Leveraging the Use of Mobile Applications to Increase Knowledge Retention in a Classroom Lecture

This research sought to determine if the use of mobile applications (e.g., iPhone® apps) had an impact on students’ ability to learn new material. A control group was compared against a group of students who used mobile devices during a statistics lecture. Students participated separately in a lecture followed by a period of either pencil and paper only or technology-assisted examples. They then took a quiz over the material. The data collected shows that the app group outperformed the control group on every question and scored 16% higher overall. A post-experimental survey found that participants in the app group felt strongly that mobile applications helped them understand the new concepts more clearly and were more confident in their ability to quickly learn this new material than the control group. Overall, this research demonstrates that technology-assisted learning positively impacts students’ learning. It also suggests that technology is changing the way people think and learn.


INTRODUCTION Background
Engineering is a challenging field. Arguably a large part of that challenge comes from the way technical material is presented to students. Research has suggested that the majority of engineering students are sensing, visual, active, and sequential learners (Felder, & Spurlin, 2005). Richard Felder also noted in his previous research that students were more motivated to learn when they were engaged in active learning and inductive guided inquiry (Felder, & Brent, 2003). This poses the question as to why so many courses are structured as direct instruction lectures in which professors verbally explain material to students with unremarkable visual aids. Using Felder's data as a guideline, there has been much research in the field of technology-assisted learning. Many of these studies have found that there is no significant difference between performance of comparison groups using traditional lecture or technology-assisted learning (i.e. the educational effectiveness was similar) (Holman, 2000;Merino, & Abel, 2003). Gee (2003) found that video games are highly motivating and Rapeepisarn et al. (2008) discussed that educational games have many elements that directly benefit learning.
The concept of "apps" has only been around since Apple ® launched the App Store℠ in July 2008. As such, there is limited research on the educational effectiveness of mobile applications (e.g., iPhone and iPad ® apps). However, as mobile apps can be presented in much the same way as other technologies, one could assume their effects on learning would be similar. This, in part, motivated the current study.
Prior to the launch of the Apple App Store, it was already noted that people were widely using mobile devices such as smartphones and PDAs to engage in "informal learning" while on the go (Clough, Jones, McAndrew, & Scanlon, 2008). There are now over 350,000 apps available on the Apple App Store, thousands of which are geared toward "every subject and every stage of learning" (Apple.com). Since the inception of this new app phenomenon, researchers have noticed a trend in the public's desire to learn through mobile devices (Jeng, Wu, Huang, Tan, & Yang, 2010). Coupling the recent research on technology-assisted learning with the vast user acceptance of mobile applications and the recent push from universities toward experiential learning, one can see the potential for educational apps to one day become a part of everyday instruction. Furthermore, apps were chosen over other technology-assisted learning options (e.g., laptop computers, SMART Board ® ) because of their increasing popularity among college students as well as their "on-the-go" functionality that does not require users to be at a computer.
Based on these trends and the fact that this study is somewhat of a "new frontier" for engineering education, this research focuses on the educational effectiveness of mobile applications. After discussion with subject matter experts, it was determined that engineering students who struggle to grasp foundational concepts such as statistics tend to have more difficulty in higher level courses that rely on students' abilities to analyze data and draw conclusions based on observations. This led to the motivation of this research.

Literature
By understanding how students learn, it becomes increasingly easier to target instruction methods that more effectively teach new subject material or challenging concepts. Since it has been shown that students learn better while they are actively engaged (Felder, & Brent, 2003), this research used mobile applications as an active learning component to gauge their effectiveness as a teaching tool.
Theories of Learning Styles. A fundamental gap exists between many student learning styles and types of instruction within engineering education. Mismatches such as these are not only detrimental to student performance but can also serve to demotivate potential engineers from completing a technical degree program. Learning modalities include visual (pictures, diagrams, charts), verbal (sound, words), and kinesthetic (touch, taste, smell) (Felder, & Silverman, 1988). Carl Jung (1971) asserts that most college engineering education tends to favor intuitors (i.e. those that prefer abstract theories) over sensors (i.e. those who prefer applications). A critical mismatch exists since most people of college age and older are visual learners and most engineering education is taught by lecture (Waldheim, 1987). As Waldheim noted, courses that utilize both text-based modules (handouts, textbooks) and visual aids (pictures, diagrams, flow charts) will be more effective at teaching technical concepts to all students, regardless of their learning preference. By the same logic, incorporating mobile apps into the curriculum will enable an instructor to more effectively encompass all three learning modalities as well as both sensors and intuitors.
Technology-assisted learning. Perhaps inspired in part by Felder and Brent's findings that people learn better while engaged, there has been much research on the educational effectiveness of technology as a form of active learning. Holman (2000) found that post-test scores of students taught by computer tutorial were not significantly different from those of students taught by traditional classroom instruction (i.e. both methods were equally effective). A study by Merino and Abel (2003) had the same result but also found that these tutorials seemed to supplement learning when paired with classroom instruction. A pre-lab computer-assisted video shown in another study was demonstrated to effectively raise performance of all students performing hands-on chemistry concepts (McNaught et al., 1995). Gee (2003) noted that the motivating effect of video games teaches people to think critically and solve problems in a complex environment. A more recent study that evaluated the use of an educational game in a mechanical engineering course found that students spent approximately twice the amount of time on their coursework outside of class and showed a deeper understanding of presented concepts as compared with students in a traditional lecture setting (Coller, & Scott, 2009).
Bloom's Taxonomy. Since its inception, Benjamin Bloom's Taxonomy of Educational Objectives has served as a framework for designing curricula based on varying depths of comprehension (Bloom, 1956). Its revised levels are: Remember, Understand, Apply, Analyze, Evaluate, and Create (Anderson & Krathwohl, 2001). Many institutions find this taxonomy especially helpful with outlining course requirements for new faculty or with developing new courses, particularly in the rapidly changing field of computer science (Starr, Manaris, & Stalvey, 2008).
For a mobile application to be an effective teaching tool, it must provide content in such a way that users can understand the relationship between presented material and user interactions allowed by a given application. A key to further use of educational applications is identifying which characteristic(s) of the mobile application contribute to which level(s) of Bloom's Taxonomy.
The higher-order thinking skills of Bloom's Taxonomy (analyze, evaluate, and create) were omitted in this study because the mobile application used in this experiment is designed specifically to provide real-time feedback to user inputs under a set of pre-selected conditions (e.g., a usersupplied mean and standard deviation). It is meant as a tool to interpret, conceptualize, and approximate statistical information based on user inputs and does not present new concepts or the relationships between them.

METHODOLOGY Experimental Design
The researcher's hypothesis was that the use of mobile applications during a lecture would result in higher quiz scores than the absence of mobile applications. The sole independent variable manipulated in this study, lecture type, consisted of two levels: without app (control group) and with app (app group). The two dependent variables used to measure the educational effectiveness of mobile applications were comprehension quiz score (compared overall and by each question) and post-experiment survey responses. A t-test was used to evaluate the difference in mean quiz scores between the control group and the app group. A Wilcoxon Rank Sum (U) test was used to analyze the difference in means of percentage correct by question.

Participants
This study included 15 males and 11 females with an average age of 20.27 that were fortuitously sampled from a population of local college students and randomly split into two equal groups of 13 (n 1 = n 2 = 13). All twenty-six participants met the qualifying criteria of no prior college coursework in statistics. It is important to note that the app group was split into two subgroups due to time conflicts but a t-test found no significant difference between the subgroups at an α level of 0.05 (i.e. they can be compared against the control group as a single group of n = 13).

Equipment and Material
Mobile device. Each member of the app group was shown how to solve examples using one of 30 iPod touch ® mobile devices, loaned by Apple as part of their seeding program. The instructor used an iPad 2 to project these examples on a screen.
Mobile application. The app used in this study was LearnStatistics, a publicly available mobile application developed by GO2STAT LLC that is designed to provide synchronous (i.e. real-time) and interactive (i.e. bi-directional between user and app) feedback to the user (Nickerson, Muntermann, Varshney, & Isaac, 2009).
After entering parameters (e.g., mean and standard deviation), LearnStatistics displays a distribution as shown in Figure 1. A moveable slidebar allows users to visualize how the probability and shaded area change in real time as truncation points (tails) are adjusted.
By understanding how the characteristics of LearnStatistics map to levels of Bloom's Taxonomy, educators will be better equipped to effectively incorporate this type of technology into the classroom. Table 1 shows a mapping of the characteristics of this app.  Comprehension Quiz. A standardized comprehension quiz was used to assess knowledge retention. Each question was mapped to at least one of the three lower levels of Bloom's Taxonomy (remember, understand, and apply; Anderson & Krathwohl, 2001).
Questionnaire and Assessment. A 7-point Likert scale (i.e. strongly disagree = 1, neutral = 4, strongly agree = 7) was used to gauge participants' comfort level with lecture pace, confidence in answers, perceived usefulness of LearnStatistics (app group only), and likelihood of future mobile application use. An Index of Learning Styles (ILS) assessment was used in attempt to discover any differences in performance between types of learning styles (Felder, & Spurlin, 2005).

Procedure
A control group simulating a standard lecture course and an app group simulating a course using active learning/inductive lecturing concepts (each containing 13 participants) were examined separately. The experimental procedure for both groups was identical with the exception of the absence/presence of mobile applications during the lecture. First, an overview of the study was given and all participants signed an informed consent form.
Lecture and examples. A lecture on the normal distribution was chosen to simulate a statistics course because it is a relatively simple concept with multiple levels of depth. An instructor gave a pre-prepared 60-minute lecture about the normal distribution followed by a 15-minute pen and paper statistics problem example period (e.g., probability calculations and z-score transformations). Note: The same instructor led each group for the same amount of time. In the app group, each participant followed along with the same examples using LearnStatistics on an iPod touch as the instructor projected the app in real time on an iPad 2.
Comprehension quiz. At the conclusion of the lecture and example period, a standardized comprehension quiz was administered to participants to gauge their ability to learn this new material. There was no time limit to the quiz. The iPod touch devices were collected from the app group prior to the quiz as the focus of this experiment was on their ability to learn with apps, not test with them.
Post-experiment. Upon individual completion of the quiz, participants filled out a post-experiment survey followed by an index of learning styles assessment.
Grading scale. In an effort to gauge the amount that participants learned about the normal distribution, a three option grading scale was used. Based on their answer, a score of 1 (completely correct), 0.5 (correct statistics concepts but minor algebra error), or a 0 (completely incorrect or blank) for each question was assigned by a single grader. Their final quiz average was determined by the total points earned over the total points possible (i.e. 14). These scores were rounded to the nearest whole percent.

Overall Quiz Score
The overall mean quiz score in terms of percent correct of the control group was 0.58 with standard deviation of 0.22. The overall mean quiz score of the app group was 0.74 with a standard deviation of 0.17. This suggests the app group performed 16% better than the control group. Sorted from highest to lowest within each group, a visual representation of these quiz scores in Figure 2 shows an obvious trend. Note that 12 of the 13 control group participants provided usable scores; one participant left multiple quiz questions blank and was discarded from analysis. After validating the normality, independence, and equal variance assumptions using multiple residual plots, a twosample t-test was used to compare mean overall quiz scores at an α level of 0.05. After comparing the computed |t 0 | test statistic of 2.135 against the t 0.05, 23 critical value of 2.069 (i.e. 2.135 > 2.069), the null hypothesis was rejected. The data was shown to have a statistically significant difference between mean overall quiz scores of the control group and the app group. Using operating characteristic (OC) curves, the power of this test was estimated as 0.25 or 25% (Natrella, 1963). This relatively low value is most likely due to the limited sample size in this experiment. The use of 0.05 for α also resulted in a smaller power. The researcher considered using α = 0.1 to compensate for a smaller sample size but felt that any results produced this way would not be considered valid based on the prevalent use of α = 0.05 in research (Montgomery, 2005).

Quiz Scores by Question
Given that the app group scored higher than the control group on each question and the answers followed the same trend in both groups (i.e. there was a tendency for each group to miss the same questions) as shown in Figure 3, a questionby-question test for significance was performed. A plot of residuals revealed that a t-test was unusable because all questions violated the normality assumption. As a result, a Wilcoxon Rank Sum test was used as a nonparametric alternative to compare the means of individual questions between the two groups at an α level of 0.05. A comparison of each individual question's U value (i.e. test statistic) against the critical value of 41 revealed that although the app group scored higher on every question than the control group, none of differences were statistically significant (Montgomery, 2005).

Survey Responses
Both groups rated that they were not very familiar with the material prior to the study but felt that the instructor was effective at explaining the concepts and felt that they understood his examples. The app group rated their confidence in learning the material quickly higher than the control. 12 out of 13 students in the app group felt that mobile applications helped them understand the concepts more clearly. The comparison of survey responses again necessitated the use of the Wilcoxon Rank Sum test due to violation of the normality assumption. No questions were found to be significant at an α level of 0.05 against the U critical value of 45.

Index of Learning Styles Assessment
Although the limited sample size in this study did not yield enough data to make a detailed comparison of learning styles, both groups appeared to show trends of 1) balanced numbers of learners within active-reflective dimension, 2) sequential-global dimension, 3) more sensing than intuitive learners, and 4) more visual than auditory learners.

DISCUSSION
It was theorized that an instructor would be able to reach both visual and verbal learners simultaneously by manipulating statistical distributions on an app using a projector. The use of technology by the app group presented active learners with the opportunity to follow along on their own app while allowing reflective learners the chance to internalize these changing conditions as they were happening. By giving them the opportunity to experiment with different outcomes using an app that provides real-time feedback, students were able to more quickly understand concepts and draw relationships.
Overall, this study showed that the app group outperformed the control group on the comprehension quiz by 16%. This was expected since mobile applications are a form of active learning and previous studies in active learning have shown similar trends (Felder, & Brent, 2003). An obvious trend of higher scores on each question was present and both groups followed the same relative trend of correct/incorrect responses. However, no quiz questions had a significant difference between groups at an α level of 0.05. Although no single question had a significantly higher mean percentage correct, the visible trend of higher performance by the app group on all questions obviously contributed to the significant difference in overall mean quiz scores. A larger sample size would increase the power of this test and perhaps yield significant question-by-question results.
Based on the overall trend of percentage correct by question and the fact the each question was mapped to one of the three lower levels of Bloom's Taxonomy (Remember, Understand, and Apply), it can be inferred that the use of mobile applications positively impacts students' abilities to learn up to these levels of complexity (although the results are not statistically significant for any one question). Further research is needed to fully examine the effects of mobile applications on levels of Bloom's Taxonomy. The postexperiment survey showed that 12 out of 13 participants in the app group felt strongly that the LearnStatistics app contributed to their learning. This feeling of confidence in learning correlates with their increased quiz performance. This suggests that the members of the app group were more motivated to learn. The fact that students attributed their increased performance to mobile application use further supports previous conclusions that students learn best while actively engaged.
Adding new technologies such as mobile applications into classroom lectures has many benefits to education. Better conceptual understanding leads to better performance on assignments and exams. These higher scores and better overall performance lead to increased confidence in learning ability with new and difficult material. This confidence may in turn contribute to increased student motivation leading to higher retention and graduation rates among students. Increased motivation has the potential to lead to more interest in pursuing higher education and possibly careers in scientific and technological fields.
What is it about mobile applications in particular that contribute to learning, as opposed to other forms of active learning, or technology-assisted learning? An app user's preexisting relationship with apps and the mobile devices on which they reside allow apps to be used as a tool, on hand at any time for today's student or working professional. The intrinsic portability of these devices allows an app user immediate access to data, when and where questions arise. Mobile applications are interactive and deliver results without the need for a web browser or a textbook. In essence, they appeal to today's generation of learners because of their portability, ease of use, and interaction leading to immediate results.

LESSONS LEARNED AND FUTURE STUDIES
Robust evidence of learning was another point in which this research was limited. This study restricted participants to those that had not taken a college statistics course. The assumption was all participants would score poorly on a pretest given their lack of coursework in the subject area; therefore this step was omitted. However, a comparison of pretest/posttest scores between the two groups would have perhaps been a better metric to evaluate the educational effectiveness of mobile applications than overall quiz score. By incorporating this method into future studies, baseline performance could be established and each participant could be assessed directly against this baseline. This may more clearly indicate the amount of learning for each participant and eliminate any uncertainty about subject matter familiarity.
This research is meant as an inquiry into the overall effectiveness of current educational practices as well as a suggestion for continuous improvement. Though it has perhaps raised more questions than it has answered, it has laid the groundwork for future studies on this emerging concept of apps in education. With the advent of new technologies, educational systems (like everything else) must adapt to changing conditions rapidly in order to remain successful.