How to advance our understanding of flipped learning : Directions and a descriptive framework for future research

With the growing number of research articles published on flipped learning, many aspects of this instructional approach have been well researched and thoughtfully discussed. At this point, how can future research advance our understanding of flipped learning? This article sheds light on three possible directions for future studies of this instructional approach, including (1) longitudinal studies, (2) examining its effects on different learning objectives, and (3) incorporating gamification into flipped courses. A descriptive framework for flipped classroom interventions is then proposed, comprising of four dimensions: (1) research background, (2) course design, (3) course activities, and (4) outcome of interventions. By applying this descriptive framework, the knowledge created in future research can be well-documented, disseminated, used, and evaluated by other practitioners and researchers. The present work can provide a foundation for further efforts to research flipped learning.


Introduction
Educators and researchers have increasingly recognized the importance of self-regulated learning and student-centered learning (Lai & Hwang, 2016).Thus, the flipped (or inverted) classroom approach has gained growing attention in the education sector (Chen, Lui, & Martinell, 2017;Karabulut-Ilgu, Jaramillo Cherrez, & Jahren, 2018;Lo, Hew, & Chen, 2017).In a typical flipped classroom, students self-regulate their learning by completing pre-class learning tasks, such as watching instructional videos or doing online exercises.This shift from direct lecturing frees up more class time for student-centered activities such as individualized instruction or collaborative learning tasks (Bishop & Verleger, 2013;Giannakos, Krogstie, & Chrisochoides, 2014;Lo et al., 2017).Hence, pre-class learning and in-class learning are two major components of a flipped classroom (Abeysekera & Dawson, 2015;Bishop & Verleger, 2013).

The increasing trend of flipped classroom research
The growing popularity of flipped learning has been accompanied by a rising number of published research articles.Few years ago, Abeysekera and Dawson (2015) conducted a search of the ERIC database for flipped classroom studies.Only eight articles were found in the database at the time of their search (June 2013).Recently, we performed a similar search using the search string "flipped class*" OR "flipped learn*" OR "flipped course*" OR "inverted class*" OR "inverted learn*" OR "inverted course*" in April 2018 (the time of finalizing this manuscript).We were able to find more than 500 documents in the ERIC database, including reports, academic journals, ERIC documents, dissertations, and books (Fig. 1).Perhaps the use of asterisk which served as a wildcard could increase the flexibility of our search string, so that more flipped classroom studies with different terminology (e.g., flipped learning, flipped class, and flipped classes) could be retrieved.Many of the search outcomes are indeed published in recent years.To further investigate the publication trend of flipped classroom research, the Web of Science was searched using the same search string.Fig. 2 shows the exponential increase in the number of documents published on flipped learning in the past few years.Consistent with the observation of some other researchers (e.g., Chen et al., 2017;Giannakos et al., 2014;Lin & Hwang, 2018b;Karabulut-Ilgu et al., 2018), the body of literature on flipped learning has expanded rapidly since 2012.From 2000 to 2017, a total of 1,852 documents were found in the Web of Science database, including proceedings papers (n = 1,065), journal articles (n = 629), meeting abstracts (n = 87), and other miscellaneous items (n = 76) such as editorial materials, reviews, book reviews, letters, news items, and corrections (Note: the numbers do not add up because one document might belong to more than one category).It is therefore questionable to claim a lack of flipped classroom research without a cautious study of these documents.

The aim and organization of the article
With this body of literature, many aspects of the flipped classroom approach should have been researched and thoughtfully discussed (Lin & Hwang, 2018a).The aim of this article is thus to identify points of departure for further research of this instructional approach.Leveraging the effort of existing reviews, this article begins with a brief overview of the findings of flipped classroom studies.Then, several possible directions of future investigations are discussed.To better document flipped classroom interventions, a descriptive framework is proposed for future research.Such a descriptive framework can ensure the knowledge created in flipped classroom studies can be well-documented, disseminated, used, and evaluated by other practitioners and researchers.

State of the literature
As a first step toward understanding the previous research of flipped learning, we can examine the review studies.Hitherto, more than 15 reviews have been published on flipped learning.The reviews by Bishop and Verleger (2013) and Giannakos et al. (2014) surveyed some of the early work on flipped learning.More recent reviews have usually focused on the flipped classroom approach as applied to particular subject disciplines such as chemistry (Seery, 2015), engineering (Karabulut-Ilgu et al., 2018;Kerr, 2015), mathematics (Lo et al., 2017), history (Lo, 2017), nursing (Betihavas, Bridgman, Kornhaber, & Cross, 2016;Presti, 2016), and medical education (Chen et al., 2017;Hew & Lo, 2018;Lin & Hwang, 2018b;Ramnanan & Pound 2017).In addition to these discipline-based reviews, several researchers have confined the scope of their reviews to particular contexts, such as K-12 education (Lo & Hew, 2017), higher education (O'Flaherty & Phillips, 2015), and Asian universities (Chua & Lateef, 2014).Besides, Voronina, Moroz, Sudarikov, Rakhimzhanova, and Muratbakeev (2017) and Zuber (2016) have specifically selected experimental studies of flipped learning for their reviews.These various existing reviews and empirical studies can enrich our understanding of the benefits, challenges, and student-learning outcomes of this instructional approach.

Benefits and challenges of the pre-class learning component
Some of the benefits and challenges of flipped learning have been widely reported.For example, carefully designed pre-class learning tasks can enable students to better engage in effective self-paced learning (Giannakos et al., 2014;Ramnanan & Pound, 2017).With prepared instructional videos, the students can pause and replay the course materials to gain better understanding (Karabulut-Ilgu et al., 2018;O'Flaherty & Phillips, 2015).If an online follow-up exercise is provided, the students can immediately practice what they have learned, and receive instant computerized feedback on their learning (Lo et al., 2017;Seery, 2015).In addition to enabling knowledge application and self-evaluation, pre-class exercises can provide the instructors with results that help to inform their design of inclass activities (Lo, 2017;Seery, 2015).In this way, the instructors can tailor-make their upcoming lessons in response to the students' pre-class efforts.For example, the instructors may focus on discussing common mistakes, or they may skip some basic materials that the students have already mastered (Lo et al., 2017;Seery, 2015).Some instructors, however, have reported negative experiences from flipped learning.First, some students do not complete the pre-class learning tasks (Lo, 2017;Lo et al., 2017).Second, the students often perceive an increased workload after flipping (Betihavas et al., 2016;Lo & Hew, 2017).In fact, it may be unrealistic to expect that every student can manage to learn all of the basic and advanced materials independently through video lecturing (Lo et al., 2017;Presti, 2016).Such a demand on students can also place a burden on the instructors to create flipped learning materials, such as instructional videos (Giannakos et al., 2014;Karabulut-Ilgu et al., 2018).It is therefore often suggested that only a reasonable amount of course materials be offloaded to preclass learning (Karabulut-Ilgu et al., 2018;Lo et al., 2017).Moreover, integrating selfregulated learning strategies into flipped learning is recommended to help students make their pre-class learning plans (Lai & Hwang, 2016).

Benefits and challenges of the in-class learning component
Concerning the in-class learning, the flipped classroom approach can enable instructors to provide their students with more individualized guidance, because of the shift from direct lecturing to coaching and problem-solving (Karabulut-Ilgu et al., 2018;O'Flaherty & Phillips, 2015).In addition to the increased opportunity for instructor-student interactions, more class time can be spent on small-group activities (Bishop & Verleger, 2013;Ramnanan & Pound, 2017).Peer instruction, cooperative learning, and collaborative learning are some of the well-established peer-assisted learning approaches that have been used in flipped courses (Giannakos et al., 2014;Lo et al., 2017).The benefits of peer-assisted learning have been thoroughly discussed, particularly from the Vygotskian perspective (Bishop & Verleger, 2013).Furthermore, various strategies for mobile technology-supported flipped learning have been proposed (Hwang, Lai, & Wang, 2015).Leveraging education technologies such as mobile devices and wireless networks can be used inside the classroom to foster the students' competencies in communication, collaboration, critical thinking, complex problem-solving, and creativity (see Hwang et al., 2015 for a review).
Despite these benefits, however, not all students can adjust to the interactive nature of a flipped learning environment (Betihavas et al., 2016;Giannakos et al., 2014).A few qualitative findings have even suggested that some students are unable to enjoy flipped learning, and they prefer receiving lectures inside the classroom (Lo et al., 2017;Seery, 2015).To avoid frustrating their students with dramatic changes to the mode of instruction, instructors can start by carefully explaining and discussing the rationales, benefits, and expectations involved in flipped learning (Lo & Hew, 2017;Lo et al., 2017).

Student learning outcomes
In many cases, flipped learning has been shown to be more effective than the traditional lecture-based approach (Chen et al., 2017;Giannakos et al., 2014;Lo & Hew, 2017;Karabulut-Ilgu et al., 2018;Kerr, 2015).A few researchers have attempted to quantify the overall effects of flipped learning compared with those of traditional lecture-based learning.Lo et al. (2017) conducted a meta-analysis of 21 studies on flipped and traditional mathematics classrooms.They found that the students in the flipped mathematics classrooms generally outperformed those in the traditional counterparts with a small but significantly positive effect size.In health professions education, Hew and Lo (2018) also found a significant effect in favor of flipped learning in their meta-analysis of 28 traditional-flipped comparison studies.Furthermore, both of these two studies revealed that the effect of flipped learning could further be promoted when instructors provided a quiz on pre-class materials at the start of face-to-face lessons.
Notwithstanding the positive results in previous research, some evidence has suggested that flipped learning may produce no improvement, or in some cases may even impair student learning (see Betihavas et al., 2016;Chen et al., 2017;Karabulut-Ilgu et al., 2018 for a review).Researchers might attribute the failure to the lack of students' self-directed learning awareness.For example, one instructor of Yang's (2017) flipped classroom lamented that "students in our school are not that learning motivated… some of them just don't care.They just don't care" (p. 7).The prompt is that the flipped classroom approach ought to be successfully implemented.Instructors' flipped classroom management skills and learning design competences may determine the efficacy of flipped learning.Therefore, instructors should learn how to manage and develop their flipped courses.Also, they can apply some established design principles (e.g., Lo et al., 2017) or ground their flipped classroom design in some existing frameworks, such as online community-based flipped classroom (Lin & Hwang, 2018a) and mobile technology-enhanced flipped classroom (e.g., Hwang et al., 2015).By doing so, the instructors can have the competences to guide their students becoming a self-directed learner.
When evaluating the overall effects of flipped learning, researchers (e.g., Chen et al., 2017;Chua & Lateef, 2014;Karabulut-Ilgu et al., 2018;Lo et al., 2017) have encountered significant challenges, because quite a few existing studies have provided insufficient data.Karabulut-Ilgu et al. (2018), for example, stated that although they reviewed many comparison studies, the majority of them did not report adequate data (e.g., numbers of student participants, mean scores of tests, or standard deviations) for a proper meta-analysis.In addition, Bishop and Verleger (2013) and Lo et al. (2017) found that some previous studies have not clearly reported the designs of the flipped classrooms concerned.Such omissions have made it difficult to conduct further analyses of flipped learning, such as the effects of different class activities (Bishop & Verleger, 2013;Lo et al., 2017).Perhaps the overall quality of their reviewed studies was not high, because they included the research papers from a wide-range of sources (e.g., conference proceedings, non-SSCI journals) in their synthesis.Developing a descriptive framework would be useful to guide researchers to report their flipped classroom interventions.

Further efforts to research flipped learning
From this brief survey of the current literature regarding flipped learning, two implications can be drawn for future research.First, our understanding of flipped learning cannot be improved if we merely replicate the previous studies and reconfirm the aforementioned findings.Second, further analyses (e.g., meta-analysis) will not be possible if the student learning data and flipped classroom designs are reported in merely summarized forms.Therefore, several directions and a descriptive framework are proposed for future research on flipped learning.

Directions for future flipped classroom research
Some researchers propose directions for future flipped classroom research.In medical education, for example, Lin and Hwang (2018b) suggested examining students' preparation degree and cognitive loading in pre-class learning.They hypothesized that pre-class loading (e.g., quantity and difficulty of pre-class materials) may affect their class preparation degree and the effectiveness of in-class learning.Further research on this aspect can advance our knowledge of how to improve our design of pre-class learning activities.Furthermore, they proposed offering flipped courses for the general public and patients.Such a use of the flipped classroom approach outside regular education contexts (e.g., K-12 and higher education) is quite a new insight and worth researching (see Lin & Hwang, 2018b for a review).
Besides, the following three types of studies offer possible directions for future research on flipped learning:

•
Longitudinal studies of flipped learning.

•
Examining the effects of flipped learning in reaching different learning objectives.
• Incorporating gamification into the flipped classroom approach.

What are the effects of flipped learning in the long run?
Previously, most studies on flipped learning have been conducted within a time span of one semester.We should exercise caution in drawing conclusions from such short-term studies, because it is possible that the students may tend to be unusually attentive when new media are first introduced (Clark, 1983).The novelty effect may thus become a confounding variable, resulting in a merely short-term boost to student performance and perception (Cheung & Slavin, 2013;Gravetter & Forzano, 2012).In their high school science classroom, Leo and Puzio (2016) examined the effect of flipped learning by flipping two chapters of their course.They were able to provide evidence that flipped learning promoted their students' levels of achievement compared to traditional lecturing.However, their study was somewhat limited by the short duration of intervention.It appears that their students were full of excitement.They asserted that the flipped classroom approach "was actually kinda cool" (p.778) and chanted "flipped classroom, flipped classroom" (p.778) during class.We therefore need to ask, what is the effect of flipped learning after its novelty disappears?
In fact, some flipped classroom instructors encountered the mid-semester slump (Bolton, 2003) in students' enthusiasm for learning.According to Bolton (2003), the midsemester slump is a phenomenon that students begin to lose their interest of learning and become indifferent or stagnate around the midpoint of a semester.As Webb and Doman (2016) reported, their "students had in fact begun to slow down and took less interest in completing the grammar assignments around the midpoint of the semester (especially weeks 7 and 8)" (p.56).This observation echoes the study of Scott, Green, and Etheridge (2016).In their 14-week flipped calculus classroom, the instructor of Scott et al. (2016) lamented that "At the end of the semester students were slacking off on preparing before class.They were running out of steam" (p.261).Longitudinal studies with a longer duration (e.g., 1 year) are thus required to better reveal the actual impact of this instructional approach.

What are the effects of flipped learning in reaching different learning objectives?
Although many studies have compared the overall scores of traditional and flipped classes, we currently know very little about the effects of flipped learning for reaching different kinds of learning objectives.Only a few studies have examined student performance in dealing with specific types of learning problems, and this approach deserves further in-depth investigation.Some examples that we found included the studies by Harrison, Saito, Markee, and Herzog (2017) and Kennedy, Beaudrie, Ernst, and St. Laurent (2015).
• Harrison et al. (2017) examined the near-and far-transfer of learning in flipped engineering education.The near-and far-transfer problems assessed students' lower-and higher-order thinking skills, respectively.However, no significant impact was found on these two thinking skills because of flipping.Kennedy et al. (2015) evaluated students' computational and conceptual knowledge in their flipped calculus course.The computational problems required students to use particular formulas, while the conceptual problems required them to recognize how certain formulas were suited to specific types of problems.Except the outperformance of their traditional class in the conceptual portion of Exam I, the students in their traditional and flipped classes scored similarly in the computational and conceptual portions of all five examinations.

•
In these two studies (i.e., Harrison et al., 2017;Kennedy et al., 2015), the differences in learning found in comparisons between traditional and flipped classes were generally non-significant across different learning objectives.These findings contradicted the proponents' belief that the flipped classroom approach would facilitate student learning of higher-order objectives (Betihavas et al., 2016;Giannakos et al., 2014;Ramnanan & Pound, 2017).Nevertheless, this conclusion was inconclusive, as it was based on only two studies.Further research is required to determine whether flipped learning is particularly effective for achieving certain types of learning objectives.

How can gamification be incorporated into the flipped classroom approach?
Gamification has already been widely used in educational contexts, with game elements such as badges, points, and leaderboards being commonly applied (Dichev & Dicheva, 2017).Gamification is a potential way to enhance student motivation in flipped learning (Lo & Hew, 2017;Lo et al., 2017).Remarkably, however, relatively few studies have been published that examine the use of game elements in flipped learning environments (see Dichev & Dicheva, 2017 for a review).We therefore know little about the benefits and challenges of incorporating gamification into this instructional approach.
Recently, several researchers report experiences of gamifying their flipped courses.For example, Yildirim (2017) gamified his flipped course of Teaching Principles and Methods with the help of a learning management system (Moodle).Various game elements were integrated into the teaching, learning, and assessment activities, such as points, badges, levels, experience prints, medals, and leaderboards.Taking medals as an example, Yildirim's (2017) students could earn medals if they spent extra time on course materials.Besides, Hung (2017) gamified in-class activities for his flipped English course.A game-like clicker application (Kahoot!) provided instant feedback on the students' task performances in the form of digital points awarded.Sound effects were also applied to create a game-like atmosphere.In both of these studies (i.e., Hung, 2017;Yildirim, 2017), the comparisons between gamified and non-gamified flipped classrooms showed that the integration of game elements could enhance the students' learning achievements and their perceptions regarding flipped learning.Although some positive evidence has been found regarding the effects of gamifying flipped courses, this evidence will remain inconclusive until more studies are published in this area.

Descriptive framework for flipped classroom interventions
Rigorously reported studies can advance our understanding of flipped learning.To improve the clarity of reporting on flipped classroom interventions, a descriptive framework (Fig. 3) with four dimensions is developed.This framework considers the factors of (1) research background, (2) course design, (3) course activities, and (4) outcome of interventions.
This four-dimensional framework is intended to define a minimal set of information and findings to be reported in future studies on flipped learning.By applying this framework, the researchers can provide a more complete picture of the practical details of the teaching methods applied (e.g., the ways of allocating pre-class and in-class course materials) for other practitioners to follow or to enrich.This framework can also help to ensure that new studies record the specific kinds of data that are necessary for other researchers to evaluate and conduct further analyses of the findings (e.g., metaanalyses).

Research background
First, the following information about research background can be provided:

•
Initial equivalence of comparison groups (if any).

•
Instructor equivalence in comparison groups (if any).
Such a description allows other readers to understand the background of a study.Readers can thus determine whether the practices and experiences reported are applicable to their own course by viewing the course title and description.
For comparison studies, the numbers of student participants in each group, their initial equivalence, and instructor equivalence are some important information that enables a meta-analysis.As a side note, rigorously designed research should use a pretest to establish initial equivalence of research groups instead of merely relying on the comparison of their pre-intervention GPA.It is possible that two classes are similar in their overall GPA but not equivalent in some specific knowledge and abilities.For example, Wong, Ip, Lopes, and Rajagopalan (2014) found that the students in the flipped class scored significantly higher than those in the traditional class in their pharmacy examination.However, when referring to the demographic characteristics of their research groups, their flipped class had a significantly higher pre-intervention pharmacy GPA than the traditional class (p = 0.04) despite the similarity between the two classes in term of their overall GPA (p = 0.32).Better prior pharmacy knowledge might become a confounding factor that resulted in such an outperformance of the flipped class (Wong et al., 2014).Therefore, both pre-and post-course assessments should be used and considered to evaluate the effect of an intervention.Otherwise, it is impossible to know whether the control and experimental groups are comparable at the start of the intervention (Cheung & Slavin, 2013).If a significant difference is found among groups, appropriated statistical tests (e.g., ANCOVA) must be used to control for this difference.

Course design
Second, we suggest the course design be clearly described.In particular, the following two aspects can vary a lot across studies:

•
Intensity of flipping (e.g., estimated percentage of course content flipped).

•
Course planning (e.g., allocations of pre-class and in-class course materials).
Different instructors may flip their course in a different manner.For example, Peterson (2016) delivered a 45-minute video lecture before class and no class time was spent on direct lecturing in his flipped statistics course.In contrast, students in the flipped calculus course of Scott et al. (2016) were only required to watch a 20-minute pre-class video.Less than one-third of their class time was spent on instructor's mini-lessons and demonstrations of solutions.As the instructor of Scott et al. (2016) commented, "some topics were easy to learn through video while others were too complex for students to understand" (p.262).So how do flipped classroom instructors allocate pre-class and inclass course materials?And what is the efficacy of their course design?To enable such an analysis across studies, researchers should declare the intensity of flipping and their way of course planning when writing up their flipped classroom study.

Course activities
Third, we suggest future studies describe their flipped learning activities portion by portion (i.e., pre-class, in-class, and post-class).For pre-class learning activities, (1) video lectures, (2) online/offline exercises, (3) text-based materials, and (4) out-of-class discussions are commonly used in flipped courses.However, there are wide variations in the way flipped classroom instructors offer these activities.To improve the clarity, the following details can be described:

•
Online/offline exercises: amounts, types of questions (e.g., multiple-choice questions, fill-in-the-blank questions), types of problems (e.g., computational or conceptual problems, near-or far-transfer problems).
For the in-class learning activities, (1) reviews of pre-class materials, (2) quizzes on pre-class materials, (3) mini-lectures, (4) individual learning tasks, and (5) small-group learning activities are commonly used in flipped courses.The following details can be described:

•
Reviews of pre-class materials: durations, depth of revisions.

•
Individual learning tasks: durations, amounts, types of problems (e.g., computational or conceptual problems, near-or far-transfer problems), amounts.
• Small-group learning activities: durations, amounts, types of problems (e.g., computational or conceptual problems, near-or far-transfer problems), specific models or approaches used (e.g., peer instruction, think-pair-share).
Post-class learning activities are not often provided in current practices of flipped classrooms (see Lo & Hew, 2017;Lo et al., 2017;Ramnanan & Pound, 2017;Seery, 2015 for a review) because the workload of post-class together with pre-class learning activities may overwhelm students.Nevertheless, post-class online/offline exercises (e.g., Clark, Kaw, & Besterfield-Sacre, 2016) and reflections (e.g., Lai & Hwang, 2016) were provided in some flipped courses.The following aspects can be described:

•
Online/offline exercises: amounts, types of questions (e.g., multiple-choice questions, fill-in-the-blank questions), types of problems (e.g., computational or conceptual problems, near-or far-transfer problems).

Outcome
Fourth, researchers should report in detail the outcome of interventions, such as (1) student achievement, (2) student perceptions, and (3) instructor perceptions.
• Student achievement: mean (or median) scores of tests, standard deviations, statistics test data.
Depending on the focus of research, not all of these three aspects must be examined and discussed.Also, the outcome of interventions is not limited to these three aspects.The most important issue is to ensure rigor in data analysis.For example, appropriate statistics test (e.g., t-test, Mann-Whitney U test) must be applied to compare the mean (or median) scores among groups.

Conclusions
Some aspects of flipped learning have already been well discussed in the literature.Future research should explore areas that have not been thoroughly addressed.In this article, three possible directions are discussed: (1) the inclusion of longitudinal studies, (2) investigations on the effects of flipped learning for reaching different learning objectives, and (3) experiments incorporating gamification into the flipped classroom approach.To

Fig. 1 .
Fig. 1.The search results of flipped classroom documents in the ERIC database as of April 2018

Fig. 2 .
Fig. 2. The number of flipped classroom documents published from 2000 to 2017 in the Web of Science database as of April 2018