Analysis of learners’ behaviors and learning outcomes in a massive open online course

This paper introduces a massive open online course (MOOC) on educational technology, and studies the factors that may influence learners’ participation and performance in the MOOC. Students’ learning records captured in the course management system and students’ feedback collected from a questionnaire survey are explored. Regression analysis is adopted to examine the correlation among perceived learning experience, learning activities and learning outcomes; data mining is applied to optimize the 282 D. Liang et al. (2014) correlation models. The findings suggest that learners’ perceived usefulness rather than perceived ease of use of the MOOC, positively influences learners’ use of the system, and consequentially, the learning outcome. In addition, learners’ previous MOOC experience is not found to have a significant impact on their learning behavior and learning outcome in general. However, the performance of less active learners is found to be influenced by their prior MOOC experience.


Introduction
The term MOOC (Massive Open Online Course) was firstly brought up by Dave Cormier of the University of Prince Edward Island in 2008 (Mehaffy, 2012). With its rapid development, not only educators and students, but also educational researchers and the media are paying more and more attention to this field (Gillani, 2013). There have been over 8,600 items containing the word "MOOC" on Google Scholar by far, while more than 3,000 of them just came out in the year of 2013.
As a report says in New York Times "The shimmery hope (of MOOC) is that free courses can bring the best education in the world to the most remote corners of the planet, help people in their careers, and expand intellectual and personal networks" (Pappano, 2012). To realize this hope, we offered an open online course based on the on-site summer school "New Media and Learning", which was hosted in Peking University from July 15th 2013 to July 26th 2013. The online course was run on the website http://class.csiec.com, which was built based on the popular open-source CMS (Course Management System) and Moodle (Modular Object-Oriented Dynamic Learning Environment). During the summer school, 312 participants registered for the online course, while 132 of them passed all the required quizzes and got a certificate.
The course contained 16 lectures given by 15 experts in this field. Seven of them were from abroad. Before every class, references and coursewares were uploaded to the course website. During the lecture, online learners could link to the live video by a click on the course website and watch it with Windows Media Player. Afterwards, video records were uploaded as well. Additionally, homework, quizzes and course forum were provided on the same site. All these are kept accessible as the fundamental resources of an online course after the summer school. As our previous conference report (Jia et al., 2013) proves, "there is no statistically significant difference between the quiz scores of the online learners and that of the on-site learners".

Related research
After a search in Web of Knowledge, we found that most of the available papers in the field of education about MOOC were on its history (Scardilli, 2013), its profit mechanism (Dellarocas & Van Alstyne, 2013) and its technical base (Aher & Lobo, 2013;Alario-Hoyos et al., 2013). Moreover, most published MOOC application reports presented descriptive statistics that could only show basic user information, e.g. demographic materials such as gender ratio and living place, education background such as academic qualification and MOOC experience, total behavior such as registration time and certification rate, and reasons for enrolling (MOOCs@Edinburgh Group, 2013;Grainger, 2013;Ho et al., 2014). In a word, there is hardly any previous study focusing on the determinants of MOOC learners' behavior and outcome.
As a result, we turned to course management system (CMS) evaluation methodology to study the CMS-based MOOC. On one hand, survey-based model is commonly used in CMS studies (Chen, 2010;Islam, 2013). Its advantage includes but not limits to convenience and abundant theoretical support. The latest research (Islam, 2013) manifests that "perceived ease of use" and "perceived usefulness" predict the CMS usage outcome. However, it should be noticed that users' feedback does not always equal to the real case. Taking Islam's survey as an example, does a "Yes" to the question "I use Moodle frequently in this academic period" means participating large amount of learning activities? As far as we are concerned, user records are able to eliminate the subjective bias here, so that the real behavior and outcome, instead of the "perceived academic performance" could be studied.
On the other hand, data mining technology has been proved effective in CMS pedagogical research (Baker & Yacef, 2009;Bovo, Sanchez, Heguy, & Duthen, 2013). Visualization, classification, clustering, association, sequential pattern analysis, as well as other methods are adopted to discover the deeper links (Romero, Ventura, Pechenizkiy, & Baker, 2010). Thereinto, classification has been used to discover potential student groups with similar characteristics and reactions to a particular pedagogical strategy; to identify learners with low motivation and to find remedial actions to lower drop-out rates; to predict students when using intelligent tutoring systems, etc (Romero, Espejo, Zafra, Romero, & Ventura, 2013). Nonetheless, data used in existing CMS mining is confined to logs and grades (Romero, Ventura, & García, 2008), which fails to consider the influence of learners' background and perceived learning experience.
This study aims to apply both of these approaches to explore the relationship among learners' perceived learning experience, learning behaviors, and learning outcomes with MOOC.

Moodle data
Despite the rich data store, course management systems provide a limited set of reporting features and do not support data mining techniques (Psaromiligkos, Orfanidou, Kytagias, & Zafiri, 2011). Therefore, activity completion reports and grades of all online users were downloaded from Moodle into Excel-compatible format (.csv) file for further processing. Instead of detailed logs used in previous research (Romero, Ventura, & García, 2008;Zafra, Romero, & Ventura, 2010), the activity completion report were used in this study to calculate activity participated. The aim was to eliminate the possibility of double counting repeated operations in one single content, or over counting the number of online interactions such as question discussing. Forum related operations in the detailed log could sum up to a much larger amount of activities than that of opening videos and downloading materials, but within the instructional design of this open online course, videos were regarded at least as important as the online interaction. What is more, the quality of the posts in the interactions differed a lot from each other. Thus, viewing and taking part in the discussion of a single question for several times were only measured as taking participation in the course once. In a word, the measurement of participation is based on learning activity, instead of operations.
The total activity participated of every learner was then counted in Excel, which included online group meeting, question discussing, reference reading, wiki editing, quiz taking, homework uploading, courseware downloading, and videos watching. Daily signin was not taken into account because its data was consistent with that of the live video watching. The record of final courseware collection download was not adopted either, considering that learners could use the everyday saved PDF to review. As a result, a sum of 115 activities in the 12 days was taken into measurement.
Regarding the grades, the average score of quizzes and homework was deemed as a valid reflection of the learning outcome for the following reasons: (1) The lectures were given by 15 experts in this field on their latest research findings, which could be considered almost equally new to every participant. Thus no pre-test was needed.
(2) Quizzes and homework were designed by the lecturers themselves to investigate whether the key points had been mastered.
(3) There was no time limitation in these quizzes and homework while related materials were always accessible. Moreover, both open-end subjective and conceptual objective items were chosen to ensure that learners could respond freely with little pressure.

Questionnaire survey
An online questionnaire (See Appendix I) was posted on the homepage of the CMS at the end of the course. The main purpose of the survey was to collect the background and perceived learning experience of the participants, which could be used as a complement of the Moodle data in our analysis.
The questionnaire contained two parts: the demographic part and the learning experience part. Questions on gender (q6), age (q7) and educational background (q1 -q5) were involved in the demographic part, which also included MOOCs experience (q8, q9), and learning place information (q10). The second part was primarily about individual experience during the online course. As Technology Acceptance Model (TAM) (Davis, 1989) and its derivations had been widely used to investigate both e-learning adoption and continuance behavior (Al-alak & Alnawas, 2011;Juhary, 2014), TAM was taken as the theoretical framework of this part.
Nasser, Cherif, and Romanowski's (2011) questionnaire based on TAM was then adopted. Questions like "I do not have computing facilities" were replaced by more MOOC-related ones. Finally, feelings on user interface (q11), system stability (q12), operative difficulty (q13) technical and other support (q14), satisfaction of individual needs (q15) as well as internationalization (q16) were asked. Other questions in part II concerned whether references uploaded before class helped content preview (q17), whether daily sign-in encouraged attendance (q18), whether quizzes and homework led to better mastering key points (q19), whether peer evaluation increased efficiency (q20) and whether the awards promoted hardworking (q21). At last, there was an item on the overall satisfaction of the course (q22). A 5-point Likert scale was designed to measure the learners' respondent to these questions, as it was widely used in investigating the subjective assessment of MOOCs (Cross, Bayyapunedi, Ravindran, Cutrell, & Thies, 2014;Romero & Usart, 2013;Rizzardini, Gütl, Chang, & Morales, 2014). Perceived ease of use" and "perceived usefulness" had been found to be determinants of e-learning system usage in the TAM based studies. We supposed the answers to q11 -q16 and q16 -q21 could separately reflect users' "perceived ease of use" and "perceived usefulness" of the system. In addition to the Likert style ones, participants were invited to answer an open ended question on their comments and suggestions to the entire open online course (q23). This questionnaire was reviewed and amended by two experts in the Graduate School of Education in Peking University before posted online.
On the final day of the summer school, every learner was encouraged to participate in the survey. Ultimately, a total of 136 questionnaires were filled out by the online group. 105 of the respondents met the requirement to get the certificate, while the overall certification rate was 132 / 176 (75%). Registrants that did not watch any videos at all were not taken into calculation here. Person Chi-square tests indicate that the sampling bias is acceptable, as shown in Table 1. After responses were exported to Excel, the processed activity completion report and grades were integrated into the same file.

User information
Within the 136 MOOC learners who participated in the survey, 119 (87.5%) are female. This proportion is best explained by the gender distribution in the field of ET (educational technology) in China since 115 (84.6%) of the respondents major in ET. 110 (80.9%) reported themselves as graduate school student. The most typical learner is a female ET master candidate who is 27 or younger.
During the course, 40.4% learners studied at home, while another 53.7% took the online course at school. The remaining 5.9% turned to internet bar or other places. 91.9% once watched open online educational resources (e.g. MIT OCW and Netease open class) and 37% had MOOC experience before.

Reliability of the questionnaire
The scale reliability of the remaining questions is examined with Statistical Product and Service Solutions (SPSS) 17.0. Table 2 presents the naming of variables for each question. The reliability analysis result is shown in Table 3. The Cronbach's Alpha, 0.89, elucidates that the entire scale used is of acceptable reliability. Little difference in the 4th column of Table 3 indicates that there is no need to adjust questions for reliability problem.  Table 4 demonstrate that the correlation between the items is strong enough to conduct a factor analysis. With principal component analysis in extraction and varimax in rotation chosen, the final result comes out as shown in Table 5. As designed, the two components extracted can be defined as perceived ease of use and perceived usefulness. Table 5 illustrates that the ratios of different factors are proper, which guarantees the content validity of the questionnaire. Sig. .000 These two factors extracted from the post-study feedback are adopted as independent variables in the linear regression. Activity participated which reflects system use, is put into dependent variable blank. Table 6 reveals that, the coefficients of the "perceived usefulness" is positive and the result is statistically significant (p = 0.014, < 0.05), which agrees with the mentioned TAM based studies. However, "perceived ease of use" does not play a significant role in the adoption of this system as far as the Likert style questions are considered. When we look into the comments and suggestions in q23, it is noticed that severe usability problems did influence the use of the system. Here are several exemplars from respondents whose activities participated are below the average (91.1).
(1) The live video suspend from time to time because of the slow Internet, which contributes to poor effect of the class. System crashes generated negative emotions and led to my absence of some activities. Hope these could be solved next time.
(2) The temporal plan of activities lacks rationality. Feelings of the online learners are not fully considered. The video quality is low and voice is not distinct. All these could have brought about dropping. To sum up, there is a big difference between online and face-to-face learning.
(3) Often, the busy network and system crashes influence my learning results.
Indeed, since the survey did not cover learners who dropped the course halfway, it is possible that low perceived ease of use is responsible for their cease of usage. However, it can be implied from the statistical analysis that as long as the usability is acceptable, there is no causal relationship between the different perceived ease of use and the disparity of learner's participation.

Analysis of learning outcome
Effects on the two elements of the grades, regular ones and the final essay score, are examined separately. Table 7 provides the output of quiz and homework score regression, which indicates that participating online activities in open online course has a positive correlation with learning outcome. Chen's (2010) model predicted that participation was a mediator of the relationship between perceived usefulness and learning outcome. To test the mediating relationship, Baron and Kenny's (1986) approach is used, which compares the effects of mediator under test on the outcome variable controlling and without controlling the predictor. The result is depicted in Table 8. The difference in Beta indicates that participation is a complete mediator of the relationship. So far, the nexus between perceived learning experience and outcome is built, i.e., the former influences use of system, and consequentially, the outcome.
While the average score of quizzes and homework is believed to reflect the daily learning outcome, the mechanism behind the performance in final essay writing seems far more complicated. Information searching level, writing ability and knowledge base all might play a part in the score. Thus, the low coefficient of "participation" in Table 9 can be explained. Furthermore, both Table 7 and Table 8 elucidate that, introduced to the regression as dummy variables, whether MOOC is taken before has no statistically significant impact on the behavior and learning outcome of open online course learners as a whole. Table 10 demonstrates that perceived usefulness and ease of use both positively influence learners' satisfaction. It can be inferred that although perceived ease of use does not immediately give rise to more active participation in the short-term online open course, their satisfaction might encourage usage of a similar system in the future, according to Seddon's model (Chen, 2010).

Further data mining
In order to verify the aforesaid conclusion and to optimize the model, data mining process is conducted. Since clustering is mostly used in grouping students or tests into related groups for individualized teaching and pedagogy adjusting (Vellido, Castro, & Nebot, 2010), its practical value to short-term open online course remains doubtful. That is because there is hardly any opportunity or obligation for a teacher to instruct the learners after the open online course. Thus, "classify" and "visualize" in Weka (2013) are chosen as approaches.
Weka is an open-source software platform that provides a collection of machine learning and data mining algorithms for data pre-processing, classification, clustering, association rules, and visualization (García, Romero, Ventura, de Castro & Calders, 2010). It supports best known classification algorithms like ID3 and C4.5 (Hämäläinen & Vinni, 2010). Hence, the data is explored with Weka 3.7.10, the newest version.

Classification
Two nominal attribute, quiz_pass(0, 1) and essay_pass(0, 1), are created to represent (1) whether a learner's average score of quizzes and homework passed 80 and (2) whether the final essay was submitted. These two conditions were required for the learners to get the certificate of the summer school. The naming of the other attributes is the same as that in the basic SPSS analysis. We adopt trees-J48 as the classifier, which is often used in e-learning data mining (Romero, Ventura, & García, 2008). J48 is realization of the C4.5 algorithm in Weka, including efficient pruning (Weka, 2013). Quiz_pass is firstly selected as grouping variables, with default test options and parameters. The outputs are demonstrated in Table. 11 and Fig. 1.
According to Table.11, the reliability of this classification is 80.88%, which is acceptable. The decision tree in Fig. 1 lends supports to some of the conclusions mentioned above and provides supplementary information to the model: (1) Participating activity positively affects the overall score of quizzes and homework. Nearly all of the learners who took part in more than 77 activities got a score over the required 80.
(2) Although there is no statistically significant relationship between the MOOC experience and the participation of all the learners as a whole, MOOC experience might play a role in influencing the performance of less active students who participated less than 77 activities. The fact that only one of the ten such students passed can be explained as experienced MOOC learner has clearer needs and expectations, which could lead to higher halfway dropping rate.
(3) Perceived usefulness, especially feelings on q20, to which extent peer evaluation increased efficiency and q19 to which extent quiz promoted mastering, not only improves the score indirectly by promoting participation, but also has direct bearing with the overall performance. This is consistent with Islam's (2013) conclusion on ordinary online course management system.

Visualization
To illustrate the correlation among the main attributes in Fig. 1, a scatter plot is chosen in the matrix of "visualize". As shown in Fig.2, x-axis represents participation, while y-axis represents quiz_score. Different colors are used to represent perceived usefulness of quiz at different levels, i.e., blue for 1, brown for 3 and orange for 5. Besides, two auxiliary lines are added manually, to indicate the threshold value of participation and the cut-off score.

Conclusion and limitation
With analysis of questionnaire feedbacks and Moodle data in a medium open online course, some of the relationship between perceived learning expereince, learning behavior, and learning outcome has been found as the following.
Firstly, the perceived usefulness of an open online course positively influences use of its system, and consequentially, the learner's outcome. Accordingly, as a practical implication of this research, we find it essential to attach more importance to the dissemination of the course, not merely for increasing the registrants. It might considerably lead to better learning outcome of the users. More specifically, not only introduction to the teaching form of the MOOC should be provided online as we did last summer (http://ei.pku.edu.cn/summer2013), but also the usefulness of every lecture ought to be emphasized during the enrollment and between classes. Besides, since MOOC experience is becoming more and more common among the learners, it could be helpful that individual needs are inquired before the course. By adjusting teaching contents and methods according to the needs, we can keep more learners with MOOC experience active, so as to improve their overall performance.
Secondly, as long as the usability is acceptable, there is no causal relationship between the different perceived ease of use and the disparity of learner's use of the system during the course. Short-term MOOC disagrees with common long-range elearning at this point. However, the stability of the CMS and the quality of the videos are suggested to be improved by quite a few users. We hope the dropping rate be lowered and the satisfaction be increased in the next summer school, which requires enhancing the robustness of the whole system. Hence, the usability of the system in the large concurrent processing environment will be one of our top concerns afterwards.
Thirdly, whether MOOC has taken before has no statistically significant impact on the behavior and outcome of open online course learners. However, MOOC experience does influence the performance of the learners that have taken part in only some of the activities.
Admittedly, though we have verified some correlation, the mechanism behind the effect of perceived usefulness in open online course has not been studied yet. Furthermore, due to paper limitation, the analysis fails to consider the entire educational background of learners and other factors, so that the outcomes of MOOC learners cannot be fully predicted so far.
Our future research will try to discover more learning mechanism of MOOC users with bigger data and more reliable survey. For example, users' learning styles will be taken into consideration in questionnaire design. What is more, we are going to apply text mining to the analysis of cooperative learning and inquiry learning in the forum of the online course system, which might reveal the detailed pattern of open online course study.