Stimulating Critical Thinking in a Virtual Learning Community with Instructor Moderations and Peer Reviews

: This mixed methods study investigated the dynamic impacts of instructor moderations and peer reviews on critical thinking (CT) in a virtual learning community. Multiple data sets were collected from online discourses, participants’ written reflections, and learning artifacts, and analyzed and triangulated with both quantitative and qualitative methods. Both instructor moderations and peer reviews had great impacts on learner’s CT in multiple ways, and stimulated CT development throughout the semester. As learners grew with more CT skills, the needs for instructor moderations decreased; yet peer reviews peaked in terms of quantity, length, and depth of discussions. Peer reviews in this study also demonstrated effective questioning patterns, which were positively accepted by students being questioned or criticized, and resulted in changes and improvements in the final learning artifacts. Practical implications for online teaching and learning and community building are discussed, together with suggestions for future research. International also taught information technology and instructional planning courses at Suleyman Demirel University. He is the co-author of book chapters and scholarly articles. He is also a member of ISPI and a presenter at Association for Educational and Communication Technology, American Educational Research Association, and International Society for Performance Improvement conferences.

also taught information technology and instructional planning courses at

Introduction
It is important yet challenging to stimulate critical thinking (CT) in virtual learning communities. Instructor moderations are crucial in online learning to promote critical thinking, especially with ill-structured tasks (Salmon, 2011;Zhang & Ge, 2006  When building virtual learning communities, instructor moderations and peer interactions are particularly important. Perkins and Murphy's study on CT in online asynchronous discussions identified four major indicators of CT: clarifications, assessments, inferences, and strategies (2006). They suggested that instructors should develop strategies for online discussions and arrange weight of CT steps based on desired learning outcomes. In a content analysis of online discussions in an applied educational psychology course, Hara and colleagues found that online discussions were more of oneway than two-way interactions because the discussions were mostly for the sake of grade (Hara, Bonk, & Angeli, 2000). Even though discussions later in the semester became more continuous and engaging, overall the percentages of cognitive and metacognitive skills reflected in online discourses were very low (Hara, Bonk, & Angeli). Meyer's study on CT further extended Garrison et al (2001)'s CT model by adding a social category, which confirmed the importance of peer interactions in promoting higher order thinking in online learning (Meyer, 2003). Schrire (2006) examined knowledge building in asynchronous discussion groups, integrating Garrison et al's Practical Inquiry Model (2001), Bloom's taxonomy of educational objectives for the cognitive domain, and the SOLO taxonomy (Biggs & Collis, 1982).
Koops, Van der Vleuten, De leng, Oei, and Snoeckx (2011) investigated the impacts of formative peer feedback on students paper preparation in an online medical course, and found that peer review helped improve learners' medical knowledge and scientific reasoning. Şendağ and Odabaşı (2009) investigated how undergraduate students' CT skills and content knowledge acquisition were influenced in an online problem-based learning (PBL) environment in a pre-test and post-test control group experimental study. The results indicated that learning in online PBL did not significantly affect content knowledge acquisition. Nevertheless, students' CT skills progressed in PBL. Osborne and colleagues (Osborne, Kriese, Tobey, & Johnson, 2009) assessed the progress of students' CT skills through online discussions in an undergraduate course. Students who showed higher-level CT skills in this study (Osborne et. al.) also demonstrated understanding and appreciative behaviors, which helped building a friendly and supportive community by addressing affective and social needs of community members. Tlhapane and Simelane's study (2010) found that students in technologyenhanced PBL outperformed those in regular PBL in CT, problem solving, and social skills in a nursing education program. Teo and Chai (2009) found that online peer critiquing among pre-service teachers facilitated the professional development progress from novice to experts, and promoted multiple perspectives in problem solving.
However it was yet to be understood how instructor moderations and peer reviews may dynamically guide and scaffold the process, during which online learners grow with CT skills in virtual learning communities. This study investigated the dynamic impacts of instructor moderations and peer reviews on CT.

Research Method
The mixed methods research (Creswell, 2003(Creswell, , 2004Creswell & Clark, 2007) investigated the dynamic impacts of instructor moderations and peer reviews on CT in an online graduate course in the United States. Multiple data sets were collected through online observation, online discourse, participants' written reflections, and students' learning artifacts, and analyzed and triangulated with both quantitative and qualitative methods (Creswell; Creswell & Clark). Qualitative and quantitative analyses were conducted with multiple raters and other techniques to ensure trustworthiness (Creswell, 2003

Context, Participants, and Procedure
15 graduate students completed the study in a 4-credit online graduate course. Participants had varied professional backgrounds in K-12, higher education, military, healthcare education, and corporate in roles as teachers, trainers, instructors, administrators, media specialists, or instructional designers. The course was built to foster a collaborative, constructive, adult learning community (Conrad, 2002;Riel & Polin, 2004), with five weekly units and one two-week unit. Major learning tasks (e.g., online discussions, peer reviews, critical friends activities  aimed to promote CT skill development via knowledge mastery as well as problem solving (PS) processes. The individual research paper assignment was an ill-structured PS task throughout the semester. Open-ended, topic-specific discussions in each unit aimed to provoke CT in content-specific knowledge mastery processes. Instructor moderations were throughout the semester, and peer reviews were required on the research paper drafts. Instructor and fellow students provided feedback to members of the community in unit learning activities, and on research papers at different stages (i.e., initial ideas, proposal, outline, and draft). Students were encouraged to make revisions after each round of instructor or peer review. All participants completed a final reflection paper at the end of the semester.

Data Collection and Analyses
Various types of data were collected, including students' online discourses, courserelated email communications with the instructor, research paper proposals, outlines, drafts, final submission with revisions, peer and instructor's feedback as text or MP3 audio files, and instructor's constant reflective notes during the semester. Instructor's MP3-format feedback was transcribed to plain text. All textual data were read, discussed, categorized, coded, recoded by both researchers and analyzed in collaborative efforts of both researchers. All online discourse data were read and re-read by both researchers and then coded and recoded with open coding technique. The researchers then jointly discussed the codes, combined or deleted some of them and then recoded the documents thoroughly with an agreed list of codes, definition and examples.  Coding of the online discourses included two aspects in this study. One was to categorize content or main message in the textual data being analyzed, and the other was to identify the sequence and levels of the discourse in the communications. Thus discourses were coded with two sets of codes: letter codes for content analysis and numerical for sequence and depth identification. Letter codes identified the categories of moderations and CT indicators, together with an identifier of types of message ("P" for original messages and "R" for responses). Numerical codes reflected the sequence and depth level of the discourse. For example, a message coded as "2.a.3. ELB-R" summarized the following information: the message was a level 2 response ("R") as a third (3) one at this level, and it was categorized as elaboration (ELB). Using discourses in forms of questions and answers as an example, all of those either in original posts (P) or responses (R) were organized as follows: Deeper understandings: Later, based on Garrison et al's Practical Inquiry Model (2001), Bloom's taxonomy of cognitive learning objectives, and the SOLO taxonomy (Biggs & Collis, 1982), all discourse with CT indicators were further coded to help understand the CT skill development progress across time.

Findings
All textual discourses identified as related to CT in the initial coding processes were further analyzed. A depth of discussion (DD) value was generated for each individual participant by topic/thread, integrating the total number of messages in such thread/topic and the level of conversation. For example, if learner A posted 2 Level-1 messages, 4 Level-2, and 3 Level 4 posts, then the value of his/her DD was calculated as: (2x1) + (4x2) + (3x4) = 22. This formula reflected that each level of conversation increased the depth of the discussion in the threaded messages.
A repeated measure of ANOVA (Field, 2005) was performed for the following three variables: (a) total number of messages (N) , (b) the length of messages in words (L), and (c) the depth of discussion values (DD). Mauchly's Test of Sphericity was performed in order to ensure the sphericity assumption (Field, 2005), with the following results: for N: χ2 (5, 14) = 11.288, p > .05; for L: χ2 (5, 14) = 11.288, p > .05; and for DD: χ2 (5, 14) = 11.288, p > .05. None of them produced significant results; and the sphericity assumption was not violated for any of these variables. Thus, the withinsubjects sphericity assumed results were used for further interpretations. The results, as summarized in Table 2 showed significant changes among the units. For all the three variables, there was a statistically significant effect of weekly discussions. N and DD had moderate effect size, while L had a large effect size value. To reveal which factor may have caused such results, each unit was compared to its consecutive unit. Table 3 summaries the comparison results. The comparisons demonstrated that there was a significant difference between Unit 1 and Unit 2 in total number of messages, in favor of Unit 2. Individual's total number of CT messages increased significantly from unit 1 to unit 2 (effective size was .82), and then remained with no significant changes from unit to unit in the rest of the semester. Similar results were found for the depth of discussions as well, which had an effect size of .79, again indicating a high effect. Likewise, depth of discussions increased significantly in unit 2 compared to unit 1, and remained with no significant changes in the rest of the semester. At the same time, the length of messages increased significantly in Unit 2 compared to Unit 1 (Effect size =.89) and during peer review of paper drafts, compared to Unit 3 (Effect size = .64).
In addition, K-related sample Friedman non-parametric test was conducted to analyze learners' responding messages. The test results demonstrated that there was a significant difference among the number of responses, χ2 (5, 17) = 17.755, p < .05. Wilcoxon Signed-Rank Test indicated that In Units 2, 3, 4, 5 and Peer Review (n = 66, 66, 46, 44, and 51 respectively), students sent more responding messages than in Unit 1 (n = 16). These findings indicate that learners may have learned in earlier unit discussions, through instructor moderations; and as a result, demonstrated increased CT skills in the rest of the semester. Such open-ended, topic-specific online discussions may have prepared learners with necessary CT skills and stimulated them to apply such skills in the research paper tasks.
The following sections report more findings of qualitative and quantitative analyses to build a deeper understanding of the dynamic impacts of instructor moderations and peer reviews on CT.

How Did Instructor Moderations Stimulate CT in a Virtual Learning
Community?
Instructor moderations were mostly motivational, cognitive, and meta-cognitive throughout the semester. During the first two weeks, the needs for instructor moderation were obvious and much greater compared to the rest of the semester. The needs for instructor moderations decreased later in the semester, and did not peak even in the research paper draft review week, probably because peer reviews were sufficient and supportive. At the earlier stage, well-structured instructions served as a powerful cognitive tool, which helped learners to organize their thoughts or to position their different views. It was also noted, however, as learning progressed, later in the semester, less-structured instructions actually stimulated more critical thinking, as reflected in learners' online discourse, and documented in their final reflections. Evidently, learners CT skills developed and they became better able to apply CT in learning tasks.
In many cases, instructor's constructive feedback had led to noticeable improvements in the research progress, which were well reflected in the changes and revisions made in the research paper, and documented in their final reflections as well. For example, when the initial research ideas were shared online, 3 out of the 17 enrolled students emailed the instructor in private, expressing concerns about this major PS task. Many ideas were either vague or too general. The instructor provided timely feedback in written text as well as recorded audio responses. For example, Diane originally proposed the following:

"This research paper will be a conceptual paper that will present a snapshot of what the literature says about adult learners and how that knowledge can be applied to Distance Education (DE) instructional design."
After two rounds of review and revisions working with the instructor, she later redefined her research paper to focus on selected instructional design models to address senior learners in online learning.
As online learners mature in the learning process, they became more critical and reflective, and demonstrated more CT skills. They had significantly fewer questions for the instructor and at the same time, they provided more critical and constructive feedback for their peers in the second half of the semester, especially after the peer review activity. It was evident that learners CT improved as reflected in the learning outcome artifacts: from their initial ideas to the revised ideas and outline, then to the first complete draft, and then to the final submission. Their comments in the final reflection papers confirmed such learning benefits of instructor moderations. The qualitative findings are consistent to the previously reported quantitative results, both indicated positive changes in individual learner's CT.

How Did Peer Reviews Stimulate CT in the Virtual Learning Community?
Learners in this course were assigned with critical friends from the class, who reviewed their open-ended, topic-specific discussions and the research paper drafts. Learners had different critical friends in different learning activities so that they could get to know one another better throughout the semester. Peer reviews occurred during unit discussions and were most intense during the research paper draft review and revision process. One student consistently reviewed the work of those not assigned as critical friends in the community. Five more students reviewed or commented on peers not assigned as their critical friends in some learning activities but not all of them.
Peer reviews stimulated critical thinking in multiple ways. Peer reviews were most effective when reviewers provided detailed suggestions, offered additional resources, or raised provoking questions, or challenged a position in the paper being reviewed. Questions in peer review were either for clarifications or provoking or challenging a position. The latter ones were noticeably more in the peer review processes.
When peer reviewers asked provoking questions or challenged a position, their fellow students were more likely to respond critically, demonstrating CT skills. For example, peer review discussions generated the most provoking questions (n=51) compared to other content-specific topics (Unit 1: n =18; Unit 2: n = 33; Unit 4 : n =13; Unit 5: n =16).
Most questions for clarifications were addressed in the online discussions and resulted in minor revisions of the final paper. When peer reviewers provided additional resources, their fellow students integrated those in their final revisions by adding new references. For example, Wendy, a peer reviewer asked: "How should these online courses be designed for effective instruction?" In response, DeAnne added substantial writings in the revised paper with practical examples from University of Texas at Brownsville and Texas Southmost College nursing programs, as well as selected online learning models to support them. Wendy also questioned whether or not distance education is appropriate for nursing education. And DeAnne later in her revisions added the following to support her position:

According to the Journal of Nursing Education, in recent years the number of courses and full-degree programs offered via the World Wide Web has increased tremendously (p 364).
Universities and colleges are taking advantage of networking technology to provide flexibility for students in nursing programs. Students who have the Internet can access resources from learning institutions to pursue degrees and certifications. Distance education allows students to take courses at their own time and allows the flexibility many nursing students desire when taking courses. Distance education also improves access to education and, in doing so, counters the shortage of nurses (AACN 2000).
Distance education also helps to counter the nations mounting nursing shortage by bringing nursing careers to people who wouldn't otherwise follow that path because they lack access to a campus, or because work, family, or economic considerations preclude a full-time, on-site education (AACN 2006). PDQS: In this pattern, peer reviewers started with positive aspects of what they had reviewed, then followed by a statement of disagreement, then more provoking questions and suggestions for revisions. Q/RD: In this pattern, students used direct citations or paraphrased texts from the reviewed paper to start the conversation, followed by their thoughts and comment accordingly, as the following example: PWilS/Q: One student came up with this structure to organize her reviews, which was then modelled by others in the learning community.
 What I learned (from your paper) In their suggestions, peer reviewers were not shy to suggest adding or deleting topics or content, with strong justifications for such suggestions. Another student adopted a similar template to organize his reviews:  What surprised me  Why questions

 General comments and observations
Responses to peer reviews, as reflected in the final research paper fell into the following categories: (1) changes made only in response to instructor's comments (n=2), (2) Minor changes (n=6), (3) major changes (n=7). For example, Andrew added the references suggested by peer reviewer, Dena, and integrated them into the paper. Cambrie asked a clarification question about a term, which led Kate to revise her use of the term throughout in the paper. Cases with major changes received a big number of provoking questions, demanding more search and research efforts. These questions stimulated in-depth discussions, critical reflections and resulted in substantial revisions or changes in the final version of papers.

Discussions
This study was conducted in a course particularly focusing on promoting CT and learning community building efforts. Evidently, both instructor Cmoderations and peer reviews have stimulated and promoted critical thinking in the virtual learning community. Instructor moderations resulted in noteworthy improvements in the learning outcomes (e.g., the final research papers) and the learning processes (e.g., reflections, unit activities, etc.). Moderations addressed affective, motivational, cognitive and meta-cognitive needs to stimulate CT and facilitate community-building efforts. Learners' CT skills developed through the various activities; and as they grew with CT skills they were able to apply the higher order thinking skills in the learning process and to build the virtual learning community. Future research may look into how open-ended, content-specific discussions, which are relatively well-structured problems, may help prepare learners to handle ill structured problems, such as an independent research project. More in-depth research is necessary to investigate how to strategically schedule different types of learning activities as well as various interventions (e.g., instructor's moderations and peer reviews, etc.) to stimulate individual CT, and to promote CT among the virtual learning communities.
Peer reviews in this study showed considerable impacts on students' knowledge development (Koops et al, 2011) as reflected in content-specific discussions as well as their PS outcome, the research papers. In addition, they also helped develop metacognitive learning skills in the learning community (Livermore, 2005;Lou, Dedic & Rosenfield, 2003;Soller et al., 1999;Wu et al.). Questioning is a powerful strategy in stimulating CT. Throughout the semester, online discourses regarding content-specific topics or research papers were all dominant by questions and responses, which effectively stimulated CT and further promoted the collaborative dynamics in the virtual learning community. Similarly, Chan, Hew & Cheung (2009) found that questioning techniques resulted in extended online discussion threads. In this study, students utilized different ways or patterns to provide constructive peer reviews, including disagreements, challenging positions, suggesting additional topics, and sharing relevant resources. When challenging a position, the well-responded patterns all started with a positive statement, or a direct quotation of the reviewed draft.
Effective and friendly questioning techniques, as those demonstrated in this study, are essential in building collaborative virtual learning communities, because they not only address the cognitive demands but also provide motivational and emotional support ( Salmon, 2011). Both students and instructors would greatly benefit from effective uses of questioning techniques, especially in online learning. Thus more research is needed to further investigate specific questioning strategies and techniques as related to learner's CT and virtual learning communities.
This study was conducted in a virtual learning community, where members utilized email, blogs, podcasts, threaded discussion boards, and instant messages for communications, collaborations and community building. With more emerging technologies, especially the social networking technologies, instructors and online learners now have more media options, as well as opportunities for more dynamic, versatile, and personalized learning (e.g., . Similar moderation and peer review strategies may apply to virtual learning communities implementing more innovative technologies. However, more research is vital to guide the practice to stimulate CT and for building collaborative, virtual learning communities in the new era.