Design of a computer-based learning environment to support diagnostic problem solving towards expertise development

Learning through problem solving is a pedagogical approach that situates learning in problem-solving contexts. As a form of constructivist learning, problem solving has received increased attention in complex and illstructured domains such as scientific inquiry and medical education. However, effective learning in problem-solving contexts is difficult to realize because problem-solving tasks often involve complex processes that are inaccessible to learners. It is important to make such complex processes visible for observation and practice, and provide learners with necessary help during the learning process. This study explored the design of a computer-based learning environment that helps medical students to externalize the sophisticated process of diagnostic problem-solving and provides them with adaptive feedback when they work with a number of simulated clinical cases. The proposed approach attempted to utilize expert knowledge to transform open-ended problemsolving experience into systematic and deliberate effort towards expertise development.

Biographical notes: Dr. Bei Yuan has just completed her PhD study at the Faculty of Education of the University of Hong Kong.She is now conducting her post-doctoral study at the KM&EL Lab of the University of Hong Kong.Her research interests include medical education, diagnostic problem solving, computer-assisted learning, and design-based research.
Dr. Minhong (Maggie) Wang is an Associate Professor in the Faculty of Education of the University of Hong Kong, and Director of the KM&EL Lab.She was a visiting scholar at Harvard Graduate School of Education, University of Cambridge, and MIT Sloan School of Management.She has been involved in multiple disciplinary research in the areas of technology-enhanced learning, complex problem solving, knowledge visualization, medical education, STEM education, adult learning, and artificial intelligence.Her research has focused on the theory-driven design and analysis of technology-enhanced learning in higher education, workplace, and school settings.Her research aims to provide learners with necessary support to deal with challenging problems in complex situations and enable them to achieve high levels of autonomy, confidence, and performance.She is the Editor-in-Chief of Knowledge Management & E-Learning, and Associate Editor of Information & Management.More details can be found at http://web3.edu.hku.hk/magwang/.
Andre W. Kushniruk is Director and Professor in Health Informatics at the School of Health Information Science at the University of Victoria, Canada.He is a fellow of the American College of Medical Informatics.Dr. Kushniruk conducts research in a number of areas including usability engineering, humancomputer interaction in healthcare and cognitive studies of end users of health information systems.He has been involved in some of the first work in providing patients with Internet access to their own health information (as stored in hospital-wide electronic health records) and he continues to work with designers and evaluators on a wide range of health information systems.

Introduction
Problem solving is a common activity in which people engage daily, and quite often it involves ill-structured or complex problems (Jonassen, 1997).Educators and researchers have paid close attention to the development of students' abilities to solve real-world problems, particularly in such complex domains as scientific inquiry and medical education.Problem-solving experience can improve learning by engaging learners in active thinking and flexible knowledge construction (Hmelo-Silver, 2004), and it is also recognized as an integral part of expertise development (Ericsson, 2008;Schmidt, Norman, & Boshuizen, 1990).
Given the constrain of classroom settings in offering learning with real-world problems, computer-based environments have been increasingly explored to support such learning in virtual environments.In spite of technology support, effective learning through problem-solving is difficult to realize because learning in such contexts involves complex processes that are inaccessible to learners (Wang, Kirschner, & Bridges, 2016).A problem-solving task often includes sophisticated processes such as exploration with information on multiple aspects, integration of problem information with domain knowledge, and reasoning with interactive components, which cannot be easily captured and mastered by students (Wang, Wu, Kinshuk, Chen, & Spector, 2013).The complexity of such processes may generate heavy cognitive load to learners, making it difficult for them to develop adequate understanding of the experience and achieve desired learning outcomes.On the other hand, the complexity of a problem-solving task is often underestimated by instructors or experts for whom many of the requisite processes have become largely automatic or subconscious because of their experience.
Given that open-ended exploration with problem-solving tasks may overburden learners (Kirschner, Sweller, & Clark, 2006), the provision of scaffolding or support to learners has been widely recognized as an important part of learning in such situations (Hmelo-Silver, Duncan, & Chinn, 2007).At the same time, research on expertise development has shown that desired learning outcomes in problem-solving contexts cannot be achieved by a mere accumulation of experience, but require systematic and deliberate effort with expert help (Ericsson, 2008;Jarodzka, Scheiter, Gerjets, & van Gog, 2010).It is important to make complex problem-solving processes visible for observation and practice, and provide learners with necessary help throughout the learning process.
In medical education, problem-based learning has been widely adopted over the past decades, and has been found to be effective in improving learners' reasoning and communication skills, fostering their abilities to cope with uncertainty, and empowering self-directed learning, although the effects of problem-based learning on learners' knowledge structure or systemic understanding of basic science has not been found to be superior to those of traditional methods (Hartling, Spooner, Tjosvold, & Oswald, 2010;Neville, 2009;Koh, Khoo, Wong, & Koh, 2008;Dochy, Segers, Van den Bossche, & Gijbels, 2003;Albanese & Mitchell, 1993).It is also noted that the implementation of problem-based learning has been considerably dependent on the instructor's personal experience, with limited studies on the design and implementation of problem-based curricula and learning environments to address the challenges of learning in such contexts (Henry, Tawfik, Jonassen, Winholtz, & Khanna, 2012).
Recent research has highlighted the importance of making thinking visible to students when they work with complex clinical problems (Delany & Golding, 2014;Bridges, Corbet, & Chan, 2015).A prior study of our research lab examined the design and effectiveness of a computer-based cognitive mapping approach that helps students to externalize the complex reasoning process as well as the knowledge underlying the reasoning process when working with clinical cases (Wang et al., 2013;Wu & Wang, 2012;Wu, Wang, Grotzer, Liu, & Johnson, 2016).In addition to making sophisticated processes accessible to learners, there is a need for investigation into how learners can be further supported by providing them with necessary support and feedback when they perform complex problem-solving tasks.
This study explored the design of a computer-based learning environment that helps medical students to externalize the sophisticated process of diagnostic problemsolving and provides adaptive feedback to students when they work with a number of simulated clinical cases.Glaucoma diagnosis was chosen as the learning subject because it is a part of the learning content of general courses in medical schools, and is considered to involve ill-structured problem solving.The learning cases in the proposed system were adapted from authentic clinical cases and academic references.Relevant expert knowledge was collected and utilized for generation of adaptive feedback to learners throughout their learning process.The clinical tasks and problem-solving contexts were designed in ways similar to realistic clinical encounters in that learners are given incomplete information of a problem and need to collect further information by selecting clinical examinations and making intermediate judgments in several rounds.A pilot evaluation was conducted with medical students to collect their initial feedback and comments on the proposed learning system, in order to refine the system before analyzing its effects on learning performance.

Theoretical foundations
Relevant learning theories and instructional strategies were integrated into the design of the proposed learning environment.Learning in problem-solving contexts is supported by situated cognition theory (Brown, Collins, & Duguid, 1989) and situated learning theory (Lave & Wenger, 1991).The two theories share a common view that situation and cognition are interdependent; cognition is a process occurring in physical and social contexts where knowledge is created and applied.Learning through problem solving positions learners in real-world problem contexts, helping them to develop critical thinking and problem-solving skills as well as consolidate and extend content knowledge.
The use of scaffolding to support learning with complex problems is aligned with the cognitive apprenticeship model, which was established on the basis of situated learning theory.According to the cognitive apprenticeship model, carrying out a complex task usually involves implicit processes; it is critical to make such processes visible for novices to observe and practice with expert help (Collins, Brown, & Holum, 1991).The cognitive apprenticeship model emphasizes that learning in problem contexts should consider situating abstract tasks in authentic contexts, making complex tasks and thinking processes visible, and providing necessary help to learners.
Making complex tasks and thinking processes visible can be linked with modelcentered learning and instruction, i.e., the use of mental models to uncover the cognitive processes and architectures to gain insight into the nature of complex problem solving (Greca & Moreira, 2000;Seel, 2003).A mental model is "what people really have in their heads and what guides their use of things" (Norman, 1983).Effective learning in problem-solving contexts requires the externalization of the implicit mental models that are associated with sequences of actions and the underlying knowledge in complex problem-solving processes (Bradley, Paul, & Seeman, 2006).Model-centered learning and instruction has two different modes-self-guided mode and expert mode (Seel, 2003).In the self-guided mode, students are expected to develop their own mental models with little support or guidance; it is more suitable to well-structured problem-solving, or when students have profound knowledge and experience in a given domain.In the expert mode, experts' mental models are externalized as support and guide to help students to solve complex problems or accomplish learning tasks; it is more suitable to ill-structured problem-solving, or when students have limited prior knowledge and experience.

Design of the learning environment
Based on the above theoretical foundations, Glaucoma Diagnosis & e-Learning System, a computer-based learning environment to support students' learning and expertise in glaucoma diagnosis was developed by including the following functions.

Exploratory problem-solving context
The exploratory problem-solving context allows students to work with a number of simulated diagnostic problems.The context was designed to be similar to real-world practice in that learners are given incomplete information of a problem and need to collect further information in a number of steps to solve the problem.
After selecting a case in the system, the student can view a primary description of the patient's background information (e.g., age, gender, and medical history) and chief complaint.Moreover, the student can conduct clinical examinations for the patient to obtain further information, as shown in Fig. 1.For each selected clinical examination, the student can view the examination results (in the form of laboratory data, images, and brief inspection reports), and make intermediate judgments based on the results.Diagnostic conclusion can be made to complete the problem-solving process.

Visualization of the problem-solving process
The diagnostic problem-solving process performed by a learner for each case can be captured as a diagnostic record and presented as a flowchart by the system (see Fig. 2).A diagnostic flowchart consists of initial information about the patient, one or more clinical examinations, intermediate judgments made according to the examination results, and a diagnostic conclusion.By capturing the diagnostic process and representing it in a visual format, the complex and tacit problem-solving process becomes accessible for review and reflection by students.

Expert support
After the completion of a case, the system will provide adaptive feedback to the student by comparing his/her problem-solving performance to the expert performance.The student can practice with the same case and receive feedback for more than once.For each diagnosis, once the degree of similarity in the diagnostic process between the student and the expert reaches 60% or more, the expert's diagnostic process will be presented to the learner for comparison and reflection, as shown in Fig. 3.Moreover, the expert's summary of the case, indicating the key points and common errors in diagnosing the case can be viewed by the student for further reflection.

Construction of knowledge in mental models
The system also includes a graphing tool, which allows students to build up a mental model to externalize the subject-matter knowledge underlying the reasoning and diagnostic processes with the cases.Students are encouraged to integrate the underlying knowledge across multiple cases in a systematic structure when building the mental model.

Methods
A pilot evaluation was conducted with medical students to collect their initial feedback and comments on the developed learning system.Twelve Year-3 students from a public medical college participated in the evaluation on a fully voluntary basis.The participants had basic knowledge and skills needed for clinical reasoning and diagnosis.
At the beginning of the experiment, students were given a face-to-face, one-hour demonstration of how to use the system for diagnostic problem-solving and reflective learning with clinical cases.A sample case was then used for practice by students under guidance.After getting familiar with the system, students started their individual learning by working with other five simulated cases within four weeks.During the individual study period, there was no teacher involvement except for assistance with technical problems.Students were prompted by the teacher to engage actively in the learning tasks.
The process of learning with the proposed system is elaborated in the following.First, the student starts the learning task by selecting a case from the system.A primary description of the patient and his/her chief complaint is then presented to the learner, including gender, age, and medical history of the patient, and symptoms such as "pain in the eye" and "blurred vision." Second, based on the initial information of the case, the student forms an initial assessment of the case and selects a sequence of clinical examinations to obtain more information of the case.Based on the examination results, the learner makes intermediate judgments and if needed selects more examinations for further exploration.After obtaining adequate information via several rounds of examination and judgment, the student makes a diagnostic conclusion for the case.
Third, after submitting the diagnostic conclusion, the student can review his/her diagnostic process captured by the system and presented in a flowchart.The system will provide feedback to the student about his/her problem-solving performance on the case (e.g., "your process includes some unnecessary examinations", "less than 60% of your diagnostic process is the same with that of the expert").The student can practice with the same case and receive individual feedback for more than once.
Fourth, once the student's performance reaches the specified degree of similarity with that of the expert, the diagnostic process of the expert can be viewed by the student.Moreover, the expert's general comments on and summative solution of the case is presented to the students to indicate some key points and common errors in diagnosing the case.
Fifth, the student needs to complete five cases in the system; during the learning process, the student is required to review and summarize the knowledge underlying these cases by drawing a mental map to represent the knowledge for reasoning and diagnosis with these cases.
At the end of the study, students were asked to participate in an open-ended questions survey to describe the strength and weakness of the learning environment and offer suggestions for improvements.

Results
The response from each of the twelve students was analyzed by capturing its main themes.The themes shared by the participants were identified and summarized in the following.
In terms of strengths of the proposed learning environment, the participants found the learning system useful and attractive, and reported their strong intention to use the system.More specially, they reported that the online learning system provided them with more flexibility and convenience in learning compared to traditional classroom settings; the visualization-based learning system helped them to observe and reflect on the complex problem-solving process; the learning program helped them to improve the diagnostic problem-solving capabilities, especially in dealing with problems of incomplete information; and the learning system enabled them to reflect on and organize the knowledge underlying the problem-solving practice in a systematic way.One participant mentioned, "The flowchart makes a complex process looks simple and clear."Another one noted, "From this study, I learnt a new approach to learning, i.e., using the mental map to organize knowledge from different cases, which helped me think more effectively." The participants also mentioned the weaknesses the learning environment and provided relevant suggestions for improvement.First of all, they mentioned that more learning cases and more learning resources, particularly multi-media learning materials, could be added to the system.Second, more detailed demonstration and descriptions were needed to elaborate how to use the system to perform the learning tasks.One student reported, "The learning resources, including the learning cases, were not adequate.In addition, other types of learning materials, such as audio-visual materials and videotaped lectures could be provided."Last but not least, more expert support could be provided to help students to complete the diagnostic tasks.For example, the expert's solution could be displayed after the student had tried the case many times but still unable to reach the specified degree of similarity with the expert solution.

Conclusion
In complex problem-solving contexts, it is critical to externalize the sophisticated process of solving an ill-structured problem and provide learners with necessary support.This study has explored a computer-based learning environment that allows learners to externalize and reflect on their problem-solving process in visual formats and helps them to identify the gap between their performance and that of the expert when working with a number of simulated clinical cases.The proposed approach attempted to utilize expert knowledge to transform open-ended problem-solving experience into systematic and deliberate effort towards expertise development.
The initial evaluation results have shown the potential of the proposed approach, which was well perceived by learners.Based on the feedback and suggestions from the participants, an improvement will be made on the developed learning environment and learning program.Further evaluation and analysis on the effectiveness of the proposed approach will be performed and reported in future work.

Fig. 3 .
Fig. 3. Expert support He holds undergraduate degrees in Psychology and Biology from Brock University, as well as an MSc in Computer Science from McMaster University and a PhD in Cognitive Psychology from McGill University.