Qualitative and quantitative analysis of the students’ perceptions to the use of 3D electronic models in problem- based learning

Faculty of Dentistry of the University of Hong Kong has introduced innovative blended problem-based learning (PBL) with the aid of 3D electronic models (e-models) to Bachelor of Dental Surgery (BDS) curriculum. Statistical results of preand post-semester questionnaire surveys illustrated compatibility of e-models in PBL settings. The students’ importance ratings of two objectives “Complete assigned tasks on time” and “Active listener”, and twenty-two facilitator evaluation items including critical thinking and group problemsolving skills had increased significantly. The students’ PBL preparation behavior, attentions to problem understanding, problem analysis, and learning resource quality were also found to be related to online support of e-models and its software. Qualitative analysis of open-ended questions with visual text Knowledge Management & E-Learning, 9(2), 128–142 129 analytic software “Leximancer” improved validity of statistical results. Using e-model functions in treatment planning, problem analysis and giving instructions provided a method of informative communication. Therefore, it is critical for the faculty to continuously provide facilitator training and quality online e-model resources to the students.

analytic software "Leximancer" improved validity of statistical results.Using e-model functions in treatment planning, problem analysis and giving instructions provided a method of informative communication.Therefore, it is critical for the faculty to continuously provide facilitator training and quality online e-model resources to the students.

Introduction
3D electronic models (e-models) is a diagnostic tool used in modern dentistry as a type of patient clinical record with better maintenance, retrieval and transferability than traditional plaster casts (Redmond, 2001).3D study e-models have been adopted into dental education thanks to the evolutionary 3D image scanning and computer-aided design (CAD) technology (Joffe, 2004).Faculty of Dentistry of the University of Hong Kong has introduced virtual resources and online support of 3D e-models into blended problem-based learning (PBL) for curriculum of Bachelor of Dental Surgery (BDS) (Yang, Zhang, & Bridges, 2012).This new method has replaced the use of traditional plaster study models in PBL to assist the students in treatment planning (Whetten, Williamson, Heo, Varnhagen, & Major, 2006) and to help them adapt better to the fastchanging modern dental technology.
Different from the passive learning environment of traditional didactic way of teaching, PBL has reconstructed the learning environment into an active and studentcentered one by utilising facilitated small group discussion and problem solving (Hmelo-Silver, 2004).It enhances individual student's understanding of teamwork (Carlisle & Ibbotson, 2005), critical thinking ability (Tiwari, Lai, So, & Yuen, 2006), problemsolving skills, self-directed learning skills, transferable skills, as well as retention of knowledge and skills (Norman & Schmidt, 1992).The role of teaching staff in PBL has changed from a didactic lecturer to an interaction-oriented facilitator, influencing the students' performance (Van Berkel & Schmidt, 2000) by providing collaborative construction of knowledge, guidance and assistance in group function (Ling & Loy-Pang, 2007).Various assessments in the process of teaching are hence critical to help the students to adapt to educational setting changes, and link their performance to specific learning outcomes (Fincham & Shuler, 2001).Student self-assessment aims to clarify PBL curriculum expectations and specific educational objectives for the students, so that they can integrate course content into learning process more actively and responsively.In addition to evaluating student's learning progress and promoting understanding of facilitator's role, facilitator-assessment serves as feedback for future course adjustment and development of better facilitation (Stassen, Doherty, & Poe, 2001).
While implementing e-models, facilitators also concern about how to effectively use their pedagogical beliefs and facilitation skills (Haith-Cooper, 2000) to maintain proper group function, which might be interrupted by both introduction and demonstration of e-models and its software.Understanding compatibility of e-models and expectations of PBL is therefore important for future curriculum design which is based on student experience rather than financial constraint (Winning & Townsend, 2007) and resource needs (Azer, 2001).The aim of the study is to analyse the compatibility of emodels in PBL settings and the students' perception of e-models.The questionnaire used in this study was modified from current questionnaires used in Faculty of Dentistry, the University of Hong Kong for student self-assessment and facilitator-assessment in PBL.They showed the faculty's expectations in PBL teaching strategies.The student's understanding of the faculty's expectations in creating the mutual expectation (Kolmos, Du, Holgaard, & Jensen, 2008) was measured from their importance ratings of the items in the questionnaire.For testing the compatibility of e-models in PBL, the consistency of the mutual expectations to the PBL teaching strategies should be shown in both the preand post-questionnaire surveys before and after implementing e-models in PBL.Also, one open-ended question was included in the questionnaire to measure the students' perception of e-models in PBL.

Methodology
The study was approved by the Institutional Review Board of the University of Hong Kong .The e-models used in PBL were supported by the software O3DM® (Kriel, 2012) which was uploaded to online learning system, allowing the students to download freely with permission from the O3DM®Company in the entire PBL session.E-models served as one of the inquiry materials in PBL to provide the patients' intraoral information for the case studies.The facilitators had to guide the students' discussions by asking the trigger questions and assisting the students in the proper use of the inquire materials.Fifty fourth-year BDS students (divided into six groups of nine students) were invited to complete the pre-and post-test questionnaire about their perception of using e-models and their expectations of PBL.Pre-test questionnaire data was collected before the students' first PBL tutorial to use e-models, while follow-up post-test survey using exactly the same questionnaire was conducted at the end of the last PBL tutorial of using e-models.The total length of the PBL sessions that the students in this study engaged in was two weeks.
The evaluation questionnaire used in this study was composed of four parts.The first part consisted of questions related to the students' personal background and the time they spent for PBL preparation.16 questions in the second part were focused on learning purposes of PBL, i.e. "Responsibility", "Proper knowledge base", "Well in reasoning" and "Communication well" and 12 specific objectives of these goals in the form of selfassessment.While the third part consisted of 35 questions for facilitator assessment, focusing on 7 goals of PBL teaching, i.e. "Display understanding of the role being a PBL facilitator", "Promote group problem solving", "Promote appropriate group function", "Promoted effective evaluation", "Facilitate your learning", "Promote your critical thinking", and "Promote your learning".Visual analogue scales (VAS) were used for questions in the second and third part of the questionnaire to quantify the students' answers.A 100-mm horizontal line with two anchor points at two extremes stating "Not important" and "Very important" was used to measure the student's continuous response to the question.The students were asked to make a vertical line along the horizontal line at the place that best represented their perceived importance of the goal or objective.1 mm in length of the horizontal line stands for 0.1 score of value.The final score ranging from 0 to 10.0 was then obtained by measuring the horizontal distance from the anchor point "Not important" to the interception of the lines.This method provided more responsive, reliable and valid measurement without any subjective or statistical weighting on rating elements (De Boer et al., 2004).The last part was an open-ended question which asks about the students' perception of using e-models.The inclusion of openended qualitative question in the questionnaire was aimed at improving validity of statistical results and quality of suggestions to curriculum designers (Noesgaard & Ørngreen, 2015).
Two sets of data and feedback collected from the questionnaire surveys were then further analysed.Quantitative analyses were carried out for data in the first three parts of the questionnaire.Means and standard deviations of the question scores were calculated.T-tests were run to analyze changes after using e-models in PBL.Regression was run to find out association between students' change of expectations and their learning experience in PBL after implementing e-models.All statistical analyses were performed using SPSS, version 23.0 (IBM Corp., Armonk, NY, USA).All statistical tests were twotailed and the level of statistical significance was set at 0.05.Furthermore, visual text analytic software "Leximancer" was used for qualitative analysis of feedback in the last part of the questionnaire."Leximancer" is a powerful text analytic software in thematic analysis of the text content by analysing word occurrence and co-occurrence statistics (Smith & Humphreys, 2006).This provided a more systematic way to understand the students' perceptions with an objective catalog of themes mentioned in the feedbacks, for the benefit of optimization of strategies with enhanced conceptual understanding of text data (Cretchley, Gallois, Chenery, & Smith, 2010).

The students' personal background
All fifty fourth-year BDS students had completed and returned the pre-and post-test questionnaires with 100% response rate.Mean age of the students was 23.34 years old.21 of them (42%) are male students and 29 (58%) are female students (see Table 1).66% students entered BDS through Joint University Programmes Admissions System (JUPAS) and 28% of them came from subsystem Early Admission Scheme of JUPAS.Most of the students (62%) came from local secondary schools in Hong Kong.The students' weekly hours of preparation for PBL in total and on Internet increased from 5.63 to 7.08 and 2.45 to 2.78 respectively after using e-models (see Table 2).The students' self-reported total time for PBL preparation before and after using e-models in PBL showed statistical significance in t-test (p = 0.044 < 0.05).

The students' importance ratings on the expectations in the student selfassessment of PBL
Mean score of the students' importance ratings in self-assessment of PBL before and after using e-models was 6.84 and 7.19 respectively.All questions about goals (see Fig. 1) and related objectives (see Fig. 2) in this part had greater importance rating in post-test data compared to pre-test data.Two of the objectives, i.e. "Complete assigned tasks on time" (p = 0.031 < 0.05) and "Active listener" (p = 0.015 < 0.05) showed statistical significance in t-tests.In addition, two of the goals, i.e. "Well in reasoning" and "Communicate well", and four of the objectives showed linear correlation between their importance ratings and the students' PBL preparation time on Internet, according to regression analysis in post-test data.

The students' importance ratings on the expectations in the facilitatorassessment of PBL
Mean score of the students' importance ratings in facilitator-assessment of PBL before and after using e-models was 6.41 and 7.11 respectively.All questions about goals (see Fig. 3) and related objectives (see Fig. 4) in this part showed increased importance ratings in the post-test data.Five out of seven goals, i.e. "Promote group problem solving" (p = 0.010 < 0.05), "Promote appropriate group function" (p = 0.045 < 0.05), "Facilitate your learning" (p = 0.037 < 0.05), "Promote your critical thinking" (p = 0.015 < 0.05) and "Promote your learning" (p = 0.013 < 0.05), and seventeen out of twenty-eight objectives demonstrated statistical significance in t-tests.In regression analysis, objective "Ask you to evaluate the quality of learning resources used" showed linear correlation between its importance rating and the students' PBL preparation time on Internet after using emodels in PBL.

The students' perceptions of using e-models
Eight themes were extracted from the content of the students' feedback in the last part of questionnaire.After taking into consideration concepts presented in the themes, related examples from the concept map (see Fig. 5) and tables of Leximancers' result, the extracted themes were then named "Comparing with solid plaster casts", "The use in lab stages", "Patient feeling", "Computational support", "Cost", "Quality of models", "Preparation of models", and "Treatment planning".The relationships of different themes were also shown in the concept map.Table 3 lists and summarizes the students' perceptions of the themes accordingly. E-models are convenient to be kept.
 E-models have no hygienic problem but solid plaster casts would be fractured, mottled, deteriorated, worn, broken, lost or be messy in a long-term storage. A good resolution monitor is required for the display of e-models.
 It is better to have the use of both e-models and plaster casts.
Theme 2: The use in lab stages  E-models are transferred to lab technicians or other dentists on referral more easily and quickly in the instant image format. Quicker communication to lab is achieved without waiting on plaster cast production and transference.
 Better communication is provided when team approach is needed.
 E-models aid in design and manufacture of plaster casts  Extra scanning procedure is needed in order to create images of e-models.
 Extra computational skills of software like O3DM or CAD-CAM are required for the lab stages such as waxing of crowns and design of prosthesis, if not actual plaster casts are still needed for lab work. Dental lab may not support the system of e-models.
Theme 3: Patient feeling  E-models are convenient and easy for the presentation facilitating the communication with patients.
 Preparation of e-models is more tolerable to patients in tooth scanning without pouring for dental impression. E-models give "Cyber" and "high tech" conception to patients.
Theme 4: Computational support  Easier measurement and calculation of spacing are supported by the software of e-models.
 The computer generated analysis results of e-models required the support of computers and software. Large memory storage space required for the records of e-models.
 Maintenance of computers and anti-viral software are required for patient records in preventing data loss and protecting data privacy.
Theme 5: Cost  The cost of plaster casts production and transference is lower.
 Scanning of the teeth of patients and turning it into digital data are expensive.
 The cost of installation and running of the supporting software of e-models is expensive.
Theme 6: Quality of models  E-models show more accurate details.
 E-models show better appearance than actual plaster casts.
Theme 7: Preparation of models  Easy copy function of e-models is convenient for dentists to compare the multiple models.
 E-models are environmental friendly.
 Mass production of plaster casts or production of prosthodontic appliances such as removable partial denture and fixed bridges will be realized by the application of 3D printing on e-models in future.
Theme 8: Treatment planning  Treatment procedures on e-models are reversible by undoing or soft copying.
 Treatment procedures on e-models can be pin point.
 The expected treatment outcomes can be stimulated.
 Treatment planning is facilitated by accurate instruction and production of e-models.
 E-models show possible prosthodontic treatment options.
 E-models can be used to diagnostic wax-up treatment.
 E-models can simulate the tooth movement.
 E-models can have occlusal analysis.
 E-models cannot simulate as articulator in checking lateral excursion.
 Face-bow cannot be done.

Discussion
Although Faculty of Dentistry, University of Hong Kong has adopted PBL in BDS curriculum since 1998, methods of instruction have always been under modification based on changes in real-life clinic settings and longitudinal effectiveness of different strategies reported by other reviewers.Blended PBL was developed to enhance both PBL teaching effectiveness and integration of e-models in clinical environment.E-models assisted in achieving goals and objectives in PBL.Students' better understanding of goals and objectives in PBL helped them to retain knowledge and skills for future application in clinical environment (Prosser & Sze, 2014).Results found in this study provided understanding of students' cognitive, behavioral and motivational regulation (Nicol & Macfarlane-Dick, 2006) to using e-models in PBL.It bridged a gap of discrepancy between expectations of the faculty and the students, thus had high value for faculty development (Steinert et al., 2006).The general increase of students' importance ratings on goals and objectives in both self-assessment and facilitator-assessment of PBL illustrated that using e-models enhanced the students' understanding of the faculty's expectations in PBL tutorials.It also implied that implementation of e-models had good compatibility with PBL and was consistent with PBL teaching strategies.
Extra computational skills were required in order to use e-models to do measurement, analysis and design, which meant that the students needed to practice using e-models as a part of their preparation for PBL tutorials.Therefore, the students were more aware of the learning objective "Complete assigned tasks on time".Besides computational skills, communicational skills were also important training objectives in dental education (Hannah, Millichamp, & Ayers, 2004).Students' were trained in active listening rather than passive listening as they changed from traditional lectures to PBL tutorials.Active listening (Rogers & Farson, 1979) required full attention to the content of the conversation in order to obtain information, show appreciation and empathise with the speaker.Active listening skills affected not only PBL learning efficiency, but also future communication with others, especially patients in clinical settings.The statistically significant result of the objective "Active listener" demonstrated that using e-models created positive influence in the process of information obtaining and knowledge exchange.
Small group teaching was one of the essential elements of PBL.Facilitators played an important role in guiding discussions and facilitating proper group functions.Facilitators helped to introduce problems and to guide the students through problemsolving process, instead of giving answers directly.Students' critical thinking ability and self-evaluation ability were main focuses of this teaching method.The students found emodels helpful in achieving these goals, shown by statistical significance found in t-test results of five questions, i.e. "Promote group problem solving", "Promote appropriate group function", "Facilitate your learning", "Promote your critical thinking" and "Promote your learning".Although general result of the other two questions were statistically non-significant, majority of subquestions in these two sections showed significance in t-test, see Fig. 4. The results confirmed that e-models enhanced the students' understanding of the facilitator's role in developing their critical thinking ability as well as self-evaluation ability from constructing the group problem solving learning environment.
While solid plaster casts had limited amount of usage, online learning system allowed the students to download study e-models for their PBL preparation outside tutorial rooms.Therefore, the student's PBL preparation was strongly encouraged, which was reflected as significant increase of time spent on PBL preparation, both total time and online study time.This correlation between the students' attention and their PBL preparation time can be useful in future curriculum design.For example, different forms of learning resources could be provided to attract the students' attention.Prolonged online PBL preparation time was found positively correlated to the students' perceived importance of study purposes, such as "Apply knowledge to the problem", "Able to contribute new information", "Willing to question" and "Critical assess data", under the categories of "Well in reasoning" and "Communication well".One possible explanation was that study e-models provided more detailed information as well as better treatment outcome simulation than traditional plaster casts.Therefore, with a better knowledge of e-models, the students might find their thinking and discussion process more interesting and productive.On the other hand, linear correlation was found between the facilitatorassessment object "Ask you to evaluate the quality of learning resources used" and increase in the students' time on internet for PBL preparation.It echoes with the calling to improve the quality of e-models (Bell, Ayoub, & Siebert, 2003) for their further application in PBL or in other courses.
The theme extraction function of "Leximancer" provided a systematic and practical way to understand the students' perceptions of using e-models from the huge text content of their feedbacks in the questionnaires.The first theme "Comparing with solid plaster casts" illustrated e-model's advantage in long-term maintenance and storage, as well as the concern of additional hardware support such as monitors for e-models.The perception "It is better to have the use of both e-models and plaster casts" implied that the students may need more time to get used to e-models in PBL.The second theme "The use in lab stages" had an overlapping area with the first theme, which corresponded to the conceptual similarity between these two themes about the concern of additional requirements needed for implementation of e-models.The concern in the second theme was focused on extra skills required for image scanning and using software in lab stages.Furthermore, the concept word "communication" linked the second theme and the third theme "Patient feeling".Communication between dentists and co-workers such as lab technicians, other dentists and patients was enhanced by e-model's advantages such as easy transportation and detailed presentation.The students' feedback about their needs of extra skills and more informative communication was consistent with statistical significance found in perceived importance of objects "Complete assigned tasks on time" and "Active listener".The fourth theme "computational support" and the fifth theme "Cost" focused on requirement and cost to set-up and maintain e-model system, including both hard-and software.Since the students' feedbacks in these two themes were mainly concerned about establishment of e-model system rather than long-term usage, little conceptual similarity was found between these two themes and the first three themes.
The sixth theme "Treatment planning" was the students' feedback about emodel's function in treatment planning.Functions such as undo, soft copy and simulation were considered beneficial in the process of treatment planning, helping the students to compare different treatment options, make diagnostic wax-up and analyse occlusal movement during treatment.E-models made it possible to visualize both treatment process and treatment outcomes, and pin point function allowed the students to give more accurate instructions as they communicated with co-workers.Meanwhile, some students thought that there was no virtual articulator function in e-model software, probably because it was not introduced by the facilitators in PBL tutorials.It proved that the students' knowledge of e-models heavily relied on facilitators' demonstration in PBL tutorials.Combining this with the statistical results, we can see why e-models made the facilitator's role stronger in PBL tutorials.In addition, e-models provided a method to study and prepare for PBL with higher accuracy, which agreed with the positive correlation found between PBL preparation time and the students' reasoning and communication abilities.The seventh theme "Preparation of models" showed that emodels were considered to be both convenient and environmental-friendly.Together with 3D printing technology, e-models may have potentiality of mass production as well (Yau, Yang, & Lin, 2016).The last theme "Quality of models" illustrated the students' concern about accuracy of details in e-models.It agreed with the correlation found between the students' perception of the facilitators-evaluation question "Ask you to evaluate the quality of learning resources used" and their time on internet for PBL preparation.The curriculum designers hence should keep on monitoring the quality of e-models, especially the accuracy of information provided by e-models.
Understanding of expectation from both the dental faculty and the students allowed curriculum designers to design and modify PBL curriculum to achieve better teaching outcome and reduce facilitators' workload.Faculty support was critical for continuous improvement of facilitators (McLean, 2003), especially when innovative technologies such as e-models were first introduced into the curriculum.Since not all facilitators were experts in using e-models, they might have difficulties in demonstrating e-model functions or finding the right time to use e-models in PBL.Facilitator training was therefore necessary to improve their skills in using e-models, so as to smoothen the process of implementing e-models into original PBL setting, and enhance the students' understanding of PBL principles (Chung, Hitchcock, Oh, Han, & Woo, 2011).Facilitators' performance in PBL tutorials could be a role-model for the students to understand, experience and adapt to PBL learning (Hmelo-Silver & Barrows, 2006).Findings from this study suggested that facilitator training should emphasize more on their skills of using e-models in problem analysis, treatment planning and giving instructions, so that the students could benefit more from informative communication, critical thinking and group problem solving process.Furthermore, the study results also suggested a demand for stronger technology support including high-quality e-models and computational skill tutorials for the students.

Limitation
Interpretation of the results should be with caution.Discussion of the implications of quantitative results might be enhanced by addressing some issues.For example, it seemed like some the changes in students' understanding might be attributed to other aspects of PBL experiences, instead of e-models usage solely.Small sample size of the study limited validity of statistical significances of the results.Comparison groups used in ttests were two data sets of the students' perception before and after using e-models in PBL.Further statistical tests should be run for different comparison groups according to the students' characteristics (Das, Mpofu, Hasan, & Stewart, 2002).Besides, analyses of this study based on measurement of the students' perception changes to PBL.It restricted generalization of the study results in teaching environments other than PBL.Also, facilitators' viewpoints were not included in this study.A further study of the facilitators' perception could provide a more dynamic picture of using e-models in PBL.Moreover, only impacts of using e-models in PBL settings were analyzed in this study.Impacts of PBL to the students' skills in using e-models were not studied, which could be one direction for future study design.Follow-up measurement of the students' performance can further evaluate their learning outcomes and analyse its relationship with the students' perception of teaching purposes.

Conclusion
The students' higher importance ratings on all goals and objectives in current student self-assessment and facilitator assessment of PBL showed an increased understanding of the faculty's expectation in blended PBL with the help of e-models.It illustrated compatibility between implementation of e-models and expectation of PBL.Qualitative analysis of open-ended questions was done with visual text analytic software Leximancer to extract themes from contents of the students' feedbacks and results were found to improve validity of statistical results.Extra skill needs required to use e-models and new informative communication made possible by e-models explained significant changes of the students' importance ratings on objectives "Complete assigned tasks on time" and "Active listener" in self-assessment.Besides, e-model functions in treatment planning, problem analysis and giving instructions supported significant changes of the students' importance ratings on facilitators' role in training critical thinking and guiding group problem-solving.Meanwhile, regression analysis was used to find correlation between the students' awareness of teaching objectives and their behavior in PBL preparation.The students' attention to objectives under self-assessment categories "Well in reasoning" and "Communication well", as well as to facilitator assessment objective "Ask you to evaluate the quality of learning resources used" were found to be associated with their increase in PBL preparation time on the Internet.It provided insights that e-model online learning resources could influence the students' behavior in PBL preparation and assist the students' to understand and analyze problems.Trainings for facilitators to enhance their skills in using e-models and integrating those skills into facilitation in PBL was hence critical for continuous development of this innovative method.Stronger support from the faculty such as high quality e-models and video tutorials to the students for related computational skills should also be available to cope with the students' needs in their PBL preparation.

Fig. 1 .Fig. 2 .
Fig. 1.The students' importance ratings of the goals in the student self-assessment

Fig. 3 .Fig. 4 .
Fig. 3.The students' importance ratings of the goals in the facilitator-assessment

Fig. 5 .
Fig. 5.The Leximancer's concept plot of students' perception of using e-models showing in eight extracted themes

Table 1
The characteristics of the 50 students Note. †JUPAS = Joint University Programmes Admissions SystemTable 2Time for PBL preparation before and after the use of e-models in PBL

Table 3
Summary of the students' perceptions of using e-models in eight themes of the Leximancer's result