Six Years of Lessons Learned in Monitoring and Evaluating Online Discussion Forums

: This paper presents the plan for evaluating virtual discussion forums held on the Implementing Best Practices in Reproductive Health (IBP) Knowledge Gateway, and its evolution over six years. Since 2005, the World Health Organization Department of Reproductive Health and Research (WHO/RHR), the Knowledge for Health (K4Health) Project based at Johns Hopkins Bloomberg School of Public Health‟s Center for Communication Programs (JHU∙CCP), and partners of the IBP Initiative have supported more than 50 virtual discussion forums on the IBP Knowledge Gateway. These discussions have provided global health practitioners with a platform to exchange evidence-based information and knowledge with colleagues working around the world. In this paper, the authors discuss challenges related to evaluating virtual discussions and present their evaluation plan for virtual discussions. The evaluation plan included the following three stages: (I) determining value of the discussion forums, (II) in-depth exploration of the data, and (III) reflection and next steps and was guided by the “Conceptual Framework for Monitoring and Evaluating Health Information Products and Services” which was published as part of the Guide to Monitoring and Evaluation of Health Information Products and Services. An analysis of data from 26 forums is presented and discussed in light of this framework. The paper also includes next steps for improving the evaluation of future virtual


Introduction
The urgent need to accelerate progress towards meeting the Millennium Development Goals, coupled with the expansion in Internet connectivity around the world presents a timely opportunity to virtually connect those working in the health sector to address challenges and to share effective practices and lessons learned. Activities supported by Information Communication Technologies (ICTs), such as online communities of practice and virtual discussions, are increasingly recognized as an important means to bring together health care workers at all levels to share knowledge and experiences (Moss, 2004;Brooks & Scott, 2006). As Meessen and colleagues suggest, "the community of practice strategy could be a real breakthrough for managing knowledge on implementation challenges" (2011; p 1007). These activities can "link colleagues in remote locations and across time zones, amplify the voice of people in remote locations, and facilitate communication and the generation of new knowledge" (Knowledge for Health Project, 2009; p. 59) --all at relatively low cost.
Since 2002, partners of the Implementing Best Practices in Reproductive Health (IBP) Initiative have collaborated on knowledge management strategies for virtual information exchange, primarily through developing the capacity, technology, and a methodology to support online communities of practice and discussion forums. The IBP Initiative is a consortium of 35 international reproductive health agencies, led by the World Health Organization"s Department of Reproductive Health and Research (WHO/RHR) and the United States Agency for International Development (USAID), who work together to improve access to reproductive health through scaling-up of best practices.
To improve communication among health professionals working around the world, WHO/RHR, USAID, and IBP partners, in collaboration with WA Research, developed a dynamic electronic communication tool to support virtual communities of practice (CoPs) called the IBP Knowledge Gateway. The Knowledge Gateway is a simple platform that allows for participation online or by email, and can be accessed in settings with low Internet bandwidth. Unlike other virtual collaboration or social networking platforms, the Knowledge Gateway was designed specifically for health and development professionals, to support access to timely and relevant information, and provide a venue in which they can immediately share local and international experiences in order to improve programs (O"Brien & Richey, 2010).
Since its launch in 2005, the Knowledge Gateway has grown to be the largest virtual communication platform in the health and development sectors, used by over 300,000 individuals worldwide. Membership in its reproductive health communities alone has increased from 300 in 2004 to nearly 30,000 in 2011. For additional information about the Knowledge Gateway, see www.knowledge-gateway.org.
The IBP Initiative defines a virtual discussion forum as a time-bound, moderated event that takes place in an online community of practice. Virtual discussion forums sponsored by the IBP Initiative grew out of an effort to increase collaboration, knowledge and information sharing and experiences among reproductive health professionals.
Fifty-one reproductive health-related online discussion forums have taken place on the IBP Knowledge Gateway between 2005 and April 2011 and post-forum evaluations of 26 of these discussions are available. Evaluating these discussions is key to (1) determining if these discussions are meeting the information needs of their target audience, (2) determining if the format and content are optimal, and (3) enhancing and informing future virtual discussions on the Knowledge Gateway and elsewhere. Li and colleagues" (2009) systematic review found that there is a lack of studies on the effectiveness of CoPs in the health sector. Most studies on CoPs in the health sector were from the UK or the US (92.3%) and were qualitative studies that described how CoPs functioned or discussed the complexity of developing and sustaining them. As a result, "the effectiveness of CoPs in this sector remained unclear" (Li et al., 2009). This paper seeks to fill this information gap.
Many of the discussions held on the Knowledge Gateway have been organized and implemented by WHO/RHR or two USAID-funded projects, the INFO Project and the Knowledge for Health (K4Health) Project, based at the Johns Hopkins Bloomberg School of Public Health"s Center for Communication Programs (JHU•CCP), in collaboration with other IBP partners. In other cases, WHO/RHR and JHU•CCP have supported or trained organizations to carry out their own virtual discussions. More recently, WHO/RHR has launched an initiative to train facilitators in Africa to conduct regional discussion forums. This paper focuses specifically on the 26 discussions supported and evaluated by WHO/RHR and JHU•CCP for which we have evaluation results.
All virtual discussion forums focused on topical issues related to reproductive health. Forums lasted from one to six weeks and were conducted primarily in English. 1 Discussion forums are generally organized around a central theme, with multiple subtopics, chosen by the members of the community and/or organization(s) sponsoring the discussion. For example, a discussion forum on emergency contraception (March 2011) focused on the following sub-topics for one or two days of the discussion: ensuring access, facing opposition, mechanism of action, post-rape care, and issues related to repeat use. Most discussions enlisted subject matter experts including researchers, program managers, providers, and in at least one case, family planning clients. Involving these experts not only added credibility to discussions, but also encouraged greater participation. Preceding each forum, organizers identified one or two moderators to facilitate the discussion. Staff from WHO/RHR and JHU•CCP often moderated forums they sponsored, or trained staff from other organizations to facilitate.
In general, the discussions organized by IBP partners follow a similar structure. Invitations to register for a discussion are circulated to the reproductive health community through announcements on listservs, communities of practice, and newsletters. Each day of the discussion, the moderator circulates a short introduction to the daily topic and a short series of questions via email to participants, who are invited to respond with their experiences, challenges, and lessons learned. The moderator collects the contributions and synthesizes them into a digest. The draft digest is then sent to the discussion"s subject matter experts, who provide additional commentary and respond to questions raised by participants. The final digest is then circulated to all participants. Although the discussion takes place primarily through email, all discussion activity is also automatically archived online in the Knowledge Gateway. Figure 1 below shows a screenshot of the Human Resources for Health (HRH) Exchange CoP on the Knowledge Gateway. All CoPs on the Knowledge Gateway have a similar structure.

Figure 1. Screenshot of a CoP in the Knowledge Gateway
The majority of the discussions organized by WHO/RHR and JHU•CCP, in collaboration with other IBP partners, have been evaluated with an online survey tool, participants to submit their contributions in Spanish or French. In July 2011, the USAID-funded K4Health Project at JHU•CCP supported the first virtual discussion to take place entirely in French. using a standard, jointly-developed format. To gain additional feedback, WHO/RHR and JHU•CCP periodically conducted in-depth telephone interviews with discussion participants and occasional surveys among the subject matter experts who helped guide the discussions.
The purpose of this paper is to discuss how the evaluation plan has changed over time, present an analysis of the evaluation data gathered from 26 online forums, and outline next steps for refining this process. In the next section, the paper discusses challenges related to evaluating virtual discussions and the use of the "Conceptual Framework for Health Information Products and Services," which informed the three stages of the evaluation strategy: Stage I: Determining value, Stage II: In-depth exploration, and Stage III: Reflection and next steps. Section three analyzes the content of previous forum evaluations, and section four provides a discussion of the results of the forum evaluations.

Challenges of Evaluating Virtual Discussion
Virtual discussions have been in existence since the 1990s and are commonly found in education and business communities and increasingly in the world of health care. Previous studies evaluating online discussion forums and communities of practice in education and business have shown that they enhance teaching, learning, and collaboration (Bezuidenhout, 2009;McNamara & Brown, 2009;McDermott, 2001). Although some basic evaluations of virtual discussions have taken place, relatively few monitoring and evaluation guidelines exist for virtual discussions and other knowledge management activities and evaluation techniques are still being refined (Li et al., 2009). Methods commonly used to evaluate virtual discussion forums include surveys, questionnaires, structured and semi-structured interviews, and focus group discussions. Other more in-depth methods of evaluation include content analysis, outcome mapping, and social networking analysis (Web2fordev, 2007).
In the health field, little research has been conducted to determine links between online discussion and subsequent health outcomes, despite the interest of donors and implementers to demonstrate results that maximize return on investment (Sullivan et al., 2007). Ranmuthugala and colleagues (2011) note the lack of empirical evidence of communities of practice on improving health care practice, indicating that much of the existing literature consists of descriptions and summaries of activities, rather than analytical evaluation.
Challenges associated with evaluating virtual discussion forums include capturing quantitative and qualitative outcomes and determining impact. There are no universallyaccepted set of standards for evaluating these activities and forum organizers are still working towards determining the best methods. Another major challenge in evaluating online discussion is choosing appropriate indicators or metrics. According to Robertson (2003), using metrics in knowledge management activities allows setting of targets, assessment of success, calculation of return on investment, and tracking the ongoing viability of the activity. Indicators, such as number of website hits and number of responses to a request for help, are often initially chosen for evaluating online CoPs (McDermott, 2001). "These activities are easy to measure, can be helpful in understanding the level of community activity, and give some indication of its health, however these measures do little to demonstrate the contribution of the community to its members or the organization," according to McDermott whose analysis included online business communities.
To understand a community, McDermott recommends identifying stakeholder needs, developing a measurement strategy, collecting data, calculating value through return on investment, and constructing community stories (2005). Hoss and Schlussel (2009) reinforce the need to go beyond basic numbers; "Measuring for the sake of measuring is fruitless and a waste of time." They state that metrics should serve the objective of "continuous improvement of knowledge management" and that "a metric should demonstrate whether knowledge is being shared and applied." Table 1 outlines a number of potential CoP metrics, suggested by Hoss and Schlussel. To address this question, JHU•CCP and WHO/RHR, in collaboration with partners of the USAID/IBP Knowledge Management Working Group (KM WG), organized a forum on the IBP Knowledge Gateway in April 2011 on metrics and evaluation entitled "Looking beyond the Numbers: Measuring the Value of CoPs for Global Health." Over 371 participants from 56 countries participated in this two-week discussion. Participants shared a number of factors that contribute to a successful CoP, including:  Easy access to the discussion;  The opportunity to interact with a diverse group of participants;  The ability to meet one's information needs;  Maintaining or increasing membership and motivation to participate;  Reaching participants with key information;  Ensuring that discussion threads are clear and easily understood;  Generating debate that is provocative, interesting and stimulating; and  Encouraging participants to discover new ways to tackle problems and to foster sustainable partnerships.
Participants shared that the CoPs they run or participate in are usually evaluated through surveys, questionnaires, and structured interviews. However, they noted that these methods may not capture the full value or contribution of CoPs. Participants also pointed out that CoP organizers, participants, and guest experts may have different views of what makes a CoP successful.

Evaluating CoPs Based on a Conceptual Framework
Since WHO/RHR, JHU•CCP, and other IBP partner organizations began facilitating and moderating online discussion forums in 2005 on the IBP Knowledge Gateway, these organizations have been concerned with evaluating them. As the IBP Initiative and the world of virtual discussions evolved, our evaluation strategy evolved as well. We originally focused largely on quantitative data discussed by McDermott that were easy to capture and demonstrated reach, such as the number of participants in each forum, the countries in which participants were located, and the number of postings during each forum. However in recent years, we have become increasingly interested in "looking beyond the numbers" and focusing on uptake and application of information and resources shared during the forum.  (Sullivan et al., 2007). The Guide suggests 29 indicators to measure the impact of health information products and services through reach, usefulness, and use, as well as collaboration and capacity building.
At the time of its development, this guide represented one of the few efforts to collect, develop, organize, and define indicators to measure reach, usefulness, and use of knowledge management products and services. However, it does have its limitations. Since the guide focused on measuring traditional information products, indicators for virtual discussion forums are not explicitly included. However, the breadth of these indicators has helped to frame the evaluation of discussion forums, such as developing questions for member surveys and interviews.
In collaboration with members of the USAID/IBP KM WG, JHU•CCP is currently revising the conceptual framework. The new version, expected in 2012, will include a more robust set of indicators for knowledge exchange and sharing and a broader analysis of knowledge management activities within global health and development organizations.

Figure 2. Conceptual Framework
The "Conceptual Framework for Monitoring and Evaluating Health Information Products and Services" provides an approach to measuring the outcomes of health information programs, including a comprehensive set of indicators and a logic model linking inputs, processes, and outputs to multiple outcome levels. The framework is intended to help users understand the pathways through which health information products and services inform policy and improve programs, practice, training, education, and research. By mapping how inputs, processes, and outputs link to one another to contribute to outcomes, the framework can guide program design, implementation, and evaluation. The framework focuses primarily on the supply side to show how program components can integrate to produce effective information products and services. The Monitoring and Evaluation section incorporates some aspects of user demand, including user satisfaction and preferences. Table 2 summarizes the conceptual framework"s performance measures.
Following the conceptual framework, early evaluations of CoPs by WHO/RHR, JHU•CCP, and other IBP partner organizations largely focused on initial outcomes of their knowledge sharing activities. After several years of collecting data at this level, WHO/RHR and JHU•CCP conducted in-depth interviews to attempt to measure intermediate outcomes as well. The next section describes the stages of this evaluation strategy, starting with collecting data on initial outcomes, moving towards looking at intermediate outcomes, and where we are today in terms of improving upon and strengthening our evaluation efforts.

Table 2. Performance Measures for Monitoring and Evaluating Health Information
Products and Services Inputs: Constitute all resources put into a project, such as human and financial capital, equipment, and facilities leveraged to carry out activities.
Processes: Activities undertaken to develop products and services to achieve a specific goal, usually using inputs; refers to how and how well an activity is carried out.

Outputs:
The products or services resulting from processes in order to reach audience groups.
Outcomes: The benefits for participants during or after their exposure to a product, service, or program. Outcomes may relate to knowledge, skills, attitudes, behavior, health condition, or health status. Three outcome levels are defined in this framework:

Initial outcomes:
The usefulness of information products and services as determined by audience satisfaction with and perceived quality of products and services.

Intermediate outcomes:
Relate to use or adaptation of information products and services to inform policy, improve programs, enhance training and education, and promote research efforts.
Intended long-term outcomes: Improvements in health condition or status at the population-level that contribute to the exposure of health care providers and allied professionals to health information or products.

Stage I: Determining Value
When IBP partners first began convening virtual discussions, a key objective was to determine how participants, donors, and other public health professionals valued these forums. As the concept of online forums in public health was relatively new, evaluations of early forums focused on "numbers" and the following outcomes in the conceptual framework:

Reach
Usefulness Use  Initial distribution  Secondary distribution  Referrals (audiences reached and knowledge accessed)  User satisfaction  User perception of quality (audiences satisfied)  Collaboration facilitated "Reach" included initial distribution indicators, such as the number of people registered to participate in an online discussion forum and the percentage of registrants from developing countries. Secondary distribution indicators included whether participants reported that they downloaded or used resources shared during the forum or forwarded discussion posts to a colleague. We measured the "usefulness" of virtual discussions by the percentage of participants who reported being satisfied with the amount and content of the discussion. We measured "use" by asking participants if they had used and/or if they intended to use information and/or resources shared during the forum in their work.

Stage II: In-Depth Exploration
Following these initial efforts, WHO/RHR, USAID, and JHU•CCP became interested in obtaining more in-depth insights from participants, such as how participation in an online discussion impacted daily work. A typical online discussion forum includes participants from a wide range of organizations around the world who may not have many opportunities for face-to-face collaboration at the organizational or project level. As a result, we focused on the following additional outcomes found in the conceptual framework in this stage of the evaluation strategy:

Environment informed
Programs and practices enhanced To measure these outcomes, we added questions to the evaluation surveys on other aspects of use of information and resources, including what (and how) participants are sharing and applying knowledge. We also conducted in-depth telephone interviews with participants to obtain a richer understanding of knowledge use and information sharing. Examples of participants" reported use of knowledge are provided later in the paper.

Stage III: Reflection and Next Steps
In the current stage of the evaluation strategy, we continue to collect data on the indicators described in Stages I and II and plan to revise the indicators and evaluation plan following the release of the next version of the conceptual framework in 2012. In addition, WHO/RHR, JHU•CCP, and other IBP Initiative partners are retrospectively codifying quantitative and qualitative data from all past discussion forums into a standard template that will be searchable and available to the public on the Knowledge Gateway public website.
WHO/RHR and JHU•CCP plan to continue to collect data to ensure that we are promoting and fostering a nurturing virtual environment that is meeting the needs of participants by increasing knowledge sharing and promotion of best practices, fostering collaboration, and reducing duplication of efforts. We will continue to survey participants of all online discussion forums held on the IBP Knowledge Gateway. Additionally, we will also conduct in-depth interviews and possibly focus group discussions with forum participants to explore use of knowledge gained during the forum and how participation in a forum advances collaboration and reduces duplication of effort.

Content of Evaluations
Fifty-one virtual discussions have been held on the Knowledge Gateway between 2005 and April 2011. However, a number of these discussions were led by organizations other than WHO/RHR and JHU•CCP and thus evaluation results were not available for this analysis. This analysis will focus on the 26 discussions supported and/or evaluated by WHO/RHR and JHU•CCP. Details on the 26 discussions can be found online (http://tinyurl.com/43cy7r7).
A standard set of questions was used in the evaluations; however in some cases, questions were removed or added to accommodate the specific needs of the cosponsoring organizations. In a few forums, slight alterations were made to the wording of some questions and/or answer options. The authors assessed these minor differences, and included the questions for analysis as long as intent was not changed. Also, in a few evaluations, one or two answer options were not included. Surveys consisted of 15 to 18 questions intended to solicit information about reach, usefulness, and use, as well as information about participant demographics and ease of participation in the forums. Each survey was administered online and took approximately 5-10 minutes to complete.
All surveys were conducted using the online survey tool Zoomerang (www.zoomerang.com). Summary reports of all discussions were produced within Zoomerang and similar questions were grouped together to determine overall results from all the discussions; average numbers as well as median numbers for each question are presented below. Because some questions were included or removed from certain surveys, the authors note the number of surveys in which each question appeared in the data presented below. Also, because some survey questions asked respondents to select multiple answer options, some percentages to do not equal one hundred.

Evaluations and Response Rates
A total of 1,684 participants responded to the 26 forum evaluations, resulting in an overall response rate of 12%. Excluding the Female Condom Programming in Low-Resource Settings discussion, in which only 3 of 500 participants responded to the evaluation, the lowest response rate was 5% in the Male and Female Sterilization forum (2008). The highest evaluation response rate was 17% in the Health Worker Migration discussion (2008). Since 2005, most evaluation response rates for forums have been in the 9-14% range. Surveys are sent out immediately after the discussion ends and no incentives are provided for completion.
In addition to the online evaluations, WHO/RHR and JHU•CCP conducted telephone interviews with 17 people who had participated in two online discussionsthe 2009 Access to Reproductive Health Essential Medicines and Contraceptives: Why is it so Difficult to Achieve? and the 2010 Kampala Conversations: Knowledge to Action for Family Planning. Interviewees were selected from a convenience sample of those who had provided us with their contact information on the forum evaluations. The interviews contained 14 to 16 questions and took approximately 30-45 minutes. Results from the interviews are presented below.

Evaluation Plan: Stage I
As discussed above, during the first evaluation stage, WHO/RHR and JHU•CCP were concerned mainly with four areas: 1) participant characteristics; 2) participation numbers; 3) initial outcomes of reach and usefulness; and 4) the intermediate outcome of use. Results of evaluations from 26 forums are below. All numbers in figures are percentages and totals that do not equal 100% are due to rounding.

Participant characteristics. Three questions asked about participant background information
The question, Please select the category that best describes your organization type, was asked in all 26 forums. Across all 26 forums, most forum respondents (34%) worked at local and international non-governmental organizations/private volunteer organizations (NGO/PVO). The next largest group (24%) worked at academic/research institutions and the next largest group (19%) worked at medical/health organizations. Smaller numbers worked in government ministry (8%), reproductive health/family planning service provider (7%), private sector (5%), USAID (3%), United Nations agency (3%), faith-based organization (1%), and news media (1%; see Figure 3).
The question, Please select the category that best describes your work, was asked in 16 forums. Of all 16 forums, most respondents (23%) worked in reproductive health/family planning services. Twenty-two percent worked in program development/management, 19% in teaching/training, 14% in research/evaluation, 11% in health/medical and/or service delivery, 7% in health communication, 4% in advocacy, 3% were students, 2% worked in policymaking, and 0% worked in journalism (see Figure  4). Most respondents from the 26 discussions (38%) heard about the forum from a listserv, 20% heard about it from a colleague/friend, 17% heard about it from the weekly IBP newsletter, 16% from the JHU•CCP K4Health Project website (www.k4health.org) or its predecessor project"s (the INFO Project) website, and 8% heard about the forum from other sources. In all 26 evaluations, respondents were asked if they posted any messages on the forum. About half (44%) did post a message during the discussion. In the 23 evaluations that included this question, most respondents (66%) participated in the forum primarily by email. Another 15% participated by logging into the Knowledge Gateway platform and 17% participated in a combination of email and online. The survey included an additional question about logging into the Knowledge Gateway; "If you participated online, did you have problems logging onto the IBP Knowledge Gateway?" Although the question was intended for the subset of participants who logged into the Knowledge Gateway, it was NOT part of a survey skip pattern, so all respondents had the ability to answer the question. This accounts for the 27% of respondents who answered, I did not log into the system. Sixty-three percent had no problems logging in and 10% experienced difficulty.

Reach. Two evaluation questions tracked initial distribution and two other questions tracked secondary distribution.
To measure initial distribution, we tracked the number of people registered to participate in each online discussion and the countries in which they worked. In total, To measure secondary distribution, we asked who downloaded resources from the community library or downloaded resources that were sent during the discussion, and who forwarded forum postings to other people (both asked in 25 evaluations). Over half (52%) of respondents reported already having downloaded resources. Although 21% reported not downloading any resources, 29% said that they planned to download resources in the future. Just over half (51%) of respondents had forwarded forum postings to other people.
Usefulness. Five questions were asked to elicit user satisfaction and perception of quality.
More than half (61%) of respondents were very satisfied, and 35% were somewhat satisfied with the content of the discussions (asked in 25 evaluations). Three percent were not satisfied with the content of the discussions. Similarly, 68% felt that there was the right amount of discussion; 20% felt there was not enough, and 12% felt that there was too much discussion (asked in 26 evaluations; Figure 5).
Most respondents (54%) felt that the forums definitely met their specific goals, 38% felt that they somewhat met their goals, and 3% felt they did not meet their goals (asked in 23 forums; see Figure 6).

Figure 6. Respondents' Opinions on the Forums Meeting their Goals
In 12 evaluations that asked this question, three-quarters of respondents liked receiving a single, digested email each day. The other quarter would have preferred to receive individual messages from contributors as they were posted. In ten evaluations that asked this question, 95% of respondents felt that involving guest experts in the discussion each day was useful and/or enriching. Two percent felt that it was not useful, and 4% had no opinion.
Use. One question was included in 23 evaluations that measured use of resources or practices discussed in the forum in participants" work.
Thirty-nine percent of respondents reported having already used resources or practices discussed in the forum in their work. Sixteen percent had not used any in their work, and 42% reported planning to use resources and practices in their work in the future. Interviewees were selected randomly from those who had indicated on the evaluation that they were willing to be contacted to provide additional information. Participants of the telephone interviews worked in 12 countries for a range of reproductive health organizations, including NGOs, USAID, WHO, and faith-based organizations.

Evaluation Plan: Stage II
The interview data were reviewed and grouped into common themes:  Environment informed (i.e., evidence-based information contributes to policy and increased resources for health.  Programs and practices enhanced (i.e., adoption of evidence-based practice and increased access facilitated)  Collaboration and capacity building facilitated (i.e., availability and access to information increased and knowledge and experience sharing increased)." These themes correspond with the conceptual framework and the interview quotes presented below are intended only to illustrate the kind of quotes received from interview respondents on each of the themes.

Environment informed
To evaluate the outcome of "environment informed," we included a question in one evaluation that spoke to "evidence-based information contributes to policy and increased resources for health." The evaluation survey following the 2008 Health Worker Migration discussion included the question; Did you find this global discussion covered issues that could be used to inform policy discussions? The majority of respondents (n=113; 93%) answered yes to this question.

Programs and practices enhanced: adoption of evidence-based practice
After the 2009 Reproductive Health Essential Medicines online discussion, interviewees discussed how information shared in the forum impacted their daily work. All said that they are using the information from the forum "somewhat" or "very much" in their daily work. Most commented on the usefulness of learning about experiences in other countries. Answers included:  I am definitely using the information. After the 2010 Kampala Conversations forum, four participants described instances where they used information shared in the forum in their work. Responses included:  I used some of the things I got from the forum in providing training for HIV [positives] in prevention and contraception.  I share publications from the internet from the forum with others, and I can see a lot of interest.  I used [the information] to teach midwifery students.
Dissemination and adoption of evidence-based practices is one of the main goals of the Knowledge Gateway and discussions that take place on the platform. Participant responses in this area confirmed that we are meeting this goal and reaffirmed commitment to continuing this method of virtual exchange.

Programs and practices enhanced: increased access facilitated
In the area of "increased access facilitated," we included one question in the interview guide, How could we improve your access to the information? Most answers related to the availability of forum exchanges, the challenges of internet connectivity, and resources in other formats including CD-ROMs and paper:  I wouldn't really know what you could do better. I think if the information that is shared with us was sent on a template, so we can look at the topics and click on those that directly link to our project area. That would be helpful. Most of the time, I found it difficult, we have a problem with internet connection, it is expensive and difficult. So it is like an elite service. The other publications we have are in paper copies. A CD-ROM and paper copy would be useful.  In Cameroon, we have poor internet facilities, but this is the way for now. We need to get information for the communities, but this needs to be in print.

Collaboration and capacity building facilitated: availability and access to information increased
Most respondents" answers focused on transferring online discussion content and resources to the community in other formats, such as CD-ROMS and paper. Two respondents also noted the need for translation of forum resources into local language. Responses included:  As the above quotes illustrate, many interviewees noted the need for other methods of information distribution, especially in areas with poor Internet connectivity. Specifically, respondents expressed a desire for CD-ROMs and paper copies of the online discussions. IBP partner organizations who conduct online discussions on the Knowledge Gateway are aware of this request and are continually seeking cost-effective methods of material distribution. However, due to the high cost of producing and disseminating CD-ROMs and printed materials around the world, this is not currently a financially sustainable option for the discussions held on the Knowledge Gateway. Forum organizers frequently create a summary of the discussion after it has ended and post it on the Knowledge Gateway as well as email it to anyone who is interested. Although this method still requires an Internet connection to access the summary report, it enables people to read and share the contents of an online discussion after it has happened, often with people who were unable to participate in the online discussion due to competing priorities. This sharing spreads the information and knowledge exchanged in the discussion even further.

Collaboration and capacity building facilitated: knowledge and experience sharing increased
In the area of "knowledge and experience sharing increased," we included two questions in two forum evaluations (2008 Health Worker Migration and 2008 Female Condom Programming in Low-Resource Settings). Three-quarters of respondents felt that there was enough discussion and input from participants from different countries. Eightyfive percent of respondents in the same two forum evaluations felt that they were able to add their voices and experiences to the discussion.
In this area, we also included one question in the interview guide, Have you discussed the information with your co-workers and colleagues? Responses included:  Exchange of knowledge and experience was also confirmed in interviews. Most interview respondents answered positively to having shared information from an online discussion with colleagues and others noted that they intended to do so. Participant responses further support our commitment to continue online discussions in the future.
In addition to interviewing participants, one discussion forum evaluation also included an evaluation survey with the discussion's subject matter experts to determine if the process for involving these individuals was useful and how it could be improved. The 2009 Reproductive Health Essential Medicines discussion forum involved 18 subject matter experts. As with other discussions, these experts drafted a short introduction to the day's topic with 2-3 questions to catalyze the discussion. They then provided commentary in response to the contributions received on that topic. A total of seven experts responded to the survey. The expert survey responses provide areas for potential improvement. Although the majority (71%) of the experts reported that they were given enough opportunity to help plan the discussion, over half (57%) felt that their role was only somewhat clear and 43% stated that they had too little time to draft their commentary in response to the contributions received. Fifty-seven percent of the experts were "somewhat" to "very satisfied" with the content of the discussion and all indicated that they would be willing to participate as experts in future discussion forums. However, experts suggested that topics of future discussions be more clearly defined; "many of the topics overlapped excessively, resulting in the same ground being covered repetitively." There were suggestions to also attract more experienced participants and coordinate the commentary among participating agencies.
While resources for monitoring and evaluation are generally limited to allow for evaluations among only the participants of discussion forums, periodic surveys of subject matter experts will help to inform and refine the process for involving these individuals.

Discussion
The authors believe that this paper offers one of the first looks at empirical data from virtual discussion forums in public health. We found that a number of indicators exist for measuring CoPs but few are well suited for measuring activities related to virtual knowledge sharing and exchange, such as discussion forums.
The present analysis has a number of limitations including courtesy bias, recall bias, convenience sampling, and response rate. However, WHO/RHR and JHU•CCP have attempted to reduce recall bias by conducting the online evaluation survey immediately following the discussion. Response rates are comparable to other online surveys that JHU•CCP has conducted.
Overall, our analysis revealed varying levels of success in the three stages of our evaluation. Stage 1: Determining value was the most successful, especially in reach and usefulness. Our results showed that the 26 forums for which we have post-forum evaluations reached nearly 14,000 participants, including many working in developing countries. These results support the idea that online discussions provide a low-cost outlet for public health practitioners (O"Brien and Richey, 2010). Moreover, about half of evaluation respondents forwarded forum postings to other people. To evaluate usefulness, we asked questions about their satisfaction and perception of quality. Almost all respondents were either "very" or "somewhat satisfied" with the content of the discussions. Most were satisfied with the amount of discussion that occurred during the forums, and almost all respondents felt that the forums definitely or somewhat met their stated goals. Measurement of use during Stage I was minimal; only one question asked about use of resources or practices discussed in the forum in participants" work. However, 39% of respondents reported having already used resources or practices discussed in the forum in their work and 42% planned to use them in the future.
Our analysis revealed three results that we would like to explore in greater depth. The first is the 35% of respondents on average who reported being somewhat satisfied or not satisfied with the content of the discussion, as compared to those who were very satisfied. We would like to identify individual factors that negatively impacted participant"s perception of satisfaction, such as the topic of discussion, the number of active participants, the preparedness of the subject matter experts and facilitators.
The second is the 25% of respondents on average who would have preferred to receive individual messages from contributors as they were posted, rather than the digest format that contains all messages from contributors. By including more questions in the telephone interview guide that deeply probe these two areas, we may be able to revise the discussion forum format to satisfy a greater number of participants.
The third result to further investigate is the small percentage of respondents who experienced difficulty logging into the Knowledge Gateway. This could be a result of slow Internet connectivity, unfamiliarity with the Knowledge Gateway, and forgetting login information. We want to ensure that everyone can log into Knowledge Gateway without difficulty.
We were less successful in measuring the intermediate outcomes included in Stage II. This was not surprising given the difficulties of measuring the impact and outcomes of virtual discussions. We collected surface level data on many intermediate outcomes; however, focusing on one or two intermediate outcomes would allow us to acquire more informative results. We would like to focus on the outcomes of enhanced programs and practices and facilitated collaboration and capacity building since these outcomes ultimately lead to improved programming and service delivery in the field of reproductive health.
We also are going to revise the indicators used to evaluate virtual discussions based on the revised "Conceptual Framework for Monitoring and Evaluating Health Information Products and Services" and the criteria put forth by participants in the "Looking beyond the Numbers: Measuring the Value of CoPs for Global Health" discussion forum. Those criteria may prove to be more useful as indictors than the indicators described by Hoss and Schlussel (2009). Once our indicator list is finalized, we will revise the questions used on surveys and interview guides. We will investigate questions such as:  Did participation in the forum give you any new programming ideas? Please provide an example.  Did participation in the forum give you any new ideas about how you could improve collaboration with your partners? Please provide an example.  Have you remained in contact with others who participated in the forum? If so, why?  Do you plan to contact others who participated in the forum in the future? If yes, for what reason?
Ideally, revised questions like these will be more effective in eliciting responses about the intermediate outcomes of enhancing programs/practices and facilitating collaboration/capacity building than the questions we have been using.
In addition to conducting additional participant interviews, we may conduct focus group discussions to give us more insight into participant collaboration and use of information.
As we are revising our evaluation plan, we will also review and update our promotion strategy to include new outlets. Moving forward, WHO/RHR and JHU•CCP will also provide public access to summaries of forums. We hope that these summaries will help forum organizers build on past experience and avoid duplication.

Conclusion
Online discussion forums have the potential to enhance communication among professionals in many fields. Evidence shows that virtual discussions contribute to advancing global learning and collaboration among international public health professionals by addressing the "knowledge to practice" gap and providing an opportunity for geographically dispersed public health professionals to come together to collaborate and share knowledge that in turn can be used to inform and improve program design and management. Virtual discussions also can serve as a venue for sharing the latest research and guidance. Thus, the contribution of virtual discussions to advancing global health include: (1) quickly convening public health practitioners around the world to share experiences at low cost, (2) identifying innovative solutions to old problems and contributing to the development and dissemination of evidence-based health information products, and (3) enhancing and informing program design and management, and ultimately improving programs and health outcomes. Our analysis shows that we are achieving the objectives of online forums by reaching an extensive network of health practitioners worldwide. Participants are able to take the new knowledge learned and apply it to their work. Moving forward, we anticipate that we will be better able to measure the contribution of forums to intermediate outcomes with the refinement of the conceptual framework and indicators to measure such programs.
Implementing Best Practices (IBP) Initiative Secretariat, Margaret D"Adamo, Knowledge Management/Information Technology Advisor, USAID, Office of Population and Reproductive Health, and Tara Sullivan, Knowledge Management Director of the Knowledge for Health (K4Health) Project based at Johns Hopkins University Center for Communication Programs (JHU•CCP) for their review of the manuscript and invaluable feedback.