How is eHealth literacy measured and what do the measurements tell us? A systematic review

The increasing use of digital services and technologies in health care calls for effective tools to evaluate the users’ eHealth literacy in order to better understand the users’ interaction with health technologies. We here present a systematic review of existing tools to measure eHealth literacy and for what these tools have been used to investigate. We identified eight tools, of which three of them are bases upon a conceptual model of eHealth literacy and the remaining five are dual tools, i.a. comprised of individual measures for health literacy and digital literacy. Of these eight tools, only one tool (The eHealth literacy Scale eHEALS) was used in other studies than the one it was originally published in. eHEALS has primarily been used to establish eHealth literacy levels in different populations. Five of the studies have been conducted by examining eHealth literacy’s impact on health outcomes, and one study has established an association between high eHealth literacy levels and increased likelihood of attending colorectal cancer screenings in a Japanese population. The two other concept-based tools, eHLS and PRE-HIT, reflect an elaborated understanding of eHealth literacy. The five dual tools were primarily used to screen for adequate and inadequate health literacy and digital literacy. In conclusion, there is very little knowledge about individuals’ eHealth literacy and how it relates to health outcomes or the clinical course of specific diseases. New tools developed for the new age of social media and new technologies should be used as eHEALS may have some limitations.


Introduction
The use of digital services is increasing within health care and may constitute either a new opportunity or barrier for the patients.
Telehealth is moving treatment and monitoring into the homes of patients.Opportunities related to new technologies such as smartphones and wearable technologies allow patients to interact with health care professionals and peers in a way that is more complex than traditional health information search on the internet.The introduction of digital health services lead to increasing expectations for patients to be able to use and engage with digital health information.People not only need to be health literate but also to have capabilities, resources, and motivation to find, understand, and appraise health information when using digital services and technology.
In response to the need of being able to characterize peoples' health literacy in a digital context, in 2006 Norman and Skinner introduced a new conceptelectronic health literacy as the ability to seek, find, understand and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem (Norman & Skinner, 2006b).Since then, synonyms to electronic health literacy, such as digital health literacy and health technology literacy, have been proposed (European Commission, 2014;Jordan-Marsh, 2011).
When healthcare is provided in the form of technology or digital services, it is essential for the clinicians to know the levels of eHealth literacy among their patients to be able to provide services that suit actual needs and abilities.When designing educational programs, that aim to increase the level of eHealth literacy, it is furthermore important for organizers and participants to be able to evaluate to the progression of the learners.In response to this need, Norman and Skinner developed eHEALS, an instrument to measure eHealth literacy (Norman & Skinner, 2006a).eHEALS is an eightitem questionnaire that is based on the conceptual model of eHealth literacy consisting of six domains, which are each divided into one analytical area and one contextual area.The analytical area consists of information literacy, media literacy, and traditional literacy.A contextual area consists of health literacy, computer literacy,and science literacy (Norman & Skinner, 2006a).Analyses of internal reliability have shown good results, but in a study from 2011 van der Vaart et al. disputed the validity of eHEALS as no correlation was found between eHEALS scores and scores on actual performance on health-related Internet tasks (van der Vaart et al., 2011) In response to these data, Norman suggested that the emergence of Web 2.0 has changed the way users interact with technology.Norman suggested that eHEALS should be adjusted in response to this development, since new digital solutions and the increased use of social media may challenge how eHealth literacy is measured meaningfully in both young and elderly persons.This called for new ways to measure eHealth literacy (Norman, 2011).This technological evolution has not only influenced the concept behind eHEALS and how we understand digital literacy, health literacy, and the other four literacies in Norman and Skinners concept.It questions whether we should create a new understanding, because the concept of health literacy has changed from being mainly a functional approach (Murphy, Davis, Long, Jackson, & Decker, 1993;Parker, Baker, Williams, & Nurss, 1995) into becoming a new multi-dimensional concept (HLS-EU Consortium, 2012;Kickbusch, Pelikan, Apfel, Tsouros, & World Health Organization, 2013;Osborne, Batterham, Elsworth, Hawkins, & Buchbinder, 2013), which may embrace domains such as classical literacy, media literacy, information seeking literacy, and science literacy, with only the computer itself and areas of media literacy left as a digital or technology literacy domain.
We have used this simplified approach in order to create a strategy to find studies measuring eHealth literacy, digital health literacy, or technology health literacy, but we have also included studies that measure health literacy and digital literacy as we find these two literacies to be the best proxies of eHealth literacy.
The purpose of this systematic review is to identify measurements of eHealth literacy and describe the conceptual models they are based on.Furthermore to illustrate areas of application, and how the measurements have provided knowledge or evidence about the participants basic state, progression during interventions or relations between clinical conditions or outcomes and eHealth literacy.Some knowledge already exists on eHealth literacy measurement.There have been published specific studies, and supplementary to these, systematic reviews have been conducted within the field of eHealth literacy.In 2011 a systematic review aimed to evaluate whether college students are to be considered an eHealth literate population (Stellefson et al., 2011).In 2014 Watkins and Xie (2014) published a systematic review on eHealth literacy interventions among seniors.Furthermore, in 2012 a review was conducted on health literacy screening tools for an eHealth setting, which included eHEALS as only measurement for eHealth literacy (Collins, Currie, Bakken, Vawdrey, & Stone, 2012).To our knowledge our review that describes available instruments for measuring eHealth literacy and their results across different populations will be the first of its kind.

Methods
To identify measurement tools for eHealth literacy we conducted a systematic review.Our review follows the 2009 Preferred Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009).PRISMA consists of a 27-item checklist, which is found in the article by Moher and on the PRISMA website (http://prisma-statement.org).
Our review includes measurements of eHealth literacy.Since the concept of eHealth literacy was not introduced until 2006, we have also included the two core components health literacy and digital literacy as search terms in order to identify studies, in which health literacy and technology-related literacies were used as a measure for what could be described as eHealth literacy.
eHealth literacy is to be understood in a broad term with origin in Norman and Skinner's conceptual model (Norman & Skinner, 2006a).This will include dual tools that comprise health literacy and digital literacy.Measurements must include competencies, and both the health and digital part of the measurement should be thoroughly uncovered and thus not only consist of health literacy in a digital setting or vice versa.An example of tools that we do not include in our study is the Research Readiness Self-Assessement tool, which primarily focuses on information literacy in a health and digital context (Hanik & Stellefson, 2011;Ivanitskaya et al., 2010).Similarly, we do not include questionnaires with a content focused on evaluating information in a digital setting, e.g. the questionnaire by Peterson-Clark that measures pharmacists' online information literacy (Peterson-Clark, Aslani, & Williams, 2010).
Articles are included if they report the development of an eHealth literacy tool or the use of an already existing tool.Included articles must be available in full text, published in peer-reviewed journals, and in English.We do not include systematic reviews, editorials, study protocols, and cases.
Our actual search was conducted in September 2015 in nine databases.As eHealth literacy is spread within technology, medicine, psychology, and sociology, we chose the databases to cover all of those.We were inspired by previous systematic reviews conducted on eHealth literacy and the chronic care model (Gammon, Berntsen, Koricho, Sygna, & Ruland, 2015;Stellefson et al., 2011).Our search-databases were ERIC, PsycINFO, PubMed/Medline, CINAHL, Embase, Cochrane Library, IEEE, ACM Digital Libraries, and Scopus.
The initial search was performed using the following search string: (("eHealth literacy" OR "electronic health literacy" OR "e-health literacy" OR "health technology literacy" OR "digital health literacy") OR (("health literacy" OR "Health Information literacy") AND ("Digital literacy" OR "technology literacy" OR "computer literacy" OR "Internet literacy"))) AND (scale OR questionnaire OR survey OR measure OR Test OR assignment OR task) In some databases this initial search string was too long to be directly implemented in the search fields, and as result we divided it into two: ("eHealth literacy" OR "electronic health literacy" OR "e-health literacy" OR "health technology literacy" OR "digital health literacy") AND (scale OR questionnaire OR survey OR measure OR Test OR assignment OR task) (("health literacy" OR "Health information literacy") AND ("Digital literacy" OR "technology literacy" OR "computer literacy" OR "Internet literacy")) AND (scale OR questionnaire OR survey OR measure OR Test OR assignment OR task) When we performed the search in PubMed, it returned 571 results, which was more than we expected.A search on individual elements of the search string identified no results for "Health technology literacy" and "Digital health technology", in which case the search engine returned similar results where all three words in each of the terms were present but not in that exact order.We removed the two search terms from the search string to avoid a mix of methods across databases.The search string was examined in the other databases, but the search results of those did only include exact matches for the search string.

Results
The initial search returned a total of 526 articles.All titles were screened in relation to inclusion criteria, and 313 articles were excluded.Abstracts of the remaining articles (n = 213) were read, which lead to exclusion of a further 132 articles.Full text was reviewed for the remaining 81 articles and in this process a further 3 articles were excluded.33 articles that contained potential tools for measuring eHealth literacy were read by both authors.The results were compared and discussed in a session, which identified 8 different tools for measuring eHealth literacy.45 articles were read to identify how eHealth literacy had been measured.The final review includes a total of 53 articles, of which 8 published articles examine tools for measuring eHealth literacy, and 45 articles comprise studies using the eHealth literacy measurement eHEALS or validations of eHEALS translations.
Out of 8 identified tools 3 of these are concept-based tools for measuring eHealth literacy (Hsu, Chiang, & Yang, 2014;Koopman, Petroski, Canfield, Stuppy, & Mehr, 2014;Norman & Skinner, 2006a), and 5 are dual tools measuring eHealth literacy using a combination of health literacy and digital literacy.The eHealth Literacy Scale (eHEALS) was used in 45 studies and the only tool used to measure eHealth literacy in more than one study.

Concept-based tools eHealth literacy scale (eHEALS), published in 2006
Norman and Skinner published the first tool for measuring eHealth literacy, the eHealth Literacy Scale (eHEALS).This scale is based on the Lily-model that comprise of six subliteracies that altogether form eHealth literacy (Norman & Skinner, 2006a;2006b).
For the questionnaire they developed an initial item pool on the basis of the conceptual model.The items were reduced through an iterative process, during which they were sent to the authors' colleagues working within eHealth for review and comments.Youths aged between 12 and 19 years were given the items to test general readability, wording, and relevance.A pilot testing was conducted in which 89 youths (aged 14 to 24 years) completed the survey and provided comments on the items.The eHEALS was subsequently modified according to comments and consisted in its final form of eight items.These eight items are scored on a 5-point Likert scale ranging from "strongly agree" to "strongly disagree".Two additional items regarding perceived usefulness and importance of health related information on the Internet were published together with the eight-items scale.(Norman & Skinner, 2006a).
Their validation was performed in a population of adolescents (aged 13 to 21 years) from 14 secondary schools in a large Canadian city.The validation of eHEALS was part of a larger study with focus on smoking cessation.Data was collected at preintervention, immediately after intervention, at 3 months' follow-up, and at 6 months' follow-up.664 participants completed the surveys (mean age 14.95 years, 370 boys and 294 girls, high number of regular information technology users) (Norman & Skinner, 2006a).
Analysis was performed on internal consistency and factor analysis.Coefficient alpha was reported 0.88 with item-scale correlations in the range of 0.51 to 0.76.A principal components analysis was performed and produced a single factor solution with eigenvalue -4.479 (56% of the variance explained).Factor loadings ranged from 0.60 to 0.84 among the eight items.Analysis of test-retest reliability was performed using Pearson product moment correlation.The eHealth literacy scores were modestly correlated between administrations of the eHEALS ranging from r-0.49 to 0.68.The intra-classcorrelation between the different scores was 0.49, suggesting that the eHEALS hasd modest stability over time (Norman & Skinner, 2006a).

eHealth literacy scale (eHLS), published in 2014
Hsu, Chiang, and Yang published a study in 2014 on the effect of individual factors on health behavior among college students and the mediating effects of eHealth literacy among students in Taiwan (Hsu, Chiang, & Yang, 2014).
It was hypothesized that eHealth literacy might mediate the association between demographic factors and health behavior.This study used the integrative model of eHealth use (IMeHU) as a framework for examining the association among individual factors, eHEalth literacy, and health behavior (Hsu, Chiang, & Yang, 2014).
The instrument for measuring eHealth literacy was developed for this specific study.The eHealth Literacy Scale (eHLS) was developed as a 12-item questionnaire, which has in three dimensions: functional, interactive, and critical eHealth literacy dimensions.Each dimension is evaluated in respect of four items, and the participants rate their practice or belief on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree).The authors mention Ishikawa's health literacy measurement, eHEALS, and Ghaddar (use of eHEALS among adolescents) as primary inspiration (Ghaddar, Valerio, Garcia, & Hansen, 2012;Ishikawa, Takeuchi, & Yano, 2008;Norman & Skinner, 2006a).Due to the structure of the questionnaire, it is considered havinge its primary inspiration from Ishikawa, which in this review will classify it as a new measurement.The measurement model was examined using an Amos 6.0 confirmatory analysis, and a review of the fit indexes revealed a chi-square/df value of 3.02, a goodness of fit index value of 0.95, and a root mean square error of approximation (RMSEA) value of 0.06.The chi-square test was significant (χ2 51=153.8,P<.001) (Hsu, Chiang, & Yang, 2014).
A total of 625 questionnaires were distributed to universities in all regions of Taiwan (distributed corresponding to the proportion of university students in each of the regions).525 usable questionnaires were collected with an effective response rate of 84% (Hsu, Chiang, & Yang, 2014).
Outcomes showed that eHealth literacy levels were higher among students with better perceived health status, who majored in medical fields, or who had great concern for their own health.Critical eHealth literacy tended to be higher among students who frequently engaged in health-related discussions.Furthermore critical eHealth literacy positively predicted all three dimensions of health behavior.Functional health literacy positively predicted eating and exercise behavior.Interactive eHealth literacy did not predict any of the health behavior dimensions.Both functional and critical eHealth literacy had a mediating dimension (Hsu, Chiang, & Yang, 2014).

PRE-HIT, published in 2014
The PRE-HIT instrument for measuring eHealth literacy was developed through an iterative process that aimed to develop an instrument with a broader scope than eHEALS (Koopman et al., 2014).Four focus group interviews were conducted with patients with chronic conditions (diabetes, hypertension, heart failure, or coronary artery disease).Grounded theory methodology was used to analyze the output data.A review of existing instruments within each of the themes from the focus groups was conducted to examine different approaches to covering the domains.This lead to 98 candidate items, and the 53 best were selected for the candidate questionnaire.All items were scored on a 4-point Likert scale ranging from Strongly disagree to Strongly agree.The questionnaire went through cognitive interviews and was adjusted accordingly.The questionnaire was subsequently administered to 200 respondents, and 195 responses were collected (Koopman et al., 2014).
The participants all had chronic conditions, their mean age was 54 years (range between 20 and 86 years), and all were ambulatory patients in one of 6 family medicine clinics in Missouri, USA.
Exploratory factor analysis identified 8 strong factors which were named by the investigators: Health information need, computer/internet experience, computer anxiety, relationship with doctor, cell phone expertise, Internet privacy concerns, and no news is good news.The candidate questionnaire consisting of 53 items was reduced to 28 items sorted into the above 8 factors.Cronbach's Alpha was reported for each factor ranging from 0.57 to 0.87.Test-retest was conducted on a selected sample 3 months after initial sample.Reliability for test-retest ranged from 0.60 to 0.85 for the 8 subscales/factors (Koopman et al., 2014).PRE-HIT is based upon the groundwork of eHEALS, but expands the understanding of eHealth literacy with the identified 8 factors (Koopman et al., 2014).

Lin et al. published in 2014
In a study from 2014 Lin et al aimed to assess the levels of health literacy and computer skills of Chinese patients with cataract.Health literacy and computer skills are measured by means of three questions to each patient (Lin et al., 2014).
The three questions assessing health literacy were developed by Chew, Bradley, and Boyko ( 2004) and are: (1) How often do you have someone to help you read hospital materials?(2) How often do you have problems learning about your medical condition because of difficulty reading hospital materials?(3) How confident are you filling out medical forms by yourself?The questions were scored on a 5-point Likert scale.High scores indicate low literacy.The scores were summed for all three questions, and scores greater than 10 were classified as low health literacy and scores of 10 or lower as adequate health literacy.The health literacy questions were tested up against the Short Test of Functional Health Literacy in Adults (STOFHLA) that uses an AUROC curve, which ranges from 0.76-0.87(95% CI).The grouped items, including a fourth item about verbal information (BRIEF) demonstrated an AUROC curve of 0.79 (95% CI) for identifying inadequate skills.Internal consistency was reported with a Cronbach's Alpha on 0.98 (Lin et al., 2014).
Computer skills were assessed asking three questions on use of computer and internet.The three questions were: (1) Have you ever used a computer?(2) Can you turn on a computer and browse the internet?(3) Do you use search engines on the internet?Respondents who answered 'yes' to all three questions were considered to have adequate computer skills.The three questions are seemingly developed by the authors themselves (Lin et al., 2014).
The measures for health literacy and computer skills were used for the authors to argue that no gold standard exists for measuring health literacy, and in particular, it does not exist for measuring health literacy in mandarin.The simple questionnaire is quick and easy to administer.It does not allow for detailed descriptions of a person's health literacy or computer skills, but is developed to screen and differ between adequate and inadequate literacy and skills.Both parts of the questionnaire are focused on a functional level.
211 Chinese patients with cataract responded.A total of 92 (43.6%) had inadequate health literacy and 204 (96.7%) had inadequate computer skills (Lin et al., 2014).Mayberry, Kripalani, Rothman, and Osborn published in 2011 In 2011 Mayberry, Kripalani, Rothman, and Osborn published a study exploring the use of patient web portals and health information technology among a population of adults with diabetes from Tennessee, United States of America.Their aim was to investigate relationships between health literacy, numeracy, and computer literacy and the usage of patient web portals and health information technology (Mayberry, Kripalani, Rothman, & Osborn, 2011).
Health literacy was measured with the same three-item health literacy screening questions (Wallace et al., 2007) that were used by Lin et al. (2014) (see previous section).The response scale was modified from a 5-point to a 6-point Likert scale to secure consistency with other measures administered in the study.Each question was assessed according to other studies, in which a score of 1-5 indicates inadequate health literacy and a score of 6 indicated adequate health literacy levels.Each question was assessed separately.The questionnaire was supplemented with items from the Subjective Numeracy Scale (Fagerlin et al., 2007).
When assessing computer literacy, the authors used computer anxiety as an indicator of computer literacy.The study used the Computer Anxiety Rating Scale that measures comfort with computers and the ability to operate a computer (Heinssen, Glass, & Knight, 1987).Responses are given on a 6-point Likert scale.The authors used a reversed scale where a sum of item scores gives a range from 4 to 24.Low scores indicate great computer anxiety.Scores in the range of 4 to16 were categorized as low computer literacy, scores from 16 to 22 as moderate, and scores from 22 to 24 as high computer literacy (Mayberry, Kripalani, Rothman, & Osborn, 2011).59 respondents diagnosed with diabetes completed the survey, 23% were identified as having limited health literacy on a least one of the health screening items.Computer literacy scores ranged from 9 to 24 with an average score of 21.8 (Mayberry, Kripalani, Rothman, & Osborn, 2011).
Associations were found between levels of health literacy and having used a computer to research for diabetes, but not between health literacy and using a diabetes patient web portal.Furthermore, numeracy and computer literacy were not associated with usage of the patient web portal or health information technology (Mayberry, Kripalani, Rothman, & Osborn, 2011).
The used tools for measuring computer literacy and health literacy are quite simple to use.They are focused on the functional aspect.

Reininger et al. published in 2013
In 2013 Reininger et al. conducted a formative evaluation of an American type 2 diabetes prevention and control website, which included a pre-questionnaire, viewing of a website, interviews, and a post-questionnaire.Several instruments were administered to the participants, inclusive of basic information on internet use, family history concerning diabetes, internet literacy, transtheoretical model staging algorithm ptretest, diabetesknowledge questionnaire pretest, theory of planned behavior questionnaire pretest, and Short Test of Functional Health Literacy in Adults (Reininger et al., 2013).
Participants were included if they were at risk of getting diabetes.The included were Caucasian, African American, or Mexican American adults aged between 18 and 54 years, who read and spoke English or Spanish, with a body mass index greater than 24, and/or who were at risk for diabetes by family history , and who has an email address, were considered eligible for participation (Reininger et al., 2013).
Internet literacy results from the post-tests indicated that those with low internet literacy were less likely to show improved diabetes knowledge scores (Reininger et al., 2013).

Taha, Sharit, and Czaja published in 2014
In 2014 Taha, Sharit, and Czaja published a study on the impact of numeracy ability and technology skills on older adults' performance of health management tasks using a patient portal (Taha, Sharit, & Czaja, 2014).
Among the background variables were perceived health, medical conditions, and attitudes toward computers.Furthermore, the questionnaire assessed use of common technologies such as ATMs, cell phones, and computers.Users of internet and computers responded to questions concerning their frequency, duration of use, and types of use (Taha, Sharit, & Czaja, 2014).
Health literacy was measured using the Test of Functional Health Literacy in Adults (TOFHLA, 50 item reading comprehension and a 17-item numeracy component) (Parker, Baker, Williams, & Nurss, 1995).TOFHLA was supplemented with two additional scales for measuring numeracy.The objective numeracy measure developed and the subjective numeracy scale (Fagerlin et al., 2007).The two scales are significantly correlated.
Together with the above, the study administered a usability questionnaire regarding the specific patient portal in the study (Taha, Sharit, & Czaja, 2014).
The study included 51 participants (mean age 69.31), of whom 31 were female and 20 were male.Eleven participants (21.6%) reported having no experience with the internet.TOFHLA scores ranged from 59 to 99.Forty-three participants (84.3%) had adequate health literacy.One participant had inadequate and seven had marginal health literacy.The subjective numeracy scale ranged from 14 to 48 (M= 32.76) and the objective numeracy scale ranged from 0 to 11 (m= 5.28), and the correlation between the two scores small but significant.52.9% of participants could not correctly answer the majority of objective numeracy questions (Taha, Sharit, & Czaja, 2014).
Web-Use skills were measured as self-perceived understanding of six items related to computer use.The respondents would rate their understanding of each item (i.e."Operational system") on a 4 point scale ranging from very well to not at all (Hargittai & Hsieh, 2012).
A discrepancy between health literacy and numeracy skills was found.Internet skills and numeracy were determined to have a significant impact on the performance of tasks (Taha, Sharit, & Czaja, 2014).
No data on validity or reliability are published together with the study.The STOFHLA, the subjective numeracy scale, the objective numeracy scale, and internet literacy have all been published with satisfactory measures for reliability and validity.Questionnaires were sent to a random sample of 484 patients of a large hospital's rheumatology clinic: response was 47% (n = 227).The questionnaires comprised four parts: socio-demographics and health characteristics, health literacy, general and health-related internet usage, and intention to use different services on a rheumatology IHCA (van der Vaart, Drossaert, Taal, & van de Laar, 2011).
Health literacy was measured using a scale by Ishikawa (Ishikawa, Takeuchi, & Yano, 2008), which is a 14-item scale consisting of three subscales: functional health literacy (five items), communicative health literacy (five items,) and critical health literacy (four items).Items of the original scale were translated into Dutch according to World Health Organization (WHO) guidelines.Cronbach's alpha was 0.86, 0.86 and 0.78 in the translated scales and corresponded to alpha-scores of the original version of the subscales (Ishikawa, Takeuchi, & Yano, 2008).
Internet use was measured by asking patients if they had Internet access in their home and, if so, how much on a five-point scale ranging from one (daily) to five (almost never) they used the Internet.Health-related internet use was measured by questions about 18 items on use of different existing online applications for information (e.g.searching for different kinds of information on rheumatology), communication (e.g.reading and posting on patient support group forums, asking questions to a physician, or making an appointment) and participation in own health (e.g.filling out self-tests and monitoring symptoms) (van der Vaart, Drossaert, Taal, & van de Laar, 2011).
In a range of 1-4, the mean score for functional health literacy was 3.4, communicative health literacy was 2.8, and critical health literacy was 2.0.Outcomes of analysis showed that the intention to use the rheumatology IHCA was not correlated with socio-demographics or health literacy scales.High levels of critical health literacy among the patients were not associated to high use of e-consultation, peer support communication, and access to their electronic medical record (van der Vaart et al., 2014).
The questionnaires in this study provide a detailed profile of participants' health literacy levels.The digital part of the tool is equally detailed, but oriented towards specific tasks and the use of digital technologies in relation to health.

Use of eHEALS
In this review eHEALS showed to be the only tool for measuring eHealth literacy used in more than one study.The eight-item questionnaire was originally developed in English.It has since been translated into Portuguese, Dutch, German, Japanese, South Korean and Spanish (Tomas, Queiros, & Rodrigues, 2013;Mitsutake, Shibata, Ishii, Okazaki, & Oka, 2011;Paramio Pérez, Almagro, Hernando Gómez, & Aguaded Gómez, 2015;Soellner, Huber, & Reder, 2014;van der Vaart et al., 2011).All the mentioned translations produced high values for reliability and internal consistency.Validation of the Dutch translation of eHEALS included a comparison of eHealth literacy scores and performance tests, and these results showed no correlations.Additionally no correlations were found regarding education and age (van der Vaart et al., 2011).41 articles were identified as using eHealth literacy.These are divided into three categories:


Studies measuring eHealth literacy as a baseline or background variable (26 studies)


Studies measuring eHealth literacy's effects on health outcomes (5 studies)


Intervention studies, where the aim is to improve eHealth literacy or implementation of an eHealth solution, with eHealth literacy as a measure before and after intervention (10 studies) Table 1 (See Appendix I) provides an overview of all 41 studies using eHEALS with a summary of each study and outcomes.
The original validation of eHEALS was conducted in a population of college students.Since 2006 the eHEALS has been used on different populations.A validation was conducted on eHEALS in an older population, which showed satisfactory results for reliability and validity, see Table 1 (Sheng & Simpson, 2013).
Very few studies have explored eHealth literacy levels' impact on health outcomes, but results from a Japanese study showed that high eHEALS scores increased the likelihood of participants to undergo Colorectal Screening for cancer, see Table 1 (Mitsutake, Shibata, Ishii, & Oka, 2012).

Results of measuring eHealth literacy
Eight tools for measuring eHealth literacy were identified in this review.Three of them were concept-based tools with the specific aim to measure eHealth literacy, while the remaining five were dual tools that used a combination of health and digital literacy in their studies.eHEALS was used as questionnaire in 41 identified studies, and was the only tool used in more than one study.The tool was originally validated in a population of adolescents which has later been supplemented with a validation of the tool in an older population (Norman & Skinner, 2006a;Sheng & Simpson, 2013).In general, the eHEALS shows satisfactory results concerning measures for reliability and validation, but has been criticized for not being valid when compared to actual performance tests (van der Vaart et al., 2011).Most studies have used eHEALS as a baseline measure and to establish eHealth literacy levels in different populations.Few have used eHEALS to study impact of eHealth literacy on health outcomes, but a study has found an association between high eHealth literacy levels and the likelihood of participating in screenings for colorectal cancer in Japan (Mitsutake, Shibata, Ishii, & Oka, 2012).Furthermore studies have been conducted with the aim of increasing eHealth literacy through interventions and successfully measured significant improvements between baseline measure to follow-up (Paek & Hove, 2012;Xie, 2011a).

Tools for measuring eHealth literacy
This review includes health and digital literacy in the search strategy to explore measurements published prior to the definition of eHealth literacy.Despite a broad scope in search strategy, Norman and Skinner's eHEALS is still the earliest work that shows up in literature.None of the seven tools in this review are dated earlier than 2010, which leaves a gap from the publication of eHEALS in 2006.The gap from 2006 to 2010 could suggest that the implementation of the eHealth literacy definition, concept, and tool was not thoroughly implemented and anchored within the fields of eHealth.It could also suggest that eHEALS does not meet the need for measuring eHealth literacy, and after eHEALS was published, studies have chosen to use other measures when looking into digital health information and use.Several of the studies using eHEALS have chosen to supplement it and administer further questionnaires on digital literacy and health literacy (Ghaddar, Valerio, Garcia, & Hansen, 2012;Xie, 2011a;Yee et al., 2014).Furthermore, studies have chosen to exclude items of the original eHEALS and supplement it with elaborated items (Duplaga, 2015;Lam & Lam, 2015;Neter & Brainin, 2012).This may indicate that eHEALS has not been considered an adequate instrument for uncovering participants' abilities to interact with digital health technologies.
The identified dual tools did not necessarily intent to measure eHealth literacy, but ended up measuring the main components of the concept.Several of the dual instruments found are focused on screening for health and digital literacy (Lin et al., 2014;Mayberry, Kripalani, Rothman, & Osborn, 2011).This indicates a need for tools which can easily be administered, and even though they may not provide a detailed description of the individual's literacy within the two domains, they will be sufficient for differentiating between adequate and inadequate literacy.The eHEALS is easy to administer, but it provides a concept-based measure for eHealth literacy without pinpointing whether inadequate eHealth literacy is a result of insufficient health literacy, digital literacy or a combination hereof.
The eHLS and PRE-HIT do not offer tools for screening, but instead elaborate on the eHealth literacy concept.The eHLS is inspired by the development within the health literacy domain and has transferred the new and broader understanding of health literacy in an eHealth literacy context.The developers of PRE-HIT acknowledges eHealth literacy as an important factor in understanding user needs in a digital healthcare setting, but they stress that the original eHealth literacy concept should be broadened to be suited for measuring in a self-management context.Since none of the identified tools have been used in more than one study, there is a need for further investigation and comparison of the three different approaches in order to elaborate tools for measuring eHealth literacy and short tools that secure easy screening processes in practice.
Appendix I. Informal caregivers were more likely to send an e-mail with links to health information, while the professional caregivers were more likely to print out health information to the care recipients.Reported Cronbach's alpha was 0.82 Only 4 items.

Brown & Dickson, 2010
To explore e-literacy among healthcare students A class of first-year master-level occupational health students.
eHEALS was distributed as a learning activity.
Results showed that the student reported confidence in their ability to critically appraise internet information.They were less confident about the ability to use the found information to make decisions without consulting a healthcare provider.

Hu & Haake, 2010
Predictors of Internet-Based Diagnoses accuracy 209 participants in an online survey.
The following hypothesis was made: The higher the level of a participant's eHL the more likely that the participant will make an accurate internet based diagnoses was not supported.
eHealth literacy was not found to be a significant predictor of diagnosis accuracy.Knapp et al., 2011;Knapp, Madden, Wang, Sloyer, & Shenkman, 2011 Assess the eHealth literacy levels of internet users among parents of children with life-threatening illnesses.
129 particpants.English and Spanish speaking parents.
Cross-sectional, telephone survey of parents whose children are in a pediatric palliative care program in Florida.Each item in the eHEALS was scored 1 if the respondent agreed, and 0 otherwise and the total score was the sum of binary responses for all the items.
The statement "I have the skills I need to evaluate the health resources I find on the Internet" had the highest level of agreement (79%).The two statements that parents had the highest level of disagreement with were related to confidence in using information received from the Internet to make health decisions (24%) and ability to distinguish between high-and lowquality information (14% No results are published on the specific outcome of eHEALS in their population, only that it is a bit higher than the general population.

Britt & Hatten, 2013
To explore whether need for cognition moderates the relationship between eHL and seeking health information online.To tests eHEALS and its connection to need for cognition.

Online survey.
Individuals with high levels of eHealth literacy were more likely to seek out health information online and had higher Need For Cognition (NFC) scores.NFC was not found to be a moderator between eHealth literacy and online seeking behavior.The authors suggest that the results point toward eHEALS as a measure for self-efficacy instead of a pure measure of literacy.

Choi & Dinitto, 2013
To explore internet use patterns, eHL and attitudes toward computer use among low-income homebound older adults.
All were part of a home delivered meals program for lowincome, disables and homebound adults.Central Texas.
Face-to-face or telephone surveys.Mean eHEALS scores were 3.53 for the younger group and 3.22 for the older group.eHEALS scores were negatively associated with age and positively associated with frequency of internet use for both age groups.In the older group depression was negatively associated with eHEALS scores.Cronbach's alpha: 0.93.

Sheng & Simpson, 2013
Explore health information orientation, internet knowledge and eHealth literacy, and their effects on seniors' use of technology for finding health information.
Surveys were distributed in South Texas, and respondents were responsible for returning them.Six hypotheses were proposed, and data was analyzed using multiple regression, simple regression and general linear models.
Results showed that eHealth literacy positively influenced the likelihood of using the Internet health information.eHealth literacy was found to have a partial mediating role, where Health information orientation and internet knowledge affect the likelihood of using internet for health information through eHealth literacy.Reported Cronbach's alpha was 0.97.Factor analysis performed using principal components analysis.

Chen & Lee, 2014
To examine the mediating effect of eHealth literacy on eHealth behaviors among college students.
College students.N=540 at a public university in Southwest U.S.
Online survey, where 4 of 8 eHEALS items were used to examine eHealth literacy.Item four was excluded before the final analysis due to low factor loading.
Results showed that eHealth literacy had a positive effect on informational eHealth behaviours.A mediating effect of eHealth literacy was found between mental health status and eHealth behaviours.Reported Cronbach's alpha was 0.84 (three items).

Cho, Park, & Lee, 2014
Cognitive factors of using health apps: systematic analysis of relationships among health consciousness, health information orientation, ehealth literacy and health app use efficacy 765 participants, aged 19 to 59 (mean 37.1 years).50% male.From metropolitan areas, middle-size cities and rural areas.Majority of participants had a college or high school degree.
Online surveys collected in South Korea.Only for respondents using mobile health apps.Conducted in Korean.
Participants with higher educational backgrounds tended to have higher levels of eHealth literacy.Men reported higher levels of eHealth literacy.No direct effect of health information orientation and eHealth literacy on the extent of health-app use.Rather these two factors were mediated by health-app use efficacy.Reported Cronbach's alpha was 0.85 Milne et al., 2014 Predictors of high eHealth literacy in primary lung cancer survivors.

participants.
Survivors of lung cancer.Toronto, Canada.
Answers for each item in eHEALS were categorized into 3 response groups: agree (eHEALS = 4 or 5), undecided (score 3), disagree (score 1 or 2).28 (33.7%)perceived themselves to have high or adequate eHL by agreeing (score 4 or 5) to at least 5 of 8 eHEALS items.Fifty five (66.3%) perceived themselves to have low or inadequate eHealh literacy.There was no statistically significant difference or trend between the high and low gorup in regards to age, gender, living at home, overall health or overall quality of life Robb & Shellenbarger, 2014 Use eHEALS to examine college students' ability to find and appraise electronic health information.

Survey.
Mean eHEALS score was 34.8 (range 24 to 40).Results reported as mean per item.No significant difference between males and females were found.Furthermore no significant differences between eHealth literacy and age, race, class standing, college major, final course grades, use of the Internet and perceived importance of the Internet.Results indicated there was a statistically significant (F (2, 56) = 4.3, p = .018)difference in eHealth literacy scores for participants who rated the Internet as useful (M = 35.7)and those who were unsure (M = 31.7).Reported Cronbach's alpha was 0.89 Strekalova, 2014 Using health information orientation to explore audience segmentation.
Online survey posted on blogs and social media.Self-reported health literacy was assessed using eHEALS.Other instruments measured Health information orientation and objective health literacy.
The study concluded that objective health literacy, health information orientation and eHealth literacy can provide a basis for the segmentation of online health information audiences.Reported Cronbach's alpha was 0.92 Only 10 completed the follow-up survey.No significant differences in subjects' baseline assessments of their internet abilities by age race or education.The increase in self-assessed skill level was statistically significant for all eight items of eHEALS.Scores for the 3 month follow-up survey remained higher than preintervention scores for most items.Preintervention compared to follow-up showed that item 1-3 were the only ones with a significant improvement.Hove, Paek, & Isaacson, 2011 Using adolescent eHealth literacy to weigh trust in commercial web sites 182 middle schoolers, USA.62,4% female and 90,1% Caucasian.The two additional questions for eHEALS were included as part of the questionnaire (now 10 items).Pre-survey mean = 3.44 and post-survey mean 3.58.Reported Cronbach's alpha was 0.85 (pre-survey) and 0.88 (post-survey) The group who had used the interactive tool, correctly answered a significantly proportion of questions.Both after and follow-up.eHL was not associated with a differential benefit from the educational intervention.Mean eHEALS score 30.Improvement in all 8 items of the eHEALS.Results showed that increased confidence lead to increased access to contemporary, reliable and important health care information on the Internet, in addition to clinicians adopting email as a regular method of communication.
van der Vaart, Drossaert, Taal, and van de Laar published in 2011 Van der Vaart et al explored current disease-related Internet use and intentions to use various online support services on a hospital-based Interactive Health Communication Application (IHCA) of patients with rheumatic diseases.The study furthermore examined which variables are associated with the intentions to use different services (van der Vaart, Drossaert, Taal, & van de Laar, 2011).

Table 1
Overview of studies using eHEALS Analyses showed significant improvements of eHL between presurvey and post intervention.Involvement in training session and expectations on it being beneficial for the respondent, were related to greater improvement of eHL.Social influences and social cognitive factors play an role in predicting and explaining the change caused by the intervention on adolescents' eHL.Study design with experimental and control group.eHEALSwas measured pre-and postintervention.The control group only received information on health literacy and was guided to locate a health web site.Experimental group watched the eSEARCH web tutorial and use the steps from eSEARCH tool to enter a health website.The experimental group showed improvement in all domains of perceived eHealth literacy.Statistically significant changes were also found in the control group, but improvement was more likely in the experimental group.