Consumer Evaluation of the Quality of Online Health Information: Systematic Literature Review of Relevant Criteria and Indicators

1More than 70% of US adults search online for health information. The information found online shapes and influences consumers’ health beliefs, intentions, health behaviors, and health care decision making. Since the inception of the internet, the quality of health information has been a source of concern for stakeholders due to the unregulated nature of the medium. This concern is furthered by the fast growth of social media and user-generated content and corroborated by more than 200 evaluation studies conducted by subject experts, which collectively suggest that the quality of consumer-oriented health information on the internet varies greatly and that the overall quality was low and remains low.

2Making decisions based on low-quality health information (eg, information that is inaccurate, incomplete, or biased) may lead to harmful consequences, such as delayed treatment or extreme anxiety, and subsequently increase consumer vulnerability. Nevertheless, evaluating the quality of information has been a major challenge for online health consumers. For example, some consumers are uncertain about the accuracy, completeness, and validity of the information they encounter; some cannot differentiate between scientific facts, empirical factors, and personal opinions; and others suffer from information overload and subsequently lack the confidence and ability to evaluate information. Studies have found that compared to health care providers or information professionals, consumers tend to give higher quality ratings to health information from both traditional health websites and social media sites.

3The ability to critically evaluate the quality of health information is an important component of health literacy, which is an important determinant of health. To enhance this ability (and related skills), it is necessary to understand how consumers evaluate the quality of health information in the current internet environment. Consumer evaluation is subjective, driven by one’s information needs. Therefore, as a starting point, we adopted a broad conceptualization that defines quality through “fitness for use”: Information is of good quality when it serves users’ needs. It is worth noting that this concept of quality is described using different terms in the existing literature, including but not limited to quality, credibility, trust, reliability, believability, and usefulness. In this review, we included articles using all these terms. We chose to be inclusive, because we want to achieve a comprehensive view of the assessments that consumers perform in the process of determining whether they would be willing to use a piece of information.

4Guided by this understanding of quality, three recent systematic reviews were identified as relevant to our current research: One review focused on identifying factors that impact consumer judgment of trustworthiness and credibility of online health information, the second one identified the antecedents of trust in health information websites, and the final one reviewed the association between low health literacy and perceived quality and trust in online health information and low literacy consumers’ ability to evaluate information quality. These reviews revealed that consumers’ quality evaluation is influenced by both source- and content-related factors. Examples of source-related factors are website design (eg, layout, visual design, and interactive features), loading speed, and the authority of the owner or sponsor. Examples of content-related factors are the authority of the author, content readability, content organization, use of evidence and citations, and the appearance of advertisements. Additionally, a number of individual characteristics were identified as influencers, including demographics (eg, age, gender, and educational attainment), perceived health status, knowledge about the content, health beliefs, and level of health literacy.

5These reviews provide an informative overview of factors that influence consumer online health information evaluation behavior but shed limited light on why and how these factors influence the evaluation behavior. From the perspective of information seeking, evaluation of information is a judgment and decision-making process that precedes users’ acceptance or rejection of received information. Judgment and decision making involve applying certain criteria, principles, or standards to form evaluations. Thus, to understand consumer quality evaluation behavior, it is necessary to understand the criteria used to guide the evaluation. Among the previously mentioned systematic reviews, only one summarized the evaluation criteria reported in five studies on consumers with low health literacy. A more comprehensive understanding of the evaluation criteria is needed. This review intends to fill this gap.

6Evaluation of the quality of online health information is a process of applying criteria to evaluate information. Thus, in addition to applying criteria, we need a better understanding of how consumers perceive online information. To achieve this goal, we deliberately differentiate between two concepts: criteria and indicators. Criteria are rules or filters that people apply to an information object to assess its value or worth. Indicators, also termed cues or markers, are perceivable elements associated with an information object that allow people to reflect on the quality of the object. Criteria are abstract, reflecting one’s values and preferences and mediating information selection decisions. Indicators are affordances of information objects that trigger or support the application of the criteria. Criteria are comparatively stable, whereas indicators are amenable to change. New indicators could emerge, and old ones could disappear with the development of new technologies and design preferences.

7In this article, we focus on the following research questions: (1) What criteria do consumers use to evaluate the quality of online health information? (2) What elements of information objects do consumers use as quality indicators? (3) Which indicators convey positive evaluations and which convey negative evaluations? (4) What is the relationship between indicators and criteria, that is, what criteria do each indicator correspond to? We argue that a more comprehensive understanding of criteria used in the evaluation process can bring some clarity to the dimensions of quality perceived by online health consumers as well as their quality evaluation process. By explicating the relationship between indicators and criteria and identifying positive and negative judgments that indicators convey, the results can also inform the design of more user-friendly health information content and information systems.

8Seven online databases, including PubMed, Web of Science, PsycINFO, CINAHL (Cumulative Index to Nursing and Allied Health), Cochrane Library, Library and Information Science Source, and Communication and Mass Media Complete, were searched in July 2017 to obtain relevant journal articles. These databases were chosen because they cover major academic disciplines that study consumer online health information search, including health, information and library science, psychology, and mass communication. Keywords, including quality, credibility, trust, reliability, accuracy, readability, relevance, and usefulness were used in combination with the keywords consumer or patient and online health information evaluation or online health information assessment. After the searches, we manually screened the references to identify relevant articles and further examined the reference lists of these articles. Additionally, we examined the references cited in the three systematic reviews mentioned above and articles that cited these reviews (using Google Scholar’s “cited by” function).

9Articles meeting the following criteria were eligible for inclusion in this review: (1) The study primarily focused on consumer evaluation of health information on the internet. Health consumers include patients, caregivers, and the general public who sought or were interested in seeking health information. This focus differentiates this review from prior reviews of health care professionals or expert evaluation of online health information for consumers. Articles that focus on media other than the internet (eg, TV and radio) were excluded. (2) The study was empirical and based on direct inquiries with health consumers where criteria were described by participants and not imposed by researchers. Articles that used only predefined evaluation criteria to survey consumers or analyze their responses without allowing new criteria to emerge were excluded. We also excluded correctional studies that focus on identifying factors (eg, source expertise) influencing consumer evaluation behavior but do not provide additional results on how quality evaluation is performed. (3) The article was published after 2002, when research on consumer evaluation of online health information began to emerge. (4) The article was written in the English language. (5) The article was published in a peer-reviewed journal.

10Figure 1 shows the process involved in identifying eligible studies. Three authors (YS, YZ, and JG) reviewed a subset of the search results by reading titles and abstracts. YS and YZ both reviewed 10% of the records (256 records in total) to check the intercoder agreement in filtering potentially relevant articles (Cohen kappa=0.83). Both YS and YZ screened the full-text articles. When there was uncertainty involved in excluding a full-text article, the other two authors provided their input.

11Full text of the 37 selected articles was imported into MAXQDA 12 (VERBI Software GmbH, Berlin, Germany) for analysis. We extracted the following information: basic characteristics of the articles (eg, year of publication, country of origin, health topics, and aims of the study), research methods, sampling techniques, participant characteristics (eg, demographics and disease experiences), source studied (eg, the internet or specific health websites), and characteristics of the search tasks (eg, self-generated vs assigned) when search tasks were involved. Guided by their corresponding definitions, indicators and the corresponding criteria were extracted from the results and discussion reported in the original papers. When no clear relationships were reported (in most of such cases, indicators were reported without mentioning the criteria. For example,.com was reported as a negative indicator of quality, but criteria by which this judgment was reached were not reported), the authors of the review derived the relationships from the participants’ direct quotes reported in the original papers, the original authors’ discussion of the results, or the interpretation of the authors of the review. Indicators were further coded into positive (+, entailing positive quality judgment), negative (–, entailing negative judgment), or both (±, entailing both positive and negative judgments). When participants commented on the absence of an indicator (eg, no author credential or no advertisements), it was coded as positive if the absence implies low quality and as negative if the absence implies high quality. The criteria were also coded into the three categories based on their correspondence with indicators.

12We analyzed the basic characteristics of the included studies using descriptive statistics. The qualitative content analysis method was used to identify themes and build categories based on the extracted information concerning criteria and indicators in an iterative manner. YS coded all the articles. YZ validated the results by comparing each assigned code to the full-text of the articles. A number of group meetings were held to discuss the codes, especially relationships between indicators and criteria. Discrepancies were discussed among all authors.

13The 37 articles included in the review were published between 2002 and 2017. They originated from 8 countries, primarily United Kingdom (N=12), United States (N=11), Australia (N=4), and the Netherlands (N=3). The characteristics of each included article are summarized in Table 1.

14Focus groups (n=17), interviews (n=16), and observations of participants performing predefined (n=11) or self-generated (n=6) search tasks were the primary research methods used in the selected articles. Observations were often used with other methods including think aloud, guided interviews, focus groups, or diaries. Fourteen articles used multiple research methods.

15Twenty-one articles focused on information on a specific health condition or issue (eg, HIV prevention, diabetes, disabilities, and chronic diseases), and the remaining articles did not specify a subject focus. Twelve studies recruited patients with a specific condition, and the others recruited people who had searched online for health information (n=6) or had a strong interest in their health or a particular condition (n=5). Twenty-eight articles involved adult participants (≥18 years old), of which 10 articles also involved older adults (>64 years old). Four studies included adolescents aged 11-17 years. The number of participants ranged from 5 to 188 (median=21). In terms of sampling technique, 26 articles used purposive sampling, five used convenience sampling, and the remaining six did not report the sampling methods.

16Regarding evaluation of internet sources, 28 articles did not specify a scope. The remaining nine articles specified or preselected sources for evaluation (eg, pediatric sun protection websites, the National Institutes of Health website, the Centers for Disease Control and Prevention website, patients.co.uk, and online forums).

17Twenty-five criteria were identified (Table 2). The definitions were derived from the codes or drawn directly from the included studies.

18Among these criteria, trustworthiness, expertise, and objectivity were reported most often in the articles, followed by transparency, popularity, and understandability. Eight criteria including relevance, familiarity, accessibility, identification, believability, accuracy, readability, and currency were reported in 10-15 articles. The remaining 11 criteria appeared in less than 10 articles.

19Indicators used by consumers to evaluate the quality of online health information were related to three aspects of online information: source, content, and design. Table 3 shows their distribution across the three categories.

20About 52% of the indicators were content related, followed by design (25%) and source factors (23%); 69% of the indicators were associated with positive quality judgment, 21% were associated with negative quality judgment, and 10% could lead to both positive and negative judgment.

21Source is the entity that creates, hosts, or distributes content. A source can be a website or the owner, creator, or sponsor of the site. Six categories of source-related quality indicators were identified: site owners/sponsors, site types, disclosures, third-party accreditations, recommendations from other systems or users, and website scope. More detailed indicators reported in the included articles, their direction of influence on quality judgment (positive, negative, or both), the corresponding criteria that guide the consumers’ appraisal of the indicators, as well as the value of the criteria (positive, negative, or both) are shown in Table 4. The indicators in the tables are self-explanatory; therefore, we focus on describing the most frequently appearing indicators in the included studies and indicators that can lead to both positive and negative judgments.

22The most frequently mentioned source-related indictors were site owners/sponsors, with sites run by reputable organizations, educational and academic institutions, and medical experts and health institutions being considered more trustworthy and offering higher levels of expertise. The second most frequently reported indicators were about disclosure. Sites that disclose their motivations were highly valued, whereas a lack of a clear statement of purpose and motivation damaged trust. The third mostly frequently reported indicators were recommendations from other systems or users. High ranks in search engines and a large number of visitors or followers were viewed as indicators of high site popularity, and subsequently, high quality. In addition, sites linked from or recommended by a trusted website or trusted others (eg, health care providers, families, and friends) were considered trustworthy.

23Mixed attitudes were found toward some indicators representing site owners/sponsors. First, most participants believed that government websites (eg, National Health Service and Centers for Disease Control and Prevention) reflect high levels of expertise and good intentions; however, some consumers suspected that the information on government websites is biased due to their agendas, and some, particularly younger generations, did not identify themselves with government sources, considering them “less cool” and not relatable. Second, most people considered sites operated by local health societies to have a high level of expertise; however, some minorities and people from nonmainstream cultures (eg, aboriginal communities) were likely to question the relevance and accuracy of the information from these sites. Third, people usually considered websites owned by commercial companies less objective and trusted more websites with no commercial interests; nevertheless, popular commercial websites such as BabyCenter.com, ParentsPlace.com, and WebMD.com were favored by some people for their expertise and comprehensiveness. Fourth, a few people viewed information from pharmaceutical company websites as “official”, whereas others considered their information biased due to the financial interests involved.

24Consumers had mixed attitudes toward the website types, particularly social media sites. Some consumers favored online discussion groups, chat rooms, and listservs because they offered first-person narratives and practical information and support from peers with whom they could identify (ie, those who have similar conditions), but some disliked such sites for their lack of objectivity and expertise. Concerning Wikipedia, some people questioned its objectivity because information can be edited by anyone on the Web, but some consumers were attracted to its encyclopedic nature and comprehensiveness.

25Consumers also had different opinions regarding sites recommended by others. Some trusted a site recommended by trusted others (eg, health care providers, families, and friends); however, some consumers recognized that recommendations from other individuals may not be relevant to their situation.

26Content refers to the information contained in a source as well as the presentation of the information. Eight categories of content-related indicators were identified: substance, writing and language, presentation, references, authorship, audience, date/updating, and advertisements. Table 5 shows these indicators, the corresponding criteria that guide the consumers’ appraisal of the indicators, and their influence on quality judgment.

27The most frequently reported content indicators were about consensus among sources. Content that appears in multiple sources, be it online sources, sources in other media (eg, newspaper, television, books, and academic journals), or health care professionals, is trusted by consumers. Writing- and language-related factors were the second most frequently reported content indicators. Consumers expect high-quality information to be error free in spelling and grammar, use straightforward language, and have a clear layout. The third most frequently reported indicators were advertisements. Consumers expect quality websites to neither depend on advertisements nor seek to make a profit. Therefore, sites with advertisements were considered less objective, be it in the form of commercial links, advertisement banners, popups, or other formats.

28Consumers had mixed attitudes toward two content types: objective facts and personal experiences. Some consumers value objective facts, whereas some dissatisfy with information that contained solely objective facts, considering it unbalanced. With regard to personal experiences, some consumers favored first-hand experiences, stories, and advice (eg, recommendations for medical gadgets, meal planning and exercising, and advice on completing medical benefit forms) from people with similar conditions for their practicality, but some had concerns that personal experiences lack objectivity, especially when it merely is a personal opinion.

29Celebrity endorsement was also an indicator with both positive and negative influences on quality evaluation. Some trusted the endorsed information but others question its objectivity due to the potential financial interest involved.

30The use of medical and technical vocabularies raised contention among consumers as well. For some consumers, high-quality information was easy to understand, that is, it exhibited less use of professional medical vocabularies or provided easy-to-understand definitions of medical jargon, especially for educational and government sites; however, for others, the use of technical vocabularies demonstrated expertise and was highly valued.

31Some consumers doubted information (especially diagnosis and treatment information) authored by other unknown consumers, but others tended to trust content written by their peers because of similar demographic or health characteristics.

32For health interventions, some consumers examined the release time and perceived newer interventions to have higher quality than the existing ones because the new intervention would have undergone more testing and research, whereas others were uncertain about the reliability of newer interventions.

33Design refers to the appearance of a website or an app and the interactions that it affords. Four categories of design-related quality indicators were identified: interface design, interaction design, navigation design, and security settings. Table 6 shows the specific indicators, the corresponding criteria that guide the consumers’ appraisal of the indicators, and their influence on quality judgment.

34The most frequently reported design indicators were related to interface design, mostly visual factors, including the overall appearance of a site, the graphics it includes, and font size. Interaction design features, including links, interactive functions, and other interactive features (eg, loading time and login requirement), were the second most frequently mentioned quality indicators. Sites with robust search capabilities (eg, easy to locate and diverse search entrance), offering useful tools (eg, self-management tools), and rendering smooth user-system interaction (eg, providing links to additional relevant sources and not having pop-ups) were perceived as high quality. Navigation-related indicators such as navigation aids and site maps were the third most frequently mentioned quality indicators.

35Mixed opinions existed concerning the interactive functions of self-management and assessment tools (eg, health calculators). Some consumers valued tailored results and advice, but some questioned the accuracy and objectivity of the information generated.

36In addition to source-, content-, and design-related factors, the evaluation of online health information quality was also affected by individual factors including individuals’ personal situation, prior knowledge or experience of a source, personal knowledge and beliefs, and intuition. Table 7 shows the specific factors, the corresponding criteria that guide the consumers’ appraisal, and their influence on quality judgment.

37Individuals’ prior knowledge and experience of a source were mentioned most frequently as factors that influence quality judgment. Consumers tended to trust sites that they had experience with, because they may already know the source to be credible, have had positive experiences with it, have seen it from advertisements on other media (eg, television and magazine), or are familiar with the organization behind the source.

38The category of personal situation was the second most frequent factor. Information relevant to individuals’ search topics (eg, hormone replacement therapy), needs and goals (eg, offering easy reading level message for younger people), specific circumstances (eg, localization), and experiences and symptoms was considered to be of high quality.

39The other two categories of individual factors were mentioned with the same frequency. One category is personal knowledge and beliefs. Consumers highly valued information consistent with their own beliefs and knowledge. The other category is intuition. Some consumers undertook “subconscious filtering” to filter out potential political and gender biased information, and some consumers relied on common sense, sensation, instinct, or “gut feelings” to evaluate information.

40In this article, we reviewed 37 empirical studies that reported consumers’ accounts of how they evaluate the quality of online health information. This review extends the existing literature by making two major conceptual contributions. First, it offers a clear conceptual understanding of the dimensions of quality of online health information perceived by consumers by differentiating criteria from indicators. Second, it explicates the relationship between webpage quality indicators (webpage elements) and the quality judgment by differentiating positive and negative influences that indicators have on judgment. In this section, we discuss each contribution and then outline practical implications and limitations of this review.

41In the existing literature, quality was often defined and assessed differently. We guided the article selection for the review using a general conceptualization that defines quality as “fitness for use”. Other authors have offered more specific conceptualizations. For example, Rieh assessed quality as the extent to which users think that the information is useful, good, current, and accurate. Bates et al measured health information quality in terms of its trustworthiness, truthfulness, readability, and completeness. Benotsch et al rated the quality of health websites on five dimensions: accuracy, amount of detail, trustworthiness-credibility, relevance, and usefulness. Eastin rated the credibility of health information on three dimensions: accuracy, believability, and factualness. The lack of consistency in measuring online health information quality suggests that there is a lack of clear conceptual understanding of what information quality means to online health consumers.

42By clearly differentiating quality judgment criteria (rules that reflect notions of value and worth) and indicators (properties of information objects to which criteria are applied to form judgments) reported in the included studies, this review identified 25 dimensions (criteria) along which consumers evaluate the quality of online health information (Table 2). Because the included articles differ on aspects such as health issues of concern, participant demographics, and sources examined, this wide range of criteria reported and the uneven distribution of the criteria across the included articles suggest that consumer evaluation of online health information may be influenced by contextual factors such as user characteristics, health conditions, and online sources. In addition to these factors, the current review, consistent with prior reviews, also identified a range of individual factors that influence quality judgment behavior, such as prior experience with a source and personal knowledge and beliefs. Therefore, future studies should attempt to identify the most influential contextual factors (including individual factors) that affect consumers’ application of quality criteria to further enhance the theoretical understanding of this behavior. Empirical studies of consumer online health information evaluation should also consider these contextual factors in research design.

43Despite the wide range, however, three criteria (trustworthiness, expertise, and objectivity) were reported in 31 articles, indicating that they are used consistently across user groups, source types, and health conditions and that they constitute core dimensions of online health information quality as perceived by consumers. The fact that trustworthiness and expertise are primary dimensions is consistent with general media source credibility research. It is not surprising that objectivity, that is, whether a source or information presents objective factual or evidence-based information, is also important for health information. Three additional criteria—transparency (reported in 21 articles), popularity (reported in 19 articles), and understandability (reported in 18 articles)—are also commonly reported and could be viewed as secondary dimensions of online health information quality. These findings imply that consumers’ perceived online health information quality could be reasonably measured by a small set of core dimensions.

44Previous reviews summarized indicators used by consumers to evaluate the quality of online information. Sbaffi and Rowley further reported the direction of the effect (ie, positive vs negative) of the (design and content) indicators. However, the situational nature of the relationship between indicators and quality judgment, that is, the fact that their relationship is not one-on-one, but dependent on users’ values and the criteria applied, was not explicitly discussed. For example, government institutions, usually associated with high level of expertise and authority, are perceived by some consumers as biased sources with which they have a hard time relating. The other example is that consumer-generated content (eg, personal blogs and listserves) indicates low objectivity and low level of expertise to some consumers, but to others, it is considered highly practical and relatable. Thus, a unique contribution of this review is that it clearly maps out the direction of the impact (ie, positive or negative) of a number of indicators on quality judgment and the underlying reasons (ie, criteria) for the impact.

45The identification and differentiation of positive and negative indicators provide clear guidance for online health information designers. They can incorporate positive indicators (eg, offering authors’ credentials and presenting information in a clear and organized way) and avoid negative indicators (eg, dead links and flash media format) to offer users better information seeking experiences. The fact that the same indicator (eg, government institutions as the source owner) can lead to different quality judgment for different people suggests that designers should also carefully investigate target users’ values and the corresponding criteria that they use to evaluate health information. This calls for active user research and user involvement in the design process.

46The results of the review also have implications for consumer education. The review revealed a wide range of criteria that consumers use to evaluate the quality of online health information. Many of the criteria, such as familiarity, identification, relevance, practicality, and usefulness, are highly subjective and situational, influenced by factors such as information needs, online information search experience, and personal beliefs. In some cases, consumers assign such criteria higher priority than more objective ones such as expertise. The review also revealed that consumers use a diverse set of quality indicators. The implications of some of the indicators are not well understood. For example, some consumers believe that the appearance of copyright information or the word “clinical” indicates high information quality. Some consumers view the fact that a website passes the screening of virus/security software as an indicator of high quality. There are also consumers assuming that third-party accreditations are indicators of information accuracy, when, in fact, the guidelines that these accreditations follow do not really check for information accuracy. Consumers need education to use more objective criteria to evaluate online health information and understand the implications of a number of quality indicators.

47This review has several limitations. First, we selected only studies where consumers explicitly described their quality evaluation behavior. These studies tend not to ask consumers to rate criteria or indicators; thus, we could not identify the importance of each indicator or criterion in quality judgment. Future reviews are needed to fill this gap. Second, we did not differentiate and compare results based on observations and results drawn from verbal inquiries as few included studies did. Eysenbach and Kohler reported discrepancies between participants’ verbal accounts of what they do to evaluate health information and what they actually did in performing search tasks (based on observations). Thus, future empirical studies are needed to shed light on this gap. Third, in the coding process, we used criterion and indicator terms from the original papers, where feasible. In cases where we needed to infer criteria from indicators, we followed the mostly commonly recognized categorization by referring to prior empirical research and reviews or inferred the criteria from participants’ quotes. However, due to the different perspectives of the authors of the original papers and the inherent overlap between terms, such as comprehensiveness and completeness, our syntheses are inevitably affected by a certain degree of subjectivity. Fourth, because most studies treated the internet as one source of information without differentiating source types (eg, regular websites and social media), we were not able to identify whether the use of evaluation criteria and indicators differs by source type.

48The quality of online health information is a complex concept involving more than two dozen dimensions, as perceived by consumers. Although a set of core dimensions can be identified, the diversity involved in consumers’ use of criteria is too obvious to ignore. Further examination of contextual factors (eg, different source and user characteristics) that influence consumers’ application of quality criteria will bring further clarity to the concept. The review identified 165 indicators, to which criteria are applied to reach a quality judgment. Indicators could be source, content, or design related; they can have a positive or negative impact on quality judgment, contingent on situations and users’ values and beliefs. The identification and differentiation of positive and negative indicators along with their respective criteria can provide clearer guidance for designers of online health websites and educational interventions. Compared to experts’ evaluation, consumers’ evaluation of online health information relies heavily on peripheral cues and is influenced by various contextual factors (eg, personal beliefs and information needs). This finding suggests that current quality evaluation checklists, which are mostly based on experts’ view of quality, may not effectively serve the needs of consumers. Consumer behavior needs to be considered in the design of interventions that intend to promote quality evaluation in online searches. At the same time, it is worth noting that criteria and indicators used by consumers merit critical evaluation, as some criteria are overly subjective and the implications of some indicators are not well understood. User education is needed to address user misconceptions and the associated suboptimal evaluation behavior.