Evaluation of New Technology-Based Tools for Dietary Intake Assessment—An ILSI Europe Dietary Intake and Exposure Task Force Evaluation

1The opportunities provided by the internet to link large scale food and nutrient databases with automated dietary recording has led to growth in the number of online dietary assessment tools. New technologies for measuring diet can be categorized according to the type of technology being used, such as Web-based or online tools, mobile systems (apps), camera-based tools, and other developing technologies, such as consumer purchase data and wearable sensors. Traditional methods relied heavily on self-reporting of foods consumed either using food frequency questionnaires (FFQ) or with paper-based recalls or diaries. All of the traditional methods lacked accuracy as a result of problems including the ability to recall food consumed, difficulties with portion size estimations or limited food composition tables. Considerable manual input and time was required for coding and converting foods recorded into nutrients. This meant that in large-scale cohort studies it was not generally possible to collect detailed food intake information, and studies relied on food frequency questionnaire data, which is subject to greater measurement error than other self-report measures. Use of computerized tools facilitated data coding, and incorporation of the automated multiple-pass method (AMPM) standardized data collection for national surveys. New methods have allowed for an expansion and potential improvement on the traditional methods. The use of the Internet makes larger-scale collection of food and nutrient information practical with lower costs and burden for both researchers and participants. Study participants can be invited to take part in research electronically via email or text. Users of new technology tools can more easily identify foods consumed through interactive searchable databases. They can provide real-time results and feedback and can include enhanced options for portion size description, such as using digital images, and more relevant lists of branded food items.

2It is often not clear how relevant a particular dietary assessment tool is for research as a result of limited information provided on the development process and lack of validation. An evaluation of new technologies to assess diet may help understanding of their potential to replace, improve, or complement traditional methods. Due to the rapid development of new technologies, existing reviews of the area quickly become out of date, including obsolete technologies such as personal digital assistants or PDAs. Highlighting features of new technologies, such as those found in Web-based recalls or apps, in comparison with tool elements reflecting traditional approaches may help to identify techniques that can enhance dietary measurement. Recently, clear guidance in terms of dietary assessment tool choice and reporting has been published. However, guidance on the development of new tools with quality criteria for their assessment is still lacking.

3In 2016, the International Life Sciences Institute (ILSI) Europe Dietary Intake and Exposure Task Force (http://ilsi.eu/task-forces/food-safety/dietary-intake-and-exposure/) established an expert group on evaluation of new methods for dietary intake assessment. The aim of the group was to review new technologies for diet assessment in terms of features, sources and quality of data, and validity. The review presented here will help to understand the relative merits of particular new tools and applications currently available for dietary intake assessment. We have critically evaluated tools, including their sources of data, applicability for research, ease of use by different population groups, and ability to handle a wide range of foods and beverages. In a second step, we also suggest guidelines for quality standards to improve reporting of dietary intake assessment tools.

4The objectives of this paper are to: (i) report on a comprehensive review of tools for dietary assessment using new technologies which are applicable for use in research, commercial, clinical and public health contexts; (ii) to develop guidelines for quality criteria required for a good quality tool; and (iii) to make recommendations for future reporting of dietary assessment tools using new technologies.

5Comprehensive literature searches were conducted to identify articles pertaining to new technologies for dietary intake assessment using key word searches with the following inclusion criteria: (1) publications were in English, (2) articles were published from January 2011 to September 2017, and (3) sufficient information was available to evaluate tool features, functions, and uses. Various search terms were used related to dietary or nutrition surveys, nutrition assessment, and the use of technologies, including mobile apps, Web-based tools, online or Internet tools, and software. PubMed, PLOS, BioMED, Science Direct and Ovid databases were used, each with slightly different search terms (Supplemental Table S1). The searches were limited to articles published after 1 January 2011 because the field of technology development for dietary intake assessment is advancing rapidly, and tools developed prior to 2011 have been previously evaluated. Dietary assessment tools were identified, details of which were available in one or more publications.

6The Expert Group, comprised of the authors of this manuscript, identified 25 attributes related to data entry, identification and quantification of foods, customization, output, usability and validity, which were used to evaluate each dietary assessment tool (Supplemental Table S2). Under the heading of Data Entry, we assessed whether the tools relied on text entry, digital images and/or bar-code scanners, and whether they also collected information about health characteristics or physical activity. For the Identification and Quantification of Foods, we assessed whether the foods or beverages were automatically identified from an image or required manual identification, the source of food composition data used, and how the intake amounts were quantified, either by weights or household measures, or estimated from digital images. In the Customization section, we assessed whether the tool allowed the user to add missing foods, custom recipes or dietary supplements, and whether the program used machine learning to adapt the list of foods to user preferences. Under Output, we considered whether the tool provided data on energy, macro- and micro-nutrient intakes, food groups consumed, time of intake and meal name, and whether the tool generated automated reports. Finally, we assessed Usability and Validity by checking whether there were any reports of user feedback, time to complete the assessment, and whether any validation studies had been conducted.

7The features of each dietary assessment tool were assessed independently by two members of the Expert Group from details provided in the publications, and any discrepancies were discussed at the Expert Group level. If the publications identified in the searches did not provide the sufficient detail to complete the assessment, additional literature, websites, contacts with authors, or tool use itself were used to attempt to fill gaps.

8The PRISMA diagram showing the search flow and inclusion/exclusion of studies appears in Figure 1. A total of 4695 articles were initially identified. Duplicates were removed and the remaining articles screened (title and abstract) to eliminate those that were not relevant to meet the project objectives, yielding a total of 800 publications related to dietary intake databases, applications, and tools. The goal of this review was to identify unique technology-based tools for dietary intake assessment, including smartphone applications, those that captured digital images of foods and beverages for the purpose of dietary intake assessment, and dietary assessment tools available from the Web or that were accessed from a personal computer (PC). From the 800 articles that mentioned dietary assessment in the title or abstract, 151 were related to new technologies for dietary intake assessment, and of these, 66 were additional references for tools already identified. Papers describing the remaining 85 tools were reviewed in detail. A further 42 were excluded following the detailed review: 14 were deemed to be not relevant because they were editorials (n = 1), review papers (n = 4), or did not describe a new tool for dietary intake assessment (n = 9); 16 were missing sufficient detail to do our evaluation; seven of the tools were developed and reported on prior to 2011, thereby meeting our exclusion criteria; and five were eliminated because the publications referred to a tool that had been subsequently renamed. In the latter case, the updated tool name was retained for our evaluation. Consequently, we included 43 unique tools in our evaluation.

9In total, from the 43 tools identified, 33 tools were for use in research or surveillance and 10 tools intended for direct consumer use (Table 1), and since several of the attributes differed between the research/surveillance tools and those designed for consumers, we separated them. Of the 33 tools used for research or surveillance, n = 21 (64%) were Web-based to be used on a computer; n = 6 (18%) were optimized to be used on smartphones; n = 3 (9%) were for PC only (not Web-based); n = 2 (6%) used wearables for data collection and n = 1 (3%) was designed to be used on a tablet. Of the 10 tools identified for consumer use, n = 8 (80%) were optimized for smartphone use and n = 2 (20%) were Web-based to be used on a computer. Of the 33 tools designed to collect dietary data for research purposes, n = 16 (48%) were designed for adults exclusively, n = 11 (33%) were for all ages, and n = 6 (18%) were exclusively for children and/or adolescents. Of the 10 tools designed for consumer use, n = 7 (70%) were for adults exclusively, while n = 3 (30%) were designed for all ages. Among all the tools designed for research purposes, n = 17 (52%) collected dietary intake over the previous 24h using dietary recalls; n = 11 (33%) collected food records, while the rest collected intakes via food frequency questionnaires (n = 3; 9%) or imaging systems (n = 2; 6%). Of the 10 tools designed for consumer use, most of them collected food records (n = 8; 80%), while n = 2 (20%) collected food frequency questionnaires.

10Although all of these tools used technology for dietary intake data collection, not all of the tools automatically coded the intake information to generate energy and nutrients (Table 1). Of the tools assessed here, 15 of the 43 (35%) were used for data capture only and required a dietitian or a coder to enter the items and portions in another tool later to estimate energy and nutrient intakes. These are identified as “not integrated into the tool” in Table 1. Another large difference in the tools was the source of food composition data and the number of items available. Tools designed to assess food consumption frequency (Evident II, Food4Me, GraFFS, IDQC, Oxford WebQ, and WebFFQ) included 135–200 individual line items (individual foods or aggregated food categories). Those designed for children varied, with SNAP and WebCaaFE including a limited list (49 and 32 foods and beverages, respectively), while WebFR and WebDASC included a more extensive list of 550 and 1300 items, respectively. Tools that relied on national food composition tables ranged from about 1000 items to more than 45,000 if branded foods were also included (e.g., myfood24), and were largely complete with respect to nutrients. The source of food composition was reported in all but one case, but the number of foods included in the database was missing for six of the tools. The daily time to complete each tool was reported in 18 of the 43 studies. The times ranged from an average low of 14 min to as much as 45–60 min, but most tools were completed within 15–35 min.

11The use of images also differed considerably among tools. TADA, Snap-N-Eat, and DietCam automatically coded foods and beverages from digital images, and RFPM used semi-automatic coding of images to facilitate data entry. GoCARB automatically coded carbohydrate content of food categories identified from images. Chest-worn cameras, like eButton or Microsoft SenseCam, captured digital images throughout the day but required subsequent coding by nutritionists for nutrient intake estimates. Several tools, CHAT, FoodNow, NANA, NuDAM, and TECH, used digital images to enhance reporting of food intakes, along with text or voice recordings. FoodLog used images as a visual diary of food intakes for patients with diabetes, and Microsoft SenseCam used images as a memory aid for food records.

12Figure 2 compares the 25 attributes evaluated according to use in research (n = 33) vs. those intended for consumer use (n = 10). The greatest differences in summary ratings occurred in the category ‘Data entry,’ where half of consumer access tools made use of photos for data entry, compared to less than a third of tools used in research or surveillance. In addition, information on health characteristics and physical activity were more prevalent in tools for consumer access (60%, six tools), compared to only 36% (12 tools) and 33% (11 tools) of research or surveillance tools, respectively. The possibility to set personal goals was identified as a unique feature in tools for consumer access. In the category ‘Food description’ differences were observed for the automated identification of foods, in particular, with 50% (5) of consumer access tools offering this functionality, compared to only 9% (3) of research and surveillance tools. With regard to the category ‘Customization,’ research and surveillance tools had proportionally more options to add missing items, customize recipes, and report use of dietary supplements. Research and surveillance tools more frequently provide detailed information on dietary intake in the ‘Output’ category, particularly for the features ‘Food groups’, ‘Time of intake’, and ‘Meal name’, but fewer of the research tools contained integrated food databases, so lacked the ability to estimate energy or nutrient intakes automatically. In contrast, all consumer access tools we identified generated automatic reports, but only 39% (13 tools) of research and surveillance tools did so. In the ‘Usability and validity’ category, a higher proportion in tools used for research or surveillance (91%; 30 tools) have conducted validation studies, compared to 30% (n = 3) consumer access tools.

13Some type of validation study was published for 33 of the 43 new technology-based tools evaluated in this review. Seven of the tools compared energy intakes with Total Energy Expenditure (TEE) from doubly-labelled water (DLW) or accelerometers (Supplementary Table S3). In the DLW studies, energy intake estimates from the new technology tools were significantly lower than the TEE in studies using the Microsoft SenseCam, NuDAM, RFPM, and TADA (differences ranging from 750 to 3745 kJ/day (179–895 kcal), whereas a different study with RFPM was within 636 kJ (152 kcal), and two studies in children using the TECH tool were within 220–330 kJ (53–79 kcal) of TEE. Two validation studies compared new technologies with TEE estimated from accelerometer data, showing that WebFR underestimated intakes by an average of 1840 kJ (440 kcal) in children 8–14 years, and FoodNow underestimated energy by 826 kJ (200 kcal) in young adults.

14Standard methods of dietary assessment, including 24-h recalls, food records or weighed portions, were used in validation studies for 19 of the new technology tools (representing 25 individual validation comparisons), and in these studies, there was much closer agreement (Figure 3). In fact, 18 of the 25 individual comparisons were within 250 kJ (about 60 kcal) of each other when comparing the tool and the traditional method. Six of the comparisons were within 400–900 kJ (95–215 kcal), and only one had a difference greater than 1000 kJ (240 kcal) compared to the traditional method. The tools NuDAM, RFPM, and TECH were assessed using both DLW and compared with standard method of dietary assessment, e.g., 24-h recall, weighed foods, or a diary.

15Macronutrient intake comparisons were available for 22 of the 25 validation comparisons (Supplemental Table S2). Protein intake estimates were the closest between traditional and new technology tools, with 18 comparisons within 5 g of the reference (average 2.1 g). Three of the protein comparisons were between 5–9 g different from the reference and only one was >10 g. Agreement was less accurate for fat with 13 comparisons within 5 g of the reference, four between 5–9 g, and three comparisons >10 g difference. Carbohydrate estimates showed the widest variation, with eight comparisons within 5 g, six between 5–9 g, and eight >10 g.

16The remaining 10 tools were validated using some other method. For example, the portions estimated from the eButton were compared to actual volumes measured by seed displacement. WebDASC and Epic-Soft were compared with biomarker data. SNAP, SNAPA, and WebCAAFE compared reported foods and beverages against observations. Results from a study using DES were compared with results from a national survey in the same population, and DAP compared FFQs with 24-h recalls collected using the same tool. VNP was evaluated by comparing the coding of 24-h recalls with DietPro 5i, a different dietary intake coding software. Lastly, GoCARB was compared with self-estimates of carbohydrates and carbohydrate intakes calculated from weighed food samples.

17The ILSI Dietary Intake and Exposure Task Force initiated this evaluation because of the rapid emergence of technologies available for dietary intake assessment coupled with concerns about a lack of quality standards for their development. Our review was anchored by a previous review and evaluation of innovative technologies for nutritional epidemiology, which assessed publications from 1995–2011. Since that review was published, personal digital assistants (PDAs) are no longer on the market, tape recorders are no longer needed for voice recording of dietary data, and cameras are integrated into smartphones, making digital image capture of foods much simpler. We focused our review on tools identified from publications in 2011–2017, and only four tools (ASA24, Nutrinet Santé, Oxford WebQ, and RFPM) were included in both this and Illner’s previous assessment.

18There is growing pressure in the area of dietary intake assessment to improve the accuracy and reduce costs of data collection and processing. New technology tools use a variety of inputs for dietary assessment, including text, voice, digital images, and bar-code scanners. Various techniques have been implemented to enhance accuracy of portion size reporting, including automatic estimation from digital images and visualization of different sized portions on a plate, as well as the ability to report quantities by weight or common household measures. Many new technology tools, especially those designed for consumer use, provide automated feedback on the individual’s nutrient intakes or dietary patterns, which may improve dietary outcomes and promote behavior change. People are now accustomed to using technology tools, like smartphones, tablets, and computers, as part of their daily life, and usability studies indicate that many prefer technology tools for dietary intake assessment over traditional methods.

19In the meantime, a number of other reviews have been published. While we deliberately chose to focus on new technologies identified from the published academic literature, other reviews have used app-store downloads as the criteria for selection. Few of the app-store tools (4%) provided details about the sources of food composition data, and only 14% provided micronutrient estimates. In contrast, half of the consumer apps in our review used a comprehensive food composition table, and 40% reported on micronutrient intakes. It is clear from the two approaches that apps with publications are more likely to include comprehensive food composition databases and, therefore, can report on a full complement of nutrients, compared to the most popular consumer apps.

20Image capture can increase accuracy and ease reporting of foods and beverages consumed. Images were used for data capture in 13 of the tools we evaluated (nine research and four consumer-based tools), either by automatically coding food intakes, passively capturing food intake throughout the day, as a method of recording intakes, or as a memory prompt. Digital images were also used to facilitate portion size estimation in over half of the tools we evaluated (53%; 19 research tools and four consumer tools). Uses ranged from automatic estimation of food volumes from digital images to visualization of different portion sizes to improve portion-size reporting.

21Validation studies were much more commonly reported for dietary assessment tools in the research setting than for those targeted to consumers. There was very good agreement between many of these tools and their reference method, a conclusion also drawn in another previous review. We found that 30 (out of 33) of the research tools and three (out of 10) of the consumer tools conducted a validation study, although the majority of comparison methods used in validation were other self-report measures and, therefore, subject to similar errors. In 72% of the comparisons (18 of 25), the new technology was within 60 kcal of the traditional method of dietary intake assessment. The differences were somewhat wider for studies with DLW, but these differences could have been due to a variety of reasons, including estimate errors from coders manually coding from images, or because eating occasions were not reported. As pointed out previously, new technologies will not resolve all of the challenges of dietary assessment, but it is also reassuring that, in many cases, results are close to traditional self-reported or memory-based recalls, which have received recent criticism for their accuracy. Objective biomarkers of dietary intake, such as DLW, urinary nitrogen or potassium, or plasma vitamin levels, are still lacking for most tools, and care must be taken to interpret validation by other means, such as direct data entry into two comparable tools, or comparison of results from a national survey, for example.

22The technology tools we reviewed were developed for use across a wide variety of geographies, including both higher and lower-income countries. Two tools in particular were developed to facilitate interviewer-assisted data collection in lower-middle income countries, illustrating the utility of technology tools, even in countries where individuals may not have access to a smartphone, personal computer, or other technology for personal monitoring. However, technology tools will have limited use for self-monitoring in countries where smartphone or personal computers are not widely available.

23Our evaluation has several notable strengths. As new tools and technologies are constantly changing, we have updated previous reviews with new tools identified from the literature and added a comprehensive evaluation of features. We have also compared features of research-based tools with those designed primarily for consumers, highlighting differences across all of our assessment topics. However, we must also acknowledge limitations in our review. The review was completed in September 2017, and it is possible that more recent publications have not been included in our review. For example, an in-depth validation of myfood24 including biomarkers was published after our assessment was completed, and others may have been missed as well. Results from validation studies comparing new technology tools to TEE or with daily energy estimations from conventional methods studies were presented, but further assessment of the quality of those studies was not assessed. We also focused on dietary assessment, per se, and have not included other new methods for assessing intakes, such as bite counters, tools that measure chews and swallows, or wrist-tracking devices that measure feeding. It is also possible that there could be other attributes that are also important, but were not covered in this review, such as ethical issues or privacy when digital devices include other identifying features. The impact of new technologies on cost will depend on the specific study design and the tools used, and this was rarely addressed in any of the publications. Finally, the search strategy may have missed some apps if key word searches did not pick up the studies, however, we used several search engines and different key word searches to minimize this risk.

24The quality of tools cannot be assessed if this information is considered to be proprietary, or is omitted from scientific publications. Our assessment included 25 attributes in the areas of data entry, food description, customization, output, and usability/validity. Based on our evaluation of new technology-based tools for dietary intake assessment we have developed best practice guidelines for reporting on new technologies for dietary assessment (Figure 4), which add to existing STROBE-nut guidelines (referring to Strengthening the Reporting of Observational Studies in Epidemiology, for nutrition epidemiology).

25The goal of the first step is to report on the purpose of the dietary assessment tool. This depends primarily on the context in which the tool has been used. Issues related to the assessment of dietary data needed for research or surveillance purposes may differ from those needed for consumer access settings. Report what you aimed to measure, in what population, and over what period of time. In addition, the definition of the specific purpose of a tool implies the identification of the population characteristics, e.g., age, sex, health status, educational level. It is also important to inform about what level of accuracy and precision was needed. For example, if a higher level of precision was required, it may be necessary to administer repeated measurements.

26The goal of the second step is to inform about the main measurement features of a given tool. These relate to the information about individual foods (e.g., generic foods or branded products), food coding systems (e.g., LanguaL) and/or standardized food classification and description system (e.g., Food EX2), nutrients or other food components reported, the number of food items contained in the tool (e.g., comprehensive food lists or specific foods rich in a specific nutrient or bioactive component), and features of the response section (e.g., whether eating occasions or time is recorded, if food groups are included). We recommend reporting not only the source of the food composition data, but also to report the number of nutrients it contains, the coverage, and how the tool has been customized to best meet the population-specific needs.

27We recommend defining the context for the tool and report if (1) a targeted tool provides relative or absolute intake estimates and (2) whether you are estimating daily intakes, habitual total dietary intakes, or temporal intake changes. It is also important to report if a given tool queries about supplementary information on physical activity, health characteristics, or use of dietary supplements.

28The goal of the third step is to report on the selection of the appropriate platform or technology of the tool. The choice for or against a specific technology type (e.g., tablet, computer, smartphone, wearable devices or multiple systems) depends strongly on the purpose and measures’ needs. Factors affecting this step are the available resources (i.e., financial, logistical and staff conditions). The level of technology-literacy of the targeted population needs to be taken in careful consideration. Other considerations include data sharing needs (i.e., how the participant/user data are exported and to whom), data storage structure and access, statistical analysis, programming language used for scripting the tool, how the individual will access the tool, and how their privacy will be maintained.

29The fourth step is to report on the customization of the features of the tool. These features, such as the type of data entry (e.g., text, voice, image capture, barcode scanning), list of foods and source of food composition data, type of portion size estimation (e.g., standardized portions, household measures or weights, pictures, automatic food volume estimations), need to be evaluated with respect to their adequacy to capture the purpose- and measures-specific needs of a given tool. One evaluation approach is to specifically assess the completeness and adequacy of the foods/recipes included in the tool in order to evaluate whether or how missing items could be added or recipes could be customized. Furthermore, the relevance of the dietary information in the output needs to be evaluated, as well as the need to provide feedback or to set goals for self-monitoring. Overall, details of the features that can be customized should be reported, and if there are any, an individual customization protocol should be developed and followed.

30The fifth step is to report on the design and pre-test of the tool. User interface, tool format, wording and order of questions (as appropriate) as well as browsers and battery storage are likely to affect design features of the platform and technology tool. When studying culturally diverse populations, these aspects become even more important (e.g., does the wording have the same meaning in different languages). As with any dietary assessment method, technology tools should be pre-tested, ideally on a sample of subjects similar to those who will ultimately be studied. The purpose is to report on the ease of use or user friendliness and to identify questions that are poorly understood, ambiguous, or evoke implausible or other undesirable responses. We recommend reporting on the completion time and acceptability for implementing the tool. In addition, report how the tool has been validated and against what standard.

31Dietary assessment methods that utilize technology provide rapid feedback to users and offer potential cost-savings for researchers. Dietary assessment methods that utilize new technology may be more appealing and engaging than paper-based methods, particularly for children and young adults. Online methods can be deployed to large groups with minimal resources compared with methods requiring in-field researchers. In addition, many of these tools provide rapid feedback to participants that may improve compliance with diet plans or research. Connectivity enables rapid and remote interaction with the participants and nutrition professionals or researchers. Combination methods may enhance the accuracy of dietary intake reporting (such as the use of digital images to improve memory and portion size estimates).

32Many of the new technology tools assessed here showed close agreement to traditional methods of dietary intake, but gaps are wider when compared to more objective measures, like TEE from doubly-labelled water, though studies using this method are limited in number. We encourage developers and researchers to publish details about their dietary assessment tools, including those designed for consumer use, and call on the research community to evaluate the validity of the tools they create and use. While we were able to extract details about many features from the tools evaluated, it often required more than one publication to find the necessary information. We recommend that descriptions of tool development and features be clearly written in publications, covering all aspects of tool development, including data entry, food description, customization features, output characteristics, sources of food composition data, and results of usability and validity studies, following the guidance provided here.