The State of the Science on Artificial Intelligence-Based Dietary Assessment Methods That Use Digital Images: A Scoping Review

Abstract Objectives Human error when estimating food intake is a major source of bias in nutrition research. This study provides an overview of literature on the accuracy of artificial intelligence (AI) methods used to analyze digital images of food compared to human coders and ground truth. Methods This scoping review included peer-reviewed journal articles reporting AI (e.g., deep learning) and image coding for food analysis. Literature was searched through August 2021 in 4 databases plus reference mining in conference papers and reviews. Eligible articles reported volume, energy, or nutrients estimated from digital food images via fully automatic AI methods. Two investigators independently screened and extracted data. Study characteristics, methods, and type of results were extracted. Results In total, 8,761 unique publications were identified; 35 papers published from 2010 to 2021 (77% after 2015) were included. Preliminary results are reported. Most papers used newly-captured images or created a digital image database (77%). Some used named image databases (e.g., Rakuten18, UEC Food-100); only ImageNet and Inselspital's dataset were used in more than one study. Most (63%) reported relying on a nutrient database to derive nutrient values; 31% specified using USDA FoodData Central databases. The most frequently reported estimation accuracy results were absolute error (AE; 46%), relative error (RE; 40%), correlation coefficient (26%), and error rate (23%). Other reported measures were Bland-Altman plots; actual and estimated values (e.g., volumes, nutrients); and tests of mean differences. Results were reported for calories (AE, 26% of all studies; RE 20%); macronutrients (AE 20%, RE 9%); volume, mass, weight, or area (AE 17%; RE 26%); and salt (AE 6%; RE 6%). Conclusions Significant resources have been devoted to investigating the ability of AI methods to conduct accurate dietary assessment using digital food images. Yet, substantial variability in the databases used and results reported prevents quantitative synthesis of overall accuracy across studies. A validated risk of bias tool is needed to compare study quality. Future research should consider using a limited number of valid databases for food images and nutrition information, and reports should at least include absolute and relative error for volume estimations. Funding Sources Supported by an NIH grant.