A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis

Background: We designed and implemented the RA2-DREAM Challenge, an international, crowdsourced competition to catalyze the development of a rapid, accurate, machine learning method to quantify radiographic damage of rheumatoid arthritis (RA) to accelerate clinical research and improve management of disease outcome in real-time manner.Methods: Existing radiographic images and expert-curated Sharp/van der Heijde (SvH) scores from two NIH-supported clinical studies (674 sets from 562 patients) were divided into training, leaderboard, and test datasets. Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1, SC1), joint space narrowing (SC2), and erosions (SC3). Containerized codes/model submitted by participants were compared to SvH scores (the gold standard) and a baseline model we created as a comparator. Teams’ performances were ranked using weighted root mean square error (RMSE). The robustness of each algorithm was assessed using bootstrapping and Bayes factor comparison. The performance was further evaluated with an independent validation dataset.Findings: A total of 173 submissions from 26 teams over seven countries in the leaderboard round and 13 submissions in the final scoring round were received. The top-performing algorithms were selected from lowest weighted RMSE from each subchallenge. Bootstrapping and Bayes factor comparison confirmed the robustness of the winning algorithms and the prediction concordance indices between the test and post-challenge validation dataset were 0·714 for SC1, 0·78 for SC2, and 0·824 for SC3, indicating high accuracy and reproducibility.Interpretation: This Challenge motivated the development of algorithms that provide feasible, accurate, and quick methods to quantify joint damage in RA. Ultimately, these methods can help drive real-world research studies on RA joint damage, and guide clinicians to optimal therapies to minimize joint damage by quantitative information on progression in real-time.Funding Information: We would like to acknowledge funding support from Bristol Myers Squibb (BMS) and the CTSA Program National Center for Data to Health (CD2H) supported by the National Center for Advancing Translational Sciences (NCATS) at the National Institutes of Health (Grant U24TR002306).Declaration of Interests: The following authors have financial interests in these companies: MM, Bristol-Myers Squibb; PG, Oryn Therapeutics, CannBioRx, Gilead; JCC, PrecisionProfile. All other authors have no competing interests to disclose.Ethics Approval Statement: The use of the Challenge data was approved by the UAB Institutional Review Board.