Large Language Models in Surgical Education: Do they Reach Human Level on Fundamentals of Robotic Surgery Test? (Preprint)

BACKGROUND Large language models are capable of answering questions as if they were engaged in active conversation with users. However, currently, there are no data on whether their performances will remain static or vary over time when answering questions in the medical domain. OBJECTIVE The aim of the present study was to assess ChatGPT and InstructGPT on multiple trials on the Fundamentals of Robotic Surgery (FRS) test. Additionally, different releases of ChatGPT were compared to establish whether its performance improved after retraining. METHODS We tested the performance of ChatGPT and InstructGPT on the 44 multiple choice questions of FRS didactic test, for which a pass mark requires 35 correct answers (79.5%). Seven attempts were performed using ChatGPT on the January 30, 2023 release and seven with the February 13, 2023 version. Three trials were performed with InstructGPT. RESULTS ChatGPT achieved a mean score of 64.6% and 65.6% respectively for the first and second release, without any significant difference between the two (p = 0.32). The score ranged from 54.5% to 72.7% with both versions. On baseline it achieved 54.5% in both releases, higher than InstructGPT (50.0%). The highest rate of correct answers of ChatGPT was observed for questions on team training and communication (77.5% with both releases), followed by those on the introduction of the robotic system (67.5% and 62.7 % respectively for the first and second versions), psychomotor skills (64.3% and 57.1%), and naming correctly clinical steps of a procedure of robot-assisted surgery (53.8% and 65.9%). CONCLUSIONS Even though ChatGPT did not pass FRS test in any of the 14 trials, the 72.7% score observed by the present study represents a remarkable result, taking into consideration the generic nature of ChatGPT as distinct from a domain specific LLM. This level represents the highest score by ChatGPT in a high-stake examination in medicine.