Pilot Study on Using Large Language Models for Educational Resource Development in Japanese Radiological Technologist Exams

Abstract In this study, we evaluated the potential of large language models (LLMs) in the development of educational materials for healthcare professional licensure exams with multiple choice questions (MCQs) in non-English-speaking countries, focusing on Japan's Radiological Technologist National Exam. We adapted MCQs from the exam, categorizing them into image-based, calculation, and textual questions, and generated explanatory texts using Microsoft Bing's Copilot. We assessed the quality of explanatory texts on a 0–4-point scale. Our findings showed that LLMs scored highly in text-based questions, demonstrating strong capabilities in processing textual information. However, we identified significant challenges in formula construction and the calculation process associated with calculation questions. Additionally, despite LLMs' multimodal functions, image-based questions received lower scores, which suggests the need for alternative approaches to these questions. In this study, we highlighted the effectiveness of LLMs in creating educational materials for medical licensure exams in non-English-speaking contexts while also noting the challenges in ensuring students' accuracy when they independently used LLM-generated information.