Predicting Surgical Resident Performance with Situational Judgment Tests

PURPOSE:Situational judgment tests (SJTs) have been proposed as an efficient, effective, and equitable approach to residency program applicant selection. This study examined how SJTs can predict milestone performance during early residency. METHOD:General surgery residency program applicants during 3 selection cycles (2018-2019, 2019-2020, 2020-2021) completed SJTs. Accreditation Council for Graduate Medical Education milestone performance data from selected applicants were collected in March and April 2019, 2020, and 2021 and from residents in March 2020, August 2020, March 2021, September 2021, and March 2022. Descriptive statistics and correlations were computed and analysis of variance tests performed to examine differences among 4 SJT performance groups: green, top 10% to 25%; yellow, next 25% to 50%; red, bottom 50%; and unknown, did not complete the SJT. RESULTS:Data were collected for 70 residents from 7 surgery residency programs. Differences were found for patient care ( F3,189 = 3.19, P = .03), medical knowledge ( F3,176 = 3.22, P = .02), practice-based learning and improvement ( F3,189 = 3.18, P = .04), professionalism ( F3,189 = 3.82, P = .01), interpersonal and communication skills ( F3,190 = 3.35, P = .02), and overall milestone score ( F3,189 = 3.44, P = .02). The green group performed better on patient care, medical knowledge, practice-based learning and improvement, professionalism, and overall milestone score. The yellow group performed better than the red group on professionalism and overall milestone score, better than the green group on interpersonal and communication skills, and better than the unknown group on all but practice-based learning and improvement. The red group outperformed the unknown group on all but professionalism and outperformed the green group on medical knowledge. CONCLUSIONS:SJTs demonstrate promise for assessing important noncognitive attributes in residency applicants and align with national efforts to review candidates more holistically and minimize potential biases.