Measuring Interrater Reliability

Interrater reliability studies (IRSs) aim to measure the extent of agreement between raters. They are performed in radiology to demonstrate adequate reproducibility of image interpretation among readers before new technologies are adopted into clinical practice. However, radiologists without formal training in statistics are generally less familiar with the underlying principles of IRSs compared with those of diagnostic accuracy studies. In addition to being infrequently reported, IRSs are often of insufficient methodological quality, especially in terms of adequate selection of subjects, raters, and reliability coefficient (1). Herein, we describe a practical, systematic, and simple approach to IRSs, adapted to a radiologic context. We aim to highlight the most important features of IRSs in radiology, while offering tips for trainees on how to approach such studies.