Development and interobserver reliability of a rating scale for lung ultrasound pathology in lower respiratory tract infection

The purpose of this study was to develop a severity rating scale for lung ultrasound pathology in lower respiratory tract infection based on multicenter expert consensus, and to test inter-rater reliability. Ten point-of-care ultrasound experts from three academic institutions developed the scale iteratively through literature review, expert opinion, and pilot testing. Clips were prospectively collected from adults suspected of COVID-19 using a 14-zone scanning protocol. Blinded reviewers independently rated four data subsets. The rating scale was refined through eight consensus-building discussions reviewing challenging cases from the first three subsets. The final scale consisted of a set of ordinal scores from 0 to 4, for five sonographic findings: B-lines, pleural line abnormalities, consolidations, pleural effusions, and overall lung aeration. Ratings from the fourth subset were analyzed to determine reliability based on intraclass correlation coefficient (ICC). A total of 11,126 cine clips from 220 patients were acquired. After excluding uninterpretable clips, the test dataset contained 81 clips and yielded an average ICC of 0.70 across the five sonographic findings (0.76 for B-lines, 0.52 for pleural line abnormalities, 0.71 for consolidations, 0.80 for pleural effusions, and 0.70 for overall lung aeration). Improvements in agreement were observed with each successive review session and dataset rating. The lung ultrasound severity scale established by multicenter expert consensus achieved moderate to good inter-rater reliability. The scale could be used clinically to standardize assessment of lower respiratory tract infection and in future studies to develop methods for automated interpretation of lung ultrasound pathology.