Multi-method appraisal of clinical quality indicators for the Emergency Medical Services in the Low- and Middle-Income Setting: The South African Perspective

Background: Quality Indicator (QI) appraisal protocols are a novel methodology that combine multiple appraisal methods in order to comprehensively assess the “appropriateness” of QIs for a particular healthcare setting. However, they remain inadequately explored compared to the single appraisal method approach. The aim of this paper was to describe and test a QI appraisal protocol versus the single method approach, against a series of QIs potentially relevant to the South African Prehospital Emergency Care setting. Methods: An appraisal protocol was developed consisting of two categorical-based appraisal methods, the Qualify tool and Rand/Appropriateness method, combined with the qualitative analysis of the discussion generated during the consensus application of each method, by a QI Appraisal Working Group. Inter-rater reliability of each individual method was assessed prior to group consensus rating. Variation in the number of non-valid QIs identified between each method and the proportion of non-valid QIs identified between each method and the protocol were compared and assessed. Results: There was mixed inter-rater reliability of the individual methods prior to the group consensus. There was similarly poor to moderate correlation of the results obtained between the individual methods (Spearman’s rank correlation = 0.42, p<0.001). From a series of 104 QIs, 11 were identified that were shared between the appraisal methods. A further 19 QIs were identified and not shared by each method, highlighting the benefits of a multimethod approach. There was little evidence to support a difference in the proportion of non-valid QIs identified between individual methods (difference=-0.03); between the Quality tool and the protocol (difference=-0.05); or between the Rand method and the protocol (difference=-0.02). The outcomes were additionally evident in the group discussion analysis, which in and of itself added further input towards understanding and appraising the appropriateness of the QIs that would not have otherwise been captured or understood by the individual methods alone. Conclusion: The utilisation of a multi-method appraisal protocol offers multiple benefits, when compared to the single appraisal approach, and can provide the confidence that the outcomes of the appraisal will ensure a strong foundation on which the measurement framework can be QI successfully implemented and employed.