cpgQA: A Benchmark Dataset for Machine Reading Comprehension Tasks on Clinical Practice Guidelines and a Case Study Using Transfer Learning.

Biomedical machine reading comprehension (bio-MRC), a crucial task in natural language processing, is a vital application of a computer-assisted clinical decision support system. It can help clinicians extract critical information effortlessly for clinical decision-making by comprehending and answering questions from biomedical text data. While recent advances in bio-MRC consider text data from resources such as clinical notes and scholarly articles, the clinical practice guidelines (CPGs) are still unexplored in this regard. CPGs are a pivotal component of clinical decision-making at the point of care as they provide recommendations for patient care based on the most up-to-date information available. Although CPGs are inherently terse compared to a multitude of articles, often, clinicians find them lengthy and complicated to use. In this paper, we define a new problem domain - bio-MRC on CPGs - where the ultimate goal is to assist clinicians in efficiently interpreting the clinical practice guidelines using MRC systems. To that end, we develop a manually annotated and subject-matter expert-validated benchmark dataset for the bio-MRC task on CPGs - cpgQA. This dataset aims to evaluate intelligent systems performing MRC tasks on CPGs. Hence, we employ the state-of-the-art MRC models to present a case study illustrating an extensive evaluation of the proposed dataset. We address the problem of lack of training data in this newly defined domain by applying transfer learning. The results show that while the current state-of-the-art models perform well with 78% exact match scores on the dataset, there is still room for improvement, warranting further research on this problem domain. We release the dataset at https://github.com/mmahbub/cpgQA.