Image similarity ranking of focal computed tomography liver lesions using a 2AFC technique.

Content-based image retrieval (CBIR) for radiological images has experienced massive growth over the past two decades, and shows great potential as a tool for use in precision medicine. A recurring challenge in CBIR evaluation has been in obtaining reference sets of images from human viewers of the system. Our work seeks to determine the feasibility of creating a reference set from images ranked by similarity from human viewers of the images. We obtained 2 sets each of 10 images of CT focal liver lesions from a database of open-access publications with and without markings showing the region containing the lesions, respectively. We created 2 sets of all 45 pair-wise combinations of the images, and displayed them to 10 volunteers, of which 2 had medical training. We used a Two-Alternative Forced Choice (2AFC) paradigm to obtain complete rankings of similarity levels in these image pairs. Analysis showed that inter-reader agreement for rankings ranged from Tau=0.21-0.69 (median=0.37) for the image pairs without any markings, and Tau=0.21-0.57 (median=0.33) for the image pairs with markings. A comparison of the regions of interests drawn by the study participants outlining the lesions in images without markings showed that participants tended to agree on images containing a single focal lesion of a single density, and inter-reader agreement for image rankings in which the regions of interest agree ranged from Tau=0.39-0.85 (median=0.58). These results show that the use of image ranking using 2AFC may be a feasible method for creating reference sets for CBIR system validation.