Fast and Scalable Method To Search for Concept Occurrences in Elsevier Content

With more than 18M articles and 610 Gb worth of text in Science Direct, it may be daunting to search for occurrences of concept labels such as "underground economy", “case-law” or "P53 inhibitor". Fear no more! We put together notebooks that can perform, at scale, exact matching of any string using Aho-Corasick algorithm. In a matter of minutes, the script allows to search for millions of concepts in a large corpus of text such as Science Direct. We leveraged this concept search method to get annotations of cell line concepts in the biomedical domain and of Omniscience concepts from our corpus. The algorithm is taking advantage of Spark distributed computation for fast processing and scalability. Combined with AnnotationQuery, an open-source library developed in-house, we can then formulate complex search queries related to the newly found concept occurrences. For example, we can get relevant concepts within the same sentence or retrieve SciBERT sentence embedding for the sentence in which the concept occurs. In this presentation we will explain you how to take advantage of the notebooks for your projects and show examples of complex search queries run directly on a large body of documents.