Benchmarking Nlp Toolkits For Enterprise Application

Natural Language Processing (NLP) is an important technology that motivates the form of AI applications today. Many NLP libraries are available for researchers and developers to perform standard NLP tasks (such as segmentation, tokenization, lemmatization, POS tagging, and NER) without the need to develop from scratch. However, there are some challenges in selecting the most suitable library such as data type, performance, and the compatibility. In this paper, we assessed five popular NLP libraries for performing the standard processing tasks on datasets crawled from different online news sources in Malaysia. The obtained results are analysed and differences of those libraries are listed. The goal of this study is to provide a clear view for users to select the suitable NLP library for their text analysis task.