A Real-Time Fault Tolerant and Scalable Recommender System Design Based on Kafka

A recommender system based on offline batch processing has the benefit of being precise in its computation and having a high level of fault tolerance. However, because of the huge quantity of data and computation, it cannot meet the real-time requirement. At the moment, the design of a real-time recommender system based on the mix of offline and online layers has several drawbacks in terms of summarizing results and preserving updates, and the suggested result is dependent on the offline layer. If user behavior changes rapidly and the offline layer’s result updates slowly, the final suggestion may not accurately represent the user’s changing interests. Additionally, as the volume of data grows, conventional recommender systems are challenging to keep up. To address these issues, the purpose of this article is to demonstrate how to utilize Apache Kafka to enhance and extend an existing message queue by adding a scalability component and expanding its fault tolerance capabilities. Apache Kafka is based on pub-sub messaging mechanism in which producers post data to a cluster, which customers subscribe to a particular topic in order to receive messages published under that topic. The makers of the communications send them and keep them in divisions. Load balancing is accomplished by the distribution of data across each cluster’s nodes. A partitioner is the component that allocates a message to a partition, and each producer comprises one partitioner. The primary goal of optimizing the message queue is to increase its data management capability in terms of performance and reliability by expanding the project to a Kafka-based broker system. In this paper, we compared the throughput for the proposed framework in comparison with the existing single node system and got some considerable good results.