A fast and accurate threat detection and prevention architecture using stream processing

Late detection of security breaches increases the risk of irreparable damages and limits any mitigation attempts. We propose a fast and accurate threat detection and prevention architecture that combines the advantages of real-time streaming with batch processing over a historical database. We create a dataset by capturing both legitimate and malicious traffic and propose two ways of combining packets into flows, one considering a time window and the other analyzing the first few packets of each flow per period. We also investigate the effectiveness of our proposal on real-world network traces obtained from a significant Brazilian network operator providing broadband Internet to their customers. We implement and evaluate three classification algorithms and two anomaly detection methods. The results show an accuracy higher than 95% and an excellent trade-off between attack detection and false-positive rates. We further propose an improved scheme based on software defined networks that automatically prevents threats by analyzing only the first few packets of a flow. The proposal promptly and efficiently blocks threats, is robust, and can scale up, even when the attacker employs spoofed IP.