Identifying Frequent Flows in Large Traffic Sets through Probabilistic Bloom Filters

In many network applications, accurate traffic measurement is critical for bandwidth management and detecting security threats such as DoS (Denial of Service) attacks. In such cases, traffic is usually modeled as a collection of flows, which are identified based on certain features such as IP address pairs. One central problem is to identify those “heavy hitter” flows, which account for a large percentage of total traffic, e.g., at least 0.1% of the link capacity. However, the challenge for this goal is that keeping an individual counter for each flow is too slow, costly, and non-scalable. In this paper, we describe a novel data structure called the Probabilistic Bloom Filter (PBF), which extends the classical bloom filter into the probabilistic direction, so that it can effectively identify heavy hitters. We analyze the performance, tradeoffs, and capacity of this data structure, as well as developing two extensions to improve its accuracy and flexibility. Our study also investigates how to calibrate this data structure’s parameters, where we prove our developed method achieves the Nash Equilibrium using game theory. We use real network traces collected on a web query server and a backbone router to test the performance of the PBF, and demonstrate that this method can accurately keep track of all objects’ frequencies, including websites and flows, so that heavy hitters can be identified with constant time computational complexity and low memory overhead.