Pathfinder: Fault Tolerance for Stream Processing Systems

Stream processing systems are able to integrate data from various sources, and to invoke self-hosted and external operators. In case of faults, such systems usually rely on the redundancy of single stream processing operators, while the relationship between the single operators is not taken into account. Hence, there is a lack of approaches towards fault-tolerant distributed stream processing which consider that stream processing applications are often composed of different operators. Within this paper, we present the Pathfinder framework which overcomes this shortcoming by enabling functional redundancy at the level of stream processing operator paths. During system runtime, Pathfinder reacts to operator failures in the main path by switching to a fault-free path with a similar functionality. To restore the main path once a failed operator has recovered, Pathfinder uses the circuit breaker pattern.