A data science pipeline synchronisation method for edge-fog-cloud continuum.

This paper presents an adaptive data delivery method for data science pipelines. While this method is feasible for processes communicating over any network, in this work we focus on edge-fog-cloud infrastructures. In a diagnostic phase, a model based on the Bernoulli principle is used to create a representation of bottlenecks in a pipeline. In a supervision phase, a watchman/sentinel cooperative system monitors the throughput of the pipeline stages to create a bottleneck-stage scheme. In a rectification phase, this system produces replicas of bottlenecks stages, mitigating the workload congestion using implicit parallelism and load balancing algorithms. This method is automatically and transparently invoked to produce a steady continuum dataflow. To test our proposal, we conducted a case study about the processing of medical and satellite data. The evaluation revealed that this method creates continuum dataflows, without neither characterising workloads nor knowing infrastructure details, which yields a competitive performance with state-of-the-art solutions.