Automatic Differentiation over Fluid Models for Holistic Load Balancing

Microservice applications consist of a set of smaller services interacting in a graph structure to deliver the full application. Jobs will traverse this graph in different paths, both depending on the type of job, but also on the current load of different service replicas. Different paths will incur different scenario-specific costs, dependent on, e.g., deployment and the underlying cloud system. In this paper, we demonstrate how automatic differentiation over data-driven fluid models can be used to optimize a running microservice application, by designing a load balancer that minimizes some holistic cost function under response time percentile constraints. The cost function is based on performance metrics from a fluid model retrieved through logs from the application. The gradient of this cost, with respect to the load balancing parameters, is calculated via automatic differentiation. This enables parameter updates, using e.g. gradient descent, that steers the application towards a setting of less cost. In an experimental evaluation on a small microservice application running on Ericsson Research Datacenter, it is shown that the method can quickly step towards optimal values while supporting complicated cost functions such as solutions to a system of ordinary differential equations.