Understanding the Neglected Cost of Serverless Cluster Management

Serverless computing enables the cloud platform to optimize resource management under the hood to improve performance and resource-efficiency. However, today's serverless cluster managers are designed by simply retrofitting legacy workload orchestration systems, despite the unique characteristics of serverless workloads. We study Knative-on-K8s as a representative state-of-the-art cluster manager for serverless and show that it can cause second-scale delays and contribute to over 65% of end-to-end latency for function invocations experiencing cold starts. These overheads occur when the cluster experiences high sandbox churn, which is common in production serverless deployments. We analyze the root cause of current cluster manager overheads for serverless workloads and propose a set of design principles to improve end-to-end latency and peak throughput by rethinking the cluster manager system architecture.