HarmonyBatch: Batching multi-SLO DNN Inference with Heterogeneous Serverless Functions

Overview

• Deep Neural Network (DNN) inference on serverless functions is gaining popularity due to potential cost savings
• Existing work focuses on optimizing batching requests from a single application with a single Service Level Objective (SLO) on CPU functions
• However, production traces show low request arrival rates, leading to long batching times and SLO violations
• There is a need for batching multiple DNN inference requests with diverse SLOs (multi-SLO DNN inference) on serverless platforms
• The potential benefits of deploying heterogeneous (CPU and GPU) functions for DNN inference have not been thoroughly explored

Plain English Explanation

In the world of cloud computing, Deep Neural Network (DNN) inference on serverless functions is becoming increasingly important. Serverless functions allow businesses to run their applications without having to manage the underlying infrastructure, which can lead to substantial cost savings.

Existing research has focused on optimizing the process of batching multiple requests for DNN inference from a single application, with a specific Service Level Objective (SLO) for the CPU functions. However, real-world data shows that the rate at which these requests arrive is often surprisingly low. This can lead to long wait times and SLO violations, where the application's performance doesn't meet the promised standards.

To address this issue, the researchers propose the need for a system that can batch multiple DNN inference requests with diverse SLOs (i.e., different performance targets) on serverless platforms. Additionally, they suggest that the potential benefits of using heterogeneous functions (both CPU and GPU) for DNN inference have not been thoroughly explored.

Technical Explanation

The researchers present HarmonyBatch, a cost-efficient resource provisioning framework designed to provide predictable performance for multi-SLO DNN inference with heterogeneous serverless functions.

First, they construct an analytical performance and cost model of DNN inference on both CPU and GPU functions, taking into account the GPU time-slicing scheduling mechanism and request arrival rate distribution.

Based on this model, they devise a two-stage merging strategy in HarmonyBatch to batch the multi-SLO DNN inference requests into application groups. The goal is to minimize the budget for function provisioning while guaranteeing diverse performance SLOs for the inference applications.

The researchers have implemented a prototype of HarmonyBatch on Alibaba Cloud Function Compute. Extensive experiments with representative DNN inference workloads demonstrate that HarmonyBatch can provide predictable performance to serverless DNN inference workloads while reducing the monetary cost by up to 82.9% compared to state-of-the-art methods.

Critical Analysis

The researchers have identified an important problem in the realm of serverless DNN inference, where low request arrival rates can lead to SLO violations. Their proposed solution, HarmonyBatch, aims to address this issue by batching multiple DNN inference requests with diverse SLOs and leveraging heterogeneous serverless functions.

While the analytical modeling and experimental results presented in the paper are promising, there are a few potential limitations and areas for further research:

1. Scalability: The paper does not explicitly discuss the scalability of HarmonyBatch as the number of applications or the complexity of DNN models increases. Further investigation into the system's performance under larger-scale deployments would be valuable.

2. Real-world Deployment: The evaluation was conducted on a prototype running on Alibaba Cloud. Applying HarmonyBatch to other serverless platforms and real-world production environments could reveal additional challenges or considerations.

3. Heterogeneous Function Utilization: The paper focuses on the potential benefits of using heterogeneous functions (CPU and GPU) for DNN inference, but it would be helpful to understand the trade-offs and optimal utilization strategies for these different function types.

4. Multi-tenant Scenarios: The current work assumes a single-tenant environment. Exploring the multi-tenant case, where multiple applications share the same serverless resources, could uncover additional complexities and optimization opportunities.

Conclusion

The researchers have presented HarmonyBatch, a cost-efficient resource provisioning framework for multi-SLO DNN inference on serverless platforms. By batching requests with diverse SLOs and leveraging heterogeneous functions, HarmonyBatch can provide predictable performance while significantly reducing monetary costs compared to existing methods.

This research highlights the importance of addressing the challenges posed by low request arrival rates in serverless DNN inference, and the potential benefits of exploring heterogeneous function deployments. As the adoption of serverless computing and DNN-powered applications continues to grow, solutions like HarmonyBatch could play a crucial role in optimizing the performance and cost-efficiency of these systems.