Missile: Fine-Grained, Hardware-Level GPU Resource Isolation for Multi-Tenant DNN Inference

Overview

• Combining high-priority, low-latency (LS) and low-priority, best-effort (BE) DNN inference services on the same GPU cluster can reduce total cost of ownership (TCO).
• Existing GPU sharing solutions are unable to avoid resource conflicts between LS and BE tasks, leading to poor latency for LS tasks and low throughput for BE tasks.
• This paper presents Missile, a GPU sharing solution for multi-tenant DNN inference on NVIDIA GPUs that isolates resources between LS and BE tasks.

Plain English Explanation

The paper discusses a way to run different types of deep neural network (DNN) inference tasks on the same GPU cluster to save money. Some tasks, like real-time video processing, need to run quickly with low delay (latency-sensitive, or LS tasks). Other tasks, like batch data processing, don't need to be as fast (best-effort, or BE tasks).

Putting LS and BE tasks on the same GPUs could be cheaper, but existing methods can't isolate the resources used by each type of task. This leads to the LS tasks being slow and the BE tasks not running as efficiently as they could.

The paper introduces Missile, a new system that can better separate the resources used by LS and BE tasks on NVIDIA GPUs. This allows the LS tasks to run quickly while still getting good performance from the BE tasks.

Technical Explanation

Missile addresses two key bottlenecks that limit existing GPU sharing solutions:

1. VRAM Channel Conflicts: Missile reveals the VRAM channel hash mapping architecture of NVIDIA GPUs and uses software-level cache coloring to eliminate conflicts between LS and BE tasks.

2. PCIe Bus Contentions: Missile isolates the PCIe bus and fairly allocates PCIe bandwidth to LS and BE tasks using a completely fair scheduler.

Through comprehensive experiments with 12 mainstream DNNs and synthetic/real-world workloads on 4 GPUs, Missile is shown to:

• Reduce tail latency for LS services by up to ~50% compared to state-of-the-art GPU sharing solutions.
• Achieve up to 6.1x higher throughput for BE jobs.
• Allocate PCIe bus bandwidth to tenants on-demand for optimal performance.

Critical Analysis

The paper provides a thorough technical explanation of Missile's design and evaluation. However, it does not discuss potential limitations or future research directions in depth.

Some areas that could be further explored include:

• Generalizability: The evaluation is focused on NVIDIA GPUs - it would be helpful to understand how Missile's techniques could apply to other GPU architectures.
• Production Feasibility: The paper does not address practical deployment challenges, such as handling dynamic workload changes or fault tolerance.
• Hardware Requirements: Missile may require specialized hardware support (e.g., cache partitioning) that may not be available in all GPU systems.

Overall, Missile presents a promising approach to improving GPU multi-tenancy, but additional research is needed to fully understand its real-world applicability and limitations.

Conclusion

This paper introduces Missile, a GPU sharing solution that can effectively isolate resources between high-priority, low-latency (LS) and low-priority, best-effort (BE) DNN inference tasks. By addressing key bottlenecks like VRAM channel conflicts and PCIe bus contention, Missile is able to achieve low latency for LS tasks while also maintaining high throughput for BE tasks.

The technical evaluation demonstrates Missile's ability to outperform state-of-the-art GPU sharing approaches. However, the paper could be strengthened by further discussion of Missile's limitations and future research directions. Overall, Missile represents an important step towards more efficient and cost-effective GPU utilization in multi-tenant environments.