Ditto: Quantization-aware Secure Inference of Transformers upon MPC

Overview

• Privacy concerns with sensitive client data and trained models like Transformers
• Secure multi-party computation (MPC) techniques used to enable secure inference, but with overhead
• Existing works aim to reduce overhead using more MPC-friendly non-linear function approximations
• Integration of quantization used in plaintext inference into the MPC domain remains unclear

Plain English Explanation

When working with sensitive client data and large language models like Transformers, privacy is a major concern. To address this, researchers use secure multi-party computation (MPC) techniques to enable secure inference, but this process can be inefficient and introduce overhead.

Previous work has tried to reduce this overhead by using more MPC-friendly approximations of the non-linear functions used in these models. However, it's still unclear how to effectively integrate the quantization techniques that are commonly used to optimize plaintext inference into the MPC domain.

To bridge this gap, the researchers propose a framework called Ditto, which enables more efficient quantization-aware secure Transformer inference. This approach significantly decreases both the computation and communication overhead, leading to improvements in overall efficiency.

Technical Explanation

The researchers first incorporate an MPC-friendly quantization into Transformer inference and employ a quantization-aware distillation procedure to maintain the model's utility. They then propose novel MPC primitives to support the type conversions that are essential in quantization and implement the quantization-aware MPC execution of secure quantized inference.

Through extensive experiments on BERT and GPT-2 models, the researchers demonstrate that their Ditto framework is about 3.14 to 4.40 times faster than the previous state-of-the-art approach, MPCFormer, and 1.44 to 2.35 times faster than the PUMA framework, with negligible utility degradation.

Critical Analysis

The paper provides a comprehensive solution to the challenge of efficiently integrating quantization techniques into the secure inference domain. However, the researchers acknowledge that their approach may still incur some overhead compared to plaintext inference, and they suggest further optimizations, such as attention-aware post-training mixed-precision quantization or genetic quantization-aware approximation of non-linear operations, as potential avenues for future research.

Additionally, the paper does not explore the impact of different quantization schemes or the trade-offs between model accuracy and inference speed in the secure inference setting. Further investigation into these areas could provide valuable insights for practitioners.

Conclusion

The Ditto framework proposed in this paper represents a significant step forward in enabling efficient and secure Transformer inference. By seamlessly integrating quantization techniques into the MPC-based secure inference process, the researchers have achieved substantial performance improvements while maintaining the model's utility. This work has important implications for the deployment of large language models in privacy-sensitive applications, paving the way for more widespread adoption of secure inference capabilities.