OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shift

Overview

• This paper examines the generalization of adversarial robustness to out-of-distribution (OOD) data, beyond just in-distribution (ID) testing.
• The authors propose a benchmark called OODRobustBench to comprehensively assess OOD adversarial robustness across various dataset and threat-wise shifts.
• Their large-scale analysis of 706 robust models shows that adversarial robustness suffers from a severe OOD generalization issue, but ID robustness can be used to predict OOD robustness.
• The authors investigate techniques to improve OOD robustness and identify several promising directions for future research.

Plain English Explanation

Existing machine learning models have made progress in becoming more robust to adversarial attacks, where small, imperceptible changes to the input can cause the model to make incorrect predictions. However, these models are typically tested only on data that comes from the same distribution as the training data, known as in-distribution (ID) testing.

The authors of this paper argue that this is a problem, because in real-world deployments, the input data may come from a different distribution, known as out-of-distribution (OOD) data. To address this issue, the authors propose a new benchmark called OODRobustBench, which tests the robustness of models to a variety of dataset-wise and threat-wise distribution shifts.

Their large-scale analysis of over 700 robust models shows that adversarial robustness does not generalize well to OOD data. However, they also find that there is a positive linear relationship between a model's ID robustness and its OOD robustness. This means that the OOD robustness of a model can be predicted from its ID robustness.

The authors then use this insight to predict that existing methods are unlikely to achieve high OOD robustness. They investigate a range of techniques and identify several promising directions for future research to improve OOD robustness, beyond what can be achieved by simply optimizing for ID robustness.

Technical Explanation

The paper starts by highlighting that existing works have made great progress in improving adversarial robustness, but typically only test their methods on data from the same distribution as the training data (in-distribution or ID testing). This is concerning, as in real-world deployments, the input data may come from a different distribution (out-of-distribution or OOD), and it is unclear how the robustness generalizes to such distribution shifts.

To address this issue, the authors propose a new benchmark called OODRobustBench, which comprehensively assesses OOD adversarial robustness using 23 dataset-wise shifts (e.g., changes in image resolution, color, or background) and 6 threat-wise shifts (e.g., unforeseen adversarial threat models). This benchmark is used to evaluate 706 robust models across 60.7K adversarial evaluations.

The large-scale analysis reveals two key findings:

1. Adversarial robustness suffers from a severe OOD generalization issue, with models performing much worse on OOD data compared to ID data.
2. There is a strong positive linear correlation between ID robustness and OOD robustness, enabling the prediction of OOD robustness from ID robustness.

Based on this observation, the authors predict that existing methods are unlikely to achieve high OOD robustness, as they are typically optimized for ID robustness. To address this, the authors investigate a wide range of techniques, including robust object detection, OOD detection, and covariate shift adaptation, and identify several promising directions for future research.

Critical Analysis

The paper provides a comprehensive and insightful analysis of the OOD generalization problem in adversarial robustness. The authors' use of the OODRobustBench benchmark to systematically evaluate a large number of robust models is a significant contribution to the field.

One potential limitation of the study is that it focuses primarily on image classification tasks, and it is unclear how the findings would translate to other domains, such as natural language processing or reinforcement learning. Additionally, the authors acknowledge that the benchmark does not cover all possible distribution shifts, and there may be other types of shifts that were not considered.

Furthermore, while the authors identify several promising research directions, the actual techniques for improving OOD robustness are not fully explored in this paper. It would be valuable to see more detailed investigations and evaluations of the proposed methods in future work.

Overall, this paper makes an important step forward in understanding the challenges of OOD generalization in adversarial robustness and provides a solid foundation for future research in this area.

Conclusion

This paper highlights the critical issue of OOD generalization in adversarial robustness, an area that has been overlooked in much of the existing research. By proposing the OODRobustBench benchmark and conducting a large-scale analysis, the authors have shown that current methods struggle to maintain their robustness when faced with distribution shifts.

The key finding that ID robustness can be used to predict OOD robustness is a valuable insight that can guide future research. However, the authors also make it clear that novel methods are needed to achieve high OOD robustness beyond what can be predicted from ID robustness.

The investigation of promising techniques, such as robust object detection and covariate shift adaptation, suggests that there are multiple avenues for improving OOD robustness. Continued research in this direction could lead to more reliable and trustworthy machine learning models that can perform well in diverse real-world scenarios.