Hacking Predictors Means Hacking Cars: Using Sensitivity Analysis to Identify Trajectory Prediction Vulnerabilities for Autonomous Driving Security

Overview

• This paper examines the susceptibility of two popular trajectory prediction models, Trajectron++ and AgentFormer, to adversarial attacks.
• The researchers conduct a sensitivity analysis to understand which inputs are most influential for the models' predictions.
• They then demonstrate how an undetectable perturbation to the image map input can induce large errors in the models' predictions, despite the models being more sensitive to perturbations in the state history.
• The paper also shows how these attacks can impact downstream planning and control, causing a vehicle to suddenly stop from moderate driving speeds.

Plain English Explanation

The researchers in this paper looked at how to trick two AI models that are used to predict the future movement of vehicles and pedestrians. These models are an important part of self-driving car systems, helping the car plan its next actions.

The researchers found that the models were most sensitive to changes in the most recent position and speed of the vehicle or person being tracked. This means that small changes to these inputs could cause the models to make big mistakes in their predictions.

However, the researchers also discovered something interesting. Even though the models were more sensitive to changes in the position and speed data, an undetectable change to the map data that the models use could also cause large errors in their predictions. This shows that these AI models for trajectory prediction are vulnerable to attacks that target the map data, not just the vehicle or pedestrian data.

The researchers then showed how these attacks on the trajectory prediction models could impact the planning and control of a self-driving car, causing it to suddenly stop from normal driving speeds. This demonstrates how these types of attacks could be a real problem for the safety and reliability of self-driving car systems.

Technical Explanation

The researchers conducted a sensitivity analysis on the Trajectron++ and AgentFormer trajectory prediction models to understand which inputs were most influential for their predictions. They found that almost all of the perturbation sensitivities for both models were concentrated within the most recent position and velocity states.

However, the researchers also demonstrated that an undetectable perturbation to the image map input, generated using the Fast Gradient Sign Method, could induce large prediction error increases in both models. This revealed that these trajectory predictors are susceptible to image-based attacks, despite their dominant sensitivity to state history perturbations.

Using an optimization-based planner and example perturbations crafted from their sensitivity analysis, the researchers showed how these attacks could cause a vehicle to come to a sudden stop from moderate driving speeds. This highlights the potential impact these types of adversarial attacks could have on the safety and reliability of autonomous vehicle systems.

Critical Analysis

The researchers provide a thorough analysis of the sensitivity of two popular trajectory prediction models to various input perturbations. Their findings suggest that these models, while heavily reliant on state history, are also vulnerable to more subtle attacks targeting the image map input.

However, the paper does not explore the generalizability of these findings to other trajectory prediction models or autonomous driving systems. The experiments were conducted on a specific dataset and scenario, and it's unclear how the results might translate to real-world driving conditions or other model architectures.

Additionally, the paper does not delve into potential mitigation strategies or defenses against these types of attacks. Further research would be needed to understand how these trajectory prediction models could be made more robust to adversarial perturbations, both in the state history and image map inputs.

Conclusion

This paper highlights the susceptibility of two widely used trajectory prediction models to adversarial attacks. The researchers demonstrated that while these models are most sensitive to perturbations in the vehicle or pedestrian state history, they can also be fooled by more subtle changes to the environmental map data.

These findings have important implications for the safety and reliability of autonomous driving systems, which rely heavily on accurate trajectory prediction to plan and execute safe maneuvers. The paper underscores the need for continued research into the robustness and security of these AI-based components, as well as the development of effective countermeasures against adversarial attacks.

As autonomous vehicle technology continues to advance, it will be crucial for researchers and developers to address these vulnerabilities to ensure the safe and trustworthy deployment of self-driving cars on public roads.