NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
2406.15349
0
0
Abstract
Benchmarking vision-based driving policies is challenging. On one hand, open-loop evaluation with real data is easy, but these results do not reflect closed-loop performance. On the other, closed-loop evaluation is possible in simulation, but is hard to scale due to its significant computational demands. Further, the simulators available today exhibit a large domain gap to real data. This has resulted in an inability to draw clear conclusions from the rapidly growing body of research on end-to-end autonomous driving. In this paper, we present NAVSIM, a middle ground between these evaluation paradigms, where we use large datasets in combination with a non-reactive simulator to enable large-scale real-world benchmarking. Specifically, we gather simulation-based metrics, such as progress and time to collision, by unrolling bird's eye view abstractions of the test scenes for a short simulation horizon. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. As we demonstrate empirically, this decoupling allows open-loop metric computation while being better aligned with closed-loop evaluations than traditional displacement errors. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights. On a large set of challenging scenarios, we observe that simple methods with moderate compute requirements such as TransFuser can match recent large-scale end-to-end driving architectures such as UniAD. Our modular framework can potentially be extended with new datasets, data curation strategies, and metrics, and will be continually maintained to host future challenges. Our code is available at https://github.com/autonomousvision/navsim.
Create account to get full access
Overview
- This paper introduces NAVSIM, a data-driven simulation framework for benchmarking autonomous vehicle systems in non-reactive environments.
- NAVSIM uses real-world driving data to create simulated scenarios that do not involve other reactive agents, allowing for more controlled and repeatable testing of autonomous vehicle algorithms.
- The authors demonstrate the effectiveness of NAVSIM by using it to evaluate the behavior of an autonomous vehicle system in different scenarios and compare its performance to human drivers.
Plain English Explanation
The paper presents a new simulation tool called NAVSIM that is designed to help test and evaluate autonomous vehicle systems. Unlike other driving simulations, NAVSIM uses real-world driving data to create scenarios where the other vehicles on the road do not react to the autonomous vehicle. This allows researchers to more precisely measure how the autonomous vehicle performs without the influence of other unpredictable drivers.
The key advantage of NAVSIM is that it provides a more controlled environment for testing autonomous vehicles. In traditional simulations, the behavior of other vehicles can complicate the evaluation of the autonomous system. By removing those reactive agents, NAVSIM enables researchers to isolate and study the autonomous vehicle's decision-making without external factors muddying the results.
The authors demonstrate the usefulness of NAVSIM by using it to assess the performance of an autonomous driving system and compare it to human drivers. This data-driven approach allows them to gain valuable insights about the autonomous system's strengths and weaknesses in a variety of common driving scenarios.
Technical Explanation
The NAVSIM framework uses real-world driving data collected from human drivers to create simulated environments for testing autonomous vehicle systems. Unlike traditional driving simulations that include reactive agents, NAVSIM generates "non-reactive" scenarios where other vehicles on the road maintain a predetermined trajectory and speed, unaffected by the autonomous vehicle's actions.
This approach allows for more controlled and repeatable experimentation, as the autonomous system's behavior can be isolated and assessed without the complicating influence of other drivers' unpredictable responses. The authors leverage techniques for diagnosing and predicting autonomous vehicle safety to evaluate the performance of their autonomous driving system within the NAVSIM environment.
The researchers demonstrate the effectiveness of NAVSIM by using it to compare the behavior of their autonomous vehicle system to that of human drivers. This data-driven benchmarking allows them to identify specific scenarios where the autonomous system excels or struggles, providing valuable insights for further development and refinement.
Critical Analysis
The NAVSIM framework represents a valuable contribution to the field of autonomous vehicle testing and evaluation. By removing the complexity of reactive agents, it enables a more focused and controlled assessment of autonomous systems, which is crucial for ensuring their safety and reliability.
However, the authors acknowledge that NAVSIM's non-reactive nature may limit its ability to capture the full complexity of real-world driving scenarios. In some cases, the lack of interaction with other drivers could lead to oversimplified or unrealistic test conditions. Further research may be needed to explore ways of incorporating some degree of driver reactivity while maintaining the benefits of NAVSIM's controlled environment.
Additionally, the authors' reliance on real-world driving data to generate simulation scenarios raises questions about the representativeness and coverage of the data. It is possible that certain driving situations or behaviors may be underrepresented in the dataset, potentially skewing the evaluation of the autonomous system.
Conclusion
The NAVSIM framework introduced in this paper represents a significant advancement in the field of autonomous vehicle simulation and testing. By creating a controlled, non-reactive environment based on real-world driving data, NAVSIM enables more precise and repeatable assessment of autonomous systems, helping to identify their strengths and weaknesses in a variety of common driving scenarios.
The insights gained from NAVSIM can inform the ongoing development and refinement of autonomous vehicle technologies, ultimately contributing to their safer and more reliable deployment. As the field of autonomous driving continues to evolve, tools like NAVSIM will play an increasingly important role in ensuring the responsible and effective advancement of these transformative technologies.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving
William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle {AA}strom, Michael Felsberg, Christoffer Petersson
0
0
We present a versatile NeRF-based simulator for testing autonomous driving (AD) software systems, designed with a focus on sensor-realistic closed-loop evaluation and the creation of safety-critical scenarios. The simulator learns from sequences of real-world driving sensor data and enables reconfigurations and renderings of new, unseen scenarios. In this work, we use our simulator to test the responses of AD models to safety-critical scenarios inspired by the European New Car Assessment Programme (Euro NCAP). Our evaluation reveals that, while state-of-the-art end-to-end planners excel in nominal driving scenarios in an open-loop setting, they exhibit critical flaws when navigating our safety-critical scenarios in a closed-loop setting. This highlights the need for advancements in the safety and real-world usability of end-to-end planners. By publicly releasing our simulator and scenarios as an easy-to-run evaluation suite, we invite the research community to explore, refine, and validate their AD models in controlled, yet highly configurable and challenging sensor-realistic environments. Code and instructions can be found at https://github.com/atonderski/neuro-ncap
4/24/2024
SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Controllable Adversaries
Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, Manmohan Chandraker
0
0
Evaluating the performance of autonomous vehicle planning algorithms necessitates simulating long-tail safety-critical traffic scenarios. However, traditional methods for generating such scenarios often fall short in terms of controllability and realism and neglect the dynamics of agent interactions. To mitigate these limitations, we introduce SAFE-SIM, a novel diffusion-based controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) the generation of realistic long-tail safety-critical scenarios that closely emulate real-world conditions, and 2) enhanced controllability, enabling more comprehensive and interactive evaluations. We develop a novel approach to simulate safety-critical scenarios through an adversarial term in the denoising process, which allows an adversarial agent to challenge a planner with plausible maneuvers while all agents in the scene exhibit reactive and realistic behaviors. Furthermore, we propose novel guidance objectives and a partial diffusion process that enables a user to control key aspects of the generated scenarios, such as the collision type and aggressiveness of the adversarial driver, while maintaining the realism of the behavior. We validate our framework empirically using the NuScenes dataset, demonstrating improvements in both realism and controllability. These findings affirm that diffusion models provide a robust and versatile foundation for safety-critical, interactive traffic simulation, extending their utility across the broader landscape of autonomous driving. For supplementary videos, visit our project at https://safe-sim.github.io/.
6/18/2024
An Approach to Systematic Data Acquisition and Data-Driven Simulation for the Safety Testing of Automated Driving Functions
Leon Eisemann, Mirjam Fehling-Kaschek, Henrik Gommel, David Hermann, Marvin Klemp, Martin Lauer, Benjamin Lickert, Florian Luettner, Robin Moss, Nicole Neis, Maria Pohle, Simon Romanski, Daniel Stadler, Alexander Stolz, Jens Ziehn, Jingxing Zhou
0
0
With growing complexity and criticality of automated driving functions in road traffic and their operational design domains (ODD), there is increasing demand for covering significant proportions of development, validation, and verification in virtual environments and through simulation models. If, however, simulations are meant not only to augment real-world experiments, but to replace them, quantitative approaches are required that measure to what degree and under which preconditions simulation models adequately represent reality, and thus, using their results accordingly. Especially in R&D areas related to the safety impact of the open world, there is a significant shortage of real-world data to parameterize and/or validate simulations - especially with respect to the behavior of human traffic participants, whom automated driving functions will meet in mixed traffic. We present an approach to systematically acquire data in public traffic by heterogeneous means, transform it into a unified representation, and use it to automatically parameterize traffic behavior models for use in data-driven virtual validation of automated driving functions.
5/6/2024
🏅
Diagnosing and Predicting Autonomous Vehicle Operational Safety Using Multiple Simulation Modalities and a Virtual Environment
Joe Beck, Shean Huff, Subhadeep Chakraborty
0
0
Even as technology and performance gains are made in the sphere of automated driving, safety concerns remain. Vehicle simulation has long been seen as a tool to overcome the cost associated with a massive amount of on-road testing for development and discovery of safety critical edge-cases. However, purely software-based vehicle models may leave a large realism gap between their real-world counterparts in terms of dynamic response, and highly realistic vehicle-in-the-loop (VIL) simulations that encapsulate a virtual world around a physical vehicle may still be quite expensive to produce and similarly time intensive as on-road testing. In this work, we demonstrate an AV simulation test bed that combines the realism of vehicle-in-the-loop (VIL) simulation with the ease of implementation of model-in-the-loop (MIL) simulation. The setup demonstrated in this work allows for response diagnosis for the VIL simulations. By observing causal links between virtual weather and lighting conditions that surround the virtual depiction of our vehicle, the vision-based perception model and controller of Openpilot, and the dynamic response of our physical vehicle under test, we can draw conclusions regarding how the perceived environment contributed to vehicle response. Conversely, we also demonstrate response prediction for the MIL setup, where the need for a physical vehicle is not required to draw richer conclusions around the impact of environmental conditions on AV performance than could be obtained with VIL simulation alone. These combine for a simulation setup with accurate real-world implications for edge-case discovery that is both cost effective and time efficient to implement.
5/14/2024