Towards Scenario- and Capability-Driven Dataset Development and Evaluation: An Approach in the Context of Mapless Automated Driving
2404.19656
0
0
🚀
Abstract
The foundational role of datasets in defining the capabilities of deep learning models has led to their rapid proliferation. At the same time, published research focusing on the process of dataset development for environment perception in automated driving has been scarce, thereby reducing the applicability of openly available datasets and impeding the development of effective environment perception systems. Sensor-based, mapless automated driving is one of the contexts where this limitation is evident. While leveraging real-time sensor data, instead of pre-defined HD maps promises enhanced adaptability and safety by effectively navigating unexpected environmental changes, it also increases the demands on the scope and complexity of the information provided by the perception system. To address these challenges, we propose a scenario- and capability-based approach to dataset development. Grounded in the principles of ISO 21448 (safety of the intended functionality, SOTIF), extended by ISO/TR 4804, our approach facilitates the structured derivation of dataset requirements. This not only aids in the development of meaningful new datasets but also enables the effective comparison of existing ones. Applying this methodology to a broad range of existing lane detection datasets, we identify significant limitations in current datasets, particularly in terms of real-world applicability, a lack of labeling of critical features, and an absence of comprehensive information for complex driving maneuvers.
Create account to get full access
Overview
- The paper discusses the critical role of datasets in defining the capabilities of deep learning models for environment perception in automated driving.
- It highlights the scarcity of published research on the dataset development process, which limits the applicability of openly available datasets and impedes the development of effective environment perception systems.
- The paper proposes a scenario- and capability-based approach to dataset development, grounded in the principles of ISO 21448 (safety of the intended functionality, SOTIF) and ISO/TR 4804.
Plain English Explanation
Deep learning models, which are at the heart of many automated driving systems, rely heavily on the quality and diversity of the datasets used to train them. However, the research on how to develop effective datasets for environment perception in automated driving has been limited. This lack of guidance on dataset development has reduced the usefulness of publicly available datasets and made it harder to create robust environment perception systems.
One area where this limitation is particularly evident is in sensor-based, mapless automated driving. While using real-time sensor data instead of pre-defined high-definition (HD) maps promises better adaptability and safety by navigating unexpected environmental changes, it also increases the demands on the scope and complexity of the information provided by the perception system.
To address these challenges, the researchers propose a scenario- and capability-based approach to dataset development. This method is grounded in the principles of ISO 21448 (safety of the intended functionality, SOTIF) and ISO/TR 4804, which helps systematically define dataset requirements. This not only aids in the development of meaningful new datasets but also enables the effective comparison of existing ones.
Technical Explanation
The researchers applied this methodology to a broad range of existing lane detection datasets and identified significant limitations. These include a lack of real-world applicability, insufficient labeling of critical features, and an absence of comprehensive information for complex driving maneuvers.
The proposed scenario- and capability-based approach facilitates the structured derivation of dataset requirements. This helps ensure that new datasets are developed to address specific challenges in environment perception, such as those encountered in sensor-based, mapless automated driving. Additionally, it enables the effective comparison of existing datasets, allowing researchers and developers to select the most appropriate ones for their needs.
Critical Analysis
The paper highlights the importance of a systematic approach to dataset development for environment perception in automated driving. By grounding the methodology in established standards like ISO 21448 and ISO/TR 4804, the researchers provide a solid foundation for dataset creation and evaluation.
However, the paper does not delve deeply into the specific limitations identified in the existing lane detection datasets. Further exploration of the nature and causes of these limitations could provide valuable insights for future dataset development efforts.
Additionally, the paper does not address the potential challenges in implementing the proposed scenario- and capability-based approach, such as the effort required to define the relevant scenarios and capabilities, or the availability of subject matter experts to guide the process.
Finally, the paper could have explored the potential for generative AI techniques to complement the dataset development process and address the scarcity of real-world data.
Conclusion
The paper highlights the critical importance of dataset development for environment perception in automated driving, a topic that has received limited attention in the research community. The proposed scenario- and capability-based approach offers a structured methodology for creating and evaluating datasets, which can significantly improve the performance and robustness of environment perception systems. By addressing the limitations of current datasets, this research paves the way for more effective sensor-based, mapless automated driving solutions that can adapt to unexpected environmental changes and enhance overall safety.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook
Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, Alois C. Knoll
0
0
Autonomous driving has rapidly developed and shown promising performance due to recent advances in hardware and deep learning techniques. High-quality datasets are fundamental for developing reliable autonomous driving algorithms. Previous dataset surveys either focused on a limited number or lacked detailed investigation of dataset characteristics. To this end, we present an exhaustive study of 265 autonomous driving datasets from multiple perspectives, including sensor modalities, data size, tasks, and contextual conditions. We introduce a novel metric to evaluate the impact of datasets, which can also be a guide for creating new datasets. Besides, we analyze the annotation processes, existing labeling tools, and the annotation quality of datasets, showing the importance of establishing a standard annotation pipeline. On the other hand, we thoroughly analyze the impact of geographical and adversarial environmental conditions on the performance of autonomous driving systems. Moreover, we exhibit the data distribution of several vital datasets and discuss their pros and cons accordingly. Finally, we discuss the current challenges and the development trend of the future autonomous driving datasets.
4/24/2024
Collaborative Perception Datasets in Autonomous Driving: A Survey
Melih Yazgan, Mythra Varun Akkanapragada, J. Marius Zoellner
0
0
This survey offers a comprehensive examination of collaborative perception datasets in the context of Vehicle-to-Infrastructure (V2I), Vehicle-to-Vehicle (V2V), and Vehicle-to-Everything (V2X). It highlights the latest developments in large-scale benchmarks that accelerate advancements in perception tasks for autonomous vehicles. The paper systematically analyzes a variety of datasets, comparing them based on aspects such as diversity, sensor setup, quality, public availability, and their applicability to downstream tasks. It also highlights the key challenges such as domain shift, sensor setup limitations, and gaps in dataset diversity and availability. The importance of addressing privacy and security concerns in the development of datasets is emphasized, regarding data sharing and dataset creation. The conclusion underscores the necessity for comprehensive, globally accessible datasets and collaborative efforts from both technological and research communities to overcome these challenges and fully harness the potential of autonomous driving.
4/23/2024
MapVision: CVPR 2024 Autonomous Grand Challenge Mapless Driving Tech Report
Zhongyu Yang, Mai Liu, Jinluo Xie, Yueming Zhang, Chen Shen, Wei Shao, Jichao Jiao, Tengfei Xing, Runbo Hu, Pengfei Xu
0
0
Autonomous driving without high-definition (HD) maps demands a higher level of active scene understanding. In this competition, the organizers provided the multi-perspective camera images and standard-definition (SD) maps to explore the boundaries of scene reasoning capabilities. We found that most existing algorithms construct Bird's Eye View (BEV) features from these multi-perspective images and use multi-task heads to delineate road centerlines, boundary lines, pedestrian crossings, and other areas. However, these algorithms perform poorly at the far end of roads and struggle when the primary subject in the image is occluded. Therefore, in this competition, we not only used multi-perspective images as input but also incorporated SD maps to address this issue. We employed map encoder pre-training to enhance the network's geometric encoding capabilities and utilized YOLOX to improve traffic element detection precision. Additionally, for area detection, we innovatively introduced LDTR and auxiliary tasks to achieve higher precision. As a result, our final OLUS score is 0.58.
6/17/2024
Collective Perception Datasets for Autonomous Driving: A Comprehensive Review
Sven Teufel, Jorg Gamerdinger, Jan-Patrick Kirchner, Georg Volk, Oliver Bringmann
0
0
To ensure safe operation of autonomous vehicles in complex urban environments, complete perception of the environment is necessary. However, due to environmental conditions, sensor limitations, and occlusions, this is not always possible from a single point of view. To address this issue, collective perception is an effective method. Realistic and large-scale datasets are essential for training and evaluating collective perception methods. This paper provides the first comprehensive technical review of collective perception datasets in the context of autonomous driving. The survey analyzes existing V2V and V2X datasets, categorizing them based on different criteria such as sensor modalities, environmental conditions, and scenario variety. The focus is on their applicability for the development of connected automated vehicles. This study aims to identify the key criteria of all datasets and to present their strengths, weaknesses, and anomalies. Finally, this survey concludes by making recommendations regarding which dataset is most suitable for collective 3D object detection, tracking, and semantic segmentation.
5/28/2024