Benchmarks for Retrospective Automated Driving System Crash Rate Analysis Using Police-Reported Crash Data

Overview

• The paper examines how to measure the safety impact of fully automated driving systems (ADS) as they become more widespread in ride-hailing services.
• It focuses on generating a benchmark crash rate for human drivers to compare against ADS crash data and identify any safety improvements.
• The researchers leverage publicly available police-reported crash data to determine human crash rates across different geographic areas with ADS deployments.
• They address the challenges in creating a fair benchmark, such as accounting for underreporting, matching crash and mileage exposure data, and controlling for factors like road type and vehicle type.

Plain English Explanation

As fully automated driving systems become more common in ride-hailing services in the US, researchers can now start to evaluate the safety impact of these systems more accurately. To do this, they need to compare the crash rates of automated driving systems to a benchmark for human drivers.

The researchers in this study used publicly available police-reported crash data to determine the crash rates for human drivers in different regions where automated driving systems are being used. This allows them to create a fair benchmark to compare against the crash data for the automated systems.

However, generating this benchmark is challenging. The researchers had to account for issues like underreporting of crashes, match the crash data to the right population of drivers and vehicles, and control for factors like road type and vehicle type that can affect crash rates.

By addressing these challenges, the researchers aim to help the research community, regulators, industry, and experts reach a consensus on how to accurately estimate benchmarks to measure the safety of automated driving systems.

Technical Explanation

The paper presents a methodology for leveraging publicly accessible police-reported crash data to generate human driver crash rate benchmarks that can be compared to crash data from fully automated driving systems (ADS).

The researchers identify several key challenges in creating this benchmark, including:

• Accounting for underreporting and different reporting thresholds in the crash data
• Identifying the appropriate population of drivers and vehicles to use for comparison
• Choosing an appropriate crash severity level to assess
• Matching crash data to accurate mileage exposure data

To address these challenges, the researchers:

• Apply established underreporting correction methodologies to the publicly available human driver crash data
• Control for geographic region, road type, and vehicle type to avoid biased results
• Discuss the dependencies and tradeoffs involved in selecting the appropriate comparison factors

The goal of this work is to contribute to the research community's ability to reach consensus on estimating accurate benchmarks for evaluating the safety impact of ADS as they become more widely deployed.

Critical Analysis

The paper provides a thoughtful and thorough approach to the challenge of generating a human driver crash rate benchmark that can be fairly compared to data from automated driving systems.

The researchers acknowledge the key limitations and complexities involved, such as the issues of underreporting and the need to control for factors like geography and vehicle type. By being transparent about these challenges, they encourage the research community to think critically about the benchmarking process and work towards a consensus.

However, the paper does not delve into some additional potential concerns, such as the reliability and consistency of the police-reported crash data, or the inherent biases that may exist in how crashes are recorded and reported. Additionally, the researchers do not discuss how their methodology might need to be adapted as the ADS technology and deployment models continue to evolve over time.

Overall, this paper makes a valuable contribution by outlining a rigorous, data-driven approach to benchmarking ADS safety and highlighting the important considerations involved. Further research and collaboration across industry, regulators, and experts will be crucial to refining and validating these methods.

Conclusion

This paper presents a framework for generating human driver crash rate benchmarks that can be used to evaluate the safety impact of fully automated driving systems as they become more widespread in ride-hailing services.

By leveraging publicly available data and addressing key challenges like underreporting and controlling for confounding factors, the researchers aim to help the research community, regulators, industry, and experts reach a consensus on accurate benchmarking methods.

As automated driving systems continue to evolve, this work represents an important step towards ensuring their safety can be reliably measured and validated against human driver performance.