Ambiguous Annotations: When is a Pedestrian not a Pedestrian?

Overview

• Datasets labeled by human annotators are widely used to train and test machine learning models.
• Researchers are increasingly concerned about the quality of these labels, as it's not always clear if a label is correct or not.
• This paper investigates the ambiguity in annotating autonomous driving datasets, an important aspect of data quality.
• The experiments show that excluding highly ambiguous data from training improves the performance of a state-of-the-art pedestrian detector.
• Understanding the properties of the dataset and the target class is crucial to safely remove ambiguous instances while maintaining the representativeness of the training data.

Plain English Explanation

Machine learning models are often trained and tested using datasets that have been labeled by human annotators. In recent years, researchers have become more interested in the quality of these labels, as it's not always easy to determine whether a label is correct or not.

This paper looks at the ambiguity in the annotation of datasets used for autonomous driving research, which is an important factor in the overall quality of the data. The experiments show that removing the most ambiguous data from the training process can actually improve the performance of a state-of-the-art pedestrian detection model, saving time and annotation costs.

However, the researchers also found that in order to safely remove ambiguous instances while still keeping the training data representative of the real world, it's crucial to have a good understanding of the properties of the dataset and the specific class being detected, like pedestrians in this case.

Technical Explanation

The paper investigates the impact of ambiguous annotations in autonomous driving datasets on the performance of a state-of-the-art pedestrian detection model. The authors propose a method to identify and remove highly ambiguous data instances from the training set, and evaluate the impact on model performance in terms of LAMR, precision, and F1 score.

The experiments were conducted on the nuScenes and Waymo datasets, using a pedestrian attribute recognition model as the base detector. The authors first defined a set of ambiguity measures to identify highly ambiguous instances, such as low-confidence annotations or disagreements between multiple annotators.

By excluding these ambiguous instances from the training set, the authors demonstrated significant improvements in the performance of the pedestrian detector, while also reducing training time and annotation costs. The results highlight the importance of understanding the properties of the dataset and the target class in order to safely remove ambiguous data without compromising the representativeness of the training data.

Critical Analysis

The paper provides a valuable contribution to the growing body of research on label quality in machine learning datasets. By focusing on the specific challenge of ambiguous annotations in autonomous driving datasets, the authors have identified an important dimension of data quality that deserves more attention.

One potential limitation of the work is that the experiments were conducted on a relatively small set of datasets, nuScenes and Waymo. It would be interesting to see if the findings hold true for a broader range of autonomous driving datasets, or even in other domains beyond self-driving cars.

Additionally, the paper does not provide much insight into the specific causes of the ambiguity in the annotations, or the factors that contribute to it. A deeper analysis of the annotation process and the characteristics of the data that lead to ambiguity could help inform better annotation strategies and dataset design.

Overall, this research highlights the importance of carefully considering label quality when working with machine learning datasets, and provides a promising approach for improving model performance by selectively excluding ambiguous data instances.

Conclusion

This paper investigates the impact of ambiguous annotations in autonomous driving datasets on the performance of a state-of-the-art pedestrian detection model. The experiments show that excluding highly ambiguous data from the training process can lead to significant improvements in model performance, while also reducing training time and annotation costs.

The key takeaway is that understanding the properties of the dataset and the target class being detected is crucial for safely removing ambiguous instances without compromising the representativeness of the training data. This research contributes to the growing body of work on label quality in machine learning, and provides valuable insights for researchers and practitioners working with complex, real-world datasets.