Understanding the Vulnerability of Skeleton-based Human Activity Recognition via Black-box Attack

Overview

• This paper explores the robustness of skeleton-based human activity recognition (HAR) models to adversarial attacks.
• The authors propose a new black-box adversarial attack approach called BASAR that exploits the interplay between the classification boundary and the natural motion manifold.
• The research finds that on-manifold adversarial samples are common in skeletal motions, challenging the belief that adversarial samples only exist off-manifold.
• The authors also propose a new adversarial training approach called mixed manifold-based adversarial training (MMAT) to defend against these attacks.

Plain English Explanation

The paper focuses on the security and reliability of human activity recognition (HAR) systems, which are used in safety-critical applications like self-driving cars. These systems use data from sensors, like cameras or wearable devices, to recognize human actions and behaviors.

The researchers discovered that these HAR systems can be fooled by "adversarial attacks" - small, often imperceptible changes to the input data that cause the system to make incorrect predictions. Previous research has shown that these attacks are possible, but they required the attacker to have full knowledge of the model's inner workings, which is a very strong assumption.

In this paper, the authors show that such attacks can also work even when the attacker only has access to the model's inputs and outputs, without knowing the model's architecture or parameters. They propose a new attack method called BASAR that exploits the relationship between the classification boundary and the natural movements that humans make.

Surprisingly, the researchers found that many of these adversarial samples are not completely unnatural or "off-manifold" (i.e., outside the normal range of human motion), as previously believed. Instead, they exist on the "manifold" of natural human movements, making them even more deceptive.

To address this threat, the authors developed a new defense mechanism called mixed manifold-based adversarial training (MMAT). This technique helps the model become more robust to both on-manifold and off-manifold adversarial samples, without significantly impacting its overall accuracy.

Technical Explanation

The paper proposes a new black-box adversarial attack approach for skeleton-based human activity recognition (HAR) models, called BASAR (Black-box Adversarial Skeleton-based Activity Recognition).

BASAR exploits the interplay between the classification boundary and the natural motion manifold to generate adversarial samples. The authors hypothesize that adversarial samples do not necessarily have to be completely unnatural or "off-manifold" (i.e., outside the range of normal human motion), as commonly believed. Instead, they may exist on the "manifold" of natural human movements, making them even more deceptive.

To validate this hypothesis, the researchers conduct extensive evaluations of BASAR across various classifiers, datasets, and attack modes. The results show that BASAR can consistently deliver successful attacks, even in a black-box setting where the attacker has limited knowledge of the target model.

Furthermore, the authors propose a new adversarial training approach called mixed manifold-based adversarial training (MMAT) to defend against these attacks. MMAT leverages the sophisticated distributions of on-manifold and off-manifold adversarial samples to improve the model's robustness without compromising its classification accuracy.

Critical Analysis

The paper makes a significant contribution by introducing the first black-box adversarial attack approach for skeleton-based HAR models and challenging the common belief that adversarial samples must be off-manifold. The authors provide a comprehensive evaluation of BASAR and demonstrate its effectiveness across various settings.

However, the paper does not address the computational complexity and practical feasibility of implementing BASAR in real-world scenarios, where the attacker may have limited resources and time. Additionally, the proposed MMAT defense mechanism is evaluated only on a limited set of models and datasets, and its generalizability to other HAR systems remains to be explored.

Further research could investigate the transferability of BASAR attacks across different HAR models and the development of more efficient and scalable defense mechanisms. Additionally, it would be valuable to understand the broader implications of on-manifold adversarial samples and their potential impact on the reliability and trustworthiness of skeleton-based HAR systems in safety-critical applications.

Conclusion

This paper presents a significant advancement in the security and robustness of skeleton-based human activity recognition (HAR) systems. By proposing the BASAR attack and the MMAT defense, the authors provide new insights into the nature of adversarial samples and the vulnerabilities of these models.

The findings challenge the common assumption that adversarial samples must be completely unnatural or off-manifold, and highlight the need for more comprehensive defense strategies that can handle both on-manifold and off-manifold adversarial threats. The proposed MMAT approach offers a promising direction for improving the reliability of HAR systems, which are critical for safety-critical applications like self-driving cars.

Overall, this research contributes to the ongoing efforts to ensure the trustworthiness and robustness of AI-powered systems, particularly in domains where human safety and security are at stake.