Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis
2211.02736
0
0
👁️
Abstract
Machine learning driven image-based controllers allow robotic systems to take intelligent actions based on the visual feedback from their environment. Understanding when these controllers might lead to system safety violations is important for their integration in safety-critical applications and engineering corrective safety measures for the system. Existing methods leverage simulation-based testing (or falsification) to find the failures of vision-based controllers, i.e., the visual inputs that lead to closed-loop safety violations. However, these techniques do not scale well to the scenarios involving high-dimensional and complex visual inputs, such as RGB images. In this work, we cast the problem of finding closed-loop vision failures as a Hamilton-Jacobi (HJ) reachability problem. Our approach blends simulation-based analysis with HJ reachability methods to compute an approximation of the backward reachable tube (BRT) of the system, i.e., the set of unsafe states for the system under vision-based controllers. Utilizing the BRT, we can tractably and systematically find the system states and corresponding visual inputs that lead to closed-loop failures. These visual inputs can be subsequently analyzed to find the input characteristics that might have caused the failure. Besides its scalability to high-dimensional visual inputs, an explicit computation of BRT allows the proposed approach to capture non-trivial system failures that are difficult to expose via random simulations. We demonstrate our framework on two case studies involving an RGB image-based neural network controller for (a) autonomous indoor navigation, and (b) autonomous aircraft taxiing.
Create account to get full access
Overview
- Machine learning driven image-based controllers allow robots to take actions based on visual feedback from their environment.
- Understanding when these controllers might lead to safety issues is important for using them in safety-critical applications.
- Existing methods use simulation-based testing to find visual inputs that cause safety violations, but struggle with complex, high-dimensional visual inputs like RGB images.
- This paper proposes a new approach that combines simulation-based analysis with Hamilton-Jacobi (HJ) reachability methods to systematically find visual inputs that could lead to safety failures.
Plain English Explanation
Robots and other autonomous systems are increasingly using machine learning models that can process visual information, like camera images, to decide how to act. This is powerful, but it also raises safety concerns - what if the visual inputs the system sees lead it to take an action that violates safety, like crashing the robot?
The researchers in this paper wanted to develop a way to find these types of "vision-based safety failures" systematically, before deploying the systems in the real world. Existing approaches use simulation to test the system, but they struggle when the visual inputs are complex, like full-color images.
The key insight in this paper is to cast the problem of finding these safety failures as a "reachability" problem. Specifically, the researchers use a mathematical technique called Hamilton-Jacobi (HJ) reachability to compute the set of states the system could reach that would violate safety.
By combining simulation-based testing with this HJ reachability analysis, the researchers can systematically find the visual inputs that could lead to safety failures, even for complex, high-dimensional visual data like RGB images. This allows them to uncover non-trivial safety issues that might be difficult to find through random simulation alone.
Technical Explanation
The core contribution of this paper is a new framework that blends simulation-based analysis with Hamilton-Jacobi (HJ) reachability methods to systematically find visual inputs that could lead to safety violations in vision-based robotic control systems.
The researchers first define a model of the robotic system, including its dynamics, the vision-based controller, and the safety specifications. They then use simulation to generate a set of representative visual inputs and the corresponding system states.
Next, they leverage HJ reachability analysis to compute an approximation of the backward reachable tube (BRT) - the set of unsafe states that the system could reach under the vision-based controller. This BRT captures both trivial and non-trivial safety failures that may be difficult to uncover through simulation alone.
By analyzing the visual inputs that correspond to the unsafe states in the BRT, the researchers can identify the key characteristics of these "vision-based safety failures." This information can then be used to engineer corrective measures or robustness improvements for the vision-based controller.
The researchers demonstrate their framework on two case studies: an RGB image-based neural network controller for autonomous indoor navigation, and an autonomous aircraft taxiing task. Their results show that the combined simulation-HJ reachability approach can effectively detect and mitigate system-level anomalies that arise from vision-based control.
Critical Analysis
The researchers acknowledge that their approach relies on the accuracy of the system model and the safety specifications, which may be challenging to obtain in practice. Additionally, the HJ reachability computation can be computationally intensive, especially for high-dimensional state spaces.
While the case studies demonstrate the efficacy of the proposed framework, further evaluation on a wider range of robotic systems and real-world scenarios would be valuable to assess its broader applicability and scalability. The researchers also note that their current approach assumes a deterministic system model, and extending it to handle stochastic uncertainties could be an area for future research.
Overall, this paper presents a promising approach to the important problem of ensuring the safety of vision-based robotic control systems. The combination of simulation-based testing and HJ reachability analysis offers a systematic way to uncover safety-critical issues that may be difficult to expose through simulation alone, which could be a valuable tool for the safe deployment of these systems in the real world.
Conclusion
This paper introduces a novel framework that combines simulation-based analysis with Hamilton-Jacobi reachability methods to systematically find visual inputs that could lead to safety violations in vision-based robotic control systems. By computing an approximation of the backward reachable tube, the researchers can identify both trivial and non-trivial safety failures, which can then be analyzed to engineer corrective measures for the vision-based controllers.
The proposed approach addresses the scalability challenges of existing simulation-based techniques when dealing with high-dimensional and complex visual inputs, such as RGB images. The case studies demonstrate the effectiveness of this framework in uncovering safety issues for autonomous indoor navigation and aircraft taxiing tasks.
Overall, this research represents an important step towards ensuring the safe deployment of vision-based robotic systems in real-world, safety-critical applications. The insights and techniques presented in this paper could have significant implications for the future development and integration of these advanced control systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers
Aryaman Gupta, Kaustav Chakraborty, Somil Bansal
0
0
Autonomous systems, such as self-driving cars and drones, have made significant strides in recent years by leveraging visual inputs and machine learning for decision-making and control. Despite their impressive performance, these vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. Such errors can cascade to catastrophic system failures and compromise system safety. In this work, we introduce a run-time anomaly monitor to detect and mitigate such closed-loop, system-level failures. Specifically, we leverage a reachability-based framework to stress-test the vision-based controller offline and mine its system-level failures. This data is then used to train a classifier that is leveraged online to flag inputs that might cause system breakdowns. The anomaly detector highlights issues that transcend individual modules and pertain to the safety of the overall system. We also design a fallback controller that robustly handles these detected anomalies to preserve system safety. We validate the proposed approach on an autonomous aircraft taxiing system that uses a vision-based controller for taxiing. Our results show the efficacy of the proposed approach in identifying and handling system-level anomalies, outperforming methods such as prediction error-based detection, and ensembling, thereby enhancing the overall safety and robustness of autonomous systems.
4/10/2024
Verification of Neural Reachable Tubes via Scenario Optimization and Conformal Prediction
Albert Lin, Somil Bansal
0
0
Learning-based approaches for controlling safety-critical systems are rapidly growing in popularity; thus, it is important to assure their performance and safety. Hamilton-Jacobi (HJ) reachability analysis is a popular formal verification tool for providing such guarantees, since it can handle general nonlinear system dynamics, bounded adversarial system disturbances, and state and input constraints. However, its computational and memory complexity scales exponentially with the state dimension, making it intractable for large-scale systems. To overcome this challenge, neural approaches, such as DeepReach, have been used to synthesize reachable tubes and safety controllers for high-dimensional systems. However, verifying these neural reachable tubes remains challenging. In this work, we propose two verification methods, based on robust scenario optimization and conformal prediction, to provide probabilistic safety guarantees for neural reachable tubes. Our methods allow a direct trade-off between resilience to outlier errors in the neural tube, which are inevitable in a learning-based approach, and the strength of the probabilistic safety guarantee. Furthermore, we show that split conformal prediction, a widely used method in the machine learning community for uncertainty quantification, reduces to a scenario-based approach, making the two methods equivalent not only for verification of neural reachable tubes but also more generally. To our knowledge, our proof is the first in the literature to show a strong relationship between conformal prediction and scenario optimization. Finally, we propose an outlier-adjusted verification approach that uses the error distribution in neural reachable tubes to recover greater safe volumes. We demonstrate the efficacy of the proposed approaches for the high-dimensional problems of multi-vehicle collision avoidance and rocket landing with no-go zones.
4/11/2024
🏋️
Bridging Dimensions: Confident Reachability for High-Dimensional Controllers
Yuang Geng, Jake Brandon Baldauf, Souradeep Dutta, Chao Huang, Ivan Ruchkin
0
0
Autonomous systems are increasingly implemented using end-to-end learning-based controllers. Such controllers make decisions that are executed on the real system, with images as one of the primary sensing modalities. Deep neural networks form a fundamental building block of such controllers. Unfortunately, the existing neural-network verification tools do not scale to inputs with thousands of dimensions -- especially when the individual inputs (such as pixels) are devoid of clear physical meaning. This paper takes a step towards connecting exhaustive closed-loop verification with high-dimensional controllers. Our key insight is that the behavior of a high-dimensional controller can be approximated with several low-dimensional controllers. To balance the approximation accuracy and verifiability of our low-dimensional controllers, we leverage the latest verification-aware knowledge distillation. Then, we inflate low-dimensional reachability results with statistical approximation errors, yielding a high-confidence reachability guarantee for the high-dimensional controller. We investigate two inflation techniques -- based on trajectories and control actions -- both of which show convincing performance in three OpenAI gym benchmarks.
5/3/2024
Scalable Surrogate Verification of Image-based Neural Network Control Systems using Composition and Unrolling
Feiyang Cai, Chuchu Fan, Stanley Bak
0
0
Verifying safety of neural network control systems that use images as input is a difficult problem because, from a given system state, there is no known way to mathematically model what images are possible in the real-world. We build on recent work that considers a surrogate verification approach, training a conditional generative adversarial network (cGAN) as an image generator in place of the real world. This enables set-based formal analysis of the closed-loop system, providing analysis beyond simulation and testing. While existing work is effective on small examples, excessive overapproximation both within a single control period and across multiple control periods limits its scalability. We propose approaches to overcome these two sources of error. First, we overcome one-step error by composing the system's dynamics along with the cGAN and neural network controller, without losing the dependencies between input states and the control outputs as in the monotonic analysis of the system dynamics. Second, we reduce multi-step error by repeating the single-step composition, essentially unrolling multiple steps of the control loop into a large neural network. We then leverage existing network verification tools to compute accurate reachable sets for multiple steps, avoiding the accumulation of abstraction error at each step. We demonstrate the effectiveness of our approach in terms of both accuracy and scalability using two case studies: an autonomous aircraft taxiing system and an advanced emergency braking system. On the aircraft taxiing system, the converged reachable set is 175% larger using the prior baseline method compared with our proposed approach. On the emergency braking system, with 24x the number of image output variables from the cGAN, the baseline method fails to prove any states are safe, whereas our improvements enable set-based safety analysis.
5/30/2024