Fusing Multi-sensor Input with State Information on TinyML Brains for Autonomous Nano-drones
2404.02567
0
0
Abstract
Autonomous nano-drones (~10 cm in diameter), thanks to their ultra-low power TinyML-based brains, are capable of coping with real-world environments. However, due to their simplified sensors and compute units, they are still far from the sense-and-act capabilities shown in their bigger counterparts. This system paper presents a novel deep learning-based pipeline that fuses multi-sensorial input (i.e., low-resolution images and 8x8 depth map) with the robot's state information to tackle a human pose estimation task. Thanks to our design, the proposed system -- trained in simulation and tested on a real-world dataset -- improves a state-unaware State-of-the-Art baseline by increasing the R^2 regression metric up to 0.10 on the distance's prediction.
Create account to get full access
Overview
- This paper explores techniques for fusing multi-sensor input and state information on tiny machine learning (TinyML) systems for autonomous nano-drones.
- The researchers developed a system that combines data from various onboard sensors and the drone's flight state to enable autonomous navigation and control.
- The system was tested on nano-drones, demonstrating its potential for enabling advanced capabilities on resource-constrained embedded devices.
Plain English Explanation
The paper focuses on creating intelligent control systems for tiny, lightweight drones called nano-drones. These nano-drones are designed to be very small and maneuverable, but they have limited computing power and memory compared to larger drones.
To make these nano-drones more autonomous and capable, the researchers developed a way to fuse, or combine, data from multiple sensors on the drone, such as cameras, gyroscopes, and altimeters. This sensor data is combined with information about the drone's current state, like its position, speed, and orientation. By using both sensor inputs and state information, the nano-drone can make more informed decisions about how to navigate its environment and carry out tasks.
The key innovation is that this data fusion and decision-making happens on the nano-drone itself, using a type of artificial intelligence called TinyML. TinyML allows complex AI models to run on small, power-efficient microcontrollers, rather than requiring a connection to a powerful remote computer. This enables the nano-drones to be truly autonomous, without needing to constantly communicate with a base station.
The researchers tested their system on real nano-drones, demonstrating that it can enable advanced autonomous capabilities even on these tiny, resource-constrained devices. This could pave the way for nano-drones to be used in a variety of applications, from search and rescue to environmental monitoring, where their small size and maneuverability are advantageous.
Technical Explanation
The paper presents a system for fusing multi-sensor input, including cameras, inertial measurement units (IMUs), and altimeters, with state information on TinyML-based control systems for autonomous nano-drones.
The researchers developed a neural network architecture that takes in data from the various onboard sensors as well as the drone's current state, including its position, velocity, and orientation. This information is processed through several neural network layers to produce control commands that enable autonomous navigation and task completion.
A key aspect of the system is its ability to run entirely on the nano-drone's embedded processor, without the need for a powerful off-board computer. This was achieved by designing a compact neural network model that can be deployed on resource-constrained TinyML hardware.
The researchers evaluated their system through extensive experiments on a fleet of nano-drones, assessing its performance on tasks like obstacle avoidance, target tracking, and mapping. The results demonstrate the effectiveness of fusing multi-sensor data and state information to achieve robust autonomous control, even on tiny drone platforms.
Critical Analysis
The paper provides a thorough technical explanation of the researchers' approach and presents compelling experimental results. However, it does not delve deeply into potential limitations or areas for further exploration.
For example, the paper does not discuss how the system might handle sensor failures or degraded performance, which could be a significant concern for safety-critical applications of nano-drones. Additionally, the evaluation is conducted in a relatively controlled laboratory setting, and the researchers acknowledge the need for further testing in more complex, real-world environments.
While the TinyML-based approach is a noteworthy achievement, the long-term viability and scalability of the system could be influenced by factors such as power consumption, computational efficiency, and the availability of suitable hardware platforms. Further research into these areas would help strengthen the practical applicability of the proposed techniques.
Conclusion
This paper presents a innovative solution for enabling advanced autonomous capabilities on resource-constrained nano-drones. By fusing multi-sensor input with state information using a TinyML-based control system, the researchers have demonstrated the potential for these tiny drones to navigate complex environments and carry out a variety of tasks without the need for constant human supervision or off-board computing resources.
The techniques described in this work could have far-reaching implications for the development of autonomous systems in a wide range of domains, from search and rescue operations to environmental monitoring and beyond. As nano-drone technology continues to evolve, the ability to endow these small, agile platforms with intelligent, self-sufficient control could unlock new frontiers in robotics and unmanned systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🤿
New!A Deep Learning-based Pest Insect Monitoring System for Ultra-low Power Pocket-sized Drones
Luca Crupi, Luca Butera, Alberto Ferrante, Daniele Palossi
0
0
Smart farming and precision agriculture represent game-changer technologies for efficient and sustainable agribusiness. Miniaturized palm-sized drones can act as flexible smart sensors inspecting crops, looking for early signs of potential pest outbreaking. However, achieving such an ambitious goal requires hardware-software codesign to develop accurate deep learning (DL) detection models while keeping memory and computational needs under an ultra-tight budget, i.e., a few MB on-chip memory and a few 100s mW power envelope. This work presents a novel vertically integrated solution featuring two ultra-low power System-on-Chips (SoCs), i.e., the dual-core STM32H74 and a multi-core GWT GAP9, running two State-of-the-Art DL models for detecting the Popillia japonica bug. We fine-tune both models for our image-based detection task, quantize them in 8-bit integers, and deploy them on the two SoCs. On the STM32H74, we deploy a FOMO-MobileNetV2 model, achieving a mean average precision (mAP) of 0.66 and running at 16.1 frame/s within 498 mW. While on the GAP9 SoC, we deploy a more complex SSDLite-MobileNetV3, which scores an mAP of 0.79 and peaks at 6.8 frame/s within 33 mW. Compared to a top-notch RetinaNet-ResNet101-FPN full-precision baseline, which requires 14.9x more memory and 300x more operations per inference, our best model drops only 15% in mAP, paving the way toward autonomous palm-sized drones capable of lightweight and precise pest detection.
7/2/2024
High-throughput Visual Nano-drone to Nano-drone Relative Localization using Onboard Fully Convolutional Networks
Luca Crupi, Alessandro Giusti, Daniele Palossi
0
0
Relative drone-to-drone localization is a fundamental building block for any swarm operations. We address this task in the context of miniaturized nano-drones, i.e., 10cm in diameter, which show an ever-growing interest due to novel use cases enabled by their reduced form factor. The price for their versatility comes with limited onboard resources, i.e., sensors, processing units, and memory, which limits the complexity of the onboard algorithms. A traditional solution to overcome these limitations is represented by lightweight deep learning models directly deployed aboard nano-drones. This work tackles the challenging relative pose estimation between nano-drones using only a gray-scale low-resolution camera and an ultra-low-power System-on-Chip (SoC) hosted onboard. We present a vertically integrated system based on a novel vision-based fully convolutional neural network (FCNN), which runs at 39Hz within 101mW onboard a Crazyflie nano-drone extended with the GWT GAP8 SoC. We compare our FCNN against three State-of-the-Art (SoA) systems. Considering the best-performing SoA approach, our model results in an R-squared improvement from 32 to 47% on the horizontal image coordinate and from 18 to 55% on the vertical image coordinate, on a real-world dataset of 30k images. Finally, our in-field tests show a reduction of the average tracking error of 37% compared to a previous SoA work and an endurance performance up to the entire battery lifetime of 4 minutes.
4/3/2024
OptiState: State Estimation of Legged Robots using Gated Networks with Transformer-based Vision and Kalman Filtering
Alexander Schperberg, Yusuke Tanaka, Saviz Mowlavi, Feng Xu, Bharathan Balaji, Dennis Hong
0
0
State estimation for legged robots is challenging due to their highly dynamic motion and limitations imposed by sensor accuracy. By integrating Kalman filtering, optimization, and learning-based modalities, we propose a hybrid solution that combines proprioception and exteroceptive information for estimating the state of the robot's trunk. Leveraging joint encoder and IMU measurements, our Kalman filter is enhanced through a single-rigid body model that incorporates ground reaction force control outputs from convex Model Predictive Control optimization. The estimation is further refined through Gated Recurrent Units, which also considers semantic insights and robot height from a Vision Transformer autoencoder applied on depth images. This framework not only furnishes accurate robot state estimates, including uncertainty evaluations, but can minimize the nonlinear errors that arise from sensor measurements and model simplifications through learning. The proposed methodology is evaluated in hardware using a quadruped robot on various terrains, yielding a 65% improvement on the Root Mean Squared Error compared to our VIO SLAM baseline. Code example: https://github.com/AlexS28/OptiState
4/30/2024
Deep learning empowered sensor fusion to improve infant movement classification
Tomas Kulvicius, Dajie Zhang, Luise Poustka, Sven Bolte, Lennart Jahn, Sarah Flugge, Marc Kraft, Markus Zweckstetter, Karin Nielsen-Saines, Florentin Worgotter, Peter B Marschik
0
0
There is a recent boom in the development of AI solutions to facilitate and enhance diagnostic procedures for established clinical tools. To assess the integrity of the developing nervous system, the Prechtl general movement assessment (GMA) is recognized for its clinical value in diagnosing neurological impairments in early infancy. GMA has been increasingly augmented through machine learning approaches intending to scale-up its application, circumvent costs in the training of human assessors and further standardize classification of spontaneous motor patterns. Available deep learning tools, all of which are based on single sensor modalities, are however still considerably inferior to that of well-trained human assessors. These approaches are hardly comparable as all models are designed, trained and evaluated on proprietary/silo-data sets. With this study we propose a sensor fusion approach for assessing fidgety movements (FMs) comparing three different sensor modalities (pressure, inertial, and visual sensors). Various combinations and two sensor fusion approaches (late and early fusion) for infant movement classification were tested to evaluate whether a multi-sensor system outperforms single modality assessments. The performance of the three-sensor fusion (classification accuracy of 94.5%) was significantly higher than that of any single modality evaluated, suggesting the sensor fusion approach is a promising avenue for automated classification of infant motor patterns. The development of a robust sensor fusion system may significantly enhance AI-based early recognition of neurofunctions, ultimately facilitating automated early detection of neurodevelopmental conditions.
6/24/2024