Towards Long Term SLAM on Thermal Imagery

Introduction

The provided text discusses the challenges of using visual Simultaneous Localization and Mapping (SLAM) systems, which rely on camera imagery, in environments with poor visibility or significant illumination changes, such as nighttime or adverse weather conditions. It proposes the use of Long-Wave Infrared (LWIR) imagery, commonly known as thermal imaging, as a promising solution to provide visibility in dark, dust-filled, or smoke-filled environments without lighting.

However, the text highlights that temperature-driven appearance changes in outdoor thermal imagery, even over a few hours, pose unique challenges, particularly in feature extraction and localization under varied environmental conditions. Existing feature-based methods are notably less effective with infrared (IR) imagery due to reduced and inconsistent feature extraction in the short term, and inverting image gradients caused by variations in LWIR energy across different objects in the long term.

The text demonstrates that inconsistent feature extraction causes the ORB-based place recognition schemes used in most state-of-the-art (SOTA) visual SLAM systems to be ineffective over temporal gaps of only a few hours. Leading feature-based SLAM systems, such as ORB-SLAM3, encounter significant difficulties. In contrast, state-of-the-art flow-based methods like DROID-SLAM provide reasonable local tracking results but lack an easily exploitable mapping/place recognition model conducive to relocalization within diurnal LWIR datasets. Other flow-based frameworks, including VINS-FUSION and Basalt, rely on BRIEF/ORB features for loop closure detection or relocalization, thus faltering with IR imagery.

The paper presents an approach to enable all-day autonomy for robotic systems using long-wave infrared (LWIR) cameras as the primary sensor. It revisits classical feature extraction techniques and highlights their limitations for LWIR imagery. The authors advocate for using Gluestick, a learning-based method, to extract and match features resistant to illumination changes.

The researchers integrate the learned feature descriptor within MCSLAM to assess its visual SLAM performance. They develop a Bag of Words (BoW) vocabulary employing SuperPoint features from LWIR images captured at various times of the day in urban, outdoor settings on the Northeastern Campus, using handheld or vehicle-mounted cameras.

To evaluate their method and compare against other SLAM systems, the authors collected comprehensive test datasets over 24-hour periods using static and mobile IR cameras with real-time kinematic (RTK) GPS ground truth. These datasets highlight the inadequacies of most existing data collections, particularly in capturing dynamic illumination conditions inherent to outdoor environments.

The experiments indicate that the Gluestick-augmented variant of MCSLAM can track features and achieve relocalization between day and night imagery.

The contributions include an extensive dataset collected with FLIR Boson thermal cameras, a BoW vocabulary using Superpoint features for effective loop closure and Visual Place Recognition (VPR) across day-night datasets, and a feature-based visual SLAM baseline using MCSLAM with Superpoint features and the Gluestick matcher, demonstrating strong local tracking and the ability to save a map during the day and accurately relocalize at night.

Related Work

The paper discusses feature matching and SLAM (Simultaneous Localization and Mapping) techniques for thermal images. Key points:

Feature Matching in Thermal Images:

• Classical feature extraction methods like ORB, SIFT, and SURF perform poorly on thermal images due to noise, low contrast, and flat field correction issues.
• Learning-based features show improvement but struggle to adapt to thermal images.
• Some methods were designed specifically for thermal imagery, but they fail to match images with dramatic intensity differences between day and night.
• The paper proposes using Gluestick, a cross-attention based matching scheme robust to challenging conditions like lighting changes and low overlap.

Thermal SLAM:

• Classical visual SLAM methods struggle with thermal images and challenging scenarios.
• Recent works fuse thermal with visible, LiDAR, or IMU data to overcome limitations, but mostly focus on odometry.
• Place recognition and loop closure are crucial for consistent long-term, multi-session mapping between day and night when images look very different.
• Bag of Words (BoW) approaches have been instrumental in efficient loop closure detection.
• The paper aims to explore day-to-night relocalization within a SLAM framework for thermal images.

The paper then describes the data collection setup for training, analysis, and benchmarking the proposed SLAM system.

Dataset

The paper describes the collection of a varied thermal imagery dataset for evaluating Day-Night relocalization and SLAM performance. Three main types of data were collected:

1. Test set of 24-hour outdoor timelapses with static cameras in semi-urban environments, captured every 10 minutes. These scenes are used for benchmarking methods where pixel-level accurate ground truth is useful.

2. A large set of monocular sequences with handheld and vehicle-mounted cameras, without ground truth trajectory information. These are used to augment the Bag-of-Words (BoW) training.

3. Three sets of matched day and night trajectories of varying sizes with stereo and side-facing cameras. These trajectories follow a pre-determined route during the day and night, allowing for place recognition evaluations between day-to-day, night-to-night, and day-to-night scenarios. Ground truth position data with RTK GPS is provided for benchmarking day-to-night and night-to-day loop closure.

The dataset was captured using FLIR Boson ADK thermal cameras with a resolution of 512x640 and a 75-degree horizontal field of view. For the paired day-night trajectories, a stereo pair with a 1.1m baseline was mounted on a Lincoln MKZ car, along with an RTK GPS antenna.

Traditional camera calibration targets were unsuitable for thermal cameras, so a wooden calibration target with a copper tape checkerboard pattern was used, allowing contrast when heated. Kalibr was employed to estimate intrinsic coefficients for all cameras and extrinsic parameters for the stereo pair.

V Method

The paper discusses several techniques used for image preprocessing, feature extraction and matching, place recognition, and SLAM pipeline development for infrared (IR) imagery.

Image Preprocessing: Raw IR images have poor contrast, so Contrast Limited Adaptive Histogram Equalization (CLAHE) is applied to increase contrast and keypoint extraction, while considering the trade-off between increasing keypoints and amplifying noise.

Feature Extraction and Matching: The Gluestick method, which uses SuperPoint features, performs significantly better than traditional methods like ORB and SIFT for matching day and night IR images. This is because learned features are more robust to noise and do not rely on gradient orientations which change with temperature. Gluestick matches lines in addition to points.

Place Recognition: An IR image vocabulary is built using DBoW2 extended for SuperPoint features matched across image pairs using Gluestick. Training data includes day and night image sequences to improve temporal generalization.

SLAM Pipeline: The Multi Camera SLAM (MCSLAM) framework is adapted by replacing ORB features with SuperPoint descriptors and improving the matching process with Gluestick. This allows using stereo IR cameras and potentially extending to camera arrays. The DBoW2 loop closure mechanism is utilized with the IR SuperPoint vocabulary for relocalization.

Experiments

The paper demonstrates a place recognition system and its effectiveness for infrared (IR) imagery. The key points are:

Evaluation on Loop Closure:

• Tested on the KRI dataset by splitting trajectories into two loops.
• Built a database from the first loop and searched it with queries from the second loop.
• Achieved 100% recall for day-day and night-night loop closures with their IR SuperPoint vocabulary, comparable to ORB features.
• For significant day-night temporal gaps, their method showed 91-93% recall while ORB features performed poorly (10-12% recall).

SLAM Evaluation:

• Tested their augmented MCSLAM system on IR trajectories for front-end tracking and relocalization across day-night gaps.
• Front-end tracking generated reasonably accurate trajectories, though tracking was lost in low-texture regions.
• For relocalization, they used the day map and relocalized the night trajectory, achieving < 3m error in most cases relative to GPS ground truth.
• Larger relocalization errors occurred when visual features were mainly on distant objects.

The paper demonstrates the effectiveness of their IR place recognition system, enabling loop closures and map reuse across significant day-night changes, which traditional methods like ORB struggle with for thermal imagery.

Conclusion and Future Work

The paper discusses the ability to achieve challenging loop closure and relocalization, enabling map reuse, within long-term infrared (IR) datasets. This is accomplished using a bag-of-words (BoW) system, making it suitable for relatively simple incorporation into existing simultaneous localization and mapping (SLAM) systems. The baseline SLAM system can generate maps using Gluestick for data association, outperforming feature-based SLAM systems that use binary descriptors.

The paper suggests future work to optimize the system for memory usage and speed, as Gluestick is not a perfect drop-in replacement for efficient matching schemes used in other SLAM methods. Improvements and optimizations could be made regarding matching across more than two frames and matching between cameras with known extrinsic parameters.

Additionally, the paper acknowledges that the datasets used are not comprehensive. Future plans include collecting larger, more diverse datasets that include other sensing modalities for comparison, enabling the retraining or fine-tuning of feature extraction and matching, building better vocabularies, and conducting more thorough analysis.