NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising

Introduction

This text discusses the progress and important properties of visual Simultaneous Localization and Mapping (SLAM) systems. SLAM systems need to incrementally build an accurate 3D model of the scene and estimate the camera position in real-time. They must also handle noisy and incomplete sensor data, and scale to large environments. The ability to synthesize novel views can benefit applications like virtual reality. Overall, the paper highlights the key requirements for real-world deployment of robust and scalable SLAM systems.

The text discusses existing visual SLAM (Simultaneous Localization and Mapping) systems and the challenges they face in representing the scene effectively. Sparse and dense SLAM systems can perform real-time pose estimation and loop closing, but they struggle to capture essential information, resulting in incomplete scene representations.

Recent advancements in deep learning have led to the development of learning-based SLAM systems, such as CodeSLAM and SceneCode, which aim to improve scene representation capabilities. Neural radiance fields (NeRF) is a promising recent technology that can encode scene geometry in fine detail using differentiable rendering and multi-layer perceptrons (MLP).

The text identifies two key challenges for dense visual SLAM: 1) the inherent limitations of consumer-grade RGB-D sensors, which produce sparse and noisy depth images, and 2) the limitations of existing methods in tracking performance within real-world indoor scenes, as they use a random pixel selection strategy.

To address these challenges, the proposed NeSLAM system includes the following:

1. A depth completion and denoising network that generates dense and precise depth images with depth uncertainty information. This geometry prior helps guide the neural point sampling and optimization process.

2. A NeRF-based self-supervised feature tracking network designed for accurate and real-time camera tracking in complex indoor environments. This network leverages the strengths of NeRF and feature tracking to enhance the system's generalization capability.

The proposed NeSLAM system demonstrates superior performance compared to recent implicit mapping approaches on various indoor RGB-D datasets.

Related Work

This paper discusses visual SLAM (Simultaneous Localization and Mapping) systems, which construct maps and track the camera's location in real-time. Traditional visual SLAM methods rely on the constructed maps, such as PTAM which uses keyframes, feature matching, and camera localization. Some sparse mapping methods use keypoints for tracking, mapping, and loop closing, making them robust to motion clutter and large indoor environments.

For learning-based SLAM, DTAM pioneered the use of dense maps and view-centric scene representation. More recent dense SLAM systems like Bundle-Fusion and Ba-Net employ bundle adjustment for pose estimation. Other methods propose new scene geometry representations, such as latent codes or probabilistic fields, to improve accuracy.

In contrast, this paper uses an implicit scene mapping approach, which allows for more accurate geometry representation and novel view synthesis. This builds on recent work in neural radiance fields (NeRF) for novel view synthesis, but addresses the challenge of poor surface prediction for reconstruction tasks. The paper discusses methods that combine NeRF with 3D geometry representation, such as signed distance fields, to enable more efficient sampling and accurate surface reconstruction without requiring ground-truth camera poses.

The paper also reviews recent work on estimating camera poses from NeRF, such as iNeRF, NeRF--, and BARF. Going further, the paper mentions iMAP and NICE-SLAM, which combine neural implicit mapping with SLAM. The key differences in this paper's approach are the use of a depth completion and denoising network for more accurate reconstruction, and a self-supervised feature tracking method for robust pose estimation.

Method

The key points from the provided text are:

System Overview:

• The system uses a three-level hierarchical feature grid and corresponding decoders to represent scene geometry and color.
• A depth completion and denoising network is used to estimate dense depth and depth uncertainty, which guide neural point sampling and NeRF optimization.
• A self-supervised feature tracking method is used for robust and accurate camera pose estimation.

Depth Completion and Denoising Network:

• The network uses sparse and noisy input depth images and RGB images to predict dense depth, depth uncertainty, and confidence maps.
• The network has a two-head encoder-decoder architecture with mirror connections to propagate spatial information.
• The network is trained using a negative log-likelihood loss on the depth and uncertainty predictions.

Neural Scene Representation:

• The system uses a hierarchical, multi-level grid feature representation with corresponding MLPs for scene geometry.
• The geometry is represented using signed distance fields (SDF) to improve representation ability.
• An additional feature grid and decoder are used for color representation.
• The system uses differentiable rendering to integrate the SDF values and colors for scene representation.

Optimization:

• The system jointly optimizes the scene feature grids, color decoder, and camera poses using carefully designed loss functions.
• Losses include depth, color, Eikonal, and ICP losses, as well as patch-wise variants for better convergence.
• The system incrementally updates the neural networks online during operation.

In summary, the system uses a novel depth completion network, hierarchical scene representation, and self-supervised feature tracking to enable robust and accurate 3D reconstruction and camera tracking.

V Experiments

This section provides implementation details and an evaluation of the proposed method on various datasets. Key points:

• The authors used specific hyperparameter settings for the number of sampling points, loss weightings, and optimizers across different datasets.

• They evaluated the method on several datasets, including Replica, ScanNet, and TUM RGB-D, to assess scene reconstruction, view synthesis, and camera tracking performance.

• On the Replica dataset, the proposed method outperformed prior SLAM systems in accuracy, completeness, and depth estimation.

• On ScanNet and the authors' own real-world dataset, the method achieved better camera tracking accuracy compared to prior NeRF-based SLAM systems.

• The authors conducted an ablation study to verify the effectiveness of the depth denoising and completion network, hierarchical scene representation with SDF, self-supervised feature tracking, and the loss function design.

• The results show these components significantly improve the reconstruction, view synthesis, and camera tracking performance of the proposed approach.

Conclusion

The paper proposes a dense SLAM system called NeSLAM that combines neural implicit scene representation with the SLAM system. It includes a depth denoising and completion network and a self-supervised feature tracking network.

The depth network provides dense depth images with depth uncertainty, which can guide the neural point sampling and enhance scene geometry consistency. The system also incorporates the Signed Distance Field (SDF) value into the hierarchical feature grid to better represent scene geometry.

The proposed NeRF-based self-supervised feature tracking network enables accurate camera tracking and enhances the robustness of the system. Extensive experiments demonstrate the system's effectiveness and accuracy in scene reconstruction, tracking, and view synthesis in complex indoor scenes.

The future work will focus on dynamic scenes, aiming to achieve high reconstruction and localization accuracy.