Stable Surface Regularization for Fast Few-Shot NeRF

Introduction

Neural Radiance Fields (NeRF) is a technique that uses Multi-Layer Perceptrons (MLPs) and a large number of training images to encode the appearance and geometry of a scene. NeRF has demonstrated its effectiveness in various applications, such as novel view synthesis, surface reconstruction, dynamic scene rendering, and lighting. However, building a neural implicit field with fewer input images is challenging, and the optimization process required for each scene can take over 10 hours, making it difficult to apply in real-world applications.

The paper discusses strategies to overcome issues in few-shot setups, such as unobserved viewpoint regularization, entropy minimization, and geometric priors. It also presents voxel representation and hash encoding as methods for fast training in novel-view synthesis tasks. However, these previous approaches require long training times or struggle with sparse input views.

The paper's key contribution is the proposal of a novel method called Annealing Signed Distance Function (ASDF) loss, which addresses the limitations of the Eikonal loss when applied to few-shot NeRF training. The Eikonal loss can fail to capture reliable color and geometry information in such settings, especially in voxel-based environments with sparse inputs. The ASDF loss enforces strong geometric smoothing early in training, enabling coarse-to-fine surface regularization and stable convergence.

By incorporating the ASDF loss, along with dense 3D predictions and multi-view consistency, the proposed algorithm can successfully synthesize novel views from few-shot training, while maintaining fast training times. The paper claims the approach demonstrates comparable performance with 30-45 times faster training speeds compared to previous methods.

Related Works

The paper discusses the recent advancements in the field of neural implicit representation, which has led to the development of several approaches to enhance the capabilities of this representation. Specifically, the paper mentions that methods like multi-view consistency, optimization based on camera pose errors, and surface reconstruction have been used to improve the performance of neural implicit representation.

The paper then focuses on the limitations of NeRF, a neural radiance field approach, which requires dense inputs for high-quality novel view synthesis and shows limited performance with sparse inputs. To address this issue, the paper discusses the use of geometric constraints and regularization techniques, such as patch-based geometry and color regularization, ray entropy minimization, and the use of pseudo-geometry and color in unseen views.

The paper also mentions that additional constraints, such as geometric priors, have been adapted to NeRF to improve its performance. Specifically, the paper discusses the use of a monocular depth oracle network, depth priors in indoor multi-view stereo, and surface normal information to enhance the optimization and performance of NeRF.

Finally, the paper states that by utilizing full information from multi-view consistency and geometric cues from sparse input views with a strong geometric smoothing loss, the authors have successfully addressed the limitations of NeRF.

Preliminary

This paper introduces NeRF, a pioneering technique for encoding the appearance and geometry of a scene within a multilayer perceptron (MLP) network to synthesize novel views. Given posed images, the MLP is trained to output color and density at a given query point. The paper proposes a differential volumetric rendering method to synthesize the color of a ray. Instead of using density, this paper designs a unique way of learning the signed distance function (SDF) to represent the scene geometry. The SDF value and its gradient with respect to the query point are used to compute the opaqueness and surface normal at each point, respectively.

Method

The paper uses OmniData [5] to extract geometric priors from the given RGB images. It then uses COLMAP [23] to obtain sparse 3D points and camera poses. To integrate this information, the method constructs multi-level feature volume grids and MLP decoders for SDF and color. The feature of query points along camera rays are sampled using trilinear interpolation, and the results are rendered with the MLP decoders. All of these processes are differentiable, allowing optimization of the feature grids and decoders using loss functions. An overview of the method is provided in Figure 3.

The paper discusses a method for surface regularization in few-shot neural radiance field (NeRF) training. It addresses the issue that the commonly used Eikonal loss is ineffective when only sparse input views are available, leading to unstable optimization and low-fidelity results.

To remedy this, the paper proposes the Annealing Signed Distance Function (ASDF) loss, which consists of two components:

1. Geometric smoothing loss: This enforces the estimated signed distance function (SDF) values to match the distance between query points and the rendered surface. The smoothing is applied adaptively, decreasing over the training process to allow the network to first optimize the coarse structure and then refine the details.

2. Weighted Eikonal loss: This regularizes the gradient of the SDF to be close to 1, as in the standard Eikonal loss.

The paper also describes the overall network architecture, which uses multi-level feature grids and MLPs to predict the SDF and color at query points. The total loss function combines the ASDF loss with rendering losses for color and normal predictions, as well as a depth loss.

The proposed method aims to enable stable and high-fidelity few-shot NeRF training by providing reliable surface supervision through the ASDF loss.

Experiments

The paper discusses the implementation details and experimental results of the proposed neural radiance field (NeRF) system.

Implementation Details:

• The system was implemented using a single TITAN RTX 24GB GPU, with adjustments to the voxel sizes for each scene to fit the GPU memory and improve performance.
• The system utilized a multi-resolution feature grid with i=4 for both SDF and color feature grids.
• The decoders were implemented using MLPs with a single layer containing 128 channels.
• The learning rates for the decoder and feature grids were set to 0.001 and 0.01, respectively.
• For the surface reconstruction parameter, an initial standard deviation of 0.3 and a learning rate of 0.001 were used.
• The Adam optimizer was used to optimize the system.
• The paper provides the values used for various loss function components, such as λN, λD, λASDF, λwEik, and iopt.
• A customized grid sampler was employed due to the lack of differentiable grid sampling in PyTorch.

Dataset and Few-Shot Training Scheme:

• The system was trained and evaluated on the ScanNet and NeRF-Real datasets, following the protocols used in previous works.
• The ScanNet dataset was used with 18-20 training images per scene and 8 images for validation and testing.
• The NeRF-Real dataset was used with 2, 5, and 10 views per scene for training.

Impact of ASDF Loss:

• The use of the ASDF loss helped stabilize the optimization process and improved the quality of the rendered color and depth images, especially in few-shot learning scenarios.

Comparison with State-of-the-Art:

• The proposed system was compared with NeRF, DSNeRF, DDPNeRF, and SCADE on the ScanNet and NeRF-Real datasets.
• The quantitative results showed that the

Conclusion

The paper presents a fast few-shot NeRF (Neural Radiance Fields) method that combines deep dense priors and structure from motion. To address the challenge of optimizing geometric information from sparse input views, the method introduces a new surface regularization loss called the Annealing Signed Distance Function (ASDF) loss. This loss enforces geometric smoothing and improves the performance of novel view synthesis with sparse input views.

The technique successfully connects deep dense priors, multi-view consistency, and multi-resolution voxel grids for novel view synthesis. It can be further enhanced by adapting recent approaches to improve optimization speed. Handling uncertainty in geometric priors may also boost performance by reducing errors.

A limitation of the method is that the ASDF loss requires hyperparameters that depend on the scene geometry or structure from motion results, such as the accuracy of camera poses. Solving this issue in an adaptive manner without heuristic tuning is identified as a potential future direction.

The work was supported by the National Research Foundation of Korea.