DerainNeRF: 3D Scene Estimation with Adhesive Waterdrop Removal

INTRODUCTION

The paper discusses the importance of 3D scene representation and estimation techniques in various applications, such as autonomous driving, robotics, virtual reality (VR), and cultural heritage preservation. Neural Radiance Fields (NeRF) have gained popularity for 3D reconstruction and scene representation due to their ability to provide continuous scene representation, handle complex scenes, and achieve state-of-the-art performance in novel-view image synthesis.

However, in real-world scenarios involving outdoor images, particularly those captured during rainy or snowy conditions, the presence of adhesive raindrops can degrade the performance of related applications like 3D reconstruction, visual perception, object detection, and tracking. To address this limitation, the paper proposes a method to simultaneously remove the adhesive raindrops from captured images and recover the underlying 3D scene implicitly, leveraging the impressive 3D scene representation capability of NeRF.

The paper proposes DerainNeRF, a NeRF-based framework that simultaneously estimates 3D scenes while removing waterdrops from images. The vanilla NeRF framework is not robust to images with adhesive waterdrops, which have random spatial distribution, irregular shapes, and complicated refraction and reflection properties.

The proposed method combines a waterdrop detection network and NeRF for 3D scene representation learning. It first uses a pre-trained deep waterdrop detector to predict the locations of waterdrops. Then, it excludes the waterdrop-covered pixels during the training of NeRF, allowing it to recover clear scenes from non-occluded pixels.

DerainNeRF is evaluated using synthetic and real datasets. The experimental results demonstrate that it effectively estimates clear 3D scenes from waterdrop images and renders novel-view clear images. Both quantitative and qualitative results show that DerainNeRF delivers superior quality compared to existing state-of-the-art image waterdrop removal methods. It is the first NeRF-based method that takes waterdrop-degraded images as input and recovers the clear scene implicitly.

RELATED WORK

The paper reviews prior work in two main areas related to the proposed method: Neural Radiance Fields (NeRF) and image adhesive waterdrop removal.

On NeRF:

• NeRF is a technique for 3D scene representation and novel view synthesis using a neural network to model the scene's volumetric radiance field.
• Many variants of NeRF have been proposed to handle large-scale scenes, challenging imaging conditions, high dynamic range, and scene editing.
• Some methods extend NeRF to handle refractive objects like water drops by modeling light refraction.
• Prior work on occlusion removal with NeRF either requires user masking or additional networks to separate backgrounds.

On image adhesive waterdrop removal:

• Early methods modeled waterdrop geometry, refraction, and reflection properties.
• Other approaches used temporal or spatial features to separate waterdrops from backgrounds.
• Recent deep learning methods use convolutional neural networks, attention mechanisms, and adversarial training for waterdrop removal.
• However, existing methods have limitations in handling large waterdrop areas or leveraging global spatial information.

No text is provided to summarize for the image adhesive waterdrop removal section.

METHOD

The paper proposes a method called DerainNeRF to remove waterdrops from multi-view images. It handles two scenarios: waterdrops fixed in the scene while the camera moves, and waterdrops static relative to the camera lens.

The method first uses the AttGAN model to detect image regions covered by waterdrops. It then excludes those regions from the training of NeRF, a technique for estimating 3D scene geometry and appearance from multi-view images.

For each input image, a binary mask is generated from the AttGAN attention map indicating waterdrop probability. NeRF is trained by masking the photometric loss for pixels covered by waterdrops according to this mask.

When waterdrops adhere to the camera lens, an additional mask is computed by averaging the attention maps over multiple frames. This accounts for areas where some waterdrops were missed due to over/underexposure in individual frames.

After training, DerainNeRF can recover the clear 3D scene without waterdrops and render novel views from arbitrary camera poses.

V EXPERIMENTS

The provided text discusses the implementation details and experimental results of the proposed DerainNeRF method for removing waterdrops from images and reconstructing clear 3D scenes. Key points are:

Implementation Details:

• Pre-trained waterdrop detector from AttGAN is used to obtain binary masks.
• ADAM optimizer is used with a decaying learning rate for DerainNeRF training.
• 200K iterations with 1024 rays per batch on an RTX 3090 GPU.
• COLMAP is used for estimating camera poses from input images.

Datasets:

• Synthetic dataset from Blender scenes with simulated waterdrops, including fixed waterdrops and moving waterdrops scenarios.
• Real indoor dataset captured with a camera and waterdrop-covered glass, including fixed glass and moving glass scenarios.
• Real outdoor dataset from a vehicle in rainy conditions with waterdrops on the camera lens.

Results:

• Qualitative and quantitative comparisons against NeRF, AttGAN, and other SOTA methods on synthetic and real datasets.
• DerainNeRF outperforms other methods in PSNR and LPIPS metrics on the synthetic dataset.
• DerainNeRF effectively removes waterdrops of various sizes and shapes on real indoor and outdoor datasets.

Ablation Study:

• Analyzes the effect of mask enhancement using average attention maps.
• Mask enhancement improves the quality of synthesized images, especially with dense waterdrops.

CONCLUSIONS

The paper introduces DerainNeRF, a novel approach for 3D scene estimation from multi-view images degraded by waterdrops, utilizing the Neural Radiance Fields (NeRF) representation. The method addresses the challenge of waterdrop removal through a comprehensive pipeline. First, a pre-trained waterdrop detector identifies and localizes waterdrops in the input images. Then, the approach estimates clear scenes by leveraging a NeRF-based network that exploits non-occluded pixels. The authors conducted a thorough evaluation of their proposed method against existing state-of-the-art techniques for image waterdrop removal, using both synthetic and real datasets. The experimental results demonstrate the superior performance of DerainNeRF compared to existing approaches.