Dual Degradation Representation for Joint Deraining and Low-Light Enhancement in the Dark

Read original: arXiv:2305.03997 - Published 6/18/2024 by Xin Lin, Jingtong Yue, Sixian Ding, Chao Ren, Lu Qi, Ming-Hsuan Yang

🤿

Overview

Challenges of real-world applications like autonomous driving, surveillance, and night photography in low-light and rainy conditions
Limitations of existing methods that struggle to handle both low-light enhancement and rain removal simultaneously
Introducing L²RIRNet, an end-to-end model designed to address these challenges

Plain English Explanation

Many real-world applications, such as autonomous driving, surveillance systems, and night photography, face significant challenges when operating in low-light and rainy conditions. Existing methods that try to either brighten low-light areas or remove rain often struggle to do both simultaneously, leading to problematic results like blurred or overexposed images.

To address these challenges, the researchers have developed an end-to-end model called L²RIRNet. This model is designed to handle both low-light enhancement and rain removal in a single process, rather than using a two-step approach. The key components of L²RIRNet are:

Dual Degradation Representation Network (DDR-Net): This part of the model learns to independently represent the degradation caused by low light and the degradation caused by rain. This dual representation helps guide the restoration process.
Restoration Network: This part of the model uses a Fourier Detail Guidance (FDG) module to leverage detailed, near-rainless images to inform the restoration of the degraded image, focusing on texture details in both the frequency and spatial domains.

By combining these two key components, L²RIRNet is able to effectively address the challenges of low-light and rain in real-world scenarios, as demonstrated by the researchers' experiments.

Technical Explanation

The paper introduces an end-to-end model called L²RIRNet, which is designed to handle both low-light enhancement and rain removal in a single process. The model consists of two main components:

Dual Degradation Representation Network (DDR-Net): This part of the model learns to independently represent the degradation caused by low light and the degradation caused by rain. The DDR-Net employs a dual degradation loss to guide this learning process, ensuring the model can effectively capture both types of degradation.
Restoration Network: This part of the model uses a Fourier Detail Guidance (FDG) module to leverage detailed, near-rainless images to inform the restoration of the degraded image. The FDG module focuses on restoring texture details in both the frequency and spatial domains, which is crucial for addressing the challenges of low-light and rain.

Additionally, the researchers contribute a dataset containing both synthetic and real-world low-light-rainy images, which they use to extensively evaluate their L²RIRNet model. The experiments demonstrate that L²RIRNet outperforms existing methods in both synthetic and complex real-world scenarios.

Critical Analysis

The paper presents a compelling approach to addressing the challenges of low-light and rain in real-world applications. The researchers' decision to develop an end-to-end model, rather than a two-step approach, is a strength, as it allows the model to learn the interdependencies between low-light enhancement and rain removal.

However, the paper does not address some potential limitations or areas for further research. For example, it would be interesting to see how L²RIRNet performs on a wider range of real-world conditions, such as cross-view nighttime image deraining, or to explore the model's robustness to more severe degradation scenarios.

Additionally, the researchers could have delved deeper into the trade-offs and design choices behind the DDR-Net and Restoration Network components, providing more insight into the rationale behind their architectural decisions.

Overall, the L²RIRNet model represents a significant step forward in addressing the challenges of low-light and rain in real-world applications. While the paper has some areas that could be explored further, it demonstrates the potential of end-to-end approaches to tackle complex computer vision problems.

Conclusion

The L²RIRNet model introduced in this paper offers a promising solution to the challenges of low-light and rain in real-world applications such as autonomous driving, surveillance, and night photography. By combining a Dual Degradation Representation Network and a Restoration Network with a Fourier Detail Guidance module, the model is able to effectively handle both low-light enhancement and rain removal simultaneously, outperforming existing methods.

The researchers' contribution of a new dataset containing synthetic and real-world low-light-rainy images also adds valuable resources for further research in this area. While the paper leaves some aspects open for further exploration, the L²RIRNet model demonstrates the potential of end-to-end approaches to tackle complex computer vision problems and paves the way for more advanced solutions in the future.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

🤿

Dual Degradation Representation for Joint Deraining and Low-Light Enhancement in the Dark

Xin Lin, Jingtong Yue, Sixian Ding, Chao Ren, Lu Qi, Ming-Hsuan Yang

Rain in the dark poses a significant challenge to deploying real-world applications such as autonomous driving, surveillance systems, and night photography. Existing low-light enhancement or deraining methods struggle to brighten low-light conditions and remove rain simultaneously. Additionally, cascade approaches like ``deraining followed by low-light enhancement'' or the reverse often result in problematic rain patterns or overly blurred and overexposed images. To address these challenges, we introduce an end-to-end model called L$^{2}$RIRNet, designed to manage both low-light enhancement and deraining in real-world settings. Our model features two main components: a Dual Degradation Representation Network (DDR-Net) and a Restoration Network. The DDR-Net independently learns degradation representations for luminance effects in dark areas and rain patterns in light areas, employing dual degradation loss to guide the training process. The Restoration Network restores the degraded image using a Fourier Detail Guidance (FDG) module, which leverages near-rainless detailed images, focusing on texture details in frequency and spatial domains to inform the restoration process. Furthermore, we contribute a dataset containing both synthetic and real-world low-light-rainy images. Extensive experiments demonstrate that our L$^{2}$RIRNet performs favorably against existing methods in both synthetic and complex real-world scenarios. All the code and dataset can be found in url{https://github.com/linxin0/Low_light_rainy}.

6/18/2024

🖼️

Not Just Streaks: Towards Ground Truth for Single Image Deraining

Yunhao Ba, Howard Zhang, Ethan Yang, Akira Suzuki, Arnold Pfahnl, Chethan Chinder Chandrappa, Celso de Melo, Suya You, Stefano Soatto, Alex Wong, Achuta Kadambi

We propose a large-scale dataset of real-world rainy and clean image pairs and a method to remove degradations, induced by rain streaks and rain accumulation, from the image. As there exists no real-world dataset for deraining, current state-of-the-art methods rely on synthetic data and thus are limited by the sim2real domain gap; moreover, rigorous evaluation remains a challenge due to the absence of a real paired dataset. We fill this gap by collecting a real paired deraining dataset through meticulous control of non-rain variations. Our dataset enables paired training and quantitative evaluation for diverse real-world rain phenomena (e.g. rain streaks and rain accumulation). To learn a representation robust to rain phenomena, we propose a deep neural network that reconstructs the underlying scene by minimizing a rain-robust loss between rainy and clean images. Extensive experiments demonstrate that our model outperforms the state-of-the-art deraining methods on real rainy images under various conditions. Project website: https://visual.ee.ucla.edu/gt_rain.htm/.

7/30/2024

Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining

Xiang Chen, Jinshan Pan, Jiangxin Dong

How to effectively explore multi-scale representations of rain streaks is important for image deraining. In contrast to existing Transformer-based methods that depend mostly on single-scale rain appearance, we develop an end-to-end multi-scale Transformer that leverages the potentially useful features in various scales to facilitate high-quality image reconstruction. To better explore the common degradation representations from spatially-varying rain streaks, we incorporate intra-scale implicit neural representations based on pixel coordinates with the degraded inputs in a closed-loop design, enabling the learned features to facilitate rain removal and improve the robustness of the model in complex scenarios. To ensure richer collaborative representation from different scales, we embed a simple yet effective inter-scale bidirectional feedback operation into our multi-scale Transformer by performing coarse-to-fine and fine-to-coarse information communication. Extensive experiments demonstrate that our approach, named as NeRD-Rain, performs favorably against the state-of-the-art ones on both synthetic and real-world benchmark datasets. The source code and trained models are available at https://github.com/cschenxiang/NeRD-Rain.

4/3/2024

🖼️

Rethinking Real-world Image Deraining via An Unpaired Degradation-Conditioned Diffusion Model

Yiyang Shen, Mingqiang Wei, Yongzhen Wang, Xueyang Fu, Jing Qin

Recent diffusion models have exhibited great potential in generative modeling tasks. Part of their success can be attributed to the ability of training stable on huge sets of paired synthetic data. However, adapting these models to real-world image deraining remains difficult for two aspects. First, collecting a large-scale paired real-world clean/rainy dataset is unavailable while regular conditional diffusion models heavily rely on paired data for training. Second, real-world rain usually reflects real-world scenarios with a variety of unknown rain degradation types, which poses a significant challenge for the generative modeling process. To meet these challenges, we propose RainDiff, the first real-world image deraining paradigm based on diffusion models, serving as a new standard bar for real-world image deraining. We address the first challenge by introducing a stable and non-adversarial unpaired cycle-consistent architecture that can be trained, end-to-end, with only unpaired data for supervision; and the second challenge by proposing a degradation-conditioned diffusion model that refines the desired output via a diffusive generative process conditioned by learned priors of multiple rain degradations. Extensive experiments confirm the superiority of our RainDiff over existing unpaired/semi-supervised methods and show its competitive advantages over several fully-supervised ones.

5/2/2024