Aggregating Local and Global Features via Selective State Spaces Model for Efficient Image Deblurring

Introduction

The provided text discusses the challenge of image deblurring, which aims to recover a sharp image from a blurry one. Conventional approaches have explicitly incorporated various priors or hand-crafted features to constrain the solution space, but designing such priors is challenging and lacks generalizability.

Deep learning-based methods, such as encoder-decoder architectures, multi-stage networks, dual networks, and generative models, have been developed as a more data-driven approach. However, the convolution operation in these models has limitations in capturing long-range dependencies.

To address these limitations, transformer-based models have been applied to image deblurring, as they can better model non-local information. However, the attention mechanism in transformers incurs high computational costs, especially for high-resolution images. Some methods have tried to alleviate this by applying self-attention across channels instead of spatial dimensions or using non-overlapping window-based self-attention, but these approaches still struggle to fully exploit the spatial information.

The paper proposes a new network called ALGNet for image deblurring. Structured state space models like Mamba can efficiently capture long-range dependencies, but they tend to neglect local pixel information and have issues with channel redundancy. To address these limitations, ALGNet has several key components:

1. An efficient global block that uses selective structured state spaces to capture long-range dependency information important for high-quality deblurring.

2. A local block that models local connectivity using simplified channel attention, addressing the problems of local pixel forgetting and channel redundancy in the state space equation.

3. An aggregate method that emphasizes the importance of the local block by recalibrating the weights when combining the global and local components.

4. Multiple scales of input and output to help with training.

The paper claims that ALGNet achieves state-of-the-art performance on image deblurring while being computationally efficient compared to existing methods.

Related Work

The provided text discusses various approaches to image deblurring, a challenging problem due to its ill-posed nature.

The first section covers hand-crafted prior-based methods, which rely on manually designed image priors to constrain the set of plausible solutions. However, designing such priors is a challenging task that often leads to complicated optimization problems.

The second section discusses CNN-based methods, which use deep neural networks to learn image priors instead of manually designing them. These methods have shown better performance than the hand-crafted prior-based approaches. However, the local receptive fields of convolutional operations limit the models' ability to efficiently eliminate long-range degradation.

The third section covers Transformer-based methods, which leverage the global receptive field of the Transformer architecture to achieve superior performance compared to CNN-based baselines. However, the quadratic time complexity of the attention mechanism in Transformers results in significant computational overhead, especially for high-resolution images.

To address the limitations of both CNN-based and Transformer-based methods, the text introduces state space models as an efficient framework for capturing long-range dependencies with linear complexity. Various state space model approaches, such as S4, S5, Mega, SGConv, and LongConv, are discussed.

The work presented in this paper leverages a selective structured state space model to capture long-range dependency information, while also addressing issues of local pixel forgetting and channel redundancy.

Method

The proposed ALGNet pipeline first outlines the overall framework. It then describes the details of the ALGBlock, which includes a global block, a local block, and a features aggregation (FA) component. The global block and local block are key parts of the ALGBlock design.

The overall pipeline of the proposed ALGNet adopts a single U-shaped architecture for image restoration. The pipeline initially applies a convolution to acquire shallow features, which then undergo a four-scale encoder sub-network. The low-resolution degraded images are incorporated into the main path through concatenation. The resulting deepest features feed into a four-scale decoder, gradually restoring features to the original size. Finally, the network refines features to generate a residual image, which is added to the degraded image to obtain the restored image.

The network is optimized using a loss function that combines character, edge, and frequency domain losses. The character loss is a Charbonnier loss, the edge loss measures the Laplacian difference, and the frequency domain loss measures the difference in the Fourier transform.

To capture long-range dependencies efficiently, the network includes a global block with linear complexity, which expands and aggregates features through parallel branches. Additionally, a local block is designed to model local connectivity and facilitate the expressive power of different channels.

Finally, a features aggregation module is introduced to emphasize the importance of the local block by recalibrating the weights when aggregating the global and local features for restoration.

V Experiments

The paper describes the experimental details and results of the proposed ALGNet method for image deblurring tasks.

The key points are:

• Datasets: The authors train ALGNet on the GoPro dataset for image motion deblurring, and evaluate it on the HIDE and RealBlur datasets as well. For single-image defocus deblurring, they use the DPDD dataset.

• Training details: ALGNet is trained using the Adam optimizer with a gradually decreasing learning rate. Data augmentation techniques like flips are used.

• Image motion deblurring results: On the GoPro dataset, ALGNet-B outperforms state-of-the-art methods like NAFNet-64 by 0.43 dB in PSNR. Even though trained only on GoPro, ALGNet achieves a 0.19 dB gain over Restormer-Local on the HIDE dataset, demonstrating good generalization. ALGNet also performs best on the real-world RealBlur dataset.

• Single-image defocus deblurring results: ALGNet outperforms leading methods on the DPDD dataset across various metrics, especially in outdoor scenes where it has a 0.15 dB PSNR improvement over IRNeXt.

• Ablation studies validate the effectiveness of ALGNet's global block, local block, and feature attention (FA) components, which together lead to significant performance gains.

• Efficiency analysis shows ALGNet achieves state-of-the-art deblurring performance with up to 84.7% reduction in computational cost and 1.5x faster inference compared to prior methods.

Conclusion

The paper proposes an efficient image deblurring network that uses a selective structured state spaces model to aggregate enriched and accurate features. The key component is the Aggregate Local and Global Block (ALGBlock), which consists of global and local components. The global block uses the selective structured state spaces model to capture long-range dependency information. The local block models local connectivity using simplified channel attention, addressing issues of local pixel forgetting and channel redundancy in the state space equation. The paper also presents an aggregate method that emphasizes the importance of the local block in restoration. Experimental results show the proposed method outperforms state-of-the-art approaches on widely used benchmarks.