DeblurGS: Gaussian Splatting for Camera Motion Blur
2404.11358
0
0
Abstract
Although significant progress has been made in reconstructing sharp 3D scenes from motion-blurred images, a transition to real-world applications remains challenging. The primary obstacle stems from the severe blur which leads to inaccuracies in the acquisition of initial camera poses through Structure-from-Motion, a critical aspect often overlooked by previous approaches. To address this challenge, we propose DeblurGS, a method to optimize sharp 3D Gaussian Splatting from motion-blurred images, even with the noisy camera pose initialization. We restore a fine-grained sharp scene by leveraging the remarkable reconstruction capability of 3D Gaussian Splatting. Our approach estimates the 6-Degree-of-Freedom camera motion for each blurry observation and synthesizes corresponding blurry renderings for the optimization process. Furthermore, we propose Gaussian Densification Annealing strategy to prevent the generation of inaccurate Gaussians at erroneous locations during the early training stages when camera motion is still imprecise. Comprehensive experiments demonstrate that our DeblurGS achieves state-of-the-art performance in deblurring and novel view synthesis for real-world and synthetic benchmark datasets, as well as field-captured blurry smartphone videos.
Create account to get full access
Overview
- This paper introduces DeblurGS, a novel method for removing camera motion blur from images using 3D Gaussian splatting.
- The key idea is to model the camera motion as a sequence of Gaussian kernels that can be efficiently applied to the image using splatting techniques.
- DeblurGS achieves state-of-the-art performance on camera motion deblurring benchmarks while being computationally efficient.
Plain English Explanation
When you take a photo, sometimes the image can come out blurry due to camera movement during the exposure. This is called camera motion blur. DeblurGS is a new technique that can remove this type of blur from images.
The core insight behind DeblurGS is to model the camera motion as a series of Gaussian-shaped "splats" that are applied to the image. Gaussian shapes are well-suited for representing camera motion, and splatting techniques allow these splats to be applied efficiently. By estimating the camera motion from the blurry image and applying the corresponding Gaussian splats, DeblurGS is able to effectively deblur the image.
Compared to other methods, DeblurGS achieves state-of-the-art deblurring performance while also being computationally efficient. This means it can produce high-quality deblurred images quickly, which is important for practical applications.
Technical Explanation
The key technical innovation of DeblurGS is the use of 3D Gaussian splatting to model and remove camera motion blur. Traditionally, camera motion blur has been modeled as a 2D blur kernel, but the authors argue that a 3D representation is more appropriate to capture the full 6 degrees of freedom of camera motion.
In DeblurGS, the camera motion is estimated as a sequence of 3D Gaussian kernels, each representing the camera position and orientation at a specific time during the exposure. These Gaussian kernels are then efficiently applied to the image using splatting techniques, which avoid the need for expensive convolution operations.
The authors show that DeblurGS outperforms state-of-the-art deblurring methods on standard benchmarks, while also being significantly faster. This is enabled by the efficient Gaussian splatting approach, as well as other architectural choices like a coarse-to-fine deblurring pipeline.
Critical Analysis
One potential limitation of DeblurGS is that it assumes the camera motion can be well-approximated by a sequence of Gaussian kernels. While this assumption holds for many real-world scenarios, there may be cases where the actual camera motion is better represented by a different model.
Additionally, the paper does not provide a thorough analysis of the sensitivity of DeblurGS to parameters like the number of Gaussian kernels used or the size of the input image. Further investigation into these areas could help users better understand the strengths and limitations of the method.
That said, the strong empirical performance of DeblurGS on benchmark datasets suggests it is a promising approach for practical camera motion deblurring applications. The efficient Gaussian splatting technique in particular is an interesting contribution that could inspire future work in related areas, such as per-Gaussian embedding-based deformation, Gaussian splatting for SLAM, or sparse controlled Gaussian splatting.
Conclusion
DeblurGS is a novel camera motion deblurring method that uses 3D Gaussian splatting to effectively remove blur from images. By modeling the camera motion as a sequence of Gaussian kernels and applying them efficiently using splatting techniques, DeblurGS achieves state-of-the-art performance while being computationally efficient.
This work demonstrates the power of clever geometric representations and efficient algorithms for solving challenging computer vision problems. The Gaussian splatting approach introduced in DeblurGS could also be applied to other areas like 3D Gaussian splatting for recovering fine details, further expanding the impact of this research.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Robust Gaussian Splatting
Franc{c}ois Darmon, Lorenzo Porzi, Samuel Rota-Bul`o, Peter Kontschieder
0
0
In this paper, we address common error sources for 3D Gaussian Splatting (3DGS) including blur, imperfect camera poses, and color inconsistencies, with the goal of improving its robustness for practical applications like reconstructions from handheld phone captures. Our main contribution involves modeling motion blur as a Gaussian distribution over camera poses, allowing us to address both camera pose refinement and motion blur correction in a unified way. Additionally, we propose mechanisms for defocus blur compensation and for addressing color in-consistencies caused by ambient light, shadows, or due to camera-related factors like varying white balancing settings. Our proposed solutions integrate in a seamless way with the 3DGS formulation while maintaining its benefits in terms of training efficiency and rendering speed. We experimentally validate our contributions on relevant benchmark datasets including Scannet++ and Deblur-NeRF, obtaining state-of-the-art results and thus consistent improvements over relevant baselines.
4/8/2024
Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion
Otto Seiskari, Jerry Ylilammi, Valtteri Kaatrasalo, Pekka Rantalankila, Matias Turkulainen, Juho Kannala, Arno Solin
0
0
High-quality scene reconstruction and novel view synthesis based on Gaussian Splatting (3DGS) typically require steady, high-quality photographs, often impractical to capture with handheld cameras. We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data suffering from motion blur and rolling shutter distortion. Our approach is based on detailed modelling of the physical image formation process and utilizes velocities estimated using visual-inertial odometry (VIO). Camera poses are considered non-static during the exposure time of a single image frame and camera poses are further optimized in the reconstruction process. We formulate a differentiable rendering pipeline that leverages screen space approximation to efficiently incorporate rolling-shutter and motion blur effects into the 3DGS framework. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods, thereby advancing 3DGS in naturalistic settings.
5/27/2024
EvaGaussians: Event Stream Assisted Gaussian Splatting from Blurry Images
Wangbo Yu, Chaoran Feng, Jiye Tang, Xu Jia, Li Yuan, Yonghong Tian
0
0
3D Gaussian Splatting (3D-GS) has demonstrated exceptional capabilities in 3D scene reconstruction and novel view synthesis. However, its training heavily depends on high-quality, sharp images and accurate camera poses. Fulfilling these requirements can be challenging in non-ideal real-world scenarios, where motion-blurred images are commonly encountered in high-speed moving cameras or low-light environments that require long exposure times. To address these challenges, we introduce Event Stream Assisted Gaussian Splatting (EvaGaussians), a novel approach that integrates event streams captured by an event camera to assist in reconstructing high-quality 3D-GS from blurry images. Capitalizing on the high temporal resolution and dynamic range offered by the event camera, we leverage the event streams to explicitly model the formation process of motion-blurred images and guide the deblurring reconstruction of 3D-GS. By jointly optimizing the 3D-GS parameters and recovering camera motion trajectories during the exposure time, our method can robustly facilitate the acquisition of high-fidelity novel views with intricate texture details. We comprehensively evaluated our method and compared it with previous state-of-the-art deblurring rendering methods. Both qualitative and quantitative comparisons demonstrate that our method surpasses existing techniques in restoring fine details from blurry images and producing high-fidelity novel views.
5/31/2024
Deblurring 3D Gaussian Splatting
Byeonghyeon Lee, Howoong Lee, Xiangyu Sun, Usman Ali, Eunbyung Park
0
0
Recent studies in Radiance Fields have paved the robust way for novel view synthesis with their photorealistic rendering quality. Nevertheless, they usually employ neural networks and volumetric rendering, which are costly to train and impede their broad use in various real-time applications due to the lengthy rendering time. Lately 3D Gaussians splatting-based approach has been proposed to model the 3D scene, and it achieves remarkable visual quality while rendering the images in real-time. However, it suffers from severe degradation in the rendering quality if the training images are blurry. Blurriness commonly occurs due to the lens defocusing, object motion, and camera shake, and it inevitably intervenes in clean image acquisition. Several previous studies have attempted to render clean and sharp images from blurry input images using neural fields. The majority of those works, however, are designed only for volumetric rendering-based neural radiance fields and are not straightforwardly applicable to rasterization-based 3D Gaussian splatting methods. Thus, we propose a novel real-time deblurring framework, Deblurring 3D Gaussian Splatting, using a small Multi-Layer Perceptron (MLP) that manipulates the covariance of each 3D Gaussian to model the scene blurriness. While Deblurring 3D Gaussian Splatting can still enjoy real-time rendering, it can reconstruct fine and sharp details from blurry images. A variety of experiments have been conducted on the benchmark, and the results have revealed the effectiveness of our approach for deblurring. Qualitative results are available at https://benhenryl.github.io/Deblurring-3D-Gaussian-Splatting/
5/28/2024