Neural '{E}tendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display
0
🧠
Sign in to get full access
Overview
- Holographic displays can generate realistic virtual and augmented reality experiences by dynamically controlling the wavefront of a coherent light beam.
- However, the limited resolution of existing display technology imposes a tradeoff between the field of view (FOV) and the display size.
- This work presents "neural étendue expanders" - a new type of optical element learned from natural images that can significantly expand the étendue (the product of the display area and the maximum diffraction angle) of holographic displays.
Plain English Explanation
Holographic displays work by carefully controlling the wavefront, or shape, of a beam of coherent light using a device called a spatial light modulator. This allows them to generate realistic 3D images and virtual/augmented reality experiences. However, the resolution of current spatial light modulators puts a limit on how wide the field of view (FOV) can be while still maintaining a large enough display area.
The étendue of a display is the product of the display size and the maximum angle of the light it can diffract. Existing holographic displays have a low étendue, forcing a tradeoff - they can either have a wide FOV or a large display, but not both.
This new research introduces "neural étendue expanders" - optical elements that are trained on natural image data to enable much higher diffraction angles. This allows holographic displays to achieve an order of magnitude increase in both horizontal and vertical FOV, while still maintaining a compact form factor and high-quality image fidelity. The authors demonstrate 64x étendue expansion of natural images in full color, with image quality over 29 dB PSNR on retinal-resolution images.
Technical Explanation
The key innovation in this work is the "neural étendue expander" - a learned optical element that can significantly expand the étendue of holographic displays. Étendue is the product of the display area and the maximum diffraction angle, and existing holographic displays are limited by their low étendue.
The authors train these neural étendue expanders on a large dataset of natural images. The resulting optical elements enable much higher diffraction angles, allowing holographic displays to achieve an order of magnitude increase in both horizontal and vertical field of view (FOV) while maintaining image fidelity.
Experimentally, the authors demonstrate 64x étendue expansion of natural images in full color, with image quality over 29 dB PSNR on retinal-resolution images. This is a significant improvement over the state-of-the-art, as it allows for both a wide FOV and a large, high-resolution display area - something that was previously limited by the underlying display technology.
The neural étendue expanders leverage advances in coordinate-based neural representations, 1-bit quantized diffractive optics, and phase-only holography. This work also has connections to research on light field rendering and holographic video compression.
Critical Analysis
While the neural étendue expanders represent a significant advance in holographic display technology, there are some potential limitations and areas for further research:
- The paper does not explore the scalability of the approach to very large display sizes or resolutions. The demonstrated results are on relatively small display areas.
- The training process for the neural étendue expanders is computationally intensive and may be challenging to deploy in real-time applications.
- The paper does not address potential issues with image artifacts or distortions introduced by the neural optical elements.
- Further research is needed to understand the fundamental limits of this approach and how it compares to alternative holographic display techniques.
Overall, this work represents an exciting step forward in overcoming the tradeoffs inherent in current holographic display technology. The neural étendue expanders offer a promising path to achieving wide field of view, high-resolution holographic experiences.
Conclusion
This research introduces a novel approach to overcoming the limitations of existing holographic display technology. By learning specialized "neural étendue expanders" from natural image data, the authors demonstrate a 64x expansion of the étendue - the key constraint that has previously forced tradeoffs between field of view and display size.
With this breakthrough, holographic displays can now achieve an order of magnitude increase in both horizontal and vertical field of view, while maintaining high-fidelity image reconstruction. This has significant implications for the development of immersive virtual and augmented reality experiences that can match the visual acuity of the human eye.
While there are still some open challenges to address, this work represents an important step forward in making high-performance holographic displays a practical reality.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🧠
0
Neural '{E}tendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display
Ethan Tseng, Grace Kuo, Seung-Hwan Baek, Nathan Matsuda, Andrew Maimone, Florian Schiffers, Praneeth Chakravarthula, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, Felix Heide
Holographic displays can generate light fields by dynamically modulating the wavefront of a coherent beam of light using a spatial light modulator, promising rich virtual and augmented reality applications. However, the limited spatial resolution of existing dynamic spatial light modulators imposes a tight bound on the diffraction angle. As a result, modern holographic displays possess low '{e}tendue, which is the product of the display area and the maximum solid angle of diffracted light. The low '{e}tendue forces a sacrifice of either the field-of-view (FOV) or the display size. In this work, we lift this limitation by presenting neural '{e}tendue expanders. This new breed of optical elements, which is learned from a natural image dataset, enables higher diffraction angles for ultra-wide FOV while maintaining both a compact form factor and the fidelity of displayed contents to human viewers. With neural '{e}tendue expanders, we experimentally achieve 64$times$ '{e}tendue expansion of natural images in full color, expanding the FOV by an order of magnitude horizontally and vertically, with high-fidelity reconstruction quality (measured in PSNR) over 29 dB on retinal-resolution images.
Read more4/30/2024
0
Large 'Etendue 3D Holographic Display with Content-adpative Dynamic Fourier Modulation
Brian Chao, Manu Gopakumar, Suyeon Choi, Jonghyun Kim, Liang Shi, Gordon Wetzstein
Emerging holographic display technology offers unique capabilities for next-generation virtual reality systems. Current holographic near-eye displays, however, only support a small 'etendue, which results in a direct tradeoff between achievable field of view and eyebox size. 'Etendue expansion has recently been explored, but existing approaches are either fundamentally limited in the image quality that can be achieved or they require extremely high-speed spatial light modulators. We describe a new 'etendue expansion approach that combines multiple coherent sources with content-adaptive amplitude modulation of the hologram spectrum in the Fourier plane. To generate time-multiplexed phase and amplitude patterns for our spatial light modulators, we devise a pupil-aware gradient-descent-based computer-generated holography algorithm that is supervised by a large-baseline target light field. Compared with relevant baseline approaches, our method demonstrates significant improvements in image quality and 'etendue in simulation and with an experimental holographic display prototype.
Read more9/6/2024
0
Pupil-Adaptive 3D Holography Beyond Coherent Depth-of-Field
Yujie Wang, Baoquan Chen, Praneeth Chakravarthula
Recent holographic display approaches propelled by deep learning have shown remarkable success in enabling high-fidelity holographic projections. However, these displays have still not been able to demonstrate realistic focus cues, and a major gap still remains between the defocus effects possible with a coherent light-based holographic display and those exhibited by incoherent light in the real world. Moreover, existing methods have not considered the effects of the observer's eye pupil size variations on the perceived quality of 3D projections, especially on the defocus blur due to varying depth-of-field of the eye. In this work, we propose a framework that bridges the gap between the coherent depth-of-field of holographic displays and what is seen in the real world due to incoherent light. To this end, we investigate the effect of varying shape and motion of the eye pupil on the quality of holographic projections, and devise a method that changes the depth-of-the-field of holographic projections dynamically in a pupil-adaptive manner. Specifically, we introduce a learning framework that adjusts the receptive fields on-the-go based on the current state of the observer's eye pupil to produce image effects that otherwise are not possible in current computer-generated holography approaches. We validate the proposed method both in simulations and on an experimental prototype holographic display, and demonstrate significant improvements in the depiction of depth-of-field effects, outperforming existing approaches both qualitatively and quantitatively by at least 5 dB in peak signal-to-noise ratio.
Read more9/4/2024
0
Wide-Field, High-Resolution Reconstruction in Computational Multi-Aperture Miniscope Using a Fourier Neural Network
Qianwan Yang, Ruipeng Guo, Guorong Hu, Yujia Xue, Yunzhe Li, Lei Tian
Traditional fluorescence microscopy is constrained by inherent trade-offs among resolution, field-of-view, and system complexity. To navigate these challenges, we introduce a simple and low-cost computational multi-aperture miniature microscope, utilizing a microlens array for single-shot wide-field, high-resolution imaging. Addressing the challenges posed by extensive view multiplexing and non-local, shift-variant aberrations in this device, we present SV-FourierNet, a novel multi-channel Fourier neural network. SV-FourierNet facilitates high-resolution image reconstruction across the entire imaging field through its learned global receptive field. We establish a close relationship between the physical spatially-varying point-spread functions and the network's learned effective receptive field. This ensures that SV-FourierNet has effectively encapsulated the spatially-varying aberrations in our system, and learned a physically meaningful function for image reconstruction. Training of SV-FourierNet is conducted entirely on a physics-based simulator. We showcase wide-field, high-resolution video reconstructions on colonies of freely moving C. elegans and imaging of a mouse brain section. Our computational multi-aperture miniature microscope, augmented with SV-FourierNet, represents a major advancement in computational microscopy and may find broad applications in biomedical research and other fields requiring compact microscopy solutions.
Read more5/31/2024