Configurable Learned Holography

Read original: arXiv:2405.01558 - Published 5/7/2024 by Yicheng Zhan, Liang Shi, Wojciech Matusik, Qi Sun, Kaan Akc{s}it

🎯

Overview

Existing holographic display technology faces a challenge in adapting to different hardware configurations due to the complexity of optical components and system settings.
Learned holography approaches have enabled rapid and high-quality hologram generation, but require retraining the model when the display hardware changes.
This work introduces a configurable learned model that can interactively compute 3D holograms from 2D images for a variety of holographic displays, without the need for retraining.

Plain English Explanation

Holographic displays are a promising technology for creating realistic 3D images that you can view without special glasses. However, the complex optical hardware used in these displays can make it difficult to adapt the software that generates the holograms when the display components change.

The researchers in this study developed a new AI model that can generate high-quality 3D holograms from regular 2D images, and can automatically adjust to work with different holographic display hardware. This means the model doesn't need to be completely retrained every time the display is updated or changed.

The key innovation is that the model can be "conditioned" on the specific parameters of the display, such as the wavelengths of light used, the pixel size, and the viewing distance. This allows it to optimize the hologram computations for that particular hardware setup, rather than having to start from scratch.

The model also supports different types of holograms, including both traditional single-color holograms and newer multi-color holograms that use multiple colored light sources. And the researchers used a clever training technique called "knowledge distillation" to make the model run faster, up to 2 times faster than previous state-of-the-art approaches.

Overall, this flexible and efficient AI model could help advance holographic display technology by making it easier to adapt to different hardware setups and configurations.

Technical Explanation

The researchers introduce a configurable learned model that can interactively compute 3D holograms from RGB-only 2D images for a variety of holographic displays. The model is conditioned on predefined hardware parameters of the display, such as working wavelengths, pixel pitch, propagation distance, and peak brightness, without requiring retraining.

The model can accommodate different hologram types, including conventional single-color and emerging multi-color holograms that use multiple color primaries. Notably, the researchers enabled the hologram computations to leverage the correlation between depth estimation and 3D hologram synthesis tasks within the learning domain, a novel approach in the literature.

To achieve interactive performance, the researchers employed knowledge distillation via a student-teacher learning strategy to streamline the model. This resulted in up to a 2x speed improvement compared to state-of-the-art models, while still generating high-quality 3D holograms for different hardware configurations.

Critical Analysis

The researchers acknowledge that their configurable model still has some limitations. For example, it may not be able to fully account for all the complex interactions between the optical components in a holographic display system. Additionally, the model is currently trained on a finite set of hardware parameters, and its ability to generalize to completely novel display configurations is not yet fully explored.

Further research could investigate ways to make the model even more flexible and adaptive, perhaps by incorporating reinforcement learning or other techniques to dynamically adjust the hologram computations based on real-time feedback from the display hardware. Exploring the integration of this model with emerging "neural etendue expander" and "phase-only hologram" techniques could also broaden its applicability.

Overall, this work represents an important step forward in making holographic displays more accessible and adaptable, which could have significant implications for a wide range of applications, from augmented reality to telepresence and beyond.

Conclusion

This study presents a configurable learned model that can interactively compute high-quality 3D holograms from 2D images for a variety of holographic display hardware, without the need for extensive retraining. By incorporating display-specific parameters and leveraging the connection between depth estimation and hologram synthesis, the model achieves up to a 2x speed improvement over previous state-of-the-art approaches.

While the model has some limitations, this research demonstrates the potential for flexible and efficient AI-powered holographic display technology that can adapt to diverse hardware configurations. Further advancements in this area could pave the way for more accessible and widespread adoption of holographic displays, with far-reaching implications for various industries and applications.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

🎯

Configurable Learned Holography

Yicheng Zhan, Liang Shi, Wojciech Matusik, Qi Sun, Kaan Akc{s}it

In the pursuit of advancing holographic display technology, we face a unique yet persistent roadblock: the inflexibility of learned holography in adapting to various hardware configurations. This is due to the variances in the complex optical components and system settings in existing holographic displays. Although the emerging learned approaches have enabled rapid and high-quality hologram generation, any alteration in display hardware still requires a retraining of the model. Our work introduces a configurable learned model that interactively computes 3D holograms from RGB-only 2D images for a variety of holographic displays. The model can be conditioned to predefined hardware parameters of existing holographic displays such as working wavelengths, pixel pitch, propagation distance, and peak brightness without having to retrain. In addition, our model accommodates various hologram types, including conventional single-color and emerging multi-color holograms that simultaneously use multiple color primaries in holographic displays. Notably, we enabled our hologram computations to rely on identifying the correlation between depth estimation and 3D hologram synthesis tasks within the learning domain for the first time in the literature. We employ knowledge distillation via a student-teacher learning strategy to streamline our model for interactive performance. Achieving up to a 2x speed improvement compared to state-of-the-art models while consistently generating high-quality 3D holograms with different hardware configurations.

5/7/2024

Quantized neural network for complex hologram generation

Yutaka Endo, Minoru Oikawa, Timothy D. Wilkinson, Tomoyoshi Shimobaba, Tomoyoshi Ito

Computer-generated holography (CGH) is a promising technology for augmented reality displays, such as head-mounted or head-up displays. However, its high computational demand makes it impractical for implementation. Recent efforts to integrate neural networks into CGH have successfully accelerated computing speed, demonstrating the potential to overcome the trade-off between computational cost and image quality. Nevertheless, deploying neural network-based CGH algorithms on computationally limited embedded systems requires more efficient models with lower computational cost, memory footprint, and power consumption. In this study, we developed a lightweight model for complex hologram generation by introducing neural network quantization. Specifically, we built a model based on tensor holography and quantized it from 32-bit floating-point precision (FP32) to 8-bit integer precision (INT8). Our performance evaluation shows that the proposed INT8 model achieves hologram quality comparable to that of the FP32 model while reducing the model size by approximately 70% and increasing the speed fourfold. Additionally, we implemented the INT8 model on a system-on-module to demonstrate its deployability on embedded platforms and high power efficiency.

9/12/2024

Holo-VQVAE: VQ-VAE for phase-only holograms

Joohyun Park, Hyeongyeop Kang

Holography stands at the forefront of visual technology innovation, offering immersive, three-dimensional visualizations through the manipulation of light wave amplitude and phase. Contemporary research in hologram generation has predominantly focused on image-to-hologram conversion, producing holograms from existing images. These approaches, while effective, inherently limit the scope of innovation and creativity in hologram generation. In response to this limitation, we present Holo-VQVAE, a novel generative framework tailored for phase-only holograms (POHs). Holo-VQVAE leverages the architecture of Vector Quantized Variational AutoEncoders, enabling it to learn the complex distributions of POHs. Furthermore, it integrates the Angular Spectrum Method into the training process, facilitating learning in the image domain. This framework allows for the generation of unseen, diverse holographic content directly from its intricately learned latent space without requiring pre-existing images. This pioneering work paves the way for groundbreaking applications and methodologies in holographic content creation, opening a new era in the exploration of holographic content.

4/3/2024

Pupil-Adaptive 3D Holography Beyond Coherent Depth-of-Field

Yujie Wang, Baoquan Chen, Praneeth Chakravarthula

Recent holographic display approaches propelled by deep learning have shown remarkable success in enabling high-fidelity holographic projections. However, these displays have still not been able to demonstrate realistic focus cues, and a major gap still remains between the defocus effects possible with a coherent light-based holographic display and those exhibited by incoherent light in the real world. Moreover, existing methods have not considered the effects of the observer's eye pupil size variations on the perceived quality of 3D projections, especially on the defocus blur due to varying depth-of-field of the eye. In this work, we propose a framework that bridges the gap between the coherent depth-of-field of holographic displays and what is seen in the real world due to incoherent light. To this end, we investigate the effect of varying shape and motion of the eye pupil on the quality of holographic projections, and devise a method that changes the depth-of-the-field of holographic projections dynamically in a pupil-adaptive manner. Specifically, we introduce a learning framework that adjusts the receptive fields on-the-go based on the current state of the observer's eye pupil to produce image effects that otherwise are not possible in current computer-generated holography approaches. We validate the proposed method both in simulations and on an experimental prototype holographic display, and demonstrate significant improvements in the depiction of depth-of-field effects, outperforming existing approaches both qualitatively and quantitatively by at least 5 dB in peak signal-to-noise ratio.

9/4/2024