ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians
2406.16815
0
0
Abstract
High-fidelity 3D garment synthesis from text is desirable yet challenging for digital avatar creation. Recent diffusion-based approaches via Score Distillation Sampling (SDS) have enabled new possibilities but either intricately couple with human body or struggle to reuse. We introduce ClotheDreamer, a 3D Gaussian-based method for generating wearable, production-ready 3D garment assets from text prompts. We propose a novel representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate optimization. DCGS represents clothed avatar as one Gaussian model but freezes body Gaussian splats. To enhance quality and completeness, we incorporate bidirectional SDS to supervise clothed avatar and garment RGBD renderings respectively with pose conditions and propose a new pruning strategy for loose clothing. Our approach can also support custom clothing templates as input. Benefiting from our design, the synthetic 3D garment can be easily applied to virtual try-on and support physically accurate animation. Extensive experiments showcase our method's superior and competitive performance. Our project page is at https://ggxxii.github.io/clothedreamer.
Create account to get full access
Overview
⢠This paper introduces ClotheDreamer, a novel system for text-guided garment generation that uses 3D Gaussian Surfaces (3D GS) to model diverse garment geometries.
⢠ClotheDreamer allows users to generate 3D garment models from text descriptions, enabling more intuitive and expressive garment design.
⢠The system leverages the power of large language models and 3D Gaussian Surfaces to generate detailed, high-quality garment meshes that capture complex shapes and folds.
Plain English Explanation
ClotheDreamer is a tool that allows you to create 3D models of clothes just by describing them in words. Instead of having to design the clothes yourself using complex 3D modeling software, you can simply type a description and the system will generate a 3D garment model for you.
The key innovation in ClotheDreamer is the use of 3D Gaussian Surfaces (3D GS) to represent the shape of the garments. Gaussian Surfaces are a mathematical way of describing 3D shapes that can capture intricate details like folds and drapes in clothing. By combining this 3D modeling technique with powerful language models, ClotheDreamer can translate your text descriptions into highly realistic 3D garment models.
This makes the process of designing clothes much more intuitive and accessible, as you don't need specialized 3D modeling skills. You can simply describe the garment you have in mind, and ClotheDreamer will bring it to life in 3D. This could be useful for fashion designers, 3D artists, or anyone who wants to create custom clothing designs.
Technical Explanation
ClotheDreamer uses a GaussianDreamer: Fast Generation from Text to 3D architecture to generate 3D garment models from text descriptions. It leverages a pre-trained language model to encode the input text, which is then used to condition the generation of 3D Gaussian Surfaces that represent the garment geometry.
The system is trained on a large dataset of 3D garment models and their corresponding text descriptions. During inference, the input text is encoded using the language model, and this representation is used to predict the parameters of the 3D Gaussian Surfaces that will form the final garment mesh.
By using 3D Gaussian Surfaces, ClotheDreamer is able to capture complex garment geometries, including folds, drapes, and other intricate details. This allows the system to generate highly realistic and diverse garment models that closely match the input text descriptions.
The authors also propose extensions to the base ClotheDreamer model, including LAGA: Layered 3D Avatar Generation and Customization via Text and DressCode: Autoregressive Sewing for Generating Garments from Text, which further expand the system's capabilities for garment generation and customization.
Critical Analysis
The ClotheDreamer system represents an impressive advancement in the field of text-to-3D generation, demonstrating the potential of large language models and 3D Gaussian Surfaces for intuitive garment design. The ability to translate natural language descriptions into high-quality 3D garment models could have significant impacts on fashion, 3D art, and other domains.
However, the paper does acknowledge some limitations of the current approach. The system is trained on a relatively small dataset of 3D garment models, which may limit its ability to generalize to more diverse or unconventional garment styles. Additionally, the 3D Gaussian Surface representation, while effective for capturing complex geometries, may not be optimal for all types of garment features, such as sharp creases or highly asymmetric shapes.
Further research could explore ways to expand the dataset, improve the 3D representation, and enhance the system's ability to handle a wider range of garment types and styles. Integrating ClotheDreamer with other 3D modeling tools or avatar generation systems, as hinted at in the related work, could also unlock new applications and use cases for this technology.
Overall, ClotheDreamer represents an exciting step forward in the quest to make 3D garment design more accessible and intuitive. As the field of text-to-3D generation continues to evolve, systems like this could revolutionize the way we think about and create clothing in the digital realm.
Conclusion
The ClotheDreamer system introduced in this paper demonstrates the power of leveraging large language models and 3D Gaussian Surfaces to enable text-guided garment generation. By translating natural language descriptions into detailed 3D garment models, ClotheDreamer offers a more intuitive and expressive approach to digital clothing design.
The ability to create 3D garments simply by describing them in words could have significant implications for fashion, 3D art, and other industries where custom clothing and textile design are important. While the current system has some limitations, the potential for further advancements in this area is substantial.
As the field of text-to-3D generation continues to evolve, systems like ClotheDreamer could become increasingly valuable tools for designers, artists, and anyone who wants to bring their clothing ideas to life in the digital realm.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
đż
GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details
Boqian Li, Xuan Li, Ying Jiang, Tianyi Xie, Feng Gao, Huamin Wang, Yin Yang, Chenfanfu Jiang
0
0
Traditional 3D garment creation is labor-intensive, involving sketching, modeling, UV mapping, and texturing, which are time-consuming and costly. Recent advances in diffusion-based generative models have enabled new possibilities for 3D garment generation from text prompts, images, and videos. However, existing methods either suffer from inconsistencies among multi-view images or require additional processes to separate cloth from the underlying human model. In this paper, we propose GarmentDreamer, a novel method that leverages 3D Gaussian Splatting (GS) as guidance to generate wearable, simulation-ready 3D garment meshes from text prompts. In contrast to using multi-view images directly predicted by generative models as guidance, our 3DGS guidance ensures consistent optimization in both garment deformation and texture synthesis. Our method introduces a novel garment augmentation module, guided by normal and RGBA information, and employs implicit Neural Texture Fields (NeTF) combined with Score Distillation Sampling (SDS) to generate diverse geometric and texture details. We validate the effectiveness of our approach through comprehensive qualitative and quantitative experiments, showcasing the superior performance of GarmentDreamer over state-of-the-art alternatives. Our project page is available at: https://xuan-li.github.io/GarmentDreamerDemo/.
5/22/2024
GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality
Taoran Yi, Jiemin Fang, Zanwei Zhou, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Xinggang Wang, Qi Tian
0
0
Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at https://taoranyi.com/gaussiandreamerpro/.
6/27/2024
DressCode: Autoregressively Sewing and Generating Garments from Text Guidance
Kai He, Kaixin Yao, Qixuan Zhang, Jingyi Yu, Lingjie Liu, Lan Xu
0
0
Apparel's significant role in human appearance underscores the importance of garment digitalization for digital human creation. Recent advances in 3D content creation are pivotal for digital human creation. Nonetheless, garment generation from text guidance is still nascent. We introduce a text-driven 3D garment generation framework, DressCode, which aims to democratize design for novices and offer immense potential in fashion design, virtual try-on, and digital human creation. We first introduce SewingGPT, a GPT-based architecture integrating cross-attention with text-conditioned embedding to generate sewing patterns with text guidance. We then tailor a pre-trained Stable Diffusion to generate tile-based Physically-based Rendering (PBR) textures for the garments. By leveraging a large language model, our framework generates CG-friendly garments through natural language interaction. It also facilitates pattern completion and texture editing, streamlining the design process through user-friendly interaction. This framework fosters innovation by allowing creators to freely experiment with designs and incorporate unique elements into their work. With comprehensive evaluations and comparisons with other state-of-the-art methods, our method showcases superior quality and alignment with input prompts. User studies further validate our high-quality rendering results, highlighting its practical utility and potential in production settings. Our project page is https://IHe-KaiI.github.io/DressCode/.
6/18/2024
đ¸
LAGA: Layered 3D Avatar Generation and Customization via Gaussian Splatting
Jia Gong, Shenyu Ji, Lin Geng Foo, Kang Chen, Hossein Rahmani, Jun Liu
0
0
Creating and customizing a 3D clothed avatar from textual descriptions is a critical and challenging task. Traditional methods often treat the human body and clothing as inseparable, limiting users' ability to freely mix and match garments. In response to this limitation, we present LAyered Gaussian Avatar (LAGA), a carefully designed framework enabling the creation of high-fidelity decomposable avatars with diverse garments. By decoupling garments from avatar, our framework empowers users to conviniently edit avatars at the garment level. Our approach begins by modeling the avatar using a set of Gaussian points organized in a layered structure, where each layer corresponds to a specific garment or the human body itself. To generate high-quality garments for each layer, we introduce a coarse-to-fine strategy for diverse garment generation and a novel dual-SDS loss function to maintain coherence between the generated garments and avatar components, including the human body and other garments. Moreover, we introduce three regularization losses to guide the movement of Gaussians for garment transfer, allowing garments to be freely transferred to various avatars. Extensive experimentation demonstrates that our approach surpasses existing methods in the generation of 3D clothed humans.
5/22/2024