Reconstruction of Manipulated Garment with Guided Deformation Prior
0
Sign in to get full access
Overview
• This paper presents a method for reconstructing manipulated garments using a guided deformation prior. • The method aims to accurately reconstruct the 3D shape of a garment that has been deformed or modified, based on a single input image. • It leverages a learned deformation prior to guide the reconstruction process, which helps to produce realistic and plausible results.
Plain English Explanation
• The paper focuses on a common problem in computer vision and graphics: reconstructing the 3D shape of a garment from a 2D image, especially when the garment has been deformed or altered in some way. • Imagine you have a photo of a shirt that has been scrunched up or pulled out of shape. The researchers wanted to develop a way to use that single photo to reconstruct the original 3D shape of the shirt, before it was manipulated. • To do this, they trained a machine learning model to learn the typical ways that garments can deform and change shape. This "deformation prior" knowledge helps guide the reconstruction process, allowing the model to produce a realistic 3D shape that matches the distorted 2D image. • The key innovation is using this deformation prior to constrain the reconstruction, rather than trying to reconstruct the 3D shape from scratch. This helps the model avoid unrealistic or implausible results.
Technical Explanation
• The paper proposes a deep learning-based method for reconstructing the 3D geometry of a manipulated garment from a single input image. • The core of the approach is a guided deformation prior, which is a neural network that has been trained to model the typical ways that garments can be deformed and distorted. • This deformation prior is used to guide the reconstruction process, helping the model produce a 3D shape that is both faithful to the input image and plausible in terms of garment physics and dynamics. • The method builds on prior work in garment transfer and unified garment manipulation, leveraging category-level information to improve the reconstruction. • Experiments on a large dataset of 3D reconstructed garments demonstrate the effectiveness of the proposed approach compared to baselines.
Critical Analysis
• One limitation of the paper is that it focuses solely on reconstructing garment geometry, and does not address the challenge of also reconstructing the appearance (e.g., texture, material properties) of the manipulated garment. • The dataset used for training and evaluation, while large, may not capture the full diversity of real-world garment deformations that could be encountered. • The deformation prior is a key component of the method, but its performance and generalization capabilities could be further analyzed and improved. • It would be interesting to see how this approach could be extended to handle more complex scenes with multiple interacting garments, as mentioned in related work on 3D reconstruction of interacting multi-person clothing.
Conclusion
• This paper presents a novel method for reconstructing the 3D shape of manipulated garments, using a guided deformation prior to produce realistic and plausible results. • The approach leverages category-level information and builds on recent advances in garment transfer and unified garment manipulation. • This work has potential applications in areas such as virtual try-on, fashion design, and augmented reality, where accurate 3D garment reconstruction is a key challenge. • While the current method is limited to garment geometry, future research could explore integrating appearance reconstruction as well, for a more comprehensive solution.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Reconstruction of Manipulated Garment with Guided Deformation Prior
Ren Li, Corentin Dumery, Zhantao Deng, Pascal Fua
Modeling the shape of garments has received much attention, but most existing approaches assume the garments to be worn by someone, which constrains the range of shapes they can assume. In this work, we address shape recovery when garments are being manipulated instead of worn, which gives rise to an even larger range of possible shapes. To this end, we leverage the implicit sewing patterns (ISP) model for garment modeling and extend it by adding a diffusion-based deformation prior to represent these shapes. To recover 3D garment shapes from incomplete 3D point clouds acquired when the garment is folded, we map the points to UV space, in which our priors are learned, to produce partial UV maps, and then fit the priors to recover complete UV maps and 2D to 3D mappings. Experimental results demonstrate the superior reconstruction accuracy of our method compared to previous ones, especially when dealing with large non-rigid deformations arising from the manipulations.
Read more5/20/2024
0
Pattern Guided UV Recovery for Realistic Video Garment Texturing
Youyi Zhan, Tuanfeng Y. Wang, Tianjia Shao, Kun Zhou
The fast growth of E-Commerce creates a global market worth USD 821 billion for online fashion shopping. What unique about fashion presentation is that, the same design can usually be offered with different cloths textures. However, only real video capturing or manual per-frame editing can be used for virtual showcase on the same design with different textures, both of which are heavily labor intensive. In this paper, we present a pattern-based approach for UV and shading recovery from a captured real video so that the garment's texture can be replaced automatically. The core of our approach is a per-pixel UV regression module via blended-weight multilayer perceptrons (MLPs) driven by the detected discrete correspondences from the cloth pattern. We propose a novel loss on the Jacobian of the UV mapping to create pleasant seams around the folding areas and the boundary of occluded regions while avoiding UV distortion. We also adopts the temporal constraint to ensure consistency and accuracy in UV prediction across adjacent frames. We show that our approach is robust to a variety type of clothes, in the wild illuminations and with challenging motions. We show plausible texture replacement results in our experiment, in which the folding and overlapping of the garment can be greatly preserved. We also show clear qualitative and quantitative improvement compared to the baselines as well. With the one-click setup, we look forward to our approach contributing to the growth of fashion E-commerce.
Read more7/16/2024
0
Neural Garment Dynamics via Manifold-Aware Transformers
Peizhuo Li, Tuanfeng Y. Wang, Timur Levent Kesdogan, Duygu Ceylan, Olga Sorkine-Hornung
Data driven and learning based solutions for modeling dynamic garments have significantly advanced, especially in the context of digital humans. However, existing approaches often focus on modeling garments with respect to a fixed parametric human body model and are limited to garment geometries that were seen during training. In this work, we take a different approach and model the dynamics of a garment by exploiting its local interactions with the underlying human body. Specifically, as the body moves, we detect local garment-body collisions, which drive the deformation of the garment. At the core of our approach is a mesh-agnostic garment representation and a manifold-aware transformer network design, which together enable our method to generalize to unseen garment and body geometries. We evaluate our approach on a wide variety of garment types and motion sequences and provide competitive qualitative and quantitative results with respect to the state of the art.
Read more7/9/2024
0
Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video
Boxiang Rong, Artur Grigorev, Wenbo Wang, Michael J. Black, Bernhard Thomaszewski, Christina Tsalicoglou, Otmar Hilliges
We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos. Our method represents garments with a combination of a 3D mesh and a Gaussian texture that encodes both the color and high-frequency surface details. This representation enables accurate registration of garment geometries to multi-view videos and helps disentangle albedo textures from lighting effects. Furthermore, we demonstrate how a pre-trained graph neural network (GNN) can be fine-tuned to replicate the real behavior of each garment. The reconstructed Gaussian Garments can be automatically combined into multi-garment outfits and animated with the fine-tuned GNN.
Read more9/14/2024