STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting
0
🛸
Sign in to get full access
Overview
- This paper introduces STAR, a novel system for generating 4D avatars (3D models that can animate) from text descriptions.
- STAR uses a skeleton-aware approach to generate realistic human motions and retarget them to a generated 3D character model.
- The system can create high-quality 4D avatars from just a text prompt, without requiring any additional input like images or motion capture data.
Plain English Explanation
STAR is a new way to create animated 3D human characters, or "4D avatars," using only text descriptions. Compared to previous text-to-avatar systems, STAR focuses on generating realistic human-like motions and seamlessly applying them to the generated 3D character model.
The key innovation in STAR is its "skeleton-aware" approach. Rather than just generating static 3D shapes, STAR first creates an underlying skeleton or "pose" for the character. It then uses this skeleton as a guide to produce natural-looking movements and animations. [This is similar to how text-guided 3D human motion generation and semantics-aware motion retargeting work.]
By generating the motion and 3D model together, STAR can create 4D avatars that look and move realistically, all from just a text prompt. This could be useful for things like virtual characters in games, movies, or metaverse applications, where customizable digital humans are important.
Technical Explanation
STAR uses a two-stage architecture to generate 4D avatars from text. First, it takes a text description as input and generates a 3D character model and an associated skeleton pose. Then, it uses this skeleton information to produce realistic human motions and seamlessly retarget them onto the 3D model.
The 3D model generation stage uses a progressive framework to gradually refine the character shape and appearance. The motion generation stage leverages a trajectory-conditioned text-to-4D generation approach, which allows the system to produce coordinated full-body movements from the text prompt.
Importantly, STAR's motion retargeting happens within the neural network, rather than as a separate post-processing step. This "in-network" retargeting helps ensure the final animations look natural and coherent with the generated 3D character.
Critical Analysis
The authors demonstrate impressive results, with STAR generating high-quality 4D avatars from a wide range of text prompts. However, the paper does note some limitations. For example, the system may struggle with highly complex or unusual motion descriptions, and the generated characters are still somewhat "generic" in appearance.
Additionally, while STAR's skeleton-aware approach is a key strength, it also means the system is reliant on having an accurate 3D skeleton representation. If the skeleton generation is inaccurate or fails, it could negatively impact the quality of the final animations.
Further research could explore ways to make STAR more robust to challenging inputs, or to provide users with more customization options for the generated characters. Incorporating additional modalities like images or videos could also help improve the realism and diversity of the created avatars.
Conclusion
Overall, STAR represents an important step forward in the field of text-to-avatar generation. By tightly integrating 3D model creation and motion generation, the system can produce high-quality 4D avatars that move naturally and realistically. This technology could have significant applications in virtual worlds, gaming, and other areas where customizable digital humans are in demand.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🛸
0
STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting
Zenghao Chai, Chen Tang, Yongkang Wong, Mohan Kankanhalli
The creation of 4D avatars (i.e., animated 3D avatars) from text description typically uses text-to-image (T2I) diffusion models to synthesize 3D avatars in the canonical space and subsequently applies animation with target motions. However, such an optimization-by-animation paradigm has several drawbacks. (1) For pose-agnostic optimization, the rendered images in canonical pose for naive Score Distillation Sampling (SDS) exhibit domain gap and cannot preserve view-consistency using only T2I priors, and (2) For post hoc animation, simply applying the source motions to target 3D avatars yields translation artifacts and misalignment. To address these issues, we propose Skeleton-aware Text-based 4D Avatar generation with in-network motion Retargeting (STAR). STAR considers the geometry and skeleton differences between the template mesh and target avatar, and corrects the mismatched source motion by resorting to the pretrained motion retargeting techniques. With the informatively retargeted and occlusion-aware skeleton, we embrace the skeleton-conditioned T2I and text-to-video (T2V) priors, and propose a hybrid SDS module to coherently provide multi-view and frame-consistent supervision signals. Hence, STAR can progressively optimize the geometry, texture, and motion in an end-to-end manner. The quantitative and qualitative experiments demonstrate our proposed STAR can synthesize high-quality 4D avatars with vivid animations that align well with the text description. Additional ablation studies shows the contributions of each component in STAR. The source code and demos are available at: href{https://star-avatar.github.io}{https://star-avatar.github.io}.
Read more6/10/2024
0
X-Oscar: A Progressive Framework for High-quality Text-guided 3D Animatable Avatar Generation
Yiwei Ma, Zhekai Lin, Jiayi Ji, Yijun Fan, Xiaoshuai Sun, Rongrong Ji
Recent advancements in automatic 3D avatar generation guided by text have made significant progress. However, existing methods have limitations such as oversaturation and low-quality output. To address these challenges, we propose X-Oscar, a progressive framework for generating high-quality animatable avatars from text prompts. It follows a sequential Geometry->Texture->Animation paradigm, simplifying optimization through step-by-step generation. To tackle oversaturation, we introduce Adaptive Variational Parameter (AVP), representing avatars as an adaptive distribution during training. Additionally, we present Avatar-aware Score Distillation Sampling (ASDS), a novel technique that incorporates avatar-aware noise into rendered images for improved generation quality during optimization. Extensive evaluations confirm the superiority of X-Oscar over existing text-to-3D and text-to-avatar approaches. Our anonymous project page: https://xmu-xiaoma666.github.io/Projects/X-Oscar/.
Read more5/3/2024
0
Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer
Zichen Geng, Caren Han, Zeeshan Hayder, Jian Liu, Mubarak Shah, Ajmal Mian
Text-driven human motion generation is an emerging task in animation and humanoid robot design. Existing algorithms directly generate the full sequence which is computationally expensive and prone to errors as it does not pay special attention to key poses, a process that has been the cornerstone of animation for decades. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text by first generating keyframes followed by in-filling. We use a Variational Autoencoder (VAE) with Kullback-Leibler regularization to project the keyframes into a latent space to reduce dimensionality and further accelerate the subsequent diffusion process. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the keyframe latents and text condition. To complete the motion sequence, we propose a text-guided Transformer designed to perform motion-in-filling, ensuring the preservation of both fidelity and adherence to the physical constraints of human motion. Experiments show that our method achieves state-of-theart results on the HumanML3D dataset outperforming others on all R-precision metrics and MultiModal Distance. KeyMotion also achieves competitive performance on the KIT dataset, achieving the best results on Top3 R-precision, FID, and Diversity metrics.
Read more5/27/2024
0
Semantics-aware Motion Retargeting with Vision-Language Models
Haodong Zhang, ZhiKe Chen, Haocheng Xu, Lei Hao, Xiaofei Wu, Songcen Xu, Zhensong Zhang, Yue Wang, Rong Xiong
Capturing and preserving motion semantics is essential to motion retargeting between animation characters. However, most of the previous works neglect the semantic information or rely on human-designed joint-level representations. Here, we present a novel Semantics-aware Motion reTargeting (SMT) method with the advantage of vision-language models to extract and maintain meaningful motion semantics. We utilize a differentiable module to render 3D motions. Then the high-level motion semantics are incorporated into the motion retargeting process by feeding the vision-language model with the rendered images and aligning the extracted semantic embeddings. To ensure the preservation of fine-grained motion details and high-level semantics, we adopt a two-stage pipeline consisting of skeleton-aware pre-training and fine-tuning with semantics and geometry constraints. Experimental results show the effectiveness of the proposed method in producing high-quality motion retargeting results while accurately preserving motion semantics.
Read more4/16/2024