CityCraft: A Real Crafter for 3D City Generation
0
Sign in to get full access
Overview
- Presents a new 3D city generation system called CityCraft that allows users to interactively create detailed urban environments
- Leverages procedural modeling and generative AI techniques to enable intuitive city-building
- Aims to make 3D city generation more accessible and enable the creation of high-quality virtual environments
Plain English Explanation
CityCraft is a new tool that makes it easier for people to build detailed 3D cities from scratch. Instead of having to manually design every building, street, and park, CityCraft uses smart algorithms to help speed up the process and make it more intuitive.
The system combines two key technologies: procedural modeling and generative AI. Procedural modeling is a way of generating content programmatically, following a set of rules. This allows CityCraft to automatically create realistic urban elements like buildings, roads, and vegetation. The generative AI component then takes this a step further, learning from examples of real cities to generate new, plausible urban layouts.
The goal is to make 3D city creation more accessible, enabling users to quickly build high-quality virtual environments for games, simulations, or other applications. Rather than requiring advanced 3D modeling skills, CityCraft allows anyone to interactively "craft" a city, experimenting with different designs and ideas.
Technical Explanation
CityCraft combines procedural modeling techniques with a generative adversarial network (GAN) to create an interactive 3D city generation system. The procedural modeling component uses shape grammars, L-systems, and other rule-based methods to generate the basic urban elements like buildings, roads, and vegetation.
The generative AI module, based on a conditional GAN architecture, then learns to compose these elements into plausible, large-scale 3D city layouts. By training on datasets of real cities, the GAN develops an understanding of common urban patterns and can generate novel yet realistic city designs. Users can interact with the system, adjusting high-level parameters to steer the generation process and experiment with different design ideas.
Key innovations include the use of multi-scale conditioning to capture dependencies between city elements at various levels of detail, as well as techniques for handling unbounded city extents. The system is evaluated through both quantitative metrics and qualitative user studies, demonstrating its ability to generate diverse and compelling 3D urban environments.
Critical Analysis
The CityCraft paper presents a compelling approach to 3D city generation, but there are a few potential limitations worth considering. While the system can produce high-quality results, it relies on training data of real cities, which may limit its ability to generate truly novel or unconventional urban designs.
Additionally, the paper does not extensively explore the impact of user interaction on the city generation process. It would be valuable to understand how the system's outputs change as users adjust the various parameters and controls, and whether this allows for the creation of highly personalized city models.
Broader questions around the societal and ethical implications of such city generation tools also merit further investigation. For example, how might CityCraft be used to plan or reimagine urban spaces, and what biases or unintended consequences could arise from its use?
Overall, the CityCraft work represents an important step forward in making 3D city generation more accessible and intuitive. However, continued research and critical analysis will be crucial to ensure these tools are developed and deployed responsibly.
Conclusion
CityCraft presents a novel approach to 3D city generation that combines procedural modeling and generative AI to enable more intuitive and accessible urban design. By leveraging these advanced techniques, the system can quickly generate detailed, plausible 3D cities that can be used in a variety of applications, from video games to urban planning.
The work demonstrates the potential for generative AI to empower non-experts to create high-quality virtual environments, opening up new possibilities for creativity, experimentation, and exploration of urban spaces. As the field of 3D city generation continues to evolve, tools like CityCraft could play a significant role in shaping the future of how we design and interact with our built environments.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
CityCraft: A Real Crafter for 3D City Generation
Jie Deng, Wenhao Chai, Junsheng Huang, Zhonghan Zhao, Qixuan Huang, Mingyan Gao, Jianshu Guo, Shengyu Hao, Wenhao Hu, Jenq-Neng Hwang, Xi Li, Gaoang Wang
City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning and monitoring solutions. Existing methods have employed a two-stage process involving city layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity and noticeable artifacts in the rendered city scenes. The rendered scenes lack variety, resembling the training images, resulting in monotonous styles. Additionally, these methods lack planning capabilities, leading to less realistic generated scenes. In this paper, we introduce CityCraft, an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is utilized to strategically make land-use plans within these layouts based on user prompts and language guidelines. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction. Furthermore, we contribute two new datasets to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban areas, corresponding satellite images, and detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance in generating realistic 3D cities.
Read more6/10/2024
📈
64
CityDreamer: Compositional Generative Model of Unbounded 3D Cities
Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu
3D city generation is a desirable yet challenging task, since humans are more sensitive to structural distortions in urban environments. Additionally, generating 3D cities is more complex than 3D natural scenes since buildings, as objects of the same class, exhibit a wider range of appearances compared to the relatively consistent appearance of objects like trees in natural scenes. To address these challenges, we propose textbf{CityDreamer}, a compositional generative model designed specifically for unbounded 3D cities. Our key insight is that 3D city generation should be a composition of different types of neural fields: 1) various building instances, and 2) background stuff, such as roads and green lands. Specifically, we adopt the bird's eye view scene representation and employ a volumetric render for both instance-oriented and stuff-oriented neural fields. The generative hash grid and periodic positional embedding are tailored as scene parameterization to suit the distinct characteristics of building instances and background stuff. Furthermore, we contribute a suite of CityGen Datasets, including OSM and GoogleEarth, which comprises a vast amount of real-world city imagery to enhance the realism of the generated 3D cities both in their layouts and appearances. CityDreamer achieves state-of-the-art performance not only in generating realistic 3D cities but also in localized editing within the generated cities.
Read more6/7/2024
0
UrbanWorld: An Urban World Model for 3D City Generation
Yu Shang, Jiansheng Chen, Hangyu Fan, Jingtao Ding, Jie Feng, Yong Li
Cities, as the most fundamental environment of human life, encompass diverse physical elements such as buildings, roads and vegetation with complex interconnection. Crafting realistic, interactive 3D urban environments plays a crucial role in constructing AI agents capable of perceiving, decision-making, and acting like humans in real-world environments. However, creating high-fidelity 3D urban environments usually entails extensive manual labor from designers, involving intricate detailing and accurate representation of complex urban features. Therefore, how to accomplish this in an automatical way remains a longstanding challenge. Toward this problem, we propose UrbanWorld, the first generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. UrbanWorld incorporates four key stages in the automatical crafting pipeline: 3D layout generation from openly accessible OSM data, urban scene planning and designing with a powerful urban multimodal large language model (Urban MLLM), controllable urban asset rendering with advanced 3D diffusion techniques, and finally the MLLM-assisted scene refinement. The crafted high-fidelity 3D urban environments enable realistic feedback and interactions for general AI and machine perceptual systems in simulations. We are working on contributing UrbanWorld as an open-source and versatile platform for evaluating and improving AI abilities in perception, decision-making, and interaction in realistic urban environments.
Read more7/17/2024
1
CityX: Controllable Procedural Content Generation for Unbounded 3D Cities
Shougao Zhang, Mengqi Zhou, Yuxi Wang, Chuanchen Luo, Rongyu Wang, Yiwei Li, Xucheng Yin, Zhaoxiang Zhang, Junran Peng
Generating a realistic, large-scale 3D virtual city remains a complex challenge due to the involvement of numerous 3D assets, various city styles, and strict layout constraints. Existing approaches provide promising attempts at procedural content generation to create large-scale scenes using Blender agents. However, they face crucial issues such as difficulties in scaling up generation capability and achieving fine-grained control at the semantic layout level. To address these problems, we propose a novel multi-modal controllable procedural content generation method, named CityX, which enhances realistic, unbounded 3D city generation guided by multiple layout conditions, including OSM, semantic maps, and satellite images. Specifically, the proposed method contains a general protocol for integrating various PCG plugins and a multi-agent framework for transforming instructions into executable Blender actions. Through this effective framework, CityX shows the potential to build an innovative ecosystem for 3D scene generation by bridging the gap between the quality of generated assets and industrial requirements. Extensive experiments have demonstrated the effectiveness of our method in creating high-quality, diverse, and unbounded cities guided by multi-modal conditions. Our project page: https://cityx-lab.github.io.
Read more8/7/2024