Generative Design through Quality-Diversity Data Synthesis and Language Models
2405.09997
0
0
Abstract
Two fundamental challenges face generative models in engineering applications: the acquisition of high-performing, diverse datasets, and the adherence to precise constraints in generated designs. We propose a novel approach combining optimization, constraint satisfaction, and language models to tackle these challenges in architectural design. Our method uses Quality-Diversity (QD) to generate a diverse, high-performing dataset. We then fine-tune a language model with this dataset to generate high-level designs. These designs are then refined into detailed, constraint-compliant layouts using the Wave Function Collapse algorithm. Our system demonstrates reliable adherence to textual guidance, enabling the generation of layouts with targeted architectural and performance features. Crucially, our results indicate that data synthesized through the evolutionary search of QD not only improves overall model performance but is essential for the model's ability to closely adhere to textual guidance. This improvement underscores the pivotal role evolutionary computation can play in creating the datasets key to training generative models for design. Web article at https://tilegpt.github.io
Create account to get full access
Overview
- This paper explores the use of quality-diversity data synthesis and language models for generative design.
- It proposes a new approach to generating diverse design solutions by combining techniques from the fields of quality-diversity optimization and large language models.
- The authors demonstrate the effectiveness of their approach on several design problems, showing that it can produce a wide range of high-quality and diverse design alternatives.
Plain English Explanation
The paper presents a new way to generate a variety of design ideas using a combination of two powerful techniques: quality-diversity optimization and large language models. Quality-diversity optimization is a method that aims to create a diverse set of high-performing solutions, rather than just a single optimal solution. Large language models are artificial intelligence systems that can generate human-like text based on patterns in a large corpus of data.
By bringing these two approaches together, the researchers created a system that can generate a wide range of design ideas, each of which is of high quality. This could be useful for designers, engineers, or other creative professionals who need to explore many different possibilities before settling on a final design.
For example, imagine you're designing a new chair. Rather than just trying to optimize for a single "best" chair, this approach would allow you to generate a diverse set of chair designs, each of which is well-crafted and functional. This could spark new ideas, help you identify unforeseen opportunities, and ultimately lead to a more innovative final product.
Technical Explanation
The paper introduces a generative design approach that combines quality-diversity optimization and large language models. Quality-diversity optimization is used to generate a diverse set of high-quality design solutions, while the language model is used to synthesize new design data that can be used to train the optimization process.
The authors demonstrate their approach on several design problems, including the design of mechanical systems and household objects. They show that their method can generate a wide range of unique and high-performing design alternatives, outperforming more traditional optimization-based approaches.
Critical Analysis
The paper provides a compelling proof-of-concept for the use of quality-diversity optimization and language models in generative design. However, the authors acknowledge that their approach is still relatively limited in the types of design problems it can handle and the complexity of the solutions it can generate.
Additionally, the paper does not address potential concerns around the use of language models in design, such as the risk of biased or unethical outputs. Further research is needed to understand the limitations and potential pitfalls of this approach, as well as to explore ways to ensure the generated designs are safe, ethical, and aligned with human values.
Conclusion
This paper presents an innovative approach to generative design that leverages the strengths of quality-diversity optimization and large language models. By combining these techniques, the researchers were able to generate a diverse set of high-quality design alternatives, which could be a valuable tool for designers, engineers, and other creative professionals.
While the approach has promising potential, further research is needed to address its limitations and ensure the ethical and responsible development of such systems. Nevertheless, this work represents an important step forward in the field of computational design and generative AI.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Large Language Models as In-context AI Generators for Quality-Diversity
Bryan Lim, Manon Flageat, Antoine Cully
0
0
Quality-Diversity (QD) approaches are a promising direction to develop open-ended processes as they can discover archives of high-quality solutions across diverse niches. While already successful in many applications, QD approaches usually rely on combining only one or two solutions to generate new candidate solutions. As observed in open-ended processes such as technological evolution, wisely combining large diversity of these solutions could lead to more innovative solutions and potentially boost the productivity of QD search. In this work, we propose to exploit the pattern-matching capabilities of generative models to enable such efficient solution combinations. We introduce In-context QD, a framework of techniques that aim to elicit the in-context capabilities of pre-trained Large Language Models (LLMs) to generate interesting solutions using few-shot and many-shot prompting with quality-diverse examples from the QD archive as context. Applied to a series of common QD domains, In-context QD displays promising results compared to both QD baselines and similar strategies developed for single-objective optimization. Additionally, this result holds across multiple values of parameter sizes and archive population sizes, as well as across domains with distinct characteristics from BBO functions to policy search. Finally, we perform an extensive ablation that highlights the key prompt design considerations that encourage the generation of promising solutions for QD.
6/6/2024
LLM2FEA: Discover Novel Designs with Generative Evolutionary Multitasking
Melvin Wong, Jiao Liu, Thiago Rios, Stefan Menzel, Yew Soon Ong
0
0
The rapid research and development of generative artificial intelligence has enabled the generation of high-quality images, text, and 3D models from text prompts. This advancement impels an inquiry into whether these models can be leveraged to create digital artifacts for both creative and engineering applications. Drawing on innovative designs from other domains may be one answer to this question, much like the historical practice of ``bionics, where humans have sought inspiration from nature's exemplary designs. This raises the intriguing possibility of using generative models to simultaneously tackle design tasks across multiple domains, facilitating cross-domain learning and resulting in a series of innovative design solutions. In this paper, we propose LLM2FEA as the first attempt to discover novel designs in generative models by transferring knowledge across multiple domains. By utilizing a multi-factorial evolutionary algorithm (MFEA) to drive a large language model, LLM2FEA integrates knowledge from various fields to generate prompts that guide the generative model in discovering novel and practical objects. Experimental results in the context of 3D aerodynamic design verify the discovery capabilities of the proposed LLM2FEA. The designs generated by LLM2FEA not only satisfy practicality requirements to a certain degree but also feature novel and aesthetically pleasing shapes, demonstrating the potential applications of LLM2FEA in discovery tasks.
6/24/2024
DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation
Anna C. Doris, Daniele Grandi, Ryan Tomich, Md Ferdous Alam, Hyunmin Cheong, Faez Ahmed
0
0
This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data-including textual design requirements, CAD images, and engineering drawings-derived from the Formula SAE student competition. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different sources. The benchmark features automatic evaluation metrics and is divided into segments-Rule Comprehension, Rule Compliance, and Rule Extraction-based on tasks that engineers perform when designing according to requirements. We evaluate state-of-the-art models like GPT4 and LLaVA against the benchmark, and our study uncovers the existing gaps in MLLMs' abilities to interpret complex engineering documentation. Key findings suggest that while MLLMs demonstrate potential in navigating technical documents, substantial limitations exist, particularly in accurately extracting and applying detailed requirements to engineering designs. This benchmark sets a foundation for future advancements in AI-supported engineering design processes. DesignQA is publicly available at: https://github.com/anniedoris/design_qa/.
4/12/2024
š ļø
Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, Joel Lehman
0
0
Reinforcement Learning from Human Feedback (RLHF) has shown potential in qualitative tasks where easily defined performance measures are lacking. However, there are drawbacks when RLHF is commonly used to optimize for average human preferences, especially in generative tasks that demand diverse model responses. Meanwhile, Quality Diversity (QD) algorithms excel at identifying diverse and high-quality solutions but often rely on manually crafted diversity metrics. This paper introduces Quality Diversity through Human Feedback (QDHF), a novel approach that progressively infers diversity metrics from human judgments of similarity among solutions, thereby enhancing the applicability and effectiveness of QD algorithms in complex and open-ended domains. Empirical studies show that QDHF significantly outperforms state-of-the-art methods in automatic diversity discovery and matches the efficacy of QD with manually crafted diversity metrics on standard benchmarks in robotics and reinforcement learning. Notably, in open-ended generative tasks, QDHF substantially enhances the diversity of text-to-image generation from a diffusion model and is more favorably received in user studies. We conclude by analyzing QDHF's scalability, robustness, and quality of derived diversity metrics, emphasizing its strength in open-ended optimization tasks. Code and tutorials are available at https://liding.info/qdhf.
6/5/2024