PET-SQL: A Prompt-enhanced Two-stage Text-to-SQL Framework with Cross-consistency

Introduction

The provided text describes a framework called Prompt-Enhanced Two-stage Text-to-SQL (PET-SQL) that aims to improve the performance of large language models (LLMs) on the task of converting natural language questions into SQL queries.

The key points are:

1. Current approaches on Text-to-SQL focus on representing schema information in prompts, but lack attention to formatting of cell values in tables, which is important for handling conditional statements.

2. The canonical schema-linking module is a separate plugin that produces limited information and is not well-suited for coding LLMs.

3. Existing post-refinement techniques like self-debugging and self-consistency have limitations and do not significantly improve performance.

4. PET-SQL addresses these challenges through: a. A reference-enhanced prompt that includes schema information, sampled cell values, and instructions to minimize SQL execution time. b. A two-stage approach that first generates a preliminary SQL (PreSQL) using few-shot demonstrations, then uses the PreSQL to link the schema and simplify the prompt for the final SQL generation. c. Cross-consistency voting across different LLMs, rather than self-consistency of a single LLM.

The proposed PET-SQL framework achieves state-of-the-art performance of 87.6% execution accuracy on the Spider benchmark, outperforming previous methods.

Related Work

The text discusses the advancements in language models and their impact on the Text2SQL task, which involves translating natural language (NL) questions into structured SQL queries. Early approaches focused on pattern matching and relation-aware self-attention mechanisms to learn representations of questions and schemas. The advent of pre-trained language models and the fine-tuning paradigm subsequently influenced these methods, leading to the use of standard sequence-to-sequence models with transformer architectures for end-to-end NL-to-SQL translation.

The development of large language models (LLMs) has further transformed this field. LLM-based in-context learning methods leverage the semantic understanding, reasoning capabilities, and zero-shot learning ability of LLMs to push the boundaries of performance on Text2SQL evaluation benchmarks. Techniques like decomposing the task into multiple components, in-context learning, and incorporating advanced reasoning methods have been explored to enhance the capabilities of LLMs and improve their performance on Text2SQL tasks.

Methodology

The provided text discusses the objective of LLM-based Text2SQL, which is to translate a natural language question Q on a database D into an executable SQL query s. The likelihood of an LLM M generating a SQL query s can be formally defined as the conditional probability distribution shown in Equation (1).

The text then summarizes the key specifications of the PET-SQL framework, which are:

1. An elaborated prompt that uses customized instructions, basic database information, and samples from stored tables.
2. Instructing the LLM to generate PreSQL, where demonstrations selected from pools using a question similarity-based strategy are prefixed to the prompt as few-shot in-context.
3. Finding question-related tables (schema linking) based on the PreSQL and prompt to yield FinSQL by the linked schema.
4. Ensuring consistency in the predicted results across multiple LLMs.

The provided text discusses prompting and SQL query generation for large language models (LLMs). Key points:

1. The use of prompting styles or templates significantly impacts LLMs' performance in generating SQL queries. The paper recommends using a combination of Code Representation and OpenAI Demonstration formats, termed "reference-enhanced representation" (RE).

2. RE includes three modifications: optimization rule (to minimize SQL execution time while ensuring correctness), cell value references (to provide sample data to help LLMs understand the database), and foreign key declarations (to help LLMs understand table relationships).

3. To further improve SQL generation, the paper proposes a "question skeleton-based PreSQL generation" approach. This retrieves similar question-SQL pairs from a training set, uses them as few-shot demonstrations, and prompts the LLM to generate a preliminary SQL query (PreSQL).

4. The paper then describes a "schema linking and FinSQL generation" step. This identifies the relevant tables and columns mentioned in the PreSQL, simplifies the prompt, and feeds it back to the LLM to generate the final SQL query (FinSQL).

5. Lastly, the paper discusses a "cross consistency" approach, which uses multiple LLMs to generate FinSQL queries and applies majority voting on the executed results to improve reliability, with an option to use fine-grained voting based on PreSQL complexity.

Overall, the paper proposes a multi-stage approach to leverage prompting and few-shot learning to improve the SQL generation capabilities of LLMs.

Experiments

The provided text describes the experimental setup and evaluation of the PET-SQL framework for the Text2SQL task on the Spider benchmark.

Key points:

• The Spider benchmark contains 8659 training instances, 1034 development instances, and 2147 test instances across 200 databases.
• Execution accuracy (EX) is used as the evaluation metric, which measures the proportion of questions where the predicted and ground-truth SQL queries produce identical execution results.
• Five large language models (LLMs) are evaluated, including coding-specific (CodeLlama-34B, SQLCoder-34B) and generic (InternLM-70B, SenseChat, GPT4-0613) models.
• PET-SQL achieves the highest EX among non-learning-based methods on the Spider test set, surpassing the second-ranked DAIL-SQL by about 1%.
• Comparisons show PET-SQL consistently outperforms DAIL-SQL across different LLMs on the test set.
• The proposed prompt (RE_p) demonstrates superior performance compared to other prompts, with improvements of 1-7% on the development and test sets.
• Ablation studies reveal the importance of the cell value references component in the prompt.
• Schema linking using GPT4 leads to significant simplification of the input prompts and improves performance across the evaluated LLMs.
• Cross-consistency voting among the five LLMs further boosts performance, outperforming self-consistency.

Conclusion

The PET-SQL framework aims to improve performance on Text2SQL tasks. It enhances the prompt and leverages consistency across large language models (LLMs). The approach achieves an 87.6% execution accuracy on the Spider leaderboard. The framework also includes a PreSQL-based schema linking method to simplify the prompt information, improving the efficiency and accuracy of LLMs in generating SQL queries. Overall, the PET-SQL framework demonstrates promising results and suggests avenues for further advancements in Text2SQL tasks.