Add comprehensive model card for CSC-SQL model (#1)
Browse files- Add comprehensive model card for CSC-SQL model (67f83713c10f23d47f88c25084ce61d5cf5c377e)
Co-authored-by: Niels Rogge <[email protected]>
README.md
ADDED
@@ -0,0 +1,141 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: cc-by-nc-4.0
|
3 |
+
pipeline_tag: text-generation
|
4 |
+
library_name: transformers
|
5 |
+
tags:
|
6 |
+
- text-to-sql
|
7 |
+
- sql
|
8 |
+
- qwen2
|
9 |
+
datasets:
|
10 |
+
- cycloneboy/bird_train
|
11 |
+
base_model: Qwen/Qwen2.5-7B-Instruct
|
12 |
+
---
|
13 |
+
|
14 |
+
# CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning
|
15 |
+
|
16 |
+
This repository contains the `CscSQL-Grpo-Qwen2.5-Coder-7B-Instruct` model, presented in the paper [CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning](https://huggingface.co/papers/2505.13271).
|
17 |
+
|
18 |
+
## Abstract
|
19 |
+
|
20 |
+
Large language models (LLMs) have demonstrated strong capabilities in translating natural language questions about relational databases into SQL queries. In particular, test-time scaling techniques such as Self-Consistency and Self-Correction can enhance SQL generation accuracy by increasing computational effort during inference. However, these methods have notable limitations: Self-Consistency may select suboptimal outputs despite majority votes, while Self-Correction typically addresses only syntactic errors. To leverage the strengths of both approaches, we propose CSC-SQL, a novel method that integrates Self-Consistency and Self-Correction. CSC-SQL selects the two most frequently occurring outputs from parallel sampling and feeds them into a merge revision model for correction. Additionally, we employ the Group Relative Policy Optimization (GRPO) algorithm to fine-tune both the SQL generation and revision models via reinforcement learning, significantly enhancing output quality. Experimental results confirm the effectiveness and generalizability of CSC-SQL. On the BIRD private test set, our 7B model achieves 71.72% execution accuracy, while the 32B model achieves 73.67%.
|
21 |
+
|
22 |
+
## Code
|
23 |
+
|
24 |
+
The official implementation, including training and evaluation scripts, can be found on GitHub: [https://github.com/CycloneBoy/csc_sql](https://github.com/CycloneBoy/csc_sql)
|
25 |
+
|
26 |
+
## Introduction
|
27 |
+
|
28 |
+
CSC-SQL is a novel method that integrates Self-Consistency and Self-Correction to enhance SQL generation accuracy. It addresses the limitations of existing test-time scaling techniques by combining their strengths. The method involves selecting the two most frequently occurring outputs from parallel sampling and feeding them into a merge revision model for correction. Furthermore, the Group Relative Policy Optimization (GRPO) algorithm is employed to fine-tune both the SQL generation and revision models via reinforcement learning, leading to significantly enhanced output quality.
|
29 |
+
|
30 |
+
The framework overview is illustrated below:
|
31 |
+
|
32 |
+

|
33 |
+
|
34 |
+
## Main Results
|
35 |
+
|
36 |
+
The CSC-SQL model achieves state-of-the-art results in Text-to-SQL generation. On the BIRD private test set, the 7B model achieves 71.72% execution accuracy, while the 32B model achieves 73.67%.
|
37 |
+
|
38 |
+
Performance Comparison of different Text-to-SQL methods on BIRD dev and test dataset:
|
39 |
+
<img src="https://github.com/CycloneBoy/csc_sql/raw/main/data/image/csc_sql_result_main.png" height="500" alt="Performance Comparison">
|
40 |
+
|
41 |
+
## Models and Datasets
|
42 |
+
|
43 |
+
The project provides various models and datasets, which can be found on Hugging Face and ModelScope:
|
44 |
+
|
45 |
+
| **Model and Dataset** | Modelscope | HuggingFace |
|
46 |
+
|---------------------------------------|-------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
|
47 |
+
| bird train and dev dataset | [π€ Modelscope](https://modelscope.cn/datasets/cycloneboy/bird_train) | [π€ HuggingFace](https://huggingface.co/datasets/cycloneboy/bird_train) |
|
48 |
+
| CscSQL-Merge-Qwen2.5-Coder-3B-Instruct | [π€ Modelscope](https://modelscope.cn/models/cycloneboy/CscSQL-Merge-Qwen2.5-Coder-3B-Instruct) | [π€ HuggingFace](https://huggingface.co/cycloneboy/CscSQL-Merge-Qwen2.5-Coder-3B-Instruct) |
|
49 |
+
| CscSQL-Merge-Qwen2.5-Coder-7B-Instruct | [π€ Modelscope](https://modelscope.cn/models/cycloneboy/CscSQL-Merge-Qwen2.5-Coder-7B-Instruct) | [π€ HuggingFace](https://huggingface.co/cycloneboy/CscSQL-Merge-Qwen2.5-Coder-7B-Instruct) |
|
50 |
+
| CscSQL-Grpo-Qwen2.5-Coder-3B-Instruct | [π€ Modelscope](https://modelscope.cn/models/cycloneboy/CscSQL-Grpo-Qwen2.5-Coder-3B-Instruct) | [π€ HuggingFace](https://huggingface.co/cycloneboy/CscSQL-Grpo-Qwen2.5-Coder-3B-Instruct) |
|
51 |
+
| CscSQL-Grpo-XiYanSQL-QwenCoder-3B-2502 | [π€ Modelscope](https://modelscope.cn/models/cycloneboy/CscSQL-Grpo-XiYanSQL-QwenCoder-3B-2502) | [π€ HuggingFace](https://huggingface.co/cycloneboy/CscSQL-Grpo-XiYanSQL-QwenCoder-3B-2502) |
|
52 |
+
| CscSQL-Grpo-Qwen2.5-Coder-7B-Instruct | [π€ Modelscope](https://modelscope.cn/models/cycloneboy/CscSQL-Grpo-Qwen2.5-Coder-7B-Instruct) | [π€ HuggingFace](https://huggingface.co/cycloneboy/CscSQL-Grpo-Qwen2.5-Coder-7B-Instruct) |
|
53 |
+
| CscSQL-Grpo-XiYanSQL-QwenCoder-7B-2502 | [π€ Modelscope](https://modelscope.cn/models/cycloneboy/CscSQL-Grpo-XiYanSQL-QwenCoder-7B-2502) | [π€ HuggingFace](https://huggingface.co/cycloneboy/CscSQL-Grpo-XiYanSQL-QwenCoder-7B-2502) |
|
54 |
+
|
55 |
+
## Usage
|
56 |
+
|
57 |
+
You can use this model with the Hugging Face `transformers` library. Here's a quick example for Text-to-SQL generation following the Qwen chat template:
|
58 |
+
|
59 |
+
```python
|
60 |
+
import torch
|
61 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
|
62 |
+
|
63 |
+
model_name = "cycloneboy/CscSQL-Grpo-Qwen2.5-Coder-7B-Instruct"
|
64 |
+
|
65 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
66 |
+
model = AutoModelForCausalLM.from_pretrained(
|
67 |
+
model_name,
|
68 |
+
device_map="auto",
|
69 |
+
torch_dtype=torch.bfloat16,
|
70 |
+
trust_remote_code=True
|
71 |
+
).eval()
|
72 |
+
|
73 |
+
# Example natural language question and a simplified database schema
|
74 |
+
question = "List the names of all employees who work in the 'Sales' department."
|
75 |
+
schema = """
|
76 |
+
CREATE TABLE employees (
|
77 |
+
employee_id INT PRIMARY KEY,
|
78 |
+
name VARCHAR(255),
|
79 |
+
department_id INT
|
80 |
+
);
|
81 |
+
|
82 |
+
CREATE TABLE departments (
|
83 |
+
department_id INT PRIMARY KEY,
|
84 |
+
department_name VARCHAR(255)
|
85 |
+
);
|
86 |
+
"""
|
87 |
+
|
88 |
+
# Construct the prompt according to the model's expected input format for Text-to-SQL
|
89 |
+
# This is typically a combination of natural language question and the schema
|
90 |
+
user_prompt = f"Question: {question}
|
91 |
+
Schema: {schema}
|
92 |
+
SQL:"
|
93 |
+
|
94 |
+
messages = [
|
95 |
+
{"role": "user", "content": user_prompt}
|
96 |
+
]
|
97 |
+
|
98 |
+
# Apply the chat template to format the input for the model
|
99 |
+
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
100 |
+
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
101 |
+
|
102 |
+
# Define generation configuration
|
103 |
+
generation_config = GenerationConfig(
|
104 |
+
do_sample=True,
|
105 |
+
temperature=0.7,
|
106 |
+
top_p=0.8,
|
107 |
+
top_k=20,
|
108 |
+
repetition_penalty=1.05,
|
109 |
+
max_new_tokens=512, # Adjust as needed for SQL query length
|
110 |
+
eos_token_id=tokenizer.eos_token_id,
|
111 |
+
pad_token_id=tokenizer.pad_token_id,
|
112 |
+
)
|
113 |
+
|
114 |
+
# Generate the SQL query
|
115 |
+
generated_ids = model.generate(
|
116 |
+
model_inputs.input_ids,
|
117 |
+
generation_config=generation_config
|
118 |
+
)
|
119 |
+
|
120 |
+
# Decode the generated SQL, skipping the input prompt
|
121 |
+
generated_sql = tokenizer.batch_decode(generated_ids[:, model_inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
|
122 |
+
|
123 |
+
print("Generated SQL Query:")
|
124 |
+
print(generated_sql)
|
125 |
+
```
|
126 |
+
|
127 |
+
## Citation
|
128 |
+
|
129 |
+
If you find our work helpful or inspiring, please feel free to cite it:
|
130 |
+
|
131 |
+
```bibtex
|
132 |
+
@misc{sheng2025cscsqlcorrectiveselfconsistencytexttosql,
|
133 |
+
title={CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning},
|
134 |
+
author={Lei Sheng and Shuai-Shuai Xu},
|
135 |
+
year={2025},
|
136 |
+
eprint={2505.13271},
|
137 |
+
archivePrefix={arXiv},
|
138 |
+
primaryClass={cs.CL},
|
139 |
+
url={https://arxiv.org/abs/2505.13271},
|
140 |
+
}
|
141 |
+
```
|