|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- mlabonne/synthetic_text_to_sql-ShareGPT |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- unsloth/Llama-3.2-1B |
|
|
tags: |
|
|
- sql-generation |
|
|
- llama |
|
|
- unsloth |
|
|
- text-to-sql |
|
|
- lora |
|
|
--- |
|
|
# Fine-tuning LLaMA 3.2 1B for SQL Generation |
|
|
|
|
|
This project is about fine-tuning a small LLaMA model (1B) to generate SQL queries from natural language. I'm using a dataset that contains examples of how people ask questions and how those get translated into SQL. |
|
|
|
|
|
## What I'm Doing |
|
|
|
|
|
* I'm starting with a pre-trained LLaMA 3.2 1B model. |
|
|
* I use a dataset called `synthetic_text_to_sql-ShareGPT` which has examples of prompts and the corresponding SQL queries. |
|
|
Dataset URL: [https://huggingface.co/datasets/mlabonne/synthetic_text_to_sql-ShareGPT](https://huggingface.co/datasets/mlabonne/synthetic_text_to_sql-ShareGPT) |
|
|
* I fine-tune the model using Unsloth libary with LoRA Adapters. This allows me to train only parts of the model, which makes it much faster and memory-efficient. |
|
|
|
|
|
## Evaluation Process |
|
|
|
|
|
The evaluation pipeline is implemented in `Evaluate_LLM.ipynb`: |
|
|
|
|
|
1. **SQL Question Generation** : Groq’s `llama3-8b-8192` model generates 10 SQL question blocks, each with table creation, inserts, and a natural language SQL question. |
|
|
|
|
|
2. **Model Answering** : Each question is passed to a local fine-tuned LLaMA model (using `llama-cpp-python`) to generate SQL queries and explanations. |
|
|
|
|
|
3. **Automated Evaluation** : Groq’s `gemma2-9b-it` model acts as an expert tutor to score each (question, answer) pair on correctness and completeness (1–10 scale) and provide feedback. |
|
|
|
|
|
4. **Summary** : Average scores and detailed feedback for all questions are output. |
|
|
|
|
|
*Note:* |
|
|
- The question generation and evaluation both use Groq's hosted models (Llama 3_8b for question generation, Gemma 2_9b for evaluation). |
|
|
- The local LLaMA_3.2_1b fine tuned model is only used for generating answers. |
|
|
- Normally, I use Gemini for evaluation, but due to Gemini being slow today, I used Groq for both question generation and evaluation in this run. |
|
|
|
|
|
## Why I’m Doing This |
|
|
|
|
|
I want to build a model that can understand plain English and generate accurate SQL queries. This can be useful for tools where people want to ask questions about their data without writing SQL themselves. |
|
|
|
|
|
## Where to Find the Model & Notebooks |
|
|
|
|
|
You can find the fine-tuned model, including the .gguf file format for easy local use, on my Hugging Face repository: |
|
|
|
|
|
👉 https://huggingface.co/Adhishtanaka/llama_3.2_1b_SQL/tree/main |
|
|
|
|
|
You can find the Jupyter Notebook files used in this project directly in this repository: |
|
|
|
|
|
- `Evaluate_LLM.ipynb`: The evaluation pipeline for the fine-tuned model. |
|
|
- `Llama3.2_1B-SQL.ipynb`: The main notebook for fine-tuning and experimentation. |
|
|
|
|
|
👉 Browse these files in the [GitHub repository](https://github.com/Adhishtanaka/llama3.2_1.b-SQL) for full code and documentation. |
