Hugging Face's logo

llmware
/
slim-sql-1b-v0

Text Generation
Transformers
PyTorch
llama
text-generation-inference
Model card Files Community
1
slim-sql-1b-v0 / README.md
doberst113080's picture
doberst113080
Update README.md
b67e8e7
preview code
|
raw
history blame
3.78 kB
metadata
license: apache-2.0

Model Card for Model ID

slim-sql-1b-v0 is the first model in the SLIM (Specialized Language Instruct Model) series.

Benchmark Tests

Evaluated against 100 test SQL queries with under 100 characters. 1 point given for exact string match, 0 given for incorrect answer.

--Accuracy Score: 86 correct out of 100

  • 8 incorrect answers attributed to query structure ordering or naming convention differences
  • 6 incorrect answers attributed to incorrect variable selection or aggregate function use

Model Description

Direct Use

slim-sql-1b-v0 is designed to generate accurate SQL queries for data retrival on simple table structures given a natural language prompt. For best results, prompts should be structured as a question to retrieve information and perform aggregate functions on one or several variables.

Bias, Risks, and Limitations

Any model can provide inaccurate or incomplete information, and should be used in conjunction with appropriate safeguards and fact-checking mechanisms.

How to Get Started with the Model

The fastest way to get started with slim is through direct import in transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM  
tokenizer = AutoTokenizer.from_pretrained("slim-sql-1b-v0")  
model = AutoModelForCausalLM.from_pretrained("slim-sql-1b-v0")

Please refer to the generation_test.py files in the Files repository, which includes 100 samples and script to test the model.

The sql-slim model was fine-tuned with a simple "<human> and <bot> wrapper", so to get the best results, wrap inference entries as:

full_prompt = "<human>: " + my_prompt + "\n" + "<bot>:"

The prompt consists of two sub-parts:

  1. Table creation prompt providing table name, variables, and variable type.
  2. Specific question or instruction based on the text passage

Training sample example: "text": ": CREATE TABLE table_name_8 ( partner VARCHAR, date VARCHAR )\nName the partner for may 2, 1993\n:SELECT partner FROM table_name_8 WHERE date = "may 2, 1993""} {"text": ": CREATE TABLE table_name_97 ( Id VARCHAR )\nName the 2012 when 2011 is qf\n:SELECT 2012 FROM table_name_97 WHERE 2011 = "qf""

Test samples are provided in this repo ("sql-slim-1b_test_questions")

If you are using a HuggingFace generation script:

# prepare prompt packaging used in fine-tuning process
new_prompt = "<human>: " + entries["context"] + "\n" + entries["query"] + "\n" + "<bot>:"

inputs = tokenizer(new_prompt, return_tensors="pt")  
start_of_output = len(inputs.input_ids[0])

#   temperature: set at 0.3 for consistency of output
#   max_new_tokens:  set at 100 - may prematurely stop a few of the summaries

outputs = model.generate(
        inputs.input_ids.to(device),
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
        do_sample=True,
        temperature=0.3,
        max_new_tokens=100,
        )

output_only = tokenizer.decode(outputs[0][start_of_output:],skip_special_tokens=True)

Model Card Contact

Dylan Oberst & llmware team