crimson3327/text_to_sql

Llama-3.2-3B Text-to-SQL

A LoRA fine-tune of Llama-3.2-3B-Instruct for generating SQL queries from a natural language question plus a table schema.

Trained locally on an AMD Radeon RX 9070 XT using Unsloth.

What this model does

Given a table schema and a question in plain English, it returns a SQL query — no explanation, no preamble, no alternative approaches. The base instruct model tends to either refuse ("I don't have access to your database") or respond with an explanatory Python example instead of raw SQL. Fine-tuning fixed that: this model reliably answers with bare SQL by default.

Prompt format

This model expects the schema to be included in the prompt. Without one, it will guess plausible-sounding table/column names rather than asking for clarification — same as any model would.

Given this schema: CREATE TABLE employees (name VARCHAR, salary INTEGER)

Answer this question in SQL: List the names of employees who earn more than 50000

Expected output:

SELECT name FROM employees WHERE salary > 50000

Use the tokenizer's chat template (apply_chat_template) with this as a single user turn — see the usage example below.

Usage

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "crimson3327/text_to_sql",
    max_seq_length = 1024,
)
FastLanguageModel.for_inference(model)

schema = "CREATE TABLE employees (name VARCHAR, salary INTEGER)"
question = "List the names of employees who earn more than 50000"

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": f"Given this schema: {schema}\n\nAnswer this question in SQL: {question}"}],
    tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(input_ids=inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training details

• Base model: unsloth/Llama-3.2-3B-Instruct
• Method: LoRA, rank 16, alpha 16, all attention + MLP projections targeted
• Dataset: b-mc2/sql-create-context (78,577 question/schema/SQL triples), trained on the first 74,000 rows
• Held-out eval set: the remaining ~4,500 rows, never seen during training
• Steps: 1,200 (batch size 2, gradient accumulation 4 — effective batch 8)
• Optimizer: AdamW, linear LR schedule, 2e-4 peak learning rate
• Checkpoint selection: best checkpoint chosen automatically by eval loss, not training loss, to avoid shipping an overfit checkpoint
• Final eval loss: 0.559

Example: base model vs. this fine-tune

Same prompt, no schema given, asked to filter employee data:

Base Llama-3.2-3B-Instruct — rewrote the task as a pandas exercise with invented sample data, offered a second alternative approach using bitwise operators, no SQL produced.

This model:

SELECT * FROM employee WHERE salary > 10000 AND Department = 'IT' AND Name = 'John Doe'

Known limitations

• Multi-table joins (3+ tables) with aggregation are inconsistent. Earlier training checkpoints produced invalid SQL (joins placed after GROUP BY/HAVING) and hallucinated columns on this pattern specifically. The checkpoint published here (trained with the held-out eval split) resolved the specific cases tested, but this remains the weakest area — verify output on complex joins before trusting it blindly.
• Ambiguous negation phrasing (e.g. "not yet shipped") may be interpreted as a literal status string rather than a negated condition (!=). Prefer explicit phrasing in questions where this matters.
• No schema validation. The model doesn't verify that referenced columns/tables exist — always pass an accurate schema and review output before executing against a real database.
• Trained on SQLite-flavored syntax (matching the source dataset); some queries may need adjustment for strict-mode PostgreSQL or other dialects.

Acknowledgements

Fine-tuned with Unsloth. Dataset: b-mc2/sql-create-context.