docs/Phi3-Text-to-SQL-Finetuning-Quantization.md

# Fine-tuning Phi-3 for Text-to-SQL, and serving it on a free CPU

I wanted a small model that could turn plain English into SQL for one specific database, run without a paid GPU, and be right often enough to be useful. This is the writeup of how that went, including the parts that did not go to plan.

If you want to poke at it, the [live demo](https://huggingface.co/spaces/Bhuvandesai/phi3-text-to-sql-studio), the [LoRA adapter](https://huggingface.co/Bhuvandesai/phi3-text-to-sql-adapter), and the [quantized GGUF](https://huggingface.co/Bhuvandesai/phi3-text-to-sql-gguf) are all up.

## Why fine-tune a small model at all

You could send this to a frontier API and move on. A lot of real projects can't. The schema is private, the per-call cost adds up at volume, you want it running on your own box, or you just want the model to know your tables instead of guessing at them. The usual answer is to take a strong small open model, teach it the task cheaply, and shrink it so it runs on hardware you already have.

That is three separate ideas, so here is the quick version of each before the numbers start.

Full fine-tuning updates every weight in the model. For a 3.8B model that means holding the weights, a gradient for each one, and two optimizer values per weight, which adds up to north of 30 GB before a single batch runs. It does not fit on a laptop GPU. LoRA avoids this by freezing the original weights and learning a small low-rank correction on the side: two thin matrices whose product stands in for the update you wanted. At rank 8 you train a few hundred thousand numbers per layer instead of millions. QLoRA goes further and loads the frozen base model in 4-bit, so the part you are not training barely uses memory.

Quantization is the other half. Weights are normally 16-bit floats. You can store them in fewer bits and the model keeps working, because it tolerates the rounding noise surprisingly well. On a CPU this helps twice over, since moving less data through memory is most of what makes inference faster. One thing worth keeping straight: the 4-bit used during training (bitsandbytes, in memory) and the 4-bit used for serving (a GGUF file read by llama.cpp) are different tools for different stages.

## The data

The database is a small synthetic company: departments, employees, products, and sales. Five departments, 13 employees, 6 products, 75 sales rows. It is deliberately tiny, but it has enough shape to be interesting. An employee's manager is another employee, so questions can need a self-join. A sale links to both an employee and a product. There are dates to filter and amounts to aggregate.

Training is 50 hand-written question and SQL pairs, with 12 more held out for testing. Every prompt carries the full schema as text, so the model is not memorizing the schema, it is learning to read one and follow it. Fifty examples is small, and the numbers further down should be read with that in mind. It works here because the task is narrow, the base model already knows SQL, and all I am really teaching it is this schema's conventions and a clean output style.

## Training, and what the numbers said

I trained on a 6 GB laptop GPU, an RTX 4050. Phi-3-mini loaded in 4-bit, LoRA at rank 8 and alpha 16, three epochs. The run took just over three minutes and never crossed 5.2 GB of memory, so it sat well inside the card.

The line I cared about from the log was this:

> trainable params: 4,456,448 || all params: 3,825,536,000 || trainable: 0.1165%

A little over a tenth of a percent of the model was actually being trained. Eval loss fell from 0.59 to about 0.06 across the three epochs, and token accuracy on the held-out set climbed from 84% to 98.4%.

| Epoch | Train loss | Eval loss | Token accuracy |
| --- | --- | --- | --- |
| 0.4 | 0.63 | 0.59 | 84% |
| 1.2 | 0.26 | 0.22 | 95% |
| 2.0 | 0.05 | 0.07 | 98% |
| 3.0 | 0.05 | 0.06 | 98% |

Two things stand out. Training and validation loss fall together and stay close, so there is no real overfitting despite the small dataset. And the curve flattens after about the second epoch, which means the third one is mostly polish and early stopping would have been fine.

## A surprise: only two of the layers actually trained

This is the part I find most interesting, and it is the kind of thing you only catch if you go looking.

I had told LoRA to adapt seven projection types in every layer: the four attention projections (query, key, value, output) and the three in the MLP (gate, up, down). That is the standard "adapt everything" choice. But Phi-3 fuses some of these for efficiency. Query, key, and value live in one combined matrix, and gate and up live in another. PEFT matches the layers you name as strings, and the names for the separate query, key, value, gate, and up projections simply do not exist in this model. Five of the seven matched nothing.

I only noticed when I opened the saved adapter and counted the tensors. Two modules had been trained: the attention output projection and the MLP down projection. That was the whole 0.12%.

The honest read is mixed. It still worked, which is the genuinely surprising part, since adapting only the two output projections was enough to reach 98% token accuracy. But it also left something on the table, because adapting the attention and gating directly would likely have helped the harder cases. The takeaway is small and a little humbling: the layer names you hand to LoRA are matched against one specific implementation, and a config that looks thorough can quietly do a fraction of what you meant. Now I check the saved tensors instead of trusting the config.

## Shrinking it to run on a CPU

To serve this without a GPU, I merged the adapter back into the base weights, converted to a 16-bit GGUF, and quantized that down with llama.cpp. The sizes:

| Format | Size | vs 16-bit |
| --- | --- | --- |
| 16-bit, merged | 7.64 GB | baseline |
| Q5_K_M | 2.76 GB | 64% smaller |
| Q4_K_M | 2.40 GB | 69% smaller |

One detail worth knowing: a "4-bit" K-quant is not actually 4 bits per weight. It keeps some sensitive tensors, like the embeddings, at higher precision, so it averages closer to 5. The file is smaller than a naive four-bits-times-parameters guess, but not as small as that math suggests. I went with the 4-bit version. It is the smallest, and as the evaluation showed, it gave up nothing against the 5-bit one on this task.

## Did the fine-tuning actually help

The right way to grade a Text-to-SQL model is execution accuracy: run the query it writes and check whether the rows come back correct. String-matching against a reference query is a poor metric, because there are many correct ways to write the same query.

On the 12 held-out questions, the fine-tuned model got 9 right by execution match, 75%, and every one of its queries was valid runnable SQL. The base Phi-3, with the same prompt and the same quantization, got 5, about 42%. So fine-tuning roughly doubled the rate of correct answers. The base model could already write valid SQL; what the fine-tuning added was writing the right SQL for this schema. Here is one it got cleanly:

```sql
-- "Total revenue per employee, highest first"
SELECT employees.name, SUM(sales.amount) AS total_revenue
FROM employees
JOIN sales ON employees.id = sales.employee_id
GROUP BY employees.name
ORDER BY total_revenue DESC;
```

The 75% is also a little hard on it. Two of the three misses were cases where it returned a reasonable answer with a different column selection than the reference, for instance selecting just the product name where the reference selected every column. Only one was a real mistake, a date filter with a malformed format string that quietly returned nothing. Counting "did the query answer the question," it is closer to 11 out of 12.

On the quantization question, the 4-bit and 5-bit models scored the same, 75% and 100% valid SQL, with the 4-bit being smaller and faster. So 4-bit cost nothing here. The only way to know that, though, was to actually run the eval. Quantization does not always come free.

## Getting it onto a free server was its own project

The model working in a notebook was maybe half the job. Getting llama.cpp to load and run on a free Hugging Face CPU Space took four separate fixes, and each was a different layer of the stack. I am writing them down because they are generic to CPU deployment, not specific to this project.

First, the prebuilt llama.cpp Python wheels. The CPU wheel index ships musl-linked builds for every version recent enough to support Phi-3, and the Space image is glibc Debian, so the library refused to load with a missing-C-library error. The only glibc wheels on that index are old enough to predate Phi-3 support. The fix was to stop using prebuilt wheels and build from source against the image's own glibc.

Second, a segfault that only showed up at inference time. An older pinned version loaded the model fine and then killed the whole process the moment it tried to generate, because its bundled llama.cpp was too old for a GGUF built by a much newer one. I reproduced it locally, found a version that worked, and pinned that.

Third, the build ran out of memory. Compiling llama.cpp with the default parallelism got the build killed on the runner. Compiling one file at a time fixed it, at the cost of a slower build.

Fourth, it was slow for a dull reason. The code set the thread count to the host's core count, but the container only has two virtual CPUs, so it was running sixteen threads on two cores and fighting itself. Capping the thread count to the actual CPUs was a real speedup.

None of these are deep on their own, but together they were the gap between "works on my machine" and "works for someone clicking a link."

## Where it stands

The demo runs the quantized model on a free two-CPU Space. It is genuinely slow, a couple of minutes per query, and almost all of that is the model rereading the schema and writing tokens one at a time without a GPU. The database answers in about a millisecond; the size of the data has nothing to do with the wait. A paid CPU or any GPU would bring it down to a second or two, but the point of the demo is that it runs at all, for free, and gets the SQL right.

If I took this further, the obvious next steps are a larger and more varied dataset, targeting the fused attention and MLP layers correctly so the whole model gets adapted, and turning on prompt caching so the fixed schema is not reprocessed on every query.

## A note on the numbers

Everything I called measured here comes from the project's own logs and files: the training log, the trainer state, llama.cpp's benchmark and server output, the sizes of files on disk, and a small harness that runs each generated query against the real database. The one figure that is an estimate is the memory that full fine-tuning would have needed, which is arithmetic rather than something I ran, because it does not fit.