scripts/CONVERT_GGUF.md

Convert the fine-tuned model to GGUF (Q4_K_M) — run on your laptop

One-time process. Produces phi3-text-to-sql-Q4_K_M.gguf (~2.3 GB), which you then push to a Hugging Face model repo. The CPU Space downloads it at startup.

Prerequisites: your trained adapter at models/phi3-text-to-sql-adapter, Python env with transformers, peft, torch (your existing .venv), and git.

---

Step 1 — Merge the adapter into the base model

python scripts/merge_adapter.py

Output: models/phi3-text-to-sql-merged/ (fp16, ~7.6 GB).

Step 2 — Get llama.cpp (for the converter + quantizer)

git clone https://github.com/ggerganov/llama.cpp
pip install -r llama.cpp/requirements.txt

Step 3 — Convert merged HF model → GGUF (F16)

python llama.cpp/convert_hf_to_gguf.py models/phi3-text-to-sql-merged \
    --outfile models/phi3-text-to-sql-f16.gguf \
    --outtype f16

Step 4 — Quantize F16 → Q4_K_M

You need the llama-quantize binary. Two options:

Option A — prebuilt binary (easiest on Windows): Download the latest llama.cpp release for your OS from https://github.com/ggerganov/llama.cpp/releases (it includes llama-quantize / llama-quantize.exe), then:

# Linux/macOS
./llama-quantize models/phi3-text-to-sql-f16.gguf \
    models/phi3-text-to-sql-Q4_K_M.gguf Q4_K_M

# Windows
.\llama-quantize.exe models\phi3-text-to-sql-merged\..\phi3-text-to-sql-f16.gguf `
    models\phi3-text-to-sql-Q4_K_M.gguf Q4_K_M

Option B — build from source:

cd llama.cpp && cmake -B build && cmake --build build --config Release -j
# binary at: build/bin/llama-quantize
./build/bin/llama-quantize ../models/phi3-text-to-sql-f16.gguf \
    ../models/phi3-text-to-sql-Q4_K_M.gguf Q4_K_M

Result: models/phi3-text-to-sql-Q4_K_M.gguf (~2.3 GB).

Step 5 — Validate quality BEFORE deploying

python scripts/validate_gguf.py --gguf models/phi3-text-to-sql-Q4_K_M.gguf

Acceptance bar: ≥ 95% execution-equivalent on data/test_dataset.jsonl. If it fails, redo Step 4 with Q5_K_M (larger/slower, higher fidelity) and re-validate.

Step 6 — Push the GGUF to Hugging Face

The Space pulls from repo Bhuvandesai/phi3-text-to-sql-gguf, file phi3-text-to-sql-Q4_K_M.gguf (override via GGUF_REPO_ID / GGUF_FILENAME env vars on the Space). Create the repo and upload:

huggingface-cli repo create phi3-text-to-sql-gguf --type model
huggingface-cli upload Bhuvandesai/phi3-text-to-sql-gguf \
    models/phi3-text-to-sql-Q4_K_M.gguf phi3-text-to-sql-Q4_K_M.gguf

(If you used Q5_K_M, set GGUF_FILENAME=phi3-text-to-sql-Q5_K_M.gguf in the Space settings and upload that file instead.)

Step 7 — Deploy

Commit and push the repo. The Dockerfile installs llama-cpp-python and the slim deps; on boot src/inference.py downloads the GGUF and serves on CPU.

---

Notes

• Do not commit the multi-GB .gguf into the Space git repo. Host it on the Hub (Step 6) so it is pulled at runtime and cached. If you ever want it bundled, the loader also accepts a local copy at models/<GGUF_FILENAME>.
• models/phi3-text-to-sql-merged/, *.gguf, and llama.cpp/ are intermediate artifacts — keep them out of git (see .gitignore).