scripts/CONVERT_GGUF.md

# Convert the fine-tuned model to GGUF (Q4_K_M) — run on your laptop

One-time process. Produces `phi3-text-to-sql-Q4_K_M.gguf` (~2.3 GB), which you
then push to a Hugging Face model repo. The CPU Space downloads it at startup.

Prerequisites: your trained adapter at `models/phi3-text-to-sql-adapter`, Python
env with `transformers`, `peft`, `torch` (your existing `.venv`), and `git`.

---

## Step 1 — Merge the adapter into the base model

```bash
python scripts/merge_adapter.py
```

Output: `models/phi3-text-to-sql-merged/` (fp16, ~7.6 GB).

## Step 2 — Get llama.cpp (for the converter + quantizer)

```bash
git clone https://github.com/ggerganov/llama.cpp
pip install -r llama.cpp/requirements.txt
```

## Step 3 — Convert merged HF model → GGUF (F16)

```bash
python llama.cpp/convert_hf_to_gguf.py models/phi3-text-to-sql-merged \
    --outfile models/phi3-text-to-sql-f16.gguf \
    --outtype f16
```

## Step 4 — Quantize F16 → Q4_K_M

You need the `llama-quantize` binary. Two options:

**Option A — prebuilt binary (easiest on Windows):**
Download the latest llama.cpp release for your OS from
https://github.com/ggerganov/llama.cpp/releases (it includes `llama-quantize`
/ `llama-quantize.exe`), then:

```bash
# Linux/macOS
./llama-quantize models/phi3-text-to-sql-f16.gguf \
    models/phi3-text-to-sql-Q4_K_M.gguf Q4_K_M
```
```powershell
# Windows
.\llama-quantize.exe models\phi3-text-to-sql-merged\..\phi3-text-to-sql-f16.gguf `
    models\phi3-text-to-sql-Q4_K_M.gguf Q4_K_M
```

**Option B — build from source:**
```bash
cd llama.cpp && cmake -B build && cmake --build build --config Release -j
# binary at: build/bin/llama-quantize
./build/bin/llama-quantize ../models/phi3-text-to-sql-f16.gguf \
    ../models/phi3-text-to-sql-Q4_K_M.gguf Q4_K_M
```

Result: `models/phi3-text-to-sql-Q4_K_M.gguf` (~2.3 GB).

## Step 5 — Validate quality BEFORE deploying

```bash
python scripts/validate_gguf.py --gguf models/phi3-text-to-sql-Q4_K_M.gguf
```

Acceptance bar: ≥ 95% execution-equivalent on `data/test_dataset.jsonl`.
If it fails, redo Step 4 with `Q5_K_M` (larger/slower, higher fidelity) and
re-validate.

## Step 6 — Push the GGUF to Hugging Face

The Space pulls from repo `Bhuvandesai/phi3-text-to-sql-gguf`, file
`phi3-text-to-sql-Q4_K_M.gguf` (override via `GGUF_REPO_ID` / `GGUF_FILENAME`
env vars on the Space). Create the repo and upload:

```bash
huggingface-cli repo create phi3-text-to-sql-gguf --type model
huggingface-cli upload Bhuvandesai/phi3-text-to-sql-gguf \
    models/phi3-text-to-sql-Q4_K_M.gguf phi3-text-to-sql-Q4_K_M.gguf
```

(If you used Q5_K_M, set `GGUF_FILENAME=phi3-text-to-sql-Q5_K_M.gguf` in the
Space settings and upload that file instead.)

## Step 7 — Deploy

Commit and push the repo. The Dockerfile installs llama-cpp-python and the slim
deps; on boot `src/inference.py` downloads the GGUF and serves on CPU.

---

### Notes
- Do **not** commit the multi-GB `.gguf` into the Space git repo. Host it on the
  Hub (Step 6) so it is pulled at runtime and cached. If you ever want it bundled,
  the loader also accepts a local copy at `models/<GGUF_FILENAME>`.
- `models/phi3-text-to-sql-merged/`, `*.gguf`, and `llama.cpp/` are intermediate
  artifacts — keep them out of git (see `.gitignore`).