docs: comprehensive training playbook — 14 bugs documented with fixes

- 14 errors catalogued with symptoms, root causes, and fixes
- Verified results section (3/3 queries pass)
- Post-training checklist with working commands
- Reproducible build sequence updated

docs/TRAINING_PLAYBOOK.md

@@ -93,74 +93,221 @@ modal app list
 - Training converged fast (loss 0.1008 by step 72/243)
 - QLoRA at 0.53% trainable params = cheap, fast, effective
 
-### Key Fix: fn.spawn() Fire-and-Forget
-- **Problem:** 5 training runs crashed because `modal run` keeps a client connection open. When the local CLI times out or disconnects, Modal cancels the running function.
-- **Solution:** Switch to `modal deploy` + `fn.spawn()` which fires the function with no client connection to kill.
-- **Result:** Training survived past step 72 where all previous runs crashed.
+### Key Fix: Fire-and-Forget via deploy + spawn
+- **Problem:** 5 training runs crashed because `modal run` keeps a client connection open. When the local CLI times out or disconnects, Modal cancels the running function. `--detach` didn't help because cancellation arrives before detach.
+- **Solution:** `modal deploy` creates a persistent app with zero client connection. Call `fn.spawn()` from Python — truly fire-and-forget.
+- **Result:** Training completed all 3 epochs (243 steps), loss converged to 0.07.
+- **EXACT COMMANDS USED:**
+```bash
+modal deploy modal_train/modal_app.py
+python3 -c "
+import modal
+fn = modal.Function.from_name('kasualdad-lfed-train', 'run_full_pipeline')
+fn.spawn()
+"
 
 ---
 
-## 3. What Went Wrong (and Fixes)
-
-### Issue 1: 5 Consecutive Training Crashes
-- **Symptom:** Training started, ran ~20-30 steps, then silently stopped
-- **Root cause:** `modal run` maintains a gRPC connection to Modal. When the local terminal session exits (timeout, sleep, Ctrl+C), Modal cancels the remote function.
-- **Fix:** Use `modal deploy` to create a persistent app, then call `fn.spawn()` from a Modal function that has no client connection. The deployed app stays alive independently.
-- **Lesson:** NEVER use `modal run` for long-running GPU work. Always deploy + spawn.
+## 3. What Went Wrong — Complete Error Catalog
 
-### Issue 2: Flash Attention 2 Broken on Modal
-- **Symptom:** Warning "Your Flash Attention 2 installation seems to be broken. Using Xformers instead."
-- **Impact:** Minimal — Xformers works fine, no perf change observed
-- **Fix:** Not needed for hackathon. For production, pin specific flash-attn + CUDA versions.
-
-### Issue 3: Only 1,289 Pairs (vs 2,000 Target)
-- **Symptom:** Generator hit dedup ceiling before reaching 2,000
-- **Root cause:** 32 templates × limited parameter combinations (5 schools, 4 years) = ~1,300 unique questions max before repeats
-- **Fix needed:** Add more templates, more parameter variety, or augment with LLM-generated rephrasings (see Expansion Plan)
-
-### Issue 4: Narrow Schema Coverage
-- **Symptom:** Training data only covers 2 tables (enrollment, attendance)
-- **Impact:** Model can only answer attendance/enrollment questions — can't handle grades, discipline, demographics, wellbeing
-- **Fix needed:** Expand schema + training pairs (see Expansion Plan)
+### CRITICAL: Training Client Disconnects (⚠️ 5 runs crashed)
+- **Symptom:** Training ran ~20-130 steps, then stopped with `Received a cancellation signal`
+- **Root cause:** `modal run` keeps a gRPC connection. When local terminal exits (timeout, sleep, Ctrl+C), Modal cancels the remote function. `--detach` delay-implies detachment — the cancellation signal arrives before detach takes effect.
+- **Fix:** `modal deploy` + call `fn.spawn()` from Python. Spawn fires with no client connection — the function runs to completion independently.
+- **Commands:**
+```bash
+modal deploy modal_train/modal_app.py
+python3 -c "
+import modal
+fn = modal.Function.from_name('kasualdad-lfed-train', 'export_and_push')
+fn.spawn()
+"
+```
+- **Lesson for future training runs:** NEVER use `modal run` for GPU work >60 seconds. ALWAYS deploy + spawn.
+
+### CRITICAL: Files Not Available in Container (2 crashes)
+- **Symptom:** `FileNotFoundError: '/root/generate_synthetic.py'` — only `modal_app.py` was uploaded.
+- **Root cause:** Modal only uploads the entry-point file. Sibling scripts must be explicitly mounted.
+- **Fix:** Add `.add_local_dir()` to the Modal image definition:
+```python
+train_image = modal.Image.debian_slim(...)
+    .add_local_dir(Path(__file__).parent, remote_path="/root")
+```
+- **Failed attempts:** `modal.Mount` (deprecated, doesn't exist in v1.4.3), `condition=` kwarg (not supported on `add_local_dir`)
+
+### CRITICAL: Cross-Device Link Error
+- **Symptom:** `OSError: [Errno 18] Invalid cross-device link: '/root/train.jsonl' -> '/data/train.jsonl'`
+- **Root cause:** `Path.rename()` / `os.rename()` fails across mount points (root mount ≠ volume mount)
+- **Fix:** Use `shutil.move()` instead of `.rename()`
+
+### CRITICAL: Pickle Error on Model Save (2 crashes)
+- **Symptom:** `PicklingError: Can't pickle <class 'trl.trainer.sft_config.SFTConfig'>`
+- **Root cause:** SFTTrainer calls `save_model()` which tries to pickle training args. SFTConfig from nested module can't be pickled.
+- **Fix (3 layers):**
+  1. `save_strategy="no"` — prevent auto-saves during training
+  2. `try/except` around `trainer.train()` to catch pickle errors
+  3. Manual save: `model.save_pretrained()` directly (bypasses trainer)
+
+### CRITICAL: CUDA Out of Memory During Merge/Export (3 crashes)
+- **Symptom:** `torch.OutOfMemoryError: Tried to allocate 14.23 GiB` during `load_adapter()`
+- **Root cause:** Training leaves model in GPU. Loading 16-bit base model for merge needs 14GB, exceeding A10G 22GB when combined.
+- **Fix:** 
+  1. Free GPU after training: `del model; gc.collect(); torch.cuda.empty_cache()`
+  2. Run `export_and_push` via `.remote()` not `.local()` — fresh container with clean GPU
+- **Failed attempts:** `load_in_4bit=True` + `merge_and_unload()` (NotImplementedError on quantized models)
+
+### CRITICAL: merge_and_unload() Not Implemented
+- **Symptom:** `NotImplementedError` when calling `merge_and_unload()` on quantized PEFT model
+- **Fix:** Load base model in FP16 via `AutoModelForCausalLM`, then apply adapter via `PeftModel.from_pretrained(base_model, adapter_path)`, THEN merge. The key insight: load base in FP16 first, apply adapter separately.
+```python
+base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
+model = PeftModel.from_pretrained(base_model, str(LORA_DIR))
+model = model.merge_and_unload()
+```
 
-### Issue 5: Memory Issue During Merge/Export
-- **Symptom:** User reported memory issues on Modal after training reached 30%
-- **Status:** Being investigated during current training run
-- **Likely cause:** Merging LoRA back into 16-bit base model requires loading full model (14GB+), which may exceed A10G 24GB when combined with training checkpoint memory
-- **Potential fix:** Free GPU memory between train and merge steps (already implemented in train.py lines 201-206), or use separate A10G container for merge step
+### GGUF Conversion: Missing Module (3 crashes)
+- **Symptom:** `No module named 'llama_cpp.convert'`
+- **Root cause:** llama-cpp-python doesn't have a built-in converter
+- **Fix:** Clone llama.cpp repo + install `gguf` package. Use `convert_hf_to_gguf.py` from the repo.
+- **Failed attempts:** Downloading single converter file (needs companion modules), `pip install gguf` in image (not picked up by cached image)
+
+### Quantization: llama-quantize Missing (2 attempts)
+- **Symptom:** `llama-quantize not found, using FP16 GGUF` → 15.2 GB model
+- **Root cause:** llama.cpp quantize binary not built
+- **Fix:** `llama_quantize()` from llama-cpp-python (v0.3.26+ has it built-in)
+```python
+from llama_cpp import llama_quantize
+llama_quantize(input_path=f16_gguf, output_path=q4_gguf, output_type="q4_k_m")
+```
+- **Failed attempts:** cmake build (dependency issues, `-j$(nproc)` shell expansion fails with subprocess)
+
+### HF Repo Push: Namespace Permission Error
+- **Symptom:** `403 Forbidden: You don't have the rights to create a model under the namespace "kasualdad"`
+- **Root cause:** HF API is case-sensitive. `kasualdad` ≠ `Kasualdad`. The token belongs to user `Kasualdad`.
+- **Fix:** Use correct case: `HF_USERNAME = "Kasualdad"`
+
+### Fine-Tuned Model Outputs Nothing (0 chars)
+- **Symptom:** Model loads but generates 0 tokens for every query
+- **Root cause:** Training used Qwen2.5 chat template (`<|im_start|>system...<|im_end|>`), but inference sent plain text. Model doesn't recognize the format.
+- **Fix:** Update `build_prompt()` in `prompts.py` to use Qwen2.5 chat template:
+```python
+prompt = (
+    f"<|im_start|>system\n{system_prompt}<|im_end|>\n"
+    f"<|im_start|>user\nQuestion: {question}<|im_end|>\n"
+    f"<|im_start|>assistant\n"
+)
+```
+- **Also:** Add `<|im_end|>` and `<|im_start|>` to stop sequences in `model_inference.py`
+
+### CUDA Wheel on CPU Spaces
+- **Symptom:** `libcuda.so.1: cannot open shared object file` on CPU Space
+- **Root cause:** `requirements.txt` had `--extra-index-url cu121` which installs CUDA-linked llama-cpp-python
+- **Fix:** Remove the CUDA wheel index. Standard PyPI wheel works on CPU + GPU.
+
+### Local Dev: Missing spaces Module
+- **Symptom:** `ModuleNotFoundError: No module named 'spaces'` when running locally
+- **Root cause:** `spaces` is an HF infrastructure-only package
+- **Fix:** Try/except import with no-op fallback:
+```python
+try:
+    import spaces
+    _gpu_decorator = spaces.GPU
+except ImportError:
+    _gpu_decorator = lambda fn: fn
+```
 
-### Issue 6: Template-Based Questions Feel Synthetic
-- **Symptom:** Questions are grammatically correct but lack the messiness of real admin questions
-- **Impact:** Model may struggle with typos, abbreviations, informal phrasing
-- **Fix:** Add LLM-augmented rephrasings (see Expansion Plan)
+### HF Space: Zero GPU Daily Limit
+- **Symptom:** "You've hit your daily Zero GPU limit"
+- **Root cause:** Free tier has limited daily GPU quota
+- **Workaround:** Switch Space hardware to CPU (model still works, just slower)
+- **Alternative:** $9/month PRO account for 8x quota
+
+### Volume Cache Serves Stale Files
+- **Symptom:** Code changes don't take effect, old errors repeat
+- **Root cause:** Modal volumes persist `.pyc` files from old runs. `import` picks up cached bytecode.
+- **Fix:** Clear module cache before importing:
+```python
+for key in list(sys.modules.keys()):
+    if "script_name" in key:
+        del sys.modules[key]
+importlib.reload(module)
+```
 
 ---
 
-## 4. Post-Training Checklist
-
-After training completes:
+## 4. Post-Training Checklist (VERIFIED WORKING)
 
 ```bash
 # 1. Verify GGUF pushed to Hub
-# Check https://huggingface.co/kasualdad/lfed-qwen2.5-coder-7b-sql-gguf
-
-# 2. Update model_inference.py to use fine-tuned model
-# Change REPO_ID and MODEL_FILE (lines ~103-107)
-
-# 3. Test locally
-python app.py
-# Ask: "How many students were chronically absent in 2023-2024?"
-
-# 4. Run tests
+# Check https://huggingface.co/build-small-hackathon/lfed-qwen2.5-coder-7b-sql-gguf
+from huggingface_hub import HfApi
+api = HfApi(token='hf_...')
+for f in api.list_repo_tree('build-small-hackathon/lfed-qwen2.5-coder-7b-sql-gguf'):
+    print(f'  {f.path} ({f.size/1e9:.2f} GB)')
+# Expected: lfed-qwen2.5-coder-7b-sql-Q4_K_M.gguf (4.68 GB)
+
+# 2. Update model_inference.py
+# Set lines 101-102:
+#   HF_REPO_ID = "build-small-hackathon/lfed-qwen2.5-coder-7b-sql-gguf"
+#   HF_MODEL_FILE = "lfed-qwen2.5-coder-7b-sql-Q4_K_M.gguf"
+
+# 3. Fix prompts.py: Qwen2.5 chat template (see Issue "Outputs Nothing")
+#   MUST use <|im_start|>system/user/assistant<|im_end|> format
+
+# 4. Fix model_inference.py stop sequences:
+#   STOP_SEQUENCES = ["\n\n", "Question:", "User:", "<|im_end|>", "<|im_start|>"]
+
+# 5. Fix app.py: spaces.GPU made optional for local dev
+#   try/except ImportError with lambda:fn fallback
+
+# 6. Test locally
+cd Kasualdad_LFED && source .venv/bin/activate
+python -c "
+from model_inference import load_model, generate_sql
+from data_engine import create_session, execute_safe
+llm = load_model()  # downloads Q4_K_M from Hub (~3s on Mac)
+raw, _ = generate_sql('How many students were chronically absent in 2023-2024?', llm=llm)
+conn = create_session()
+sql, df = execute_safe(conn, raw)
+print(df)  # Should show: chronic_count = 435
+"
+
+# 7. Run tests
 pytest tests/ -v
 
-# 5. Deploy to HF Space
-git push space main
+# 8. Commit + push to Space
+git add -A && git commit -m "feat: fine-tuned Q4_K_M model" && git push space main
+
+# 9. Verify Space
+# Open https://huggingface.co/spaces/build-small-hackathon/Kasualdad_LFED
+# Click first example query — should return 435
 ```
 
 ---
 
-## 5. Reproducing from Scratch
+## 6. Verified Results (2026-06-08)
+
+### Training
+| Metric | Value |
+|---|---|
+| Model | Qwen2.5-Coder-7B-Instruct → QLoRA fine-tuned |
+| Training data | 1,289 NL→SQL pairs, 32 templates |
+| Epochs | 3 (243 steps) |
+| Final loss | 0.07 (converged from 2.6) |
+| Training time | ~7 minutes on A10G |
+| GGUF output | Q4_K_M, 4.68 GB |
+
+### Inference (3/3 test queries pass)
+| Query | SQL | Result |
+|---|---|---|
+| "How many students were chronically absent in 2023-2024?" | `SELECT COUNT(*) ... WHERE is_chronically_absent = TRUE` | **435** ✅ |
+| "Show total enrollment per school for 2024-2025, sorted highest first." | `SELECT school_name, SUM(student_count) ... ORDER BY total_enrollment DESC` | Correct ranking ✅ |
+| "What percentage of students at Lincoln Elementary were chronically absent?" | `ROUND(COUNT(CASE WHEN ...) * 100.0 / COUNT(*), 1)` | **13.7%** ✅ |
+
+### Deployed URLs
+- **HF Space:** https://huggingface.co/spaces/build-small-hackathon/Kasualdad_LFED
+- **Model repo:** https://huggingface.co/build-small-hackathon/lfed-qwen2.5-coder-7b-sql-gguf
+- **Modal app:** https://modal.com/apps/flucido/main/deployed/kasualdad-lfed-train
 
 ```bash
 # 1. Clone