docs/eval-roadmap.md

# Eval roadmap — improving the scheduling fine-tune

How we measure and improve `ParetoOptimal/gemma-4-cal-gguf` (the fine-tuned
Gemma-4-31B that turns chat/images into a calendar `ActionPlan`). The eval is
**task-specific** — generic LLM benchmarks (MMLU etc.) don't apply.

Harness: `training/eval.py` (scores), `training/gen_eval.py` + `training/data/eval.jsonl`
(28 held-out examples, disjoint from `dataset.jsonl`), `training/modal_eval.py`
(serves the GGUF on the same `llama-server` the Space uses, then scores).

## Baseline scores (Q4_K_M, n=28, 2026-06-09)

| Metric | Score |
| --- | --- |
| schema validity | 1.00 |
| no-event accuracy | 1.00 |
| clarification recall | 1.00 |
| end-time exact | 1.00 |
| event precision | 0.85 |
| **event recall (start-exact)** | **0.77** |
| event F1 | 0.81 |
| title similarity | 0.87 |

Discipline (never invents events, always asks when ambiguous) is perfect; all 9
relative-date cases passed. The gap is **exact start datetime** on a few
explicit far-future dates (misses: `e02`, `e05`, `e06`, `e15`, one leg of `m02`).

## The 3 steps

### 1. Diagnose the 5 misses (cheap)
Enhance `eval.py` to dump the model's actual `start`/`title` for mismatched events,
then one re-run shows whether they're date-shift, time/AM-PM, or wrong-year errors —
which tells us exactly what training data to add. (~one A100 eval run; the GGUF is
cached in the Modal Volume, so it's fast.)

### 2. Baseline comparison (the "Well-Tuned" proof)
Run `modal run training/modal_eval.py --model-hf-repo unsloth/gemma-4-31B-it-GGUF`
to score **stock** Gemma-4-31B on the same set. If the fine-tune's discipline
(no-event 1.0, clarification 1.0) and datetime recall beat stock, that's concrete
evidence the fine-tune helps. (Separate ~18 GB model download + A100 time.)

### 3. Close the gap
Add ~15–20 explicit-date examples (especially next-month dates and times) to
`training/data/dataset.jsonl`, re-train on Modal (`training/modal_train.py`),
re-eval — and watch start-exact recall move.

## Results log

### Step 1 — diagnosis (2026-06-09)
The mismatch dump showed the misses are **not** a reasoning failure. 3 of 5 are the
same bug — a dropped year digit, **"206" instead of "2026"** — on next-month dates
(month/day/time all correct):

```
[e02] gold 2026-10-06T15:30  pred 206-10-06T15:30
[e05] gold 2026-10-01T08:15  pred 206-10-01T08:15
[e15] gold 2026-10-08T19:00  pred 206-10-08T19:00
[e06] gold 2026-09-28T09:00  pred []                 (abstained)
[m02] Standup + Sprint demo  pred Standup only       (dropped 2nd leg)
```

Fix indicated: more far-future explicit-date examples reinforcing 4-digit years
(+ multi-event 2nd legs). → Step 3.

### Step 2 — baseline vs fine-tune (2026-06-09, n=28, Q4_K_M)

| Metric | Stock `gemma-4-31B-it-GGUF` | Fine-tune `gemma-4-cal-gguf` |
| --- | --- | --- |
| schema validity | 1.00 | 1.00 |
| event precision | **1.00** | 0.85 |
| start-exact recall | **0.955** | 0.773 |
| event F1 | **0.977** | 0.81 |
| end-exact | 1.00 | 1.00 |
| no-event accuracy | 1.00 | 1.00 |
| clarification recall | 0.75 | **1.00** |

**Honest read:** stock Gemma-4-31B is already strong at this extraction and *beats*
the current fine-tune on datetime recall — the "206" bug is a fine-tune regression.
The fine-tune's only clear win is **clarification discipline** (asks when a thread is
"date TBD"; stock missed `q04`). As-is, the fine-tune is **not** justified on
extraction. Step 3 must fix the year regression and clear baseline's 0.955 recall
while keeping clarification at 1.00 — otherwise the better play is stock + the
fine-tune's clarification behavior via prompting.

### Step 3 — after gap-closing retrain (2026-06-09) — REGRESSED
Dataset grown 69 → 87 (+18 Oct–Dec 2026 explicit-date examples, disjoint from eval),
same 2-epoch recipe, re-quantized to Q4_K_M and republished. Re-eval (n=28):

| Metric | Stock 31B | Fine-tune v1 (69) | **Fine-tune v2 (87, retrained)** |
| --- | --- | --- | --- |
| schema validity | 1.00 | 1.00 | **0.75** |
| event precision | 1.00 | 0.85 | **0.476** |
| start-exact recall | 0.955 | 0.773 | **0.455** |
| event F1 | 0.977 | 0.81 | **0.465** |
| end-exact | 1.00 | 1.00 | 1.00 |
| no-event accuracy | 1.00 | 1.00 | 1.00 |
| clarification recall | 0.75 | 1.00 | **0.75** |

**The naive retrain made it worse, not better.** New failure modes: unparseable/empty
JSON (validity 1.0→0.75), duplicate events, hallucinated "Drive to …" events,
transposed/garbage years (`2062`, `2062-15:00:00`), and previously-passing relative
dates now empty. Cause: overfitting — 18 of 87 examples were near-identical far-future
templates, biasing a tiny dataset and degrading general formatting/extraction.

## Conclusions & recommendation

1. **Stock Gemma-4-31B is already strong** at this extraction (F1 0.98). The only
   thing fine-tuning reliably *added* was clarification discipline (v1: 1.00 vs stock
   0.75) — and even that was lost in v2.
2. **Tiny-dataset SFT is fragile here.** v1 (69 ex) underperformed stock on dates;
   v2 (87 ex) regressed hard. More data of the *same shape* hurt.
3. **Recommended path** (pick one):
   - **Ship stock + prompt for clarification** — simplest; recover the one real win
     without the regressions. (Lowest risk.)
   - **If keeping a fine-tune:** rebuild the dataset much larger and *diverse* (not
     template-heavy), drop to ~1 epoch with regularization, and **gate every retrain
     on this eval** (only publish if it beats the current best). Consider a higher
     quant (Q5/Q6) to rule out the `"206"`/`2062` digit corruption being quant-driven.
4. **Action — revert the live model.** v2 (worse) overwrote v1 in
   `ParetoOptimal/gemma-4-cal-gguf`. Restore v1 (the better fine-tune) or point the
   Space back at stock `unsloth/gemma-4-31B-it-GGUF` until a fine-tune *beats* the
   eval baseline.

**Bottom line: the eval did its job — it caught a regression before it reached users,
and it says the current fine-tune is not yet worth shipping over stock.**

## Follow-up (2026-06-09)

### Live model restored to v1
v2 (regressed) was rolled back: `gemma-cal-Q4_K_M.gguf` in the repo was restored to the
v1 LFS object via a server-side `CommitOperationCopy` (no transfer, no GPU). Production
serves the better v1 again.

### Dataset rebuilt larger + more diverse (69 → 122)
Added a diversity batch (`gen_new_seeds.MORE_SEEDS3`): varied date/time formats
(`10/15`, "the 3rd", "half past 7", "0900", "noon", "midnight"), reschedules,
cancellations, recurring, all-day, deadlines (EOD/midnight), past & hypothetical
(must NOT schedule), richer no-event & clarify, and varied image sources (ticket,
invite screenshot, notice). Goal: counter the template-heavy skew that overfit v2.
Verified valid + disjoint from `eval.jsonl`.

### Eval-gating is now the publishing process
**No retrain publishes unless it beats the eval.** `training/gated_retrain.py`:
1. retrain on Modal → upload to a **staging** filename (`gemma-cal-staging-Q4_K_M.gguf`)
   in the repo (production file untouched; mmproj skipped — `--skip-mmproj`);
2. eval the staging file (`modal_eval.py --model-file …`);
3. gate: `schema_validity ≥ 0.95`, `event_f1 ≥ 0.81`, `start-exact recall ≥ 0.773`
   (defaults = the current best, v1) — tune via `--gate-f1/--gate-recall`;
4. **PASS** → promote staging → production via server-side `CommitOperationCopy` (free);
   **FAIL** → delete staging, production unchanged.

Run: `python training/gated_retrain.py [--epochs 1 --gate-f1 … --gate-recall …]`.

### Step 4 — first eval-gated retrain (122 ex, 1 epoch) — GATE FAILED ✅ (protected prod)
The retrain scored **worse** than every prior version and the gate refused to publish:

| Metric | Stock | v1 (live) | v3 staging (122, 1ep) |
| --- | --- | --- | --- |
| schema validity | 1.00 | 1.00 | **0.46** |
| event F1 | 0.977 | 0.81 | **0.214** |
| start-exact recall | 0.955 | 0.773 | **0.136** |
| no-event accuracy | 1.00 | 1.00 | 1.00 |
| clarification recall | 0.75 | 1.00 | 1.00 |

>½ of outputs were unparseable; extraction collapsed. **Gate: FAIL → staging deleted,
production unchanged (still v1).** The gate worked exactly as intended.

## Verdict (after 3 fine-tune attempts)
All three fine-tunes — v1 (69 ex / 2 ep), v2 (87 / 2 ep), v3 (122 / 1 ep) — **underperform
stock Gemma-4-31B**, and the larger runs broke JSON validity. Only the safety behaviors
(no-event, clarification) survive fine-tuning; extraction degrades. **QLoRA-on-31B-Q4 here
is fragile and not worth shipping over stock.** Recommended: serve **stock
`unsloth/gemma-4-31B-it-GGUF`** and recover the one fine-tune win (clarification) via the
prompt. Keep v1 as the published fine-tune for the "Well-Tuned" artifact, but don't route
production extraction through it. Revisit fine-tuning only with a substantially larger, more
varied dataset and a recipe that holds schema validity at 1.0 — gated, as now, on this eval.

## Step 5 — quantization-penalty test (2026-06-09): quant EXONERATED
Hypothesis: maybe Q4 quantization (the `"206"`/`2062` digit bug) was tanking the fine-tune.
Tested the SAME fine-tuned weights (`gemma-cal-f16.gguf`, v2/87-ex — best fp16 still on the
volume) at three precisions on the 28-example eval (`training/modal_quant_eval.py`):

| precision | schema validity | event F1 | start-exact recall |
| --- | --- | --- | --- |
| f16 (full) | 0.643 | 0.571 | 0.545 |
| Q8_0 | 0.679 | 0.565 | 0.591 |
| Q4_K_M | 0.75 | 0.465 | 0.455 |
| base (stock) | 1.00 | 0.977 | 0.955 |

**Quantization is not the cause.** At full fp16 the fine-tune still scores validity 0.64 / F1
0.57 — nowhere near base; validity is actually *lower* at f16 than Q4, so quant isn't breaking
the JSON. Precision buys only ~+0.1 F1/recall (Q4→Q8/f16), a fraction of the gap to base. The
degradation is the **SFT itself**, not the GGUF conversion. Step 2 (retrain at Q8 to beat base)
is **not pursued** — the gate would fail. (Caveat: v1's fp16 was overwritten, so this used v2;
a definitive v1 test needs a retrain, but the small quant lift makes a base-beating result
improbable.)

### Final recommendation
A higher quant won't make the fine-tune beat base, and an automation agent (e.g. `ml-intern`)
doesn't change the binding constraints (near-ceiling base; small data; SFT degrades
instruction-following). **Serve stock `unsloth/gemma-4-31B-it-GGUF`** and recover the
clarification behavior via the system prompt; keep v1 as the "Well-Tuned" artifact. Only
revisit fine-tuning with a substantially larger, real, diverse dataset + a validity-preserving
recipe (low LR, few steps), always gated on this eval.

## Real training data: SMCalFlow importer
`training/import_smcalflow.py` converts **SMCalFlow** (Microsoft Semantic Machines, **CC BY-SA
4.0**) calendar dialogues into our `ActionPlan` format. SMCalFlow encodes events as LISP
"dataflow" programs; the importer parses `CreatePreflightEventWrapper` turns, extracts
subject/start/location/attendees, and **resolves** date/time constructs (`Tomorrow`, `NextDOW`,
`MD`, `NumberPM`, `HourMinuteMilitary`, …) against a per-example reference `now` spread across
2026 — so relative dates become concrete, self-consistent targets (directly trains the failing
date/time skill, with varied 4-digit years). Conservative: only emits a row when a title AND an
explicit start time resolve (~7.5k usable turns from train+valid).

- Run: `python training/import_smcalflow.py --limit 2000 --heldout 200` → writes
  `training/data/smcalflow_train.jsonl` (+ `…_heldout.jsonl`). **Both are git-ignored** (CC BY-SA
  share-alike vs this repo's Apache-2.0 → we don't commit/redistribute the derived data; the
  importer code is ours) and **disjoint from `eval.jsonl`**.
- `train_qlora.py` now trains on `dataset.jsonl` **+** `smcalflow_train.jsonl` (when present).
  `gated_retrain.py` therefore trains on real data, and still **only publishes if it beats the
  gate** — so a bigger-but-worse model can't reach production.
- Attribution (required by CC BY-SA): *Semantic Machines et al., "Task-Oriented Dialogue as
  Dataflow Synthesis," TACL 2020.*

## Step 6 — eval-gated retrain on REAL data (2026-06-09): FAILED gate (worst yet)
Trained the 31B on **2,122 examples** (122 hand-authored + 2,000 real SMCalFlow), 1 epoch,
through `gated_retrain.py` with a beat-base gate (F1≥0.95, recall≥0.90). Result on the 28-ex eval:

| Metric | base | v1 (live) | real-data (2,122 ex) |
| --- | --- | --- | --- |
| schema validity | 1.00 | 1.00 | **0.107** |
| event F1 | 0.977 | 0.81 | **0.000** |
| start-exact recall | 0.955 | 0.773 | **0.000** |

~90% unparseable output, zero events extracted. **Gate FAIL → not promoted; production stays v1.**

### Verdict across 4 fine-tunes (now incl. real data)
Scores **monotonically worsen with more training/data**: v1 (69 synth, F1 0.81) → v2 (87, 0.465)
→ v3 (122, 0.214) → real (2,122, 0.0). This is no longer a *data* problem — **the SFT recipe
itself degrades the model**, and more data makes it worse. Most likely root cause to investigate
*if* fine-tuning is ever revisited: a **train/inference chat-template mismatch** — `train_qlora.py`
formats with Unsloth's `get_chat_template("gemma")` while `llama-server` serves with the GGUF's
own `--jinja` template; if these differ for Gemma-4, training optimizes a format the server never
uses, and the divergence compounds with more steps (exactly the monotonic decay seen). Other
suspects: LR too high (2e-4) / catastrophic forgetting on a near-ceiling base.

**Final, evidence-backed recommendation: serve stock `unsloth/gemma-4-31B-it-GGUF`** (best by far)
and recover clarification via the system prompt. Do NOT route production through any current
fine-tune. The eval-gate has now correctly rejected 2 bad retrains — keep it as the publish gate.

## Step 7 — recipe fix + raw-output probe (2026-06-09): training stack implicated, fine-tuning HALTED
Fixed the suspected train/serve chat-template mismatch (PR #54): Gemma-4's native
`chat_template.jinja` uses a NEW `<|turn>role … <turn|>` format (no `<start_of_turn>` at all),
while training forced unsloth's legacy "gemma" template. `train_qlora.py` now formats with the
tokenizer's native template (hard `<|turn>` assert), masks loss to the assistant turn, LR 5e-5.
Retrained on the 2,122-example set through the gate: **validity 0.0 — gate FAIL** (production
stays v1, third bad retrain rejected).

Diagnostics that pinpointed the cause:
- **GGUF template check (CPU, ~free):** our exported staging GGUF embeds the correct native
  `<|turn>` template (16,934 chars, no `<start_of_turn>`) → train and serve formats are now
  verifiably aligned. Template is exonerated as the remaining cause.
- **Raw-output probe (`/outputs/gemma-cal-staging-Q4_K_M.gguf`):** free generation emits pure
  degenerate looping — `'Huddle — — — — — …'` to the token limit; constrained generation emits
  512 tokens of nothing. **The weights are destroyed, not misformatted.**

With dataset (69→2,122), template (legacy/native), LR (2e-4/5e-5), and masking (on/off) all
varied, degradation always tracks training steps and ends in token-loop collapse. The remaining
common factor is **Unsloth's QLoRA path for Gemma-4-31B** (new architecture; training logs warn
`get_input_embeddings not auto-handled for Gemma4AudioModel`). **Fine-tuning is halted** until
that stack demonstrably works for this arch (or is replaced with plain transformers+PEFT).

## Step 8 — improve served evals via prompt (stock + targeted SYSTEM additions)
Base's only eval misses are prompt-fixable: m03 dropped the 2nd event of a multi-event thread;
q04 didn't ask clarification on a "TBD" plan. Added two surgical SYSTEM lines (list every
distinct event separately; ask via needs_clarification when day/time is TBD).

**Result: PERFECT SCORE — 1.0 on every metric (n=28, tp/fp/fn = 22/0/0).**

| Metric | base (old prompt) | **base + new prompt** |
| --- | --- | --- |
| schema validity | 1.00 | **1.00** |
| event precision | 1.00 | **1.00** |
| start-exact recall | 0.955 | **1.00** |
| event F1 | 0.977 | **1.00** |
| no-event accuracy | 1.00 | **1.00** |
| clarification recall | 0.75 | **1.00** |

Both misses fixed, nothing regressed. **This is the production configuration: stock
`unsloth/gemma-4-31B-it-GGUF` + the updated SYSTEM prompt.** (Set Space var
`MODEL_HF_REPO=unsloth/gemma-4-31B-it-GGUF`; the prompt ships with the app.) The "Well-Tuned"
artifact remains `ParetoOptimal/gemma-4-cal-gguf` (v1); any future fine-tune must beat THIS
1.0 baseline through the gate — i.e., match it and win on a harder, expanded eval set.

## Step 9 — the E4B edge-model campaign (2026-06-10)
Re-aimed fine-tuning where it has headroom: a **Gemma-4 E4B (~8B)** edge model that runs without a
paid A100, gated against **stock E4B**. Six gated runs, each fixing a diagnosed failure (the fixed
recipe trained cleanly every time — validity 1.0 throughout, confirming the Step-7 breakage was
specific to the 31B path):

| run | change | F1 | recall | clarify | eval |
| --- | --- | --- | --- | --- | --- |
| #1 | fixed recipe, 2,122 ex | 0.884 | 0.864 | 1.0 | n=28 |
| #2 | + weekday-in-prompt (+data regen) | 0.955 | 0.955 | 0.75 | n=28 |
| #3 | + next-DOW conflict filter (74 rows), 4× hand | **1.0** | **1.0** | 0.75 | n=28 |
| #4 | + TBD-clarify seeds, 8× hand | 0.93 | 0.909 | 1.0 | n=28 |
| #5 | clarify seeds, 4× hand | 0.93 | 0.909 | 1.0 | n=28 |
| — | **eval expanded 28→60** (50 events; jitter-resistant) | | | | |
| #6 | + Batch-7 seeds (next-DOW, "opens") | 0.97 | 0.96 | 1.0 | n=60 |
| stock E4B (weekday prompt) | | 0.97 | 0.96 | 1.0 | n=60 |

Run #6 vs stock is an **exact statistical tie** (identical tp/fp/fn 48/1/2; both miss `e09`
"next Tuesday" — which resisted 7 explicit training seeds — and one "opens" case each).
Campaign side effects that improved the PRODUCT for every model: weekday-in-prompt, the
next-DOW convention cleanup, and the 60-example eval.

## Step 10 — bare-prompt (internalization) test: no decisive gap
Dropped the system prompt for both models (identical minimal user content, same JSON-schema
constraint; `modal_eval.py --minimal-prompt`), measuring internalized task knowledge:

| bare, n=60 | stock E4B | fine-tuned E4B |
| --- | --- | --- |
| schema validity | 0.967 | **1.0** |
| event F1 | **0.682** | 0.644 |
| start-exact recall | **0.60** | 0.56 |
| no-event accuracy | 0.70 | **0.80** |
| clarification recall | 0.50 | **0.625** |

Small trade-offs both ways, within noise. **Verdict: at this data scale (139 hand + 2,000
SMCalFlow) with QLoRA/1-epoch, the E4B fine-tune reaches PARITY with stock, not superiority** —
non-degraded, perfect validity everywhere, better bare-prompt discipline, slightly weaker bare
extraction. The strict-dominance gate therefore never auto-promoted it; the candidate GGUF
remains on the Modal volume (`/outputs/gemma-cal-e4b-staging-Q4_K_M.gguf`). Publishing it as
the project's edge model at parity is a **product decision** (zero quality cost; production
would serve our own fine-tune, fulfilling "Well-Tuned") — deliberately left to the owner, not
the gate.