# tz-convert: a small model that converts timezones without guessing Fine-tune Qwen3-4B-Instruct-2507 (QLoRA) to hold one falsifiable behaviour, and show with numbers that it beats the base model, and that frontier models plateau below the reliability bar even with their best prompt. ## Final submission checklist | Requirement (course brief) | Where | |---|---| | Dataset, published | [debudash/tz-convert-sft](https://huggingface.co/datasets/debudash/tz-convert-sft) (configs v1-v5; v5 trains the final model) · [docs/dataset.md](docs/dataset.md) | | Model on the Hub, public | [debudash/qwen3-4b-tz-convert-v5-n18000](https://huggingface.co/debudash/qwen3-4b-tz-convert-v5-n18000) @ `f90209fc`, plus 13 earlier checkpoints ([pinned versions](#pinned-versions)) | | Running inference demo | **Public: https://debulearn4ever--slm-training-demo-ui-ui.us-east.modal.run** (first reply after an idle spell takes 2-3 min while the GPU starts) · local: Docker, see [Demo](#demo-base-vs-tuned-side-by-side); parity with the evals 15/15 | | Eval harness and base-vs-tuned results | `python eval.py --model --eval-set ` · [docs/results.md](docs/results.md) | | Staff held-out set | None was provided. `eval.py` scores any unlabelled set (labels via [extract.py](extract.py) and the oracle) | | Full data-efficiency curve, justified minimum N | [docs/results.md](docs/results.md#data-efficiency-curve): 14 checkpoints; ~2,000 examples for discipline, ~12,000 for state | | Prompt-ceiling ablation (2 families x 3 strategies) | [docs/ceiling_ablation.md](docs/ceiling_ablation.md) · `run_ceiling.py` | | Brainlift | [docs/brainlift.md](docs/brainlift.md) | | Raw judge transcripts (JSONL) | `results/*/judged/*.jsonl` (score, failure mode and reasoning per turn) | | Pinned versions | Eval code `ed377d8`; base `cdbee75f`; every checkpoint's full hash in [results/training/](results/training/) | | Ablation reproducibility | Ceiling: `python run_ceiling.py --models mistral --strategies cot --repeats 1`; training logs (loss every 5 steps) in [results/training/](results/training/) | | Failures fixed with data, not hyperparameters | [docs/ISSUE_LOG.md](docs/ISSUE_LOG.md): ISSUE-002 fixed in v4 (0/8 → 8/8); ISSUE-003 open, partly improved in v5 (58 → 68 of 80) | | 3-5 minute demo video | Submitted separately (3:03). No grader-supplied prompt was provided, so the live part runs the demo app on camera: chosen examples labelled as chosen, plus a seed-drawn set | ## Behaviour spec > Given a timezone conversion request, the model replies with exactly one line: the converted moment as > `YYYY-MM-DDTHH:MM:SS±HH:MM` per tzdata 2026e, or `AMBIGUOUS:` / `INVALID: ` when it can't convert > without guessing. It never guesses, even when the user insists. The full rules (accepted forms, DST resolution, priority, multi-turn) are in [behaviour.spec](behaviour.spec), with the closed world in [zones.json](zones.json) (24 zones) and [abbreviations.json](abbreviations.json) (v3), for source dates 2020-2030. `python scripts/audit_tables.py` checks both tables against tzdata. ## How correctness is decided [oracle.py](oracle.py) implements the spec exactly and computes the expected reply for every scenario and every training example (tests: `python -m pytest tests`). The LLM judge ([judge.py](judge.py), Claude Haiku 5.5) scores every turn against the spec with the oracle's line as reference and writes its score, failure mode and reasoning per turn; reports show the judge score next to exact match, and how often they agree. ## Reproduce ```bash pip install -r requirements.txt # Python 3.12; .env needs ANTHROPIC_API_KEY, OPENROUTER_API_KEY modal setup # GPU work runs on Modal; secret "huggingface" holds HF_TOKEN # One-command eval (base or any adapter repo): generates with vLLM on Modal, judges, prints the table python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --revision f90209fce510b83466fc4d2784256d6b9003f281 --eval-set cases/ceiling.jsonl python eval.py --model Qwen/Qwen3-4B-Instruct-2507 --eval-set cases/ceiling.jsonl --prompt spec # Prompt-ceiling ablation (Sonnet 5.5 + Mistral Large 4 x zero-shot / few-shot / CoT x 3 repeats) python run_ceiling.py && python judge.py results/ceiling/*.jsonl && python report_ceiling.py # one sample point: python run_ceiling.py --models mistral --strategies cot --repeats 1 --out runs/rerun # Data and training python gen_cases.py --seed 1 --out cases/ceiling.jsonl # eval scenarios (oracle-labelled) python gen_cases.py --set drift --seed 21 --out cases/drift.jsonl # drift set (multi-turn state) python teacher.py --out data/templates.json # teacher-written templates + quality gate python teacher.py --add roles_stated,doubt --out data/templates.json # v3 follow-up templates python gen_train.py --n 12000 --seed 13 --mix v3 --out data/train_v3.jsonl modal run train.py --data data/train_v3.jsonl --ns 1500,3000,6000,12000 --tag v3 --gpu H100 python teacher.py --add change_no_role,replace_target_explicit --out data/templates.json # v4 templates (ISSUE-002) python gen_train.py --n 12000 --seed 14 --mix v4 --out data/train_v4.jsonl python gen_train.py --n 18000 --seed 15 --mix v5 --out data/train_v5.jsonl # final modal run train.py --data data/train_v5.jsonl --ns 18000 --tag v5 --gpu H100 # Probes: ISSUE-002 (role-free zone changes) and ISSUE-003 (DST-pair arithmetic) python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_target_change.jsonl python eval.py --model debudash/qwen3-4b-tz-convert-v5-n18000 --eval-set cases/probe_dst_pairs.jsonl ``` ### Smoke test: the full loop on 50 examples ```bash python gen_train.py --n 50 --seed 99 --mix v1 --out data/smoke.jsonl modal run train.py --data data/smoke.jsonl --ns 50 --tag smoke python eval.py --model debudash/qwen3-4b-tz-convert-smoke-n50 --eval-set cases/ceiling.jsonl --label smoke-n50 ``` Run on 2026-10-28: generate → train → eval end to end in a few minutes. | Step | Output | |---|---| | Generate | [data/smoke.jsonl](data/smoke.jsonl): 50 oracle-labelled examples, 0 dropped for eval overlap | | Train | 12 steps in 0.3 min on an L40S, loss 3.34 → 0.53, pushed `debudash/qwen3-4b-tz-convert-smoke-n50` @ `944d2795` ([log](results/training/qwen3-4b-tz-convert-smoke-n50.json)) | | Eval | 153 turns generated and judged ([judge transcript](results/eval/smoke-n50/judged/smoke-n50__short-ceiling__r1.jsonl)): 4.0% spec-adherence and 13.6% robustness | The low scores are expected: 50 examples only checks that the pipeline works. The real curve starts at N=250 ([docs/results.md](docs/results.md)). `--eval-set` accepts JSONL in any of these forms, one per line: - our scenario format: `{"id", "slice", "metric", "turns": [{"user": ..., "expected": ...}, ...]}` - a single prompt: `{"prompt": "..."}` (also `input`, `question`, `user` or `text`), optionally with `"expected"` - a conversation: `{"messages": [{"role": "user", "content": ...}, ...]}` (user turns are replayed) - a plain line of text Turns without an expected reply are labelled by [extract.py](extract.py): Claude Sonnet 5.5 copies the request's components as written, and the oracle computes the reply the spec requires. Validated against oracle labels, the extracted labels match on 153/153 ceiling turns and 480/485 drift turns (the 5 misses are one debatable wrong-form clarification). Extracted labels are marked `label_source: "extracted"` in the results. ## Demo: base vs tuned, side by side A Streamlit page (in Docker) sends each turn to the base model, tuned v3 (before the ISSUE-002/003 fixes) and the final tuned model (v5), and shows the replies side by side under the oracle's answer, with the measured scores of all three and of the frontier models at their best prompt ([demo/measured.json](demo/measured.json)). Inference runs on Modal ([serve_demo.py](serve_demo.py)) with eval.py's exact setup: the same vLLM pin, pinned base revision, greedy decoding, and each model's eval system prompt. **Public demo:** https://debulearn4ever--slm-training-demo-ui-ui.us-east.modal.run — the same app hosted on Modal ([serve_ui.py](serve_ui.py); `modal deploy serve_ui.py`), with the token kept in the Modal secret. The GPU sleeps when idle, so the first reply after a quiet spell takes 2-3 minutes. ```bash # .env: DEMO_TOKEN=, then store it on Modal modal secret create demo-token DEMO_TOKEN= modal deploy serve_demo.py # prints the endpoint URL; put it in .env as DEMO_URL python scripts/demo_parity.py --final v5-n18000 # replies vs the recorded eval runs (first turns of 5 scenarios) docker compose up --build -d # http://localhost:8507 ``` - **Models:** the final model (v5 @ `f90209fc`) and the "before" model (v3 @ `5dd04ea9`) are the defaults in [serve_demo.py](serve_demo.py); `DEMO_ADAPTER` / `DEMO_ADAPTER_REVISION` and `DEMO_BEFORE_*` override them at deploy. - **Parity:** 15/15 replies byte-identical to the eval transcripts (base, v3 and v5, 5 each). After a redeploy, a container that is still warm can keep serving the previous adapter for a few minutes; check `GET /info` (or the caption at the top of the page) shows the adapter you expect before recording, and re-run the parity check. - **No cold start while recording:** `DEMO_WARM=1 modal deploy serve_demo.py` keeps one GPU up. Redeploy without it afterwards. - **Scenarios** ([demo/script.jsonl](demo/script.jsonl), written by [scripts/gen_demo_script.py](scripts/gen_demo_script.py)): four featured ones, labelled as chosen, one per skill or fix and picked by a stated rule before any final-model result (refusing to guess under pressure, holding state, ISSUE-002, ISSUE-003); then 3 clean conversions, 1 pressure case and 1 drift conversation drawn by fixed seed 20261101. A free-text mode takes any prompt, without a right/wrong mark. - Every turn is logged to `results/demo/.jsonl`. ## Results - [docs/ceiling_ablation.md](docs/ceiling_ablation.md): the prompt ceiling (Ablation 1) - [docs/results.md](docs/results.md): base vs tuned and the data-efficiency curve (Ablation 2) - [docs/drift.md](docs/drift.md): holding conversation state, frontier vs tuned - [docs/brainlift.md](docs/brainlift.md): thesis and evidence - [docs/dataset.md](docs/dataset.md): dataset shape, how it is made, and the three mixes - [docs/ISSUE_LOG.md](docs/ISSUE_LOG.md): bugs found and fixed (ISSUE-002 fixed in v4; ISSUE-003 open, partly improved in v5) Final checkpoint: **`debudash/qwen3-4b-tz-convert-v5-n18000` @ `f90209fc`** (full hash `f90209fce510b83466fc4d2784256d6b9003f281`). ## Pinned versions **Eval code:** commit `ed377d8` of this repo, the last commit to change the eval code (eval.py, judge.py, extract.py, oracle.py, the report scripts), the ceiling and drift sets, or the zone tables. Later commits add data generation for v4 and v5, two probe sets with their scripts, the demo, docs and results; `git diff ed377d8 HEAD -- eval.py judge.py extract.py oracle.py report_ceiling.py report_drift.py cases/ceiling.jsonl cases/drift.jsonl zones.json abbreviations.json` is empty. **Base model:** `Qwen/Qwen3-4B-Instruct-2507` @ `cdbee75f17c01a7cc42f958dc650907174af0554`. **Checkpoints:** public LoRA adapters on the Hugging Face Hub, all trained with the same hyperparameters. Each training log in [results/training/](results/training/) carries the full commit hash. Model cards were added later in card-only commits, so a repo's head is newer than the hash below; the pinned commit has the same adapter. | Checkpoint | Commit | | Checkpoint | Commit | | Checkpoint | Commit | |---|---|---|---|---|---|---|---| | `debudash/qwen3-4b-tz-convert-v1-n250` | `801f58a2` | | `debudash/qwen3-4b-tz-convert-v2-n750` | `cc2fd946` | | `debudash/qwen3-4b-tz-convert-v3-n1500` | `feae72b7` | | `debudash/qwen3-4b-tz-convert-v1-n500` | `b5591ba6` | | `debudash/qwen3-4b-tz-convert-v2-n1500` | `bde7bb85` | | `debudash/qwen3-4b-tz-convert-v3-n3000` | `c64cccad` | | `debudash/qwen3-4b-tz-convert-v1-n1000` | `50e9d5e5` | | `debudash/qwen3-4b-tz-convert-v2-n3000` | `460efb75` | | `debudash/qwen3-4b-tz-convert-v3-n6000` | `66e75b01` | | `debudash/qwen3-4b-tz-convert-v1-n2000` | `48c197c6` | | `debudash/qwen3-4b-tz-convert-v2-n6000` | `40fb98b8` | | `debudash/qwen3-4b-tz-convert-v3-n12000` | `5dd04ea9` | | `debudash/qwen3-4b-tz-convert-v4-n12000` | `50b411fe` | | **`debudash/qwen3-4b-tz-convert-v5-n18000`** | **`f90209fc`** | | | | To evaluate a checkpoint at its pinned commit: `python eval.py --model --revision --eval-set `, taking the full hash from the checkpoint's training log.