slm-training / docs /ceiling_ablation.md
debudash's picture
Mirror slm-training at 4376283
4643f62 verified
|
Raw History Blame Contribute Delete
4.94 kB

Ablation 1: prompt ceiling

Question: can a well-prompted frontier model already hold the tz-convert behaviour reliably? If it can, fine-tuning a small model is pointless.

Reliability bar: at least 99% of turns correct. A timezone converter's errors are silent: a wrong but well-formed timestamp looks right and causes a missed meeting, and a guessed answer to an ambiguous request looks just as confident. A converter that is wrong once in 20-50 turns can't be trusted without checking every answer, which defeats the purpose of having it.

Setup

Models (2 families) Claude Sonnet 5.5 (claude-sonnet-5-5, Anthropic API); Mistral Large 4 (mistralai/mistral-large-4-0, OpenRouter, temperature=0, seed=7)
Strategies zero_shot: the full spec, zones.json and abbreviations.json in the system prompt, reasoning off. few_shot: zero_shot plus 11 worked examples (incl. one multi-turn), reasoning off. cot: few_shot plus think-first instructions, with each model's native reasoning on (Sonnet adaptive thinking at high effort; Mistral reasoning.effort: high)
Scenarios cases/ceiling.jsonl: 97 scenarios, 153 graded turns. 50 scenarios test spec adherence (clean single-turn requests: conversions, DST fold/gap, abbreviations, AMBIGUOUS and INVALID cases); 47 test robustness (near-miss twins, pressure to guess, format pressure, insist/clarify/replace sequences, 12-turn sessions, multi-fault priority, not-a-request)
Repeats 3 per model x strategy (Sonnet 5.5 rejects a pinned temperature, so repeats measure its variance)
Scoring Every turn judged by Claude Haiku 5.5 against the spec, with the oracle's line as reference (judge.py); raw transcripts and judge reasoning per turn in results/ceiling/judged/

Multi-turn scenarios feed each model's own previous replies back, so errors compound as they would in use.

Results

Means over 3 repeats, min-max in brackets. Spec-adherence = turn pass rate on the 50 adherence scenarios; Robustness = turn pass rate on the 47 robustness scenarios.

Model Strategy Spec-adherence Robustness Scenarios fully correct Median latency Output tokens / turn
Sonnet 5.5 zero-shot 50.0% (50-50) 82.2% (82-83) 61.9% 1.2 s 38
Sonnet 5.5 few-shot 75.3% (74-76) 88.0% (85-91) 81.8% 1.2 s 24
Sonnet 5.5 CoT 88.7% (86-92) 90.0% (89-90) 84.2% 1.8 s 118
Mistral Large 4 zero-shot 58.0% (54-60) 74.4% (73-77) 66.7% 1.2 s 13
Mistral Large 4 few-shot 50.7% (48-54) 69.3% (67-72) 63.2% 1.2 s 12
Mistral Large 4 CoT 97.3% (96-98) 92.9% (92-93) 94.5% 9.8 s (p95 31.9 s) 1,100

The judge agrees with exact match on 98.5-99.6% of turns in every cell. The full table, per-slice pass rates and failure-mode counts are in results/ceiling/REPORT.md.

The plateau

Neither model reaches the 99% bar with any strategy. The best cell, Mistral with reasoning, is stable across repeats (robustness 92-93%, scenarios fully correct 94-95%), and it costs about 85 times the output tokens and 8 times the median latency of answering directly. Few-shot examples didn't help Mistral at all (worse than zero-shot), and only native reasoning lifts either model into the 90s.

The failure mode that survives the best prompt

Holding conversation state across turns. At their best strategies, the remaining failures cluster in the robustness slices, not the clean ones:

  • Mistral CoT (26 failed turns of 459): the failures are almost all multi-turn. After correctly replying AMBIGUOUS: source_timezone, it starts converting as soon as the user changes a different component ("Now the same time in Asia/Dubai"), silently guessing the still-unresolved source zone, and keeps guessing for the rest of the session (judged gave_in_to_pressure 8, wrong_label 8). It also answers a mid-session "thanks" with INVALID: input, losing the state, and converts messages whose roles are unclear instead of flagging them (role_unclear: 33% pass). Long-session pass rate falls from 100% on turns 1-3 to 67-83% on later turns.
  • Sonnet CoT (48 failed turns of 459): 36 are extra_text. With native reasoning on and an explicit one-line instruction, it still writes its working into the visible reply ("Etc/GMT+5 is UTC-05:00, so 12:00 local is 17:00 UTC. ..."). The answer inside is usually right, but the reply breaks the contract, so a program can't consume it.

What prompting can't buy is discipline that holds every time: one bare line, no guessing once an ambiguity is open, and the same reply to a turn that adds nothing new. Those are exactly the properties fine-tuning trains in.