Download docs/ceiling_ablation.md from debudash/slm-training: direct link, hf CLI and curl.
- Browser
- Download file 4.94 kB
-
https://huggingface.co/debudash/slm-training/resolve/main/docs/ceiling_ablation.md
- Command line
-
hf download hf://debudash/slm-training/docs/ceiling_ablation.md
-
curl -L -o ceiling_ablation.md https://huggingface.co/debudash/slm-training/resolve/main/docs/ceiling_ablation.md
Ablation 1: prompt ceiling
Question: can a well-prompted frontier model already hold the tz-convert behaviour reliably? If it can, fine-tuning a small model is pointless.
Reliability bar: at least 99% of turns correct. A timezone converter's errors are silent: a wrong but well-formed timestamp looks right and causes a missed meeting, and a guessed answer to an ambiguous request looks just as confident. A converter that is wrong once in 20-50 turns can't be trusted without checking every answer, which defeats the purpose of having it.
Setup
| Models (2 families) | Claude Sonnet 5.5 (claude-sonnet-5-5, Anthropic API); Mistral Large 4 (mistralai/mistral-large-4-0, OpenRouter, temperature=0, seed=7) |
| Strategies | zero_shot: the full spec, zones.json and abbreviations.json in the system prompt, reasoning off. few_shot: zero_shot plus 11 worked examples (incl. one multi-turn), reasoning off. cot: few_shot plus think-first instructions, with each model's native reasoning on (Sonnet adaptive thinking at high effort; Mistral reasoning.effort: high) |
| Scenarios | cases/ceiling.jsonl: 97 scenarios, 153 graded turns. 50 scenarios test spec adherence (clean single-turn requests: conversions, DST fold/gap, abbreviations, AMBIGUOUS and INVALID cases); 47 test robustness (near-miss twins, pressure to guess, format pressure, insist/clarify/replace sequences, 12-turn sessions, multi-fault priority, not-a-request) |
| Repeats | 3 per model x strategy (Sonnet 5.5 rejects a pinned temperature, so repeats measure its variance) |
| Scoring | Every turn judged by Claude Haiku 5.5 against the spec, with the oracle's line as reference (judge.py); raw transcripts and judge reasoning per turn in results/ceiling/judged/ |
Multi-turn scenarios feed each model's own previous replies back, so errors compound as they would in use.
Results
Means over 3 repeats, min-max in brackets. Spec-adherence = turn pass rate on the 50 adherence scenarios; Robustness = turn pass rate on the 47 robustness scenarios.
| Model | Strategy | Spec-adherence | Robustness | Scenarios fully correct | Median latency | Output tokens / turn |
|---|---|---|---|---|---|---|
| Sonnet 5.5 | zero-shot | 50.0% (50-50) | 82.2% (82-83) | 61.9% | 1.2 s | 38 |
| Sonnet 5.5 | few-shot | 75.3% (74-76) | 88.0% (85-91) | 81.8% | 1.2 s | 24 |
| Sonnet 5.5 | CoT | 88.7% (86-92) | 90.0% (89-90) | 84.2% | 1.8 s | 118 |
| Mistral Large 4 | zero-shot | 58.0% (54-60) | 74.4% (73-77) | 66.7% | 1.2 s | 13 |
| Mistral Large 4 | few-shot | 50.7% (48-54) | 69.3% (67-72) | 63.2% | 1.2 s | 12 |
| Mistral Large 4 | CoT | 97.3% (96-98) | 92.9% (92-93) | 94.5% | 9.8 s (p95 31.9 s) | 1,100 |
The judge agrees with exact match on 98.5-99.6% of turns in every cell. The full table, per-slice pass rates and failure-mode counts are in results/ceiling/REPORT.md.
The plateau
Neither model reaches the 99% bar with any strategy. The best cell, Mistral with reasoning, is stable across repeats (robustness 92-93%, scenarios fully correct 94-95%), and it costs about 85 times the output tokens and 8 times the median latency of answering directly. Few-shot examples didn't help Mistral at all (worse than zero-shot), and only native reasoning lifts either model into the 90s.
The failure mode that survives the best prompt
Holding conversation state across turns. At their best strategies, the remaining failures cluster in the robustness slices, not the clean ones:
- Mistral CoT (26 failed turns of 459): the failures are almost all multi-turn. After correctly replying
AMBIGUOUS: source_timezone, it starts converting as soon as the user changes a different component ("Now the same time in Asia/Dubai"), silently guessing the still-unresolved source zone, and keeps guessing for the rest of the session (judgedgave_in_to_pressure8,wrong_label8). It also answers a mid-session "thanks" withINVALID: input, losing the state, and converts messages whose roles are unclear instead of flagging them (role_unclear: 33% pass). Long-session pass rate falls from 100% on turns 1-3 to 67-83% on later turns. - Sonnet CoT (48 failed turns of 459): 36 are
extra_text. With native reasoning on and an explicit one-line instruction, it still writes its working into the visible reply ("Etc/GMT+5 is UTC-05:00, so 12:00 local is 17:00 UTC. ..."). The answer inside is usually right, but the reply breaks the contract, so a program can't consume it.
What prompting can't buy is discipline that holds every time: one bare line, no guessing once an ambiguity is open, and the same reply to a turn that adds nothing new. Those are exactly the properties fine-tuning trains in.