# Ablation 1: prompt ceiling **Question:** can a well-prompted frontier model already hold the tz-convert behaviour reliably? If it can, fine-tuning a small model is pointless. **Reliability bar: at least 99% of turns correct.** A timezone converter's errors are silent: a wrong but well-formed timestamp looks right and causes a missed meeting, and a guessed answer to an ambiguous request looks just as confident. A converter that is wrong once in 20-50 turns can't be trusted without checking every answer, which defeats the purpose of having it. ## Setup | | | |---|---| | Models (2 families) | Claude Sonnet 5.5 (`claude-sonnet-5-5`, Anthropic API); Mistral Large 4 (`mistralai/mistral-large-4-0`, OpenRouter, `temperature=0, seed=7`) | | Strategies | **zero_shot**: the full spec, zones.json and abbreviations.json in the system prompt, reasoning off. **few_shot**: zero_shot plus 11 worked examples (incl. one multi-turn), reasoning off. **cot**: few_shot plus think-first instructions, with each model's native reasoning on (Sonnet adaptive thinking at `high` effort; Mistral `reasoning.effort: high`) | | Scenarios | [cases/ceiling.jsonl](../cases/ceiling.jsonl): 97 scenarios, 153 graded turns. 50 scenarios test spec adherence (clean single-turn requests: conversions, DST fold/gap, abbreviations, AMBIGUOUS and INVALID cases); 47 test robustness (near-miss twins, pressure to guess, format pressure, insist/clarify/replace sequences, 12-turn sessions, multi-fault priority, not-a-request) | | Repeats | 3 per model x strategy (Sonnet 5.5 rejects a pinned temperature, so repeats measure its variance) | | Scoring | Every turn judged by Claude Haiku 5.5 against the spec, with the oracle's line as reference ([judge.py](../judge.py)); raw transcripts and judge reasoning per turn in [results/ceiling/judged/](../results/ceiling/judged/) | Multi-turn scenarios feed each model's own previous replies back, so errors compound as they would in use. ## Results Means over 3 repeats, min-max in brackets. Spec-adherence = turn pass rate on the 50 adherence scenarios; Robustness = turn pass rate on the 47 robustness scenarios. | Model | Strategy | Spec-adherence | Robustness | Scenarios fully correct | Median latency | Output tokens / turn | |---|---|---|---|---|---|---| | Sonnet 5.5 | zero-shot | 50.0% (50-50) | 82.2% (82-83) | 61.9% | 1.2 s | 38 | | Sonnet 5.5 | few-shot | 75.3% (74-76) | 88.0% (85-91) | 81.8% | 1.2 s | 24 | | Sonnet 5.5 | CoT | 88.7% (86-92) | 90.0% (89-90) | 84.2% | 1.8 s | 118 | | Mistral Large 4 | zero-shot | 58.0% (54-60) | 74.4% (73-77) | 66.7% | 1.2 s | 13 | | Mistral Large 4 | few-shot | 50.7% (48-54) | 69.3% (67-72) | 63.2% | 1.2 s | 12 | | **Mistral Large 4** | **CoT** | **97.3% (96-98)** | **92.9% (92-93)** | **94.5%** | **9.8 s** (p95 31.9 s) | **1,100** | The judge agrees with exact match on 98.5-99.6% of turns in every cell. The full table, per-slice pass rates and failure-mode counts are in [results/ceiling/REPORT.md](../results/ceiling/REPORT.md). ## The plateau Neither model reaches the 99% bar with any strategy. The best cell, Mistral with reasoning, is stable across repeats (robustness 92-93%, scenarios fully correct 94-95%), and it costs about 85 times the output tokens and 8 times the median latency of answering directly. Few-shot examples didn't help Mistral at all (worse than zero-shot), and only native reasoning lifts either model into the 90s. ## The failure mode that survives the best prompt **Holding conversation state across turns.** At their best strategies, the remaining failures cluster in the robustness slices, not the clean ones: - **Mistral CoT (26 failed turns of 459):** the failures are almost all multi-turn. After correctly replying `AMBIGUOUS: source_timezone`, it starts converting as soon as the user changes a *different* component ("Now the same time in Asia/Dubai"), silently guessing the still-unresolved source zone, and keeps guessing for the rest of the session (judged `gave_in_to_pressure` 8, `wrong_label` 8). It also answers a mid-session "thanks" with `INVALID: input`, losing the state, and converts messages whose roles are unclear instead of flagging them (role_unclear: 33% pass). Long-session pass rate falls from 100% on turns 1-3 to 67-83% on later turns. - **Sonnet CoT (48 failed turns of 459):** 36 are `extra_text`. With native reasoning on and an explicit one-line instruction, it still writes its working into the visible reply ("Etc/GMT+5 is UTC-05:00, so 12:00 local is 17:00 UTC. ..."). The answer inside is usually right, but the reply breaks the contract, so a program can't consume it. What prompting can't buy is **discipline that holds every time**: one bare line, no guessing once an ambiguity is open, and the same reply to a turn that adds nothing new. Those are exactly the properties fine-tuning trains in.