Download docs/drift.md from debudash/slm-training: direct link, hf CLI and curl.
- Browser
- Download file 5.45 kB
-
https://huggingface.co/debudash/slm-training/resolve/main/docs/drift.md
- Command line
-
hf download hf://debudash/slm-training/docs/drift.md
-
curl -L -o drift.md https://huggingface.co/debudash/slm-training/resolve/main/docs/drift.md
Drift eval: holding conversation state across turns
Question: the prompt-ceiling ablation found that the failure surviving the best prompt is multi-turn state (guessing after a flag, losing state mid-session). Does the fine-tuned model hold state better than prompted frontier models?
Answer: yes. The final model (v5) holds 91.4% of conversations, against 76.4% for Sonnet and 57.1% for Mistral with reasoning, with no state failures. v2 held state far worse than the frontier models. v3 added the multi-turn state patterns and matched Sonnet (75.7%); v4 fixed role-free zone changes (ISSUE-002, 80.0%); v5 added DST-pair arithmetic data (ISSUE-003), which cut the wrong hours that carry into later turns from 27 to 5.
| Model | Conversations held | Turns correct | State failures (wrong label + gave in) | Wrong hours |
|---|---|---|---|---|
| Qwen3-4B tuned v5, N=18000 (final) | 91.4% | 98.1% | 0 | 5 |
| Qwen3-4B tuned v4, N=12000 | 80.0% | 93.8% | 0 | 27 |
| Qwen3-4B tuned v3, N=12000 | 75.7% | 91.8% | 1 | 33 |
| Sonnet 5.5, CoT (2 runs) | 76.4% | 94.8% | 2 (plus 40 turns of leaked reasoning) | |
| Mistral Large 4, CoT (2 runs) | 57.1% | 88.6% | 71 |
Results: results/drift/. The rest of this document is the v3 analysis and, below it, the original v2 analysis that led to the v3 data change.
v3 result
| Model | Conversations held start to finish | Turns correct | State failures (wrong label + gave in) |
|---|---|---|---|
| Sonnet 5.5, CoT (2 runs) | 76.4% | 94.8% | 2 (plus 40 turns of leaked reasoning) |
| Qwen3-4B tuned v3, N=12000 | 75.7% | 91.8% | 1 |
| Mistral Large 4, CoT (2 runs) | 57.1% | 88.6% | 71 |
| Qwen3-4B tuned v2, N=6000 | 12.9% | 54.0% | 42 |
By pattern, v3 beats both frontier models on clarification threads (100% vs 95% / 5%), late pressure (80% vs 60% / 75%) and filler chains (60% vs 35% / 40%); Sonnet stays ahead when an open flag must survive a change to another component (100% vs 70%) and on corrections (95% vs 80%). v3 gets 99% of first turns right (frontier 84-87%) but decays to 75-84% by turns 7-9 as a wrong hour carries forward; Sonnet holds 97-100% there.
Caveat: v3's training data contains these seven patterns, with different wording and values (no user turn matches verbatim), so this is an in-distribution test of the patterns. No staff held-out set was provided, so there is no out-of-distribution score; the nearest evidence is the ISSUE-002 probe, where unscripted wording exposed a failure the drift set couldn't see.
The rest of this document is the original v2 analysis that led to the v3 data change.
Setup
cases/drift.jsonl: 70 conversations, 485 graded turns, 5-9 turns each, in 7 patterns: an open ambiguity that must survive changes to other components, filler and doubt that must repeat the last reply, an ambiguous-date thread with pressure and a wrong-form answer, unclear roles then roles stated, clean turns followed by an ambiguous request under escalating pressure, a refusal then a correction then a late ambiguity, and fresh requests interleaved with follow-ups. Conversions use whole-hour zones without DST where possible, so the arithmetic is easy. Every turn is labelled by the oracle. 12 turns ("And in Asia/Dubai?") match generic training phrasings verbatim; the conversations themselves are new.
Frontier models at their best strategy (CoT), 2 repeats; base and tuned via eval.py, greedy.
Results
| Model | Conversations held start to finish | Turns correct |
|---|---|---|
| Sonnet 5.5, CoT | 76.4% (76-77) | 94.8% |
| Mistral Large 4, CoT | 57.1% (54-60) | 88.6% |
| Qwen3-4B base, full spec prompt | 0.0% | 15.5% |
| Qwen3-4B tuned (v2, N=6000) | 12.9% | 54.0% |
Per-pattern results, pass rate by turn number and failure modes are in results/drift/REPORT.md.
What it shows
Frontier models drift. Even with reasoning on, Mistral gives in to pressure (38 turns) or uses the wrong label (33) as conversations go on, and holds only 5% of ambiguous-date threads and 20% of unclear-role conversations. Sonnet holds state better but writes its reasoning into the reply (40 turns), and holds only 35% of filler-heavy conversations. Neither is near the 99% bar.
The tuned model drifts more, mostly through arithmetic that carries forward. 77% of its first turns are right; 171 of its 223 failed turns are wrong_time, usually exactly 1 hour off, and a wrong value is then repeated through every follow-up and filler turn. It also has real state errors (wrong_label 23, gave_in_to_pressure 19, sometimes converting to the previous target after a change).
Why: the v2 training data doesn't contain the hardest patterns in this eval (an open flag surviving a change to another component, unclear roles resolved mid-conversation, composed date+target+source changes), and its conversion accuracy is still rising with data (results.md).
Next (v3 data, for the final submission)
- Multi-turn conversations in all seven drift patterns, written with teacher templates (not the eval phrasings), as about a third of the dataset.
- More conversions per zone pair and season, the bottleneck behind the carried-forward errors.
- Re-run this eval on the v3 checkpoints alongside the curve.