APPENDIX-looping.md

Appendix: the tool-call loop defect, and the seed sweep against it

Every number here is measured and its artifact is kept.

What the defect is

A turn ends with the model emitting the same tool call over and over instead of stopping — sometimes the identical call, sometimes the same tool with varied arguments. The worst example we recorded ran 313 consecutive tool calls in a single turn. In production this is not a quality problem, it is an availability one: the turn never terminates, and it consumes the budget of whatever called it.

Scope of the claim — read this before quoting any number here

Everything below was measured on one checkpoint: mistralai/Ministral-3-8B-Instruct-2512-BF16 at revision f6fae9795746f63c9be8344932f01275f3c63734, served as Q8_0 GGUF through llama.cpp c812c543f, on our production-derived suite.

This is not a claim about "Mistral models". A family name predicts neither the behaviour nor the licence of its next release, and this project has an expensive worked example: Ministral-8B-Instruct-2410 is research-licensed while Ministral-3-8B-Instruct-2512 is apache-2.0 — same product name, opposite answers. We have not measured any other Mistral checkpoint, so we say nothing about them. What we can say is that our pinned base exhibits the defect, and that fine-tuning moves the rate in both directions depending on nothing but the seed.

The base already loops

population	loop rate	worst response
full faithful set (1,133 tool-turns, 1,965 items)	🔴 0.09 % (1 turn)	111 calls
select-vera (146 tool-turns)	🔴 0.68 % (1 turn)	111 calls
screen-vera (212 tool-turns)	🔴 0.47 % (1 turn)	111 calls

The first row is the number to quote, and all three rows are the same single event seen through different denominators — one looping turn, 111 calls, a single tool name repeated 108 times. On 146 tool-turns that one turn is 0.68 %; on 1,133 it is 0.09 %. Nothing about the base changed between those measurements, and quoting the largest of them because it is the most convenient would be reporting a sample size as a finding.

The scan completed 2026-10-02 over 1,965 of 1,965 collectable items (Q8_0, temperature 0, cap 4096), and its scorer output is kept at benchmarks/base-scans/faithful-all-q8-cap4096/scores.json. The name-repeat histogram is the useful shape: 1,059 turns repeat a tool name once, 52 call no tool, a thin tail reaches 10 — and then one turn sits at 108, with nothing in between. The defect is rare and catastrophic, not a gradient, which is why a threshold-based gate works at all and why an aggregate rate understates it: the cost of the one turn is an unterminated production request, not 0.09 % of a quality score.

So the defect is not introduced by fine-tuning. That matters for reading everything below: a trained arm with a non-zero rate has not necessarily been damaged, and an arm at zero has genuinely removed something the base does.

How a loop is detected

Three independent readings, any one of which condemns a turn:

rule	threshold	what it catches
maxIdenticalCallRepeat	≥ 8	the identical (name, arguments) signature repeated
toolCallCount	≥ 200	a runaway trajectory of any shape
maxToolNameRepeat	≥ 24	the same tool NAME repeated, arguments ignored

The third rule exists because a model that loops while varying one argument field looks like progress to a signature-based check and is the same failure.

The name threshold was calibrated against production traffic, not guessed. At 24 it is deliberately generous: a stricter setting of 12 condemns a real production turn that called web_search 15 times with distinct site: queries — legitimate work. A detector that cannot tell a loop from a thorough search is worse than none, because it teaches you to distrust it.

The sweep

Ten runs, seeds 42–51. Identical dataset (l2-3f301662871f), recipe (LoRA rank 8, dropout 0.05, scale 1.0, attention projections only, lr 2e-5 cosine), checkpoint (150), quantisation and scoring population. The seed is the only difference.

A seed is not a quantity — it is the initial state of a pseudo-random generator, and here it controls three things: LoRA A initialisation, every dropout mask, and data order. We checked that consecutive seeds are not secretly similar: across MLX's RNG and numpy's permutation, mean |r| between adjacent seeds (0.0037 and 0.0181) sits at or below the noise floor and is no higher than between distant values like 1009 and 999983. 42–51 samples the same distribution that scattered values would.

Results

validFirstToolMatchPct on select-vera, 146 tool-turns, cap 4096, temperature 0.

seed	score	Δ vs base	loop rate	worst response	gate
base (untrained)	80.82 ‡	—	🔴 0.68 %	272	—
46	96.58	🟢 +15.76 pts (+19.5 %)	0.00 %	13	passes
48	96.58	🟢 +15.76 pts (+19.5 %)	🔴 0.68 %	116	excluded
43	95.21	🟢 +14.39 pts (+17.8 %)	🔴 0.68 %	292	excluded
42	92.92 †	not comparable	0.00 %	13	passes
45	93.15	🟢 +12.33 pts (+15.3 %)	0.00 %	—	passes
50	93.15	🟢 +12.33 pts (+15.3 %)	🔴 1.37 %	—	excluded
49	92.47	🟢 +11.65 pts (+14.4 %)	🔴 5.48 %	—	excluded
44	91.10	🟢 +10.28 pts (+12.7 %)	🔴 6.85 %	313	excluded
47	90.41	🟢 +9.59 pts (+11.9 %)	0.00 %	—	passes
51	94.52	🟢 +13.70 pts (+17.0 %)	0.00 %	15	passes

‡ The base was collected directly on these 266 items at the same cap and temperature, three times with identical results; this is the figure the card quotes. (Re-scoring the base's 404-item screen responses on this population gives 74.66 instead — a re-score of a different run, not a measurement of this one.)

† Seed 42 is not measured on this population. It was the original arm and was scored on screen-vera — 404 items, 212 tool-turns — before the sweep standardised on select-vera's 266, so 92.92 is its screen-vera figure and is not comparable to the rest of this column.

Seed 42's artifact-backed figures are: report-vera-sealed 95.45 (66 turns, tying seed 46), production temperature 0.35 95.21 (146 turns), and the loop-provocation slice 100.0. Those are the numbers to quote for it.

Five of the ten seeds loop at least once. The rate ranges from 0.00 % to 6.85 % — ten of 146 tool-turns — from one recipe on one dataset. This is real variance in what gets trained, not measurement noise: the loop VERDICT agreed 20/20 across two collections of the same artifact (see "Is it the sampler?").

Is it the sampler, or the model?

The screen is not byte-deterministic. Re-collecting the 20 busiest items against the same artifact gave 14/20 identical tool-call sequences and 19/20 identical text; one item ran 119 calls one time and 158 the next.

That is the right alarm to raise and it does not make the gate unsound, because the gate is a binary per-item verdict and a 119-call and a 158-call runaway are the same verdict. Measured per item with the real scorer, over the same 20 items:

fact	agreement
looped	20/20
firstToolMatch	20/20
validFirstToolMatch	20/20

Those 20 were chosen as the busiest — adversarially, where instability compounds — which makes the agreement stronger evidence than a random sample. So the difference between seed 42's zero and seed 44's 6.85 % is a property of the models, not of the sampling.

The gate, and what it cost

Looping is a gate, not a ranking dimension. Any non-zero rate excludes the seed outright; survivors are then ranked on score. There is no ranking within non-zero, because 3 loops versus 4 is noise while zero versus non-zero is not.

For most of the sweep this was free — the best score happened also to be clean. Then seed 48 scored 96.58, identical to seed 46, 141 of 146 tool-turns for both, and loops once. It is excluded and seed 46 ships. Two models indistinguishable on the ranking metric, and the tie broken by the defect rather than by noise in the third decimal.

"You ran ten and shipped the best one" — yes, and here is why the headline survives

This table invites a fair objection: ten runs were made, the one that scored highest on select-vera is the one being published, so the published selection score is the maximum of ten draws and is biased upward by construction. That is correct, and it is why the headline number is not a select-vera number.

The populations have different jobs, fixed before the sweep ran:

population	role	may inform selection?
screen-vera, select-vera	screening and ranking the sweep	yes — that is their purpose
report-vera-sealed (138 items, 66 tool-turns)	the reported figure	no — held back, scored once

Seed 46 was chosen on select-vera at 96.58, and that figure is reported here as what it is: the selection score, the maximum of ten, not a performance claim. The published claim is 95.45 on report-vera-sealed against the base's 86.36, measured after selection was already settled, on items that informed no seed or checkpoint choice (they were part of the earlier screen on which training recipes were compared — README §2).

Two consequences worth stating plainly, because they cut against us:

• The sealed figure is lower than the selection figure (95.45 against 96.58), which is the direction selection bias predicts. We report the lower one.
• Seed 42 ties on the sealed population (95.45, the same 63 of 66 tool-turns) while scoring 2 points lower on the selection population. With 66 tool-turns one turn is 1.5 points, so the sealed measurement cannot separate the top two — and the card says so rather than presenting the selection ordering as a ranking.

What a reader cannot check, and should know they cannot. Nine of the ten seeds are not published, and neither are the evaluation suites — they are built from production captures. So every row in the table above except seed 46's rests on our word. The scorer outputs for all of them are kept, and the exact code, data and artifact behind each is recorded, but that record is internal. We state the rows because omitting them while claiming "selected under a loop gate" would be the weaker position, not because a reader can independently verify them.

The runner-up, and why seed 46 was chosen anyway

Seed 42 ties this model on the reported population (95.45, 63 of 66) and on the loop-provocation slice (100.0). It was separated on three other measurements:

	seed 46 (this model)	seed 42	Δ (46 − 42)
report-vera-sealed, temp 0 (66 turns)	95.45	95.45	±0
production temperature 0.35 (146 turns)	97.26	95.21	🟢 +2.05 pts (+2.2 %)
worst response on sealed	6 calls	12 calls	🟢 −6 calls
loop-provocation slice	100.0 (worst 13)	100.0 (worst 10)	±0
select-vera, temp 0 (146 turns)	96.58	never measured	—

Seed 42 has no select-vera temperature-0 run under the published recipe: it was the original arm, scored on screen-vera — 404 items, 212 tool-turns, 92.92 — before the sweep standardised on select-vera's 266 items, so that row cannot be filled with a comparable number and is left empty rather than approximated.

The reported figure alone could not choose between them. Saying so is the point: with 66 tool-turns, one turn is 1.5 points, and presenting a tie as a ranking would be presenting a coin-flip as a measurement. The separation that survives is production temperature, where the two were measured on the same 146 tool-turns.

What these numbers do not establish

• "No loop observed on 266 items" is not "never loops." A dedicated provocation slice, built from turns that actually provoked loops, is the stronger test and is reported separately.
• A gate verdict belongs to the population it was measured on. Seed 45 scored 0.00 % on select-vera and passed, then logged 1.52 % with a 124-call response on report-vera-sealed. "No loop observed on 266 items" is the honest wording, and this is what it was protecting against.
• At production temperature 0.35, none of the three shortlisted seeds looped (worst responses 12–19 calls), against 0.00–1.52 % at temperature 0 on the same items. This is suggestive and not a finding: the base has not been run at 0.35, so whether sampling suppresses the defect or this population simply does not provoke it either way is unmeasured.
• Run-to-run variance of the aggregate score is unbounded on this evidence — 20 items were re-collected, not 266 — so a seed ordering separated by a fraction of a point is not trustworthy. It is sound for a coarse cut.
• One checkpoint, one suite, one quantisation. See "Scope of the claim".