# Appendix: the tool-call loop defect, and the seed sweep against it

*Every number here is measured and its artifact is kept.*

## What the defect is

A turn ends with the model emitting the same tool call over and over instead of
stopping — sometimes the identical call, sometimes the same tool with varied
arguments. The worst example we recorded ran **313 consecutive tool calls** in a
single turn. In production this is not a quality problem, it is an availability
one: the turn never terminates, and it consumes the budget of whatever called it.

## Scope of the claim — read this before quoting any number here

Everything below was measured on **one checkpoint**:
`mistralai/Ministral-3-8B-Instruct-2512-BF16` at revision
`f6fae9795746f63c9be8344932f01275f3c63734`, served as Q8_0 GGUF through
llama.cpp `c812c543f`, on **our** production-derived suite.

**This is not a claim about "Mistral models".** A family name predicts neither the
behaviour nor the licence of its next release, and this project has an expensive
worked example: `Ministral-8B-Instruct-2410` is research-licensed while
`Ministral-3-8B-Instruct-2512` is `apache-2.0` — same product name, opposite
answers. We have not measured any other Mistral checkpoint, so we say nothing
about them. What we can say is that **our pinned base exhibits the defect**, and
that fine-tuning moves the rate in both directions depending on nothing but the
seed.

## The base already loops

| population | loop rate | worst response |
|---|---:|---:|
| **full faithful set (1,133 tool-turns, 1,965 items)** | **🔴 0.09 %** (1 turn) | **111 calls** |
| `select-vera` (146 tool-turns) | 🔴 0.68 % (1 turn) | 111 calls |
| `screen-vera` (212 tool-turns) | 🔴 0.47 % (1 turn) | 111 calls |

**The first row is the number to quote**, and all three rows are the same single
event seen through different denominators — one looping turn, 111 calls, a single
tool name repeated 108 times. On 146 tool-turns that one turn is 0.68 %; on 1,133
it is 0.09 %. Nothing about the base changed between those measurements, and
quoting the largest of them because it is the most convenient would be reporting a
sample size as a finding.

The scan completed 2026-10-02 over **1,965 of 1,965 collectable items** (Q8_0,
temperature 0, cap 4096), and its scorer output is kept at
`benchmarks/base-scans/faithful-all-q8-cap4096/scores.json`. The name-repeat
histogram is the useful shape: 1,059 turns repeat a tool name once, 52 call no
tool, a thin tail reaches 10 — and then one turn sits at **108**, with nothing in
between. **The defect is rare and catastrophic, not a gradient**, which is why a
threshold-based gate works at all and why an aggregate rate understates it: the
cost of the one turn is an unterminated production request, not 0.09 % of a
quality score.

So the defect is **not introduced by fine-tuning**. That matters for reading
everything below: a trained arm with a non-zero rate has not necessarily been
damaged, and an arm at zero has genuinely *removed* something the base does.

## How a loop is detected

Three independent readings, any one of which condemns a turn:

| rule | threshold | what it catches |
|---|---:|---|
| `maxIdenticalCallRepeat` | ≥ 8 | the identical `(name, arguments)` signature repeated |
| `toolCallCount` | ≥ 200 | a runaway trajectory of any shape |
| `maxToolNameRepeat` | ≥ 24 | the same tool NAME repeated, **arguments ignored** |

The third rule exists because a model that loops while varying one argument field
looks like progress to a signature-based check and is the same failure.

**The name threshold was calibrated against production traffic, not guessed.**
At 24 it is deliberately generous: a stricter setting of 12 condemns a real
production turn that called `web_search` 15 times with distinct `site:` queries —
legitimate work. A detector that cannot tell a loop from a thorough search is
worse than none, because it teaches you to distrust it.

## The sweep

Ten runs, seeds 42–51. **Identical** dataset (`l2-3f301662871f`), recipe (LoRA
rank 8, dropout 0.05, scale 1.0, attention projections only, lr 2e-5 cosine),
checkpoint (150), quantisation and scoring population. The seed is the only
difference.

A seed is not a quantity — it is the initial state of a pseudo-random generator,
and here it controls three things: LoRA `A` initialisation, every dropout mask,
and data order. We checked that consecutive seeds are not secretly similar:
across MLX's RNG and numpy's permutation, mean |r| between adjacent seeds (0.0037
and 0.0181) sits at or below the noise floor and is no higher than between
distant values like 1009 and 999983. **42–51 samples the same distribution that
scattered values would.**

## Results

`validFirstToolMatchPct` on `select-vera`, 146 tool-turns, cap 4096, temperature 0.

| seed | score | Δ vs base | loop rate | worst response | gate |
|---|---:|---:|---:|---:|---|
| base (untrained) | 80.82 ‡ | — | 🔴 0.68 % | 272 | — |
| **46** | **96.58** | **🟢 +15.76 pts (+19.5 %)** | **0.00 %** | 13 | **passes** |
| 48 | 96.58 | 🟢 +15.76 pts (+19.5 %) | 🔴 0.68 % | 116 | excluded |
| 43 | 95.21 | 🟢 +14.39 pts (+17.8 %) | 🔴 0.68 % | 292 | excluded |
| 42 | 92.92 † | not comparable | 0.00 % | 13 | passes |
| 45 | 93.15 | 🟢 +12.33 pts (+15.3 %) | 0.00 % | — | passes |
| 50 | 93.15 | 🟢 +12.33 pts (+15.3 %) | 🔴 1.37 % | — | excluded |
| 49 | 92.47 | 🟢 +11.65 pts (+14.4 %) | 🔴 5.48 % | — | excluded |
| 44 | 91.10 | 🟢 +10.28 pts (+12.7 %) | 🔴 6.85 % | 313 | excluded |
| 47 | 90.41 | 🟢 +9.59 pts (+11.9 %) | 0.00 % | — | passes |
| 51 | 94.52 | 🟢 +13.70 pts (+17.0 %) | 0.00 % | 15 | passes |

‡ The base was collected directly on these 266 items at the same cap and
temperature, three times with identical results; this is the figure the card
quotes. (Re-scoring the base's 404-item screen responses on this population gives
74.66 instead — a re-score of a different run, not a measurement of this one.)

† **Seed 42 is not measured on this population.** It was the original arm and was
scored on `screen-vera` — 404 items, 212 tool-turns — before the sweep standardised
on `select-vera`'s 266, so 92.92 is its screen-vera figure and is **not comparable
to the rest of this column**.

Seed 42's artifact-backed figures are: `report-vera-sealed` **95.45** (66 turns,
tying seed 46), production temperature 0.35 **95.21** (146 turns), and the
loop-provocation slice **100.0**. Those are the numbers to quote for it.

**Five of the ten seeds loop at least once.** The rate ranges from 0.00 %
to 6.85 % — ten of 146 tool-turns — from one recipe on one dataset. This is real
variance in what gets trained, not measurement noise: the loop VERDICT agreed
20/20 across two collections of the same artifact (see "Is it the sampler?").

## Is it the sampler, or the model?

The screen is **not** byte-deterministic. Re-collecting the 20 busiest items
against the same artifact gave 14/20 identical tool-call sequences and 19/20
identical text; one item ran 119 calls one time and 158 the next.

That is the right alarm to raise and it does **not** make the gate unsound,
because the gate is a **binary** per-item verdict and a 119-call and a 158-call
runaway are the same verdict. Measured per item with the real scorer, over the
same 20 items:

| fact | agreement |
|---|---|
| `looped` | **20/20** |
| `firstToolMatch` | 20/20 |
| `validFirstToolMatch` | 20/20 |

Those 20 were chosen as the **busiest** — adversarially, where instability
compounds — which makes the agreement stronger evidence than a random sample.
**So the difference between seed 42's zero and seed 44's 6.85 % is a property of
the models, not of the sampling.**

## The gate, and what it cost

Looping is a **gate**, not a ranking dimension. Any non-zero rate excludes the
seed outright; survivors are then ranked on score. There is no ranking *within*
non-zero, because 3 loops versus 4 is noise while zero versus non-zero is not.

For most of the sweep this was free — the best score happened also to be clean.
Then seed 48 scored **96.58, identical to seed 46**, 141 of 146 tool-turns for
both, and loops once. It is excluded and seed 46 ships. Two models
indistinguishable on the ranking metric, and the tie broken by the defect rather
than by noise in the third decimal.

## "You ran ten and shipped the best one" — yes, and here is why the headline survives

This table invites a fair objection: ten runs were made, the one that scored
highest on `select-vera` is the one being published, so the published selection
score is the maximum of ten draws and is biased upward by construction. That is
correct, and it is why **the headline number is not a `select-vera` number.**

The populations have different jobs, fixed before the sweep ran:

| population | role | may inform selection? |
|---|---|---|
| `screen-vera`, `select-vera` | screening and ranking the sweep | **yes — that is their purpose** |
| `report-vera-sealed` (138 items, 66 tool-turns) | the reported figure | **no — held back, scored once** |

Seed 46 was chosen on `select-vera` at **96.58**, and that figure is reported here
as what it is: the selection score, the maximum of ten, not a performance claim.
The published claim is **95.45 on `report-vera-sealed` against the base's 86.36**,
measured after selection was already settled, on items that informed no seed or
checkpoint choice (they were part of the earlier screen on which training recipes
were compared — README §2).

Two consequences worth stating plainly, because they cut against us:

- **The sealed figure is lower than the selection figure** (95.45 against 96.58),
  which is the direction selection bias predicts. We report the lower one.
- **Seed 42 ties on the sealed population** (95.45, the same 63 of 66 tool-turns)
  while scoring 2 points lower on the selection population. With 66 tool-turns one
  turn is 1.5 points, so the sealed measurement cannot separate the top two — and
  the card says so rather than presenting the selection ordering as a ranking.

**What a reader cannot check, and should know they cannot.** Nine of the ten seeds
are not published, and neither are the evaluation suites — they are built from
production captures. So every row in the table above except seed 46's rests on our
word. The scorer outputs for all of them are kept, and the exact code, data and
artifact behind each is recorded, but that record is internal. We state the rows
because omitting them while claiming "selected under a loop gate" would be the
weaker position, not because a reader can independently verify them.

## The runner-up, and why seed 46 was chosen anyway

Seed 42 ties this model on the reported population (95.45, 63 of 66) and on the
loop-provocation slice (100.0). It was separated on three other measurements:

| | seed 46 (this model) | seed 42 | Δ (46 − 42) |
|---|---:|---:|---:|
| `report-vera-sealed`, temp 0 (66 turns) | 95.45 | 95.45 | ±0 |
| **production temperature 0.35** (146 turns) | **97.26** | 95.21 | **🟢 +2.05 pts (+2.2 %)** |
| worst response on sealed | **6 calls** | 12 calls | 🟢 −6 calls |
| loop-provocation slice | 100.0 (worst 13) | 100.0 (worst 10) | ±0 |
| `select-vera`, temp 0 (146 turns) | **96.58** | **never measured** | — |

Seed 42 has no `select-vera` temperature-0 run under the published recipe: it was
the original arm, scored on `screen-vera` — 404 items, 212 tool-turns, **92.92** —
before the sweep standardised on `select-vera`'s 266 items, so that row cannot be
filled with a comparable number and is left empty rather than approximated.

**The reported figure alone could not choose between them.** Saying so is the
point: with 66 tool-turns, one turn is 1.5 points, and presenting a tie as a
ranking would be presenting a coin-flip as a measurement. The separation that
survives is production temperature, where the two were measured on the same 146
tool-turns.

## What these numbers do not establish

- **"No loop observed on 266 items" is not "never loops."** A dedicated
  provocation slice, built from turns that actually provoked loops, is the
  stronger test and is reported separately.
- **A gate verdict belongs to the population it was measured on.** Seed 45 scored
  0.00 % on `select-vera` and passed, then logged **1.52 % with a 124-call
  response** on `report-vera-sealed`. "No loop observed on 266 items" is the
  honest wording, and this is what it was protecting against.
- **At production temperature 0.35, none of the three shortlisted seeds looped**
  (worst responses 12–19 calls), against 0.00–1.52 % at temperature 0 on the same
  items. This is suggestive and **not a finding**: the base has not been run at
  0.35, so whether sampling suppresses the defect or this population simply does
  not provoke it either way is unmeasured.
- **Run-to-run variance of the aggregate score is unbounded** on this evidence —
  20 items were re-collected, not 266 — so a seed ordering separated by a
  fraction of a point is not trustworthy. It is sound for a coarse cut.
- **One checkpoint, one suite, one quantisation.** See "Scope of the claim".