Documentation update: Claude comparisons move to APPENDIX-claude-comparison.md; training-data, held-out and lineage notes corrected; Ollama download command updated for huggingface_hub 2.x; adapter_config.json without local paths
3a3d1b3 verified | # Appendix: the tool-call loop defect, and the seed sweep against it | |
| *Every number here is measured and its artifact is kept.* | |
| ## What the defect is | |
| A turn ends with the model emitting the same tool call over and over instead of | |
| stopping β sometimes the identical call, sometimes the same tool with varied | |
| arguments. The worst example we recorded ran **313 consecutive tool calls** in a | |
| single turn. In production this is not a quality problem, it is an availability | |
| one: the turn never terminates, and it consumes the budget of whatever called it. | |
| ## Scope of the claim β read this before quoting any number here | |
| Everything below was measured on **one checkpoint**: | |
| `mistralai/Ministral-3-8B-Instruct-2512-BF16` at revision | |
| `f6fae9795746f63c9be8344932f01275f3c63734`, served as Q8_0 GGUF through | |
| llama.cpp `c812c543f`, on **our** production-derived suite. | |
| **This is not a claim about "Mistral models".** A family name predicts neither the | |
| behaviour nor the licence of its next release, and this project has an expensive | |
| worked example: `Ministral-8B-Instruct-2410` is research-licensed while | |
| `Ministral-3-8B-Instruct-2512` is `apache-2.0` β same product name, opposite | |
| answers. We have not measured any other Mistral checkpoint, so we say nothing | |
| about them. What we can say is that **our pinned base exhibits the defect**, and | |
| that fine-tuning moves the rate in both directions depending on nothing but the | |
| seed. | |
| ## The base already loops | |
| | population | loop rate | worst response | | |
| |---|---:|---:| | |
| | **full faithful set (1,133 tool-turns, 1,965 items)** | **π΄ 0.09 %** (1 turn) | **111 calls** | | |
| | `select-vera` (146 tool-turns) | π΄ 0.68 % (1 turn) | 111 calls | | |
| | `screen-vera` (212 tool-turns) | π΄ 0.47 % (1 turn) | 111 calls | | |
| **The first row is the number to quote**, and all three rows are the same single | |
| event seen through different denominators β one looping turn, 111 calls, a single | |
| tool name repeated 108 times. On 146 tool-turns that one turn is 0.68 %; on 1,133 | |
| it is 0.09 %. Nothing about the base changed between those measurements, and | |
| quoting the largest of them because it is the most convenient would be reporting a | |
| sample size as a finding. | |
| The scan completed 2026-10-02 over **1,965 of 1,965 collectable items** (Q8_0, | |
| temperature 0, cap 4096), and its scorer output is kept at | |
| `benchmarks/base-scans/faithful-all-q8-cap4096/scores.json`. The name-repeat | |
| histogram is the useful shape: 1,059 turns repeat a tool name once, 52 call no | |
| tool, a thin tail reaches 10 β and then one turn sits at **108**, with nothing in | |
| between. **The defect is rare and catastrophic, not a gradient**, which is why a | |
| threshold-based gate works at all and why an aggregate rate understates it: the | |
| cost of the one turn is an unterminated production request, not 0.09 % of a | |
| quality score. | |
| So the defect is **not introduced by fine-tuning**. That matters for reading | |
| everything below: a trained arm with a non-zero rate has not necessarily been | |
| damaged, and an arm at zero has genuinely *removed* something the base does. | |
| ## How a loop is detected | |
| Three independent readings, any one of which condemns a turn: | |
| | rule | threshold | what it catches | | |
| |---|---:|---| | |
| | `maxIdenticalCallRepeat` | β₯ 8 | the identical `(name, arguments)` signature repeated | | |
| | `toolCallCount` | β₯ 200 | a runaway trajectory of any shape | | |
| | `maxToolNameRepeat` | β₯ 24 | the same tool NAME repeated, **arguments ignored** | | |
| The third rule exists because a model that loops while varying one argument field | |
| looks like progress to a signature-based check and is the same failure. | |
| **The name threshold was calibrated against production traffic, not guessed.** | |
| At 24 it is deliberately generous: a stricter setting of 12 condemns a real | |
| production turn that called `web_search` 15 times with distinct `site:` queries β | |
| legitimate work. A detector that cannot tell a loop from a thorough search is | |
| worse than none, because it teaches you to distrust it. | |
| ## The sweep | |
| Ten runs, seeds 42β51. **Identical** dataset (`l2-3f301662871f`), recipe (LoRA | |
| rank 8, dropout 0.05, scale 1.0, attention projections only, lr 2e-5 cosine), | |
| checkpoint (150), quantisation and scoring population. The seed is the only | |
| difference. | |
| A seed is not a quantity β it is the initial state of a pseudo-random generator, | |
| and here it controls three things: LoRA `A` initialisation, every dropout mask, | |
| and data order. We checked that consecutive seeds are not secretly similar: | |
| across MLX's RNG and numpy's permutation, mean |r| between adjacent seeds (0.0037 | |
| and 0.0181) sits at or below the noise floor and is no higher than between | |
| distant values like 1009 and 999983. **42β51 samples the same distribution that | |
| scattered values would.** | |
| ## Results | |
| `validFirstToolMatchPct` on `select-vera`, 146 tool-turns, cap 4096, temperature 0. | |
| | seed | score | Ξ vs base | loop rate | worst response | gate | | |
| |---|---:|---:|---:|---:|---| | |
| | base (untrained) | 80.82 β‘ | β | π΄ 0.68 % | 272 | β | | |
| | **46** | **96.58** | **π’ +15.76 pts (+19.5 %)** | **0.00 %** | 13 | **passes** | | |
| | 48 | 96.58 | π’ +15.76 pts (+19.5 %) | π΄ 0.68 % | 116 | excluded | | |
| | 43 | 95.21 | π’ +14.39 pts (+17.8 %) | π΄ 0.68 % | 292 | excluded | | |
| | 42 | 92.92 β | not comparable | 0.00 % | 13 | passes | | |
| | 45 | 93.15 | π’ +12.33 pts (+15.3 %) | 0.00 % | β | passes | | |
| | 50 | 93.15 | π’ +12.33 pts (+15.3 %) | π΄ 1.37 % | β | excluded | | |
| | 49 | 92.47 | π’ +11.65 pts (+14.4 %) | π΄ 5.48 % | β | excluded | | |
| | 44 | 91.10 | π’ +10.28 pts (+12.7 %) | π΄ 6.85 % | 313 | excluded | | |
| | 47 | 90.41 | π’ +9.59 pts (+11.9 %) | 0.00 % | β | passes | | |
| | 51 | 94.52 | π’ +13.70 pts (+17.0 %) | 0.00 % | 15 | passes | | |
| β‘ The base was collected directly on these 266 items at the same cap and | |
| temperature, three times with identical results; this is the figure the card | |
| quotes. (Re-scoring the base's 404-item screen responses on this population gives | |
| 74.66 instead β a re-score of a different run, not a measurement of this one.) | |
| β **Seed 42 is not measured on this population.** It was the original arm and was | |
| scored on `screen-vera` β 404 items, 212 tool-turns β before the sweep standardised | |
| on `select-vera`'s 266, so 92.92 is its screen-vera figure and is **not comparable | |
| to the rest of this column**. | |
| Seed 42's artifact-backed figures are: `report-vera-sealed` **95.45** (66 turns, | |
| tying seed 46), production temperature 0.35 **95.21** (146 turns), and the | |
| loop-provocation slice **100.0**. Those are the numbers to quote for it. | |
| **Five of the ten seeds loop at least once.** The rate ranges from 0.00 % | |
| to 6.85 % β ten of 146 tool-turns β from one recipe on one dataset. This is real | |
| variance in what gets trained, not measurement noise: the loop VERDICT agreed | |
| 20/20 across two collections of the same artifact (see "Is it the sampler?"). | |
| ## Is it the sampler, or the model? | |
| The screen is **not** byte-deterministic. Re-collecting the 20 busiest items | |
| against the same artifact gave 14/20 identical tool-call sequences and 19/20 | |
| identical text; one item ran 119 calls one time and 158 the next. | |
| That is the right alarm to raise and it does **not** make the gate unsound, | |
| because the gate is a **binary** per-item verdict and a 119-call and a 158-call | |
| runaway are the same verdict. Measured per item with the real scorer, over the | |
| same 20 items: | |
| | fact | agreement | | |
| |---|---| | |
| | `looped` | **20/20** | | |
| | `firstToolMatch` | 20/20 | | |
| | `validFirstToolMatch` | 20/20 | | |
| Those 20 were chosen as the **busiest** β adversarially, where instability | |
| compounds β which makes the agreement stronger evidence than a random sample. | |
| **So the difference between seed 42's zero and seed 44's 6.85 % is a property of | |
| the models, not of the sampling.** | |
| ## The gate, and what it cost | |
| Looping is a **gate**, not a ranking dimension. Any non-zero rate excludes the | |
| seed outright; survivors are then ranked on score. There is no ranking *within* | |
| non-zero, because 3 loops versus 4 is noise while zero versus non-zero is not. | |
| For most of the sweep this was free β the best score happened also to be clean. | |
| Then seed 48 scored **96.58, identical to seed 46**, 141 of 146 tool-turns for | |
| both, and loops once. It is excluded and seed 46 ships. Two models | |
| indistinguishable on the ranking metric, and the tie broken by the defect rather | |
| than by noise in the third decimal. | |
| ## "You ran ten and shipped the best one" β yes, and here is why the headline survives | |
| This table invites a fair objection: ten runs were made, the one that scored | |
| highest on `select-vera` is the one being published, so the published selection | |
| score is the maximum of ten draws and is biased upward by construction. That is | |
| correct, and it is why **the headline number is not a `select-vera` number.** | |
| The populations have different jobs, fixed before the sweep ran: | |
| | population | role | may inform selection? | | |
| |---|---|---| | |
| | `screen-vera`, `select-vera` | screening and ranking the sweep | **yes β that is their purpose** | | |
| | `report-vera-sealed` (138 items, 66 tool-turns) | the reported figure | **no β held back, scored once** | | |
| Seed 46 was chosen on `select-vera` at **96.58**, and that figure is reported here | |
| as what it is: the selection score, the maximum of ten, not a performance claim. | |
| The published claim is **95.45 on `report-vera-sealed` against the base's 86.36**, | |
| measured after selection was already settled, on items that informed no seed or | |
| checkpoint choice (they were part of the earlier screen on which training recipes | |
| were compared β README Β§2). | |
| Two consequences worth stating plainly, because they cut against us: | |
| - **The sealed figure is lower than the selection figure** (95.45 against 96.58), | |
| which is the direction selection bias predicts. We report the lower one. | |
| - **Seed 42 ties on the sealed population** (95.45, the same 63 of 66 tool-turns) | |
| while scoring 2 points lower on the selection population. With 66 tool-turns one | |
| turn is 1.5 points, so the sealed measurement cannot separate the top two β and | |
| the card says so rather than presenting the selection ordering as a ranking. | |
| **What a reader cannot check, and should know they cannot.** Nine of the ten seeds | |
| are not published, and neither are the evaluation suites β they are built from | |
| production captures. So every row in the table above except seed 46's rests on our | |
| word. The scorer outputs for all of them are kept, and the exact code, data and | |
| artifact behind each is recorded, but that record is internal. We state the rows | |
| because omitting them while claiming "selected under a loop gate" would be the | |
| weaker position, not because a reader can independently verify them. | |
| ## The runner-up, and why seed 46 was chosen anyway | |
| Seed 42 ties this model on the reported population (95.45, 63 of 66) and on the | |
| loop-provocation slice (100.0). It was separated on three other measurements: | |
| | | seed 46 (this model) | seed 42 | Ξ (46 β 42) | | |
| |---|---:|---:|---:| | |
| | `report-vera-sealed`, temp 0 (66 turns) | 95.45 | 95.45 | Β±0 | | |
| | **production temperature 0.35** (146 turns) | **97.26** | 95.21 | **π’ +2.05 pts (+2.2 %)** | | |
| | worst response on sealed | **6 calls** | 12 calls | π’ β6 calls | | |
| | loop-provocation slice | 100.0 (worst 13) | 100.0 (worst 10) | Β±0 | | |
| | `select-vera`, temp 0 (146 turns) | **96.58** | **never measured** | β | | |
| Seed 42 has no `select-vera` temperature-0 run under the published recipe: it was | |
| the original arm, scored on `screen-vera` β 404 items, 212 tool-turns, **92.92** β | |
| before the sweep standardised on `select-vera`'s 266 items, so that row cannot be | |
| filled with a comparable number and is left empty rather than approximated. | |
| **The reported figure alone could not choose between them.** Saying so is the | |
| point: with 66 tool-turns, one turn is 1.5 points, and presenting a tie as a | |
| ranking would be presenting a coin-flip as a measurement. The separation that | |
| survives is production temperature, where the two were measured on the same 146 | |
| tool-turns. | |
| ## What these numbers do not establish | |
| - **"No loop observed on 266 items" is not "never loops."** A dedicated | |
| provocation slice, built from turns that actually provoked loops, is the | |
| stronger test and is reported separately. | |
| - **A gate verdict belongs to the population it was measured on.** Seed 45 scored | |
| 0.00 % on `select-vera` and passed, then logged **1.52 % with a 124-call | |
| response** on `report-vera-sealed`. "No loop observed on 266 items" is the | |
| honest wording, and this is what it was protecting against. | |
| - **At production temperature 0.35, none of the three shortlisted seeds looped** | |
| (worst responses 12β19 calls), against 0.00β1.52 % at temperature 0 on the same | |
| items. This is suggestive and **not a finding**: the base has not been run at | |
| 0.35, so whether sampling suppresses the defect or this population simply does | |
| not provoke it either way is unmeasured. | |
| - **Run-to-run variance of the aggregate score is unbounded** on this evidence β | |
| 20 items were re-collected, not 266 β so a seed ordering separated by a | |
| fraction of a point is not trustworthy. It is sound for a coarse cut. | |
| - **One checkpoint, one suite, one quantisation.** See "Scope of the claim". | |