wcamon/circus-0.2-binding-diagnosis

Tool-Name Binding in Small Language Models Is Generative, Not Copy-Grounded: A Counterfactual Diagnosis and a Data-Side Fix

Wei Ciao Wu (independent researcher), with Claude (Anthropic) as AI research assistant.

📄 Read the paper (PDF) · LaTeX source included · 13 pages

Preprint, 2026-07. Part of the circus-0.2 research line (1B-parameter from-scratch bilingual MoE).

TL;DR

Reliable tool calling requires binding: emitting a tool name byte-identical to an entry in the in-context schema. A 1B from-scratch MoE (circus-0.2) plateaus at ~59% held-out binding despite a 2.93B-token agentic mid-train, while emission is healthy. Counterfactual probes localize the failure completely:

• Given gold reasoning, the model copies the correct name 98% (125/128), fixing 25/25 previously failed cases; in free rollouts the wrong name always appears first in the model's own chain-of-thought (25/25) and is then copied faithfully.
• Failed names are semantic paraphrases (check_email_validity for email_validate_regex): the model generates names from tool semantics and parametric memory instead of copying from the schema. An architecture control rules out the NoPE/SWA attention stack (91% exact copy under worst-case distractors).
• A renamed-schema evaluation decomposes the base 59% into ~47% genuine copying + ~12% convention collision (invented canonical-style names that happen to match).
• Data-side fix — name randomization: consistently renaming a fraction of training tools to semantically unpredictable identifiers makes generation-from-semantics score zero, forcing the copy circuit. Renamed-schema binding rises 47% → 62% → 69% with dose (control 53%), then saturates: a 5× token scale-up holds at 69%. The ceiling is emission-bound, not copy-bound — valid-tool rate is capped by ~80% call-emission, while copy discipline conditioned on emission reaches ~86% (control 76%). Out-of-distribution zero-shot binding is the one dose-responsive axis (renamed-OOD 30% → 66% on a 100-prompt 4-domain follow-up), climbing sharply then plateauing.

Citation

@misc{wu2026bindingdiagnosis,
  title  = {Tool-Name Binding in Small Language Models Is Generative, Not
            Copy-Grounded: A Counterfactual Diagnosis and a Data-Side Fix},
  author = {Wei Ciao Wu},
  year   = {2026},
  note   = {Preprint. https://huggingface.co/wcamon/circus-0.2-binding-diagnosis}
}

Related

• Companion paper: Memory as an Additive Side-Path (STEM niches, combine mechanisms, learned depth allocation)