| --- |
| license: cc-by-4.0 |
| language: |
| - en |
| tags: |
| - research-paper |
| - tool-calling |
| - function-calling |
| - agentic |
| - mixture-of-experts |
| - diagnosis |
| --- |
| |
| # Tool-Name Binding in Small Language Models Is Generative, Not Copy-Grounded: A Counterfactual Diagnosis and a Data-Side Fix |
|
|
| **Wei Ciao Wu** (independent researcher), with Claude (Anthropic) as AI research assistant. |
|
|
| 📄 **[Read the paper (PDF)](binding-diagnosis.pdf)** · LaTeX source included · 13 pages |
|
|
| Preprint, 2026-07. Part of the **circus-0.2** research line (1B-parameter from-scratch bilingual MoE). |
|
|
| ## TL;DR |
|
|
| Reliable tool calling requires **binding**: emitting a tool name byte-identical to an entry in the in-context schema. A 1B from-scratch MoE (circus-0.2) plateaus at ~59% held-out binding despite a 2.93B-token agentic mid-train, while emission is healthy. Counterfactual probes localize the failure completely: |
|
|
| - Given **gold reasoning**, the model copies the correct name 98% (125/128), fixing 25/25 previously failed cases; in free rollouts the wrong name always appears **first in the model's own chain-of-thought** (25/25) and is then copied faithfully. |
| - Failed names are semantic paraphrases (`check_email_validity` for `email_validate_regex`): the model **generates names from tool semantics and parametric memory instead of copying from the schema**. An architecture control rules out the NoPE/SWA attention stack (91% exact copy under worst-case distractors). |
| - A renamed-schema evaluation decomposes the base 59% into ~47% genuine copying + ~12% *convention collision* (invented canonical-style names that happen to match). |
| - **Data-side fix — name randomization**: consistently renaming a fraction of training tools to semantically unpredictable identifiers makes generation-from-semantics score zero, forcing the copy circuit. Renamed-schema binding rises 47% → 62% → 69% with dose (control 53%), then **saturates**: a 5× token scale-up holds at 69%. The ceiling is **emission-bound, not copy-bound** — valid-tool rate is capped by ~80% call-emission, while copy discipline *conditioned on emission* reaches ~86% (control 76%). Out-of-distribution zero-shot binding is the one dose-responsive axis (renamed-OOD 30% → 66% on a 100-prompt 4-domain follow-up), climbing sharply then plateauing. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{wu2026bindingdiagnosis, |
| title = {Tool-Name Binding in Small Language Models Is Generative, Not |
| Copy-Grounded: A Counterfactual Diagnosis and a Data-Side Fix}, |
| author = {Wei Ciao Wu}, |
| year = {2026}, |
| note = {Preprint. https://huggingface.co/wcamon/circus-0.2-binding-diagnosis} |
| } |
| ``` |
|
|
| ## Related |
|
|
| - Companion paper: [Memory as an Additive Side-Path (STEM niches, combine mechanisms, learned depth allocation)](https://huggingface.co/wcamon/circus-0.2-stem-architecture) |
|
|
