Upload folder using huggingface_hub

README.md

@@ -9,6 +9,7 @@ tags:
 - on-device
 - small-model
 - grammar-constrained-decoding
+- edge
 library_name: pytorch
 model-index:
 - name: thimble-v6
@@ -23,6 +24,16 @@ model-index:
     - type: exact_match
       value: 33.1
       name: Seal-Tools in-domain
+  - task:
+      type: text-generation
+      name: Function calling (ordered strict exact match)
+    dataset:
+      name: Seal-Tools out-of-domain
+      type: seal-tools
+    metrics:
+    - type: exact_match
+      value: 28.1
+      name: Seal-Tools out-of-domain
   - task:
       type: text-generation
       name: Function calling (ordered strict exact match)
@@ -43,78 +54,118 @@ model-index:
     - type: exact_match
       value: 52.5
       name: DroidCall
+  - task:
+      type: text-generation
+      name: Function calling (ordered strict exact match)
+    dataset:
+      name: BFCL v4 single-turn
+      type: bfcl
+    metrics:
+    - type: exact_match
+      value: 23.5
+      name: BFCL v4 single-turn
 ---
 
-# Thimble v6
-
-**Thimble** is a **48.12M-parameter** factorized tool-calling model that beats
-[Needle 2](https://cactuscompute.com/needle) (Cactus Compute, 45M, 153B training
-tokens) on **three of its five published benchmark tables** — including
-Seal-Tools, the suite its evaluation is built around — while training on
-**~1B tokens (150x less)** for a total cost of about $260.
-
-| Suite | Thimble v6 | Needle 2 (45M) |
-|---|---|---|
-| **Seal-Tools in-domain** | **33.1** | 32.6 |
-| **Mobile Actions** | **86.3** | 63.7 |
-| **DroidCall** | **52.5** | 17.0 |
-| Seal-Tools out-of-domain | 28.1 | **28.7** |
-| BFCL v4 single-turn | 23.5 | **42.6** |
-
-Metric: ordered strict exact match — a row passes only if function names, call
-order, and every argument value match exactly. Needle 2's numbers are from
-their published tables. The Seal-in margin (+0.5 on 700 rows) is within
-sampling noise and is stated as measured, not as a decisive gap. Full
-disclosures (parameter-class matching, a champion-selection protocol failure,
-the DroidCall split caveat) live in the
-[GitHub repo](https://github.com/nikshepsvn/tiny-toolcall)'s RESULTS.md.
-
-## How it works
-
-- **Deep-thin gated trunk**: d=448, 20 layers, GQA 8/4, SwiGLU x2.0, QK-norm,
-  sandwich RMSNorm, tied embeddings; 16,384-token BPE with digits as
-  singletons.
-- **Grammar-constrained decoding**: JSON structure and argument keys are
-  force-fed from the tool schemas; the model is consulted at exactly five
-  choice points (refuse, tool name, optional-include, value, stop). Malformed
-  JSON and hallucinated parameter names are unreachable by construction —
-  100% well-formed output.
-- **Plan-conditioned retrieval**: the legal tool-name set is re-retrieved
-  between emitted calls, conditioned on the calls already made.
-- **Error-driven training data**: the v6 corpus was synthesized against the
-  measured failure buckets of the previous version (spurious optional
-  arguments, wrong-slot entity binding, date canonicalization), then
-  **annealed into the learning-rate decay phase** of a continued run — the
-  controlled twin experiment showing anneal-beats-scratch for corrective data
-  is in the repo.
-
-## Files
-
-- `thimble-v6.pt` — the released checkpoint (`v6c_ema`): PyTorch
-  `state_dict` plus config dict (`torch.load(..., weights_only=False)`,
-  keys: `model`, `cfg`).
-- `tokenizer.json` — 16,384-entry BPE vocabulary + merges.
-
-## Usage
-
-Inference requires the grammar decoder from the repo (the checkpoint is a
-plain transformer; the decoding harness is where the guarantees live):
+# 🧵 Thimble
+
+**A 48M-parameter tool-calling model that beats [Needle 2](https://cactuscompute.com/needle)
+on 3 of its 5 published benchmarks — including the one it's named after — with 150× less training data.**
+
+[**GitHub (code, evals, full experimental record)**](https://github.com/nikshepsvn/tiny-toolcall) · MIT · 48.12M params · 11.5MB at 2-bit · $260 total build cost
+
+![Results](results.png)
+
+## TL;DR
+
+| Suite | Thimble v6 | Needle 2 (45M) | |
+|---|---:|---:|---|
+| Seal-Tools in-domain (700) | **33.1** | 32.6 | ✅ their flagship suite |
+| Mobile Actions (961) | **86.3** | 63.7 | ✅ +22.6 |
+| DroidCall (200) | **52.5** | 17.0 | ✅ 3.1× |
+| Well-formed JSON | **100.0** | 93.4 | ✅ by construction |
+| Seal-Tools out-of-domain (654) | 28.1 | **28.7** | ❌ −0.6 |
+| BFCL v4 single-turn (3,641) | 23.5 | **42.6** | ❌ their data moat |
+
+Metric: **ordered strict exact match** — a row passes only if the function names,
+call order, and *every* argument value match. Their metric, their published
+numbers, unmodified. Needle 2 trained on **153B tokens**; Thimble saw **~1B**.
+
+Two things to know before quoting the table: the Seal-in margin (+0.5 on 700
+rows) is within sampling noise and we say so, and the pre-registered model
+selector actually picked a sibling checkpoint that scored worse — the failure is
+diagnosed, both models' results are published, and the full story is in
+[RESULTS.md](https://github.com/nikshepsvn/tiny-toolcall/blob/master/RESULTS.md).
+
+## Why a thimble beats a needle
+
+**1. Tool calling is five decisions, not a generation problem.** A grammar
+compiled from the tool schemas force-feeds all JSON structure — braces, quotes,
+and every argument key. The model is consulted at exactly five choice points:
+*refuse or call · which tool · include this optional? · what value · stop or
+continue*. Malformed JSON, hallucinated parameter names, and calls to
+nonexistent tools are **unreachable, not unlikely**. At 45M parameters, capacity
+spent learning that `{` follows `[` is capacity wasted.
+
+**2. Every training example earns its place.** Row accuracy factors as
+`P(name sequence) × pⁿ`. Each version measured which factor was binding and
+attacked only that. The final data round was synthesized directly against the
+previous model's diagnosed failure buckets — spurious optional arguments,
+wrong-slot entity binding, date canonicalization — with a mid-training causal
+check (+3.3 points at constant LR, attributable to the corrective data alone).
+
+**3. Anneal, don't retrain.** A controlled twin experiment: the corrective
+corpus fed from scratch *diluted* (28.4); the same corpus **annealed into the
+learning-rate decay phase** of a continued run *concentrated* (33.1). The decay
+phase is where a WSD-trained model crystallizes — that's where the good data
+belongs.
+
+## What didn't work (measured, not guessed)
+
+The most reusable part of the project. Each idea was killed by an A/B, not an argument:
+
+| idea | result |
+|---|---|
+| Span-copy heads | −30 pts |
+| Pointer/copy head | −16 pts |
+| Down-weighting grammar-forced tokens (RFT-style) | −12 pts — structure tokens carry call-sequencing signal |
+| From-scratch retrain on corrective data | −4.7 vs annealing |
+| Field-set reranking | −1.4 — training had already fixed its target bucket |
+| Beam / RL / best-of-N | oracle-capped below target |
+| RLOO fine-tune on the annealed checkpoint | diverges at every LR — sharp minima and policy gradients don't mix |
+| Matching Seal's gold numeric typing | not learnable — 74% of params are mixed-convention noise |
+
+## Model details
+
+| | |
+|---|---|
+| Parameters | 48.12M (fp32; ~11.5MB at Needle's own 2-bit standard vs their 14MB) |
+| Architecture | deep-thin gated trunk: d=448, 20 layers, GQA 8/4, SwiGLU ×2.0, QK-norm, sandwich RMSNorm, tied embeddings |
+| Tokenizer | 16,384 BPE, digits as singletons, JSON structural chars as singletons |
+| Context | 768 tokens |
+| Decoding | grammar-constrained, five choice points, plan-conditioned retrieval between calls |
+| Training | Muon (trunk) + AdamW, WSD schedule, EMA, weighted CE matched to the error distribution, decay-phase data annealing |
+
+## Files & usage
+
+- `thimble-v6.pt` — checkpoint (`torch.load(..., weights_only=False)` → `{"model": state_dict, "cfg": dict}`)
+- `tokenizer.json` — BPE vocab + merges
+
+The guarantees live in the decoding harness, so inference goes through the repo:
 
 ```bash
 git clone https://github.com/nikshepsvn/tiny-toolcall
 cd tiny-toolcall && uv venv && uv pip install -e .
-# place thimble-v6.pt in checkpoints/, tokenizer.json in data/
+# put thimble-v6.pt in checkpoints/, tokenizer.json in data/
 python scripts/final_eval.py --ckpt thimble-v6 --suite seal-tools-in
 ```
 
-## Training provenance
+## Integrity
 
-Trained on public function-calling corpora (xlam, ToolACE, Dolci, Glaive,
-official benchmark train splits) plus validated synthetic data
-(deepseek-v4-flash teacher, stepwise-validated, evidence-filtered). Every
-training row passed an 8-gram contamination firewall against every evaluation
-query of every reported suite, BFCL included. Nothing was ever tuned on an
-eval set; the full experimental record — including every negative result —
-is in the repo.
+Public corpora (xlam, ToolACE, Dolci, Glaive, official benchmark train splits)
+plus stepwise-validated, evidence-filtered synthetic data. Every training row
+passed an **8-gram contamination firewall against every evaluation query of
+every reported suite** (BFCL included). Champion selection by held-out dev loss
+only; nothing was ever tuned on an eval set; every negative result is published.
 
-MIT. Built by one person and an AI assistant in about a week of evenings.
+*Built by one person and an AI assistant in about a week of evenings, for about
+the price of a game console. The failures are the useful part.*