Update report: generalization stress-test, planner ceiling, two-tier solution

report.md

@@ -209,3 +209,124 @@ method verified vs published sequence (…35,110…): True
 - `agentic_demo.py` — the exact, runnable script that produced Section 6 (fast mode only). It
   contains the endpoint pool, the general geometry tools, the constrained-extraction solver, the
   generality tests, and the web-search method verifier.
+
+
+---
+
+# 11. Generalization stress-test & the planner ceiling (UPDATE)
+
+After the polygon result, we stress-tested whether the **same** harness generalizes to very
+different queries. **It does not, as originally built** — and the reason is important for the next
+phase. This section documents the experiment, the precise failure points, the diagnosis, and the
+recommended fix.
+
+## 11.1 Test queries (3 different "shapes" of problem)
+
+| # | Query | Correct answer | Natural tool |
+|---|---|---|---|
+| Q1 | Regular hexagon, connect all pairs except one shorter diagonal — triangles of any size? | **82** | COMPUTE (polygon skill) |
+| Q2 | "Today is December 2 2012. In a few weeks something will happen that hasn't happened since 1987. What is it?" | **a year with no repeating digits** (1987 → 2013) | WEB_SEARCH (known riddle) |
+| Q3 | "siapa nama tionghoa hary tanoe" (Hary Tanoe's Chinese name) | **陈明立 / Chen Mingli** | WEB_SEARCH (fact) |
+
+## 11.2 What happened (a general router over a tool registry, fast mode)
+
+We built a general router: planner picks a tool ∈ {WEB_SEARCH, POLYGON_TRIANGLES, CODE}, then the
+tool runs. Results:
+
+| Query | Tool chosen | Output | Failure |
+|---|---|---|---|
+| Q1 | CODE ❌ (should be POLYGON) | "12" | **routing wrong** |
+| Q2 | WEB_SEARCH ✓ | "2012 phenomenon" ❌ | **bad search query** (no reformulation) |
+| Q3 | WEB_SEARCH ✓ | "Hary Tanoesoedibjo" ❌ | **extraction wrong** (missed 陈明立) |
+
+A "v2" with **binary** routing + multi-query search was **worse**: it mis-classified Q1 as
+non-computational, and produced junk reformulations ("The Great Wall of China"; "Hary Tanoe is a
+Thai name…").
+
+**Thinking mode for the planner** (route + reformulate + extract): **65–131 s per query and returned
+EMPTY content** — the chain-of-thought consumed the whole token budget before any answer (same
+`finish_reason=length` trap). Not viable.
+
+## 11.3 Isolation test — is it retrieval or cognition?
+
+We fed the extractor **good** search results (the queries a strong planner *would* have written) and
+asked the **fast** 0.8B to extract:
+
+- Q3 (Hary Tanoe): with good results pooled → still answered **"Hary Tanoesoedibjo"** (his own name), not 陈明立.
+- Q2 (riddle): the snippet literally said *"Between the years 1987–2013, there was no single year
+  comprised [of all different digits]"* → the model answered **"2013"** (grabbed the wrong span;
+  missed the concept "no repeating digits").
+
+→ Even with correct retrieval, the fast 0.8B **reads and synthesizes results poorly**.
+
+## 11.4 Diagnosis — the architecture is right; the 0.8B's open cognition is the ceiling
+
+The pattern (planner → tool registry → verify) is correct and *does* solve the polygon class.
+What breaks on Q2/Q3 is **open-ended cognition**, in three distinct places — all **judgment, not
+output format** (the JSON was always well-formed):
+
+1. **Routing nuance** — choosing the right tool when categories overlap (Q1 → CODE instead of POLYGON).
+2. **Query crafting** — turning a question/riddle into a good search query (inferring "2013", translating "nama tionghoa" → "Chinese name").
+3. **Reading & synthesizing** results into the intended answer (picking 陈明立 / "no repeating digits").
+
+Fast mode cannot fix these (capability ceiling). Thinking mode is too slow and empties the output.
+
+**What the 0.8B *can* do reliably:** constrained classification + filling a tiny, explicit form
+(this is exactly why the polygon skill works). 
+
+**Design rule going forward:**
+> Every decision the 0.8B makes must be reducible to a tiny, explicit, constrained choice.
+> Anything that needs open synthesis (free query writing, multilingual reading, lateral reasoning)
+> is beyond the 0.8B and must be handled by a stronger planner or a pre-built deterministic skill.
+
+## 11.5 Recommended solution — two-tier: strong planner + fast workers ⭐
+
+This is what Codex / Claude Code effectively are: a capable "brain" orchestrating cheaper actions.
+
+```
+        ┌─────────────────────────────────────────────┐
+        │  TIER-1 PLANNER  (capable model: 3B–8B / API) │
+        │  routing · query crafting · reading results · │
+        │  multi-step decisions · verification          │
+        └───────────────┬───────────────────────────────┘
+                        │ delegates mechanical sub-tasks
+        ┌───────────────▼──────────────────────────��────┐
+        │  TIER-2 WORKERS  (fleet of 0.8B Spaces, fast)  │
+        │  constrained classification · form-filling ·   │
+        │  extraction over a SMALL span · bulk parallel  │
+        │  steps · tool argument formatting              │
+        └────────────────────────────────────────────────┘
+                        │ tools
+        WEB_SEARCH · POLYGON_TRIANGLES · CODE-EXEC · (more skills…)
+```
+
+- **Q1** → planner routes to the polygon skill (already works) → 82.
+- **Q2** → planner reasons "a few weeks after Dec 2012 = 2013", crafts query, reads the snippet → "no repeating digits".
+- **Q3** → planner translates intent, crafts "Hary Tanoesoedibjo Chinese name", reads 陈明立 → Chen Mingli.
+
+Latency stays low: the planner is **one short call**; the 0.8B fleet does the high-volume work.
+
+### Alternative (0.8B-only) — limited generality
+Keep everything on the 0.8B **only if** every step is a constrained scaffold (per-skill form
+builders, *templated* query builders per intent, constrained-span extraction). This generalizes
+**only to the skills you pre-build** — not to arbitrary open questions. Use this if a stronger model
+is not available, and grow the skill library over time.
+
+## 11.6 Updated roadmap
+
+1. **Add a Tier-1 planner model** to the pool (a 3B–8B Space, or a hosted API) — highest-leverage change.
+2. Keep the **0.8B fleet as Tier-2 workers** (pool with health-check + failover; one in-flight per Space).
+3. **Tool/skill registry**: WEB_SEARCH (with planner-crafted queries), POLYGON_TRIANGLES, CODE-EXEC,
+   and a growing set of constrained skills. Each skill = description + constrained input schema + deterministic executor.
+4. **Standard verification**: numeric cross-check, second-method agreement, search anchor (e.g. the
+   published triangle sequence 1,8,35,110,…).
+5. **Benchmark suite** spanning compute / lookup / riddle / multilingual queries to measure accuracy
+   and latency, and to guard against regressions and overfitting.
+
+## 11.7 Bottom line
+
+- The agentic **architecture is sound and general**; the polygon class is fully solved, fast, no thinking.
+- A **0.8B alone cannot be the planner** for open-ended questions — proven on Q2/Q3 across fast mode,
+  binary routing, thinking mode, and an isolation test.
+- **Fix:** a **two-tier** system — a stronger planner brain over the fast 0.8B worker fleet — makes the
+  *same* architecture answer all three query types correctly while staying fast.