report.md

Agentic System Report — Making a 0.8B Model Smart with Architecture, not Weights

Status: Working prototype proven. Benchmark puzzle solved correctly (82) in ~3s, fast mode only (no thinking), generalizes to other variants, method externally verified. Audience: Whoever continues development of the agentic UI/runtime (Claude-Code / Codex style) on top of our self-hosted Qwen3.5-0.8B Spaces. Companion file: agentic_demo.py (runnable, self-contained — the exact script behind the results below).

---

1. Goal

We serve a tiny Qwen3.5-0.8B (Q4_K_M GGUF) on llama.cpp behind an OpenAI-compatible proxy (HF Spaces). It is fast but weak: on the benchmark puzzle it answers wrong in both fast and thinking modes. The question: can an agentic architecture (long, multi-step, tool-using, decision-making) make the system smart even though the model is weak — like Codex/Claude Code — ideally without the slow thinking mode?

Answer: Yes. The intelligence is moved out of the model's weights and into the harness (planner + constrained extraction + general tools + verification). Fast mode is enough.

---

2. Infrastructure inventory

Component	Value
Model	Qwen3.5-0.8B.Q4_K_M.gguf (~518 MB, 752M params, n_vocab 248320, n_embd 1024)
Runtime	llama.cpp llama-server (release b9664), CPU-basic (2 vCPU), -t 2, -b 256, --no-mmap, --parallel 1
Context	n_ctx default 32768 (native train 262144 via N_CTX)
Output cap	MAX_TOKENS = -1 (unlimited) by default
Modes	thinking:"on" / thinking:"off" (a.k.a. enable_thinking) — controlled via request param, NOT a /no_think text token
Reasoning	--reasoning-format auto → chain-of-thought lands in reasoning_content, answer stays clean
Tool calling	Embedded in chat_template.jinja (<tools> block) — emits clean tool_calls JSON
Endpoints (Spaces)	https://p2test2-test.hf.space/v1, https://anon334test-test11.hf.space/v1 (more to be added)
Web search (verifier)	https://s09ais09ai9s-s9a0js9ajs9ajsjkjkj.hf.space — SearXNG-compatible: GET /search?q=...&format=json → {query, results:[{url,title,content}], answers, ...}

2.1 Correct usage notes (gotchas discovered)

• Disable thinking with the param, not text. Sending /no_think in the message body does nothing here; the custom chat template only honors thinking:"off" / enable_thinking:false. Forgetting this leaves thinking ON, which eats the max_tokens budget on reasoning_content and the structured answer never gets emitted (we saw finish_reason=length, empty content).
• Always set max_tokens per request in agent steps. Do not rely on -1 (unlimited) inside a loop — a runaway generation will fill the context and stall.
• response_format: json_schema works (pass-through to llama.cpp). With thinking off + a small schema, routing/classification is reliable and fast.

---

3. The benchmark problem & ground truth

Draw a regular hexagon and connect every pair of vertices except one. The pair you don't connect are not on opposite sides of the hexagon but along a shorter diagonal. How many triangles of any size are in this figure?

Correct answer: 82. Single-shot 0.8B (fast) answered "4"; thinking also wrong.

Independent computation (our count_triangles) confirms:

Figure	Triangles
K6 full (all 15 chords)	110
Hexagon minus one SHORT diagonal (dist 2)	82
Hexagon minus one LONG/opposite diagonal (dist 3)	76

82 is identical for all short diagonals (symmetry) → it is a genuine general result, not luck.

---

4. Experiments & findings (the path to the design)

These are the empirical lessons that shaped the architecture. Read this section before changing the design.

1. Tool-calling & JSON-schema work cleanly on this model (fast mode), so it can be the brain of an agent loop. Format reliability is good.
2. "Search-first / easy path" does NOT find the answer. We tried 4 reasonable queries; 82 never appears — the exact variant is too niche. Only the full hexagon (110) is well-published. → A strong planner is not one that assumes the cheap path exists; it is one that escalates when the cheap path fails and verifies.
3. Long generations are SLOW and time out. Asking the model to write a full geometry script (~1500 tokens) exceeded 120s on CPU and timed out. Short generations (50–150 tokens) return in 3–15s. → Keep model outputs tiny. Never make the weak model write long algorithms.
4. The weak model fails at GENERATING structure, even with self-reflection. When asked to emit the exact set of drawn/removed pairs, it misread "connect every pair except one" and excluded the 6 sides (→ 56). A reflection/critique step detected "this is wrong" (ok=false) but it could not produce the correct fix and looped on the same error. → Reflection alone does not rescue a model that cannot generate the right structure.
5. The weak model SUCCEEDS at constrained CLASSIFICATION. Reframed as a tiny multiple-choice form — {n, num_removed, removed_type∈{SIDE,SHORT_DIAGONAL,LONG_OR_OPPOSITE,NONE}} — it answered correctly in ~3–4s (fast mode). Deterministic code then built the figure and the general tool counted → 82. This is the breakthrough.
6. It generalizes (not overfit). The same pipeline solved 4 variants correctly; the counting method matches the published sequence 1, 8, 35, 110, 287, … (triangles from all diagonals of a regular n-gon) for n=5 (35) and n=6 (110).
7. Endpoint pool needs failover, not blind round-robin. Because --parallel 1 keeps a Space busy generating even after a client timeout, round-robin sent the next call into the stuck endpoint. Fix: rotate, short timeouts, cooldown/skip busy endpoints, prefer idle ones. This is exactly why multiple Spaces exist — treat them as a pool.

---

5. Final architecture (universal pattern)

                    ┌────────────────────────────────────────────────┐
   user task  ─────▶│  PLANNER (⚡fast, constrained)                  │
                    │  classify domain + cheapest viable path         │
                    └───────────────┬────────────────────────────────┘
                                    │  route by domain
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
  ⚡ CONSTRAINED EXTRACTION    🔎 WEB SEARCH (retrieval)    ⚡ other extractors
  (tiny JSON form / enum)      (facts / known results)     (calculator inputs, …)
        │                           │
        ▼                           │
  🔧 DETERMINISTIC ASSEMBLY         │   ← code, not the model
  (build exact structure)          │
        │                           │
        ▼                           │
  🔧 GENERAL TOOL (compute)         │   ← e.g. count_triangles(), calculator, code-exec
        │                           │
        └───────────┬───────────────┘
                    ▼
            🔎 VERIFIER (cross-check)
            compare result vs an independently-known fact
            (e.g. published sequence 35,110) and/or a 2nd method
                    │
                    ▼
                ✅ answer

Mode policy: everything in fast (non-thinking) mode. Thinking mode is NOT required and is avoided for latency. (If a future hard step ever needs it, gate it narrowly.)

Why it works: the model only ever does what it is good at — short, constrained classification. All long/precise work is done by deterministic code and general tools. Verification guards against the model's residual judgment errors.

---

6. Validated results (from agentic_demo.py, fast mode only)

==============================================================================
AGENTIC DEMO  --  fast mode only (no thinking)
==============================================================================
[    4s] got=  82  truth=  82  PASS   form={'n': 6, 'num_removed': 1, 'removed_type': 'SHORT_DIAGONAL'}
[    6s] got= 110  truth= 110  PASS   form={'n': 6, 'num_removed': 0, 'removed_type': 'NONE'}
[    9s] got=  76  truth=  76  PASS   form={'n': 6, 'num_removed': 1, 'removed_type': 'LONG_OR_OPPOSITE'}
[   11s] got=  35  truth=  35  PASS   form={'n': 5, 'num_removed': 0, 'removed_type': 'NONE'}
------------------------------------------------------------------------------
4/4 passed in 11s (~2.7s/task, no thinking mode)
method verified vs published sequence (…35,110…): True

Benchmark puzzle → 82, in ~4s, fast mode, no hardcoding.

---

7. Universal design principles (reusable beyond this puzzle)

1. Classify, don't generate. Give the weak model constrained choices (JSON-schema enums). It is reliable at picking; it is unreliable at authoring structures or long text.
2. Offload heavy logic to general tools. Coordinates, intersections, counting, arithmetic, code execution — deterministic and reusable for any instance, not one puzzle.
3. Planner = cheapest path first + escalate on failure + verification gate. Do not assume the easy path exists; detect when it fails and switch.
4. Verify against something independently knowable. Cross-check a computed result with a known fact (published value) or a second independent method. Search is the verifier, not always the answerer.
5. Keep model outputs tiny. Long generations are slow on CPU and error-prone. Tools produce the bulk; the model emits a few tokens.
6. Treat Spaces as a failover pool. Health-check / cooldown, prefer idle endpoints, one in-flight request per Space (CPU is small). Add more Spaces to scale throughput.

---

8. Known constraints & operational notes

• CPU latency: ~3–5s for tiny fast-mode calls; >120s for ~1500-token generations (avoid).
• Busy-endpoint trap: a timed-out request keeps the Space generating; that endpoint stays busy. Use cooldowns and prefer idle Spaces.
• thinking:"off" is mandatory for speed and for the structured-output budget; /no_think text is ignored.
• Security: the HF token was shared in plaintext during development — rotate/regenerate it.
• Generality scope: the {n, num_removed, removed_type} form is specific to polygon figure-counting. The universal part is the pattern (planner → constrained extraction → general tool → verify). New domains = new extractor schema + new tool ("skills").

---

9. Roadmap / next steps (for the agentic UI Space)

1. Package the harness as a small runtime: endpoint pool (health-check + failover), planner → router → extractor(s) → tool(s) → verifier, fast mode default, tiny per-step outputs.
2. Add a tool/skill registry with per-domain extractors:
   • math/arithmetic → calculator / sympy
   • counting/geometry → count_triangles and friends (generalize to regions, intersections)
   • facts/lookup → web search
   • general compute → sandboxed code-exec (short outputs only)
3. Planner routing that picks domain + cheapest path, with an escalation policy.
4. Verification layer standardized: numeric cross-check, second-method agreement, search anchor.
5. Benchmark suite (diverse tasks) to measure accuracy and latency vs single-shot, and to guard against regressions / overfitting.
6. Scale-out: register N Spaces in the pool; one in-flight per Space; round-robin among idle ones.

---

10. Files

• report.md — this document.
• agentic_demo.py — the exact, runnable script that produced Section 6 (fast mode only). It contains the endpoint pool, the general geometry tools, the constrained-extraction solver, the generality tests, and the web-search method verifier.

---

11. Generalization stress-test & the planner ceiling (UPDATE)

After the polygon result, we stress-tested whether the same harness generalizes to very different queries. It does not, as originally built — and the reason is important for the next phase. This section documents the experiment, the precise failure points, the diagnosis, and the recommended fix.

11.1 Test queries (3 different "shapes" of problem)

#	Query	Correct answer	Natural tool
Q1	Regular hexagon, connect all pairs except one shorter diagonal — triangles of any size?	82	COMPUTE (polygon skill)
Q2	"Today is December 2 2012. In a few weeks something will happen that hasn't happened since 1987. What is it?"	a year with no repeating digits (1987 → 2013)	WEB_SEARCH (known riddle)
Q3	"siapa nama tionghoa hary tanoe" (Hary Tanoe's Chinese name)	陈明立 / Chen Mingli	WEB_SEARCH (fact)

11.2 What happened (a general router over a tool registry, fast mode)

We built a general router: planner picks a tool ∈ {WEB_SEARCH, POLYGON_TRIANGLES, CODE}, then the tool runs. Results:

Query	Tool chosen	Output	Failure
Q1	CODE ❌ (should be POLYGON)	"12"	routing wrong
Q2	WEB_SEARCH ✓	"2012 phenomenon" ❌	bad search query (no reformulation)
Q3	WEB_SEARCH ✓	"Hary Tanoesoedibjo" ❌	extraction wrong (missed 陈明立)

A "v2" with binary routing + multi-query search was worse: it mis-classified Q1 as non-computational, and produced junk reformulations ("The Great Wall of China"; "Hary Tanoe is a Thai name…").

Thinking mode for the planner (route + reformulate + extract): 65–131 s per query and returned EMPTY content — the chain-of-thought consumed the whole token budget before any answer (same finish_reason=length trap). Not viable.

11.3 Isolation test — is it retrieval or cognition?

We fed the extractor good search results (the queries a strong planner would have written) and asked the fast 0.8B to extract:

• Q3 (Hary Tanoe): with good results pooled → still answered "Hary Tanoesoedibjo" (his own name), not 陈明立.
• Q2 (riddle): the snippet literally said "Between the years 1987–2013, there was no single year comprised [of all different digits]" → the model answered "2013" (grabbed the wrong span; missed the concept "no repeating digits").

→ Even with correct retrieval, the fast 0.8B reads and synthesizes results poorly.

11.4 Diagnosis — the architecture is right; the 0.8B's open cognition is the ceiling

The pattern (planner → tool registry → verify) is correct and does solve the polygon class. What breaks on Q2/Q3 is open-ended cognition, in three distinct places — all judgment, not output format (the JSON was always well-formed):

1. Routing nuance — choosing the right tool when categories overlap (Q1 → CODE instead of POLYGON).
2. Query crafting — turning a question/riddle into a good search query (inferring "2013", translating "nama tionghoa" → "Chinese name").
3. Reading & synthesizing results into the intended answer (picking 陈明立 / "no repeating digits").

Fast mode cannot fix these (capability ceiling). Thinking mode is too slow and empties the output.

What the 0.8B can do reliably: constrained classification + filling a tiny, explicit form (this is exactly why the polygon skill works).

Design rule going forward:

Every decision the 0.8B makes must be reducible to a tiny, explicit, constrained choice. Anything that needs open synthesis (free query writing, multilingual reading, lateral reasoning) is beyond the 0.8B and must be handled by a stronger planner or a pre-built deterministic skill.

11.5 Recommended solution — two-tier: strong planner + fast workers ⭐

This is what Codex / Claude Code effectively are: a capable "brain" orchestrating cheaper actions.

        ┌─────────────────────────────────────────────┐
        │  TIER-1 PLANNER  (capable model: 3B–8B / API) │
        │  routing · query crafting · reading results · │
        │  multi-step decisions · verification          │
        └───────────────┬───────────────────────────────┘
                        │ delegates mechanical sub-tasks
        ┌───────────────▼───────────────────────────────┐
        │  TIER-2 WORKERS  (fleet of 0.8B Spaces, fast)  │
        │  constrained classification · form-filling ·   │
        │  extraction over a SMALL span · bulk parallel  │
        │  steps · tool argument formatting              │
        └────────────────────────────────────────────────┘
                        │ tools
        WEB_SEARCH · POLYGON_TRIANGLES · CODE-EXEC · (more skills…)

• Q1 → planner routes to the polygon skill (already works) → 82.
• Q2 → planner reasons "a few weeks after Dec 2012 = 2013", crafts query, reads the snippet → "no repeating digits".
• Q3 → planner translates intent, crafts "Hary Tanoesoedibjo Chinese name", reads 陈明立 → Chen Mingli.

Latency stays low: the planner is one short call; the 0.8B fleet does the high-volume work.

Alternative (0.8B-only) — limited generality

Keep everything on the 0.8B only if every step is a constrained scaffold (per-skill form builders, templated query builders per intent, constrained-span extraction). This generalizes only to the skills you pre-build — not to arbitrary open questions. Use this if a stronger model is not available, and grow the skill library over time.

11.6 Updated roadmap

1. Add a Tier-1 planner model to the pool (a 3B–8B Space, or a hosted API) — highest-leverage change.
2. Keep the 0.8B fleet as Tier-2 workers (pool with health-check + failover; one in-flight per Space).
3. Tool/skill registry: WEB_SEARCH (with planner-crafted queries), POLYGON_TRIANGLES, CODE-EXEC, and a growing set of constrained skills. Each skill = description + constrained input schema + deterministic executor.
4. Standard verification: numeric cross-check, second-method agreement, search anchor (e.g. the published triangle sequence 1,8,35,110,…).
5. Benchmark suite spanning compute / lookup / riddle / multilingual queries to measure accuracy and latency, and to guard against regressions and overfitting.

11.7 Bottom line

• The agentic architecture is sound and general; the polygon class is fully solved, fast, no thinking.
• A 0.8B alone cannot be the planner for open-ended questions — proven on Q2/Q3 across fast mode, binary routing, thinking mode, and an isolation test.
• Fix: a two-tier system — a stronger planner brain over the fast 0.8B worker fleet — makes the same architecture answer all three query types correctly while staying fast.