docs/harness_research.md

Harness research — what big tech recommends for a small-model "suit" (2026-08-09)

Multi-source digest (OpenAI, Anthropic, Google, Meta, Microsoft, DeepSeek, MIT). Purpose: encode the battle-tested system-layer doctrine behind skills/ tiny-model-harness. Full claim: every rule below cites a source.

1. Agent = model + tools + instructions, with guardrails (OpenAI)

• "A practical guide to building agents" (OpenAI, 2025): an agent is three components — MODEL, TOOLS, INSTRUCTIONS — plus GUARDRAILS. Tools should have standardized definitions and fall into data/action/compute types. Start with ONE agent and add tools; go multi-agent only when it pays. Guardrails are first-class: relevance classifier, safety classifier, PII filter, rules-based protections (blocklists, regex), output validation, and human-in-the-loop.
• Map to FSI: model = TinyLiquid; tools = BM25, web/tor, workspace, helix memory; instructions = SOP library (the Codex-style "task bar" agent.py already implements: plan + ledger + guardrails). MISSING: input guardrails (relevance/safety/injection) — built 2026-08-09 in research/guardrails.py.

2. Simple composable patterns; workflows before agents (Anthropic)

• "Building effective agents" (Anthropic, 2024-12): the most successful implementations use simple, composable patterns, not frameworks. Use deterministic WORKFLOWS (prompt chaining, routing, parallelization, evaluator-optimizer, orchestrator-workers) where the path is predictable; use model-driven AGENTS only where decisions are needed at runtime. Add guardrails (off-topic detection, output validation) and plan for human intervention. "Find the simplest solution and only increase complexity when needed."
• Map: our rule spine (verify.py) + constrained decode + calibration table are deterministic workflows; the head only runs on fall-through. Correct shape.

3. Context engineering: context is a finite resource (Anthropic)

• "Effective context engineering for AI agents" (Anthropic, 2025-09): curate what goes into the window — structure it, retrieve instead of stuffing, compact/summarize old material, dedupe, write-before-you-read, keep the goal visible. Long sessions degrade; the system must manage its own context.
• Map: helix memory + workspace artifacts + SOP blocks are our context curation. GAP: no budget/compaction guard in the TUI for long rabbit-hole sessions — planned next.

4. Calibration: format matters; verbalized confidence is anti-calibrated

• "Language Models (Mostly) Know What They Know" (Anthropic, 2022): models are well-calibrated only in the RIGHT format; smaller models are worse.
• arXiv 2408.11774 (2024): verbalized confidence is less honest than probabilistic confidence; ORCE (2026), Direct Confidence Alignment (2025): verbalized labels need alignment to measured accuracy.
• Rule (applied 2026-08-09): confidence labels are inputs to a CALIBRATION TABLE (research/calibration.py), never trusted directly; the decision spine (research/decision.py) votes with measured per-bucket accuracy and abstains below threshold (selective prediction — conformal selective prediction arXiv 2026; governance-ready SLM recipe 2025).

5. Verification must be EXTERNAL, not self-critique

• "Large Language Models Cannot Self-Correct Reasoning Yet" (arXiv 2310.01798, 2023): intrinsic self-correction fails; feedback must be external.
• "Small Language Models Need Strong Verifiers to Self-Correct" (arXiv 2404.09931, 2024): small models especially need STRONG (external/learned) verifiers, not weak self-critique.
• CRITIC (Microsoft, arXiv 2305.11738): tool-interactive critiquing — search and code execution act as the critic. Chain-of-Verification (Meta, arXiv 2309.09308): draft -> verification questions -> answer independently -> revise.
• Map: deterministic rule spine (research/verify.py) + retrieval = the external critic. Built 2026-08-09: research/verify_loop.py (draft -> plan checks -> retrieve -> deterministic compare -> revise + trace).

6. Self-consistency: sample and vote — but weight it

• Self-Consistency (Google, arXiv 2203.11171): sample N reasoning paths, majority vote beats greedy. Universal SC (2311.08110): no manual CoT needed.
• Rule (applied): keep N small (3-5) on the fall-through path; weight by CALIBRATED reliability (research/decision.py), not naive majority.

7. Multi-agent debate improves factuality (MIT/Google, arXiv 2305.14325)

• Independent agents debate/aggregate; improves factuality and reasoning.
• Map: dual-mind fusion (research/fusion.py) + 4-angle orchestrator (research/orchestrator.py). GAP: fusion raises confidence naively on agreement — replace with calibrated merge (decision.py) when the champion calibration lands.

8. RL with verifiable rewards (DeepSeek-R1, arXiv 2501.12948; Reasoning Gym

arXiv 2025-05)

• Outcome supervision on verifiable problems incentivizes correct reasoning. Reasoning Gym provides a verifier library.
• Map: our constrained verdict space + decision spine are a verifier; RLVR on verdicts is recorded as the training-side unlock (not this week).

9. Distillation: small CAN beat big with the right curriculum

• "Distilling Step-by-Step" (Google, arXiv 2305.02301): CoT distillation lets smaller models OUTPERFORM larger with less data. Supports the KD pipeline (tiny-model-kd) and the handcrafted-gold rule.

10. Training pipeline research — what big tech actually does (2026-08-10)

Multi-source digest for the tiny-model-pipeline skill. Every claim sourced.

1. SmolLM (HF, 2024) — huggingface.co/blog/smollm: curated corpus (Cosmopedia v2 synthetic textbooks/stories by Mixtral + FineWeb-Edu + code), trapezoidal LR with cooldown = 20% of training, 49,152 vocab tokenizer trained on the corpus, embedding tying, context 2048. Instruct: SFT lr 3e-4 (Zephyr-Gemma alignment handbook recipe) then DPO 1 epoch (HelpSteer 135M/1.7B, dpo-mix-7k 360M).
2. SmolLM2 (HF, 2025) — arXiv 2502.02737: 135M on 2T tokens, multi-stage training mixing web/math/code/instruction data; data-mix rates refined at each stage from previous-stage eval (small-scale ablations + manual refinement). Instruct: SFT (SmolTalk) then DPO (UltraFeedback).
3. Zephyr (2023) — arXiv 2310.16944: dSFT then dDPO, few hours, no sampling during fine-tune.
4. Phi-3 (MS, 2024) — arXiv 2404.14219: 3.8B on 3.3T tokens of heavily filtered web + synthetic; data quality is the lever at small scale.
5. DeepSeek-R1 (2025) — arXiv 2501.12948: RL with verifiable rewards (RLVR) incentivizes reasoning without human-annotated traces.
6. LIMA (2023) — arXiv 2305.11206: 1,000 hand-curated examples shape style; supports the handcrafted-gold directive.
7. Forgetting — arXiv 2401.05605 (LoRA still forgets; inverse-linear perf/forgetting tradeoff, not fixable by early stop/rank) and 2502.06042 (injecting ~1%+ pretraining data into the finetune mixture prevents drift/overfit).
8. Liquid LFM2 (2025) — liquid.ai/research/lfm2-technical-report (the architecture family we use): training pipeline = tempered decoupled Top-K distillation objective (avoids support mismatch); curriculum learning with difficulty-ordered data; three-stage post-training = supervised fine-tuning -> length-normalized preference optimization -> model merging. 350M-8.3B, 32K context.

Audit of OUR pipeline vs the recipe (2026-08-10)

MATCH: AdamW (0.9,0.95) wd 0.1 grad-clip 1.0; cosine LR+warmup (GPT-3/LLaMA standard; SmolLM uses trapezoidal+cooldown); BF16; embedding tying; SFT then DPO 1 epoch; handcrafted gold (LIMA/phi-1, better than synthetic here); replay 0.5; curriculum stages; eval gates every stage. GAP: 8k tokenizer (merge artifacts; SmolLM 49k, 16k retrain queued). GAP: model merging after preference (LFM2 3rd stage) — naive averaging failed at 25M; TIES/task-vector untested at 50M. GAP: length-normalized preference optimization (LFM2) — our DPO full-epoch collapsed to "abstain"; length norm / IPO may fix. GAP: RLVR (DeepSeek-R1) — verifier exists (constrained verdicts + decision spine); recorded unlock. GAP: explicit cooldown tail (SmolLM 20%) — cosine covers it; optional.