pico-type / walkthrough.md

Upload walkthrough.md with huggingface_hub

db5fa1a verified 10 days ago

28 kB

pico-type — Walkthrough

Living document. Every agent/harness picking up this project should start here. Also, you must keep updating as you go further.

1. What this is

pico-type is a tiny (~1.5M params), byte-level, multi-head content classifier. Input: up to 1024 raw bytes (clipboard text, file bytes, image header, etc.). Output: structured label set in one forward pass.

Built per the locked-in plan in docs/PLAN.md (recovered from the original opencode session — see §11).

Why this exists

Existing clipboard tools are regex-only (ClipGate, 13 types) or LLM-powered (needs Ollama, GB-scale). Existing tiny classifiers do one job. No model does all of them in one sub-5MB forward pass with a multi-head output.

Deliverables

HuggingFace model (Apache-2.0) with 4 Matryoshka tiers ✅ (ONNX exported)
Python CLI (picotype) ✅
Gradio Space app ✅ (gradio_app.py — label lists fixed Jun 18)
Python MCP server ✅ (model/pico_type/mcp_server.py)
Rust MCP server ✅ (crates/picotype-mcp/)
pytest smoke tests ✅ (tests/test_smoke.py)
HF model card ✅ (MODEL_CARD.md)
Badge'd README ✅
Rust CLI (crates/picotype/) ✅
Chrome extension scaffold + icons ✅ (MV3)
PyPI package pico-type v0.1.3 ✅ (pip install pico-type)
arXiv paper — in progress (draft with final numbers)
Raycast/Alfred/VSCode extensions — pending

2. Locked-in architecture (from `docs/PLAN.md` §2)

Inputs (≤1024 UTF-8 bytes, masked/padded)
  │
  ├─ ByteEmbed (256 → 96d, learned)
  │
  ├─ 3× Conv1D block (kernel 3, 5, 7) + GELU + residual  → 192d
  │
  ├─ 2× BiAttention block (d=192, 4 heads, RoPE θ=500k)
  │
  ├─ Pool = [mean ‖ max ‖ std]  → 576d shared trunk
  │
  └─ 7 Matryoshka heads (Linear at 16/64/192/576 dim slices)
       ├─ h_coarse        (12)   — primary type
       ├─ h_modality      (8)    — textual / binary-image / …
       ├─ h_subtype       (24)   — JSON/YAML/CSV/HTML…  (if coarse ∈ {config, markup, data})
       ├─ h_code_lang     (62)   — if coarse=code, + "undetected" fallback
       ├─ h_text_lang     (30)   — if coarse=text, + "undetected" fallback
       ├─ h_file_mime     (90)   — if coarse ∈ {image, file, archive, binary} or modality=binary_*, + "undetected"
       └─ h_risk          (6)    — sigmoid multi-label: api_key, jwt, ssh_key, password, email, phone

Tier matrix

Tier	Dim slice	Params (actual)	INT8 size
`pico-type-tiny`	16	1.43M	1.37 MB
`pico-type-small`	64	1.45M	1.38 MB
`pico-type-base`	192	1.48M	1.41 MB
`pico-type-pro`	576	1.56M	1.49 MB

Actual sizes came in under the plan's targets (0.5/1.5/3.5/8 MB INT8). Trunk dominates; if we need to shrink tiny further, reduce trunk_dim 192→128 or num_attn_layers 2→1.

Output schema (always returned, with confidence)

{
  "coarse": "code",
  "modality": "textual",
  "subtype": null,
  "code_language": "python",
  "text_language": null,
  "file_mime": null,
  "risk_flags": [],
  "confidence": 0.94,
  "modality_confidence": 0.91,
  "model_tier": "base"
}

3. Repo layout

classifier-model/
├── .venv/                          # Python 3.11 venv (torch, numpy<2, safetensors, pyyaml)
├── .git/
├── checkpoints/                    # best.pt, ONNX models, eval results
├── docs/
│   └── PLAN.md                    # Full architecture plan
├── gradio_app.py                  # Gradio Space app
├── model/
│   └── pico_type/
│       ├── __init__.py            # re-exports public API
│       ├── labels.py              # vocabularies + decode_output
│       ├── arch.py                # PicoType model
│       ├── data.py                # synthetic generator
│       ├── train.py               # multi-task trainer
│       ├── eval.py                # eval harness
│       ├── distill.py             # KD pipeline
│       ├── export.py              # ONNX export
│       ├── cli.py                 # Python CLI (picotype)
│       └── mcp_server.py          # MCP server (stdio)
├── tests/
│   └── test_smoke.py              # pytest smoke tests
├── spaces/
│   └── requirements.txt           # HF Space dependencies
├── MODEL_CARD.md                  # HF model card
├── README.md                      # badge'd README
├── LICENSE                        # Apache-2.0
├── pyproject.toml
└── walkthrough.md                 # this file

4. Public API (current)

from model.pico_type import (
    PicoType, PicoTypeConfig, TIERS,            # model
    COARSE_LABELS, MODALITY_LABELS, SUBTYPE_LABELS,
    CODE_LANG_LABELS, TEXT_LANG_LABELS,
    FILE_MIME_LABELS, RISK_LABELS,
    UNDETECTED, decode_output,                  # labels + decoder
)
from model.pico_type.arch import encode_bytes   # bytes → (tokens, mask) tensors

Smoke test

source .venv/bin/activate
python -m model.pico_type.arch

5. Key invariants (DO NOT break these)

These are tied to the locked-in plan and the on-disk checkpoints. Changing them means re-training + re-publishing.

Vocab sizes (asserted in `labels.py`)

Head	Size	Plan target
coarse	12	12 ✓
modality	8	8 ✓
subtype	24	24 ✓
code_lang	62	62 ✓
text_lang	30	30 ✓
file_mime	90	90 ✓
risk	6	6 ✓

Gating (enforced in `decode_output`, must also be enforced in training loss)

subtype is only valid when coarse ∈ {config, markup, data}
code_lang is only valid when coarse == code
text_lang is only valid when coarse == text
file_mime is only valid when coarse ∈ {image, file, archive, binary} or modality starts with binary_
risk is always valid (multi-label)

`UNDETECTED` behavior

For code_lang / text_lang / file_mime: the model has N logits (62/30/90), no separate "undetected" class. If max_softmax < undetected_threshold (default 0.4), the decoder returns "undetected" instead of the argmax label.
For risk: per-class sigmoid; class is flagged if sigmoid(logit) ≥ risk_threshold (default 0.5).

Matryoshka slicing

The shared trunk emits a 576d vector. Each MatryoshkaHead slices x[..., :tier_dim] then applies its tier-specific nn.Linear(tier_dim, num_classes). All 4 tier linears live in the model (so a single checkpoint contains all tiers); at inference, only the chosen tier's linears are loaded — parameter_count(tier) reflects this.

Byte input

0 is the pad byte (matches the 0th row of the embedding)
max_bytes default 1024
Inputs longer than max_bytes are truncated (not rejected)
Mask is 1 for real bytes, 0 for pad — passed to attention and pool

6. Environment

# venv already created at .venv
source .venv/bin/activate
python --version  # 3.11
python -c "import torch, numpy, safetensors, yaml; print('ok')"

Python 3.14 was tried first but has no torch wheels — we use Python 3.11. This is documented because the failed attempt is in the opencode session history.

7. What's done ✅

Training, Deployment & Publishing

Training: 1700 synthetic steps → 5000 synthetic steps (eval_loss 1.61) → 4000 real-data fine-tuning steps (2 rounds: 10 samples/lang, then 25 samples/lang from GitHub). Eval loss 1.9848 after final round. best.pt updated.
Eval results (1000 samples, base tier, 5.6ms):
- coarse 100%, modality 100%, subtype 98.4%, code_lang 53.9%, text_lang 100%, file_mime 100%, risk mAP 100%
- code_lang improved +9.9% (43.96% → 53.85%) via real GitHub code samples
ONNX export: All 4 tiers re-exported from best.pt (~9 MB each, self-contained single file, opset 18, external_data=False).
HF Model: huggingface.co/eulogik/pico-type — ONNX models at root level + checkpoints/ directory, model card, paper scaffold.
HF Space: huggingface.co/spaces/eulogik/pico-type — Gradio app label lists fixed (Jun 18 2026). Root cause: gradio_app.py had different label ordering than labels.py (text_lang had "ar","hi" instead of "id","ms"; file_mime was completely different set). Also fixed np.bool_→bool for NumPy 2.x compat (Space runs Python 3.13).
PyPI: pico-type v0.1.3 published at https://pypi.org/project/pico-type/0.1.3/.
GitHub: github.com/eulogik/pico-type — main branch, CI passes.

Known Training Issues

MPS OOM: batch_size 64 causes MPS OOM (19+ GiB allocated). Fixed by reducing to batch_size=16 and train_tiers=('base',).
MPS graph cache: Writes to system /tmp, was filling disk when free space <1GB. ~9GB now available, OK.
code_lang stuck at ~44% on synthetic-only: Real-data fine-tuning pushed it to 53.9% with 25 samples/lang from GitHub. Further improvement likely needs more diverse real data (error messages, SQL, HTML, binary headers).

HF Space fix (Jun 18)

Root cause: gradio_app.py had different label list ordering than model/pico_type/labels.py. Model outputs numeric logits at specific indices, but Space app decoded them with wrong label lists:
- text_lang: had "ar","hi" at positions 26-27 instead of "id","ms"
- file_mime: completely different set and ordering (started with "text/html" instead of "application/pdf")
Also fixed: np.bool_→bool (removed in NumPy 2.x, needed for Space's Python 3.13)
Also fixed: Removed unused pathlib import, removed numpy<2 pin from Space requirements
New ONNX files copied to model repo root level (Space downloads from root, not checkpoints/ directory)

`model/pico_type/labels.py`

All 7 vocabularies (sizes match plan exactly, asserted at import time)
decode_output(logits, tier, undetected_threshold, risk_threshold) — respects all gating rules, applies UNDETECTED threshold
HEAD_NUM_CLASSES dict for heads that need to query class counts
label_for(head, idx) helper
Constants: UNDETECTED, ALL_HEADS, SUBTYPE_GATED_BY, CODE_LANG_GATED_BY, TEXT_LANG_GATED_BY, FILE_MIME_GATED_BY

`model/pico_type/arch.py`

PicoTypeConfig dataclass with all hyperparams
ByteEmbed — nn.Embedding(256, 96) init normal std=0.02
ConvBlock — Conv1d → LayerNorm → GELU → Dropout, residual via 1×1 projection when dims change
RotaryPosEmb — precomputed cos/sin cache, auto-grows if seq exceeds cache
AttnBlock — pre-norm, fused QKV, RoPE on Q/K, F.scaled_dot_product_attention, MLP w/ 4× expansion
Pool — mean ‖ max ‖ std over masked positions (handles padding correctly)
MatryoshkaHead — nn.ModuleDict of nn.Linear per tier
PicoType — top-level model
encode_bytes(data, max_len, pad) — bytes → (LongTensor[B, L], LongTensor[B, L])
smoke_test() — instantiates model, runs a forward, returns param counts
__main__ block runs the smoke test
NaN fix in AttnBlock: F.scaled_dot_product_attention with a boolean mask where all entries are False (sample has no padding) produces NaN on CPU. Fixed by converting to float (-inf for masked positions, 0 for valid) and guarding with mask.all().item(). See AttnBlock.forward for the guard.

`model/pico_type/data.py`

SyntheticGenerator(seed) — generates one balanced sample at a time from 11 buckets: code, text, config, markup, data, link, error, image, file, secret, archive, binary
11 generator methods (_gen_code, _gen_text, _gen_config, _gen_markup, _gen_data, _gen_link, _gen_error, _gen_image, _gen_file, _gen_secret, _gen_archive, _gen_binary)
Sample dataclass with data: bytes, label fields (int for single-label heads, list[int] for risk), IGNORE_INDEX = -100 for gated heads
SyntheticDataset(generator, size) — wraps generator for DataLoader compatibility
Code templates for all 62 languages across 18 syntax groups (Python-like, C-like, JS-like, Lisp-like, etc.) — uses re.sub with ${kind} placeholders
Word lists for all 30 text languages
Binary magic-byte headers for PDF, ZIP, GZIP, ELF, SQLite, Parquet, TIFF, PNG, JPEG, WASM, DEB, TTF, plus archive formats (7z, RAR, TAR, XZ, BZ2)
_detect_risk runs on text samples (AWS key, JWT, SSH key, password detection)
label_counts() returns class distribution for debugging
smoke_test() generates 500 samples and prints coverage per head

`crates/picotype-mcp/` (new)

Rust MCP server crate implementing JSON-RPC 2.0 over stdio transport
Same protocol as Python MCP server: initialize handshake + tools/call for classify and classify_file
Uses ort (ONNX Runtime) for inference, respects PICOTYPE_MODEL_DIR and PICOTYPE_TIER env vars
Tested and working: builds in ~30s, passes initialize and tools/call requests

`extensions/chrome/icons/` (new)

16x16, 48x48, 128x128 PNG icons with indigo rounded-square design and "P" letter
Resolves missing icons referenced in manifest.json

Training update (synthetic)

Continued training from scratch (no prior checkpoints available) for 5000 steps
Config: lr=5e-4, warmup=200, batch_size=16 (MPS), train_size=20000, eval_size=500
Eval loss improved: 31.97 → 28.52 (step 0) → 1.97 (step 2500) → 1.61 (step 5000)
Per-head results (step 5000, synthetic-only): coarse=100%, modality=100%, subtype=98%, code_lang=44%, text_lang=97.5%, file_mime=100%, risk=100%
code_lang stuck at ~44% — synthetic code templates too limited; confirmed need for real data
Saved best.pt, final.pt, step_{500,1000,...,4500}.pt in checkpoints/

Real-data fine-tuning

model/pico_type/realdata.py — fetches real source code from GitHub Search API per language
model/pico_type/train_real.py — mixed real+synthetic training (30% real), lr=3e-4, batch=16
Round 1 (10/lang, 2000 steps): code_lang 44%→43.96% (marginal)
Round 2 (25/lang, ~1500 samples, 2000 steps): code_lang 43.96%→53.85% ✅
Final eval (1000 samples, best.pt): coarse=100%, modality=100%, subtype=98.4%, code_lang=53.9%, text_lang=100%, file_mime=100%, risk=100%, inference=5.6ms
5 languages with 0 GitHub results: fsharp, lisp, vim, fortran, vb (unsupported search names)

Real-data fine-tuning (new)

model/pico_type/realdata.py — fetches real source code from GitHub Search API per language, builds mixed datasets
model/pico_type/train_real.py — fine-tuning loop mixing real code + synthetic data (30% ratio), lower LR (3e-4)
Round 1: 10 samples/language (423 total), 2000 steps — code_lang 41.7%→43.96%
Round 2: 25 samples/language (~1500 total), 2000 steps — code_lang 43.96%→53.85%
GitHub token used, 0.5s delay/request to respect rate limits
5 languages returned 0 samples: fsharp, lisp, vim, fortran, vb (unsupported GitHub search names)

Real-world eval

14 hand-curated samples across 10 coarse categories
Overall accuracy: 71.4% (10/14)
Correct: python, js, JSON, text, link, password, AWS key, SSH key, ZIP, bash
Wrong: SQL (→text), HTML (→code), error trace (→text), PNG header (→config)
Confirms synthetic data gap: SQL, HTML, errors, and binary headers need real training data

PyPI publish

Bumped version from 0.1.0→0.1.2 (0.1.1 already existed)
Built and uploaded to PyPI: pico-type==0.1.2 at https://pypi.org/project/pico-type/0.1.2/
v0.1.3: Published with best.pt (53.9% code_lang), updated eval results, paper scaffold

`model/pico_type/init.py`

Re-exports the public API (already present in the repo when we recovered)

`model/pico_type/train.py`

TrainConfig dataclass — lr, warmup, total_steps, batch_size, grad_clip, per-head weights, etc.
collate_fn(batch) — pads variable-length samples, creates input_ids, attention_mask, labels dict
MultiTaskLoss(weights) — CE per head (ignore_index=-100 for gated heads) + BCE for risk. Skips any head with zero valid labels in batch (returns 0.0). Applies per-head weights (coarse=3.0, modality=2.0, code_lang=1.5, text_lang=1.5, others=1.0).
get_lr(step, config) — linear warmup → cosine decay
train(config) — full training loop:
- SyntheticGenerator + SyntheticDataset for train/eval
- AdamW, separate param groups (trunk w/ weight_decay, Matryoshka heads w/o)
- BF16 AMP (CUDA) or FP32 (CPU/MPS)
- Gradient clipping at 1.0
- Logs every log_every steps, eval every eval_every, save every save_every
- Saves best.pt (lowest eval loss), final.pt, plus periodic step_{N}.pt
load_checkpoint(path, model, optimizer) — loads state dict
Known issues fixed: NaN in SDP with all-valid mask (use float -inf instead of boolean mask); NaN from CE on all-ignore labels (skip head); Python 3.14 has no torch wheels (use 3.11)

`model/pico_type/eval.py`

EvalConfig dataclass — checkpoint, tier, eval_size, batch_size
evaluate(config) — generates synthetic eval set, runs forward pass for all 7 heads
Per-head HeadMetrics: accuracy, per-class precision/recall/F1, confusion matrix
RiskMetrics: per-class average precision (sklearn-free implementation)
run_eval() — CLI: python -m model.pico_type.eval --eval-size 1000 --checkpoint checkpoints/best.pt
_average_precision(y_true, y_scores) — area under PR curve via trapezoidal rule

Hindi support + v0.1.4 (Jun 18)

Replaced ms (Malay, index 29) with hi (Hindi) in TEXT_LANG_LABELS. Added Hindi word list to _TEXT_WORDS in data.py.
Fine-tuned 800 steps — Hindi text now correctly detected as text_lang=hi.
code_lang improved to 61.3% on synthetic eval after Hindi fine-tune (unexpected side benefit).
Published as v0.1.4 at https://pypi.org/project/pico-type/0.1.4/

Diverse synthetic training + real-world eval improvement (Jun 18)

Created diverse generator: patched _gen_config, _gen_markup, _gen_text, _gen_error with realistic patterns (markdown with code blocks, Python tracebacks, JS errors, nested YAML, env vars, tech prose, emails, conversations).
Real-world eval accuracy: 22.7% → 52.4% (2.3x better, 11/21 correct)
- coarse classification fixed for config, error, text, markup
- code_lang still "undetected" for real snippets (Python def, JS function, C #include, etc.)
Key change: coarse head weight increased to 8.0 (was 3.0) to prevent "default to code" behavior
Synthetic eval regression: coarse 100%→99.8%, code_lang 61.3%→38.7%, text_lang 100%→98.8% (expected — harder data)

ONNX re-export + v0.1.5 (Jun 18)

Re-exported all 4 ONNX tiers from best.pt (diverse-trained checkpoint)
Published as v0.1.5 at https://pypi.org/project/pico-type/0.1.5/
ONNX files pushed to HF model repo root, Space redeployed

Quantization attempts (deferred)

INT8: ONNX shape inference fails — [ShapeInferenceError] Inferred shape (192) vs (12). Multi-head architecture (shared 192-dim pooled vector → 7 linear layers with different output dims) confuses shape inference.
FP16: onnxconverter_common.float16 succeeds but produces type mismatch errors in ONNX Runtime. Could not resolve with op_block_list.

Fresh training from scratch experiment (Jun 18, discarded)

Tried training from scratch with diverse generator + 647 real GitHub samples (30% ratio) + high coarse weight (8.0)
4000 steps, batch_size=16, best eval loss 2.42
Result: Worse than fine-tuned model — real-world accuracy only 28.6% vs 52.4% from fine-tuned best.pt
Lesson: Training from scratch with high coarse weight over-prioritizes coarse classification at expense of code_lang/text_lang. Fine-tuning from a good synthetic checkpoint with gradual head-weight adjustments works better.
Status: Discarded. Production model remains checkpoints/best.pt (52.4% real-world accuracy).

v0.1.6: Docs, model card, HF fixes (Jun 18)

Uploaded walkthrough.md, docs/PLAN.md, MODEL_CARD.md to HF model repo (fixes broken links on HF model page)
Updated README.md with badges (PyPI, CI, DOI), eulogik branding, real-world accuracy metrics
Updated MODEL_CARD.md with full training/architecture details, citation, eulogik branding
Updated paper/main.tex with real-world eval, diverse generator details, higher coarse weight
Created HF org card content for eulogik organization page (paste at huggingface.co/eulogik)

8. What's next (from plan §3–§6, in order)

| # | File | What it does | Status | |---|---|---|---|---|---| | 1 | data.py | Synthetic generator + dataset for multi-head training. 11 buckets, all 12 coarse classes, code/word templates for all 62/30 langs. | ✅ done | | 2 | train.py | Multi-task trainer. AdamW + cosine, bf16, per-head loss weighting, gradient clipping, checkpoint save/load. resume_from field for continuing training. | ✅ done | | 3 | eval.py | Eval harness: per-head accuracy/PRF1, confusion matrix, risk AP, inference timing. CLI entry point. | ✅ done | | 4 | distill.py | KD from per-head teachers (deberta-v3-small, CodeBERTa-lang-id, xlm-roberta-lang-detect). T=2.0, α=0.7. | ✅ done | | 5 | export.py | ONNX export (opset 18), int8, tract, gguf. | ✅ done | | 6 | cli.py | Python CLI (picotype) — stdin/file/clipboard input → ONNX inference → JSON output | ✅ done | | 7 | mcp_server.py | MCP server (stdio transport) for Claude/Cursor/VSCode | ✅ done | | 8 | gradio_app.py | Gradio Space app for HF Spaces | ✅ done | | 9 | tests/test_smoke.py | pytest smoke tests (8 tests: arch, data, ONNX, CLI, labels) | ✅ done | | 10 | MODEL_CARD.md | HuggingFace model card with eval results | ✅ done | | 11 | README.md | Overhauled with badges, perf table, deploy links | ✅ done | | 12 | spaces/requirements.txt | Dependencies for HF Space deployment | ✅ done | | 13 | HF Model + Space | Published to huggingface.co/eulogik/pico-type (model) and /spaces/eulogik/pico-type (Space) | ✅ done | | 14 | PyPI publish | pico-type v0.1.0→v0.1.2→v0.1.3 on PyPI | ✅ done | | 15 | crates/picotype/ | Rust CLI w/ ONNX runtime. | ✅ done | | 16 | crates/picotype-mcp/ | Rust MCP server (stdio). | ✅ done | | 17 | extensions/chrome/ | Chrome MV3 scaffolded + icons created. Raycast, Alfred, VSCode — pending. | partial | | 18 | paper/ | arXiv LaTeX scaffolded, final numbers updated (paper/main.tex). | in progress | | 19 | Training | 1700 → 5000 synthetic steps (loss 1.61) → 4000 real-data steps (25/lang, code_lang 53.9%). | ✅ done | | 20 | ONNX re-export | All 4 tiers re-exported from best.pt (post real-data fine-tune). | ✅ done | | 21 | Real-data pipeline | realdata.py + train_real.py — GitHub code fetcher, mixed dataset, fine-tuning loop. | ✅ done | | 22 | HF Space fix | Label lists synced to labels.py, np.bool_→bool for NumPy 2, requirements updated. | ✅ done | | 23 | INT8 quantization | Shape inference issue with multi-head architecture — needs graph fix. | pending | | 24 | HuggingFace Hub push | ONNX, best.pt, eval results, paper, README all pushed. | ✅ done |

9. Open decisions (from plan §8 — still open)

HF handle: pico-type (dash) for model card, picotype (no dash) for CLI binary. Proposed, not confirmed.
Tier naming: tiny/small/base/pro (matches Sentence-Transformers convention). Proposed, not confirmed.
License: Apache-2.0 (matches CommonLingua base). Proposed, not confirmed.
arXiv target: cs.CL (primary) + cs.LG. Co-authors: open question.
Tagline: "One tiny model, one forward pass, every clipboard." Proposed, not confirmed.

10. Quick recipes

Instantiate the model

from model.pico_type import PicoType, PicoTypeConfig

cfg = PicoTypeConfig(max_bytes=1024)
model = PicoType(cfg)
print(model.tier_sizes())  # {tiny: 1434344, small: 1445480, base: 1475176, pro: 1564264}

Run a forward + decode

from model.pico_type.arch import encode_bytes
from model.pico_type.labels import decode_output

model.eval()
x, mask = encode_bytes(b'def hi(): return 1', max_len=1024)
with torch.no_grad():
    logits = model(x, mask, tier='base')
out = decode_output(logits, tier='base')
# {'coarse': ..., 'modality': ..., 'subtype': ..., 'code_language': ...,
#  'text_language': ..., 'file_mime': ..., 'risk_flags': [...],
#  'confidence': ..., 'modality_confidence': ..., 'model_tier': 'base'}

Convert to a single-tier checkpoint (for release)

Use parameter_count(tier) to get the param count for that tier. To build a release checkpoint, you would: train full model → for each tier, save only trunk.* + heads.*.linears.{tier}.* → export.

11. Session history (why this file exists)

The user was working on this project in opencode. The session (ses_16dd3d39fffer9xnBUQYBS3u5z — "Tiny model for clipboard content classification") crashed mid-execution while writing model/pico_type/arch.py. Opencode had to be re-installed; the user thought files might be lost.

They weren't. The full session data was recovered intact from:

~/.local/share/opencode/opencode.db (262MB SQLite)
~/.local/share/opencode/storage/session_diff/ses_16dd3d39fffer9xnBUQYBS3u5z.json (128MB JSON)
The trash (~/.Trash/opencode) only contained opencode Desktop app data (different product, irrelevant to the CLI session).

From the recovery we:

Extracted the full plan → docs/PLAN.md
Re-wrote arch.py (the file the crashed session was aborting on) and labels.py (never written in original session)
Wrote this walkthrough.md so the next agent/harness has full context

User's 3 original prompts

"I want to build a really tiny model which categorises/classifies content. eg if we pass clipboard copied content, it should classify that as text, image, rich text, link, code(with language name), file with file type etc. Deep research the existing models on huggingface etc. go through new research papers and find an opportunity / gap to make this model in the most efficient manner plus make it really popular. feel free to suggest anything"
"continue asking questions and ahead. btw, the text language should be identified too like code language. if language not detected, it should simply return text + undetected or code undetected"
"go"

Where the original session was at crash

23 messages, 95 parts
Plan had been finalized (10KB markdown)
Approved with "go"
Switched to build agent
Set up Python 3.11 venv, installed torch/numpy/safetensors/pyyaml
Created the full directory tree (model/pico_type/, model/configs/, crates/picotype/, etc.)
Wrote model/pico_type/__init__.py ✅
Wrote arch.py (the file the user originally saw being written) — wait, the original session was aborted on the arch.py write. The __init__.py is in the repo. We re-wrote arch.py from scratch using the plan + the small preview from the original write tool input.

12. Anti-patterns to avoid (learned)

Python 3.14 has no torch wheels. Always use the venv's Python 3.11.
Don't double-count Matryoshka head params when iterating named_modules() — ModuleDict is visited separately from its children. Use named_parameters() and check .linears.{tier}. in the name.
Gating heads are not always-on. subtype, code_lang, text_lang, file_mime must mask their loss when not applicable. The decoder handles this; the trainer must too.
UNK/undetected is a decoder-side decision, not a model class. The model has N logits; the decoder thresholds.