# Model A — From-Scratch Mini Chat Companion — Build Checklist

Pure NumPy (CuPy swap allowed for GPU). Tiny. **General chat only — no code, no tools.** Build + test each yourself; don't move on until current item works. Every autograd op passes gradcheck; every layer passes a shape/grad test; training passes a one-paragraph overfit before scaling.

## Scope

| Property | Target |
|---|---|
| Capability | short English small-talk / companion replies |
| Explicitly NOT | code, math tools, retrieval, function-calling |
| Params | ~1–8M (CPU-trainable) |
| Arch | modern-tiny: RoPE + RMSNorm + SwiGLU + GQA, weight-tied head |
| Framework | pure NumPy (`np`), optional `ZYN_BACKEND=cuda` CuPy swap |
| Tokenizer | small byte-BPE, vocab ~4k, chat special tokens |

## Foundation

- [x] Folder layout (`mla/`, `tests/`, `data/`, `checkpoints/`, `serve/`)
- [x] Backend switch (NumPy / CuPy, dtype float64 for gradcheck / float32 GPU)
- [x] Numerical gradient checker (finite diff + rel-error)
- [x] Autograd Tensor (add, mul, matmul, reshape, transpose, sum, gather)
- [x] Activations as ops (silu/gelu, exp, log, softmax, rsqrt)
- [x] Reverse-mode backward (topological order) + broadcast-aware grads

## Data (chat only)

- [x] Collect + clean small-talk / dialogue corpus (chit-chat, no code)
- [x] Chat formatting (`<|user|>` / `<|assistant|>` turns, `<bos>`/`<eos>`)
- [x] Train/val split (by conversation, seed=42) + dedup
- [x] Byte-BPE tokenizer (train ~4k, encode, decode, save/load)
- [x] Chat special tokens reserved before byte base
- [x] Batching (packed turns → next-token x/y windows)

## Model (modern-tiny decoder)

- [x] Token embeddings (tied to output head)
- [x] RoPE (rotary positional embeddings)
- [x] RMSNorm (Pre-LN placement)
- [x] Self-attention (Q/K/V, scaled, causal mask)
- [x] Grouped-query attention (fewer KV heads)
- [x] QK-Norm before RoPE (stability)
- [x] SwiGLU MLP (gated, 2/3 hidden-dim rule)
- [x] Residual connections
- [x] Stack blocks → full model + config

## Pretraining

- [x] Cross-entropy loss (ignore-index for padding)
- [x] AdamW optimizer + grad clipping
- [x] LR warmup + cosine decay
- [x] Sanity: overfit one dialogue (loss < 0.05)
- [x] Training loop + checkpoint save/load/resume

## Chat fine-tune

- [x] Instruction/chat dataset + loss masking (train assistant turns only)
- [x] Fine-tune from pretrain checkpoint
- [x] Persona/system prompt conditioning (companion tone)

## Evaluation

- [x] Val loss + perplexity
- [x] Next-token accuracy
- [x] Sample chat turns per checkpoint (manual read)
- [x] Refusal check: coding prompt → stays in-scope (no code attempt)

## Inference

- [x] Sampling (greedy, temperature, top-k, top-p)
- [x] KV-cache (logits identical to full forward)
- [x] Chat runtime (render turns → generate → stop on `<eos>`)

## Hosting

- [ ] Package model + tokenizer + config bundle
- [ ] Inference-only load mode
- [ ] Serving API (`/chat`, `/health`)
- [ ] Dockerfile + env config
- [ ] Deploy + smoke test live URL

## Scaling (later)

- [ ] NumPy → CuPy for GPU train/infer
- [ ] Bigger config + larger chat corpus
- [ ] Batched/concurrent serving