File size: 3,156 Bytes
ae9e4fe 00c9d45 ae9e4fe 00c9d45 ae9e4fe 00c9d45 ae9e4fe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | # Model A — From-Scratch Mini Chat Companion — Build Checklist
Pure NumPy (CuPy swap allowed for GPU). Tiny. **General chat only — no code, no tools.** Build + test each yourself; don't move on until current item works. Every autograd op passes gradcheck; every layer passes a shape/grad test; training passes a one-paragraph overfit before scaling.
## Scope
| Property | Target |
|---|---|
| Capability | short English small-talk / companion replies |
| Explicitly NOT | code, math tools, retrieval, function-calling |
| Params | ~1–8M (CPU-trainable) |
| Arch | modern-tiny: RoPE + RMSNorm + SwiGLU + GQA, weight-tied head |
| Framework | pure NumPy (`np`), optional `ZYN_BACKEND=cuda` CuPy swap |
| Tokenizer | small byte-BPE, vocab ~4k, chat special tokens |
## Foundation
- [x] Folder layout (`mla/`, `tests/`, `data/`, `checkpoints/`, `serve/`)
- [x] Backend switch (NumPy / CuPy, dtype float64 for gradcheck / float32 GPU)
- [x] Numerical gradient checker (finite diff + rel-error)
- [x] Autograd Tensor (add, mul, matmul, reshape, transpose, sum, gather)
- [x] Activations as ops (silu/gelu, exp, log, softmax, rsqrt)
- [x] Reverse-mode backward (topological order) + broadcast-aware grads
## Data (chat only)
- [x] Collect + clean small-talk / dialogue corpus (chit-chat, no code)
- [x] Chat formatting (`<|user|>` / `<|assistant|>` turns, `<bos>`/`<eos>`)
- [x] Train/val split (by conversation, seed=42) + dedup
- [x] Byte-BPE tokenizer (train ~4k, encode, decode, save/load)
- [x] Chat special tokens reserved before byte base
- [x] Batching (packed turns → next-token x/y windows)
## Model (modern-tiny decoder)
- [x] Token embeddings (tied to output head)
- [x] RoPE (rotary positional embeddings)
- [x] RMSNorm (Pre-LN placement)
- [x] Self-attention (Q/K/V, scaled, causal mask)
- [x] Grouped-query attention (fewer KV heads)
- [x] QK-Norm before RoPE (stability)
- [x] SwiGLU MLP (gated, 2/3 hidden-dim rule)
- [x] Residual connections
- [x] Stack blocks → full model + config
## Pretraining
- [x] Cross-entropy loss (ignore-index for padding)
- [x] AdamW optimizer + grad clipping
- [x] LR warmup + cosine decay
- [x] Sanity: overfit one dialogue (loss < 0.05)
- [x] Training loop + checkpoint save/load/resume
## Chat fine-tune
- [x] Instruction/chat dataset + loss masking (train assistant turns only)
- [x] Fine-tune from pretrain checkpoint
- [x] Persona/system prompt conditioning (companion tone)
## Evaluation
- [x] Val loss + perplexity
- [x] Next-token accuracy
- [x] Sample chat turns per checkpoint (manual read)
- [x] Refusal check: coding prompt → stays in-scope (no code attempt)
## Inference
- [x] Sampling (greedy, temperature, top-k, top-p)
- [x] KV-cache (logits identical to full forward)
- [x] Chat runtime (render turns → generate → stop on `<eos>`)
## Hosting
- [ ] Package model + tokenizer + config bundle
- [ ] Inference-only load mode
- [ ] Serving API (`/chat`, `/health`)
- [ ] Dockerfile + env config
- [ ] Deploy + smoke test live URL
## Scaling (later)
- [ ] NumPy → CuPy for GPU train/infer
- [ ] Bigger config + larger chat corpus
- [ ] Batched/concurrent serving
|