| --- |
| language: en |
| pipeline_tag: text-generation |
| tags: |
| - assembly |
| - from-scratch |
| - chat |
| --- |
| |
| # Mnemonic 126M |
|
|
| A small chat model where every part (data pipeline, tokenizer, GPU training, quantization and inference) is hand-written assembly: x86-64 on the CPU, PTX on the GPU, WebAssembly in the browser. |
|
|
| Code: https://github.com/Supergoatscriptguy/Mnemonic |
|
|
| | file | weights | size | |
| |---|---|---| |
| | `mnemonic-q8.mnm` | int8, one scale per row | 121 MB | |
| | `mnemonic-q4.mnm` | int4, one scale per group of 32 | 75 MB | |
|
|
| **Model.** Llama-style decoder: 16 layers, d_model 768, 12 query / 4 key-value heads, SwiGLU (ffn 2048), RoPE, RMSNorm, tied embeddings, no biases. Context 1024, byte-level BPE vocab of 32768. |
| |
| **Training.** Pretrained on 5B tokens of FineWeb-Edu on one RTX 5070 Ti (19 hours), then fine-tuned for chat on smol-smoltalk plus a small identity set. |
| |
| **Format.** `.mnm` is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (`chat/engine.asm`, `site/engine.wat`). It is not a transformers checkpoint. |
| |
| **Limitations.** It's a small model: it makes confident mistakes, is weak at math, and loses the thread in long conversations. |
| |
| Made by Supergoatscriptguy. |
| |
