model card

README.md

@@ -0,0 +1,29 @@
+---
+language: en
+pipeline_tag: text-generation
+tags:
+- assembly
+- from-scratch
+- chat
+---
+
+# Mnemonic 126M
+
+A small chat model where every part (data pipeline, tokenizer, GPU training, quantization and inference) is hand-written assembly: x86-64 on the CPU, PTX on the GPU, WebAssembly in the browser.
+
+Code: https://github.com/Supergoatscriptguy/Mnemonic
+
+| file | weights | size |
+|---|---|---|
+| `mnemonic-q8.mnm` | int8, one scale per row | 121 MB |
+| `mnemonic-q4.mnm` | int4, one scale per group of 32 | 75 MB |
+
+**Model.** Llama-style decoder: 16 layers, d_model 768, 12 query / 4 key-value heads, SwiGLU (ffn 2048), RoPE, RMSNorm, tied embeddings, no biases. Context 1024, byte-level BPE vocab of 32768.
+
+**Training.** Pretrained on 5B tokens of FineWeb-Edu on one RTX 5070 Ti (19 hours), then fine-tuned for chat on smol-smoltalk plus a small identity set.
+
+**Format.** `.mnm` is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (`chat/engine.asm`, `site/engine.wat`). It is not a transformers checkpoint.
+
+**Limitations.** It's a small model: it makes confident mistakes, is weak at math, and loses the thread in long conversations.
+
+Made by Supergoatscriptguy.