language: en
pipeline_tag: text-generation
tags:
- assembly
- from-scratch
- chat
Mnemonic 126M
A small chat model where every part (data pipeline, tokenizer, GPU training, quantization and inference) is hand-written assembly: x86-64 on the CPU, PTX on the GPU, WebAssembly in the browser.
Code: https://github.com/Supergoatscriptguy/Mnemonic
| file | weights | size |
|---|---|---|
mnemonic-q8.mnm |
int8, one scale per row | 121 MB |
mnemonic-q4.mnm |
int4, one scale per group of 32 | 75 MB |
Model. Llama-style decoder: 16 layers, d_model 768, 12 query / 4 key-value heads, SwiGLU (ffn 2048), RoPE, RMSNorm, tied embeddings, no biases. Context 1024, byte-level BPE vocab of 32768.
Training. Pretrained on 5B tokens of FineWeb-Edu on one RTX 5070 Ti (19 hours), then fine-tuned for chat on smol-smoltalk plus a small identity set.
Format. .mnm is the project's own format (a 4 KB header, then the tensors), read by the engines in the repo (chat/engine.asm, site/engine.wat). It is not a transformers checkpoint.
Limitations. It's a small model: it makes confident mistakes, is weak at math, and loses the thread in long conversations.
Made by Supergoatscriptguy.