Diogenes 0.2 Β· 55M β SFT final
Diogenes is a 55M-parameter byte-level language model jointly with a content memory and its reader: one run, no pretrained init, no tokenizer. This is the final supervised fine-tuned model β 3,097 chat updates (Dolly + OASST1) on top of the 380k pretrained base. Pretraining checkpoint: Counfield/Diogenes-0.2-55M-260K-early.
The design point: the LM itself does not memorize facts. A small reader (858k params) maintains and reads a rolling content memory through 4 hop channels and 10 passage anchors. On held-out single-fact recall, accuracy is 100% with the reader on and 0% with it off β the facts live in the memory path, not in the weights.
The model follows the Recurrent Loop Transformer (RLT) design: the base core runs feedback-off within a bounded attention window, the recurrent path is carried by a persistent content memory and its reader.
Architecture
Diogenes follows the Recurrent Loop Transformer (RLT) design. The core runs feedback-off: the recurrence is instantiated through the memory path β the store is written during training and re-injected at read time, alongside the base LM's own distribution, so information older than the base's fixed window remains reachable at every scored position.
The bytes are the tokens. The vocabulary is the 256 raw UTF-8 byte values β no tokenizer, no BPE, no subwords. Documents are separated by \n\n<|doc|>\n\n. Chat turns are User: ...\n\nAssistant: ... + EOT (0x04); the loss covers assistant bytes and EOT only.
The base holds a fixed window. Feedback-off transformer ("HoldCore"), banded attention, bounded memory limit: attention cost linear in length. Per 8,192-byte row it scores the last 1,024 positions and attends over ~2,048 bytes; older text is invisible to it by design.
Content memory carries the rest. Every 3rd training step writes memory rows (4 Γ 8,192 bytes) into a rolling content store persisted inside the checkpoint; at inference that store gives the model reach beyond its window.
The reader is candidate-first. 858k params (width 128): it scores byte candidates β direct context repeats, 4-hop chains, 10 passage anchors, plus a NULL "no answer" slot β against a summary of the base's hidden states, and a learned gate picks the base distribution or a candidate per byte.
| Base params | 55,078,145 |
| Reader params | 858,436 |
| Vocabulary | 256 (raw UTF-8 bytes) |
| Base width / heads | 1536 / 8 |
| Reader width | 128 |
| Hop channels / passage anchors | 4 / 10 |
| Document separator | `\n\n< |
| Chat format | User: ...\n\nAssistant: ... + EOT (0x04) |
| SFT recipe | epochs 2, batch 16, replay 0.25, EOT weight 10, lr 2e-4 (base) / 3e-4 (reader), AMP |
| Seed | 401 |
Benchmarks
| benchmark | score |
|---|---|
| language modeling, bits/byte | 1.1492 |
| sft val, bits/byte | 1.2567 |
| turn completion (greedy, gen 400) | 0.72 |
| repetition rate | 0.42 |
| natural text, bits/byte (memory on) | 1.3392 |
| single-fact recall, held-out | 1.00 |
| chain recall, 3 hops | 0.875 |
| chain recall, 4 hops | 1.00 |
| chain recall, 5 hops | 1.00 |
| broken-chain detection | 1.00 |
| absent-entity rejection | 1.00 |
| cross recall, forward | 1.00 |
| cross recall, backward | 0.9688 |
| cross recall, natural | 0.9375 |
Use and limitations
- Chat: temperature β 0.7, top-p β 0.9, repetition penalty β 1.1; stop at EOT (0x04) or 900 bytes. Greedy decoding loops.
- Arithmetic is weak at this scale.
- English-centric; no tokenizer β feed raw UTF-8 bytes.
- Checkpoint:
latest.pt(step 3,097) +config.json, custommlineformat; not Transformers-compatible. Load withmline.cmem_sft.load_startor thescripts/panel/server.pypanel.
- Downloads last month
- 70
