Diakosmos
Diakosmos (διάκοσμος, "the ordering of a world") is a byte-level harmonic recurrent language model,
99,823,934 parameters, written and trained from scratch in the
Alpha language on a rented RTX 3070. It has no tokenizer and
no attention matrix: each byte enters as a fixed constellation of phasors (the wire_table), and each
of 14 blocks blends a decaying harmonic recurrence over the past with a planar rotation between seven
wire groups. Its design is harmonic-gpt's leading model.
Try it: lsalpha.com/training/diakosmos/chat.
Files
| File | What |
|---|---|
diakosmos-base-15360.safetensors |
the pretrained model (update 15,360) |
diakosmos-chat-960.safetensors |
the chat fine-tune's first stage (960 updates on conversations) |
diakosmos-chat-short-320.safetensors |
its second stage (320 updates on short replies): the served chat model |
models.json |
which file each served model is, and its held-out scores |
diakosmos.py |
a PyTorch byte-a-step decoder for these weights (the reference model's arithmetic) |
The tensors carry harmonic-gpt's parameter names; wire_table is the fixed byte encoding (256 x 1330).
Training
- Pretraining: 2.01 GB of OpenWebText as UTF-8 bytes, read once in 2,048-byte windows: 15,360 updates of 64 windows, 24.5 GPU hours and 5.29 kWh on one RTX 3070. Held-out OpenWebText: 1.4194 bits per byte (Alpha's GPT-2 small, on the same bytes and card: 1.4417 after 25.2 GPU hours).
- Chat fine-tune: 93,125 conversations under 2 KB from smol-smoltalk (without its code), everyday-conversations, oasst2 (each English tree's best-ranked path) and dolly-15k, 114 MB, packed with OpenWebText the pretraining never read (about 9%). 960 updates from the pretrained weights, fresh optimizer state, peak learning rate 6e-4 (a fifth of pretraining's), warmup 50, cosine to a tenth; 1.55 GPU hours.
- Second stage: 320 more updates on the 58,588 conversations whose every reply is at most 400 bytes (peak 3e-4, warmup 20, fresh optimizer state; 0.52 GPU hours). The first stage had learned SmolTalk's long replies (median 800 bytes) and rarely ended them; after this stage it ends most replies within a few hundred bytes and stays closer to the question.
- Held-out conversations (1,774, bits per byte on the replies, most of them long): pretrained 1.461, first stage 0.950, second stage 0.983.
Format
Roles are plain text and every reply ends with the byte 0x04:
User: <message>
Assistant: <reply>\x04
User: <message>
Assistant:
Generation stops at 0x04. Prompt and reply together fit 2,047 bytes.
Use
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import diakosmos as dk # diakosmos.py from this repository
model = dk.Diakosmos(load_file(hf_hub_download('ajaxdavis/diakosmos', 'diakosmos-chat-short-320.safetensors')))
reply, ended, steps = dk.generate(model, b'User: Hi! How are you today?\nAssistant: ', temperature=0.7, min_p=0.1, seed=1)
print(reply.decode('utf-8', errors='replace'))
Limitations
It is small and read little: 2 GB of text, about a twentieth of what GPT-2 small read. Asked to continue simple facts ("The capital of France is", "Romeo and Juliet was written by", ...), the pretrained model completed none of six correctly in 24 tries; the chat model does answer "The capital of France is Paris.", but most of what it says is fluent, assistant-like and often off topic or wrong. It is a measurement of a new architecture and a new compiler, not an assistant to rely on.
License
No license has been chosen for these weights yet. The training data's own licenses apply to it: OpenWebText (CC0 for the compilation), smol-smoltalk, everyday-conversations and oasst2 (Apache 2.0), dolly-15k (CC BY-SA 3.0).