SCALE — GPT-3 capability without GPT-3 parameter/GPU-farm scale
Thesis
ATOM does not aim to match GPT-3’s ~175B parameters or its training-cluster footprint. The goal is comparable useful intelligence — coherent continuum behavior over language — via a different capacity substrate:
| GPT-3 world | ATOM world |
|---|---|
| ~175B shared weights | Small shared rule set Θ (field / dynamics / surface) |
| GPU farms, huge batches | CPU / single-box friendly continuum |
| Capacity ≈ parameter count | Capacity ≈ persistent structured matter |
| One-shot dense pretrain | Infinite resume + consolidation + streaming ingest |
Parameters stay modest. What grows is the toroidal field, structural atoms, aggregates, abstractions, and consolidation — computational matter that persists across ticks and runs.
Why streaming ingest matters
A full corpus must not be forced into a giant in-RAM packet list. Streaming
full-dataset ingest (src/io/stream_corpus.py, --stream --data-glob) reads
UTF-8-safe shards online, atomizes with encode_bytes(..., flush=False),
flushes at shard ends, and yields (current, target) transitions forever
(--loop-shards). Complete datasets become train-able on one machine without
matching GPT-3’s hardware narrative.
Capacity story (honest)
- Not claimed: “we beat GPT-3 on every benchmark with 64 modes.”
- Claimed: the architectural bet is that structured persistence + streaming complete data can approach capability without parameter/GPU-farm scale.
- Metrics (byte CE, packet ppl) are training signals, not chat-quality proofs. Judge generations and long-horizon field behavior separately.
Practical scale path
- Shard corpora under
data/shards/*.txt(or any glob). - Train with
--stream --data-glob ... --loop-shards --no-episode-reset. - Resume forever; let consolidation accumulate; soft-flush atoms, not the field.
- Grow Θ only when dynamics demand it — not to chase 175B vanity counts.
See also: docs/TRAINING.md, ATOM_RULES.md, ATOM_NATIVE_TOKENIZATION.md.
Persistence test (field farm vs CE-only)
CE down + fluent-looking metrics without field reconstruction still means Transformer regime without the farm. Capacity that matters for ATOM is persistent structured matter in the toroidal field, not parameter count alone.
Latest probe (tools/probe_field_persistence.py):
- Mean RMS after prompt / after 20 gen: 0.003229 / 0.022723
- Off-diag cosine surface logits / alpha: 0.998711 / 0.709268
- Verdict: field differs across prompts but surface logits nearly identical — field not reliably read by surface (Transformer-regime risk / surface-only byte-LM behavior at decode)
See docs/PERSISTENCE_TEST.md and docs/FIELD_PROBE.md.