File size: 2,810 Bytes
9f1076a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # SCALE — GPT-3 capability without GPT-3 parameter/GPU-farm scale
## Thesis
ATOM does **not** aim to match GPT-3’s ~175B parameters or its training-cluster
footprint. The goal is comparable **useful intelligence** — coherent continuum
behavior over language — via a different capacity substrate:
| GPT-3 world | ATOM world |
|-------------|------------|
| ~175B shared weights | Small shared rule set Θ (field / dynamics / surface) |
| GPU farms, huge batches | CPU / single-box friendly continuum |
| Capacity ≈ parameter count | Capacity ≈ **persistent structured matter** |
| One-shot dense pretrain | Infinite resume + consolidation + streaming ingest |
Parameters stay modest. What grows is the **toroidal field**, structural atoms,
aggregates, abstractions, and consolidation — computational matter that
persists across ticks and runs.
## Why streaming ingest matters
A full corpus must not be forced into a giant in-RAM packet list. Streaming
full-dataset ingest (`src/io/stream_corpus.py`, `--stream --data-glob`) reads
UTF-8-safe shards online, atomizes with `encode_bytes(..., flush=False)`,
flushes at shard ends, and yields `(current, target)` transitions forever
(`--loop-shards`). Complete datasets become train-able on one machine without
matching GPT-3’s hardware narrative.
## Capacity story (honest)
- **Not claimed:** “we beat GPT-3 on every benchmark with 64 modes.”
- **Claimed:** the architectural bet is that structured persistence + streaming
complete data can approach *capability* without *parameter/GPU-farm* scale.
- Metrics (byte CE, packet ppl) are training signals, not chat-quality proofs.
Judge generations and long-horizon field behavior separately.
## Practical scale path
1. Shard corpora under `data/shards/*.txt` (or any glob).
2. Train with `--stream --data-glob ... --loop-shards --no-episode-reset`.
3. Resume forever; let consolidation accumulate; soft-flush atoms, not the field.
4. Grow Θ only when dynamics demand it — not to chase 175B vanity counts.
See also: `docs/TRAINING.md`, `ATOM_RULES.md`, `ATOM_NATIVE_TOKENIZATION.md`.
## Persistence test (field farm vs CE-only)
CE down + fluent-looking metrics **without** field reconstruction still means
**Transformer regime without the farm.** Capacity that matters for ATOM is
persistent structured matter in the toroidal field, not parameter count alone.
Latest probe (`tools/probe_field_persistence.py`):
- Mean RMS after prompt / after 20 gen: 0.003229 / 0.022723
- Off-diag cosine surface logits / alpha: 0.998711 / 0.709268
- Verdict: field differs across prompts but surface logits nearly identical — field not reliably read by surface (Transformer-regime risk / surface-only byte-LM behavior at decode)
See `docs/PERSISTENCE_TEST.md` and `docs/FIELD_PROBE.md`.
|