Crowd v1.5 DUST checkpoint (300M)
A single weight snapshot that several people train in turn. One trainer holds the baton, trains for a while, and pushes it back with the round number bumped. There is no server: the repo is the coordinator state. Peers are trusted, and the round number is what stops a slow trainer from overwriting newer work.
| field | value |
|---|---|
| preset | 300m β 18 layers, width 1024, 16 heads, context 2048 |
| parameters | 306,222,080 (tied embeddings) |
| weights | float32 safetensors, 1.22 GB |
| routes | dust (active), backprop (kept, unused) |
| round | status.json: round |
The earlier 1B seed is preserved at git tag seed-1b if you want to compare.
Live 24 h training results
Two volunteers are running the DUST estimator for 24 hours, uncapped, on real
FineWeb-Edu tokens (dataset: peva3/crowd-v15-tokens,
58 M train / 2 M held-out, fixed BPE-4096). The curves are republished every
20 minutes under results/:
| machine | hardware | arm | protocol |
|---|---|---|---|
results/rtx4060ti |
RTX 4060 Ti, 16 GB, CUDA 12.4 | backprop Adam (12 min baseline) then DUST | results/rtx4060ti/RESULTS.md |
results/gfx1150 |
Radeon 890M, 30 GB shared, ROCm 7.2 | DUST | results/gfx1150/RESULTS.md |
Each directory holds the raw *.jsonl the trainer emitted (one line per log
interval), a *.curve.csv, an SVG curve and a *.summary.json. Nothing in
those files is edited after the fact; a run is only ever appended to.
Configuration for both DUST arms: 300M preset, seq 512, batch 8, population 32,
--mode all, relative noise scale (sigma_rel = 0.05), --noise-device device,
1-bit residual codec (ngd1r) upload through the real coordinator, outer
optimizer lr 0.7 / momentum 0.9.
Honest expectation: the estimator needs no backward pass and no optimizer
state, but it evaluates 3 x population forward passes per step, so evidence
tokens per second is roughly 1 / (population x draw_chunk) of a backprop
token. These two cards are the throughput measurement, not a convergence claim.
Files
| file | what it is |
|---|---|
weights.safetensors |
float32 tensors under canonical parameter names |
status.json |
crowd-v15-coordinator-1 β round, baseSha256, history |
manifest.json |
crowd-v15-model-1 β names, sizes, shapes, baseSha256 |
seed.json |
crowd-v15-seed-1 β preset, params, init seed, genesis digest |
lock.json |
crowd-v15-baton-1 β advisory lease (holder, expiry) |
baseSha256 is not the hash of the file. It is sha256 over sorted tensor
names plus the float32 values rounded to bfloat16, so re-saving or changing
dtype does not move it. verify recomputes it before and after training.
Get the code
git clone https://github.com/peva3/Crowd-v1.5 && cd Crowd-v1.5
python -m pip install -e . # torch, numpy, safetensors, fastapi
Fine-grained token: https://huggingface.co/settings/tokens β Fine-grained β
Repositories: write on peva3/Crowd-v1.5-dust only. Send it over DM, not in
a commit or a chat log that gets forwarded.
Train a round
export HF_TOKEN=hf_... # fine-grained, write access to THIS repo only
# 1. take the baton
python tools/hf_baton.py pull --repo peva3/Crowd-v1.5-dust --dir state
python tools/hf_baton.py verify --dir state
# 2. train DUST locally on your own tokens (any tokenizer-compatible uint16 bin)
python tools/hf_baton.py train --repo peva3/Crowd-v1.5-dust --dir state \
--data /path/to/tok --device cuda \
--steps 200 --batch 2 --seq-len 512 --population 32 --draw-chunk 4 \
--mode all --sigma-mode relative --sigma-rel 0.05 \
--lr 0.05 --momentum 0.9 --push
# 3. hand it back (done by --push above; or explicitly:)
python tools/hf_baton.py push --repo peva3/Crowd-v1.5-dust --dir state
train writes weights.safetensors, bumps round, appends a history entry
(steps, tokens, loss, population, sigma, lr), rewrites manifest.json and the
lock, and refuses to push if the remote round already moved past yours.
Configs that fit
| hardware | command sketch | measured |
|---|---|---|
| 16 GB consumer GPU | --batch 2 --seq-len 512 --population 32 --draw-chunk 4 --mode all |
57 s/step, 17.9 evidence tok/s, 8.46 GB peak |
| 8 GB GPU | --batch 1 --seq-len 256 --population 32 --draw-chunk 4 --mode all |
~1/4 the tokens per step |
| CPU only | --device cpu --compute-dtype float32 --batch 1 --seq-len 128 --population 16 --draw-chunk 4 |
slow but valid |
--mode layer isolates each layer's noise the way the paper does and roughly
doubles gradient alignment, but it costs ~20x more forwards per step. all is
the sane default at 300M.
Rules of the baton
- Pull before you train. Never train on a checkpoint you did not pull.
- Push one round ahead. If
pushsays the remote round is ahead, pull again and re-apply your work; do not--forceover someone else's round. lock.jsonis advisory (24 h lease). The round guard is the enforcement.- Keep the token in your environment, never in a file you commit.
- Report what you ran: the
traincommand is echoed intohistory.