You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Crowd v1.5 DUST checkpoint (300M)

A single weight snapshot that several people train in turn. One trainer holds the baton, trains for a while, and pushes it back with the round number bumped. There is no server: the repo is the coordinator state. Peers are trusted, and the round number is what stops a slow trainer from overwriting newer work.

field value
preset 300m β€” 18 layers, width 1024, 16 heads, context 2048
parameters 306,222,080 (tied embeddings)
weights float32 safetensors, 1.22 GB
routes dust (active), backprop (kept, unused)
round status.json: round

The earlier 1B seed is preserved at git tag seed-1b if you want to compare.

Live 24 h training results

Two volunteers are running the DUST estimator for 24 hours, uncapped, on real FineWeb-Edu tokens (dataset: peva3/crowd-v15-tokens, 58 M train / 2 M held-out, fixed BPE-4096). The curves are republished every 20 minutes under results/:

machine hardware arm protocol
results/rtx4060ti RTX 4060 Ti, 16 GB, CUDA 12.4 backprop Adam (12 min baseline) then DUST results/rtx4060ti/RESULTS.md
results/gfx1150 Radeon 890M, 30 GB shared, ROCm 7.2 DUST results/gfx1150/RESULTS.md

Each directory holds the raw *.jsonl the trainer emitted (one line per log interval), a *.curve.csv, an SVG curve and a *.summary.json. Nothing in those files is edited after the fact; a run is only ever appended to.

Configuration for both DUST arms: 300M preset, seq 512, batch 8, population 32, --mode all, relative noise scale (sigma_rel = 0.05), --noise-device device, 1-bit residual codec (ngd1r) upload through the real coordinator, outer optimizer lr 0.7 / momentum 0.9.

Honest expectation: the estimator needs no backward pass and no optimizer state, but it evaluates 3 x population forward passes per step, so evidence tokens per second is roughly 1 / (population x draw_chunk) of a backprop token. These two cards are the throughput measurement, not a convergence claim.

Files

file what it is
weights.safetensors float32 tensors under canonical parameter names
status.json crowd-v15-coordinator-1 β€” round, baseSha256, history
manifest.json crowd-v15-model-1 β€” names, sizes, shapes, baseSha256
seed.json crowd-v15-seed-1 β€” preset, params, init seed, genesis digest
lock.json crowd-v15-baton-1 β€” advisory lease (holder, expiry)

baseSha256 is not the hash of the file. It is sha256 over sorted tensor names plus the float32 values rounded to bfloat16, so re-saving or changing dtype does not move it. verify recomputes it before and after training.

Get the code

git clone https://github.com/peva3/Crowd-v1.5 && cd Crowd-v1.5
python -m pip install -e .          # torch, numpy, safetensors, fastapi

Fine-grained token: https://huggingface.co/settings/tokens β†’ Fine-grained β†’ Repositories: write on peva3/Crowd-v1.5-dust only. Send it over DM, not in a commit or a chat log that gets forwarded.

Train a round

export HF_TOKEN=hf_...            # fine-grained, write access to THIS repo only

# 1. take the baton
python tools/hf_baton.py pull --repo peva3/Crowd-v1.5-dust --dir state
python tools/hf_baton.py verify --dir state

# 2. train DUST locally on your own tokens (any tokenizer-compatible uint16 bin)
python tools/hf_baton.py train --repo peva3/Crowd-v1.5-dust --dir state \
    --data /path/to/tok --device cuda \
    --steps 200 --batch 2 --seq-len 512 --population 32 --draw-chunk 4 \
    --mode all --sigma-mode relative --sigma-rel 0.05 \
    --lr 0.05 --momentum 0.9 --push

# 3. hand it back (done by --push above; or explicitly:)
python tools/hf_baton.py push --repo peva3/Crowd-v1.5-dust --dir state

train writes weights.safetensors, bumps round, appends a history entry (steps, tokens, loss, population, sigma, lr), rewrites manifest.json and the lock, and refuses to push if the remote round already moved past yours.

Configs that fit

hardware command sketch measured
16 GB consumer GPU --batch 2 --seq-len 512 --population 32 --draw-chunk 4 --mode all 57 s/step, 17.9 evidence tok/s, 8.46 GB peak
8 GB GPU --batch 1 --seq-len 256 --population 32 --draw-chunk 4 --mode all ~1/4 the tokens per step
CPU only --device cpu --compute-dtype float32 --batch 1 --seq-len 128 --population 16 --draw-chunk 4 slow but valid

--mode layer isolates each layer's noise the way the paper does and roughly doubles gradient alignment, but it costs ~20x more forwards per step. all is the sane default at 300M.

Rules of the baton

  1. Pull before you train. Never train on a checkpoint you did not pull.
  2. Push one round ahead. If push says the remote round is ahead, pull again and re-apply your work; do not --force over someone else's round.
  3. lock.json is advisory (24 h lease). The round guard is the enforcement.
  4. Keep the token in your environment, never in a file you commit.
  5. Report what you ran: the train command is echoed into history.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support