Covenant72B
Covenant72B is the largest permissionless collaboratively trained language model trained entirely from scratch at the 72 billion parameter scale.
It is being trained with 20+ globally distributed participants coordinated via decentralized infrastructure on the Bittensor blockchain.
Checkpoint-One marks the first release, corresponding to 200 billion tokens processed. Model files are available in the Checkpoint-One branch. Future checkpoints will be updated here.
Training Details
| Property | Value |
|---|---|
| Model size | 72B |
| Architecture | LLaMA-style |
| Target token budget | 1.2T (210B for current checkpoint) |
| Compute participants | 20+ |
| Minimal compute per participant | 8×B200 or equivalent |
| Dataset | DCLM-baseline |
| Optimizer | SparseLoCo (communication-efficient optimizer) |
Performance on Benchmarks
All results are 0-shot acc-norm (%)
| Model | Compute Environment / Permissions | Size | Tokens | ARC-C | ARC-E | PIQA | OpenBookQA | HellaSwag | Winogrande | MMLU |
|---|---|---|---|---|---|---|---|---|---|---|
| Intellect-1 | Over the internet / White List | 10B | 1T | 44.8 | 71.6 | 77.7 | 43.6 | 70.5 | 63.1 | 32.7 |
| Psyche Consilience-7Y9 | Over the internet / White List | 40B | 1.2T | 31.1 | 55.8 | 76.1 | 34.8 | 63.7 | 57.0 | 24.2 |
| Covenant72B – Checkpoint One | Over the internet / Permissionless | 70B | 210B | 46.2 | 72.6 | 79.2 | 43.0 | 73.5 | 70.3 | 38.0 |
| K2 Checkpoint 54 | Centralized Cluster | 65B | 210B | 41.8 | 69.5 | 80.1 | 42.4 | 74.9 | 68.9 | 33.7 |
For more details, refer to Checkpoint One on Templar Research.
- Downloads last month
- 7
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support