| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - mlfoundations/dclm-baseline-1.0-parquet |
| | language: |
| | - en |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # Covenant-72B |
| |
|
| | ## Model Overview |
| |
|
| | **Covenant-72B** is the largest permissionless collaboratively trained language |
| | model, trained entirely from scratch at the 72 billion parameter scale on 1.1 |
| | trillion tokens of English text. |
| |
|
| |  |
| |
|
| | For more details, see the [technical report](https://arxiv.org/abs/2603.08163). |
| | This is a base model. See [Covenant-72B-Chat](https://huggingface.co/1Covenant/Covenant-72B-Chat) for the instruction-tuned variant. |
| |
|
| | **Covenant-72B** was trained with 20+ globally distributed participants |
| | coordinated via decentralized infrastructure on the Bittensor blockchain. |
| | Unlike prior collaborative training efforts that use whitelisted compute, |
| | Covenant-72B is the first to achieve this scale with fully permissionless |
| | participation. Training used the SparseLoCo communication-efficient optimizer |
| | to reduce bandwidth requirements across distributed nodes. |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | import torch |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model = AutoModelForCausalLM.from_pretrained( |
| | "1Covenant/Covenant-72B", |
| | torch_dtype=torch.bfloat16, |
| | device_map="auto", |
| | ) |
| | tokenizer = AutoTokenizer.from_pretrained("1Covenant/Covenant-72B") |
| | |
| | input_text = "The theory of general relativity" |
| | input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device) |
| | output_ids = model.generate(input_ids, max_new_tokens=100) |
| | print(tokenizer.decode(output_ids[0], skip_special_tokens=True)) |
| | ``` |
| |
|
| | ## Model Details |
| |
|
| | - **Compute Participants**: 20+ independent contributors on Bittensor |
| | - **Minimum Compute per Participant**: 8×B200 or equivalent |
| | - **Model License**: Apache 2.0 |
| |
|
| | ## Technical Specifications |
| |
|
| | | Parameter | Value | |
| | | ------------------------- | ------------------------------ | |
| | | Parameter Size | 72B | |
| | | Architecture | LLaMA-style (LlamaForCausalLM) | |
| | | Number of Layers | 80 | |
| | | Number of Attention Heads | 64 (8 KV heads) | |
| | | Hidden Size | 8192 | |
| | | Intermediate Size | 28672 | |
| | | Head Dimension | 128 | |
| | | Vocabulary Size | 262,144 | |
| |
|
| | **Training Details**: |
| |
|
| | - **Dataset**: [DCLM-baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet) |
| | - **Tokens**: 1.1 Trillion |
| | - **Optimizer**: SparseLoCo (communication-efficient optimizer) |
| |
|
| | ## Performance on Benchmarks |
| |
|
| | _All results are 0-shot acc_norm (%) unless noted._ |
| |
|
| | | Model | Size | Tokens | ARC-C | ARC-E | PIQA | OBQA | HellaSwag | WinoGrande\* | MMLU\* | |
| | | :----------------- | ---: | -----: | ----: | ----: | ----: | ----: | --------: | -----------: | -----: | |
| | | **Covenant-72B** | 72B | 1.1T | 56.83 | 80.93 | 81.56 | 44.00 | 80.61 | 75.85 | 67.11 | |
| | | INTELLECT-1 | 10B | 1T | 44.80 | 71.76 | 77.37 | 43.80 | 70.26 | 63.30 | 32.69 | |
| | | Psyche Consilience | 40B | 1.2T | 31.14 | 55.77 | 76.12 | 35.20 | 63.67 | 56.99 | 24.23 | |
| | | LLM360 K2 ckpt_108 | 65B | 420B | 45.73 | 70.54 | 80.90 | 43.20 | 78.23 | 71.90 | 50.01 | |
| | | LLM360 K2 | 65B | 1.4T | 53.75 | 75.97 | 82.54 | 48.00 | 82.86 | 76.40 | 65.51 | |
| | | LLaMA-2-7B | 7B | 2T | 45.05 | 73.82 | 78.73 | 44.20 | 76.18 | 69.38 | 41.73 | |
| | | LLaMA-2-70B | 70B | 2T | 57.42 | 79.55 | 82.59 | 49.40 | 84.34 | 80.43 | 65.63 | |
| | |
| | _\*WinoGrande uses acc; MMLU uses acc._ |
| | |
