File size: 4,436 Bytes
0c1c001 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
library_name: peft
tags:
- qbc
- aethersmind
- decentralised-training
- diloco
- lora
- mixture-of-experts
model-index:
- name: aether-v7.2-unified
results:
- task:
type: text-generation
dataset:
name: MMLU (full, 14042 questions, uncapped)
type: mmlu
metrics:
- type: accuracy
value: 0.7109
- task:
type: text-generation
dataset:
name: GSM8K (full, 1319 questions, uncapped)
type: gsm8k
metrics:
- type: accuracy
value: 0.7498
---
# aether-v7.2-unified
The Sephirot adapter served by [qbc.network](https://qbc.network), and the first
adapter produced by our decentralised training loop to pass a full evaluation and
be promoted into the live served model. It succeeds
[aether-v7.1-unified](https://huggingface.co/QuantumAI-Blockchain/aether-v7.1-unified).
It is a small adapter over a **frozen Qwen2.5-7B-Instruct base (Q4_K_M)**: ten
low-rank experts with a top-2 router, about 1.18 million trainable parameters. The
base model is not redistributed here; load `Qwen/Qwen2.5-7B-Instruct` and apply this
adapter on top. Serving precision is BF16.
## On-chain attestation
This model's identity is attested on the QBC chain. The on-chain `weight_root` is a
SHA-256 over (base, adapter, tokenizer, config):
- **weight_root:** `0x3e99b3d2f4e4d5eacbb3fe8d7e2f709dbfc48deb34407035383b3a5263e29e4e`
- **registry round:** 2, **finalised** by a 4-of-5 validator quorum
- **submitted at block:** 1698355
- **content root of the trained adapter:** `0x157a01b9`
## How it was trained
Trained with a DiLoCo-style local-training-plus-aggregation loop across a fleet of
ordinary 8GB consumer GPUs. A vocab-chunked backward pass keeps a full 7B adapter
step inside roughly 7.5GB of video memory, so commodity cards can train the served
model. Training runs as short payable rounds of about 35 minutes: a worker draws a
fresh shuffled batch from a large corpus, trains the adapter locally, and submits a
candidate. A candidate is accepted only if it beats the previous accepted candidate
on a fixed held-out set it never trains on. Ten rounds produced seven accepted
units, and the best by full held-out evaluation is this release.
## Evaluation
Measured on this adapter over the frozen base, as a snapshot from the promotion
evaluation (not a live probe):
| Measurement | Base | This adapter | Change |
| --- | --- | --- | --- |
| Held-out cross-entropy (clean 1000-example corpus, nats) | 2.9014 | 2.5609 | -0.3405 |
| MMLU (full 14042, uncapped) | 0.7116 | 0.7109 | -0.07pp (flat) |
| GSM8K (full 1319, uncapped) | 0.7248 | 0.7498 | +2.50pp |
The BF16 weights served here reproduce the held-out CE of the F32 candidate to four
decimals, so the -0.3405 nats generalisation gain holds at serving precision.
Generalisation on unseen text improved, general knowledge held flat, and multi-step
maths reasoning improved, at no measured cost to base capability. Against the
previously-served adapter, the promotion gate measured a further held-out CE
improvement (2.5912 to 2.4811 on the gate's own holdout).
## Scope and honesty
This adapter is the product of decentralised **training** with an owner-gated
promotion into the served model. This is Stage 1 of a four-stage plan. It is **not**
yet a permissionless model update: that stage is designed, not built, and is gated
on robust aggregation that survives a malicious contributor, a stake-and-slash
economic layer, and honest independent re-derivation of the claimed improvement.
During this work our own evaluation flattered us three times, through memorisation on
training data, a sampling-window bug, and a mixed-scale gate that false-rejected a
good round. Each was caught and fixed before any number here was trusted. The full
account is in the
[engineering write-up](https://qbc.network/blog/public-gpus-improved-the-served-model).
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base, "QuantumAI-Blockchain/aether-v7.2-unified")
```
## Licence
Released under Apache-2.0, matching the Qwen2.5-7B-Instruct base licence.
|