Trim README: drop training details, condense architecture and acceptance notes
Browse files
README.md
CHANGED
|
@@ -6,28 +6,21 @@ private: true
|
|
| 6 |
# Muse-Spark-1.2-DSpark
|
| 7 |
|
| 8 |
DSpark speculative-decoding draft model for **Muse-Spark-1.2-816B-A42B-NVFP4-open**
|
| 9 |
-
(816B-A42B MoE target, NVFP4).
|
| 10 |
-
(commit `6f102eebc1aa`).
|
| 11 |
|
| 12 |
## DSpark architecture
|
| 13 |
|
| 14 |
-
- 5-layer decoder-only draft
|
| 15 |
-
|
| 16 |
-
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
- Proposes
|
| 20 |
-
with Markov head (rank 512)
|
| 21 |
-
- Trained from scratch (FP32 master weights, 3,600 AdamW updates, LR 3e-4 cosine) on on-policy
|
| 22 |
-
data regenerated by the target; loss 0.1·CE + 1.8·TV with positional decay γ=4
|
| 23 |
- ~10.6 GB, 74 tensors, safetensors
|
| 24 |
|
| 25 |
-
## Acceptance rates
|
| 26 |
|
| 27 |
-
|
| 28 |
-
7 probabilistic draft tokens, adaptive verification off; acceptance arms at temperature 1.0 /
|
| 29 |
-
top-p 0.95 (MQA at 0.7). Acceptance = accepted / proposed draft tokens; tokens/cycle includes
|
| 30 |
-
the target token; positions divide by all verification cycles.
|
| 31 |
|
| 32 |
| Dataset | Requests | Acceptance | Tokens/cycle | Positions 1–7 (%) |
|
| 33 |
|---|---:|---:|---:|---|
|
|
@@ -53,4 +46,4 @@ GSM8K MQA accuracy (separate from acceptance): **94.01%** (1,319 requests, 5-sho
|
|
| 53 |
"rejection_sample_method":"block"}
|
| 54 |
```
|
| 55 |
|
| 56 |
-
Requires
|
|
|
|
| 6 |
# Muse-Spark-1.2-DSpark
|
| 7 |
|
| 8 |
DSpark speculative-decoding draft model for **Muse-Spark-1.2-816B-A42B-NVFP4-open**
|
| 9 |
+
(816B-A42B MoE target, NVFP4).
|
|
|
|
| 10 |
|
| 11 |
## DSpark architecture
|
| 12 |
|
| 13 |
+
- 5-layer decoder-only draft (hidden 8192, 128/16 attention/KV heads, FFN 16384), BF16,
|
| 14 |
+
causal sliding-window attention (2048)
|
| 15 |
+
- Full 202,048-token target vocabulary; frozen target LM head
|
| 16 |
+
- Conditioned on target hidden states tapped at layers [13, 29, 45, 57, 61] plus the
|
| 17 |
+
final hidden state
|
| 18 |
+
- Proposes 7 tokens per step (sample-from-anchor); learned Markov confidence head (rank 512)
|
|
|
|
|
|
|
|
|
|
| 19 |
- ~10.6 GB, 74 tensors, safetensors
|
| 20 |
|
| 21 |
+
## Acceptance rates
|
| 22 |
|
| 23 |
+
Measured with 7 probabilistic draft tokens; temperature 1.0 / top-p 0.95 (MQA at 0.7).
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
| Dataset | Requests | Acceptance | Tokens/cycle | Positions 1–7 (%) |
|
| 26 |
|---|---:|---:|---:|---|
|
|
|
|
| 46 |
"rejection_sample_method":"block"}
|
| 47 |
```
|
| 48 |
|
| 49 |
+
Requires a Muse Spark–compatible serving runtime.
|