ZixiQi commited on
Commit
41e7828
·
verified ·
1 Parent(s): 519a663

Trim README: drop training details, condense architecture and acceptance notes

Browse files
Files changed (1) hide show
  1. README.md +10 -17
README.md CHANGED
@@ -6,28 +6,21 @@ private: true
6
  # Muse-Spark-1.2-DSpark
7
 
8
  DSpark speculative-decoding draft model for **Muse-Spark-1.2-816B-A42B-NVFP4-open**
9
- (816B-A42B MoE target, NVFP4). Current head: **fresh_fp32_20260920 final, step 3600**
10
- (commit `6f102eebc1aa`).
11
 
12
  ## DSpark architecture
13
 
14
- - 5-layer decoder-only draft: hidden 8192, 128 attention heads / 16 KV heads, FFN 16384, BF16
15
- - Causal sliding-window attention (window 2048); RoPE theta 500,000; positions up to 1M
16
- - Reads target hidden states tapped at layers [13, 29, 45, 57, 61] (of 62) and reuses the
17
- last target hidden state as an auxiliary input
18
- - Full target vocabulary (202,048 tokens) with the frozen target LM head; identity `d2t` mapping
19
- - Proposes **7 tokens per step** (`block_size=7`, sample-from-anchor); learned confidence head
20
- with Markov head (rank 512)
21
- - Trained from scratch (FP32 master weights, 3,600 AdamW updates, LR 3e-4 cosine) on on-policy
22
- data regenerated by the target; loss 0.1·CE + 1.8·TV with positional decay γ=4
23
  - ~10.6 GB, 74 tensors, safetensors
24
 
25
- ## Acceptance rates (full suite, block verification)
26
 
27
- Served with the Muse Spark mpk runtime, TP4/EP, BF16, eager, no prefix caching;
28
- 7 probabilistic draft tokens, adaptive verification off; acceptance arms at temperature 1.0 /
29
- top-p 0.95 (MQA at 0.7). Acceptance = accepted / proposed draft tokens; tokens/cycle includes
30
- the target token; positions divide by all verification cycles.
31
 
32
  | Dataset | Requests | Acceptance | Tokens/cycle | Positions 1–7 (%) |
33
  |---|---:|---:|---:|---|
@@ -53,4 +46,4 @@ GSM8K MQA accuracy (separate from acceptance): **94.01%** (1,319 requests, 5-sho
53
  "rejection_sample_method":"block"}
54
  ```
55
 
56
- Requires the Muse Spark mpk serving runtime (`zixi/muse-spark-hf-refresh-20260916`).
 
6
  # Muse-Spark-1.2-DSpark
7
 
8
  DSpark speculative-decoding draft model for **Muse-Spark-1.2-816B-A42B-NVFP4-open**
9
+ (816B-A42B MoE target, NVFP4).
 
10
 
11
  ## DSpark architecture
12
 
13
+ - 5-layer decoder-only draft (hidden 8192, 128/16 attention/KV heads, FFN 16384), BF16,
14
+ causal sliding-window attention (2048)
15
+ - Full 202,048-token target vocabulary; frozen target LM head
16
+ - Conditioned on target hidden states tapped at layers [13, 29, 45, 57, 61] plus the
17
+ final hidden state
18
+ - Proposes 7 tokens per step (sample-from-anchor); learned Markov confidence head (rank 512)
 
 
 
19
  - ~10.6 GB, 74 tensors, safetensors
20
 
21
+ ## Acceptance rates
22
 
23
+ Measured with 7 probabilistic draft tokens; temperature 1.0 / top-p 0.95 (MQA at 0.7).
 
 
 
24
 
25
  | Dataset | Requests | Acceptance | Tokens/cycle | Positions 1–7 (%) |
26
  |---|---:|---:|---:|---|
 
46
  "rejection_sample_method":"block"}
47
  ```
48
 
49
+ Requires a Muse Spark–compatible serving runtime.