DFlash models have rope_theta=500000 but main model uses default 10000 — causes positional encoding mismatch at >32K context
#8
by valentijnvenus - opened
Issue: rope_theta mismatch between dFlash and main model
The dFlash speculative decoding models (both BF16 and INT4) have rope_theta: 500000.0 in config.json,
while the main model (poolside/Laguna-S-2.1 and cyankiwi/Laguna-S-2.1-AWQ-INT4) have rope_theta: NOT SET
(defaults to 10000.0).
This 50x difference in rope_theta causes the dFlash draft model to produce predictions in a different
positional encoding space than the main model.
Observed symptoms
- At ≤32K context: dFlash works with ~47% average acceptance rate
- At 64K+ context: acceptance drops to 0-20%, causing garbage output
- Per-position: position 1 ~80-90%, positions 2-7 near 0% at long context
- At 0% acceptance, draft tokens are never accepted — wasted GPU cycles and output corruption
Model configs comparison
| Parameter | Main (poolside/Laguna-S-2.1) | DFlash BF16 | DFlash INT4 |
|-------------------------|------------------------------|-------------|-------------|
| rope_theta | 10000 (default) | 500000.0 | 500000.0 |
| max_position_embeddings | 1048576 | 1048576 | 262144 |
| num_hidden_layers | 48 | 6 | 6 |
| hidden_size | 3072 | 3072 | 3072 |
Proposed fix
Either:
1. Retrain the dFlash model with rope_theta=10000 to match the main model
2. Update the main model config to rope_theta=500000 (if intentional)
Until resolved, dFlash speculative decoding is only reliable up to ~32K context.
Hi @valentijnvenus , thanks for opening this thread! I'm just closing the loop here: for every latest precision base model, please pull the latest associated DFlash for that precision.