| --- |
| license: apache-2.0 |
| tags: |
| - reinforcement-learning |
| - contextual-bandit |
| - offline-rl |
| - speculative-decoding |
| - energy-efficiency |
| - sglang |
| --- |
| |
| # EAGLE3 Speculative Decoding -- Energy-Aware Policy Models |
|
|
| Eight models, one problem: pick `(speculative_num_steps, |
| speculative_eagle_topk, speculative_num_draft_tokens)` for sglang + EAGLE3 so |
| GPU energy utilization lands inside a 95-98% band. All trained on the |
| [`eagle3-speculative-decoding-energy-sweep`](https://huggingface.co/datasets/Pradheep1647/eagle3-speculative-decoding-energy-sweep) |
| dataset. Full writeup, sweep mechanism, and live-validated results: |
| [project README](https://github.com/HyperKuvid-Labs/energy_throttling_llms). |
| |
| **Shared I/O contract** -- input is a 4-dim state |
| `[batch_size/8.0, gpu_temp_c/100.0, gpu_mem_used_mb/8192.0, gpu_util_pct/100.0]`; |
| output is an index into the same 19-action space (`RL/policy.py` in the repo |
| above), decoded to the three sglang flags. |
| |
| | file | algorithm | format | load | |
| |---|---|---|---| |
| | `mlp_bandit/policy.pth` | contextual bandit (QNetwork) | torch state_dict | `QNetwork(4, 19).load_state_dict(torch.load(...))` | |
| | `lookup_table/model.json` | per-bs empirical best | JSON | `json.load(open(...))` -- `{batch_size: {"config": [...], "mean_reward": ...}}` | |
| | `linucb/model.npz` | LinUCB | numpy | `np.load(...)` -- `theta` (per-action weight vectors), `A` (per-action design matrices), `actions` | |
| | `thompson_sampling/model.npz` | Bayesian linear TS | numpy | `np.load(...)` -- `mean`, `cov` (per-action posteriors), `actions` | |
| | `gbt/model.joblib` | gradient boosted trees | sklearn | `joblib.load(...)` -- `GradientBoostingRegressor`, features `[state, action_idx/19]` | |
| | `doubly_robust/model.joblib` | doubly robust (direct model) | sklearn | `joblib.load(...)` -- `LinearRegression`, IPS correction term is not persisted (recomputed from raw sweep data at pick time) | |
| | `cql/policy.pth` | conservative Q-learning | torch state_dict | `QNetwork(4, 19).load_state_dict(torch.load(...))` | |
| | `bcq/policy.pth` | discrete BCQ | torch state_dict | `torch.load(...)` -- dict with `behavior_state_dict`, `q_state_dict`, `actions` | |
|
|
| ## Which one to actually use |
|
|
| `mlp_bandit`, `lookup_table`, and `doubly_robust` agree on every batch size |
| and are the live-validated picks. `cql` and `bcq` collapsed to the |
| non-speculative baseline past `bs=1` (overly conservative default |
| hyperparameters against this reward scale) and are **not recommended** -- |
| kept here for completeness, not as a suggested pick. See the project README |
| for the full live A/B numbers per algorithm. |
|
|
| ## Hardware this was validated on |
|
|
| RTX 4060 Laptop GPU (8GB), `unsloth/Llama-3.2-1B-Instruct` target + |
| `rescommons/SpecForge-EAGLE3-Llama-3.2-1B-Instruct` draft, 80W power cap. |
| Picks are specific to this hardware/model pair -- retrain on the linked |
| dataset (or a fresh sweep) before trusting these on different hardware. |
|
|
