|
|
We propose **PyroDash**, a token-level dynamic reasoning paradigm for collaborative inference between small and large language models. PyroDash enables the small model to autonomously emit the control token `<|llm_offload|>` during autoregressive streaming decoding; the collaboration engine then dynamically offloads the local reasoning chain to a large model based on this control signal. This approach requires neither an additional router model nor retraining of the large model, and is naturally compatible with closed-source LLM services. |
<\|llm_offload\|> |
| SFT dataset | [EasyHard-24K](https://huggingface.co/datasets/pyromind/easyhard-24k) |
| GRPO dataset | [BytedTsinghua-SIA/DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k) |
| Expert LLM (train/eval) | GLM-5.2-FP8 (frozen) |
| Precision | bfloat16 |
## Quick Start
### 1. Setup
```bash
git clone https://github.com/PyroMind-Dynamics/pyroDash.git
cd pyroDash
pip install -r requirements.txt
```
### 2. Run evaluation (`evaluation/math_eval.sh`)
Edit placeholders in [`evaluation/math_eval.sh`](https://github.com/PyroMind-Dynamics/pyroDash/blob/main/evaluation/math_eval.sh), then:
```bash
bash evaluation/math_eval.sh
```
The script (1) starts a local **vLLM** server for the small model on port `8001`, (2) runs `math_eval.py`, and (3) stops vLLM on exit.
#### Parameters
| Variable / flag | Meaning | Example |
|-----------------|---------|---------|
| `MODEL` | Local merged model path (vLLM serve + tokenizer) | `/path/to/your/merged_model` |
| `--glm-base-url` | OpenAI-compatible API for the large/relay model | `http://your-glm-host:8000/v1` |
| `--glm-api-key` | API key for that endpoint | `your-glm-api-key` |
| `--glm-model` | Served model name on the GLM side | `your-glm-model` |
| `--output-dir` | Per-dataset JSON output directory | `./results_500` |
| `--datasets` | Benchmarks (space-separated) | `gsm8k minerva olympiad aime2024 aime2025` |
Tokenizer must include the special token `<|llm_offload|>`.
## Results
