Typhoon OCR V2 0.8B: EAGLE-3 draft head
This repository holds an optional draft head for Typhoon OCR V2 0.8B. It makes the model generate text faster in vLLM through speculative decoding. It does not work on its own: you always run it together with typhoon-ai/typhoon-ocr2-0.8b. If you do not need the speed-up, you do not need this repository.
At each step, the draft head proposes the next 3 tokens, and Typhoon OCR V2 0.8B checks them in a single forward pass and keeps the ones it agrees with. In our tests about 3.1 tokens come out of each step instead of 1, which makes generation 1.5 to 1.7 times faster, depending on the load. For the 2B model, use typhoon-ai/typhoon-ocr2-2b-eagle3 instead.
Usage
The draft head works with vLLM only. We tested it with vLLM 0.33.0. It does not work with Transformers, llama.cpp (GGUF), Ollama or MLX.
vLLM server
vllm serve typhoon-ai/typhoon-ocr2-0.8b \
--served-model-name typhoon-ocr2 \
--max-model-len 40960 \
--limit-mm-per-prompt '{"image": 1}' \
--mm-processor-kwargs '{"min_pixels": 2048, "max_pixels": 16777216}' \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--speculative-config '{"method": "eagle3", "model": "typhoon-ai/typhoon-ocr2-0.8b-eagle3", "num_speculative_tokens": 3}'
This is the server command from the Typhoon OCR V2 0.8B model card with one added line, --speculative-config. Requests, prompts and outputs are the same as described there.
vLLM offline
from vllm import LLM
llm = LLM(
model="typhoon-ai/typhoon-ocr2-0.8b",
max_model_len=40960,
limit_mm_per_prompt={"image": 1},
mm_processor_kwargs={"min_pixels": 2048, "max_pixels": 16777216},
speculative_config={
"method": "eagle3",
"model": "typhoon-ai/typhoon-ocr2-0.8b-eagle3",
"num_speculative_tokens": 3,
},
)
Use the prompts, sampling settings (greedy, temperature=0) and enable_thinking=False chat template option from the main model card.
Speed
All measurements use vLLM 0.33.0 in its default compiled mode on one H100 GPU, greedy decoding, 3 draft tokens and 30 pages that were held out from the draft head's training data.
| Setting | Without draft head | With draft head | Speed-up |
|---|---|---|---|
| One request at a time | 327 tokens/s | 546 tokens/s | 1.7ร |
| 16 requests at a time | 1,488 tokens/s | 2,570 tokens/s | 1.5ร |
On average 3.1 tokens come out of each step (the maximum is 4), and 70% of the proposed tokens are accepted. An earlier measurement, made while another job was running on the same GPU, gave 1.7ร in both settings. The gain is usually largest when the GPU is not fully loaded, and it becomes smaller as the number of concurrent requests grows.
The speed-up is smaller than for the 2B model (1.9 to 2.5ร), because the draft head of the smaller model predicts its tokens less accurately.
Accuracy
With greedy decoding, speculative decoding is meant to keep the model's output unchanged, because every token is checked by Typhoon OCR V2 0.8B. In practice the output can still differ slightly. Checking several tokens at once changes the order of GPU arithmetic a little, and when two tokens are almost tied, the other one can win. Repeated runs without the draft head differ in the same way: two runs of Typhoon OCR V2 0.8B in vLLM 0.33.0 give the same output on only 151 of the 320 MDPBench pages.
We ran MDPBench (320 English and Thai pages) once with the draft head and four times without it:
| MDPBench, vLLM 0.33.0 | Without draft head (4 runs) | With draft head (1 run) |
|---|---|---|
| Overall | 76.8 (76.3 to 76.9) | 76.9 |
| English pages | 76.8 (76.5 to 76.9) | 77.2 |
| Thai pages | 76.8 (76.1 to 77.1) | 76.6 |
| Pages with repetition loops | 6 to 11 | 7 |
Values without the draft head are the mean and range of the four runs. The run with the draft head is within or next to that range. Against the four runs, its overall change is +0.2 points, with a 95% interval from โ1.0 to +1.3 (bootstrap over pages and runs), so we found no change in accuracy. Its output is identical to a run without the draft head on 152 to 160 of the 320 pages, about as often as two runs without it agree with each other.
Training
The draft head was trained with speculators (EAGLE-3, online training), with the same data and settings as the 2B draft head.
- Data. The same 20,000 page images and prompts as for the 2B draft head, sampled from the SFT and GRPO training data of Typhoon OCR V2, with extra weight on full-page JSON parsing and on Thai pages. The answers were generated by Typhoon OCR V2 0.8B itself (greedy decoding), so the draft head learns to predict this model's own outputs. After removing outputs that looped or came back empty and 16 images that could not be read, 18,988 conversations remained.
- Model. One transformer layer that reads hidden states from layers 2, 12 and 21 of Typhoon OCR V2 0.8B, with a draft vocabulary of the 32,000 most frequent tokens.
- Training. 5 epochs, learning rate 1e-4, sequences of up to 16,384 tokens, on one H100 GPU. On held-out conversations, the first, second and third proposed tokens are correct 86%, 74% and 64% of the time.
Files
| File | Content |
|---|---|
model.safetensors |
the draft head's weights (bfloat16), including a copy of the input embeddings |
config.json, config.py |
the speculators configuration that vLLM reads |
Limitations
- vLLM only, tested with vLLM 0.33.0. Other versions may need a different configuration.
- It is trained for Typhoon OCR V2 0.8B and does not work with Typhoon OCR V2 2B or any other model.
- Outputs can differ slightly from decoding without the draft head (see Accuracy). If you need outputs that exactly match a run without it, do not use it.
- The speed was measured on 30 pages, and the MDPBench comparison uses one run with the draft head. The speed-up was measured on document pages similar to the training data; on very different inputs, fewer proposed tokens may be accepted and the speed-up will be smaller.
License
Apache 2.0, the same as Typhoon OCR V2 0.8B.
- Downloads last month
- 53
Model tree for typhoon-ai/typhoon-ocr2-0.8b-eagle3
Base model
Qwen/Qwen3.5-0.8B-Base