---
tags:
  - custom_generate
library_name: transformers
---

# ASD — Approximate Speculative Decoding

A training-free, verifier-side acceptance policy for **greedy assisted decoding** in Hugging Face Transformers.

Standard greedy speculative decoding discards the entire draft suffix at the first position where the draft token differs from the target argmax. ASD relaxes this strictly deterministic check with three explicit, bounded rules so that near-optimal draft tokens can still be accepted.

> Paper: [ASD: Approximate Speculative Decoding](https://arxiv.org/abs/2608.03447)
> Reference implementation: https://github.com/Kissmetothemoon/ASD

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3-14B"
assistant_id = "your-draft-model"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
assistant_model = AutoModelForCausalLM.from_pretrained(assistant_id, device_map="auto")

inputs = tokenizer("The quick brown", return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    assistant_model=assistant_model,
    custom_generate="ynFeng/approximate-speculative-decoding",
    trust_remote_code=True,
    assistant_asd_budget=2.0,
    assistant_asd_local_ratio=0.25,
    assistant_asd_max_mismatches=2,
)
```

## Parameters

| Name | Type | Default | Description |
|------|------|---------|-------------|
| `assistant_asd_budget` | `float` | `None` | Request-level cumulative regret budget `B`. `None` = unlimited (only ratio / mismatch caps apply). `0.0` forces strict argmax verification. |
| `assistant_asd_local_ratio` | `float` | `None` | Max allowed `r_i / (K - i)` for a relaxed mismatch at draft position `i`, where `K` is the candidate length. `None` = unlimited. |
| `assistant_asd_max_mismatches` | `int` | `None` | Max number of relaxed mismatches accepted per block. `None` = unlimited. `0` forces strict verification. |

If any parameter is `0`, the policy **exactly reduces to strict greedy verification** — this is the built-in correctness baseline.

## Rules

For each drafted token `x_i` with target logits `z_i`:

1. `r_i = max_v z_i(v) - z_i(x_i)` (`r_i = 0` iff `x_i` is the target argmax).
2. Accept the draft prefix while **all** hold:
   - request-level cumulative regret `<= assistant_asd_budget`
   - local regret `r_i / (K - i) <= assistant_asd_local_ratio`
   - relaxed mismatches in the current block `<= assistant_asd_max_mismatches`
3. On the first infeasible position, commit the target argmax as the bonus token.

## Empirical results (paper)

Qwen3-14B + DSpark block-7 draft, 8×L20, greedy:

| Setting | Throughput | GSM8K |
|---------|-----------|-------|
| Strict verification | 66.21 tok/s | 79.682 % |
| ASD (`B=2.0625`) | 75.04 tok/s (+13.34 %) | 79.454 % (-0.227 pp) |
| ASD (`B=0`) | token-identical to strict on all 1319 requests | identical |

## Requirements

```text
transformers>=5.19.0,<5.20.0
torch>=2.0.0
```

The acceptance loop mirrors `GenerationMixin._assisted_decoding` internals, so the
transformers minor version is pinned. Verified on `transformers==5.19.0`, `torch==2.6.0`.

## Validation

Checked with Qwen2.5-1.5B-Instruct (target) + Qwen2.5-0.5B-Instruct (draft) on an
NVIDIA L20 (`scripts/example.py` reproduces):

- `assistant_asd_budget=0` and no-knob calls are **token-identical** to native
  greedy assisted decoding on every tested prompt (also with `prompt_lookup_num_tokens`);
- relaxed runs are deterministic across repeated calls (no RNG anywhere in the policy);
- `do_sample=True`, combining with `assistant_ensemble_weight`, ASD knobs without a
  draft source, and negative budgets all raise `ValueError`.

ASD also works with prompt-lookup drafting (`prompt_lookup_num_tokens`) — unlike
`assistant_ensemble_weight`, the regret rule needs target logits only.

## License

Apache-2.0