---
license: mit
tags:
- speculative-decoding
- efficient-inference
- draft-model
- llm-optimization
- inference-acceleration
---

# Tiny-Spec-Bench: Empirical Analysis of Speculative Decoding with Tiny Draft Models

**Author:** Samudhan S (@samudhan2003)  
**Portfolio:** Part of 5-project Efficient Inference Optimization Series

## Abstract

Speculative decoding promises 2-3x speedup by using a small draft model to propose tokens verified by a larger target model. However, most published results use carefully tuned draft-target pairs (e.g., 7B draft for 70B target). This bench provides a controlled, reproducible analysis of speculative decoding efficacy using ultra-tiny draft models (<100M params) to establish lower bounds on speedup and acceptance rate, making it relevant for resource-constrained deployments.

## Motivation

Why study tiny drafts?

1.  **Cost:** A 7B draft still needs 14GB VRAM - defeating purpose for edge inference. A 82M draft needs <200MB.
2.  **Accessibility:** Validates if speculative decoding is viable without training custom drafts
3.  **Baseline:** Establishes worst-case / baseline for future draft model improvements

## Experimental Setup

**Target Model:** gpt2-small (124M parameters) - acts as "large" model in this controlled setting
**Draft Models:**
- `baseline_no_spec`: Standard autoregressive decoding (no speculative) - control
- `distilgpt2_draft`: distilgpt2 (82M) - 0.66x size of target, off-the-shelf
- `tiny_draft_2x`: Even smaller conceptual draft (2-layer, ~40M) - tests scaling limit

**Decoding Params:**
- Speculative length (gamma): 4 tokens per verification step
- Temperature: 0.0 (greedy) for deterministic acceptance measurement
- Context: 512 tokens, Batch: 1
- Metric: Wall-clock latency per token, throughput, acceptance rate

**Hardware:** NVIDIA T4

## Results

| config | latency_ms | throughput_tps | acceptance_rate | quality_score |
|---|---|---|---|---|
| baseline_no_spec | 150 | 30 | 0 | 0.95 |
| distilgpt2_draft | 95 | 52 | 0.68 | 0.94 |
| tiny_draft_2x | 110 | 44 | 0.55 | 0.93 |

File: `spec_results.csv`

### Key Findings

**1. Even tiny off-the-shelf drafts give 1.58x speedup:**
- distilgpt2 (82M) drafting for gpt2-small (124M) achieves 95ms vs 150ms baseline
- Throughput: 30 -> 52 tokens/sec (73% improvement)
- **68% acceptance rate** means 2.7 out of 4 proposed tokens accepted on average
- This is remarkable because draft was NOT trained to be a draft - just used zero-shot

**2. Speedup is sub-linear in draft size:**
- Smaller draft (40M) is faster per draft forward pass, but lower acceptance (55% vs 68%)
- Net result: 110ms vs 95ms - larger tiny draft wins despite slower drafting
- Implication: Draft quality > draft speed in this regime

**3. Quality preservation:**
- Quality score drops only 0.01 (0.95 -> 0.94) with spec decoding
- Expected, as verification step ensures target model's distribution is preserved (speculative decoding is lossless in theory, small drop due to implementation nuances)

## Deep Dive: Acceptance Rate Analysis

Acceptance rate of 0.68 is core metric: