README.md

metadata

base_model:
  - meta-llama/Llama-3.1-8B-Instruct
license: llama3.1
pipeline_tag: text-generation
library_name: vllm

Llama-3.1-SwiftKV-8B-Instruct: Accelerated with SuffixDecoding

The Llama-3.1-SwiftKV-8B-Instruct model is an iteration of the Llama-3.1-8B model optimized with SwiftKV, an inference optimization technique that goes beyond traditional key-value (KV) cache compression. This model is designed to be effectively accelerated by SuffixDecoding, an extreme speculative decoding method for emerging AI applications, as presented in the paper SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications.

Project Page (SuffixDecoding): https://suffix-decoding.github.io Code Repository (Arctic Inference): https://github.com/snowflakedb/ArcticInference

SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications

SuffixDecoding is a novel method that utilizes efficient suffix trees to cache long token sequences from prompts and previous outputs. By adaptively speculating more tokens when acceptance likelihood is high and fewer when it is low, SuffixDecoding effectively exploits opportunities for longer speculations while conserving computation when those opportunities are limited. Evaluations on agentic benchmarks, including SWE-Bench and Text-to-SQL, demonstrate that SuffixDecoding achieves speedups of up to 5.3x, outperforming state-of-the-art methods -- 2.8x faster than model-based approaches like EAGLE-2/3 and 1.9x faster than model-free approaches such as Token Recycling.

SwiftKV: Accelerating Enterprise LLM Workloads

SwiftKV is a series of inference optimizations that goes beyond traditional key-value (KV) cache compression. This method reduces computational overhead during prompt processing by combining model rewiring and knowledge-preserving self-distillation, allowing prefill tokens to skip up to half the model's layers. SwiftKV achieves up to 2x improvements in throughput, latency, and cost efficiency with minimal accuracy loss, making LLM deployments more performant and economically viable.

For more details about SwiftKV:

• ❄️ SwiftKV: Accelerating Enterprise LLM Workloads with Knowledge Preserving Compute Reduction (blog)
• 📝 SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (arXiv)
• 🚀 Getting started guide

Arctic Inference: The Orchestrating Framework

SuffixDecoding, along with SwiftKV and other innovations, are components of Arctic Inference, an open-source vLLM plugin from Snowflake AI Research that delivers the fastest and most cost-effective open-source inference for LLMs and Embeddings.

Revisions

• release-2508 (Aug 2025): Updated model weights for long-context up to 128K
• release-2412 (Dec 2024): Initial model release

Performance Metrics (SwiftKV)

To evaluate SwiftKV’s performance, we focus on the following key metrics (see more details in our blog):

• Combined throughput: The total number of input and output tokens processed per second. This determines:
  • For batch processing, the time required to complete jobs.
  • For interactive use, the volume of concurrent requests a system can handle.
• TTFT: The latency between a user request and receiving the first token in the response.
• TPOT: The latency between subsequent tokens after the first token.

Combined input and output throughput for Llama 3.1 70B (left) and Llama 3.1 405B (right) across a range of input lengths (bottom).

TTFT (top) and TPOT (bottom) for input lengths 2000 (left), 8000 (middle), and 32000 (right) for Llama 3.1 405B fp8 model. For each experiment, a range of different request arrival rates is simulated. Each request generates 256 output tokens.

Eval Metrics (SwiftKV)

For a full breakdown on evaluation metrics and performance impact please refer to our blog and arXiv paper but below we've outlined some relevant evaluation metrics.

Llama-3.1-405B-Instruct-FP8	Arc Challenge	Winogrande	HellaSwag	TruthfulQA	MMLU	MMLU cot	GSM8K	Avg
Baseline	94.7	87.0	88.3	64.7	87.5	88.1	96.1	86.6
50% SingleInputKV	94.0	86.3	88.1	64.2	85.7	87.5	95.2	85.9

Llama-3.1-8B-Instruct	Arc Challenge	Winogrande	HellaSwag	TruthfulQA	MMLU	MMLU cot	GSM8K	Avg
Baseline	82.00	77.90	80.40	54.56	67.90	70.63	82.56	73.71
50% SingleInputKV	80.38	78.22	79.30	54.54	67.30	69.73	79.45	72.70

Sample Usage (with SuffixDecoding via vLLM)

To leverage SuffixDecoding with this model, use arctic-inference to patch vLLM.

First, install the arctic-inference package:

pip install arctic-inference[vllm]

Once installed, Arctic Inference automatically patches vLLM. You can then use vLLM's familiar APIs with SuffixDecoding and other optimizations enabled.

Serving

ARCTIC_INFERENCE_ENABLED=1 vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct \
    --quantization "fp8" \
    --tensor-parallel-size 1 \
    --ulysses-sequence-parallel-size 2 \
    --enable-shift-parallel \
    --speculative-config '{
        "method": "arctic",
        "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true,
        "disable_by_batch_size": 64
    }'

Offline Inference

Save the following script to arctic_example.py:

import vllm
from vllm import LLM, SamplingParams

vllm.plugins.load_general_plugins()

llm = LLM(
    model="Snowflake/Llama-3.1-SwiftKV-8B-Instruct",
    quantization="fp8",
    tensor_parallel_size=1,
    ulysses_sequence_parallel_size=2,
    enable_shift_parallel=True,
    speculative_config={
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": True,
        "disable_by_batch_size": 64,
    },
)

conversation = [
    {
        "role": "user",
        "content": "Write an essay about the importance of higher education.",
    },
]

sampling_params = SamplingParams(temperature=0.0, max_tokens=800)

outputs = llm.chat(conversation, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

Run the script with Arctic Inference enabled:

ARCTIC_INFERENCE_ENABLED=1 python arctic_example.py

Citation

If you find this model or the related work useful, please consider citing the Arctic Inference and SuffixDecoding papers:

@misc{arcticinference2025,
      title={Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI},
      author={Samyam Rajbhandari and Mert Hidayetoglu and Aurick Qiao and Ye Wang and Juncheng Yang and Jeff Rasley and Michael Wyatt and Yuxiong He},
      year={2025},
      url={https://arxiv.org/abs/2507.11830},
}

@misc{suffixdecoding2024,
      title={SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications},
      author={Ye Wang and Samyam Rajbhandari and Juncheng Yang and Aurick Qiao and Mert Hidayetoglu and Michael Wyatt and Jeff Rasley and Yuxiong He},
      year={2024},
      eprint={2411.04975},
      archivePrefix={arXiv},\
      primaryClass={cs.CL},\
      url={https://arxiv.org/abs/2411.04975},\
}