Improve model card: Add SuffixDecoding context, vLLM usage, project & repo links

README.md

@@ -1,55 +1,166 @@
 ---
-license: llama3.1
 base_model:
 - meta-llama/Llama-3.1-8B-Instruct
+license: llama3.1
+pipeline_tag: text-generation
+library_name: vllm
 ---
-# SwiftKV 
 
-The Snowflake AI Research team is releasing a series of SwiftKV optimized Llama-3.1 models. [SwiftKV](https://arxiv.org/abs/2410.03960) is a series of inference optimizations that goes beyond traditional key-value (KV) cache compression. This method reduces computational overhead during prompt processing by combining model rewiring and knowledge-preserving self-distillation, allowing prefill tokens to skip up to half the model's layers. SwiftKV achieves up to 2x improvements in throughput, latency, and cost efficiency with minimal accuracy loss, making LLM deployments more performant and economically viable.
+# Llama-3.1-SwiftKV-8B-Instruct: Accelerated with SuffixDecoding
+
+The `Llama-3.1-SwiftKV-8B-Instruct` model is an iteration of the Llama-3.1-8B model optimized with **SwiftKV**, an inference optimization technique that goes beyond traditional key-value (KV) cache compression. This model is designed to be effectively accelerated by **SuffixDecoding**, an extreme speculative decoding method for emerging AI applications, as presented in the paper [SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications](https://huggingface.co/papers/2411.04975).
+
+**Project Page (SuffixDecoding)**: [https://suffix-decoding.github.io](https://suffix-decoding.github.io)
+**Code Repository (Arctic Inference)**: [https://github.com/snowflakedb/ArcticInference](https://github.com/snowflakedb/ArcticInference)
+
+## SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
+
+SuffixDecoding is a novel method that utilizes efficient suffix trees to cache long token sequences from prompts and previous outputs. By adaptively speculating more tokens when acceptance likelihood is high and fewer when it is low, SuffixDecoding effectively exploits opportunities for longer speculations while conserving computation when those opportunities are limited. Evaluations on agentic benchmarks, including SWE-Bench and Text-to-SQL, demonstrate that SuffixDecoding achieves speedups of up to 5.3x, outperforming state-of-the-art methods -- 2.8x faster than model-based approaches like EAGLE-2/3 and 1.9x faster than model-free approaches such as Token Recycling.
+
+## SwiftKV: Accelerating Enterprise LLM Workloads
+
+SwiftKV is a series of inference optimizations that goes beyond traditional key-value (KV) cache compression. This method reduces computational overhead during prompt processing by combining model rewiring and knowledge-preserving self-distillation, allowing prefill tokens to skip up to half the model's layers. SwiftKV achieves up to 2x improvements in throughput, latency, and cost efficiency with minimal accuracy loss, making LLM deployments more performant and economically viable.
 
-For more details about SwiftKV and how to use it: 
+For more details about SwiftKV:
 * ❄️ [SwiftKV: Accelerating Enterprise LLM Workloads with Knowledge Preserving Compute Reduction (blog)](https://www.snowflake.com/engineering-blog/swiftkv-llm-compute-reduction/)
 * 📝 [SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (arXiv)](https://arxiv.org/abs/2410.03960)
 * 🚀 [Getting started guide](https://github.com/Snowflake-Labs/vllm/tree/swiftkv/examples/swiftkv)
 
+## Arctic Inference: The Orchestrating Framework
+
+SuffixDecoding, along with SwiftKV and other innovations, are components of **Arctic Inference**, an open-source vLLM plugin from Snowflake AI Research that delivers the fastest and most cost-effective open-source inference for LLMs and Embeddings.
+
+<p align="middle">
+<img src="https://github.com/snowflakedb/ArcticInference/raw/main/projects/arctic_inference/imgs/figure1.png" alt="Arctic Inference overview diagram" width="800">
+</p>
+
 ## Revisions
 
-* **release-2508 (Aug 2025):** Updated model weights for long-context up to 128K
-* **release-2412 (Dec 2024):** Initial model release
+*   **release-2508 (Aug 2025):** Updated model weights for long-context up to 128K
+*   **release-2412 (Dec 2024):** Initial model release
 
-## Performance Metrics
+## Performance Metrics (SwiftKV)
 
 To evaluate SwiftKV’s performance, we focus on the following key metrics (see more details in our [blog](https://www.snowflake.com/engineering-blog/swiftkv-llm-compute-reduction/)):
-* Combined throughput: The total number of input and output tokens processed per second. This determines:
-  * For batch processing, the time required to complete jobs.
-  * For interactive use, the volume of concurrent requests a system can handle.
-* TTFT: The latency between a user request and receiving the first token in the response.
-* TPOT: The latency between subsequent tokens after the first token.
+*   Combined throughput: The total number of input and output tokens processed per second. This determines:
+    *   For batch processing, the time required to complete jobs.
+    *   For interactive use, the volume of concurrent requests a system can handle.
+*   TTFT: The latency between a user request and receiving the first token in the response.
+*   TPOT: The latency between subsequent tokens after the first token.
 
 Combined input and output throughput for Llama 3.1 70B (left) and Llama 3.1 405B (right) across a range of input lengths (bottom).
 <img src="figure-4-full.png" alt="performance plot of llama-405B w. swiftkv" width="800">
 
-TTFT (top) and TPOT (bottom) for input lengths 2000 (left), 8000 (middle), and 32000 (right) for Llama 3.1 405B fp8 model. For each experiment, a range of different request arrival rates is simulated. Each request generates 256 output tokens. 
+TTFT (top) and TPOT (bottom) for input lengths 2000 (left), 8000 (middle), and 32000 (right) for Llama 3.1 405B fp8 model. For each experiment, a range of different request arrival rates is simulated. Each request generates 256 output tokens.
 <img src="figure-6.png" alt="performance plot of llama-405B w. swiftkv" width="700">
 
+## Eval Metrics (SwiftKV)
 
-## Eval Metrics
-
-For a full breakdown on evaluation metrics and performance impact please refer to our [blog](https://www.snowflake.com/engineering-blog/swiftkv-llm-compute-reduction/) and [arXiv paper]((https://arxiv.org/abs/2410.03960)) but below we've outlined some relevant evaluation metrics.
+For a full breakdown on evaluation metrics and performance impact please refer to our [blog](https://www.snowflake.com/engineering-blog/swiftkv-llm-compute-reduction/) and [arXiv paper](https://arxiv.org/abs/2410.03960) but below we've outlined some relevant evaluation metrics.
 
 | Llama-3.1-405B-Instruct-FP8 | Arc Challenge | Winogrande | HellaSwag | TruthfulQA | MMLU | MMLU cot | GSM8K | Avg |
-|-----------|---------------|------------|-----------|------------|------|----------|-------|-----|
-| Baseline | 94.7 | 87.0 | 88.3 | 64.7 | 87.5 | 88.1 | 96.1 | **86.6** |
-| 50% SingleInputKV | 94.0 | 86.3 | 88.1 | 64.2 | 85.7 | 87.5 | 95.2 | **85.9** |
+|:----------------------------|:--------------|:-----------|:----------|:-----------|:-----|:---------|:------|:----|
+| Baseline                    | 94.7          | 87.0       | 88.3      | 64.7       | 87.5 | 88.1     | 96.1  | **86.6** |
+| 50% SingleInputKV           | 94.0          | 86.3       | 88.1      | 64.2       | 85.7 | 87.5     | 95.2  | **85.9** |
 
 | Llama-3.1-8B-Instruct | Arc Challenge | Winogrande | HellaSwag | TruthfulQA | MMLU | MMLU cot | GSM8K | Avg |
-|-----------|---------------|------------|-----------|------------|------|----------|-------|-----|
-| Baseline | 82.00 | 77.90 | 80.40 | 54.56 | 67.90 | 70.63 | 82.56 | **73.71** |
-| 50% SingleInputKV | 80.38 | 78.22 | 79.30 | 54.54 | 67.30 | 69.73 | 79.45 | **72.70** |
-
-## Get started by serving SwiftKV on vLLM
-
-Instructions on how to use vLLM for both evaluation and performance benchmarks:
-https://github.com/Snowflake-Labs/vllm/tree/swiftkv/examples/swiftkv
-
+|:----------------------|:--------------|:-----------|:----------|:-----------|:-----|:---------|:------|:----|
+| Baseline              | 82.00         | 77.90      | 80.40     | 54.56      | 67.90 | 70.63    | 82.56 | **73.71** |
+| 50% SingleInputKV     | 80.38         | 78.22      | 79.30     | 54.54      | 67.30 | 69.73    | 79.45 | **72.70** |
+
+## Sample Usage (with SuffixDecoding via vLLM)
+
+To leverage SuffixDecoding with this model, use `arctic-inference` to patch `vLLM`.
+
+First, install the `arctic-inference` package:
+```bash
+pip install arctic-inference[vllm]
+```
+
+Once installed, Arctic Inference automatically patches vLLM. You can then use `vLLM`'s familiar APIs with SuffixDecoding and other optimizations enabled.
+
+#### Serving
+
+```console
+ARCTIC_INFERENCE_ENABLED=1 vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct \
+    --quantization "fp8" \
+    --tensor-parallel-size 1 \
+    --ulysses-sequence-parallel-size 2 \
+    --enable-shift-parallel \
+    --speculative-config '{
+        "method": "arctic",
+        "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
+        "num_speculative_tokens": 3,
+        "enable_suffix_decoding": true,
+        "disable_by_batch_size": 64
+    }'
+```
+
+#### Offline Inference
+
+Save the following script to `arctic_example.py`:
+
+```python
+import vllm
+from vllm import LLM, SamplingParams
+
+vllm.plugins.load_general_plugins()
+
+llm = LLM(
+    model="Snowflake/Llama-3.1-SwiftKV-8B-Instruct",
+    quantization="fp8",
+    tensor_parallel_size=1,
+    ulysses_sequence_parallel_size=2,
+    enable_shift_parallel=True,
+    speculative_config={
+        "method": "arctic",
+        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
+        "num_speculative_tokens": 3,
+        "enable_suffix_decoding": True,
+        "disable_by_batch_size": 64,
+    },
+)
+
+conversation = [
+    {
+        "role": "user",
+        "content": "Write an essay about the importance of higher education.",
+    },
+]
+
+sampling_params = SamplingParams(temperature=0.0, max_tokens=800)
+
+outputs = llm.chat(conversation, sampling_params=sampling_params)
+
+print(outputs[0].outputs[0].text)
+```
+
+Run the script with Arctic Inference enabled:
+
+```console
+ARCTIC_INFERENCE_ENABLED=1 python arctic_example.py
+```
+
+## Citation
+
+If you find this model or the related work useful, please consider citing the Arctic Inference and SuffixDecoding papers:
+
+```bibtex
+@misc{arcticinference2025,
+      title={Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI},
+      author={Samyam Rajbhandari and Mert Hidayetoglu and Aurick Qiao and Ye Wang and Juncheng Yang and Jeff Rasley and Michael Wyatt and Yuxiong He},
+      year={2025},
+      url={https://arxiv.org/abs/2507.11830},
+}
+
+@misc{suffixdecoding2024,
+      title={SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications},
+      author={Ye Wang and Samyam Rajbhandari and Juncheng Yang and Aurick Qiao and Mert Hidayetoglu and Michael Wyatt and Jeff Rasley and Yuxiong He},
+      year={2024},
+      eprint={2411.04975},
+      archivePrefix={arXiv},\
+      primaryClass={cs.CL},\
+      url={https://arxiv.org/abs/2411.04975},\
+}
+```
 <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=99c9bfbe-21ae-47b6-9cf3-5c881758bd27" />