Qwen3.5-2B-SpeedX

Qwen3.5-2B-SpeedX is an experimental, text-only recurrent conversion of Qwen/Qwen3.5-2B, designed for fast autoregressive inference and constant-size recurrent memory.

The original Qwen3.5-2B text backbone mixes native Gated DeltaNet (GDN) layers with full-attention layers. SpeedX replaces the six full-attention layers with distilled native GDN layers, producing a 24/24 recurrent token-mixing stack.

This is an experimental research checkpoint, not an official Qwen release.

Highlights

• ~1.92B parameters
• 24 / 24 recurrent Gated DeltaNet layers
• Full-attention layers 3, 7, 11, 15, 19, 23 replaced with native GDN
• No token-wise full-attention KV cache in the converted text stack
• Recurrent decode state is constant-size with respect to context length
• Hugging Face Transformers compatible through custom remote code
• Designed for native Qwen GDN kernels
• Experimental Triton, CUDA Graph, and FP8 inference runtime
• Text-generation only; upstream multimodal/vision components are not included

Architecture

Original Qwen3.5-2B text pattern:

GDN → GDN → GDN → Full Attention

SpeedX:

GDN → GDN → GDN → GDN

repeated across all 24 text layers.

The recurrent state follows

[ S_t = F(S_{t-1}, x_t) ]

so decode-state memory is approximately

[ M_{\mathrm{state}} = O(1) ]

with respect to context length, instead of a token-wise KV cache whose memory grows approximately as

[ M_{\mathrm{KV}} = O(T). ]

NVIDIA L4 benchmark

Measured on one NVIDIA L4 22.03 GiB, batch size 1, greedy decoding.

Exact BF16 CUDA Graph

Context	Official Qwen3.5-2B	SpeedX exact	Speedup
256	25.90 tok/s	60.75 tok/s	2.35×
1,024	26.63 tok/s	60.58 tok/s	2.28×
4,096	26.06 tok/s	60.67 tok/s	2.33×
16,384	26.46 tok/s	60.68 tok/s	2.29×

Experimental FP8 runtime

Context	Official Qwen3.5-2B	SpeedX FP8	Speedup
256	25.90 tok/s	79.27 tok/s	3.06×
1,024	26.63 tok/s	78.85 tok/s	2.96×
4,096	26.06 tok/s	78.47 tok/s	3.01×
16,384	26.46 tok/s	78.38 tok/s	2.96×

These results use the project's optimized runtime. Standard Hugging Face model.generate() should not be assumed to reproduce the CUDA-Graph / FP8 numbers above.

At 16K context, an optimized exact recurrent path measured approximately:

• Official Qwen3.5-2B prefill: 10.85k tok/s
• SpeedX prefill: 13.79k tok/s
• Ratio: ~1.27×

Quick start

Custom model code is included, so trust_remote_code=True is required.

pip install -U \
  "transformers @ git+https://github.com/huggingface/transformers.git@e453228ef83ce0d756f7621ef8607220ebf5da6a" \
  "kernels>=0.16.0,<0.17" \
  "accelerate>=1.1.0" \
  "safetensors>=0.8.0"

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "summerMC/Qwen3.5-2B-SpeedX"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
).eval()

inputs = tokenizer(
    "Explain recurrent memory in neural networks.",
    return_tensors="pt",
).to("cuda")

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Chat example

messages = [
    {"role": "system", "content": "You are a precise technical assistant."},
    {"role": "user", "content": "Compare recurrent state memory with a Transformer KV cache."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(text, return_tensors="pt").to("cuda")

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Conversion

The original text stack contains 18 native GDN layers and six full-attention layers.

SpeedX preserves the native GDN layers and replaces full-attention layers:

3, 7, 11, 15, 19, 23

with native Qwen3_5GatedDeltaNet modules.

The replacement layers are initialized from nearby native GDN layers and distilled against the original full-attention outputs.

The experimental layer-wise objective combines normalized MSE and cosine distance:

[ \mathcal{L}

\frac{|y_S-y_T|_2^2} {\mathbb{E}[y_T^2]+\epsilon} + \lambda \left(1-\cos(y_S,y_T)\right). ]

Limitations

Experimental distillation

This checkpoint was created primarily to validate the recurrent conversion and high-speed runtime. The smoke-distillation schedule is short, so it should not be assumed to preserve all reasoning, factual, multilingual, or benchmark quality of the original Qwen3.5-2B.

Longer distillation or continued pretraining is recommended for serious deployment.

Text only

The upstream Qwen3.5-2B is multimodal. SpeedX contains the converted text-generation stack and does not provide the original vision stack.

Custom code

Review the repository code before enabling:

trust_remote_code=True

Optimized runtime vs checkpoint

The ~60 tok/s exact and ~79 tok/s FP8 figures rely on runtime-specific optimizations including:

• native GDN kernels
• Triton fusion
• persistent CUDA Graph greedy decoding
• selective precision / FP8 execution

The normal Hugging Face loading path is primarily a compatibility path.

Intended use

SpeedX is intended for research on:

• recurrent language models
• fixed-state decoding
• KV-cache-free decoding
• long-context recurrent memory
• Gated DeltaNet architectures
• CUDA Graph inference
• recurrent-model quantization

Demo

Hugging Face Space:

summerMC/Qwen-speed

Base model

Derived from:

Qwen/Qwen3.5-2B

Please review the upstream model card and license as well.

License

Apache-2.0, following the upstream Qwen3.5-2B model metadata.

Citation

If you use SpeedX in experiments, cite the upstream Qwen work and identify this checkpoint as an experimental recurrent conversion of Qwen3.5-2B.

A dedicated SpeedX technical report has not yet been published.