File size: 4,350 Bytes
1403b2b 5f73b41 1403b2b 5f73b41 1403b2b 302b676 5f73b41 302b676 5f73b41 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
---
language:
- en
- zh
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
---
# BlockFFN-XLarge
This is the original 1.2B BlockFFN checkpoint used in the paper *BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity* for acceleration tests.
Links: [[Paper](https://arxiv.org/pdf/2507.08771)] [[Codes](https://github.com/thunlp/BlockFFN)]
### Introduction
**BlockFFN** presents a novel Mixture-of-Experts (MoE) architecture designed to enhance activation sparsity at both token and chunk levels, making LLMs more acceleration-friendly, especially for end-side devices. This approach integrates a new router for differentiable and flexible routing and is optimized with CLS-aware training objectives. The model achieves superior performance and significant speedup on end-side devices.
### How to Use
You can explore the core implementation of **BlockFFN** in the [GitHub repository](https://github.com/thunlp/BlockFFN). You can load and use this model simply by using `AutoTokenizer` and `AutoModelForCausalLM`.
#### Text Generation
```python
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "SparseLLM/BlockFFN-XLarge" # Or other BlockFFN models like SparseLLM/BlockFFN-XLarge-sft
pipe = pipeline(
"text-generation",
model_name,
tokenizer=AutoTokenizer.from_pretrained(model_name),
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
print(pipe("The key to life is", max_new_tokens=20, do_sample=True)[0]["generated_text"])
```
#### Get Expert Routing Probabilities
Based on expert routing probabilities, **BlockFFN** enables mechanistic interpretability by understanding which sparse features are activated to which token. Following the standard MoE approach, you can obtain expert routing probabilities for all layers by setting `output_router_probs=True`. The example below demonstrates how to compute and analyze the expert activation patterns:
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"SparseLLM/BlockFFN-XLarge",
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("SparseLLM/BlockFFN-XLarge")
inputs = tokenizer("City and County of San Francisco", return_tensors="pt")
outputs = model(**inputs.to(model.device), output_router_probs=True)
# Get full expert routing probabilities: [batch_size, seq_len, moe_heads, moe_experts**2]
# Note: The output format for router_probs might vary based on the specific BlockFFN implementation details.
# This example assumes a common structure for illustration.
if hasattr(outputs, 'router_probs') and outputs.router_probs is not None:
for layer_idx, layer_router_probs in enumerate(outputs.router_probs):
print(f"Layer {layer_idx} Router Probs Shape: {layer_router_probs.shape}")
# Example: Analyze first token's expert activation in the first layer
if layer_router_probs.shape[1] > 0: # Check if there are tokens
first_token_probs = layer_router_probs[0, 0] # batch_idx, token_idx
# Assuming first_token_probs is [num_heads, num_experts]
# Sum across heads to get overall expert importance
expert_activations = first_token_probs.sum(dim=0)
activated_experts = (expert_activations > 1e-2).nonzero(as_tuple=True)[0]
decoded_token = tokenizer.decode(inputs.input_ids[0, 0])
print(f"Token: '{decoded_token}' (Layer {layer_idx}) Activated Experts Count: {len(activated_experts)}")
# print(f"Activated Expert Indices: {activated_experts.tolist()}")
else:
print("Model output does not contain 'router_probs'.")
```
### Citation
If you find our work useful for your research, please kindly cite our paper as follows:
```
@article{song2025blockffn,
title={{BlockFFN}: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity},
author={Chenyang Song and Weilin Zhao and Xu Han and Chaojun Xiao and Yingfa Chen and Yuxuan Li and Zhiyuan Liu and Maosong Sun},
journal={arXiv preprint arXiv:2507.08771},
year={2025},
url={https://arxiv.org/pdf/2507.08771},
}
``` |