|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- Qwen/Qwen3-Embedding-4B |
|
|
library_name: sentence-transformers |
|
|
--- |
|
|
## Description |
|
|
This is one [CSRv2](https://arxiv.org/abs/2602.05735) model finetuned on [MTEB](https://huggingface.co/mteb) |
|
|
sts datasets with [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) as backbone. |
|
|
|
|
|
For more details, including benchmark evaluation, hardware requirements, and inference performance, please |
|
|
refer to our [Github](https://github.com/Y-Research-SBU/CSRv2). |
|
|
|
|
|
## Sentence Transformer Usage |
|
|
You can evaluate this model loaded by Sentence Transformers with the following code snippet (take STS12 as one example): |
|
|
```python |
|
|
import mteb |
|
|
from sentence_transformers import SparseEncoder |
|
|
model = SparseEncoder( |
|
|
"Y-Research-Group/CSRv2-sts", |
|
|
trust_remote_code=True |
|
|
) |
|
|
model.prompts = { |
|
|
"STS12": "Instruct: Retrieve semantically similar text\n Query:" |
|
|
} |
|
|
task = mteb.get_tasks(tasks=["STS12"]) |
|
|
evaluation = mteb.MTEB(tasks=task) |
|
|
evaluation.run( |
|
|
model, |
|
|
eval_splits=["test"], |
|
|
output_folder="./results/STS12", |
|
|
show_progress_bar=True |
|
|
encode_kwargs={"convert_to_sparse_tensor": False, "batch_size": 8} |
|
|
) # MTEB don't support sparse tensors yet, so we need to convert to dense tensors |
|
|
``` |
|
|
|
|
|
It is suggested that you use our [default prompts](https://github.com/Y-Research-SBU/CSRv2/blob/main/text/dataset_to_prompt.json) |
|
|
in evaluation. |
|
|
|
|
|
## Multi-TopK Support |
|
|
|
|
|
Our model supports different sparsity levels due to the utilization of **Multi-TopK** loss in training. |
|
|
You can change sparsity model by adjusting the `k` parameter` in the file `3_SparseAutoEncoder/config.json`. |
|
|
We set sparsity level to 2 by default. |
|
|
|
|
|
For instance, if you want to evaluate with sparsity level $K=8$ (which means there are 8 activated neurons in |
|
|
each embedding vector), the `3_SparseAutoEncoder/config.json` should look like this: |
|
|
|
|
|
```json |
|
|
{ |
|
|
"input_dim": 2560, |
|
|
"hidden_dim": 10240, |
|
|
"k": 8, |
|
|
"k_aux": 1024, |
|
|
"normalize": false, |
|
|
"dead_threshold": 30 |
|
|
} |
|
|
``` |
|
|
|
|
|
## CSRv2 Qwen Series |
|
|
We will release a series of [CSRv2](https://arxiv.org/abs/2602.05735) models finetuned on common tasks in |
|
|
[MTEB](https://huggingface.co/mteb) with [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) |
|
|
as backbone. These tasks are: |
|
|
|
|
|
- **[Classification](https://huggingface.co/Y-Research-Group/CSRv2-classification)** |
|
|
- **[Clustering](https://huggingface.co/Y-Research-Group/CSRv2-clustering)** |
|
|
- **[Retrieval](https://huggingface.co/Y-Research-Group/CSRv2-retrieval)** |
|
|
- **[STS](https://huggingface.co/Y-Research-Group/CSRv2-sts)** |
|
|
- **[Pair_classification](https://huggingface.co/Y-Research-Group/CSRv2-pair_classification)** |
|
|
- **[Reranking](https://huggingface.co/Y-Research-Group/CSRv2-reranking)** |
|
|
|
|
|
## Citation |
|
|
```bibtex |
|
|
@inproceedings{guo2026csrv2, |
|
|
title={{CSR}v2: Unlocking Ultra-sparse Embeddings}, |
|
|
author={Guo, Lixuan and Wang, Yifei and Wen, Tiansheng and Wang, Yifan and Feng, Aosong and Chen, Bo and Jegelka, Stefanie and You, Chenyu}, |
|
|
booktitle={International Conference on Learning Representations (ICLR)}, |
|
|
year={2026} |
|
|
} |
|
|
``` |
