Gausson/pythia-160m-deduped-SepLLM

Please refer to the SepLLM paper - ICML 2025 and our GitHub repository for using this model.

To use the checkpoint of this model, you must install the transformers-4.38.0.post1+sepllm-py3-none-any.whl released from our GitHub repository. Below are the reference script for testing and a sample of test results. We conducted testing using lm_eval==0.4.0.

This model has the same config as Gausson/pythia-160m-deduped-n128-SepLLM.

CUDA_LAUNCH_BLOCKING=1
lm_eval --model hf \
    --model_args pretrained=Gausson/pythia-160m-deduped-SepLLM \
    --tasks  arc_challenge,arc_easy,lambada_openai,logiqa,piqa,sciq,winogrande,wsc,wikitext  \
    --num_fewshot 5 \
    --device cuda:0\
    --batch_size 32

hf (pretrained=Gausson/pythia-160m-deduped-SepLLM), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 32
|    Tasks     |Version|Filter|n-shot|    Metric     |   | Value |   |Stderr|
|--------------|------:|------|-----:|---------------|---|------:|---|------|
|arc_challenge |      1|none  |     5|acc            |↑  | 0.2014|±  |0.0117|
|              |       |none  |     5|acc_norm       |↑  | 0.2346|±  |0.0124|
|arc_easy      |      1|none  |     5|acc            |↑  | 0.4731|±  |0.0102|
|              |       |none  |     5|acc_norm       |↑  | 0.4520|±  |0.0102|
|lambada_openai|      1|none  |     5|acc            |↑  | 0.3315|±  |0.0066|
|              |       |none  |     5|perplexity     |↓  |30.1605|±  |1.0128|
|logiqa        |      1|none  |     5|acc            |↑  | 0.2273|±  |0.0164|
|              |       |none  |     5|acc_norm       |↑  | 0.2857|±  |0.0177|
|piqa          |      1|none  |     5|acc            |↑  | 0.6464|±  |0.0112|
|              |       |none  |     5|acc_norm       |↑  | 0.6447|±  |0.0112|
|sciq          |      1|none  |     5|acc            |↑  | 0.8260|±  |0.0120|
|              |       |none  |     5|acc_norm       |↑  | 0.8150|±  |0.0123|
|wikitext      |      2|none  |     5|bits_per_byte  |↓  | 0.9207|±  |   N/A|
|              |       |none  |     5|byte_perplexity|↓  | 1.8931|±  |   N/A|
|              |       |none  |     5|word_perplexity|↓  |30.3488|±  |   N/A|
|winogrande    |      1|none  |     5|acc            |↑  | 0.5304|±  |0.0140|
|wsc           |      1|none  |     5|acc            |↑  | 0.3750|±  |0.0477|

If you find our work helpful, please consider giving us a star ⭐ @ our GitHub repository and citing our paper. We greatly appreciate your support 😄

@inproceedings{chen2025sepllm,
  title={{SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator}},
  author={Chen, Guoxuan and Shi, Han and Li, Jiawei and Gao, Yihang and Ren, Xiaozhe and Chen, Yimeng and Jiang, Xin and Li, Zhenguo and Liu, Weiyang and Huang, Chao},
  booktitle={International Conference on Machine Learning},
  year={2025},
  note={Also available at arXiv:2412.12094}
}