Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
Paper • 2401.16421 • Published
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("hzy00/BiPE_ALiBi-151M")
model = AutoModelForCausalLM.from_pretrained("hzy00/BiPE_ALiBi-151M")Quick Links
References
For more information, please refer to our paper and GitHub repository.
Paper: Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
GitHub: BiPE
Authors: Zhenyu He*, Guhao Feng*, Shengjie Luo*, Kai Yang, Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, Liwei Wang
- Downloads last month
- 3
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="hzy00/BiPE_ALiBi-151M")