| --- |
| datasets: |
| - EleutherAI/pile |
| language: |
| - en |
| --- |
| # Model Card |
|
|
| This model is pretrained Based model. |
|
|
| As a quality reference, we include a pretrained Mamba model provided here: https://huggingface.co/hazyresearch/mamba-1b, and a pretrained Attention (Llama architecture) model provided here: https://huggingface.co/hazyresearch/attn-1b |
|
|
| All three checkpoints are pretrained on 10Bn tokens of the Pile in the exact same data order using next token prediction. |
|
|
|
|
| ### Model Sources |
|
|
| The model implementation and training code that produced the model are provided here: https://github.com/HazyResearch/based |
|
|
| ### Uses |
|
|
| The purpose of this work is to evaluate the language modeling quality of a new efficient architecture, Based. |
|
|
| We include a series of benchmarks that you can use to evaluate quality: |
| - FDA: https://huggingface.co/datasets/hazyresearch/based-fda |
| - SWDE: https://huggingface.co/datasets/hazyresearch/based-swde |
| - SQUAD: https://huggingface.co/datasets/hazyresearch/based-squad |
|
|
|
|
| ## Citation |
|
|
| Please consider citing this paper if you use our work: |
|
|
| ``` |
| @article{arora2024simple, |
| title={Simple linear attention language models balance the recall-throughput tradeoff}, |
| author={Arora, Simran and Eyuboglu, Sabri and Zhang, Michael and Timalsina, Aman and Alberti, Silas and Zinsley, Dylan and Zou, James and Rudra, Atri and Ré, Christopher}, |
| journal={arXiv:2402.18668}, |
| year={2024} |
| } |
| ``` |
|
|
| Please reach out to simarora@stanford.edu, eyuboglu@stanford.edu, and mzhang20@stanford.edu with questions. |
|
|
