File size: 1,093 Bytes
4ca4e4c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | # Repeat After Me: Transformers are Better than State Space Models at Copying
## About
This part of the repository was built from [Repeat After Me: Transformers are Better than State Space Models at Copying](https://arxiv.org/abs/2402.01032), specifically the synthetic experiment part.
This repository gathers the experiments for the paper . The experiments divide in two parts:
## Installation
<tt>pip install causal-conv1d>=1.1.0</tt> : an efficient implementation of a simple causal Conv1d layer used inside the Mamba block.
<tt>pip install mamba-ssm</tt> : the core Mamba package.
<tt>pip install names</tt> : names package to randomly sample names in the phone-book experiment.
Other requirements:
- Linux
- NVIDIA GPU
- PyTorch 1.12+
- CUDA 11.6+
- transformers 4.35+
- datasets 2.14+
## We need to cite
```
@article{jelassi2024repeat,
title={Repeat after me: Transformers are better than state space models at copying},
author={Jelassi, Samy and Brandfonbrener, David and Kakade, Sham M and Malach, Eran},
journal={arXiv preprint arXiv:2402.01032},
year={2024}
}
```
|