| # Repeat After Me: Transformers are Better than State Space Models at Copying | |
| ## About | |
| This part of the repository was built from [Repeat After Me: Transformers are Better than State Space Models at Copying](https://arxiv.org/abs/2402.01032), specifically the synthetic experiment part. | |
| This repository gathers the experiments for the paper . The experiments divide in two parts: | |
| ## Installation | |
| <tt>pip install causal-conv1d>=1.1.0</tt> : an efficient implementation of a simple causal Conv1d layer used inside the Mamba block. | |
| <tt>pip install mamba-ssm</tt> : the core Mamba package. | |
| <tt>pip install names</tt> : names package to randomly sample names in the phone-book experiment. | |
| Other requirements: | |
| - Linux | |
| - NVIDIA GPU | |
| - PyTorch 1.12+ | |
| - CUDA 11.6+ | |
| - transformers 4.35+ | |
| - datasets 2.14+ | |
| ## We need to cite | |
| ``` | |
| @article{jelassi2024repeat, | |
| title={Repeat after me: Transformers are better than state space models at copying}, | |
| author={Jelassi, Samy and Brandfonbrener, David and Kakade, Sham M and Malach, Eran}, | |
| journal={arXiv preprint arXiv:2402.01032}, | |
| year={2024} | |
| } | |
| ``` | |