Repeat After Me: Transformers are Better than State Space Models at Copying
About
This part of the repository was built from Repeat After Me: Transformers are Better than State Space Models at Copying, specifically the synthetic experiment part.
This repository gathers the experiments for the paper . The experiments divide in two parts:
Installation
pip install causal-conv1d>=1.1.0 : an efficient implementation of a simple causal Conv1d layer used inside the Mamba block. pip install mamba-ssm : the core Mamba package. pip install names : names package to randomly sample names in the phone-book experiment.
Other requirements:
- Linux
- NVIDIA GPU
- PyTorch 1.12+
- CUDA 11.6+
- transformers 4.35+
- datasets 2.14+
We need to cite
@article{jelassi2024repeat,
title={Repeat after me: Transformers are better than state space models at copying},
author={Jelassi, Samy and Brandfonbrener, David and Kakade, Sham M and Malach, Eran},
journal={arXiv preprint arXiv:2402.01032},
year={2024}
}