## Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models

This repository includes all of the experiments relevant to the associated paper. 

- `micro`: A collection of old experiments with pytorch transformers and mamba. Not currently used, but might be useful as a reference.
- `micro_hf`: A collection of experiments for micro-scale experiments, using huggingface models. `run.ipynb` will execute the experiments, and `process.ipynb` can be used to analyze the results. Each individual run does not take long to run on an RTX 4090. 
- `mini`: A collection of larger experiments, mostly ported from the ones in the Repeat After Me paper. Added to this are hybrid models implemented in huggingface.