---
library_name: transformers
pipeline_tag: text-generation
tags:
- custom_code
- causal-lm
- rex
---

# REX1 300M

REX is a recursive decoder-only Transformer language model. This repository uses custom
Transformers code, so load it with `trust_remote_code=True`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(".")
```
.pt`.

## Training Notes

- Tokenizer: `gpt2`
- Context length: `1024`
- Training output dir: `runs/rex-300m-mixed-2`

This is a base language model checkpoint and is not instruction-aligned unless noted.