README.md

metadata

library_name: transformers
pipeline_tag: text-generation
tags:
  - custom_code
  - causal-lm
  - rex

REX1 300M

REX is a recursive decoder-only Transformer language model. This repository uses custom Transformers code, so load it with trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(".")

.pt`.

Training Notes

• Tokenizer: gpt2
• Context length: 1024
• Training output dir: runs/rex-300m-mixed-2

This is a base language model checkpoint and is not instruction-aligned unless noted.