ASTERIZER/LUNA-Tokenizer
Token Classification • Updated • 28
LUNA Tokenizer
Shared GPT-NeoX tokenizer (vocab 50304) for LUNA-100M and LUNA-300M — the single tokenizer behind both models.
Shared GPT-NeoX tokenizer (vocab 50304) for LUNA-100M and LUNA-300M — the single tokenizer behind both models.