Ezaris Tokenizer Collection Multilingual BPE tokenizer family (128K/64K/8K) for LUNA-1B, with benchmark vs the original GPT-NeoX tokenizer. • 1 item • Updated 1 day ago
Ezaris Tokenizer Collection Multilingual BPE tokenizer family (128K/64K/8K) for LUNA-1B, with benchmark vs the original GPT-NeoX tokenizer. • 1 item • Updated 1 day ago
Ezaris Program Collection LUNA-1B by ASTERIZER: 1B-parameter multilingual model, 2.4TB training corpus, and 128K/64K tokenizer. • 4 items • Updated 1 day ago
Ezaris Tokenizer Collection Multilingual BPE tokenizer family (128K/64K/8K) for LUNA-1B, with benchmark vs the original GPT-NeoX tokenizer. • 1 item • Updated 1 day ago
LUNA Tokenizer Collection Shared GPT-NeoX tokenizer (vocab 50304) for LUNA-100M and LUNA-300M — the single tokenizer behind both models. • 1 item • Updated 2 days ago