arthu1/north-tokenizer

North Tokenizer

Shared SentencePiece tokenizer for the North Star model family:

• North Air 1 — 124M, fast
• North Star 1 — 198M, balanced
• Wind Arc 1.5 — 198M, deep reasoning

Vocabulary size: 32,000 Format: SentencePiece (.model file)

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
ids = sp.encode("Hello world", out_type=int)