Aquiles-ai/Chargaff-Tokenizer

Chargaff UTF-8 byte-level tokenizer (512)

Byte-level BPE tokenizer for Chargaff, our DNA prediction model. No training, no merges: 1 token per UTF-8 byte.

Functional equivalent of the Evo2 CharLevelTokenizer (raw UTF-8 bytes) with its own id layout - ids do not match the original Evo2 (A=32 here vs A=65 there). Do not load Evo2 checkpoints directly; remap the embedding or train from scratch.

Vocab

• 0..255: sorted(ByteLevel.alphabet()) - all 256 UTF-8 bytes, deterministic order

• 256: <eos> (also used as bos_token, like Evo2)

• 257: <pad>

• 258..511: <unused_258> … <unused_511> filler to reach vocab_size=512

• Model: BPE(vocab, merges=[], unk_token=None)

• Pre-tokenizer: ByteLevel(add_prefix_space=False, use_regex=False) - raw bytes, no GPT-2 regex splits

• Decoder: ByteLevel()

• model_max_length=1048576, padding_side=right

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Aquiles-ai/Chargaff-Tokenizer")
tok.encode("ACGT")          # [32, 34, 38, 51]
tok.decode([32, 34, 38, 51])  # "ACGT"
tok(["ACGT", "ACGTN"], padding=True)
# {'input_ids': [[32, 34, 38, 51, 257], [32, 34, 38, 51, 45]]}

Limitations

• Fertility ~1 token/byte: CJK/emoji expand 3-4x. Fine for ACGTN, wasteful for multilingual text.
• No subword semantics (no merges).
• Id layout differs from Evo2 original, so embeddings are not interchangeable.