jescy525/archon-tokenizer-v2

ARCHON Tokenizer v2

BPE ByteLevel (GPT-2 style) tokenizer used by ARCHON ASI. Trained on jescy525/archon-corpus (regen-variant text augmentation, ~101 GB).

Vocab info

Aspect	Value
Type	BPE ByteLevel (GPT-2 regex pre-tokenizer)
Base vocab	32,000 BPE tokens
Reserved special slots (IDs 0-15)	16
Added ChatML/tool tokens (IDs 32000-32005)	6
Total vocab size	32,006
Byte fallback	Native (ByteLevel covers all 256 bytes)
Normalizer	None
model_max_length	4096

Special tokens — 22 total

Active in chat_template

ID	Token	Role
1	<bos>	Auto-added by post_processor
2	<eos>	Auto-added by post_processor
0	<pad>	Padding
32000	<|im_start|>	ChatML turn start
32001	<|im_end|>	ChatML turn end

Reserved but inactive in default chat_template

ID	Token	Status
3	<unk>	Declared but never emitted (ByteLevel = byte fallback)
4	<sep>	Reserved
5-11	<|teach|>, <|evolve|>, <|query|>, <|code|>, <|/code|>, <|rust|>, <|python|>	Legacy AETHER/Cipher tokens — embeddings trained but unused in current chat_template
12-13	<|system|>, <|user|>	Available but chat_template uses raw role string instead
14	<|archon|>	Reserved system tag
15	<|think|>	Reasoning marker (model trained on but not in chat_template)
32002	<|assistant|>	Reserved (chat_template uses raw string)
32003	<|tool_call|>	Function-call start (trained, not in current chat_template)
32004	<|tool_result|>	Function-result start (trained, not in current chat_template)
32005	<|task_type|>	Optional task tag

Note (2026-05-28) : 14/22 special tokens have trained embeddings but are not generated by the default chat_template. Consider using tokenizer.apply_chat_template with a custom template to leverage them (see v2.1 enhanced template in tokenizer_config.json).

Chat template

Default = simple ChatML:

<|im_start|>role
content<|im_end|>

v2.1 enhanced template adds support for tool_calls, tool_results, and <|think|> reasoning.

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("jescy525/archon-tokenizer-v2")
text = tok.apply_chat_template(
    [{"role": "user", "content": "Hello ARCHON"}],
    tokenize=False, add_generation_prompt=True,
)
ids = tok.encode(text)

Known issues & status (2026-05-28 audit)

Issue	Severity	Note
Avg fertility ~39 tokens/100 chars	⚠️ HIGH	~56% above Llama 2 baseline (25 t/100c). Reduces effective context @ seq=4096 to ~10,500 chars vs 16,400 for Llama 2.
~50 polluted long tokens in vocab (e.g. ManufacturingAndIndustrialEngineeringHandler)	⚠️ MEDIUM	From regen-variant/ LLM-augmented corpus containing legacy class names. Embeddings trained but unused → dead weight ~0.15-0.3% vocab.
~100+ raw number tokens in vocab ('5848', '9937', etc.)	⚠️ LOW	Probable artifact of timestamps/IDs in training data.
add_prefix_space inconsistency: pre_tokenizer=false, decoder=true	⚠️ LOW	Possible phantom-space roundtrip edge case. Fixed in v2.1.
No NFC normalizer	⚠️ LOW	Risk of unicode-composed vs decomposed duplicates (mainly French accents). Not fixed in v2.1 to preserve trained embeddings. Target for v3.
14/22 special tokens orphan (trained but not in chat_template)	⚠️ MEDIUM	Capacity dead weight. Activatable via custom chat_template (v2.1).

Training corpus (BPE)

Trained on text from jescy525/archon-corpus:

• regen-variant/ (25 JSONL files, ~101 GB) — LLM-augmented text variations
• memories-archon/ (narratives, mostly EN with some FR observations_sur_soi)
• meta/ and meta-orchestration/ (small metadata)

Network flow datasets (archon-comm-v1-sources-light UNSW-NB15, archon-comm-v1-sources-packets CIC-IDS2017) were NOT used for BPE training (they are tabular Parquet, not text).

ARCHON SFT v1 lang reality

Despite the language: [en, fr] tag, sampling 700 rows × 5 SFT groups shows 100% English content. The bilingual capacity in the vocab is currently unused by ARCHON SFT v1.

Roadmap

• v2.1 (2026-05-28): Enhanced chat_template with tool_call / tool_result / think support. decoder.add_prefix_space=false for consistency. Same vocab.json → fully backward-compatible with all 51 trained ckpts (jescy525/archon-sft-v1-ckpts).
• v3 (planned): Re-train on filtered ARCHON-pure corpus, vocab 65K, fertility target <28 t/100c, remove polluted Handler tokens, decide EN-only vs bilingual based on data reality.