| --- |
| license: apache-2.0 |
| tags: |
| - tokenizers |
| - BPE |
| - sentencepiece |
| - code-generation |
| --- |
| |
| # samg_mm_tokenizer (NexusBPE) |
|
|
| Multilingual + action SentencePiece BPE tokenizer for **SAM-G**, the AMEFORGE |
| ~30M generalist instruction-action model. Includes atomic [CHAT]/[ACTION] |
| mode tokens, the 10 SAM action domains, and JSON scaffolding atoms. |
|
|
| ## Configuration |
| - Vocabulary size: **32000** |
| - Model type: BPE |
| - Byte fallback: enabled |
| - Digit splitting: enabled (digits 0-9 are guaranteed atomic) |
| - Whitespace normalization: disabled (`identity` rule) — indentation preserved |
|
|
| ## Special atomic tokens |
| Mode tokens (`[CHAT]`, `[ACTION]`), domains (`<ros>`, `<mqtt>`, …), JSON keys |
| (`"op":`, `"params":`, …), code keywords (`def`, `fn`, `struct`, `async`, …), |
| operators and structural tags are all guaranteed single tokens. |
|
|
| ## Usage |
| ```python |
| import sentencepiece as spm |
| sp = spm.SentencePieceProcessor() |
| sp.Load("samg_mm_tokenizer.model") |
| print(sp.EncodeAsPieces('turn on the lamp [ACTION] {"domain":"home","op":"set_state"}')) |
| ``` |
|
|
