---
license: apache-2.0
tags:
- tokenizers
- BPE
- sentencepiece
- code-generation
---

# samg_mm_tokenizer (NexusBPE)

Multilingual + action SentencePiece BPE tokenizer for **SAM-G**, the AMEFORGE
~30M generalist instruction-action model. Includes atomic [CHAT]/[ACTION]
mode tokens, the 10 SAM action domains, and JSON scaffolding atoms.

## Configuration
- Vocabulary size: **32000**
- Model type: BPE
- Byte fallback: enabled
- Digit splitting: enabled (digits 0-9 are guaranteed atomic)
- Whitespace normalization: disabled (`identity` rule) — indentation preserved

## Special atomic tokens
Mode tokens (`[CHAT]`, `[ACTION]`), domains (`<ros>`, `<mqtt>`, …), JSON keys
(`"op":`, `"params":`, …), code keywords (`def`, `fn`, `struct`, `async`, …),
operators and structural tags are all guaranteed single tokens.

## Usage
```python
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("samg_mm_tokenizer.model")
print(sp.EncodeAsPieces('turn on the lamp [ACTION] {"domain":"home","op":"set_state"}'))
```