README.md

metadata

license: apache-2.0
tags:
  - tokenizers
  - BPE
  - sentencepiece
  - code-generation

samg_mm_tokenizer (NexusBPE)

Multilingual + action SentencePiece BPE tokenizer for SAM-G, the AMEFORGE ~30M generalist instruction-action model. Includes atomic [CHAT]/[ACTION] mode tokens, the 10 SAM action domains, and JSON scaffolding atoms.

Configuration

• Vocabulary size: 32000
• Model type: BPE
• Byte fallback: enabled
• Digit splitting: enabled (digits 0-9 are guaranteed atomic)
• Whitespace normalization: disabled (identity rule) — indentation preserved

Special atomic tokens

Mode tokens ([CHAT], [ACTION]), domains (<ros>, <mqtt>, …), JSON keys ("op":, "params":, …), code keywords (def, fn, struct, async, …), operators and structural tags are all guaranteed single tokens.

Usage

import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("samg_mm_tokenizer.model")
print(sp.EncodeAsPieces('turn on the lamp [ACTION] {"domain":"home","op":"set_state"}'))