Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,132 @@
+---
+license: apache-2.0
+language: [en]
+tags:
+- tokenizer
+- sentencepiece
+- bpe
+- structured-action-model
+- agentic-ai
+- robotics
+- iot
+- workflow-automation
+inference: false
+library_name: sentencepiece
+---
+
+# SAM Tokenizer — `AMFORGE/sam_tokenizer`
+
+The official tokenizer for **SAM (Structured Action Model)** by **AMFORGE**.
+
+A SentencePiece BPE tokenizer purpose-built for structured action generation
+across **10 domains**: robotics, HTTP/REST, MQTT/IoT, databases, workflows,
+e-commerce, vehicles, smart home, calendar/email, and filesystem.
+
+## Why a custom tokenizer
+
+Generic LLM tokenizers shred robotics and JSON tokens into fragments
+(`0.5` → `0`, `.`, `5`). This kills numeric precision and bloats sequence
+length. SAM's tokenizer enforces **atomic numerics** via reserved symbols,
+keeping every coordinate, status code, port, and angle as a single token.
+
+## Design principles (the "SparsForos philosophy")
+
+| Setting | Value | Reason |
+|---|---|---|
+| `model_type` | BPE | Best balance for mixed natural-language + JSON |
+| `character_coverage` | 0.9999 | 1.0 causes infinite SPM loop with rare chars |
+| `byte_fallback` | False | Disables silent byte-level splits |
+| `split_digits` | False | Keeps `0.5` as ONE token, not `0 . 5` |
+| `normalization_rule_name` | identity | No unicode normalization side-effects |
+| `num_threads` | 2 | Higher counts freeze Kaggle/Jupyter kernels |
+| `hard_vocab_limit` | True | No silent vocab expansion |
+
+## Statistics
+
+- **Vocab size**: 12000
+- **Reserved domain symbols**: 6806
+- **Atomic numeric tokens**: 6320
+- **Structural tags + JSON Schema primitives**: 64
+- **Multi-domain operations + keys**: 294
+- **Domain values (units, currencies, modes, rooms…)**: 135
+- **Training corpus**: 200,000 synthetic multi-domain lines
+
+## Reserved structural tags
+
+```
+<SCHEMA> </SCHEMA>     # JSON Schema conditioning
+<TASK>   </TASK>       # Natural language instruction
+<DOMAIN> </DOMAIN>     # Generic domain wrapper
+<JSON>   </JSON>       # JSON output wrapper
+<ACTION> </ACTION>     # Action block
+<META>   </META>       # Metadata
+```
+
+## Reserved domain tags
+
+```
+<ROS>        <HTTP>       <MQTT>       <DB>         <WORKFLOW>
+<ECOMMERCE>  <VEHICLE>    <HOME>       <CAL>        <FILE>
+```
+
+## Reserved numeric ranges (all atomic)
+
+| Range | Step | Use |
+|---|---|---|
+| `-10.00` … `10.00` | 0.01 | Spatial coords (robotics, vehicle, IoT) |
+| `0.00` … `10.00` | 0.05 | Velocities (m/s) |
+| `0.00` … `1.00` | 0.01 | Force, ratios, probabilities |
+| `0.0` … `60.0` | 0.5 | Wait/timeout durations (s) |
+| `0` … `999` | 1 | Small integers (counts, retries, indices) |
+| HTTP status | discrete | 100,200,201,…,500,501,…,511 |
+| Network ports | discrete | 22,80,443,1883,3306,5432,6379,8080,… |
+| Frequencies (Hz) | discrete | 1,5,10,20,…,5000 |
+| Common angles | discrete | radians + degrees (0, π/2, π, 45°, 90°, 180°, …) |
+
+## Quick start
+
+```python
+from huggingface_hub import hf_hub_download
+import sentencepiece as spm
+
+# Public — no token required
+path = hf_hub_download(repo_id="AMFORGE/sam_tokenizer", filename="sam_tokenizer.model")
+sp = spm.SentencePieceProcessor()
+sp.Load(path)
+
+# Atomic numeric encoding — try this with any generic tokenizer for comparison
+text = '<ROS><TASK>move to x=0.5 y=-1.2 z=0.8</TASK> => [{"op":"move","x":0.5,"y":-1.2,"z":0.8}]'
+ids = sp.EncodeAsIds(text)
+print(f"Tokens: {len(ids)}")
+print(f"Decoded: {sp.DecodeIds(ids)}")
+```
+
+## Atomicity verification
+
+Verify that floats stay as single tokens:
+
+```python
+for v in ["0.5", "-1.2", "1.57", "0.01", "3.14"]:
+    pieces = sp.EncodeAsPieces(v)
+    real = [p for p in pieces if p not in ("\u2581", " ")]
+    assert len(real) == 1, f"{v} got split: {real}"
+print("[OK] All floats atomic.")
+```
+
+## Used by
+
+- [`AMFORGE/sam-v1`](https://huggingface.co/AMFORGE/sam-v1) — the SAM model
+
+## Citation
+
+```bibtex
+@misc{sam_tokenizer_2026,
+  title  = {SAM Tokenizer: Atomic Multi-Domain BPE for Structured Action Generation},
+  author = {AMFORGE},
+  year   = {2026},
+  url    = {https://huggingface.co/AMFORGE/sam_tokenizer}
+}
+```
+
+Built with the SparsForos philosophy by **AMFORGE** —
+https://huggingface.co/AMFORGE