---
license: apache-2.0
library_name: transformers
tags:
  - grt
  - geometric-rotation-transformer
  - tokenizer
  - polar-embedding
  - phase-angle-embedding
  - rotational-transformer
  - qwen3
  - matterwave
base_model: MatterWaveAI/grt-30m
---

# GRT-30M-Tokenizer (`MatterWaveAI/grt-30m-tokenizer`)

Canonical **BPE Tokenizer (`151,669` tokens)** and shared **Polar Phase-Angle Token Embedding Bank (`19,565,302` parameters, `0 nn.Linear` layers)** for the **GRT-30M Research Suite**:
- [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (Base Causal LM)
- [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat) (ChatML Conversational SFT)
- [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor) (4-Expert Mixture-of-Rotations Causal LM)
- [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding) (Circular-Pooled Bi-Encoder)
- [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker) (Rotational Cross-Encoder Reranker)
- [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl) (100% Rotational Vision-Language Model)

---

## What Is Included

| File | Size | Contents |
| :--- | :---: | :--- |
| `tokenizer.json` | `11.4 MB` | Rust `PreTrainedTokenizerFast` BPE vocabulary (`151,669` tokens, Qwen 3 compatible + ChatML special tokens) |
| `tokenizer_config.json` | `1.2 KB` | Tokenizer configuration including the ChatML `chat_template` (`<|im_start|>` / `<|im_end|>`) |
| `polar_embedding.safetensors` | `78.3 MB` | Shared polar phase-angle embedding table & tied cosine head scale (`19,565,302` FP32 parameters) |
| `config.json` | `310 B` | Polar embedding geometry metadata (`vocab_size = 151669`, `hidden_size = 256`, `n_pairs = 128`) |

### Polar Embedding Tensors (`polar_embedding.safetensors`)

In the Geometric Rotation Transformer (`GRT`), token embeddings live in polar coordinates on `128` orthogonal `2D` planes (`hidden_size = 256`) rather than an unstructured Euclidean matrix:
```text
e(token_id)[2p]   = exp(log_magnitude[token_id]) · cos(phase_angles[token_id, p])
e(token_id)[2p+1] = exp(log_magnitude[token_id]) · sin(phase_angles[token_id, p])
```

| Tensor Name | Shape | Parameters | Role |
| :--- | :---: | :---: | :--- |
| `embedding.phase_angles` | `[151669, 128]` | `19,413,632` | Per-token phase angles `θ` on the 128-torus `(S¹)^128` (tied between input embedding and output `GRTCosineHead`) |
| `embedding.log_magnitude` | `[151669]` | `151,669` | Per-token radial log-magnitude offset |
| `lm_head.output_scale` | `[]` (scalar) | `1` | Raw softplus temperature scale for the tied polar cosine language modeling head |
| **Total** | — | **`19,565,302`** | **`66.8%` of `grt-30m` total parameters (shared identically across all 6 `grt-30m*` models)** |

---

## Quickstart

### 1. Load the Tokenizer (`transformers` / `vLLM`)

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-30m-tokenizer")
tokens = tokenizer("The Givens Rotational Transformer replaces dense matrices with")
print(tokens["input_ids"])
```

### 2. Load the Shared Polar Embedding Table (`safetensors`)

```python
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("MatterWaveAI/grt-30m-tokenizer", "polar_embedding.safetensors")
polar_bank = load_file(path)

phase_angles = polar_bank["embedding.phase_angles"]   # [151669, 128]
log_magnitude = polar_bank["embedding.log_magnitude"] # [151669]
output_scale = polar_bank["lm_head.output_scale"]     # []
print(phase_angles.shape, log_magnitude.shape, output_scale.shape)
```

### 3. Harmonic Upcycling to Larger `d_model` (`256 → 512` or `1024`)

When scaling a new GRT backbone to a wider `d_model = 256 · k` (`n_pairs = 128 · k`), tile `embedding.phase_angles` by `k` along dim `-1` (`phase_angles.repeat(1, k)`) to preserve all pairwise token cosine similarities at initialization.

---

## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)

- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

---

## License

Apache 2.0

---

## Citation

```bibtex
@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
```