File size: 5,590 Bytes
78b5a99 9578c4b dda2c42 9578c4b 1d0101c 9578c4b 1d0101c 9578c4b 5f17cb7 9578c4b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | ---
license: apache-2.0
library_name: transformers
tags:
- grt
- geometric-rotation-transformer
- tokenizer
- polar-embedding
- phase-angle-embedding
- rotational-transformer
- qwen3
- matterwave
base_model: MatterWaveAI/grt-30m
---
# GRT-30M-Tokenizer (`MatterWaveAI/grt-30m-tokenizer`)
Canonical **BPE Tokenizer (`151,669` tokens)** and shared **Polar Phase-Angle Token Embedding Bank (`19,565,302` parameters, `0 nn.Linear` layers)** for the **GRT-30M Research Suite**:
- [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (Base Causal LM)
- [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat) (ChatML Conversational SFT)
- [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor) (4-Expert Mixture-of-Rotations Causal LM)
- [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding) (Circular-Pooled Bi-Encoder)
- [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker) (Rotational Cross-Encoder Reranker)
- [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl) (100% Rotational Vision-Language Model)
---
## What Is Included
| File | Size | Contents |
| :--- | :---: | :--- |
| `tokenizer.json` | `11.4 MB` | Rust `PreTrainedTokenizerFast` BPE vocabulary (`151,669` tokens, Qwen 3 compatible + ChatML special tokens) |
| `tokenizer_config.json` | `1.2 KB` | Tokenizer configuration including the ChatML `chat_template` (`<|im_start|>` / `<|im_end|>`) |
| `polar_embedding.safetensors` | `78.3 MB` | Shared polar phase-angle embedding table & tied cosine head scale (`19,565,302` FP32 parameters) |
| `config.json` | `310 B` | Polar embedding geometry metadata (`vocab_size = 151669`, `hidden_size = 256`, `n_pairs = 128`) |
### Polar Embedding Tensors (`polar_embedding.safetensors`)
In the Geometric Rotation Transformer (`GRT`), token embeddings live in polar coordinates on `128` orthogonal `2D` planes (`hidden_size = 256`) rather than an unstructured Euclidean matrix:
```text
e(token_id)[2p] = exp(log_magnitude[token_id]) · cos(phase_angles[token_id, p])
e(token_id)[2p+1] = exp(log_magnitude[token_id]) · sin(phase_angles[token_id, p])
```
| Tensor Name | Shape | Parameters | Role |
| :--- | :---: | :---: | :--- |
| `embedding.phase_angles` | `[151669, 128]` | `19,413,632` | Per-token phase angles `θ` on the 128-torus `(S¹)^128` (tied between input embedding and output `GRTCosineHead`) |
| `embedding.log_magnitude` | `[151669]` | `151,669` | Per-token radial log-magnitude offset |
| `lm_head.output_scale` | `[]` (scalar) | `1` | Raw softplus temperature scale for the tied polar cosine language modeling head |
| **Total** | — | **`19,565,302`** | **`66.8%` of `grt-30m` total parameters (shared identically across all 6 `grt-30m*` models)** |
---
## Quickstart
### 1. Load the Tokenizer (`transformers` / `vLLM`)
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-30m-tokenizer")
tokens = tokenizer("The Givens Rotational Transformer replaces dense matrices with")
print(tokens["input_ids"])
```
### 2. Load the Shared Polar Embedding Table (`safetensors`)
```python
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
path = hf_hub_download("MatterWaveAI/grt-30m-tokenizer", "polar_embedding.safetensors")
polar_bank = load_file(path)
phase_angles = polar_bank["embedding.phase_angles"] # [151669, 128]
log_magnitude = polar_bank["embedding.log_magnitude"] # [151669]
output_scale = polar_bank["lm_head.output_scale"] # []
print(phase_angles.shape, log_magnitude.shape, output_scale.shape)
```
### 3. Harmonic Upcycling to Larger `d_model` (`256 → 512` or `1024`)
When scaling a new GRT backbone to a wider `d_model = 256 · k` (`n_pairs = 128 · k`), tile `embedding.phase_angles` by `k` along dim `-1` (`phase_angles.repeat(1, k)`) to preserve all pairwise token cosine similarities at initialization.
---
## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
---
## License
Apache 2.0
---
## Citation
```bibtex
@techreport{householter2026grt,
title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
author = {Householter, David},
institution = {MatterWave AI},
year = {2026},
url = {https://matterwave.ai}
}
```
|