Update README: remove GNU Free Software Statement, keep Apache 2.0 and matterwave.ai citation
1d0101c verified | license: apache-2.0 | |
| library_name: transformers | |
| tags: | |
| - grt | |
| - geometric-rotation-transformer | |
| - tokenizer | |
| - polar-embedding | |
| - phase-angle-embedding | |
| - rotational-transformer | |
| - qwen3 | |
| - matterwave | |
| base_model: MatterWaveAI/grt-30m | |
| # GRT-30M-Tokenizer (`MatterWaveAI/grt-30m-tokenizer`) | |
| Canonical **BPE Tokenizer (`151,669` tokens)** and shared **Polar Phase-Angle Token Embedding Bank (`19,565,302` parameters, `0 nn.Linear` layers)** for the **GRT-30M Research Suite**: | |
| - [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (Base Causal LM) | |
| - [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat) (ChatML Conversational SFT) | |
| - [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor) (4-Expert Mixture-of-Rotations Causal LM) | |
| - [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding) (Circular-Pooled Bi-Encoder) | |
| - [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker) (Rotational Cross-Encoder Reranker) | |
| - [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl) (100% Rotational Vision-Language Model) | |
| --- | |
| ## What Is Included | |
| | File | Size | Contents | | |
| | :--- | :---: | :--- | | |
| | `tokenizer.json` | `11.4 MB` | Rust `PreTrainedTokenizerFast` BPE vocabulary (`151,669` tokens, Qwen 3 compatible + ChatML special tokens) | | |
| | `tokenizer_config.json` | `1.2 KB` | Tokenizer configuration including the ChatML `chat_template` (`<|im_start|>` / `<|im_end|>`) | | |
| | `polar_embedding.safetensors` | `78.3 MB` | Shared polar phase-angle embedding table & tied cosine head scale (`19,565,302` FP32 parameters) | | |
| | `config.json` | `310 B` | Polar embedding geometry metadata (`vocab_size = 151669`, `hidden_size = 256`, `n_pairs = 128`) | | |
| ### Polar Embedding Tensors (`polar_embedding.safetensors`) | |
| In the Geometric Rotation Transformer (`GRT`), token embeddings live in polar coordinates on `128` orthogonal `2D` planes (`hidden_size = 256`) rather than an unstructured Euclidean matrix: | |
| ```text | |
| e(token_id)[2p] = exp(log_magnitude[token_id]) · cos(phase_angles[token_id, p]) | |
| e(token_id)[2p+1] = exp(log_magnitude[token_id]) · sin(phase_angles[token_id, p]) | |
| ``` | |
| | Tensor Name | Shape | Parameters | Role | | |
| | :--- | :---: | :---: | :--- | | |
| | `embedding.phase_angles` | `[151669, 128]` | `19,413,632` | Per-token phase angles `θ` on the 128-torus `(S¹)^128` (tied between input embedding and output `GRTCosineHead`) | | |
| | `embedding.log_magnitude` | `[151669]` | `151,669` | Per-token radial log-magnitude offset | | |
| | `lm_head.output_scale` | `[]` (scalar) | `1` | Raw softplus temperature scale for the tied polar cosine language modeling head | | |
| | **Total** | — | **`19,565,302`** | **`66.8%` of `grt-30m` total parameters (shared identically across all 6 `grt-30m*` models)** | | |
| --- | |
| ## Quickstart | |
| ### 1. Load the Tokenizer (`transformers` / `vLLM`) | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-30m-tokenizer") | |
| tokens = tokenizer("The Givens Rotational Transformer replaces dense matrices with") | |
| print(tokens["input_ids"]) | |
| ``` | |
| ### 2. Load the Shared Polar Embedding Table (`safetensors`) | |
| ```python | |
| import torch | |
| from huggingface_hub import hf_hub_download | |
| from safetensors.torch import load_file | |
| path = hf_hub_download("MatterWaveAI/grt-30m-tokenizer", "polar_embedding.safetensors") | |
| polar_bank = load_file(path) | |
| phase_angles = polar_bank["embedding.phase_angles"] # [151669, 128] | |
| log_magnitude = polar_bank["embedding.log_magnitude"] # [151669] | |
| output_scale = polar_bank["lm_head.output_scale"] # [] | |
| print(phase_angles.shape, log_magnitude.shape, output_scale.shape) | |
| ``` | |
| ### 3. Harmonic Upcycling to Larger `d_model` (`256 → 512` or `1024`) | |
| When scaling a new GRT backbone to a wider `d_model = 256 · k` (`n_pairs = 128 · k`), tile `embedding.phase_angles` by `k` along dim `-1` (`phase_angles.repeat(1, k)`) to preserve all pairwise token cosine similarities at initialization. | |
| --- | |
| ## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face | |
| 1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) | |
| 2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat) | |
| 3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor) | |
| 4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding) | |
| 5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker) | |
| 6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl) | |
| 7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer) | |
| - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf) | |
| --- | |
| ## License | |
| Apache 2.0 | |
| --- | |
| ## Citation | |
| ```bibtex | |
| @techreport{householter2026grt, | |
| title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture}, | |
| author = {Householter, David}, | |
| institution = {MatterWave AI}, | |
| year = {2026}, | |
| url = {https://matterwave.ai} | |
| } | |
| ``` | |