README.md

metadata

license: apache-2.0
library_name: transformers
tags:
  - grt
  - geometric-rotation-transformer
  - tokenizer
  - polar-embedding
  - phase-angle-embedding
  - rotational-transformer
  - qwen3
  - matterwave
base_model: MatterWaveAI/grt-30m

GRT-30M-Tokenizer (MatterWaveAI/grt-30m-tokenizer)

Canonical BPE Tokenizer (151,669 tokens) and shared Polar Phase-Angle Token Embedding Bank (19,565,302 parameters, 0 nn.Linear layers) for the GRT-30M Research Suite:

• MatterWaveAI/grt-30m (Base Causal LM)
• MatterWaveAI/grt-30m-chat (ChatML Conversational SFT)
• MatterWaveAI/grt-30m-mor (4-Expert Mixture-of-Rotations Causal LM)
• MatterWaveAI/grt-30m-embedding (Circular-Pooled Bi-Encoder)
• MatterWaveAI/grt-30m-reranker (Rotational Cross-Encoder Reranker)
• MatterWaveAI/grt-30m-vl (100% Rotational Vision-Language Model)

---

What Is Included

File	Size	Contents
tokenizer.json	11.4 MB	Rust PreTrainedTokenizerFast BPE vocabulary (151,669 tokens, Qwen 3 compatible + ChatML special tokens)
tokenizer_config.json	1.2 KB	Tokenizer configuration including the ChatML chat_template (`<
polar_embedding.safetensors	78.3 MB	Shared polar phase-angle embedding table & tied cosine head scale (19,565,302 FP32 parameters)
config.json	310 B	Polar embedding geometry metadata (vocab_size = 151669, hidden_size = 256, n_pairs = 128)

Polar Embedding Tensors (polar_embedding.safetensors)

In the Geometric Rotation Transformer (GRT), token embeddings live in polar coordinates on 128 orthogonal 2D planes (hidden_size = 256) rather than an unstructured Euclidean matrix:

e(token_id)[2p]   = exp(log_magnitude[token_id]) · cos(phase_angles[token_id, p])
e(token_id)[2p+1] = exp(log_magnitude[token_id]) · sin(phase_angles[token_id, p])

Tensor Name	Shape	Parameters	Role
embedding.phase_angles	[151669, 128]	19,413,632	Per-token phase angles θ on the 128-torus (S¹)^128 (tied between input embedding and output GRTCosineHead)
embedding.log_magnitude	[151669]	151,669	Per-token radial log-magnitude offset
lm_head.output_scale	[] (scalar)	1	Raw softplus temperature scale for the tied polar cosine language modeling head
Total	—	19,565,302	66.8% of grt-30m total parameters (shared identically across all 6 grt-30m* models)

---

Quickstart

1. Load the Tokenizer (transformers / vLLM)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-30m-tokenizer")
tokens = tokenizer("The Givens Rotational Transformer replaces dense matrices with")
print(tokens["input_ids"])

2. Load the Shared Polar Embedding Table (safetensors)

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("MatterWaveAI/grt-30m-tokenizer", "polar_embedding.safetensors")
polar_bank = load_file(path)

phase_angles = polar_bank["embedding.phase_angles"]   # [151669, 128]
log_magnitude = polar_bank["embedding.log_magnitude"] # [151669]
output_scale = polar_bank["lm_head.output_scale"]     # []
print(phase_angles.shape, log_magnitude.shape, output_scale.shape)

3. Harmonic Upcycling to Larger d_model (256 → 512 or 1024)

When scaling a new GRT backbone to a wider d_model = 256 · k (n_pairs = 128 · k), tile embedding.phase_angles by k along dim -1 (phase_angles.repeat(1, k)) to preserve all pairwise token cosine similarities at initialization.

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}