README.md

metadata

license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - grt
  - geometric-rotation-transformer
  - mixture-of-rotations
  - mor
  - moe
  - rotational-transformer
  - givens-rotations
  - clifford-algebra
  - matterwave
base_model: MatterWaveAI/grt-30m

GRT-30M-MoR (MatterWaveAI/grt-30m-mor)

GRT-30M-MoR (31,634,086 parameters, zero nn.Linear layers) is the Mixture-of-Rotations (MoR) upcycled variant of MatterWaveAI/grt-30m (29,305,498 parameters).

Instead of routing tokens through multiple separate dense nn.Linear feed-forward matrices, GRT-30M-MoR exploits the abelian Lie group structure of planar 2×2 Givens rotations (SO(2)^P at each butterfly stage): the router blends the E = 4 expert rotation angles and log-scales in parameter space before executing a single cos / sin Givens rotation cascade per stage — providing 4× FFN rotational capacity at 1× trigonometric rotation compute.

---

Model Details

Property	Value
Architecture	GRTForCausalLM (model_type: "grt", Mixture-of-Rotations)
Base model	MatterWaveAI/grt-30m
Total stored parameters	31,634,086 (31.63M, +2,328,588 over grt-30m)
Non-embedding backbone parameters	12,068,785 (12.07M across 12 MoR blocks)
Number of FFN rotational experts (ffn_n_experts)	4 per block (48 rotational FFN experts across 12 layers)
MoR parameterization (mor_mode)	"direct" (expert_expand_angles [4, 21, 512], expert_expand_scales [4, 21, 1024], expert_contract_angles [4, 21, 512], expert_contract_scales [4, 21, 1024])
Router architecture (mor_router_mode)	"rotational" (GRTRotationalRouter — cosine phase-angle alignment on (S¹)^128, 0 nn.Linear layers)
nn.Linear layers	0
hidden_size (d_model) / intermediate_size (d_ff)	256 / 1,024
num_hidden_layers / num_attention_heads	12 / 4 (head_dim = 64)
n_qkv_rotations / n_mixer_rotations / ffn_n_rotations	11 / 1 / 21
Vocabulary	151,669 (Qwen 3 BPE, tied polar cosine head)

---

Architectural Highlights

1. 4× FFN Rotational Capacity at 1× Rotation Compute (_mor_rotate): Each of the 12 GRTFeedForward blocks maintains ffn_n_experts = 4 rotational cascades. Given per-token router probabilities w[n, e] (sum_e w[n, e] = 1), the effective rotation angle and positive scale at stage k, pair p / dimension d are blended prior to rotation:

   θ_blended[n, k, p] = Σ_{e=0..3} w[n, e] · θ_expert[e, k, p]
   s_blended[n, k, d] = Σ_{e=0..3} w[n, e] · softplus(s_raw_expert[e, k, d])
   

   A single Givens rotation (cos θ_blended, sin θ_blended) is then applied per pair.
2. 100% Rotational Router (GRTRotationalRouter, 0 nn.Linear Layers): Tokens are routed to experts via scaled cosine alignment between L2-normalized planar coordinate pairs (u_p, v_p) and E = 4 learned phase-angle prototypes (cos φ_{e,p}, sin φ_{e,p}) with learned softplus(output_scale) temperature.
3. Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (PairAdam): Initialized from MatterWaveAI/grt-30m via zero-sum Walsh–Hadamard angular perturbations (Σ_e Δ^(e) = 0) and specialized on a 4-domain corpus (rotational_geometry, science_and_physics, code_and_algorithms, reasoning_and_language) on an Intel Arc 140T XPU (xpu:0) with PairAdam and Switch-style load balancing:

Metric	Initial Upcycle	After MoR Specialization (PairAdam)
Held-Out Validation Loss	5.8467	0.5115
Held-Out Validation Perplexity	346.10	1.67
Domain Loss — code_and_algorithms	5.4092	0.4334
Domain Loss — rotational_geometry	6.6421	0.4991
Domain Loss — science_and_physics	5.3337	0.5293
Domain Loss — reasoning_and_language	6.1720	0.6442
Mean Expert Routing Load (E = 4)	[0.25, 0.25, 0.25, 0.25]	[0.250, 0.248, 0.256, 0.245] (entropy = 1.386 / 1.386)

---

Quickstart (transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-mor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)

inputs = tokenizer("The Givens Rotational Transformer replaces dense matrices with", return_tensors="pt")
out_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out_ids[0], skip_special_tokens=True))

Quickstart (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(model="MatterWaveAI/grt-30m-mor", model_impl="transformers", dtype="float32")
outputs = llm.generate(
    ["The Givens Rotational Transformer replaces dense matrices with"],
    SamplingParams(temperature=0.0, max_tokens=32),
)
print(outputs[0].outputs[0].text)

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}