---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - grt
  - geometric-rotation-transformer
  - mixture-of-rotations
  - mor
  - moe
  - rotational-transformer
  - givens-rotations
  - clifford-algebra
  - matterwave
base_model: MatterWaveAI/grt-30m
---

# GRT-30M-MoR (`MatterWaveAI/grt-30m-mor`)

**GRT-30M-MoR** (`31,634,086` parameters, **zero `nn.Linear` layers**) is the **Mixture-of-Rotations (`MoR`)** upcycled variant of [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` parameters).

Instead of routing tokens through multiple separate dense `nn.Linear` feed-forward matrices, **GRT-30M-MoR** exploits the abelian Lie group structure of planar `2×2` Givens rotations (`SO(2)^P` at each butterfly stage): the router blends the `E = 4` expert rotation angles and log-scales in parameter space **before** executing a single `cos / sin` Givens rotation cascade per stage — providing **`4×` FFN rotational capacity at `1×` trigonometric rotation compute**.

---

## Model Details

| Property | Value |
| :--- | :--- |
| **Architecture** | `GRTForCausalLM` (`model_type: "grt"`, Mixture-of-Rotations) |
| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
| **Total stored parameters** | `31,634,086` (`31.63M`, `+2,328,588` over `grt-30m`) |
| **Non-embedding backbone parameters** | `12,068,785` (`12.07M` across 12 MoR blocks) |
| **Number of FFN rotational experts (`ffn_n_experts`)** | `4` per block (`48` rotational FFN experts across 12 layers) |
| **MoR parameterization (`mor_mode`)** | `"direct"` (`expert_expand_angles [4, 21, 512]`, `expert_expand_scales [4, 21, 1024]`, `expert_contract_angles [4, 21, 512]`, `expert_contract_scales [4, 21, 1024]`) |
| **Router architecture (`mor_router_mode`)** | `"rotational"` (`GRTRotationalRouter` — cosine phase-angle alignment on `(S¹)^128`, **0 `nn.Linear` layers**) |
| **`nn.Linear` layers** | **`0`** |
| **`hidden_size` (`d_model`) / `intermediate_size` (`d_ff`)** | `256` / `1,024` |
| **`num_hidden_layers` / `num_attention_heads`** | `12` / `4` (`head_dim = 64`) |
| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
| **Vocabulary** | `151,669` (Qwen 3 BPE, tied polar cosine head) |

---

## Architectural Highlights

1. **4× FFN Rotational Capacity at 1× Rotation Compute (`_mor_rotate`)**:
   Each of the 12 `GRTFeedForward` blocks maintains `ffn_n_experts = 4` rotational cascades. Given per-token router probabilities `w[n, e]` (`sum_e w[n, e] = 1`), the effective rotation angle and positive scale at stage `k`, pair `p` / dimension `d` are blended prior to rotation:
   ```text
   θ_blended[n, k, p] = Σ_{e=0..3} w[n, e] · θ_expert[e, k, p]
   s_blended[n, k, d] = Σ_{e=0..3} w[n, e] · softplus(s_raw_expert[e, k, d])
   ```
   A single Givens rotation `(cos θ_blended, sin θ_blended)` is then applied per pair.
2. **100% Rotational Router (`GRTRotationalRouter`, 0 `nn.Linear` Layers)**:
   Tokens are routed to experts via scaled cosine alignment between L2-normalized planar coordinate pairs `(u_p, v_p)` and `E = 4` learned phase-angle prototypes `(cos φ_{e,p}, sin φ_{e,p})` with learned `softplus(output_scale)` temperature.
3. **Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (`PairAdam`)**:
   Initialized from `MatterWaveAI/grt-30m` via zero-sum Walsh–Hadamard angular perturbations (`Σ_e Δ^(e) = 0`) and specialized on a 4-domain corpus (`rotational_geometry`, `science_and_physics`, `code_and_algorithms`, `reasoning_and_language`) on an Intel Arc 140T XPU (`xpu:0`) with `PairAdam` and Switch-style load balancing:

| Metric | Initial Upcycle | After MoR Specialization (`PairAdam`) |
| :--- | :---: | :---: |
| **Held-Out Validation Loss** | `5.8467` | **`0.5115`** |
| **Held-Out Validation Perplexity** | `346.10` | **`1.67`** |
| **Domain Loss — `code_and_algorithms`** | `5.4092` | **`0.4334`** |
| **Domain Loss — `rotational_geometry`** | `6.6421` | **`0.4991`** |
| **Domain Loss — `science_and_physics`** | `5.3337` | **`0.5293`** |
| **Domain Loss — `reasoning_and_language`** | `6.1720` | **`0.6442`** |
| **Mean Expert Routing Load (`E = 4`)** | `[0.25, 0.25, 0.25, 0.25]` | **`[0.250, 0.248, 0.256, 0.245]`** (`entropy = 1.386 / 1.386`) |

---

## Quickstart (`transformers`)

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-mor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)

inputs = tokenizer("The Givens Rotational Transformer replaces dense matrices with", return_tensors="pt")
out_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
```

## Quickstart (`vLLM`)

```python
from vllm import LLM, SamplingParams

llm = LLM(model="MatterWaveAI/grt-30m-mor", model_impl="transformers", dtype="float32")
outputs = llm.generate(
    ["The Givens Rotational Transformer replaces dense matrices with"],
    SamplingParams(temperature=0.0, max_tokens=32),
)
print(outputs[0].outputs[0].text)
```

---

## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)

- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

---

## License

Apache 2.0

---

## Citation

```bibtex
@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
```