| ---
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - grt
|
| - geometric-rotation-transformer
|
| - mixture-of-rotations
|
| - mor
|
| - moe
|
| - rotational-transformer
|
| - givens-rotations
|
| - clifford-algebra
|
| - matterwave
|
| base_model: MatterWaveAI/grt-30m
|
| ---
|
|
|
| # GRT-30M-MoR (`MatterWaveAI/grt-30m-mor`)
|
|
|
| **GRT-30M-MoR** (`31,634,086` parameters, **zero `nn.Linear` layers**) is the **Mixture-of-Rotations (`MoR`)** upcycled variant of [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` parameters).
|
|
|
| Instead of routing tokens through multiple separate dense `nn.Linear` feed-forward matrices, **GRT-30M-MoR** exploits the abelian Lie group structure of planar `2×2` Givens rotations (`SO(2)^P` at each butterfly stage): the router blends the `E = 4` expert rotation angles and log-scales in parameter space **before** executing a single `cos / sin` Givens rotation cascade per stage — providing **`4×` FFN rotational capacity at `1×` trigonometric rotation compute**.
|
|
|
| ---
|
|
|
| ## Model Details
|
|
|
| | Property | Value |
|
| | :--- | :--- |
|
| | **Architecture** | `GRTForCausalLM` (`model_type: "grt"`, Mixture-of-Rotations) |
|
| | **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
|
| | **Total stored parameters** | `31,634,086` (`31.63M`, `+2,328,588` over `grt-30m`) |
|
| | **Non-embedding backbone parameters** | `12,068,785` (`12.07M` across 12 MoR blocks) |
|
| | **Number of FFN rotational experts (`ffn_n_experts`)** | `4` per block (`48` rotational FFN experts across 12 layers) |
|
| | **MoR parameterization (`mor_mode`)** | `"direct"` (`expert_expand_angles [4, 21, 512]`, `expert_expand_scales [4, 21, 1024]`, `expert_contract_angles [4, 21, 512]`, `expert_contract_scales [4, 21, 1024]`) |
|
| | **Router architecture (`mor_router_mode`)** | `"rotational"` (`GRTRotationalRouter` — cosine phase-angle alignment on `(S¹)^128`, **0 `nn.Linear` layers**) |
|
| | **`nn.Linear` layers** | **`0`** |
|
| | **`hidden_size` (`d_model`) / `intermediate_size` (`d_ff`)** | `256` / `1,024` |
|
| | **`num_hidden_layers` / `num_attention_heads`** | `12` / `4` (`head_dim = 64`) |
|
| | **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
|
| | **Vocabulary** | `151,669` (Qwen 3 BPE, tied polar cosine head) |
|
|
|
| ---
|
|
|
| ## Architectural Highlights
|
|
|
| 1. **4× FFN Rotational Capacity at 1× Rotation Compute (`_mor_rotate`)**:
|
| Each of the 12 `GRTFeedForward` blocks maintains `ffn_n_experts = 4` rotational cascades. Given per-token router probabilities `w[n, e]` (`sum_e w[n, e] = 1`), the effective rotation angle and positive scale at stage `k`, pair `p` / dimension `d` are blended prior to rotation:
|
| ```text
|
| θ_blended[n, k, p] = Σ_{e=0..3} w[n, e] · θ_expert[e, k, p]
|
| s_blended[n, k, d] = Σ_{e=0..3} w[n, e] · softplus(s_raw_expert[e, k, d])
|
| ```
|
| A single Givens rotation `(cos θ_blended, sin θ_blended)` is then applied per pair.
|
| 2. **100% Rotational Router (`GRTRotationalRouter`, 0 `nn.Linear` Layers)**:
|
| Tokens are routed to experts via scaled cosine alignment between L2-normalized planar coordinate pairs `(u_p, v_p)` and `E = 4` learned phase-angle prototypes `(cos φ_{e,p}, sin φ_{e,p})` with learned `softplus(output_scale)` temperature.
|
| 3. **Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (`PairAdam`)**:
|
| Initialized from `MatterWaveAI/grt-30m` via zero-sum Walsh–Hadamard angular perturbations (`Σ_e Δ^(e) = 0`) and specialized on a 4-domain corpus (`rotational_geometry`, `science_and_physics`, `code_and_algorithms`, `reasoning_and_language`) on an Intel Arc 140T XPU (`xpu:0`) with `PairAdam` and Switch-style load balancing:
|
|
|
| | Metric | Initial Upcycle | After MoR Specialization (`PairAdam`) |
|
| | :--- | :---: | :---: |
|
| | **Held-Out Validation Loss** | `5.8467` | **`0.5115`** |
|
| | **Held-Out Validation Perplexity** | `346.10` | **`1.67`** |
|
| | **Domain Loss — `code_and_algorithms`** | `5.4092` | **`0.4334`** |
|
| | **Domain Loss — `rotational_geometry`** | `6.6421` | **`0.4991`** |
|
| | **Domain Loss — `science_and_physics`** | `5.3337` | **`0.5293`** |
|
| | **Domain Loss — `reasoning_and_language`** | `6.1720` | **`0.6442`** |
|
| | **Mean Expert Routing Load (`E = 4`)** | `[0.25, 0.25, 0.25, 0.25]` | **`[0.250, 0.248, 0.256, 0.245]`** (`entropy = 1.386 / 1.386`) |
|
|
|
| ---
|
|
|
| ## Quickstart (`transformers`)
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| model_id = "MatterWaveAI/grt-30m-mor"
|
| tokenizer = AutoTokenizer.from_pretrained(model_id)
|
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)
|
|
|
| inputs = tokenizer("The Givens Rotational Transformer replaces dense matrices with", return_tensors="pt")
|
| out_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
|
| print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
|
| ```
|
|
|
| ## Quickstart (`vLLM`)
|
|
|
| ```python
|
| from vllm import LLM, SamplingParams
|
|
|
| llm = LLM(model="MatterWaveAI/grt-30m-mor", model_impl="transformers", dtype="float32")
|
| outputs = llm.generate(
|
| ["The Givens Rotational Transformer replaces dense matrices with"],
|
| SamplingParams(temperature=0.0, max_tokens=32),
|
| )
|
| print(outputs[0].outputs[0].text)
|
| ```
|
|
|
| ---
|
|
|
| ## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
|
|
|
| 1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
|
| 2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
|
| 3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
|
| 4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
|
| 5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
|
| 6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
|
| 7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
|
| 8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
|
| 9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
|
|
|
| - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
|
|
|
| ---
|
|
|
| ## License
|
|
|
| Apache 2.0
|
|
|
| ---
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @techreport{householter2026grt,
|
| title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
|
| author = {Householter, David},
|
| institution = {MatterWave AI},
|
| year = {2026},
|
| url = {https://matterwave.ai}
|
| }
|
| ```
|
| |
