---
license: apache-2.0
tags:
  - grt
  - geometric-rotation-transformer
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - circular-pooling
  - rotational
  - qwen3
datasets:
  - sentence-transformers/all-nli
  - sentence-transformers/stsb
language:
  - en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: MatterWaveAI/grt-30m
---

# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)

An **embedding-only Geometric Rotation Transformer (`GRTModel`)** contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using **planar-pair `circular` pooling** (`sentence-transformers`).

Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, **`circular` pooling** decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.

---

## Model Details

| Property | Value |
|---|---|
| **Architecture** | `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) |
| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
| **Pooling strategy** | `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) |
| **Output dimensionality** | `256` (`128` planar `(x, y)` pairs, L2-normalized) |
| **Total stored parameters** | `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) |
| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 rotational blocks) |
| **`hidden_size` (`d_model`)** | `256` |
| **`num_hidden_layers` (`n_blocks`)** | `12` |
| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
| **`intermediate_size` (`d_ff`)** | `1,024` |
| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
| **Vocabulary** | `151,669` (Qwen 3 BPE) |
| **Max sequence length** | `512` (supports up to `2,048` positions) |

---

## Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

Starting from the pretrained [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) causal checkpoint (`lm_head.output_scale` stripped to form a headless `GRTModel` encoder), the `151,669` lexical `embedding.phase_angles` (`19.41M` parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the **`9,740,196` rotational backbone parameters + `151,669` token `log_magnitude` weights (`9,891,865` trainable parameters)** in `FP32` on an **Intel Arc 140T GPU (`xpu:0`, 32 GB)** using fused oneAPI DPC++ SYCL Givens and Clifford kernels (`~75s` total wall-clock time):

| Stage | Dataset & Budget | Objective & Hyperparameters |
|---|---|---|
| **Parameter Freezing** | `embedding.phase_angles` frozen (`19.41M` params) | `9,891,865` trainable parameters (`12` rotational blocks + `embedding.log_magnitude`); max grad norm `1.0` |
| **Stage 1: AllNLI Contrastive Alignment** | `sentence-transformers/all-nli` (`triplet` split) · `150 steps` (`4,800` triplets, `<1%` of corpus) | In-batch `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`), `batch_size = 32`, `max_length = 96`, `PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)` |
| **Stage 2: STS-B Rank & Metric Calibration** | `sentence-transformers/stsb` (`train` split, `5,749` pairs) · `2 epochs` (`358 steps`) | `CoSENT` rank-consistency loss (`τ = 20.0`) + `2.0 × CosineMSE`, `batch_size = 32`, `max_length = 128`, `PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)` |

---

## Evaluation Results (STS-B)

Evaluated on the **STS Benchmark (`sentence-transformers/stsb`)** validation and test splits using `circular` pooling and cosine similarity:

| Split | Metric | `GRT-30M` (Pretrained Base) | `GRT-30M-Embedding` (`PairAdam`) | `AdamW` Ablation¹ | Delta (`PairAdam` vs. Base) |
|---|---|---:|---:|---:|---:|
| **STS-B Validation** | **Spearman (`×100`)** | `45.51` | **`71.79`** | `72.89` | **`+26.28`** |
| **STS-B Test** | **Spearman (`×100`)** | `28.49` | **`62.42`** | `63.17` | **`+33.93`** |
| **STS-B Test** | **Pearson (`×100`)** | `29.81` | **`63.30`** | `63.72` | **`+33.49`** |

> ¹ *Optimizer comparison note:* The published checkpoint is trained with `PairAdam` (zero weight decay and `[-π, π]` wrapping on rotation angles, `max_angular_step = 0.05`, and rank-1 factored second moments with a `3×` learning-rate multiplier on `Cl(2,0)` coupling tensors) for consistency with `GRT-30M` pretraining. In a head-to-head run on the identical `508`-step schedule and random seed (`42`), standard `AdamW` (`1×` coupling LR, full diagonal second moments) reaches `72.89` Val / `63.17` Test Spearman (`+1.10` / `+0.75` points), indicating that while `PairAdam`'s factored moments and `3×` coupling LR multiplier stabilize large-batch (`524k` tok/step) pretraining from scratch, standard diagonal moments at `1×` coupling LR are slightly gentler during short, small-batch (`B = 32`) downstream fine-tuning.

---

## Usage

### 1. `sentence-transformers` (Recommended)

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")

sentences = [
    "Plants convert sunlight into chemical energy through photosynthesis.",
    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
    "The stock market experienced volatility during the afternoon session.",
]

embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities)
```

### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)

```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-embedding"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()


def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
    radii = pairs.norm(dim=-1, keepdim=True)
    directions = pairs / torch.clamp(radii, min=1e-9)
    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)


inputs = tokenizer(
    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**inputs)
    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])

print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
```

---

## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)

- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

---

## License

Apache 2.0

---

## Citation

```bibtex
@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
```