Update README: remove GNU Free Software Statement, keep Apache 2.0 and matterwave.ai citation

57fc8c9 verified about 17 hours ago

8.86 kB

---
license: apache-2.0
tags:
  - grt
  - geometric-rotation-transformer
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - circular-pooling
  - rotational
  - qwen3
datasets:
  - sentence-transformers/all-nli
  - sentence-transformers/stsb
language:
  - en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: MatterWaveAI/grt-30m
---

# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)

An **embedding-only Geometric Rotation Transformer (`GRTModel`)** contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using **planar-pair `circular` pooling** (`sentence-transformers`).

Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, **`circular` pooling** decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.

---

## Model Details

| Property | Value |
|---|---|
| **Architecture** | `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) |
| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
| **Pooling strategy** | `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) |
| **Output dimensionality** | `256` (`128` planar `(x, y)` pairs, L2-normalized) |
| **Total stored parameters** | `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) |
| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 rotational blocks) |
| **`hidden_size` (`d_model`)** | `256` |
| **`num_hidden_layers` (`n_blocks`)** | `12` |
| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
| **`intermediate_size` (`d_ff`)** | `1,024` |
| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
| **Vocabulary** | `151,669` (Qwen 3 BPE) |
| **Max sequence length** | `512` (supports up to `2,048` positions) |

---

## Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

Starting from the pretrained [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) causal checkpoint (`lm_head.output_scale` stripped to form a headless `GRTModel` encoder), the `151,669` lexical `embedding.phase_angles` (`19.41M` parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the **`9,740,196` rotational backbone parameters + `151,669` token `log_magnitude` weights (`9,891,865` trainable parameters)** in `FP32` on an **Intel Arc 140T GPU (`xpu:0`, 32 GB)** using fused oneAPI DPC++ SYCL Givens and Clifford kernels (`~75s` total wall-clock time):

| Stage | Dataset & Budget | Objective & Hyperparameters |
|---|---|---|
| **Parameter Freezing** | `embedding.phase_angles` frozen (`19.41M` params) | `9,891,865` trainable parameters (`12` rotational blocks + `embedding.log_magnitude`); max grad norm `1.0` |
| **Stage 1: AllNLI Contrastive Alignment** | `sentence-transformers/all-nli` (`triplet` split) · `150 steps` (`4,800` triplets, `<1%` of corpus) | In-batch `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`), `batch_size = 32`, `max_length = 96`, `PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)` |
| **Stage 2: STS-B Rank & Metric Calibration** | `sentence-transformers/stsb` (`train` split, `5,749` pairs) · `2 epochs` (`358 steps`) | `CoSENT` rank-consistency loss (`τ = 20.0`) + `2.0 × CosineMSE`, `batch_size = 32`, `max_length = 128`, `PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)` |

---

## Evaluation Results (STS-B)

Evaluated on the **STS Benchmark (`sentence-transformers/stsb`)** validation and test splits using `circular` pooling and cosine similarity:

| Split | Metric | `GRT-30M` (Pretrained Base) | `GRT-30M-Embedding` (`PairAdam`) | `AdamW` Ablation¹ | Delta (`PairAdam` vs. Base) |
|---|---|---:|---:|---:|---:|
| **STS-B Validation** | **Spearman (`×100`)** | `45.51` | **`71.79`** | `72.89` | **`+26.28`** |
| **STS-B Test** | **Spearman (`×100`)** | `28.49` | **`62.42`** | `63.17` | **`+33.93`** |
| **STS-B Test** | **Pearson (`×100`)** | `29.81` | **`63.30`** | `63.72` | **`+33.49`** |

> ¹ *Optimizer comparison note:* The published checkpoint is trained with `PairAdam` (zero weight decay and `[-π, π]` wrapping on rotation angles, `max_angular_step = 0.05`, and rank-1 factored second moments with a `3×` learning-rate multiplier on `Cl(2,0)` coupling tensors) for consistency with `GRT-30M` pretraining. In a head-to-head run on the identical `508`-step schedule and random seed (`42`), standard `AdamW` (`1×` coupling LR, full diagonal second moments) reaches `72.89` Val / `63.17` Test Spearman (`+1.10` / `+0.75` points), indicating that while `PairAdam`'s factored moments and `3×` coupling LR multiplier stabilize large-batch (`524k` tok/step) pretraining from scratch, standard diagonal moments at `1×` coupling LR are slightly gentler during short, small-batch (`B = 32`) downstream fine-tuning.

---

## Usage

### 1. `sentence-transformers` (Recommended)

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")

sentences = [
    "Plants convert sunlight into chemical energy through photosynthesis.",
    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
    "The stock market experienced volatility during the afternoon session.",
]

embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities)
```

### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)

```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-embedding"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()


def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
    radii = pairs.norm(dim=-1, keepdim=True)
    directions = pairs / torch.clamp(radii, min=1e-9)
    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)


inputs = tokenizer(
    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**inputs)
    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])

print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
```

---

## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)

- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

---

## License

Apache 2.0

---

## Citation

```bibtex
@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}
```

	---
	license: apache-2.0
	tags:
	- grt
	- geometric-rotation-transformer
	- sentence-transformers
	- feature-extraction
	- sentence-similarity
	- circular-pooling
	- rotational
	- qwen3
	datasets:
	- sentence-transformers/all-nli
	- sentence-transformers/stsb
	language:
	- en
	library_name: sentence-transformers
	pipeline_tag: sentence-similarity
	base_model: MatterWaveAI/grt-30m
	---

	# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)

	An embedding-only Geometric Rotation Transformer (`GRTModel`) contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using planar-pair `circular` pooling (`sentence-transformers`).

	Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, `circular` pooling decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.

	---

	## Model Details

	\| Property \| Value \|
	\|---\|---\|
	\| Architecture \| `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) \|
	\| Base model \| [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) \|
	\| Pooling strategy \| `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) \|
	\| Output dimensionality \| `256` (`128` planar `(x, y)` pairs, L2-normalized) \|
	\| Total stored parameters \| `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) \|
	\| Non-embedding backbone parameters \| `9,740,197` (`9.74M` across 12 rotational blocks) \|
	\| `hidden_size` (`d_model`) \| `256` \|
	\| `num_hidden_layers` (`n_blocks`) \| `12` \|
	\| `num_attention_heads` (`n_heads`) \| `4` (`head_dim = 64`) \|
	\| `intermediate_size` (`d_ff`) \| `1,024` \|
	\| `n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations` \| `11` / `1` / `21` \|
	\| Vocabulary \| `151,669` (Qwen 3 BPE) \|
	\| Max sequence length \| `512` (supports up to `2,048` positions) \|

	---

	## Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

	Starting from the pretrained [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) causal checkpoint (`lm_head.output_scale` stripped to form a headless `GRTModel` encoder), the `151,669` lexical `embedding.phase_angles` (`19.41M` parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the `9,740,196` rotational backbone parameters + `151,669` token `log_magnitude` weights (`9,891,865` trainable parameters) in `FP32` on an Intel Arc 140T GPU (`xpu:0`, 32 GB) using fused oneAPI DPC++ SYCL Givens and Clifford kernels (`~75s` total wall-clock time):

	\| Stage \| Dataset & Budget \| Objective & Hyperparameters \|
	\|---\|---\|---\|
	\| Parameter Freezing \| `embedding.phase_angles` frozen (`19.41M` params) \| `9,891,865` trainable parameters (`12` rotational blocks + `embedding.log_magnitude`); max grad norm `1.0` \|
	\| Stage 1: AllNLI Contrastive Alignment \| `sentence-transformers/all-nli` (`triplet` split) · `150 steps` (`4,800` triplets, `<1%` of corpus) \| In-batch `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`), `batch_size = 32`, `max_length = 96`, `PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)` \|
	\| Stage 2: STS-B Rank & Metric Calibration \| `sentence-transformers/stsb` (`train` split, `5,749` pairs) · `2 epochs` (`358 steps`) \| `CoSENT` rank-consistency loss (`τ = 20.0`) + `2.0 × CosineMSE`, `batch_size = 32`, `max_length = 128`, `PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)` \|

	---

	## Evaluation Results (STS-B)

	Evaluated on the STS Benchmark (`sentence-transformers/stsb`) validation and test splits using `circular` pooling and cosine similarity:

	\| Split \| Metric \| `GRT-30M` (Pretrained Base) \| `GRT-30M-Embedding` (`PairAdam`) \| `AdamW` Ablation¹ \| Delta (`PairAdam` vs. Base) \|
	\|---\|---\|---:\|---:\|---:\|---:\|
	\| STS-B Validation \| Spearman (`×100`) \| `45.51` \| `71.79` \| `72.89` \| `+26.28` \|
	\| STS-B Test \| Spearman (`×100`) \| `28.49` \| `62.42` \| `63.17` \| `+33.93` \|
	\| STS-B Test \| Pearson (`×100`) \| `29.81` \| `63.30` \| `63.72` \| `+33.49` \|

	> ¹ Optimizer comparison note: The published checkpoint is trained with `PairAdam` (zero weight decay and `[-π, π]` wrapping on rotation angles, `max_angular_step = 0.05`, and rank-1 factored second moments with a `3×` learning-rate multiplier on `Cl(2,0)` coupling tensors) for consistency with `GRT-30M` pretraining. In a head-to-head run on the identical `508`-step schedule and random seed (`42`), standard `AdamW` (`1×` coupling LR, full diagonal second moments) reaches `72.89` Val / `63.17` Test Spearman (`+1.10` / `+0.75` points), indicating that while `PairAdam`'s factored moments and `3×` coupling LR multiplier stabilize large-batch (`524k` tok/step) pretraining from scratch, standard diagonal moments at `1×` coupling LR are slightly gentler during short, small-batch (`B = 32`) downstream fine-tuning.

	---

	## Usage

	### 1. `sentence-transformers` (Recommended)

	```python
	from sentence_transformers import SentenceTransformer

	model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")

	sentences = [
	"Plants convert sunlight into chemical energy through photosynthesis.",
	"Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
	"The stock market experienced volatility during the afternoon session.",
	]

	embeddings = model.encode(sentences)
	similarities = model.similarity(embeddings, embeddings)
	print(similarities)
	```

	### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)

	```python
	import torch
	import torch.nn.functional as F
	from transformers import AutoModel, AutoTokenizer

	model_id = "MatterWaveAI/grt-30m-embedding"
	tokenizer = AutoTokenizer.from_pretrained(model_id)
	model = AutoModel.from_pretrained(model_id).eval()


	def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
	pairs = token_embeddings.float().unflatten(-1, (-1, 2))
	mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
	radii = pairs.norm(dim=-1, keepdim=True)
	directions = pairs / torch.clamp(radii, min=1e-9)
	denom = torch.clamp(mask.sum(dim=1), min=1e-9)
	pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
	return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)


	inputs = tokenizer(
	["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
	padding=True,
	truncation=True,
	return_tensors="pt",
	)
	with torch.no_grad():
	outputs = model(**inputs)
	embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])

	print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
	```

	---

	## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face

	1. Base Causal LM: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
	2. Chat Fine-Tune: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
	3. 4-Expert Mixture-of-Rotations (`MoR`): [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
	4. Circular-Pooled Sentence Embedding: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
	5. Cross-Encoder Reranker: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
	6. Vision-Language (`GRTVL`): [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
	7. Canonical Tokenizer & Polar Embedding Bank: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)

	- Whitepaper: [The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)

	---

	## License

	Apache 2.0

	---

	## Citation

	```bibtex
	@techreport{householter2026grt,
	title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
	author = {Householter, David},
	institution = {MatterWave AI},
	year = {2026},
	url = {https://matterwave.ai}
	}
	```