README.md

---
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- yat-kernel
- distillation
- mteb
- modernbert
base_model: nomic-ai/modernbert-embed-base
model-index:
- name: modernbert-embed-base-yat
  results:
  - task: {type: STS, name: Semantic Textual Similarity}
    dataset: {type: mteb/stsbenchmark-sts, name: MTEB STSBenchmark}
    metrics: [{type: cosine_spearman, value: 0.8147}]
  - task: {type: STS, name: Semantic Textual Similarity}
    dataset: {type: mteb/sts12-sts, name: MTEB STS12}
    metrics: [{type: cosine_spearman, value: 0.7182}]
  - task: {type: STS, name: Semantic Textual Similarity}
    dataset: {type: mteb/sts16-sts, name: MTEB STS16}
    metrics: [{type: cosine_spearman, value: 0.8139}]
---

# modernbert-embed-base-yat

`nomic-ai/modernbert-embed-base` with **every GeGLU feed-forward block replaced by a
sigmoid-gated Yat-kernel MLP** (an alignment / inverse-distance kernel primitive, no GELU/GeGLU).
Only the 22 feed-forward blocks are changed; attention, embeddings and norms are the base model's.

The Yat FFNs are fit by **end-to-end last-layer distillation**: freeze everything but the FFNs and
train them so the model's *final* hidden state matches the frozen GeGLU teacher (normalized MSE on
the last-layer hidden states + a cosine term on the mean-pooled embedding), one epoch over all-nli
sentences. Matching only the final representation — rather than imitating each GeGLU block pointwise,
which hits a function-class ceiling — lets the kernel layers reallocate computation and **recover full
teacher parity**.

## Evaluation (MTEB STS, cosine Spearman)

| Task | base `modernbert-embed-base` (GeGLU) | **this model** (Yat) |
|---|---|---|
| STSBenchmark | 0.835 | 0.815 |
| STS12 | 0.676 | **0.718** |
| STS16 | 0.835 | 0.814 |
| **average** | **0.782** | **0.782** |

The Yat-kernel swap reaches the same average STS as the GeGLU base (and is ahead of it on STS12).
Scores are reproduced after a Hub round-trip (`trust_remote_code=True`). The bundled custom
architecture (`modeling_yatmodernbert.py`) is loaded automatically.

## Usage

```python
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("mlnomad/modernbert-embed-base-yat", trust_remote_code=True)
emb = m.encode(["A man is eating food.", "A man is eating a meal."])
```

## The Yat FFN

```
g(x) = ( softplus(a) * (x·W + b)^2 / (||x - W||^2 + exp(le)) * sigmoid(gate(x)) ) @ A + c
```

a non-negative rational (alignment-over-distance) kernel feature with a sigmoid gate, replacing the
GeGLU map `d -> 4d -> d` at the same hidden width.

## Notes

- Reaches teacher parity by distillation; this is the ceiling of distillation (matches, does not beat
  the base). A light contrastive fine-tune of the FFNs alone does **not** exceed the base.
- Quantizes losslessly to int8 (weight-only PTQ leaves STS unchanged); below 8 bits the rational
  kernel is more sensitive than GeGLU under naive uniform quantization.

Part of the ⵟ-kernel research project (kernel-native replacements for transformer FFNs).