File size: 3,149 Bytes
0835b2f ae839f4 0835b2f ae839f4 0835b2f ae839f4 23dc9c7 ae839f4 0835b2f ae839f4 0835b2f ae839f4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- yat-kernel
- distillation
- mteb
- modernbert
base_model: nomic-ai/modernbert-embed-base
model-index:
- name: modernbert-embed-base-yat
results:
- task: {type: STS, name: Semantic Textual Similarity}
dataset: {type: mteb/stsbenchmark-sts, name: MTEB STSBenchmark}
metrics: [{type: cosine_spearman, value: 0.8147}]
- task: {type: STS, name: Semantic Textual Similarity}
dataset: {type: mteb/sts12-sts, name: MTEB STS12}
metrics: [{type: cosine_spearman, value: 0.7182}]
- task: {type: STS, name: Semantic Textual Similarity}
dataset: {type: mteb/sts16-sts, name: MTEB STS16}
metrics: [{type: cosine_spearman, value: 0.8139}]
---
# modernbert-embed-base-yat
`nomic-ai/modernbert-embed-base` with **every GeGLU feed-forward block replaced by a
sigmoid-gated Yat-kernel MLP** (an alignment / inverse-distance kernel primitive, no GELU/GeGLU).
Only the 22 feed-forward blocks are changed; attention, embeddings and norms are the base model's.
The Yat FFNs are fit by **end-to-end last-layer distillation**: freeze everything but the FFNs and
train them so the model's *final* hidden state matches the frozen GeGLU teacher (normalized MSE on
the last-layer hidden states + a cosine term on the mean-pooled embedding), one epoch over all-nli
sentences. Matching only the final representation — rather than imitating each GeGLU block pointwise,
which hits a function-class ceiling — lets the kernel layers reallocate computation and **recover full
teacher parity**.
## Evaluation (MTEB STS, cosine Spearman)
| Task | base `modernbert-embed-base` (GeGLU) | **this model** (Yat) |
|---|---|---|
| STSBenchmark | 0.835 | 0.815 |
| STS12 | 0.676 | **0.718** |
| STS16 | 0.835 | 0.814 |
| **average** | **0.782** | **0.782** |
The Yat-kernel swap reaches the same average STS as the GeGLU base (and is ahead of it on STS12).
Scores are reproduced after a Hub round-trip (`trust_remote_code=True`). The bundled custom
architecture (`modeling_yatmodernbert.py`) is loaded automatically.
## Usage
```python
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("mlnomad/modernbert-embed-base-yat", trust_remote_code=True)
emb = m.encode(["A man is eating food.", "A man is eating a meal."])
```
## The Yat FFN
```
g(x) = ( softplus(a) * (x·W + b)^2 / (||x - W||^2 + exp(le)) * sigmoid(gate(x)) ) @ A + c
```
a non-negative rational (alignment-over-distance) kernel feature with a sigmoid gate, replacing the
GeGLU map `d -> 4d -> d` at the same hidden width.
## Notes
- Reaches teacher parity by distillation; this is the ceiling of distillation (matches, does not beat
the base). A light contrastive fine-tune of the FFNs alone does **not** exceed the base.
- Quantizes losslessly to int8 (weight-only PTQ leaves STS unchanged); below 8 bits the rational
kernel is more sensitive than GeGLU under naive uniform quantization.
Part of the ⵟ-kernel research project (kernel-native replacements for transformer FFNs).
|