gomyk/modernbert-student-modernbert_L4_uniform

modernbert_L4_uniform

Lightweight sentence encoder created from answerdotai/ModernBERT-base via layer pruning + vocabulary pruning.

Model Details

Property	Value
Teacher	answerdotai/ModernBERT-base
Architecture	ModernBERT (pruned)
Hidden dim	768
Layers	4 / 22
Layer indices	[0, 7, 14, 21]
Strategy	4 layers, evenly spaced from ModernBERT (22L)
Parameters	55,607,040
Model size (FP32)	137.7MB
Distilled	No

Architecture

==============================================================
  TEACHER: ModernBERT  →  STUDENT: 4L / 24,978 vocab
==============================================================

            TEACHER                        STUDENT          
  ───────────────────────────    ───────────────────────────

  ┌─────────────────────────┐    ┌─────────────────────────┐
  │   Input Tokens          │    │   Input Tokens          │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌────────────┴────────────┐    ┌────────────┴────────────┐
  │  Embeddings             │    │  Embeddings (pruned)    │
  │  vocab:  50,368         │    │  vocab:  24,978         │
  │  dim:  768              │    │  dim:  768              │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌─────────────────────────┐    ┌─────────────────────────┐
  │  Layer  0               │ ──►  │  Layer  0 ← L0         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  1               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  2               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  3               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  4               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  5               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  6               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  7               │ ──►  │  Layer  1 ← L7         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  8               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  9               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 10               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 11               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 12               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 13               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer 14               │ ──►  │  Layer  2 ← L14        │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer 15               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 16               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 17               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 18               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 19               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 20               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer 21               │ ──►  │  Layer  3 ← L21        │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌────────────┴────────────┐    ┌────────────┴────────────┐
  │  Mean Pooling           │    │  Mean Pooling           │
  │  → 768d embedding       │    │  → 768d embedding       │
  └─────────────────────────┘    └─────────────────────────┘

  Size: 495.8MB (FP32)           →  137.7MB (FP32)
  Params: 129,980,160        →  36,107,520
  Reduction: 72.2%
==============================================================

Quick Start

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("modernbert_L4_uniform", trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "안녕하세요",
    "Bonjour, comment allez-vous?",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 768)

MTEB Evaluation Results

Overall Average: 39.64%

Task Group	Average
Classification	46.21%
Clustering	27.69%
STS	44.42%

Classification

Task	Average	Details
AmazonCounterfactualClassification	62.15%	en: 65.7%, de: 62.22%, en-ext: 61.47%
Banking77Classification	45.76%	default: 45.76%
ImdbClassification	57.29%	default: 57.29%
MTOPDomainClassification	49.25%	es: 53.78%, en: 53.67%, de: 49.66%
MassiveIntentClassification	29.75%	zh-CN: 42.91%, ja: 38.38%, zh-TW: 37.92%
MassiveScenarioClassification	30.03%	zh-CN: 45.57%, zh-TW: 38.84%, en: 37.16%
ToxicConversationsClassification	55.82%	default: 55.82%
TweetSentimentExtractionClassification	39.6%	default: 39.6%

Clustering

Task	Average	Details
ArXivHierarchicalClusteringP2P	47.97%	default: 47.97%
ArXivHierarchicalClusteringS2S	45.05%	default: 45.05%
BiorxivClusteringP2P.v2	15.49%	default: 15.49%
MedrxivClusteringP2P.v2	23.89%	default: 23.89%
MedrxivClusteringS2S.v2	18.83%	default: 18.83%
StackExchangeClustering.v2	32.31%	default: 32.31%
StackExchangeClusteringP2P.v2	29.14%	default: 29.14%
TwentyNewsgroupsClustering.v2	8.87%	default: 8.87%

STS

Task	Average	Details
BIOSSES	42.22%	default: 42.22%
SICK-R	54.53%	default: 54.53%
STS12	40.85%	default: 40.85%
STS13	46.53%	default: 46.53%
STS14	48.23%	default: 48.23%
STS15	61.72%	default: 61.72%
STS17	28.67%	es-es: 67.31%, en-en: 63.62%, ko-ko: 53.02%
STS22.v2	28.6%	zh: 60.37%, es: 54.2%, it: 51.29%
STSBenchmark	48.43%	default: 48.43%

Training

Created via layer pruning + vocabulary pruning (no additional training):

1. Teacher: answerdotai/ModernBERT-base (22 layers, 768d)
2. Layer selection: [0, 7, 14, 21] - 4 layers, evenly spaced from ModernBERT (22L)
3. Vocab pruning: Corpus-based filtering for target languages

Supported Languages (18)

ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl