Upload L6_bottom with MTEB results

README.md

@@ -4,18 +4,17 @@ tags:
 - sentence-transformers
 - intent-classification
 - multilingual
-- distillation
 - layer-pruning
+- vocab-pruning
 library_name: sentence-transformers
 pipeline_tag: sentence-similarity
 license: apache-2.0
 ---
 
-# Intent Classifier Student: L6_bottom
+# L6_bottom
 
-Distilled multilingual sentence encoder for intent classification (Action / Recall / Other).
-
-Created by **layer pruning** from `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`.
+Lightweight multilingual sentence encoder optimized for intent classification.
+Created from `paraphrase-multilingual-MiniLM-L12-v2` via layer pruning + corpus-based vocabulary pruning.
 
 ## Model Details
 
@@ -24,76 +23,86 @@ Created by **layer pruning** from `sentence-transformers/paraphrase-multilingual
 | Teacher | paraphrase-multilingual-MiniLM-L12-v2 |
 | Architecture | XLM-RoBERTa (pruned) |
 | Hidden dim | 384 |
-| Layers | 6 (from 12) |
+| Layers | 6 / 12 |
 | Layer indices | [0, 1, 2, 3, 4, 5] |
 | Strategy | 6 layers, bottom half (syntactic-focused) |
-| Est. params | 106,825,344 |
-| Est. FP32 | 407.5MB |
-| Est. INT8 | 101.9MB |
-| Est. INT8 + vocab pruned | 30.5MB |
+| Vocab size | ~38,330 (pruned from 250K) |
+| Parameters | 26,184,576 |
+| Safetensors size | 98.1MB |
+| Distilled | No |
 
 ## Supported Languages (18)
 
 ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl
 
-## Intended Use
-
-This is a **student encoder** designed to be used as the backbone for a lightweight
-3-class intent classifier (Action / Recall / Other) in multilingual dialogue systems.
-
-- **Action**: User requests an action (book, order, change settings, etc.)
-- **Recall**: User asks about past events or stored information
-- **Other**: Greetings, chitchat, emotions, etc.
-
-## Usage
+## Quick Start
 
 ```python
 from sentence_transformers import SentenceTransformer
 
 model = SentenceTransformer("L6_bottom")
-embeddings = model.encode(["예약 좀 해줘", "지난번 주문 뭐였지?", "안녕하세요"])
-print(embeddings.shape)  # (3, 384)
+
+sentences = [
+    "예약 좀 해줘",           # Korean
+    "What did I order?",     # English
+    "今日はいい天気ですね",    # Japanese
+    "Reserva una mesa",      # Spanish
+]
+
+embeddings = model.encode(sentences)
+print(embeddings.shape)  # (4, 384)
 ```
 
-## MTEB Results
+## MTEB Evaluation Results
+
+**Overall Average: 57.05%**
 
 ### MassiveIntentClassification
 
-**Average: 55.88%**
+**Average: 54.7%**
 
 | Language | Score |
 |----------|-------|
-| ar | 48.23% |
-| en | 60.82% |
-| es | 56.89% |
-| ko | 57.58% |
+| ar | 46.36% |
+| en | 59.84% |
+| es | 56.11% |
+| ko | 56.49% |
 
 ### MassiveScenarioClassification
 
-**Average: 60.75%**
+**Average: 59.39%**
 
 | Language | Score |
 |----------|-------|
-| ar | 53.04% |
-| en | 65.8% |
-| es | 60.99% |
-| ko | 63.19% |
+| ar | 50.55% |
+| en | 64.52% |
+| es | 60.31% |
+| ko | 62.19% |
+
+
+
+## Training
+
+This model was created via **layer pruning + vocabulary pruning**:
 
+1. **Teacher**: `paraphrase-multilingual-MiniLM-L12-v2` (12 layers, 384 hidden dim)
+2. **Layer selection**: `[0, 1, 2, 3, 4, 5]` - 6 layers, bottom half (syntactic-focused)
+3. **Vocab pruning**: 250K -> ~38K tokens (corpus-based filtering for 18 target languages)
+4. **No additional training** - weights are directly copied from the teacher
 
+A distilled version of this model is also available with improved performance.
 
-## Training / Distillation
 
-This model was created via **layer pruning** (no additional training):
-1. Load teacher: `paraphrase-multilingual-MiniLM-L12-v2` (12 layers, 384 hidden)
-2. Select layers: `[0, 1, 2, 3, 4, 5]`
-3. Copy embedding weights + selected layer weights
-4. Wrap with mean pooling for sentence embeddings
+## Compression Summary
 
-For deployment, vocabulary pruning (250K → ~55K tokens) and INT8 quantization
-are applied to meet the ≤50MB size constraint.
+| Stage | Vocab | Layers | Size |
+|-------|-------|--------|------|
+| Teacher (original) | 250,002 | 12 | ~480MB |
+| + Layer pruning | 250,002 | 6 | ~407MB |
+| + Vocab pruning | ~38,330 | 6 | ~98MB |
 
 ## Limitations
 
-- Layer pruning without fine-tuning may lose some quality vs. proper knowledge distillation
-- Vocabulary pruning limits the model to the target 18 languages
+- Vocabulary pruning restricts the model to the 18 target languages
 - Designed for short dialogue utterances, not long documents
+- Layer pruning may reduce performance on complex semantic tasks

config.json

@@ -21,5 +21,5 @@
   "transformers_version": "4.56.2",
   "type_vocab_size": 2,
   "use_cache": true,
-  "vocab_size": 250037
+  "vocab_size": 38330
 }

config_sentence_transformers.json

@@ -3,7 +3,7 @@
   "__version__": {
     "sentence_transformers": "5.3.0",
     "transformers": "4.56.2",
-    "pytorch": "2.10.0+cpu"
+    "pytorch": "2.10.0+cu128"
   },
   "prompts": {
     "query": "",

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d48bea38209b27ea02c4f79948b58ed600a6f46353aed9774b87f99b963b61ba
-size 428039432
+oid sha256:75aade5a2325bfa6346cc282b70cbad0525ffc5add0ef159448f2df61b1260e7
+size 102857288

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cad551d5600a84242d0973327029452a1e3672ba6313c2a3c3d69c4310e12719
-size 17082987
+oid sha256:0ab1d8ad18d647b10254a627ba87f4f8dac8aea96ca026510f5f883fe2e6532e
+size 2816831

tokenizer_config.json

@@ -32,7 +32,7 @@
       "single_word": false,
       "special": true
     },
-    "250001": {
+    "38329": {
       "content": "<mask>",
       "lstrip": true,
       "normalized": false,