Publish TinyE5-L6-384 Safetensors

Browse files

Files changed (8) hide show

README.md +100 -3
config.json +25 -0
embedding_config.json +11 -0
model.safetensors +3 -0
special_tokens_map.json +37 -0
tokenizer.json +0 -0
tokenizer_config.json +65 -0
vocab.txt +0 -0

README.md CHANGED Viewed

@@ -1,3 +1,100 @@
----
-license: apache-2.0
----

+# TinyE5-L6-384
+**TinyE5-L6-384** is a compact **384-dimensional text embedding model** built from
+`sentence-transformers/all-MiniLM-L6-v2` and fine-tuned for semantic search and
+information retrieval using E5-style query/passage prefixes.
+The model uses **mean pooling + L2 normalization** and is designed for applications
+where model size, CPU latency, and deployment efficiency matter.
+## Highlights
+- **6 Transformer layers**
+- **384-dimensional embeddings**
+- **22.7M parameters**
+- Only the **last 2 Transformer layers** were fine-tuned
+- **Mean pooling**
+- **L2-normalized embeddings**
+- E5-style `query:` and `passage:` prefixes
+- FP32 and dynamically quantized **INT8 ONNX** deployment
+- INT8 ONNX size: **21.8 MB**
+- Measured INT8 CPU throughput: **377.7 texts/s**
+---
+## Model Architecture
+| Property | Value |
+|---|---|
+| Base model | `sentence-transformers/all-MiniLM-L6-v2` |
+| Transformer layers | 6 |
+| Hidden / embedding size | 384 |
+| Total parameters | 22,713,216 |
+| Fine-tuned layers | Last 2 Transformer layers |
+| Trainable parameters | 3,548,928 |
+| Trainable percentage | 15.62% |
+| Pooling | Mean pooling |
+| Normalization | L2 normalization |
+| Query prefix | `query: ` |
+| Passage prefix | `passage: ` |
+| Training max length | 128 tokens |
+---
+## Benchmark Results
+### Quality
+| Model                           |        Size | STS Spearman ↑ | SciFact Recall@10 ↑ | SciFact nDCG@10 ↑ |
+| ------------------------------- | ----------: | -------------: | ------------------: | ----------------: |
+| **TinyE5-L6-384 (Safetensors)** |     86.7 MB |         0.8138 |              0.7148 |            0.5701 |
+| **TinyE5-L6-384 (FP32 ONNX)**   |     86.2 MB |         0.8138 |              0.7148 |            0.5701 |
+| **TinyE5-L6-384 (INT8 ONNX)**   | **21.8 MB** |         0.8070 |              0.7259 |            0.5791 |
+| E5-small-v2                     |      133 MB |     **0.8574** |          **0.8093** |        **0.6797** |
+### CPU Inference Performance
+| Model                  | Precision |        Size |        Latency ↓ |      Throughput ↑ |
+| ---------------------- | --------- | ----------: | ---------------: | ----------------: |
+| TinyE5-L6-384 ONNX     | FP32      |     86.2 MB |     3.65 ms/text |     274.3 texts/s |
+| **TinyE5-L6-384 ONNX** | **INT8**  | **21.8 MB** | **2.65 ms/text** | **377.7 texts/s** |
+### INT8 vs FP32 ONNX
+| Metric            |          FP32 |              INT8 |           Change |
+| ----------------- | ------------: | ----------------: | ---------------: |
+| Model size        |       86.2 MB |       **21.8 MB** | **~75% smaller** |
+| CPU latency       |  3.65 ms/text |  **2.65 ms/text** |   **~27% lower** |
+| CPU throughput    | 274.3 texts/s | **377.7 texts/s** |  **~38% higher** |
+| STS Spearman      |    **0.8138** |            0.8070 |          -0.0068 |
+| SciFact Recall@10 |        0.7148 |        **0.7259** |          +0.0111 |
+| SciFact nDCG@10   |        0.5701 |        **0.5791** |          +0.0090 |
+> **Summary:** INT8 quantization reduces TinyE5-L6-384 to just **21.8 MB** while increasing CPU throughput to approximately **378 texts/s**, with only a small change in semantic similarity performance and no degradation on the tested SciFact retrieval benchmark.
+## Docker Example
+```
+services:
+  embedding-server:
+    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
+    ports:
+      - "80:80"
+    volumes:
+      - ./data:/data
+    command:
+      - --model-id
+      - GrowBitLabs/TinyE5-L6-384-INT8
+      - --pooling
+      - mean
+```
+### variants:
+```
+GrowBitLabs/TinyE5-L6-384
+GrowBitLabs/TinyE5-L6-384-ONNX
+GrowBitLabs/TinyE5-L6-384-INT8
+```
+## Attribution
+Developed by **[GrowBit Labs](https://growbitlabs.com)**.

config.json ADDED Viewed

	@@ -0,0 +1,25 @@

+{
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

embedding_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "name": "TinyE5-L6-384",
+  "embedding_dimension": 384,
+  "pooling": "mean",
+  "normalize": true,
+  "query_prefix": "query: ",
+  "passage_prefix": "passage: ",
+  "base_model": "sentence-transformers/all-MiniLM-L6-v2",
+  "training_dataset": "sentence-transformers/msmarco-bm25/triplet",
+  "train_last_n_layers": 2
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1230da4b753a3712e7811e9f1b9068f35097569cd1b0987960a44a14c91b52fc
+size 90864192

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,37 @@

+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,65 @@

+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

vocab.txt ADDED Viewed

The diff for this file is too large to render. See raw diff