File size: 5,894 Bytes
7d44eb7 bf92763 1629152 bf92763 1629152 bf92763 1629152 bf92763 1629152 bf92763 1629152 bf92763 7d44eb7 bf92763 1629152 7d44eb7 bf92763 7d44eb7 1629152 bf92763 7d44eb7 1629152 7d44eb7 1629152 7d44eb7 bf92763 7d44eb7 1629152 7d44eb7 bf92763 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 | ---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: feature-extraction
base_model: sentence-transformers/all-MiniLM-L6-v2
tags:
- sentence-transformers
- embeddings
- text-embeddings
- semantic-search
- information-retrieval
- onnx
- int8
- minilm
- e5
---
# TinyE5-L6-384
**TinyE5-L6-384** is a compact **384-dimensional text embedding model** built from
`sentence-transformers/all-MiniLM-L6-v2` and fine-tuned for semantic search and
information retrieval using E5-style query/passage prefixes.
The model uses **mean pooling + L2 normalization** and is designed for applications
where model size, CPU latency, and deployment efficiency matter.
## Highlights
- **6 Transformer layers**
- **384-dimensional embeddings**
- **22.7M parameters**
- Only the **last 2 Transformer layers** were fine-tuned
- **Mean pooling**
- **L2-normalized embeddings**
- E5-style `query:` and `passage:` prefixes
- FP32 and dynamically quantized **INT8 ONNX** deployment
- INT8 ONNX size: **21.8 MB**
- Measured INT8 CPU throughput: **377.7 texts/s**
---
## Model Architecture
| Property | Value |
|---|---|
| Base model | `sentence-transformers/all-MiniLM-L6-v2` |
| Transformer layers | 6 |
| Hidden / embedding size | 384 |
| Total parameters | 22,713,216 |
| Fine-tuned layers | Last 2 Transformer layers |
| Trainable parameters | 3,548,928 |
| Trainable percentage | 15.62% |
| Pooling | Mean pooling |
| Normalization | L2 normalization |
| Query prefix | `query: ` |
| Passage prefix | `passage: ` |
| Training max length | 128 tokens |
---
## Benchmark Results
### Quality
| Model | Size | STS Spearman ↑ | SciFact Recall@10 ↑ | SciFact nDCG@10 ↑ |
| ------------------------------- | ----------: | -------------: | ------------------: | ----------------: |
| **TinyE5-L6-384 (Safetensors)** | 86.7 MB | 0.8138 | 0.7342 | 0.5897 |
| **TinyE5-L6-384 (FP32 ONNX)** | 86.2 MB | 0.8138 | 0.7342 | 0.5897 |
| **TinyE5-L6-384 (INT8 ONNX)** | **21.8 MB** | 0.8058 | 0.7599 | 0.6020 |
### CPU Inference Performance
| Model | Precision | Size | Latency ↓ | Throughput ↑ | Session RAM ↓ |
| ---------------------- | --------- | ----------: | ---------------: | -----------------: | ------------: |
| TinyE5-L6-384 ONNX | FP32 | 86.2 MB | 2.35 ms/text | 424.7 texts/s | 90.4 MB |
| **TinyE5-L6-384 ONNX** | **INT8** | **21.8 MB** | **0.77 ms/text** | **1299.4 texts/s** | **11.2 MB** |
### INT8 vs FP32 ONNX
| Metric | FP32 | INT8 | Change |
| ----------------- | ------------: | -----------------: | -----------------: |
| Model size | 86.2 MB | **21.8 MB** | **~74.7% smaller** |
| CPU latency | 2.35 ms/text | **0.77 ms/text** | **~67.2% lower** |
| CPU throughput | 424.7 texts/s | **1299.4 texts/s** | **~205.9% higher** |
| Session RAM | 90.4 MB | **11.2 MB** | **~87.6% lower** |
| STS Spearman | **0.8138** | 0.8058 | -0.0080 |
| SciFact Recall@10 | 0.7342 | **0.7599** | **+0.0257** |
| SciFact nDCG@10 | 0.5897 | **0.6020** | **+0.0123** |
### End-to-End Benchmark Time
| Model | STS Time ↓ | SciFact Time ↓ |
| ------------------------------- | ---------: | -------------: |
| **TinyE5-L6-384 (Safetensors)** | 0.72s | 3.71s |
| TinyE5-L6-384 (FP32 ONNX) | 8.82s | 265.58s |
| TinyE5-L6-384 (INT8 ONNX) | 5.46s | 179.11s |
> **Summary:** INT8 quantization reduces TinyE5-L6-384 from **86.2 MB to 21.8 MB (~74.7% smaller)**, lowers CPU inference latency from **2.35 ms/text to 0.77 ms/text (~67.2% lower)**, and increases CPU throughput from **424.7 to 1299.4 texts/s (~3.06× throughput)**. Session RAM also drops from **90.4 MB to 11.2 MB (~87.6% lower)**. STS Spearman decreases slightly from **0.8138 to 0.8058**, while SciFact Recall@10 improves from **0.7342 to 0.7599** and nDCG@10 improves from **0.5897 to 0.6020**. The all-MiniLM-L6-v2 baseline still achieves the highest overall quality, with **0.8194 STS Spearman, 0.7923 Recall@10, and 0.6494 nDCG@10**.
## Docker Example
### INT8 ONNX
```yaml
services:
embedding-server:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
ports:
- "80:80"
volumes:
- ./data:/data
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- int8-onnx
- --pooling
- mean
```
### Available Variants
All variants are published under the same Hugging Face repository:
```text
GrowBitLabs/tinye5
```
| Variant | Revision | Model |
| ---------------- | ----------- | ------------------- |
| Safetensors FP32 | `main` | `model.safetensors` |
| ONNX FP32 | `fp32-onnx` | `onnx/model.onnx` |
| ONNX INT8 | `int8-onnx` | `onnx/model.onnx` |
### Safetensors
```yaml
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- main
- --pooling
- mean
```
### FP32 ONNX
```yaml
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- fp32-onnx
- --pooling
- mean
```
### INT8 ONNX
```yaml
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- int8-onnx
- --pooling
- mean
```
For CPU deployment, the **INT8 ONNX** revision is recommended when model size and inference throughput are the priority.
## Attribution
Developed by **[GrowBit Labs](https://growbitlabs.com)**.
|