---
language:
  - en
license: apache-2.0
library_name: transformers
pipeline_tag: feature-extraction
base_model: sentence-transformers/all-MiniLM-L6-v2
tags:
  - sentence-transformers
  - embeddings
  - text-embeddings
  - semantic-search
  - information-retrieval
  - onnx
  - int8
  - minilm
  - e5
---

# TinyE5-L6-384

**TinyE5-L6-384** is a compact **384-dimensional text embedding model** built from
`sentence-transformers/all-MiniLM-L6-v2` and fine-tuned for semantic search and
information retrieval using E5-style query/passage prefixes.

The model uses **mean pooling + L2 normalization** and is designed for applications
where model size, CPU latency, and deployment efficiency matter.

## Highlights

- **6 Transformer layers**
- **384-dimensional embeddings**
- **22.7M parameters**
- Only the **last 2 Transformer layers** were fine-tuned
- **Mean pooling**
- **L2-normalized embeddings**
- E5-style `query:` and `passage:` prefixes
- FP32 and dynamically quantized **INT8 ONNX** deployment
- INT8 ONNX size: **21.8 MB**
- Measured INT8 CPU throughput: **377.7 texts/s**

---

## Model Architecture

| Property | Value |
|---|---|
| Base model | `sentence-transformers/all-MiniLM-L6-v2` |
| Transformer layers | 6 |
| Hidden / embedding size | 384 |
| Total parameters | 22,713,216 |
| Fine-tuned layers | Last 2 Transformer layers |
| Trainable parameters | 3,548,928 |
| Trainable percentage | 15.62% |
| Pooling | Mean pooling |
| Normalization | L2 normalization |
| Query prefix | `query: ` |
| Passage prefix | `passage: ` |
| Training max length | 128 tokens |

---

## Benchmark Results

### Quality

| Model                           |        Size | STS Spearman ↑ | SciFact Recall@10 ↑ | SciFact nDCG@10 ↑ |
| ------------------------------- | ----------: | -------------: | ------------------: | ----------------: |
| **TinyE5-L6-384 (Safetensors)** |     86.7 MB |         0.8138 |              0.7342 |            0.5897 |
| **TinyE5-L6-384 (FP32 ONNX)**   |     86.2 MB |         0.8138 |              0.7342 |            0.5897 |
| **TinyE5-L6-384 (INT8 ONNX)**   | **21.8 MB** |         0.8058 |              0.7599 |            0.6020 |

### CPU Inference Performance

| Model                  | Precision |        Size |        Latency ↓ |       Throughput ↑ | Session RAM ↓ |
| ---------------------- | --------- | ----------: | ---------------: | -----------------: | ------------: |
| TinyE5-L6-384 ONNX     | FP32      |     86.2 MB |     2.35 ms/text |      424.7 texts/s |       90.4 MB |
| **TinyE5-L6-384 ONNX** | **INT8**  | **21.8 MB** | **0.77 ms/text** | **1299.4 texts/s** |   **11.2 MB** |

### INT8 vs FP32 ONNX

| Metric            |          FP32 |               INT8 |             Change |
| ----------------- | ------------: | -----------------: | -----------------: |
| Model size        |       86.2 MB |        **21.8 MB** | **~74.7% smaller** |
| CPU latency       |  2.35 ms/text |   **0.77 ms/text** |   **~67.2% lower** |
| CPU throughput    | 424.7 texts/s | **1299.4 texts/s** | **~205.9% higher** |
| Session RAM       |       90.4 MB |        **11.2 MB** |   **~87.6% lower** |
| STS Spearman      |    **0.8138** |             0.8058 |            -0.0080 |
| SciFact Recall@10 |        0.7342 |         **0.7599** |        **+0.0257** |
| SciFact nDCG@10   |        0.5897 |         **0.6020** |        **+0.0123** |

### End-to-End Benchmark Time

| Model                           | STS Time ↓ | SciFact Time ↓ |
| ------------------------------- | ---------: | -------------: |
| **TinyE5-L6-384 (Safetensors)** |      0.72s |          3.71s |
| TinyE5-L6-384 (FP32 ONNX)       |      8.82s |        265.58s |
| TinyE5-L6-384 (INT8 ONNX)       |      5.46s |        179.11s |

> **Summary:** INT8 quantization reduces TinyE5-L6-384 from **86.2 MB to 21.8 MB (~74.7% smaller)**, lowers CPU inference latency from **2.35 ms/text to 0.77 ms/text (~67.2% lower)**, and increases CPU throughput from **424.7 to 1299.4 texts/s (~3.06× throughput)**. Session RAM also drops from **90.4 MB to 11.2 MB (~87.6% lower)**. STS Spearman decreases slightly from **0.8138 to 0.8058**, while SciFact Recall@10 improves from **0.7342 to 0.7599** and nDCG@10 improves from **0.5897 to 0.6020**. The all-MiniLM-L6-v2 baseline still achieves the highest overall quality, with **0.8194 STS Spearman, 0.7923 Recall@10, and 0.6494 nDCG@10**.

## Docker Example

### INT8 ONNX

```yaml
services:
  embedding-server:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
    ports:
      - "80:80"
    volumes:
      - ./data:/data
    command:
      - --model-id
      - GrowBitLabs/tinye5
      - --revision
      - int8-onnx
      - --pooling
      - mean
```

### Available Variants

All variants are published under the same Hugging Face repository:

```text
GrowBitLabs/tinye5
```

| Variant          | Revision    | Model               |
| ---------------- | ----------- | ------------------- |
| Safetensors FP32 | `main`      | `model.safetensors` |
| ONNX FP32        | `fp32-onnx` | `onnx/model.onnx`   |
| ONNX INT8        | `int8-onnx` | `onnx/model.onnx`   |

### Safetensors

```yaml
command:
  - --model-id
  - GrowBitLabs/tinye5
  - --revision
  - main
  - --pooling
  - mean
```

### FP32 ONNX

```yaml
command:
  - --model-id
  - GrowBitLabs/tinye5
  - --revision
  - fp32-onnx
  - --pooling
  - mean
```

### INT8 ONNX

```yaml
command:
  - --model-id
  - GrowBitLabs/tinye5
  - --revision
  - int8-onnx
  - --pooling
  - mean
```

For CPU deployment, the **INT8 ONNX** revision is recommended when model size and inference throughput are the priority.

## Attribution

Developed by **[GrowBit Labs](https://growbitlabs.com)**.