TinyE5-L6-384

TinyE5-L6-384 is a compact 384-dimensional text embedding model built from sentence-transformers/all-MiniLM-L6-v2 and fine-tuned for semantic search and information retrieval using E5-style query/passage prefixes.

The model uses mean pooling + L2 normalization and is designed for applications where model size, CPU latency, and deployment efficiency matter.

Highlights

• 6 Transformer layers
• 384-dimensional embeddings
• 22.7M parameters
• Only the last 2 Transformer layers were fine-tuned
• Mean pooling
• L2-normalized embeddings
• E5-style query: and passage: prefixes
• FP32 and dynamically quantized INT8 ONNX deployment
• INT8 ONNX size: 21.8 MB
• Measured INT8 CPU throughput: 377.7 texts/s

---

Model Architecture

Property	Value
Base model	sentence-transformers/all-MiniLM-L6-v2
Transformer layers	6
Hidden / embedding size	384
Total parameters	22,713,216
Fine-tuned layers	Last 2 Transformer layers
Trainable parameters	3,548,928
Trainable percentage	15.62%
Pooling	Mean pooling
Normalization	L2 normalization
Query prefix	query:
Passage prefix	passage:
Training max length	128 tokens

---

Benchmark Results

Quality

Model	Size	STS Spearman ↑	SciFact Recall@10 ↑	SciFact nDCG@10 ↑
TinyE5-L6-384 (Safetensors)	86.7 MB	0.8138	0.7342	0.5897
TinyE5-L6-384 (FP32 ONNX)	86.2 MB	0.8138	0.7342	0.5897
TinyE5-L6-384 (INT8 ONNX)	21.8 MB	0.8058	0.7599	0.6020

CPU Inference Performance

Model	Precision	Size	Latency ↓	Throughput ↑	Session RAM ↓
TinyE5-L6-384 ONNX	FP32	86.2 MB	2.35 ms/text	424.7 texts/s	90.4 MB
TinyE5-L6-384 ONNX	INT8	21.8 MB	0.77 ms/text	1299.4 texts/s	11.2 MB

INT8 vs FP32 ONNX

Metric	FP32	INT8	Change
Model size	86.2 MB	21.8 MB	~74.7% smaller
CPU latency	2.35 ms/text	0.77 ms/text	~67.2% lower
CPU throughput	424.7 texts/s	1299.4 texts/s	~205.9% higher
Session RAM	90.4 MB	11.2 MB	~87.6% lower
STS Spearman	0.8138	0.8058	-0.0080
SciFact Recall@10	0.7342	0.7599	+0.0257
SciFact nDCG@10	0.5897	0.6020	+0.0123

End-to-End Benchmark Time

Model	STS Time ↓	SciFact Time ↓
TinyE5-L6-384 (Safetensors)	0.72s	3.71s
TinyE5-L6-384 (FP32 ONNX)	8.82s	265.58s
TinyE5-L6-384 (INT8 ONNX)	5.46s	179.11s

Summary: INT8 quantization reduces TinyE5-L6-384 from 86.2 MB to 21.8 MB (~74.7% smaller), lowers CPU inference latency from 2.35 ms/text to 0.77 ms/text (~67.2% lower), and increases CPU throughput from 424.7 to 1299.4 texts/s (~3.06× throughput). Session RAM also drops from 90.4 MB to 11.2 MB (~87.6% lower). STS Spearman decreases slightly from 0.8138 to 0.8058, while SciFact Recall@10 improves from 0.7342 to 0.7599 and nDCG@10 improves from 0.5897 to 0.6020. The all-MiniLM-L6-v2 baseline still achieves the highest overall quality, with 0.8194 STS Spearman, 0.7923 Recall@10, and 0.6494 nDCG@10.

Docker Example

INT8 ONNX

services:
  embedding-server:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
    ports:
      - "80:80"
    volumes:
      - ./data:/data
    command:
      - --model-id
      - GrowBitLabs/tinye5
      - --revision
      - int8-onnx
      - --pooling
      - mean

Available Variants

All variants are published under the same Hugging Face repository:

GrowBitLabs/tinye5

Variant	Revision	Model
Safetensors FP32	main	model.safetensors
ONNX FP32	fp32-onnx	onnx/model.onnx
ONNX INT8	int8-onnx	onnx/model.onnx

Safetensors

command:
  - --model-id
  - GrowBitLabs/tinye5
  - --revision
  - main
  - --pooling
  - mean

FP32 ONNX

command:
  - --model-id
  - GrowBitLabs/tinye5
  - --revision
  - fp32-onnx
  - --pooling
  - mean

INT8 ONNX

command:
  - --model-id
  - GrowBitLabs/tinye5
  - --revision
  - int8-onnx
  - --pooling
  - mean

For CPU deployment, the INT8 ONNX revision is recommended when model size and inference throughput are the priority.

Attribution

Developed by GrowBit Labs.