mlx-community/bge-m3-mlx-4bit

BGE-M3 MLX (4-bit Quantized)

This is the BAAI/bge-m3 model converted to MLX format with 4-bit quantization for Apple Silicon.

Model Description

BGE-M3 is a versatile embedding model capable of:

• Dense retrieval
• Sparse retrieval
• Multi-vector (ColBERT) retrieval

This 4-bit quantized version offers the smallest footprint while maintaining reasonable quality.

Model Details

Property	Value
Architecture	XLM-RoBERTa
Precision	4-bit (affine quantization)
Embedding Dimension	1024
Max Sequence Length	8192
Model Size	~321 MB
Quantization Group Size	64
Languages	100+ languages

Size Comparison

Version	Size	Compression
FP16	1.1 GB	-
8-bit	592 MB	46%
6-bit	457 MB	58%
4-bit	321 MB	71%

Usage

With MLX

from mlx_embeddings.utils import load_model, load_tokenizer
import mlx.core as mx

model_path = "mlx-community/bge-m3-mlx-4bit"

# Load model and tokenizer
model = load_model(model_path)
tokenizer = load_tokenizer(model_path)

# Generate embeddings
text = "Hello, world!"
tokens = tokenizer.encode(text)
input_ids = mx.array([tokens])
output = model(input_ids)
embedding = output.last_hidden_state.mean(axis=1)  # Mean pooling

print(f"Embedding shape: {embedding.shape}")  # (1, 1024)

With oMLX

# Start oMLX server
omlx serve --model-dir /path/to/models

# Get embeddings via API
curl http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "bge-m3-mlx-4bit", "input": "Your text here"}'

Quantization Details

• Method: Affine quantization
• Bits per weight: 4
• Group size: 64
• Source: Converted from MLX FP16 version

Performance

Tested on macOS with Apple Silicon:

• Successfully generates 1024-dimensional embeddings
• Supports multilingual text
• Lowest memory footprint among all variants

Recommended Use Cases

• Memory-constrained environments
• High-throughput embedding generation
• Applications where some quality loss is acceptable for reduced memory

License

This model inherits the MIT license from the original BAAI/bge-m3 model.

Citation

@article{bge_m3,
  title={BGE M3-Embedding: Accurate, Efficient and Versatile Text Embedding},
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Zhang, Zheng},
  journal={arXiv preprint arXiv:2402.03216},
  year={2024}
}

Disclaimer

This is an unofficial MLX conversion of the BAAI/bge-m3 model. For the original model and official implementations, please refer to BAAI/bge-m3.