multilingual-e5-base-pms-32768

This model is a 60.0% smaller version of intfloat/multilingual-e5-base optimized for 32768 language via vocabulary pruning.

Total vocabulary size: 32768 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming

Language Distribution

• pms: 32768 tokens

This pruned model should perform similarly to the original model for 32768 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.

Usage

You can use this model with the Transformers library:

from transformers import AutoModel, AutoTokenizer

model_name = "Lumberjackk/multilingual-e5-base-pms-32768"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Model Statistics

• Original model size: 278.0M parameters
• Pruned model size: 111.2M parameters
• Size reduction: 60.0%
• Vocabulary reduction: 86.9%