README.md

---
language:
- multilingual
tags:
- transformers
- pytorch
- sentence-transformers
- feature-extraction
- retrieval
- semantic-similarity
- classification
- clustering
- bitext-mining
- reranking
library_name: transformers
pipeline_tag: feature-extraction
license: mit
---

# global-embedder

**Multilingual Dense Text Embedding Family – Decoder‑Only, Instruction‑Aware, State‑of‑the‑Art**

---

## Model Overview

`global-embedder` is a family of multilingual text embedding models built on a decoder‑only Transformer architecture. Unlike traditional encoder‑based embedders, this model uses **last‑token pooling** and L2 normalization to produce dense, fixed‑size vector representations from long input sequences (up to 32,768 tokens). It is designed to serve as a universal backbone for retrieval, semantic similarity, clustering, classification, bitext mining, and reranking across over 80 languages.

The key differentiator lies in its **instruction‑aware training**: query‑side natural language instructions allow the model to adapt to diverse downstream tasks without fine‑tuning. By leveraging contrastive learning on a large‑scale mixture of multilingual datasets and knowledge distillation for the smaller variants, `global-embedder` achieves top performance on the Multilingual MTEB v2 benchmark as of its release date.

| Variant | Parameters | Embedding Dim | Max Tokens | MTEB v2 Score |
|---------|------------|---------------|------------|---------------|
| `global-embedder-270m` | 270M | 640 | 32,768 | 66.5 |
| `global-embedder-0.6b`  | 0.6B | 1,024 | 32,768 | 69.0 |
| `global-embedder-27b`   | 27B  | 5,376 | 32,768 | **74.3** |

---

## Intended Uses & Limitations

### Primary Use Cases
- **Information Retrieval** – Dense passage retrieval for web search, FAQ matching, and enterprise knowledge bases.
- **Semantic Similarity** – Computing pairwise sentence/document similarity for deduplication or clustering.
- **Classification & Clustering** – Producing input features for downstream classifiers or unsupervised grouping.
- **Bitext Mining** – Aligning sentences between different languages for translation or parallel corpus creation.
- **Reranking** – Improving first‑stage retrieval results by re‑scoring candidates.

### Out‑of‑Scope Scenarios
- Generative tasks (text summarisation, dialogue, translation) – this model is not designed for autoregressive generation.
- Real‑time latency‑sensitive applications without hardware acceleration – the 27B variant requires substantial compute.
- Tasks requiring explicit reasoning or fact‑grounded knowledge – embeddings capture semantic similarity, not factual correctness.

### Known Biases and Limitations
- The training data, while multilingual, may contain regional and cultural biases that can surface in the embeddings.
- Performance varies across languages; high‑resource languages (English, Chinese, Spanish) generally achieve better scores than low‑resource ones.
- The model does not inherently distinguish between factual and hypothetical statements – it treats all input text semantically.

---

## How to Use

### Option 1: Sentence Transformers (Recommended)

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("global-embedder-27b", model_kwargs={"dtype": "auto"})

# Use a pre‑configured instruction for web search queries
queries = [
    "how much protein should a female eat",
    "summit define"
]
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day...",
    "Definition of summit for English Language Learners: the highest point of a mountain..."
]

query_emb = model.encode(queries, prompt_name="web_search_query")
doc_emb = model.encode(documents)

scores = (query_emb @ doc_emb.T) * 100
print(scores.tolist())
```

For a custom instruction, pass the `prompt` parameter:

```python
query_emb = model.encode(
    queries,
    prompt="Instruct: Retrieve semantically similar passages\nQuery: "
)
```

### Option 2: Transformers (Raw Pooling)

```python
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

def last_token_pool(last_hidden_states, attention_mask):
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        seq_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), seq_lengths]

def instruct_query(task, query):
    return f'Instruct: {task}\nQuery: {query}'

task = "Given a web search query, retrieve relevant passages"
queries = [instruct_query(task, q) for q in ["how much protein should a female eat", "summit define"]]
documents = ["...", "..."]

tokenizer = AutoTokenizer.from_pretrained("global-embedder-27b")
model = AutoModel.from_pretrained("global-embedder-27b", dtype="auto").cuda()
model.eval()

batch = tokenizer(queries + documents, max_length=32768, padding=True, truncation=True, return_tensors="pt")
batch = {k: v.cuda() for k, v in batch.items()}

with torch.no_grad():
    outputs = model(**batch)
    embeddings = last_token_pool(outputs.last_hidden_state, batch["attention_mask"])
    embeddings = F.normalize(embeddings, p=2, dim=1)

scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
```

### Inference Widget (Feature Extraction)

You can test the model directly on the Hub using the inference widget (if hosted). Select the `feature-extraction` pipeline and input your text. Note that the widget returns raw hidden states – for a pooled embedding, please use the code above.

---

## Training Data

The model was trained on a **large‑scale proprietary mixture of multilingual datasets** encompassing:

- Web‑crawled corpora (general domain)
- Parallel bitext for cross‑lingual alignment
- Question‑answer pairs and search logs
- Public benchmark datasets (e.g., MS‑MARCO, Natural Questions, XQuAD, and others – anonymised)

All data underwent deduplication, language identification, and filtering for quality (e.g., removal of toxic or low‑perplexity content). The final training set covers more than 80 languages, with a balanced sampling strategy to mitigate the dominance of high‑resource languages.

---

## Training Procedure

- **Architecture**: Decoder‑only Transformer with causal attention, adapted for embedding via last‑token pooling.
- **Objective**: Contrastive learning with in‑batch negatives and hard‑negative mining. Instruction‑based query formatting was applied during training.
- **Optimizer**: AdamW with weight decay (`[WEIGHT_DECAY]`) and linear warmup.
- **Hyperparameters** (varies by size – representative for 27B):
  - Batch size: `[BATCH_SIZE]` (global with gradient accumulation)
  - Learning rate: `[LEARNING_RATE]`
  - Epochs: `[EPOCHS]`
  - Precision: mixed‑precision (BF16)
- **Hardware**: Distributed training across `[NUMBER_OF_GPUS]` NVIDIA A100 (80GB) GPUs.
- **Knowledge Distillation** (for 270M and 0.6B variants): Teacher embeddings from the larger 27B model were used to augment the contrastive loss.

---

## Evaluation Results

Performance measured on the **Multilingual MTEB v2** benchmark, which covers retrieval, clustering, classification, similarity, and reranking tasks across 50+ languages. The reported score is the **average across all tasks** (macro‑average).

| Variant | Average MTEB v2 Score |
|---------|------------------------|
| `global-embedder-270m` | 66.5 |
| `global-embedder-0.6b`  | 69.0 |
| `global-embedder-27b`   | **74.3** |

Detailed per‑task breakdowns are available in the evaluation suite (`[EVALUATION_SUITE_LINK]`). Reproduced scores may vary by ±0.2 due to software versions.

---

## Environmental Impact

Training the 27B variant required approximately `[TRAINING_HOURS]` hours on `[NUMBER_OF_GPUS]` A100 GPUs. Estimated CO₂ emissions (using the Machine Learning Impact calculator) are `[CO2_EMISSIONS]` kg CO₂ equivalent. We are committed to reducing future footprints via model distillation and efficient scaling.

---

## Bias, Risks, and Limitations

- **Geographic and Cultural Bias**: The training data over‑represents Western and East Asian internet content, which may skew embeddings for underrepresented regions.
- **Stereotypical Associations**: As with all large language models, the embeddings may encode societal stereotypes present in the training corpora.
- **Misuse Potential**: The model could be used to amplify surveillance or profiling if applied to sensitive personal data without proper safeguards.
- **Adversarial Robustness**: The model has not been extensively tested against adversarial perturbations; embeddings might be vulnerable to deliberate input manipulation.

**Mitigations**: We recommend deploying the model with a bias‑audit layer for high‑stakes applications, and always combining with human‑in‑the‑loop oversight. Users should not rely solely on model outputs for critical decisions.