README.md

---
language:
- pt
license: mit
tags:
- mteb
- baseline
- random-baseline
- portuguese
- brazilian-portuguese
pipeline_tag: sentence-similarity
---

# MTEB(por) — Random Baseline Encoder

> ⚠️ **This is NOT a trained model.** It is the **chance-level floor** reference for the
> [MTEB(por, v2)](https://huggingface.co/MTEB-BR) Brazilian-Portuguese embedding benchmark.

It maps each input text to a deterministic, L2-normalized **random** vector (seeded by a hash of
the text). It carries **zero semantic signal** — two textually-different but semantically-similar
sentences get unrelated vectors — so it scores at chance level on every task family (STS,
retrieval, classification, clustering, reranking, regression).

## Why a random baseline?

1. **Interpretability** — it anchors every number. Is `0.30` on a retrieval task *good* or
   near-random? Only the floor answers that.
2. **Task discrimination** — if a real model scores near the floor on a task, that task does not
   discriminate. A concrete empirical sanity check.
3. **Convention** — mirrors `mteb/baseline-random-encoder` from the upstream MTEB leaderboard.

## Design

- Each text `t` → `rng = numpy.random.default_rng(sha256("42|" + t))` → `v = rng.standard_normal(768)` → `v / ‖v‖`.
- **Deterministic per text** (fully reproducible), **dim 768**, **seed 42**.
- No weights, no GPU, no training.

## Reproduce

```python
import hashlib
import numpy as np

DIM, SEED = 768, 42

def encode(texts: list[str]) -> np.ndarray:
    """Deterministic per-text L2-normalized random vectors (chance-level floor)."""
    out = np.empty((len(texts), DIM), dtype=np.float32)
    for i, t in enumerate(texts):
        h = int(hashlib.sha256((str(SEED) + "|" + (t or "")).encode()).hexdigest(), 16) % (2**32)
        v = np.random.default_rng(h).standard_normal(DIM).astype(np.float32)
        out[i] = v / (np.linalg.norm(v) + 1e-9)
    return out
```

The full evaluation script (`run_random_baseline.py`, using the same pinned-revision MTEB(por)
tasks as the benchmarked models) is included in this repo.

## Floor scores — MTEB(por, v2)

**Retrieval (nDCG@10)**

| Task | Floor |
|---|---|
| MedPTRetrieval | 0.0083 |
| FaQuADIR | 0.0235 |
| Quati | 0.0 |
| FaqBacenRetrieval | 0.0027 |
| JurisTCU | 0.0 |
| BRTaxQAR | 0.0129 |

**Reranking (MAP)**

| Task | Floor |
|---|---|
| QuatiReranking | 0.1804 |
| JurisTCUReranking | 0.1434 |
| PortuLexRRIP | 0.1415 |

**STS (Spearman)**

| Task | Floor |
|---|---|
| AssinSTS | 0.005 |
| Assin2STS | -0.0288 |

**Pair classification (AP)**

| Task | Floor |
|---|---|
| AssinRTE | 0.2328 |
| InferBR | 0.3556 |

**Classification (acc/AP)**

| Task | Floor |
|---|---|
| HateBR | 0.5016 |
| ToxSynPT | 0.495 |
| FactckBrClassification | 0.322 |
| OlidBrMultilabelClassification | 0.2035 |
| BrighterEmotionMultilabelClassification | 0.2027 |

**Clustering (V-measure)**

| Task | Floor |
|---|---|
| MedPTClustering | 0.5289 |
| WikipediaPTCategoriesClusteringP2P | 0.3248 |
| JurisTCUClusteringP2P | 0.1225 |
| SciELOClusteringP2P | 0.0859 |
| StackoverflowPtClustering | 0.3353 |
| CamaraProposicoesClustering | 0.4912 |

**Regression (Spearman)**

| Task | Floor |
|---|---|
| BrighterEmotionIntensityRegression | 0.0223 |
| EnemEssayRegression | -0.0783 |
| NarrativeEssaysBRRegression | -0.0526 |

*Floor is non-zero for clustering (the V-measure of a random partition is not 0) and for
classification (chance ≈ 1/num-classes); real models score well above it on every task.*

## Citation

Part of the **MTEB(por)** benchmark by the `MTEB-BR` project. The floor is computed with the
identical pinned-SHA tasks used for every benchmarked model.