File size: 3,681 Bytes
84ddb77 03c1a93 84ddb77 03c1a93 84ddb77 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | ---
language:
- pt
license: mit
tags:
- mteb
- baseline
- random-baseline
- portuguese
- brazilian-portuguese
pipeline_tag: sentence-similarity
---
# MTEB(por) — Random Baseline Encoder
> ⚠️ **This is NOT a trained model.** It is the **chance-level floor** reference for the
> [MTEB(por, v2)](https://huggingface.co/MTEB-BR) Brazilian-Portuguese embedding benchmark.
It maps each input text to a deterministic, L2-normalized **random** vector (seeded by a hash of
the text). It carries **zero semantic signal** — two textually-different but semantically-similar
sentences get unrelated vectors — so it scores at chance level on every task family (STS,
retrieval, classification, clustering, reranking, regression).
## Why a random baseline?
1. **Interpretability** — it anchors every number. Is `0.30` on a retrieval task *good* or
near-random? Only the floor answers that.
2. **Task discrimination** — if a real model scores near the floor on a task, that task does not
discriminate. A concrete empirical sanity check.
3. **Convention** — mirrors `mteb/baseline-random-encoder` from the upstream MTEB leaderboard.
## Design
- Each text `t` → `rng = numpy.random.default_rng(sha256("42|" + t))` → `v = rng.standard_normal(768)` → `v / ‖v‖`.
- **Deterministic per text** (fully reproducible), **dim 768**, **seed 42**.
- No weights, no GPU, no training.
## Reproduce
```python
import hashlib
import numpy as np
DIM, SEED = 768, 42
def encode(texts: list[str]) -> np.ndarray:
"""Deterministic per-text L2-normalized random vectors (chance-level floor)."""
out = np.empty((len(texts), DIM), dtype=np.float32)
for i, t in enumerate(texts):
h = int(hashlib.sha256((str(SEED) + "|" + (t or "")).encode()).hexdigest(), 16) % (2**32)
v = np.random.default_rng(h).standard_normal(DIM).astype(np.float32)
out[i] = v / (np.linalg.norm(v) + 1e-9)
return out
```
The full evaluation script (`run_random_baseline.py`, using the same pinned-revision MTEB(por)
tasks as the benchmarked models) is included in this repo.
## Floor scores — MTEB(por, v2)
**Retrieval (nDCG@10)**
| Task | Floor |
|---|---|
| MedPTRetrieval | 0.0083 |
| FaQuADIR | 0.0235 |
| Quati | 0.0 |
| FaqBacenRetrieval | 0.0027 |
| JurisTCU | 0.0 |
| BRTaxQAR | 0.0129 |
**Reranking (MAP)**
| Task | Floor |
|---|---|
| QuatiReranking | 0.1804 |
| JurisTCUReranking | 0.1434 |
| PortuLexRRIP | 0.1415 |
**STS (Spearman)**
| Task | Floor |
|---|---|
| AssinSTS | 0.005 |
| Assin2STS | -0.0288 |
**Pair classification (AP)**
| Task | Floor |
|---|---|
| AssinRTE | 0.2328 |
| InferBR | 0.3556 |
**Classification (acc/AP)**
| Task | Floor |
|---|---|
| HateBR | 0.5016 |
| ToxSynPT | 0.495 |
| FactckBrClassification | 0.322 |
| OlidBrMultilabelClassification | 0.2035 |
| BrighterEmotionMultilabelClassification | 0.2027 |
**Clustering (V-measure)**
| Task | Floor |
|---|---|
| MedPTClustering | 0.5289 |
| WikipediaPTCategoriesClusteringP2P | 0.3248 |
| JurisTCUClusteringP2P | 0.1225 |
| SciELOClusteringP2P | 0.0859 |
| StackoverflowPtClustering | 0.3353 |
| CamaraProposicoesClustering | 0.4912 |
**Regression (Spearman)**
| Task | Floor |
|---|---|
| BrighterEmotionIntensityRegression | 0.0223 |
| EnemEssayRegression | -0.0783 |
| NarrativeEssaysBRRegression | -0.0526 |
*Floor is non-zero for clustering (the V-measure of a random partition is not 0) and for
classification (chance ≈ 1/num-classes); real models score well above it on every task.*
## Citation
Part of the **MTEB(por)** benchmark by the `MTEB-BR` project. The floor is computed with the
identical pinned-SHA tasks used for every benchmarked model.
|