Test datasets

README.md

@@ -388,6 +388,24 @@ Para fine-tuning, `scripts/train_place_retriever.py` acepta JSONL con `query`,
 `positive` y `hard_negatives`. El artefacto resultante se configura mediante
 `PLACES_EMBEDDING_MODEL`; no se incluye un modelo ficticio preentrenado en el repo.
 
+La evaluación del retriever debe usar el corpus global, no cuatro candidatos aislados
+por fila. Validation incluye 60 consultas y el test sintético 80 consultas, cada uno
+sobre 20 documentos propios y no vistos, con lenguaje
+coloquial, errores ortográficos, necesidades implícitas y frases contrastivas. Para
+comparar el modelo base y el fine-tuned sobre exactamente el mismo corpus:
+
+```powershell
+python scripts/evaluate_place_retriever.py `
+  --test-file data/training/places_retrieval_v1/test.jsonl `
+  --model base=intfloat/multilingual-e5-base `
+  --model fine_tuned=C:\ruta\al\places-e5-retriever-v1 `
+  --output-json artifacts/retriever-evaluation.json
+```
+
+Además de Top-1, Recall@k, MRR y nDCG@10, el reporte separa resultados para
+`colloquial`, `misspelling`, `implicit`, `contrastive` y otras dificultades. El
+prefijo `query:`/`passage:` se aplica dentro del evaluador.
+
 El extractor de intencion se entrena por separado con
 `scripts/train_place_intent_bert.py`. Su JSONL contiene `text` y spans abiertos
 `{start, end, slot}`; los slots permitidos son `CATEGORY`, `PREFERENCE`,

data/training/manifest.json

@@ -27,13 +27,13 @@
     },
     "validation": {
       "path": "data/training/places_retrieval_v1/validation.jsonl",
-      "records": 20,
-      "sha256": "418c93e09aae4783354287a6976517d132b2dd461879a73cc408ae9f0def066e"
+      "records": 60,
+      "sha256": "9f61e3467e907f195f14015f15acf78f7d4568fb2198b86516724ce1088a8241"
     },
     "test": {
       "path": "data/training/places_retrieval_v1/test.jsonl",
-      "records": 20,
-      "sha256": "010036e21b1515dd262728064259a4b526c8f36058ba6e33b026ba89fb103408"
+      "records": 80,
+      "sha256": "d99fd7094e7cc999b62ee264d88280e62e464c9a9562be8ebdeb1a9f06d64b37"
     }
   }
 }

data/training/places_retrieval_v1/README.md

@@ -4,7 +4,10 @@ Dataset JSONL sintético para arrancar el bi-encoder E5 de Places. Cada fila con
 
 - `query`: consulta sin prefijo E5.
 - `positive`: documento con formato `structured-place-v3`.
-- `hard_negatives`: tres lugares plausibles pero incorrectos del mismo split.
+- `hard_negatives`: lugares plausibles pero incorrectos del mismo split. Train
+  contiene tres; validation y test contienen siete para evitar evaluaciones triviales.
+- `challenge_tags` (validation y test): tipo de dificultad de la consulta, por ejemplo
+  `colloquial`, `misspelling`, `implicit`, `contrastive` o `constraint`.
 
 Los splits no comparten queries ni documentos positivos. No agregues `query:` o
 `passage:` a los archivos: el script de entrenamiento aplica esos prefijos.
@@ -12,15 +15,34 @@ Los splits no comparten queries ni documentos positivos. No agregues `query:` o
 Conteos:
 
 - `train.jsonl`: 160 pares.
-- `validation.jsonl`: 20 pares de conceptos no vistos.
-- `test.jsonl`: 20 pares de conceptos no vistos.
-- Total: 200 registros.
+- `validation.jsonl`: 60 consultas sobre 20 conceptos no vistos (tres formulaciones
+  por documento relevante).
+- `test.jsonl`: 80 consultas sobre 20 conceptos no vistos (cuatro formulaciones
+  por documento relevante).
+- Total: 300 registros; el split usado para entrenar permanece en 160 pares.
 
 Las sedes derivadas tienen nombres, documentos, consultas y atributos discriminantes
 propios. Variantes del mismo concepto se excluyen de los hard negatives explícitos
 para no etiquetar como incorrecto un lugar que también podría satisfacer la consulta.
 Validation y test conservan un solo documento por cada concepto no visto; así sus positivos
-no compiten contra una sede hermana igualmente relevante durante la evaluación.
+no compiten contra una sede hermana igualmente relevante durante la evaluación. Test repite
+el documento relevante para consultas canónicas, coloquiales, implícitas y con errores de
+escritura. Eso es intencional: varios intents lingüísticos pueden compartir el mismo qrel.
+Validation también contiene reformulaciones difíciles para elegir épocas y checkpoints sin
+consultar repetidamente el holdout final.
+
+No evalúes cada fila únicamente contra sus `hard_negatives`. Usa el corpus global
+deduplicado para que las 80 consultas compitan contra los 20 documentos de test:
+
+```bash
+python scripts/evaluate_place_retriever.py \
+  --test-file data/training/places_retrieval_v1/test.jsonl \
+  --model base=intfloat/multilingual-e5-base \
+  --model fine_tuned=/ruta/al/modelo
+```
+
+El resultado incluye Top-1, Recall@3/5/10, MRR, nDCG@10 y métricas separadas por
+`challenge_tags`. Elige épocas con validation; usa test solo para la comparación final.
 
 Este corpus cubre vocabulario abierto y sirve para iniciar el fine-tuning, pero no es
 suficiente por sí solo para una decisión productiva. Sustituir o ampliar train con

scripts/build_place_training_datasets.py

@@ -294,6 +294,207 @@ PLACE_SPECS: tuple[PlaceSpec, ...] = (
 )
 
 
+# Validation and test deliberately contain several queries for the same relevant
+# document. Validation supports epoch/model selection without inspecting test.
+# Test remains the final, harder holdout.
+# This is closer to production retrieval than a one-query/one-document lookup:
+# the model must survive slang, misspellings, implicit needs and contrastive
+# wording while ranking against the complete test corpus.  Keep these variants
+# out of train to prevent evaluation leakage.
+RETRIEVAL_VALIDATION_QUERY_VARIANTS: dict[
+    str,
+    tuple[tuple[str, tuple[str, ...]], ...],
+] = {
+    "Pupusas Centroamérica": (
+        ("se me antoja comida salvadoreña de masa rellena", ("implicit", "paraphrase")),
+        ("unas pupusitas con curtido dónde", ("colloquial", "implicit")),
+    ),
+    "Crêpe Maison": (
+        ("quiero algo enrollado que pueda ser dulce o salado", ("implicit", "contrastive")),
+        ("crepas pa cenar, no solo postre", ("colloquial", "contrastive")),
+    ),
+    "Planetario Orión": (
+        ("quiero ver el cielo proyectado sin salir de la ciudad", ("implicit", "paraphrase")),
+        ("plan de estrellas y astronomía bajo techo", ("implicit", "constraint")),
+    ),
+    "Aqua Centro": (
+        ("donde hay carriles pa echarme unos largos", ("colloquial", "implicit")),
+        ("alberca techada con clases para nadar", ("paraphrase", "constraint")),
+    ),
+    "Clave Oculta": (
+        ("plan con acertijos pa escapar con la banda", ("colloquial", "implicit")),
+        ("cuarto temático donde resolvemos pistas", ("implicit", "paraphrase")),
+    ),
+    "Vinilo Sur": (
+        ("dónde consigo discos de los grandotes", ("colloquial", "implicit")),
+        ("ando cazando elepés usados", ("colloquial", "paraphrase")),
+    ),
+    "Centro Presente": (
+        ("necesito respirar y apagar la mente un rato", ("implicit", "colloquial")),
+        ("un lugar silencioso con meditación guiada", ("paraphrase", "constraint")),
+    ),
+    "Lúpulo Local": (
+        ("cheve artesanal y que expliquen cómo la hacen", ("colloquial", "implicit")),
+        ("quiero degustar varias cervezas locales", ("paraphrase",)),
+    ),
+    "Sushi Viajero": (
+        ("sushi que vaya pasando frente a la mesa", ("implicit", "paraphrase")),
+        ("japonés con banda de platitos", ("colloquial", "implicit")),
+    ),
+    "Cero Residuo": (
+        ("comprar despensa llevando mis propios frascos", ("implicit", "constraint")),
+        ("tienda a granel sin tanto empaque", ("paraphrase", "constraint")),
+    ),
+    "Salto Alto": (
+        ("dónde llevo a brincar a los chamacos", ("colloquial", "implicit")),
+        ("parque techado lleno de camas elásticas", ("paraphrase", "implicit")),
+    ),
+    "Teatro Bel Canto": (
+        ("quiero escuchar voces líricas en un teatro", ("implicit", "paraphrase")),
+        ("algún recinto con bel canto y música de cámara", ("paraphrase",)),
+    ),
+    "Finca Aroma": (
+        ("quiero ver de dónde sale el café antes de la taza", ("implicit", "paraphrase")),
+        ("paseo entre cafetales con tostado", ("implicit", "paraphrase")),
+    ),
+    "Raqueta Sur": (
+        ("dónde rentan raqueta y cancha de squash", ("constraint", "paraphrase")),
+        ("quiero pegarle a la pelota contra la pared", ("implicit", "colloquial")),
+    ),
+    "Fábrica Lab": (
+        ("taller para fabricar prototipos con impresora 3d", ("implicit", "paraphrase")),
+        ("necesito herramientas y electrónica compartidas", ("implicit", "constraint")),
+    ),
+    "Estudio Obturador": (
+        ("renta de ciclorama y luces por hora", ("constraint", "paraphrase")),
+        ("lugar equipado pa una sesión de fotos", ("colloquial", "implicit")),
+    ),
+    "Verso Café": (
+        ("cafecito con micro abierto para leer versos", ("colloquial", "implicit")),
+        ("dónde escucho poesía mientras tomo algo", ("implicit", "paraphrase")),
+    ),
+    "Baño Nube": (
+        ("quiero un baño de vapor árabe", ("implicit", "paraphrase")),
+        ("spa con circuito de vapor tipo turco", ("paraphrase", "constraint")),
+    ),
+    "Cosecha Directa": (
+        ("verduras directo de quien las cosecha", ("implicit", "paraphrase")),
+        ("mercadito semanal de productores", ("colloquial", "paraphrase")),
+    ),
+    "Taco Verde": (
+        ("tacos sin nada de origen animal", ("implicit", "constraint")),
+        ("taquitos de setas, cero carne", ("colloquial", "contrastive")),
+    ),
+}
+
+
+RETRIEVAL_TEST_QUERY_VARIANTS: dict[
+    str,
+    tuple[tuple[str, tuple[str, ...]], ...],
+] = {
+    "Arepa Morena": (
+        ("traigo antojo de una reina pepiada", ("colloquial", "implicit")),
+        ("onde venden arepitas chidas", ("colloquial", "misspelling")),
+        ("algo venezolano relleno y hecho al momento", ("implicit", "paraphrase")),
+    ),
+    "Michi Café": (
+        ("un cafecito pa echar el rato con michis", ("colloquial", "implicit")),
+        ("cafeteria con gatitos rescatados", ("misspelling", "paraphrase")),
+        ("quiero tomar algo rodeado de mininos", ("implicit", "paraphrase")),
+    ),
+    "Laboratorio Curioso": (
+        ("algo chido pa que los peques hagan experimentos", ("colloquial", "implicit")),
+        ("museo no, mejor talleres cientificos para morritos", ("contrastive", "colloquial")),
+        ("donde llevo a mi hijo a aprender ciencia jugando", ("implicit", "paraphrase")),
+    ),
+    "Santuario Monarca": (
+        ("quiero ver un buen de mariposas en un jardin", ("colloquial", "misspelling")),
+        ("algún lugar de conservación de monarcas", ("implicit", "paraphrase")),
+        ("plan tranqui entre plantas y maripositas", ("colloquial", "implicit")),
+    ),
+    "Risa Abierta": (
+        ("donde hay standap esta noche", ("misspelling", "colloquial")),
+        ("quiero echarme unas risas con micro abierto", ("colloquial", "implicit")),
+        ("un plan de comedia en vivo, no cine", ("contrastive", "paraphrase")),
+    ),
+    "Café Babel": (
+        ("cafecito pa practicar inglés con la banda", ("colloquial", "implicit")),
+        ("intercambio d idiomas en un lugar para tomar algo", ("misspelling", "paraphrase")),
+        ("quiero conversar con gente en otras lenguas", ("implicit", "paraphrase")),
+    ),
+    "Miga Verde": (
+        ("algo dulce pero cero ingredientes animales", ("implicit", "contrastive")),
+        ("panecito vegano pa'l antojo", ("colloquial", "paraphrase")),
+        ("pasteles sin leche ni huevo, qué hay", ("implicit", "constraint")),
+    ),
+    "Bosque Domo": (
+        ("acampar sí pero sin sufrirle", ("colloquial", "implicit")),
+        ("un domito cómodo en el bosque", ("colloquial", "paraphrase")),
+        ("naturaleza con cama y techo, plan de finde", ("implicit", "colloquial")),
+    ),
+    "La Media Luna": (
+        ("se me antoja algo relleno y horneadito", ("implicit", "colloquial")),
+        ("onde hay empanadas dulces y saladas", ("misspelling", "paraphrase")),
+        ("un lugar de medias lunas no, de empanadas", ("contrastive", "adversarial")),
+    ),
+    "Herpetario Verde": (
+        ("quiero ver víboras y lagartos", ("implicit", "paraphrase")),
+        ("un plan educativo con animalitos de sangre fría", ("implicit", "paraphrase")),
+        ("donde conocen los peques a los reptiles", ("colloquial", "implicit")),
+    ),
+    "Museo del Barro": (
+        ("quiero ver piezas antiguas hechas de barro", ("implicit", "paraphrase")),
+        ("museo d alfareria de la region", ("misspelling", "paraphrase")),
+        ("un plan cultural sobre vasijas y cerámica", ("implicit", "paraphrase")),
+    ),
+    "Pista Furia": (
+        ("dónde entrenan ese deporte rudo en patines", ("implicit", "paraphrase")),
+        ("quiero darle al roler derbi", ("misspelling", "colloquial")),
+        ("pista para aprender derby sobre ruedas", ("paraphrase", "implicit")),
+    ),
+    "Voz Estudio": (
+        ("necesito una cabina que no meta ruido pa grabar", ("colloquial", "implicit")),
+        ("donde puedo producir mi podcast", ("paraphrase",)),
+        ("un estudio con tratamiento acustico y edición", ("misspelling", "implicit")),
+    ),
+    "Invernadero Café": (
+        ("cafecito entre un montón de plantas", ("colloquial", "implicit")),
+        ("un lugar verde para tomar café sin estar afuera", ("contrastive", "implicit")),
+        ("cafeteria tipo jardin techado", ("misspelling", "paraphrase")),
+    ),
+    "Robot Peques": (
+        ("algo pa que mi morrito arme robots", ("colloquial", "implicit")),
+        ("tayer infantil de programacion y electronica", ("misspelling", "paraphrase")),
+        ("donde enseñan tecnología construyendo cosas", ("implicit", "paraphrase")),
+    ),
+    "Remo Río": (
+        ("quiero remar pero necesito que me presten todo", ("implicit", "constraint")),
+        ("renta de kayak pa novatos con alguien que guíe", ("colloquial", "constraint")),
+        ("plan en el río con remo y equipo seguro", ("implicit", "paraphrase")),
+    ),
+    "Quesería Sierra": (
+        ("quiero probar quesitos de la región", ("colloquial", "implicit")),
+        ("una tienda donde den degustacion de queso artesanal", ("misspelling", "paraphrase")),
+        ("ando buscando lácteos locales, sobre todo quesos", ("implicit", "paraphrase")),
+    ),
+    "Circo Aire": (
+        ("quiero aprender a colgarme de telas sin matarme", ("colloquial", "implicit")),
+        ("clases d acrobacia aerea pa principiantes", ("misspelling", "colloquial")),
+        ("una escuela de circo con equilibrio y telas", ("paraphrase",)),
+    ),
+    "Reserva Alas": (
+        ("plan para pajarear con binoculares", ("colloquial", "implicit")),
+        ("un sitio con observatorios para avistar plumíferos", ("paraphrase", "implicit")),
+        ("quiero caminar por una reserva viendo aves", ("paraphrase",)),
+    ),
+    "Bar Cero": (
+        ("quiero salir de noche sin ponerme peda", ("colloquial", "implicit")),
+        ("bar con tragos cero alcohol y ambiente tranqui", ("colloquial", "constraint")),
+        ("mocktels y musiquita pero nada de chela", ("misspelling", "colloquial", "constraint")),
+    ),
+}
+
+
 def main() -> None:
     INTENT_ROOT.mkdir(parents=True, exist_ok=True)
     RETRIEVAL_ROOT.mkdir(parents=True, exist_ok=True)
@@ -409,16 +610,24 @@ def _retrieval_records(split: str) -> list[dict[str, object]]:
             and _concept_key(candidate) != _concept_key(spec)
         ]
         rotated = other_family[index % len(other_family) :] + other_family[: index % len(other_family)]
-        negative_specs = _unique_specs((*same_family, *rotated))[:3]
-        if len(negative_specs) < 3:
+        negative_limit = 3 if split == "train" else 7
+        negative_specs = _unique_specs((*same_family, *rotated))[:negative_limit]
+        if len(negative_specs) < negative_limit:
             raise RuntimeError(f"Not enough hard negatives for {spec.name}")
-        records.append(
-            {
-                "query": spec.query,
+        query_variants = [(spec.query, ("canonical",))]
+        if split == "validation":
+            query_variants.extend(RETRIEVAL_VALIDATION_QUERY_VARIANTS[spec.name])
+        elif split == "test":
+            query_variants.extend(RETRIEVAL_TEST_QUERY_VARIANTS[spec.name])
+        for query, challenge_tags in query_variants:
+            record: dict[str, object] = {
+                "query": query,
                 "positive": documents[spec.name],
                 "hard_negatives": [documents[item.name] for item in negative_specs],
             }
-        )
+            if split != "train":
+                record["challenge_tags"] = list(challenge_tags)
+            records.append(record)
     return records
 
 
@@ -544,8 +753,11 @@ def _validate_cross_split_uniqueness() -> None:
             positive = str(record["positive"])
             if query in retrieval_seen:
                 raise ValueError(f"Duplicate retrieval query in {retrieval_seen[query]} and {split}: {query}")
-            if positive in positive_seen:
-                raise ValueError(f"Duplicate positive document in {positive_seen[positive]} and {split}")
+            positive_owner = positive_seen.get(positive)
+            if positive_owner is not None and positive_owner != split:
+                raise ValueError(
+                    f"Duplicate positive document in {positive_owner} and {split}"
+                )
             retrieval_seen[query] = split
             positive_seen[positive] = split
 

scripts/evaluate_place_retriever.py

@@ -0,0 +1,250 @@
+"""Evaluate E5 place retrievers against one shared, global candidate corpus.
+
+Unlike a per-row positive-versus-negatives check, every query is ranked against
+all unique positives and hard negatives in the JSONL file.  This avoids the
+four-candidate ceiling that made both the base and fine-tuned models score 1.0.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+
+DEFAULT_BASE_MODEL = "intfloat/multilingual-e5-base"
+
+
+def read_evaluation_rows(path: Path) -> list[dict[str, Any]]:
+    if not path.is_file():
+        raise FileNotFoundError(f"Evaluation dataset not found: {path}")
+
+    rows: list[dict[str, Any]] = []
+    for line_number, raw_line in enumerate(
+        path.read_text(encoding="utf-8").splitlines(),
+        start=1,
+    ):
+        if not raw_line.strip():
+            continue
+        payload = json.loads(raw_line)
+        query = _required_text(payload.get("query"), line_number, "query")
+        positive = _required_text(payload.get("positive"), line_number, "positive")
+        negatives = payload.get("hard_negatives", [])
+        if not isinstance(negatives, list):
+            raise ValueError(
+                f"Line {line_number}: hard_negatives must be a list of strings"
+            )
+        hard_negatives = [
+            _required_text(value, line_number, "hard_negatives")
+            for value in negatives
+        ]
+        raw_tags = payload.get("challenge_tags", ["unclassified"])
+        if not isinstance(raw_tags, list) or not raw_tags:
+            raise ValueError(
+                f"Line {line_number}: challenge_tags must be a non-empty list"
+            )
+        challenge_tags = [
+            _required_text(value, line_number, "challenge_tags")
+            for value in raw_tags
+        ]
+        rows.append(
+            {
+                "query": query,
+                "positive": positive,
+                "hard_negatives": hard_negatives,
+                "challenge_tags": challenge_tags,
+            }
+        )
+
+    if not rows:
+        raise ValueError("At least one evaluation example is required")
+    return rows
+
+
+def build_global_corpus(rows: Iterable[dict[str, Any]]) -> list[str]:
+    """Return all candidate documents once, preserving their first-seen order."""
+
+    corpus: list[str] = []
+    seen: set[str] = set()
+    materialized = list(rows)
+    for row in materialized:
+        candidates = (row["positive"], *row.get("hard_negatives", []))
+        for candidate in candidates:
+            if candidate not in seen:
+                seen.add(candidate)
+                corpus.append(candidate)
+    return corpus
+
+
+def evaluate_model(
+    model: Any,
+    rows: list[dict[str, Any]],
+    *,
+    batch_size: int = 32,
+) -> dict[str, Any]:
+    corpus = build_global_corpus(rows)
+    document_indexes = {document: index for index, document in enumerate(corpus)}
+    query_texts = [f"query: {row['query']}" for row in rows]
+    passage_texts = [f"passage: {document}" for document in corpus]
+
+    query_embeddings = model.encode(
+        query_texts,
+        batch_size=batch_size,
+        normalize_embeddings=True,
+        convert_to_numpy=True,
+        show_progress_bar=True,
+    )
+    passage_embeddings = model.encode(
+        passage_texts,
+        batch_size=batch_size,
+        normalize_embeddings=True,
+        convert_to_numpy=True,
+        show_progress_bar=True,
+    )
+    scores = np.asarray(query_embeddings) @ np.asarray(passage_embeddings).T
+    positive_indexes = [document_indexes[row["positive"]] for row in rows]
+    ranks = calculate_ranks(scores, positive_indexes)
+    tags = [row["challenge_tags"] for row in rows]
+    return summarize_ranks(ranks, tags=tags, documents=len(corpus))
+
+
+def calculate_ranks(
+    scores: np.ndarray,
+    positive_indexes: list[int],
+) -> list[int]:
+    if scores.ndim != 2:
+        raise ValueError("scores must be a two-dimensional query-document matrix")
+    if scores.shape[0] != len(positive_indexes):
+        raise ValueError("scores and positive_indexes must contain the same queries")
+
+    ranks: list[int] = []
+    for query_index, positive_index in enumerate(positive_indexes):
+        if positive_index < 0 or positive_index >= scores.shape[1]:
+            raise ValueError(f"Invalid positive index for query {query_index}")
+        order = np.argsort(-scores[query_index], kind="stable")
+        position = np.flatnonzero(order == positive_index)
+        ranks.append(int(position[0]) + 1)
+    return ranks
+
+
+def summarize_ranks(
+    ranks: list[int],
+    *,
+    tags: list[list[str]],
+    documents: int,
+) -> dict[str, Any]:
+    if not ranks or len(ranks) != len(tags):
+        raise ValueError("ranks and tags must be non-empty and have equal length")
+
+    overall = _rank_metrics(ranks)
+    tag_ranks: defaultdict[str, list[int]] = defaultdict(list)
+    for rank, row_tags in zip(ranks, tags):
+        for tag in set(row_tags):
+            tag_ranks[tag].append(rank)
+
+    return {
+        "queries": len(ranks),
+        "documents": documents,
+        **overall,
+        "per_challenge": {
+            tag: {"queries": len(values), **_rank_metrics(values)}
+            for tag, values in sorted(tag_ranks.items())
+        },
+        "ranks": ranks,
+    }
+
+
+def _rank_metrics(ranks: list[int]) -> dict[str, float]:
+    values = np.asarray(ranks, dtype=np.float64)
+    discounted_gains_at_10 = np.where(
+        values <= 10,
+        1.0 / np.log2(values + 1.0),
+        0.0,
+    )
+    return {
+        "top1": float(np.mean(values <= 1)),
+        "recall_at_3": float(np.mean(values <= 3)),
+        "recall_at_5": float(np.mean(values <= 5)),
+        "recall_at_10": float(np.mean(values <= 10)),
+        "mrr": float(np.mean(1.0 / values)),
+        "ndcg_at_10": float(np.mean(discounted_gains_at_10)),
+        "mean_rank": float(np.mean(values)),
+    }
+
+
+def _required_text(value: Any, line_number: int, field: str) -> str:
+    if not isinstance(value, str) or not value.strip():
+        raise ValueError(f"Line {line_number}: {field} must be a non-empty string")
+    return " ".join(value.split())
+
+
+def _parse_model_specs(values: list[str]) -> list[tuple[str, str]]:
+    if not values:
+        return [("base", DEFAULT_BASE_MODEL)]
+    parsed: list[tuple[str, str]] = []
+    labels: set[str] = set()
+    for value in values:
+        if "=" not in value:
+            raise ValueError("Each --model must use LABEL=MODEL_OR_PATH")
+        label, model_path = (part.strip() for part in value.split("=", 1))
+        if not label or not model_path:
+            raise ValueError("Each --model must use LABEL=MODEL_OR_PATH")
+        if label in labels:
+            raise ValueError(f"Duplicate model label: {label}")
+        labels.add(label)
+        parsed.append((label, model_path))
+    return parsed
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--test-file", required=True)
+    parser.add_argument(
+        "--model",
+        action="append",
+        default=[],
+        metavar="LABEL=MODEL_OR_PATH",
+        help="Repeat to compare multiple models on exactly the same corpus.",
+    )
+    parser.add_argument("--device", default=None)
+    parser.add_argument("--batch-size", type=int, default=32)
+    parser.add_argument("--output-json")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = _parse_args()
+    if args.batch_size < 1:
+        raise ValueError("--batch-size must be greater than zero")
+    try:
+        from sentence_transformers import SentenceTransformer
+    except ImportError as exc:
+        raise RuntimeError(
+            "Install requirements-training.txt before evaluating retrievers"
+        ) from exc
+
+    rows = read_evaluation_rows(Path(args.test_file))
+    results: dict[str, Any] = {}
+    for label, model_path in _parse_model_specs(args.model):
+        print(f"Evaluating {label}: {model_path}")
+        model = SentenceTransformer(model_path, device=args.device)
+        results[label] = evaluate_model(
+            model,
+            rows,
+            batch_size=args.batch_size,
+        )
+
+    output = json.dumps(results, ensure_ascii=False, indent=2)
+    print(output)
+    if args.output_json:
+        output_path = Path(args.output_json)
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        output_path.write_text(output + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

tests/test_evaluate_place_retriever.py

@@ -0,0 +1,51 @@
+from pathlib import Path
+
+import numpy as np
+
+from scripts.evaluate_place_retriever import (
+    build_global_corpus,
+    calculate_ranks,
+    read_evaluation_rows,
+    summarize_ranks,
+)
+
+
+ROOT = Path(__file__).resolve().parents[1]
+TEST_DATA = ROOT / "data" / "training" / "places_retrieval_v1" / "test.jsonl"
+VALIDATION_DATA = (
+    ROOT / "data" / "training" / "places_retrieval_v1" / "validation.jsonl"
+)
+
+
+def test_challenge_evaluation_uses_one_global_candidate_pool() -> None:
+    for path, expected_queries in ((VALIDATION_DATA, 60), (TEST_DATA, 80)):
+        rows = read_evaluation_rows(path)
+        corpus = build_global_corpus(rows)
+
+        assert len(rows) == expected_queries
+        assert len(corpus) == 20
+        assert set(corpus) == {row["positive"] for row in rows}
+
+
+def test_rank_metrics_are_calculated_over_all_documents_and_by_challenge() -> None:
+    scores = np.asarray(
+        [
+            [0.9, 0.8, 0.1, 0.0],
+            [0.8, 0.9, 0.7, 0.1],
+            [0.4, 0.3, 0.2, 0.1],
+        ]
+    )
+    ranks = calculate_ranks(scores, [0, 2, 3])
+    summary = summarize_ranks(
+        ranks,
+        tags=[["canonical"], ["colloquial"], ["colloquial", "misspelling"]],
+        documents=4,
+    )
+
+    assert ranks == [1, 3, 4]
+    assert summary["queries"] == 3
+    assert summary["documents"] == 4
+    assert summary["top1"] == 1 / 3
+    assert summary["recall_at_3"] == 2 / 3
+    assert summary["per_challenge"]["colloquial"]["queries"] == 2
+    assert summary["per_challenge"]["misspelling"]["mean_rank"] == 4.0

tests/test_place_training_datasets.py

@@ -4,6 +4,8 @@ from pathlib import Path
 
 from scripts.build_place_training_datasets import (
     INTENT_CONCEPTS,
+    RETRIEVAL_TEST_QUERY_VARIANTS,
+    RETRIEVAL_VALIDATION_QUERY_VARIANTS,
     _concept_key,
     _expanded_place_specs,
     _place_document,
@@ -52,7 +54,7 @@ def test_intent_dataset_is_valid_open_vocabulary_and_leak_free() -> None:
 
 def test_retrieval_dataset_matches_training_contract_and_has_unique_positives() -> None:
     seen_queries: set[str] = set()
-    seen_positives: set[str] = set()
+    positive_splits: dict[str, str] = {}
     counts: dict[str, int] = {}
 
     for split in ("train", "validation", "test"):
@@ -62,21 +64,24 @@ def test_retrieval_dataset_matches_training_contract_and_has_unique_positives()
         counts[split] = len(rows)
         for row in rows:
             assert row["query"].casefold() not in seen_queries
-            assert row["positive"] not in seen_positives
+            positive_owner = positive_splits.get(row["positive"])
+            assert positive_owner in (None, split)
             seen_queries.add(row["query"].casefold())
-            seen_positives.add(row["positive"])
+            positive_splits[row["positive"]] = split
             assert row["positive"].startswith("Nombre: ")
             assert "Tipo registrado:" in row["positive"]
             assert "Descripcion:" in row["positive"]
             assert "Etiquetas:" in row["positive"]
-            assert len(row["hard_negatives"]) == 3
-            assert len(set(row["hard_negatives"])) == 3
+            expected_negatives = 3 if split == "train" else 7
+            assert len(row["hard_negatives"]) == expected_negatives
+            assert len(set(row["hard_negatives"])) == expected_negatives
             assert row["positive"] not in row["hard_negatives"]
             assert not row["query"].startswith("query: ")
             assert not row["positive"].startswith("passage: ")
 
-    assert counts == {"train": 160, "validation": 20, "test": 20}
-    assert 150 <= sum(counts.values()) <= 200
+    assert counts == {"train": 160, "validation": 60, "test": 80}
+    assert 150 <= counts["train"] <= 200
+    assert len(positive_splits) == 200
 
 
 def test_retrieval_hard_negatives_never_use_a_sibling_of_the_positive() -> None:
@@ -88,20 +93,61 @@ def test_retrieval_hard_negatives_never_use_a_sibling_of_the_positive() -> None:
         document_concepts = {
             _place_document(spec): _concept_key(spec) for spec in specs
         }
-        assert len(rows) == len(specs)
+        if split == "train":
+            assert len(rows) == len(specs)
         if split != "train":
             assert len({_concept_key(spec) for spec in specs}) == len(specs)
-        for spec, row in zip(specs, rows):
-            positive_concept = _concept_key(spec)
-            assert document_concepts[row["positive"]] == positive_concept
+        assert {row["positive"] for row in rows} == set(document_concepts)
+        for row in rows:
+            positive_concept = document_concepts[row["positive"]]
             assert all(
                 document_concepts[negative] != positive_concept
                 for negative in row["hard_negatives"]
             )
 
 
+def test_retrieval_evaluation_has_curated_challenges_and_repeated_qrels() -> None:
+    assert len(RETRIEVAL_VALIDATION_QUERY_VARIANTS) == 20
+    assert len(RETRIEVAL_TEST_QUERY_VARIANTS) == 20
+    for split, expected_rows, expected_repetitions in (
+        ("validation", 60, 3),
+        ("test", 80, 4),
+    ):
+        path = DATA / "places_retrieval_v1" / f"{split}.jsonl"
+        rows = [
+            json.loads(line)
+            for line in path.read_text(encoding="utf-8").splitlines()
+        ]
+        positive_counts = Counter(row["positive"] for row in rows)
+        tag_counts = Counter(
+            tag
+            for row in rows
+            for tag in row["challenge_tags"]
+        )
+
+        assert len(rows) == expected_rows
+        assert set(positive_counts.values()) == {expected_repetitions}
+        assert tag_counts["canonical"] == 20
+        assert tag_counts["colloquial"] >= 10
+        assert tag_counts["implicit"] >= 20
+
+    test_rows = [
+        json.loads(line)
+        for line in (DATA / "places_retrieval_v1" / "test.jsonl")
+        .read_text(encoding="utf-8")
+        .splitlines()
+    ]
+    test_tag_counts = Counter(
+        tag for row in test_rows for tag in row["challenge_tags"]
+    )
+    assert test_tag_counts["misspelling"] >= 10
+    assert test_tag_counts["contrastive"] >= 4
+
+
 def test_dataset_manifest_matches_generated_files() -> None:
     manifest = json.loads((DATA / "manifest.json").read_text(encoding="utf-8"))
     assert manifest["schema_version"] == 1
     assert manifest["intent"]["train"]["records"] == 132
     assert manifest["retrieval"]["train"]["records"] == 160
+    assert manifest["retrieval"]["validation"]["records"] == 60
+    assert manifest["retrieval"]["test"]["records"] == 80