Spaces:
Sleeping
feat(audit): Phase 3 partielle — câblage des features inachevées (S2, S4, S6)
Browse filesTrois features réellement inachevées débloquées par l'audit
code-quality. Plutôt que de supprimer du code "inutilisé", on
branche ce qui a une vraie valeur produit.
**3.4 (S4) — Sur-normalisation LLM agrégée corpus-wide**
``aggregate_over_normalization`` existait mais :
- 0 ``@register_corpus_aggregator`` → jamais exécuté par les hooks
- module pas importé par ``evaluation/metrics/__init__.py``
(seulement en docstring)
- ``synthetic.py`` réimplémentait l'agrégation à la main
Câblage propre :
- Ajout du hook décoré ``_aggregate_over_normalization_hook``
(profils ``philological``, ``diagnostics``, ``full``) qui extrait
l'info depuis ``DocumentResult.pipeline_metadata["over_normalization"]``
et délègue à la fonction pure (rétrocompat préservée).
- Nouveau champ ``EngineReport.aggregated_over_normalization`` +
round-trip JSON ``as_dict``/``from_dict``.
- Helper ``_from_metadata_dict`` reconstruit
``OverNormalizationResult`` depuis le dict stocké, gère les
erreurs de typage avec ``logger.warning("[over_normalization]...")``.
- Module ajouté à ``evaluation/metrics/__init__.py`` pour déclencher
l'auto-enregistrement à l'import.
Tests : ``test_over_normalization_hook.py`` (8 tests) — registry,
profils, fonction pure, hook, malformed dict, round-trip JSON.
**3.5 (S6) — Test live tesseract : marker guard**
L'audit avait flaggé ``test_tesseract_live.py`` comme « skip top-level
inconditionnel ». Vérification : le skip est en réalité **conditionnel**
(``if shutil.which("tesseract") is None``) et le marker
``@pytest.mark.live`` est bien posé. Aucun bug — l'audit s'est
trompé. Mais on ajoute un garde-fou pour éviter qu'une nouvelle
fonction de test dans ``tests/integration/live/`` n'oublie le marker
(fait s'exécuter le test en CI standard et casse sans clé API).
Tests : ``test_live_test_markers.py`` (2 tests) — AST scan des
``test_*`` au top-level de ``tests/integration/live/``, échoue si
manque ``@pytest.mark.live``.
**3.2 (S2) — Journal des fallbacks d'importer**
Le détecteur narratif ``IMPORTER_FALLBACK_TRIGGERED`` était écrit
(history.py:280) et attendait ``benchmark_data["importer_fallbacks"]``,
mais le wiring intermédiaire manquait :
1. ``HTRUnitedCatalogue.from_remote`` quand DNS/réseau échoue
→ loguait mais n'appelait pas ``record_fallback`` (alors que
HuggingFace et le ``_parse_yml_catalogue`` le faisaient).
Ajout de l'appel + ``extra={"url", "fallback_used": "demo"}``.
2. ``app/services/benchmark_runner.py`` : 2 sites de production de
``BenchmarkResult`` (``_run_benchmark_unified``,
``_run_benchmark_with_partial``) — aucun ne consommait le journal.
Ajout de ``consume_fallback_log()`` en fin de run + stockage
dans ``BenchmarkResult.metadata["importer_fallbacks"]``.
3. ``reports/html/data.build_report_data`` ne propageait pas
``metadata.importer_fallbacks`` dans ``report_data``. Ajout
de la clé ``importer_fallbacks`` (liste vide si rien).
Résultat : pipeline end-to-end fonctionnel — un fallback HTR-United
en mode démo apparaît désormais dans la synthèse narrative du
rapport HTML, avec traçabilité (URL distante + raison de l'échec).
Tests : ``test_importer_fallback_wiring.py`` (8 tests E2E) — du
``record_fallback`` jusqu'au ``Fact`` rendu dans la prose de
``build_synthesis``. Régression couverte : si
``HTRUnitedCatalogue.from_remote`` oublie d'appeler ``record_fallback``
dans son except, le test ``test_htr_united_fallback_records_entry``
échoue.
**Bilan**
Suite : 4 750 passed, 16 skipped, 8 deselected, 2 xfailed.
+18 tests vs Phase 2 (8+2+8). Ruff propre, sync-counters CI vert,
auto-incrémenté à 4 750 (cohérent avec la prose CLAUDE.md/README.md).
Phase 3 partielle — restent les sous-phases 3.1 (backend pure-Python
robustness) et 3.3 (exposer NormalizationProfile.from_yaml en CLI/API).
- picarones/adapters/corpus/htr_united.py +10 -1
- picarones/app/services/benchmark_runner.py +22 -0
- picarones/evaluation/benchmark_result.py +16 -0
- picarones/evaluation/metrics/__init__.py +2 -0
- picarones/evaluation/metrics/over_normalization.py +80 -1
- picarones/reports/html/data/__init__.py +5 -0
- tests/architecture/test_live_test_markers.py +83 -0
- tests/evaluation/metrics/test_over_normalization_hook.py +217 -0
- tests/integration/test_importer_fallback_wiring.py +196 -0
|
@@ -259,12 +259,21 @@ class HTRUnitedCatalogue:
|
|
| 259 |
entries = _parse_yml_catalogue(raw)
|
| 260 |
return cls(entries, source="remote")
|
| 261 |
except (urllib.error.URLError, Exception) as exc:
|
| 262 |
-
# Fallback démo avec avertissement
|
|
|
|
|
|
|
| 263 |
logger.warning(
|
| 264 |
"[HTR-United] impossible de charger le catalogue distant (%s) : %s. "
|
| 265 |
"Utilisation des données de démonstration.",
|
| 266 |
_CATALOGUE_URL, exc,
|
| 267 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 268 |
return cls.from_demo()
|
| 269 |
|
| 270 |
def search(
|
|
|
|
| 259 |
entries = _parse_yml_catalogue(raw)
|
| 260 |
return cls(entries, source="remote")
|
| 261 |
except (urllib.error.URLError, Exception) as exc:
|
| 262 |
+
# Fallback démo avec avertissement. Phase 3.2 audit
|
| 263 |
+
# code-quality : enregistrement de l'incident pour le
|
| 264 |
+
# détecteur narratif ``IMPORTER_FALLBACK_TRIGGERED``.
|
| 265 |
logger.warning(
|
| 266 |
"[HTR-United] impossible de charger le catalogue distant (%s) : %s. "
|
| 267 |
"Utilisation des données de démonstration.",
|
| 268 |
_CATALOGUE_URL, exc,
|
| 269 |
)
|
| 270 |
+
from picarones.adapters.corpus._fallback_log import record_fallback
|
| 271 |
+
record_fallback(
|
| 272 |
+
importer="htr_united",
|
| 273 |
+
operation="catalogue_remote_fetch",
|
| 274 |
+
error=exc,
|
| 275 |
+
extra={"url": _CATALOGUE_URL, "fallback_used": "demo"},
|
| 276 |
+
)
|
| 277 |
return cls.from_demo()
|
| 278 |
|
| 279 |
def search(
|
|
@@ -372,11 +372,25 @@ def run_result_to_benchmark_result(
|
|
| 372 |
),
|
| 373 |
)
|
| 374 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 375 |
return BenchmarkResult(
|
| 376 |
corpus_name=corpus.name,
|
| 377 |
corpus_source=str(corpus.source_path) if corpus.source_path else None,
|
| 378 |
document_count=len(documents),
|
| 379 |
engine_reports=engine_reports,
|
|
|
|
| 380 |
)
|
| 381 |
|
| 382 |
|
|
@@ -1532,11 +1546,19 @@ def _run_benchmark_with_partial(
|
|
| 1532 |
# ``all_doc_results``.
|
| 1533 |
_delete_partial(partial_path)
|
| 1534 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1535 |
return BenchmarkResult(
|
| 1536 |
corpus_name=corpus.name,
|
| 1537 |
corpus_source=str(corpus.source_path) if corpus.source_path else None,
|
| 1538 |
document_count=len(corpus.documents),
|
| 1539 |
engine_reports=engine_reports,
|
|
|
|
| 1540 |
)
|
| 1541 |
|
| 1542 |
|
|
|
|
| 372 |
),
|
| 373 |
)
|
| 374 |
|
| 375 |
+
# Phase 3.2 audit code-quality — consommer le journal des
|
| 376 |
+
# fallbacks d'importer (HTR-United, HuggingFace, etc.). La liste
|
| 377 |
+
# est vidée à la fin du benchmark pour que le run suivant
|
| 378 |
+
# n'hérite pas des incidents du précédent. Le détecteur narratif
|
| 379 |
+
# ``IMPORTER_FALLBACK_TRIGGERED`` (history.py:280) lit
|
| 380 |
+
# ``benchmark_data["importer_fallbacks"]`` propagé par
|
| 381 |
+
# ``build_report_data``.
|
| 382 |
+
from picarones.adapters.corpus._fallback_log import consume_fallback_log
|
| 383 |
+
fallbacks = consume_fallback_log()
|
| 384 |
+
metadata: dict[str, Any] = {}
|
| 385 |
+
if fallbacks:
|
| 386 |
+
metadata["importer_fallbacks"] = fallbacks
|
| 387 |
+
|
| 388 |
return BenchmarkResult(
|
| 389 |
corpus_name=corpus.name,
|
| 390 |
corpus_source=str(corpus.source_path) if corpus.source_path else None,
|
| 391 |
document_count=len(documents),
|
| 392 |
engine_reports=engine_reports,
|
| 393 |
+
metadata=metadata,
|
| 394 |
)
|
| 395 |
|
| 396 |
|
|
|
|
| 1546 |
# ``all_doc_results``.
|
| 1547 |
_delete_partial(partial_path)
|
| 1548 |
|
| 1549 |
+
# Phase 3.2 audit code-quality — cf. _run_benchmark_unified.
|
| 1550 |
+
from picarones.adapters.corpus._fallback_log import consume_fallback_log
|
| 1551 |
+
fallbacks = consume_fallback_log()
|
| 1552 |
+
metadata: dict[str, Any] = {}
|
| 1553 |
+
if fallbacks:
|
| 1554 |
+
metadata["importer_fallbacks"] = fallbacks
|
| 1555 |
+
|
| 1556 |
return BenchmarkResult(
|
| 1557 |
corpus_name=corpus.name,
|
| 1558 |
corpus_source=str(corpus.source_path) if corpus.source_path else None,
|
| 1559 |
document_count=len(corpus.documents),
|
| 1560 |
engine_reports=engine_reports,
|
| 1561 |
+
metadata=metadata,
|
| 1562 |
)
|
| 1563 |
|
| 1564 |
|
|
@@ -364,6 +364,17 @@ class EngineReport:
|
|
| 364 |
delta_median, delta_min, delta_max, n_over_normalized,
|
| 365 |
n_under_normalized, over_normalized_rate}``. ``None`` si
|
| 366 |
aucun document n'avait de ``readability_metrics``."""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 367 |
|
| 368 |
def __post_init__(self) -> None:
|
| 369 |
if not self.aggregated_metrics and self.document_results:
|
|
@@ -450,6 +461,8 @@ class EngineReport:
|
|
| 450 |
)
|
| 451 |
if self.aggregated_readability is not None:
|
| 452 |
d["aggregated_readability"] = self.aggregated_readability
|
|
|
|
|
|
|
| 453 |
return d
|
| 454 |
|
| 455 |
@classmethod
|
|
@@ -487,6 +500,9 @@ class EngineReport:
|
|
| 487 |
"aggregated_numerical_sequences",
|
| 488 |
),
|
| 489 |
aggregated_readability=data.get("aggregated_readability"),
|
|
|
|
|
|
|
|
|
|
| 490 |
)
|
| 491 |
|
| 492 |
|
|
|
|
| 364 |
delta_median, delta_min, delta_max, n_over_normalized,
|
| 365 |
n_under_normalized, over_normalized_rate}``. ``None`` si
|
| 366 |
aucun document n'avait de ``readability_metrics``."""
|
| 367 |
+
# Phase 3.4 audit code-quality (2026-05) — câblage de
|
| 368 |
+
# ``aggregate_over_normalization`` (classe 10 de la taxonomie).
|
| 369 |
+
aggregated_over_normalization: Optional[dict] = None
|
| 370 |
+
"""Sur-normalisation LLM agrégée corpus-wide.
|
| 371 |
+
|
| 372 |
+
Format ``{score, total_correct_ocr_words, over_normalized_count,
|
| 373 |
+
document_count}`` produit par
|
| 374 |
+
:func:`picarones.evaluation.metrics.over_normalization.aggregate_over_normalization`.
|
| 375 |
+
``None`` si aucun document n'a porté de
|
| 376 |
+
``pipeline_metadata["over_normalization"]`` (cas d'un benchmark
|
| 377 |
+
OCR seul, sans étape LLM)."""
|
| 378 |
|
| 379 |
def __post_init__(self) -> None:
|
| 380 |
if not self.aggregated_metrics and self.document_results:
|
|
|
|
| 461 |
)
|
| 462 |
if self.aggregated_readability is not None:
|
| 463 |
d["aggregated_readability"] = self.aggregated_readability
|
| 464 |
+
if self.aggregated_over_normalization is not None:
|
| 465 |
+
d["aggregated_over_normalization"] = self.aggregated_over_normalization
|
| 466 |
return d
|
| 467 |
|
| 468 |
@classmethod
|
|
|
|
| 500 |
"aggregated_numerical_sequences",
|
| 501 |
),
|
| 502 |
aggregated_readability=data.get("aggregated_readability"),
|
| 503 |
+
aggregated_over_normalization=data.get(
|
| 504 |
+
"aggregated_over_normalization",
|
| 505 |
+
),
|
| 506 |
)
|
| 507 |
|
| 508 |
|
|
@@ -57,6 +57,7 @@ from picarones.evaluation.metrics import ( # noqa: F401
|
|
| 57 |
longitudinal,
|
| 58 |
marginal_cost,
|
| 59 |
module_policy,
|
|
|
|
| 60 |
pricing,
|
| 61 |
rare_tokens,
|
| 62 |
robustness_projection,
|
|
@@ -83,6 +84,7 @@ __all__ = [
|
|
| 83 |
"longitudinal",
|
| 84 |
"marginal_cost",
|
| 85 |
"module_policy",
|
|
|
|
| 86 |
"pricing",
|
| 87 |
"rare_tokens",
|
| 88 |
"robustness_projection",
|
|
|
|
| 57 |
longitudinal,
|
| 58 |
marginal_cost,
|
| 59 |
module_policy,
|
| 60 |
+
over_normalization,
|
| 61 |
pricing,
|
| 62 |
rare_tokens,
|
| 63 |
robustness_projection,
|
|
|
|
| 84 |
"longitudinal",
|
| 85 |
"marginal_cost",
|
| 86 |
"module_policy",
|
| 87 |
+
"over_normalization",
|
| 88 |
"pricing",
|
| 89 |
"rare_tokens",
|
| 90 |
"robustness_projection",
|
|
@@ -22,9 +22,19 @@ la graphie originale.
|
|
| 22 |
|
| 23 |
from __future__ import annotations
|
| 24 |
|
|
|
|
| 25 |
from dataclasses import dataclass, field
|
| 26 |
from typing import Optional
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
@dataclass
|
| 30 |
class OverNormalizationResult:
|
|
@@ -111,7 +121,16 @@ def detect_over_normalization(
|
|
| 111 |
|
| 112 |
|
| 113 |
def aggregate_over_normalization(results: list[Optional[OverNormalizationResult]]) -> dict:
|
| 114 |
-
"""Agrège les résultats de sur-normalisation sur un ensemble de documents.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
valid = [r for r in results if r is not None]
|
| 116 |
if not valid:
|
| 117 |
return {"score": None, "total_correct_ocr_words": 0, "over_normalized_count": 0}
|
|
@@ -126,3 +145,63 @@ def aggregate_over_normalization(results: list[Optional[OverNormalizationResult]
|
|
| 126 |
"over_normalized_count": total_over,
|
| 127 |
"document_count": len(valid),
|
| 128 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
from __future__ import annotations
|
| 24 |
|
| 25 |
+
import logging
|
| 26 |
from dataclasses import dataclass, field
|
| 27 |
from typing import Optional
|
| 28 |
|
| 29 |
+
from picarones.evaluation.metric_hooks import (
|
| 30 |
+
PROFILE_DIAGNOSTICS,
|
| 31 |
+
PROFILE_FULL,
|
| 32 |
+
PROFILE_PHILOLOGICAL,
|
| 33 |
+
register_corpus_aggregator,
|
| 34 |
+
)
|
| 35 |
+
|
| 36 |
+
logger = logging.getLogger(__name__)
|
| 37 |
+
|
| 38 |
|
| 39 |
@dataclass
|
| 40 |
class OverNormalizationResult:
|
|
|
|
| 121 |
|
| 122 |
|
| 123 |
def aggregate_over_normalization(results: list[Optional[OverNormalizationResult]]) -> dict:
|
| 124 |
+
"""Agrège les résultats de sur-normalisation sur un ensemble de documents.
|
| 125 |
+
|
| 126 |
+
Fonction pure utilitaire — reçoit directement une liste de
|
| 127 |
+
:class:`OverNormalizationResult` (typiquement le retour de
|
| 128 |
+
:func:`detect_over_normalization`). Pour l'agrégation à partir
|
| 129 |
+
d'une liste de :class:`DocumentResult` produite par un benchmark,
|
| 130 |
+
le hook décoré :func:`_aggregate_over_normalization_hook`
|
| 131 |
+
(auto-enregistré) extrait l'information depuis
|
| 132 |
+
``dr.pipeline_metadata["over_normalization"]``.
|
| 133 |
+
"""
|
| 134 |
valid = [r for r in results if r is not None]
|
| 135 |
if not valid:
|
| 136 |
return {"score": None, "total_correct_ocr_words": 0, "over_normalized_count": 0}
|
|
|
|
| 145 |
"over_normalized_count": total_over,
|
| 146 |
"document_count": len(valid),
|
| 147 |
}
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
# ---------------------------------------------------------------------------
|
| 151 |
+
# Hook d'agrégation corpus-level — Phase 3.4 audit code-quality (2026-05)
|
| 152 |
+
# ---------------------------------------------------------------------------
|
| 153 |
+
#
|
| 154 |
+
# Le calcul ``detect_over_normalization`` est branché en amont (synthétique
|
| 155 |
+
# + pipelines OCR+LLM réels) et stocke son résultat dans
|
| 156 |
+
# ``dr.pipeline_metadata["over_normalization"]`` (déjà sous forme de dict
|
| 157 |
+
# via ``OverNormalizationResult.as_dict()``). Le hook ci-dessous
|
| 158 |
+
# l'extrait et invoque l'agrégateur pur ; la valeur retournée alimente
|
| 159 |
+
# l'attribut ``EngineReport.aggregated_over_normalization``.
|
| 160 |
+
#
|
| 161 |
+
# Profils : disponible pour ``philological`` (analyse fine du LLM),
|
| 162 |
+
# ``diagnostics`` (audit du pipeline) et ``full``.
|
| 163 |
+
|
| 164 |
+
|
| 165 |
+
def _from_metadata_dict(meta: Optional[dict]) -> Optional[OverNormalizationResult]:
|
| 166 |
+
"""Reconstruit un :class:`OverNormalizationResult` depuis le dict
|
| 167 |
+
stocké dans ``pipeline_metadata`` (forme ``as_dict()``)."""
|
| 168 |
+
if not isinstance(meta, dict):
|
| 169 |
+
return None
|
| 170 |
+
try:
|
| 171 |
+
return OverNormalizationResult(
|
| 172 |
+
total_correct_ocr_words=int(meta.get("total_correct_ocr_words", 0)),
|
| 173 |
+
over_normalized_count=int(meta.get("over_normalized_count", 0)),
|
| 174 |
+
over_normalized_passages=list(meta.get("over_normalized_passages", []) or []),
|
| 175 |
+
)
|
| 176 |
+
except (TypeError, ValueError) as exc:
|
| 177 |
+
logger.warning(
|
| 178 |
+
"[over_normalization] dict metadata mal formé, ignoré : %s", exc,
|
| 179 |
+
)
|
| 180 |
+
return None
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
@register_corpus_aggregator(
|
| 184 |
+
name="over_normalization",
|
| 185 |
+
attribute="aggregated_over_normalization",
|
| 186 |
+
profiles=(PROFILE_PHILOLOGICAL, PROFILE_DIAGNOSTICS, PROFILE_FULL),
|
| 187 |
+
)
|
| 188 |
+
def _aggregate_over_normalization_hook(doc_results: list) -> Optional[dict]:
|
| 189 |
+
"""Agrégateur corpus-level — auto-enregistré.
|
| 190 |
+
|
| 191 |
+
Extrait ``pipeline_metadata["over_normalization"]`` de chaque
|
| 192 |
+
document, reconstruit un :class:`OverNormalizationResult`, et
|
| 193 |
+
délègue à :func:`aggregate_over_normalization` (logique pure).
|
| 194 |
+
Retourne ``None`` si aucun document n'avait de données — pas
|
| 195 |
+
d'attribut ajouté au :class:`EngineReport` dans ce cas.
|
| 196 |
+
"""
|
| 197 |
+
extracted = [
|
| 198 |
+
_from_metadata_dict(
|
| 199 |
+
getattr(dr, "pipeline_metadata", {}).get("over_normalization")
|
| 200 |
+
if hasattr(dr, "pipeline_metadata")
|
| 201 |
+
else None
|
| 202 |
+
)
|
| 203 |
+
for dr in doc_results
|
| 204 |
+
]
|
| 205 |
+
if not any(r is not None for r in extracted):
|
| 206 |
+
return None
|
| 207 |
+
return aggregate_over_normalization(extracted)
|
|
@@ -126,6 +126,11 @@ def build_report_data(
|
|
| 126 |
"taxonomy_intra_doc": compute_taxonomy_intra_doc_section(benchmark),
|
| 127 |
# Sprint 91 (A.II.6) : matrice de coût marginal entre paires de moteurs.
|
| 128 |
"marginal_cost": compute_marginal_cost_section(engines_summary),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
}
|
| 130 |
|
| 131 |
|
|
|
|
| 126 |
"taxonomy_intra_doc": compute_taxonomy_intra_doc_section(benchmark),
|
| 127 |
# Sprint 91 (A.II.6) : matrice de coût marginal entre paires de moteurs.
|
| 128 |
"marginal_cost": compute_marginal_cost_section(engines_summary),
|
| 129 |
+
# Phase 3.2 audit code-quality — incidents d'importer (fallback
|
| 130 |
+
# mode démo HTR-United, fallback recherche HuggingFace, etc.)
|
| 131 |
+
# propagés au détecteur narratif ``IMPORTER_FALLBACK_TRIGGERED``.
|
| 132 |
+
# Liste vide si aucun fallback n'a eu lieu.
|
| 133 |
+
"importer_fallbacks": (benchmark.metadata or {}).get("importer_fallbacks", []),
|
| 134 |
}
|
| 135 |
|
| 136 |
|
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Phase 3.5 audit code-quality — les tests dans
|
| 2 |
+
``tests/integration/live/`` doivent porter le marker
|
| 3 |
+
``@pytest.mark.live`` sur **chacune** de leurs fonctions de test.
|
| 4 |
+
|
| 5 |
+
Contexte : ``pyproject.toml`` déclare le marker ``live`` comme
|
| 6 |
+
« tests d'intégration contre vraie API/binaire (Tesseract,
|
| 7 |
+
Anthropic, OpenAI, Mistral) ; exclus par défaut, opt-in via
|
| 8 |
+
``pytest -m live`` ». Le filtre ``addopts = '-m "not live and not
|
| 9 |
+
network"'`` les déselectionne au runner par défaut.
|
| 10 |
+
|
| 11 |
+
Si une fonction dans ``tests/integration/live/`` oublie le marker,
|
| 12 |
+
elle s'exécute lors du ``pytest tests/`` standard et :
|
| 13 |
+
|
| 14 |
+
- échoue sur les runners sans la dep cloud → faux échec CI ;
|
| 15 |
+
- consomme du quota API (clé en CI = facture surprise) ;
|
| 16 |
+
- introduit une dépendance réseau non documentée.
|
| 17 |
+
|
| 18 |
+
L'agent d'audit avait flaggé ``test_tesseract_live.py`` comme
|
| 19 |
+
« skip top-level inconditionnel ». Vérification : le skip est en
|
| 20 |
+
fait **conditionnel** (``if shutil.which("tesseract") is None``),
|
| 21 |
+
ce qui est légitime — un test live qui peut s'exécuter seulement
|
| 22 |
+
si le binaire est présent. Mais le garde-fou ci-dessous évite
|
| 23 |
+
qu'une nouvelle fonction de test oublie le marker.
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import ast
|
| 29 |
+
from pathlib import Path
|
| 30 |
+
|
| 31 |
+
import pytest
|
| 32 |
+
|
| 33 |
+
LIVE_DIR = Path(__file__).resolve().parents[1] / "integration" / "live"
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _test_functions(path: Path) -> list[tuple[str, ast.FunctionDef | ast.AsyncFunctionDef]]:
|
| 37 |
+
"""Liste les fonctions ``test_*`` au top-level d'un fichier."""
|
| 38 |
+
tree = ast.parse(path.read_text(encoding="utf-8"))
|
| 39 |
+
out: list[tuple[str, ast.FunctionDef | ast.AsyncFunctionDef]] = []
|
| 40 |
+
for node in tree.body:
|
| 41 |
+
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name.startswith("test_"):
|
| 42 |
+
out.append((node.name, node))
|
| 43 |
+
return out
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _has_live_marker(fn: ast.FunctionDef | ast.AsyncFunctionDef) -> bool:
|
| 47 |
+
for deco in fn.decorator_list:
|
| 48 |
+
# ``@pytest.mark.live`` ou ``@pytest.mark.live(reason=...)``
|
| 49 |
+
if isinstance(deco, ast.Attribute) and deco.attr == "live":
|
| 50 |
+
return True
|
| 51 |
+
if isinstance(deco, ast.Call) and isinstance(deco.func, ast.Attribute) and deco.func.attr == "live":
|
| 52 |
+
return True
|
| 53 |
+
return False
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def _live_test_files() -> list[Path]:
|
| 57 |
+
if not LIVE_DIR.exists():
|
| 58 |
+
return []
|
| 59 |
+
return [
|
| 60 |
+
p for p in sorted(LIVE_DIR.glob("test_*.py"))
|
| 61 |
+
if p.name != "__init__.py" and p.name != "conftest.py"
|
| 62 |
+
]
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
@pytest.mark.parametrize("path", _live_test_files(), ids=lambda p: p.name)
|
| 66 |
+
def test_every_function_in_live_dir_has_live_marker(path: Path) -> None:
|
| 67 |
+
"""Chaque ``test_*`` dans ``tests/integration/live/`` porte ``@pytest.mark.live``.
|
| 68 |
+
|
| 69 |
+
Sinon le test peut s'exécuter en CI standard et casser sur
|
| 70 |
+
l'absence de clé API / binaire externe.
|
| 71 |
+
"""
|
| 72 |
+
missing: list[str] = []
|
| 73 |
+
for name, fn in _test_functions(path):
|
| 74 |
+
if not _has_live_marker(fn):
|
| 75 |
+
missing.append(f" {path.name}:{fn.lineno} :: {name}")
|
| 76 |
+
|
| 77 |
+
assert not missing, (
|
| 78 |
+
f"Fonctions dans {LIVE_DIR.name}/ sans ``@pytest.mark.live`` :\n"
|
| 79 |
+
+ "\n".join(missing)
|
| 80 |
+
+ "\n\nAjouter ``@pytest.mark.live`` au-dessus de chaque test "
|
| 81 |
+
"qui hit une API/un binaire externe — sinon le test "
|
| 82 |
+
"s'exécute sans opt-in et peut faire échouer le CI standard."
|
| 83 |
+
)
|
|
@@ -0,0 +1,217 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Phase 3.4 audit code-quality — la sur-normalisation LLM est
|
| 2 |
+
désormais agrégée automatiquement via le registre
|
| 3 |
+
:mod:`picarones.evaluation.metric_hooks`.
|
| 4 |
+
|
| 5 |
+
Avant la Phase 3.4, ``aggregate_over_normalization`` existait dans
|
| 6 |
+
``picarones/evaluation/metrics/over_normalization.py`` mais :
|
| 7 |
+
|
| 8 |
+
- n'avait aucun ``@register_corpus_aggregator`` ;
|
| 9 |
+
- le module n'était même pas importé par ``evaluation/metrics/__init__.py``
|
| 10 |
+
(mentionné en docstring uniquement) ;
|
| 11 |
+
- ``synthetic.py`` réimplémentait l'agrégation manuellement
|
| 12 |
+
(duplication silencieuse).
|
| 13 |
+
|
| 14 |
+
Le hook ``_aggregate_over_normalization_hook`` (auto-enregistré)
|
| 15 |
+
extrait désormais l'info depuis
|
| 16 |
+
``DocumentResult.pipeline_metadata["over_normalization"]`` et
|
| 17 |
+
alimente ``EngineReport.aggregated_over_normalization`` pour les
|
| 18 |
+
profils ``philological``, ``diagnostics`` et ``full``.
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
from picarones.evaluation.benchmark_result import DocumentResult, EngineReport
|
| 24 |
+
from picarones.evaluation.metric_hooks import (
|
| 25 |
+
PROFILE_DIAGNOSTICS,
|
| 26 |
+
PROFILE_FULL,
|
| 27 |
+
PROFILE_MINIMAL,
|
| 28 |
+
PROFILE_PHILOLOGICAL,
|
| 29 |
+
PROFILE_STANDARD,
|
| 30 |
+
_all_corpus_aggregator_names,
|
| 31 |
+
run_corpus_aggregators,
|
| 32 |
+
select_corpus_aggregators,
|
| 33 |
+
)
|
| 34 |
+
from picarones.evaluation.metric_result import MetricsResult
|
| 35 |
+
from picarones.evaluation.metrics.over_normalization import (
|
| 36 |
+
OverNormalizationResult,
|
| 37 |
+
aggregate_over_normalization,
|
| 38 |
+
)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
# --------------------------------------------------------------------------
|
| 42 |
+
# Auto-enregistrement
|
| 43 |
+
# --------------------------------------------------------------------------
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def test_over_normalization_aggregator_is_registered() -> None:
|
| 47 |
+
"""L'import de ``picarones.evaluation.metrics`` doit déclencher
|
| 48 |
+
l'enregistrement de l'agrégateur ``over_normalization``."""
|
| 49 |
+
import picarones.evaluation.metrics # noqa: F401 — déclenchement
|
| 50 |
+
|
| 51 |
+
assert "over_normalization" in _all_corpus_aggregator_names(), (
|
| 52 |
+
"Le hook ``_aggregate_over_normalization_hook`` n'est pas "
|
| 53 |
+
"enregistré. Vérifier que ``over_normalization`` est dans "
|
| 54 |
+
"``picarones/evaluation/metrics/__init__.py`` (Phase 3.4)."
|
| 55 |
+
)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def test_aggregator_in_correct_profiles() -> None:
|
| 59 |
+
"""L'agrégateur doit être actif pour ``philological``,
|
| 60 |
+
``diagnostics``, ``full`` — pas pour ``minimal`` ni ``standard``."""
|
| 61 |
+
import picarones.evaluation.metrics # noqa: F401
|
| 62 |
+
|
| 63 |
+
for profile in (PROFILE_PHILOLOGICAL, PROFILE_DIAGNOSTICS, PROFILE_FULL):
|
| 64 |
+
names = [a.name for a in select_corpus_aggregators(profile)]
|
| 65 |
+
assert "over_normalization" in names, (
|
| 66 |
+
f"Profil ``{profile}`` n'inclut pas l'agrégateur over_normalization."
|
| 67 |
+
)
|
| 68 |
+
|
| 69 |
+
for profile in (PROFILE_MINIMAL, PROFILE_STANDARD):
|
| 70 |
+
names = [a.name for a in select_corpus_aggregators(profile)]
|
| 71 |
+
assert "over_normalization" not in names, (
|
| 72 |
+
f"Profil ``{profile}`` ne devrait pas inclure over_normalization."
|
| 73 |
+
)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
# --------------------------------------------------------------------------
|
| 77 |
+
# Fonction pure aggregate_over_normalization (rétrocompat)
|
| 78 |
+
# --------------------------------------------------------------------------
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def test_pure_aggregate_empty_list_returns_zero() -> None:
|
| 82 |
+
"""Pas de docs → score None, compteurs à zéro (rétrocompat de la
|
| 83 |
+
fonction utilitaire pure)."""
|
| 84 |
+
out = aggregate_over_normalization([])
|
| 85 |
+
assert out == {
|
| 86 |
+
"score": None,
|
| 87 |
+
"total_correct_ocr_words": 0,
|
| 88 |
+
"over_normalized_count": 0,
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def test_pure_aggregate_sums_counts() -> None:
|
| 93 |
+
"""L'agrégation somme les compteurs bruts puis recalcule le score."""
|
| 94 |
+
r1 = OverNormalizationResult(
|
| 95 |
+
total_correct_ocr_words=100,
|
| 96 |
+
over_normalized_count=10,
|
| 97 |
+
)
|
| 98 |
+
r2 = OverNormalizationResult(
|
| 99 |
+
total_correct_ocr_words=50,
|
| 100 |
+
over_normalized_count=5,
|
| 101 |
+
)
|
| 102 |
+
out = aggregate_over_normalization([r1, r2, None]) # None ignoré
|
| 103 |
+
assert out == {
|
| 104 |
+
"score": 0.1, # 15 / 150
|
| 105 |
+
"total_correct_ocr_words": 150,
|
| 106 |
+
"over_normalized_count": 15,
|
| 107 |
+
"document_count": 2,
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
# --------------------------------------------------------------------------
|
| 112 |
+
# Hook décoré — extraction depuis DocumentResult.pipeline_metadata
|
| 113 |
+
# --------------------------------------------------------------------------
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def _make_dr(
|
| 117 |
+
doc_id: str,
|
| 118 |
+
over_norm_dict: dict | None,
|
| 119 |
+
) -> DocumentResult:
|
| 120 |
+
return DocumentResult(
|
| 121 |
+
doc_id=doc_id,
|
| 122 |
+
image_path=f"/tmp/{doc_id}.png",
|
| 123 |
+
ground_truth="fait",
|
| 124 |
+
hypothesis="fait",
|
| 125 |
+
metrics=MetricsResult(cer=0.0, wer=0.0),
|
| 126 |
+
duration_seconds=1.0,
|
| 127 |
+
ocr_intermediate="faict",
|
| 128 |
+
pipeline_metadata=(
|
| 129 |
+
{"over_normalization": over_norm_dict}
|
| 130 |
+
if over_norm_dict is not None
|
| 131 |
+
else {}
|
| 132 |
+
),
|
| 133 |
+
)
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def test_hook_returns_none_when_no_pipeline_metadata() -> None:
|
| 137 |
+
"""Benchmark OCR seul (sans LLM) → aucun ``pipeline_metadata``,
|
| 138 |
+
donc le hook retourne ``None`` et ``aggregated_over_normalization``
|
| 139 |
+
reste à ``None``."""
|
| 140 |
+
import picarones.evaluation.metrics # noqa: F401
|
| 141 |
+
|
| 142 |
+
docs = [_make_dr("d1", None), _make_dr("d2", None)]
|
| 143 |
+
out = run_corpus_aggregators(PROFILE_FULL, docs)
|
| 144 |
+
assert "aggregated_over_normalization" not in out
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def test_hook_aggregates_from_pipeline_metadata() -> None:
|
| 148 |
+
"""Pipeline OCR+LLM → ``pipeline_metadata["over_normalization"]``
|
| 149 |
+
est extrait et agrégé."""
|
| 150 |
+
import picarones.evaluation.metrics # noqa: F401
|
| 151 |
+
|
| 152 |
+
docs = [
|
| 153 |
+
_make_dr("d1", {
|
| 154 |
+
"score": 0.1,
|
| 155 |
+
"total_correct_ocr_words": 100,
|
| 156 |
+
"over_normalized_count": 10,
|
| 157 |
+
"over_normalized_passages": [],
|
| 158 |
+
}),
|
| 159 |
+
_make_dr("d2", {
|
| 160 |
+
"score": 0.2,
|
| 161 |
+
"total_correct_ocr_words": 50,
|
| 162 |
+
"over_normalized_count": 10,
|
| 163 |
+
"over_normalized_passages": [],
|
| 164 |
+
}),
|
| 165 |
+
]
|
| 166 |
+
out = run_corpus_aggregators(PROFILE_PHILOLOGICAL, docs)
|
| 167 |
+
assert "aggregated_over_normalization" in out
|
| 168 |
+
result = out["aggregated_over_normalization"]
|
| 169 |
+
# 20 over-normalized / 150 correct OCR = 0.1333
|
| 170 |
+
assert result["over_normalized_count"] == 20
|
| 171 |
+
assert result["total_correct_ocr_words"] == 150
|
| 172 |
+
assert result["document_count"] == 2
|
| 173 |
+
assert 0.13 < result["score"] < 0.14
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
def test_hook_resilient_to_malformed_dict() -> None:
|
| 177 |
+
"""Si un document a un ``pipeline_metadata["over_normalization"]``
|
| 178 |
+
mal formé (manque un champ, valeur non castable), il est skipé
|
| 179 |
+
avec un warning — l'agrégateur n'échoue pas."""
|
| 180 |
+
import picarones.evaluation.metrics # noqa: F401
|
| 181 |
+
|
| 182 |
+
docs = [
|
| 183 |
+
_make_dr("d1", {"total_correct_ocr_words": 100, "over_normalized_count": 5}),
|
| 184 |
+
_make_dr("d2", {"total_correct_ocr_words": "garbage", "over_normalized_count": 0}),
|
| 185 |
+
_make_dr("d3", None),
|
| 186 |
+
]
|
| 187 |
+
out = run_corpus_aggregators(PROFILE_FULL, docs)
|
| 188 |
+
# d1 est valide → l'agrégateur retourne un dict, même si d2 est ignoré
|
| 189 |
+
assert "aggregated_over_normalization" in out
|
| 190 |
+
assert out["aggregated_over_normalization"]["over_normalized_count"] == 5
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
# --------------------------------------------------------------------------
|
| 194 |
+
# Sérialisation EngineReport
|
| 195 |
+
# --------------------------------------------------------------------------
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def test_engine_report_round_trip_with_over_normalization() -> None:
|
| 199 |
+
"""Le champ ``aggregated_over_normalization`` est préservé par
|
| 200 |
+
``as_dict`` / ``from_dict``."""
|
| 201 |
+
er = EngineReport(
|
| 202 |
+
engine_name="tesseract+ministral",
|
| 203 |
+
engine_version="5.3.0",
|
| 204 |
+
engine_config={},
|
| 205 |
+
document_results=[],
|
| 206 |
+
aggregated_over_normalization={
|
| 207 |
+
"score": 0.15,
|
| 208 |
+
"total_correct_ocr_words": 200,
|
| 209 |
+
"over_normalized_count": 30,
|
| 210 |
+
"document_count": 5,
|
| 211 |
+
},
|
| 212 |
+
)
|
| 213 |
+
d = er.as_dict()
|
| 214 |
+
assert d["aggregated_over_normalization"]["score"] == 0.15
|
| 215 |
+
|
| 216 |
+
rebuilt = EngineReport.from_dict(d)
|
| 217 |
+
assert rebuilt.aggregated_over_normalization == er.aggregated_over_normalization
|
|
@@ -0,0 +1,196 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Phase 3.2 audit code-quality — end-to-end du journal de fallback.
|
| 2 |
+
|
| 3 |
+
Vérifie que la chaîne complète fonctionne :
|
| 4 |
+
|
| 5 |
+
1. Un importer (HTR-United) dégrade en mode démo →
|
| 6 |
+
``record_fallback`` côté importer.
|
| 7 |
+
2. Le runner consomme via ``consume_fallback_log()`` et stocke dans
|
| 8 |
+
``BenchmarkResult.metadata["importer_fallbacks"]``.
|
| 9 |
+
3. ``build_report_data`` propage la liste dans
|
| 10 |
+
``report_data["importer_fallbacks"]``.
|
| 11 |
+
4. Le détecteur narratif ``detect_importer_fallback`` (history.py:280)
|
| 12 |
+
produit un ``Fact(FactType.IMPORTER_FALLBACK_TRIGGERED, ...)``.
|
| 13 |
+
5. ``build_synthesis`` rend une phrase qui mentionne l'incident.
|
| 14 |
+
|
| 15 |
+
Avant la Phase 3.2 : étapes 2-3 manquaient — le détecteur ne
|
| 16 |
+
recevait jamais de données malgré l'API ``_fallback_log`` câblée
|
| 17 |
+
côté importer.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
from __future__ import annotations
|
| 21 |
+
|
| 22 |
+
import pytest
|
| 23 |
+
|
| 24 |
+
from picarones.adapters.corpus._fallback_log import (
|
| 25 |
+
consume_fallback_log,
|
| 26 |
+
peek_fallback_log,
|
| 27 |
+
record_fallback,
|
| 28 |
+
reset_fallback_log,
|
| 29 |
+
)
|
| 30 |
+
from picarones.domain.facts import FactType
|
| 31 |
+
from picarones.evaluation.benchmark_result import BenchmarkResult
|
| 32 |
+
from picarones.reports.html.data import build_report_data
|
| 33 |
+
from picarones.reports.narrative import build_synthesis
|
| 34 |
+
from picarones.reports.narrative.detectors.history import detect_importer_fallback
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
@pytest.fixture(autouse=True)
|
| 38 |
+
def _clean_fallback_log() -> None:
|
| 39 |
+
"""Le journal est un singleton thread-safe — on le vide avant
|
| 40 |
+
et après chaque test pour éviter les contaminations croisées."""
|
| 41 |
+
reset_fallback_log()
|
| 42 |
+
yield
|
| 43 |
+
reset_fallback_log()
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
# --------------------------------------------------------------------------
|
| 47 |
+
# Étape 1 : record_fallback est appelable + sérialise correctement
|
| 48 |
+
# --------------------------------------------------------------------------
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def test_record_fallback_appends_entry() -> None:
|
| 52 |
+
record_fallback(
|
| 53 |
+
importer="htr_united",
|
| 54 |
+
operation="catalogue_remote_fetch",
|
| 55 |
+
error=RuntimeError("DNS timeout"),
|
| 56 |
+
extra={"url": "https://example.org/cat.yml"},
|
| 57 |
+
)
|
| 58 |
+
entries = peek_fallback_log()
|
| 59 |
+
assert len(entries) == 1
|
| 60 |
+
assert entries[0]["importer"] == "htr_united"
|
| 61 |
+
assert entries[0]["operation"] == "catalogue_remote_fetch"
|
| 62 |
+
assert "DNS timeout" in entries[0]["error"]
|
| 63 |
+
assert entries[0]["extra"]["url"] == "https://example.org/cat.yml"
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def test_htr_united_fallback_records_entry(monkeypatch: pytest.MonkeyPatch) -> None:
|
| 67 |
+
"""``HTRUnitedCatalogue.from_remote`` doit appeler ``record_fallback``
|
| 68 |
+
quand le réseau échoue (régression : avant Phase 3.2 le warning
|
| 69 |
+
log était là, le record manquait)."""
|
| 70 |
+
import urllib.error
|
| 71 |
+
|
| 72 |
+
from picarones.adapters.corpus.htr_united import HTRUnitedCatalogue
|
| 73 |
+
|
| 74 |
+
def _boom(*_a, **_kw):
|
| 75 |
+
raise urllib.error.URLError("simulated DNS failure")
|
| 76 |
+
|
| 77 |
+
monkeypatch.setattr(
|
| 78 |
+
"picarones.adapters.corpus.htr_united.urllib.request.urlopen",
|
| 79 |
+
_boom,
|
| 80 |
+
)
|
| 81 |
+
cat = HTRUnitedCatalogue.from_remote(timeout=1)
|
| 82 |
+
assert cat.source == "demo" # fallback effectif
|
| 83 |
+
|
| 84 |
+
entries = peek_fallback_log()
|
| 85 |
+
assert len(entries) == 1
|
| 86 |
+
assert entries[0]["importer"] == "htr_united"
|
| 87 |
+
assert entries[0]["operation"] == "catalogue_remote_fetch"
|
| 88 |
+
assert entries[0]["extra"]["fallback_used"] == "demo"
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
# --------------------------------------------------------------------------
|
| 92 |
+
# Étape 4 : le détecteur narratif émet un Fact à partir de la liste
|
| 93 |
+
# --------------------------------------------------------------------------
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def test_detector_emits_fact_from_benchmark_data() -> None:
|
| 97 |
+
benchmark_data = {
|
| 98 |
+
"importer_fallbacks": [
|
| 99 |
+
{
|
| 100 |
+
"importer": "htr_united",
|
| 101 |
+
"operation": "catalogue_remote_fetch",
|
| 102 |
+
"error": "URLError(...)",
|
| 103 |
+
"extra": {"fallback_used": "demo"},
|
| 104 |
+
},
|
| 105 |
+
],
|
| 106 |
+
}
|
| 107 |
+
facts = detect_importer_fallback(benchmark_data)
|
| 108 |
+
assert len(facts) == 1
|
| 109 |
+
assert facts[0].type is FactType.IMPORTER_FALLBACK_TRIGGERED
|
| 110 |
+
assert facts[0].payload["importer"] == "htr_united"
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def test_detector_silent_when_no_fallback() -> None:
|
| 114 |
+
"""Pas de clé → pas de Fact."""
|
| 115 |
+
assert detect_importer_fallback({}) == []
|
| 116 |
+
assert detect_importer_fallback({"importer_fallbacks": []}) == []
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
# --------------------------------------------------------------------------
|
| 120 |
+
# Étape 3 : build_report_data propage metadata.importer_fallbacks
|
| 121 |
+
# --------------------------------------------------------------------------
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def _empty_benchmark_with_metadata(metadata: dict) -> BenchmarkResult:
|
| 125 |
+
"""Benchmark sans engine (suffisant pour tester la propagation
|
| 126 |
+
de ``metadata.importer_fallbacks`` vers ``report_data``)."""
|
| 127 |
+
return BenchmarkResult(
|
| 128 |
+
corpus_name="t",
|
| 129 |
+
corpus_source=None,
|
| 130 |
+
document_count=0,
|
| 131 |
+
engine_reports=[],
|
| 132 |
+
metadata=metadata,
|
| 133 |
+
)
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def test_build_report_data_propagates_fallbacks() -> None:
|
| 137 |
+
bench = _empty_benchmark_with_metadata({
|
| 138 |
+
"importer_fallbacks": [
|
| 139 |
+
{"importer": "htr_united", "operation": "catalogue_remote_fetch",
|
| 140 |
+
"error": "URLError(timeout)"},
|
| 141 |
+
],
|
| 142 |
+
})
|
| 143 |
+
data = build_report_data(bench, images_b64={})
|
| 144 |
+
assert "importer_fallbacks" in data
|
| 145 |
+
assert len(data["importer_fallbacks"]) == 1
|
| 146 |
+
assert data["importer_fallbacks"][0]["importer"] == "htr_united"
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def test_build_report_data_empty_when_no_fallback() -> None:
|
| 150 |
+
bench = _empty_benchmark_with_metadata({})
|
| 151 |
+
data = build_report_data(bench, images_b64={})
|
| 152 |
+
assert data["importer_fallbacks"] == []
|
| 153 |
+
|
| 154 |
+
|
| 155 |
+
# --------------------------------------------------------------------------
|
| 156 |
+
# Étape 5 : build_synthesis fait remonter l'incident dans la prose
|
| 157 |
+
# --------------------------------------------------------------------------
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def test_build_synthesis_mentions_fallback_in_french() -> None:
|
| 161 |
+
"""La synthèse française doit produire au moins un fragment
|
| 162 |
+
textuel qui mentionne l'importer en mode dégradé."""
|
| 163 |
+
data = {
|
| 164 |
+
"engines": [],
|
| 165 |
+
"ranking": [],
|
| 166 |
+
"importer_fallbacks": [
|
| 167 |
+
{
|
| 168 |
+
"importer": "htr_united",
|
| 169 |
+
"operation": "catalogue_remote_fetch",
|
| 170 |
+
"error": "URLError(timeout)",
|
| 171 |
+
"extra": {"fallback_used": "demo"},
|
| 172 |
+
},
|
| 173 |
+
],
|
| 174 |
+
}
|
| 175 |
+
out = build_synthesis(data, lang="fr", max_facts=5)
|
| 176 |
+
# Le texte rendu doit contenir au moins le nom de l'importer.
|
| 177 |
+
rendered = " ".join(out.get("paragraphs", []) or []) + " " + str(out)
|
| 178 |
+
assert "htr_united" in rendered.lower() or "htr-united" in rendered.lower(), (
|
| 179 |
+
f"La synthèse FR ne mentionne pas l'importer HTR-United malgré "
|
| 180 |
+
f"un fallback enregistré. Sortie : {out!r}"
|
| 181 |
+
)
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
# --------------------------------------------------------------------------
|
| 185 |
+
# Étape 2 : consume vide bien la liste (anti-contamination cross-run)
|
| 186 |
+
# --------------------------------------------------------------------------
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def test_consume_clears_the_log() -> None:
|
| 190 |
+
record_fallback(importer="a", operation="x")
|
| 191 |
+
record_fallback(importer="b", operation="y")
|
| 192 |
+
first = consume_fallback_log()
|
| 193 |
+
assert len(first) == 2
|
| 194 |
+
|
| 195 |
+
second = consume_fallback_log()
|
| 196 |
+
assert second == [] # vidé par le premier consume
|