docs(evals): M1 classifier report, per-corpus OOD reads and production decision (#3)

## What & why
Consolidates the M1 training campaign (3 arms + 2 generalization audits
+
published baseline, all on identical EN test documents) into the
evaluation
report, and records the production decision: en_chunked_weighted.

## Changes
- docs/evals/m1_classifier_report.md: comparison tables, ablation
reading,
multilingual verdict, per-corpus domain-shift quantification,
advantaged-
baseline analysis, limitations
- training/configs/eval_{cambridge,elg}_only.toml: evaluation-only
configs for
pure per-corpus OOD reads (same test docs via per-stratum seeding)
- docs/adr/0003: production decision recorded

## Headline
Doc-level (n=99): macro-F1 0.798 / QWK 0.936 / adjacent 0.990 —
chunk-aggregate
beats truncation by +0.14 accuracy. Domain shift quantified per corpus:
−16–19 pts exact accuracy on an unseen source, adjacent accuracy
untouched
(graceful degradation). The published baseline (likely advantaged by
train/test
overlap) reaches 0.403 macro-F1 through our pipeline and 0.579 in its
native
regime — still −0.22 below our system: the margin is the chunk-aggregate
system, not the encoder.

## How to test
Numbers reproducible via training/README.md (seed 13); reads from
MLflow.

docs/adr/0003-datasets-and-licensing.md

@@ -66,3 +66,5 @@ the symmetric run for elg (domain-shift measurement, not the production split).
 **Known limitations (mirrored in README):** only 24 A1 documents → per-level
 metrics reported, weak A1 recall expected at doc level; product mitigation: A1
 recommendations lean on short texts, which is pedagogically appropriate.
+
+**Production decision (2026-06-12):** en_chunked_weighted — see docs/evals/m1_classifier_report.md

docs/evals/m1_classifier_report.md

@@ -0,0 +1,165 @@
+# M1 — CEFR classifier: evaluation report
+
+**Date:** 2026-06-12 · **Tracking:** MLflow (`sqlite:///mlflow.db`), experiment `cefr-classifier`
+**Decision:** production config = **`en_chunked_weighted`** (arm 1). Rationale below.
+
+## Setup
+
+Policies, data mix and experiment arms are defined in
+[ADR 0003](../adr/0003-datasets-and-licensing.md); pipeline in `training/`
+(PR `feat(ml): CEFR dataset builder and XLM-R training pipeline`).
+
+- Base model: `xlm-roberta-base`, fine-tuned 4 epochs, class-weighted CE,
+  fp16, effective batch 16, RTX 3070 (~10–27 min/run).
+- Split: document-level, stratified by corpus × level, seed 13. **Per-stratum
+  seeding makes the English test documents identical across every arm** (unit
+  test: `test_arm_comparability_*`), so all rows below are directly comparable.
+- EN test set: **99 documents** (31 cambridge_exams + 68 elg_cefr; supports
+  A1:2, A2:20, B1:26, B2:24, C1:15, C2:12) and **1,279 sentences**
+  (cefr_sp + readme).
+- Headline view: document-level (the product use case: recommending a reading
+  text at the learner's level), chunk probabilities aggregated per document by
+  expected-rank mean.
+
+Statistical honesty up front: at n=99, the 95% CI on accuracy is roughly
+±0.09; per-corpus reads (n=31, n=68) are directional only; A1 (2 docs) and C2
+(12 docs) document-level F1 values are anecdotal.
+
+## Headline: EN document-level (n=99)
+
+| run | accuracy | adjacent | macro-F1 | QWK | f1_A1² | f1_B1 | f1_C1 | f1_C2 |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| **en_chunked_weighted** | **0.788** | 0.990 | **0.798** | **0.936** | 0.800 | 0.634 | 0.800 | 0.909 |
+| multilingual | 0.717 | 0.990 | 0.681 | 0.918 | 0.500 | 0.605 | 0.545 | 0.923 |
+| en_truncated | 0.646 | 0.990 | 0.684 | 0.892 | 1.000 | 0.537 | 0.320 | 0.800 |
+| audit_no_cambridge | 0.707 | 0.990 | 0.665 | 0.903 | 0.667 | 0.727 | 0.485 | 0.500 |
+| audit_no_elg | 0.576 | 0.980 | 0.475 | 0.851 | 0.000 | 0.524 | 0.424 | 0.600 |
+| baseline¹ (our chunk pipeline) | 0.566 | 0.970 | 0.403 | 0.824 | 0.000 | 0.681 | 0.390 | 0.000 |
+| baseline¹ (native, truncated docs) | 0.687 | 0.990 | 0.579 | 0.898 | 0.000 | 0.681 | 0.571 | 0.762 |
+
+¹ `UniversalCEFR/xlm-roberta-base-cefr-all-classifier`. ² A1 support = 2 docs.
+
+## EN sentence-level (n=1,279)
+
+| run | accuracy | adjacent | macro-F1 | QWK | f1_A1 (n=30) |
+|---|---:|---:|---:|---:|---:|
+| en_chunked_weighted | 0.622 | 0.983 | 0.611 | 0.820 | 0.492 |
+| multilingual | 0.603 | 0.981 | 0.624 | 0.813 | **0.563** |
+| en_truncated | 0.608 | 0.980 | 0.594 | 0.811 | 0.459 |
+| audit_no_cambridge | 0.625 | 0.984 | 0.618 | 0.821 | 0.535 |
+| audit_no_elg | 0.618 | 0.981 | 0.623 | 0.818 | 0.507 |
+| baseline¹ | **0.815** | 0.991 | **0.775** | **0.901** | 0.735 |
+
+## Findings
+
+### 1. Chunk-aggregate beats truncation — the ablation that justifies the architecture
+
+Same data, same test documents, only the length policy differs:
+**+0.14 accuracy / +0.11 macro-F1 / +0.044 QWK** for chunk-aggregate over
+truncate-at-512. The damage is exactly where ADR 0003 predicted: long, high-level
+cambridge texts (truncated f1_C1 collapses to 0.32; the truncated f1_A1 = 1.0 is
+the 2-doc anecdote — A1 documents are short, truncation cannot hurt them).
+Expected-rank aggregation also absorbs chunk-level noise: passage-level accuracy
+is ~0.62 while document-level reaches 0.79.
+
+### 2. Multilingual training: A1 hypothesis half-confirmed, headline cost too high
+
+The Welsh hypothesis (764 A1 short texts lifting A1 recall) shows in the
+predicted direction at sentence level (f1_A1 0.563 vs 0.492) but on only 30
+positives — weak evidence. Meanwhile the EN document headline drops by 0.12
+macro-F1 / 0.07 accuracy: capacity spreads across 13 languages and the class
+weights now reflect the multilingual distribution. **Verdict: not worth it for
+the English-only product.** The run is kept as the reference point for future
+target languages: across all 13 languages it reaches document macro-F1 0.568 /
+QWK 0.762 (n=648 docs) — the starting line when a second language ships.
+
+### 3. Generalization audits: real domain shift, gracefully absorbed
+
+Mixed-test reads first (excluded corpus = 31/99 or 68/99 of the test):
+removing cambridge from training costs −0.13 macro-F1 on the mixed test;
+removing elg costs −0.32 and zeroes A1 (elg is the *only* source of A1
+documents). That apparent asymmetry is mostly a **composition effect** — elg
+is 69% of the document test and the sole A1 provider. Per-corpus pure reads
+(same test documents in every run, thanks to per-stratum seeding, via the
+`eval_*_only` configs; accuracy / adjacent / QWK — macro-F1 omitted: 6-way
+macro is misleading on single corpora with absent levels) show a shift of
+comparable magnitude on both sides:
+
+| corpus (test docs) | en_chunked_weighted (in-domain) | audit model (corpus never seen) |
+|---|---|---|
+| cambridge (31) | 0.871 / 1.000 / 0.966 | 0.677 / 1.000 / 0.900 |
+| elg (68) | 0.750 / 0.985 / 0.919 | 0.588 / 0.985 / 0.850 |
+
+Reading: an unseen source costs **16–19 points of exact accuracy** (n=31/68 →
+directional), while **adjacent accuracy is untouched** (1.000 and 0.985,
+identical in- and out-of-domain) and QWK stays ≥ 0.85: degradation is graceful —
+errors move from exact to off-by-one, never further. Consistency check: the two
+in-domain reads recompose the headline exactly ((31×0.871 + 68×0.750)/99 = 0.788).
+
+Production consequence: both corpora stay in training, and on authentic
+unseen-source texts the model should be trusted at **level ± 1** rather than
+exact level — which still supports the recommendation use case.
+
+### 4. The published baseline: the decisive margin is the system, not the encoder
+
+Two-sided, two-staged result. At sentence level the baseline dominates (0.815
+vs 0.622 accuracy) — but it was, to the best of our knowledge, trained on the
+full UniversalCEFR collection, which almost certainly includes our test
+sentences. It is an **advantaged baseline**: "better model" and "saw the test
+data" cannot be separated, so sentence-level numbers are reported without a
+winner declared.
+
+At the document level: through *our* chunk-aggregate pipeline the baseline
+collapses (macro-F1 0.403; f1_A1 = f1_C2 = 0.000) — its chunk-level probability
+distributions, never trained for this regime, are flat enough that the
+expected-rank mean pulls everything toward the center. In its **native regime**
+(full documents, tokenizer-truncated) it recovers substantially — macro-F1
+0.579, f1_C2 back to 0.762 — confirming that part of the collapse was an
+evaluation-regime artifact, which we report rather than hide. But even at its
+native best, the advantaged baseline stays well below the production system
+(0.687 vs 0.788 accuracy, 0.579 vs 0.798 macro-F1, 0.898 vs 0.936 QWK), and is
+merely on par with our own truncated arm (0.687/0.579 vs 0.646/0.684 acc/macro
+— mixed). **The decisive margin is therefore not the encoder; it is the
+chunk-aggregate system fine-tuned end-to-end**, a regime the baseline cannot
+exploit. Bottom line: on the product task, our system outperforms the published
+baseline by **+0.22 macro-F1**, each system evaluated in its best regime, with
+the baseline likely advantaged by train/test overlap.
+
+## Decision
+
+**Ship `en_chunked_weighted`** (`models/cefr/en_chunked_weighted/`): best
+headline on the product metric (macro-F1 0.798, QWK 0.936, adjacent 0.990 —
+1/99 documents off by more than one level), architecture validated by ablation,
+preprocessing parameters frozen in `preprocessing.json` next to the weights.
+Next step (separate PR): ONNX int8 export + CPU inference service + Reading tab.
+
+Deferred, recorded: multilingual checkpoint for future target languages;
+ordinal head (CORAL/CORN, ADR 0003 arm 4) as a possible follow-up — QWK 0.936
+leaves limited headroom, so it is not on the critical path.
+
+## Limitations
+
+- n=99 test documents (±~0.09 CI); A1 = 2 docs, C2 = 12 docs at document level.
+- B1 remains the weakest mid class (f1 0.634); errors are off-by-one at the
+  notoriously fuzzy B1/B2 boundary (adjacent accuracy 0.990).
+- Chunk labels are weak labels (inherited from the document) — accepted,
+  documented, and absorbed by aggregation, but passage-level metrics
+  underestimate the model for that reason.
+- Reference corpora are exam/pedagogical texts; authentic web texts are
+  out-of-domain — expect the audit-sized cost (−16–19 points of exact accuracy,
+  adjacent accuracy intact).
+- All training data is research/non-commercial licensed (ADR 0003); the
+  fine-tuned checkpoint inherits CC BY-NC-SA terms if ever published.
+
+## Reproducibility
+
+```bash
+uv run --group train python training/train_cefr.py --config training/configs/<arm>.toml
+uv run --group train python training/eval_cefr.py --model <checkpoint|hub-id> --config training/configs/<cfg>.toml
+uv run --group train mlflow ui --backend-store-uri sqlite:///mlflow.db
+```
+
+Seed 13 everywhere; arms: `en_only`, `multilingual`, `en_truncated`,
+`audit_no_cambridge`, `audit_no_elg`; per-corpus reads: `eval_cambridge_only`,
+`eval_elg_only`; baseline native mode: `eval_cefr.py` with the `en_truncated`
+config.

training/configs/eval_cambridge_only.toml

@@ -0,0 +1,23 @@
+# Evaluation-only config: the cambridge test split in isolation, for per-corpus
+# OOD reads. Same seed => exactly the same 31 cambridge test documents as every
+# training arm (per-stratum seeding). Never used for training.
+
+[run]
+name = "eval_cambridge_only"
+seed = 13
+
+[tracking]
+uri = "sqlite:///mlflow.db"
+experiment = "cefr-classifier"
+
+[data]
+subsets = ["UniversalCEFR/cambridge_exams_en"]
+chunking = true
+target_words = 200
+max_words = 300
+ratios = [0.8, 0.1, 0.1]
+exclude_corpora_from_train = []
+
+[model]
+base = "xlm-roberta-base"
+max_length = 512

training/configs/eval_elg_only.toml

@@ -0,0 +1,23 @@
+# Evaluation-only config: the elg test split in isolation, for per-corpus
+# OOD reads. Same seed => exactly the same 68 elg test documents as every
+# training arm (per-stratum seeding). Never used for training.
+
+[run]
+name = "eval_elg_only"
+seed = 13
+
+[tracking]
+uri = "sqlite:///mlflow.db"
+experiment = "cefr-classifier"
+
+[data]
+subsets = ["UniversalCEFR/elg_cefr_en"]
+chunking = true
+target_words = 200
+max_words = 300
+ratios = [0.8, 0.1, 0.1]
+exclude_corpora_from_train = []
+
+[model]
+base = "xlm-roberta-base"
+max_length = 512