docs/evals/m1_classifier_report.md

# M1 — CEFR classifier: evaluation report

**Date:** 2026-06-12 · **Tracking:** MLflow (`sqlite:///mlflow.db`), experiment `cefr-classifier`
**Decision:** production config = **`en_chunked_weighted`** (arm 1). Rationale below.

## Setup

Policies, data mix and experiment arms are defined in
[ADR 0003](../adr/0003-datasets-and-licensing.md); pipeline in `training/`
(PR `feat(ml): CEFR dataset builder and XLM-R training pipeline`).

- Base model: `xlm-roberta-base`, fine-tuned 4 epochs, class-weighted CE,
  fp16, effective batch 16, RTX 3070 (~10–27 min/run).
- Split: document-level, stratified by corpus × level, seed 13. **Per-stratum
  seeding makes the English test documents identical across every arm** (unit
  test: `test_arm_comparability_*`), so all rows below are directly comparable.
- EN test set: **99 documents** (31 cambridge_exams + 68 elg_cefr; supports
  A1:2, A2:20, B1:26, B2:24, C1:15, C2:12) and **1,279 sentences**
  (cefr_sp + readme).
- Headline view: document-level (the product use case: recommending a reading
  text at the learner's level), chunk probabilities aggregated per document by
  expected-rank mean.

Statistical honesty up front: at n=99, the 95% CI on accuracy is roughly
±0.09; per-corpus reads (n=31, n=68) are directional only; A1 (2 docs) and C2
(12 docs) document-level F1 values are anecdotal.

## Headline: EN document-level (n=99)

| run | accuracy | adjacent | macro-F1 | QWK | f1_A1² | f1_B1 | f1_C1 | f1_C2 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| **en_chunked_weighted** | **0.788** | 0.990 | **0.798** | **0.936** | 0.800 | 0.634 | 0.800 | 0.909 |
| multilingual | 0.717 | 0.990 | 0.681 | 0.918 | 0.500 | 0.605 | 0.545 | 0.923 |
| en_truncated | 0.646 | 0.990 | 0.684 | 0.892 | 1.000 | 0.537 | 0.320 | 0.800 |
| audit_no_cambridge | 0.707 | 0.990 | 0.665 | 0.903 | 0.667 | 0.727 | 0.485 | 0.500 |
| audit_no_elg | 0.576 | 0.980 | 0.475 | 0.851 | 0.000 | 0.524 | 0.424 | 0.600 |
| baseline¹ (our chunk pipeline) | 0.566 | 0.970 | 0.403 | 0.824 | 0.000 | 0.681 | 0.390 | 0.000 |
| baseline¹ (native, truncated docs) | 0.687 | 0.990 | 0.579 | 0.898 | 0.000 | 0.681 | 0.571 | 0.762 |

¹ `UniversalCEFR/xlm-roberta-base-cefr-all-classifier`. ² A1 support = 2 docs.

## EN sentence-level (n=1,279)

| run | accuracy | adjacent | macro-F1 | QWK | f1_A1 (n=30) |
|---|---:|---:|---:|---:|---:|
| en_chunked_weighted | 0.622 | 0.983 | 0.611 | 0.820 | 0.492 |
| multilingual | 0.603 | 0.981 | 0.624 | 0.813 | **0.563** |
| en_truncated | 0.608 | 0.980 | 0.594 | 0.811 | 0.459 |
| audit_no_cambridge | 0.625 | 0.984 | 0.618 | 0.821 | 0.535 |
| audit_no_elg | 0.618 | 0.981 | 0.623 | 0.818 | 0.507 |
| baseline¹ | **0.815** | 0.991 | **0.775** | **0.901** | 0.735 |

## Findings

### 1. Chunk-aggregate beats truncation — the ablation that justifies the architecture

Same data, same test documents, only the length policy differs:
**+0.14 accuracy / +0.11 macro-F1 / +0.044 QWK** for chunk-aggregate over
truncate-at-512. The damage is exactly where ADR 0003 predicted: long, high-level
cambridge texts (truncated f1_C1 collapses to 0.32; the truncated f1_A1 = 1.0 is
the 2-doc anecdote — A1 documents are short, truncation cannot hurt them).
Expected-rank aggregation also absorbs chunk-level noise: passage-level accuracy
is ~0.62 while document-level reaches 0.79.

### 2. Multilingual training: A1 hypothesis half-confirmed, headline cost too high

The Welsh hypothesis (764 A1 short texts lifting A1 recall) shows in the
predicted direction at sentence level (f1_A1 0.563 vs 0.492) but on only 30
positives — weak evidence. Meanwhile the EN document headline drops by 0.12
macro-F1 / 0.07 accuracy: capacity spreads across 13 languages and the class
weights now reflect the multilingual distribution. **Verdict: not worth it for
the English-only product.** The run is kept as the reference point for future
target languages: across all 13 languages it reaches document macro-F1 0.568 /
QWK 0.762 (n=648 docs) — the starting line when a second language ships.

### 3. Generalization audits: real domain shift, gracefully absorbed

Mixed-test reads first (excluded corpus = 31/99 or 68/99 of the test):
removing cambridge from training costs −0.13 macro-F1 on the mixed test;
removing elg costs −0.32 and zeroes A1 (elg is the *only* source of A1
documents). That apparent asymmetry is mostly a **composition effect** — elg
is 69% of the document test and the sole A1 provider. Per-corpus pure reads
(same test documents in every run, thanks to per-stratum seeding, via the
`eval_*_only` configs; accuracy / adjacent / QWK — macro-F1 omitted: 6-way
macro is misleading on single corpora with absent levels) show a shift of
comparable magnitude on both sides:

| corpus (test docs) | en_chunked_weighted (in-domain) | audit model (corpus never seen) |
|---|---|---|
| cambridge (31) | 0.871 / 1.000 / 0.966 | 0.677 / 1.000 / 0.900 |
| elg (68) | 0.750 / 0.985 / 0.919 | 0.588 / 0.985 / 0.850 |

Reading: an unseen source costs **16–19 points of exact accuracy** (n=31/68 →
directional), while **adjacent accuracy is untouched** (1.000 and 0.985,
identical in- and out-of-domain) and QWK stays ≥ 0.85: degradation is graceful —
errors move from exact to off-by-one, never further. Consistency check: the two
in-domain reads recompose the headline exactly ((31×0.871 + 68×0.750)/99 = 0.788).

Production consequence: both corpora stay in training, and on authentic
unseen-source texts the model should be trusted at **level ± 1** rather than
exact level — which still supports the recommendation use case.

### 4. The published baseline: the decisive margin is the system, not the encoder

Two-sided, two-staged result. At sentence level the baseline dominates (0.815
vs 0.622 accuracy) — but it was, to the best of our knowledge, trained on the
full UniversalCEFR collection, which almost certainly includes our test
sentences. It is an **advantaged baseline**: "better model" and "saw the test
data" cannot be separated, so sentence-level numbers are reported without a
winner declared.

At the document level: through *our* chunk-aggregate pipeline the baseline
collapses (macro-F1 0.403; f1_A1 = f1_C2 = 0.000) — its chunk-level probability
distributions, never trained for this regime, are flat enough that the
expected-rank mean pulls everything toward the center. In its **native regime**
(full documents, tokenizer-truncated) it recovers substantially — macro-F1
0.579, f1_C2 back to 0.762 — confirming that part of the collapse was an
evaluation-regime artifact, which we report rather than hide. But even at its
native best, the advantaged baseline stays well below the production system
(0.687 vs 0.788 accuracy, 0.579 vs 0.798 macro-F1, 0.898 vs 0.936 QWK), and is
merely on par with our own truncated arm (0.687/0.579 vs 0.646/0.684 acc/macro
— mixed). **The decisive margin is therefore not the encoder; it is the
chunk-aggregate system fine-tuned end-to-end**, a regime the baseline cannot
exploit. Bottom line: on the product task, our system outperforms the published
baseline by **+0.22 macro-F1**, each system evaluated in its best regime, with
the baseline likely advantaged by train/test overlap.

## Decision

**Ship `en_chunked_weighted`** (`models/cefr/en_chunked_weighted/`): best
headline on the product metric (macro-F1 0.798, QWK 0.936, adjacent 0.990 —
1/99 documents off by more than one level), architecture validated by ablation,
preprocessing parameters frozen in `preprocessing.json` next to the weights.
**Deployment parity (int8 ONNX service).** The deployed service (ONNX int8,
service-side chunking) re-scored on the same 99 test documents reaches
accuracy 0.768 / macro-F1 0.753 / QWK 0.927, with identical adjacent accuracy
(0.990) — torch reference: 0.788 / 0.798 / 0.936. Two documents flip under
quantization; the macro-F1 move is amplified by tiny per-level supports
(A1 = 2 docs). Export faithfulness gate: max |fp32 − torch| logit diff =
1.4e-05 (legacy tracer + eager attention) — the int8 delta is the quantization
itself, not the export. Artifact: 279 MB (from 1,112 MB), published at
huggingface.co/blizzarman/polyglot-tutor-cefr-onnx.

Deferred, recorded: multilingual checkpoint for future target languages;
ordinal head (CORAL/CORN, ADR 0003 arm 4) as a possible follow-up — QWK 0.936
leaves limited headroom, so it is not on the critical path.

## Limitations

- n=99 test documents (±~0.09 CI); A1 = 2 docs, C2 = 12 docs at document level.
- B1 remains the weakest mid class (f1 0.634); errors are off-by-one at the
  notoriously fuzzy B1/B2 boundary (adjacent accuracy 0.990).
- Chunk labels are weak labels (inherited from the document) — accepted,
  documented, and absorbed by aggregation, but passage-level metrics
  underestimate the model for that reason.
- Reference corpora are exam/pedagogical texts; authentic web texts are
  out-of-domain — expect the audit-sized cost (−16–19 points of exact accuracy,
  adjacent accuracy intact).
- All training data is research/non-commercial licensed (ADR 0003); the
  fine-tuned checkpoint inherits CC BY-NC-SA terms if ever published.

## Reproducibility

```bash
uv run --group train python training/train_cefr.py --config training/configs/<arm>.toml
uv run --group train python training/eval_cefr.py --model <checkpoint|hub-id> --config training/configs/<cfg>.toml
uv run --group train mlflow ui --backend-store-uri sqlite:///mlflow.db
```

Seed 13 everywhere; arms: `en_only`, `multilingual`, `en_truncated`,
`audit_no_cambridge`, `audit_no_elg`; per-corpus reads: `eval_cambridge_only`,
`eval_elg_only`; baseline native mode: `eval_cefr.py` with the `en_truncated`
config.