card: state the held-out calibration error, the shipped 0.0054 is in-sample

README.md

@@ -42,15 +42,23 @@ category, not a claim of compatibility or endorsement.
 
 ## Results (development)
 
-Measured on the held-out development split (17,501 items). Calibration is the
-point of this model, and the expected calibration error is low.
+Measured on the development split (17,501 items), which is held out from
+training. Calibration is the point of this model, and the expected calibration
+error is low.
 
 | Metric | Value |
 | --- | --- |
-| ECE (calibration error) | 0.0054 |
+| ECE, temperature fitted on this split | 0.0054 |
+| ECE, temperature fitted on a disjoint half | 0.0089 |
 | Temperature | 1.82 |
 | Overall accuracy (micro) | 0.824 |
 
+Two ECE figures, because the number depends on where the temperature came from.
+The shipped temperature was fitted on this same split, so 0.0054 is an in-sample
+figure and optimistic. Fitting the temperature on one half and measuring on the
+other gives 0.0089, which is what to expect on new data of the same kind. Both
+are reproducible from the code repository; the second is the one to quote.
+
 Per task:
 
 | Task | Accuracy | n |