A small real-world calibration check on Laya typed-decisions
I ran convaiinnovations/laya-typed-decisions on 24
customer-service decisions while building Second Thought,
an open-source calibration and active-learning layer for
System One / typed-decision models.
On this small sample:
- Accuracy: 87.5% (21/24)
- ECE: 0.62
The sample is far too small to make a general claim about
the checkpoint's calibration, so I'm treating this as a
case study rather than a benchmark.
What I found interesting is that this is directionally
consistent with the model card's own warning that the
checkpoint's probability calibration should be refit on
held-out data before relying on probabilities for
downstream decisions.
Second Thought is designed around exactly that problem:
model prediction
→ probability distribution
→ calibration
→ identify decisions worth reviewing
→ correction
→ training/evaluation data.
Repository:
https://github.com/KNambiarDJsc/second-thought
I'd be interested in whether others have tested the
checkpoint on held-out real-world data.
@NambiarONS Second Thought's calibration-to-review loop is relevant to what I'm testing with Seb-9B, a local text/image decision model I built: https://huggingface.co/ironbcc/seb-9b
On my 6,920-row evaluation suite, Seb's ECE is 1.97% and accuracy on the most confident 80% of cases is 96.9%. Those are development evaluation results, not evidence that the same calibration transfers to customer-service data: the suite was used during checkpoint selection, and a fresh human-labelled evaluation is still pending.
Your customer-service cases would be an interesting independent check, particularly comparing calibration before/after refitting and accuracy at fixed review coverage. If you're adding more decision-model backends to Second Thought, Seb's card documents how to retrieve the distribution over allowed options through vLLM.