convaiinnovations

/

laya

Like 5.77k

Independent head-to-head on a sealed 1,190-case manifest: laya 0.686 vs Jev 1.13.0 0.907 vs Qwen2.5-1.5B PCD 0.605, with paired intervals, plus three adapter findings

Posting an independent evaluation of laya against jev-1.13.0 and a Qwen2.5-1.5B parallel constrained decoding baseline. Everything is public and reproducible: https://github.com/instax-dutta/sysone-bench, report at results/v2/report-20260926/REPORT.md. The same write-up is on GitHub with the full tables: https://github.com/NandhaKishorM/laya/issues/555

Checkpoint scope, stated first because it is the easiest thing to misread. This run used the default root (English) checkpoint for every suite, multilingual included. laya-multilingual, laya-typed-decisions and the Router were not evaluated. The multilingual row below therefore measures the English checkpoint on non-English text and is not a statement about the multilingual checkpoint. Version: laya 0.3.11, weights commit 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851. Happy to rerun against main and against the multilingual and Router checkpoints.

Setup

• 1,190 cases and 1,550 typed questions across 9 suites: triage (5-way intent choice plus three nouls), guardrails (noul), moderation (noul), and the public sets agnews (4 labels), emotion (6), banking77 (12 intents), mnli (3-way), sst5 (5-level score), multilingual intent (6 labels, 5 languages).
• One sealed manifest, raw bytes a938cc2483a592dc84e0d5baac12594491bcaa5b4ceb6b7c3b0def71b36297bd, logical digest 4272a7a25ebcb324235696ad808544e157a31e3f9c04421da731a08dfb9d5768. Checksum verified before the run and again inside the container. Seed 42. Same states, same question dicts, same order for every model.
• CPU only, no GPU. Each run container capped at 4 CPUs and 12 GB. laya and Qwen ran locally against a dedicated model cache. Jev was called over the network from the same host, so Jev latency is a deployment observation, not a lab number.

Results, evaluation split, 952 cases, 1,240 scored decisions

Paired per suite, Jev minus laya, clustered by case because questions inside one case share a state: 20,000-replicate state-cluster bootstrap for the interval, 20,000 permutation replicates for the p-value, Holm corrected across the nine suites.

Triage is the one suite we do not claim. The bootstrap interval clears zero, the permutation test does not survive correction. A 5.7-point gap on 192 decisions is not established, and it is where both models are already strongest.

Calibration is close on choice questions across all three, ECE between 0.0009 and 0.0028. Refitting on the calibration split helped the PCD baseline and hurt laya on choice, and helped both open models on noul, so temperature scaling is not a dependable fix. Latency per case, p50/p95: Jev 314/387 ms, laya 588/1,364 ms, PCD 3,948/13,471 ms.

The gold is the weak part, and it should temper everything above

Our labels are hand-reviewed, not behavioral. One human reviewer read all 1,190 cases and corrected an AI draft of every answer, protocol human-reviewed-ai-assisted-v1. There was no second independent reviewer and no adjudication, so this dataset has no inter-annotator agreement, no kappa and no adjudication artifact. provenance.json records "independent_human_review": false and "adjudication": false.

Evals that grade from real outcomes are stronger than anything we produced. Do not place these numbers next to outcome-graded results. The per-language multilingual table is the weakest evidence here, for that reason plus the checkpoint scope above.

Three adapter findings, each reproducible on the current package

These cost us three failed runs, so they are worth a maintainer's five minutes.

1. The agent exposes no revision or checkpoint identity. After laya.load("convaiinnovations/laya") the agent carries model_id, cfg, device and model, but no revision, checkpoint_revision, model_revision, commit, _commit_hash or checkpoint attribute on the agent, its model, its config or cfg. model.config is a transformers config and does not carry the checkpoint commit either. Anything recording which weights served a result has to resolve the commit out of band, for example from the directory name of the cached rl_agent_config.json snapshot. We resolve it offline from the cache and fail closed when it cannot be resolved. Surfacing the resolved snapshot path or commit on the agent would remove the guesswork for every downstream consumer.

2. Choice probabilities are rounded to four decimals, so they do not sum to one exactly. With five labels the raw values summed to 0.9999, which fails a strict unit-sum contract at 1e-6 tolerance. Dividing by the observed total and absorbing the residual on the largest entry is order-preserving, so the argmax and the reported choice do not move. Either the docs should say transport rounding is expected, or _decode_answers could emit more decimals.

3. Answers carry fields that strict answer schemas reject. Every answer includes action: {"act_probability": ...}; score answers add legend and probabilities; noul answers add confidence. Our contract is exact about answer fields, so we project each answer onto the declared set at the adapter edge rather than loosening the contract. Flagging it because anyone integrating against a strict schema hits the same wall, and action in particular reads like debug telemetry that may not be intended as part of the answer payload.

Reproducing

The sealed manifest and checksum are in the repo. The report regenerates byte-identically, figures included:

python -m benchmark.report --laya <run> --jev <run> --qwen <run> \
  --manifest datasets/v2/manifest.jsonl --output-root <new-report-dir>
python -m benchmark.graphics --figure-source <new-report-dir>/figures/figure-source.json \
  --output-root <new-report-dir>/figures

benchmark.report refuses to emit anything unless its recomputed per-decision counts and per-suite accuracies reproduce each run's own sealed summary.json, so the published numbers cannot drift from the raw predictions. Run IDs: laya-assisted-20260925-5, jev-assisted-20260925, qwen-assisted-20260925.

Corrections welcome, especially on the checkpoint scope, since that is the row most likely to be misread.