laya-grounded

A Fine-Tune of convaiinnovations/laya that fixes two measured Failures in the base Model: it could not read a Fact stated verbatim in its Input, and it answered a Question and that Question's own Negation both with "no" at high Confidence.

Benchmarks and Method: Luni/laya-jev-benchmark.

This is not a drop-in Replacement for the base Model

It is better at Grounding, Contradiction and Calibration. It is worse at Phishing and at Routing Stability. Read the Limitations before swapping it in. For Phishing specifically, keep using the base Model.

What it fixes

The base Model, given this Ticket:

I've been trying to export my data for three days and the button just spins forever. I'm on the Pro plan and I have a compliance audit on Monday. This is the second time I've written in.

Question	base	this Model
Has this Customer contacted Support before?	0.20	0.89
Is the Customer on a paid Plan?	0.50	0.98
P(needs a Human) + P(a Bot can resolve it)	0.09	1.22
P(Phishing) + P(legitimate Sender)	1.73	1.00

The first two Facts are written in the Text. The last two are Pairs of logical Inverses and should sum to about 1.0.

On an eleven-Assertion Probe Suite the base Model fails 7. This one fails 2.

	base	this Model
Probe Failures (of 11)	7	2
Grounding	2 of 5	5 of 5
Contradiction	0 of 3	2 of 3
held-out Macro Accuracy		0.840
held-out Macro ECE		0.156
typed-decisions	0.360	0.676
Latency, one Question, RTX 5090	10.7 ms	10.7 ms

Per Source on the held-out Split: boolq 0.963, snli 0.945, anli 0.960, multi_nli 0.956, scitail 0.947, enron_spam 0.993, phishing_email 0.985, clinc_oos 0.908, civil_comments 0.962, yelp_stars 0.653, support_tickets 0.353.

Limitations

All four are understood and all four are fixable. None of them were fixed here because each needs another Training Run.

1. Phishing got worse

	base	this Model
PhishNChips Accuracy, calibrated	0.611	0.512
AUROC	0.679	0.569

Cause. The Mixture includes zefang-liu/phishing-email-dataset and puyang2025/seven-phishing-email-datasets. Both are older Corpora. The Model learned their Distribution and it does not transfer to the 2026 PhishNChips Set. Note also that the second of those has visible Word-Noise injected into the Bodies, which is probably part of the Problem.

Fix. Drop both Sources, or hold out a PhishNChips-like Split during Training and early-stop on it instead of on in-Distribution Loss. Right now nothing in the Training Loop notices this Regression, which is the actual Bug.

Until then. Use the base Model for Phishing.

2. Routing Stability got worse

Same Ticket, same Decision, only the Option Names change. The base Model gets 2 of 3 right, this Model gets 1 of 3.

Cause. Training augmented choice Items by permuting Option Order, not by renaming them. That teaches Order Invariance, which was never the Problem. The Model keys off Option Name Semantics, so account versus billing versus finance moves the Verdict even when the Ticket does not change.

Fix. Augment with Synonym Option Names and paraphrased Criteria Text, not just permuted Order. The Machinery for this already exists in the Conversion Code, it was pointed at the wrong Axis.

Until then. Do not build Schemas at Runtime with this Model. Fix your Option Names and evaluate on those exact Names.

3. Non-commercial Licence

Cause. The Mixture includes facebook/anli and Tobi-Bueck/customer-support-tickets, both CC-BY-NC-4.0, so this Checkpoint inherits that. The base Model is Apache 2.0 and unaffected.

Fix. Drop those two Sources and retrain. That removes about 27k of 180k Items, all of it Entailment and Ticket Routing, so expect a small Loss on those two Task Families and no Change elsewhere. About 40 Minutes on one RTX 5090.

4. It is a Generalist now, not a Specialist

On typed-decisions it scores 0.676. A Version fine-tuned only on that Task scores 0.767.

Cause. This is the expected Trade. The Task-specific Fine-Tune also scores above the 0.735 Teacher Agreement Ceiling, which means it is memorising Label Noise rather than understanding the Task better.

Fix. If you have one narrow Task, fine-tune this Checkpoint further on it. Starting from here rather than from the base Model should be strictly better, since the Grounding and Contradiction Behaviour carries over.

5. Calibration is Distribution-specific

The reported ECE of 0.156 is on the held-out Split of the Training Mixture. On a genuinely new Distribution the Confidences will need refitting.

Fix. Platt-scale on a few hundred labelled Examples from your own Data. bench/platt.py in the Benchmark Repo does this. Temperature Scaling will not work, it has no Bias Term and cannot move a Threshold across 0.5.

Usage

Same API as the base Model.

import laya
agent = laya.load("Luni/laya-grounded")

state = "I'm on the Pro plan and this is the second time I've written in."
questions = {
    "contacted_before": {"type": "noul",
        "instructions": "Has this customer contacted support about this before?"},
    "paid_plan": {"type": "noul",
        "instructions": "Is the customer on a paid plan?"},
}
print(agent.predict(state, questions)["answers"])

confidence is normalised Shannon Entropy, 1 - H(p)/log(k), not the top Probability. For a Boolean Question it is a pure Function of the Probability and carries no extra Information.

Training

180k Items from public Datasets, 3 Epochs, 55 Minutes on one RTX 5090. Grounding from BoolQ and the unanswerable Half of SQuAD v2. Contradiction from SNLI, MultiNLI, ANLI and SciTail. Every Boolean Source with a sensible Inverse was emitted as a logical Pair, so 51.6% of the Mixture is Pairs.

Objective is the base Model's RLCD Reward plus a Semantic Consistency Penalty over those Pairs, -log(p_A(1-p_B) + (1-p_A)p_B) from Xu et al. 2018, lambda ramped 0.05 to 0.30.

A Control Arm with that Penalty switched off scored better on the Probes, 1 Failure against 2. The Data fixed the Contradictions, not the Loss. What the Penalty bought was Calibration: the Control answers every Grounding Probe at exactly Confidence 1.00, this Model answers 0.89, 0.98 and 0.62, and is better on both Accuracy and ECE. That is why this Checkpoint is the published one.

Credits

Base Model by Convai Innovations, Apache 2.0. Backbone ModernBERT-large by Answer.AI and LightOn. Full Attribution for every Dataset and Paper in the Benchmark Repo.