laya-clinc150

A fine-tune of Laya that sorts requests to a voice assistant into the 150 intents of CLINC150, across banking, travel, cooking, work, small talk and more, and answers "none of these" to requests that are none of them. It gives a probability for every option in one forward pass, and its confidence can be trusted. So it can act on the requests it is sure about and send the rest to a fallback.

Requests a model was never built for are the hard part of any deployed assistant. CLINC150 was made to measure exactly that: its test holds 1,000 out-of-scope queries beside the 4,500 in-scope ones.

Results

All 5,500 queries of CLINC150's "plus" test set, one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-18.

Untuned Laya This model
In-scope queries right (of 4,500) 2,352 (52.3%) 4,277 (95.0%)
95% confidence interval 50.8 to 53.7% 94.4 to 95.6%
Out-of-scope queries caught (of 1,000) 490 (49.0%) 716 (71.6%)
Mean stated confidence, all 5,500 0.666 0.955
Calibration error (ECE, lower is better) 0.251 0.050

It runs a little overconfident: it states 0.955 on average and is right on 90.8% of all 5,500 queries. Queries it scores 0.95 to 0.98 are right 93.1% of the time, and those at 0.98 or more 97.9%.

Acting only when sure. During the fine-tune, Laya held back 400 training queries and fitted a threshold on them, aiming at 99% right among the answers above it. That threshold is 0.9687. The model acts on a query only when it names one of the 150 intents at or above it; "none of these" and anything below it go to the fallback. Applied once to the test:

  • It acts on 3,795 of the 4,500 in-scope queries (84.3%), and 3,741 of those are right (98.6%).
  • It lets through 83 of the 1,000 out-of-scope queries (8.3%).

With the cut chosen on the test itself, which describes this test and is optimistic for new queries, and counting every answer above the cut, "none of these" included: 95% right while answering 88.1% of queries, 98% right at 36.4%, and 99% right only at 0.2% (13 queries).

Speed. One query alone: median 24 ms, 95th percentile 26 ms. In batches of 32: about 13 ms a query.

Against published results. The CLINC150 paper (Larson et al. 2019, Table 2, "OOS+" column, models trained with the out-of-scope queries as a class) reports BERT at 96.7% in-scope accuracy and 59.2% out-of-scope recall, and Rasa at 90.9% and 66.0%. Without the threshold, this model is 1.7 points behind BERT in scope (95.0%; BERT's number lies above this model's whole 95% interval) and 12.4 points ahead on out-of-scope recall (71.6%, interval 68.7 to 74.3%). It is ahead of Rasa on both. With the threshold it stops 91.7% of out-of-scope queries but acts on only 84.3% of in-scope ones, a trade the paper's models were not scored on. The paper does not say which size of BERT it used; Laya has 421 million parameters.

Where it is weakest. Out-of-scope queries: 284 of the 1,000 get an intent, most often directions (16), vaccines (13) and order (11). Among the intents, transactions and calendar are right 76.7% of the time each, then yes, restaurant reviews, how busy, pin change and report fraud at 80% each. The most common mix-up between two intents is transactions taken for spending history (6 of 30).

Use

pip install laya==0.4.31
import json

import laya
from huggingface_hub import hf_hub_download

REPO = "VibeHash/laya-clinc150"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
    questions = json.load(f)

result = model.predict("set a timer for ten minutes", questions, min_confidence=0.9687)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained" or answer["choice"] == "none of these":
    print("fallback; best guess:", answer["choice"])
else:
    print(answer["choice"], answer["answer_confidence"])

What it returned on the CPU:

Query Answer Confidence At 0.9687
set a timer for ten minutes timer 0.9696 acts
how much money is in my checking account balance 0.9776 acts
book a table for two at 7 tonight restaurant reservation 0.9672 fallback
what is the boiling point of mercury none of these 0.9724 fallback
help me write a cover letter none of these 0.8896 fallback

The results above were measured on a GPU, and this table on the CPU. On the first 100 test queries the two gave the same answers; their confidences differed by 0.0002 at the median, but by 0.07 for one query. A query close to the threshold can fall on either side of it depending on the hardware.

  • Ask the question in questions.json exactly as written: the same instruction and the same 151 options in the same order. labels.json maps each option back to CLINC150's label name; "none of these" is the label oos.
  • min_confidence is the threshold fitted in training. Leave it out to get an answer for every query, or pick your own cut from the trade-off above.
  • predict_batch(texts, questions, batch_size=32) scores many queries in shared forward passes.
  • To make sure you load exactly these weights, pass expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"} to laya.load. SHA256SUMS lists every file.

Limits

  • English queries to a general assistant, written by crowd workers. Other products and other languages were not tested.
  • "Out of scope" in CLINC150 means outside its 150 intents, collected the same way. Requests that are off-topic in another way were not tested.
  • One fine-tune. How much a second seed would differ was not measured.
  • The threshold was fitted on 400 training queries, aiming at 99% right. On the test, 96.5% of the queries it acted on were right (3,741 of 3,878, counting the 83 out-of-scope queries it let through as wrong); on in-scope queries alone, 98.6%. The training data is 1.6% out of scope and the test 18.2%. Since 8.3% of out-of-scope queries got through, the more off-topic requests a deployment sees, the further it falls below 99%.

Training

Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, RLCD loss), with two changes, both fixed before any fine-tuned model was scored:

  • head_max_len 512 instead of 192. The 151 options take 471 tokens; at 192 Laya cuts them short until many read the same. The longest query in the data is 37 tokens, so no query is cut.
  • target_error 0.01 instead of 0.10, so the threshold aims at 99% right.

The options are CLINC150's label names with underscores written as spaces; oos is written "none of these". The training set holds 15,250 queries (100 for each intent and 250 out of scope); Laya trained on 14,850 and held back the other 400 to fit its temperature and the threshold. Training took 50 minutes on one RTX 4070 SUPER. The validation split was not used. The test was read only by the scorer, through laya.load(...).predict_batch, the same call a user makes. 2 of the 5,500 test queries also appear in the training data; they are left in, as in the official split.

Reproduce

The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-tasks/.

hf download VibeHash/laya-clinc150 --include "code/*" --local-dir laya-clinc150
cd laya-clinc150/code
pip install torch --index-url https://download.pytorch.org/whl/cu126   # or the build for your GPU
pip install -r requirements.txt
python -m layatasks.data clinc150              # CLINC150 at a pinned revision; writes and locks the test
python -m layatasks.train clinc150 --seed 0    # 50 minutes on an RTX 4070 SUPER
python -m layatasks.evaluate clinc150 --model ~/.laya-tasks/clinc150/runs/seed0/model --name seed0

Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed. eval.json holds the full scores behind this card, train.json the training record.

Licence and credits

Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint, doi:10.20944/preprints202611.0781.v1. Trained on CLINC150 (CC BY 3.0): Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction, EMNLP-IJCNLP 2019. See NOTICE.

Downloads last month
15
Safetensors
Model size
0.4B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for VibeHash/laya-clinc150

Finetuned
(259)
this model

Dataset used to train VibeHash/laya-clinc150

Evaluation results