Instructions to use VibeHash/laya-clinc150 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use VibeHash/laya-clinc150 with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
laya-clinc150
A fine-tune of Laya that sorts requests to a voice assistant into the 150 intents of CLINC150, across banking, travel, cooking, work, small talk and more, and answers "none of these" to requests that are none of them. It gives a probability for every option in one forward pass, and its confidence can be trusted. So it can act on the requests it is sure about and send the rest to a fallback.
Requests a model was never built for are the hard part of any deployed assistant. CLINC150 was made to measure exactly that: its test holds 1,000 out-of-scope queries beside the 4,500 in-scope ones.
Results
All 5,500 queries of CLINC150's "plus" test set, one fine-tune (seed 0), one RTX 4070 SUPER, 2026-11-18.
| Untuned Laya | This model | |
|---|---|---|
| In-scope queries right (of 4,500) | 2,352 (52.3%) | 4,277 (95.0%) |
| 95% confidence interval | 50.8 to 53.7% | 94.4 to 95.6% |
| Out-of-scope queries caught (of 1,000) | 490 (49.0%) | 716 (71.6%) |
| Mean stated confidence, all 5,500 | 0.666 | 0.955 |
| Calibration error (ECE, lower is better) | 0.251 | 0.050 |
It runs a little overconfident: it states 0.955 on average and is right on 90.8% of all 5,500 queries. Queries it scores 0.95 to 0.98 are right 93.1% of the time, and those at 0.98 or more 97.9%.
Acting only when sure. During the fine-tune, Laya held back 400 training queries and fitted a threshold on them, aiming at 99% right among the answers above it. That threshold is 0.9687. The model acts on a query only when it names one of the 150 intents at or above it; "none of these" and anything below it go to the fallback. Applied once to the test:
- It acts on 3,795 of the 4,500 in-scope queries (84.3%), and 3,741 of those are right (98.6%).
- It lets through 83 of the 1,000 out-of-scope queries (8.3%).
With the cut chosen on the test itself, which describes this test and is optimistic for new queries, and counting every answer above the cut, "none of these" included: 95% right while answering 88.1% of queries, 98% right at 36.4%, and 99% right only at 0.2% (13 queries).
Speed. One query alone: median 24 ms, 95th percentile 26 ms. In batches of 32: about 13 ms a query.
Against published results. The CLINC150 paper (Larson et al. 2019, Table 2, "OOS+" column, models trained with the out-of-scope queries as a class) reports BERT at 96.7% in-scope accuracy and 59.2% out-of-scope recall, and Rasa at 90.9% and 66.0%. Without the threshold, this model is 1.7 points behind BERT in scope (95.0%; BERT's number lies above this model's whole 95% interval) and 12.4 points ahead on out-of-scope recall (71.6%, interval 68.7 to 74.3%). It is ahead of Rasa on both. With the threshold it stops 91.7% of out-of-scope queries but acts on only 84.3% of in-scope ones, a trade the paper's models were not scored on. The paper does not say which size of BERT it used; Laya has 421 million parameters.
Where it is weakest. Out-of-scope queries: 284 of the 1,000 get an intent, most often directions (16), vaccines (13) and order (11). Among the intents, transactions and calendar are right 76.7% of the time each, then yes, restaurant reviews, how busy, pin change and report fraud at 80% each. The most common mix-up between two intents is transactions taken for spending history (6 of 30).
Use
pip install laya==0.4.31
import json
import laya
from huggingface_hub import hf_hub_download
REPO = "VibeHash/laya-clinc150"
model = laya.load(REPO)
with open(hf_hub_download(REPO, "questions.json")) as f:
questions = json.load(f)
result = model.predict("set a timer for ten minutes", questions, min_confidence=0.9687)
answer = result["answers"]["intent"]
if answer["abstention"] == "abstained" or answer["choice"] == "none of these":
print("fallback; best guess:", answer["choice"])
else:
print(answer["choice"], answer["answer_confidence"])
What it returned on the CPU:
| Query | Answer | Confidence | At 0.9687 |
|---|---|---|---|
| set a timer for ten minutes | timer | 0.9696 | acts |
| how much money is in my checking account | balance | 0.9776 | acts |
| book a table for two at 7 tonight | restaurant reservation | 0.9672 | fallback |
| what is the boiling point of mercury | none of these | 0.9724 | fallback |
| help me write a cover letter | none of these | 0.8896 | fallback |
The results above were measured on a GPU, and this table on the CPU. On the first 100 test queries the two gave the same answers; their confidences differed by 0.0002 at the median, but by 0.07 for one query. A query close to the threshold can fall on either side of it depending on the hardware.
- Ask the question in
questions.jsonexactly as written: the same instruction and the same 151 options in the same order.labels.jsonmaps each option back to CLINC150's label name; "none of these" is the labeloos. min_confidenceis the threshold fitted in training. Leave it out to get an answer for every query, or pick your own cut from the trade-off above.predict_batch(texts, questions, batch_size=32)scores many queries in shared forward passes.- To make sure you load exactly these weights, pass
expected_sha256={"model.safetensors": "<hash from SHA256SUMS>"}tolaya.load.SHA256SUMSlists every file.
Limits
- English queries to a general assistant, written by crowd workers. Other products and other languages were not tested.
- "Out of scope" in CLINC150 means outside its 150 intents, collected the same way. Requests that are off-topic in another way were not tested.
- One fine-tune. How much a second seed would differ was not measured.
- The threshold was fitted on 400 training queries, aiming at 99% right. On the test, 96.5% of the queries it acted on were right (3,741 of 3,878, counting the 83 out-of-scope queries it let through as wrong); on in-scope queries alone, 98.6%. The training data is 1.6% out of scope and the test 18.2%. Since 8.3% of out-of-scope queries got through, the more off-topic requests a deployment sees, the further it falls below 99%.
Training
Laya's own laya.train.finetune at its default settings (4 epochs, effective batch 64, RLCD loss), with two
changes, both fixed before any fine-tuned model was scored:
head_max_len512 instead of 192. The 151 options take 471 tokens; at 192 Laya cuts them short until many read the same. The longest query in the data is 37 tokens, so no query is cut.target_error0.01 instead of 0.10, so the threshold aims at 99% right.
The options are CLINC150's label names with underscores written as spaces; oos is written "none of these". The
training set holds 15,250 queries (100 for each intent and 250 out of scope); Laya trained on 14,850 and held back
the other 400 to fit its temperature and the threshold. Training took 50 minutes on one RTX 4070 SUPER. The
validation split was not used. The test was read only by the scorer, through laya.load(...).predict_batch, the
same call a user makes. 2 of the 5,500 test queries also appear in the training data; they are left in, as in the
official split.
Reproduce
The code that built and scored this model is in code/. Data, weights and runs go under ~/.laya-tasks/.
hf download VibeHash/laya-clinc150 --include "code/*" --local-dir laya-clinc150
cd laya-clinc150/code
pip install torch --index-url https://download.pytorch.org/whl/cu126 # or the build for your GPU
pip install -r requirements.txt
python -m layatasks.data clinc150 # CLINC150 at a pinned revision; writes and locks the test
python -m layatasks.train clinc150 --seed 0 # 50 minutes on an RTX 4070 SUPER
python -m layatasks.evaluate clinc150 --model ~/.laya-tasks/clinc150/runs/seed0/model --name seed0
Every source is pinned by revision and SHA-256, and the scorer refuses a test file whose SHA-256 changed.
eval.json holds the full scores behind this card, train.json the training record.
Licence and credits
Apache 2.0. A fine-tune of Laya by Convai Innovations (Apache 2.0); please cite their preprint,
doi:10.20944/preprints202611.0781.v1. Trained on CLINC150
(CC BY 3.0): Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction,
EMNLP-IJCNLP 2019. See NOTICE.
- Downloads last month
- 15
Model tree for VibeHash/laya-clinc150
Base model
convaiinnovations/layaDataset used to train VibeHash/laya-clinc150
Evaluation results
- In-scope accuracy on CLINC150 (plus)test set self-reported0.950
- Out-of-scope recall on CLINC150 (plus)test set self-reported0.716
- Expected calibration error (15 bins) on CLINC150 (plus)test set self-reported0.050