flow-laya-intent

A Laya checkpoint fine-tuned to classify messages sent to Flow, the AI assistant for the Frappe / ERPNext desk, so Flow can answer common requests with a read-only tool instead of calling a chat model.

Fine-tuned from convaiinnovations/laya (Apache-2.0) with Laya's own RLCD training script and per-type temperature calibration.

What it answers

It is trained on exactly these two typed questions; ask them verbatim (they are FINETUNED_QUESTIONS in Flow's flow/lib/laya_router.py):

questions = {
    "intent": {
        "type": "choice",
        "instructions": "What kind of request is this message to an ERP assistant?",
        "criteria": {
            "HOWTO": "asks how to create, add or set up a record by hand, or for the steps to do it",
            "DEFINITION": "asks what a term, document type or field means or is used for",
            "WORKFLOW": "asks about a business process with several steps, approvals or documents",
            "RECORD_LOOKUP": "asks to count, find, list or show existing records or their details",
            "TROUBLESHOOTING": "reports an error message, a failure, or something that is not working",
        },
    },
    "asks_count": {
        "type": "noul",
        "instructions": "Does the message ask how many records there are (a count or number)?",
    },
}

Usage

import laya

agent = laya.load("path/or/repo-id/of/this/model")
answers = agent.system_one("how do I add a new supplier?", questions)["answers"]
answers["intent"]["choice"]             # "HOWTO"
answers["intent"]["answer_confidence"]  # calibrated confidence

In Flow, download the checkpoint and point the site at it:

bench --site <site> set-config flow_laya_enabled 1 --parse
bench --site <site> set-config flow_laya_model_path /path/to/flow-laya-intent
bench --site <site> set-config flow_laya_threshold 0.85 --parse

ONNX (no PyTorch)

laya.onnx + laya.onnx.data are an fp32 export of the same checkpoint (Laya's scripts/export_onnx.py). Flow runs them with its own runner (flow/lib/laya_onnx.py: onnxruntime + tokenizers, no PyTorch), and picks it automatically when the repo contains laya.onnx, so the router works on hosts like Frappe Cloud. Checked against Laya on 730 messages: identical token sequences and decisions, max probability difference 0.0001. On CPU it takes about 2–3 s per message and about 1.7 GB of RAM per process.

On a server without the files, set HF_TOKEN (read access) and point the site at the repo:

bench --site <site> set-config flow_laya_model_path aditya-walue/flow-laya-intent

Training data

1,500 synthetic messages (300 per intent), generated from one Frappe site's real metadata: its DocTypes and field labels, and the error messages raised in its installed apps' code. DocTypes, error messages and two phrasings per intent are held out, so validation and test measure generalisation to unseen record types, errors and wording. No real user messages were used for training. Gold targets are soft (0.92 on the label), so calibrated confidence tops out near 0.92.

Evaluation

Set	n	Accuracy	ECE	Base model accuracy
Validation (synthetic)	300	100.0%	0.079	—
Test (synthetic, unseen DocTypes/errors)	400	99.3%	0.078	50.7%
Real Flow messages (never trained on)	30	100.0%	0.079	63.3%

Test-set confusion: all errors were 3 DEFINITION questions in held-out phrasings ("tell me what X represents in Y", "what does the X on Y do") read as RECORD_LOOKUP.

Accuracy vs. coverage on the synthetic test set:

Threshold	Coverage	Accuracy
0.80	99.8%	99.2%
0.85	99.5%	99.5%
0.90	99.3%	99.5%
0.95	0%	— (confidence caps near 0.92)

Recommended routing threshold: 0.85. Latency is about 100 ms per message on an Apple-silicon GPU.

Limitations

• Train and test share templates (beyond the held-out phrasings), so synthetic accuracy likely overstates accuracy on real traffic.
• The real-message set is small (30) and has no WORKFLOW and one DEFINITION example; re-evaluate on real usage before relying on the numbers.
• Trained on one site's apps: DocTypes and errors from other apps may be less well covered.
• English only.

License

Apache-2.0, as the base model. Laya is by the Laya authors (NandhaKishorM/laya).