eric-ml-nlp/Delveta-LayaChoice-v2

Delveta LayaChoice v2

Delveta-LayaChoice-v2 is a capability-selection decision model with an explicit reject option. Given a user turn and a short list of candidate capabilities, it returns either the single capability that should handle the request or the dedicated REJECT decision when none of the candidates applies. It is a full-parameter fine-tune of Laya's multilingual decision model on Delveta's capability-selection task; it is not a general-purpose chat model.

This release is the recall_threshold_082 run (layachoice_v2_082_r1) and supersedes the earlier epoch-7 artifact. It uses the dynamic K+1 option contract: each question carries K capability cards plus one REJECT card, with K between 0 and 3, so the model always scores between 1 and 4 options instead of a fixed four-slot layout.

The repository ships two self-contained precision directories. Load either one directly; they do not depend on each other or on files at the repository root.

• FP32 model — FP32/model.safetensors, 1.29 GB (decimal). Reference artifact, byte-for-byte the published weights.
• FP16 model — FP16/model.safetensors, 644 MB. Half-precision copy of the same weights, validated by smoke test (see §8). It reduces download and on-disk size only: with the stock loader the parameters are still stored as FP32 in memory. See §8.3.

Task	choose one of K capability cards + 1 REJECT card (K = 0..3)
Option slots	dynamic K+1, min 1, max 4 (observed 1, 2, 3, 4)
Base model	convaiinnovations/laya (multilingual), revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851, subfolder multilingual
Encoder	jhu-clsp/mmBERT-base (~322M parameters)
Languages	Chinese (zh), English (en), mixed (mixed)
Selected checkpoint	epoch 9 (validation top-1 0.996676 on 2,407 rows)
Weights	FP32 (1.29 GB) · FP16 (644 MB, storage only)
Temperature	[3.070228338241577, 1.0, 1.0] (choice slot calibrated)
Training code	https://github.com/Eric-LLMs/Delveta/tree/main/fine_tuning/laya_finetune/V2
Training evidence	eric-ml-nlp/Delveta-LayaChoice-v2-checkpoints

1. What this release changes

• Dynamic option sets. The bundle declares option_slots = {"rule": "K+1", "min": 1, "max": 4}, and rows contain 1–4 options including the REJECT card. A question with three candidates is a four-option question; a question with no applicable candidate is a single REJECT option.
• New frozen data bundle. recall_threshold_082, flat installed layout: 36,197 train rows, 2,407 validation rows, 9,729 test rows.
• New selected epoch. Epoch 9, chosen on validation top-1 (see §4).
• New calibration. Temperature fitted on the held-out calibration split: 3.0702 in the choice slot.
• Dual-precision layout. The FP32 artifact moved into FP32/, and an FP16 copy was added under FP16/. See §8 for the differences and verified output.

2. Training data

The training bundle is frozen and hash-verified. Training reads only manifest.json and the three split files.

Split	Rows	en	zh	mixed
train	36,197	16,499	15,505	4,193
calibration (held out from train)	400	—	—	—
effective train	35,797	—	—	—
val	2,407	1,116	1,030	261
test	9,729	4,438	4,132	1,159

• Card view: B_noprov. Every row has exactly one REJECT card whose text equals the manifest's frozen reject card.
• Splits are disjoint by id and source_query_id.
• Calibration rows are selected deterministically from the final shuffled train order: random.Random(20260922), first min(400, n_train // 10) = 400 indices; the manifest records the indices and row IDs.
• Bundle SHA256:
  • manifest.json 5bb80da5…
  • train.jsonl 1a947233…
  • val.jsonl 943000f2…
  • test.jsonl 3abd7b3c…

Dataset: https://huggingface.co/datasets/eric-ml-nlp/Delveta-LayaChoice-v2-Data

3. Training configuration

The run used the frozen V2 configuration without modification:

Parameter	Value
Epochs	10
Micro batch	8
Gradient accumulation	4
Group size (GRPO)	4
Encoder LR	2.5e-5
Head LR	1.0e-4
Weight decay	0.01
Sigma schedule	0.4 → 0.1
CE weight	1.0
w_sph / w_rps	0.75 / 1.0
Gradient clip	1.0
Seed / calibration seed	42 / 20260922
Precision	BF16 autocast, GradScaler disabled
Token budget	option 256, head 768, max 1,024
Optimizer updates	11,190 (verified equal to planned)
Hardware	1 × NVIDIA L4 24 GB
Training time	17,673 s ≈ 4.91 h
Runtime	Python 3.10.20, torch 2.14.1+cu130, transformers 5.17.0, laya 0.3.21

The loss is the frozen GRPO-style group-advantage objective (w_sph 0.75, w_rps 1.0) plus CE_WEIGHT = 1.0; one full-state checkpoint is written per epoch.

4. Epoch selection

Selection is argmax validation top-1, ties resolved to the earlier epoch. The test split is never read during training or selection.

Epoch	Train loss	Val top-1	Val ECE	Reject recall	Reject FPR	Reject precision
1	0.354976	0.990445	0.009463	0.945455	0.003201	0.967442
2	0.114831	0.981305	0.018557	0.945455	0.013260	0.877637
3	0.052656	0.989198	0.009794	0.909091	0.000914	0.990099
4	0.033588	0.989614	0.010101	0.931818	0.002743	0.971564
5	0.020002	0.972995	0.026366	0.922727	0.019204	0.828571
6	0.012646	0.994184	0.005269	0.963636	0.001372	0.986047
7	0.004379	0.995015	0.005989	0.963636	0.000914	0.990654
8	0.003872	0.995430	0.004793	0.968182	0.000914	0.990698
9	0.004162	0.996676	0.003710	0.981818	0.000914	0.990826
10	0.001076	0.995430	0.004196	0.968182	0.000914	0.990698

Epoch 9 wins on validation top-1; epoch 10 does not overtake it.

5. Results

Validation (2,407 rows, uncalibrated checkpoint temperature)

Metric	Value
Top-1	0.996676
ECE	0.003710
Reject recall	0.981818
Reject FPR	0.000914
Reject precision	0.990826

Test (9,729 rows, fitted temperature 3.0702)

Metric	Value
Top-1	0.998355 (9,713 / 9,729)
ECE	0.002576
Mean confidence	0.995827
Mean gold probability	0.994382
Reject recall	0.996441 (840 / 843)
Reject precision	0.995261
Reject FPR	0.000450
Confusion	TP 840, FP 4, FN 3, TN 8,882

By language (test top-1):

Language	n	Top-1
en	4,438	0.998197
mixed	1,159	0.999137
zh	4,132	0.998306

By target kind (test top-1):

Target	n	Top-1
capability	8,886	0.998537
reject	843	0.996441

All test numbers above come from benchmark/final_test.report.json in the training evidence repository.

6. Zero-shot baseline comparison

The baseline is the unmodified official base model on the same frozen test split, with the checkpoint temperature [1.0, 1.0, 1.0] and no calibration.

Metric	Zero-shot baseline	Fine-tuned (epoch 9)
Top-1	0.416795	0.998355
ECE	0.206293	0.002576
Reject recall	0.595492	0.996441
Reject precision	0.128751	0.995261
Reject FPR	0.382287	0.000450

Top-1 improvement: +0.581560.

7. Calibration

Temperature is fitted per question type on the 400 held-out calibration rows (seed 20260922) with LBFGS, then clamped to [0.5, 5.0].

Field	Value
Raw fitted value (choice slot)	3.070228338241577
Written value	3.070228338241577 (no clamping)
Temperature vector	[3.070228338241577, 1.0, 1.0]
Base temperature	[1.0, 1.0, 1.0]
Fallback	1.2 (not used)
Calibration rows	400

Temperature changes confidence, not the top-1 decision. It is fitted on in-distribution held-out rows and does not guarantee calibrated confidence under distribution shift.

8. Precision layout

Both directories are self-contained: each has its own weights, encoder config, tokenizer, code, dataset manifest and agent config. A/ never points at files in B/.

Delveta-LayaChoice-v2/
  README.md                     this file
  SHA256SUMS                    checksums for both precision directories
  FP32/
    model.safetensors           float32 weights (1.29 GB)
    encoder/config.json
    tokenizer/tokenizer.json
    tokenizer/tokenizer_config.json
    code/layachoice_v2_spec.py
    code/layachoice_v2_dataset.py
    code/layachoice_v2_render.py
    manifest.json               frozen dataset manifest
    rl_agent_config.json        agent config, temperature, precision: fp32
    PRECISION_MANIFEST.json     precision, source revision, conversion, hashes
    SHA256SUMS
  FP16/
    model.safetensors           float16 weights (644 MB)
    encoder/config.json
    tokenizer/tokenizer.json
    tokenizer/tokenizer_config.json
    code/layachoice_v2_spec.py
    code/layachoice_v2_dataset.py
    code/layachoice_v2_render.py
    manifest.json
    rl_agent_config.json        agent config, temperature, precision: fp16
    PRECISION_MANIFEST.json
    SHA256SUMS

8.1 What differs

	FP32	FP16
Weight dtype	float32	float16
Weight file	1.29 GB (decimal)	644 MB
Weight SHA256	964f8fe5dde2812ad3902d8237035bfb109885d4fefef590307bc681bd8fab14	ce550b6a85d0a3a96083f842fa0432c2efa9d19f3acea6f9a093fa7140949f97
Tensors	170 / 170 float32	170 / 170 float16
Parameter count	321,908,998	321,908,998
Keys and shapes	unchanged	identical to FP32
Source	the published epoch-9 artifact, byte-for-byte	cast from the FP32 artifact

The FP16 file was produced with safetensors.torch.load_file → Tensor.to(torch.float16) → safetensors.torch.save_file. Every tensor in this checkpoint is floating point, so all 170 were cast; no integer or boolean tensor was present. Keys, shapes, parameter names and the model graph are unchanged. Nothing was retrained or quantized.

8.2 Which file to use

• FP32/ — reference artifact. Load this when you need the exact published weights and the smallest numerical deviation from the training run.
• FP16/ — same weights after a lossless-in-structure fp16 cast. Choose it when download size or on-disk size matters and a sub-1e-4 change in probabilities is acceptable. See the measured drift below. It does not reduce CPU memory or GPU VRAM under the stock loader; §8.3 explains why.

8.3 Checkpoint storage vs. runtime precision

The FP16 directory changes the checkpoint storage dtype, not the runtime parameter dtype. laya==0.3.21 builds the model with float32 parameters (build_model) and copies the checkpoint tensors into them (load_state_dict); PyTorch casts the values to the parameter dtype during the copy. The stock laya.Agent therefore holds float32 parameters for both directories, so weight memory on CPU and weight VRAM on GPU are the same in both cases (approximately 1.29 GB decimal). The parameter dtype becomes float32; the values loaded from FP16 remain the FP16-rounded ones.

Compute precision is a separate setting:

• On CUDA the Agent enables autocast by default; it can be disabled explicitly, on the OOM fallback path, or when an operator is unsupported. MPS additionally gates autocast per call by batch size.
• On CPU autocast is off by default; LAYA_CPU_AMP=bf16 enables BF16, not FP16.
• Mixed precision is not a guaranteed speedup; measure it on the target hardware.

Runtime parameter memory only drops when the parameters themselves are stored in 16-bit or INT8 form — through an explicit cast or a runtime that genuinely stores and executes quantized weights. A half-precision parameter cast requires keeping the decision head in float32:

agent = laya.Agent("/path/to/FP16", device="cuda")
agent.model.half()
agent.model.act_head.float()   # forward casts pooled features to float32

Measured on CPU with the FP16 directory: parameters 1,228.0 MiB → 614.4 MiB; warm forward 290–382 ms → 747–840 ms (no native FP16 units). act_head must stay float32 because DecisionModel.forward does pooled = h[:, 0].float() before the head (laya/common.py:347); a fully-half model raises mat1 and mat2 must have the same dtype, but got Float and Half. CUDA half-precision with autocast disabled fails the same way. With half parameters and autocast enabled, CUDA worked in the 10-row smoke test.

Savings from INT8 depend on which tensors are converted. The token-embedding table is about 750 MiB of the 1.29 GB, so quantizing Linear layers alone leaves most of the footprint in place. Benchmark speed and accuracy before adopting either path.

8.4 Measured output difference

Smoke input: the first 10 rows of the frozen V2 test split, batch size 1, the same question text, temperature 3.0702.

Comparison	Max abs logit diff	Max abs probability diff	Same argmax
FP32 vs FP16 loaded by stock laya.Agent (CPU)	0.000687	7.2e-7	10 / 10
FP32 vs FP16 loaded by stock laya.Agent (CUDA)	0.000685	7.2e-7	10 / 10
FP32 vs true FP16 under CUDA BF16 autocast	0.078094	1.1e-4	10 / 10

The stock-loader rows differ only because the stored fp16 values are rounded; the model then runs in FP32. A three-row end-to-end agent.predict smoke against both assembled directories returned the same choice on all rows (cap-pdf-extract-text, cap-bash, cap-web-search, all matching gold).

These numbers are a smoke check on 10 rows, not an accuracy benchmark. The 0.998355 test top-1 in §5 comes from the FP32 artifact and is not re-measured here for FP16.

8.5 Download

# whole repository
hf download eric-ml-nlp/Delveta-LayaChoice-v2 --local-dir Delveta-LayaChoice-v2

# only one precision
hf download eric-ml-nlp/Delveta-LayaChoice-v2 --include "FP32/*" \
  --local-dir Delveta-LayaChoice-v2
hf download eric-ml-nlp/Delveta-LayaChoice-v2 --include "FP16/*" \
  --local-dir Delveta-LayaChoice-v2

8.6 Verification

Each directory carries a SHA256SUMS covering every file in that directory. Verify from inside the directory:

cd FP32 && sha256sum -c SHA256SUMS
cd FP16 && sha256sum -c SHA256SUMS

From the repository root, the top-level SHA256SUMS covers both trees.

9. Installation and inference

The deployment artifact loads through the public laya entry point. The code/layachoice_v2_render.py module must be installed before constructing the Agent; without it the stock short option cap applies and the option cards are truncated. Both precision directories ship the same code/.

pip install "laya==0.3.21" torch transformers
export PYTHONPATH="/path/to/Delveta-LayaChoice-v2/FP32/code:$PYTHONPATH"

import layachoice_v2_render as R   # 256-token option builder
R.install()                        # patch the option builder BEFORE the Agent
import laya

# choose one directory; each is self-contained
agent = laya.Agent("/path/to/Delveta-LayaChoice-v2/FP32", device="cuda")
# agent = laya.Agent("/path/to/Delveta-LayaChoice-v2/FP16", device="cuda")

state = "user turn or conversation state"
ans = agent.predict(
    state,
    {"capability": {
        "type": "choice",
        "instructions": "Which capability should handle the user's request?",
        # K capability cards (0..3) plus exactly one REJECT card
        "criteria": {
            "cap-...": "...",
            "cap-...": "...",
            "cap-...": "...",
            "REJECT": "None of the listed capabilities applies.",
        },
    }},
    max_len=1024,
    head_max_len=768,
)
choice = ans["answers"]["capability"]["choice"]   # capability id or "REJECT"

Notes:

• The Agent reads max_len, head_max_len and the temperature vector from rl_agent_config.json. The precision directory's config records "precision": "fp32" or "precision": "fp16" and "weight_file".
• The question must contain between 1 and 4 criteria, including exactly one REJECT card. This mirrors the training-time dynamic contract.
• R.install() is required; the stock option budget does not match the trained pipeline.
• REJECT is a decision, not an error: it means no candidate applies and the caller should escalate or ask for clarification.

10. Evaluation method

• Validation runs every epoch on the frozen 2,407-row validation split and is used only to select the checkpoint.
• Test runs once on the frozen 9,729-row test split with the fitted temperature; it is never read during training or selection.
• Top-1 is exact match between the selected option and the gold slot. The calibration split is removed from training before the first optimizer step.
• Reject metrics treat REJECT as the positive class: recall = TP / (TP + FN), FPR = FP / (FP + TN).
• ECE is computed on the option distribution after temperature scaling.

11. Reproducibility

• Base model revision and subfolder are pinned in manifest.json and rl_agent_config.json.
• Data hashes are frozen in manifest.json; each precision directory's SHA256SUMS covers every file in that directory, and the root SHA256SUMS covers both trees.
• The FP16 artifact records its source revision, weight SHA256 and conversion method in FP16/PRECISION_MANIFEST.json; FP32/PRECISION_MANIFEST.json does the same for the reference artifact.
• Seeds: training 42, calibration 20260922; the calibration indices and IDs are recorded in the manifest.
• Training entry point: bash run.sh all --epochs 10 --out <fresh-directory>.
• Full training and evaluation evidence (per-epoch validation reports, selection record, calibration, baseline, final test report, curves, log, manifest): eric-ml-nlp/Delveta-LayaChoice-v2-checkpoints.

12. Limitations

• The model is trained for in-domain capability selection on the frozen recall_threshold_082 bundle. It does not generalise automatically to unseen capability taxonomies, card renderings, or domains.
• Validation and test rows come from the same frozen benchmark family; the very high test metrics should be read as in-domain performance, not open-domain robustness.
• The model assumes the question contains exactly one REJECT card and 0–3 capability cards. Feeding a question without REJECT, or with more than three capability cards, is outside the trained contract.
• Confidence is calibrated on 400 held-out in-distribution rows. Under distribution shift, both top-1 and calibration may degrade.
• The 5-row loader smoke in the export stage is a load check, not an accuracy measurement; the reported accuracy figures come from the full test evaluation.
• FP16 is a storage-precision copy. With the stock laya loader it runs as FP32 and does not reduce runtime parameter memory or VRAM; the measured drift in §8.4 is a 10-row smoke check, not a full test re-evaluation.

13. Citation and links

• Training code: https://github.com/Eric-LLMs/Delveta/tree/main/fine_tuning/laya_finetune/V2
• Training evidence and checkpoints: https://huggingface.co/eric-ml-nlp/Delveta-LayaChoice-v2-checkpoints
• Base model: https://huggingface.co/convaiinnovations/laya
• FP32 weights: https://huggingface.co/eric-ml-nlp/Delveta-LayaChoice-v2/tree/main/FP32
• FP16 weights: https://huggingface.co/eric-ml-nlp/Delveta-LayaChoice-v2/tree/main/FP16