Tiny Aya Hallucination Guard - Models
Collection
Linear probe weights, LayerMix stacker, and conformal thresholds for Tiny Aya Global (Lexsi Track 2b). • 1 item • Updated
Linear probes on the residual stream of CohereLabs/tiny-aya-global that predict whether a greedy answer is wrong, plus conformal thresholds for a three-way abstain / answer / retrieve policy.
| name | position | best layer (1-based) | best-layer AUROC | LayerMix CV AUROC |
|---|---|---|---|---|
linear_ans_last |
ans_last | 19 | 0.8717 | 0.8762 |
linear_q_last |
q_last | 22 | 0.8403 | 0.8467 |
Each probe directory contains weights.safetensors (per-layer L2
logistic regression + LayerMix stacker), metadata.json (CV AUROCs,
config hash, feature hash), per_layer_auroc.csv, and
manifest.json.
Calibration split: N=None, pos_rate=None
| α | τ_abs | calib coverage | calib risk | risk UCB | guarantee met |
|---|---|---|---|---|---|
| 0.05 | 0.0000 | 0.000 | 0.000 | 1.000 | False |
| 0.10 | 0.2010 | 0.006 | 0.000 | 0.079 | True |
| 0.15 | 0.2563 | 0.042 | 0.109 | 0.143 | True |
| 0.20 | 0.2915 | 0.062 | 0.162 | 0.192 | True |
See policy/thresholds.json for the full output (τ_ret, tau grid, config hash).
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import json
p = snapshot_download("basab1142/tinyaya-hguard-probes")
w = load_file(f"{p}/linear_ans_last/weights.safetensors")
meta = json.loads(open(f"{p}/linear_ans_last/metadata.json").read())
# w["weights"] shape (n_layers, d_model)
# w["bias"] shape (n_layers,)
# w["scaler_mean"], w["scaler_std"] for z-scoring the residual stream
# w["layermix_stacker_weights"] combines meta["layermix"]["top_k_layers_1based"]
Full pipeline, data, and eval harness:
https://github.com/basab1142/hallucination-detection-and-abstention
(or the companion dataset repo — see configs/default.yaml).