Thanabordee/AudioLaya-poc

Audio-Laya — Direct Speech-to-Decision PoC (checkpoints)

Trainable delta weights for an English-only proof-of-concept that classifies call audio as legitimate or spam directly from audio, with no ASR on the inference path.

16 kHz mono call audio
  -> frozen Whisper-small.en encoder (768)
  -> trainable projector (768 -> 1024)
  -> append to frozen ModernBERT text states
  -> Laya decision head
  -> P(legitimate), P(spam)

Whisper and Laya stay frozen and are not redistributed here — these files contain only the parameters that were trained. Whisper's decoder is never called on this path.

Files

File	Experiment	Trainable params	Val macro F1	Size
projector_head/best.pt	B: projector + Laya decision head	28,086,785	1.000	108 MB
projector_only/best.pt	A: projector only	1,838,592	1.000	7.1 MB
projector_semantic/best.pt	A + transcript semantic alignment (10 epochs)	1,838,592	0.996	7.1 MB
projector_semantic_3epochs/best.pt	A + semantic alignment, 3-epoch sensitivity check	1,838,592	0.993	7.1 MB

Each checkpoint is a dict with experiment, epoch, validation_macro_f1, trainable_state, trainable_parameter_count, and model_ids. In A only the projector is in trainable_state; in B it also holds Laya's type embedding, decision Transformer, and option scorer.

The semantic variants add spam cross-entropy + 0.05 × InfoNCE(projected audio, paired transcript) at temperature 0.07. Transcripts are training-time supervision only — inference reads raw audio.

Usage

These weights require the code in ThanabordeeN/AudioLaya:

git clone https://github.com/ThanabordeeN/AudioLaya.git
cd AudioLaya && python -m venv .venv && . .venv/bin/activate
python -m pip install -r requirements.txt

# fetch a checkpoint from this repo into runs/projector_head/best.pt, then:
python -m src.inference path/to/call.wav --checkpoint runs/projector_head/best.pt

The first run also downloads openai/whisper-small.en and convaiinnovations/laya.

Results

Same 282-call internal test split (146 Harper legitimate, 136 FTC spam; 576 chunks), chunk probabilities averaged per original call:

Method	Transcript at inference	Macro F1	AUROC
C: Whisper ASR -> standard Laya	Yes	0.892	0.949
A: projector only	No	0.996	1.000
B: projector + head	No	1.000	1.000

On 14 legitimate external AppTek calls, A and B produced 0 false alarms.

Accuracy across all three evaluation sets on a common 0--1 axis, with each set's majority-class accuracy as a dashed line. In-domain robocall screening is at ceiling for the direct-audio variants; zero-shot transfer to MInDS-14 collapses onto the majority-class baseline. Hatching marks route C, which consumes transcript text and is not a matched input route for A and B. The three-epoch urgency bar is exploratory. Figure and script: src/plot_accuracy_comparison.py in the GitHub repository.

Zero-shot transfer (no target-task training) was weak: MInDS-14 14-way intent reached 0.080 (A) and 0.062 (B) accuracy against a 0.085 majority baseline, while the transcript path (C) reached 0.936. Urgency tone reached 0.450 (A) and 0.725 (B) accuracy. The semantic-alignment variant improved held-out audio->transcript Recall@1 from 0.002 to 0.333 but did not lift MInDS accuracy above its majority baseline — embedding alignment and downstream choice-head generalization are separate problems.

Limitations — read before using

• Labels are confounded with corpus. Every positive example is from FTC and every negative from HarperValleyBank. The high in-domain scores are a pipeline comparison, not evidence of real-world spam detection. A matched-source, source-disjoint evaluation is required first.
• Not calibrated. Laya emits a calibration warning; do not read the probabilities as calibrated confidence.
• English only, and the model inherits the coverage limits of whisper-small.en and Laya.
• Chunked input. Audio is cut into non-overlapping chunks up to 30 s; only the trained paths were evaluated. Zero-shot transfer to unseen tasks is near or below chance.
• Training audio is not redistributed. The source datasets are linked in the GitHub README.

License

MIT for the released weights (see the GitHub repository). Base models keep their own terms — check the openai/whisper-small.en and convaiinnovations/laya model cards.

Citation

@software{nammungkun2026audiolaya,
  author = {Nammungkun, Thanabodee},
  title  = {Audio-Laya: Direct Speech-to-Decision PoC},
  year   = {2026},
  url    = {https://github.com/ThanabordeeN/AudioLaya}
}