AlphaAvatar/persona-speaker-attribute-onnx

AlphaAvatar Persona - Speaker Attribute (wav2vec2-large-robust 6L age/gender, ONNX)

Speaker age/gender model used by the AlphaAvatar Persona plugin (alphaavatar.persona.speaker.attribute). It maps a raw 16 kHz waveform to a pooled 1024-d hidden state, an age score and three gender logits.

This repository replaces the branch-based storage in AlphaAvatar/plugins-persona (branch speaker_attribute_onnx), which is now legacy.

Licence warning. The upstream weights are CC-BY-NC-SA-4.0 (non-commercial, share-alike). The legacy repository declared apache-2.0 for this artifact; that declaration does not match the verified provenance and is corrected here. Commercial use is not permitted by the upstream licence.

Artifact

File	model.onnx
SHA256	75c5cc3debc2013215cee5f331a66b59bd7205da170fb61abab46fc4507df7be
Size	346.6 MiB
ONNX IR / opset	7 / 12
Exported by	pytorch 1.13.1
Parameters	90,808,836

model.onnx is a byte-identical copy of w2v2l6.onnx from AlphaAvatar/plugins-persona at commit 530618cc4ebd8c1aa2a995fc12a180d594535d3f (branch speaker_attribute_onnx). Only the file name changed; the SHA256 above equals the Git-LFS object id on that branch. No conversion or re-export was performed.

Provenance (verified, not assumed)

Upstream: audeering/wav2vec2-large-robust-6-ft-age-gender at revision a681b720dafd12b9dd7b6d13fb437c7b6b197fd3 (model.safetensors, SHA256 4a0f33264b690efcdee7d6cb803404486f9f93fd58355bbb0e6aa6f40db1e2e8), licensed CC-BY-NC-SA-4.0. Paper: https://arxiv.org/abs/2306.16962.

Two independent checks:

1. Weights. All 102 of 102 ONNX initializers that kept their PyTorch parameter names (wav2vec2.*, age.*, gender.*) are bit-identical to the same-named tensors in the upstream checkpoint. Zero mismatches, max absolute difference 0.0.

2. Known-answer test. The upstream model card publishes the output for signal = np.zeros((1, 16000)). This ONNX reproduces it:

   	age	child	female	male
   upstream card	0.3079211	0.00848487	0.0051472	0.9863679
   this ONNX (softmaxed)	0.30792558	0.00848555	0.00514768	0.98636677

   Max absolute difference 1.13e-06.

Not established: who ran the export, and with what script (unknown in the manifest). audEERING publish their own ONNX export at doi:10.5281/zenodo.7761387; this file was not compared against that release, because it was not retrieved here.

Input / output

input   signal          float32  [1, time]        raw mono waveform @ 16 kHz
output  hidden_states   float32  [1, 1024]        mean-pooled last-layer states
output  logits_age      float32  [1, 1]           ~0..1, i.e. 0..100 years
output  logits_gender   float32  [1, 3]           RAW logits, order [child, female, male]

Two things that are easy to get wrong

1. logits_gender are logits, not probabilities. The upstream PyTorch forward() applies torch.softmax to the gender head; this ONNX export does not. For the card's silent input it returns [-1.60957503, -2.10939479, 3.14608812], which sums to -0.57. Apply softmax yourself.

2. The index order is [child, female, male] - index 0 is child. Softmaxing the vector above gives [0.00849, 0.00515, 0.98637], which lines up with the card's child / female / male columns to 1.1e-06.

Preprocessing

The upstream model expects the waveform to be normalised by Wav2Vec2FeatureExtractor with do_normalize=true, i.e. zero mean and unit variance, (x - mean) / sqrt(var + 1e-7). This is not part of the graph and must be applied by the caller.

AlphaAvatar drives this over a 3.0 s window with a 1.0 s hop (window_size_samples=48000, step_size_samples=16000).

Postprocessing

years  = float(np.clip(logits_age, 0.0, 1.0)) * 100.0
probs  = softmax(logits_gender, axis=-1)
label  = ["child", "female", "male"][int(np.argmax(probs))]

Known issues in the current AlphaAvatar caller

Verified against AlphaAvatar commit 65c83b7dfb29134e190f80e12493317f54b14a10. These are not fixed by this artifact; they are recorded here as evidence.

id	severity	issue
GENDER_LABEL_ORDER_MISMATCH	high	speaker_cache.py sets _gender_labels = ["female", "male", "child"]. The verified order is ["child", "female", "male"], so derived gender is systematically mislabelled.
MISSING_WAVEFORM_NORMALISATION	medium	speaker_attribute_runner.py passes the raw waveform straight to the graph, skipping the zero-mean / unit-variance step the upstream model requires.
NON_COMMERCIAL_UPSTREAM_LICENCE	high	The legacy repository declared apache-2.0; the verified upstream is cc-by-nc-sa-4.0.

See model_manifest.json -> known_issues for the full evidence trail.

Examples

examples/input.json describes the exact input (the upstream card's silent zeros((1, 16000)) known-answer case) and examples/output.json records this model's outputs for it, so the artifact can be re-verified from a clean checkout.

Licence

CC-BY-NC-SA-4.0, inherited from the verified upstream. Non-commercial use only; derivatives must carry the same licence. See LICENSE and NOTICE.