| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - audio |
| - speech |
| - embedding |
| - retrieval |
| - feature-extraction |
| - efficientat |
| - matryoshka |
| - memory-augmentation |
| library_name: pytorch |
| pipeline_tag: feature-extraction |
| datasets: |
| - custom |
| --- |
| |
| # AS-20M |
|
|
| `AS-20M` is a standalone audio + speech embedding encoder for |
| human-memory augmentation workloads. It uses a native `mn20_as` EfficientAT |
| backbone with the speech/audio LoRA training merged into the released weights, |
| so inference does not require loading a separate adapter. |
|
|
| Canonical name: |
|
|
| - `AS` = audio + speech |
| - `20M` = 19,837,720 loaded parameters, rounded to integer millions |
|
|
| ## Runtime Contract |
|
|
| Input is mono audio resampled to 32 kHz. The expected preprocessing is the |
| EfficientAT mel frontend used during training: |
|
|
| - sample rate: `32000` |
| - FFT: `1024` |
| - window length: `800` |
| - hop size: `320` |
| - mel bins: `128` |
|
|
| The model emits a 1280-dimensional embedding. For Matryoshka runtime profiles, |
| truncate and renormalize: |
|
|
| ```text |
| z1280 = l2norm(model(audio)) |
| z768 = l2norm(z1280[0:768]) |
| z512 = l2norm(z1280[0:512]) |
| z256 = l2norm(z1280[0:256]) |
| z128 = l2norm(z1280[0:128]) |
| ``` |
|
|
| ## Artifacts |
|
|
| - `AS-20M.safetensors`: standalone native EfficientAT embedding model |
| - `config.json`: release and architecture metadata |
| - `preprocessor_config.json`: waveform and mel frontend contract |
| - `manifest.json`: file hashes and source checkpoint lineage |
|
|
| ## Training Summary |
|
|
| This checkpoint was continued from the balanced native `mn20_as` student and |
| trained on an audio-heavy mix of synthetic speech/audio alignment data. The |
| published artifact contains merged weights, not a runtime LoRA adapter. |
|
|
| Source checkpoint: |
|
|
| ```text |
| triembed/checkpoints/mn20_native_merged_aistmix_audioheavy100k175k175k_continue_from_balanced_20260426T143137Z/latest_model.pt |
| ``` |
|
|
| Merged LoRA source: |
|
|
| ```text |
| triembed/checkpoints/mn20_native_lora_aistmix_audioheavy100k175k175k_continue_from_balanced_20260426T143137Z/latest_model.pt |
| ``` |
|
|
| ## Local Gate Metrics |
|
|
| The checkpoint-local heldout gate reported audio-side consistency metrics: |
|
|
| | Metric | Score | |
| |---|---:| |
| | audio cosine | 0.8108 | |
| | embedding Pearson | 0.7953 | |
| | similarity Pearson | 0.8853 | |
|
|
| Internal training runs also tracked text-audio retrieval against a companion |
| text embedding space. Those numbers are not reported here as standalone model |
| capabilities because this release artifact does not include a text encoder. |
|
|
| ## MAEB Audio-Only Comparison |
|
|
| This comparison uses the same 20 MAEB audio-only tasks for all three |
| standalone audio encoders. Cross-modal text-audio MAEB tasks are excluded |
| because base `mn20_as` and Whisper-Tiny do not include a compatible text |
| encoder; no text adapters were invented for those baselines. |
|
|
| Validation: each run completed 20/20 tasks with `exception_count=0`. |
|
|
| | Model | Params | Native output | Mean primary | |
| |---|---:|---:|---:| |
| | base `mn20_as` | 17.9M | 1920d audio feature | 0.3977 | |
| | Whisper-Tiny encoder | 8.2M encoder / 37.8M full | 384d pooled encoder state | 0.3320 | |
| | `AS-20M` | 19.8M | 1280d embedding | 0.4083 | |
|
|
| | Task | base `mn20_as` | Whisper-Tiny | `AS-20M` | |
| |---|---:|---:|---:| |
| | BeijingOpera | 0.8470 | 0.5933 | 0.8349 | |
| | BirdCLEF | 0.2070 | 0.0730 | 0.1730 | |
| | CREMADPairClassification | 0.5458 | 0.5752 | 0.5475 | |
| | CREMA_D | 0.2804 | 0.2995 | 0.3351 | |
| | CREMA_DClustering | 0.0229 | 0.0955 | 0.0943 | |
| | CommonLanguageAgeDetection | 0.1401 | 0.2108 | 0.1799 | |
| | FSD2019Kaggle | 0.5734 | 0.0964 | 0.6230 | |
| | GTZANAudioReranking | 0.8298 | 0.6340 | 0.7747 | |
| | GTZANGenre | 0.8260 | 0.4550 | 0.7300 | |
| | IEMOCAPGender | 0.7790 | 0.5269 | 0.7712 | |
| | JamAltArtistA2ARetrieval | 0.8981 | 0.6786 | 0.8490 | |
| | MInDS14 | 0.0818 | 0.1057 | 0.0967 | |
| | MridinghamTonic | 0.3434 | 0.3080 | 0.3450 | |
| | NMSQAPairClassification | 0.4714 | 0.4360 | 0.5875 | |
| | SIBFLEURS | 0.1515 | 0.1554 | 0.1456 | |
| | VehicleSoundClustering | 0.0065 | 0.1194 | 0.0162 | |
| | VoxCelebSA | 0.2377 | 0.1673 | 0.2601 | |
| | VoxPopuliAccentPairClassification | 0.5158 | 0.5196 | 0.5235 | |
| | VoxPopuliGenderClustering | 0.0057 | 0.0008 | 0.0014 | |
| | VoxPopuliLanguageID | 0.1900 | 0.5900 | 0.2780 | |
|
|
| Interpretation: `AS-20M` is slightly ahead on the 20-task audio-only mean, |
| while base `mn20_as` remains stronger on several music/general-audio tasks. |
| Whisper-Tiny is competitive on some speech/language-adjacent tasks, but it is |
| not a general audio embedding model and is weaker on broad environmental-audio |
| coverage in this comparison. |
|
|
| Artifacts: |
|
|
| - `triembed/results/maeb_audio_only_3model_final_20260505T215838Z.md` |
| - `triembed/results/maeb_audio_only_3model_final_20260505T215838Z.json` |
|
|
| ## Limitations |
|
|
| `AS-20M` is an audio embedding model only. It does not transcribe speech, |
| classify audio events directly, or embed text. Text-audio retrieval requires |
| a separate compatible text encoder/head that is not included in this release |
| artifact. |
|
|
