| --- |
| license: apache-2.0 |
| tags: |
| - audio |
| - audio-embeddings |
| - text-to-audio-retrieval |
| - persian |
| - multilingual |
| - visualears |
| - audiomosaic |
| --- |
| |
| # Audio Mosaicist-1 |
|
|
| Audio Mosaicist-1 is a multilingual text-to-AudioMosaic bridge for searching large unstructured audio corpora with natural-language prompts. It maps Qwen3 text embeddings into the frozen AudioMosaic acoustic `z` space, then retrieves and localizes candidate audio events. |
|
|
| This repository contains the bridge artifacts and helper scripts. It does not contain the full private Persian audio corpus used to build the current VisualEars mining index. |
|
|
| ## What it is |
|
|
| - Text tower: `mlx-community/Qwen3-Embedding-0.6B-4bit-DWQ` for the current build. |
| - Audio tower: frozen AudioMosaic ViT-B/16 pretrained encoder. |
| - Bridge: small 1024 -> 128 projection plus category/audio centroids and text prototypes. |
| - Current anchor languages: English and Persian. |
| - Current event/noise anchors: 15 VisualEars environment categories. |
|
|
| ## Included files |
|
|
| - `projection_category_qwen3_to_audiomosaic_z.npy`: category-level Qwen3-text to AudioMosaic-z bridge. |
| - `projection_exact_anchor_qwen3_to_audiomosaic_z.npy`: exact-anchor bridge. |
| - `text_category_prototypes.npy` and `text_category_prototypes.jsonl`: text-side category prototypes. |
| - `audio_category_centroids.npy`: AudioMosaic-side category centroids. |
| - `audio_anchor_index.with_inferred_categories.parquet`: anchor metadata used for the current bridge. |
| - `categories.json`: current category list. |
| - `audiomosaic_text_bridge_query.py`: minimal query helper. |
| - `scripts/asr_query_audiomosaic_text_bridge.py`: full-corpus search helper used in the VisualEars mining run. |
| - `scripts/asr_localize_audiomosaic_events.py`: multi-scale timestamp localizer for retrieved candidates. |
| - `scripts/asr_make_audio_mosaicist_extension_pack.py`: helper for building local-language extension packs. |
|
|
| ## Quick use |
|
|
| Download the repo and point the query script at an existing AudioMosaic embedding index: |
|
|
| ```bash |
| python scripts/asr_query_audiomosaic_text_bridge.py \ |
| --bridge-dir Audio-Mosaicist-1 \ |
| --index /path/to/audiomosaic_index \ |
| --prompts prompts.jsonl \ |
| --out search_results.jsonl |
| ``` |
|
|
| Then recover exact audio rows and localize the event windows: |
|
|
| ```bash |
| python scripts/asr_localize_audiomosaic_events.py \ |
| --candidates search_results.jsonl \ |
| --manifest exact_manifest_rows.jsonl \ |
| --out localized_events.jsonl \ |
| --repo-dir /path/to/audiomosaic-vit-b16-pretrained \ |
| --device cuda:0 |
| ``` |
|
|
| The helper scripts are intentionally plain Python so people can adapt them to their own storage layout. |
|
|
| ## Language coverage |
|
|
| The bridge was calibrated with English and Persian prompts, so those two are validated. Because Qwen3 Embedding is multilingual, prompts in other Qwen-supported languages may work through cross-lingual alignment, but they are not yet measured. For non-English/Persian use, treat this as zero-shot until you add local-language calibration prompts or anchors. |
|
|
| ## Extending to a new language |
|
|
| You do not need a carefully structured dataset. The easiest extension is a JSONL/CSV with any of these fields when available: |
|
|
| - `audio_path` or HF dataset locator fields (`source`, `file_path`, `row_index`, `audio_col`) |
| - `label` or `caption` in your language |
| - `lang`, for example `ar`, `hi`, `tr`, `de` |
| - optional `category` if you already know it |
|
|
| Recommended extension workflow: |
|
|
| 1. Embed your audio anchors with AudioMosaic into 128-d `z` vectors. |
| 2. Embed your local-language labels/prompts with Qwen3 Embedding. |
| 3. Add the new text/audio pairs to the bridge training table. |
| 4. Retrain or adapter-train the 1024 -> 128 projection and text prototypes. |
| 5. Evaluate category hit@k and a small manual localization smoke before trusting mining output. |
|
|
| For unlabeled audio, first mine clusters in AudioMosaic space, manually name a small number of representative clusters in your language, then use those labels as bridge anchors. |
|
|
| You can start an extension pack with: |
|
|
| ```bash |
| python scripts/asr_make_audio_mosaicist_extension_pack.py \ |
| --input your_audio_or_labels.csv \ |
| --out audio_mosaicist_extension_your_lang.jsonl \ |
| --lang your_language_code |
| ``` |
|
|
| The extension pack is a staging file: embed the listed audio with AudioMosaic, embed the labels with Qwen3 or another multilingual text encoder, then fit a small adapter or refit the bridge. |
|
|
| ## Timestamp precision |
|
|
| AudioMosaic retrieval is coarse. The production path is: |
|
|
| 1. Prompt -> AudioMosaic `z` query. |
| 2. Full-corpus top-k retrieval. |
| 3. Exact manifest join to recover the chiseled audio row. |
| 4. Multi-scale localization: |
| - coarse calibrated AudioMosaic-style window score |
| - fine relative sliding-window score inside the candidate region |
| - optional energy trimming for boundary cleanup |
|
|
| Fine-window scores are relative, not fully calibrated probabilities. For VisualEars-grade event alerts, the next model should train a dedicated fine event localizer on mined/mixed spans. |
|
|
| ## Current limitations |
|
|
| - The public artifact is a bridge and mining toolkit, not a full end-to-end event detector. |
| - The current category set is 15 VisualEars pilot categories, not the full 69-category target taxonomy. |
| - Fine localization scores are useful for ranking and QA, but should not be treated as calibrated probabilities. |
| - Non-English/Persian prompting should be evaluated with local anchors before production use. |
|
|
