Qwen3-ASR-0.6B-Agent

A speech-in agentic tool-caller fine-tuned from Qwen/Qwen3-ASR-0.6B. It hears a spoken request (zh-TW / English / code-switch), emits a search_contacts tool call, runs a multi-turn disambiguation flow ("which department?" → refined call), and speaks back to the caller in their own language — Traditional Chinese to a Chinese caller, English to an English one. I.e. it does what a Qwen3-Omni-0.6B would do, except Alibaba never shipped a small dense Omni. This is that model, reconstructed: the Qwen3-ASR AuT audio encoder (frozen) + a LoRA-re-elicited Qwen3 decoder.

Conversation and tool use, in one 0.6B model. Each turn the model decides whether to emit a <tool_call> or a natural-language reply to the user, and drives the whole multi-turn flow itself.

Language mirroring (default / main): zh and code-switched callers get zh-TW replies; English callers get English. The earlier English-only adapter is preserved at revision v1-english-replies.

Why it's interesting

On a bilingual zh-TW phone-attendant task it beats Qwen2.5-Omni-3B while being 1/5 the parameters, ~3× faster on CPU, fp16 ~1 GB — and it answers the caller in their own language:

Metric	Qwen3-ASR-0.6B-Agent	Qwen2.5-Omni-3B
Single-turn agent (1500 clips)	94.3% / 1.3% misroute	92.6% / 1.7%
Mandarin (zh-TW)	99.2% / 0.1%	98.2% / 0.4%
Multi-turn disambiguation (free-running)	93.8%	—
Spoken reply language	mirrors caller (zh-TW / en)	—

Why it wins: Qwen3-ASR's AuT encoder (purpose-built on ~20M h) out-perceives Omni's Whisper-derived encoder; for a narrow tool-calling task, a great encoder + a competent 0.6B decoder beats a good encoder + a 3B decoder.

Conversation + tool use (the agent loop)

The model emits both <tool_call> turns and dedicated natural-language turns to the user. Example (a Chinese caller asking for an ambiguous name):

caller (zh audio): 「我找鄭柏翰」
  model → <tool_call>{"name":"search_contacts","arguments":{"query":"鄭柏翰"}}</tool_call>
  tool  ← [4 people named "Carol Cheng" in different depts]
  model → 找到多位Carol Cheng，分別在會計部、業務部、物流部、工程部，請問您要找哪個部門?   ← reply to user
caller (zh audio): 「業務部」
  model → <tool_call>{"name":"search_contacts","arguments":{"query":"Carol Cheng","department":"Sales"}}</tool_call>
  tool  ← [unique: Carol Cheng, Sales, ext 9438]
  model → 為您轉接業務部的Carol Cheng，分機9438。                                          ← reply to user

An English caller gets the same flow with English replies ("I found several people named … Which department?" → "Connecting you to … in Sales, extension 9438.").

Usage

This LoRA loads on top of the base via the qwen-asr package's modeling code (it carries the qwen3_asr model class; targets transformers==4.57.6).

import torch
from qwen_asr.core.transformers_backend.modeling_qwen3_asr import Qwen3ASRForConditionalGeneration
from qwen_asr.core.transformers_backend.processing_qwen3_asr import Qwen3ASRProcessor
from peft import PeftModel

proc = Qwen3ASRProcessor.from_pretrained("Qwen/Qwen3-ASR-0.6B")
base = Qwen3ASRForConditionalGeneration.from_pretrained("Qwen/Qwen3-ASR-0.6B", dtype=torch.float16)
model = PeftModel.from_pretrained(base.thinker.to("cuda").eval(), "Luigi/Qwen3-ASR-0.6B-Agent").eval()

SYS = ('You are a phone attendant for a Taiwan office. To find a colleague, call the directory tool '
       'by writing exactly <tool_call>{"name":"search_contacts","arguments":{"query":"<name as heard>"}}'
       '</tool_call>. If several people share that name, ask which department, then call the tool again '
       'adding "department":"<dept>". After a unique result, connect the caller; if none match, say not found.')

text = f"<|im_start|>system\n{SYS}<|im_end|>\n<|im_start|>user\n<|audio_pad|><|im_end|>\n<|im_start|>assistant\n"
enc = proc(text=text, audio=[your_16k_mono_waveform], sampling_rate=16000, return_tensors="pt")
enc = {k: (v.cuda() if torch.is_tensor(v) else v) for k, v in enc.items()}
enc["input_features"] = enc["input_features"].half()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
    out = model.generate(**enc, max_new_tokens=64, do_sample=False, eos_token_id=151645)
print(proc.tokenizer.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
# -> <tool_call>{"name": "search_contacts", "arguments": {"query": "蔡孟儒"}}</tool_call>

The agent loop: parse the <tool_call>, run your directory lookup, feed back a <|im_start|>tool\n<tool_response>[...]</tool_response><|im_end|> turn, and continue.

Reproduce from scratch

Everything needed to rebuild this adapter is public:

• Dataset — Luigi/contact-attendant-zhtw: the ~14.4k request clips + 13.5k multi-turn dialogs + directory.csv + the data-generation scripts.
• Code — the reproduce/ folder in this repo: the trainer, the three eval scripts, the fuzzy resolver.py/tools.py, the scorer, pinned requirements.txt, and a full step-by-step REPRODUCE.md.

# 1. env (CUDA 12.4, py3.11): torch 2.6 + transformers 4.57.6 + peft + qwen-asr (see REPRODUCE.md)
pip install -r reproduce/requirements.txt
pip install --no-deps qwen-asr==0.0.6 qwen-omni-utils    # + neutralize its av import (REPRODUCE.md §1)
# 2. data
huggingface-cli download Luigi/contact-attendant-zhtw --repo-type dataset --local-dir data-repro
# 3. mirror the caller's language in the reply turns (zh-TW for zh/mix, English for en)
python ../reproduce/relangify_replies.py \
    --in data-repro/dialogs_phase3_train.jsonl --out data-repro/dialogs_phase3_train_ml.jsonl
# 4. train (from inside data-repro/ so clips/... resolve)
cd data-repro && python ../reproduce/train_qwen3asr_agent.py \
    --train dialogs_phase3_train_ml.jsonl --out ../runs/qwen3asr-agent --epochs 2

For the original English-only adapter (revision v1-english-replies), skip step 3 and train on dialogs_phase3_train.jsonl directly.

The recipe

• Base Qwen/Qwen3-ASR-0.6B; we keep its thinker and freeze the AuT audio encoder + projector — reuse the ~20M-h speech↔text alignment and only re-elicit the deliberately de-instruction-tuned decoder to follow prompts / emit tool calls.
• LoRA on the decoder only — regex model\.layers\.\d+\.(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj), r=32, α=64, dropout=0.05, CAUSAL_LM; LoRA params upcast to fp32 for fp16-grad stability.
• fp16, bsz 1 × grad-accum 8, lr 1e-4, 2 epochs, gradient checkpointing — fits an 8 GB GTX-1070.
• Data format: each dialog → ChatML with <|audio_pad|> in user turns; Hermes <tool_call>{…}</tool_call> / <tool_response>[…]</tool_response> as plain text (the ASR tokenizer has no tool template). Loss masked to assistant spans only.
• Eval (numbers above): reproduce/eval_qwen3asr.py (single-turn) + eval_qwen3asr_mt.py / eval_qwen3asr_freerun.py (multi-turn), scored by eval.py.

Training data: ~13.5k synthetic multi-turn zh-TW/en contact-attendant dialogs (request audio → tool call → tool response → reply; incl. department-disambiguation), audio fully synthetic (edge-tts). The assistant reply turns are language-mirrored by reproduce/relangify_replies.py (zh-TW for zh/mix callers, English for en) — tool-call turns are byte-identical, so tool-calling accuracy is unchanged.

Notes & limits

• OOD not_found rejection is the soft axis (~76%) — handle it in your resolver (confidence / distribution-aware threshold), not the model.
• Built for a closed directory + a fuzzy resolver downstream (so perception only needs to land phonetically in-set). General open-domain agentic use is out of scope for a 0.6B.

License: Apache-2.0 (inherits the base). Not affiliated with or endorsed by Alibaba/Qwen.