README.md

metadata

license: apache-2.0
language:
  - en
  - zh
library_name: transformers
base_model: Qwen/Qwen3-Omni-30B-A3B-Captioner
tags:
  - audio
  - audio-language-model
  - instruction-following
  - rubric-based-evaluation
  - judge-model
pipeline_tag: audio-text-to-text

AnyAudio-Judge-30B

AnyAudio-Judge-30B is the dynamic rubric-based audio judge reported in the paper. It is initialized from Qwen3-Omni-30B-A3B-Captioner and fine-tuned on the AnyAudio-Judge SFT Corpus.

For each binary rubric item describing one verifiable aspect of an audio caption, the model predicts yes / no and produces a one-sentence evidence string drawn from what it heard in the audio. Aggregating the per-item soft probabilities yields a fine-grained alignment score that is significantly more sensitive to partial mismatches than a single holistic match/mismatch judgment.

Companion benchmark: cucl2/AnyAudio-Judge-Bench Companion corpus: cucl2/AnyAudio-Judge-Corpus Smaller variant: cucl2/AnyAudio-Judge-7B

Headline numbers (AnyAudio-Judge Bench, accuracy ↑)

Model	Avg (en)	Avg (zh)
Qwen3-Omni-30B-A3B-Captioner (dynamic rubric, no fine-tuning)	76.77	76.66
Gemini-2.5-Pro (holistic)	77.72	80.01
AnyAudio-Judge-30B (this checkpoint)	84.45	85.26

Training

• Base: Qwen3-Omni-30B-A3B-Captioner
• Corpus: 105K (audio, instruction, rubric, CoT) tuples
• Stage 1 — SFT: full-parameter fine-tuning, 1 epoch, 16 × H20 96GB, lr 1e-5, per-device bs 4
• Stage 2 — GRPO (separate release): LoRA r=16, α=32, 1 epoch on 8,454 hard samples

This release contains the SFT-only stage (matching the "+SFT" row of the ablation table). The +GRPO improvement reported in the paper can be reproduced by running GRPO on top of this checkpoint.

Usage

from anyaudio_judge import AnyAudioJudge, decompose_instruction

caption = "A gentle, delicate female voice, with soft and smooth pitch, calm and restrained throughout."
rubric  = decompose_instruction(caption)

judge = AnyAudioJudge.from_pretrained("cucl2/AnyAudio-Judge-30B")
result = judge.judge("./demo.wav", rubric)
print("alignment_score:", result.score)
for item in result.items:
    print(item.question, "->", item.answer)

License

Apache-2.0, inheriting the license of the base Qwen3-Omni-30B-A3B-Captioner model.

Citation

@misc{anyaudiojudge2026,
  title  = {AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following},
  author = {Anonymous Authors},
  year   = {2026},
  note   = {Preprint, under submission}
}