google

/

gemma-4-12B-it

like 1.32k

Follow

Google 61.9k

Endpoint: http://192.168.44.186:8000/v1/chat/completions
Model: Gemma4
Baseline files: 19 WAV files, 10 seconds each

Summary
Language-only accuracy: 14/19
Full quality prompt language accuracy: 14/19
Average request latency: 6.06s
Original WAV signal check: no baseline file had >=0.999 full-scale clipping by sample threshold.
Language-Only Errors
07_id_Indonesian_r1546_10s.wav expected Indonesian, detected Arabic, confidence high
10_ko_Korean_r1546_10s.wav expected Korean, detected Japanese, confidence high
11_ms_Malay_r1546_10s.wav expected Malay, detected Indonesian, confidence high
12_nl_Dutch_r1546_10s.wav expected Dutch, detected German, confidence high
18_vi_Vietnamese_r1546_10s.wav expected Vietnamese, detected Thai, confidence high
Full Prompt Language Errors
07_id_Indonesian_r1546_10s.wav expected Indonesian, detected Arabic, confidence high
11_ms_Malay_r1546_10s.wav expected Malay, detected Indonesian, confidence high
13_pl_Polish_r1546_10s.wav expected Polish, detected Russian, confidence high
18_vi_Vietnamese_r1546_10s.wav expected Vietnamese, detected Chinese, confidence high
19_zh_Chinese_r1546_10s.wav expected Chinese, detected , confidence
Stress Tests
clipped_chinese expected Chinese audio with obvious clipping/distortion; model languages=["Chinese"]; quality=clean; clipping=none; peak/rms=0.0/-4.75 dBFS; clipped_pct=14.10354
low_volume_english expected English audio, very low volume; model languages=["English"]; quality=clean; clipping=none; peak/rms=-28.54/-43.33 dBFS; clipped_pct=0.0
noisy_english expected English audio with added white noise; model languages=["English"]; quality=clean; clipping=none; peak/rms=-1.75/-16.93 dBFS; clipped_pct=0.0
mixed_english_chinese expected First English, then Chinese; model languages=["English", "Chinese"]; quality=clean; clipping=none; peak/rms=-3.21/-17.91 dBFS; clipped_pct=0.0
mixed_chinese_japanese expected First Chinese, then Japanese; model languages=["Japanese"]; quality=clean; clipping=none; peak/rms=-2.9/-16.75 dBFS; clipped_pct=0.0
Additional Manual Stress Observations
Reversed English speech was detected as Arabic speech with poor quality, not as reversed/garbled English.
Pure generated noise was detected as no speech, but incorrectly described as silent.
True silence caused the model to ask for an audio file instead of returning structured no-speech analysis.
Files
Raw JSON: /Users/ai/Desktop/linux/gemma4_audio_deep_eval/raw_results.json
Language CSV: /Users/ai/Desktop/linux/gemma4_audio_deep_eval/language_detection.csv
Quality CSV: /Users/ai/Desktop/linux/gemma4_audio_deep_eval/quality_assessment.csv
Stress CSV: /Users/ai/Desktop/linux/gemma4_audio_deep_eval/stress_tests.csv
主要结果：

19 个 10 秒语音，语言-only 测试：14/19 正确。
综合 prompt 测试：14/19 正确。
常错语言：Indonesian、Malay、Dutch、Polish、Vietnamese，容易被判成相近或无关语言。
中文能识别为 Chinese，但长摘要时会重复、跑偏、截断。
混合英语+中文能识别出两种语言。
混合中文+日语只识别成 Japanese，漏掉中文。
明显削波样本，信号检测 clipped_pct=14.1%，模型仍判 clean / none。
低音量、加噪声，也倾向判 clean。
倒放英文被误判成 Arabic，不会稳定识别“乱码/倒放”。
纯噪声能判断无语音，但描述成 silent；真静音反而要求“提供音频”。