Historical methods for the earlier 25,159-example subset and throughput study. The current model card uses the full-suite evaluation methods.
Methods and interpretation
The eight-task adapted local evaluation covers FLEURS en_us/cmn_hans_cn test, LibriSpeech test-clean/test-other, MMAR, public MMAU test-mini, MMSU and MMLU-Pro test with five-shot validation demonstrations. It has 25,159 acceptance and 1,024 earlier development examples. Selection used inspected outcomes, including acceptance outcomes: these are descriptive revalidations, not an untouched holdout, published-score reproduction, equivalence test or coverage of Ming's other tasks.
Greedy text output caps are ASR 512, audio multiple-choice 1,024 and MMLU-Pro 2,048 new tokens. The evaluation uses 10-gram blocking (a local adaptation for MC/text); Ming internal thinking is disabled for MMLU-Pro. ASR uses corpus WER/CER. Primary MC accuracy keeps every example and counts capped/malformed answers wrong. MMAR/MMAU canonicalize anchored unambiguous labels to option text. MMSU counts malformed answers wrong although its upstream scorer drops them. Raw-parsed accuracy is a separate diagnostic.
Tiny reuses historical native-vLLM BF16 reference against the corrected NVFP4 candidate: 16 routed NVFP4 layers, with BF16 routed layers 5, 10, 15, 18, 20, 21 and 22. Both arms used input limit 8,192, client concurrency 512 and stream interval 8 on separate schedules. The HF-only RoPE correction does not imply an HF RoPE error in the native BF16 reference. Schedule-dependent greedy responses and reused outcomes prevent isolated quantization attribution or sub-0.5 pp reproducibility certification.
results/quality.csv uses percentages; deltas and intervals use percentage points. MC intervals are conservative exact discordance-marginal Clopper-Pearson differences with exact two-sided McNemar p-values and paired counts. Tiny's Holm p-values cover only four MC tests. ASR uses 20,000 paired percentile corpus-rate bootstrap replicates, seed 20261025: utterances for FLEURS (speaker IDs unavailable), speakers for LibriSpeech. Tiny's LibriSpeech speaker clusters number 40/33. The CSV reports the primary speaker sensitivity; source reports also retain utterance sensitivity. Nominal 95% and Bonferroni-eight columns assume exchangeable units; MC requires independent pairs and bootstrap coverage is approximate. Multiplicity corrections do not repair selection bias; zero-width intervals do not establish equivalence.
Hardware: NVIDIA RTX PRO 6000 Blackwell, exact SM120 (12.0), measured physical97887MiB/95.593GiB; host RAM about125GiB. Serving: torch 2.13.0, vllm 0.30.0, vllm_omni 0.1.dev3352+g41edeb7fe, flashinfer_python 0.6.18.post1, triton 3.7.1. Preparation: torch 2.8.0, transformers 5.14.1, accelerate 1.10.1, nvidia_modelopt 0.47.0, safetensors 0.8.0. Serving/preparation Torch versions describe distinct environments. Required serving plugins: ming_nvfp4, vllm_omni_register_models. Native VLLM_CUTLASS executes NVFP4 on SM120; Tiny's retained BF16 routed layers use Triton. Portability was not validated by these reports.
Quality profiles: Tiny context 16,384 / server capacity 256, graphs/async, queued client concurrency 512.
results/performance.csv gives median output tokens/s over three fixed-workload trials; audio means audio input/text output, not speech synthesis. Tiny: context 16,384 / capacity 1,024; text client concurrency 512 with 256 output tokens, audio concurrency 1,024 with 128 output tokens; 2,048 warmup plus 2,048 timed requests each.
Thinker model allocation is Tiny BF16 15.97 GiB and Tiny NVFP4 9.50 GiB. It excludes KV cache, workspace, CUDA graphs and other allocations; it is neither checkpoint size nor total GPU memory. Workload hashes identify fixed inputs. evidence_id resolves to relative local artifact names/SHA-256 in provenance.json; original source files remain unchanged and are not distributed. This export checks report hashes/completion, counts, deltas and shared primary values. It does not rerun inference, rehash bulk weights/audio or newly validate every prediction. Only aggregate data is included.
Benchmark references from the local catalog: FLEURS, LibriSpeech, MMAR, MMAU, MMSU, MMLU-Pro. Source datasets retain their own terms; no examples are redistributed.