Local integration results — 2026-09-18
This report preserves the initial sequential bfloat16 baseline. The current default uses cached, batched float32 computation; its validation is in CACHE-RESULTS.md. The smoke script now exercises the current backend, so rerunning it need not reproduce this historical report's predictions.
The frozen Gemma 4 E2B model successfully processed text, image, speech, video, and video-with-speech inputs through native multimodal inference. All three primitive contracts were exercised on every input category.
15/15 inference calls completed; 13/15 simple expected answers matched. No model training or calibration validation has occurred. These deliberately tiny synthetic fixtures are integration checks, not a quality benchmark or evidence of general multimodal reliability.
| Input | Choice | Noul | Score's highest-probability level |
|---|---|---|---|
| Text describing a cat and no dogs | Correct | Correct | Incorrect |
| Uniform red image | Correct | Correct | Correct |
| Synthesized speech mentioning a dog | Correct | Correct | Correct |
| Red-then-blue video | Correct temporal order | Correct | Correct |
| Red-then-blue video with dog speech | Correct speech classification | Incorrect | Correct visual grade |
The text grading failure assigned roughly 99.98% probability to the wrong level. The combined video/speech input classified the spoken animal correctly, then incorrectly answered the separate dog-presence proposition. These errors directly show why a typed output and highly concentrated probabilities are not enough. Both failures remain in the saved report.
The inference run used one loaded model and sequential questions. First-use kernel compilation, preprocessing, media encoders, and decoder execution all contribute to the recorded per-call times. These are single observations, not controlled warm/cold latency measurements. The processor also emitted an empty-mel-filter warning at load time; broader speech-quality validation remains necessary.
Reproduce
From the project directory, using the existing environment:
python scripts/create_fixtures.py --output-dir work/media
python scripts/smoke.py \
--model /absolute/path/to/full-gemma-4-e2b-mlx-checkpoint \
--media work/media \
--report work/smoke-reproduced.json
pytest -q -o cache_dir=work/pytest-cache
Fixture creation requires macOS's say with the Samantha voice, Pillow, and ffmpeg. It creates a plain color image, synthetic speech, a four-second color sequence, and the same sequence with speech. No user audio or video is read.
The full inference outputs, expected answers, failures, timings, and local model path are in RESULTS.json. The tested dependency versions are in requirements-tested.txt.
Model provenance read from the reused checkpoint's source.json:
- Repository:
mlx-community/gemma-4-e2b-it-4bit - Revision:
238767527555cb75a05732a84dff5d6ba0dd6809 - Backend: MLX-VLM 0.7.1; MLX 0.32.2; Transformers 5.17.0.
- Host: Apple Silicon, 32 GiB physical memory.
Contract and math checks
23 automated tests passed. They cover exclusive and independent probability semantics, weighted grading, Noul criteria ordering, validation failures, finite/stable scores, input-path validation, proper scoring rules, and temperature recovery on a synthetic known-frequency example with separate evaluation outcomes. Lint and formatting checks also passed.
The temperature recovery test verifies the calibration implementation's math. It does not establish calibration of Gemma's predictions. Frozen-model smoke fixtures were not used to fit a temperature or train a head.