resoajoe/quality-gate-nano

quality-gate — is this generated video frame usable?

47,122 parameters · 188 KB ONNX · 0.131 ms/frame on a Jetson AGX Orin CPU (7,641 fps)

SCOPE — read this before using it

What it is for: flagging catastrophically degraded frames in chunked video generation — frames where the scene has smeared, posterised, or lost chromatic coherence. It exists to decide "redraw this chunk or keep it" inside a generation pipeline.

What it is NOT for:

• Not an aesthetic or quality judge. It detects collapse, not mediocrity. A frame can be sharp, coherent and completely wrong (incorrect subject, impossible geometry) and pass.
• Not a general image-quality metric. It was trained only on interior-room scenes from two video generators. It has no calibration against human preference, IQA benchmarks, or any standard metric.
• Not a deepfake / AI-generated-image detector. Every training frame is generated. It cannot distinguish real from synthetic.
• Not validated on any domain other than the one described below.

Measured performance

Trained on 1,840 chunks from 34 generation arms across two model families (LTX-Video-2B and Wan 2.2 TI2V-5B), 16,320 frames, 36.1% collapsed.

seed	accuracy	majority baseline	lift	recall (collapsed)	precision
0	0.902	0.749	+0.153	0.672	0.913
1	0.941	0.586	+0.356	0.905	0.951
2	0.930	0.725	+0.205	0.940	0.962
mean	0.924 ± 0.017	—	+0.238 ± 0.086	0.84 ± 0.12	0.94 ± 0.02

Read the lift, not the accuracy. The class balance varies by split, so raw accuracy tracks the prior. Lift over the majority baseline is the honest figure.

Recall varies a lot across seeds (0.67–0.94). Which arms land in the test set matters. If your application cannot tolerate missing a third of collapsed chunks, do not deploy this alone.

How it was evaluated — split by ARM, not by frame

This matters more than the numbers. Three successive evaluations of this same task gave 1.000, +0.036 and +0.238, and only the last is real:

1. Random FRAME split → accuracy 1.000. Frames within a chunk are consecutive video frames, i.e. near-duplicates. A frame split puts copies of the same chunk on both sides and the model memorises chunks.
2. Split by CHUNK → lift +0.036. Better, but within an arm the collapse rate is near-constant (one arm is 98% collapsed, another 8%), so the model can learn "this looks like arm X" instead of learning collapse.
3. Split by ARM → lift +0.238. Entire generation runs held out. The test conditions were never seen in any form. This is the number above.

Labels

Ground truth comes from chunk_qc.inspect_chunk — an existing detector using SIFT keypoint count, Laplacian sharpness and face presence, probed at four points within each chunk. Labels are therefore that detector's opinion, not a human's. This model distils a ~100 ms multi-stage CV pipeline into a 0.131 ms forward pass; it inherits that detector's biases and cannot exceed it.

Input and the crop assumption

float32[N,3,64,64], RGB, scaled to [0,1], resized from a full frame.

It was trained on whole frames where collapse affects the entire image. It is not a crop-and-classify model. Nano vision models are known to be unlearnable below roughly 1% object-to-frame ratio against real backgrounds — so for any task where the target is small, crop to it first. That is not this model's regime; localised artifacts in an otherwise good frame are a documented failure mode below.

Known failure modes

• Localised damage. A frame with one corrupt region and an otherwise clean scene will likely pass. Global degradation is what it learned.
• Recall instability — see the seed spread above.
• Two generators only. LTX-Video-2B and Wan 2.2. Other models' failure signatures may differ.
• One scene type. Interior rooms, fixed camera. No outdoor, motion, or multi-subject footage.
• Inherits chunk_qc's blind spots, including its inability to detect mediocrity.

Learnability score

A training-free learnability ratio R (per loglens-learnability) scores this task at 0.337, the highest of eight tasks tested. However, that predictor failed validation as a general gate — across eight tasks it ranked them at Spearman ρ ≈ 0.57–0.62, so its score should not be read as evidence about this model. The measured lift above is the evidence. See NEGATIVE_generalised_R.md in the study repo.

Usage

import onnxruntime as ort, numpy as np, cv2
s = ort.InferenceSession("quality_gate.onnx", providers=["CPUExecutionProvider"])
img = cv2.resize(cv2.imread("frame.jpg"), (64, 64))          # BGR, full frame
x = img.astype(np.float32).transpose(2, 0, 1)[None] / 255.0
collapsed = int(s.run(None, {"image": x})[0].argmax(1)[0])   # 1 = collapsed

Latency (measured, Jetson AGX Orin, ONNX Runtime CPU)

mode	ms/frame	fps
1 thread	0.313	3,191
default threads	0.131	7,641
batch 32	0.061	16,513

Related

Part of the LogLens study. Tools: tripod, warpcost, vae-mtf, nullcheck.

MIT.

Deployment note: cap the ONNX Runtime thread pool

Measured on a Jetson AGX Orin. ONNX Runtime sizes its intra-op thread pool to the CPU core count, and those workers spin-wait between inferences. Running two 47K-parameter models this way left ~18 threads busy-waiting at ~10.6% of a core each — about 1.9 cores burned continuously to run inferences that take 0.28 ms. A model this small cannot use intra-op parallelism at all.

so = ort.SessionOptions()
so.intra_op_num_threads = 1
so.inter_op_num_threads = 1
so.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
so.add_session_config_entry("session.intra_op.allow_spinning", "0")
sess = ort.InferenceSession("quality_gate.onnx", sess_options=so, providers=["CPUExecutionProvider"])

Measured effect on the same workload: idle CPU 192% → 16.5% of one core, active 231% → 88%, thread count 45 → 22, throughput unchanged. On edge hardware this is the difference between "runs alongside everything else" and "saturates the machine".

What "scalar baseline" means on this card

Every margin quoted here is against a stated baseline, because a margin without one is not a measurement. The baseline is the best single-threshold classifier over ten cheap statistics, fitted optimistically:

mean · std · lapvar · hf (high-frequency energy ratio) · grad (Sobel magnitude) · entropy · centre_edge · radial_slope · row_fft_peak · col_fft_peak

The last four are spatially aware, added after an earlier six-statistic baseline — all global aggregates — was found to systematically overstate model value on spatially structured tasks. A baseline that cannot see where anything is loses to a CNN by default. On one test task that flaw inflated an apparent margin from +0.060 to +0.261.

Two questions are asked with it, and they disagree:

• in-sample — threshold fitted on the data it is scored on. Deliberately generous. Answers is there structure beyond a low-order statistic?
• transferred — threshold fitted on the training corpus, applied unchanged to the target. Answers what should I ship? On one task the in-sample figure was 0.954 and the transferred figure 0.565.

Where this card quotes a single scalar figure without qualification, it is the in-sample one.