DEFERRED_PLANS.md

Deferred plans

Designs that were worked out and verified but deliberately not implemented yet.

Implement async_frame_load for real

async_frame_load on the process_video endpoint is currently a documented no-op: the parameter is accepted by app.py and samv2_handler.run_sam_video_inference but never used. It is a vestige of the old samv2 package's async_loading_frames, lost in the transformers rewrite (commit 0f11cd9). The design below implements it as a producer/consumer overlap of CPU frame preprocessing with GPU propagation; it was verified against the installed transformers 5.9.0 source (transformers/models/sam2_video/) in Aug 2026 — re-verify the cited internals if transformers has been upgraded since.

Value: overlaps ~10–20 s of CPU preprocessing (for a ~900-frame video) with GPU propagation inside the billed ZeroGPU window, and avoids the eager path's transient ~11 GB fp32 CPU stack (frames get preprocessed per-chunk instead of all at once).

Design

• Producer thread preprocesses JPEG chunks (CHUNK = 8) on CPU via processor.video_processor(videos=pils, device="cpu", return_tensors="pt").pixel_values_videos[0] and appends each frame via session.add_new_frame(frame, idx) — the session class is designed for streaming append (casts to session dtype, stores on video_storage_device, GIL-atomic dict insert). A threading.Condition guards a _loaded counter and an _error slot; the consumer calls wait_for(frame_idx, timeout_s=90), which blocks until the frame is available, raises TimeoutError on a stalled producer, and re-raises producer exceptions as RuntimeError from original.
• Consumer: propagate_in_video_iterator is unusable — it reads inference_session.num_frames once at iteration start to build a fixed processing_order. Instead run a manual loop calling model(inference_session=session, frame_idx=i, reverse=...) per frame (exactly what the iterator does internally). Forward order range(start_frame_idx, num_total); reverse range(start_frame_idx, -1, -1) if start_frame_idx > 0 else empty. Total frame count is known upfront from the extracted JPEG file list.
• Numeric-parity gate: the only num_frames-sensitive numeric in forward (with streaming=False) is min(num_total_frames, config.max_object_pointers_in_encoder=16) in _get_object_pointers (it also normalizes the object pointers' temporal sine PE). Gate the consumer on len(processed_frames) >= min(N, 16) before the first forward call (wait_for uses target = max(frame_idx + 1, min_ready)) → bit-identical results to the eager path.
• Streaming session setup: init_video_session() with no video arg leaves session.video_height/video_width = None; set them manually after init (plain attributes). Mask seeding (add_inputs_to_inference_session(input_masks=...) + the _use_mask_as_output forward) needs only frame ref_frame_idx to be present — call loader.wait_for(ref_frame_idx) before the seeding forward.
• Bidirectional tracking: the forward pass ends at num_total - 1, which forces the producer to have finished, so the reverse pass (nonzero ref_frame_idx) always runs against a complete session.
• Thread-locality: torch.inference_mode() / torch.autocast decorators are thread-local. The producer does plain CPU preprocessing (needs neither); ALL model forward calls must stay on the main thread.
• Cleanup: wrap propagation in try/finally: loader.join() + move shutil.rmtree(vframes_dir) into the finally (today an exception leaks the frames dir — worth fixing even without async).
• Keep the eager path as the async_frame_load=False branch. When implemented, update the process_video docstring (currently says "reserved; currently has no effect") — it is the MCP tool schema, so load the gradio-mcp-tool-docstrings skill first.