neural_codec/README.md

neural_codec — DCVC-RT patch selection

Internal package for DCVC-RT neural-codec patch selection: the codec's per-frame bit-cost map decides which video patches to feed the VLM (regions the codec spends more bits on — motion / new detail — are kept; predictable background is dropped), as the neural alternative to the traditional HEVC (cv-preinfer) path.

Usage, environment setup, and the controllable codec.dcvc parameters are documented in the top-level ../README.md. This file is an internal file reference.

Files

File	Role
dcvc_rt_engine.py	Loads DCVC-RT intra/inter nets; DMCIBitmap / DMCBitmap add compute_bitmap (per-frame (H/16, W/16) bit-cost map) via a streaming reset_sequence / step API. Loads the bundled DCVC-RT source from DCVC/ (no env var); checkpoints default to dcvc_rt_intra.tar / dcvc_rt_inter.tar in this dir (DCVC_INTRA_TAR / DCVC_INTER_TAR to override).
codec_dcvc_config.py	Single source of truth — reads the codec.dcvc block of ../processor/preprocessor_config.json.
dcvc_readiness_gen.py	Config-driven generator: runs the readiness pipeline (codec_tools/) with DCVC bit-cost as the score source. Invoked by the model's codec path (../processor/codec_video_processing_magevl.py::_run_dcvc_rt).
reproduce_bench.py	Reproduce the evaluated selection for one video (config-driven; presets cap12 / b50 / s95_b50).
codec_loader.py	Load precomputed assets → model inputs via the release codec helpers.
infer_dcvc_rt.py	Standalone end-to-end demo (--asset_dir or --video).
precompute_dcvc_rt.py	Standalone, CLI-flag-driven batch precompute (video(s) → assets).
canvas_assembler.py	Top-k patch selection + canvas packing used by the standalone precompute path.
codec_tools/	Vendored readiness pipeline (frame sampling, grouping, 2×2-block selection, canvas packing).
DCVC/	Bundled DCVC-RT source (MIT, microsoft/DCVC): the src/ package the engine imports + the CUDA-kernel source under src/layers/extensions/inference/. No external checkout / DCVC_RT_ROOT needed.
dcvc_rt_intra.tar / dcvc_rt_inter.tar	DCVC-RT checkpoints.

Notes / limitations

• The bit-cost map is the summed y-bits estimate from DCVC-RT's Gaussian entropy model (the dominant, spatially-resolved term); the small hyperprior z term is omitted (it is a ranking signal). It is not run through the RANS arithmetic coder.
• patch=16 is mandatory — it must match the image processor (preprocessor_config.json: patch_size=16, merge_size=2). The codec.patch=14 field is a separate cv-preinfer internal and does not apply here.
• Canvases are square; non-16:9 videos are letterboxed, so wide videos waste some budget on padding. Tune codec.dcvc.max_pixels if needed.
• DCVC-RT decodes every frame 0..max(sampled) to keep temporal references valid, so long videos are slow — use codec.dcvc.max_side and multiple GPUs.
• The DCVC CUDA kernels fall back to pytorch when the compiled extension is unavailable (slower but numerically fine, and deterministic on the fallback path).