| # neural_codec — DCVC-RT patch selection |
| |
| Internal package for **DCVC-RT** neural-codec patch selection: the codec's per-frame |
| **bit-cost map** decides which video patches to feed the VLM (regions the codec spends more |
| bits on — motion / new detail — are kept; predictable background is dropped), as the neural |
| alternative to the traditional HEVC (`cv-preinfer`) path. |
| |
| > **Usage, environment setup, and the controllable `codec.dcvc` parameters are documented in |
| > the top-level [`../README.md`](../README.md).** This file is an internal file reference. |
| |
| ## Files |
| |
| | File | Role | |
| |------|------| |
| | `dcvc_rt_engine.py` | Loads DCVC-RT intra/inter nets; `DMCIBitmap` / `DMCBitmap` add `compute_bitmap` (per-frame `(H/16, W/16)` bit-cost map) via a streaming `reset_sequence` / `step` API. Loads the bundled DCVC-RT source from `DCVC/` (no env var); checkpoints default to `dcvc_rt_intra.tar` / `dcvc_rt_inter.tar` in this dir (`DCVC_INTRA_TAR` / `DCVC_INTER_TAR` to override). | |
| | `codec_dcvc_config.py` | Single source of truth — reads the `codec.dcvc` block of `../processor/preprocessor_config.json`. | |
| | `dcvc_readiness_gen.py` | Config-driven generator: runs the readiness pipeline (`codec_tools/`) with DCVC bit-cost as the score source. Invoked by the model's codec path (`../processor/codec_video_processing_magevl.py::_run_dcvc_rt`). | |
| | `reproduce_bench.py` | Reproduce the evaluated selection for one video (config-driven; presets `cap12` / `b50` / `s95_b50`). | |
| | `codec_loader.py` | Load precomputed assets → model inputs via the release codec helpers. | |
| | `infer_dcvc_rt.py` | Standalone end-to-end demo (`--asset_dir` or `--video`). | |
| | `precompute_dcvc_rt.py` | Standalone, CLI-flag-driven batch precompute (video(s) → assets). | |
| | `canvas_assembler.py` | Top-k patch selection + canvas packing used by the standalone precompute path. | |
| | `codec_tools/` | Vendored readiness pipeline (frame sampling, grouping, 2×2-block selection, canvas packing). | |
| | `DCVC/` | Bundled DCVC-RT source (MIT, [microsoft/DCVC](https://github.com/microsoft/DCVC)): the `src/` package the engine imports + the CUDA-kernel source under `src/layers/extensions/inference/`. No external checkout / `DCVC_RT_ROOT` needed. | |
| | `dcvc_rt_intra.tar` / `dcvc_rt_inter.tar` | DCVC-RT checkpoints. | |
|
|
| ## Notes / limitations |
|
|
| - The bit-cost map is the summed **y-bits** estimate from DCVC-RT's Gaussian entropy model |
| (the dominant, spatially-resolved term); the small hyperprior `z` term is omitted (it is a |
| ranking signal). It is **not** run through the RANS arithmetic coder. |
| - `patch=16` is mandatory — it must match the image processor |
| (`preprocessor_config.json: patch_size=16, merge_size=2`). The `codec.patch=14` field is a |
| separate cv-preinfer internal and does not apply here. |
| - Canvases are square; non-16:9 videos are letterboxed, so wide videos waste some budget on |
| padding. Tune `codec.dcvc.max_pixels` if needed. |
| - DCVC-RT decodes every frame `0..max(sampled)` to keep temporal references valid, so long |
| videos are slow — use `codec.dcvc.max_side` and multiple GPUs. |
| - The DCVC CUDA kernels fall back to pytorch when the compiled extension is unavailable |
| (slower but numerically fine, and deterministic on the fallback path). |
|
|
