neural_codec/README.md

# neural_codec — DCVC-RT patch selection

Internal package for **DCVC-RT** neural-codec patch selection: the codec's per-frame
**bit-cost map** decides which video patches to feed the VLM (regions the codec spends more
bits on — motion / new detail — are kept; predictable background is dropped), as the neural
alternative to the traditional HEVC (`cv-preinfer`) path.

> **Usage, environment setup, and the controllable `codec.dcvc` parameters are documented in
> the top-level [`../README.md`](../README.md).** This file is an internal file reference.

## Files

| File | Role |
|------|------|
| `dcvc_rt_engine.py` | Loads DCVC-RT intra/inter nets; `DMCIBitmap` / `DMCBitmap` add `compute_bitmap` (per-frame `(H/16, W/16)` bit-cost map) via a streaming `reset_sequence` / `step` API. Loads the bundled DCVC-RT source from `DCVC/` (no env var); checkpoints default to `dcvc_rt_intra.tar` / `dcvc_rt_inter.tar` in this dir (`DCVC_INTRA_TAR` / `DCVC_INTER_TAR` to override). |
| `codec_dcvc_config.py` | Single source of truth — reads the `codec.dcvc` block of `../processor/preprocessor_config.json`. |
| `dcvc_readiness_gen.py` | Config-driven generator: runs the readiness pipeline (`codec_tools/`) with DCVC bit-cost as the score source. Invoked by the model's codec path (`../processor/codec_video_processing_magevl.py::_run_dcvc_rt`). |
| `reproduce_bench.py` | Reproduce the evaluated selection for one video (config-driven; presets `cap12` / `b50` / `s95_b50`). |
| `codec_loader.py` | Load precomputed assets → model inputs via the release codec helpers. |
| `infer_dcvc_rt.py` | Standalone end-to-end demo (`--asset_dir` or `--video`). |
| `precompute_dcvc_rt.py` | Standalone, CLI-flag-driven batch precompute (video(s) → assets). |
| `canvas_assembler.py` | Top-k patch selection + canvas packing used by the standalone precompute path. |
| `codec_tools/` | Vendored readiness pipeline (frame sampling, grouping, 2×2-block selection, canvas packing). |
| `DCVC/` | Bundled DCVC-RT source (MIT, [microsoft/DCVC](https://github.com/microsoft/DCVC)): the `src/` package the engine imports + the CUDA-kernel source under `src/layers/extensions/inference/`. No external checkout / `DCVC_RT_ROOT` needed. |
| `dcvc_rt_intra.tar` / `dcvc_rt_inter.tar` | DCVC-RT checkpoints. |

## Notes / limitations

- The bit-cost map is the summed **y-bits** estimate from DCVC-RT's Gaussian entropy model
  (the dominant, spatially-resolved term); the small hyperprior `z` term is omitted (it is a
  ranking signal). It is **not** run through the RANS arithmetic coder.
- `patch=16` is mandatory — it must match the image processor
  (`preprocessor_config.json: patch_size=16, merge_size=2`). The `codec.patch=14` field is a
  separate cv-preinfer internal and does not apply here.
- Canvases are square; non-16:9 videos are letterboxed, so wide videos waste some budget on
  padding. Tune `codec.dcvc.max_pixels` if needed.
- DCVC-RT decodes every frame `0..max(sampled)` to keep temporal references valid, so long
  videos are slow — use `codec.dcvc.max_side` and multiple GPUs.
- The DCVC CUDA kernels fall back to pytorch when the compiled extension is unavailable
  (slower but numerically fine, and deterministic on the fallback path).