Add codec video backend & docs (README.md)

README.md

@@ -23,10 +23,34 @@ The model is distributed as a HuggingFace `transformers` checkpoint with custom
 
 ## Requirements
 
+### Base (image + frame-sampling video)
+
 ```bash
 pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord
 ```
 
+### Optional: codec video backend
+
+The model ships a second video backend (`video_backend="codec"`) that replaces
+uniform frame sampling with codec-aware **canvas packing** driven by motion
+vectors and bit-cost — typically yielding stronger long-video accuracy at the
+same token budget. To enable it you need two extra pieces:
+
+```bash
+# 1. The cv-preinfer CLI (PyPI: codec-video-prep) drives canvas extraction.
+pip install codec-video-prep opencv-python
+
+# 2. A working `ffmpeg` binary must be on PATH.
+#    Verify with: ffmpeg -version
+```
+
+**ffmpeg version:** ffmpeg **4.4.x – 7.x** is recommended.
+
+The codec backend additionally needs **POSIX `flock`** (already present on
+Linux/macOS) for the on-disk result cache, and roughly **2 GB free disk** under
+`$ONLINE_CODEC_CACHE_DIR` (defaults to `$HF_HOME/online_codec_v2`) per
+processed video.
+
 ## Quick start
 
 The repository ships a ready-to-run `demo_inference.py` that covers both image and video paths.
@@ -90,11 +114,54 @@ out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
 print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
 ```
 
+### Video — codec backend (recommended for long videos)
+
+The codec backend is exposed as a single processor kwarg
+(`video_backend="codec"`). Everything else — canvas extraction via
+`cv-preinfer`, on-disk caching, patch-position bookkeeping, chat-template
+rewriting — happens inside `processor(...)`:
+
+```python
+# Make sure: `pip install codec-video-prep opencv-python` and ffmpeg on PATH.
+messages = [{"role": "user", "content": [
+    {"type": "video"},
+    {"type": "text", "text": "Describe what happens in this long video."},
+]}]
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+inputs = processor(
+    text=[text],
+    videos=["long_clip.mp4"],
+    video_backend="codec",
+    max_pixels=150000,          # per-canvas pixel budget; lower if OOM
+    return_tensors="pt",
+    padding=True,
+    # Optional: override codec defaults from preprocessor_config.json
+    # codec_config={"target_canvas": 32, "group_size": 32, "images_per_group": 4},
+)
+inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}
+
+out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
+print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+```
+
+Defaults for the codec pipeline live in `preprocessor_config.json` under the
+`"codec"` key (`target_canvas=32`, `group_size=32`, `images_per_group=4`,
+`patch=14`, `min_group_frames=8`, `max_group_frames=64`); they can be
+overridden per call via `codec_config={...}`.
+
+**Short-video behaviour:** if the input video has fewer frames than
+`target_canvas` requires (or fewer than `min_group_frames`), a `UserWarning`
+is emitted and inference proceeds with however many canvases `cv-preinfer`
+can actually form. For very short clips, falling back to the frame-sampling
+backend is usually a better choice.
+
 ## Notes
 
 - The vision tower is a OneVision-style encoder; the language backbone is **Qwen3-8B**.
-- `chat_template.jinja` follows the Qwen3 chat format and emits `<|vision_start|>...<|vision_end|>` placeholders; the processor expands them per-frame for video.
-- Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation.
+- `chat_template.jinja` follows the Qwen3 chat format and emits `<|vision_start|>...<|vision_end|>` placeholders; the processor expands them per-frame (frames backend) or per-canvas-patch-run (codec backend).
+- Two video backends are available via `processor(..., video_backend=...)`: **`"frames"`** (default, uniform sampling) and **`"codec"`** (canvas packing via `cv-preinfer`, requires `codec-video-prep` + `ffmpeg`).
+- Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends.
 
 ## License