Bundle vllm_paddleocr_seqpos plugin; card: real plugin install path

README.md

@@ -110,9 +110,17 @@ print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special
 ### vLLM (production)
 
 Serve greedy (`temperature=0`) with **vLLM ≥ 0.26**. The **only** extra requirement is
-the sequential image-token position regime (above): install the `vllm_paddleocr_seqpos`
-plugin and set `OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential`. Without it, structured
-pages loop and accuracy collapses.
+the sequential image-token position regime (above). Stock vLLM always serves PaddleOCR-VL
+in the *grid* regime and never reads `mm_token_type_ids`, so you need a tiny vLLM general
+plugin — **it ships in this repo** under `vllm_paddleocr_seqpos/`:
+
+```bash
+pip install "git+https://huggingface.co/BDRC/tibetan-ocr#subdirectory=vllm_paddleocr_seqpos"
+export OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential   # 'grid' / unset = no-op
+```
+
+Then serve as usual. Without this, structured (book/list) pages skip or merge lines and
+CER regresses badly.
 
 <!-- The snippets above are the intended native path; run them once on a GPU box
      against deploy/fast_inference/bench.py (the source-of-truth harness) before

vllm_paddleocr_seqpos/README.md

@@ -0,0 +1,111 @@
+# vllm-paddleocr-seqpos
+
+A tiny [vLLM](https://github.com/vllm-project/vllm) plugin that makes PaddleOCR-VL serve in the
+**`sequential` image-token position regime** (1D positions for image tokens) instead of vLLM's
+built-in **`grid`** regime (2D image-grid M-RoPE).
+
+Use it to serve any PaddleOCR-VL checkpoint that was **trained/validated with
+`image_token_positions=sequential`** (i.e. with `mm_token_type_ids` zeroed) — e.g.
+`elie_v6_coarse_grow26_ep2`. Without it, vLLM silently serves such a checkpoint in the wrong
+regime and the model regresses.
+
+## Background: the two regimes
+
+PaddleOCR-VL uses M-RoPE. A per-token mask, `mm_token_type_ids`, decides how *image* tokens are
+positioned:
+
+- **`sequential`** — mask zeroed → image tokens get plain 1D positions, exactly like text
+  (M-RoPE degenerates to standard 1D RoPE). This is the regime our checkpoints are trained in.
+- **`grid`** — mask marked → image tokens get native 2D image-grid M-RoPE.
+
+The HF reference path honors both (`deploy/fast_inference/bench.py --backend hf
+--image-token-positions {sequential,grid}`), and `patch_checkpoint.py` warns that serving the
+`sequential` regime needs an *engine-side* override. This package is that override for vLLM.
+
+## Why a plugin is needed (and why there's no simpler knob)
+
+vLLM's in-tree model `vllm/model_executor/models/paddleocr_vl.py` (checked on 0.27.1) computes
+positions itself in `PaddleOCRVLForConditionalGeneration.get_mrope_input_positions`. It
+**unconditionally builds the 2D grid** from `image_grid_thw`/`mm_features` and **never reads
+`mm_token_type_ids`**. Consequences:
+
+- There is **no `mm_processor_kwargs` / processor knob** to switch it to sequential — the regime
+  lives in vLLM's own position code, not in the processor output vLLM consumes.
+- `get_mrope_input_positions` runs in a **separate `EngineCore`/worker process**, so a
+  monkeypatch applied in the driver process (before `LLM(...)`) does **not** reach it.
+
+vLLM does, however, call `vllm.general_plugins` entry points in **process0, the EngineCore
+process, and every worker** (`vllm/plugins/__init__.py`). So the robust, version-stable fix is a
+general plugin that overrides that one method wherever it runs. That's all this package does.
+
+## What it does
+
+On load, and only when the environment variable `OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential`,
+it replaces `get_mrope_input_positions` with a version that returns plain 1D sequential positions
+(`arange(n)` broadcast across the 3 M-RoPE sections, `mrope_position_delta=0`) for the whole
+sequence, image tokens included. The image features still merge via `input_ids ==
+config.image_token_id`, so zeroing positions does **not** remove the image — it only changes the
+position encoding.
+
+It is a **strict no-op** for any other value (`grid`, unset), so grid-trained checkpoints are
+untouched. It is idempotent (vLLM may load plugins more than once per process).
+
+## Install
+
+Install into the **same Python environment that runs vLLM** (e.g. the DLAMI `/opt/pytorch` venv
+on the serving box). No dependencies are declared on purpose — the plugin only touches
+`vllm`/`torch`/`numpy`, which are already present in a serving env.
+
+```bash
+/opt/pytorch/bin/pip install /path/to/deploy/vllm_paddleocr_seqpos
+```
+
+Verify the entry point is registered:
+
+```bash
+python -c "from importlib.metadata import entry_points; \
+print([e.value for e in entry_points(group='vllm.general_plugins') if e.name=='paddleocr_seqpos'])"
+# -> ['vllm_paddleocr_seqpos:apply']
+```
+
+## Use
+
+Set the env var in the process that constructs `LLM(...)` **before** the engine starts; the
+EngineCore/worker children inherit it:
+
+```bash
+export OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential   # or 'grid' / unset to disable (no-op)
+```
+
+Then serve/construct vLLM as usual. On activation you'll see a one-line WARNING:
+`paddleocr-seqpos: forcing SEQUENTIAL (1D) image-token M-RoPE for PaddleOCR-VL ...`.
+
+In the OCR web-app this is automated: `ocr_app.worker.paddle_engine` resolves the checkpoint's
+regime (from `experiment_config.json` / `info.json`, defaulting to `sequential`) and sets this
+env var itself before loading the engine.
+
+## Verification (vLLM 0.27.1, A10G)
+
+Against the production path (chat-template prompt, pyvips long-side 2500, greedy + DRY), on the
+6-page `TSAM-CHOE.pdf`, under default multiprocessing:
+
+| page | `grid` (stock vLLM) | `sequential` (this plugin) |
+| ---- | ------------------- | -------------------------- |
+| 2    | 816 gen-tokens      | **479 gen-tokens** (matches the HF `--image-token-positions sequential` oracle) |
+| 4    | drops a catalog entry (`༤༨` → `རི་ཁྲོད`) | correct `༤༨ ཆོས་གསུམ / ༤༩ རི་ཁྲོད / ༥༠` |
+
+`grid` stays at 816 with the plugin installed but the flag unset/`grid` (confirmed no-op).
+
+## Alternative
+
+If you'd rather not run a serving-side plugin, **train/validate the checkpoint in the `grid`
+regime** (`--image-token-positions grid`) and record `image_token_positions: grid` in its
+`experiment_config.json`. Grid is vLLM's native path, so serving then needs no plugin. This
+package exists so you don't *have* to retrain to serve existing `sequential` checkpoints.
+
+## Files
+
+- `vllm_paddleocr_seqpos/__init__.py` — the plugin (`apply()` entry point + the sequential
+  `get_mrope_input_positions`).
+- `pyproject.toml` — declares the `vllm.general_plugins` → `paddleocr_seqpos = vllm_paddleocr_seqpos:apply`
+  entry point.

vllm_paddleocr_seqpos/pyproject.toml

@@ -0,0 +1,19 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "vllm-paddleocr-seqpos"
+version = "0.1.0"
+description = "vLLM general plugin: serve PaddleOCR-VL with sequential (1D) image-token M-RoPE."
+requires-python = ">=3.10"
+# No install deps: the plugin only touches vllm/torch/numpy, which are already present in the
+# serving environment (importing them itself would wrongly pin versions).
+
+[project.entry-points."vllm.general_plugins"]
+# vLLM loads this in process0, the EngineCore process, and every worker process, and calls it
+# with no args. The plugin is a no-op unless OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential.
+paddleocr_seqpos = "vllm_paddleocr_seqpos:apply"
+
+[tool.setuptools]
+packages = ["vllm_paddleocr_seqpos"]

vllm_paddleocr_seqpos/vllm_paddleocr_seqpos/__init__.py

@@ -0,0 +1,82 @@
+"""vLLM general plugin: force PaddleOCR-VL into the *sequential* image-token M-RoPE regime.
+
+Why this exists
+---------------
+PaddleOCR-VL uses M-RoPE. The ``mm_token_type_ids`` mask selects the image-token position
+regime: **sequential** (mask zeroed -> image tokens get plain 1D positions, like text) vs
+**grid** (mask marked -> native 2D image-grid M-RoPE). vLLM's in-tree PaddleOCR-VL model
+(``vllm/model_executor/models/paddleocr_vl.py``) does not read ``mm_token_type_ids`` at all;
+its ``get_mrope_input_positions`` *unconditionally* builds the 2D grid from ``image_grid_thw``.
+So stock vLLM can only serve the ``grid`` regime, and there is no processor/kwarg knob to
+change it.
+
+Checkpoints trained/validated in the ``sequential`` regime (e.g. ``elie_v6_coarse_grow26_ep2``)
+therefore suffer a train/serve mismatch under vLLM: line skips/merges on structured book/list
+pages and a large CER regression. This plugin closes that gap by overriding the one method so
+image tokens get 1D sequential positions -- verified on vLLM 0.27.1 to reproduce the HF
+``--image-token-positions sequential`` oracle exactly (TSAM-CHOE page 2: 479 generated tokens,
+vs 816 under grid, with the catalog numbering restored).
+
+How it loads
+------------
+vLLM calls ``vllm.general_plugins`` entry points with no args in process0, the EngineCore
+process, and every worker process (``vllm/plugins/__init__.py``), which is exactly where
+``get_mrope_input_positions`` runs. The override is a class-attribute swap, applied once
+(idempotent -- plugins may be loaded multiple times per process).
+
+Gating
+------
+The plugin is a **no-op** unless the environment variable ``OCR_VLLM_IMAGE_TOKEN_POSITIONS``
+equals ``sequential`` (case-insensitive). The serving process (``ocr_app.worker.paddle_engine``)
+sets this from the checkpoint's resolved regime before constructing the engine; the child
+processes inherit it. Anything else (``grid``/unset) leaves vLLM's native behaviour untouched.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+ENV_FLAG = "OCR_VLLM_IMAGE_TOKEN_POSITIONS"
+_state = {"patched": False}
+
+
+def _sequential_mrope_positions(self, input_tokens, mm_features) -> tuple[Any, int]:  # noqa: ANN001, ARG001
+    """1D sequential positions for the whole sequence (image tokens included).
+
+    Equivalent to zeroing ``mm_token_type_ids``: every token (text and image) gets the plain
+    ``0..n-1`` position broadcast across the 3 M-RoPE sections, so M-RoPE degenerates to
+    standard 1D RoPE. ``mrope_position_delta`` is 0 so decode continues at ``n, n+1, ...``.
+    """
+    import numpy as np
+    import torch
+
+    n = len(input_tokens)
+    positions = np.broadcast_to(np.arange(n, dtype=np.int64), (3, n))
+    return torch.from_numpy(np.ascontiguousarray(positions)), 0
+
+
+def apply() -> None:
+    """Entry point invoked by vLLM in every process. Idempotent; gated by env."""
+    if _state["patched"]:
+        return
+    regime = os.environ.get(ENV_FLAG, "").strip().lower()
+    if regime != "sequential":
+        logger.debug("paddleocr-seqpos: %s=%r != 'sequential'; not patching", ENV_FLAG, regime)
+        return
+    try:
+        from vllm.model_executor.models import paddleocr_vl as mod
+    except Exception:
+        logger.exception("paddleocr-seqpos: could not import PaddleOCR-VL model; not patching")
+        return
+
+    mod.PaddleOCRVLForConditionalGeneration.get_mrope_input_positions = _sequential_mrope_positions
+    _state["patched"] = True
+    logger.warning(
+        "paddleocr-seqpos: forcing SEQUENTIAL (1D) image-token M-RoPE for PaddleOCR-VL "
+        "(mm_token_type_ids-zeroed regime). Grid-trained checkpoints must NOT set %s=sequential.",
+        ENV_FLAG,
+    )