README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-4B-Instruct
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - ocr
  - document-parsing
  - document-ai
  - layout
  - vision-language-model
  - qwen3_vl
---

# Hanji Parse 4B

**Hanji Parse 4B** is a document-parsing vision-language model fine-tuned from
[`Qwen/Qwen3-VL-4B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct).
Given a page image, it emits a JSON array of layout-grounded content blocks —
each block a semantic section of the page (a paragraph, a heading with its
content, a key-value panel, or a whole table) with a bounding box and its
transcribed text. Tables are transcribed as GitHub-Flavored Markdown inside a
single block. Figures, photos, and signatures are returned as image blocks.

## Output contract

The model returns **JSON only**: an array of records

```json
[{"bbox_2d": [x1, y1, x2, y2], "text_content": "..."}]
```

- `bbox_2d` is `[left, top, right, bottom]` in **normalized 0–1000 page
  coordinates** (divide by 1000 and multiply by the page width/height to
  recover pixel boxes).
- Blocks are **semantic sections**, typically 5–30 per page — not one record
  per line, cell, or field.
- Tables come back as **one block** containing a GitHub-Flavored Markdown
  table; every cell rides inside the markdown.
- Non-text graphics (photos, charts, stamps, signatures) come back with
  `text_content` = `"<image>"`.
- Checkboxes are transcribed inline as `[x]` / `[ ]` before their label.
- An empty page returns exactly `[]`.

## Usage — read this before running the model

### 1. Image preprocessing

- Downscale so the image is at most **2,000,000 pixels** (2 MP), preserving
  aspect ratio. **Never upscale.**
- Floor each dimension to a **multiple of 32** (Qwen3-VL uses 16-px patches
  with 2×2 spatial merge → 32 px per visual token).
- Use LANCZOS resampling; feed the result as a PNG.

```python
from PIL import Image

MAX_PIXELS, PATCH = 2_000_000, 32

def preprocess(img: Image.Image) -> Image.Image:
    w, h = img.size
    scale = min(1.0, (MAX_PIXELS / (w * h)) ** 0.5)
    w, h = int(w * scale) // PATCH * PATCH, int(h * scale) // PATCH * PATCH
    return img.convert("RGB").resize((w, h), Image.LANCZOS)
```

### 2. The prompt

Send the page image followed by exactly this text as the user turn. Do not
paraphrase, extend, or reformat it.

````text
Detect every BLOCK in this document and return a JSON array.

Schema: [{"bbox_2d":[x1,y1,x2,y2], "text_content":"..."}]
Coordinates: normalized 0-1000 page coordinates; [x1,y1,x2,y2] = [left,top,right,bottom].

EMPTY PAGE: If the page has no legible content, return exactly []. Otherwise, transcribe every legible content block; a page with only one legible item is not empty.

Your DEFAULT is to GROUP. Most such pages form 5-30 records; sparse pages may form only 1-4. More than 30 remains unusual. A
"block" is a semantic SECTION (a panel, a heading + its content, a key-value group,
or a whole table), NOT a single line, cell, or field. If you are emitting one record
per line, per cell, or per form field, STOP - that is WRONG. When a region is not a
clean table, you must STILL group it into section blocks; never fall back to
one-record-per-element.

Block categories:
- Text records: ONE record per block - a heading TOGETHER WITH the lines beneath it,
  a paragraph, a list, or a key-value field group. text_content = the block's text,
  with "\n" between its lines. DO NOT emit one record per line.
- Table records: a table or dense grid of cells, rendered as GitHub-Flavored Markdown
  (| col | col |\n|---|---|\n| cell | cell |). Never emit one record per cell or
  per row - the markdown carries every cell. A repeated item|amount list (receipt
  lines, menu items) IS a table.
  HEADERLESS TABLES: if a table has no visible column headings, DO NOT invent any.
  Render only the visible rows/cells in their observed order. If Markdown syntax needs
  a separator row, use empty header cells rather than synthetic names like "Column 1".
  TABLE CELL TEXT: cell contents must be plain visible text. Do NOT add Markdown
  emphasis or formatting inside cells (no **bold**, _italics_, backticks, or headings)
  unless that formatting is the only way to preserve information that is visible on
  the page.
  TALL TABLES: A logical table on one page is ONE block regardless of row count. Include
  every visible row in one GFM table. Its bbox_2d must tightly enclose the full table
  from the first row through the last row.
- Image records: ONE record per photo, figure, chart, scan, or non-text graphic.
  text_content = "<image>". Handwritten signatures, cursive e-signatures, initials,
  signature scribbles, and signature marks are ALWAYS images - do NOT transcribe or
  guess them, even if partly readable. Printed labels such as "Signature:" remain text.
  Do NOT emit for logos < 40 px wide.

CRITICAL - transcribe MEANING, not layout glyphs:
- Fill-in / blank lines: emit ONLY the label, NOT the blank. Write "Name:" - never
  "Name:________________". For signature fields, keep the printed label as text and
  emit the actual signature mark itself as an image record with text_content="<image>".
- NEVER reproduce decorative rules or separators - rows of *, -, _, =, ., or any
  repeated glyph. Omit them entirely; they are not content.
- Checkboxes / Y-N / selection fields: write the field and its options on one line with the marks inline - "<row label> Y [x] N [ ]", "<label>: [x] Yes [ ] No"; "[x]" filled, "[ ]" empty; one record per field, not per option.
- text_content must equal the VISIBLE text of the block - never pad, repeat, or
  continue a character run. No single text block exceeds ~20 lines; split a longer
  section at its sub-headings.

Grouping rules:
- A section heading and the content beneath it (down to the next heading) form ONE block.
- A field label and its value are ONE block - EVEN in a dense report header. Write
  "Visit Date: 08/12/2025" as one record; NEVER split the label from its value
  ("Visit Date:" + "08/12/2025" as two records is WRONG).
- On forms, when a key has an associated value, merge the key and value into the same
  block and bbox so the association is explicit. If several related key-value fields
  are visually grouped, emit the group as one block with one "Label: value" line per field.
- A bordered or visually-grouped PANEL (e.g. a "PRESCRIBER INFORMATION" box with all
  its fields) is ONE block - join its label:value pairs with "\n".
- A row of related cells that is NOT a clean table (a lab-result line, a transaction
  line) is ONE block - join the cells into one line; NEVER one record per cell.
- NEVER emit a bare value, a single cell, or a lone field as its own record.
- Keep COLUMNS separate: two side-by-side panels (e.g. Patient | Ordering Provider,
  Bill To | Ship To) are TWO blocks - never merge across the gutter.
- DO NOT merge unrelated neighboring panels or sections.
- Group by semantic relationship, NOT by bbox size. Small unrelated regions must stay
  separate. Example: a page title at the top-left and a page number at the top-right
  are TWO blocks, even if both boxes are small and on the same horizontal band. Only
  group items that belong to the same section, panel, list, table, or key-value group.
- Prefer CORRECT grouping over tight boxes: a block's bbox may be wide and may lightly
  touch a neighbor - do NOT over-split a section just to keep boxes small or separate.
- Page-edge content: if visible text touches or sits near the page boundary, inspect the
  full edge carefully and make the bbox include the entire visible glyphs/block, even if
  the box must start at 0 or end at 1000. Do NOT shrink edge boxes inward.
- Bbox coverage is strict: every transcribed character in text_content MUST be inside
  that record's bbox_2d, with no clipped letters. This matters most for small regions,
  rotated/non-horizontal text, page-edge text, headers/footers, stamps, and fax strips.
  Use a tight box around the actual region, but never make it so tight that any visible
  character you transcribed falls outside the box.

Output JSON only.
Checkboxes: transcribe every checkbox as [x] if marked or [ ] if unmarked, placed before its label (e.g. "[x] Allergies reviewed"; Y/N pairs as "Y [x] N [ ]"). Include every checkbox, including checkbox grids and Y/N option pairs.
````

### 3. Decoding

- **Greedy**: `temperature 0.0`, `top_p 1.0`, no repetition penalty.
- `max_new_tokens = 8192`.
- Strongly recommended: **JSON-schema-constrained decoding** (xgrammar in
  SGLang, `guided_json` in vLLM) with this schema:

```json
{
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "bbox_2d": {
        "type": "array",
        "items": {"type": "integer"},
        "minItems": 4,
        "maxItems": 4
      },
      "text_content": {"type": "string"}
    },
    "required": ["bbox_2d", "text_content"]
  }
}
```

### Quickstart (transformers)

```python
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL = "hanji-dev/hanji-parse-4b"
model = AutoModelForImageTextToText.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(MODEL)

image = preprocess(Image.open("page.png"))  # see preprocessing above
PROMPT = "..."  # the exact prompt above

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": PROMPT},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### Serving (SGLang)

N-gram speculative decoding gives a large decode speedup on this output
format (the repeated JSON keys draft extremely well):

```bash
python -m sglang.launch_server \
  --model-path hanji-dev/hanji-parse-4b \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 8192 \
  --speculative-algorithm NGRAM \
  --speculative-num-draft-tokens 16 \
  --speculative-ngram-max-bfs-breadth 10 \
  --enable-deterministic-inference \
  --context-length 16384
```

A ready-to-run server that implements the full preprocessing + prompt contract
(and a schema-extraction API around it) is available at
https://github.com/youlearn-ai/hanji.

## License

Apache-2.0. Fine-tuned from
[`Qwen/Qwen3-VL-4B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct)
(Apache-2.0, © Alibaba Cloud / the Qwen team).