| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-VL-4B-Instruct |
| pipeline_tag: image-text-to-text |
| library_name: transformers |
| tags: |
| - ocr |
| - document-parsing |
| - document-ai |
| - layout |
| - vision-language-model |
| - qwen3_vl |
| --- |
| |
| # Hanji Parse 4B |
|
|
| **Hanji Parse 4B** is a document-parsing vision-language model fine-tuned from |
| [`Qwen/Qwen3-VL-4B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct). |
| Given a page image, it emits a JSON array of layout-grounded content blocks — |
| each block a semantic section of the page (a paragraph, a heading with its |
| content, a key-value panel, or a whole table) with a bounding box and its |
| transcribed text. Tables are transcribed as GitHub-Flavored Markdown inside a |
| single block. Figures, photos, and signatures are returned as image blocks. |
|
|
| ## Output contract |
|
|
| The model returns **JSON only**: an array of records |
|
|
| ```json |
| [{"bbox_2d": [x1, y1, x2, y2], "text_content": "..."}] |
| ``` |
|
|
| - `bbox_2d` is `[left, top, right, bottom]` in **normalized 0–1000 page |
| coordinates** (divide by 1000 and multiply by the page width/height to |
| recover pixel boxes). |
| - Blocks are **semantic sections**, typically 5–30 per page — not one record |
| per line, cell, or field. |
| - Tables come back as **one block** containing a GitHub-Flavored Markdown |
| table; every cell rides inside the markdown. |
| - Non-text graphics (photos, charts, stamps, signatures) come back with |
| `text_content` = `"<image>"`. |
| - Checkboxes are transcribed inline as `[x]` / `[ ]` before their label. |
| - An empty page returns exactly `[]`. |
|
|
| ## Usage — read this before running the model |
|
|
| ### 1. Image preprocessing |
|
|
| - Downscale so the image is at most **2,000,000 pixels** (2 MP), preserving |
| aspect ratio. **Never upscale.** |
| - Floor each dimension to a **multiple of 32** (Qwen3-VL uses 16-px patches |
| with 2×2 spatial merge → 32 px per visual token). |
| - Use LANCZOS resampling; feed the result as a PNG. |
|
|
| ```python |
| from PIL import Image |
| |
| MAX_PIXELS, PATCH = 2_000_000, 32 |
| |
| def preprocess(img: Image.Image) -> Image.Image: |
| w, h = img.size |
| scale = min(1.0, (MAX_PIXELS / (w * h)) ** 0.5) |
| w, h = int(w * scale) // PATCH * PATCH, int(h * scale) // PATCH * PATCH |
| return img.convert("RGB").resize((w, h), Image.LANCZOS) |
| ``` |
|
|
| ### 2. The prompt |
|
|
| Send the page image followed by exactly this text as the user turn. Do not |
| paraphrase, extend, or reformat it. |
|
|
| ````text |
| Detect every BLOCK in this document and return a JSON array. |
| |
| Schema: [{"bbox_2d":[x1,y1,x2,y2], "text_content":"..."}] |
| Coordinates: normalized 0-1000 page coordinates; [x1,y1,x2,y2] = [left,top,right,bottom]. |
| |
| EMPTY PAGE: If the page has no legible content, return exactly []. Otherwise, transcribe every legible content block; a page with only one legible item is not empty. |
| |
| Your DEFAULT is to GROUP. Most such pages form 5-30 records; sparse pages may form only 1-4. More than 30 remains unusual. A |
| "block" is a semantic SECTION (a panel, a heading + its content, a key-value group, |
| or a whole table), NOT a single line, cell, or field. If you are emitting one record |
| per line, per cell, or per form field, STOP - that is WRONG. When a region is not a |
| clean table, you must STILL group it into section blocks; never fall back to |
| one-record-per-element. |
| |
| Block categories: |
| - Text records: ONE record per block - a heading TOGETHER WITH the lines beneath it, |
| a paragraph, a list, or a key-value field group. text_content = the block's text, |
| with "\n" between its lines. DO NOT emit one record per line. |
| - Table records: a table or dense grid of cells, rendered as GitHub-Flavored Markdown |
| (| col | col |\n|---|---|\n| cell | cell |). Never emit one record per cell or |
| per row - the markdown carries every cell. A repeated item|amount list (receipt |
| lines, menu items) IS a table. |
| HEADERLESS TABLES: if a table has no visible column headings, DO NOT invent any. |
| Render only the visible rows/cells in their observed order. If Markdown syntax needs |
| a separator row, use empty header cells rather than synthetic names like "Column 1". |
| TABLE CELL TEXT: cell contents must be plain visible text. Do NOT add Markdown |
| emphasis or formatting inside cells (no **bold**, _italics_, backticks, or headings) |
| unless that formatting is the only way to preserve information that is visible on |
| the page. |
| TALL TABLES: A logical table on one page is ONE block regardless of row count. Include |
| every visible row in one GFM table. Its bbox_2d must tightly enclose the full table |
| from the first row through the last row. |
| - Image records: ONE record per photo, figure, chart, scan, or non-text graphic. |
| text_content = "<image>". Handwritten signatures, cursive e-signatures, initials, |
| signature scribbles, and signature marks are ALWAYS images - do NOT transcribe or |
| guess them, even if partly readable. Printed labels such as "Signature:" remain text. |
| Do NOT emit for logos < 40 px wide. |
| |
| CRITICAL - transcribe MEANING, not layout glyphs: |
| - Fill-in / blank lines: emit ONLY the label, NOT the blank. Write "Name:" - never |
| "Name:________________". For signature fields, keep the printed label as text and |
| emit the actual signature mark itself as an image record with text_content="<image>". |
| - NEVER reproduce decorative rules or separators - rows of *, -, _, =, ., or any |
| repeated glyph. Omit them entirely; they are not content. |
| - Checkboxes / Y-N / selection fields: write the field and its options on one line with the marks inline - "<row label> Y [x] N [ ]", "<label>: [x] Yes [ ] No"; "[x]" filled, "[ ]" empty; one record per field, not per option. |
| - text_content must equal the VISIBLE text of the block - never pad, repeat, or |
| continue a character run. No single text block exceeds ~20 lines; split a longer |
| section at its sub-headings. |
| |
| Grouping rules: |
| - A section heading and the content beneath it (down to the next heading) form ONE block. |
| - A field label and its value are ONE block - EVEN in a dense report header. Write |
| "Visit Date: 08/12/2025" as one record; NEVER split the label from its value |
| ("Visit Date:" + "08/12/2025" as two records is WRONG). |
| - On forms, when a key has an associated value, merge the key and value into the same |
| block and bbox so the association is explicit. If several related key-value fields |
| are visually grouped, emit the group as one block with one "Label: value" line per field. |
| - A bordered or visually-grouped PANEL (e.g. a "PRESCRIBER INFORMATION" box with all |
| its fields) is ONE block - join its label:value pairs with "\n". |
| - A row of related cells that is NOT a clean table (a lab-result line, a transaction |
| line) is ONE block - join the cells into one line; NEVER one record per cell. |
| - NEVER emit a bare value, a single cell, or a lone field as its own record. |
| - Keep COLUMNS separate: two side-by-side panels (e.g. Patient | Ordering Provider, |
| Bill To | Ship To) are TWO blocks - never merge across the gutter. |
| - DO NOT merge unrelated neighboring panels or sections. |
| - Group by semantic relationship, NOT by bbox size. Small unrelated regions must stay |
| separate. Example: a page title at the top-left and a page number at the top-right |
| are TWO blocks, even if both boxes are small and on the same horizontal band. Only |
| group items that belong to the same section, panel, list, table, or key-value group. |
| - Prefer CORRECT grouping over tight boxes: a block's bbox may be wide and may lightly |
| touch a neighbor - do NOT over-split a section just to keep boxes small or separate. |
| - Page-edge content: if visible text touches or sits near the page boundary, inspect the |
| full edge carefully and make the bbox include the entire visible glyphs/block, even if |
| the box must start at 0 or end at 1000. Do NOT shrink edge boxes inward. |
| - Bbox coverage is strict: every transcribed character in text_content MUST be inside |
| that record's bbox_2d, with no clipped letters. This matters most for small regions, |
| rotated/non-horizontal text, page-edge text, headers/footers, stamps, and fax strips. |
| Use a tight box around the actual region, but never make it so tight that any visible |
| character you transcribed falls outside the box. |
| |
| Output JSON only. |
| Checkboxes: transcribe every checkbox as [x] if marked or [ ] if unmarked, placed before its label (e.g. "[x] Allergies reviewed"; Y/N pairs as "Y [x] N [ ]"). Include every checkbox, including checkbox grids and Y/N option pairs. |
| ```` |
|
|
| ### 3. Decoding |
|
|
| - **Greedy**: `temperature 0.0`, `top_p 1.0`, no repetition penalty. |
| - `max_new_tokens = 8192`. |
| - Strongly recommended: **JSON-schema-constrained decoding** (xgrammar in |
| SGLang, `guided_json` in vLLM) with this schema: |
|
|
| ```json |
| { |
| "type": "array", |
| "items": { |
| "type": "object", |
| "properties": { |
| "bbox_2d": { |
| "type": "array", |
| "items": {"type": "integer"}, |
| "minItems": 4, |
| "maxItems": 4 |
| }, |
| "text_content": {"type": "string"} |
| }, |
| "required": ["bbox_2d", "text_content"] |
| } |
| } |
| ``` |
|
|
| ### Quickstart (transformers) |
|
|
| ```python |
| import torch |
| from PIL import Image |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| |
| MODEL = "hanji-dev/hanji-parse-4b" |
| model = AutoModelForImageTextToText.from_pretrained( |
| MODEL, dtype=torch.bfloat16, device_map="auto" |
| ) |
| processor = AutoProcessor.from_pretrained(MODEL) |
| |
| image = preprocess(Image.open("page.png")) # see preprocessing above |
| PROMPT = "..." # the exact prompt above |
| |
| messages = [{ |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": image}, |
| {"type": "text", "text": PROMPT}, |
| ], |
| }] |
| inputs = processor.apply_chat_template( |
| messages, add_generation_prompt=True, tokenize=True, |
| return_dict=True, return_tensors="pt", |
| ).to(model.device) |
| |
| out = model.generate(**inputs, max_new_tokens=8192, do_sample=False) |
| print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### Serving (SGLang) |
|
|
| N-gram speculative decoding gives a large decode speedup on this output |
| format (the repeated JSON keys draft extremely well): |
|
|
| ```bash |
| python -m sglang.launch_server \ |
| --model-path hanji-dev/hanji-parse-4b \ |
| --attention-backend fa3 \ |
| --mm-attention-backend fa3 \ |
| --mem-fraction-static 0.85 \ |
| --chunked-prefill-size 8192 \ |
| --speculative-algorithm NGRAM \ |
| --speculative-num-draft-tokens 16 \ |
| --speculative-ngram-max-bfs-breadth 10 \ |
| --enable-deterministic-inference \ |
| --context-length 16384 |
| ``` |
|
|
| A ready-to-run server that implements the full preprocessing + prompt contract |
| (and a schema-extraction API around it) is available at |
| https://github.com/youlearn-ai/hanji. |
|
|
| ## License |
|
|
| Apache-2.0. Fine-tuned from |
| [`Qwen/Qwen3-VL-4B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) |
| (Apache-2.0, © Alibaba Cloud / the Qwen team). |
|
|
