nvidia/NVIDIA-Nemotron-Parse-2.0

Model Overview

Description:

NVIDIA Nemotron Parse 2.0 transforms document images into structured, machine-readable representations with text, layout classes, bounding boxes, and reading-order information. Given a Red, Green, Blue (RGB) document image and a task prompt, the model produces formatted text and spatial annotations for document elements such as titles, paragraphs, captions, tables, charts, page headers, page footers, footnotes, pictures, and bibliography entries. Compared with NVIDIA Nemotron Parse v1.2, NVIDIA Nemotron Parse 2.0 adds an approximately 20k-token vocabulary expansion for more efficient multilingual support, chart-aware document parsing with the <class_Chart> class token, and updated training coverage for chart/table-heavy documents. NVIDIA Nemotron Parse 2.0 is intended for document understanding, information retrieval, data extraction, and multimodal data-curation workflows.

This model is ready for commercial or non-commercial use.

License/Terms of Use:

Governing Terms: Your use of this model is governed by the NVIDIA Open Model License Agreement. Use of the tokenizer included in this model is governed by the CC-BY-4.0 license.

Deployment Geography:

Global

Use Case:

NVIDIA Nemotron Parse 2.0 is designed for developers and teams building document intelligence, retrieval-augmented generation (RAG), curator, extractor, and agentic AI applications. It can be used to convert scanned or rendered PDFs, presentation slides, forms, reports, tables, and mixed-content document pages into structured outputs for downstream indexing, retrieval, analytics, model training-data creation, and human-in-the-loop review.

Capability Highlights:

• Expanded multilingual OCR support, with substantial gains on CJK and Indic-script document text.
  
• Improved handwritten-text extraction for document pages containing informal, handwritten, or note-like content.
  
• Chart-to-table parsing that can identify chart regions and convert visible chart information into structured text for downstream use.
  
• Improved table handling, including stronger table detection, structure recovery, and text extraction on table-heavy documents.
  

Release Date:

Hugging Face August 3, 2026 via URL

References(s):

• Hugging Face Transformers mBART documentation
  
• NVIDIA C-RADIO
  

Model Architecture:

Architecture Type: Transformer-based vision-encoder-decoder model

Network Architecture:

• Vision Encoder: ViT-H model based on NVIDIA C-RADIO
  
• Adapter Layer: 1D convolutions and normalization layers that compress the vision latent sequence before decoding
  
• Decoder: mBART decoder with 10 blocks
  
• Auxiliary Prediction Head: One training-time decoder prediction head is preserved separately in auxiliary_prediction_heads.safetensors.extra for future multi-token prediction research. Standard generation uses the tied decoder input/output embeddings; the default model.safetensors, Transformers examples, and vLLM examples do not load this auxiliary head.
  
• Tokenizer: The tokenizer contains 72,256 entries, including an approximately 20k-token expansion over NVIDIA Nemotron Parse v1.2 to improve multilingual token efficiency. The model also includes task/control tokens such as <predict_bbox>, <predict_classes>, <predict_text_in_pic>, and <predict_no_text_in_pic>, plus the chart class token <class_Chart>. Use of the tokenizer included in this model is governed by the CC-BY-4.0 license.
  
• Number of Parameters: < 1B
  

Input(s):

Input Type(s): Image, Text

Input Format(s):

• Image: Red, Green, Blue (RGB)
  
• Text: Prompt string
  

Input Parameters:

• Image: Two-Dimensional (2D)
  
• Text: One-Dimensional (1D)
  

Other Properties Related to Input:

• Recommended maximum input resolution (Width, Height): 1664, 2048
  
• Recommended minimum input resolution (Width, Height): 1024, 1280
  
• Channel Count: 3
  
• Prompt format: one task prompt composed from the supported control tokens. The default prompt extracts bounding boxes, semantic classes, and text in markdown format: </s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>. The model can emit chart regions using <class_Chart> when chart content is detected.
  

Output(s):

Output Type(s): Text

Output Format(s): String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: Nemotron Parse 2.0 returns a string encoding document text, semantic element classes, and bounding boxes. Postprocessing utilities in this repository can transform generated bounding boxes back to original image coordinates and convert table or chart-associated text into LaTeX, HTML, markdown, JSON, or CSV where supported.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA hardware and software frameworks, the model achieves faster training and inference times compared to CPU-only solutions.

Quick Start

Install dependencies in your environment

You can use a public image nvcr.io/nvidia/pytorch:25.03-py3 with the following library versions installed on top:

pip install accelerate==1.12.0
pip install albumentations==2.0.8
pip install transformers==5.6.1
pip install timm==1.0.22
pip install open_clip_torch==3.2.0
pip install einops==0.8.1

Usage example

import torch
from PIL import Image, ImageDraw
from transformers import AutoModel, AutoProcessor, AutoTokenizer, GenerationConfig
from postprocessing import extract_classes_bboxes, transform_bbox_to_original, postprocess_text

# Load model and processor
model_path = "nvidia/NVIDIA-Nemotron-Parse-2.0"  # Or use a local path
device = "cuda:0"

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(model_path)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# Load image
image = Image.open("path/to/your/image.jpg")
task_prompt = "</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>"
# task_prompt = "</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>"

# Process image
inputs = processor(images=[image], text=task_prompt, return_tensors="pt", add_special_tokens=False).to(device)

generation_config = GenerationConfig.from_pretrained(model_path, trust_remote_code=True)

# Generate text
outputs = model.generate(**inputs, generation_config=generation_config)

# Decode the generated text
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]

Postprocessing

from PIL import ImageDraw
from postprocessing import extract_classes_bboxes, transform_bbox_to_original, postprocess_text

classes, bboxes, texts = extract_classes_bboxes(generated_text)
bboxes = [transform_bbox_to_original(bbox, image.width, image.height) for bbox in bboxes]

# Specify output formats for postprocessing
table_format = "latex"  # latex | HTML | markdown | json | json_hierarchical | csv
text_format = "markdown"  # markdown | plain
blank_text_in_figures = False  # set True to remove text inside 'Picture' class
texts = [
    postprocess_text(
        text,
        cls=cls,
        table_format=table_format,
        text_format=text_format,
        blank_text_in_figures=blank_text_in_figures,
    )
    for text, cls in zip(texts, classes)
]

for cl, bb, txt in zip(classes, bboxes, texts):
    print(cl, ": ", txt)

draw = ImageDraw.Draw(image)
for bbox in bboxes:
    draw.rectangle((bbox[0], bbox[1], bbox[2], bbox[3]), outline="red")

Table output formats

Supported values for table_format: latex | HTML | markdown | json | json_hierarchical | csv

Inference with vLLM

Nemotron Parse 2.0 can be served with vLLM when using a vLLM build that includes Nemotron Parse remote-code support. On A100 and A10 systems, we recommend running vllm serve with --attention-backend=TRITON_ATTN.

Install the following dependencies on top of the serving image:

pip install albumentations timm open_clip_torch einops

This export keeps lm_head.weight tied to decoder.embed_tokens.weight and does not materialize a duplicate output-head tensor. Current vLLM 0.20 Nemotron Parse builds create a separate output head unless patched. If your vLLM build does not already support tied Nemotron Parse output embeddings, fetch the included runtime patch and add it to PYTHONPATH before starting vLLM:

PATCH_ROOT=$(python - <<'PY'
from huggingface_hub import snapshot_download
print(snapshot_download(
    "nvidia/NVIDIA-Nemotron-Parse-2.0",
    allow_patterns="vllm_tied_patch/sitecustomize.py",
))
PY
)
export PYTHONPATH="${PATCH_ROOT}/vllm_tied_patch:${PYTHONPATH}"

vLLM Inference example

Option 1: end-to-end Python inference

from vllm import LLM, SamplingParams
from PIL import Image


def main():
    sampling_params = SamplingParams(
        temperature=0,
        top_k=1,
        repetition_penalty=1.1,
        max_tokens=9000,
        skip_special_tokens=False,
    )

    llm = LLM(
        model="nvidia/NVIDIA-Nemotron-Parse-2.0",
        max_num_seqs=64,
        limit_mm_per_prompt={"image": 1},
        dtype="bfloat16",
        trust_remote_code=True,
    )

    image = Image.open("<YOUR-IMAGE-PATH>")

    prompts = [
        {
            "prompt": "</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>",
            "multi_modal_data": {
                "image": image,
            },
        },
        {
            "encoder_prompt": {
                "prompt": "",
                "multi_modal_data": {
                    "image": image,
                },
            },
            "decoder_prompt": "</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>",
        },
    ]

    outputs = llm.generate(prompts, sampling_params)

    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Decoder prompt: {prompt!r}, Generated text: {generated_text!r}")


if __name__ == "__main__":
    main()

Option 2: vLLM serve

vllm serve nvidia/NVIDIA-Nemotron-Parse-2.0 \
    --dtype bfloat16 \
    --max-num-seqs 8 \
    --limit-mm-per-prompt '{"image": 1}' \
    --trust-remote-code \
    --port 8000 \
    --chat-template chat_template.jinja

Then run inference through the OpenAI-compatible API:

import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
)

with open("<YOUR-IMAGE-PATH>", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode("utf-8")

prompt_text = "</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>"

resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-Parse-2.0",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": prompt_text,
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{img_b64}",
                    },
                },
            ],
        }
    ],
    max_tokens=9000,
    temperature=0.0,
    extra_body={
        "repetition_penalty": 1.1,
        "top_k": 1,
        "skip_special_tokens": False,
    },
)
print(resp.choices[0].message.content)

Prompt and logits-processor options

The recommended default prompt extracts bounding boxes, semantic classes, and text in markdown format:

</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>

To extract text that appears inside embedded images or figures, use:

</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>

If only bounding boxes and classes are needed, use:

</s><s><predict_bbox><predict_classes><output_no_text><predict_no_text_in_pic>

This repository includes two optional logits processors:

• NemotronParseRepetitionStopProcessor: detects repeating n-grams during generation and forces the model to close the coordinate block when repeated structured output suggests a potential hallucination.
• NemotronParseTableInsertionLogitsProcessor: forces every block to follow a table structure, which can be useful when running the model on table image crops.

Please refer to example_with_processor.py for Python-model usage. With vLLM, export logitsprocs/ to PYTHONPATH and pass the desired processor to vllm serve, for example:

vllm serve nvidia/NVIDIA-Nemotron-Parse-2.0 \
  --dtype bfloat16 \
  --max-num-seqs 4 \
  --limit-mm-per-prompt '{"image": 1}' \
  --attention-backend=TRITON_ATTN \
  --trust-remote-code \
  --logits-processors nemotron_parse_vllm_logitprocs:NemotronParseTableInsertionLogitsProcessor \
  --port 8000

An example of inference with the vLLM OpenAI-compatible server is available in vllm_example.py.

Software Integration:

Runtime Engine(s):

• Transformers
  
• vLLM
  

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
  
• NVIDIA Blackwell
  
• NVIDIA Hopper
  
• NVIDIA Turing
  

Supported Operating System(s):

• Linux
  

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels is essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

Nemotron Parse 2.0

Training, Testing, and Evaluation Datasets:

Training Dataset

Data Modality:

• Image
  
• Text
  

Image Training Data Size:

• 1 Million to 1 Billion Images
  

Text Training Data Size:

• 1 Billion to 10 Trillion Tokens
  

Data Collection Method by dataset:

• Hybrid: Automated, Human, Synthetic
  

Labeling Method by dataset:

• Hybrid: Automated, Human, Synthetic
  

Properties: The training set contains millions of image-text items aggregated across large document, table, and layout datasets. The data consists of document-page and table images paired with OCR text, bounding boxes, and layout labels. Sources include rendered digital documents, scientific papers, PDFs, Wikipedia-style pages, and synthetic document, table, word, and character renderings. Annotations come from OCR and layout models, third-party OCR services, synthetic-generation pipelines, and human labeling.

Testing and Evaluation Dataset

Testing and evaluation use internal and public document-understanding benchmarks that cover OCR quality, layout structure, table parsing, reading order, and visual grounding. Dataset collection and labeling methods are hybrid and include automated, model-derived, manually labeled, and synthetic annotations.

Evaluation Results

Benchmark coverage:

• ParseBench evaluates document parsing across text fidelity, semantic formatting, tables, charts, and visual grounding with layout boxes/classes.
  
• IndicVisionBench evaluates OCR quality for Indic-language pages, including Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu.
  
• MOSCAR evaluates multilingual synthetic OCR across broad script coverage, including Latin, Arabic, Cyrillic, Chinese, Hangul, Japanese, Indic scripts, Hebrew, Thai, Greek, and others.
  
• OmniDocBench Notes (Handwriting) evaluates text-block edit distance on note-style document pages from the data_source: note slice.
  

The following results compare NVIDIA Nemotron Parse 2.0 with NVIDIA Nemotron Parse v1.2 on internal and public evaluation benchmarks. NVIDIA Nemotron Parse 2.0 results use the equal-weight checkpoint soup from training checkpoints 58k, 60k, 62k, 64k, and 66k. Unless marked otherwise, metrics are higher-is-better; arrows indicate the direction of the 2.0 change relative to v1.2.

Benchmark	Metric	NVIDIA Nemotron Parse v1.2	NVIDIA Nemotron Parse 2.0	Change
ParseBench	Overall score	0.5782	0.6391	↑ +0.0609
OmniDocBench Notes (Handwriting)	Text edit distance (lower is better)	0.9739	0.3395	↓ -0.6343
IndicVisionBench	Overall ANLS character	0.0612	0.7203	↑ +0.6592
MOSCAR (Multilingual)	Overall BoC F1	0.4410	0.9102	↑ +0.4692

Inference:

Acceleration Engine: Transformers, vLLM
Test Hardware:

• H100
  
• A100
  

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content. If an input image contains people, personal health information, confidential business data, or intellectual property, the model may extract or reproduce visible text from that content.

For more detailed information on ethical considerations for this model, please see the Model Card++ subcards: Bias, Explainability, Safety & Security, and Privacy.
Please report model quality, risk, security vulnerabilities, or NVIDIA AI Concerns here.