taresco/KarantaOCR-TrimmedVocab

KarantaOCR: Efficient Document Processing for African Languages

Karanta OCR

Model Description

KarantaOCR is an open-source document OCR and processing model designed for high-accuracy text extraction in African languages. The model focuses on preserving language-specific characters and diacritics that are often lost, normalized, or mis-transcribed by existing OCR systems.

KarantaOCR is fine-tuned from Qwen/Qwen2.5-VL-3B-Instruct, a vision-language model that combines a strong vision encoder with a large language model. Through targeted curriculum fine-tuning, KarantaOCR extends these capabilities to robust document understanding across diverse PDF formats and multilingual settings.

Vocabulary Trimming

KarantaOCR-TrimmedVocab is an optimized version of taresco/KarantaOCR where we aggressively removed unused token embeddings to save GPU memory and improve efficiency. The original model inherited a very large tokenizer vocabulary (around 150k tokens) from its base model. In practice, a big chunk of those tokens are never used in our OCR workloads. Since the input embeddings are tied to the output language-model head, a smaller vocabulary means

• less memory usage,
• fewer output logits to compute at every generation step,
• and faster, cheaper inference overall.

Because we’re mostly removing dead or irrelevant tokens, this trimming doesn’t hurt performance.

Our approach is straightforward:

• We identify tokens that never (or almost never) appear in our OCR training data.
• We remove those tokens from both the tokenizer and the embedding / LM head.
• We keep all special tokens and everything the model actually needs to function normally.

Even within the target languages, some tokens are extremely rare — odd substrings, unused symbols, or artifacts from multilingual training. Using the same corpus we trained on, we compute token frequencies and drop the least frequent ones below a threshold. With this frequency-based trimming, we reduced the vocabulary from ~150,000 tokens down to ~94,000 tokens, without sacrificing OCR quality. What we get in return is a leaner model that’s better suited for deployment and large-scale document processing.

Training Data

KarantaOCR was trained using a two-stage curriculum fine-tuning strategy.

Stage 1: General OCR Training

• 100,000 documents sampled from Allenai OCRMix
• Purpose: learn general OCR skills across layouts, fonts, tables, and document structures

Stage 2: African Language Fine-Tuning

• 50,000 PDFs containing text in 10 African languages, crawled from the web

• Domains include:

  • Religious texts
  • Legal documents
  • Dictionaries
  • Novels
  • Other long-form and structured documents

This stage emphasizes accurate transcription of diacritics, special characters, and region-specific typography.

---

Capabilities

KarantaOCR supports:

• High-accuracy text extraction from PDFs

• Table extraction and structured document understanding

• Robust handling of:

  • Multi-column layouts
  • Headers and footers
  • Mixed scanned and digital PDFs

While improved performance on African languages was our priority, KarantaOCR maintains strong performance on English and other high-resource languages, making it suitable for mixed-language document collections.

Evaluation

KarantaOCR is evaluated on the OLMOocr benchmark using pass-rate accuracy. Scores are reported as averages across JSONL files with 95% confidence intervals.

Model	Avg Score ↑	95% CI
KarantaOCR	74.1%	± 1.1
KarantaOCR-TrimmedVocab	73.5%	± 1.1
RoLMOCR	74.4%	± 1.0
NanoNetsOCR-2	68.8%	± 1.1
OLMOCR	65.8%	± 0.9

Results by Documet Type (%)

JSONL File	KarantaOCR	RoLMOCR	NanoNetsOCR-2	OLMOCR	KarantaOCR-TrimmedVocab
arxiv_math	74.2	76.8	73.7	68.9	73.9
baseline	99.4	97.9	99.5	85.0	99.4
headers_footers	95.3	94.1	32.8	96.4	95.7
long_tiny_text	72.2	61.3	92.1	81.9	71.7
multi_column	75.6	70.0	82.5	84.0	75.3
old_scans	41.3	42.4	41.4	42.0	40.9
old_scans_math	70.3	80.1	44.1	0.0	70.3
table_tests	64.3	72.2	84.2	68.3	60.7

How to Use

KarantaOCR processes PDF documents by rendering pages into images and combining them with structured prompts for inference.

Load the Model and Processor

import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

def load_model(model_path: str, device_map: str = "auto", dtype: str = "auto"):
    model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
        model_path,
        torch_dtype=getattr(torch, dtype) if dtype != "auto" else "auto",
        device_map=device_map,
    )
    return model

def load_processor(processor_name: str, min_pixels=None, max_pixels=None):
    if min_pixels and max_pixels:
        return AutoProcessor.from_pretrained(
            processor_name, min_pixels=min_pixels, max_pixels=max_pixels
        )
    return AutoProcessor.from_pretrained(processor_name)

Prepare a PDF Page for Inference

from jinja2 import Template

def render_pdf_to_base64png(
    local_pdf_path: str, page_num: int, target_longest_image_dim: int = 2048
) -> str:
    longest_dim = max(get_pdf_media_box_width_height(local_pdf_path, page_num))

    # Convert PDF page to PNG using pdftoppm
    pdftoppm_result = subprocess.run(
        [
            "pdftoppm",
            "-png",
            "-f",
            str(page_num),
            "-l",
            str(page_num),
            "-r",
            str(
                target_longest_image_dim * 72 / longest_dim
            ),  # 72 pixels per point is the conversion factor
            local_pdf_path,
        ],
        timeout=120,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    assert pdftoppm_result.returncode == 0, pdftoppm_result.stderr
    return base64.b64encode(pdftoppm_result.stdout).decode("utf-8")

def build_message(image_url: str, system_prompt: str, page: int = 0):
    image_base64 = render_pdf_to_base64png(image_url, page, TARGET_IMAGE_DIM)

    prompt = [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": system_prompt
                },
                {
                    "type": "image",
                    "image": f"data:image/png;base64,{image_base64}",
                },
            ],
        }
    ]
    return prompt

Run OCR Inference

from qwen_vl_utils import process_vision_info

def run_inference(model, processor, messages, max_new_tokens=128, device="cuda"):
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

    image_inputs, _ = process_vision_info(messages)
    inputs = processor(
        text=[text],
        images=image_inputs,
        padding=False,
        return_tensors="pt",
    ).to(device)

    generated_ids = model.generate(**inputs, max_new_tokens=max_new_tokens)
    trimmed_ids = [
        out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
    ]

    outputs = processor.batch_decode(
        trimmed_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )
    return outputs[0]

End-to-End Example

model = load_model("taresco/KarantaOCR")
processor = load_processor("taresco/KarantaOCR")

prompt = """Below is the image of one page of a PDF document.
Just return the plain text representation of this document as if you were reading it naturally.
Turn equations into a LaTeX representation, and tables into markdown format. Remove the headers and footers, but keep references and footnotes.
Read any natural handwriting.
This is likely one page out of several in the document, so be sure to preserve any sentences that come from the previous page, or continue onto the next page, exactly as they are.
If there is no text at all that you think you should read, you can output null.
if the document contains diacritics, please include them in the output.
Do not hallucinate.
"""

messages = build_message(
    image_url="example.pdf",
    system_prompt=prompt,
    page=0
)

output_text = run_inference(model, processor, messages)
print(output_text)