JustANormalTinkerer/hayai-ocr-v2

Hayai OCR v2

Hayai is a compact vision-to-text OCR model. It pairs a SigLIP2 vision encoder with a causal transformer decoder to read text directly out of images, without a separate detection stage.

Architecture

• Vision encoder: google/siglip2-base-patch16-naflex, used at native aspect ratio via NaFlex patching (no forced resizing/cropping).
• Projector: a two-layer MLP mapping vision features into the decoder's hidden size.
• Decoder: a 12-layer causal transformer over vision tokens followed by text tokens, using:
  • Grouped-query attention (8 query heads, 2 key/value heads)
  • RMSNorm on queries and keys, and on layer inputs
  • SwiGLU feed-forward blocks
  • 2D rotary position embeddings over the vision tokens, 1D rotary over the text tokens
  • Block-causal attention: vision tokens attend to each other freely, text tokens attend causally to all vision tokens and to preceding text tokens
• Input and output token embeddings are tied.

The model has no separate object/text detector — the decoder attends over the full set of vision patch tokens and generates the transcription autoregressively.

Usage

from transformers import AutoModel, AutoProcessor, PreTrainedTokenizerFast
from PIL import Image

model = AutoModel.from_pretrained("JustANormalTinkerer/hayai-ocr-v2", trust_remote_code=True)
tokenizer = PreTrainedTokenizerFast.from_pretrained("JustANormalTinkerer/hayai-ocr-v2")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-naflex")

model.eval()

image = Image.open("example.png").convert("RGB")
inputs = processor(images=[image], max_num_patches=256, return_tensors="pt")

with torch.no_grad():
    texts = model.generate(
        pixel_values=inputs["pixel_values"],
        pixel_attention_mask=inputs["pixel_attention_mask"],
        spatial_shapes=inputs["spatial_shapes"],
        tokenizer=tokenizer,
        max_new_tokens=128,
        repetition_penalty=1.20,
    )

print(texts[0])

trust_remote_code=True is required because the model uses a custom architecture (configuration_hayai.py, modeling_hayai.py) hosted alongside the weights.

Eval

The model was evaluated on a L4 GPU on modal.

Finetuning Dataset:

Model Name	Mean CER	Throughput (FPS)
Hayai OCR v2	8.52%	37.25
PaddleOCR-VL-For-Manga	24.66%	3.60

Pretraining Dataset test split (1k images randomly picked)

Model Name	Mean CER	Throughput (FPS)
Hayai OCR v2	10.56%	31.95
PaddleOCR-VL-For-Manga	38.69%	2.22

Training

Training happened in two stages.

Base training: 15 epochs on JustANormalTinkerer/hayai-dataset-merged (~1M images), at roughly 7 hours per epoch. Compute was split across two hardware setups over the course of training: 2x T4 GPUs provided by Kaggle, and L4 GPUs provided by Modal.

Fine-tuning: starting from the base checkpoint, 9 further epochs on JustANormalTinkerer/hayai-finetuning-dataset (roughly 2,000 labeled images), using the same training setup described below.

• Optimizer: Muon for the decoder's 2D weight matrices, AdamW for embeddings, norms, residual scales, and the vision encoder, with separate learning rates for the decoder and the vision tower
• Schedule: cosine decay with linear warmup (5% of total steps), floor at 10% of peak LR
• Augmentation: random rotation, affine, perspective, color jitter, grayscale, blur, and sharpness adjustments applied to training images
• Precision: mixed precision (FP16) training and inference, with gradient scaling
• Evaluation: Character Error Rate (CER) computed on NFKC-normalized, whitespace-stripped text, using Levenshtein distance

Text normalization

Both training and evaluation normalize text before comparison:

1. Unicode NFKC normalization
2. Removal of all whitespace (spaces, tabs, newlines)

Downstream users comparing model output to ground truth should apply the same normalization for consistent CER numbers.

Limitations

• The fine-tuning stage used a relatively small set (~2,000 images); performance on domains far from that fine-tuning distribution — even if covered during base training — may regress somewhat toward base-model behavior.
• English language capability is limited.
• Generation is greedy (argmax) decoding with a repetition penalty; there is no beam search or sampling option built into generate.
• The model expects images preprocessed by the SigLIP2 NaFlex processor; passing raw tensors in another format will not work.