masoniis/magi-onnx

magi — ONNX export (detection only, no OCR)

ONNX export of ragavsachdeva/magi, containing only the detection and association sub-networks. The OCR sub-network has been intentionally omitted so users can substitute their own OCR model appropriate to their context.

Files

File	Description
detection_transformer.onnx	ConditionalDETR backbone — preprocessed manga page → object tokens + reference points
crop_embedding_model.onnx	ViTMAE — 224×224 character crop images → CLS embeddings
bbox_predictor.onnx	MLP — object tokens → bounding boxes (xywh, normalised)
class_labels_classifier.onnx	Linear — object tokens → class logits (character / text / panel)
text_character_matching_head.onnx	MLP — text+character token pairs → association score
character_character_matching_head.onnx	MLP — character+character token pairs → clustering score
is_this_text_a_dialogue.onnx	MLP — text tokens → dialogue vs. non-dialogue confidence

Inference order

1. Preprocess the manga page image using MagiProcessor from the original repo (or reimplement: resize to 1024×1024, normalise, build pixel mask)
2. Run detection_transformer.onnx → last_hidden_state, reference_points
3. Slice object tokens: last_hidden_state[:, :-5] (last 5 are special tokens)
4. Run bbox_predictor.onnx and class_labels_classifier.onnx on object tokens
5. Crop detected character regions and run crop_embedding_model.onnx
6. Run character_character_matching_head.onnx to cluster characters
7. Run text_character_matching_head.onnx to associate speech bubbles to speakers
8. Run is_this_text_a_dialogue.onnx to filter non-dialogue text
9. Run your own OCR on the detected text bounding boxes

Source

Conversion script: github.com/masoniis/magi-onnx

License

The provided model and datasets are available for unrestricted use in personal, research, non-commercial, and not-for-profit endeavors. For any other usage scenarios, kindly contact me via email, providing a detailed description of your requirements, to establish a tailored licensing arrangement. My contact information can be found on my website: ragavsachdeva.github.io

Citation

@misc{sachdeva2024manga,
      title={The Manga Whisperer: Automatically Generating Transcriptions for Comics},
      author={Ragav Sachdeva and Andrew Zisserman},
      year={2024},
      eprint={2401.10224},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}