harness-race/pi-r2-probe

pi-r2 — DETR fine-tuned on Beyond Words (LOC)

Object detection model fine-tuned from facebook/detr-resnet-50 (Apache-2.0) on the biglam/loc_beyond_words dataset (CC0): crowd-sourced bounding-box annotations of World War I-era newspaper pages from the Library of Congress Chronicling America collection.

Model detail

• Architecture: DETR (DEtection TRansformer) with a ResNet-50 backbone, 6 encoder/6 decoder transformer layers, 100 object queries.
• Base model: facebook/detr-resnet-50 — license Apache-2.0 (shareable).
• Dataset: biglam/loc_beyond_words — license CC0-1.0 (public domain).
• Classes (7): Photograph, Illustration, Map, Comics/Cartoon, Editorial Cartoon, Headline, Advertisement
• Image size: resized so the longest edge ≤ 1200 px (aspect ratio preserved), padded per batch via a pixel_mask. Training used random horizontal flips and random scale (70–100% of the max size).
• Optimizer: AdamW (LR 5e-05, weight decay 0.0001), cosine schedule with 5% warmup, fp16, gradient clipping 0.1.
• Class imbalance: images containing rare classes (Map, Editorial Cartoon, Illustration, Comics) are oversampled with a WeightedRandomSampler when building training batches.
• Training budget: 0.19 minutes, 24 steps, batch size 4 on an NVIDIA A10G.

Validation results (COCO protocol, pycocotools)

Evaluated on the biglam/loc_beyond_words validation split (32 images) with COCO-style IoU-matched metrics (area = all, max detections = 300, NMS IoU threshold 0.75).

Metric	Value
mAP @[0.5:0.95]	1.843759794973911e-06
mAP @0.50	6.675969148985493e-05
mAP @0.75	0.0
AR @100	0.0021307798481711524

Per-class AP @0.50:

Class	AP@0.50
Photograph	-

Raw per-image predictions and this training script are included in this repo (eval_predictions.json, train_detr.py).

Usage

from transformers import AutoImageProcessor, DetrForObjectDetection
from PIL import Image
import torch

repo = "harness-race/pi-r2-probe"
processor = AutoImageProcessor.from_pretrained(repo)
model = DetrForObjectDetection.from_pretrained(repo)

img = Image.open("newspaper_page.jpg").convert("RGB")
inputs = processor(images=img, return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

score_threshold = 0.5
for logits, box in zip(out.logits[0], out.pred_boxes[0]):
    prob = logits.softmax(-1)
    cls_idx, score = prob[:, :-1].max(-1)
    if score.item() > score_threshold:
        cx, cy, bw, bh = box.tolist()
        W, H = img.size
        x1, y1 = (cx - bw/2)*W, (cy - bh/2)*H
        x2, y2 = (cx + bw/2)*W, (cy + bh/2)*H
        print(model.config.id2label[cls_idx.item()], round(score.item(), 3), [round(x1), round(y1), round(x2), round(y2)])

Intended use & limitations

• Trained on historically scanned newspaper pages (circa 1910–1920); other domains or modern document layouts will degrade accuracy.
• The dataset is strongly class-imbalanced (Headline/Advertisement dominate; Map and Editorial Cartoon are rare), so per-class accuracy varies widely (see table above).
• DETR emits up to 100 box proposals per image; in very dense pages some objects may be missed.

Licenses

• Base model facebook/detr-resnet-50: Apache-2.0.
• Dataset biglam/loc_beyond_words: CC0-1.0 (Public Domain Dedication).
• This fine-tuned model: Apache-2.0.