🔍 VT-DETR — Textbook Layout Detection

RF-DETR Large fine-tuned for Vietnamese textbook document layout analysis

---

Overview

VT-DETR stands for Vietnamese Textbook Detection Transformer — a fine-tuned variant of RF-DETR Large targeting automated detection of structural elements in Vietnamese textbook pages. It identifies four layout classes with high accuracy, enabling downstream document parsing, indexing, and educational content extraction pipelines.

Property	Value
Base model	RF-DETR Large (pretrained)
Task	Object Detection
Domain	Document Layout / Textbook Pages
Language	Vietnamese
Input resolution	640 × 640

---

Classes

ID	Label	Description
0	Object-detection	Generic detectable region
1	Figure	Diagrams, illustrations, charts
2	Icon	Small symbolic graphics
3	Table	Tabular data structures

---

Demo

Input	Output

---

Benchmark

✅ Best validation metrics

Metric	Value
mAP @ [0.50 : 0.95]	0.8304
mAP @ 0.50	0.9216
mAP @ 0.75	0.8992
Precision	0.9471
Recall	0.8997
F1	0.8875

📉 Last epoch metrics

Metric	Value
mAP @ [0.50 : 0.95]	0.8161
mAP @ 0.50	0.9074
mAP @ 0.75	0.8811
Precision	0.9184
Recall	0.8272
F1	0.8694

Metrics sourced from outputs/rfdetr_textbook/metrics.csv (full training run). A separate single-image eval run in outputs/exp_rfdetr_textbook/eval/metrics.json (num_images=1) is not representative and excluded from this report.

---

Training Configuration

model:
  size: large

training:
  epochs: 20
  batch_size: 4
  grad_accum_steps: 8
  resolution: 640
  device: cuda

augmentation:
  mosaic: true
  mixup: 0.1

early_stopping:
  enabled: true
  monitor: mAP50:95
  patience_logs: 4
  min_delta: 0.005

Dataset split (COCO format)

data_resplit/
├── train/
│   └── _annotations.coco.json
├── valid/
│   └── _annotations.coco.json
└── test/
    └── _annotations.coco.json

---

Usage

Option 1 — HuggingFace Transformers (recommended)

pip install transformers rfdetr

from transformers import AutoModel, AutoImageProcessor
from PIL import Image

processor = AutoImageProcessor.from_pretrained(
    "hietraan/vt-detr",
    trust_remote_code=True
)
model = AutoModel.from_pretrained(
    "hietraan/vt-detr",
    trust_remote_code=True
)

image = Image.open("page.png")
inputs = processor.preprocess(image)
outputs = model(inputs["images"], threshold=0.5)
results = processor.post_process_object_detection(outputs, threshold=0.5)

# results[0]["boxes"]  → xyxy coordinates (tensor)
# results[0]["scores"] → confidence scores (tensor)
# results[0]["labels"] → class ids (tensor)
print(results)

Note: trust_remote_code=True is required because this model uses a custom modeling_vtdetr.py. The checkpoint (134MB) is downloaded automatically to `/.cache/huggingface/` on first run.

Option 2 — RF-DETR directly

pip install "rfdetr[train]"

from rfdetr import RFDETRLarge
from huggingface_hub import hf_hub_download
from PIL import Image

weights = hf_hub_download(repo_id="hieptran318204/vt-detr", filename="checkpoint_best_total.pth")
model = RFDETRLarge(pretrain_weights=weights)

image = Image.open("page.png")
detections = model.predict(image, threshold=0.5)
print(detections)

Train / Evaluate (local)

python -m rfdetr_finetune.train
python -m rfdetr_finetune.evaluate

---

Repository Structure

hieptran318204/vt-detr
├── README.md                    # Model card
├── config.json                  # HuggingFace model config
├── preprocessor_config.json     # Image processor config
├── modeling_vtdetr.py           # Custom modeling code (trust_remote_code)
├── checkpoint_best_total.pth    # Best checkpoint (by mAP50:95)
├── inference.webp               # Demo output
└── page077.png                  # Demo input

---

Limitations

• Trained and evaluated on Vietnamese textbook domain; generalization to other document types is not guaranteed.
• Test set benchmark not yet finalized — reported metrics are from the validation split.
• Robustness may be improved with additional data augmentation, larger datasets, or LR schedule tuning.

---

License

This model is released under the Apache License 2.0, inherited from the base RF-DETR Large model by Roboflow.

Copyright 2025 Roboflow, Inc.
Licensed under the Apache License, Version 2.0. See LICENSE for details.

Fine-tuned weights and additional code in this repository are also distributed under Apache 2.0.

---

Citation

If you use this model or the RF-DETR framework, please cite:

Base model (RF-DETR Large):

@misc{rf-detr,
  title        = {RF-DETR: Neural Architecture Search for Real-Time Detection Transformers},
  author       = {Isaac Robinson and Peter Robicheaux and Matvei Popov and Deva Ramanan and Neehar Peri},
  year         = {2025},
  eprint       = {2511.09554},
  archivePrefix= {arXiv},
  primaryClass = {cs.CV},
  url          = {https://arxiv.org/abs/2511.09554}
}

Base weights: Roboflow/rf-detr-large — Apache 2.0

This fine-tuned model (VT-DETR):

@misc{vtdetr2025,
  title        = {VT-DETR: RF-DETR Large Fine-tuned for Textbook Layout Detection},
  author       = {hieptran318204},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/hieptran318204/vt-detr}}
}

---

_{Fine-tuned with ❤️ for Vietnamese educational document processing}