OlleOltorp/rfdetr-minecraft-mobs

RF-DETR Large for Minecraft mobs

Flights past mobs in twelve places the model never saw: the game's true boxes on the left, the model's on the right.

Finds every Minecraft Java 1.21.11 mob (87 + the player) in game pictures. The baseline for the Minecraft Mobs dataset.

• Training code, method and how to make your own dataset: minecraft-mob-dataset

Results

On the test split: 5,583 pictures from places the model never saw.

mAP50-95	mAP50	mobs found at score 0.5	of those, wrong class
0.602	0.741	73 %	5 %

• Size matters most. Mobs within 8 blocks score 0.78, past 128 blocks 0.12. Boxes 8 pixels tall or less score 0.06, over 128 pixels 0.84.
• By place: caves score best (0.74), under water 0.48, aerial views hardest (0.33). Time of day and weather change the score by less than 0.04.
• Best classes: magma cube 0.87, ender dragon 0.86, ghast 0.84.
• Weakest: tropical fish 0.14, pufferfish 0.25, tadpole 0.29 (small water mobs with few labels), creaking 0.32 and wither skeleton 0.39 (look-alikes).

Full scores per class, distance, size and place: the baseline notebook.

Use

import json
from huggingface_hub import hf_hub_download
from rfdetr import RFDETRLarge

repo = "OlleOltorp/rfdetr-minecraft-mobs"
model = RFDETRLarge(pretrain_weights=hf_hub_download(repo, "checkpoint_best_total.pth"))
names = json.load(open(hf_hub_download(repo, "classes.json")))

detections = model.predict("picture.png", threshold=0.5)
print([names[c] for c in detections.class_id])

model.predict squeezes the picture to a square. The model was trained on 16:9 pictures at 1248×704, so for the scores above use mobdataset.model.infer.predict from the GitHub repo, which keeps the shape.

Training

RF-DETR Large from its COCO weights, fine-tuned for 10 epochs on the train split (39,158 pictures): 16 pictures per update, learning rate 1e-4 with a quarter-epoch warm-up and cosine decay, seed 0. One RTX 4070, about 50 minutes per epoch. Settings and the command to repeat it are in the GitHub repo (baselines/rf-detr/).

Limits

• Game pictures only, vanilla graphics: no shader packs or resource packs.
• Java 1.21.11 mobs; later versions may change how mobs look.
• Far and tiny mobs are often missed (see above).

Citation

@misc{oltorp2026rfdetrminecraftmobs,
  author = {Olle Oltorp},
  title = {RF-DETR Large for Minecraft mobs},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/OlleOltorp/rfdetr-minecraft-mobs}
}

Licence

CC BY-NC 4.0. Minecraft is a trademark of Mojang; not affiliated with or endorsed by Mojang or Microsoft.