README.md

metadata

title: WhatBird
emoji: 🐦
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: 6.17.3
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Bird ID — fast classifier + 1B vision-language model
tags:
  - build-small-hackathon
  - backyard-ai
  - tiny-titan
  - small-models
  - track:backyard
  - sponsor:openbmb
  - achievement:offgrid

🐦 WhatBird

Identify a bird from a photo with a two-stage pipeline: a fast on-device classifier proposes the species, then a compact vision-language model confirms the call and explains it from visible field marks.

• Specialist classifier — fast & on-device. A YOLO26-x image classifier (exported to ONNX, runs on CPU) trained on CUB-200-2011 + Birdsnap + iNaturalist-2021 merged into 1,532 species (78.3% top-1 / 91.9% top-5). It narrows 1,532 classes to a handful of candidates in milliseconds.
• Vision-language model — visual reasoning. MiniCPM-V 4.6 (just 1B parameters) looks at the photo together with the shortlist, confirms the species (or flags a mismatch), and explains why from visible field marks — exactly the call a narrow classifier struggles with on look-alikes (e.g. Alder vs. Least Flycatcher).

Both models are small (≤4B) — the point is the division of labour, not raw scale: the specialist is cheap and covers the long tail of 1,532 species; the vision-language model is invoked only on the shortlist, where its reasoning adds the most value.

How the two models collaborate

They don't just run in sequence — control is handed to whichever model is more reliable, gated on the classifier's top-1 confidence:

Classifier top-1	Who leads	Behavior
≥ 0.80 (confident)	the specialist classifier	The VLM only confirms and explains — it cannot overturn a confident, in-domain call. (Stops a 1B VLM from "correcting" a correct 94% answer into a look-alike.)
0.50 – 0.80 (unsure)	the VLM, within the shortlist	Re-ranks the top-5 from what it sees.
< 0.50 (lost)	the VLM, open-vocabulary	The shortlist is a weak hint; the VLM may name a species not in the 1,532 and the result is flagged "not in shortlist".

In practice that means: a clear shot (samples/sample_2.jpg, Acorn Woodpecker, 100%) sits in the top tier — the classifier is sure and the VLM only confirms. A look-alike Empidonax flycatcher (samples/sample_1.jpg) drops the classifier to ~40%, below the threshold, so the VLM reasons over the shortlist from visible field marks — exactly where a 1B vision model earns its keep. For a bird beyond all 1,532 classes, the open-vocabulary tier lets the VLM name it directly, flagged "not in shortlist".

The 1,532-class set is now global, not just North-American — it merges iNaturalist-2021 on top of CUB + Birdsnap. So the European Robin (samples/robin.jpg), which the earlier 562-species model couldn't place, is now identified directly by the classifier. Small model covers an ever-wider base; large model reasons over what's left.

Running it

pip install -r requirements.txt
python app.py

The large-model backend needs no configuration: MiniCPM-V switches on automatically wherever a GPU can serve it (an HF ZeroGPU Space, or a local CUDA box). Without a GPU the app falls back to a lightweight templated explanation and stays fully functional. (WHATBIRD_DESCRIBER=stub|minicpm or WHATBIRD_MODEL_ID=... override the auto-detection.)

Models

Model	Role	Size	Score	Notes
yolo26x_v2 (ONNX)	specialist classifier	116 MB	78.3% top-1 / 91.9% top-5	shipped, CPU, 1,532 classes
yolo26s_v2	specialist classifier (light)	15 MB (.pt)	70.9% top-1 / 87.4% top-5	smaller/faster alternative
MiniCPM-V 4.6	re-ranker / explainer	1B	—	2.6 GB, Apache-2.0

Built for the Build Small Hackathon.