| --- |
| title: WhatBird |
| emoji: 🐦 |
| colorFrom: green |
| colorTo: blue |
| sdk: gradio |
| sdk_version: 6.17.3 |
| python_version: "3.12" |
| app_file: app.py |
| pinned: false |
| license: apache-2.0 |
| short_description: Bird ID — fast classifier + 1B vision-language model |
| tags: |
| - build-small-hackathon |
| - backyard-ai |
| - tiny-titan |
| - small-models |
| - track:backyard |
| - sponsor:openbmb |
| - achievement:offgrid |
| --- |
| |
| # 🐦 WhatBird |
|
|
| [](http://xhslink.com/o/9AHjKKUQ1B) |
| [](http://xhslink.com/o/2jdfwxk4IDj) |
|
|
| Identify a bird from a photo with a two-stage pipeline: a fast on-device |
| classifier proposes the species, then a compact vision-language model confirms |
| the call and explains it from visible field marks. |
|
|
| - **Specialist classifier — fast & on-device.** A YOLO26-x image classifier |
| (exported to ONNX, runs on CPU) trained on CUB-200-2011 + Birdsnap + |
| iNaturalist-2021 merged into **1,532 species** (78.3% top-1 / 91.9% top-5). It |
| narrows 1,532 classes to a handful of candidates in milliseconds. |
| - **Vision-language model — visual reasoning.** [MiniCPM-V 4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6) |
| (just **1B** parameters) looks at the photo together with the shortlist, |
| confirms the species (or flags a mismatch), and explains *why* from visible |
| field marks — exactly the call a narrow classifier struggles with on |
| look-alikes (e.g. Alder vs. Least Flycatcher). |
|
|
| Both models are small (≤4B) — the point is the division of labour, not raw |
| scale: the specialist is cheap and covers the long tail of 1,532 species; the |
| vision-language model is invoked only on the shortlist, where its reasoning adds |
| the most value. |
|
|
| ## How the two models collaborate |
|
|
| They don't just run in sequence — control is handed to whichever model is more |
| reliable, gated on the classifier's top-1 confidence: |
|
|
| | Classifier top-1 | Who leads | Behavior | |
| |------------------|-----------|----------| |
| | **≥ 0.80** (confident) | the **specialist classifier** | The VLM only *confirms and explains* — it cannot overturn a confident, in-domain call. (Stops a 1B VLM from "correcting" a correct 94% answer into a look-alike.) | |
| | **0.50 – 0.80** (unsure) | the **VLM, within the shortlist** | Re-ranks the top-5 from what it sees. | |
| | **< 0.50** (lost) | the **VLM, open-vocabulary** | The shortlist is a weak hint; the VLM may name a species *not in the 1,532* and the result is flagged "not in shortlist". | |
|
|
| In practice that means: a clear shot (`samples/sample_2.jpg`, Acorn Woodpecker, |
| 100%) sits in the top tier — the classifier is sure and the VLM only confirms. A |
| look-alike *Empidonax* flycatcher (`samples/sample_1.jpg`) drops the classifier |
| to ~40%, below the threshold, so the VLM reasons over the shortlist from visible |
| field marks — exactly where a 1B vision model earns its keep. For a bird beyond |
| all 1,532 classes, the open-vocabulary tier lets the VLM name it directly, |
| flagged "not in shortlist". |
|
|
| The 1,532-class set is now **global**, not just North-American — it merges |
| iNaturalist-2021 on top of CUB + Birdsnap. So the **European Robin** |
| (`samples/robin.jpg`), which the earlier 562-species model couldn't place, is now |
| identified directly by the classifier. Small model covers an ever-wider base; |
| large model reasons over what's left. |
|
|
| ## Running it |
|
|
| ```bash |
| pip install -r requirements.txt |
| python app.py |
| ``` |
|
|
| The large-model backend needs no configuration: MiniCPM-V switches on |
| automatically wherever a GPU can serve it (an HF ZeroGPU Space, or a local CUDA |
| box). Without a GPU the app falls back to a lightweight templated explanation and |
| stays fully functional. (`WHATBIRD_DESCRIBER=stub|minicpm` or |
| `WHATBIRD_MODEL_ID=...` override the auto-detection.) |
|
|
| ## Models |
|
|
| | Model | Role | Size | Score | Notes | |
| |-------|------|------|-------|-------| |
| | yolo26x_v2 (ONNX) | specialist classifier | 116 MB | 78.3% top-1 / 91.9% top-5 | shipped, CPU, 1,532 classes | |
| | yolo26s_v2 | specialist classifier (light) | 15 MB (.pt) | 70.9% top-1 / 87.4% top-5 | smaller/faster alternative | |
| | MiniCPM-V 4.6 | re-ranker / explainer | **1B** | — | 2.6 GB, Apache-2.0 | |
|
|
| Built for the [Build Small Hackathon](https://huggingface.co/spaces/build-small-hackathon). |
|
|
