---
title: WhatBird
emoji: 🐦
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: 6.17.3
python_version: "3.12"
app_file: app.py
pinned: false
license: apache-2.0
short_description: Bird ID — fast classifier + 1B vision-language model
tags:
  - build-small-hackathon
  - backyard-ai
  - tiny-titan
  - small-models
  - track:backyard
  - sponsor:openbmb
  - achievement:offgrid
---

# 🐦 WhatBird

[![REDnote](https://img.shields.io/badge/REDnote-whatbird-ff2442?style=flat-square)](http://xhslink.com/o/9AHjKKUQ1B)
[![Demo video](https://img.shields.io/badge/Demo-video-2f855a?style=flat-square)](http://xhslink.com/o/2jdfwxk4IDj)

Identify a bird from a photo with a two-stage pipeline: a fast on-device
classifier proposes the species, then a compact vision-language model confirms
the call and explains it from visible field marks.

- **Specialist classifier — fast & on-device.** A YOLO26-x image classifier
  (exported to ONNX, runs on CPU) trained on CUB-200-2011 + Birdsnap +
  iNaturalist-2021 merged into **1,532 species** (78.3% top-1 / 91.9% top-5). It
  narrows 1,532 classes to a handful of candidates in milliseconds.
- **Vision-language model — visual reasoning.** [MiniCPM-V 4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6)
  (just **1B** parameters) looks at the photo together with the shortlist,
  confirms the species (or flags a mismatch), and explains *why* from visible
  field marks — exactly the call a narrow classifier struggles with on
  look-alikes (e.g. Alder vs. Least Flycatcher).

Both models are small (≤4B) — the point is the division of labour, not raw
scale: the specialist is cheap and covers the long tail of 1,532 species; the
vision-language model is invoked only on the shortlist, where its reasoning adds
the most value.

## How the two models collaborate

They don't just run in sequence — control is handed to whichever model is more
reliable, gated on the classifier's top-1 confidence:

| Classifier top-1 | Who leads | Behavior |
|------------------|-----------|----------|
| **≥ 0.80** (confident) | the **specialist classifier** | The VLM only *confirms and explains* — it cannot overturn a confident, in-domain call. (Stops a 1B VLM from "correcting" a correct 94% answer into a look-alike.) |
| **0.50 – 0.80** (unsure) | the **VLM, within the shortlist** | Re-ranks the top-5 from what it sees. |
| **< 0.50** (lost) | the **VLM, open-vocabulary** | The shortlist is a weak hint; the VLM may name a species *not in the 1,532* and the result is flagged "not in shortlist". |

In practice that means: a clear shot (`samples/sample_2.jpg`, Acorn Woodpecker,
100%) sits in the top tier — the classifier is sure and the VLM only confirms. A
look-alike *Empidonax* flycatcher (`samples/sample_1.jpg`) drops the classifier
to ~40%, below the threshold, so the VLM reasons over the shortlist from visible
field marks — exactly where a 1B vision model earns its keep. For a bird beyond
all 1,532 classes, the open-vocabulary tier lets the VLM name it directly,
flagged "not in shortlist".

The 1,532-class set is now **global**, not just North-American — it merges
iNaturalist-2021 on top of CUB + Birdsnap. So the **European Robin**
(`samples/robin.jpg`), which the earlier 562-species model couldn't place, is now
identified directly by the classifier. Small model covers an ever-wider base;
large model reasons over what's left.

## Running it

```bash
pip install -r requirements.txt
python app.py
```

The large-model backend needs no configuration: MiniCPM-V switches on
automatically wherever a GPU can serve it (an HF ZeroGPU Space, or a local CUDA
box). Without a GPU the app falls back to a lightweight templated explanation and
stays fully functional. (`WHATBIRD_DESCRIBER=stub|minicpm` or
`WHATBIRD_MODEL_ID=...` override the auto-detection.)

## Models

| Model | Role | Size | Score | Notes |
|-------|------|------|-------|-------|
| yolo26x_v2 (ONNX) | specialist classifier | 116 MB | 78.3% top-1 / 91.9% top-5 | shipped, CPU, 1,532 classes |
| yolo26s_v2 | specialist classifier (light) | 15 MB (.pt) | 70.9% top-1 / 87.4% top-5 | smaller/faster alternative |
| MiniCPM-V 4.6 | re-ranker / explainer | **1B** | — | 2.6 GB, Apache-2.0 |

Built for the [Build Small Hackathon](https://huggingface.co/spaces/build-small-hackathon).