WhatDoWeSee-EfficientNetV2-B0

Tags what is visible in an image with any number of 20 attributes, such as people, text, chart, diagram, table, map or aerial view. A small, fast EfficientNetV2-B0 model (24 MB ONNX, about 14 ms per image on one CPU thread) for sorting and describing large image collections.

This is v0.4.0 (2026-10-16): trained on 28,495 images labelled by three open vision-language models (Clef-omni, Gemma 4 26B, Qwen3-VL 32B), learning from how many of them agree. Try it in your browser: WhatDoWeSee Space.

Attributes

An image gets every attribute that is clearly recognisable, also when blurry or in the background; tiny or unrecognisable details don't count. Depicted things count (a drawn dog is Animals).

• People (people): one or more humans, or clearly human faces, hands or bodies
• Animals (animals): animals, birds, fish or insects
• Plants (nature): living green things: trees, plants, flowers, crops; not grass or a lawn as ground, not sea, sky or rocks (those are Landscape)
• Landscape (landscape): an outdoor view of land, sea, sky, mountains, rocks, rivers or a city as a whole
• Buildings / structures (buildings): buildings, interiors where the room or its architecture is clearly visible, and other built structures: bridges, towers, dams, monuments, ruins
• Vehicles (vehicles): cars, bikes, trains, ships, planes or other vehicles
• Food (food): food or drinks
• Text (text): running text, sentences, lists or a page that make up a substantial part of the image; labels, captions, titles and signs don't count
• Handwriting (handwriting): handwritten text, notes or signatures
• Chart (chart): a data plot: values on axes or as proportions (bar, line, pie, scatter, area chart or similar)
• Diagram (diagram): structure or relationships without measured values: a flowchart, schematic, process or relationship diagram; also labelled drawings that show how something is built (anatomy, parts, cross-sections)
• Table (table): text or numbers laid out in rows and columns, with or without grid lines; not the furniture
• Map (map): a geographic map, floor plan or route map
• Logo / icon (logo-icon): a logo, icon, emblem, badge or pictogram
• Formula (formula): mathematical, chemical or other notation
• Screen / UI (screen): an app, website or device interface
• Artwork (artwork): made as art: a painting, illustration, cartoon or sculpture, as the image itself or shown in it; not drawings made to explain (those are Diagram)
• Instruments (instruments): musical instruments: guitar, piano, violin, drums, wind instruments and the like
• Furniture (furniture): chairs, tables, beds, sofas, desks, cabinets, shelves and other furniture (a data table is Table)
• Aerial (aerial): the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map

Usage

Download a fixed version (recommended), so a later release never changes your results:

from huggingface_hub import hf_hub_download

model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-EfficientNetV2-B0", "model.onnx", revision="v0.4.0")
config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-EfficientNetV2-B0", "config.json", revision="v0.4.0")

Then:

import json
import numpy as np
import onnxruntime as ort
from PIL import Image

config = json.load(open(config_path))
session = ort.InferenceSession(model_path)
size = config["input"]["shape"][-1]
mean = np.array(config["input"]["mean"], dtype=np.float32)
std = np.array(config["input"]["std"], dtype=np.float32)

image = Image.open("photo.jpg").convert("RGB").resize((size, size), Image.BILINEAR)
x = ((np.asarray(image, dtype=np.float32) / 255 - mean) / std).transpose(2, 0, 1)[None]
probabilities = 1 / (1 + np.exp(-session.run(None, {"pixel_values": x})[0][0]))
present = [label for label, p in zip(config["labels"], probabilities)
           if p >= config["thresholds"][label]]

Preprocessing: RGB, resized to 224×224 (antialiased bilinear, no crop, aspect ratio not kept), scaled to [0, 1], normalised with the ImageNet mean and standard deviation, NCHW float32. Each label has its own threshold in config.json, tuned on a validation split.

Evaluation

On 3435 held-out teacher-labelled images: mean average precision 0.91, macro F1 0.85 at the tuned thresholds.

label	name	positives	AP	F1
people	People	626	0.95	0.89
animals	Animals	268	0.87	0.82
nature	Plants	382	0.89	0.82
landscape	Landscape	438	0.93	0.84
buildings	Buildings / structures	422	0.91	0.84
vehicles	Vehicles	185	0.82	0.74
food	Food	106	0.76	0.72
text	Text	1035	0.95	0.90
handwriting	Handwriting	243	0.86	0.80
chart	Chart	260	0.98	0.95
diagram	Diagram	235	0.94	0.93
table	Table	275	0.97	0.91
map	Map	169	0.95	0.92
logo-icon	Logo / icon	551	0.84	0.77
formula	Formula	30	0.90	0.81
screen	Screen / UI	377	0.97	0.92
artwork	Artwork	474	0.94	0.88
instruments	Instruments	64	0.87	0.85
furniture	Furniture	183	0.81	0.73
aerial	Aerial	161	0.97	0.93

How it compares

Against its teachers on the 3,435 test images: mean AP 0.91, macro F1 0.85 at 0.5 (MobileNetV4 v0.4.0-rc.1 on the same images: 0.88 and 0.83). On the 5,158 validation images macro F1 at 0.5 is 0.857, about the same as v0.3.0's 0.858, but v0.3.0 trained on most of those images; on the part it never saw, v0.3.0 reaches 0.74. About 14 ms per image on one CPU thread. Against v0.3.0 on 60 independent hand-labelled images: slightly worse (mean AP 0.81 against 0.87, macro F1 at 0.5 0.76 against 0.77 to 0.78). Better on Animals, Screen / UI, Map and Artwork; weaker on People, Landscape, Handwriting and Logo / icon. Hand labels and teacher labels differ in what they count, and per-label scores on the 60 rest on 1 to 24 images: indicative only. A larger EfficientViT-B2 model scored higher (confirm macro F1 0.86) at four times the CPU time; it was not released.

Training

• Backbone: tf_efficientnetv2_b0.in1k (ImageNet-pretrained, via timm), with a new 20-way sigmoid head; 30 epochs, AdamW, binary cross-entropy per label.
• Data: The same teachers and images as WhatDoWeSee-MobileNetV4 v0.4.0-rc.1: Cloudflare Clef-omni, Gemma 4 26B-A4B and Qwen3-VL 32B, run on EuroHPC (Discoverer+), answered the attribute question for all 44,361 images of the full-v1 corpus. No hosted API and no hand labels are involved; the 681 hand-labelled images are left out entirely, and near-duplicates (pHash) stay in one split. New in v0.4.0: instead of a yes/no target, the model learns from the vote count (all three agree 0.95, two 0.35, one 0.10, none 0.02). Trained on 28,495 images, 30 epochs from ImageNet weights, best epoch picked on 1,441 held-out training images. The backbone (tf_efficientnetv2_b0, 224 px) won a champion-challenger selection over 35 backbones and several training variants, scored on 5,158 validation images where all three teachers agree; thresholds are tuned on those. Test: 3,435 teacher-labelled images it never trained, tuned or was selected on.
• The images come from public collections: photo datasets, document and form scans, charts, diagrams, screenshots, maps, artwork and icons.

Limitations

• Small text-related details (a small table inside a document, a signature) are harder than whole-image subjects.
• The training images come from many public collections, some under research-only or non-commercial terms. Check whether that matters for your use.

Licence

The model weights and config.json are released under the Apache License 2.0, like the EfficientNetV2-B0 backbone they start from. The licence covers the model, not the images it was trained on.

Versions

Each version on test images it never trained on (mean AP and macro F1). The test sets differ between some versions, so compare versions through the notes of each version:

version	date	test images	mean AP	macro F1
v0.4.0	2026-10-16	3435	0.91	0.85	trained on 28,495 images labelled by three open vision-language models (Clef-omni, Gemma 4 26B, Qwen3-VL 32B), learning from how many of them agree