371 MB
513 files
Updated about 1 month ago
Name
Size
images
tuples
README.md12.8 kB
xet
ground_truth.json25.3 kB
xet
object_location.json4.87 kB
xet
prompt_templates.json5.55 kB
xet
README.md

brands

A VADE entity whose objects are company logos instead of country flags. Identical file layout to data/flags/, identical tuple schema, scored by the same eval/score.py.

The point of a second entity is that a method which finds an attribute subspace in one should find it in the other. Flags and brands differ in the way that matters — a flag is a fixed-shape emblem, a logo is a mark the model may be reading rather than recognising — while everything around them is held constant.

128 brands · 3 attributes · 90 train / 38 test · 101,133 intervention rows

Model-independent. Ground truth is external — Wikidata, not any model's answers — so the entity is the same benchmark for every model, and Cause/Iso scores are comparable across them. Which cells a given model can actually read off a logo is measured separately and applied downstream; see Per-model pruning.

This changed on 2026-09-04. Until then a cell existed only where Qwen2.5-VL-7B-Instruct answered it stably and in agreement with Wikidata, and the stored string was that model's own wording. That made the entity an artifact of one model: coverage was ragged (124/93/100), Iso was optimistic because the removed cells were exactly the shaky ones, and a baseline accuracy sweep against it was circular — it scored 97.2% by asking the model to reproduce answers admitted because the model had produced them. The old ground truth and tuples are kept in build/_archive_model_conditioned/; every number reported from this entity before that date came from them.

Attributes

Every brand carries every attribute Wikidata supplies.

attribute brands source
hq_country 128 P159/P17, mapped to short common form by build/country_short_form.json
founded_year 128 P571, first 4-digit year of inception
industry 127 P452 resolved into twelve sectors by build/sector_map.json, plus build/industry_overrides.json
all three 127

Labels are stored lowercase, one plain string per cell, no accept sets — the same contract as flags and animals. eval/score.py's contains_label() looks for the gold label inside the generation, which is why hq_country uses the short common form: gold china matches a generation of "the People's Republic of China", and the reverse does not.

One ragged cell. Airbus has no industry: its P452 statement is "aircraft and space construction", and no sector in the twelve-item menu covers aerospace. The menu is baked into prompt_templates.json, so extending it would invalidate every probe already run. build/step9_build_tuples.py emits an Iso probe only where both brands in a pair have the attribute, so Airbus contributes country and year rows and no industry rows. That is the only place raggedness survives.

The industry overrides are the entity's only hand-entered attribute data. Twelve brands where P452 resolution returns nothing — no keyword hit, or a tie between two sectors. Each override in build/industry_overrides.json quotes the brand's actual P452 statements alongside the call, so it can be audited without re-querying. The two weakest are Samsung and LG, whose only P452 statement is "conglomerate"; both were assigned electronics on the mark's public identity, which is judgement, not Wikidata.

Pornhub was removed from the pool on 2026-09-04 — a suitability call, not a data-quality one; its cells resolved cleanly. See build/purge_extra.json.

Per-model pruning

"Can this model read this attribute off this logo?" is a question about a model, not about the world, so it lives one layer down, in models/ — the same arrangement flags and animals use:

models/run_accuracy_sweep.py   zero-shot baseline: every (item × attribute ×
                               template), one generation, checked against this
                               entity's own ground truth
                               → models/<model_slug>/brands_predictions.jsonl
                               → models/<model_slug>/brands_summary.{json,md}

models/prune_tuples.py         drops any tuples row whose gradient target the
                               model cannot support at baseline — a cause row
                               needs the SOURCE read correctly, an iso row needs
                               the BASE
                               → models/<model_slug>/brands/tuples/<attr>/{train,test}.jsonl
                               → models/<model_slug>/brands_tuples_prune_summary.{json,md}

eval/score.py accepts --tuples_dir, so a pruned tree drops straight in.

Expect the pruned trees to be noticeably smaller here than for flags, which loses 3–8%: founded_year in particular is where a 7B model answers about a different corporate entity (see below), and every one of those items removes both the cause rows that target it and the iso rows that rely on it.

recognition_freeform templates ship in prompt_templates.json but no tuples are built from them. Run it alongside as an unscored diagnostic: it answers "did I edit one fact, or swap the whole brand?", which nothing else does. Keep it out of final_score.

Why founding year is the weak attribute

Under the old model-conditioned ground truth this attribute admitted only 70% of brands, and the reason is worth keeping because it now predicts what pruning will remove. Not format — every answer is one word. It is corporate lineage: the model says Marvel Comics 1939 where Wikidata says 1961, Yamaha Motor 1887 (the piano company) against 1955 (the motorcycle spin-off), KFC 1952 against 1930. The model is answering about a different corporate entity, not guessing.

Those items are no longer removed from the entity — they are in the tuples, and prune_tuples.py removes them per model. The disagreement is now measured rather than assumed away, which also means a model with better corporate knowledge keeps rows this one loses.

Why ground truth has to be external

The clean pass showed what happens without it. Five of the six country mismatches answered "United States" — Lenovo, DeepSeek and both East India Companies among them. That is the model's fallback when it does not know, and it is perfectly stable across templates. Consistency with itself cannot catch that; an independent source can. This is the argument for Wikidata deciding the label, and it is also the argument against a model deciding which cells exist.

ticker was tried and dropped

A registry fact, the brands equivalent of flags' calling code — the strongest idea available on paper. It does not survive a 7B model:

stability across 5 templates: 46% mean, 0% unanimous
141 distinct answers across 200 generations

Nintendo   NINT. / "NTDO" / NINT. / "NTDO" / NINT.     real: 7974
AB Volvo   VOV.  / "VOLVO" / VVO. / "VOLVO" / VVO.     real: VOLV-B
Shell      SHELL. / "RDSA" / SHEL. / "RDSA" / SHELL.   real: SHEL

Not one brand of 40 gave the same ticker five times. The model builds a plausible abbreviation from the company name and which one depends on the phrasing. Alphabet came back GOOGL — US large caps are attested, the rest is confabulated.

industry uses the model's vocabulary, not GICS

Three variants, head to head on 40 brands:

variant stable ≥4/5 unanimous distinct answers
GICS sector menu 72% 40% 32
model-vocabulary menu 98% 90% 14
no menu 85% 40% 40

GICS forces categories the model does not hold — Volkswagen to Industrials, Nintendo to Consumer Discretionary, Rolex to Industrials — so it picks differently depending on phrasing. Given words it uses unprompted it answers the same way 90% of the time.

Deriving the label set from the model is circular, and the guard is that ground truth stays external: Wikidata P452, resolved into the twelve sectors in build/sector_map.json by highest matching-statement count. The model supplies the wording; Wikidata decides admission.

How this differs from flags

One shared object box — flags' box. An interchange intervention copies token positions between runs, so it only means anything if the same positions hold the entity in both images, and the size of that box is the size of the intervention. Brands uses 112×56 at (112,140), byte-identical token indices to flags:

flags    8 of 144 tokens (5.6%)   100% of the box is the flag
brands   8 of 144 tokens (5.6%)   median 50% of the box is the mark
animals 36 of 144 tokens (25%)    median 34% of the box is the animal

Measured per base/source pair, the number of tokens whose pixels actually differ is a median of 8 for flags and 8 for brands — the two entities now receive the same-sized edit, so their Cause and Iso curves are directly comparable.

Until 2026-09-05 brands used animals' 168×168 box: 36 tokens, a median of 24 differing per pair. That is close to handing the model the other image, and it showed — Cause was already high in the earliest layers, which is the signature of entity substitution rather than attribute editing. Old images and every result measured against them are in build/_archive_box168/.

Marks are scaled to fit inside the box, not to fill it, and the padding is the canvas gray, identical in every image. Aspect ratios run from 0.75:1 roundels to 13:1 wordmarks, so a wide wordmark renders short: median mark height is 33 px inside the 56 px box, minimum 8 px, and 23 of 128 come out under 20 px tall. That is the price of the smaller box, and it is paid in legibility — expect baseline accuracy below what the 168 box gave, most of it in the attributes that lean on reading the name. Per-image rendered sizes are in build/aligned_boxes_flagbox.json.

Wordmarks are included. Textless-only was measured and is unaffordable: being textless is independent of fame, so the two filters multiply down to ~13 usable brands against ~99 without the fame cut. So entity identity partly arrives by reading rather than by recognising a shape — that belongs in the writeup.

The pool is fame-selected. Model knowledge tracks fame hard (founding-year admission runs 39% below 50 sitelinks against 89% above 100), so the pool is cut at sitelinks ≥ 70 before anything else.

Three attributes, not four. Ticker was measured and dropped rather than carried.

row_index is continuous across files. In flags it restarts at 0 per split, so ("capital", 0) names one row in train.jsonl and a different one in test.jsonl. score.py keys on (attribute, row_index) alone, so scoring a train predictions file against the test split grades every answer against the wrong question — and the missing-prediction check cannot catch it, because the index ranges overlap. Continuous numbering makes that impossible here without touching the scorer.

Scoring

eval/score.pyreads this entity as-is —--attributedefaults toalland enumerates the entity's owntuples/` subdirectories, so nothing is hardcoded to flags' four attributes any more.

python eval/score.py --predictions <file> --entity brands --split test
python eval/score.py --predictions <file> --entity brands --split test \
    --tuples_dir models/Qwen2.5-VL-7B-Instruct/brands/tuples

2a20e2a (brands resized and pruned):brands/README.md

build/

Not part of the entity contract — score.py only reads tuples/. Kept here so the pipeline sits next to what it produced.

step2b_render_aligned.py renders marks into the shared patch-aligned box
step3_reshortlist.py fame cut over the Wikidata pool
step3c_purge_and_labels.py removes non-brands, recovers missing names
step2_template_audit.py flags templates asking for a different answer form
check_setup.py / run_pass.py the GPU scripts — see RUNPOD.md
step8_ground_truth.py SUPERSEDED — clean-pass answers → model-conditioned ground_truth.json
step8b_ground_truth_full.py Wikidata → ground_truth.json, every brand, no model
country_short_form.json Wikidata country label → the short form used as the gold label
industry_overrides.json the 12 hand-adjudicated sectors, each quoting its P452 statements
step9_build_tuples.py ground truth → tuples/
step10_object_location.py writes and verifies object_location.json
clean_pass.jsonl the raw answers everything above was derived from
sector_map.json the twelve sectors and their P452 ground truth
_archive_model_conditioned/ the pre-2026-09-04 ground_truth.json and tuples/
_to_delete/ superseded files and unreferenced images; no delete permission on this mount
Total size
371 MB
Files
513
Last updated
Oct 21
Pre-warmed CDN
US EU US EU

Contributors