Debug Agent commited on
Commit
282557d
Β·
1 Parent(s): 8a1c0ed

Move harness logos to assets/harnesses/ with graceful missing-file fallback

Browse files

The previous two passes shipped placeholder SVG badges in assets/ directly
(first plain letters, then generic CLI prompt glyphs). Neither was a real
logo, and the review feedback was to ship the slot without the art so the
actual logos can be dropped in by hand later.

Restructure so the code no longer assumes any specific file exists:

- New dedicated folder assets/harnesses/ with only a README documenting the
expected filenames and the mapping from agent_name -> file stem. No
placeholder artwork ships with the repo.

- leaderboard_transformer.py: replace the hardcoded HARNESS_LOGO_PATHS map
with HARNESS_LOGO_STEMS + a small resolver in get_harness_icon() that
tries {stem}.svg then {stem}.png under assets/harnesses/ and returns
None when no file exists. The three cases that yield None (empty
agent_name, unregistered agent_name, or no file on disk) all make the
scatter-plot caller skip the harness layer, so every point falls back to
a single model-provider marker β€” the exact same rendering the canonical
OpenHands pages use via has_mixed_agents. No crash, no warning, no
empty-placeholder box.

- Delete the four old assets/harness-*.svg placeholders so the repo is
clean on a fresh checkout.

This means rollout can be incremental: drop in assets/harnesses/openhands.svg
first, the OpenHands and OpenHands Sub-agents rows start showing composite
markers on next app restart, and the other three harnesses keep rendering
single markers until their files land too.

Smoke-tested both paths:
- Empty assets/harnesses/: 7 alt-agent rows -> 7 scatter markers (1 per point).
- With a single claude-code.svg present: 7 rows -> 9 markers, per-row
counts [2,2,1,1,1,1,1] matching the two Claude Code rows.

assets/harness-claude-code.svg DELETED
assets/harness-codex-cli.svg DELETED
assets/harness-gemini-cli.svg DELETED
assets/harness-openhands.svg DELETED
assets/harnesses/README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Agent harness logos
2
+
3
+ This folder holds the **bottom half** of the composite scatter markers used
4
+ on the [Alternative Agents](../../alternative_agents_page.py) page. Each
5
+ point on that scatter stacks two logos: the model provider on top (from
6
+ `assets/logo-*.svg`) and the harness on the bottom (from this folder).
7
+
8
+ ## Expected filenames
9
+
10
+ The scatter code looks up a logo by the exact `agent_name` string that the
11
+ `push-to-index` workflow writes into the index repo's `metadata.json`, then
12
+ maps it through `HARNESS_LOGO_STEMS` in `leaderboard_transformer.py`. Keep
13
+ these filenames in sync with that map.
14
+
15
+ | `agent_name` (in index repo) | File in this folder |
16
+ | --- | --- |
17
+ | `Claude Code` | `claude-code.svg` or `claude-code.png` |
18
+ | `Codex` | `codex-cli.svg` or `codex-cli.png` |
19
+ | `Gemini CLI` | `gemini-cli.svg` or `gemini-cli.png` |
20
+ | `OpenHands` | `openhands.svg` or `openhands.png` |
21
+ | `OpenHands Sub-agents` | `openhands.svg` or `openhands.png` (shared with `OpenHands`) |
22
+
23
+ Both `.svg` and `.png` are accepted β€” the resolver tries `.svg` first, then
24
+ `.png`. **Prefer SVG when possible**: the HuggingFace Space rejects new
25
+ binary files on plain `git push` and routes PNGs through Xet, so an SVG is
26
+ one less thing to set up.
27
+
28
+ ## When a file is missing
29
+
30
+ The scatter falls back to a single marker (just the model provider logo) β€”
31
+ exactly the same rendering path the canonical OpenHands pages use. Nothing
32
+ crashes and nothing prints a warning in normal operation. This means you
33
+ can roll out logos one harness at a time without waiting for all four.
34
+
35
+ ## Sizing and shape
36
+
37
+ - Square canvas. The composite marker is drawn at a fixed aspect ratio, so
38
+ a non-square logo will get squished.
39
+ - Any SVG `viewBox` works β€” the renderer base64-encodes the file as-is and
40
+ Plotly scales it to the marker's `sizex` / `sizey`. Around `80Γ—80` to
41
+ `256Γ—256` is a good source size.
42
+ - Leave some internal padding (β‰ˆ10%) so the logo doesn't touch the marker
43
+ edge when two are stacked.
44
+ - No background is required, but a rounded-square coloured tile reads well
45
+ at small sizes because it gives each harness a distinct silhouette even
46
+ when the inner glyph isn't fully legible. Look at the existing
47
+ `assets/logo-*.svg` files for the canonical model provider logos if you
48
+ want a visual reference for sizing.
49
+
50
+ ## Adding a new harness
51
+
52
+ 1. Decide on the exact `agent_name` that the push-to-index workflow writes
53
+ for the new harness (see `AGENT_NAME_BY_TYPE` in
54
+ `OpenHands/evaluation/push-to-index-job/scripts/push_to_index_from_archive.py`).
55
+ 2. Add an entry to `HARNESS_LOGO_STEMS` in
56
+ [`leaderboard_transformer.py`](../../leaderboard_transformer.py) that
57
+ maps the display name to a stem.
58
+ 3. Drop `{stem}.svg` (or `.png`) into this folder.
59
+ 4. Reload the app and look at `/alternative-agents`.
leaderboard_transformer.py CHANGED
@@ -247,37 +247,57 @@ def get_marker_icon(model_name: str, openness: str, mark_by: str) -> dict:
247
  return get_company_from_model(model_name)
248
 
249
 
250
- # Map the agent_name stored in the index repo's metadata.json to a harness
251
- # logo file. Kept in sync with AGENT_NAME_BY_TYPE in OpenHands/evaluation
252
- # push_to_index_from_archive.py β€” if a new ACP harness lands there, add the
253
- # corresponding display name and a matching asset here. Unknown agent_name
254
- # values fall through to None so the scatter plot just draws the model logo
255
- # alone (no blank placeholder).
256
- HARNESS_LOGO_PATHS: dict[str, str] = {
257
- "Claude Code": "assets/harness-claude-code.svg",
258
- "Codex": "assets/harness-codex-cli.svg",
259
- "Gemini CLI": "assets/harness-gemini-cli.svg",
260
- "OpenHands": "assets/harness-openhands.svg",
261
- "OpenHands Sub-agents": "assets/harness-openhands.svg",
 
 
 
 
 
 
262
  }
 
 
263
 
264
 
265
  def get_harness_icon(agent_name: Optional[str]) -> Optional[dict]:
266
- """Return {'path', 'name'} for the harness logo, or None if unknown.
267
 
268
  Consumed by the Alternative Agents scatter plot to draw a composite
269
- marker (model provider on top, harness on bottom). Empty/missing
270
- agent_name yields None so the caller can skip the harness layer and
271
- fall back to a plain model logo β€” which is what canonical OpenHands
272
- pages hit, since their DataTransformer.view() drops the Agent column
273
- entirely when there's only one agent.
 
 
 
 
 
 
 
 
274
  """
275
  if not agent_name:
276
  return None
277
- path = HARNESS_LOGO_PATHS.get(str(agent_name).strip())
278
- if path is None:
279
  return None
280
- return {"path": path, "name": agent_name}
 
 
 
 
281
 
282
 
283
  # Standard layout configuration for all charts
 
247
  return get_company_from_model(model_name)
248
 
249
 
250
+ # Map the agent_name stored in the index repo's metadata.json to a file stem
251
+ # inside assets/harnesses/. Kept in sync with AGENT_NAME_BY_TYPE in
252
+ # OpenHands/evaluation push_to_index_from_archive.py β€” if a new ACP harness
253
+ # lands there, add the corresponding display name and a matching stem here.
254
+ #
255
+ # The scatter plot looks for {stem}.svg first, then {stem}.png in
256
+ # assets/harnesses/. This repo intentionally ships only a README in that
257
+ # folder: drop the logo files in by hand (SVG preferred, PNG works too via
258
+ # HF Xet) and they'll be picked up on the next app restart. If the file is
259
+ # missing, get_harness_icon() returns None and the scatter falls back to the
260
+ # single-marker path β€” same rendering the canonical OpenHands pages use β€”
261
+ # so logos can be added one harness at a time without breaking anything.
262
+ HARNESS_LOGO_STEMS: dict[str, str] = {
263
+ "Claude Code": "claude-code",
264
+ "Codex": "codex-cli",
265
+ "Gemini CLI": "gemini-cli",
266
+ "OpenHands": "openhands",
267
+ "OpenHands Sub-agents": "openhands",
268
  }
269
+ HARNESS_LOGO_DIR = "assets/harnesses"
270
+ HARNESS_LOGO_EXTENSIONS = ("svg", "png")
271
 
272
 
273
  def get_harness_icon(agent_name: Optional[str]) -> Optional[dict]:
274
+ """Return {'path', 'name'} for the harness logo, or None if not usable.
275
 
276
  Consumed by the Alternative Agents scatter plot to draw a composite
277
+ marker (model provider on top, harness on bottom). Returns None in any
278
+ of three cases, all of which make the caller skip the harness layer:
279
+
280
+ - ``agent_name`` is empty or missing from the dataframe row.
281
+ - ``agent_name`` isn't in ``HARNESS_LOGO_STEMS`` (new harness that
282
+ hasn't been registered yet β€” register it and drop in a logo).
283
+ - The logo file for that stem doesn't exist in ``assets/harnesses/``
284
+ yet (the repo ships only the README).
285
+
286
+ That third case is the important one: it lets the Alternative Agents
287
+ page work immediately after checkout even when the harness logo files
288
+ haven't been dropped in. The corresponding points just render like a
289
+ canonical-page marker (model logo only) until the file is added.
290
  """
291
  if not agent_name:
292
  return None
293
+ stem = HARNESS_LOGO_STEMS.get(str(agent_name).strip())
294
+ if stem is None:
295
  return None
296
+ for ext in HARNESS_LOGO_EXTENSIONS:
297
+ path = f"{HARNESS_LOGO_DIR}/{stem}.{ext}"
298
+ if os.path.exists(path):
299
+ return {"path": path, "name": agent_name}
300
+ return None
301
 
302
 
303
  # Standard layout configuration for all charts