Spaces:
Running
Move harness logos to assets/harnesses/ with graceful missing-file fallback
Browse filesThe previous two passes shipped placeholder SVG badges in assets/ directly
(first plain letters, then generic CLI prompt glyphs). Neither was a real
logo, and the review feedback was to ship the slot without the art so the
actual logos can be dropped in by hand later.
Restructure so the code no longer assumes any specific file exists:
- New dedicated folder assets/harnesses/ with only a README documenting the
expected filenames and the mapping from agent_name -> file stem. No
placeholder artwork ships with the repo.
- leaderboard_transformer.py: replace the hardcoded HARNESS_LOGO_PATHS map
with HARNESS_LOGO_STEMS + a small resolver in get_harness_icon() that
tries {stem}.svg then {stem}.png under assets/harnesses/ and returns
None when no file exists. The three cases that yield None (empty
agent_name, unregistered agent_name, or no file on disk) all make the
scatter-plot caller skip the harness layer, so every point falls back to
a single model-provider marker β the exact same rendering the canonical
OpenHands pages use via has_mixed_agents. No crash, no warning, no
empty-placeholder box.
- Delete the four old assets/harness-*.svg placeholders so the repo is
clean on a fresh checkout.
This means rollout can be incremental: drop in assets/harnesses/openhands.svg
first, the OpenHands and OpenHands Sub-agents rows start showing composite
markers on next app restart, and the other three harnesses keep rendering
single markers until their files land too.
Smoke-tested both paths:
- Empty assets/harnesses/: 7 alt-agent rows -> 7 scatter markers (1 per point).
- With a single claude-code.svg present: 7 rows -> 9 markers, per-row
counts [2,2,1,1,1,1,1] matching the two Claude Code rows.
- assets/harness-claude-code.svg +0 -8
- assets/harness-codex-cli.svg +0 -8
- assets/harness-gemini-cli.svg +0 -8
- assets/harness-openhands.svg +0 -12
- assets/harnesses/README.md +59 -0
- leaderboard_transformer.py +41 -21
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Agent harness logos
|
| 2 |
+
|
| 3 |
+
This folder holds the **bottom half** of the composite scatter markers used
|
| 4 |
+
on the [Alternative Agents](../../alternative_agents_page.py) page. Each
|
| 5 |
+
point on that scatter stacks two logos: the model provider on top (from
|
| 6 |
+
`assets/logo-*.svg`) and the harness on the bottom (from this folder).
|
| 7 |
+
|
| 8 |
+
## Expected filenames
|
| 9 |
+
|
| 10 |
+
The scatter code looks up a logo by the exact `agent_name` string that the
|
| 11 |
+
`push-to-index` workflow writes into the index repo's `metadata.json`, then
|
| 12 |
+
maps it through `HARNESS_LOGO_STEMS` in `leaderboard_transformer.py`. Keep
|
| 13 |
+
these filenames in sync with that map.
|
| 14 |
+
|
| 15 |
+
| `agent_name` (in index repo) | File in this folder |
|
| 16 |
+
| --- | --- |
|
| 17 |
+
| `Claude Code` | `claude-code.svg` or `claude-code.png` |
|
| 18 |
+
| `Codex` | `codex-cli.svg` or `codex-cli.png` |
|
| 19 |
+
| `Gemini CLI` | `gemini-cli.svg` or `gemini-cli.png` |
|
| 20 |
+
| `OpenHands` | `openhands.svg` or `openhands.png` |
|
| 21 |
+
| `OpenHands Sub-agents` | `openhands.svg` or `openhands.png` (shared with `OpenHands`) |
|
| 22 |
+
|
| 23 |
+
Both `.svg` and `.png` are accepted β the resolver tries `.svg` first, then
|
| 24 |
+
`.png`. **Prefer SVG when possible**: the HuggingFace Space rejects new
|
| 25 |
+
binary files on plain `git push` and routes PNGs through Xet, so an SVG is
|
| 26 |
+
one less thing to set up.
|
| 27 |
+
|
| 28 |
+
## When a file is missing
|
| 29 |
+
|
| 30 |
+
The scatter falls back to a single marker (just the model provider logo) β
|
| 31 |
+
exactly the same rendering path the canonical OpenHands pages use. Nothing
|
| 32 |
+
crashes and nothing prints a warning in normal operation. This means you
|
| 33 |
+
can roll out logos one harness at a time without waiting for all four.
|
| 34 |
+
|
| 35 |
+
## Sizing and shape
|
| 36 |
+
|
| 37 |
+
- Square canvas. The composite marker is drawn at a fixed aspect ratio, so
|
| 38 |
+
a non-square logo will get squished.
|
| 39 |
+
- Any SVG `viewBox` works β the renderer base64-encodes the file as-is and
|
| 40 |
+
Plotly scales it to the marker's `sizex` / `sizey`. Around `80Γ80` to
|
| 41 |
+
`256Γ256` is a good source size.
|
| 42 |
+
- Leave some internal padding (β10%) so the logo doesn't touch the marker
|
| 43 |
+
edge when two are stacked.
|
| 44 |
+
- No background is required, but a rounded-square coloured tile reads well
|
| 45 |
+
at small sizes because it gives each harness a distinct silhouette even
|
| 46 |
+
when the inner glyph isn't fully legible. Look at the existing
|
| 47 |
+
`assets/logo-*.svg` files for the canonical model provider logos if you
|
| 48 |
+
want a visual reference for sizing.
|
| 49 |
+
|
| 50 |
+
## Adding a new harness
|
| 51 |
+
|
| 52 |
+
1. Decide on the exact `agent_name` that the push-to-index workflow writes
|
| 53 |
+
for the new harness (see `AGENT_NAME_BY_TYPE` in
|
| 54 |
+
`OpenHands/evaluation/push-to-index-job/scripts/push_to_index_from_archive.py`).
|
| 55 |
+
2. Add an entry to `HARNESS_LOGO_STEMS` in
|
| 56 |
+
[`leaderboard_transformer.py`](../../leaderboard_transformer.py) that
|
| 57 |
+
maps the display name to a stem.
|
| 58 |
+
3. Drop `{stem}.svg` (or `.png`) into this folder.
|
| 59 |
+
4. Reload the app and look at `/alternative-agents`.
|
|
@@ -247,37 +247,57 @@ def get_marker_icon(model_name: str, openness: str, mark_by: str) -> dict:
|
|
| 247 |
return get_company_from_model(model_name)
|
| 248 |
|
| 249 |
|
| 250 |
-
# Map the agent_name stored in the index repo's metadata.json to a
|
| 251 |
-
#
|
| 252 |
-
# push_to_index_from_archive.py β if a new ACP harness
|
| 253 |
-
# corresponding display name and a matching
|
| 254 |
-
#
|
| 255 |
-
#
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 262 |
}
|
|
|
|
|
|
|
| 263 |
|
| 264 |
|
| 265 |
def get_harness_icon(agent_name: Optional[str]) -> Optional[dict]:
|
| 266 |
-
"""Return {'path', 'name'} for the harness logo, or None if
|
| 267 |
|
| 268 |
Consumed by the Alternative Agents scatter plot to draw a composite
|
| 269 |
-
marker (model provider on top, harness on bottom).
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 274 |
"""
|
| 275 |
if not agent_name:
|
| 276 |
return None
|
| 277 |
-
|
| 278 |
-
if
|
| 279 |
return None
|
| 280 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 281 |
|
| 282 |
|
| 283 |
# Standard layout configuration for all charts
|
|
|
|
| 247 |
return get_company_from_model(model_name)
|
| 248 |
|
| 249 |
|
| 250 |
+
# Map the agent_name stored in the index repo's metadata.json to a file stem
|
| 251 |
+
# inside assets/harnesses/. Kept in sync with AGENT_NAME_BY_TYPE in
|
| 252 |
+
# OpenHands/evaluation push_to_index_from_archive.py β if a new ACP harness
|
| 253 |
+
# lands there, add the corresponding display name and a matching stem here.
|
| 254 |
+
#
|
| 255 |
+
# The scatter plot looks for {stem}.svg first, then {stem}.png in
|
| 256 |
+
# assets/harnesses/. This repo intentionally ships only a README in that
|
| 257 |
+
# folder: drop the logo files in by hand (SVG preferred, PNG works too via
|
| 258 |
+
# HF Xet) and they'll be picked up on the next app restart. If the file is
|
| 259 |
+
# missing, get_harness_icon() returns None and the scatter falls back to the
|
| 260 |
+
# single-marker path β same rendering the canonical OpenHands pages use β
|
| 261 |
+
# so logos can be added one harness at a time without breaking anything.
|
| 262 |
+
HARNESS_LOGO_STEMS: dict[str, str] = {
|
| 263 |
+
"Claude Code": "claude-code",
|
| 264 |
+
"Codex": "codex-cli",
|
| 265 |
+
"Gemini CLI": "gemini-cli",
|
| 266 |
+
"OpenHands": "openhands",
|
| 267 |
+
"OpenHands Sub-agents": "openhands",
|
| 268 |
}
|
| 269 |
+
HARNESS_LOGO_DIR = "assets/harnesses"
|
| 270 |
+
HARNESS_LOGO_EXTENSIONS = ("svg", "png")
|
| 271 |
|
| 272 |
|
| 273 |
def get_harness_icon(agent_name: Optional[str]) -> Optional[dict]:
|
| 274 |
+
"""Return {'path', 'name'} for the harness logo, or None if not usable.
|
| 275 |
|
| 276 |
Consumed by the Alternative Agents scatter plot to draw a composite
|
| 277 |
+
marker (model provider on top, harness on bottom). Returns None in any
|
| 278 |
+
of three cases, all of which make the caller skip the harness layer:
|
| 279 |
+
|
| 280 |
+
- ``agent_name`` is empty or missing from the dataframe row.
|
| 281 |
+
- ``agent_name`` isn't in ``HARNESS_LOGO_STEMS`` (new harness that
|
| 282 |
+
hasn't been registered yet β register it and drop in a logo).
|
| 283 |
+
- The logo file for that stem doesn't exist in ``assets/harnesses/``
|
| 284 |
+
yet (the repo ships only the README).
|
| 285 |
+
|
| 286 |
+
That third case is the important one: it lets the Alternative Agents
|
| 287 |
+
page work immediately after checkout even when the harness logo files
|
| 288 |
+
haven't been dropped in. The corresponding points just render like a
|
| 289 |
+
canonical-page marker (model logo only) until the file is added.
|
| 290 |
"""
|
| 291 |
if not agent_name:
|
| 292 |
return None
|
| 293 |
+
stem = HARNESS_LOGO_STEMS.get(str(agent_name).strip())
|
| 294 |
+
if stem is None:
|
| 295 |
return None
|
| 296 |
+
for ext in HARNESS_LOGO_EXTENSIONS:
|
| 297 |
+
path = f"{HARNESS_LOGO_DIR}/{stem}.{ext}"
|
| 298 |
+
if os.path.exists(path):
|
| 299 |
+
return {"path": path, "name": agent_name}
|
| 300 |
+
return None
|
| 301 |
|
| 302 |
|
| 303 |
# Standard layout configuration for all charts
|