Add token efficiency analysis and visualizations

README.md

@@ -15,26 +15,103 @@ tags:
 
 # Coding Agent Leaderboard
 
+Compare coding-agent models and harnesses across benchmark performance, cost, latency, and token usage.
+
+## Leaderboard views
+
+### Cost vs Performance
+
+The Cost vs Performance scatter plot compares benchmark score with mean cost per task. Use its controls to select the benchmark, grouping, color palette, image background, and optional point labels.
+
+Point labels are disabled by default to keep the chart uncluttered. When enabled, concise run labels are shown without replacing the full hover details.
+
+### Token Efficiency
+
+The **Efficiency** tab shows that score alone is insufficient: agents with similar benchmark performance can use substantially different token budgets.
+
+The tab provides:
+
+- benchmark filtering, including **All benchmarks**
+- total, input, output, or solved-task-normalized token metrics
+- linear or logarithmic token axes
+- coloring by model, harness, or benchmark
+- optional point labels
+- an optional token-efficiency Pareto frontier
+- a sortable token-efficiency ranking table
+
+#### Metric definitions
+
+Scores are stored internally as fractions from 0 to 1 and displayed as percentages.
+
+- **Total Tokens Per Task**: mean total token usage for one task.
+- **Input Tokens Per Task**: mean input token usage for one task.
+- **Output Tokens Per Task**: mean output token usage for one task.
+- **Cache Tokens Per Task**: mean cached token usage for one task.
+- **Tokens Per Solved Task**: `Total Tokens Per Task / Score`, where Score is the fractional value from 0 to 1.
+
+For example, a run using 10,000 total tokens per task with a score of 0.5 has 20,000 Tokens Per Solved Task.
+
+#### Missing and invalid token data
+
+A token measurement is available only when its value is present and greater than zero. Missing, zero, and negative token values are excluded from token-efficiency plots rather than treated as perfect efficiency.
+
+Tokens Per Solved Task is also unavailable when the score is zero or negative, which avoids division by zero. The UI reports how many runs were omitted because the selected token metric was missing or non-positive.
+
+#### Pareto frontier
+
+A run is token Pareto-efficient when no other displayed run:
+
+- uses fewer or equal tokens, and
+- achieves an equal or higher score,
+
+with at least one of those comparisons being a strict improvement.
+
+The dashed frontier connects the efficient runs. Points below and to the right of the frontier are dominated: another run achieves at least as much score with no more token usage.
+
+The dotted horizontal lines at **70%** and **80%** are fixed capability guides for visual orientation. They do not affect Pareto membership and are not statistical thresholds.
+
+#### Color palettes and themes
+
+The Cost vs Performance and Efficiency charts share the same palette registry and support:
+
+- Default
+- Pastel
+- Bold
+- Safe
+- Grayscale
+- Viridis
+- Plasma
+- Cividis
+
+All palettes are available with light and dark chart backgrounds. The grayscale endpoints use near-white and near-black shades to retain practical contrast on both backgrounds.
+
 ## Adding a New Leaderboard Entry
 
-Create a PR adding a new entry into the `results/` folder. 
-Check out `results/qwen3-6-35b-nvfp4-claude-code.json`(./results/qwen3-6-35b-nvfp4-claude-code.json) for an example result.
+Create a pull request adding a new entry to the `results/` folder. See [`results/qwen3-6-35b-nvfp4-claude-code.json`](./results/qwen3-6-35b-nvfp4-claude-code.json) for an example result.
+
+Do not change the result JSON schema for token-efficiency analysis. The feature uses token and performance metrics already present in the existing result model.
 
 ## Development
 
-1. Install dependencies
+1. Install dependencies:
+
+   ```sh
+   pip install -r requirements.txt
+
+   # or
+
+   uv venv
+   uv pip install -r requirements.txt
+   ```
 
-    ```sh
-    pip install -r requirements.txt
+2. Run the app:
 
-    # or
+   ```sh
+   python app.py
+   ```
 
-    uv venv
-    uv pip install -r requirements.txt
-    ```
+3. Run tests:
 
-2. Run the app
-   
-    ```sh
-    python app.py
-    ```
+   ```sh
+   pytest
+   ```

app.py

@@ -51,19 +51,39 @@ from src.charts import (
     clean_markdown_link,
     create_leaderboard_benchmark_plot,
     create_score_vs_cost_plot,
+    create_score_vs_tokens_plot,
 )
 from src.display.text_blocks import (
     HOW_TO_USE_TEXT,
     INTRODUCTION_TEXT,
     LLM_BENCHMARKS_TEXT,
 )
-from src.leaderboard import get_benchmark_names, get_benchmark_run_df, get_score_vs_cost_df
+from src.leaderboard import (
+    ALL_BENCHMARKS,
+    TOKEN_METRICS,
+    get_analysis_df,
+    get_benchmark_names,
+    get_benchmark_run_df,
+    get_score_vs_cost_df,
+    get_token_efficiency_df,
+    get_token_efficiency_table_df,
+)
 
 REPO_ID = "taagarwa/coding-agent-leaderboard"
 TOKEN = os.environ.get("HF_TOKEN")
 API = HfApi(token=TOKEN)
 COLOR_BY_CHOICES = ["Model", "Harness"]
-COLOR_PALETTE_CHOICES = ["Citrus", "Okabe-Ito", "High contrast", "Rainbow"]
+EFFICIENCY_COLOR_BY_CHOICES = ["Model", "Harness", "Benchmark"]
+COLOR_PALETTE_CHOICES = [
+    "Citrus",
+    "Okabe-Ito",
+    "High contrast",
+    "Rainbow",
+    "Grayscale",
+    "Viridis",
+    "Plasma",
+    "Cividis",
+]
 DEFAULT_COLOR_PALETTE = "Citrus"
 PLOT_BACKGROUND_CHOICES = ["Dark", "White"]
 DEFAULT_PLOT_BACKGROUND = "Dark"
@@ -91,6 +111,7 @@ BENCHMARK_NAMES = get_benchmark_names()
 DEFAULT_BENCHMARK = BENCHMARK_NAMES[0] if BENCHMARK_NAMES else None
 BENCHMARK_RUN_DF = get_benchmark_run_df()
 SCORE_VS_COST_DF = get_score_vs_cost_df()
+ANALYSIS_DF = get_analysis_df()
 
 
 def render_leaderboard_benchmark_plot(
@@ -111,6 +132,7 @@ def render_leaderboard_benchmark_plot(
 def render_score_vs_cost_plot(
     benchmark_name,
     color_by,
+    show_labels=False,
     color_palette=DEFAULT_COLOR_PALETTE,
     plot_background=DEFAULT_PLOT_BACKGROUND,
 ):
@@ -118,11 +140,46 @@ def render_score_vs_cost_plot(
         SCORE_VS_COST_DF,
         benchmark_name=benchmark_name,
         color_by=color_by,
+        show_labels=show_labels,
         palette_name=color_palette,
         background_name=plot_background,
     )
 
 
+def render_token_efficiency(
+    benchmark_name,
+    token_metric,
+    color_by,
+    x_scale,
+    show_pareto_frontier,
+    show_labels,
+    color_palette=DEFAULT_COLOR_PALETTE,
+    plot_background=DEFAULT_PLOT_BACKGROUND,
+):
+    plot_df = get_token_efficiency_df(
+        benchmark_name=benchmark_name,
+        token_metric=token_metric,
+        analysis_df=ANALYSIS_DF,
+    )
+    table_df = get_token_efficiency_table_df(
+        benchmark_name=benchmark_name,
+        analysis_df=ANALYSIS_DF,
+    )
+    exclusion_count = plot_df.attrs.get("exclusion_count", 0)
+    note = f"{exclusion_count} runs excluded because {token_metric.lower()} was missing or non-positive."
+    figure = create_score_vs_tokens_plot(
+        plot_df,
+        token_metric=token_metric,
+        color_by=color_by,
+        x_scale=x_scale,
+        show_pareto_frontier=show_pareto_frontier,
+        show_labels=show_labels,
+        palette_name=color_palette,
+        background_name=plot_background,
+    )
+    return figure, table_df, note
+
+
 def build_header_html(df):
     n_results = len(df)
     n_models = df["Model"].nunique()
@@ -270,37 +327,117 @@ with demo:
                     label="Image background",
                     elem_classes="color-control",
                 )
+                cost_labels = gr.Checkbox(
+                    value=False,
+                    label="Show point labels",
+                    elem_classes="color-control",
+                )
             score_vs_cost_plot = gr.Plot(
                 value=render_score_vs_cost_plot(
                     DEFAULT_BENCHMARK,
                     "Model",
+                    False,
                     DEFAULT_COLOR_PALETTE,
                     DEFAULT_PLOT_BACKGROUND,
                 ),
                 show_label=False,
                 elem_classes="responsive-plot",
             )
-            cost_benchmark.change(
-                fn=render_score_vs_cost_plot,
-                inputs=[cost_benchmark, cost_color_by, cost_palette, cost_background],
-                outputs=score_vs_cost_plot,
+            cost_controls = [
+                cost_benchmark,
+                cost_color_by,
+                cost_labels,
+                cost_palette,
+                cost_background,
+            ]
+            for control in cost_controls:
+                control.change(
+                    fn=render_score_vs_cost_plot,
+                    inputs=cost_controls,
+                    outputs=score_vs_cost_plot,
+                )
+
+        with gr.Tab("⚡ Efficiency"):
+            gr.Markdown(
+                "### Token Efficiency\n"
+                "Score alone does not show how much token budget an agent uses to achieve it. "
+                "Tokens Per Solved Task divides mean tokens per task by the fractional score. "
+                "The dashed Pareto frontier connects runs for which no other run uses fewer or equal tokens "
+                "while achieving an equal or higher score; points below and to the right are dominated. "
+                "The dotted 70% and 80% horizontal lines are fixed capability guides only—they do not affect "
+                "Pareto membership and are not statistical thresholds."
             )
-            cost_color_by.change(
-                fn=render_score_vs_cost_plot,
-                inputs=[cost_benchmark, cost_color_by, cost_palette, cost_background],
-                outputs=score_vs_cost_plot,
+            with gr.Row():
+                efficiency_benchmark = gr.Dropdown(
+                    choices=[ALL_BENCHMARKS, *BENCHMARK_NAMES],
+                    value=ALL_BENCHMARKS,
+                    label="Benchmark",
+                )
+                efficiency_metric = gr.Dropdown(
+                    choices=list(TOKEN_METRICS),
+                    value="Tokens Per Solved Task",
+                    label="Token metric",
+                )
+                efficiency_color_by = gr.Radio(
+                    choices=EFFICIENCY_COLOR_BY_CHOICES,
+                    value="Model",
+                    label="Color by",
+                )
+                efficiency_scale = gr.Radio(choices=["Log", "Linear"], value="Log", label="X-axis scale")
+            with gr.Row():
+                efficiency_pareto = gr.Checkbox(value=True, label="Show Pareto frontier")
+                efficiency_labels = gr.Checkbox(value=False, label="Show point labels")
+                efficiency_palette = gr.Dropdown(
+                    choices=COLOR_PALETTE_CHOICES,
+                    value=DEFAULT_COLOR_PALETTE,
+                    label="Color palette",
+                )
+                efficiency_background = gr.Dropdown(
+                    choices=PLOT_BACKGROUND_CHOICES,
+                    value=DEFAULT_PLOT_BACKGROUND,
+                    label="Image background",
+                )
+
+            initial_efficiency = render_token_efficiency(
+                ALL_BENCHMARKS,
+                "Tokens Per Solved Task",
+                "Model",
+                "Log",
+                True,
+                False,
+                DEFAULT_COLOR_PALETTE,
+                DEFAULT_PLOT_BACKGROUND,
             )
-            cost_palette.change(
-                fn=render_score_vs_cost_plot,
-                inputs=[cost_benchmark, cost_color_by, cost_palette, cost_background],
-                outputs=score_vs_cost_plot,
+            efficiency_note = gr.Markdown(initial_efficiency[2])
+            efficiency_plot = gr.Plot(
+                value=initial_efficiency[0],
+                show_label=False,
+                elem_classes="responsive-plot",
             )
-            cost_background.change(
-                fn=render_score_vs_cost_plot,
-                inputs=[cost_benchmark, cost_color_by, cost_palette, cost_background],
-                outputs=score_vs_cost_plot,
+            gr.Markdown("#### Token-efficiency ranking")
+            efficiency_table = gr.Dataframe(
+                value=initial_efficiency[1],
+                interactive=False,
+                show_label=False,
             )
 
+            efficiency_controls = [
+                efficiency_benchmark,
+                efficiency_metric,
+                efficiency_color_by,
+                efficiency_scale,
+                efficiency_pareto,
+                efficiency_labels,
+                efficiency_palette,
+                efficiency_background,
+            ]
+            for control in efficiency_controls:
+                control.change(
+                    fn=render_token_efficiency,
+                    inputs=efficiency_controls,
+                    outputs=[efficiency_plot, efficiency_table, efficiency_note],
+                )
+
         with gr.Tab("🏃 Benchmark Runs"):
             benchmark_runs = init_benchmark_runs(BENCHMARK_RUN_DF)
 

src/charts.py

@@ -5,11 +5,21 @@ import re
 from typing import Literal
 
 import pandas as pd
+import plotly.colors as pc
 import plotly.graph_objects as go
 from plotly.graph_objs._figure import Figure
 
 ColorBy = Literal["Model", "Harness"]
-PaletteName = Literal["Citrus", "Okabe-Ito", "High contrast", "Rainbow"]
+PaletteName = Literal[
+    "Citrus",
+    "Okabe-Ito",
+    "High contrast",
+    "Rainbow",
+    "Grayscale",
+    "Viridis",
+    "Plasma",
+    "Cividis",
+]
 PlotBackground = Literal["Dark", "White"]
 DEFAULT_PALETTE: PaletteName = "Citrus"
 DEFAULT_BACKGROUND: PlotBackground = "Dark"
@@ -92,6 +102,32 @@ PLOT_BACKGROUNDS: dict[PlotBackground, dict[str, str]] = {
 }
 
 
+
+CAPABILITY_TIER_BOUNDARIES = (70.0, 80.0)
+
+
+def add_capability_tier_guides(fig: Figure, background_name: str | None = DEFAULT_BACKGROUND) -> None:
+    """Add fixed score-band guides without affecting Pareto calculations."""
+    theme = get_plot_background(background_name)
+    for boundary in CAPABILITY_TIER_BOUNDARIES:
+        fig.add_hline(
+            y=boundary,
+            line={"width": 1.5, "dash": "dot", "color": theme["zero_line_color"]},
+            layer="below",
+        )
+        fig.add_annotation(
+            x=1.0,
+            xref="paper",
+            xanchor="right",
+            y=boundary,
+            yref="y",
+            yshift=8,
+            text=f"{boundary:.0f}% guide",
+            showarrow=False,
+            font={"size": 11, "color": theme["text_muted"]},
+        )
+
+
 def clean_markdown_link(value: object) -> str:
     """Return human-readable text from Markdown links used in leaderboard tables."""
     text = str(value).replace("<sup>*</sup>", "")
@@ -101,25 +137,40 @@ def clean_markdown_link(value: object) -> str:
     return text
 
 
-MODEL_PALETTES: dict[PaletteName, list[str]] = {
+COLOR_PALETTES: dict[PaletteName, list[str]] = {
     "Citrus": MODEL_FALLBACK_PALETTE,
-    # Full Okabe-Ito palette. It is categorical and colorblind-friendly.
-    "Okabe-Ito": ["#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7", "#999999"],
-    # High-contrast colors are intentionally broad, not just orange/yellow variants.
+    "Okabe-Ito": [
+        "#E69F00",
+        "#56B4E9",
+        "#009E73",
+        "#F0E442",
+        "#0072B2",
+        "#D55E00",
+        "#CC79A7",
+        "#999999",
+    ],
     "High contrast": ["#FFD166", "#06D6A0", "#118AB2", "#EF476F", "#A78BFA", "#F97316", "#22D3EE", "#E5E7EB"],
-    # Categorical rainbow-style palette. This is not a continuous colorscale; it is
-    # sampled as discrete colors so each category gets a distinct color.
     "Rainbow": ["#E6194B", "#F58231", "#FFE119", "#3CB44B", "#42D4F4", "#4363D8", "#911EB4", "#F032E6", "#469990", "#9A6324"],
+    # Near-white and near-black endpoints remain visible against both supported backgrounds.
+    "Grayscale": ["#E2E8F0", "#CBD5E1", "#94A3B8", "#64748B", "#475569", "#334155", "#1E293B", "#111827"],
+    "Viridis": list(pc.sequential.Viridis),
+    "Plasma": list(pc.sequential.Plasma),
+    "Cividis": list(pc.sequential.Cividis),
 }
 
+# Harness categories use the same central registry, with the default palette retaining
+# its established cyan/blue/violet identity.
 HARNESS_PALETTES: dict[PaletteName, list[str]] = {
+    **COLOR_PALETTES,
     "Citrus": HARNESS_FALLBACK_PALETTE,
-    # Use the same broad Okabe-Ito family but with a different starting point so the
-    # harness mode does not visually mirror the model mode.
-    "Okabe-Ito": ["#0072B2", "#D55E00", "#CC79A7", "#009E73", "#56B4E9", "#E69F00", "#F0E442", "#999999"],
-    "High contrast": ["#38BDF8", "#34D399", "#A78BFA", "#F472B6", "#FACC15", "#FB923C", "#22D3EE", "#E5E7EB"],
-    "Rainbow": ["#4363D8", "#E6194B", "#3CB44B", "#F58231", "#911EB4", "#42D4F4", "#F032E6", "#FFE119", "#469990", "#9A6324"],
 }
+MODEL_PALETTES = COLOR_PALETTES
+
+
+def get_color_palette(name: str | None) -> list[str]:
+    """Return a copy of the requested palette, falling back to Citrus."""
+    palette_name = normalize_palette_name(name)
+    return list(COLOR_PALETTES[palette_name])
 
 
 def normalize_palette_name(palette_name: str | None) -> PaletteName:
@@ -160,7 +211,7 @@ def get_color(name: str, color_by: ColorBy, palette_name: str | None = DEFAULT_P
 def palette_colors_for(color_by: ColorBy, palette_name: str | None = DEFAULT_PALETTE) -> list[str]:
     palette_key = normalize_palette_name(palette_name)
     palettes = MODEL_PALETTES if color_by == "Model" else HARNESS_PALETTES
-    return palettes[palette_key]
+    return list(palettes[palette_key])
 
 
 def color_map_for(
@@ -261,7 +312,9 @@ def prepare_benchmark_run_plot_df(dataframe: pd.DataFrame) -> pd.DataFrame:
     plot_df["Model Label"] = plot_df["Model"].map(clean_markdown_link)
     plot_df["Harness Label"] = plot_df["Harness"].map(clean_markdown_link)
     plot_df["Benchmark Label"] = plot_df["Benchmark"].map(clean_markdown_link)
-    plot_df["Run Label"] = plot_df["Model Label"] + "<br>" + plot_df["Harness Label"]
+    plot_df["Run Label"] = (
+        plot_df["Model Label"] + " / " + plot_df["Harness Label"]
+    )
     plot_df["Score"] = pd.to_numeric(plot_df["Score"], errors="coerce")
     return plot_df
 
@@ -324,10 +377,24 @@ def create_leaderboard_benchmark_plot(
     return fig
 
 
+def scatter_label_kwargs(
+    dataframe: pd.DataFrame,
+    show_labels: bool,
+    preferred_columns: tuple[str, ...] = ("Run Label", "Label"),
+) -> dict[str, object]:
+    """Return consistent Plotly scatter label arguments without affecting hover data."""
+    if not show_labels:
+        return {"mode": "markers", "text": None, "textposition": "top center"}
+    label_column = next((column for column in preferred_columns if column in dataframe.columns), None)
+    labels = dataframe[label_column] if label_column else None
+    return {"mode": "markers+text", "text": labels, "textposition": "top center"}
+
+
 def create_score_vs_cost_plot(
     dataframe: pd.DataFrame,
     benchmark_name: str | None,
     color_by: ColorBy = "Model",
+    show_labels: bool = False,
     palette_name: str | None = DEFAULT_PALETTE,
     background_name: str | None = DEFAULT_BACKGROUND,
 ) -> Figure:
@@ -351,14 +418,13 @@ def create_score_vs_cost_plot(
     fig = go.Figure()
 
     for group, group_df in plot_df.groupby(color_by, sort=True):
+        label_kwargs = scatter_label_kwargs(group_df, show_labels)
         fig.add_trace(
             go.Scatter(
                 x=group_df["Cost Per Task (USD)"],
                 y=group_df["Score"],
-                mode="markers+text",
                 name=str(group),
-                text=group_df["Label"],
-                textposition="top center",
+                **label_kwargs,
                 marker={
                     "size": 15,
                     "color": colors[str(group)],
@@ -383,3 +449,141 @@ def create_score_vs_cost_plot(
         legend_title_text=color_by,
     )
     return apply_plot_theme(fig, background_name)
+
+
+def create_score_vs_tokens_plot(
+    dataframe: pd.DataFrame,
+    token_metric: str = "Tokens Per Solved Task",
+    color_by: Literal["Model", "Harness", "Benchmark"] = "Model",
+    x_scale: Literal["Linear", "Log"] = "Log",
+    show_pareto_frontier: bool = True,
+    show_labels: bool = False,
+    palette_name: str | None = DEFAULT_PALETTE,
+    background_name: str | None = DEFAULT_BACKGROUND,
+) -> Figure:
+    """Plot score against a positive token metric, optionally with its Pareto frontier."""
+    from src.leaderboard import get_token_pareto_frontier_df
+
+    if dataframe is None or dataframe.empty:
+        return empty_figure("No valid token data available.", background_name)
+    if token_metric not in dataframe.columns:
+        return empty_figure(f"Token metric not available: {token_metric}.", background_name)
+    if color_by not in dataframe.columns:
+        return empty_figure(f"Color dimension not available: {color_by}.", background_name)
+
+    plot_df = dataframe.copy()
+    plot_df[token_metric] = pd.to_numeric(plot_df[token_metric], errors="coerce")
+    plot_df["Score (%)"] = pd.to_numeric(plot_df["Score (%)"], errors="coerce")
+    plot_df = plot_df.dropna(subset=[token_metric, "Score (%)"])
+    plot_df = plot_df[plot_df[token_metric] > 0]
+    if plot_df.empty:
+        return empty_figure("No valid token data available.", background_name)
+
+    palette_color_by: ColorBy = color_by if color_by in ("Model", "Harness") else "Model"
+    colors = color_map_for(plot_df[color_by], palette_color_by, palette_name)
+    theme = get_plot_background(background_name)
+    fig = go.Figure()
+    hover_columns = [
+        "Model",
+        "Harness",
+        "Benchmark",
+        "Score (%)",
+        "Input Tokens Per Task",
+        "Output Tokens Per Task",
+        "Cache Tokens Per Task",
+        "Total Tokens Per Task",
+        "Cost Per Task",
+        "Total Time Per Task",
+        "Agent Time Per Task",
+    ]
+    for column in hover_columns:
+        if column not in plot_df:
+            plot_df[column] = None
+
+    for group, group_df in plot_df.groupby(color_by, sort=True):
+        label_kwargs = scatter_label_kwargs(group_df, show_labels)
+        fig.add_trace(
+            go.Scatter(
+                x=group_df[token_metric],
+                y=group_df["Score (%)"],
+                name=str(group),
+                **label_kwargs,
+                marker={
+                    "size": 13,
+                    "color": colors[str(group)],
+                    "line": {"width": 1, "color": theme["marker_line_color"]},
+                },
+                customdata=group_df[hover_columns],
+                hovertemplate=(
+                    "<b>%{customdata[0]}</b><br>"
+                    "Harness: %{customdata[1]}<br>"
+                    "Benchmark: %{customdata[2]}<br>"
+                    "Score: %{customdata[3]:.1f}%<br>"
+                    "Input tokens/task: %{customdata[4]:,.0f}<br>"
+                    "Output tokens/task: %{customdata[5]:,.0f}<br>"
+                    "Cache tokens/task: %{customdata[6]:,.0f}<br>"
+                    "Total tokens/task: %{customdata[7]:,.0f}<br>"
+                    "Cost/task: $%{customdata[8]:.4f}<br>"
+                    "Total time/task: %{customdata[9]:,.0f}s<br>"
+                    "Agent time/task: %{customdata[10]:,.0f}s"
+                    "<extra></extra>"
+                ),
+            )
+        )
+
+    if show_pareto_frontier:
+        frontier_df = get_token_pareto_frontier_df(plot_df, token_metric)
+        if not frontier_df.empty:
+            fig.add_trace(
+                go.Scatter(
+                    x=frontier_df[token_metric],
+                    y=frontier_df["Score (%)"],
+                    mode="lines+markers",
+                    name="Pareto frontier",
+                    line={"width": 3, "dash": "dash", "color": theme["text_primary"]},
+                    marker={
+                        "size": 10,
+                        "symbol": "diamond-open",
+                        "color": theme["text_primary"],
+                        "line": {"width": 2, "color": theme["text_primary"]},
+                    },
+                    customdata=frontier_df[["Run Label"]],
+                    hovertemplate=(
+                        "<b>Pareto frontier</b><br>"
+                        "%{customdata[0]}<br>"
+                        f"{token_metric}: %{{x:,.0f}}<br>"
+                        "Score: %{y:.1f}%<extra></extra>"
+                    ),
+                )
+            )
+
+    add_capability_tier_guides(fig, background_name)
+
+    fig.update_layout(
+        title=None,
+        xaxis={"title": token_metric, "type": "log" if x_scale == "Log" else "linear"},
+        yaxis={"title": "Score (%)", "range": [0, 105]},
+        legend_title_text=color_by,
+    )
+    return apply_plot_theme(fig, background_name)
+
+
+def create_token_pareto_frontier_plot(
+    dataframe: pd.DataFrame,
+    token_metric: str = "Tokens Per Solved Task",
+    color_by: Literal["Model", "Harness", "Benchmark"] = "Model",
+    x_scale: Literal["Linear", "Log"] = "Log",
+    show_labels: bool = False,
+    palette_name: str | None = DEFAULT_PALETTE,
+    background_name: str | None = DEFAULT_BACKGROUND,
+) -> Figure:
+    return create_score_vs_tokens_plot(
+        dataframe=dataframe,
+        token_metric=token_metric,
+        color_by=color_by,
+        x_scale=x_scale,
+        show_pareto_frontier=True,
+        show_labels=show_labels,
+        palette_name=palette_name,
+        background_name=background_name,
+    )

src/leaderboard.py

@@ -1,5 +1,5 @@
-from pathlib import Path
 import json
+from pathlib import Path
 
 import pandas as pd
 
@@ -12,6 +12,47 @@ BENCHMARK_SORT_ORDER = {
     "SWE-Bench Pro -- Ansible": 1,
 }
 
+ALL_BENCHMARKS = "All benchmarks"
+TOKEN_METRICS = (
+    "Total Tokens Per Task",
+    "Input Tokens Per Task",
+    "Output Tokens Per Task",
+    "Tokens Per Solved Task",
+)
+ANALYSIS_COLUMNS = [
+    "Benchmark",
+    "Model",
+    "Harness",
+    "Run Label",
+    "Category",
+    "Score",
+    "Score (%)",
+    "Tasks",
+    "Errors",
+    "Input Tokens Per Task",
+    "Cache Tokens Per Task",
+    "Output Tokens Per Task",
+    "Total Tokens Per Task",
+    "Tokens Per Solved Task",
+    "Cost Per Task",
+    "Total Time Per Task",
+    "Agent Time Per Task",
+    "Token Data Available",
+]
+TOKEN_EFFICIENCY_TABLE_COLUMNS = [
+    "Benchmark",
+    "Model",
+    "Harness",
+    "Score (%)",
+    "Total Tokens Per Task",
+    "Input Tokens Per Task",
+    "Output Tokens Per Task",
+    "Cache Tokens Per Task",
+    "Tokens Per Solved Task",
+    "Tasks",
+    "Errors",
+]
+
 
 def benchmark_sort_key(name: str) -> tuple[int, str]:
     return (BENCHMARK_SORT_ORDER.get(name, 99), name)
@@ -40,6 +81,136 @@ def get_benchmark_names(results: list[Result] | None = None) -> list[str]:
     return sorted({r.benchmark.name for r in results}, key=benchmark_sort_key)
 
 
+def get_analysis_df(results: list[Result] | None = None) -> pd.DataFrame:
+    """Return one normalized analysis row per benchmark result."""
+    if results is None:
+        results = get_results()
+
+    rows = []
+    for result in results:
+        metrics = result.metrics
+        model_label = result.model.repo or result.model.name
+        total_tokens = metrics.mean_tokens_per_task
+        token_data_available = total_tokens is not None and total_tokens > 0
+        tokens_per_solved_task = (
+            total_tokens / metrics.score
+            if token_data_available and metrics.score > 0
+            else None
+        )
+        rows.append(
+            {
+                "Benchmark": result.benchmark.name,
+                "Model": model_label,
+                "Harness": result.harness.name,
+                "Run Label": f"{model_label} / {result.harness.name}",
+                "Category": "FOSS" if result.model.is_oss and result.harness.is_oss else "Proprietary",
+                "Score": metrics.score,
+                "Score (%)": metrics.score * 100,
+                "Tasks": metrics.n_tasks,
+                "Errors": metrics.n_errors,
+                "Input Tokens Per Task": metrics.mean_input_tokens_per_task,
+                "Cache Tokens Per Task": metrics.mean_cache_tokens_per_task,
+                "Output Tokens Per Task": metrics.mean_output_tokens_per_task,
+                "Total Tokens Per Task": total_tokens,
+                "Tokens Per Solved Task": tokens_per_solved_task,
+                "Cost Per Task": metrics.mean_cost_usd_per_task,
+                "Total Time Per Task": metrics.mean_total_time_seconds_per_task,
+                "Agent Time Per Task": metrics.mean_agent_time_seconds_per_task,
+                "Token Data Available": token_data_available,
+            }
+        )
+
+    return pd.DataFrame(rows, columns=ANALYSIS_COLUMNS)
+
+
+def get_token_efficiency_df(
+    benchmark_name: str | None = ALL_BENCHMARKS,
+    token_metric: str = "Tokens Per Solved Task",
+    analysis_df: pd.DataFrame | None = None,
+) -> pd.DataFrame:
+    """Filter normalized analysis data for a selected positive token metric.
+
+    The number of rows excluded because the selected metric is missing or
+    non-positive is available as ``dataframe.attrs["exclusion_count"]``.
+    """
+    if token_metric not in TOKEN_METRICS:
+        raise ValueError(f"Unsupported token metric: {token_metric}")
+
+    dataframe = get_analysis_df() if analysis_df is None else analysis_df.copy()
+    if benchmark_name and benchmark_name != ALL_BENCHMARKS:
+        dataframe = dataframe[dataframe["Benchmark"] == benchmark_name].copy()
+
+    if dataframe.empty:
+        dataframe.attrs["exclusion_count"] = 0
+        return dataframe
+
+    metric_values = pd.to_numeric(dataframe[token_metric], errors="coerce")
+    valid_mask = metric_values.notna() & (metric_values > 0)
+    exclusion_count = int((~valid_mask).sum())
+    dataframe = dataframe.loc[valid_mask].copy()
+    dataframe[token_metric] = metric_values.loc[valid_mask]
+    dataframe = dataframe.sort_values(
+        ["Tokens Per Solved Task", "Score (%)", "Benchmark", "Model", "Harness"],
+        ascending=[True, False, True, True, True],
+        na_position="last",
+    )
+    dataframe.attrs["exclusion_count"] = exclusion_count
+    return dataframe
+
+
+def get_token_efficiency_table_df(
+    benchmark_name: str | None = ALL_BENCHMARKS,
+    analysis_df: pd.DataFrame | None = None,
+) -> pd.DataFrame:
+    dataframe = get_token_efficiency_df(
+        benchmark_name=benchmark_name,
+        token_metric="Tokens Per Solved Task",
+        analysis_df=analysis_df,
+    )
+    table_df = dataframe.reindex(columns=TOKEN_EFFICIENCY_TABLE_COLUMNS).copy()
+    table_df.attrs.update(dataframe.attrs)
+    return table_df
+
+
+def get_token_pareto_frontier_df(
+    dataframe: pd.DataFrame,
+    token_metric: str,
+    score_column: str = "Score (%)",
+) -> pd.DataFrame:
+    """Return deterministic non-dominated points for lower tokens/higher score."""
+    if token_metric not in dataframe.columns or score_column not in dataframe.columns:
+        return dataframe.iloc[0:0].copy()
+
+    candidates = dataframe.copy()
+    candidates[token_metric] = pd.to_numeric(candidates[token_metric], errors="coerce")
+    candidates[score_column] = pd.to_numeric(candidates[score_column], errors="coerce")
+    candidates = candidates.dropna(subset=[token_metric, score_column])
+    candidates = candidates[candidates[token_metric] > 0]
+    if candidates.empty:
+        return candidates
+
+    tie_breakers = [column for column in ("Benchmark", "Model", "Harness", "Run Label") if column in candidates]
+    candidates = candidates.sort_values(
+        [token_metric, score_column, *tie_breakers],
+        ascending=[True, False, *([True] * len(tie_breakers))],
+        kind="mergesort",
+    )
+
+    frontier_indices: list[object] = []
+    best_score_at_lower_tokens = float("-inf")
+    for _, token_group in candidates.groupby(token_metric, sort=True):
+        group_best_score = float(token_group[score_column].max())
+        if group_best_score > best_score_at_lower_tokens:
+            frontier_indices.extend(token_group[token_group[score_column] == group_best_score].index.tolist())
+            best_score_at_lower_tokens = group_best_score
+
+    return candidates.loc[frontier_indices].sort_values(
+        [token_metric, score_column, *tie_breakers],
+        ascending=[True, False, *([True] * len(tie_breakers))],
+        kind="mergesort",
+    )
+
+
 def get_benchmark_run_df():
     results = get_results()
 
@@ -81,43 +252,13 @@ def get_benchmark_run_df():
 
 
 def get_score_vs_cost_df():
-    results = get_results()
-
-    rows = []
-    for result in results:
-        mean_cost = result.metrics.mean_cost_usd_per_task
-        if mean_cost is None:
-            continue
-
-        model_label = result.model.repo or result.model.name
-        harness_label = result.harness.name
-
-        rows.append(
-            {
-                "Label": f"{model_label} / {harness_label}",
-                "Model": model_label,
-                "Harness": harness_label,
-                "Benchmark": result.benchmark.name,
-                "Category": "FOSS" if result.model.is_oss and result.harness.is_oss else "Proprietary",
-                "Score": round(result.metrics.score * 100, 1),
-                "Cost Per Task (USD)": round(mean_cost, 2),
-            }
-        )
-
-    columns = [
-        "Label",
-        "Model",
-        "Harness",
-        "Benchmark",
-        "Category",
-        "Score",
-        "Cost Per Task (USD)",
-    ]
-    score_vs_cost_df = pd.DataFrame(rows, columns=columns)
+    analysis_df = get_analysis_df()
+    score_vs_cost_df = analysis_df.dropna(subset=["Cost Per Task"]).copy()
+    columns = ["Label", "Model", "Harness", "Benchmark", "Category", "Score", "Cost Per Task (USD)"]
     if score_vs_cost_df.empty:
-        return score_vs_cost_df
+        return pd.DataFrame(columns=columns)
 
-    return score_vs_cost_df.sort_values(
-        ["Benchmark", "Score"],
-        ascending=[True, False],
-    )
+    score_vs_cost_df["Label"] = score_vs_cost_df["Run Label"]
+    score_vs_cost_df["Score"] = score_vs_cost_df["Score (%)"].round(1)
+    score_vs_cost_df["Cost Per Task (USD)"] = score_vs_cost_df["Cost Per Task"].round(2)
+    return score_vs_cost_df[columns].sort_values(["Benchmark", "Score"], ascending=[True, False])

tests/test_token_efficiency.py

@@ -0,0 +1,241 @@
+import pandas as pd
+import plotly.graph_objects as go
+
+from src.charts import create_score_vs_tokens_plot, create_token_pareto_frontier_plot
+from src.leaderboard import (
+    ALL_BENCHMARKS,
+    ANALYSIS_COLUMNS,
+    get_analysis_df,
+    get_token_efficiency_df,
+    get_token_pareto_frontier_df,
+)
+from src.models import Benchmark, Environment, Harness, Metrics, Model, Result
+
+
+def make_result(
+    *,
+    benchmark: str = "Benchmark A",
+    model: str = "model-a",
+    harness: str = "harness-a",
+    score: float = 0.5,
+    total_tokens: int | None = 100,
+    input_tokens: int | None = 60,
+    cache_tokens: int | None = 10,
+    output_tokens: int | None = 30,
+) -> Result:
+    return Result(
+        benchmark=Benchmark(name=benchmark, repo="repo", num_tasks=10, url="https://example.com/benchmark"),
+        harness=Harness(name=harness, skills=[], is_oss=True, url="https://example.com/harness"),
+        model=Model(
+            name=model,
+            repo=None,
+            is_oss=True,
+            num_params=1,
+            precision="fp16",
+            url="https://example.com/model",
+        ),
+        environment=Environment(name="env", url="https://example.com/env"),
+        metrics=Metrics(
+            score=score,
+            n_tasks=10,
+            n_errors=1,
+            mean_input_tokens_per_task=input_tokens,
+            mean_cache_tokens_per_task=cache_tokens,
+            mean_output_tokens_per_task=output_tokens,
+            mean_tokens_per_task=total_tokens,
+            mean_cost_usd_per_task=0.25,
+            mean_total_time_seconds_per_task=12,
+            mean_agent_time_seconds_per_task=10,
+        ),
+    )
+
+
+def test_get_analysis_df_columns_and_derived_values():
+    dataframe = get_analysis_df([make_result(score=0.25, total_tokens=200)])
+
+    assert set(ANALYSIS_COLUMNS).issubset(dataframe.columns)
+    assert dataframe.loc[0, "Score (%)"] == 25
+    assert dataframe.loc[0, "Tokens Per Solved Task"] == 800
+    assert bool(dataframe.loc[0, "Token Data Available"]) is True
+
+
+def test_missing_zero_and_negative_tokens_are_unavailable():
+    dataframe = get_analysis_df(
+        [
+            make_result(model="missing", total_tokens=None),
+            make_result(model="zero", total_tokens=0),
+            make_result(model="negative", total_tokens=-10),
+        ]
+    )
+
+    assert dataframe["Token Data Available"].tolist() == [False, False, False]
+    assert dataframe["Tokens Per Solved Task"].isna().all()
+
+
+def test_zero_score_does_not_divide_by_zero():
+    dataframe = get_analysis_df([make_result(score=0, total_tokens=100)])
+
+    assert pd.isna(dataframe.loc[0, "Tokens Per Solved Task"])
+    assert bool(dataframe.loc[0, "Token Data Available"]) is True
+
+
+def test_token_efficiency_filtering_and_exclusion_count():
+    analysis_df = get_analysis_df(
+        [
+            make_result(model="valid", total_tokens=100),
+            make_result(model="zero", total_tokens=0),
+            make_result(model="missing", total_tokens=None),
+        ]
+    )
+
+    filtered = get_token_efficiency_df(
+        token_metric="Total Tokens Per Task",
+        analysis_df=analysis_df,
+    )
+
+    assert filtered["Model"].tolist() == ["valid"]
+    assert filtered.attrs["exclusion_count"] == 2
+
+
+def test_all_benchmarks_and_single_benchmark_filtering():
+    analysis_df = get_analysis_df(
+        [
+            make_result(benchmark="Benchmark A", model="a"),
+            make_result(benchmark="Benchmark B", model="b"),
+        ]
+    )
+
+    all_rows = get_token_efficiency_df(ALL_BENCHMARKS, analysis_df=analysis_df)
+    one_benchmark = get_token_efficiency_df("Benchmark B", analysis_df=analysis_df)
+
+    assert set(all_rows["Benchmark"]) == {"Benchmark A", "Benchmark B"}
+    assert one_benchmark["Benchmark"].tolist() == ["Benchmark B"]
+
+
+def test_pareto_frontier_identification():
+    dataframe = pd.DataFrame(
+        {
+            "Run Label": ["a", "b", "c", "d"],
+            "Total Tokens Per Task": [100, 200, 300, 400],
+            "Score (%)": [50, 60, 55, 80],
+        }
+    )
+
+    frontier = get_token_pareto_frontier_df(dataframe, "Total Tokens Per Task")
+
+    assert frontier["Run Label"].tolist() == ["a", "b", "d"]
+
+
+def test_pareto_ties_are_deterministic_and_not_falsely_dominated():
+    dataframe = pd.DataFrame(
+        {
+            "Run Label": ["z", "a", "dominated", "higher"],
+            "Model": ["z", "a", "d", "h"],
+            "Total Tokens Per Task": [100, 100, 100, 200],
+            "Score (%)": [50, 50, 40, 60],
+        }
+    )
+
+    frontier = get_token_pareto_frontier_df(dataframe, "Total Tokens Per Task")
+
+    assert frontier["Run Label"].tolist() == ["a", "z", "higher"]
+    assert "dominated" not in frontier["Run Label"].tolist()
+
+
+def test_chart_functions_return_figures_for_valid_data():
+    dataframe = get_analysis_df([make_result(), make_result(model="model-b", score=0.7, total_tokens=200)])
+
+    scatter = create_score_vs_tokens_plot(dataframe, token_metric="Total Tokens Per Task")
+    pareto = create_token_pareto_frontier_plot(dataframe, token_metric="Total Tokens Per Task")
+
+    assert isinstance(scatter, go.Figure)
+    assert isinstance(pareto, go.Figure)
+    assert scatter.layout.xaxis.type == "log"
+
+
+def test_empty_and_fully_invalid_chart_data_are_graceful():
+    empty = create_score_vs_tokens_plot(pd.DataFrame(), token_metric="Total Tokens Per Task")
+    invalid_df = get_analysis_df([make_result(total_tokens=0)])
+    invalid = create_score_vs_tokens_plot(invalid_df, token_metric="Total Tokens Per Task")
+
+    assert isinstance(empty, go.Figure)
+    assert isinstance(invalid, go.Figure)
+    assert len(empty.layout.annotations) == 1
+    assert len(invalid.layout.annotations) == 1
+
+
+def test_palette_lookup_new_palettes_fallback_and_copy():
+    from src.charts import COLOR_PALETTES, get_color_palette
+
+    for palette_name in ("Grayscale", "Viridis", "Plasma", "Cividis"):
+        assert get_color_palette(palette_name) == COLOR_PALETTES[palette_name]
+        assert get_color_palette(palette_name) is not COLOR_PALETTES[palette_name]
+
+    fallback = get_color_palette("unknown")
+    assert fallback == COLOR_PALETTES["Citrus"]
+    fallback.append("#000000")
+    assert "#000000" not in COLOR_PALETTES["Citrus"]
+
+
+def test_cost_and_token_scatter_labels_toggle_consistently():
+    from src.charts import create_score_vs_cost_plot
+
+    cost_df = pd.DataFrame(
+        {
+            "Benchmark": ["Benchmark A"],
+            "Model": ["model-a"],
+            "Harness": ["harness-a"],
+            "Score": [50.0],
+            "Cost Per Task (USD)": [0.25],
+            "Label": ["model-a<br>harness-a"],
+        }
+    )
+    token_df = get_analysis_df([make_result()])
+
+    cost_without = create_score_vs_cost_plot(cost_df, "Benchmark A", show_labels=False)
+    cost_with = create_score_vs_cost_plot(cost_df, "Benchmark A", show_labels=True)
+    token_without = create_score_vs_tokens_plot(token_df, show_labels=False, show_pareto_frontier=False)
+    token_with = create_score_vs_tokens_plot(token_df, show_labels=True, show_pareto_frontier=False)
+
+    assert cost_without.data[0].mode == "markers"
+    assert cost_with.data[0].mode == "markers+text"
+    assert list(cost_with.data[0].text) == ["model-a<br>harness-a"]
+    assert token_without.data[0].mode == "markers"
+    assert token_with.data[0].mode == "markers+text"
+    assert list(token_with.data[0].text) == ["model-a / harness-a"]
+    assert cost_without.data[0].hovertemplate == cost_with.data[0].hovertemplate
+    assert token_without.data[0].hovertemplate == token_with.data[0].hovertemplate
+
+
+def test_label_toggle_handles_empty_data():
+    from src.charts import create_score_vs_cost_plot
+
+    cost = create_score_vs_cost_plot(pd.DataFrame(), "Benchmark A", show_labels=True)
+    tokens = create_score_vs_tokens_plot(pd.DataFrame(), show_labels=True)
+
+    assert isinstance(cost, go.Figure)
+    assert isinstance(tokens, go.Figure)
+    assert len(cost.layout.annotations) == 1
+    assert len(tokens.layout.annotations) == 1
+
+
+def test_score_vs_tokens_plot_adds_capability_guide_lines():
+    dataframe = pd.DataFrame(
+        {
+            "Benchmark": ["bench"],
+            "Model": ["model"],
+            "Harness": ["harness"],
+            "Run Label": ["model / harness"],
+            "Score (%)": [75.0],
+            "Tokens Per Solved Task": [1000.0],
+        }
+    )
+
+    figure = create_score_vs_tokens_plot(dataframe, show_pareto_frontier=False)
+
+    horizontal_lines = [
+        shape
+        for shape in figure.layout.shapes
+        if shape.type == "line" and shape.y0 == shape.y1
+    ]
+    assert {shape.y0 for shape in horizontal_lines} == {70.0, 80.0}