refactor.md

# Refactor Proposal: Cleaner GRM Leaderboard Space

## Goal

Build a cleaner version of the Game Ready Leaderboard that keeps the same information, copy direction, dark NVIDIA-inspired theme, and green accent system, but removes the layout constraints currently caused by the Gradio page structure.

The main product change is to consolidate stacked sections into a smaller number of interactive surfaces. In particular, the current Leaderboard plus three collapsed Per-Benchmark Score Breakdown tables should become one score explorer with filters, toggles, and expandable benchmark detail views.

The updated benchmark and score plan should be treated as coming from `ref/GRM Eval - Benchmarks PRD.pdf`.

## Recommended Stack

Use Streamlit as the next implementation target.

Reasons:

- Hugging Face Spaces supports Streamlit directly through README metadata with `sdk: streamlit`.
- Streamlit gives enough native interactivity for filters, segmented controls, tabs, tables, expanders, and sidebar controls without building a custom frontend stack.
- The existing Python data files can be reused with minimal migration.
- The app can stay frontend-only and file-backed, matching the current Hugging Face Space model.
- Custom CSS can preserve the current dark surface, muted text, and NVIDIA green accent style.

Alternative: a static React/Vite Space would give better table UX and visual polish, especially with TanStack Table, but it would require a larger rewrite and a separate build step. For this repo, Streamlit is the best next step unless the target is a fully custom product UI.

## Space Metadata Changes

If moving to Streamlit, update the README frontmatter:

```yaml
---
title: GRM Leaderboard
colorFrom: gray
colorTo: blue
sdk: streamlit
app_file: streamlit_app.py
pinned: false
---
```

Recommended files:

- `streamlit_app.py`: page composition, controls, CSS injection, table rendering.
- `benchmarks.py`: benchmark registry and PRD-derived metadata.
- `scores.py`: score data.
- `scoring.py`: weighted scoring and ranking logic.
- `theme.py` or `ui_theme.py`: reusable color tokens and CSS strings.
- `data_views.py`: table-shaping helpers for leaderboard, benchmark matrix, and benchmark detail views.

Keep `app.py` during migration until the Streamlit version is accepted, then remove or archive the Gradio entry point.

## Product Shape

### Current Pain

The current Gradio app stacks the experience as:

1. Overview
2. Leaderboard
3. Proprietary model toggle
4. Per-Benchmark Score Breakdown, split into three collapsed category tables
5. Evaluation Suite
6. Benchmark Details
7. Separate GRM-Bench tab

This works, but it makes users scroll and mentally reconcile several tables that all describe the same score system.

### Proposed Shape

Use two top-level pages or tabs:

1. `Leaderboard`
2. `Benchmark Library`

The first page is the primary product surface. The second page is the methodology/reference surface.

## Page 1: Leaderboard

### Header

Keep the existing hero identity and tone:

- Eyebrow: `NVIDIA Game Ready Evaluation`
- H1: `Game Ready Leaderboard`
- Supporting copy: `An open game model evaluation surface for comparing LLMs across roleplay, gameplay actions, and practical in-game reasoning.`

Keep the overview copy nearby, but make it less dominant. Use a short default summary with an expandable `About GRM Score` section containing the full paragraphs and formulas.

### Score Explorer

Replace the separate Leaderboard and Per-Benchmark Score Breakdown sections with one consolidated score explorer.

Controls:

- Model visibility: `All models` / `Open-source only`
- Score view: `Summary` / `Category` / `Benchmark matrix`
- Category filter: `All`, `Roleplay`, `Actions`, `General`
- Benchmark priority filter: `P0`, `P1`, `P2`, `All`
- Benchmark source filter: `All`, `External`, `GRM-Bench`
- Domain filter for PRD domains such as conversational breadth/coherence, multi-turn tool calling, input resilience, input grounding, restriction adherence, and common sense.
- Search box for model or benchmark name.
- Optional checkbox: `Show missing scores`.

Default columns in `Summary` view:

| Rank | Model | GRM Score | Roleplay | Actions | General |
| --- | --- | --- | --- | --- | --- |

Default columns in `Category` view:

| Rank | Model | GRM Score | Selected Category Score | Core Avg | Supplementary Avg | Missing |
| --- | --- | --- | --- | --- | --- | --- |

Default columns in `Benchmark matrix` view:

| Rank | Model | GRM Score | Roleplay | Actions | General | Selected benchmark columns... |
| --- | --- | --- | --- | --- | --- | --- |

Behavior:

- The same table changes shape based on the selected view.
- Category filters control which benchmark score columns appear.
- Benchmark priority/source/domain filters reduce the matrix without changing the overall GRM calculation unless a separate `Recalculate using visible benchmarks` toggle is enabled.
- Clicking or selecting a model reveals a detail panel below the table with category breakdowns and strongest/weakest benchmark scores.
- Clicking or selecting a benchmark reveals the benchmark description, priority, weight, source, paper link, and PRD summary.

This makes the per-benchmark breakdown part of the main leaderboard instead of a second section users have to open one category at a time.

### Score Cards

Above the table, show small dense stats rather than big decorative cards:

- Top model
- Best open-source model
- Number of models
- Number of benchmarks with active score data
- Latest data source: `ref/GRM Eval - Benchmarks PRD.pdf` or generated data snapshot date

These should be compact and utilitarian, not marketing-style hero cards.

### Notes

Keep the current warning that scores are placeholder/static until replaced, but attach it to the table caption or data status line instead of a separate section note.

Suggested copy:

`Current score values are static placeholders unless replaced with benchmark outputs from the PRD-backed data pipeline.`

## Page 2: Benchmark Library

This page consolidates Evaluation Suite, Benchmark Details, and GRM-Bench authored benchmark descriptions.

### Benchmark Registry Table

One searchable/filterable table:

| Category | Domain | Benchmark | Source | Priority | Weight | Included in GRM | Description |
| --- | --- | --- | --- | --- | --- | --- | --- |

This replaces the current Evaluation Suite table and gives it the extra PRD fields.

### Benchmark Detail Drawer or Expander

Selecting a benchmark opens details below the table:

- Summary
- Methodology
- Detection scope, where applicable
- Paper/source link
- Test sample availability
- Whether it is included in GRM score

For Streamlit, this can be a `st.expander` or a selected-row-driven detail panel. If row selection is awkward in Streamlit, use a selectbox above the detail panel.

### GRM-Bench Authored Families

Fold the authored GRM-Bench tab into the Benchmark Library page. Use a segmented control or filter rather than a separate top-level tab.

PRD-backed authored dimensions:

| Dimension | Phase | Included in GRM | Notes |
| --- | --- | --- | --- |
| Coherence | Phase 1 | Yes | Logical soundness, contradictions, role confusion, knowledge boundaries, false premises. |
| Response Diversity | Phase 1 | Yes | Avoids repetitive language, sentence frames, and stale delivery. |
| Prompt Robustness | Phase 1 | Yes | Handles the same intent across terse, verbose, structured, and noisy prompts. |
| Persona-Aligned Actions | Phase 1 | Yes | Ensures personality instructions are followed in tool use and planning. |
| Tool Recovery | Phase 1 | Yes | Recovers from missed calls, malformed arguments, explicit failures, and fabricated outputs. |
| State Adaption / State Tracking | Phase 1 | Yes | Tracks changing world state without stale values or entity drift. |
| Emotional Intelligence | Sanity / non-scored | No | Important for immersion but marked as not included in GRM score in the PRD. |
| Role Diversity | Phase 2 / non-scored | No | Diverse characters, accents, speech patterns, and scenarios. |
| Spatial & Temporal Awareness | Phase 2 | TBD | Correctly tracks entities, locations, navigation, and state over time. |
| Group Conversations | Phase 2 | TBD | Maintains roleplay quality when multiple parties participate. |
| Self Triggering | Phase 2 | TBD | Judges when to trigger events from situational awareness without a user prompt. |

Note: normalize the spelling in the UI to `State Adaptation`, but preserve any original PRD labels in source metadata if needed.

## Updated Data Model

The current benchmark registry has `name`, `category`, `calc_weight`, `description`, `summary`, and `paper`.

The PRD implies a richer registry:

```python
BENCHMARKS = [
    {
        "id": "grm_coherence",
        "name": "GRM - Coherence",
        "category": "ROLEPLAY",
        "domain": "Conversational breadth/coherence",
        "source": "GRM-Bench",
        "phase": "Phase 1",
        "priority": "P0",
        "calc_weight": 1.0,
        "included_in_grm": True,
        "description": "Logically sound and coherent across turns without contradictions.",
        "summary": "...",
        "methodology": "...",
        "detection_scope": [...],
        "paper": None,
    },
]
```

Recommended score shape:

```python
MODEL_SCORES = {
    "Qwen3.5-4B": {
        "grm_coherence": 41.4,
        "multichallenge": 49.0,
    }
}
```

Use 0-100 values directly if the PRD data is already in that scale. The current app stores 0.0-1.0 values and formats them as percentages; the refactor should pick one canonical scale and document it. I recommend storing 0-100 because the PRD table is already expressed that way.

Recommended model metadata:

```python
MODEL_METADATA = {
    "Qwen3.5-4B": {
        "family": "Qwen",
        "size": "4B",
        "open_weights": True,
        "precision": None,
    }
}
```

## PRD Benchmark Suite Mapping

The Streamlit version should update the visible suite to match the PRD table, not the older placeholder-only registry.

### Roleplay, 33 percent

Domain: conversational breadth/coherence.

| Benchmark | Priority | Weight | Source |
| --- | --- | --- | --- |
| MultiChallenge | P0 | 1.0 | External |
| RoleBench | P0 | 1.0 | External |
| RoleMRC | P1 | 0.5 | External |
| EQ-Bench v3 | P1 | 0.5 | External |
| GRM - Coherence | P0 | 1.0 | GRM-Bench |
| GRM - Resp. Div | P0 | 1.0 | GRM-Bench |

### Actions, 33 percent

Domain: multi-turn tool calling.

| Benchmark | Priority | Weight | Source |
| --- | --- | --- | --- |
| BFCL v3 | P0 | 1.0 | External |
| When2Call-MT | P0 | 1.0 | External |
| ToolSandbox | P0 | 1.0 | External |
| Tau2-Bench | P0 | 1.0 | External |
| BFCL v4 | P1 | 0.5 | External |
| GRM - Tool Rec | P0 | 1.0 | GRM-Bench |

### General, 33 percent

Domains: input resilience, input grounding, restriction adherence, and common sense.

| Domain | Benchmark | Priority | Weight | Source |
| --- | --- | --- | --- | --- |
| Input resilience | IHEval | P0 | 1.0 | External |
| Input resilience | RULER | P0 | 1.0 | External |
| Input resilience | LongMemEval | P1 | 0.5 | External |
| Input resilience | AgentIF | P1 | 0.5 | External |
| Input resilience | GRM - Prompt Rob | P0 | 1.0 | GRM-Bench |
| Input resilience | GRM - State Adapt | P0 | 1.0 | GRM-Bench |
| Input grounding | GaRAGe | P0 | 1.0 | External |
| Input grounding | RAGTruth | P1 | 0.5 | External |
| Restriction adherence | StructEval-T | P1 | 0.5 | External |
| Restriction adherence | IFBench | P1 | 0.5 | External |
| Restriction adherence | GRM - Persona Act | P0 | 1.0 | GRM-Bench |
| Common sense | GSM8K | P2 | 0.25 | External |
| Common sense | HumanEval | P2 | 0.25 | External |
| Common sense | MBPP | P2 | 0.25 | External |

## Scoring Rules

Keep the current top-level scoring model:

```text
GRM Score = (Roleplay + Actions + General) / 3
Category Score = sum(score x weight) / sum(weight)
```

Refinements:

- Only benchmarks with `included_in_grm = True` count toward category scores.
- Missing scores are skipped in the weighted average, but the UI should display missing counts.
- Non-scored dimensions like Emotional Intelligence and Role Diversity appear in the Benchmark Library, not in the GRM calculation.
- If a user turns on `Recalculate using visible benchmarks`, label the result as an exploratory filtered score, not the official GRM Score.

## Visual System

Preserve the current theme direction:

```css
:root {
  --bg-top: #202327;
  --bg-bottom: #0f1012;
  --surface: #15181b;
  --surface-strong: #24282d;
  --surface-alt: #1d2126;
  --text-main: #f5f7f8;
  --text-muted: #c1c6cb;
  --text-soft: #a2a8ae;
  --accent: #76b900;
  --rule: rgba(255, 255, 255, 0.08);
}
```

Streamlit CSS targets should style:

- App background gradient.
- Main content width around 1260px.
- Compact table headers with uppercase muted text.
- Alternating table rows using `#1c2024` and `#24282d`.
- Top ranked row highlight using a dark green-tinted row like `#252d1d`.
- Accent links and selected controls with `#76b900`.
- Dense control rows rather than stacked controls.

Avoid a marketing-style redesign. This should feel like a sharp evaluation tool: compact, scannable, and built for repeated comparison.

## Streamlit Layout Sketch

```python
st.set_page_config(
    page_title="GRM Score - Game Ready Leaderboard",
    layout="wide",
)

inject_theme_css()

render_header()
render_about_grm_expander()

tab_leaderboard, tab_library = st.tabs(["Leaderboard", "Benchmark Library"])

with tab_leaderboard:
    controls = render_score_controls()
    leaderboard_frame = build_score_explorer_frame(controls)
    st.dataframe(
        leaderboard_frame,
        use_container_width=True,
        hide_index=True,
    )
    render_selected_model_or_benchmark_details(controls)

with tab_library:
    filters = render_benchmark_filters()
    benchmark_frame = build_benchmark_registry_frame(filters)
    st.dataframe(benchmark_frame, use_container_width=True, hide_index=True)
    render_benchmark_detail_panel(filters)
```

## Migration Plan

1. Add PRD fields to `benchmarks.py`: `id`, `domain`, `source`, `phase`, `priority`, and `included_in_grm`.
2. Convert `scores.py` to canonical 0-100 values and PRD model names.
3. Update `scoring.py` to use benchmark IDs and `included_in_grm`.
4. Add table view helpers in `data_views.py`.
5. Build `streamlit_app.py` with the two-page/tab structure.
6. Port the existing CSS tokens into Streamlit CSS injection.
7. Update README metadata to `sdk: streamlit` and `app_file: streamlit_app.py`.
8. Keep `app.py` until parity is verified, then remove the Gradio dependency from `requirements.txt`.
9. Verify locally with `streamlit run streamlit_app.py`.
10. Deploy to Hugging Face Space and confirm the Space launches with the new SDK metadata.

## Acceptance Criteria

- The first screen shows the header, concise overview, score controls, and leaderboard without requiring deep scrolling.
- The Leaderboard and Per-Benchmark Score Breakdown are one consolidated interactive table surface.
- Users can filter by category, domain, priority, source, model visibility, and benchmark name.
- Users can switch between summary, category, and benchmark matrix views without leaving the leaderboard.
- The Benchmark Library includes Evaluation Suite, Benchmark Details, and GRM-Bench authored descriptions in one place.
- The visual theme matches the current dark NVIDIA-accented design.
- The scoring logic matches the existing formula and the PRD weights.
- Non-scored dimensions are visible but clearly excluded from official GRM Score.
- The Hugging Face Space runs without an external database or backend.

## Open Decisions

- Whether to keep official score storage as 0-100 values, matching the PRD, or preserve the current 0.0-1.0 internal scale.
- Whether Streamlit's built-in dataframe is sufficient, or whether the app needs `streamlit-aggrid` for stronger filtering, pinned columns, and row selection.
- Whether non-scored PRD dimensions should live in the main Benchmark Library by default or behind a `Show future/non-scored dimensions` toggle.
- Whether the old Gradio app should remain as `app_gradio.py` for a short transition period.