refactor.md

Refactor Proposal: Cleaner GRM Leaderboard Space

Goal

Build a cleaner version of the Game Ready Leaderboard that keeps the same information, copy direction, dark NVIDIA-inspired theme, and green accent system, but removes the layout constraints currently caused by the Gradio page structure.

The main product change is to consolidate stacked sections into a smaller number of interactive surfaces. In particular, the current Leaderboard plus three collapsed Per-Benchmark Score Breakdown tables should become one score explorer with filters, toggles, and expandable benchmark detail views.

The updated benchmark and score plan should be treated as coming from ref/GRM Eval - Benchmarks PRD.pdf.

Recommended Stack

Use Streamlit as the next implementation target.

Reasons:

• Hugging Face Spaces supports Streamlit directly through README metadata with sdk: streamlit.
• Streamlit gives enough native interactivity for filters, segmented controls, tabs, tables, expanders, and sidebar controls without building a custom frontend stack.
• The existing Python data files can be reused with minimal migration.
• The app can stay frontend-only and file-backed, matching the current Hugging Face Space model.
• Custom CSS can preserve the current dark surface, muted text, and NVIDIA green accent style.

Alternative: a static React/Vite Space would give better table UX and visual polish, especially with TanStack Table, but it would require a larger rewrite and a separate build step. For this repo, Streamlit is the best next step unless the target is a fully custom product UI.

Space Metadata Changes

If moving to Streamlit, update the README frontmatter:

---
title: GRM Leaderboard
colorFrom: gray
colorTo: blue
sdk: streamlit
app_file: streamlit_app.py
pinned: false
---

Recommended files:

• streamlit_app.py: page composition, controls, CSS injection, table rendering.
• benchmarks.py: benchmark registry and PRD-derived metadata.
• scores.py: score data.
• scoring.py: weighted scoring and ranking logic.
• theme.py or ui_theme.py: reusable color tokens and CSS strings.
• data_views.py: table-shaping helpers for leaderboard, benchmark matrix, and benchmark detail views.

Keep app.py during migration until the Streamlit version is accepted, then remove or archive the Gradio entry point.

Product Shape

Current Pain

The current Gradio app stacks the experience as:

1. Overview
2. Leaderboard
3. Proprietary model toggle
4. Per-Benchmark Score Breakdown, split into three collapsed category tables
5. Evaluation Suite
6. Benchmark Details
7. Separate GRM-Bench tab

This works, but it makes users scroll and mentally reconcile several tables that all describe the same score system.

Proposed Shape

Use two top-level pages or tabs:

1. Leaderboard
2. Benchmark Library

The first page is the primary product surface. The second page is the methodology/reference surface.

Page 1: Leaderboard

Header

Keep the existing hero identity and tone:

• Eyebrow: NVIDIA Game Ready Evaluation
• H1: Game Ready Leaderboard
• Supporting copy: An open game model evaluation surface for comparing LLMs across roleplay, gameplay actions, and practical in-game reasoning.

Keep the overview copy nearby, but make it less dominant. Use a short default summary with an expandable About GRM Score section containing the full paragraphs and formulas.

Score Explorer

Replace the separate Leaderboard and Per-Benchmark Score Breakdown sections with one consolidated score explorer.

Controls:

• Model visibility: All models / Open-source only
• Score view: Summary / Category / Benchmark matrix
• Category filter: All, Roleplay, Actions, General
• Benchmark priority filter: P0, P1, P2, All
• Benchmark source filter: All, External, GRM-Bench
• Domain filter for PRD domains such as conversational breadth/coherence, multi-turn tool calling, input resilience, input grounding, restriction adherence, and common sense.
• Search box for model or benchmark name.
• Optional checkbox: Show missing scores.

Default columns in Summary view:

Rank	Model	GRM Score	Roleplay	Actions	General

Default columns in Category view:

Rank	Model	GRM Score	Selected Category Score	Core Avg	Supplementary Avg	Missing

Default columns in Benchmark matrix view:

Rank	Model	GRM Score	Roleplay	Actions	General	Selected benchmark columns...

Behavior:

• The same table changes shape based on the selected view.
• Category filters control which benchmark score columns appear.
• Benchmark priority/source/domain filters reduce the matrix without changing the overall GRM calculation unless a separate Recalculate using visible benchmarks toggle is enabled.
• Clicking or selecting a model reveals a detail panel below the table with category breakdowns and strongest/weakest benchmark scores.
• Clicking or selecting a benchmark reveals the benchmark description, priority, weight, source, paper link, and PRD summary.

This makes the per-benchmark breakdown part of the main leaderboard instead of a second section users have to open one category at a time.

Score Cards

Above the table, show small dense stats rather than big decorative cards:

• Top model
• Best open-source model
• Number of models
• Number of benchmarks with active score data
• Latest data source: ref/GRM Eval - Benchmarks PRD.pdf or generated data snapshot date

These should be compact and utilitarian, not marketing-style hero cards.

Notes

Keep the current warning that scores are placeholder/static until replaced, but attach it to the table caption or data status line instead of a separate section note.

Suggested copy:

Current score values are static placeholders unless replaced with benchmark outputs from the PRD-backed data pipeline.

Page 2: Benchmark Library

This page consolidates Evaluation Suite, Benchmark Details, and GRM-Bench authored benchmark descriptions.

Benchmark Registry Table

One searchable/filterable table:

Category	Domain	Benchmark	Source	Priority	Weight	Included in GRM	Description

This replaces the current Evaluation Suite table and gives it the extra PRD fields.

Benchmark Detail Drawer or Expander

Selecting a benchmark opens details below the table:

• Summary
• Methodology
• Detection scope, where applicable
• Paper/source link
• Test sample availability
• Whether it is included in GRM score

For Streamlit, this can be a st.expander or a selected-row-driven detail panel. If row selection is awkward in Streamlit, use a selectbox above the detail panel.

GRM-Bench Authored Families

Fold the authored GRM-Bench tab into the Benchmark Library page. Use a segmented control or filter rather than a separate top-level tab.

PRD-backed authored dimensions:

Dimension	Phase	Included in GRM	Notes
Coherence	Phase 1	Yes	Logical soundness, contradictions, role confusion, knowledge boundaries, false premises.
Response Diversity	Phase 1	Yes	Avoids repetitive language, sentence frames, and stale delivery.
Prompt Robustness	Phase 1	Yes	Handles the same intent across terse, verbose, structured, and noisy prompts.
Persona-Aligned Actions	Phase 1	Yes	Ensures personality instructions are followed in tool use and planning.
Tool Recovery	Phase 1	Yes	Recovers from missed calls, malformed arguments, explicit failures, and fabricated outputs.
State Adaption / State Tracking	Phase 1	Yes	Tracks changing world state without stale values or entity drift.
Emotional Intelligence	Sanity / non-scored	No	Important for immersion but marked as not included in GRM score in the PRD.
Role Diversity	Phase 2 / non-scored	No	Diverse characters, accents, speech patterns, and scenarios.
Spatial & Temporal Awareness	Phase 2	TBD	Correctly tracks entities, locations, navigation, and state over time.
Group Conversations	Phase 2	TBD	Maintains roleplay quality when multiple parties participate.
Self Triggering	Phase 2	TBD	Judges when to trigger events from situational awareness without a user prompt.

Note: normalize the spelling in the UI to State Adaptation, but preserve any original PRD labels in source metadata if needed.

Updated Data Model

The current benchmark registry has name, category, calc_weight, description, summary, and paper.

The PRD implies a richer registry:

BENCHMARKS = [
    {
        "id": "grm_coherence",
        "name": "GRM - Coherence",
        "category": "ROLEPLAY",
        "domain": "Conversational breadth/coherence",
        "source": "GRM-Bench",
        "phase": "Phase 1",
        "priority": "P0",
        "calc_weight": 1.0,
        "included_in_grm": True,
        "description": "Logically sound and coherent across turns without contradictions.",
        "summary": "...",
        "methodology": "...",
        "detection_scope": [...],
        "paper": None,
    },
]

Recommended score shape:

MODEL_SCORES = {
    "Qwen3.5-4B": {
        "grm_coherence": 41.4,
        "multichallenge": 49.0,
    }
}

Use 0-100 values directly if the PRD data is already in that scale. The current app stores 0.0-1.0 values and formats them as percentages; the refactor should pick one canonical scale and document it. I recommend storing 0-100 because the PRD table is already expressed that way.

Recommended model metadata:

MODEL_METADATA = {
    "Qwen3.5-4B": {
        "family": "Qwen",
        "size": "4B",
        "open_weights": True,
        "precision": None,
    }
}

PRD Benchmark Suite Mapping

The Streamlit version should update the visible suite to match the PRD table, not the older placeholder-only registry.

Roleplay, 33 percent

Domain: conversational breadth/coherence.

Benchmark	Priority	Weight	Source
MultiChallenge	P0	1.0	External
RoleBench	P0	1.0	External
RoleMRC	P1	0.5	External
EQ-Bench v3	P1	0.5	External
GRM - Coherence	P0	1.0	GRM-Bench
GRM - Resp. Div	P0	1.0	GRM-Bench

Actions, 33 percent

Domain: multi-turn tool calling.

Benchmark	Priority	Weight	Source
BFCL v3	P0	1.0	External
When2Call-MT	P0	1.0	External
ToolSandbox	P0	1.0	External
Tau2-Bench	P0	1.0	External
BFCL v4	P1	0.5	External
GRM - Tool Rec	P0	1.0	GRM-Bench

General, 33 percent

Domains: input resilience, input grounding, restriction adherence, and common sense.

Domain	Benchmark	Priority	Weight	Source
Input resilience	IHEval	P0	1.0	External
Input resilience	RULER	P0	1.0	External
Input resilience	LongMemEval	P1	0.5	External
Input resilience	AgentIF	P1	0.5	External
Input resilience	GRM - Prompt Rob	P0	1.0	GRM-Bench
Input resilience	GRM - State Adapt	P0	1.0	GRM-Bench
Input grounding	GaRAGe	P0	1.0	External
Input grounding	RAGTruth	P1	0.5	External
Restriction adherence	StructEval-T	P1	0.5	External
Restriction adherence	IFBench	P1	0.5	External
Restriction adherence	GRM - Persona Act	P0	1.0	GRM-Bench
Common sense	GSM8K	P2	0.25	External
Common sense	HumanEval	P2	0.25	External
Common sense	MBPP	P2	0.25	External

Scoring Rules

Keep the current top-level scoring model:

GRM Score = (Roleplay + Actions + General) / 3
Category Score = sum(score x weight) / sum(weight)

Refinements:

• Only benchmarks with included_in_grm = True count toward category scores.
• Missing scores are skipped in the weighted average, but the UI should display missing counts.
• Non-scored dimensions like Emotional Intelligence and Role Diversity appear in the Benchmark Library, not in the GRM calculation.
• If a user turns on Recalculate using visible benchmarks, label the result as an exploratory filtered score, not the official GRM Score.

Visual System

Preserve the current theme direction:

:root {
  --bg-top: #202327;
  --bg-bottom: #0f1012;
  --surface: #15181b;
  --surface-strong: #24282d;
  --surface-alt: #1d2126;
  --text-main: #f5f7f8;
  --text-muted: #c1c6cb;
  --text-soft: #a2a8ae;
  --accent: #76b900;
  --rule: rgba(255, 255, 255, 0.08);
}

Streamlit CSS targets should style:

• App background gradient.
• Main content width around 1260px.
• Compact table headers with uppercase muted text.
• Alternating table rows using #1c2024 and #24282d.
• Top ranked row highlight using a dark green-tinted row like #252d1d.
• Accent links and selected controls with #76b900.
• Dense control rows rather than stacked controls.

Avoid a marketing-style redesign. This should feel like a sharp evaluation tool: compact, scannable, and built for repeated comparison.

Streamlit Layout Sketch

st.set_page_config(
    page_title="GRM Score - Game Ready Leaderboard",
    layout="wide",
)

inject_theme_css()

render_header()
render_about_grm_expander()

tab_leaderboard, tab_library = st.tabs(["Leaderboard", "Benchmark Library"])

with tab_leaderboard:
    controls = render_score_controls()
    leaderboard_frame = build_score_explorer_frame(controls)
    st.dataframe(
        leaderboard_frame,
        use_container_width=True,
        hide_index=True,
    )
    render_selected_model_or_benchmark_details(controls)

with tab_library:
    filters = render_benchmark_filters()
    benchmark_frame = build_benchmark_registry_frame(filters)
    st.dataframe(benchmark_frame, use_container_width=True, hide_index=True)
    render_benchmark_detail_panel(filters)

Migration Plan

1. Add PRD fields to benchmarks.py: id, domain, source, phase, priority, and included_in_grm.
2. Convert scores.py to canonical 0-100 values and PRD model names.
3. Update scoring.py to use benchmark IDs and included_in_grm.
4. Add table view helpers in data_views.py.
5. Build streamlit_app.py with the two-page/tab structure.
6. Port the existing CSS tokens into Streamlit CSS injection.
7. Update README metadata to sdk: streamlit and app_file: streamlit_app.py.
8. Keep app.py until parity is verified, then remove the Gradio dependency from requirements.txt.
9. Verify locally with streamlit run streamlit_app.py.
10. Deploy to Hugging Face Space and confirm the Space launches with the new SDK metadata.

Acceptance Criteria

• The first screen shows the header, concise overview, score controls, and leaderboard without requiring deep scrolling.
• The Leaderboard and Per-Benchmark Score Breakdown are one consolidated interactive table surface.
• Users can filter by category, domain, priority, source, model visibility, and benchmark name.
• Users can switch between summary, category, and benchmark matrix views without leaving the leaderboard.
• The Benchmark Library includes Evaluation Suite, Benchmark Details, and GRM-Bench authored descriptions in one place.
• The visual theme matches the current dark NVIDIA-accented design.
• The scoring logic matches the existing formula and the PRD weights.
• Non-scored dimensions are visible but clearly excluded from official GRM Score.
• The Hugging Face Space runs without an external database or backend.

Open Decisions

• Whether to keep official score storage as 0-100 values, matching the PRD, or preserve the current 0.0-1.0 internal scale.
• Whether Streamlit's built-in dataframe is sufficient, or whether the app needs streamlit-aggrid for stronger filtering, pinned columns, and row selection.
• Whether non-scored PRD dimensions should live in the main Benchmark Library by default or behind a Show future/non-scored dimensions toggle.
• Whether the old Gradio app should remain as app_gradio.py for a short transition period.