V1 including mock results

Makefile

@@ -1,13 +0,0 @@
-.PHONY: style format
-
-
-style:
-	python -m black --line-length 119 .
-	python -m isort .
-	ruff check --fix .
-
-
-quality:
-	python -m black --check --line-length 119 .
-	python -m isort --check-only .
-	ruff check .

README.md

@@ -11,36 +11,103 @@ short_description: Duplicate this leaderboard to initialize your own!
 sdk_version: 5.19.0
 ---
 
-# Start the configuration
-
-Most of the variables to change for a default leaderboard are in `src/env.py` (replace the path for your leaderboard) and `src/about.py` (for tasks).
-
-Results files should have the following format and be stored as json files:
-```json
-{
-    "config": {
-        "model_dtype": "torch.float16", # or torch.bfloat16 or 8bit or 4bit
-        "model_name": "path of the model on the hub: org/model",
-        "model_sha": "revision on the hub",
-    },
-    "results": {
-        "task_name": {
-            "metric_name": score,
-        },
-        "task_name2": {
-            "metric_name": score,
-        }
-    }
-}
+# 🔒 LLM PII Detection Leaderboard
+
+A comprehensive benchmark for evaluating language models' performance in detecting and handling personally identifiable information (PII) across various document types and scenarios.
+
+## ✨ Features
+
+- **Beautiful Modern UI**: Elegant dark theme with gradient styling and smooth animations
+- **Comprehensive Metrics**: Precision, Recall, F1 Score, Over-detection Rate, Processing Time, and Cost
+- **Domain-Specific Analysis**: Specialized evaluation across Healthcare, Financial, Government, Legal, and Personal documents
+- **Performance Cards**: Professional model performance cards perfect for presentations and reports
+- **Interactive Filtering**: Filter by model type, document type, and sort by any metric
+- **Real-time Updates**: Dynamic table updates and score visualizations
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+git clone https://github.com/your-username/LLM-PII-Detection-Leaderboard.git
+cd LLM-PII-Detection-Leaderboard
+pip install -r requirements.txt
+```
+
+### Run the Application
+
+```bash
+python app.py
+```
+
+The leaderboard will be available at `http://localhost:7860`
+
+## 📊 Key Metrics
+
+- **Overall Accuracy**: Percentage of correctly identified and classified PII entities
+- **Precision**: Of all flagged items, how many were actually PII (avoiding false positives)
+- **Recall**: Of all PII present, how many were successfully detected (avoiding false negatives)
+- **F1 Score**: Harmonic mean balancing precision and recall
+- **Over-detection Rate**: Percentage of non-PII incorrectly flagged (lower is better)
+
+## 🏗️ Project Structure
+
 ```
+LLM-PII-Detection-Leaderboard/
+├── app.py                 # Main application entry point
+├── pii_leaderboard.py     # Core leaderboard functionality
+├── data_loader.py         # Data loading and styling configuration
+├── requirements.txt       # Python dependencies
+└── README.md             # This file
+```
+
+## 🎨 Design Philosophy
+
+This leaderboard combines the slim architecture of agent-leaderboard with the beautiful design elements from DocumentProcessing Leaderboard Nutrient, featuring:
+
+- **Minimal Dependencies**: Only essential packages (Gradio, Pandas, NumPy)
+- **Clean Architecture**: Simple, maintainable code structure
+- **Professional Styling**: Modern dark theme with custom color palette
+- **Interactive Elements**: Score bars, rank badges, and performance cards
+- **Responsive Design**: Works beautifully on all screen sizes
+
+## 🔧 Customization
+
+### Adding New Models
+
+Update the `sample_data` dictionary in `data_loader.py` with your model's performance metrics.
+
+### Changing Colors
+
+Modify the `COLORS` dictionary in `data_loader.py` to customize the color scheme.
+
+### Adding New Metrics
+
+1. Add the metric to your data structure
+2. Update the table generation in `pii_leaderboard.py`
+3. Add appropriate styling and score bars
+
+## 📈 Performance
+
+The leaderboard currently evaluates 8 leading language models across:
+- **5 Document Types**: Healthcare, Financial, Government, Legal, Personal
+- **6 Key Metrics**: Accuracy, Precision, Recall, F1, Over-detection Rate, Cost & Time
+- **Real-world Scenarios**: Synthetic industry documents with embedded PII
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Test thoroughly
+5. Submit a pull request
 
-Request files are created automatically by this tool.
+## 📄 License
 
-If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.
+This project is licensed under the MIT License - see the LICENSE file for details.
 
-# Code logic for more complex edits
+## 🙏 Acknowledgments
 
-You'll find 
-- the main table' columns names and properties in `src/display/utils.py`
-- the logic to read all results and request files, then convert them in dataframe lines, in `src/leaderboard/read_evals.py`, and `src/populate.py`
-- the logic to allow or filter submissions in `src/submission/submit.py` and `src/submission/check_validity.py`
+- Inspired by the elegant design of DocumentProcessing Leaderboard Nutrient
+- Built with the slim architecture approach of agent-leaderboard
+- Powered by Gradio for the beautiful web interface

app.py

@@ -1,204 +1,14 @@
-import gradio as gr
-from gradio_leaderboard import Leaderboard, ColumnFilter, SelectColumns
-import pandas as pd
-from apscheduler.schedulers.background import BackgroundScheduler
-from huggingface_hub import snapshot_download
-
-from src.about import (
-    CITATION_BUTTON_LABEL,
-    CITATION_BUTTON_TEXT,
-    EVALUATION_QUEUE_TEXT,
-    INTRODUCTION_TEXT,
-    LLM_BENCHMARKS_TEXT,
-    TITLE,
-)
-from src.display.css_html_js import custom_css
-from src.display.utils import (
-    BENCHMARK_COLS,
-    COLS,
-    EVAL_COLS,
-    EVAL_TYPES,
-    AutoEvalColumn,
-    ModelType,
-    fields,
-    WeightType,
-    Precision
-)
-from src.envs import API, EVAL_REQUESTS_PATH, EVAL_RESULTS_PATH, QUEUE_REPO, REPO_ID, RESULTS_REPO, TOKEN
-from src.populate import get_evaluation_queue_df, get_leaderboard_df
-from src.submission.submit import add_new_eval
-
+import warnings
 
-def restart_space():
-    API.restart_space(repo_id=REPO_ID)
-
-### Space initialisation
-try:
-    print(EVAL_REQUESTS_PATH)
-    snapshot_download(
-        repo_id=QUEUE_REPO, local_dir=EVAL_REQUESTS_PATH, repo_type="dataset", tqdm_class=None, etag_timeout=30, token=TOKEN
-    )
-except Exception:
-    restart_space()
-try:
-    print(EVAL_RESULTS_PATH)
-    snapshot_download(
-        repo_id=RESULTS_REPO, local_dir=EVAL_RESULTS_PATH, repo_type="dataset", tqdm_class=None, etag_timeout=30, token=TOKEN
-    )
-except Exception:
-    restart_space()
+warnings.filterwarnings("ignore")
 
-
-LEADERBOARD_DF = get_leaderboard_df(EVAL_RESULTS_PATH, EVAL_REQUESTS_PATH, COLS, BENCHMARK_COLS)
-
-(
-    finished_eval_queue_df,
-    running_eval_queue_df,
-    pending_eval_queue_df,
-) = get_evaluation_queue_df(EVAL_REQUESTS_PATH, EVAL_COLS)
-
-def init_leaderboard(dataframe):
-    if dataframe is None or dataframe.empty:
-        raise ValueError("Leaderboard DataFrame is empty or None.")
-    return Leaderboard(
-        value=dataframe,
-        datatype=[c.type for c in fields(AutoEvalColumn)],
-        select_columns=SelectColumns(
-            default_selection=[c.name for c in fields(AutoEvalColumn) if c.displayed_by_default],
-            cant_deselect=[c.name for c in fields(AutoEvalColumn) if c.never_hidden],
-            label="Select Columns to Display:",
-        ),
-        search_columns=[AutoEvalColumn.model.name, AutoEvalColumn.license.name],
-        hide_columns=[c.name for c in fields(AutoEvalColumn) if c.hidden],
-        filter_columns=[
-            ColumnFilter(AutoEvalColumn.model_type.name, type="checkboxgroup", label="Model types"),
-            ColumnFilter(AutoEvalColumn.precision.name, type="checkboxgroup", label="Precision"),
-            ColumnFilter(
-                AutoEvalColumn.params.name,
-                type="slider",
-                min=0.01,
-                max=150,
-                label="Select the number of parameters (B)",
-            ),
-            ColumnFilter(
-                AutoEvalColumn.still_on_hub.name, type="boolean", label="Deleted/incomplete", default=True
-            ),
-        ],
-        bool_checkboxgroup_label="Hide models",
-        interactive=False,
+import gradio as gr
+from pii_leaderboard import create_app
+
+if __name__ == "__main__":
+    demo = create_app()
+    demo.launch(
+        server_name="0.0.0.0",
+        server_port=7860,
+        share=False
     )
-
-
-demo = gr.Blocks(css=custom_css)
-with demo:
-    gr.HTML(TITLE)
-    gr.Markdown(INTRODUCTION_TEXT, elem_classes="markdown-text")
-
-    with gr.Tabs(elem_classes="tab-buttons") as tabs:
-        with gr.TabItem("🏅 LLM Benchmark", elem_id="llm-benchmark-tab-table", id=0):
-            leaderboard = init_leaderboard(LEADERBOARD_DF)
-
-        with gr.TabItem("📝 About", elem_id="llm-benchmark-tab-table", id=2):
-            gr.Markdown(LLM_BENCHMARKS_TEXT, elem_classes="markdown-text")
-
-        with gr.TabItem("🚀 Submit here! ", elem_id="llm-benchmark-tab-table", id=3):
-            with gr.Column():
-                with gr.Row():
-                    gr.Markdown(EVALUATION_QUEUE_TEXT, elem_classes="markdown-text")
-
-                with gr.Column():
-                    with gr.Accordion(
-                        f"✅ Finished Evaluations ({len(finished_eval_queue_df)})",
-                        open=False,
-                    ):
-                        with gr.Row():
-                            finished_eval_table = gr.components.Dataframe(
-                                value=finished_eval_queue_df,
-                                headers=EVAL_COLS,
-                                datatype=EVAL_TYPES,
-                                row_count=5,
-                            )
-                    with gr.Accordion(
-                        f"🔄 Running Evaluation Queue ({len(running_eval_queue_df)})",
-                        open=False,
-                    ):
-                        with gr.Row():
-                            running_eval_table = gr.components.Dataframe(
-                                value=running_eval_queue_df,
-                                headers=EVAL_COLS,
-                                datatype=EVAL_TYPES,
-                                row_count=5,
-                            )
-
-                    with gr.Accordion(
-                        f"⏳ Pending Evaluation Queue ({len(pending_eval_queue_df)})",
-                        open=False,
-                    ):
-                        with gr.Row():
-                            pending_eval_table = gr.components.Dataframe(
-                                value=pending_eval_queue_df,
-                                headers=EVAL_COLS,
-                                datatype=EVAL_TYPES,
-                                row_count=5,
-                            )
-            with gr.Row():
-                gr.Markdown("# ✉️✨ Submit your model here!", elem_classes="markdown-text")
-
-            with gr.Row():
-                with gr.Column():
-                    model_name_textbox = gr.Textbox(label="Model name")
-                    revision_name_textbox = gr.Textbox(label="Revision commit", placeholder="main")
-                    model_type = gr.Dropdown(
-                        choices=[t.to_str(" : ") for t in ModelType if t != ModelType.Unknown],
-                        label="Model type",
-                        multiselect=False,
-                        value=None,
-                        interactive=True,
-                    )
-
-                with gr.Column():
-                    precision = gr.Dropdown(
-                        choices=[i.value.name for i in Precision if i != Precision.Unknown],
-                        label="Precision",
-                        multiselect=False,
-                        value="float16",
-                        interactive=True,
-                    )
-                    weight_type = gr.Dropdown(
-                        choices=[i.value.name for i in WeightType],
-                        label="Weights type",
-                        multiselect=False,
-                        value="Original",
-                        interactive=True,
-                    )
-                    base_model_name_textbox = gr.Textbox(label="Base model (for delta or adapter weights)")
-
-            submit_button = gr.Button("Submit Eval")
-            submission_result = gr.Markdown()
-            submit_button.click(
-                add_new_eval,
-                [
-                    model_name_textbox,
-                    base_model_name_textbox,
-                    revision_name_textbox,
-                    precision,
-                    weight_type,
-                    model_type,
-                ],
-                submission_result,
-            )
-
-    with gr.Row():
-        with gr.Accordion("📙 Citation", open=False):
-            citation_button = gr.Textbox(
-                value=CITATION_BUTTON_TEXT,
-                label=CITATION_BUTTON_LABEL,
-                lines=20,
-                elem_id="citation-button",
-                show_copy_button=True,
-            )
-
-scheduler = BackgroundScheduler()
-scheduler.add_job(restart_space, "interval", seconds=1800)
-scheduler.start()
-demo.queue(default_concurrency_limit=40).launch()

data_loader.py

@@ -0,0 +1,428 @@
+import pandas as pd
+import os
+
+# PII Detection Categories
+PII_CATEGORIES = {
+    "Overall": ["Overall Accuracy"],
+    "Entity Types": ["Names", "Addresses", "Phone Numbers", "SSN", "Medical IDs", "Financial Info"],
+    "Document Types": ["Healthcare", "Financial", "Government", "Legal", "Personal"],
+    "Performance": ["Precision", "Recall", "F1 Score"],
+    "Efficiency": ["Processing Time", "Cost per Document"]
+}
+
+# Model type definitions
+MODEL_TYPES = {
+    "Proprietary": "🔒",
+    "Open Source": "🔓"
+}
+
+def load_data():
+    """Load and preprocess the PII detection evaluation data from CSV file."""
+    
+    # Load from CSV file
+    csv_path = "results/pii_detection_results.csv"
+    
+    if not os.path.exists(csv_path):
+        raise FileNotFoundError(f"Results file not found: {csv_path}. Please ensure the CSV file exists in the results folder.")
+    
+    df = pd.read_csv(csv_path)
+    
+    # Clean and prepare data
+    df = df.fillna('')
+    
+    # Round numeric columns for better display
+    numeric_cols = [
+        'Overall Accuracy', 'Precision', 'Recall', 'F1 Score', 'Over-redaction Rate',
+        'Processing Time (s)', 'Cost per Document ($)',
+        'Healthcare Accuracy', 'Financial Accuracy', 'Government Accuracy', 
+        'Legal Accuracy', 'Personal Accuracy'
+    ]
+    
+    for col in numeric_cols:
+        if col in df.columns:
+            df[col] = pd.to_numeric(df[col], errors='coerce').round(3)
+    
+    return df
+
+# Color palette matching DocumentProcessing style
+COLORS = {
+    # Light mode colors
+    "white": "#FFFFFF",
+    "disc_pink": "#DE9DCC", 
+    "code_coral": "#F25E45",
+    "data_green": "#6EB579",
+    "digital_pollen": "#F0C968",
+    "warm_black": "#1A1414",
+    "off_white": "#EFEBE7",
+    "pixel_mist": "#E2DBD9", 
+    "soft_grey": "#C2B8AE",
+    "warm_grey": "#67594B",
+    
+    # Dark mode colors
+    "disc_pink_dm": "#4F2B45",
+    "code_coral_dm": "#672D23", 
+    "data_green_dm": "#2B412F",
+    "digital_pollen_dm": "#5B481A",
+}
+
+# Header content with PII detection focus
+HEADER_CONTENT = f"""
+<style>
+/* Import fonts */
+@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&display=swap');
+
+/* Root variables with custom color palette */
+:root {{
+    --bg-primary: #1A1414;
+    --bg-secondary: rgba(239, 235, 231, 0.03);
+    --bg-card: rgba(239, 235, 231, 0.02);
+    --border-subtle: rgba(239, 235, 231, 0.08);
+    --border-default: rgba(239, 235, 231, 0.12);
+    --border-strong: rgba(239, 235, 231, 0.2);
+    --text-primary: #EFEBE7;
+    --text-secondary: #C2B8AE;
+    --text-muted: #67594B;
+    --accent-primary: #DE9DCC;
+    --accent-secondary: #F25E45;
+    --accent-tertiary: #6EB579;
+    --accent-quaternary: #F0C968;
+    --glow-primary: rgba(222, 157, 204, 0.4);
+    --glow-secondary: rgba(242, 94, 69, 0.4);
+    --glow-tertiary: rgba(110, 181, 121, 0.4);
+}}
+
+/* Global font and background */
+.gradio-container {{
+    font-family: 'Inter', -apple-system, BlinkMacSystemFont, sans-serif !important;
+    background: var(--bg-primary) !important;
+    color: var(--text-primary) !important;
+}}
+
+/* Headers and text */
+h1, h2, h3, h4 {{
+    color: var(--text-primary) !important;
+    font-weight: 700 !important;
+    font-family: 'Inter', sans-serif !important;
+}}
+
+p, span, div {{
+    color: var(--text-primary) !important;
+    font-family: 'Inter', sans-serif !important;
+}}
+
+/* Dark containers */
+.dark-container {{
+    background: var(--bg-card);
+    border: 1px solid var(--border-subtle);
+    border-radius: 20px;
+    padding: 28px;
+    box-shadow: 0 8px 32px rgba(0, 0, 0, 0.4);
+    backdrop-filter: blur(10px);
+    position: relative;
+    overflow: hidden;
+}}
+
+/* Section headers */
+.section-header {{
+    display: flex;
+    align-items: center;
+    gap: 12px;
+    margin-bottom: 24px;
+}}
+
+.section-icon {{
+    filter: drop-shadow(0 0 12px currentColor);
+    transition: all 0.3s ease;
+}}
+
+/* Enhanced table styling */
+.v2-table-container {{
+    background: var(--bg-card);
+    border-radius: 16px;
+    overflow: hidden;
+    border: 1px solid var(--border-subtle);
+    box-shadow: 0 8px 32px rgba(0, 0, 0, 0.3);
+    backdrop-filter: blur(10px);
+}}
+
+.v2-styled-table {{
+    width: 100%;
+    border-collapse: collapse;
+    font-family: 'Inter', sans-serif;
+    font-size: 14px;
+}}
+
+.v2-styled-table thead {{
+    background: linear-gradient(135deg, var(--accent-primary), var(--accent-secondary));
+}}
+
+.v2-styled-table th {{
+    padding: 16px 12px;
+    text-align: left;
+    color: white;
+    font-weight: 600;
+    font-size: 13px;
+    text-transform: uppercase;
+    letter-spacing: 0.05em;
+    border: none;
+    position: relative;
+}}
+
+.v2-styled-table td {{
+    padding: 14px 12px;
+    border-bottom: 1px solid var(--border-subtle);
+    color: var(--text-primary);
+    transition: all 0.2s ease;
+    vertical-align: middle;
+}}
+
+.v2-styled-table tbody tr {{
+    transition: all 0.3s ease;
+    background: var(--bg-secondary);
+}}
+
+.v2-styled-table tbody tr:nth-child(even) {{
+    background: var(--bg-card);
+}}
+
+.v2-styled-table tbody tr:hover {{
+    background: rgba(222, 157, 204, 0.1);
+    box-shadow: 0 0 20px var(--glow-primary);
+    transform: scale(1.01);
+}}
+
+.model-name {{
+    font-weight: 600;
+    color: var(--accent-primary);
+    transition: all 0.2s ease;
+}}
+
+.numeric-cell {{
+    text-align: center;
+    font-family: 'SF Mono', monospace;
+    font-weight: 500;
+}}
+
+.score-cell {{
+    padding: 8px 12px;
+}}
+
+/* Scrollbar styling */
+::-webkit-scrollbar {{
+    width: 8px;
+    height: 8px;
+}}
+
+::-webkit-scrollbar-track {{
+    background: var(--bg-secondary);
+    border-radius: 4px;
+}}
+
+::-webkit-scrollbar-thumb {{
+    background: var(--accent-secondary);
+    border-radius: 4px;
+}}
+
+::-webkit-scrollbar-thumb:hover {{
+    background: var(--accent-primary);
+}}
+</style>
+
+<div style="
+    background: var(--bg-primary);
+    padding: 4rem 2rem;
+    border-radius: 16px;
+    margin-bottom: 0;
+    transition: all 0.3s ease;
+    position: relative;
+">
+    <div style="max-width: 72rem; margin: 0 auto;">
+        <div style="text-align: center; margin-bottom: 4rem;">
+            <h1 style="
+                font-size: 4rem;
+                font-weight: 800;
+                line-height: 1.1;
+                background: linear-gradient(45deg, var(--accent-primary), var(--accent-secondary));
+                -webkit-background-clip: text;
+                -webkit-text-fill-color: transparent;
+                margin-bottom: 0.5rem;
+            ">
+                🔒 LLM PII Detection Leaderboard
+            </h1>
+            
+            <p style="
+                color: var(--text-secondary);
+                font-size: 1.25rem;
+                line-height: 1.75;
+                max-width: 800px;
+                margin: 0 auto;
+                text-align: center;
+            ">
+                Comprehensive benchmark for language models' performance in detecting and redacting 
+                personally identifiable information (PII) across various document types and scenarios.
+                <span style="
+                    background: linear-gradient(to right, var(--accent-tertiary), var(--accent-quaternary));
+                    -webkit-background-clip: text;
+                    -webkit-text-fill-color: transparent;
+                    display: block;
+                    margin-top: 1rem;
+                    font-size: 1.5rem;
+                    font-weight: 500;
+                ">
+                    "How well do LLMs protect sensitive information?"
+                </span>
+            </p>
+        </div>
+        
+        <div style="
+            display: grid;
+            grid-template-columns: repeat(3, 1fr);
+            gap: 1.5rem;
+            margin-top: 4rem;
+        ">
+            <div style="
+                background: var(--bg-secondary);
+                border: 1px solid var(--border-subtle);
+                border-radius: 1rem;
+                padding: 2rem;
+                transition: all 0.3s ease;
+                text-align: center;
+            ">
+                <div style="
+                    font-size: 4rem;
+                    font-weight: 800;
+                    margin-bottom: 1rem;
+                    background: linear-gradient(45deg, var(--accent-primary), var(--accent-secondary));
+                    -webkit-background-clip: text;
+                    -webkit-text-fill-color: transparent;
+                ">8</div>
+                <div style="color: var(--text-secondary); font-size: 1.5rem; margin-bottom: 1.5rem;">
+                    Language Models
+                </div>
+                <div style="font-size: 1.125rem; line-height: 1.75; color: var(--text-primary);">
+                    Leading proprietary & open source
+                </div>
+                <div style="color: var(--text-secondary); margin-top: 0.5rem;">
+                    GPT-4o, Claude, Gemini, LLaMA, Mistral
+                </div>
+            </div>
+
+            <div style="
+                background: var(--bg-secondary);
+                border: 1px solid var(--border-subtle);
+                border-radius: 1rem;
+                padding: 2rem;
+                transition: all 0.3s ease;
+                text-align: center;
+            ">
+                <div style="
+                    font-size: 4rem;
+                    font-weight: 800;
+                    margin-bottom: 1rem;
+                    background: linear-gradient(45deg, var(--accent-tertiary), var(--accent-quaternary));
+                    -webkit-background-clip: text;
+                    -webkit-text-fill-color: transparent;
+                ">5</div>
+                <div style="color: var(--text-secondary); font-size: 1.5rem; margin-bottom: 1.5rem;">
+                    Document Types
+                </div>
+                <div style="font-size: 1.125rem; line-height: 1.75; color: var(--text-primary);">
+                    Real-world scenarios
+                </div>
+                <div style="color: var(--text-secondary); margin-top: 0.5rem;">
+                    Healthcare, Financial, Government, Legal, Personal
+                </div>
+            </div>
+
+            <div style="
+                background: var(--bg-secondary);
+                border: 1px solid var(--border-subtle);
+                border-radius: 1rem;
+                padding: 2rem;
+                transition: all 0.3s ease;
+                text-align: center;
+            ">
+                <div style="
+                    font-size: 4rem;
+                    font-weight: 800;
+                    margin-bottom: 1rem;
+                    background: linear-gradient(45deg, var(--accent-secondary), var(--accent-primary));
+                    -webkit-background-clip: text;
+                    -webkit-text-fill-color: transparent;
+                ">94.1%</div>
+                <div style="color: var(--text-secondary); font-size: 1.5rem; margin-bottom: 1.5rem;">
+                    Best Accuracy
+                </div>
+                <div style="font-size: 1.125rem; line-height: 1.75; color: var(--text-primary);">
+                    State-of-the-art performance
+                </div>
+                <div style="color: var(--text-secondary); margin-top: 0.5rem;">
+                    GPT-4o leading precision & recall
+                </div>
+            </div>
+        </div>
+    </div>
+</div>
+"""
+
+# Methodology section adapted for PII detection
+METHODOLOGY = """
+<div style="max-width: 1200px; margin: 0 auto; padding: 2rem; color: var(--text-secondary); line-height: 1.7; font-size: 1rem;">
+    <h1 style="font-size: 2.5rem; font-weight: 700; margin: 3rem 0 1.5rem; color: var(--text-primary); 
+               background: linear-gradient(to right, var(--accent-primary), var(--accent-secondary)); 
+               -webkit-background-clip: text; -webkit-text-fill-color: transparent;">
+        Methodology
+    </h1>
+    
+    <p>Our evaluation methodology assesses language models' capabilities in detecting and handling personally identifiable information (PII) across realistic document scenarios. Each model is tested on synthetic documents containing embedded PII entities across 5 document categories.</p>
+
+    <h2 style="font-size: 1.75rem; font-weight: 600; margin: 2rem 0 1rem; color: var(--text-primary);">
+        Evaluation Process
+    </h2>
+    
+    <ul style="list-style: none; padding: 0; margin: 1rem 0;">
+        <li style="padding-left: 2rem; position: relative; margin: 1rem 0; display: flex; align-items: flex-start;">
+            <span style="content: ''; position: absolute; left: 0; top: 0.75rem; width: 8px; height: 8px; 
+                         background: var(--accent-primary); border-radius: 50%; 
+                         box-shadow: 0 0 0 2px rgba(222, 157, 204, 0.2);"></span>
+            <span style="color: var(--accent-primary); font-weight: 600;">Model Selection:</span> 
+            We evaluate leading language models across proprietary and open-source categories
+        </li>
+        <li style="padding-left: 2rem; position: relative; margin: 1rem 0; display: flex; align-items: flex-start;">
+            <span style="content: ''; position: absolute; left: 0; top: 0.75rem; width: 8px; height: 8px; 
+                         background: var(--accent-primary); border-radius: 50%; 
+                         box-shadow: 0 0 0 2px rgba(222, 157, 204, 0.2);"></span>
+            <span style="color: var(--accent-primary); font-weight: 600;">PII Detection:</span> 
+            Each model processes documents with instructions to identify and classify PII entities
+        </li>
+        <li style="padding-left: 2rem; position: relative; margin: 1rem 0; display: flex; align-items: flex-start;">
+            <span style="content: ''; position: absolute; left: 0; top: 0.75rem; width: 8px; height: 8px; 
+                         background: var(--accent-primary); border-radius: 50%; 
+                         box-shadow: 0 0 0 2px rgba(222, 157, 204, 0.2);"></span>
+            <span style="color: var(--accent-primary); font-weight: 600;">Performance Metrics:</span> 
+            Precision, Recall, F1 Score, Over-detection Rate, Processing Time, and Cost
+        </li>
+        <li style="padding-left: 2rem; position: relative; margin: 1rem 0; display: flex; align-items: flex-start;">
+            <span style="content: ''; position: absolute; left: 0; top: 0.75rem; width: 8px; height: 8px; 
+                         background: var(--accent-primary); border-radius: 50%; 
+                         box-shadow: 0 0 0 2px rgba(222, 157, 204, 0.2);"></span>
+            <span style="color: var(--accent-primary); font-weight: 600;">Domain Analysis:</span> 
+            Specialized evaluation across Healthcare, Financial, Government, Legal, and Personal documents
+        </li>
+    </ul>
+
+    <h2 style="font-size: 1.75rem; font-weight: 600; margin: 2rem 0 1rem; color: var(--text-primary);">
+        Key Metrics Explained
+    </h2>
+    
+    <div style="background: var(--bg-secondary); border: 1px solid var(--border-subtle); border-radius: 12px; padding: 1.5rem; margin: 1.5rem 0;">
+        <ul style="list-style: none; padding: 0; margin: 0;">
+            <li style="margin: 1rem 0;"><span style="color: var(--accent-tertiary); font-weight: 600;">Overall Accuracy:</span> Percentage of correctly identified and classified PII entities</li>
+            <li style="margin: 1rem 0;"><span style="color: var(--accent-tertiary); font-weight: 600;">Precision:</span> Of all flagged items, how many were actually PII (avoiding false positives)</li>
+            <li style="margin: 1rem 0;"><span style="color: var(--accent-tertiary); font-weight: 600;">Recall:</span> Of all PII present, how many were successfully detected (avoiding false negatives)</li>
+            <li style="margin: 1rem 0;"><span style="color: var(--accent-tertiary); font-weight: 600;">F1 Score:</span> Harmonic mean balancing precision and recall</li>
+            <li style="margin: 1rem 0;"><span style="color: var(--accent-secondary); font-weight: 600;">Over-detection Rate:</span> Percentage of non-PII incorrectly flagged (lower is better)</li>
+        </ul>
+    </div>
+</div>
+""" 

pii_leaderboard.py

@@ -0,0 +1,976 @@
+import gradio as gr
+import pandas as pd
+import tempfile
+import os
+from data_loader import (
+    load_data,
+    PII_CATEGORIES,
+    HEADER_CONTENT,
+    METHODOLOGY,
+    COLORS,
+    MODEL_TYPES
+)
+
+def get_rank_badge(rank):
+    """Generate HTML for rank badge with appropriate styling"""
+    badge_styles = {
+        1: ("1st", f"linear-gradient(145deg, {COLORS['digital_pollen']}, {COLORS['digital_pollen']})", COLORS['warm_black']),
+        2: ("2nd", f"linear-gradient(145deg, {COLORS['soft_grey']}, {COLORS['warm_grey']})", COLORS['white']),
+        3: ("3rd", f"linear-gradient(145deg, {COLORS['code_coral']}, {COLORS['code_coral_dm']})", COLORS['white']),
+    }
+
+    if rank in badge_styles:
+        label, gradient, text_color = badge_styles[rank]
+        return f"""
+            <div style="
+                display: inline-flex;
+                align-items: center;
+                justify-content: center;
+                min-width: 48px;
+                padding: 4px 12px;
+                background: {gradient};
+                color: {text_color};
+                border-radius: 6px;
+                font-weight: 600;
+                font-size: 0.9em;
+                box-shadow: 0 2px 4px rgba(0, 0, 0, 0.2);
+            ">
+                {label}
+            </div>
+        """
+    return f"""
+        <div style="
+            display: inline-flex;
+            align-items: center;
+            justify-content: center;
+            min-width: 28px;
+            color: var(--text-secondary);
+            font-weight: 500;
+        ">
+            {rank}
+        </div>
+    """
+
+def get_type_badge(model_type):
+    """Generate HTML for model type badge"""
+    bg_color = COLORS['disc_pink'] if model_type == 'Proprietary' else COLORS['data_green']
+    return f"""
+        <div style="
+            display: inline-flex;
+            align-items: center;
+            padding: 4px 8px;
+            background: {bg_color};
+            color: white;
+            border-radius: 4px;
+            font-size: 0.85em;
+            font-weight: 500;
+        ">
+            {model_type}
+        </div>
+    """
+
+def get_score_bar(score, is_inverse=False):
+    """Generate HTML for score bar with gradient styling"""
+    if pd.isna(score) or score == '':
+        score = 0
+    else:
+        score = float(score)
+    
+    width = score * 100
+    
+    # For over-detection rate, use inverse coloring (lower is better)
+    if is_inverse:
+        gradient = f"linear-gradient(90deg, {COLORS['data_green']}, {COLORS['code_coral']})"
+    else:
+        gradient = f"linear-gradient(90deg, {COLORS['code_coral']}, {COLORS['data_green']})"
+    
+    return f"""
+        <div style="display: flex; align-items: center; gap: 12px; width: 100%;">
+            <div style="
+                flex-grow: 1;
+                height: 8px;
+                background: rgba(239, 235, 231, 0.1);
+                border-radius: 4px;
+                overflow: hidden;
+                max-width: 200px;
+            ">
+                <div style="
+                    width: {width}%;
+                    height: 100%;
+                    background: {gradient};
+                    border-radius: 4px;
+                    transition: width 0.3s ease;
+                "></div>
+            </div>
+            <span style="
+                font-family: 'SF Mono', monospace;
+                font-weight: 600;
+                color: var(--text-primary);
+                min-width: 60px;
+            ">{score:.3f}</span>
+        </div>
+    """
+
+def create_pii_leaderboard():
+    """Create the main PII detection leaderboard interface"""
+    
+    def load_leaderboard_data():
+        """Load and prepare the leaderboard data"""
+        return load_data()
+    
+    def generate_html_table(filtered_df, document_type, sort_by):
+        """Generate styled HTML table with rank badges and score bars"""
+        table_html = """
+        <div class="v2-table-container">
+            <table class="v2-styled-table">
+                <thead>
+                    <tr>
+                        <th style="width: 80px;">Rank</th>
+                        <th>Model</th>
+                        <th style="width: 120px;">Type</th>
+                        <th>Vendor</th>
+                        <th style="width: 200px;">Overall Accuracy</th>
+                        <th style="width: 150px;">Precision</th>
+                        <th style="width: 150px;">Recall</th>
+                        <th style="width: 150px;">F1 Score</th>
+                        <th style="width: 160px;">Over-detection Rate</th>
+                        <th>Cost/Doc ($)</th>
+                        <th>Time (s)</th>
+                    </tr>
+                </thead>
+                <tbody>
+        """
+        
+        # Generate table rows
+        for idx, (_, row) in enumerate(filtered_df.iterrows()):
+            rank = idx + 1
+            table_html += f"""
+                <tr>
+                    <td>{get_rank_badge(rank)}</td>
+                    <td class="model-name">{row['Model']}</td>
+                    <td>{get_type_badge(row['Model Type'])}</td>
+                    <td>{row['Vendor']}</td>
+            """
+            
+            # Get appropriate values based on document type filter
+            if document_type != "All":
+                # For specific document type, show domain-specific scores
+                accuracy_col = f'{document_type} Accuracy'
+                accuracy = row.get(accuracy_col, row.get('Overall Accuracy', ''))
+            else:
+                # For "All", show overall accuracy
+                accuracy = row.get('Overall Accuracy', '')
+            
+            precision = row.get('Precision', '')
+            recall = row.get('Recall', '')
+            f1 = row.get('F1 Score', '')
+            over_detection = row.get('Over-redaction Rate', '')
+            cost = row.get('Cost per Document ($)', '')
+            time = row.get('Processing Time (s)', '')
+            
+            # Add score bars
+            if accuracy != '':
+                table_html += f'<td class="score-cell">{get_score_bar(accuracy)}</td>'
+            else:
+                table_html += '<td class="numeric-cell">-</td>'
+                
+            if precision != '':
+                table_html += f'<td class="score-cell">{get_score_bar(precision)}</td>'
+            else:
+                table_html += '<td class="numeric-cell">-</td>'
+                
+            if recall != '':
+                table_html += f'<td class="score-cell">{get_score_bar(recall)}</td>'
+            else:
+                table_html += '<td class="numeric-cell">-</td>'
+                
+            if f1 != '':
+                table_html += f'<td class="score-cell">{get_score_bar(f1)}</td>'
+            else:
+                table_html += '<td class="numeric-cell">-</td>'
+                
+            if over_detection != '':
+                table_html += f'<td class="score-cell">{get_score_bar(over_detection, is_inverse=True)}</td>'
+            else:
+                table_html += '<td class="numeric-cell">-</td>'
+            
+            # Format cost and time
+            if cost != '':
+                cost_display = f'${float(cost):.3f}'
+            else:
+                cost_display = '-'
+                
+            if time != '':
+                time_display = f'{float(time):.1f}'
+            else:
+                time_display = '-'
+            
+            table_html += f"""
+                    <td class="numeric-cell">{cost_display}</td>
+                    <td class="numeric-cell">{time_display}</td>
+                </tr>
+            """
+        
+        table_html += """
+                </tbody>
+            </table>
+        </div>
+        """
+        
+        return table_html
+    
+    def filter_and_sort_data(document_type, model_type_filter, sort_by, sort_order):
+        """Filter and sort the leaderboard data"""
+        df = load_leaderboard_data()
+        filtered_df = df.copy()
+        
+        # Document type filtering
+        if document_type != "All":
+            # Only show models that have data for this document type
+            doc_col = f'{document_type} Accuracy'
+            if doc_col in filtered_df.columns:
+                filtered_df = filtered_df[filtered_df[doc_col] != '']
+        
+        # Model type filtering
+        if model_type_filter != "All":
+            if model_type_filter == "Open Source":
+                filtered_df = filtered_df[filtered_df['Model Type'] == 'Open Source']
+            elif model_type_filter == "Proprietary":
+                filtered_df = filtered_df[filtered_df['Model Type'] == 'Proprietary']
+        
+        # Sorting
+        sort_column = sort_by
+        if document_type != "All" and sort_by == 'Overall Accuracy':
+            sort_column = f'{document_type} Accuracy'
+        
+        if sort_column in filtered_df.columns:
+            ascending = (sort_order == "Ascending")
+            # For over-detection rate, flip the logic (lower is better)
+            if sort_by == "Over-redaction Rate":
+                ascending = not ascending
+            filtered_df = filtered_df.sort_values(by=sort_column, ascending=ascending, na_position='last')
+        
+        return generate_html_table(filtered_df, document_type, sort_by)
+    
+    def generate_performance_card(model_name):
+        """Generate HTML for the model performance card"""
+        if not model_name:
+            return """<div style="text-align: center; color: var(--text-secondary); padding: 40px;">
+                Please select a model to generate its performance card
+            </div>"""
+        
+        df = load_leaderboard_data()
+        model_data = df[df['Model'] == model_name]
+        
+        if model_data.empty:
+            return """<div style="text-align: center; color: var(--text-secondary); padding: 40px;">
+                Model not found in the database
+            </div>"""
+        
+        row = model_data.iloc[0]
+        
+        # Get overall rank
+        df_with_accuracy = df[df['Overall Accuracy'] != ''].copy()
+        df_with_accuracy['Overall Accuracy'] = pd.to_numeric(df_with_accuracy['Overall Accuracy'], errors='coerce')
+        df_sorted = df_with_accuracy.sort_values('Overall Accuracy', ascending=False).reset_index(drop=True)
+        try:
+            rank = df_sorted[df_sorted['Model'] == model_name].index[0] + 1
+        except:
+            rank = 'N/A'
+        
+        # Format values
+        def format_value(val, decimals=3, prefix='', suffix=''):
+            if pd.isna(val) or val == '':
+                return 'N/A'
+            return f"{prefix}{float(val):.{decimals}f}{suffix}"
+        
+        # Determine model type icon
+        type_icon = "🔓" if row['Model Type'] == 'Open Source' else "🔒"
+         
+        # Calculate performance stars
+        def get_performance_stars(value, max_val=1.0):
+            if pd.isna(value) or value == '':
+                return '⭐' * 0
+            score = float(value) / max_val
+            if score >= 0.9:
+                return '⭐' * 5
+            elif score >= 0.8:
+                return '⭐' * 4
+            elif score >= 0.7:
+                return '⭐' * 3
+            elif score >= 0.6:
+                return '⭐' * 2
+            else:
+                return '⭐' * 1
+         
+        # Create HTML
+        card_html = f"""
+        <div class="performance-card">
+            <div class="card-header">
+                <h1 class="card-model-name">{model_name}</h1>
+                <div class="card-stars">
+                    {get_performance_stars(row['Overall Accuracy'])}
+                </div>
+            </div>
+           
+           <div class="metrics-grid" style="margin-bottom: 24px;">
+               <div class="metric-item">
+                   <div class="metric-icon" style="color: var(--accent-primary);">🏆</div>
+                   <div class="metric-label">Overall Rank</div>
+                   <div class="metric-value">#{rank}</div>
+               </div>
+               
+               <div class="metric-item">
+                   <div class="metric-icon" style="color: var(--accent-primary);">🎯</div>
+                   <div class="metric-label">Overall Accuracy</div>
+                   <div class="metric-value">{format_value(row['Overall Accuracy'])}</div>
+               </div>
+               
+               <div class="metric-item">
+                   <div class="metric-icon" style="color: var(--accent-secondary);">📊</div>
+                   <div class="metric-label">Precision</div>
+                   <div class="metric-value">{format_value(row['Precision'])}</div>
+               </div>
+               
+               <div class="metric-item">
+                   <div class="metric-icon" style="color: var(--accent-tertiary);">🔍</div>
+                   <div class="metric-label">Recall</div>
+                   <div class="metric-value">{format_value(row['Recall'])}</div>
+               </div>
+               
+               <div class="metric-item">
+                   <div class="metric-icon" style="color: var(--accent-quaternary);">💰</div>
+                   <div class="metric-label">Cost/Doc</div>
+                   <div class="metric-value">{format_value(row['Cost per Document ($)'], 3, '$')}</div>
+               </div>
+               
+               <div class="metric-item">
+                   <div class="metric-icon" style="color: var(--text-primary);">⚡</div>
+                   <div class="metric-label">Processing Time</div>
+                   <div class="metric-value">{format_value(row['Processing Time (s)'], 1, '', 's')}</div>
+               </div>
+           </div>
+           
+           <div class="domains-section" style="margin-top: 24px;">
+               <h3 class="domains-title">📄 Document Type Performance</h3>
+               <div class="domains-grid">
+        """
+        
+        # Add document type scores
+        doc_types = [
+            ('🏥', 'Healthcare'),
+            ('💰', 'Financial'),
+            ('🏛️', 'Government'),
+            ('⚖️', 'Legal'),
+            ('👤', 'Personal')
+        ]
+        
+        for doc_icon, doc_type in doc_types:
+            accuracy_col = f'{doc_type} Accuracy'
+            accuracy_value = row.get(accuracy_col, '')
+            
+            if accuracy_value != '' and not pd.isna(accuracy_value):
+                score_display = f"{float(accuracy_value):.3f}"
+                score_color = "var(--accent-primary)"
+            else:
+                score_display = "N/A"
+                score_color = "var(--text-muted)"
+            
+            card_html += f"""
+                <div class="domain-item">
+                    <div class="domain-name">{doc_icon}</div>
+                    <div style="font-size: 0.7rem; color: var(--text-secondary); margin-bottom: 2px;">{doc_type}</div>
+                    <div class="domain-score" style="color: {score_color};">{score_display}</div>
+                </div>
+            """
+        
+        card_html += f"""
+                </div>
+            </div>
+            
+            <div class="card-footer">
+                <div class="card-url">
+                    <strong>LLM PII Detection Leaderboard</strong>
+                </div>
+            </div>
+        </div>
+        """
+        
+        return card_html
+    
+    def download_performance_card(model_name):
+        """Generate and return downloadable HTML file for the model performance card"""
+        if not model_name:
+            return None
+            
+        card_html = generate_performance_card(model_name)
+        
+        # Create a complete HTML document
+        full_html = f"""
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>{model_name} - Performance Card</title>
+    <style>
+        @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&display=swap');
+        
+        :root {{
+            --bg-primary: #1A1414;
+            --bg-secondary: rgba(239, 235, 231, 0.03);
+            --bg-card: rgba(239, 235, 231, 0.02);
+            --border-subtle: rgba(239, 235, 231, 0.08);
+            --text-primary: #EFEBE7;
+            --text-secondary: #C2B8AE;
+            --text-muted: #67594B;
+            --accent-primary: #DE9DCC;
+            --accent-secondary: #F25E45;
+            --accent-tertiary: #6EB579;
+            --accent-quaternary: #F0C968;
+            --glow-primary: rgba(222, 157, 204, 0.4);
+        }}
+        
+        body {{
+            margin: 0;
+            padding: 40px;
+            background: var(--bg-primary);
+            font-family: 'Inter', sans-serif;
+            color: var(--text-primary);
+        }}
+        
+        .performance-card {{
+            background: linear-gradient(145deg, rgba(26, 20, 20, 0.98) 0%, rgba(222, 157, 204, 0.05) 100%);
+            border: 2px solid var(--accent-primary);
+            border-radius: 24px;
+            padding: 32px;
+            max-width: 700px;
+            margin: 0 auto;
+            box-shadow: 
+                0 20px 40px rgba(0, 0, 0, 0.5),
+                0 0 80px rgba(222, 157, 204, 0.2);
+        }}
+        
+        .card-header {{
+            text-align: center;
+            margin-bottom: 24px;
+        }}
+        
+        .card-model-name {{
+            font-size: 2rem;
+            font-weight: 800;
+            background: linear-gradient(135deg, var(--accent-primary) 0%, var(--accent-secondary) 100%);
+            -webkit-background-clip: text;
+            -webkit-text-fill-color: transparent;
+            margin-bottom: 8px;
+            line-height: 1.2;
+        }}
+        
+        .card-stars {{
+            font-size: 1.2rem;
+            margin: 8px 0;
+        }}
+        
+        .metrics-grid {{
+            display: grid;
+            grid-template-columns: repeat(2, 1fr);
+            gap: 16px;
+            margin: 24px 0;
+        }}
+        
+        .metric-item {{
+            display: flex;
+            flex-direction: column;
+            align-items: center;
+            padding: 16px;
+            background: rgba(239, 235, 231, 0.05);
+            border-radius: 12px;
+            border: 1px solid var(--border-subtle);
+        }}
+        
+        .metric-icon {{
+            font-size: 1.5rem;
+            margin-bottom: 8px;
+        }}
+        
+        .metric-label {{
+            font-size: 0.85rem;
+            color: var(--text-secondary);
+            margin-bottom: 4px;
+            text-align: center;
+        }}
+        
+        .metric-value {{
+            font-size: 1.1rem;
+            font-weight: 700;
+            color: var(--text-primary);
+            text-align: center;
+        }}
+        
+        .domains-section {{
+            margin-top: 24px;
+        }}
+        
+        .domains-title {{
+            color: var(--text-primary);
+            font-size: 1.2rem;
+            margin-bottom: 16px;
+            text-align: center;
+        }}
+        
+        .domains-grid {{
+            display: grid;
+            grid-template-columns: repeat(5, 1fr);
+            gap: 12px;
+        }}
+        
+        .domain-item {{
+            display: flex;
+            flex-direction: column;
+            align-items: center;
+            padding: 12px;
+            background: rgba(239, 235, 231, 0.03);
+            border-radius: 8px;
+            border: 1px solid var(--border-subtle);
+        }}
+        
+        .domain-name {{
+            font-size: 1.2rem;
+            margin-bottom: 4px;
+        }}
+        
+        .domain-score {{
+            font-size: 0.9rem;
+            font-weight: 600;
+        }}
+        
+        .card-footer {{
+            text-align: center;
+            margin-top: 24px;
+            padding-top: 16px;
+            border-top: 1px solid var(--border-subtle);
+        }}
+        
+        .card-url {{
+            color: var(--text-secondary);
+            font-size: 0.9rem;
+        }}
+    </style>
+</head>
+<body>
+    {card_html}
+</body>
+</html>
+"""
+        
+        # Create a temporary file
+        with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix=f'_{model_name.replace(" ", "_")}_performance_card.html', encoding='utf-8') as f:
+            f.write(full_html)
+            return f.name
+    
+    # Load initial data
+    initial_df = load_leaderboard_data()
+    initial_table = filter_and_sort_data("All", "All", "Overall Accuracy", "Descending")
+    
+    # Display header
+    gr.HTML(HEADER_CONTENT)
+    
+    # Document type filter section
+    gr.HTML("""
+    <div class="dark-container" style="margin-bottom: 32px;">
+        <div class="section-header">
+            <span class="section-icon" style="color: var(--accent-primary);">📄</span>
+            <h3 style="margin: 0; color: var(--text-primary); font-size: 1.5rem; font-family: 'Inter', sans-serif; font-weight: 700;">
+                Document Type Analysis
+            </h3>
+        </div>
+        <p style="color: var(--text-secondary); margin-bottom: 20px; font-size: 1.1rem; font-family: 'Inter', sans-serif;">
+            Select a document type to see specialized PII detection performance
+        </p>
+    """)
+    
+    with gr.Row():
+        document_type_filter = gr.Radio(
+            choices=["All", "Healthcare", "Financial", "Government", "Legal", "Personal"],
+            value="All",
+            label="",
+            interactive=True,
+            elem_classes=["document-type-radio"]
+        )
+    
+    gr.HTML("</div>")
+    
+    # Filter controls
+    gr.HTML("""
+    <div class="dark-container" style="margin-bottom: 24px;">
+        <div class="section-header">
+            <span class="section-icon" style="color: var(--accent-secondary);">🔍</span>
+            <h3 style="margin: 0; color: var(--text-primary); font-size: 1.5rem; font-family: 'Inter', sans-serif; font-weight: 700;">
+                Filters & Sorting
+            </h3>
+        </div>
+    """)
+    
+    with gr.Row():
+        with gr.Column(scale=1):
+            model_type_filter = gr.Radio(
+                choices=["All", "Open Source", "Proprietary"],
+                value="All", 
+                label="🔓 Model Access",
+                elem_classes=["compact-radio"]
+            )
+        
+        with gr.Column(scale=1):
+            sort_by = gr.Dropdown(
+                choices=["Overall Accuracy", "Precision", "Recall", "F1 Score", "Over-redaction Rate", "Cost per Document ($)", "Processing Time (s)"],
+                value="Overall Accuracy",
+                label="📊 Sort By",
+                elem_classes=["dropdown"]
+            )
+        
+        with gr.Column(scale=1):
+            sort_order = gr.Radio(
+                choices=["Descending", "Ascending"],
+                value="Descending",
+                label="🔄 Sort Order",
+                elem_classes=["compact-radio"]
+            )
+    
+    gr.HTML("</div>")
+    
+    # Main leaderboard table
+    gr.HTML("""
+    <div class="dark-container" style="margin-bottom: 24px;">
+        <div class="section-header">
+            <span class="section-icon" style="color: var(--accent-primary);">📈</span>
+            <h3 style="margin: 0; color: var(--text-primary); font-size: 1.5rem; font-family: 'Inter', sans-serif; font-weight: 700;">
+                PII Detection Performance Leaderboard
+            </h3>
+        </div>
+        <div class="dataframe-container">
+    """)
+    
+    leaderboard_table = gr.HTML(initial_table)
+    
+    gr.HTML("""
+        </div>
+    </div>""")
+    
+    # Performance Card Section
+    gr.HTML("""
+    <div class="dark-container" style="margin-top: 32px;">
+        <div class="section-header">
+            <span class="section-icon" style="color: var(--accent-primary);">🎯</span>
+            <h3 style="margin: 0; color: var(--text-primary); font-size: 1.5rem; font-family: 'Inter', sans-serif; font-weight: 700;">
+                Model Performance Card
+            </h3>
+        </div>
+        <p style="color: var(--text-secondary); margin-bottom: 20px; font-size: 1.1rem; font-family: 'Inter', sans-serif;">
+            Comprehensive performance card for any model - perfect for presentations and reports
+        </p>
+        
+        <div style="display: flex; gap: 24px; align-items: flex-start;">
+            <div style="flex: 0 0 280px;">
+                <div style="background: rgba(239, 235, 231, 0.03); border: 1px solid var(--border-subtle); 
+                            border-radius: 16px; padding: 20px; position: sticky; top: 20px;">
+    """)
+    
+    card_model_selector = gr.Dropdown(
+        choices=initial_df['Model'].tolist(),
+        value=initial_df['Model'].tolist()[0] if len(initial_df) > 0 else None,
+        label="🤖 Select Model",
+        info="Choose a model to view its performance card",
+        elem_classes=["dropdown"]
+    )
+    
+    gr.HTML("""
+                </div>
+            </div>
+            
+            <div style="flex: 1; min-width: 0;" id="card-display-container">
+    """)
+    
+    # Card display area
+    initial_model = initial_df['Model'].tolist()[0] if len(initial_df) > 0 else None
+    initial_card_html = generate_performance_card(initial_model) if initial_model else ""
+    card_display = gr.HTML(value=initial_card_html, elem_id="performance-card-html")
+    
+    # Download button below the card
+    gr.HTML("""
+        <div style="margin-top: 24px; text-align: center;">
+    """)
+    
+    download_button = gr.DownloadButton(
+        label="📥 Download Performance Card",
+        value=None,
+        variant="primary",
+        elem_classes=["download-card-btn"]
+    )
+    
+    gr.HTML("""
+        </div>
+            </div>
+        </div>
+    </div>""")
+    
+    # Add performance card CSS
+    gr.HTML(f"""
+    <style>
+    .performance-card {{
+        background: linear-gradient(145deg, rgba(26, 20, 20, 0.98) 0%, rgba(222, 157, 204, 0.05) 100%);
+        border: 2px solid var(--accent-primary);
+        border-radius: 24px;
+        padding: 32px;
+        max-width: 700px;
+        margin: 0 auto;
+        position: relative;
+        overflow: hidden;
+        box-shadow: 
+            0 20px 40px rgba(0, 0, 0, 0.5),
+            0 0 80px rgba(222, 157, 204, 0.2),
+            inset 0 0 120px rgba(222, 157, 204, 0.05);
+    }}
+    
+    .card-header {{
+        text-align: center;
+        margin-bottom: 24px;
+        position: relative;
+        z-index: 1;
+    }}
+    
+    .card-model-name {{
+        font-size: 2rem;
+        font-weight: 800;
+        background: linear-gradient(135deg, var(--accent-primary) 0%, var(--accent-secondary) 100%);
+        -webkit-background-clip: text;
+        -webkit-text-fill-color: transparent;
+        margin-bottom: 8px;
+        text-shadow: 0 0 40px var(--glow-primary);
+        line-height: 1.2;
+    }}
+    
+    .card-stars {{
+        font-size: 1.2rem;
+        margin: 8px 0;
+    }}
+    
+    .metrics-grid {{
+        display: grid;
+        grid-template-columns: repeat(2, 1fr);
+        gap: 16px;
+        margin: 24px 0;
+    }}
+    
+    .metric-item {{
+        display: flex;
+        flex-direction: column;
+        align-items: center;
+        padding: 16px;
+        background: rgba(239, 235, 231, 0.05);
+        border-radius: 12px;
+        border: 1px solid var(--border-subtle);
+        transition: all 0.3s ease;
+    }}
+    
+    .metric-item:hover {{
+        transform: translateY(-2px);
+        border-color: var(--accent-primary);
+        box-shadow: 0 8px 16px rgba(222, 157, 204, 0.3);
+    }}
+    
+    .metric-icon {{
+        font-size: 1.5rem;
+        margin-bottom: 8px;
+    }}
+    
+    .metric-label {{
+        font-size: 0.85rem;
+        color: var(--text-secondary);
+        margin-bottom: 4px;
+        text-align: center;
+    }}
+    
+    .metric-value {{
+        font-size: 1.1rem;
+        font-weight: 700;
+        color: var(--text-primary);
+        text-align: center;
+    }}
+    
+    .domains-section {{
+        margin-top: 24px;
+    }}
+    
+    .domains-title {{
+        color: var(--text-primary);
+        font-size: 1.2rem;
+        margin-bottom: 16px;
+        text-align: center;
+    }}
+    
+    .domains-grid {{
+        display: grid;
+        grid-template-columns: repeat(5, 1fr);
+        gap: 12px;
+    }}
+    
+    .domain-item {{
+        display: flex;
+        flex-direction: column;
+        align-items: center;
+        padding: 12px;
+        background: rgba(239, 235, 231, 0.03);
+        border-radius: 8px;
+        border: 1px solid var(--border-subtle);
+        transition: all 0.3s ease;
+    }}
+    
+    .domain-item:hover {{
+        border-color: var(--accent-primary);
+        transform: scale(1.02);
+    }}
+    
+    .domain-name {{
+        font-size: 1.2rem;
+        margin-bottom: 4px;
+    }}
+    
+    .domain-score {{
+        font-size: 0.9rem;
+        font-weight: 600;
+    }}
+    
+    .card-footer {{
+        text-align: center;
+        margin-top: 24px;
+        padding-top: 16px;
+        border-top: 1px solid var(--border-subtle);
+    }}
+    
+    .card-url {{
+        color: var(--text-secondary);
+        font-size: 0.9rem;
+    }}
+    
+    /* Additional styling for radio buttons and specific components */
+    .document-type-radio .wrap {{
+        display: flex !important;
+        gap: 12px !important;
+        flex-wrap: wrap !important;
+        justify-content: center !important;
+    }}
+    
+    .document-type-radio .wrap > label {{
+        flex: 1 !important;
+        min-width: 140px !important;
+        max-width: 180px !important;
+        padding: 12px 16px !important;
+        background: var(--bg-card) !important;
+        border: 2px solid var(--border-default) !important;
+        border-radius: 12px !important;
+        cursor: pointer !important;
+        transition: all 0.3s ease !important;
+        text-align: center !important;
+        font-weight: 500 !important;
+    }}
+    
+    .document-type-radio .wrap > label:hover {{
+        border-color: var(--accent-primary) !important;
+        transform: translateY(-2px) !important;
+    }}
+    
+    .document-type-radio .wrap > label:has(input[type="radio"]:checked) {{
+        background: transparent !important;
+        border-color: var(--accent-primary) !important;
+        color: var(--text-primary) !important;
+        font-weight: 600 !important;
+        box-shadow: 0 8px 16px var(--glow-primary) !important;
+    }}
+    
+    .document-type-radio input[type="radio"] {{
+        display: none !important;
+    }}
+    
+    .compact-radio .wrap > label {{
+        padding: 8px 12px !important;
+        font-size: 0.85rem !important;
+        min-width: auto !important;
+        max-width: 120px !important;
+    }}
+    
+    .download-card-btn {{
+        background: linear-gradient(135deg, var(--accent-primary), var(--accent-secondary)) !important;
+        color: white !important;
+        border: none !important;
+        padding: 12px 24px !important;
+        border-radius: 12px !important;
+        font-weight: 600 !important;
+        font-size: 0.95rem !important;
+        transition: all 0.3s ease !important;
+        box-shadow: 0 4px 16px rgba(222, 157, 204, 0.4) !important;
+    }}
+    
+    .download-card-btn:hover {{
+        transform: translateY(-2px) !important;
+        box-shadow: 0 6px 20px rgba(222, 157, 204, 0.6) !important;
+    }}
+    </style>
+    """)
+    
+    # Update functions
+    def update_table(*args):
+        return filter_and_sort_data(*args)
+    
+    def update_card(model_name):
+        return generate_performance_card(model_name)
+    
+    # Connect update functions to components
+    filter_inputs = [document_type_filter, model_type_filter, sort_by, sort_order]
+    
+    for input_component in filter_inputs:
+        input_component.change(
+            fn=update_table,
+            inputs=filter_inputs,
+            outputs=[leaderboard_table]
+        )
+    
+    # Update card when model selection changes
+    card_model_selector.change(
+        fn=update_card,
+        inputs=[card_model_selector],
+        outputs=[card_display]
+    )
+    
+    # Download card functionality
+    def update_download_button(model_name):
+        if model_name:
+            file_path = download_performance_card(model_name)
+            return file_path
+        return None
+    
+    card_model_selector.change(
+        fn=update_download_button,
+        inputs=[card_model_selector],
+        outputs=[download_button]
+    )
+    
+    # Methodology section
+    gr.HTML(f"""
+    <div class="dark-container" style="margin-top: 32px;">
+        {METHODOLOGY}
+    </div>
+    """)
+
+def create_app():
+    """Create the main Gradio application"""
+    with gr.Blocks(
+        theme=gr.themes.Default(),
+        title="🔒 LLM PII Detection Leaderboard"
+    ) as app:
+        create_pii_leaderboard()
+    
+    return app
+
+if __name__ == "__main__":
+    demo = create_app()
+    demo.launch() 

pyproject.toml

@@ -1,13 +0,0 @@
-[tool.ruff]
-# Enable pycodestyle (`E`) and Pyflakes (`F`) codes by default.
-select = ["E", "F"]
-ignore = ["E501"] # line too long (black is taking care of this)
-line-length = 119
-fixable = ["A", "B", "C", "D", "E", "F", "G", "I", "N", "Q", "S", "T", "W", "ANN", "ARG", "BLE", "COM", "DJ", "DTZ", "EM", "ERA", "EXE", "FBT", "ICN", "INP", "ISC", "NPY", "PD", "PGH", "PIE", "PL", "PT", "PTH", "PYI", "RET", "RSE", "RUF", "SIM", "SLF", "TCH", "TID", "TRY", "UP", "YTT"]
-
-[tool.isort]
-profile = "black"
-line_length = 119
-
-[tool.black]
-line-length = 119

requirements.txt

@@ -1,16 +1,3 @@
-APScheduler
-black
-datasets
 gradio
-gradio[oauth]
-gradio_leaderboard==0.0.13
-gradio_client
-huggingface-hub>=0.18.0
-matplotlib
-numpy
 pandas
-python-dateutil
-tqdm
-transformers
-tokenizers>=0.15.0
-sentencepiece
+numpy

results/pii_detection_results.csv

@@ -0,0 +1,9 @@
+Model,Model Type,Vendor,Overall Accuracy,Precision,Recall,F1 Score,Over-redaction Rate,Processing Time (s),Cost per Document ($),Healthcare Accuracy,Financial Accuracy,Government Accuracy,Legal Accuracy,Personal Accuracy
+GPT-4o,Proprietary,OpenAI,0.941,0.945,0.938,0.941,0.023,2.3,0.012,0.952,0.938,0.933,0.941,0.940
+Claude-3.5-Sonnet,Proprietary,Anthropic,0.928,0.932,0.924,0.928,0.031,3.1,0.015,0.939,0.925,0.920,0.928,0.927
+Gemini-1.5-Pro,Proprietary,Google,0.915,0.919,0.911,0.915,0.038,2.8,0.008,0.926,0.912,0.907,0.915,0.914
+LLaMA-3.1-70B,Open Source,Meta,0.882,0.887,0.877,0.882,0.052,4.2,0.003,0.893,0.879,0.874,0.882,0.881
+Mistral-Large,Proprietary,Mistral AI,0.871,0.875,0.867,0.871,0.048,3.7,0.011,0.882,0.868,0.863,0.871,0.870
+GPT-4o-mini,Proprietary,OpenAI,0.856,0.860,0.852,0.856,0.061,1.8,0.002,0.867,0.853,0.848,0.856,0.855
+Claude-3-Haiku,Proprietary,Anthropic,0.834,0.838,0.830,0.834,0.078,2.1,0.006,0.845,0.831,0.826,0.834,0.833
+Gemini-1.5-Flash,Proprietary,Google,0.821,0.825,0.817,0.821,0.085,2.4,0.004,0.832,0.818,0.813,0.821,0.820 

src/about.py

@@ -1,72 +0,0 @@
-from dataclasses import dataclass
-from enum import Enum
-
-@dataclass
-class Task:
-    benchmark: str
-    metric: str
-    col_name: str
-
-
-# Select your tasks here
-# ---------------------------------------------------
-class Tasks(Enum):
-    # task_key in the json file, metric_key in the json file, name to display in the leaderboard 
-    task0 = Task("anli_r1", "acc", "ANLI")
-    task1 = Task("logiqa", "acc_norm", "LogiQA")
-
-NUM_FEWSHOT = 0 # Change with your few shot
-# ---------------------------------------------------
-
-
-
-# Your leaderboard name
-TITLE = """<h1 align="center" id="space-title">Demo leaderboard</h1>"""
-
-# What does your leaderboard evaluate?
-INTRODUCTION_TEXT = """
-Intro text
-"""
-
-# Which evaluations are you running? how can people reproduce what you have?
-LLM_BENCHMARKS_TEXT = f"""
-## How it works
-
-## Reproducibility
-To reproduce our results, here is the commands you can run:
-
-"""
-
-EVALUATION_QUEUE_TEXT = """
-## Some good practices before submitting a model
-
-### 1) Make sure you can load your model and tokenizer using AutoClasses:
-```python
-from transformers import AutoConfig, AutoModel, AutoTokenizer
-config = AutoConfig.from_pretrained("your model name", revision=revision)
-model = AutoModel.from_pretrained("your model name", revision=revision)
-tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
-```
-If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.
-
-Note: make sure your model is public!
-Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted!
-
-### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
-It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`!
-
-### 3) Make sure your model has an open license!
-This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗
-
-### 4) Fill up your model card
-When we add extra information about models to the leaderboard, it will be automatically taken from the model card
-
-## In case of model failure
-If your model is displayed in the `FAILED` category, its execution stopped.
-Make sure you have followed the above steps first.
-If everything is done, check you can launch the EleutherAIHarness on your model locally, using the above command without modifications (you can add `--limit` to limit the number of examples per task).
-"""
-
-CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
-CITATION_BUTTON_TEXT = r"""
-"""

src/display/css_html_js.py

@@ -1,105 +0,0 @@
-custom_css = """
-
-.markdown-text {
-    font-size: 16px !important;
-}
-
-#models-to-add-text {
-    font-size: 18px !important;
-}
-
-#citation-button span {
-    font-size: 16px !important;
-}
-
-#citation-button textarea {
-    font-size: 16px !important;
-}
-
-#citation-button > label > button {
-    margin: 6px;
-    transform: scale(1.3);
-}
-
-#leaderboard-table {
-    margin-top: 15px
-}
-
-#leaderboard-table-lite {
-    margin-top: 15px
-}
-
-#search-bar-table-box > div:first-child {
-    background: none;
-    border: none;
-}
- 
-#search-bar {
-    padding: 0px;
-}
-
-/* Limit the width of the first AutoEvalColumn so that names don't expand too much */
-#leaderboard-table td:nth-child(2),
-#leaderboard-table th:nth-child(2) {
-    max-width: 400px;
-    overflow: auto;
-    white-space: nowrap;
-}
-
-.tab-buttons button {
-    font-size: 20px;
-}
-
-#scale-logo {
-    border-style: none !important;
-    box-shadow: none;
-    display: block;
-    margin-left: auto;
-    margin-right: auto;
-    max-width: 600px;
-}
-
-#scale-logo .download {
-    display: none;
-}
-#filter_type{
-    border: 0;
-    padding-left: 0;
-    padding-top: 0;
-}
-#filter_type label {
-    display: flex;
-}
-#filter_type label > span{
-    margin-top: var(--spacing-lg);
-    margin-right: 0.5em;
-}
-#filter_type label > .wrap{
-    width: 103px;
-}
-#filter_type label > .wrap .wrap-inner{  
-    padding: 2px;
-}
-#filter_type label > .wrap .wrap-inner input{
-    width: 1px
-}
-#filter-columns-type{
-    border:0;
-    padding:0.5;
-}
-#filter-columns-size{
-    border:0;
-    padding:0.5;
-}
-#box-filter > .form{
-    border: 0
-}
-"""
-
-get_window_url_params = """
-    function(url_params) {
-        const params = new URLSearchParams(window.location.search);
-        url_params = Object.fromEntries(params);
-        return url_params;
-    }
-    """

src/display/formatting.py

@@ -1,27 +0,0 @@
-def model_hyperlink(link, model_name):
-    return f'<a target="_blank" href="{link}" style="color: var(--link-text-color); text-decoration: underline;text-decoration-style: dotted;">{model_name}</a>'
-
-
-def make_clickable_model(model_name):
-    link = f"https://huggingface.co/{model_name}"
-    return model_hyperlink(link, model_name)
-
-
-def styled_error(error):
-    return f"<p style='color: red; font-size: 20px; text-align: center;'>{error}</p>"
-
-
-def styled_warning(warn):
-    return f"<p style='color: orange; font-size: 20px; text-align: center;'>{warn}</p>"
-
-
-def styled_message(message):
-    return f"<p style='color: green; font-size: 20px; text-align: center;'>{message}</p>"
-
-
-def has_no_nan_values(df, columns):
-    return df[columns].notna().all(axis=1)
-
-
-def has_nan_values(df, columns):
-    return df[columns].isna().any(axis=1)

src/display/utils.py

@@ -1,110 +0,0 @@
-from dataclasses import dataclass, make_dataclass
-from enum import Enum
-
-import pandas as pd
-
-from src.about import Tasks
-
-def fields(raw_class):
-    return [v for k, v in raw_class.__dict__.items() if k[:2] != "__" and k[-2:] != "__"]
-
-
-# These classes are for user facing column names,
-# to avoid having to change them all around the code
-# when a modif is needed
-@dataclass
-class ColumnContent:
-    name: str
-    type: str
-    displayed_by_default: bool
-    hidden: bool = False
-    never_hidden: bool = False
-
-## Leaderboard columns
-auto_eval_column_dict = []
-# Init
-auto_eval_column_dict.append(["model_type_symbol", ColumnContent, ColumnContent("T", "str", True, never_hidden=True)])
-auto_eval_column_dict.append(["model", ColumnContent, ColumnContent("Model", "markdown", True, never_hidden=True)])
-#Scores
-auto_eval_column_dict.append(["average", ColumnContent, ColumnContent("Average ⬆️", "number", True)])
-for task in Tasks:
-    auto_eval_column_dict.append([task.name, ColumnContent, ColumnContent(task.value.col_name, "number", True)])
-# Model information
-auto_eval_column_dict.append(["model_type", ColumnContent, ColumnContent("Type", "str", False)])
-auto_eval_column_dict.append(["architecture", ColumnContent, ColumnContent("Architecture", "str", False)])
-auto_eval_column_dict.append(["weight_type", ColumnContent, ColumnContent("Weight type", "str", False, True)])
-auto_eval_column_dict.append(["precision", ColumnContent, ColumnContent("Precision", "str", False)])
-auto_eval_column_dict.append(["license", ColumnContent, ColumnContent("Hub License", "str", False)])
-auto_eval_column_dict.append(["params", ColumnContent, ColumnContent("#Params (B)", "number", False)])
-auto_eval_column_dict.append(["likes", ColumnContent, ColumnContent("Hub ❤️", "number", False)])
-auto_eval_column_dict.append(["still_on_hub", ColumnContent, ColumnContent("Available on the hub", "bool", False)])
-auto_eval_column_dict.append(["revision", ColumnContent, ColumnContent("Model sha", "str", False, False)])
-
-# We use make dataclass to dynamically fill the scores from Tasks
-AutoEvalColumn = make_dataclass("AutoEvalColumn", auto_eval_column_dict, frozen=True)
-
-## For the queue columns in the submission tab
-@dataclass(frozen=True)
-class EvalQueueColumn:  # Queue column
-    model = ColumnContent("model", "markdown", True)
-    revision = ColumnContent("revision", "str", True)
-    private = ColumnContent("private", "bool", True)
-    precision = ColumnContent("precision", "str", True)
-    weight_type = ColumnContent("weight_type", "str", "Original")
-    status = ColumnContent("status", "str", True)
-
-## All the model information that we might need
-@dataclass
-class ModelDetails:
-    name: str
-    display_name: str = ""
-    symbol: str = "" # emoji
-
-
-class ModelType(Enum):
-    PT = ModelDetails(name="pretrained", symbol="🟢")
-    FT = ModelDetails(name="fine-tuned", symbol="🔶")
-    IFT = ModelDetails(name="instruction-tuned", symbol="⭕")
-    RL = ModelDetails(name="RL-tuned", symbol="🟦")
-    Unknown = ModelDetails(name="", symbol="?")
-
-    def to_str(self, separator=" "):
-        return f"{self.value.symbol}{separator}{self.value.name}"
-
-    @staticmethod
-    def from_str(type):
-        if "fine-tuned" in type or "🔶" in type:
-            return ModelType.FT
-        if "pretrained" in type or "🟢" in type:
-            return ModelType.PT
-        if "RL-tuned" in type or "🟦" in type:
-            return ModelType.RL
-        if "instruction-tuned" in type or "⭕" in type:
-            return ModelType.IFT
-        return ModelType.Unknown
-
-class WeightType(Enum):
-    Adapter = ModelDetails("Adapter")
-    Original = ModelDetails("Original")
-    Delta = ModelDetails("Delta")
-
-class Precision(Enum):
-    float16 = ModelDetails("float16")
-    bfloat16 = ModelDetails("bfloat16")
-    Unknown = ModelDetails("?")
-
-    def from_str(precision):
-        if precision in ["torch.float16", "float16"]:
-            return Precision.float16
-        if precision in ["torch.bfloat16", "bfloat16"]:
-            return Precision.bfloat16
-        return Precision.Unknown
-
-# Column selection
-COLS = [c.name for c in fields(AutoEvalColumn) if not c.hidden]
-
-EVAL_COLS = [c.name for c in fields(EvalQueueColumn)]
-EVAL_TYPES = [c.type for c in fields(EvalQueueColumn)]
-
-BENCHMARK_COLS = [t.value.col_name for t in Tasks]
-

src/envs.py

@@ -1,25 +0,0 @@
-import os
-
-from huggingface_hub import HfApi
-
-# Info to change for your repository
-# ----------------------------------
-TOKEN = os.environ.get("HF_TOKEN") # A read/write token for your org
-
-OWNER = "demo-leaderboard-backend" # Change to your org - don't forget to create a results and request dataset, with the correct format!
-# ----------------------------------
-
-REPO_ID = f"{OWNER}/leaderboard"
-QUEUE_REPO = f"{OWNER}/requests"
-RESULTS_REPO = f"{OWNER}/results"
-
-# If you setup a cache later, just change HF_HOME
-CACHE_PATH=os.getenv("HF_HOME", ".")
-
-# Local caches
-EVAL_REQUESTS_PATH = os.path.join(CACHE_PATH, "eval-queue")
-EVAL_RESULTS_PATH = os.path.join(CACHE_PATH, "eval-results")
-EVAL_REQUESTS_PATH_BACKEND = os.path.join(CACHE_PATH, "eval-queue-bk")
-EVAL_RESULTS_PATH_BACKEND = os.path.join(CACHE_PATH, "eval-results-bk")
-
-API = HfApi(token=TOKEN)

src/leaderboard/read_evals.py

@@ -1,196 +0,0 @@
-import glob
-import json
-import math
-import os
-from dataclasses import dataclass
-
-import dateutil
-import numpy as np
-
-from src.display.formatting import make_clickable_model
-from src.display.utils import AutoEvalColumn, ModelType, Tasks, Precision, WeightType
-from src.submission.check_validity import is_model_on_hub
-
-
-@dataclass
-class EvalResult:
-    """Represents one full evaluation. Built from a combination of the result and request file for a given run.
-    """
-    eval_name: str # org_model_precision (uid)
-    full_model: str # org/model (path on hub)
-    org: str 
-    model: str
-    revision: str # commit hash, "" if main
-    results: dict
-    precision: Precision = Precision.Unknown
-    model_type: ModelType = ModelType.Unknown # Pretrained, fine tuned, ...
-    weight_type: WeightType = WeightType.Original # Original or Adapter
-    architecture: str = "Unknown" 
-    license: str = "?"
-    likes: int = 0
-    num_params: int = 0
-    date: str = "" # submission date of request file
-    still_on_hub: bool = False
-
-    @classmethod
-    def init_from_json_file(self, json_filepath):
-        """Inits the result from the specific model result file"""
-        with open(json_filepath) as fp:
-            data = json.load(fp)
-
-        config = data.get("config")
-
-        # Precision
-        precision = Precision.from_str(config.get("model_dtype"))
-
-        # Get model and org
-        org_and_model = config.get("model_name", config.get("model_args", None))
-        org_and_model = org_and_model.split("/", 1)
-
-        if len(org_and_model) == 1:
-            org = None
-            model = org_and_model[0]
-            result_key = f"{model}_{precision.value.name}"
-        else:
-            org = org_and_model[0]
-            model = org_and_model[1]
-            result_key = f"{org}_{model}_{precision.value.name}"
-        full_model = "/".join(org_and_model)
-
-        still_on_hub, _, model_config = is_model_on_hub(
-            full_model, config.get("model_sha", "main"), trust_remote_code=True, test_tokenizer=False
-        )
-        architecture = "?"
-        if model_config is not None:
-            architectures = getattr(model_config, "architectures", None)
-            if architectures:
-                architecture = ";".join(architectures)
-
-        # Extract results available in this file (some results are split in several files)
-        results = {}
-        for task in Tasks:
-            task = task.value
-
-            # We average all scores of a given metric (not all metrics are present in all files)
-            accs = np.array([v.get(task.metric, None) for k, v in data["results"].items() if task.benchmark == k])
-            if accs.size == 0 or any([acc is None for acc in accs]):
-                continue
-
-            mean_acc = np.mean(accs) * 100.0
-            results[task.benchmark] = mean_acc
-
-        return self(
-            eval_name=result_key,
-            full_model=full_model,
-            org=org,
-            model=model,
-            results=results,
-            precision=precision,  
-            revision= config.get("model_sha", ""),
-            still_on_hub=still_on_hub,
-            architecture=architecture
-        )
-
-    def update_with_request_file(self, requests_path):
-        """Finds the relevant request file for the current model and updates info with it"""
-        request_file = get_request_file_for_model(requests_path, self.full_model, self.precision.value.name)
-
-        try:
-            with open(request_file, "r") as f:
-                request = json.load(f)
-            self.model_type = ModelType.from_str(request.get("model_type", ""))
-            self.weight_type = WeightType[request.get("weight_type", "Original")]
-            self.license = request.get("license", "?")
-            self.likes = request.get("likes", 0)
-            self.num_params = request.get("params", 0)
-            self.date = request.get("submitted_time", "")
-        except Exception:
-            print(f"Could not find request file for {self.org}/{self.model} with precision {self.precision.value.name}")
-
-    def to_dict(self):
-        """Converts the Eval Result to a dict compatible with our dataframe display"""
-        average = sum([v for v in self.results.values() if v is not None]) / len(Tasks)
-        data_dict = {
-            "eval_name": self.eval_name,  # not a column, just a save name,
-            AutoEvalColumn.precision.name: self.precision.value.name,
-            AutoEvalColumn.model_type.name: self.model_type.value.name,
-            AutoEvalColumn.model_type_symbol.name: self.model_type.value.symbol,
-            AutoEvalColumn.weight_type.name: self.weight_type.value.name,
-            AutoEvalColumn.architecture.name: self.architecture,
-            AutoEvalColumn.model.name: make_clickable_model(self.full_model),
-            AutoEvalColumn.revision.name: self.revision,
-            AutoEvalColumn.average.name: average,
-            AutoEvalColumn.license.name: self.license,
-            AutoEvalColumn.likes.name: self.likes,
-            AutoEvalColumn.params.name: self.num_params,
-            AutoEvalColumn.still_on_hub.name: self.still_on_hub,
-        }
-
-        for task in Tasks:
-            data_dict[task.value.col_name] = self.results[task.value.benchmark]
-
-        return data_dict
-
-
-def get_request_file_for_model(requests_path, model_name, precision):
-    """Selects the correct request file for a given model. Only keeps runs tagged as FINISHED"""
-    request_files = os.path.join(
-        requests_path,
-        f"{model_name}_eval_request_*.json",
-    )
-    request_files = glob.glob(request_files)
-
-    # Select correct request file (precision)
-    request_file = ""
-    request_files = sorted(request_files, reverse=True)
-    for tmp_request_file in request_files:
-        with open(tmp_request_file, "r") as f:
-            req_content = json.load(f)
-            if (
-                req_content["status"] in ["FINISHED"]
-                and req_content["precision"] == precision.split(".")[-1]
-            ):
-                request_file = tmp_request_file
-    return request_file
-
-
-def get_raw_eval_results(results_path: str, requests_path: str) -> list[EvalResult]:
-    """From the path of the results folder root, extract all needed info for results"""
-    model_result_filepaths = []
-
-    for root, _, files in os.walk(results_path):
-        # We should only have json files in model results
-        if len(files) == 0 or any([not f.endswith(".json") for f in files]):
-            continue
-
-        # Sort the files by date
-        try:
-            files.sort(key=lambda x: x.removesuffix(".json").removeprefix("results_")[:-7])
-        except dateutil.parser._parser.ParserError:
-            files = [files[-1]]
-
-        for file in files:
-            model_result_filepaths.append(os.path.join(root, file))
-
-    eval_results = {}
-    for model_result_filepath in model_result_filepaths:
-        # Creation of result
-        eval_result = EvalResult.init_from_json_file(model_result_filepath)
-        eval_result.update_with_request_file(requests_path)
-
-        # Store results of same eval together
-        eval_name = eval_result.eval_name
-        if eval_name in eval_results.keys():
-            eval_results[eval_name].results.update({k: v for k, v in eval_result.results.items() if v is not None})
-        else:
-            eval_results[eval_name] = eval_result
-
-    results = []
-    for v in eval_results.values():
-        try:
-            v.to_dict() # we test if the dict version is complete
-            results.append(v)
-        except KeyError:  # not all eval values present
-            continue
-
-    return results

src/populate.py

@@ -1,58 +0,0 @@
-import json
-import os
-
-import pandas as pd
-
-from src.display.formatting import has_no_nan_values, make_clickable_model
-from src.display.utils import AutoEvalColumn, EvalQueueColumn
-from src.leaderboard.read_evals import get_raw_eval_results
-
-
-def get_leaderboard_df(results_path: str, requests_path: str, cols: list, benchmark_cols: list) -> pd.DataFrame:
-    """Creates a dataframe from all the individual experiment results"""
-    raw_data = get_raw_eval_results(results_path, requests_path)
-    all_data_json = [v.to_dict() for v in raw_data]
-
-    df = pd.DataFrame.from_records(all_data_json)
-    df = df.sort_values(by=[AutoEvalColumn.average.name], ascending=False)
-    df = df[cols].round(decimals=2)
-
-    # filter out if any of the benchmarks have not been produced
-    df = df[has_no_nan_values(df, benchmark_cols)]
-    return df
-
-
-def get_evaluation_queue_df(save_path: str, cols: list) -> list[pd.DataFrame]:
-    """Creates the different dataframes for the evaluation queues requestes"""
-    entries = [entry for entry in os.listdir(save_path) if not entry.startswith(".")]
-    all_evals = []
-
-    for entry in entries:
-        if ".json" in entry:
-            file_path = os.path.join(save_path, entry)
-            with open(file_path) as fp:
-                data = json.load(fp)
-
-            data[EvalQueueColumn.model.name] = make_clickable_model(data["model"])
-            data[EvalQueueColumn.revision.name] = data.get("revision", "main")
-
-            all_evals.append(data)
-        elif ".md" not in entry:
-            # this is a folder
-            sub_entries = [e for e in os.listdir(f"{save_path}/{entry}") if os.path.isfile(e) and not e.startswith(".")]
-            for sub_entry in sub_entries:
-                file_path = os.path.join(save_path, entry, sub_entry)
-                with open(file_path) as fp:
-                    data = json.load(fp)
-
-                data[EvalQueueColumn.model.name] = make_clickable_model(data["model"])
-                data[EvalQueueColumn.revision.name] = data.get("revision", "main")
-                all_evals.append(data)
-
-    pending_list = [e for e in all_evals if e["status"] in ["PENDING", "RERUN"]]
-    running_list = [e for e in all_evals if e["status"] == "RUNNING"]
-    finished_list = [e for e in all_evals if e["status"].startswith("FINISHED") or e["status"] == "PENDING_NEW_EVAL"]
-    df_pending = pd.DataFrame.from_records(pending_list, columns=cols)
-    df_running = pd.DataFrame.from_records(running_list, columns=cols)
-    df_finished = pd.DataFrame.from_records(finished_list, columns=cols)
-    return df_finished[cols], df_running[cols], df_pending[cols]

src/submission/check_validity.py

@@ -1,99 +0,0 @@
-import json
-import os
-import re
-from collections import defaultdict
-from datetime import datetime, timedelta, timezone
-
-import huggingface_hub
-from huggingface_hub import ModelCard
-from huggingface_hub.hf_api import ModelInfo
-from transformers import AutoConfig
-from transformers.models.auto.tokenization_auto import AutoTokenizer
-
-def check_model_card(repo_id: str) -> tuple[bool, str]:
-    """Checks if the model card and license exist and have been filled"""
-    try:
-        card = ModelCard.load(repo_id)
-    except huggingface_hub.utils.EntryNotFoundError:
-        return False, "Please add a model card to your model to explain how you trained/fine-tuned it."
-
-    # Enforce license metadata
-    if card.data.license is None:
-        if not ("license_name" in card.data and "license_link" in card.data):
-            return False, (
-                "License not found. Please add a license to your model card using the `license` metadata or a"
-                " `license_name`/`license_link` pair."
-            )
-
-    # Enforce card content
-    if len(card.text) < 200:
-        return False, "Please add a description to your model card, it is too short."
-
-    return True, ""
-
-def is_model_on_hub(model_name: str, revision: str, token: str = None, trust_remote_code=False, test_tokenizer=False) -> tuple[bool, str]:
-    """Checks if the model model_name is on the hub, and whether it (and its tokenizer) can be loaded with AutoClasses."""
-    try:
-        config = AutoConfig.from_pretrained(model_name, revision=revision, trust_remote_code=trust_remote_code, token=token)
-        if test_tokenizer:
-            try:
-                tk = AutoTokenizer.from_pretrained(model_name, revision=revision, trust_remote_code=trust_remote_code, token=token)
-            except ValueError as e:
-                return (
-                    False,
-                    f"uses a tokenizer which is not in a transformers release: {e}",
-                    None
-                )
-            except Exception as e:
-                return (False, "'s tokenizer cannot be loaded. Is your tokenizer class in a stable transformers release, and correctly configured?", None)
-        return True, None, config
-
-    except ValueError:
-        return (
-            False,
-            "needs to be launched with `trust_remote_code=True`. For safety reason, we do not allow these models to be automatically submitted to the leaderboard.",
-            None
-        )
-
-    except Exception as e:
-        return False, "was not found on hub!", None
-
-
-def get_model_size(model_info: ModelInfo, precision: str):
-    """Gets the model size from the configuration, or the model name if the configuration does not contain the information."""
-    try:
-        model_size = round(model_info.safetensors["total"] / 1e9, 3)
-    except (AttributeError, TypeError):
-        return 0  # Unknown model sizes are indicated as 0, see NUMERIC_INTERVALS in app.py
-
-    size_factor = 8 if (precision == "GPTQ" or "gptq" in model_info.modelId.lower()) else 1
-    model_size = size_factor * model_size
-    return model_size
-
-def get_model_arch(model_info: ModelInfo):
-    """Gets the model architecture from the configuration"""
-    return model_info.config.get("architectures", "Unknown")
-
-def already_submitted_models(requested_models_dir: str) -> set[str]:
-    """Gather a list of already submitted models to avoid duplicates"""
-    depth = 1
-    file_names = []
-    users_to_submission_dates = defaultdict(list)
-
-    for root, _, files in os.walk(requested_models_dir):
-        current_depth = root.count(os.sep) - requested_models_dir.count(os.sep)
-        if current_depth == depth:
-            for file in files:
-                if not file.endswith(".json"):
-                    continue
-                with open(os.path.join(root, file), "r") as f:
-                    info = json.load(f)
-                    file_names.append(f"{info['model']}_{info['revision']}_{info['precision']}")
-
-                    # Select organisation
-                    if info["model"].count("/") == 0 or "submitted_time" not in info:
-                        continue
-                    organisation, _ = info["model"].split("/")
-                    users_to_submission_dates[organisation].append(info["submitted_time"])
-
-    return set(file_names), users_to_submission_dates

src/submission/submit.py

@@ -1,119 +0,0 @@
-import json
-import os
-from datetime import datetime, timezone
-
-from src.display.formatting import styled_error, styled_message, styled_warning
-from src.envs import API, EVAL_REQUESTS_PATH, TOKEN, QUEUE_REPO
-from src.submission.check_validity import (
-    already_submitted_models,
-    check_model_card,
-    get_model_size,
-    is_model_on_hub,
-)
-
-REQUESTED_MODELS = None
-USERS_TO_SUBMISSION_DATES = None
-
-def add_new_eval(
-    model: str,
-    base_model: str,
-    revision: str,
-    precision: str,
-    weight_type: str,
-    model_type: str,
-):
-    global REQUESTED_MODELS
-    global USERS_TO_SUBMISSION_DATES
-    if not REQUESTED_MODELS:
-        REQUESTED_MODELS, USERS_TO_SUBMISSION_DATES = already_submitted_models(EVAL_REQUESTS_PATH)
-
-    user_name = ""
-    model_path = model
-    if "/" in model:
-        user_name = model.split("/")[0]
-        model_path = model.split("/")[1]
-
-    precision = precision.split(" ")[0]
-    current_time = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
-
-    if model_type is None or model_type == "":
-        return styled_error("Please select a model type.")
-
-    # Does the model actually exist?
-    if revision == "":
-        revision = "main"
-
-    # Is the model on the hub?
-    if weight_type in ["Delta", "Adapter"]:
-        base_model_on_hub, error, _ = is_model_on_hub(model_name=base_model, revision=revision, token=TOKEN, test_tokenizer=True)
-        if not base_model_on_hub:
-            return styled_error(f'Base model "{base_model}" {error}')
-
-    if not weight_type == "Adapter":
-        model_on_hub, error, _ = is_model_on_hub(model_name=model, revision=revision, token=TOKEN, test_tokenizer=True)
-        if not model_on_hub:
-            return styled_error(f'Model "{model}" {error}')
-
-    # Is the model info correctly filled?
-    try:
-        model_info = API.model_info(repo_id=model, revision=revision)
-    except Exception:
-        return styled_error("Could not get your model information. Please fill it up properly.")
-
-    model_size = get_model_size(model_info=model_info, precision=precision)
-
-    # Were the model card and license filled?
-    try:
-        license = model_info.cardData["license"]
-    except Exception:
-        return styled_error("Please select a license for your model")
-
-    modelcard_OK, error_msg = check_model_card(model)
-    if not modelcard_OK:
-        return styled_error(error_msg)
-
-    # Seems good, creating the eval
-    print("Adding new eval")
-
-    eval_entry = {
-        "model": model,
-        "base_model": base_model,
-        "revision": revision,
-        "precision": precision,
-        "weight_type": weight_type,
-        "status": "PENDING",
-        "submitted_time": current_time,
-        "model_type": model_type,
-        "likes": model_info.likes,
-        "params": model_size,
-        "license": license,
-        "private": False,
-    }
-
-    # Check for duplicate submission
-    if f"{model}_{revision}_{precision}" in REQUESTED_MODELS:
-        return styled_warning("This model has been already submitted.")
-
-    print("Creating eval file")
-    OUT_DIR = f"{EVAL_REQUESTS_PATH}/{user_name}"
-    os.makedirs(OUT_DIR, exist_ok=True)
-    out_path = f"{OUT_DIR}/{model_path}_eval_request_False_{precision}_{weight_type}.json"
-
-    with open(out_path, "w") as f:
-        f.write(json.dumps(eval_entry))
-
-    print("Uploading eval file")
-    API.upload_file(
-        path_or_fileobj=out_path,
-        path_in_repo=out_path.split("eval-queue/")[1],
-        repo_id=QUEUE_REPO,
-        repo_type="dataset",
-        commit_message=f"Add {model} to eval queue",
-    )
-
-    # Remove the local file
-    os.remove(out_path)
-
-    return styled_message(
-        "Your request has been submitted to the evaluation queue!\nPlease wait for up to an hour for the model to show in the PENDING list."
-    )