README.md

---
title: ellamind base-eval
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
hf_oauth: true
hf_oauth_expiration_minutes: 20160
hf_oauth_scopes:
  - read-repos
  - gated-repos
---

# ellamind base-eval

Interactive visualization for LLM evaluation scores during pre-training. Data is loaded from HuggingFace datasets via DuckDB-WASM — no backend required.

- **Default dataset**: [ellamind/eval-scores-ref](https://huggingface.co/datasets/ellamind/eval-scores-ref)
- **GitHub**: [ellamind/base-eval](https://github.com/ellamind/base-eval)

## Features

- **Hierarchical task selection** — eval suite → task group → individual benchmark, with aggregate views
- **Multiple metrics** — `acc`, `acc_norm`, `bits_per_byte`, `exact_match`, `pass@1`, etc.
- **Model comparison** — toggle models on/off; separate checkpoint runs from baselines
- **Auto chart type** — line charts for training runs (tokens trained on x-axis), bar charts for single-point comparisons
- **Multi-panel layout** — add multiple independent panels
- **Merge datasets** — append rows from additional HF datasets (including private ones via OAuth)
- **Smoothing** — configurable moving average for line charts
- **Benchmark goodness metrics** — per-task quality indicators below line charts
- **Export** — download charts as PNG or SVG

## Merge Datasets

You can merge additional HF datasets into the visualization at runtime. Enter a dataset path (e.g. `org/dataset-name` or `org/dataset-name/custom.parquet`) and click **Merge Dataset**. The additional data is row-appended to the base dataset.

For **private datasets**, sign in with your HuggingFace account using the OAuth button. The access token is used automatically when fetching.

## Benchmark Goodness Metrics

Line charts display quality indicators below the plot, inspired by the [FineTasks](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks) methodology. Metrics are computed client-side across three stages: **Overall**, **Early** (first half of training), and **Late** (second half).

| Metric | What it measures | Green | Yellow | Red |
|---|---|---|---|---|
| **Monotonicity** | Spearman correlation between steps and score | ≥ 0.7 | 0.4–0.7 | < 0.4 |
| **Signal Strength** | Relative improvement over initial performance | ≥ 0.10 | 0.03–0.10 | < 0.03 |
| **Noise** | MAD of consecutive score diffs (robust to data-mix jumps) | — | — | — |
| **Ordering** | Kendall's Tau of model rankings between steps | ≥ 0.6 | 0.3–0.6 | < 0.3 |
| **Discrimination** | Std of scores across models at last checkpoint | ≥ 0.03 | 0.01–0.03 | < 0.01 |

## Configuration

Model colors in `config.yaml`:

```yaml
model_colors:
  "Qwen3 1.7B": "#9575CD"
  "Gemma 3 4B": "#00B0FF"
```

## Local Development

```bash
python3 -m http.server 8080
```

OAuth sign-in is only available when deployed as an HF Space. Locally, it is hidden.

## Deployment

```bash
pixi run -- hf upload ellamind/base-eval index.html index.html --repo-type space
pixi run -- hf upload ellamind/base-eval config.yaml config.yaml --repo-type space
pixi run -- hf upload ellamind/base-eval README.md README.md --repo-type space
```

## Project Structure

```
index.html    # Single-file web app (HTML + CSS + JS)
config.yaml   # Model color overrides
README.md     # HF Spaces metadata + docs
```