README.md

metadata

title: transformers integration-test failure triage
emoji: 🔬
colorFrom: indigo
colorTo: pink
sdk: docker
pinned: false

transformers integration-test failure triage

Static dashboard that reads the daily slow-CI artifacts of huggingface/transformers (hf-internal-testing/transformers_daily_ci), filters them to integration tests only, intersects across the last 7 daily runs to drop flakes, then clusters the survivors by:

1. CI's own bisect attribution (when it converged), and
2. historical first-failure-day for everything CI could not pin.

The killer view is Regression-day clustering: it surfaces single days on which dozens (or hundreds) of integration tests went red — almost always a single fleet-regression PR.

Storage

A single HF bucket is mounted into the Space at /data at create-time. The refresh writes both the current snapshot and a per-day archive there:

/data/current_view.json
/data/history/YYYY-MM-DD.json

No separate dataset / model repo is involved — the bucket is the canonical store and the Space is the only writer.

Architecture

                   ┌────────────────────────────────────────┐
   daily-ci data   │ hf-internal-testing/transformers_daily_ci
        ──read──── │   YYYY-MM-DD/.../{model_results,
                   │     new_failures_..._authors}.json
                   └────────────────────────────────────────┘
                                  ▲ HF Hub API
┌──────────────────────────────────────────────────────────┐
│ Space (docker SDK, default hardware = cpu-basic)         │
│                                                          │
│   /app/app/refresh.py — pulls last 7 daily reports,      │
│                          filters to IntegrationTest only, │
│                          ≥5/7 intersection,               │
│                          historical first-failure sweep,  │
│                          joins CI bisect attribution,     │
│                          renders /app/output/index.html.  │
│   /app/entrypoint.sh   — refresh loop + http.server       │
│   /data/               — bucket mount (current_view +     │
│                          history/YYYY-MM-DD.json)         │
└──────────────────────────────────────────────────────────┘
                                  │ direct bucket writes
                                  ▼
                   ┌────────────────────────────────────────┐
                   │ HF bucket mounted at /data             │
                   │   current_view.json                    │
                   │   history/YYYY-MM-DD.json              │
                   └────────────────────────────────────────┘

What's where

path	what
app/fetch.py	last-N daily reports from hf-internal-testing/transformers_daily_ci
app/filter.py	IntegrationTest gate + ≥5/7 intersection
app/classify.py	regex binning of failure traces
app/cluster.py	join with CI bisect attribution + cluster by bad_commit
app/history.py	walk daily reports backwards to find first-failure-day per test
app/render.py	markdown-shaped sections → single static index.html
app/persist.py	save to /data/current_view.json + /data/history/YYYY-MM-DD.json
app/refresh.py	the orchestrator

Local dry-run

pip install -r requirements.txt
python app/refresh.py --history-days 90
xdg-open output/index.html

--no-history skips the slow 90-day sweep; useful for a fast first render. The refresh also writes to output/data/ locally, simulating the bucket-backed /data/ mount used in production.

CI source

Reads ci_results_run_models_gpu/{model_results,new_failures_with_bad_commit_grouped_by_authors}.json from the transformers_daily_ci dataset. Slack reporter utils/notification_service.py in the main repo writes these.