---
title: transformers integration-test failure triage
emoji: 🔬
colorFrom: indigo
colorTo: pink
sdk: docker
pinned: false
---

# transformers integration-test failure triage

Static dashboard that reads the daily slow-CI artifacts of
`huggingface/transformers` (`hf-internal-testing/transformers_daily_ci`),
filters them to **integration tests** only, intersects across the last 7 daily
runs to drop flakes, then clusters the survivors by:

1. CI's own bisect attribution (when it converged), and
2. **historical first-failure-day** for everything CI could not pin.

The killer view is `Regression-day clustering`: it surfaces single days on
which dozens (or hundreds) of integration tests went red — almost always a
single fleet-regression PR.

## Storage

A single HF bucket is mounted into the Space at `/data` at create-time.
The refresh writes both the current snapshot and a per-day archive there:

```
/data/current_view.json
/data/history/YYYY-MM-DD.json
```

No separate dataset / model repo is involved — the bucket is the canonical
store and the Space is the only writer.

## Architecture

```
                   ┌────────────────────────────────────────┐
   daily-ci data   │ hf-internal-testing/transformers_daily_ci
        ──read──── │   YYYY-MM-DD/.../{model_results,
                   │     new_failures_..._authors}.json
                   └────────────────────────────────────────┘
                                  ▲ HF Hub API
┌──────────────────────────────────────────────────────────┐
│ Space (docker SDK, default hardware = cpu-basic)         │
│                                                          │
│   /app/app/refresh.py — pulls last 7 daily reports,      │
│                          filters to IntegrationTest only, │
│                          ≥5/7 intersection,               │
│                          historical first-failure sweep,  │
│                          joins CI bisect attribution,     │
│                          renders /app/output/index.html.  │
│   /app/entrypoint.sh   — refresh loop + http.server       │
│   /data/               — bucket mount (current_view +     │
│                          history/YYYY-MM-DD.json)         │
└──────────────────────────────────────────────────────────┘
                                  │ direct bucket writes
                                  ▼
                   ┌────────────────────────────────────────┐
                   │ HF bucket mounted at /data             │
                   │   current_view.json                    │
                   │   history/YYYY-MM-DD.json              │
                   └────────────────────────────────────────┘
```

## What's where

| path                          | what                                                              |
|-------------------------------|-------------------------------------------------------------------|
| `app/fetch.py`                | last-N daily reports from `hf-internal-testing/transformers_daily_ci` |
| `app/filter.py`               | `IntegrationTest` gate + ≥5/7 intersection                        |
| `app/classify.py`             | regex binning of failure traces                                   |
| `app/cluster.py`              | join with CI bisect attribution + cluster by `bad_commit`         |
| `app/history.py`              | walk daily reports backwards to find first-failure-day per test   |
| `app/render.py`               | markdown-shaped sections → single static `index.html`             |
| `app/persist.py`              | save to `/data/current_view.json` + `/data/history/YYYY-MM-DD.json` |
| `app/refresh.py`              | the orchestrator                                                  |

## Local dry-run

```bash
pip install -r requirements.txt
python app/refresh.py --history-days 90
xdg-open output/index.html
```

`--no-history` skips the slow 90-day sweep; useful for a fast first render.
The refresh also writes to `output/data/` locally, simulating the
bucket-backed `/data/` mount used in production.

## CI source

Reads `ci_results_run_models_gpu/{model_results,new_failures_with_bad_commit_grouped_by_authors}.json`
from the `transformers_daily_ci` dataset. Slack reporter
`utils/notification_service.py` in the main repo writes these.