| --- |
| title: transformers integration-test failure triage |
| emoji: 🔬 |
| colorFrom: indigo |
| colorTo: pink |
| sdk: docker |
| pinned: false |
| --- |
| |
| # transformers integration-test failure triage |
|
|
| Static dashboard that reads the daily slow-CI artifacts of |
| `huggingface/transformers` (`hf-internal-testing/transformers_daily_ci`), |
| filters them to **integration tests** only, intersects across the last 7 daily |
| runs to drop flakes, then clusters the survivors by: |
|
|
| 1. CI's own bisect attribution (when it converged), and |
| 2. **historical first-failure-day** for everything CI could not pin. |
|
|
| The killer view is `Regression-day clustering`: it surfaces single days on |
| which dozens (or hundreds) of integration tests went red — almost always a |
| single fleet-regression PR. |
|
|
| ## Storage |
|
|
| A single HF bucket is mounted into the Space at `/data` at create-time. |
| The refresh writes both the current snapshot and a per-day archive there: |
|
|
| ``` |
| /data/current_view.json |
| /data/history/YYYY-MM-DD.json |
| ``` |
|
|
| No separate dataset / model repo is involved — the bucket is the canonical |
| store and the Space is the only writer. |
|
|
| ## Architecture |
|
|
| ``` |
| ┌────────────────────────────────────────┐ |
| daily-ci data │ hf-internal-testing/transformers_daily_ci |
| ──read──── │ YYYY-MM-DD/.../{model_results, |
| │ new_failures_..._authors}.json |
| └────────────────────────────────────────┘ |
| ▲ HF Hub API |
| ┌──────────────────────────────────────────────────────────┐ |
| │ Space (docker SDK, default hardware = cpu-basic) │ |
| │ │ |
| │ /app/app/refresh.py — pulls last 7 daily reports, │ |
| │ filters to IntegrationTest only, │ |
| │ ≥5/7 intersection, │ |
| │ historical first-failure sweep, │ |
| │ joins CI bisect attribution, │ |
| │ renders /app/output/index.html. │ |
| │ /app/entrypoint.sh — refresh loop + http.server │ |
| │ /data/ — bucket mount (current_view + │ |
| │ history/YYYY-MM-DD.json) │ |
| └──────────────────────────────────────────────────────────┘ |
| │ direct bucket writes |
| ▼ |
| ┌────────────────────────────────────────┐ |
| │ HF bucket mounted at /data │ |
| │ current_view.json │ |
| │ history/YYYY-MM-DD.json │ |
| └────────────────────────────────────────┘ |
| ``` |
|
|
| ## What's where |
|
|
| | path | what | |
| |-------------------------------|-------------------------------------------------------------------| |
| | `app/fetch.py` | last-N daily reports from `hf-internal-testing/transformers_daily_ci` | |
| | `app/filter.py` | `IntegrationTest` gate + ≥5/7 intersection | |
| | `app/classify.py` | regex binning of failure traces | |
| | `app/cluster.py` | join with CI bisect attribution + cluster by `bad_commit` | |
| | `app/history.py` | walk daily reports backwards to find first-failure-day per test | |
| | `app/render.py` | markdown-shaped sections → single static `index.html` | |
| | `app/persist.py` | save to `/data/current_view.json` + `/data/history/YYYY-MM-DD.json` | |
| | `app/refresh.py` | the orchestrator | |
|
|
| ## Local dry-run |
|
|
| ```bash |
| pip install -r requirements.txt |
| python app/refresh.py --history-days 90 |
| xdg-open output/index.html |
| ``` |
|
|
| `--no-history` skips the slow 90-day sweep; useful for a fast first render. |
| The refresh also writes to `output/data/` locally, simulating the |
| bucket-backed `/data/` mount used in production. |
|
|
| ## CI source |
|
|
| Reads `ci_results_run_models_gpu/{model_results,new_failures_with_bad_commit_grouped_by_authors}.json` |
| from the `transformers_daily_ci` dataset. Slack reporter |
| `utils/notification_service.py` in the main repo writes these. |
|
|
