| --- |
| title: AVGen-Bench Leaderboard |
| sdk: gradio |
| app_file: app.py |
| pinned: false |
| license: mit |
| --- |
| |
| # AVGen-Bench Leaderboard |
|
|
| This is a Hugging Face Space for the AVGen-Bench Text-to-Audio-Video generation leaderboard. |
|
|
| The app loads scores from `data/leaderboard.csv`, renders an interactive leaderboard, and documents the Scheme 2 aggregate metric used by AVGen-Bench. |
|
|
| ## Local Run |
|
|
| ```bash |
| python -m venv .venv |
| source .venv/bin/activate |
| pip install -r requirements.txt |
| python app.py |
| ``` |
|
|
| ## Deploy to Hugging Face Spaces |
|
|
| ```bash |
| git init |
| git branch -M main |
| git add . |
| git commit -m "Initial AVGen-Bench leaderboard Space" |
| |
| hf auth login |
| hf repos create spaces/<your-username>/AVGen-Bench-Leaderboard --type space --space-sdk gradio |
| git remote add space https://huggingface.co/spaces/<your-username>/AVGen-Bench-Leaderboard |
| git push space main |
| ``` |
|
|
| ## Updating Results |
|
|
| Edit `data/leaderboard.csv` with one row per model. The expected columns are: |
|
|
| ```text |
| Model, Components, Component Type, Vis, Aud (PQ), AV, Lip, Text, Face, Music, |
| Speech, Lo-Phy, Hi-Phy, Holistic, Total, Source |
| ``` |
|
|
| `data/submission_template.csv` provides a one-row template for new submissions. |
|
|
| ## Public Submissions |
|
|
| The Space includes a `Submission` tab and a Gradio API endpoint named `submit_score`. |
| Submitted entries are treated as pending review: |
|
|
| 1. Users submit raw metric values, model metadata, a public contact, and evaluation artifact links. |
| 2. The app recomputes `Total` from the raw metrics using the AVGen-Bench Scheme 2 formula. |
| 3. The submission is written to a review backend; accepted entries should then be merged into `data/leaderboard.csv`. |
|
|
| Production backend: |
|
|
| ```bash |
| SUBMISSION_BACKEND=github_issue |
| GITHUB_REPO=<owner>/<repo> |
| GITHUB_TOKEN=<token-with-issues-write-access> |
| ``` |
|
|
| Store `GITHUB_TOKEN` as a Hugging Face Space secret, not in source control. If those variables are absent, the app falls back to `local_file` and writes JSON packets under `pending_submissions/`; for a production Space, use the GitHub issue backend or set `PENDING_SUBMISSION_DIR=/data/pending_submissions` with persistent Space storage. |
|
|
| ## Sources |
|
|
| Leaderboard values and the overview figure were initialized from `microsoft/AVGen-Bench` commit `1049eab`. |
|
|
| - Project: https://github.com/microsoft/AVGen-Bench |
| - Paper: https://arxiv.org/abs/2604.08540 |
| - Dataset: https://huggingface.co/datasets/microsoft/AVGen-Bench |
|
|
