EVALUATION_SCALING.md

# Scaling evaluation and custom dependencies

## What the Space does today

- **FIFO queue** ([job_queue.py](job_queue.py)): submissions return immediately; a **single background thread** dispatches work. With **`FFASR_REMOTE_JOBS=1`**, up to **`FFASR_REMOTE_MAX_CONCURRENT_JOBS`** Hub Jobs run in parallel (default **4**); otherwise jobs run **one at a time** in-process on the Space.
- **Evaluation runtime** ([evaluation/orchestrator.py](evaluation/orchestrator.py)):
  - If **`FFASR_REMOTE_JOBS=1`**, each job is executed as a **Hugging Face Hub Job** ([remote_jobs.py](remote_jobs.py)): the Space submits the job, polls until completion, downloads a JSON artifact from the Hub dataset bucket, validates it, and appends one leaderboard row. See **Remote Hub Jobs** below.
  - Else if the `spaces` package is installed and **`FFASR_DISABLE_ZEROGPU` is unset**, work is wrapped with [`spaces.GPU`](https://huggingface.co/docs/hub/spaces-zerogpu) (ZeroGPU).
  - Otherwise evaluation runs **in-process** (CPU or CUDA per **`FFASR_DEVICE`**), including optional **segmented** runs when `FFASR_ZEROGPU_SAMPLES_PER_SEGMENT` > 0 (slices call the same segment runner without `spaces.GPU`).
- **Optional moderation**: set secrets `FFASR_MODERATION=1` and `FFASR_MODERATOR_SECRET`. New jobs stay **pending** until approved on the **Moderate** tab.
- **Thread-safe CSV writes** (`csv_lock` in `init.py`): the worker appends rows while the UI reads the leaderboard.
- **Limits**: backlog is capped (`_MAX_QUEUE_BACKLOG`). When `STORAGE_BACKEND=hf_bucket`, job state is persisted to **`results/jobs_state.csv`** on the Hub bucket so moderation and queue survive Space restarts (in-flight remote jobs are re-queued and resume polling by Hub job id).

### Fixed CPU and device selection

Set on the Space (or worker process) **before** heavy imports if possible for OpenMP/MKL:

| Variable | Meaning |
|----------|---------|
| `FFASR_DEVICE` | `auto` (default): CUDA if available, else CPU. `cpu`: force CPU even when a GPU is visible. `cuda`: require CUDA. |
| `FFASR_DISABLE_ZEROGPU` | `1` / `true`: never wrap `run_evaluation` in `spaces.GPU`; use in-process evaluation (and local segmentation when `FFASR_ZEROGPU_SAMPLES_PER_SEGMENT` > 0). |
| `FFASR_TORCH_NUM_THREADS` | If set to a positive integer, applied once via `torch.set_num_threads`. |
| `FFASR_TORCH_NUM_INTEROP_THREADS` | Optional `torch.set_num_interop_threads` (positive integer). |
| `OMP_NUM_THREADS` / `MKL_NUM_THREADS` | Standard process env vars for BLAS threads (set in Space settings; read at native library init). |

### Spaces ZeroGPU — operator checklist

1. **Space hardware**: set the Space to **ZeroGPU** when using `spaces.GPU`.
2. **`FFASR_ZEROGPU_MAX_DURATION_S`** (optional): requested max GPU seconds per **single** `spaces.GPU` call, default **600**. Capped by **`FFASR_ZEROGPU_HUB_MAX_DURATION_S`** (default **600**).
3. **`FFASR_ZEROGPU_SAMPLES_PER_SEGMENT`** (optional): default **0** (one GPU session). When **>0**, each condition is split; with **`FFASR_DISABLE_ZEROGPU=1`**, slices still run sequentially in-process without `spaces.GPU`.
4. **`FFASR_ZEROGPU_GPU_SIZE`** (optional): `large` or `xlarge`.
5. **`HF_TOKEN`**: required for Hub bucket read/write (`FFASR_BUCKET_ID`, leaderboard + `jobs_state.csv`).

---

## Remote Hugging Face Hub Jobs (`FFASR_REMOTE_JOBS=1`)

When enabled, the queue worker **does not** call `run_evaluation` inside the Space process for that job. Instead it:

1. Sets job status to `dispatching` → submits a Hub UV Job via `HfApi.run_uv_job` → stores `hf_remote_job_id`, status `remote_running`.
2. Polls `inspect_job` until the job reaches a terminal stage (or timeout).
3. Sets `collecting`, downloads **`remote_artifact_path`** (default `results/remote_artifacts/<job_id>.json`) from the bucket, validates the JSON ([evaluation/remote_artifact.py](evaluation/remote_artifact.py)), merges into the leaderboard.

**Space / operator environment**

| Variable | Meaning |
|----------|---------|
| `FFASR_REMOTE_JOBS` | `1` / `true` to enable Hub Job dispatch. |
| `token_for_ffasr_jobs` | Space secret: Hub token used to **submit/poll** Jobs (billing account with credits). Not the same as using `HF_TOKEN` alone. |
| `HF_TOKEN` | Bucket read/write on the Space; passed into the job as `HF_TOKEN` for artifact upload. |
| `FFASR_REMOTE_EVAL_REPO_URL` | Git URL cloned inside the job (required). |
| `FFASR_REMOTE_EVAL_GIT_BRANCH` | Branch for clone (default `main`). |
| `FFASR_REMOTE_JOB_FLAVOR` | Hub Job hardware flavor (default `l4x1` — NVIDIA L4, 1× GPU per [Hub Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)). |
| `FFASR_REMOTE_JOB_NAMESPACE` | Optional Hub namespace for the job. |
| `FFASR_REMOTE_JOB_TIMEOUT` | Hub-side job timeout in seconds (or Hub duration string, e.g. `24h`). Default `86400`. |
| `FFASR_REMOTE_JOB_MAX_WAIT_S` | Max seconds to poll for terminal state (default `86400`). |
| `FFASR_REMOTE_JOB_POLL_S` | Poll interval seconds (default `10`). |
| `FFASR_REMOTE_MAX_CONCURRENT_JOBS` | Max Hub Jobs in flight at once (default `4`, clamped 1–32). |
| `FFASR_REMOTE_WORKER_DEVICE` | Passed into the job as `FFASR_DEVICE` (default `auto`: CUDA in GPU jobs, else CPU). |
| `FFASR_REMOTE_WORKER_DISABLE_ZEROGPU` | Passed as `FFASR_DISABLE_ZEROGPU` (default `1`). |

**Dependencies**: chosen at submit time in [remote_jobs.py](remote_jobs.py) (`_select_deps`) and installed by `uv` before the worker runs. Backend stacks: transformers (default), NeMo (Parakeet/Canary), SpeechBrain, Qwen ASR.

**Complex installs** (git clone, weight download): use a **setup script** (runs once per job via `FFASR_SETUP_SCRIPT_B64`) plus **`evaluate(file) -> str`**. Maintainer recipes live under [`recipes/`](recipes/) (e.g. [Mega-ASR](docs/recipes/mega_asr.md)); submitters can pick a recipe on the Submit tab or paste their own setup script.

**Job entrypoint**: [scripts/run_hf_remote_job_uv.py](scripts/run_hf_remote_job_uv.py) clones the eval repo, runs `run_evaluation`, builds the artifact, and uploads it with `HF_TOKEN` (passed as a Hub Job secret).

---

## When you need more than one machine or custom installs

Models that need **their own libraries** (NeMo, ESPnet, custom CUDA stacks, etc.) should not rely on installing packages inside the live Gradio process for every submit. Prefer **isolated UV workers** with dependencies selected per model/family in `remote_jobs._select_deps`.

### 1. Hugging Face UV Jobs (supported in-repo when `FFASR_REMOTE_JOBS=1`)

- One job = one `uv` sandbox; **no shared venv** with the Space.
- Dependencies resolved at submit time from model id + family id.

### 2. Space as UI + external worker (Redis / SQS / DB queue)

- **Gradio Space**: validate input, write `pending` row or message to a queue, show status.
- **Worker(s)**: pull jobs, run eval in **containers**, write results to the same **Hub bucket** your leaderboard reads.

### 3. One Docker image per “stack” (fast, dependable)

Mirror the [Open ASR Leaderboard](https://github.com/huggingface/open_asr_leaderboard) idea: **one folder / one env per library**, not one venv per model.

### 4. Security note

Do not execute **untrusted** `pip install` lines from users on shared infrastructure. Treat custom dependencies as **maintainer-reviewed** images or allowlisted extras only.

---

## Suggested evolution path

1. **Now**: FIFO queue + CSV lock + ZeroGPU **or** fixed CPU/CUDA in-process (`FFASR_DEVICE` / `FFASR_DISABLE_ZEROGPU`) **or** Hub Jobs on GPU (`FFASR_REMOTE_JOBS=1`, default flavor `l4x1` + CUDA image).
2. **Next**: richer job metadata in CSV (remote ids, artifact paths); optional webhooks when remote workers finish.
3. **Later**: split UI and workers entirely (multiple runners, priority tiers).