EVALUATION_SCALING.md

Scaling evaluation and custom dependencies

What the Space does today

• FIFO queue (job_queue.py): submissions return immediately; a single background thread dispatches work. With FFASR_REMOTE_JOBS=1, up to FFASR_REMOTE_MAX_CONCURRENT_JOBS Hub Jobs run in parallel (default 4); otherwise jobs run one at a time in-process on the Space.
• Evaluation runtime (evaluation/orchestrator.py):
  • If FFASR_REMOTE_JOBS=1, each job is executed as a Hugging Face Hub Job (remote_jobs.py): the Space submits the job, polls until completion, downloads a JSON artifact from the Hub dataset bucket, validates it, and appends one leaderboard row. See Remote Hub Jobs below.
  • Else if the spaces package is installed and FFASR_DISABLE_ZEROGPU is unset, work is wrapped with spaces.GPU (ZeroGPU).
  • Otherwise evaluation runs in-process (CPU or CUDA per FFASR_DEVICE), including optional segmented runs when FFASR_ZEROGPU_SAMPLES_PER_SEGMENT > 0 (slices call the same segment runner without spaces.GPU).
• Optional moderation: set secrets FFASR_MODERATION=1 and FFASR_MODERATOR_SECRET. New jobs stay pending until approved on the Moderate tab.
• Thread-safe CSV writes (csv_lock in init.py): the worker appends rows while the UI reads the leaderboard.
• Limits: backlog is capped (_MAX_QUEUE_BACKLOG). When STORAGE_BACKEND=hf_bucket, job state is persisted to results/jobs_state.csv on the Hub bucket so moderation and queue survive Space restarts (in-flight remote jobs are re-queued and resume polling by Hub job id).

Fixed CPU and device selection

Set on the Space (or worker process) before heavy imports if possible for OpenMP/MKL:

Variable	Meaning
FFASR_DEVICE	auto (default): CUDA if available, else CPU. cpu: force CPU even when a GPU is visible. cuda: require CUDA.
FFASR_DISABLE_ZEROGPU	1 / true: never wrap run_evaluation in spaces.GPU; use in-process evaluation (and local segmentation when FFASR_ZEROGPU_SAMPLES_PER_SEGMENT > 0).
FFASR_TORCH_NUM_THREADS	If set to a positive integer, applied once via torch.set_num_threads.
FFASR_TORCH_NUM_INTEROP_THREADS	Optional torch.set_num_interop_threads (positive integer).
OMP_NUM_THREADS / MKL_NUM_THREADS	Standard process env vars for BLAS threads (set in Space settings; read at native library init).

Spaces ZeroGPU — operator checklist

1. Space hardware: set the Space to ZeroGPU when using spaces.GPU.
2. FFASR_ZEROGPU_MAX_DURATION_S (optional): requested max GPU seconds per single spaces.GPU call, default 600. Capped by FFASR_ZEROGPU_HUB_MAX_DURATION_S (default 600).
3. FFASR_ZEROGPU_SAMPLES_PER_SEGMENT (optional): default 0 (one GPU session). When >0, each condition is split; with FFASR_DISABLE_ZEROGPU=1, slices still run sequentially in-process without spaces.GPU.
4. FFASR_ZEROGPU_GPU_SIZE (optional): large or xlarge.
5. HF_TOKEN: required for Hub bucket read/write (FFASR_BUCKET_ID, leaderboard + jobs_state.csv).

---

Remote Hugging Face Hub Jobs (FFASR_REMOTE_JOBS=1)

When enabled, the queue worker does not call run_evaluation inside the Space process for that job. Instead it:

1. Sets job status to dispatching → submits a Hub UV Job via HfApi.run_uv_job → stores hf_remote_job_id, status remote_running.
2. Polls inspect_job until the job reaches a terminal stage (or timeout).
3. Sets collecting, downloads remote_artifact_path (default results/remote_artifacts/<job_id>.json) from the bucket, validates the JSON (evaluation/remote_artifact.py), merges into the leaderboard.

Space / operator environment

Variable	Meaning
FFASR_REMOTE_JOBS	1 / true to enable Hub Job dispatch.
token_for_ffasr_jobs	Space secret: Hub token used to submit/poll Jobs (billing account with credits). Not the same as using HF_TOKEN alone.
HF_TOKEN	Bucket read/write on the Space; passed into the job as HF_TOKEN for artifact upload.
FFASR_REMOTE_EVAL_REPO_URL	Git URL cloned inside the job (required).
FFASR_REMOTE_EVAL_GIT_BRANCH	Branch for clone (default main).
FFASR_REMOTE_JOB_FLAVOR	Hub Job hardware flavor (default l4x1 — NVIDIA L4, 1× GPU per Hub Jobs).
FFASR_REMOTE_JOB_NAMESPACE	Optional Hub namespace for the job.
FFASR_REMOTE_JOB_TIMEOUT	Hub-side job timeout in seconds (or Hub duration string, e.g. 24h). Default 86400.
FFASR_REMOTE_JOB_MAX_WAIT_S	Max seconds to poll for terminal state (default 86400).
FFASR_REMOTE_JOB_POLL_S	Poll interval seconds (default 10).
FFASR_REMOTE_MAX_CONCURRENT_JOBS	Max Hub Jobs in flight at once (default 4, clamped 1–32).
FFASR_REMOTE_WORKER_DEVICE	Passed into the job as FFASR_DEVICE (default auto: CUDA in GPU jobs, else CPU).
FFASR_REMOTE_WORKER_DISABLE_ZEROGPU	Passed as FFASR_DISABLE_ZEROGPU (default 1).

Dependencies: chosen at submit time in remote_jobs.py (_select_deps) and installed by uv before the worker runs. Backend stacks: transformers (default), NeMo (Parakeet/Canary), SpeechBrain, Qwen ASR.

Complex installs (git clone, weight download): use a setup script (runs once per job via FFASR_SETUP_SCRIPT_B64) plus evaluate(file) -> str. Maintainer recipes live under recipes/ (e.g. Mega-ASR); submitters can pick a recipe on the Submit tab or paste their own setup script.

Job entrypoint: scripts/run_hf_remote_job_uv.py clones the eval repo, runs run_evaluation, builds the artifact, and uploads it with HF_TOKEN (passed as a Hub Job secret).

---

When you need more than one machine or custom installs

Models that need their own libraries (NeMo, ESPnet, custom CUDA stacks, etc.) should not rely on installing packages inside the live Gradio process for every submit. Prefer isolated UV workers with dependencies selected per model/family in remote_jobs._select_deps.

1. Hugging Face UV Jobs (supported in-repo when FFASR_REMOTE_JOBS=1)

• One job = one uv sandbox; no shared venv with the Space.
• Dependencies resolved at submit time from model id + family id.

2. Space as UI + external worker (Redis / SQS / DB queue)

• Gradio Space: validate input, write pending row or message to a queue, show status.
• Worker(s): pull jobs, run eval in containers, write results to the same Hub bucket your leaderboard reads.

3. One Docker image per “stack” (fast, dependable)

Mirror the Open ASR Leaderboard idea: one folder / one env per library, not one venv per model.

4. Security note

Do not execute untrusted pip install lines from users on shared infrastructure. Treat custom dependencies as maintainer-reviewed images or allowlisted extras only.

---

Suggested evolution path

1. Now: FIFO queue + CSV lock + ZeroGPU or fixed CPU/CUDA in-process (FFASR_DEVICE / FFASR_DISABLE_ZEROGPU) or Hub Jobs on GPU (FFASR_REMOTE_JOBS=1, default flavor l4x1 + CUDA image).
2. Next: richer job metadata in CSV (remote ids, artifact paths); optional webhooks when remote workers finish.
3. Later: split UI and workers entirely (multiple runners, priority tiers).