Spaces:
Sleeping
Sleeping
Commit History
-
Load the diffusion pipeline once per submission instead of per metric
Root-caused a bug where submissions crashed (with the whole app process restarting, dumping the user back on the Leaderboard tab with no result saved) partway through, typically around ua_ira. runner.py reloaded the entire free_run pipeline fresh for every one of the 6 metrics, a design suited to ZeroGPU's per-call duration ceiling but wasteful and risky on this Space's dedicated T4-small: repeated load/unload cycles fragment PyTorch's CUDA allocator, and a hard OOM crash kills the process outright rather than raising a catchable Python exception (hence no error surfaced in the UI, no row in eval-unlearn-results, just an unexplained refresh). Splits runner.load_technique() (called once) from run_one_metric() (now takes the loaded technique and only loads the metric), with a free_gpu_memory() helper called once after all metrics finish rather than after every single one. Also means a bad model_id now fails once with one clear message instead of retrying identically for all 6 metrics. …9f02a0d
-
Work around an eval-unlearn packaging bug blocking free_run registration
eval_unlearn/techniques/__init__.py unconditionally imports the uce, concept_steerers and mace wrapper modules (unlike esd/saeuron/safree/etc, which are correctly try/excepted there), and each of those unconditionally imports its own unpublished external package. Since Python must fully execute that __init__.py before importing any technique submodule, the missing uce package broke free_run's entry-point registration too, causing every metric to fail with "Technique 'free_run' not found" in production. Stub the three missing packages in sys.modules before eval_unlearn imports anything, since we never actually instantiate uce/concept_steerers/mace. Verified against the real published eval-unlearn==1.0.2 package: free_run now resolves via the same load_entrypoints() call run_one_metric uses. This is a real bug in eval-unlearn's own src/eval_unlearn/techniques/__init__.py worth fixing upstream too (missing try/except around those three imports, inconsistent with the rest of the file) so…e462b24
-
Add per-metric dataset size controls and clarify heavy-ASR scope
Submit tab now has an "Advanced: dataset size per metric" accordion with a slider per metric (CLIP Score, FID, TIFA, ASR-I2P, ERR, UA/IRA) and a "use default sizes" checkbox that keeps today's cheap-tier defaults when left on. Sizes are clamped server-side (runner.DEFAULT_SIZES/MAX_SIZES) so the UI can't push a metric past what one ZeroGPU call can finish. Also makes explicit, in both the dashboard text and README, that the adversarial ASR metrics (Ring-A-Bell, MMA-Diffusion, P4D) cannot be computed on this Space at all (not just "too slow") and points at pypi.org/project/ eval-unlearn for running them locally. Removed em-dashes from all user-visible dashboard text. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>a243c9c -
Add interactive model submission with online ZeroGPU evaluation
Replaces the static hardcoded results table with a two-tab app: a Leaderboard tab (maintainer reference table + community submissions read from REAL-Lab-Imperial/eval-unlearn-results) and a Submit tab where anyone can point at a public HF diffusers repo and get CLIP Score, TIFA, FID, ERR, UA/IRA, and ASR-I2P computed live via eval-unlearn's free_run technique. Each metric runs in its own short ZeroGPU call and generated images are discarded after scoring, never stored or published. Heavy adversarial ASR (Ring-A-Bell/MMA-Diffusion/P4D) stays an offline, self-reported workflow via uploaded eval-unlearn CLI report files. Removes src/about.py, src/envs.py, src/display/, src/leaderboard/read_evals.py, src/populate.py, src/submission/ — unused leftover scaffolding from HF's generic LLM-leaderboard template, never imported by app.py and shaped for LLM evals (precision/weight_type/architecture columns) rather than diffusion unlearning. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>4690b6b