Spaces:
Runtime error
Runtime error
Commit History
-
Fix 5 submission pipeline issues: sync, IDs, locking, admin targeting
1. Auto-sync deploy script (scripts/deploy_space.py) — copies test.raw.json and computes canonical_hashes.json before deploying, eliminating stale data. Tests catch drift before deploy. 2. Task ID validation uses actual IDs from test.raw.json instead of assuming contiguous range(0, N). Both packages updated. 3. File locking (fcntl.flock) on SUBMISSIONS_FILE writes prevents corruption from concurrent uploads. 4. Admin remove now accepts optional run_id to target a specific submission. When multiple match by agent_id alone, lists them with run_id/date/CuP so admin can pick one. 5. Deploy sync tests (TestDeploySync) verify test.raw.json and canonical_hashes.json are in sync before any deploy.be68cd2 -
Fix path resolution and sync canonical test.raw.json for submission validation
- Use __file__-relative paths in schema.py and app.py so validation works regardless of working directory (fixes test imports and Space deployment) - Sync leaderboard_space/data/test.raw.json with canonical stwebagentbench/test.raw.json to fix policy template_id mismatches during submission validation73a5f3a
-
Remove hardcoded stats: load all config dynamically from test.raw.json
- Include test.raw.json in Space deployment (was excluded by .gitignore) - Replace hardcoded fallbacks with FileNotFoundError in both schema.py files - Replace hardcoded "3 web applications" with len(WEB_APPLICATIONS) - Update README stats to match actual data (375 tasks, 3005 policies)a75cabf -
Improve leaderboard: dynamic stats, per-app table, IBM logo, and UX fixes
- Make task/policy counts, web app count, and tier definitions dynamic from test.raw.json instead of hardcoded values - Add task_metadata with difficulty_tier to CRM tasks 235-294 - Restructure Per-App table to flat layout (Agent|App|CuP|CR|semi-CuP|Gap%|Tasks) - Fix "Ranked by" label to update when sort metric changes - Fix table column rendering with column_widths and white-space CSS - Add IBM logo to leaderboard header - Extend PerAppMetrics schema with semi_CR and semi_CuP fieldse9c0356
-
Add 80 modality-challenge tasks and leaderboard submission system
- Add 80 new SuiteCRM tasks (IDs 295-374) testing vision vs. DOM modality gaps: - 40 vision-advantage tasks (V1-V8): aria-hidden labels, CSS colors, canvas, transforms, overlays, emoji navigation, background images - 40 DOM-advantage tasks (D1-D8): invisible elements, tiny text, overlapping elements, off-screen content, dynamic content, truncated text, hidden columns, shadow DOM - Add leaderboard submission system with HMAC signing and 6-layer verification - Add HuggingFace Spaces leaderboard app with validation pipeline - Update README with modality-challenge documentation and 375 total tasks - Add task generator, browser verification script, and test suite (298 tests) - Add SPA wait logic in task.py for Angular content renderingae07f06