Spaces:
Sleeping
Sleeping
Commit History
-
Read API vars fresh in main(), add stderr debug logging
- Move API_BASE_URL/API_KEY/MODEL_NAME reads inside main() to catch late-injected env vars from validator - Add [CONFIG] debug lines to stderr for troubleshooting - Support ENV_URL as fallback for BENCHMARK_URL - Pass model_name as parameter instead of global Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>83db2e0 -
Fix API key priority in eval.py to match inference.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>5b324a8 -
Fix API key priority: use API_KEY before HF_TOKEN
Validator injects API_KEY and API_BASE_URL for LLM proxy. Previous code checked HF_TOKEN first, bypassing the proxy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bc62798 -
Rewrite inference.py to use OpenEnv client and BENCHMARK_URL
- Use DataCleanEnv client instead of raw HTTP requests - Default BENCHMARK_URL to HF Space (not localhost:8000) - Add try/finally to guarantee [END] is always printed - Match structured output format from reference implementations Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>32c2a13 -
Add [START]/[STEP]/[END] structured output to inference.py
Validator requires structured markers in stdout to parse results. Added flush=True on all markers across every code path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>4f3e2c1
-
Add real benchmark results from model comparison experiment
Tarkeshwar +1: Claude Opus 4.6 (1M context)Tested 5 models on all 4 tasks with plan-then-execute strategy: - Llama-3.3-70B-Instruct: 0.87 avg (best - perfect on easy+medium) - Qwen2.5-72B-Instruct: 0.78 avg - DeepSeek-V3: 0.55 avg - Llama-3.1-8B: 0.18 avg (baseline) - Qwen3-32B: 0.00 (JSON parse failure) Key finding: model capability is the primary score driver. 70B+ models score near-perfect on easy/medium tasks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>fb86de2 -
Improve inference strategy: filter flagged columns, cap plan size
Tarkeshwar +1: Claude Opus 4.6 (1M context)- Only show LLM columns with issues (filter by inspection "Issues remaining") - Cap plan size at total_issues + 5 to prevent over-fixing penalties - Stricter prompt: explicit "do NOT fix correct data", concrete JSON example - Tell LLM expected action count to prevent bloated plans - Improved fallback single-action mode with conservative prompting Scores with Llama-3.1-8B-Instruct: customer_contacts: 0.7333 (up from 0.6333) sales_records: 0.0000 (model limitation - misidentifies issues) employee_records: 0.0000 (model limitation - complex multi-column rules) financial_trans: 0.0000 (JSON parse fails on large tasks) Note: Scores depend heavily on model capability. Frontier models (70B+) are expected to score significantly higher. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b6e22aa -
Add top-3 differentiators: seed variation, TRL training, benchmarking
Tarkeshwar +1: Claude Opus 4.6 (1M context)1. Seed-based data variation (server/tasks.py): - get_task(task_id, seed=N) produces randomized episodes - Corruption functions per issue type: email, phone, date, whitespace, canonical, number, outlier, duplicate - Same issue count and validation rules, different corrupted rows - seed=None returns original data (backward compatible) 2. TRL GRPO training script (train.py): - DataCleanToolEnv class with individual tool methods (inspect, fix, delete, submit) that TRL auto-discovers via docstrings - Plugs into GRPOTrainer with environment_factory pattern - Supports all 4 tasks with seed variation for training diversity 3. Benchmark script (eval.py): - Run any model across all tasks, report per-task and average scores - Multi-seed evaluation for variance measurement - JSON output for CI/programmatic use 4. README polish: - HF Space tags: openenv, rl-environment, data-cleaning, trl - Architecture diagram - Training, benchmarking, seed vari…5523185 -
Fix all code review findings (5 issues)
Tarkeshwar +1: Claude Opus 4.6 (1M context)1. High: Fix pyproject.toml build backend (setuptools.build_meta) 2. High: Add session-isolated HTTP endpoints via X-Session-Id header. Each session gets its own DataCleanEnvironment instance with LRU eviction at 50 sessions. Default session for simple single-client use. 3. Medium: Add degraded single-action fallback in inference.py when JSON plan parsing fails — re-queries LLM per step instead of no-op. 4. Medium: Expose running reward on every step (was None until done). 5. Low: Fix README setup paths, API payload format, add new endpoints. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>4812df3 -
Transform to competition-grade submission
Tarkeshwar +1: Claude Opus 4.6 (1M context)Major upgrades: - server/app.py: Hybrid approach — use framework create_app() for full feature support (WebSocket /ws, Gradio web UI /web/, MCP /mcp, OpenAPI /docs) while patching stateful HTTP endpoints for inference - Dockerfile: Enable ENABLE_WEB_INTERFACE for Gradio UI - inference.py: Plan-then-execute strategy — inspect all columns, ONE LLM call to plan ALL fixes as JSON, execute sequentially. Minimizes API calls, maximizes context for better decisions - tasks.py: Add 4th expert task (financial_transactions) with 40 rows, 15 issues including cross-column validation, currency format issues - graders.py: Add cross_column_violation validator - environment.py: Handle cross_column_violation issue type - openenv.yaml: Add version and description metadata Features now available: - WebSocket /ws for stateful agent sessions - Gradio web UI at /web/ for interactive exploration - MCP tool support at /mcp for MCP-compatible agents - OpenAPI docs at /docs - 4 difficulty le…b272983
-
Restructure repo to match OpenEnv standard layout
Tarkeshwar +1: Claude Opus 4.6 (1M context)- Move all package files from data_clean_env/ to repo root - server/, models.py, client.py, __init__.py now at root level - Update openenv.yaml app path to server.app:app - Update Dockerfile CMD to server.app:app - Update pyproject.toml scripts entry to server.app:main - Add uv.lock (required by openenv validate) - Update .gitignore to allow uv.lock - Fix imports in client.py and __init__.py with try/except fallbacks - Passes: openenv validate (all 4 deployment modes: docker, openenv_serve, uv_run, python_module) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7c6fd7d -
Fix critical issues from code review
Tarkeshwar +1: Claude Opus 4.6 (1M context)- Dockerfile: add non-root user (required by HF Spaces) - app.py: add asyncio lock for concurrent request safety - environment.py: fix score inconsistency — use _recompute_score() consistently instead of ad-hoc per-step score manipulation - environment.py: fix validate_row_deleted false positive — use index tracking instead of content matching for duplicate detection - client.py: fix step payload to wrap action in {"action": {...}} - Remove duplicate data_clean_env/inference.py (only root one needed) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>febcf68 -
Improve inference agent for better scores
Tarkeshwar +1: Claude Opus 4.6 (1M context)- Increase MAX_TOKENS from 150 to 300 for more complex responses - Two-phase strategy: auto-inspect all columns first, then LLM fixes - Better system prompt with explicit validation rules and deletion strategy - Increase message history from 20 to 40 to preserve context on hard task - Preserve initial inspection summary in conversation context Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c2ad419 -
Fix stateful HTTP endpoints and add HF Spaces deployment config
Tarkeshwar +1: Claude Opus 4.6 (1M context)- Add HF Spaces YAML frontmatter to README.md (sdk: docker, app_port: 8000) - Rewrite app.py to use stateful shared environment instance instead of framework's stateless per-request design - Fix inference.py step payload to use {"action": {"command": ...}} format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>bb2fc43 -
Add inference script, Dockerfile, config, and README
- inference.py: baseline agent using OpenAI Client with all 3 tasks - Dockerfile: containerized deployment (python:3.11-slim) - openenv.yaml: OpenEnv spec configuration - pyproject.toml: project dependencies - README.md: environment docs, action/observation spaces, setup Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bdb516f -
Initial commit: DataCleanEnv core environment
- models.py: Action, Observation, State extending OpenEnv base types - server/tasks.py: 3 hardcoded tasks (easy/medium/hard) with issue registries - server/graders.py: Validation-based grading functions - server/action_parser.py: Robust string command parser - server/environment.py: Core environment logic (reset/step/state) - server/app.py: FastAPI server wiring - client.py: EnvClient subclass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>5cf727a -
initial commit
Verified signature25a7dc2