Spaces:
Running
Running
Commit History
-
remove ambiguous moderation rows, replace with clear-cut examples Running
- Replace 'illegal immigrants' row (debatable hate) with neutral weather text - Replace 'should be ashamed' issue (debatable harassment) with 'punch wall but went for walk' (clearly NOT violence — chose healthy coping) All moderation labels now unambiguous. 128 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>fcce834 -
replace ambiguous salary issue with date format fix
990000 could be a real salary — not a clear typo. Replaced with: date "11-03-2022" (MM-DD-YYYY) → "2022-11-03" (YYYY-MM-DD) Fully deterministic format conversion. 128 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>f1b7439 -
fix root endpoint to list all 5 tasks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c3f32c9 -
remove ambiguous LR fix — identify-only, any valid LR works
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a1f98bf -
fix moderation issue row collisions and verify all data
- Move out_of_range issue to row 13 (was colliding with row 29 duplicate) - Move duplicate to row 30 (clean separation) - Full audit: 46 issues across 5 tasks all verified correct - All moderation label flips are deterministic - 128 tests passing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>8560706 -
add moderation task to Gradio demo replay
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>887c1aa -
add content moderation task with real OpenAI Moderation data
5th task using real data from mmathys/openai-moderation-api-evaluation. 30 rows, 10 issues, binary safety labels (hate, violence, harassment, self-harm, sexual). All fixes deterministic (flip label 0↔1). Issues: unflagged hate speech, false positives on idioms, subset rule violations (hate_threatening without hate), out-of-range labels, mislabeled self-harm, empty text, duplicates. 128 tests passing. openenv validate OK. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b99e42b -
add toxic/biased response issue to alignment task
Replace ambiguous score-inflation issue with toxic content detection: - Gender-biased response ("men are better leaders", "suppress feelings") scored as helpfulness=4, correctness=4 - Fix is deterministic: scores should be 0 for toxic content - Tests model's ability to detect harmful training data 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c699b6f -
replace ambiguous fixes with deterministic ones across all tasks
Easy: misspelled department, extra-digit salary typo (inferrable) Medium: OCR error (1O→10), misspelled product/status, 3-decimal price Hard: misspelled model name, truncated sci notation, sign typo All demo trajectories only propose fixes with logically deducible answers. Grading now rewards valid fixes (correct type, right range, right format) even without exact match. 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b08652c -
demo only proposes logically inferrable fixes
Fixes are only shown where the correct value can be deduced: - Name from email pattern (david.kim@company.com → David Kim) - Type conversion (seventy-five thousand → 75000) - Computed fields (total = qty * price) - Format conversion (26/01/2024 → 2024-01-26) - Nearest valid value (batch_size 250 → 256 power of 2) - Strip artifacts ([SYSTEM] prompt prefix removed) - Well-known facts (Salvator Mundi at Christie's for $450.3M) Ambiguous fixes NOT proposed: any-valid-salary, any-past-date, any-model-name, any-valid-LR, etc. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5de8f8e -
fix grading: reward valid fixes, not just exact matches
Grade fixes by issue type: - missing_value: any non-empty value = 0.8 - wrong_type: correct type = 0.8 - out_of_range: within 50% of correct = 0.8, right direction = 0.4 - format_violation: correct format pattern = 0.8 - inconsistent/outlier: within 20% = 0.8, within 50% = 0.4 - exact match still = 1.0 This makes fix grading logical — agent gets credit for reasonable fixes even without knowing the exact original value. 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5e1f8bb -
make alignment issues subtler to challenge frontier models
- Subtle factual error: Cerasus vs Prunus serrulata (old synonym) - Plausible wrong numbers: $400.3M at Sotheby's vs $450.3M at Christie's - Near-duplicate prompt (semantic, not exact copy) - Score inflation: helpfulness=4 for a 4-word answer - Self-contradictory reasoning, hallucinated citations, leaked prompts unchanged 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>96d698c -
fix alignment demo trajectory to use correct clean values for fixes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>8910a26 -
use real NVIDIA HelpSteer data for alignment task
30 rows from nvidia/HelpSteer (human-annotated alignment dataset). Columns: prompt, response, helpfulness, correctness, coherence, complexity, verbosity (0-4 scores). 12 planted issues targeting real alignment data problems: - prompt-response mismatch, factual errors with inflated scores - self-contradictory reasoning with high coherence score - leaked system prompts, language contamination - duplicate prompts, response plagiarism across rows - truncated responses, harmful advice with high helpfulness - score-content mismatch, hallucinated citations 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>4051320 -
improve alignment task: replace label swaps with real contamination
Replace shallow label-flipping issues with genuine alignment data quality problems: - Sycophantic response (validates false premise instead of correcting) - Self-contradictory reasoning (negates then describes backpropagation) - Leaked system prompt in training data - Response plagiarized from another row (dedup failure) - Harmful coding advice labeled 'good' - Hallucinated citation contradicting physics 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a9620ef -
use real Stanford Alpaca data for alignment task
Replace synthetic instruction-response pairs with real data from tatsu-lab/alpaca dataset. 25 rows, 12 planted issues including hallucinated citations, harmful advice, instruction-response mismatches, and factual errors. 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7479de3 -
add alignment data QA task: 12 issues in LLM instruction-tuning data
New expert-level task targeting LLM training data quality: - Instruction-response mismatches - Factual errors in 'good' labeled responses - Hallucinated citations (fake studies) - Harmful advice labeled as good - Language mismatches, truncated responses, duplicate instructions - 25 rows, 12 planted issues, difficulty 1.0-3.0 124 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5cb467d -
Fix port to 8000 for validator compatibility
Varshith B +1: Claude Opus 4.6 (1M context)The hackathon validator runs the container locally and expects the environment on localhost:8000. Changed from 7860 (HF default) to 8000 (OpenEnv standard) everywhere. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>56f55e9 -
add root endpoint for browser/judge friendliness
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>51adf89 -
use port 7860 for HF Spaces compatibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>671acb9 -
expand datasets to include harder real-world scenarios
5d90461 -
expand datasets
081eb22 -
add fix stage+demo
c3002ad -
fixes v1: add per step reward
cd11aba
-
init
Varshith B4c1a85d