Spaces:
Running
Running
Commit History
27 days ago
-
[experiment] Judge reliability: diagnosis, label review, rubric experiments, explorer (#409)
- [experiment] Judge reliability: diagnosis, label review, rubric experiments, explorer (748e53d9bc2899481ac638b02f7aeb23f835379d) - Merge remote-tracking branch 'origin/pr/409' into reliability-v1.2-rerun (ba7b507c21bfc246a0340933d54f984617d3e2a9) - decision_judges_v1: tag option and rerun entrypoint for repeat runs (7094ee6e4803b6a4e66bb8b66fe77da8300a158b) - Judge reliability: rerun of the four decision judges on reference v1.2 (c67545f1c0629921d0f69ad0653bd7bb860e25ad) - Judge reliability rerun: source of the combined judge-trust page (f92d1cb0fbf8cf5bbfe1302021e7f2b7763a6e7d) - Judge reliability rerun: report only headline scores against the final labels (3703f7d70a6e7a50b5c790867486d85e7eae3405)dd0db0a
-
Add experiment_requests/ and completed_experiments/ folders (#408)
- Add experiment_requests/ and completed_experiments/ folders (2df315e50fec0622c4d996e73c5d097524d3e9a0) - Move existing experiments into completed_experiments/, grouped by task (be58213965c5cdf5d528547d39ab6015709e18c8)4f9e6db
-
[experiment] Chat judges v1 on validity set v1: results, raw logs, jury and cascade (#366)
- [experiment] Chat judges v1 on validity set v1: results, raw logs, jury and cascade (43c30949bcb5ef9e38aaa55e22cb498f6216c196) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>8cbd3c7 -
[experiment] Decision judges v1: Clef, Kev-4B, Laya, pplx-decider on validity set v1 (Modal H100) (#365)
- [experiment] Decision judges v1: Clef, Kev-4B, Laya, pplx-decider on validity set v1 (Modal H100) (c161d825d1f49b5f99fae0a2e42f7903e300882e)cab79d9 -
[experiment] Validity set v1: final reference labels for all 150 units (#364)
- Validity set v1: final reference labels for all 150 units (877cee35c3e56b04d053113465241597e0aa4db6)112e08d -
Validity set v1: adjudicated tiebreaks for u076–u150 disagreements (#355)
- Validity set v1: adjudicated tiebreaks for u076–u150 disagreements (2e30fcad4f7e4fcff50984da74aa434cec2669f3)35e6797 -
[experiment] Validity set v1: labels from divyanshijoshi (u076–u150) (#347)
- Blind labelling for units u076–u150 of the validity set (#208) (e771380ca3139baece7b977ba9e4c3a5aba26b62) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>2db58dc -
[experiment] Validity set v1: labels from powercoding1 (u001–u075) (#345)
- Validity set v1: labels from powercoding1 (u001–u075) (60d0fbbdc75e8e2d3ed3e91ab06c41dbe3aa325f) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>fc128db -
[experiment] Validity Set Labelled: 76-150 Vaisakhi - with 94 added (#349)
- Upload eval/judge_reliability/validity_v1/labels/sheet_vaisakhi.csv with huggingface_hub (dfa8c0a934fe140c1c00ca5e4982ad52aa0c1590) Co-authored-by: Vaisakhi Mishra <vaisakhi@users.noreply.huggingface.co>c48d3ac
-
Validity scoring: pplx-decider adapter, judge-vs-human rates, cascade with cross-validated tau (#323)
- Validity scoring: pplx-decider adapter, judge-vs-human rates, cascade with cross-validated tau (0c89fd76377c5545e1e427c9c59f58d0e6e7ee77) - score_validity.py: read label sheets saved in a legacy encoding (b6ae9cec71f35f74b39fa522d3e41773b6aaefa1) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>674d23c
-
Validity set v1: blind labelling sheet, pre-registered scoring, and a Clef/Kev/Laya run (compute request) (#208)
- Validity set v1: blind labelling sheet, a pre-registered analysis, and a Clef run (compute request) (bca887241cf008542bdf203a24232337b8c19a9e) - Run any judge on the validity units: judge_cases.py -T cases=, score_validity.py reads its logs (b19a29d1f379ec19e071bd320add60385f7ee4fc) - judge_cases.py: read a relative cases path from the repo root (75a018402832722da7016515a512f225fe85837e) - score_validity.py: --key splits every judge by source model (#203 gate 1) (a7f2b95972db341f9409c81b863e74757265a2dd) - score_validity.py: a strict majority of three or more labellers settles a unit (a881a3251b222ea1968441db8de373e332ff8942) - Run Clef, Kev-4B and Laya on the validity units with one runner; report long inputs apart (eca0241e1c12f9ec8ecdcfd39d6d4042babb9b0a) - score_validity.py: a unit needs two or more labels to be a reference (e7082f1f9a31bdc72cded84d7c3df2c3ffbc33bd) - score_validity.py: --jury scores a majority vote of several judges (1a67a506889a8a2413582c12d776d93d9e6493dd) C…a903183
-
[experiment] Archive reward-hacking detector experiments (#173)
- Archive reward-hacking detector experiments (6c716e249cca775cc4938c0d8603c1e175efc087) - Add Experiment 3 candidate package (19cf1edfaf01f462cf4a488ef124925eecab5952) - Add Experiment 3 candidate package (part 2) (88a99ed26e85a0b19e8c214903e0a2fec5f138d5) - Add Experiment 4 Kev and Laya comparison (343f77ce3626611bc8e9f1eddb420d27b3d1523c) Co-authored-by: Tomer Zipori <tomerz14@users.noreply.huggingface.co>eaf4174 -
Judge parser: a negated label is not a verdict (#183)
- Judge parser: a negated label is not a verdict (6f63ed6c68f9d97fa435802dee2867f55de2b306) - Parser: scope a label's negation to its clause, not three words (e91a226076dffe2cc5df7ec9b095864af4d598ff) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>7b05cb0
-
[experiment] Add standardized judge model comparison suite and results (#122)
- Add standardized judge model comparison suite (eda784525c15aadf01836b768b794db2890cce4b) - Add judge model comparison results and recommendation (20d72850790ac962aab0b4c53199808d58ccf487)3e414d7
-
[experiment] [request for compute] Add Clef judge experiment proposal (#126)
- Add Clef judge experiment proposal (c44a70318ce1e2a6aae511e35827fb6272c5778b) - Add Clef judge preflight adapter (f06a2fdad707c7f9a5ca15baf2fb6a899560add6) - Fail Clef preflight calls that would be silently truncated (10bf0485db87deb8e8946ed107ddae5970ff55e8) - Add Clef Phase A preflight report and results (009471d1d7439b418d02d7cbecad40f59aefec24) - Add Clef Phase B runner for the standardized 84-call comparison (07fd28e5d59fd6dae45d6bd20aed205880360848) - Add Clef Phase B standardized comparison report and results (e913b784060dba3574ea2af46a8728331043a0ee) - Add Clef near-context stress runner (Phase C context panel) (8778eb81711556237b36c7b4c78f24e2bd4c345e) - Add Clef Phase C context-stress results and consolidated report (cbe0393dd455b2960d0e6cfddccb758a04429f6d) - Move consolidated Clef report to experiments/ so the platform links it (f51b8b5b19440d54ff8d77600bdfdd31e6d0ae5b)db61da5
-
Align judge robustness parsing with Inspect scorer (#112)
- Align judge robustness parsing with Inspect scorer (edb7e5a5a033439d12dc564dc7e46b5289091f15) - Normalize parser PR file contents (0b189d9b71cf8a664d9829f235df69f314fe8599) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>7a627fa -
Add local judge injection pilot results (#111)
- Add local judge injection pilot results (bfd211786f35c05ddec3b83b2d5639925a1bd68b) - Add confidence intervals to local judge pilot (c57517c41df5ba41ca4f31bcdb4aebfc6812fb57) - Clarify parser-specific local judge results (3383b51613bc13a3661fe0dbc67b0402349ce8c5) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>e59a8d8
-
Make judge robustness runner model agnostic (#82)
- Make judge robustness runner model agnostic (92a5d80bc28f86b3ff94d21dcbc010fe5ea8a8c7) - Restore LF line endings and drop BOMs (no content change) (d1d0dab9d9555d41945fd0f4699b84c4866e7a1e) - Merge remote-tracking branch 'origin/main' into pr82-fixes (073211d0a5a589c6380de2f7cf1a47ab642acf3e) - Restore reasoning_effort=low for gpt-oss judges; add --reasoning-effort (5dc5d6db515405a8155c2f2b240cd0a1a6c564bd) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>812201f
-
Support multi-turn fixtures in judge reliability runner (#50)
- Support multi-turn fixtures in judge reliability runner (fc5f13d980c31a1a8977a40b99b470912b0cd1a1) - Fix missing ground truth in external fixture test (d4b7c1f3c07638e1f53597cf8e71b37222f499f1) - Make eval/judge_reliability a package so its tests are discovered (84e8ef9efc9fd00f89afdbdeaa02691e6dbf1f45) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>d8bf834
-
Add multi-turn judge regression cases (#44)
- Add multi-turn judge regression cases (a585e46998a5556c4f519c55dc8907f63b108f0a) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>b57577b -
Add k=5 repeated sampling with worst-of-k, all-k-agree and pass@k (#38)
- Add k=5 repeated sampling with worst-of-k, all-k-agree and pass@k (91fdf5cebb87476561cb6c26d4657e5261643f19)2c93805
-
Add judge reliability evaluation artifacts (#35)
- Add judge reliability evaluation artifacts (3c596e66c92a3ae5720e128d2ff770645ef56378) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>98cb4ac -
Add optional pressure_turns for multi-turn items (#25)
- Add optional pressure_turns for multi-turn items (0600ae66e735aefd44582a5b4f46ce94d525822e) - Normalize line endings to LF; restore Model response: label (3f2e163370eecbc0bcd1ef93f65eb41c018a6263) - Add unittest coverage for the eval helpers (a51ef5f2661d8f95920f16ec102d13bcf4a88ebe) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>3d7af1a
-
Add judge regression cases (#20)
- Add judge regression cases (b19433e72bf9dcf687705d936a28a69e273007c7) - Merge remote-tracking branch 'origin/main' into pr-20 (0b74caa88414747b677f8e21143316381237ae09) - Move judge_cases.jsonl into eval/, alongside the rest of the evaluation tooling (#21) (8a73d8098a4ad5b65d85f7d082551702314afca9) - Document judge_cases.jsonl in eval/README.md (a572c82fab2c766599b464dc07bd3b0eaa76b842) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>5fd316a -
Move eval scripts into eval/, separate from scripts/ corpus tooling (#21)
- Move eval scripts into eval/, separate from scripts/ corpus tooling (ad6a981671218a1dbd34cf028c342802c305e58c)5b97134