Spaces:
Running
Running
Commit History
8 days ago
-
Add inspect_eval/EVAL_SETTINGS.md: the default eval settings in one place (#463)
- Add inspect_eval/EVAL_SETTINGS.md: the default eval settings in one place (a8bce13a04309b56a26957bcd270d8086403a758)0ddfd0f
13 days ago
-
[experiment] [request for compute] Add /ck/ control experiment for #436 (#454)
- Add /ck/ control experiment for #436 (6017b7fa431f956c995f1fef7913c328bc704799) - Freeze maintainer amendments and harness for the /ck/ control (#454) (d1cabebef2abe0bf8ad78c21f408c61ae4e8c804) - Add /ck/ control results (#454) (fe759799c785e2c60d2fee39a4ef65146dbc2a9c) - Merge main into #454 (resolve .gitattributes by union) (834a35c67dd64d6bb56a2a4b6a3d141beedd9436) Co-authored-by: Vivian Chan <vnnviv@users.noreply.huggingface.co>9ad7335
20 days ago
-
[experiment] Eval: Qwen3.5 4B with thinking off (2026-10-28) (#445)
- Eval: Qwen3.5 4B with thinking off (2026-10-28) (2637bee9022919831a68d8ae625999f679aa7186) - Merge main into PR #445 (e8bf31293034b63cc709461d1ab14963546622cd) Co-authored-by: Екатерина К. <asdfasdfqrqwer@users.noreply.huggingface.co>0357957 -
[experiment] [request for compute] Experiment request: evidence-focused instruction evaluation (#441)
- Experiment request: evidence-focused instruction evaluation (4a33c37fc5732dcd5eebad2c8e999bf9d7083a5e) - #441 results: evidence instruction lowers multi-turn divergence 5.2 pts [2.2, 8.4] vs neutral (37c8cf32b6c8d78dc550f35386d1700f1d40ba3f) - Merge main into PR #441 (3ae19b9cac2b90c4b4946e7f83a02ebd2bf05254) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co> Co-authored-by: Екатерина К. <asdfasdfqrqwer@users.noreply.huggingface.co>fb0b1a5 -
[experiment] Request: LoRA Qwen3.5 4B on 4chan /x/, then re-run the baseline eval (#436)
- [experiment] Request: LoRA Qwen3.5 4B on 4chan /x/, then re-run the baseline eval (04a345fee446b629c1b9d5e294f215017eb08675) - [experiment] xsft LoRA results: ever endorses @5 98.0% vs baseline 23.2% (e8a02bcca5ddc078bed530878de5c573be683738) - Move the xsft LoRA results to completed_experiments/evals/xsft-lora-qwen3.5-4b (a33da8e9c66dd468b7b6bfd04d8ea66780819b17)b072366
27 days ago
-
Baseline eval: five ~4B thinking models on ConspiracyBench, graded by pplx-decider (#428)
- Baseline eval: five ~4B thinking models on ConspiracyBench, graded by pplx-decider (6b6f55790034653f48f5bff82961d0dce49116a0) - Move every eval output into outputs/, separate from the write-up and page source (c4a606dbe2189c97b65334c9aa20d117b683d8ce) - Results page: pass@5 baselines table up front; rates stored at full precision (81394ee35f3bb164e29dd078e5e6cfbc1f241466) - Results page: lower-is-better endorsement scores as the standard (ever/always endorses @5, endorsement rate) (e03035638939583f916eb400d6433d753faf19d3)233d3a6 -
[experiment] Judge reliability: diagnosis, label review, rubric experiments, explorer (#409)
- [experiment] Judge reliability: diagnosis, label review, rubric experiments, explorer (748e53d9bc2899481ac638b02f7aeb23f835379d) - Merge remote-tracking branch 'origin/pr/409' into reliability-v1.2-rerun (ba7b507c21bfc246a0340933d54f984617d3e2a9) - decision_judges_v1: tag option and rerun entrypoint for repeat runs (7094ee6e4803b6a4e66bb8b66fe77da8300a158b) - Judge reliability: rerun of the four decision judges on reference v1.2 (c67545f1c0629921d0f69ad0653bd7bb860e25ad) - Judge reliability rerun: source of the combined judge-trust page (f92d1cb0fbf8cf5bbfe1302021e7f2b7763a6e7d) - Judge reliability rerun: report only headline scores against the final labels (3703f7d70a6e7a50b5c790867486d85e7eae3405)dd0db0a
-
Add experiment_requests/ and completed_experiments/ folders (#408)
- Add experiment_requests/ and completed_experiments/ folders (2df315e50fec0622c4d996e73c5d097524d3e9a0) - Move existing experiments into completed_experiments/, grouped by task (be58213965c5cdf5d528547d39ab6015709e18c8)4f9e6db
-
Stress test the solver (B5): the whole corpus and unusual-but-valid input (#350)
- Stress test the solver (B5): the whole corpus and unusual-but-valid input (e321e406aa3fbec1735b6cd04e98738d48925ef1) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>377a40e -
Stress test solver (B4) (#334)
- Stress test the solver (B4): what #178 settled (2f2be8319fe96dc11700c0fbf2e813981bfeb3c9) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>61dad6b
-
Validity set v1: blind labelling sheet, pre-registered scoring, and a Clef/Kev/Laya run (compute request) (#208)
- Validity set v1: blind labelling sheet, a pre-registered analysis, and a Clef run (compute request) (bca887241cf008542bdf203a24232337b8c19a9e) - Run any judge on the validity units: judge_cases.py -T cases=, score_validity.py reads its logs (b19a29d1f379ec19e071bd320add60385f7ee4fc) - judge_cases.py: read a relative cases path from the repo root (75a018402832722da7016515a512f225fe85837e) - score_validity.py: --key splits every judge by source model (#203 gate 1) (a7f2b95972db341f9409c81b863e74757265a2dd) - score_validity.py: a strict majority of three or more labellers settles a unit (a881a3251b222ea1968441db8de373e332ff8942) - Run Clef, Kev-4B and Laya on the validity units with one runner; report long inputs apart (eca0241e1c12f9ec8ecdcfd39d6d4042babb9b0a) - score_validity.py: a unit needs two or more labels to be a reference (e7082f1f9a31bdc72cded84d7c3df2c3ffbc33bd) - score_validity.py: --jury scores a majority vote of several judges (1a67a506889a8a2413582c12d776d93d9e6493dd) C…a903183
-
Scorer cost: cap the judge reply at its budget, add speedtest --same-labels, document measured settings (#254)
- Scorer cost: cap the judge reply at its budget, add speedtest --same-labels, document measured settings (25576287ef3bd314ab8665ccd6746633f5df7278) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>0a0e9d3
-
speedtest.py: warn on logs it can't time, list the slowest samples (#170)
- speedtest.py: warn on logs it can't time, list the slowest samples (7f65ba94af13264b4e6aef35751a1b8d024db732) - speedtest.py: null, not 0, for what a log without samples can't measure (e3d4cb99a5569b858f8931988b7288b0e461e31c) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>25b0d70 -
Score cut-off replies invalid and report invalid turns by reason (#200)
- Score cut-off replies invalid and report invalid turns by reason (1b03d614d9e25c70816b91b901c41ebcc01727fa) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>ae066d9 -
Check the metrics against hand-computed values, in report.py and cb_metrics.py (#198)
- Check the metrics against hand-computed values, in report.py and cb_metrics.py (a3dfa35e30a1f95676a7aacfe7f1b92a4468dda5) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>e5186a0 -
Judge parser: a negated label is not a verdict (#183)
- Judge parser: a negated label is not a verdict (6f63ed6c68f9d97fa435802dee2867f55de2b306) - Parser: scope a label's negation to its clause, not three words (e91a226076dffe2cc5df7ec9b095864af4d598ff) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>7b05cb0
-
Add a solver speed test and report Qwen3.5 4B results on Ollama (#119)
- Add a solver speed test and report Qwen3.5 4B results on Ollama (f65c817320e809671b75d03227efb0098cd23095) - Merge main into PR #119: keep "When samples fail" and "Speed test" (920ec43877747a69036814b86701bb20f80a50dc) - speedtest: clear error when --same-prompts gets a non-OpenAI-style log (492323bd9efef8711c9f51abd53eff2ad10ea075) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>21315cf -
stress test the solver(B3) (#144)
- Handle valid-but-awkward text: line separators, lone surrogates, system_prompt braces (c72dd643671c647fc832fd871c39856e1eff4726) - Leave the system_prompt table row as it was (b75e111a8a7708abb6ee504610bc4516530efdfb) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>8e2de47
-
Score an empty reply as invalid instead of asking the judge (#137)
- Score an empty reply as invalid instead of asking the judge (da82bbd4bdd2a993d5194ce01f799181e2ec3bce) Co-authored-by: Haleema Tallat <powercoding1@users.noreply.huggingface.co>40bcfdd -
Stress test the solver - Infrastructure failures(B2) (#141)
- Bound model-call retries by default and report the root cause of errored samples (f9fada94c23993070d58c4b9f051982b1b9a3610) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>f290118
-
Stress test the solver - Infrastructure failures(B1) (#121)
- Show errored samples in the report and mark cut-off transcripts incomplete (cf800a04d960ddafc8b13ce69b7dd7d6dfb4af07) Co-authored-by: Gowrish Kumar Reddy <gowrishkumar@users.noreply.huggingface.co>be303a8 -
Add local judge injection pilot results (#111)
- Add local judge injection pilot results (bfd211786f35c05ddec3b83b2d5639925a1bd68b) - Add confidence intervals to local judge pilot (c57517c41df5ba41ca4f31bcdb4aebfc6812fb57) - Clarify parser-specific local judge results (3383b51613bc13a3661fe0dbc67b0402349ce8c5) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>e59a8d8
-
Stress test the solver with malformed data and fix the issues (#86)
- Stress-test malformed data edge cases with review fixes (#86) (096716f82ab62e1cca39c6f679973aef1bb41c09) - Merge remote-tracking branch 'origin/main' into pr86-fixes (caf6b582da40a4c113d0f132532fae05da377175) - Evaluate the corpus only by default; pending items only when asked (d38aad64150df756006c646332a6414a521b943e) Co-authored-by: Divyanshi Joshi <divyanshijoshi@users.noreply.huggingface.co>999e97b -
[experiment] roster: Qwen2.5-7B judge and notes from a remote (Freestyle VM) run (#72)
- roster: Qwen2.5-7B judge and notes from a remote (Freestyle VM) run (8a9970616399cd0ac315d24282b33982df67a8b6)310c284 -
Make judge robustness runner model agnostic (#82)
- Make judge robustness runner model agnostic (92a5d80bc28f86b3ff94d21dcbc010fe5ea8a8c7) - Restore LF line endings and drop BOMs (no content change) (d1d0dab9d9555d41945fd0f4699b84c4866e7a1e) - Merge remote-tracking branch 'origin/main' into pr82-fixes (073211d0a5a589c6380de2f7cf1a47ab642acf3e) - Restore reasoning_effort=low for gpt-oss judges; add --reasoning-effort (5dc5d6db515405a8155c2f2b240cd0a1a6c564bd) Co-authored-by: Chaewon Yoon <melover24@users.noreply.huggingface.co>812201f -
Add a context budget checked before each model and judge call (#87)
- Add a context budget checked before each model and judge call (5d47cee3a29907e3d0a7ff58a596f627d7b9514e) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>c6d2cf7
-
Re-land #68 with the solver and scorer in their own folders (#71)
- Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (960f718216c5a02613985d7e4209752396c3a3a7) - Re-land #68 with the solver and scorer in their own folders (29f92190763634945b32cf6f468057e48c73b463) - Merge main (#70 revert) into the re-land branch (42bb2975411d46b8fab03302ee66316fbe562ef5)f66ab8f -
Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (#70)
- Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (960f718216c5a02613985d7e4209752396c3a3a7)6879d90 -
Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)
- Inspect solver and scorer infrastructure: repeated sampling, metrics, report (c3d9a88dda8edab427042e5f4e0af5363426c01f)29572a0
-
Add an Inspect pilot of ConspiracyBench in inspect_eval/ (#54)
- Add an Inspect pilot of ConspiracyBench in inspect_eval/ (8e4e50fb6b14367b7cbe9cc790fab0cfd3c70811) Co-authored-by: EMMANUEL NWANGUMA <Emar7@users.noreply.huggingface.co>b3f14c5