ConspiracyBench / inspect_eval

Commit History

8 days ago

  1. Add inspect_eval/EVAL_SETTINGS.md: the default eval settings in one place (#463)⁠

    0ddfd0f

13 days ago

  1. [experiment] [request for compute] Add /ck/ control experiment for #436 (#454)⁠

    stationhouse +1: vnnviv
    9ad7335

20 days ago

  1. [experiment] Eval: Qwen3.5 4B with thinking off (2026-10-28) (#445)⁠

    stationhouse +1: asdfasdfqrqwer
    0357957
  2. [experiment] [request for compute] Experiment request: evidence-focused instruction evaluation (#441)⁠

    stationhouse +2: melover24, asdfasdfqrqwer
    fb0b1a5
  3. [experiment] Request: LoRA Qwen3.5 4B on 4chan /x/, then re-run the baseline eval (#436)⁠

    b072366

27 days ago

  1. Baseline eval: five ~4B thinking models on ConspiracyBench, graded by pplx-decider (#428)⁠

    233d3a6
  2. [experiment] Judge reliability: diagnosis, label review, rubric experiments, explorer (#409)⁠

    dd0db0a

  1. Add experiment_requests/ and completed_experiments/ folders (#408)⁠

    4f9e6db

  1. Stress test the solver (B5): the whole corpus and unusual-but-valid input (#350)⁠

    stationhouse +1: gowrishkumar
    377a40e
  2. Stress test solver (B4) (#334)⁠

    stationhouse +1: gowrishkumar
    61dad6b

  1. Validity set v1: blind labelling sheet, pre-registered scoring, and a Clef/Kev/Laya run (compute request) (#208)⁠

    stationhouse +1: powercoding1
    a903183

  1. Scorer cost: cap the judge reply at its budget, add speedtest --same-labels, document measured settings (#254)⁠

    stationhouse +1: powercoding1
    0a0e9d3

  1. speedtest.py: warn on logs it can't time, list the slowest samples (#170)⁠

    stationhouse +1: powercoding1
    25b0d70
  2. Score cut-off replies invalid and report invalid turns by reason (#200)⁠

    stationhouse +1: Emar7
    ae066d9
  3. Check the metrics against hand-computed values, in report.py and cb_metrics.py (#198)⁠

    stationhouse +1: powercoding1
    e5186a0
  4. Judge parser: a negated label is not a verdict (#183)⁠

    stationhouse +1: powercoding1
    7b05cb0

  1. Add a solver speed test and report Qwen3.5 4B results on Ollama (#119)⁠

    stationhouse +1: powercoding1
    21315cf
  2. stress test the solver(B3) (#144)⁠

    stationhouse +1: gowrishkumar
    8e2de47

  1. Score an empty reply as invalid instead of asking the judge (#137)⁠

    stationhouse +1: powercoding1
    40bcfdd
  2. Stress test the solver - Infrastructure failures(B2) (#141)⁠

    stationhouse +1: gowrishkumar
    f290118

  1. Stress test the solver - Infrastructure failures(B1) (#121)⁠

    stationhouse +1: gowrishkumar
    be303a8
  2. Add local judge injection pilot results (#111)⁠

    stationhouse +1: melover24
    e59a8d8

  1. Stress test the solver with malformed data and fix the issues (#86)⁠

    stationhouse +1: divyanshijoshi
    999e97b
  2. [experiment] roster: Qwen2.5-7B judge and notes from a remote (Freestyle VM) run (#72)⁠

    310c284
  3. Make judge robustness runner model agnostic (#82)⁠

    stationhouse +1: melover24
    812201f
  4. Add a context budget checked before each model and judge call (#87)⁠

    stationhouse +1: Emar7
    c6d2cf7

  1. Re-land #68 with the solver and scorer in their own folders (#71)⁠

    f66ab8f
  2. Revert "Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)" (#70)⁠

    6879d90
  3. Inspect solver and scorer infrastructure: repeated sampling, metrics, report (#68)⁠

    29572a0

  1. Add an Inspect pilot of ConspiracyBench in inspect_eval/ (#54)⁠

    stationhouse +1: Emar7
    b3f14c5