improving content

ASSESSMENT.md

@@ -1,291 +0,0 @@
-# Critical Assessment: Eleusis Benchmark Article
-
-## Executive Summary
-
-The article presents an interesting benchmark with solid methodology and rich data. The main structural issue is that the **Results section tells a fragmented story about guessing behavior**, spreading related insights across 6+ subsections without a clear narrative arc. The key message—that metacognition matters and models have distinct "scientific personalities"—gets lost in the noise.
-
-Additionally, there are **data consistency issues** between the text and the underlying data files that need resolution before publication.
-
----
-
-## 1. Critical Issues
-
-### 1.1 Data Inconsistencies
-
-The numbers in the text don't match `summary.txt`. For example:
-
-| Metric | In Text | In summary.txt |
-|--------|---------|----------------|
-| Claude Opus 4.5 avg score | 15.88 (CLAUDE.md) | 14.46 |
-| Kimi K2 avg score | 14.53 (CLAUDE.md) | 10.31 |
-| GPT 5.2 High rank | "third place" | Actually 1st by avg_score (14.85) |
-
-**Action needed:** Audit all numbers in the text against the latest data files.
-
-### 1.2 Results Section: Scattered Narrative
-
-The guessing behavior story is currently spread across:
-
-1. "Confidence and Calibration" - calibration curves, confidence distribution
-2. "Guessing Strategy" - score vs failed guesses
-3. "The Caution-Recklessness Trade-off" - early correct turns, caution scatter
-4. "Alternative Scoring Systems" - score stack breakdown
-5. "Analysis of the reckless guessing behavior" - double-down rate
-
-These all address the same fundamental question: **How do models decide when to commit?** But the current structure forces readers to piece together the story themselves.
-
-**Problem:** A reader finishing the Results section doesn't have a clear mental model of "what makes some models better than others."
-
----
-
-## 2. Suggested Restructuring
-
-### Option A: Reorganize Around the Key Insight
-
-**Proposed Results structure:**
-
-```
-## Results
-
-### Overall Performance (keep as-is)
-   Brief overview, scatter plot of score vs tokens
-
-### Finding the Rule: Who Gets It Right?
-   - Success rates by model
-   - Performance by rule complexity
-   - Brief: what capabilities matter for finding rules
-
-### Knowing When You Know: The Metacognition Challenge
-   [This is the heart of the article - elevate it]
-   - The caution-recklessness trade-off (central framing)
-   - Caution analysis: early correct turns, GPT 5.2 waits too long
-   - Recklessness analysis: failed guesses, double-down rates
-   - The scatter plot showing the trade-off (Figure 6)
-   - Why Claude Opus wins: good enough at finding + great at timing
-
-### Confidence and Calibration
-   - Calibration curves (all models overconfident)
-   - Confidence distribution when guessing
-   - Brief: why calibration enables good timing decisions
-
-### Alternative Scoring: Robustness Check
-   - Score stack shows the penalty different behaviors pay
-   - Confirms that metacognition, not just rule-finding, drives scores
-```
-
-**Benefits:**
-- The key message (metacognition matters) becomes structurally prominent
-- Reader builds understanding progressively: first "can they solve it?", then "do they know when they've solved it?"
-- Eliminates the feeling of "lots of charts, hard to synthesize"
-
-### Option B: Two-Act Structure
-
-```
-## Results
-
-### Act 1: The Leaderboard (compact)
-   - Overall performance scatter
-   - Success rates
-   - One paragraph summary: "Models vary from 70% to 96% success rate..."
-
-### Act 2: The Real Story—Scientific Temperaments
-   [Frame models as having distinct "personalities"]
-
-   The Cautious Achiever: GPT 5.2 High
-   - Highest success rate, but 3rd in score
-   - Figure: excess caution distribution
-   - Lost ~3.6 points per round to over-caution
-
-   The Balanced Scientist: Claude Opus 4.5
-   - Not the best at finding rules, but best at knowing when
-   - Commits quickly, accepts occasional wrong guesses
-
-   The Reckless Guesser: Claude Haiku 4.5 / DeepSeek R1
-   - Commits before sufficient evidence
-   - Double-down behavior after failures
-
-   Visualizing the Trade-off
-   - Caution vs recklessness scatter (the key figure)
-   - Score stack showing what each "personality" costs
-
-### Calibration: Why Timing Is Hard
-   - Overconfidence makes timing decisions unreliable
-   - Even well-performing models poorly calibrated
-```
-
-**Benefits:**
-- Memorable framing (scientific personalities)
-- Natural story arc
-- Each model type is clearly characterized
-
----
-
-## 3. Missing Content
-
-### 3.1 Figures Marked as TODO
-
-- **Learning curves figure** (analysis.mdx:22) - Would show within-round dynamics
-- **Failure mode distribution** (analysis.mdx:55) - Stacked bar by model
-
-**Recommendation:** The learning curves figure would be valuable if you have the data. The failure mode classification might be hard to automate reliably—consider whether a few qualitative examples serve the purpose better.
-
-### 3.2 Human Baseline
-
-Mentioned in limitations but this is a significant gap. Without human performance, readers can't judge if 92% success is impressive or trivial.
-
-**Options:**
-- Run a small human study (even N=5 would help)
-- Cite related work on human performance in similar inductive reasoning tasks
-- Frame it explicitly as "relative comparison between models" not absolute capability assessment
-
-### 3.3 Example Turn Figure
-
-benchmark.mdx shows the JSON output format but doesn't illustrate what a complete turn looks like in context (game state → reasoning → decision).
-
-**Recommendation:** Add a figure showing:
-```
-[Current board state visualization]
-[Model reasoning excerpt]
-[Decision: play 4♣, confidence 6, don't guess yet]
-[Outcome: accepted/rejected]
-```
-
-This makes the task concrete for readers.
-
----
-
-## 4. The "Deeper Analysis" Section
-
-Currently a grab-bag of interesting observations with TODOs. Your instinct to replace with "Discussion" is right.
-
-### Proposed: Discussion Section
-
-```
-## Discussion
-
-### What Explains the Performance Gap?
-   - Metacognition (knowing when you know) is the key differentiator
-   - Success rate alone doesn't predict score (GPT 5.2 vs Opus example)
-   - Calibration enables good timing, but no model is well-calibrated
-
-### Open vs Proprietary Models
-   - Kimi K2 competitive on rule-finding
-   - But open models trend toward reckless guessing (training objective differences?)
-   - Opportunity: calibration tuning could improve open model performance
-
-### Failure Modes [keep the accordion, it's useful]
-
-### Implications for AI-Assisted Science
-   - The caution-recklessness trade-off mirrors real scientific decision-making
-   - An overconfident AI assistant could lead researchers astray
-   - An overcautious one wastes resources on unnecessary verification
-```
-
-### Move to Appendix
-
-- Symmetric rules analysis (interesting but niche)
-- Confirmation bias (preliminary, needs more work)
-- Detailed qualitative examples (unless you expand them significantly)
-
----
-
-## 5. Framing Suggestions
-
-### 5.1 Lead with the Surprise
-
-Current opening of Results is fine, but the key insight (metacognition matters) comes too late. Consider foreshadowing in the introduction:
-
-> "We found something surprising: the model with the highest success rate doesn't have the highest score. What matters isn't just finding the answer—it's knowing when you've found it."
-
-### 5.2 The "Scientific Personality" Frame
-
-This is potentially memorable and shareable. Models as:
-- **The Perfectionist** (GPT 5.2 High): Always wants more evidence
-- **The Pragmatist** (Claude Opus 4.5): Good enough evidence is enough
-- **The Gambler** (Claude Haiku 4.5): Guesses based on vibes
-
-This framing:
-- Makes the article more accessible to non-specialists
-- Creates natural anchors for discussion
-- Is scientifically defensible (behavioral clustering is real)
-
-### 5.3 The Decision Theory Angle
-
-You mention the optimal guessing threshold (0.67 confidence) briefly. This could be expanded:
-
-> "Given perfect calibration, the optimal strategy is to guess whenever confidence exceeds 67%. But no model is well-calibrated. GPT 5.2 High effectively uses a threshold of ~95%; Claude Haiku 4.5 seems to use ~50%."
-
-This quantifies the "personalities" and connects to calibration.
-
----
-
-## 6. Minor Issues
-
-### 6.1 Typos/Grammar
-
-- results.mdx:38: "overconfident : for instance" → extra space before colon
-- results.mdx:39: "GPT 5.2 is the best calibrated" → should be "GPT 5.2 High"
-- results.mdx:51: "closed to Claude Opus 4.5" → "close to"
-- results.mdx:103: "constrats" → "contrasts"
-- analysis.mdx:60: "GPT OSS 120B also performs respectably at 12.0" → check number
-
-### 6.2 Caption Numbering
-
-Figure 7 appears twice (score-stack and reckless-guessing). Fix numbering.
-
-### 6.3 Model Names Consistency
-
-Inconsistent capitalization and naming:
-- "Claude Opus 4.5" vs "Claude 4.5 Opus"
-- "GPT 5.2 High" vs "Gpt 5.2 High" (in data files)
-- "DeepSeek R1" vs "Deepseek R1"
-
----
-
-## 7. Ideas for Additional Content
-
-### 7.1 Interactive "Play a Round" Demo
-
-Let readers play one round against a rule to experience the task. Even a simple version would be compelling. (This could be a stretch goal.)
-
-### 7.2 Model-Specific Breakdowns
-
-You have per-model PNG files (`model_claude_opus_4_5.png`, etc.). Consider:
-- Appendix section with one page per model
-- Or: expandable accordion for each model's detailed stats
-
-### 7.3 Token Efficiency Discussion
-
-You show score vs tokens in Figure 1 but don't discuss it much. Gemini 3 Flash achieves decent results with 4x fewer tokens than Opus—is that worth highlighting for practitioners?
-
-### 7.4 Prompt Sensitivity
-
-You note this as a limitation but could briefly test: what if you told models to be more cautious? More aggressive? (Could be future work suggestion.)
-
----
-
-## 8. Prioritized Action Items
-
-### Must Fix
-1. Audit all numbers against latest data files
-2. Fix duplicate Figure 7 numbering
-3. Fix typos listed above
-
-### Should Do
-4. Reorganize Results section (Option A or B above)
-5. Rename "Deeper Analysis" to "Discussion" and restructure
-6. Add foreshadowing of key insight in introduction
-
-### Nice to Have
-7. Add example turn figure in benchmark.mdx
-8. Expand "scientific personalities" framing
-9. Human baseline (even informal)
-10. Per-model detail pages in appendix
-
----
-
-## 9. Summary
-
-The benchmark and data are solid. The article's main weakness is structural: it has too many charts telling pieces of the same story without a clear narrative spine. The fix is to reorganize around **the key insight** (metacognition matters more than raw rule-finding ability) and **the key visual** (the caution-recklessness scatter plot).
-
-Your target message—"Models differ dramatically because metacognition matters, and this is an opportunity for improvement"—is supported by the data but not yet prominently surfaced by the article structure.

ASSESSMENT_V2.md

@@ -1,197 +0,0 @@
-# Revised Assessment: Eleusis Benchmark Article (v2)
-
-## Executive Summary
-
-The article has improved significantly since the first assessment. The **Results section is now well-structured** with a clear narrative arc: overall performance → the metacognition insight → caution/recklessness trade-off → calibration → performance by rule. The key message about metacognition is now prominent and supported by the logical flow.
-
-The main remaining issues are:
-1. **Data inconsistencies** between text and data files (numbers are outdated)
-2. **The "Deeper Analysis" section** needs restructuring—much of it now duplicates the improved Results section
-3. Minor typos
-
----
-
-## 1. What's Working Well
-
-### 1.1 Results Section Structure
-The new structure is excellent:
-```
-Results
-├── Overall Performance (intro)
-├── Pure discovery vs metacognition (the key insight, early!)
-├── Caution-Recklessness Trade-off (central analysis)
-├── Confidence and Calibration (supporting evidence)
-└── Performance by Rule (rule-level breakdown)
-```
-
-This addresses the main criticism from v1: readers now build understanding progressively and the metacognition insight is front and center.
-
-### 1.2 Figure Flow
-Figures now tell a coherent story:
-- Fig 1: Overview (where does each model sit?)
-- Fig 2: Score breakdown (what drives score differences?)
-- Fig 3: Caution vs recklessness (the key trade-off)
-- Fig 4: Calibration (why is timing hard?)
-- Fig 5: Guess rate (how do models decide when to commit?)
-- Fig 6-7: Rule-level analysis (drill-down)
-
-### 1.3 New Guess Rate Analysis (Figure 5)
-This is a valuable addition that wasn't in the original. It shows how models operationalize their confidence into actual decisions, connecting calibration to behavior.
-
-### 1.4 Clear Messaging
-Lines like "knowing when to commit is as important as finding the rule" now appear early and are reinforced throughout.
-
----
-
-## 2. Critical Issues
-
-### 2.1 Data Inconsistencies (Must Fix)
-
-The text still uses outdated numbers. Current data (from `summary.txt` and `overall_performance.json`) vs text:
-
-| Metric | In Text | Actual Data |
-|--------|---------|-------------|
-| Claude Opus 4.5 avg score | 15.9 (conclusion.mdx:10) | **17.0** (avg_floored_score) |
-| Claude Opus 4.5 success rate | 92% (conclusion.mdx:10) | **83%** |
-| Claude Haiku 4.5 success rate | 70% (conclusion.mdx:10) | **56%** |
-| Claude Haiku 4.5 failed guesses | 7.5/round (analysis.mdx:15) | **3.95/round** |
-| Kimi K2 avg score | 14.5 (analysis.mdx:60) | **16.2** |
-| GPT OSS 120B score | 12.0 (analysis.mdx:60) | **12.9** |
-| GPT 5.2 High early correct turns | 3.6 (multiple places) | **3.56** ✓ (close enough) |
-
-**Action:** Audit all numbers in `results.mdx`, `analysis.mdx`, and `conclusion.mdx` against the latest data files.
-
-### 2.2 Typos Still Present
-
-| Location | Issue |
-|----------|-------|
-| results.mdx:20 | "closed to Claude Opus 4.5" → "close to" |
-| results.mdx:85 | "overconfident : for instance" → remove space before colon |
-| results.mdx:86 | "GPT 5.2 is the best calibrated" → "GPT 5.2 High" |
-| results.mdx:102 | "THis is somehow" → "This is somehow" |
-
----
-
-## 3. The "Deeper Analysis" Section
-
-### 3.1 Current Problem
-
-The "Deeper Analysis" section is now partially redundant. It covers:
-1. **Metacognition** (duplicates Results § "Pure discovery vs metacognition")
-2. **Learning Curves** (TODO, placeholder)
-3. **Failure Modes** (valuable, keep)
-4. **Open vs Closed Models** (brief, could be expanded)
-5. **Symmetric Rules** (interesting niche finding)
-6. **Confirmation Bias** (preliminary, incomplete)
-7. **Qualitative Observations** (nice examples, but disconnected)
-
-### 3.2 Recommended Restructure
-
-Rename to "Discussion" and reorganize:
-
-```markdown
-## Discussion
-
-### What Explains the Performance Gap?
-- Brief synthesis: metacognition > raw ability
-- The caution-recklessness trade-off determines ranking more than success rate
-- Move the GPT 5.2 High / Claude Opus 4.5 / Claude Haiku 4.5 characterizations here
-  (but avoid repeating numbers already in Results)
-
-### Scientific Temperaments
-- This is where the "scientific personality" framing could shine
-- The Perfectionist (GPT 5.2 High): needs too much evidence
-- The Pragmatist (Claude Opus 4.5): good-enough is good enough
-- The Gambler (Claude Haiku 4.5): acts on insufficient evidence
-- Link to real-world science: these map to actual failure modes in research
-
-### Failure Modes [keep the accordion, it's excellent]
-- Already well-written, just tighten the taxonomy
-
-### Open vs Proprietary Models
-- Currently too brief (1 paragraph)
-- Could expand: why might open models trend reckless? (RLHF differences?)
-- Kimi K2's success is notable—worth highlighting more
-
-### Implications for AI-Assisted Science
-- Currently in Conclusion but could be expanded here
-- An overconfident assistant leads researchers astray
-- An overcautious assistant wastes resources
-- The calibration problem is particularly concerning
-
-### Move to Appendix (or delete)
-- Learning Curves (TODO) → either implement or remove
-- Symmetric Rules → niche, move to appendix or cut
-- Confirmation Bias → too preliminary, either expand significantly or cut
-- Qualitative Observations → keep 1-2 good examples, cut the rest
-```
-
-### 3.3 Delete the Redundancy
-
-The current Metacognition subsection (analysis.mdx:7-16) largely repeats what's now better expressed in Results. Either:
-- Delete it entirely and rely on Results
-- Or transform it into the "Scientific Temperaments" narrative frame (more memorable)
-
----
-
-## 4. Missing Content (Lower Priority)
-
-### 4.1 TODOs Still Present
-- Learning curves figure (analysis.mdx:22) — either implement or remove the placeholder
-- Failure mode distribution stacked bar (analysis.mdx:55) — nice to have, not critical
-
-### 4.2 Human Baseline
-Still missing. Consider adding a sentence like: "Without human performance data on the same rules, we cannot assess whether these success rates represent strong or weak performance in absolute terms—only that models differ substantially among themselves."
-
-### 4.3 Example Turn Figure
-Would still be valuable in benchmark.mdx to make the task concrete. A simple 3-panel showing:
-```
-[Board state] → [Model reasoning excerpt] → [Decision output]
-```
-
----
-
-## 5. Minor Polish
-
-### 5.1 Model Name Consistency
-Some inconsistencies remain:
-- "Grok 4.1 Fast Reasoning" vs "Grok 4 1 Fast Reasoning" (in data)
-- "DeepSeek R1" vs "Deepseek R1" (in data)
-- Decide on one capitalization style and apply consistently
-
-### 5.2 The "floored" Score
-The article doesn't explain that scores below 0 are floored to 0. This affects interpretation—might be worth a brief mention in the Benchmark section or a sidenote.
-
-### 5.3 Sidenote on Optimal Threshold
-Results.mdx mentions the 0.67 optimal threshold but doesn't explain why. A brief derivation in a sidenote would help:
-> For a perfectly calibrated model: E[guess at p] = p×(points remaining) - (1-p)×2. Setting E[guess] > E[wait 1 turn] gives p > 2/3 ≈ 0.67.
-
----
-
-## 6. Summary of Recommended Actions
-
-### Must Do
-1. ☐ Fix all data inconsistencies (audit numbers against data files)
-2. ☐ Fix typos listed in §2.2
-3. ☐ Remove or transform redundant content in "Deeper Analysis"
-
-### Should Do
-4. ☐ Rename "Deeper Analysis" → "Discussion"
-5. ☐ Restructure Discussion per §3.2
-6. ☐ Either implement Learning Curves figure or remove the TODO
-
-### Nice to Have
-7. ☐ Add "Scientific Temperaments" framing
-8. ☐ Add example turn figure in benchmark.mdx
-9. ☐ Explain the score flooring mechanism
-10. ☐ Expand Open vs Proprietary discussion
-
----
-
-## 7. Overall Assessment
-
-**Grade: B+ (up from B-)**
-
-The structural problems identified in v1 are largely resolved. The article now tells a clear story: models vary in their "scientific temperament," and metacognition—knowing when you know—matters as much as raw reasoning ability.
-
-The remaining work is mostly cleanup (data consistency, typos) and deciding what to do with the Deeper Analysis section. The article is close to publication-ready once the numbers are fixed.

app/astro.config.mjs

@@ -65,7 +65,6 @@ export default defineConfig({
         bibliography: 'src/content/bibliography.bib',
         linkCitations: true,
         csl: "apa",
-        noCite: false,
         suppressBibliography: false,
       }],
       rehypeReferencesAndFootnotes,

app/src/content/article.mdx

@@ -1,6 +1,6 @@
 ---
-title: "Are LLMs any good at the Science Game?"
-subtitle: "Evaluating scientific reasoning using the card game Eleusis"
+title: "Are LLMs any good at the Game of Science?"
+subtitle: "Evaluating scientific reasoning and metacognition using the card game Eleusis"
 description: "A benchmark for evaluating LLM scientific reasoning using the card game Eleusis, testing iterative hypothesis formation, calibration, and strategic experimentation."
 authors:
   - name: "David Louapre"
@@ -25,7 +25,7 @@ showPdf: true
 import Introduction from "./chapters/eleusis/introduction.mdx";
 import Benchmark from "./chapters/eleusis/benchmark.mdx";
 import Results from "./chapters/eleusis/results.mdx";
-import Analysis from "./chapters/eleusis/analysis.mdx";
+import Discussion from "./chapters/eleusis/discussion.mdx";
 import Conclusion from "./chapters/eleusis/conclusion.mdx";
 import Appendix from "./chapters/eleusis/appendix.mdx";
 
@@ -35,7 +35,7 @@ import Appendix from "./chapters/eleusis/appendix.mdx";
 
 <Results />
 
-<Analysis />
+<Discussion />
 
 <Conclusion />
 

app/src/content/bibliography.bib

@@ -1,130 +1,66 @@
-@inproceedings{vaswani2017attention,
-  title     = {Attention Is All You Need},
-  author    = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {
-               }Lukasz and Polosukhin, Illia},
-  booktitle = {Advances in Neural Information Processing Systems},
-  year      = {2017}
-}
-
-@book{mckinney2017python,
-  title     = {Python for Data Analysis},
-  author    = {McKinney, Wes},
-  publisher = {O'Reilly Media},
-  address   = {Sebastopol, CA},
-  year      = {2017},
-  edition   = {2},
-  isbn      = {978-1491957660}
-}
-
-@inproceedings{he2016resnet,
-  title     = {Deep Residual Learning for Image Recognition},
-  author    = {He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
-  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
-  pages     = {770--778},
-  year      = {2016},
-  doi       = {10.1109/CVPR.2016.90},
-  url       = {https://doi.org/10.1109/CVPR.2016.90}
-}
-
-@article{silver2017mastering,
-  title   = {Mastering the game of Go without human knowledge},
-  author  = {Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and others},
-  journal = {Nature},
-  volume  = {550},
-  number  = {7676},
-  pages   = {354--359},
-  year    = {2017},
-  month   = {oct},
-  doi     = {10.1038/nature24270},
-  url     = {https://www.nature.com/articles/nature24270}
-}
-
-@techreport{openai2023gpt4,
-  title         = {GPT-4 Technical Report},
-  author        = {{OpenAI}},
-  institution   = {OpenAI},
-  year          = {2023},
-  number        = {arXiv:2303.08774},
+@misc{chollet2019measure,
+  title         = {On the Measure of Intelligence},
+  author        = {Chollet, François},
+  year          = {2019},
+  howpublished  = {arXiv preprint},
   archiveprefix = {arXiv},
-  eprint        = {2303.08774},
-  primaryclass  = {cs.CL},
-  url           = {https://arxiv.org/abs/2303.08774}
-}
-
-@phdthesis{doe2020thesis,
-  title   = {Learning Efficient Representations for Large-Scale Visual Recognition},
-  author  = {Doe, Jane},
-  school  = {Massachusetts Institute of Technology},
-  address = {Cambridge, MA},
-  year    = {2020},
-  doi     = {10.5555/mit-2020-xyz}
-}
-
-@incollection{cover2006entropy,
-  title     = {Entropy, Relative Entropy, and Mutual Information},
-  author    = {Cover, Thomas M. and Thomas, Joy A.},
-  booktitle = {Elements of Information Theory},
-  publisher = {Wiley},
-  address   = {Hoboken, NJ},
-  edition   = {2},
-  year      = {2006},
-  pages     = {13--55},
-  isbn      = {978-0471241959}
+  eprint        = {1911.01547},
+  primaryclass  = {cs.AI},
+  url           = {https://arxiv.org/abs/1911.01547}
 }
 
-@misc{zenodo2021dataset,
-  title        = {ImageNet-21K Subset (Version 2.0)},
-  author       = {Smith, John and Lee, Alice and Kumar, Ravi},
-  year         = {2021},
-  howpublished = {Dataset on Zenodo},
-  doi          = {10.5281/zenodo.1234567},
-  url          = {https://doi.org/10.5281/zenodo.1234567},
-  note         = {Accessed 2025-09-01}
+@article{abbott1977eleusis,
+  title   = {The New Eleusis},
+  author  = {Abbott, Robert},
+  journal = {Games \& Puzzles},
+  year    = {1977},
+  note    = {Updated rules for the Eleusis card game, originally published in 1956}
 }
 
-@misc{sklearn2024,
-  title        = {scikit-learn: Machine Learning in Python (Version 1.4)},
-  author       = {Pedregosa, Fabian and Varoquaux, Ga{"e}l and Gramfort, Alexandre and others},
-  year         = {2024},
-  howpublished = {Software},
-  doi          = {10.5281/zenodo.592264},
-  url          = {https://scikit-learn.org}
-}
-
-@inproceedings{smith2024privacy,
-  title     = {Privacy-Preserving Training with Low-Precision Secure Aggregation},
-  author    = {Smith, Emily and Zhang, Wei and Rossi, Marco and Patel, Neha},
-  booktitle = {Proceedings of the 41st International Conference on Machine Learning},
-  editor    = {Smith, A. and Johnson, B.},
+@inproceedings{guo2017calibration,
+  title     = {On Calibration of Modern Neural Networks},
+  author    = {Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q.},
+  booktitle = {Proceedings of the 34th International Conference on Machine Learning},
   series    = {Proceedings of Machine Learning Research},
-  volume    = {235},
-  pages     = {12345--12367},
-  address   = {Vienna, Austria},
+  volume    = {70},
+  pages     = {1321--1330},
+  year      = {2017},
   publisher = {PMLR},
-  month     = {jul},
-  year      = {2024},
-  url       = {https://proceedings.mlr.press/v235/}
+  url       = {https://proceedings.mlr.press/v70/guo17a.html}
 }
 
-@article{kingma2015adam,
-  title         = {Adam: A Method for Stochastic Optimization},
-  author        = {Kingma, Diederik P. and Ba, Jimmy},
-  journal       = {International Conference on Learning Representations (ICLR)},
-  year          = {2015},
-  archiveprefix = {arXiv},
-  eprint        = {1412.6980},
-  primaryclass  = {cs.LG},
-  url           = {https://arxiv.org/abs/1412.6980}
+@article{nickerson1998confirmation,
+  title   = {Confirmation Bias: A Ubiquitous Phenomenon in Many Guises},
+  author  = {Nickerson, Raymond S.},
+  journal = {Review of General Psychology},
+  volume  = {2},
+  number  = {2},
+  pages   = {175--220},
+  year    = {1998},
+  doi     = {10.1037/1089-2680.2.2.175},
+  url     = {https://journals.sagepub.com/doi/abs/10.1037/1089-2680.2.2.175}
 }
 
-@misc{raffel2020t5,
-  title         = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
-  author        = {Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and others},
-  year          = {2020},
-  howpublished  = {arXiv preprint},
-  archiveprefix = {arXiv},
-  eprint        = {1910.10683},
-  primaryclass  = {cs.LG},
-  doi           = {10.48550/arXiv.1910.10683},
-  url           = {https://arxiv.org/abs/1910.10683}
+@article{flavell1979metacognition,
+  title   = {Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry},
+  author  = {Flavell, John H.},
+  journal = {American Psychologist},
+  volume  = {34},
+  number  = {10},
+  pages   = {906--911},
+  year    = {1979},
+  doi     = {10.1037/0003-066X.34.10.906},
+  url     = {https://psycnet.apa.org/record/1980-09388-001}
+}
+
+@article{lichtenstein1977calibration,
+  title   = {Do Those Who Know More Also Know More About How Much They Know?},
+  author  = {Lichtenstein, Sarah and Fischhoff, Baruch},
+  journal = {Organizational Behavior and Human Performance},
+  volume  = {20},
+  number  = {2},
+  pages   = {159--183},
+  year    = {1977},
+  doi     = {10.1016/0030-5073(77)90001-0},
+  url     = {https://www.sciencedirect.com/science/article/abs/pii/0030507377900010}
 }

app/src/content/chapters/eleusis/analysis.mdx

@@ -1,100 +0,0 @@
-import Note from "../../../components/Note.astro";
-import Sidenote from "../../../components/Sidenote.astro";
-import Accordion from "../../../components/Accordion.astro";
-
-## Deeper Analysis
-
-### Metacognition: Knowing What You Know
-
-The caution-recklessness analysis reveals that metacognition, the ability to accurately assess one's own knowledge, is a key differentiator between models. Consider two extremes:
-
-**GPT 5.2 High** has excellent rule-finding ability (96% success rate), fairly good calibration, but is overly cautious. It averages 3.6 turns after discovering the correct rule before making the guess, leading to lost points despite its high accuracy. It frequently discovers the correct rule early but doesn't *believe* it has sufficient evidence. This excessive caution costs an average of 3.6 points per successful round, enough to drop it from first to third place overall.
-
-**Claude Opus 4.5** achieves the best balance: high success rate (92%) with well-calibrated timing (only 0.9 early correct turns and 2.8 failed guesses). It is not as well calibrated as GPT 5.2 High, and would benefit from better calibration while keeping its slightly risk-taking approach.
-
-**Claude Haiku 4.5** has the opposite problem: poor rule-finding (70% success) combined with overconfident metacognition. It commits to guesses without sufficient evidence, accumulating 7.5 failed guesses per round on average—the highest of any model.
-
-### Learning Curves
-
-How do models improve within a single round? We tracked confidence and hypothesis quality over turn number to understand the learning dynamics.
-
-<Note variant="info">
-  **TODO**: Add figure showing line plot of average confidence by turn number, colored by eventual success/failure.
-</Note>
-
-Key observations:
-- **Successful rounds** typically show steadily increasing confidence with occasional drops when hypotheses are revised
-- **Failed rounds** often show erratic confidence or premature plateaus where models become stuck on incorrect hypotheses
-- **Acceptance rate decreases** over time as obvious cards are exhausted from the hand
-
-<Sidenote>
-  The turn-by-turn reasoning traces provide rich data for understanding model behavior beyond simple success/failure metrics.
-</Sidenote>
-
-### Failure Modes
-
-When models fail, why? We identified several recurring patterns:
-
-<Accordion title="Failure mode taxonomy" open>
-
-1. **Premature guessing**: High confidence, wrong rule, insufficient evidence. The model becomes convinced too early based on limited data. This is the dominant failure mode for Claude Haiku 4.5 (7.5 failed guesses/round).
-
-2. **Hypothesis fixation**: Stuck on wrong rule despite contradictory evidence. The model fails to update when new observations conflict with its theory.
-
-3. **Overfitting**: Rule matches all observations but is more specific than the actual rule (e.g., guessing "only red hearts" when the rule is "only red cards").
-
-4. **Underfitting**: Rule is too simple and fails to capture necessary conditions (e.g., guessing "black cards" when rule is "black even cards").
-
-5. **Position blindness**: Fails on rules depending on position in mainline or relationship to previous cards.
-
-6. **Excessive caution**: The model finds the correct rule but doesn't trust its conclusion. GPT 5.2 High exemplifies this—waiting an average of 3.6 turns after finding the answer, costing significant points.
-
-</Accordion>
-
-<Note variant="info">
-  **TODO**: Add stacked bar chart showing distribution of failure modes by model.
-</Note>
-
-### Open vs Closed Models
-
-A notable finding is the competitive performance of open-weight models. Kimi K2, available with open weights, achieves the second-highest score (14.5) and outperforms several proprietary models including GPT 5 Mini Medium and Gemini 3 Flash. The open-weight GPT OSS 120B also performs respectably at 12.0.
-
-However, open models tend toward more aggressive guessing strategies. Kimi K2 averages 4.0 failed guesses per round (vs. 2.8 for Claude Opus 4.5), and GPT OSS 20B has 6.2. This may reflect differences in training objectives or RLHF tuning between open and proprietary models.
-
-### Symmetric Rules
-
-An interesting test: are symmetric rules equally difficult? For example, "only spades" vs "only non-spades" should be logically equivalent in difficulty, but models might have biases.
-
-We found that:
-- Negative rules ("not X") are generally harder than positive rules ("only X")
-- Rules involving rare events (low acceptance rate) are harder than rules with high acceptance rates
-- This may reflect training data biases where positive examples are more common
-
-### Confirmation Bias
-
-Do models exhibit confirmation bias—preferring to play cards that confirm their current hypothesis rather than cards that could falsify it?
-
-<Sidenote>
-  A good scientist designs experiments that could prove them wrong, not just experiments that confirm what they already believe.
-</Sidenote>
-
-Preliminary analysis suggests:
-- Models do show some tendency toward confirmation-seeking behavior
-- When confident in a hypothesis, models prefer "safe" plays that are likely to be accepted
-- Strategic exploration (playing cards specifically to test hypothesis boundaries) is rare
-
-### Qualitative Observations
-
-Examining individual reasoning traces reveals interesting patterns:
-
-<Accordion title="Example: Hypothesis revision">
-
-In one game with the rule "alternating odd/even ranks," a model initially hypothesized "increasing ranks" based on the first few accepted cards. When a lower-ranked card was accepted, instead of abandoning the hypothesis entirely, the model revised it to "ranks must differ from previous." This partial update eventually led to discovering the true rule—a good example of iterative refinement.
-
-</Accordion>
-
-<Accordion title="Example: Fixation failure">
-
-With the rule "only face cards (J, Q, K)," one model became fixated on "only red cards" after the first three accepted cards happened to be red face cards. Despite subsequently seeing black face cards accepted, the model kept trying to reconcile observations with a color-based rule, eventually running out of turns.
-
-</Accordion>

app/src/content/chapters/eleusis/appendix.mdx

@@ -1,5 +1,6 @@
 import Accordion from "../../../components/Accordion.astro";
 import Note from "../../../components/Note.astro";
+import Sidenote from "../../../components/Sidenote.astro";
 
 ## Appendix: Detailed Methods
 
@@ -26,50 +27,50 @@ All models were evaluated with the following settings:
 
 | Parameter | Value |
 |-----------|-------|
-| Temperature | 0.0 (deterministic) |
-| Max tokens | 4096 |
+| Temperature | 0.7 |
+| Max tokens | 16384 |
 | Retries | 3 (on API failures) |
 
-Reasoning models were allowed their default reasoning budgets. Standard models used base inference without additional chain-of-thought prompting beyond what's included in the game prompt.
+Reasoning models were allowed their default reasoning budgets.
 
 </Accordion>
 
 ### Rule Checking
 
-<Accordion title="Rule verification methodology">
+
 
 Rules are created by hand and expressed in natural language. Each rule is then compiled into a Python function using an LLM, with manual verification of correctness.
 
-When the model outputs a guessed rule, we:
-1. Compile the guess into a Python function using the same LLM
-2. Test the compiled function against all cards played in that game
-3. Mark the guess as correct only if it matches the true rule's behavior on all observations
+When the model outputs a guessed rule, we
+    TODO: explain
 
-This simulation-based approach avoids issues with semantic equivalence in natural language. For instance, "same color as previous card" and "red cards only" might be equivalent given a specific game history starting with a red card, but would differ on other histories.
+This simulation-based approach avoids issues with semantic equivalence in natural language. For instance, "same color as previous card" and "red cards only" might be equivalent given a specific game history starting with a red card, but would differ on other histories. Also the simulation approach avoids declaring as different two rules that, given the current state of the game, will behave identically. For instance if the rule was "same color", as soon as the first card has been drawn, the model might guess "red cards only" or "black cards only" depending on the color of the first card. Both guesses are semantically different but functionally equivalent given the current game state.
 
-</Accordion>
 
-### Prompt Structure
 
-<Accordion title="Full prompt template">
+### Additional results
 
-The prompt includes:
+#### Learning Curves
 
-1. **Game rules**: Complete explanation of how Eleusis works, without mentioning the game's name to avoid potential training data leakage
+How do models improve within a single round? We tracked confidence and hypothesis quality over turn number to understand the learning dynamics.
 
-2. **Scoring system**: Explicit explanation of the scoring formula and strategic implications
+<Note variant="info">
+    **TODO**: Add figure showing line plot of average confidence by turn number, colored by eventual success/failure.
+</Note>
 
-3. **Response format**: JSON schema specifying required fields (reasoning, card choice, tentative rule, confidence, guess decision)
+Key observations:
+- **Successful rounds** typically show steadily increasing confidence with occasional drops when hypotheses are revised
+- **Failed rounds** often show erratic confidence or premature plateaus where models become stuck on incorrect hypotheses
+- **Acceptance rate decreases** over time as obvious cards are exhausted from the hand
 
-4. **Game state**: Current mainline, all sidelines, current hand, and reasoning from the previous 3 turns
+<Sidenote>
+    The turn-by-turn reasoning traces provide rich data for understanding model behavior beyond simple success/failure metrics.
+</Sidenote>
 
-5. **Format reminders**: Instructions for confidence scale interpretation (7 = 70% probability)
 
-</Accordion>
 
 ### Evaluation Metrics
 
-<Accordion title="Metric definitions">
 
 - **Success rate**: Fraction of games where the model correctly identified the rule before running out of turns
 
@@ -81,22 +82,6 @@ The prompt includes:
 
 - **Turns to success**: For successful games, mean number of turns before correct guess
 
-</Accordion>
-
-### References
-
-<Accordion title="Bibliography">
-
-- Abbott, R. (1963). "Eleusis" — Original game rules and design philosophy
-
-- Guo, C., et al. (2017). "On Calibration of Modern Neural Networks" — Foundational work on neural network calibration
-
-- Chollet, F. (2019). "On the Measure of Intelligence" — ARC benchmark and discussion of abstract reasoning
-
-- Recent LLM reasoning benchmarks: GSM8K, MATH, ARC-AGI, BIG-Bench, etc.
-
-</Accordion>
-
 <Note>
   Full code, data, and model outputs are available in the benchmark repository.
 </Note>

app/src/content/chapters/eleusis/benchmark.mdx

@@ -1,20 +1,18 @@
 import Sidenote from "../../../components/Sidenote.astro";
-import Note from "../../../components/Note.astro";
-import Accordion from "../../../components/Accordion.astro";
 
-## The Eleusis Benchmark
+## 1. The Eleusis Benchmark
 
 ### The Original Game
 
-In the original Eleusis card game, one player acts as the "dealer" (sometimes called "God" or "Nature") and secretly invents a rule determining which cards can be legally played. The other players don't know this rule—they must discover it through experimentation.
+In the original Eleusis card game, one player acts as the "dealer" (sometimes called "God" or "Nature") and secretly invents a rule determining which cards can be legally played. The other players (called "scientists") don't know this rule, they must discover it through experimentation.
 
-Players take turns playing cards from their hand onto a central "mainline." If a card satisfies the secret rule, it's accepted and added to the mainline. If it violates the rule, it's rejected and placed in a "sideline" below the mainline at that position. Over time, the pattern of accepted and rejected cards provides evidence about the hidden rule.
+Players take turns playing cards from their hand onto a central "mainline." If a card satisfies the secret rule, the dealer accepts it and it is added to the mainline. If it violates the rule, it's rejected and placed in a "sideline" below the mainline at that position. Over time, the pattern of accepted and rejected cards provides evidence about the hidden rule.
 
 <Sidenote>
   The name "Eleusis" comes from the ancient Greek mystery cult, where initiates gradually discovered hidden truths.
 </Sidenote>
 
-At any point, a player can attempt to guess the rule; correctly identifying it ends the game. A specific scoring system rewards efficiency in discovering the rule while penalizing reckless guessing.
+At any point, a player can attempt to guess the rule; correctly identifying it ends the game, but a wrong guess incurs a penalty. The game continues until someone correctly identifies the rule. A specific scoring system rewards efficiency in discovering the rule while penalizing reckless guessing.
 
 ### Our Adaptation
 
@@ -26,15 +24,16 @@ On each turn, the player selects a card from their hand to play. If the card sat
 
 When correctly guessing the rule, the player scores as many points as the number of turns spent, and each wrong guess deducts a penalty of 2 points:
 
-$$\text{score} = (30 - \text{turns\_elapsed} + 1) - 2 \times \text{num_wrong\_guesses}$$
+$$\text{score} = (30 - \text{turns\_elapsed} + 1) - 2 \times \text{num\_wrong\_guesses}$$
 
-A player who correctly identifies the rule on turn 13 with no wrong guesses scores 18 points; one who made 3 wrong guesses along the way scores only 12. If because of penalties the score drops to zero or below, the round stops and the final score is recorded as zero (similar to a scientist having wasted all their resources).
+For instance, a player who correctly identifies the rule on turn 13 with no wrong guesses scores 18 points; one who made 3 wrong guesses along the way scores only 12. If because of penalties the score drops to zero or below, the current round ends and the final score is recorded as zero (similar to a scientist having wasted all their resources).
 
 This creates an interesting tension: guessing early yields more points if correct, but wrong guesses are costly. The optimal strategy requires accurately assessing one's own confidence and acting accordingly.
 
 ### Rule Library
+In the original game, the dealer has to invent a secret rule on the spot. However, for benchmarking LLMs, we need a fixed set of rules to ensure comparability across model runs. We created a library of 26 hand-crafted rules spanning a range of types and complexity. Some rules involve simply card properties (e.g., "only red cards"), while others depend on the sequence of previously accepted cards (e.g., "card rank must be higher than previous card"). The rule might involve rank, suits, color or a combination thereof, and may include positional dependencies.
 
-We created a library of 26 hand-crafted rules spanning a range of types and complexity. Some rules involve simply card properties (e.g., "only red cards"), while others depend on the sequence of previously accepted cards (e.g., "card rank must be higher than previous card"). The rule might involve rank, suits, color or a combination thereof, and may include positional dependencies.
+Here are some example rules from our library, with a tentative categorization:
 
 | Category | Examples |
 |----------|----------|
@@ -48,7 +47,7 @@ Each rule is played 3 times with different random seeds (affecting the initial h
 
 ### What the LLM Must Do
 
-On each turn, the model receives the complete game state: the mainline of accepted cards, the sidelines of rejected cards at each position, its current hand, and its history of reasoning from the previous turns.
+On each turn, the model gets prompted with the rule of the game and the complete game state: the mainline of accepted cards, the sidelines of rejected cards at each position, its current hand, and its history of reasoning from the previous turns.
 
 The model is free to reason, but it is asked to output a structured response containing:
 
@@ -69,6 +68,6 @@ Example output
 }
 ```
 
-This structure lets us analyze not just whether models succeed, but *how* they reason: Do they update hypotheses appropriately when evidence contradicts them? Do they explore strategically or play conservatively? Is their stated confidence calibrated to their actual accuracy? In particular, forcing the model to articulate a tentative rule and a confidence level in it (even if they don't want to guess it yet) allows us to (secretely) evaluate it nonetheless, which will be useful for measuring calibration and guessing abilities.
+**This structure lets us analyze not just whether models succeed, but *how* they reason:** Do they update hypotheses appropriately when evidence contradicts them? Do they explore strategically or play conservatively? Is their stated confidence calibrated to their actual accuracy? In particular, forcing the model to articulate a tentative rule and a confidence level in it (even if they don't want to guess it yet) allows us to (secretely) evaluate it nonetheless, which will be useful for measuring calibration and guessing abilities.
 
 

app/src/content/chapters/eleusis/conclusion.mdx

@@ -3,23 +3,12 @@ import Sidenote from "../../../components/Sidenote.astro";
 
 ## Conclusion
 
-### Key Findings
-
-Our evaluation of ten LLMs on the Eleusis benchmark reveals several important insights:
-
-1. **LLMs can do inductive reasoning**—but with significant variation. Claude Opus 4.5 leads with 92% success rate and 15.9 average score, while Claude Haiku 4.5 achieves only 70% success and 9.1 average score—a substantial gap on the same benchmark.
-
-2. **Metacognition matters as much as reasoning**. Finding the correct rule is only half the challenge; knowing *when* you've found it is equally important. GPT 5.2 High has the highest success rate (96%) but only ranks third overall because it waits too long to commit—an average of 3.6 turns after finding the answer.
-
-3. **There's a caution-recklessness trade-off**. Models cluster into distinct behavioral styles: cautious achievers (GPT 5.2 High), balanced performers (Claude Opus 4.5), and reckless guessers (Claude Haiku 4.5). The best results come from accurate metacognition, not from either extreme.
+The Eleusis benchmark offers a window into capabilities that matter for real-world scientific reasoning: iterative hypothesis refinement, strategic experimentation, and calibrated confidence. Perhaps most importantly, it reveals the critical role of *metacognition*—the ability to accurately assess one's own knowledge state.
 
-4. **Open models are competitive**. Kimi K2 (open weights) achieves the second-highest score, outperforming several proprietary models. The gap between open and closed models appears to be closing on reasoning tasks.
+Our results suggest that raw reasoning ability is necessary but not sufficient. GPT 5.2 High can find the answer more often than any other model, yet loses to Claude Opus 4.5 because it doesn't know when to commit. Claude Haiku 4.5 commits readily but often before it should. The winning strategy requires both: strong inductive reasoning *and* accurate self-assessment.
 
-5. **Calibration remains imperfect**—models don't always know what they don't know. Most models show systematic overconfidence at high stated confidence levels.
+As LLMs are increasingly deployed to assist with scientific research, understanding these limitations becomes crucial. A model that is brilliant at generating hypotheses but doesn't know when to trust them could either lead researchers down unproductive paths (if overconfident) or waste time on unnecessary verification (if overcautious). The Eleusis benchmark provides one lens for evaluating and improving these capabilities—measuring not just what models know, but whether they know what they know.
 
-<Sidenote>
-  The 7-point gap between best and worst models (15.9 vs 9.1) suggests this benchmark captures meaningful capability differences.
-</Sidenote>
 
 ### Limitations
 
@@ -45,16 +34,7 @@ Several directions for future work:
 
 - **Human comparisons**: Collecting human performance data would provide crucial context for interpreting model capabilities.
 
-- **Interactive exploration**: Building tools to explore individual game traces could help researchers understand model reasoning more deeply.
+- **Prompt engineering**: Exploring how different prompt designs affect performance and metacognitive accuracy. In particular, can we compensate for bad calibration or guessing strategies via prompting?
 
-<Note variant="info">
-  The benchmark is open source. Try it yourself and contribute new rules or model evaluations!
-</Note>
-
-### Final Thoughts
-
-The Eleusis benchmark offers a window into capabilities that matter for real-world scientific reasoning: iterative hypothesis refinement, strategic experimentation, and calibrated confidence. Perhaps most importantly, it reveals the critical role of *metacognition*—the ability to accurately assess one's own knowledge state.
-
-Our results suggest that raw reasoning ability is necessary but not sufficient. GPT 5.2 High can find the answer more often than any other model, yet loses to Claude Opus 4.5 because it doesn't know when to commit. Claude Haiku 4.5 commits readily but often before it should. The winning strategy requires both: strong inductive reasoning *and* accurate self-assessment.
-
-As LLMs are increasingly deployed to assist with scientific research, understanding these limitations becomes crucial. A model that is brilliant at generating hypotheses but doesn't know when to trust them could either lead researchers down unproductive paths (if overconfident) or waste time on unnecessary verification (if overcautious). The Eleusis benchmark provides one lens for evaluating and improving these capabilities—measuring not just what models know, but whether they know what they know.
+- **Confirmation bias analysis**:
+Do models exhibit confirmation bias—preferring to play cards that confirm their current hypothesis rather than cards that could falsify it? It would require LLM as a judge analysis.

app/src/content/chapters/eleusis/discussion.mdx

@@ -0,0 +1,30 @@
+import Note from "../../../components/Note.astro";
+import Sidenote from "../../../components/Sidenote.astro";
+import Accordion from "../../../components/Accordion.astro";
+
+## Discussion
+
+### Inductive abilities & metacognition
+
+TODO: summarize main findings about the fact that performance depends on both inductive reasoning and metacognitive calibration.
+
+Primary factor : inductive reasoning and carefully choosing the next experiment.
+
+Then Different scientific personalities, which is on a different axis than raw reasoning ability, play a crucial role in performance.
+
+TODO: refine this
+- The Perfectionist (GPT 5.2 High): needs too much evidence
+- The Balanced (Gemini 3 Flash Preview Low): good tradeoff thanks to bad calibration compensated by caution, but not the best at inductive reasoning
+- The Pragmatist (Claude Opus 4.5): good-enough is good enough
+- The Gambler (Claude Haiku 4.5): acts on insufficient evidence
+
+
+### Open vs Closed Models
+
+A notable finding is the competitive performance of open-weight models. Kimi K2, available with open weights, achieves the second-highest score (16.2) and outperforms several proprietary models including GPT 5.2. DeepSeekR1 scores 13.3 and the open-weight GPT OSS 120B also performs respectably at 12.ç.
+
+However, open models all tend toward more aggressive guessing strategies mediated by bad calibration, leading to lower overall scores despite reasonable inductive abilities. This suggests that while open models can match proprietary ones in raw reasoning, they may lack the nuanced metacognitive skills needed for optimal performance in this benchmark.
+
+
+
+

app/src/content/chapters/eleusis/introduction.mdx

@@ -3,34 +3,34 @@ import Image from "../../../components/Image.astro";
 
 import exampleSequence from "../../assets/image/example_sequence.png";
 
-Large language models are increasingly being deployed as tools for scientific research—analyzing data, generating hypotheses, and even designing experiments. But how well do they actually embody the scientific method?
+Large language models are increasingly being deployed as tools for scientific research : analyzing data, generating hypotheses, and even designing experiments. But how well do they actually embody the scientific method?
 
 <Sidenote>
   Read time: 15–20 minutes.
 </Sidenote>
 
-Most reasoning benchmarks test whether models can solve well-defined problems: given premises, derive a conclusion. The ARC challenge, for instance, evaluates inductive reasoning on visual patterns. These benchmarks capture important capabilities, but they miss something fundamental about how science actually works.
+Most reasoning benchmarks test whether models can solve well-defined problems: given premises, derive a conclusion. The ARC challenge [@chollet2019measure], for instance, evaluates inductive reasoning on visual patterns. **These benchmarks capture important capabilities, but they miss something fundamental about how science actually works.**
 
-Real scientific reasoning is not a single inference step. It's an iterative agentic process of observation, hypothesis formation, experimentation, and refinement, often spanning many cycles before reaching a conclusion. It requires not just logical ability, but also *strategic* thinking: which experiment to run next, how much evidence is enough, when to commit to a theory versus when to keep exploring.
+First, real scientific reasoning is not a single inference step. It's an iterative agentic process of observation, hypothesis formation, experimentation, and refinement, often spanning many cycles before reaching a conclusion. It requires not just logical ability, but also *strategic thinking*: which experiment to run next, how much evidence is enough, when to commit to a theory versus when to keep exploring.
 
-Beyond pure reasoning, effective science depends on psychological factors that are rarely evaluated: **calibration** (does my confidence match my actual accuracy?), **metacognition** (how certain am I about my uncertainty?), and resistance to **cognitive biases** like confirmation bias (seeking only evidence that supports my current hypothesis instead of trying to challenge it). A scientist who is brilliant at deduction but overconfident in weak theories will waste resources pursuing dead ends. One who is well-calibrated but overly cautious may never publish.
+Also, beyond pure reasoning, effective science depends on psychological factors that are rarely evaluated: **calibration** (does my confidence match my actual accuracy?) [@lichtenstein1977calibration], **metacognition** (how certain am I about my uncertainty?) [@flavell1979metacognition], and resistance to **cognitive biases** like confirmation bias (seeking only evidence that supports my current hypothesis instead of trying to challenge it) [@nickerson1998confirmation]. A scientist who is brilliant at deduction but overconfident in weak theories will waste resources pursuing dead ends. One who is well-calibrated but overly cautious may never publish.
 
 We wanted to test whether LLMs can exhibit these deeper aspects of scientific reasoning. To do this, we turned to an unlikely source: a 1950s card game called Eleusis.
 
 ## The Eleusis Game
 
-Eleusis was designed by Robert Abbott explicitly to simulate the process of scientific discovery. In the original game, one player invents a secret rule governing which cards can be played, and other players must deduce the rule through experimentation, by playing cards and observing whether they are accepted or rejected.
+Eleusis was designed by @abbott1977eleusis explicitly to simulate the process of scientific discovery. In the original game, one player invents a secret rule governing which cards can be played, and other players must deduce the rule through experimentation, by playing cards and observing whether they are accepted or rejected.
 
 **Eleusis is a microcosm of the scientific method:** the rule is a hidden law of nature, each card play is an experiment, and the sequence of accepted and rejected cards is the accumulating evidence.
 
 <Image
   src={exampleSequence}
   alt="Example Eleusis game sequence with the secret rule 'alternating colors': mainline shows 5♠, J♥, J♠, A♦, 6♣ following the pattern, while the sideline below shows rejected cards 10♣ after J♠, and Q♥ and 2♦ after A♦"
-  caption="An example Eleusis game with the secret rule 'alternating colors'. The main line (top) shows the sequence accepted cards: 5♠ → J♥ → J♠ → A♦ → 6♣, alternating between black and red. The sideline (bottom) shows cards that were tried but rejected because they are violating the pattern, for instance 10♣ after J♠, or Q♥ and 2♦ after A♦."
+  caption="An example Eleusis game. The secret rule here is 'colors must alternate'. The main line (top) shows the sequence of accepted cards: 5♠ → J♥ → J♠ → A♦ → 6♣, alternating between black and red. The sideline (bottom) shows cards that were tried but rejected because they are violating the rule, for instance 10♣ after J♠, or Q♥ and 2♦ after A♦."
   id="fig-example-sequence"
   preserveColors
 />
 
 We built a benchmark around Eleusis to evaluate LLMs on this iterative, hypothesis-driven reasoning. Rather than testing knowledge retrieval or instruction-following, our benchmark asks: *can models act like scientists?* Can they observe evidence, form hypotheses, design informative experiments, and refine their theories? Can they calibrate their confidence appropriately and know when they've gathered enough evidence to commit to a conclusion?
 
-These skills are fundamental not just to science, but to debugging code, diagnosing problems, and everyday reasoning under uncertainty.
+These skills are fundamental not just to science, but to debugging code, medical diagnosis, and everyday reasoning under uncertainty.

app/src/content/chapters/eleusis/results.mdx

@@ -4,7 +4,7 @@ import Note from "../../../components/Note.astro";
 import Sidenote from "../../../components/Sidenote.astro";
 import HtmlEmbed from "../../../components/HtmlEmbed.astro";
 
-## Results
+## 2. Results
 
 ### Overall Performance
 
@@ -17,44 +17,50 @@ We evaluated ten models on the Eleusis benchmark, including both proprietary and
   wide
 />
 
-Performance varies dramatically among tested models. Claude Opus 4.5 achieves top performance with moderate token usage. The open-weight model Kimi K2 comes second and performs competitively with the best proprietary models, outperforming GPT 5.2 High and being closed to Claude Opus 4.5, but at the price of a 2.5× larger reasoning budget.
+Performance varies dramatically among tested models.
 
-GPT 5.2 High and Grok 4.1 Fast Reasoning show a similar performance but GPT 5.2 High is significantly more token efficient.
+* **Claude Opus 4.5** achieves top performance with 17.0 score and moderate token usage. The open-weight model **Kimi K2 Thinking** comes second at 16.2 and performs competitively with the best proprietary models (outperforming GPT 5.2 High and being close to Claude Opus 4.5), but at the price of a 2.5× larger reasoning budget.
 
-GPT-5-Mini, GPT OSS-120B and Gemini 3 Flash Preview Low cluster in the mid-tier (around 13) with moderate token usage.While Deepseek R1, an open-weight model specialized for reasoning tasks, achieves a similar score with much larger token count.
+* **GPT 5.2 High** and **Grok 4.1 Fast Reasoning** show a similar performance around 15, but GPT 5.2 High is 3 times more token efficient.
 
-Finally, GPT-OSS 20B and Claude Haiku 4.5 lag behind, scoring between 11 and 12 with moderate token usage.
+* **GPT-5-Mini**, **GPT OSS-120B** and **Gemini 3 Flash Preview Low** cluster in the mid-tier (around 13) with low token usage. While Deepseek R1, an open-weight model specialized for reasoning tasks, achieves a similar score but with a much larger token count.
 
-As we mentionned, this score reflects not only the pure model's ability to find the correct rule, but also its metacognitive skills: knowing when to commit, how confident it is, and how to balance exploration vs. exploitation. To distinguish these factors, we also computed an alternative "no-stakes" score that removes penalties for wrong guesses and counts tentative rules as guesses. This allows us to isolate pure rule-discovery ability from metacognitive skills.
+* Finally, **GPT-OSS 20B** and **Claude Haiku 4.5** lag behind, scoring between 11 and 12 with moderate token usage.
+
+As we mentionned, this score reflects not only the pure model's ability to find the correct rule, but also its metacognitive skills: knowing when to commit, how confident it is, and how to balance exploration vs. exploitation. To distinguish these factors, we also computed an alternative "no-stakes" score that removes penalties for wrong guesses and counts tentative rules as guesses.
 
 ### Pure discovery versus metacognition
 
-The following chart shows the score of each model, and which score it would have achieved under a "no stakes" scenario where guessing is free and systematic.
+We use the same game data but we applied a different scoring system to reflect the pure ability to discover the rule, without the metacognitive aspect of knowing when to commit. **In this "no stakes" scenario, guessing is free and systematic**: at each turn, if the model has the correct tentative rule, it is considered to have guessed it correctly (even if it didn't formally attempt to guess); if the tentative rule is incorrect, it is considered a wrong guess, but without penalty.
+
+The following chart shows the initial score of each model, and which (higher) score it would have achieved under the "no stakes" scenario. This allows us to isolate pure rule-discovery ability from metacognitive skills.
 
 <HtmlEmbed
     src="score-stack.html"
-    caption="<strong>Figure 2:</strong> Score breakdown under alternative scoring systems. Blue shows raw score (standard scoring). while green shows no-stakes gain (additional gain from removing wrong-guess penalties). Models sorted by total no-stakes score."
+    caption="<strong>Figure 2:</strong> Score breakdown under alternative scoring systems. Blue shows raw score (standard scoring), while green shows no-stakes gain (additional gain from systematic guessing and removing wrong-guess penalties). Models sorted by total no-stakes score."
     id="fig-score-stack"
     wide
 />
 
-Even if using this alternative scoring does not change a lot the relative ranking of models, it reveals important differences in their behavior. GPT 5.2 High and Claude Haiku 4.5 are the two models with the largest difference between raw and no-stakes scores (more than 4) while Gemini and Kimi K2 have the smallest difference (less than 3).
+Even if using this alternative scoring does not change a lot the relative ranking of models, it reveals important differences in their behavior.
+
+* GPT 5.2 High and Claude Haiku 4.5 are the two models with the largest difference between raw and no-stakes scores (more than 4), suggesting they are the most penalized by wrong guesses or delayed guessing.
+* On the other hand, Gemini 3 Flash Preview Low and Kimi K2 have the smallest difference (less than 3) and benefit the least from this alternative scoring, indicating a better balance between discovery and metacognition.
 
-They might be two reason for the difference between the raw and the no-stakes scores:
+They might be two reasons for the difference between the raw and the no-stakes scores:
 1. The model is reckless and makes a lot of wrong guesses, incurring penalties.
 2. The model is too cautious and waits too long before guessing, missing out on points.
 
 We analyze these two aspects in more details below.
 
-
 ### The Caution-Recklessness Trade-off
 
-To estimate how reckless or cautious a model is, we can compute the average number of failed guesses per round (recklessness). It directly relates to how many points a model loses due to wrong guesses.
+To estimate how reckless or cautious a model is, we can compute **the average number of failed guesses per round** (recklessness). It directly relates to how many points a model loses due to wrong guesses.
 
-To estimate caution, we can compute on average how many turns a model waits while having the correct tentative rule before actually guessing it. This relates to how many points a model loses by waiting too long to commit.
+To estimate caution, we can compute on average **how many turns a model waits while having the correct tentative rule before actually guessing it**. This relates to how many points a model loses by waiting too long to commit.
 
 <Sidenote>
-    This trade-off mirrors a fundamental tension in science: being overconfident too early might risk false positives, leading to wasted resources and reputational damage; being overly cautious can delay discoveries and allow others to scoop you. Scientists must balance the risk of trying to publish too early and risk being wrong, wait too long and lose priority (or in our case, points).
+    This trade-off mirrors a fundamental tension in science: being overconfident too early might risk false positives, leading to wasted resources and reputational damage; being overly cautious can delay discoveries, waste resources and allow others to scoop you. Scientists must balance the risk of trying to publish too early and risk being wrong, wait too long and lose priority (or in our case, points).
 </Sidenote>
 
 <HtmlEmbed
@@ -63,32 +69,33 @@ To estimate caution, we can compute on average how many turns a model waits whil
     id="fig-caution-reckless"
 />
 
+How should we interpret those values ? Knowing that a failed guess costs 2 points, while each turn of delay costs 1 point, the optimal number of failed guesses per round should be around 0.5 (i.e., 1 failed guess every 2 rounds) to balance the two sources of loss. We can see that most models are above that threshold, indicating **a clear tendency towards recklessness**. This is confirmed by the fact that they have a low caution value (most models wait around 1 turn or less on average before guessing when they have the correct rule).
 
-How should we interpret those values ? Knowing that a failed guess costs 2 points, while each turn of delay costs 1 point, the optimal number of failed guesses per round should be around 0.5 (i.e., 1 failed guess every 2 rounds) to balance the two sources of loss. We can see that most models are above that threshold, indicating a tendency towards recklessness. This is confirmed by the fact that they have a low caution value (most models wait around 1 turns on average before guessing when they have the correct rule).
-
-On the other hand, GPT 5.2 High has a singular behavior with very few failed guesses (0.28 per round) but a high caution (waiting 3.5 turns on average before guessing when it has the correct rule). Gemini 3 Flash Preview Low and GPT 5 Mini Medium are intermediate in both dimensions, Gemini achieving a better balance with on average 2 points lost due to recklessness and 2 points lost due to caution.
+On the other hand, **GPT 5.2 High has a singular behavior** with very few failed guesses (0.28 per round) but a high caution (waiting 3.5 turns on average before guessing when it has the correct rule). Gemini 3 Flash Preview Low and GPT 5 Mini Medium are intermediate in both dimensions, Gemini achieving a better balance with on average 2 points lost due to caution and 2 points lost due to recklessness (1 failed guess every round on average).
 
 To try to understand deeper the causes of recklessness and caution, we now turn to an analysis of confidence and guessing strategies.
 
 ### Confidence and Calibration
 
-Models are asked to output their confidence level, with clear instructions on what it means (7 = 70% probability of being correct, etc.). Even when they don't guess, they report their tentative rule. When confidence ≥5, we test whether they would have guessed correctly, even if they didn't formally attempted to guess. This allows us to evaluate calibration: does reported confidence match actual accuracy?
+Models are asked to output their confidence level, with clear instructions on what it means (7 = 70% probability of being correct, etc.). Even when they don't guess, they report their tentative rule. When confidence ≥5, we test whether they would have guessed correctly, even if they didn't formally attempt to do so. **This allows us to evaluate calibration: does reported confidence match actual accuracy?** This is particularly relevant as modern neural networks have been shown to be poorly calibrated [@guo2017calibration].
 
 <HtmlEmbed
   src="calibration-curves.html"
-  caption="<strong>Figure 4:</strong> Calibration curves for each model. A perfectly calibrated model would follow the diagonal. Points below the line indicate overconfidence: they correspond to confidence levels where actual success rates are lower than reported. Click legend items to show/hide models."
+  caption="<strong>Figure 4:</strong> Calibration curves for each model (for reported confidence ≥5). A perfectly calibrated model would follow the diagonal. Points below the line indicate overconfidence: they correspond to confidence levels where actual success rates are lower than reported. Click legend items to show/hide models."
   id="fig-calibration"
 />
 
 The calibration analysis reveals several patterns:
 
-- **All models are overconfident** : for instance when they report 80% confidence, their actual success rates are often closer to 20% !
-- GPT 5.2 is the best calibrated model overall.
+- **All models are very overconfident** : for instance when they report 80% confidence, their actual success rates are often closer to 20% !
+- GPT 5.2 is the best calibrated model overall, being the closest to the diagonal line, although it is still slightly overconfident.
 - Even models with a strong performance like Claude Opus 4.5 and Kimi K2 show significant overconfidence.
 
-Is overconfidence a problem ? It depends on how the model decides to act on it.
+Is overconfidence a problem ? In our setting, not necessarily; it depends on how the model decides to act on it.
 
-For a perfectly calibrated model, as the expected loss for a failed guess is twice the expected opportunity cost of waiting one turn, the optimal confidence threshold for guessing is 0.67 (i.e., guess when you believe your tentative rule has at least a 67% chance of being correct). But do model follow such a strategy ? For this, we can look at how often models guess at each confidence level.
+**For a perfectly calibrated model**, as the expected loss for a failed guess is twice the expected opportunity cost of waiting one turn, **the optimal confidence threshold for guessing is 0.67** (i.e., guess when you believe your tentative rule has at least a 67% chance of being correct). But do model follow such a strategy ?
+
+For this, we can look at how often models guess at each reported confidence level. This is shown in the following figure. For each confidence level (from 5 to 10), we compute the guess rate: the fraction of turns the model actually attempts to guess when reporting that confidence.
 
 
 <HtmlEmbed
@@ -97,22 +104,20 @@ For a perfectly calibrated model, as the expected loss for a failed guess is twi
   id="fig-confidence"
 />
 
-We can see that some models like Grok 4.1 or Gemini 3 will essentially only guess when very confident (9 or 10). Most other models will guess at confidence levels above 8 and rarely below. The two Claude models show different behaviors: Claude Opus 4.5 tends to guess more agressively at confidence level 8, while Claude Haiku 4.5 guesses even at confidence level 7.
-
-We can see that models on average are more cautious than the optimal decision-theoretic strategy for a perfectly calibrated model, which would guess as soon as confidence exceeds 67%. THis is somehow a good thing, given that all models are overconfident. By raising the bar for guessing, they reduce the risk of wrong guesses and compensate for their poor calibration.
+Once again, we observe significant differences from one model to another. Grok 4.1 and Gemini 3 will essentially only guess when very confident (9 or 10). Most other models will also often guess at confidence levels above 8 and rarely below. The two Claude models show different behaviors: Claude Opus 4.5 tends to guess more aggressively at confidence level 8, while Claude Haiku 4.5 often guesses even at confidence level 7.
 
-This is particularly true for Gemini 3 Flash Preview Low which is very cautious despite being overconfident, and this is probably what helps it achieve a good balance between failed guesses and lost opportunity cost. It is also consistent with the fact that it's the model with the smallest difference between raw and no-stakes scores.
+We can see that **models on average are more cautious than the optimal decision-theoretic strategy** for a perfectly calibrated model, which would guess as soon as confidence exceeds 67%. This is somehow a good thing for them, given that all models are overconfident. **By raising the threshold for guessing, they reduce the risk of wrong guesses and compensate for their poor calibration.**
 
-The case of GPT 5.2 High is different: it is both fairly well calibrated and very cautious, leading to very few failed guesses but a high opportunity cost due to delayed guessing. This suggests that GPT 5.2 High could improve its performance by being more agressive in guessing once it has a correct tentative rule.
+This is particularly true for Gemini 3 Flash Preview Low which is very cautious, guessing only 1/3 of the time at reported confidence 9 ! This compensates its overconfidence, which is probably what helps it achieve a good balance between failed guesses and lost opportunity cost. This is reflected in our "no-stakes" analysis by the fact that it's the model with the smallest difference between raw and no-stakes scores.
 
+The case of GPT 5.2 High is different: it is both fairly well calibrated and very cautious, leading to very few failed guesses but a high opportunity cost due to delayed guessing. This suggests that GPT 5.2 High could improve its performance by being more aggressive in guessing once it has a correct tentative rule, especially at confidence level 8.
 
-### Performance by Rule
 
-Not all rules are created equal. Some rules are discovered quickly by all models (e.g. "All cards must be red") while others prove consistently challenging (e.g. "increase rank after a red card, decrease after a black").
+### Performance by Rule Complexity
 
-It is not easy to quantify rule complexity, as it depends on multiple factors: the inherent logical complexity of the rule, how familiar the concept is to models, and how much evidence is needed to distinguish it from alternatives. We create a crude complexity score for each rule based on the complexity code implementation, as measured by cyclomatic complexity and Abstract Syntax Tree node count.
+Not all rules are created equal. Some rules are discovered quickly by all models (e.g. *"all cards must be red"*) while others prove consistently challenging (e.g. *"increase rank after a red card, decrease after a black"*).
 
-The following figure breaks down performance by rule across all models and runs.
+The following figure breaks down performance by rule across all models and runs, displaying the average success rate per rule on the left (how often the rule was found), and individual run scores as colored dots for each model on the right.
 
 <HtmlEmbed
   src="by-rule.html"
@@ -121,9 +126,15 @@ The following figure breaks down performance by rule across all models and runs.
   wide
 />
 
-We can see that the most complex rules are devastating for the reckless models like Claude Haiku 4.5 and DeepSeek R1, which often negative scores on these rules due to multiple wrong guesses. Even the best models struggle on the hardest rules, but their superior metacognition allows them to avoid catastrophic failures.
+It confirms that some rules are consistently easy, with low variance in score across models, while others are hard for all models. To analyse this, we need a way to quantify rule complexity. This is not straightforward since it depends on multiple factors: the inherent logical complexity of the rule, how familiar the concept is to models, and how much evidence is needed to distinguish it from alternatives.
+
+We created a crude complexity score for each rule based on the complexity of its code implementation, as measured by *cyclomatic complexity* and *Abstract Syntax Tree node count*. Combining these two metrics into a unique indicator
+
+$$\text{cyclomatic\_complexity} + 0.14 * \text{node\_count}$$
 
-The following plot breaks down the relative score of each model (as measured by score on the rule divided by average score on all rules) against the complexity metrics of each rule.
+The coefficient 0.14 was chosen to maximize correlation with average success rate across models. The achieved correlation being -0.66. This indicates that as expected more complex rules tend to have lower success rates, and validates our complexity metric as a useful proxy for rule difficulty, despite its limitations.
+
+The following plot breaks down the success rate of each model per complexity quartile.
 
 <HtmlEmbed
   src="complexity-analysis.html"
@@ -131,6 +142,21 @@ The following plot breaks down the relative score of each model (as measured by
   id="fig-complexity"
 />
 
-<Note variant="info">
-  Interestingly, code complexity (cyclomatic complexity, AST node count) doesn't perfectly predict difficulty. Semantically simple rules like "only face cards" can be harder than structurally complex rules if the semantic concept is unfamiliar to models.
-</Note>
+
+Interestingly, code complexity (as measured by our combination of cyclomatic complexity and AST node count) doesn't perfectly predict difficulty, as semantic concepts also play a role. For instance a rule like "only face cards" has a complexity equivalent to "only A, 2 and 3", but the former is easier for models (and humans !) due to familiarity with the semantic category of face cards.
+
+Also rules involving rare events (low acceptance rate). Only aces is harder than "only even ranks" despite being simpler, simply because models need more evidence to confirm it.
+
+An interesting test: are symmetric rules equally difficult? For example, "only spades" vs "only non-spades" should be logically equivalent in difficulty, but models might have biases.
+For instance average score on "only spades" is 25, but "no spades" is 20.
+
+### Complexity of rules produced
+
+#### Overly Complex Rules
+Failure mode: models have a tendency to produce over complicated rules, even if they were informed that the rule is typically one sentence. They can produce tentative rules like "...".
+
+TODO : Backup this with examples from logs and "guess complexity" vs "actual complexity".
+
+
+#### Overfitting Rules
+We have observed qualitative evidence of model producing overfit rules that explain all observations so far, but fail to generalize. For instance if all accepted cards so far are red, and happens to be only number cards (simply because no red face card has been tried yet), the model may hypothesize "only red number cards" rather than the simpler "only red cards."