| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>LotteryElicitationEnv: An OpenEnv Benchmark for Post-training Strategic Question Design via Lottery Elicitation as Proxy</title> |
| <meta name="description" content="LotteryElicitationEnv: an OpenEnv-native benchmark for post-training strategic question design via Lottery Elicitation as Proxy. With GRPO training pipeline."> |
| <link rel="preconnect" href="https://fonts.googleapis.com"> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> |
| <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600;700;800&family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet"> |
| |
| <script type="module"> |
| import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs'; |
| mermaid.initialize({ |
| startOnLoad: true, |
| theme: 'dark', |
| themeVariables: { |
| primaryColor: '#6366f1', |
| primaryTextColor: '#e2e8f0', |
| primaryBorderColor: '#818cf8', |
| lineColor: '#818cf8', |
| secondaryColor: '#1e293b', |
| tertiaryColor: '#172033', |
| background: '#0f172a', |
| mainBkg: '#1e293b', |
| nodeBorder: '#818cf8', |
| clusterBkg: '#172033', |
| clusterBorder: '#334155', |
| titleColor: '#e2e8f0', |
| edgeLabelBackground: '#1e293b', |
| nodeTextColor: '#e2e8f0' |
| }, |
| flowchart: { curve: 'basis', htmlLabels: true }, |
| fontFamily: 'Inter, sans-serif' |
| }); |
| </script> |
| <style> |
| :root { |
| --bg: #0f172a; --surface: #1e293b; --surface-2: #172033; --border: #334155; |
| --text: #e2e8f0; --muted: #94a3b8; --accent: #6366f1; |
| --accent2: #818cf8; --green: #22c55e; --red: #ef4444; |
| --orange: #f59e0b; --radius: 12px; |
| } |
| * { margin: 0; padding: 0; box-sizing: border-box; } |
| html { scroll-behavior: smooth; } |
| body { font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; |
| background: var(--bg); color: var(--text); line-height: 1.7; |
| -webkit-font-smoothing: antialiased; } |
| .container { max-width: 820px; margin: 0 auto; padding: 2rem 1.5rem 4rem; } |
| |
| .topnav { position: sticky; top: 0; z-index: 10; background: rgba(15,23,42,.85); |
| backdrop-filter: blur(10px); border-bottom: 1px solid var(--border); |
| padding: .9rem 1.5rem; display: flex; justify-content: space-between; |
| align-items: center; font-size: .88rem; } |
| .topnav .brand { font-weight: 700; color: var(--text); text-decoration: none; |
| display: flex; align-items: center; gap: .5rem; } |
| .topnav .brand .dot { width: 8px; height: 8px; border-radius: 50%; |
| background: var(--green); box-shadow: 0 0 8px rgba(34,197,94,.6); } |
| .topnav .links { display: flex; gap: 1.25rem; } |
| .topnav .links a { color: var(--muted); text-decoration: none; transition: color .15s; } |
| .topnav .links a:hover { color: var(--accent2); } |
| |
| .hero { text-align: center; padding: 4rem 0 2.5rem; } |
| .hero-badge { display: inline-block; background: rgba(99,102,241,.15); color: var(--accent2); |
| padding: .4rem 1.1rem; border-radius: 20px; font-size: .78rem; font-weight: 600; |
| letter-spacing: .08em; margin-bottom: 1.25rem; |
| border: 1px solid rgba(99,102,241,.3); text-transform: uppercase; } |
| .hero h1 { font-size: clamp(2rem, 4.2vw, 3.2rem); font-weight: 800; letter-spacing: -.025em; |
| line-height: 1.15; |
| background: linear-gradient(135deg, #e2e8f0 25%, #6366f1 100%); |
| -webkit-background-clip: text; -webkit-text-fill-color: transparent; |
| background-clip: text; } |
| .hero .subtitle { color: var(--muted); font-size: 1.15rem; max-width: 640px; |
| margin: 1rem auto 0; } |
| .hero .byline { color: var(--muted); font-size: .85rem; margin-top: 1.5rem; |
| font-style: italic; } |
| .banner { width: 100%; border-radius: var(--radius); margin: 2rem 0 3rem; |
| border: 1px solid var(--border); } |
| |
| .badges { display: flex; justify-content: center; gap: .6rem; flex-wrap: wrap; |
| margin: 1.5rem 0; } |
| .badges img { height: 22px; } |
| |
| .btn-group { display: flex; gap: .75rem; justify-content: center; margin: 2rem 0; |
| flex-wrap: wrap; } |
| .btn { display: inline-flex; align-items: center; gap: .45rem; padding: .6rem 1.35rem; |
| background: var(--accent); color: white; border-radius: 8px; font-size: .88rem; |
| font-weight: 600; text-decoration: none; transition: all .2s; } |
| .btn:hover { background: var(--accent2); transform: translateY(-1px); } |
| .btn-outline { background: transparent; border: 1px solid var(--border); color: var(--text); } |
| .btn-outline:hover { border-color: var(--accent); color: var(--accent2); |
| background: rgba(99,102,241,.08); } |
| |
| .toc { background: var(--surface); border: 1px solid var(--border); border-radius: var(--radius); |
| padding: 1.25rem 1.5rem; margin: 0 0 2.5rem; } |
| .toc h3 { font-size: .82rem; font-weight: 700; letter-spacing: .08em; text-transform: uppercase; |
| color: var(--accent2); margin-bottom: .85rem; } |
| .toc ol { list-style: none; counter-reset: toc; display: flex; flex-wrap: wrap; gap: .35rem .8rem; |
| margin: 0; padding: 0; } |
| .toc ol li { counter-increment: toc; font-size: .88rem; } |
| .toc ol li::before { content: counter(toc) "."; color: var(--accent); font-weight: 700; |
| font-size: .8rem; margin-right: .3rem; } |
| .toc ol li a { color: var(--muted); text-decoration: none; transition: color .15s; } |
| .toc ol li a:hover { color: var(--accent2); } |
| |
| section { margin: 3.5rem 0; } |
| section h2 { font-size: 1.55rem; font-weight: 800; letter-spacing: -.01em; |
| margin-bottom: 1rem; color: var(--text); |
| border-left: 3px solid var(--accent); padding-left: .9rem; } |
| section h3 { font-size: 1.1rem; font-weight: 700; margin: 2rem 0 .75rem; |
| color: var(--accent2); } |
| section p { color: #cbd5e1; margin-bottom: 1rem; font-size: 1.02rem; } |
| section p strong { color: var(--text); } |
| section ul, section ol { color: #cbd5e1; margin: 1rem 0 1rem 1.5rem; } |
| section ul li, section ol li { margin-bottom: .5rem; font-size: 1rem; } |
| section ul li strong, section ol li strong { color: var(--text); } |
| |
| blockquote { border-left: 3px solid var(--accent2); |
| background: rgba(99,102,241,.06); padding: 1.1rem 1.25rem; |
| margin: 1.5rem 0; border-radius: 0 8px 8px 0; |
| color: #e2e8f0; font-size: 1.02rem; } |
| |
| .table-wrap { margin: 1.5rem 0; overflow-x: auto; |
| background: var(--surface); border: 1px solid var(--border); |
| border-radius: var(--radius); } |
| table { width: 100%; border-collapse: collapse; font-size: .92rem; } |
| th { background: rgba(99,102,241,.1); color: var(--accent2); |
| font-size: .72rem; font-weight: 700; letter-spacing: .06em; |
| text-transform: uppercase; padding: .85rem 1rem; text-align: left; } |
| td { padding: .7rem 1rem; border-top: 1px solid var(--border); color: #cbd5e1; } |
| td.num { text-align: right; font-variant-numeric: tabular-nums; |
| font-family: 'JetBrains Mono', monospace; font-size: .88rem; } |
| tr:hover td { background: rgba(99,102,241,.04); } |
| td strong, th strong { color: var(--text); } |
| .task-id { font-family: 'JetBrains Mono', monospace; font-weight: 700; |
| color: var(--accent2); font-size: .85rem; } |
| tr.avg-row td { background: rgba(99,102,241,.08); font-weight: 700; |
| color: var(--text); } |
| tr.novel td:first-child { color: #fca5a5; } |
| |
| pre { background: #0b1120; border: 1px solid var(--border); |
| border-radius: var(--radius); padding: 1.1rem 1.25rem; overflow-x: auto; |
| margin: 1.25rem 0; font-family: 'JetBrains Mono', monospace; |
| font-size: .85rem; line-height: 1.6; color: #d1d5db; } |
| pre .c { color: #64748b; } |
| code { font-family: 'JetBrains Mono', monospace; font-size: .88em; |
| background: rgba(99,102,241,.12); color: var(--accent2); |
| padding: .1em .35em; border-radius: 4px; } |
| pre code { background: none; color: inherit; padding: 0; font-size: 1em; } |
| |
| figure { margin: 2rem 0; } |
| figure img { width: 100%; border-radius: var(--radius); |
| border: 1px solid var(--border); } |
| figcaption { text-align: center; color: var(--muted); font-size: .85rem; |
| margin-top: .75rem; } |
| |
| .mermaid-wrap { margin: 2rem 0; background: var(--surface); border: 1px solid var(--border); |
| border-radius: var(--radius); padding: 1.5rem 1rem; overflow-x: auto; } |
| .mermaid-wrap .mermaid { display: flex; justify-content: center; } |
| .mermaid-caption { text-align: center; color: var(--muted); font-size: .85rem; |
| margin-top: .75rem; } |
| |
| .episode-trace { background: var(--surface); border: 1px solid var(--border); |
| border-radius: var(--radius); padding: 1.25rem 1.5rem; margin: 1.5rem 0; |
| position: relative; } |
| .episode-trace::before { content: ''; position: absolute; left: 1.5rem; top: 2.5rem; |
| bottom: 1.25rem; width: 2px; background: var(--border); } |
| .trace-step { position: relative; padding-left: 2rem; margin-bottom: 1.25rem; } |
| .trace-step:last-child { margin-bottom: 0; } |
| .trace-step .step-marker { position: absolute; left: -.45rem; top: .2rem; width: 12px; |
| height: 12px; border-radius: 50%; border: 2px solid var(--accent); |
| background: var(--bg); z-index: 1; } |
| .trace-step .step-marker.terminal { background: var(--red); border-color: var(--red); } |
| .trace-step .step-marker.good { background: var(--green); border-color: var(--green); } |
| .trace-step .step-label { font-family: 'JetBrains Mono', monospace; font-size: .78rem; |
| color: var(--accent2); font-weight: 700; margin-bottom: .25rem; } |
| .trace-step .step-content { font-size: .9rem; color: #cbd5e1; } |
| .trace-step .step-content code { font-size: .82em; } |
| .trace-verdict { margin-top: 1rem; padding: .75rem 1rem; border-radius: 8px; |
| font-size: .9rem; font-weight: 600; } |
| .trace-verdict.bad { background: rgba(239,68,68,.1); border: 1px solid rgba(239,68,68,.3); |
| color: #fca5a5; } |
| .trace-verdict.good { background: rgba(34,197,94,.1); border: 1px solid rgba(34,197,94,.3); |
| color: #86efac; } |
| |
| .callout { text-align: center; padding: 2rem 1.5rem; margin: 3rem 0; |
| background: linear-gradient(135deg, rgba(99,102,241,.08), rgba(129,140,248,.04)); |
| border: 1px solid rgba(99,102,241,.25); border-radius: var(--radius); } |
| .callout .q { font-size: 1.25rem; font-weight: 700; color: var(--text); |
| font-style: italic; margin-bottom: .5rem; } |
| .callout .sub { color: var(--muted); font-size: .95rem; } |
| |
| .footer { text-align: center; padding: 3rem 0 1rem; color: var(--muted); |
| font-size: .85rem; border-top: 1px solid var(--border); margin-top: 3rem; } |
| .footer a { color: var(--accent2); text-decoration: none; margin: 0 .5rem; } |
| .footer a:hover { text-decoration: underline; } |
| @media (max-width: 640px) { |
| .container { padding: 1rem 1rem 3rem; } |
| .hero { padding: 2.5rem 0 1.5rem; } |
| .topnav .links { display: none; } |
| section h2 { font-size: 1.3rem; } |
| table { font-size: .82rem; } |
| th, td { padding: .55rem .6rem; } |
| .toc ol { flex-direction: column; } |
| .episode-trace { padding: 1rem; } |
| .episode-trace::before { left: 1rem; } |
| } |
| |
| .math-display { |
| margin: 1.25rem 0; |
| padding: 1rem 1.25rem 1.15rem; |
| overflow-x: auto; |
| background: var(--surface); |
| border: 1px solid var(--border); |
| border-radius: var(--radius); |
| text-align: center; |
| } |
| .math-display mjx-container[jax="CHTML"][display="true"] { margin: 0.65em 0 !important; } |
| .math-display mjx-container { color: #e2e8f0 !important; } |
| .math-note { font-size: .9rem; color: var(--muted); margin-top: .35rem; margin-bottom: 0; } |
| </style> |
| |
| <script> |
| window.MathJax = { |
| tex: { |
| inlineMath: [['\\(', '\\)']], |
| displayMath: [['\\[', '\\]']] |
| }, |
| options: { |
| renderActions: { |
| addMenu: [0, '', ''] |
| } |
| } |
| }; |
| </script> |
| <script defer src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" id="MathJax-script"></script> |
| </head> |
| <body> |
|
|
| <nav class="topnav"> |
| <a href="#top" class="brand"><span class="dot"></span> LotteryElicitationEnv Blog</a> |
| <div class="links"> |
| <a href="#why">Why</a> |
| <a href="#design">Design</a> |
| <a href="#scoring">Scoring</a> |
| <a href="#traces">Traces</a> |
| <a href="#results">Results</a> |
| <a href="#engineering">Engineering</a> |
| <a href="https://huggingface.co/spaces/yashu2000/LotteryElicitationEnv" target="_blank">Live Space ↗</a> |
| </div> |
| </nav> |
|
|
| <div class="container" id="top"> |
|
|
| <div class="hero"> |
| <div class="hero-badge">OpenEnv · AgentBeats Phase 2</div> |
| <h1>LotteryElicitationEnv</h1> |
| <p class="subtitle">An OpenEnv Benchmark Where LLMs Learn to Design Strategic Questions for Preference Elicitation.</p> |
| <div class="badges"> |
| <a href="https://github.com/sharma-yash01/LotteryElicitationEnv" target="_blank"><img src="https://img.shields.io/badge/GitHub-Repository-181717?logo=github" alt="GitHub"/></a> |
| <a href="https://huggingface.co/spaces/yashu2000/LotteryElicitationEnv" target="_blank"><img src="https://img.shields.io/badge/HF%20Space-Live%20Demo-FFD21E?logo=huggingface&logoColor=black" alt="HF Space"/></a> |
| <img src="https://img.shields.io/badge/OpenEnv-Native-4B8BBE" alt="OpenEnv"/> |
| <img src="https://img.shields.io/badge/Tasks-10-brightgreen" alt="Variable | Up to 10 Tasks"/> |
| <img src="https://img.shields.io/badge/Training-GRPO-orange" alt="GRPO"/> |
| </div> |
| <div class="byline">AgentBeats Phase 2 · OpenEnv Challenge Submission | Yashaswi Sharma (University of Southern California) | Defu Cao (USC) | Muyan Weng (USC)</div> |
| </div> |
|
|
| <img src="banner.png" alt="LotteryElicitationEnv: adaptive lottery elicitation cycle — propose pairs, observe choices, recover hidden parameters" class="banner"/> |
|
|
| <div class="btn-group"> |
| <a class="btn" href="https://huggingface.co/spaces/yashu2000/LotteryElicitationEnv" target="_blank">Live Environment Space →</a> |
| <a class="btn btn-outline" href="https://github.com/sharma-yash01/LotteryElicitationEnv" target="_blank">GitHub Repo</a> |
| </div> |
|
|
| |
| <nav class="toc" id="toc"> |
| <h3>Contents</h3> |
| <ol> |
| <li><a href="#why">Lottery Elicitation as Proxy for Strategic Question Design</a></li> |
| <li><a href="#matters">Why This Benchmark Matters</a></li> |
| <li><a href="#prior-work">Prior Work & Novelty</a></li> |
| <li><a href="#design">What LotteryElicitationEnv Is</a></li> |
| <li><a href="#env-design">Environment Design</a></li> |
| <li><a href="#openenv">Why OpenEnv</a></li> |
| <li><a href="#scoring">Scoring: Terminal Reward + Format Shaping</a></li> |
| <li><a href="#architecture">Architecture & Training Pipeline</a></li> |
| <li><a href="#traces">Episode Traces & Reward Hacking</a></li> |
| <li><a href="#results">Results: What We Found</a></li> |
| <li><a href="#engineering">Engineering Lessons</a></li> |
| <li><a href="#foundations">Foundations & Citations</a></li> |
| <li><a href="#quickstart">Quick Start</a></li> |
| <li><a href="#future">Future Work</a></li> |
| </ol> |
| </nav> |
|
|
| |
| <section id="why"> |
| <h2>Lottery Elicitation as Proxy for Strategic Question Design</h2> |
| <p>Most LLM benchmarks test what a model <strong>can say</strong>. They rarely test whether it can <strong>uncover hidden structure through action</strong>. <strong>LotteryElicitationEnv</strong> is an OpenEnv-native environment where an agent designs sequences of lottery pairs to recover a simulated respondent's prospect-theory parameters: risk aversion (γ) and loss aversion (λ).</p> |
| <p>Each episode is adaptive. The agent proposes a lottery pair, observes a binary choice from a closed-form respondent, updates its belief in-context, and on the final turn commits a parameter estimate. Reward arrives only at termination, grounded entirely in arithmetic: mean-squared error against the ground-truth θ*, Holt–Laury consistency of the implied preferences, and an efficiency bonus for stopping early when confident.</p> |
| <p>The challenge is not language. It is <strong>verifiable experimental design</strong>: which lottery pair, right now, is most informative about (γ, λ) given the history so far?</p> |
| <p>The falsifiable claim: <em>a GRPO-trained LLM can recover (γ, λ) more efficiently than the fixed 10-pair Holt–Laury (2002) battery that economists still use as the status-quo elicitation protocol.</em></p> |
| </section> |
|
|
| |
| <section id="matters"> |
| <h2>Why this benchmark matters</h2> |
| <p>Preference elicitation is a foundational problem in behavioral economics, marketing science, medical triage, and alignment. The dominant practice is still a <strong>fixed menu</strong>: every subject answers the same pre-specified list. An adaptive protocol that picks the next question given past answers should be strictly more sample-efficient, but hand-designing a Bayesian optimal experimental design (BOED) over the prospect-theory likelihood is expensive. We ask whether an LLM can <em>learn</em> that policy via RL.</p> |
| <p>The methodology is <strong>transferable</strong>. Any latent-parameter elicitation with an auditable forward model — medical triage (which symptoms to ask about next), educational diagnostics (which question reveals a student's misconception), alignment preference learning (which comparison is most informative about a human's utility function) — fits the same MDP template. Lotteries are the proxy; the capability is <strong>adaptive experimental design</strong>.</p> |
| <p>Every reward signal here is <strong>ground-truth arithmetic</strong>, not a judge. The environment samples θ*, runs a closed-form prospect-theory respondent, and scores the agent against the stored parameters. There is no circularity and no LLM judge in the loop.</p> |
| </section> |
|
|
| |
| <section id="prior-work"> |
| <h2>Prior work & novelty</h2> |
| <p>Most "LLMs + economics" work lands in one of three buckets. None occupies the cell we target:</p> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Prior work bucket</th><th>What it does</th><th>What it does not</th></tr></thead> |
| <tbody> |
| <tr><td><strong>Static economic QA</strong><br><span style="font-size:.85em;color:var(--muted)">Recon (Zhou et al., <a href="https://arxiv.org/abs/2506.00577" target="_blank" style="color:var(--accent2)">arXiv:2506.00577</a>, 2025)</span></td><td>SFT + GRPO on curated economic reasoning items, graded by rules on text</td><td>No sequential active choice, no continuous latent recovery</td></tr> |
| <tr><td><strong>LLMs as agents in games</strong><br><span style="font-size:.85em;color:var(--muted)">RLVR Negotiation (Liu et al., <a href="https://arxiv.org/abs/2604.09855" target="_blank" style="color:var(--accent2)">arXiv:2604.09855</a>, 2026); EconAgent (Li et al., ACL 2024)</span></td><td>RL in strategic or macro simulations, verifiable reward on surplus or budget</td><td>No parameter inference from a known simulator</td></tr> |
| <tr><td><strong>LLMs as subjects</strong><br><span style="font-size:.85em;color:var(--muted)">Horton's <em>Homo silicus</em> (<a href="https://arxiv.org/abs/2301.07543" target="_blank" style="color:var(--accent2)">arXiv:2301.07543</a>); "PT Fails for LLMs" (<a href="https://arxiv.org/abs/2508.08992" target="_blank" style="color:var(--accent2)">arXiv:2508.08992</a>)</span></td><td>Measures whether LLMs <em>are</em> PT-rational</td><td>Does not train them to <em>query</em> a PT-rational counterpart</td></tr> |
| <tr class="novel"><td><strong>LotteryElicitationEnv (ours)</strong></td><td>Sequential MDP, structured JSON lottery actions, terminal reward from ground-truth θ*, non-learned respondent</td><td>Not a human study (yet)</td></tr> |
| </tbody> |
| </table> |
| </div> |
| <blockquote>To our knowledge, no prior work trains an LLM to <strong>adaptively design lottery pairs</strong> against a non-learned prospect-theory respondent with terminal ground-truth rewards under the GRPO + OpenEnv contract. The task, action space, and reward semantics are new; the method (RLVR / GRPO on verifiable signals) is shared with the Recon and negotiation-RLVR lineages.</blockquote> |
| <p><strong>Important caveat</strong> from the "PT Fails for LLMs" literature: we do <em>not</em> claim the policy LLM is PT-rational. We train it to <strong>design experiments</strong> against a PT-rational simulator with known θ*. The LLM is the experimenter, not the subject.</p> |
| </section> |
|
|
| |
| <section id="design"> |
| <h2>What LotteryElicitationEnv is</h2> |
| <blockquote>An OpenEnv-native sequential MDP in which an LLM agent designs lottery pairs, a closed-form prospect-theory respondent answers, and the agent is rewarded on the final turn for how accurately it recovers the respondent's hidden (γ, λ).</blockquote> |
| <p>Each episode proceeds like this:</p> |
| <ul> |
| <li>The environment samples θ* = (γ*, λ*) from configured priors.</li> |
| <li>On turn <em>t</em>, the agent emits a structured JSON action containing two lotteries, and optionally a current estimate of θ and an early-termination flag.</li> |
| <li>The respondent computes expected utility under prospect theory, picks A or B deterministically (optional <strong>Fechner logistic noise</strong> via <code>noise_std</code>), and returns the choice plus the running history.</li> |
| <li>Mid-episode reward is always zero. On termination the environment computes a decomposed reward from the final θ estimate.</li> |
| <li>Termination happens when the agent sets <code>terminate_early=true</code>, runs out of turns, or submits on the final allowed turn.</li> |
| </ul> |
| <p>The agent's interface is deliberately minimal: <strong>raw JSON output</strong>, no tool-call protocol, no markdown parsing. The LLM emits text, the training client parses and steps the environment over WebSocket.</p> |
| </section> |
|
|
| |
| <section id="env-design"> |
| <h2>Environment design</h2> |
| <p>The core contract is three Pydantic types exchanged over the OpenEnv WebSocket:</p> |
| <pre><code><span class="c"># Action (agent → env)</span> |
| class LotteryElicitationAction(Action): |
| lottery_a: Lottery <span class="c"># 2–3 outcomes, probs sum to 1.0</span> |
| lottery_b: Lottery |
| theta_estimate: Optional[dict] <span class="c"># {"gamma": float, "lambda": float}</span> |
| terminate_early: bool = False |
|
|
| <span class="c"># Observation (env → agent)</span> |
| class LotteryElicitationObservation(Observation): |
| step_idx, steps_remaining, max_steps: int |
| history: list[dict] <span class="c"># [{lottery_a, lottery_b, choice}, ...]</span> |
| last_choice: Optional[str] <span class="c"># "A" | "B"</span> |
| gamma_range, lambda_range: tuple[float, float] |
| min_outcome_value, max_outcome_value: float |
| done: bool; reward: Optional[float]; metadata: dict |
|
|
| <span class="c"># State (hidden from agent)</span> |
| true_gamma, true_lambda, gamma_mse, lambda_mse, hl_accuracy</code></pre> |
| <p>A <strong>Lottery</strong> is 2 or 3 outcomes with probabilities that must sum to 1.0. Values lie in <code>[min_outcome_value, max_outcome_value]</code>, both surfaced on every observation so the agent cannot drift off-spec.</p> |
| <p>The <strong>respondent</strong> is pure arithmetic. No LLM, no heuristic:</p> |
| <div class="math-display" aria-label="Prospect-theory value function, expected utility, and choice rule"> |
| \[ |
| v(x) = x^{\gamma}\;\text{if } x \geq 0, |
| \qquad |
| v(x) = -\lambda\,(-x)^{\gamma}\;\text{if } x \lt 0 |
| \] |
| \[ |
| \mathrm{EU}(L) = \sum_{i} p_i\, v(x_i) |
| \] |
| \[ |
| \text{choice} = |
| \mathrm{A}\;\text{if }\mathrm{EU}(A) \geq \mathrm{EU}(B), |
| \quad\text{else }\mathrm{B} |
| \] |
| </div> |
| <p>A <strong>two-stage curriculum</strong> shapes the training distribution:</p> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Stage</th><th>γ sampled</th><th>λ sampled</th><th>Purpose</th></tr></thead> |
| <tbody> |
| <tr><td><strong>Stage 1</strong></td><td>Uniform[γ<sub>lo</sub>, γ<sub>hi</sub>]</td><td>Fixed at 2.25</td><td>Shorten credit assignment, learn risk curvature first</td></tr> |
| <tr><td><strong>Stage 2</strong></td><td>Uniform[γ<sub>lo</sub>, γ<sub>hi</sub>]</td><td>Uniform[λ<sub>lo</sub>, λ<sub>hi</sub>]</td><td>Full two-parameter elicitation</td></tr> |
| </tbody> |
| </table> |
| </div> |
| <p>Curriculum is honored both in <code>EnvConfig</code> and at <code>env.reset(curriculum_stage=...)</code>, so a single server can serve both stages to different sessions concurrently.</p> |
| </section> |
|
|
| |
| <section id="openenv"> |
| <h2>Why OpenEnv</h2> |
| <p>OpenEnv gives us three things that matter for this submission: (1) a standard WebSocket environment contract consumable by TRL's <code>rollout_func</code>, (2) per-session state with <code>SUPPORTS_CONCURRENT_SESSIONS=True</code> and <code>max_concurrent_envs=64</code>, so DDP ranks can hammer the same Space without cross-talk, and (3) a uniform deployment path. The same env code runs in-process for tests, as a Docker container for development, and as a Hugging Face Space during training and evaluation.</p> |
| <p><strong>No in-process environment imports from PT</strong> — everything crosses the wire, exactly like OpenEnv intends. No new abstractions were invented. Base types only: <code>EnvClient</code>, <code>Environment</code>, Pydantic <code>Action</code> / <code>Observation</code>. All extensions (curriculum stage, reward breakdown, history) ride on <code>metadata</code>. No new method signatures, no fork. The env ships with <code>openenv.yaml</code>, a <code>Dockerfile</code>, and a live Hugging Face Space.</p> |
| </section> |
|
|
| |
| <section id="scoring"> |
| <h2>Scoring: a three-part terminal reward, plus format shaping</h2> |
| <p>The environment reward is terminal only. No mid-episode credit. On the final step the env computes:</p> |
| <div class="math-display" aria-label="MSE and Holt-Laury accuracy terms"> |
| \[ |
| \mathrm{mse\_component} |
| = -\Biggl[ |
| \frac{(\hat{\gamma}-\gamma^*)^2}{(\gamma_{\mathrm{hi}}-\gamma_{\mathrm{lo}})^2} |
| + |
| \frac{(\hat{\lambda}-\lambda^*)^2}{(\lambda_{\mathrm{hi}}-\lambda_{\mathrm{lo}})^2} |
| \Biggr] |
| \] |
| \[ |
| \mathrm{hl\_accuracy} |
| = \frac{1}{10}\sum_{k=1}^{10} |
| \mathbf{1}\bigl[s_k(\hat{\theta}) = s_k(\theta^*)\bigr] |
| \] |
| </div> |
| <p class="math-note">Here \(s_k(\cdot)\) is the predicted Holt–Laury choice on menu pair \(k\) under the implied parameters.</p> |
| <div class="math-display" aria-label="Efficiency bonus and total terminal reward"> |
| \[ |
| \mathrm{efficiency\_bonus} |
| = \frac{\max\bigl(0,\; S_{\max} - S_{\mathrm{taken}}\bigr)}{S_{\max}} |
| \] |
| \[ |
| R |
| = w_{\mathrm{mse}}\cdot \mathrm{mse\_component} |
| + w_{\mathrm{HL}}\cdot \mathrm{hl\_accuracy} |
| + w_{\mathrm{eff}}\cdot \mathrm{efficiency\_bonus} |
| \] |
| </div> |
| <p class="math-note">Mapping to code: \(S_{\max}\) is <code>max_steps</code>, \(S_{\mathrm{taken}}\) is <code>steps_taken</code>, and \((w_{\mathrm{mse}}, w_{\mathrm{HL}}, w_{\mathrm{eff}})\) are <code>mse_weight</code>, <code>holt_laury_weight</code>, <code>efficiency_weight</code> in <code>EnvConfig</code>.</p> |
| <p>Defaults live in <code>EnvConfig</code>:</p> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Component</th><th>Weight</th><th>What it rewards</th></tr></thead> |
| <tbody> |
| <tr><td><strong>MSE</strong></td><td class="num">1.0</td><td>Closeness of θ̂ to ground truth in normalized range</td></tr> |
| <tr><td><strong>Holt–Laury accuracy</strong></td><td class="num">0.5</td><td>Behavioral consistency: θ̂ should predict the same HL choices as θ*</td></tr> |
| <tr><td><strong>Efficiency bonus</strong></td><td class="num">0.1</td><td>Stopping early when confident (not just guessing and quitting)</td></tr> |
| <tr><td><strong>Missing θ penalty</strong></td><td class="num">−2.0</td><td>Final-turn action with no valid <code>theta_estimate</code></td></tr> |
| </tbody> |
| </table> |
| </div> |
| <p>The training package adds one optional auxiliary reward: a <strong>format score</strong> (fraction of turns that produced valid JSON, weighted at 0.1–0.75) into the GRPO advantage. It is intended as training wheels, removable once the model reliably emits structured output (>90% validity).</p> |
| <blockquote><strong>Why three components:</strong> MSE alone rewards a lucky guess. Holt–Laury accuracy alone lets the agent memorize the HL menu without recovering θ. Efficiency alone rewards guess-and-quit. The product of incentives forces the agent to <em>actually identify</em> the parameters, not just match a proxy. As we show in the <a href="#traces">Episode Traces</a> section, the efficiency bonus interacts with curriculum design in a way that can <em>create</em> reward-hacking fixed points.</blockquote> |
| </section> |
|
|
| |
| <section id="architecture"> |
| <h2>Architecture & training pipeline</h2> |
| <p>The project is two strictly separated packages: <strong>LotteryElicitationEnv</strong> (the OpenEnv environment) and <strong>LotteryElicitationPT</strong> (the GRPO training client). They communicate exclusively over WebSocket — no in-process imports.</p> |
|
|
| <div class="mermaid-wrap"> |
| <pre class="mermaid"> |
| flowchart LR |
| subgraph PT ["LotteryElicitationPT (Training)"] |
| GRPO["GRPOTrainer<br/>TRL 1.0.0"] |
| RF["rollout_func"] |
| VLLM["vLLM<br/>colocate/server"] |
| PARSE["action_parser<br/>JSON + guardrails"] |
| end |
| subgraph ENV ["LotteryElicitationEnv (OpenEnv)"] |
| WS["FastAPI<br/>WebSocket"] |
| RESP["PT Respondent<br/>v(x) = x^γ ..."] |
| REW["Reward<br/>MSE + HL + eff."] |
| end |
|
|
| GRPO --> RF |
| RF --> VLLM |
| VLLM -->|"generate"| PARSE |
| PARSE -->|"JSON action"| WS |
| WS --> RESP |
| RESP -->|"choice A/B"| WS |
| WS -->|"observation"| RF |
| REW -->|"terminal reward"| WS |
| </pre> |
| <p class="mermaid-caption">Figure 1. System architecture. PT never imports env-side types — everything crosses the WebSocket.</p> |
| </div> |
|
|
| <p>Training uses <strong>GRPO</strong> (Group Relative Policy Optimization), a critic-free RL algorithm ideal for terminal-only rewards. We use TRL 1.0.0's <code>rollout_func</code> contract for explicit control over the generate → parse → step loop, avoiding TRL's Qwen3-only <code>add_response_schema</code> allowlist.</p> |
| <p>The rollout function manages: chat-template tokenization with <code>enable_thinking=False</code>, vLLM generation (colocate or server mode), think-block stripping, null-safe JSON parsing with 18 regression tests, probability normalization, and episode logging to <code>reward_logs.jsonl</code>.</p> |
| </section> |
|
|
| |
| <section id="traces"> |
| <h2>Episode traces & reward hacking</h2> |
| <p>The central research-grade finding of this submission is not a converged checkpoint — it is a <strong>structural diagnosis of how GRPO collapses on multi-turn verifiable-reward environments</strong>. We show the failure, explain the mechanism, and prescribe the fix stack.</p> |
|
|
| <h3>The reward-hacking episode: guess-and-quit</h3> |
| <p>Under Stage 1 training (λ fixed at 2.25), the policy collapsed to the following single-turn episode:</p> |
|
|
| <div class="episode-trace"> |
| <div class="trace-step"> |
| <div class="step-marker terminal"></div> |
| <div class="step-label">Turn 1 (and only turn)</div> |
| <div class="step-content"> |
| <strong>Agent action:</strong><br> |
| <code>{"lottery_a": {"outcomes": [{"value": 50, "probability": 0.5}, {"value": 10, "probability": 0.5}]}, "lottery_b": {"outcomes": [{"value": 30, "probability": 1.0}]}, "theta_estimate": {"gamma": 1.0, "lambda": 2.25}, "terminate_early": true}</code> |
| <br><br> |
| <strong>What happens:</strong> The agent ignores the respondent entirely. It emits a fixed guess (γ=1.0, λ=2.25) and terminates immediately. The lottery pair is irrelevant — the respondent's choice is never used. |
| </div> |
| </div> |
| <div class="trace-verdict bad"> |
| Reward: efficiency bonus (0.9 × 0.1 = 0.09) + partial MSE credit (because λ=2.25 is literally correct under Stage 1) + some HL accuracy. Every rollout in the GRPO group gets the <strong>same reward</strong> → zero within-group variance → zero advantage → zero gradient → no learning. |
| </div> |
| </div> |
|
|
| <h3>The telemetry that reveals collapse</h3> |
| <p>From 322 logged episodes on Qwen3-1.7B and Qwen2.5-7B-Instruct runs:</p> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Signal</th><th>Value</th><th>What it means</th></tr></thead> |
| <tbody> |
| <tr><td><code>frac_reward_zero_std</code></td><td class="num">≈ 1.0</td><td>Every completion in the GRPO group gets identical reward</td></tr> |
| <tr><td><code>loss</code>, <code>grad_norm</code></td><td class="num">≈ 0</td><td>No gradient signal — policy is frozen at a fixed point</td></tr> |
| <tr><td><code>entropy</code></td><td class="num">≈ 1e-5</td><td>Policy has collapsed to a single deterministic output</td></tr> |
| <tr><td><code>clipped_ratio</code></td><td class="num">≈ 1.0</td><td>No policy update being applied</td></tr> |
| <tr><td>Episodes hitting <code>max_steps</code></td><td class="num">98.4%</td><td>Before collapse, most episodes used all 10 turns</td></tr> |
| <tr><td>Tokens/step (early → late)</td><td class="num">~450 → ~55</td><td>Cold-start verbosity converges; OOM is a cold-start problem</td></tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| <h3>Why this happens: the mechanism</h3> |
| <p>Stage 1 fixes λ=2.25 in the data-generating process to simplify credit assignment. But this creates a <strong>partially-correct shortcut</strong>: guessing λ=2.25 is always exactly right for that parameter. Combined with the efficiency bonus rewarding early termination, the model discovers that <em>guess-and-quit on turn 1</em> yields a stable reward. Since every rollout in the GRPO group finds the same shortcut, within-group reward variance drops to zero, GRPO's group-relative advantage becomes zero, and the gradient vanishes. The policy is stuck at a fixed point that is <strong>partially correct by construction</strong>.</p> |
|
|
| <blockquote>This failure mode is general, not specific to our environment. <strong>Any GRPO run on a multi-turn verifiable-reward env with a partially-right-but-cheap shortcut has this bug latent.</strong></blockquote> |
|
|
| <h3>The prescribed fix stack</h3> |
| <ol> |
| <li><strong>Phase A — Bug fixes (done):</strong> Null-safe parsing (<code>_safe_float</code>, <code>_safe_int</code>), think-block stripping, hard-cap on <code>completion_ids</code>, probability normalization. 18 regression tests.</li> |
| <li><strong>Phase B — Training mechanics:</strong> Raise <code>max_completion_length</code>, shorter episodes (5 turns for Stage 1), stronger <code>format_weight</code> (0.1 → 0.75), tune learning rate.</li> |
| <li><strong>Phase C — Kill reward hacking:</strong> Move to Stage 2 (both γ and λ sampled), lower <code>efficiency_weight</code>, raise <code>mse_weight</code>, force minimum-turn exploration before <code>terminate_early</code> can fire, randomize Stage 1 λ to a narrow band around 2.25.</li> |
| <li><strong>Phase D — Advanced shaping:</strong> Information-gain reward term using <code>history</code> to directly incentivize informative lottery pairs.</li> |
| </ol> |
|
|
| <h3>What a good episode should look like</h3> |
| <p>For contrast, here is what the agent <em>should</em> learn to do — an adaptive 5-turn elicitation for a respondent with γ*=0.6, λ*=3.0:</p> |
|
|
| <div class="episode-trace"> |
| <div class="trace-step"> |
| <div class="step-marker good"></div> |
| <div class="step-label">Turn 1 — Probe risk aversion (γ)</div> |
| <div class="step-content"> |
| Lottery A: (100, 0.5; 0, 0.5) vs Lottery B: (40, 1.0)<br> |
| <strong>Choice: B</strong> — respondent is risk-averse (γ < 1), prefers the safe option. |
| </div> |
| </div> |
| <div class="trace-step"> |
| <div class="step-marker good"></div> |
| <div class="step-label">Turn 2 — Narrow γ range</div> |
| <div class="step-content"> |
| Lottery A: (60, 0.7; 10, 0.3) vs Lottery B: (35, 1.0)<br> |
| <strong>Choice: A</strong> — the expected-value gap is large enough that moderate risk aversion still picks A. |
| </div> |
| </div> |
| <div class="trace-step"> |
| <div class="step-marker good"></div> |
| <div class="step-label">Turn 3 — Introduce losses to probe λ</div> |
| <div class="step-content"> |
| Lottery A: (80, 0.5; −20, 0.5) vs Lottery B: (20, 1.0)<br> |
| <strong>Choice: B</strong> — loss aversion makes the mixed gamble unattractive. Constrains λ > 2. |
| </div> |
| </div> |
| <div class="trace-step"> |
| <div class="step-marker good"></div> |
| <div class="step-label">Turn 4 — Refine λ estimate</div> |
| <div class="step-content"> |
| Lottery A: (90, 0.6; −10, 0.4) vs Lottery B: (30, 1.0)<br> |
| <strong>Choice: B</strong> — still avoids losses. λ likely > 2.5. |
| </div> |
| </div> |
| <div class="trace-step"> |
| <div class="step-marker good"></div> |
| <div class="step-label">Turn 5 — Submit estimate + terminate early</div> |
| <div class="step-content"> |
| <code>{"theta_estimate": {"gamma": 0.65, "lambda": 2.9}, "terminate_early": true}</code><br> |
| Efficiency bonus: 5 unused turns → 0.5 × 0.1 = 0.05. MSE is low. HL accuracy is high. |
| </div> |
| </div> |
| <div class="trace-verdict good"> |
| The agent used losses to probe λ (turns 3–4), safe-vs-risky comparisons to probe γ (turns 1–2), and terminated early with a close estimate. This is what adaptive experimental design looks like. |
| </div> |
| </div> |
| </section> |
|
|
| |
| <section id="results"> |
| <h2>Results: what we found</h2> |
|
|
| <h3>Baselines</h3> |
| <p>The environment bundles two deterministic baselines. Both run in-process without a GPU:</p> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Baseline</th><th>Policy</th><th>What it isolates</th></tr></thead> |
| <tbody> |
| <tr><td><strong>Random lottery</strong></td><td>Sample valid lottery pairs uniformly; return prior midpoint as θ̂</td><td>Lower bound: beating it proves the model learned something</td></tr> |
| <tr><td><strong>Holt–Laury fixed battery</strong></td><td>Replay the canonical 10-pair menu; grid-search fit (γ, λ) at 0.01 resolution</td><td>Status-quo comparison from experimental economics</td></tr> |
| </tbody> |
| </table> |
| </div> |
| <p>The eval harness adds: zero-shot API LLMs, local vLLM-served LLMs, and a trained HF policy loaded from disk.</p> |
|
|
| <h3>Baseline performance targets</h3> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Metric</th><th>Holt–Laury fixed</th><th>Random lottery</th><th>Target for trained policy</th></tr></thead> |
| <tbody> |
| <tr><td>γ MSE (normalized)</td><td class="num">≈ 0.02</td><td class="num">high</td><td class="num">lower than HL, at fewer steps</td></tr> |
| <tr><td>λ MSE (normalized)</td><td class="num">≈ 0.3</td><td class="num">high</td><td class="num">lower than HL</td></tr> |
| <tr><td>HL accuracy</td><td class="num">≈ 0.9</td><td class="num">≈ 0.5</td><td class="num">≥ HL</td></tr> |
| <tr><td>Steps used</td><td class="num">10 / 10</td><td class="num">10 / 10</td><td class="num">< 10 via <code>terminate_early</code></td></tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| <h3>Current training status</h3> |
| <p>The pipeline is validated end-to-end. Convergence to a baseline-beating checkpoint is blocked by two factors: the <strong>reward-hacking fixed point</strong> diagnosed above (now understood, fix stack prescribed), and <strong>Unsloth's multi-GPU incompatibility</strong> with our FSDP + vLLM topology (see <a href="#unsloth">Unsloth section</a>), which prevented scaling beyond single-GPU training within the submission window.</p> |
| <p>What we have demonstrated:</p> |
| <ul> |
| <li><strong>Full multi-turn rollouts</strong> against the live OpenEnv WebSocket, with structured JSON actions parsed through null-safe guardrails and episode logs persisting to <code>reward_logs.jsonl</code>.</li> |
| <li><strong>Stable 1×H100 colocate runs on Qwen3-1.7B</strong> — hundreds of optimizer steps with non-zero reward variance before vLLM memory-sync spikes required lowering <code>LEPT_VLLM_GPU_UTIL</code> from 0.90 to 0.75–0.80.</li> |
| <li><strong>Qwen2.5-7B-Instruct colocate run</strong> reaching step 82/1400 (~1 hour wall time) — the first run to generate parseable JSON lotteries intermittently, before crashing on a <code>float(None)</code> edge case (fixed in Session 12).</li> |
| <li><strong>322-episode telemetry</strong> from run-1: 98.4% of episodes hit max_steps=10; token length converged from ~4,000–5,000 to ~55/step by episode 200+. The cold-start OOM problem is solved.</li> |
| <li><strong>Infrastructure for FSDP v1/v2</strong>, vLLM server/colocate, NCCL padding, and three deployment targets (Lambda, AWS EC2, CARC Discovery).</li> |
| </ul> |
| <p>The research story — can a GRPO-trained LLM beat Holt–Laury's 24-year-old fixed battery? — is the experiment this submission sets up. The reward-hacking diagnosis is the finding we contribute now.</p> |
| </section> |
|
|
| |
| <section id="engineering"> |
| <h2>Engineering lessons</h2> |
| <p>Building a real GRPO + OpenEnv + vLLM training pipeline on a multi-turn, verifiable-reward environment surfaced five categories of structural issues. We document them because the next OpenEnv submission will hit every one.</p> |
|
|
| <h3 id="nccl">NCCL desync under variable-length episodes</h3> |
| <p>In <code>vllm_mode=server</code>, every <code>trainer.vllm_generation.generate()</code> call performs <code>gather_object → all_gather_object → broadcast_object_list</code>. Our rollout is <code>while not session.done</code>, so different DDP ranks make different numbers of <code>generate()</code> calls per episode. NCCL collectives are sequence-numbered: different call counts per rank = permanent desync.</p> |
| <p><strong>Symptoms:</strong> tqdm stuck at <code>0/14</code>, GPUs 0–6 pinned at ~32 GiB, GPU 7 idle (vLLM). After ~30 minutes, the NCCL watchdog fires with <code>last enqueued: 529 vs last completed: 527</code> on rank 1. Then <code>UnpicklingError</code> as ranks deserialize off-by-one collective buffers.</p> |
| <p><strong>Fix:</strong> fixed-count padding — every rank performs exactly 8 <code>generate()</code> calls per episode:</p> |
| <pre><code>DIST_SERVER_GENERATES_PER_EPISODE = 8 |
| per_episode_generate_cap = min(max_episode_turns, 8) |
| <span class="c"># After the real loop terminates, issue (8 - num_real_generates)</span> |
| <span class="c"># dummy generates under _temporary_vllm_max_tokens(..., 1).</span> |
| <span class="c"># Outputs are DISCARDED. Guarded with try/finally.</span></code></pre> |
| <p>Active only when <code>vllm_mode == "server"</code> and <code>world_size > 1</code>. Reward, logprobs, and credit assignment are byte-identical to the unpadded case. <strong>Any TRL <code>rollout_func</code> user running variable-length rollouts in server mode has this bug latent.</strong></p> |
|
|
| <h3 id="unsloth">Unsloth & multi-GPU scaling</h3> |
| <p>We invested significant time integrating <strong>Unsloth</strong> for efficient fine-tuning. Unsloth's multi-GPU support did not work with our distributed FSDP + vLLM server topology — specifically, the weight-sharding and vLLM weight-sync paths are incompatible with Unsloth's kernel replacements. This prevented us from scaling beyond single-GPU colocate training within the submission window, which in turn limited the training budget available to push past the reward-hacking fixed point.</p> |
| <p>We document this so the next OpenEnv submitter can avoid the same dead end: if your pipeline requires FSDP + vLLM server mode, Unsloth is not currently a compatible acceleration path.</p> |
|
|
| <h3 id="hygiene">Memory, parsing & model hygiene</h3> |
| <p>Four issues that each crashed training runs, consolidated with their fixes:</p> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Issue</th><th>Root cause</th><th>Fix</th></tr></thead> |
| <tbody> |
| <tr> |
| <td><strong>CUDA OOM</strong> despite low <code>max_completion_length</code></td> |
| <td><code>_rollout_one_episode</code> concatenated generated tokens <em>and</em> observation suffixes across 10 turns → 4,000–5,000 token sequences per episode</td> |
| <td>Hard-cap <code>completion_ids</code> to <code>max_completion_length</code>; strip think blocks from training tensor; <code>PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True</code></td> |
| </tr> |
| <tr> |
| <td><strong>Qwen3 <code><think></code> blocks</strong> despite <code>reasoning_mode=off</code></td> |
| <td><code>enable_thinking=False</code> is prompt-side only; Qwen3 hybrid models still emit verbose think traces that truncate before JSON</td> |
| <td><code>_strip_think_blocks</code> regex (closed + unclosed); re-encode only stripped JSON for <code>completion_ids</code></td> |
| </tr> |
| <tr> |
| <td><strong><code>float(None)</code></strong> crash killing all DDP ranks</td> |
| <td>LLM emitted <code>{"value": null, "probability": 0.5}</code>; parser checked key presence but not None-ness. One rank died → gloo cascade killed all others</td> |
| <td><code>_safe_float</code> / <code>_safe_int</code> guardrails; 18 regression tests; fallback action instead of crash</td> |
| </tr> |
| <tr> |
| <td><strong>FSDP1 <code>_is_root</code> assertion</strong> in server mode</td> |
| <td>TRL 1.0.0's <code>_sync_fsdp1_params_to_vllm</code> calls <code>summon_full_params</code> per child module, corrupting the FSDP root flag (TRL PR #3582, unmerged)</td> |
| <td>Default to <code>vllm_mode=colocate</code>; opt-in FSDP2 behind <code>LEPT_FSDP2_SHARDING=1</code> flag</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| <h3>Key takeaways</h3> |
| <ul> |
| <li><strong>Terminal-only rewards + GRPO is a sweet spot</strong> — but only after you kill the reward-hacking fixed points. Curriculum and reward weighting must be staged carefully.</li> |
| <li><strong><code>max_completion_length</code> ≠ what TRL trains on.</strong> The rollout function keeps appending per-turn generations + observation suffixes until you explicitly hard-cap.</li> |
| <li><strong>Qwen2.5-Instruct > Qwen3-thinking-off</strong> on structured JSON output for this task, despite Qwen3's newer release date.</li> |
| <li><strong>Verifiable reward >>> LLM judge.</strong> A closed-form simulator means training reduces to "can the model hit the right arithmetic target" — no second-order "what does the judge like" pathology.</li> |
| </ul> |
| </section> |
|
|
| |
| <section id="positioning"> |
| <h2>Where this submission sits</h2> |
| <div class="mermaid-wrap"> |
| <pre class="mermaid"> |
| quadrantChart |
| title LLMs + Economics: Task Structure vs Agent Role |
| x-axis "Static / Fixed Tasks" --> "Sequential / Adaptive Tasks" |
| y-axis "LLM as Subject" --> "LLM as Experimenter" |
| quadrant-1 "Our target" |
| quadrant-2 "Unexplored" |
| quadrant-3 "Most prior work" |
| quadrant-4 "Emerging" |
| "Recon (GRPO on econ QA)": [0.2, 0.35] |
| "Homo silicus (Horton)": [0.15, 0.2] |
| "PT Fails for LLMs": [0.25, 0.15] |
| "RLVR Negotiation": [0.7, 0.45] |
| "EconAgent (macro sim)": [0.6, 0.35] |
| "LotteryElicitationEnv": [0.85, 0.85] |
| </pre> |
| <p class="mermaid-caption">Figure 2. Positioning of LotteryElicitationEnv relative to prior work. We occupy the high-sequential, high-experimenter quadrant that no prior work targets.</p> |
| </div> |
| </section> |
|
|
| |
| <section id="foundations"> |
| <h2>Foundations & citations</h2> |
| <div class="table-wrap"> |
| <table> |
| <thead><tr><th>Foundation</th><th>Role in this project</th><th>Citation</th></tr></thead> |
| <tbody> |
| <tr><td><strong>Cumulative prospect theory</strong></td><td>Closed-form respondent: piecewise value function, expected utility, binary choice</td><td>Tversky & Kahneman, <em>J. Risk & Uncertainty</em> 5(4), 1992</td></tr> |
| <tr><td><strong>Holt–Laury risk elicitation</strong></td><td>Fixed 10-pair battery, HL accuracy reward term, grid-search baseline</td><td>Holt & Laury, <em>American Economic Review</em> 92(5), 2002</td></tr> |
| <tr><td><strong>Bayesian OED</strong></td><td>Motivation for adaptive > fixed; hand-derived BOED called "expensive"</td><td>Chaloner & Verdinelli, <em>Statistical Science</em> 10(3), 1995</td></tr> |
| <tr><td><strong>OpenEnv</strong></td><td>Gym-style reset/step, WebSocket transport, HF Space deployment</td><td><a href="https://huggingface.co/blog/openenv" target="_blank" style="color:var(--accent2)">HF Blog: Introducing OpenEnv</a></td></tr> |
| <tr><td><strong>TRL + GRPO</strong></td><td>GRPOTrainer, custom rollout_func, remote env rollouts</td><td>Shao et al., <a href="https://arxiv.org/abs/2402.03300" target="_blank" style="color:var(--accent2)">arXiv:2402.03300</a> (DeepSeekMath)</td></tr> |
| <tr><td><strong>ReasoningEconomicsEnv/PT</strong></td><td>Sibling project — structural template for two-repo split, rollout_func, DDP padding</td><td>Same monorepo</td></tr> |
| </tbody> |
| </table> |
| </div> |
| </section> |
|
|
| |
| <section id="quickstart"> |
| <h2>Quick start</h2> |
| <pre><code><span class="c"># 1. Run the env locally (Python in-process)</span> |
| pip install -e LotteryElicitationEnv |
| python -m lottery_elicitation_env.server.app |
|
|
| <span class="c"># 2. Or pull the HF Space</span> |
| export ENV_BASE_URL="https://yashu2000-lotteryelicitationenv.hf.space" |
|
|
| <span class="c"># 3. Train with GRPO (1xH100 colocate)</span> |
| cd LotteryElicitationPT |
| bash scripts/bootstrap_lambda.sh |
| bash scripts/preflight_lambda.sh |
| bash scripts/run_grpo_lambda.sh |
|
|
| <span class="c"># 4. Evaluate a checkpoint against baselines</span> |
| python -m lottery_elicitation_pt.eval.evaluate \ |
| --policy hf --model ./outputs/ckpt-last \ |
| --episodes 200 --baselines random,holt_laury</code></pre> |
| <p>All episodes are seeded and reproducible from <code>(env_seed, curriculum_stage, θ_prior)</code>. No external fixtures, no live API, no human labels.</p> |
| </section> |
|
|
| <div class="callout"> |
| <div class="q">Can an LLM learn to ask better questions than the economist's 24-year-old playbook?</div> |
| <div class="sub">Adaptive elicitation under a verifiable reward is the test. The pipeline is built. The reward-hacking diagnosis is the first finding.</div> |
| </div> |
|
|
| |
| <section id="future"> |
| <h2>Future work</h2> |
| <ul> |
| <li><strong>Run Stage 2 to convergence</strong> (γ+λ sampled, <code>max_steps=10</code>) and publish the comparison table: Random vs Holt–Laury-Fixed vs Trained-HF across γ-MSE, λ-MSE, HL accuracy, total reward, average steps.</li> |
| <li><strong>Information-gain reward term</strong> using <code>history</code> — directly incentivize informative lottery pairs to kill guess-and-quit at the source.</li> |
| <li><strong>Bayesian OED baseline</strong> for a strong intellectual benchmark that upper-bounds what adaptive elicitation can achieve.</li> |
| <li><strong>Logistic-noise respondent</strong> (<code>noise_std > 0</code>) for a realism ablation — how much does the sim-to-real gap depend on clean deterministic choices?</li> |
| <li><strong>SFT warm-up</strong> on valid-JSON lottery examples before GRPO, to skip the cold-start OOM entirely.</li> |
| <li><strong>Port the NCCL padding pattern upstream into TRL</strong> — the bug is general, the fix is simple.</li> |
| <li><strong>Human-subjects pilot</strong> to measure sim-to-real transfer: real people ≠ prospect theory, but the adaptive-elicitation capability might still beat Holt–Laury on genuine risk preferences.</li> |
| </ul> |
| </section> |
|
|
| |
| <section> |
| <h2>Conclusion</h2> |
| <p><strong>LotteryElicitationEnv</strong> reframes an economics problem as a verifiable RL task. A non-learned prospect-theory respondent, structured JSON lottery actions, and a terminal reward grounded in MSE against ground-truth θ* give us a sequential MDP where every component is auditable.</p> |
| <p>The infrastructure contributions — NCCL desync padding for variable-length rollouts, reward-hacking diagnosis under GRPO with partially-correct shortcuts, think-block hygiene for training tensors, null-safe JSON parsing that prevents DDP cascade failures — are the lessons the next OpenEnv + TRL 1.0 + multi-turn submission will need.</p> |
| <p>The research question remains open: <em>can a GRPO-trained LLM beat Holt–Laury's fixed battery?</em> The pipeline to answer it is built, validated, and documented. The reward-hacking diagnosis is the first finding we contribute.</p> |
| </section> |
|
|
| <div class="footer"> |
| <p>LotteryElicitationEnv · AgentX OpenEnv Track · UC Berkeley RDI</p> |
| <p style="margin-top:.5rem;"> |
| <a href="https://github.com/sharma-yash01/LotteryElicitationEnv" target="_blank">GitHub</a> · |
| <a href="https://huggingface.co/spaces/yashu2000/LotteryElicitationEnv" target="_blank">HF Space</a> · |
| <a href="https://github.com/meta-pytorch/OpenEnv" target="_blank">OpenEnv Framework</a> · |
| <a href="https://huggingface.co/docs/trl/en/openenv" target="_blank">TRL x OpenEnv</a> |
| </p> |
| </div> |
|
|
| </div> |
| </body> |
| </html> |
|
|
