FinCosmos / index.html
YupengCao's picture
FinCosmos leaderboard v1 (E1 results; replay to follow)
40d09bb verified
Raw History Blame Contribute Delete
12 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>FinCosmos · Can We Trust Agents with Financial Work? · The Fin AI</title>
<meta name="description" content="FinCosmos evaluates agents doing financial work in the open world: a simulated investor with a hidden strategy, data from the open Web, and programs that check what the agent delivers." />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,500;0,9..144,600;1,9..144,500&family=Newsreader:ital,opsz,wght@0,6..72,400;0,6..72,500;1,6..72,400;1,6..72,500&family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<link rel="icon" href="./assets/finai-logo.svg" type="image/svg+xml" />
<link rel="stylesheet" href="./style.css" />
</head>
<body>
<div class="progress" id="progress"></div>
<div class="glow" aria-hidden="true"></div>
<div class="edge edge-l" aria-hidden="true"><span class="e-name">FinCosmos</span></div>
<div class="edge edge-r" aria-hidden="true"><span class="e-name">The Fin AI</span></div>
<div class="tip" id="tip" role="tooltip"></div>
<header class="header" id="header">
<div class="wrap nav">
<a href="https://thefin.ai/" class="brand" target="_blank" rel="noreferrer"><img src="./assets/finai-logo.svg" alt="" />The Fin AI<span class="brand-sep">/</span><span class="brand-fc">FinCosmos</span></a>
<div class="nav-right">
<nav class="links" id="navLinks">
<a href="#replay" data-sec="replay">Replay</a>
<a href="#leaderboard" data-sec="leaderboard">Leaderboard</a>
<a href="#graders" data-sec="graders">Silent failures</a>
<a href="#workflow" data-sec="workflow">Workflow</a>
<a href="#learning" data-sec="learning">Learning</a>
<a href="#benchmark" data-sec="benchmark">Benchmark</a>
</nav>
<a class="btn btn--sm btn--ghost nav-data" href="https://huggingface.co/datasets/TheFinAI/FinCosmos" target="_blank" rel="noreferrer">Data <span class="arrow">↗</span></a>
<a class="btn btn--sm" data-paper href="#" target="_blank" rel="noreferrer">Paper <span class="arrow">↗</span></a>
</div>
</div>
</header>
<main id="top">
<section class="hero" id="hero">
<div class="wrap hero-grid">
<div class="hero-copy">
<a class="award reveal" href="#graders"><span class="medal" aria-hidden="true"><svg viewBox="0 0 24 24"><circle cx="12" cy="12" r="3.2" fill="#c9a24a"/><circle cx="12" cy="12" r="6.6" fill="none" stroke="#c9a24a" stroke-width="1.2" opacity=".55"/><circle cx="12" cy="12" r="10" fill="none" stroke="#2b6cb0" stroke-width="1" stroke-dasharray="2 2.4" opacity=".5"/></svg></span>Open-world evaluation · <b><span data-n="sessions"></span> sessions, <span data-n="accepted"></span> accepted</b> →</a>
<h1 class="wordmark reveal">FinCosmos</h1>
<p class="hero-sub reveal">The investor signed off.<br />The backtest was <em>wrong</em>.</p>
<p class="lead reveal">Agents now search the Web, write code, and hand back backtests that people act on. FinCosmos leaves the investor's intent, the data, and time open, and then checks what the agent delivers. Every agent, <span class="rotator" id="rotator"></span>, ran the same ten tasks.</p>
<div class="stats reveal" id="stats"></div>
<div class="ctas reveal">
<a class="btn btn--lg" href="#replay">Spot the failure</a>
<a class="btn btn--lg btn--ghost" href="#leaderboard">See the leaderboard</a>
</div>
<p class="paper-links reveal" id="paperLinks"></p>
</div>
<figure class="hero-viz reveal">
<figcaption><span>Orbit map · one star per session</span><a href="#graders">Why none are accepted →</a></figcaption>
<div class="xs-chips" id="orbitChips" role="tablist" aria-label="Choose an agent"></div>
<div class="orbit-wrap">
<div class="xs-side xs-side--l"><b class="c-signed" id="orbitSigned"></b><span>signed off by the investor</span></div>
<div class="orbit" id="orbit"></div>
<div class="xs-side xs-side--r"><b class="c-pass" id="orbitAccepted"></b><span>reach the accepted orbit</span></div>
</div>
<div class="orbit-key"><span><i class="dot dot--signed"></i>signed off</span><span><i class="dot dot--agent"></i>not signed off</span><span><i class="dot dot--base"></i>no-tool baseline</span><span>farther out = higher quality before vetoes</span></div>
<div class="xs-ends" id="orbitEnds"></div>
<p class="hero-claim">Signing off is <em>not</em> evidence that the work is right.</p>
</figure>
</div>
<div class="ticker" id="ticker" aria-label="Leaderboard ticker"><div class="ticker-track" id="tickerTrack"></div></div>
</section>
<section class="section" id="replay">
<div class="wrap">
<div class="sec-head reveal">
<span class="kicker">Replay</span>
<h2>The investor <em>signed off</em>. Find what went wrong.</h2>
<p class="note">Real sessions from the benchmark, trimmed. In each one the simulated investor was satisfied. Mark what you think the delivery got wrong, then open it.</p>
</div>
<div class="game reveal">
<article class="q-card">
<div class="q-top"><span class="q-kick" id="rKick"></span><span class="q-n" id="rN"></span></div>
<p class="r-ask" id="rAsk"></p>
<div class="chat" id="rChat"></div>
<div class="reported" id="rReported"></div>
</article>
<aside class="a-card">
<div class="a-head"><span>What the delivery contains</span><span class="a-score" id="rScore"></span></div>
<p class="a-prompt" id="rPrompt">Which parts are wrong? Mark any number of them.</p>
<div class="checks" id="rChecks"></div>
<div class="verdict" id="rVerdict" hidden></div>
<div class="q-foot">
<button class="btn btn--ghost btn--sm" id="rReveal" type="button">Reveal without guessing</button>
<button class="btn btn--sm" id="rCheck" type="button">Open the delivery</button>
<button class="btn btn--sm" id="rNext" type="button" hidden>Next session →</button>
</div>
</aside>
</div>
</div>
</section>
<section class="section section--tint" id="leaderboard">
<div class="wrap">
<div class="sec-head reveal">
<span class="kicker">Leaderboard</span>
<h2>Ten tasks, three sessions each, <em>no</em> acceptance yet</h2>
<p class="note">Ranked by acceptance rate, then by quality before vetoes. Open a row for the breakdown. Click a heading to rank by it.</p>
</div>
<div class="board reveal"><div class="scroll"><table class="lb" id="lb"></table></div></div>
<div class="keys reveal">
<p><b>Accepted</b> the investor is satisfied and none of the 15 vetoes fails. <b>Signed off</b> the investor is satisfied. <b>Quality</b> pre-veto quality on 0 to 100, with a 95% bootstrap interval over tasks; the five category scores use the same scale. <b>Vetoes</b> failed per session, of 15. <b>Asked</b> share of the details the investor did not state up front that the agent raised. <b>Cost</b> API-equivalent USD per session.</p>
<p class="ramp-key">Darker blue means higher. <span class="ramp"></span></p>
</div>
</div>
</section>
<section class="section" id="graders">
<div class="wrap">
<div class="sec-head reveal">
<span class="kicker">Silent failures</span>
<h2>Graded by sign-off, they pass. <em>Checked</em>, none do.</h2>
<p class="note">The same sessions under three graders. Switch the grader and watch what counts as a success.</p>
</div>
<div class="seg reveal" id="graderSeg" role="tablist" aria-label="Choose a grader"></div>
<div class="gr reveal">
<div class="card gr-main">
<div class="gr-total"><b id="grTotal"></b><span id="grTotalNote"></span></div>
<div class="hb" id="grBars"></div>
</div>
<div class="card gr-why">
<h3>Why the signed-off sessions fail</h3>
<p class="note">Share of the <span data-n="signed"></span> signed-off sessions that fail each group of checks. Program checks catch every one.</p>
<div class="hb hb--rust" id="grWhy"></div>
</div>
</div>
</div>
</section>
<section class="section section--tint" id="workflow">
<div class="wrap">
<div class="sec-head reveal">
<span class="kicker">Workflow</span>
<h2>Agents break at <em>both ends</em> of the work</h2>
<p class="note">Sessions passing each step, of 30 per agent. The last column follows the four agents' 120 sessions through every step in turn.</p>
</div>
<div class="card reveal"><div class="scroll"><table class="fn" id="funnel"></table></div></div>
<div class="sec-head sec-head--sub reveal">
<h3 class="h3-big">They ask about <em>half</em> of what they were never told</h3>
<p class="note">The opening request states about three of the strategy's seven details. The investor answers what the agent asks and corrects what it gets wrong, but volunteers nothing.</p>
</div>
<div class="decl reveal">
<div class="card"><h3>Unstated details the agent raised</h3><div class="hb" id="elCov"></div></div>
<div class="card"><h3>Never-stated details the delivery gets wrong</h3><div class="hb hb--rust" id="elAssume"></div></div>
</div>
</div>
</section>
<section class="section" id="learning">
<div class="wrap">
<div class="sec-head reveal">
<span class="kicker">Learning from feedback</span>
<h2>With FinCosmos feedback, an agent <em>learned all ten</em></h2>
<p class="note">Claude Code with a persistent memory practiced each task until its first acceptance. After every graded session it read the leak-free report, which gives scores and evidence but never reveals the hidden strategy, and edited its memory.</p>
</div>
<div class="learn reveal">
<div class="card learn-stats" id="learnStats"></div>
<div class="card learn-chart">
<div class="legend"><span><i class="ln ln--gray"></i>Each task</span><span><i class="ln ln--k"></i>Median</span></div>
<div class="chart" id="learnChart"></div>
<p class="note">Failed vetoes per session: in the first session, averaged over the first and the last five sessions before acceptance, and in the accepted session. Hover a line for the task.</p>
</div>
</div>
</div>
</section>
<section class="section section--tint" id="benchmark">
<div class="wrap">
<div class="sec-head reveal">
<span class="kicker">Benchmark</span>
<h2>The intent, the data, and time stay <em>open</em></h2>
<p class="note">Existing benchmarks state the task, hand over curated data, or enforce the cutoff for the agent. Each of these removes a silent failure instead of measuring it.</p>
</div>
<div class="m-grid reveal" id="parts"></div>
<div class="sec-head sec-head--sub reveal">
<h3 class="h3-big">Fifteen vetoes, and <em>one</em> is enough to fail</h3>
<p class="note">A session is accepted only when the investor is satisfied and every veto passes. The count shows how many of the <span data-n="signed"></span> signed-off sessions fail each veto.</p>
</div>
<div class="vgrid reveal" id="vgrid"></div>
<div class="trust reveal" id="trust"></div>
</div>
</section>
</main>
<footer class="foot"><div class="wrap foot-in"><span>FinCosmos · The Fin AI</span><span class="foot-links" id="footLinks"></span></div></footer>
<script src="./data.js"></script>
<script src="./app.js"></script>
</body>
</html>