Spaces:

glayguo
/

evalarc

Running

App Files Files Community

evalarc / coding /index.html

glayguo

Publish verified EvalArc 483f84999ee1

5508abc verified 14 days ago

Raw

History Blame Contribute Delete

21.5 kB

	<!doctype html>
	<html lang="en"><meta charset="utf-8">
	<meta name="viewport" content="width=device-width,initial-scale=1">
	<meta http-equiv="Content-Security-Policy"
	content="default-src 'none'; style-src 'unsafe-inline'; base-uri 'none'; form-action 'none'">
	<title>EvalArc · Grader audit</title>
	<style>

	:root{color-scheme:dark;font:16px/1.6 system-ui,sans-serif;background:#101618;color:#e0ece7}
	body{max-width:1080px;margin:60px auto;padding:0 24px}
	.eyebrow{color:#86dbaf;letter-spacing:.18em;font-size:12px;font-weight:700}
	h1{font-size:clamp(32px,6vw,64px);letter-spacing:-.045em;line-height:1.1;margin:16px 0}
	p{color:#acbeb5;max-width:760px}.cards{display:flex;gap:20px;flex-wrap:wrap;margin:36px 0}
	.card{border:1px solid #33463d;border-radius:12px;padding:18px 26px;flex:1;min-width:150px}
	.number{font-size:38px;color:#96e8b9;font-weight:650}.label{color:#b7c6bf}
	.scroll{overflow:auto}table{border-collapse:collapse;width:100%;font-size:14px}
	td,th{text-align:left;padding:14px;border-bottom:1px solid #33463d}th{color:#86dbaf}
	code{font-family:ui-monospace,monospace}h2{margin-top:42px}
	.dimension{display:flex;gap:16px;align-items:center;max-width:620px;margin:10px 0}
	.dimension span:first-child{width:150px}meter{flex:1;accent-color:#86dbaf}
	footer{margin-top:48px;border-top:1px solid #33463d;padding-top:20px;font-size:13px}
	.metadata{overflow-wrap:anywhere;font-size:13px}
	pre{overflow:auto;padding:18px;background:#17221c;font-size:12px;max-height:480px}
	summary{cursor:pointer;color:#96e8b9}details{margin:16px 0}
	summary{min-height:44px;padding:10px 0}
	a:focus-visible,summary:focus-visible{outline:3px solid #96e8b9;outline-offset:4px}
	.audit-control{border:1px solid #33463d;border-radius:12px;padding:12px 18px}
	.audit-control summary{overflow-wrap:anywhere}.audit-control p{overflow-wrap:anywhere}
	.audit-coverage{border-left:3px solid #edb68d;padding:12px 20px;background:#17221c}
	@media(max-width:640px){body{margin:28px auto;padding:0 16px}.cards{gap:12px}
	.card{min-width:0;flex-basis:100%}.dimension{flex-wrap:wrap}.audit-control{padding:10px 14px}}
	a{color:#96e8b9}.failed,.agent_error,.environment_error{color:#edb68d}
	.passed{color:#96e8b9}td{overflow-wrap:anywhere}
	.scroll table{min-width:680px}td:first-child{min-width:170px;overflow-wrap:normal}
	.change-cards{display:none}.change-card{border:1px solid #33463d;border-radius:12px;padding:16px}
	.change-card h3{font-size:16px;margin:0;overflow-wrap:anywhere}.change-card p{margin:10px 0 0}
	@media(max-width:640px){.change-table{display:none}.change-cards{display:grid;gap:16px}}

	</style>
	<div class="eyebrow">EVALARC / DEVELOPMENT AUDIT</div>
	<h1>Test the grader.<br>Then trust the signal.</h1>
	<p>Behavioral controls for task-specific agent evaluations. Correct and faulty
	submissions are evaluated against the same externally enforced contract.</p>
	<div class="cards"><div class="card"><div class="number">8/8</div><div class="label">declared faults detected</div></div><div class="card"><div class="number">1.000</div><div class="label">reference correctness</div></div><div class="card"><div class="number">docker</div><div class="label">execution backend</div></div></div><section class="audit-coverage" aria-labelledby="coverage-title"><h2 id="coverage-title">Which faults depend on one case?</h2><p><strong>3 of 8</strong> declared faults have exactly one distinct detecting case in the assessed records. Repeating a case under more seeds does not increase this margin. A surviving fault has margin zero; an environment failure remains unassessed.</p><p>Margins below are derived from the saved failing case IDs. Rendering an older audit does not rerun its controls.</p></section><h2>Does the grader detect plausible defects?</h2><div class="scroll" tabindex="0" role="region" aria-label="Declared fault coverage"><table><thead><tr><th>Negative control</th><th>Target</th><th>Candidate score</th><th>Result</th><th title="Cases that caught this fault independently. One means the suite loses this fault if that case changes.">Margin</th><th>Evidence</th></tr></thead><tbody><tr><td><code>ack-without-work</code></td><td>basic</td><td>0.000</td><td>Detected</td><td>3</td><td><a href="#fault-0">Inspect recorded cases</a></td></tr><tr><td><code>memory-only</code></td><td>persistence</td><td>0.783</td><td>Detected</td><td>3</td><td><a href="#fault-1">Inspect recorded cases</a></td></tr><tr><td><code>partial-batch</code></td><td>transactions</td><td>0.900</td><td>Detected</td><td>1</td><td><a href="#fault-2">Inspect recorded cases</a></td></tr><tr><td><code>cas-always-wins</code></td><td>compare_swap</td><td>0.850</td><td>Detected</td><td>2</td><td><a href="#fault-3">Inspect recorded cases</a></td></tr><tr><td><code>boolean-equals-one</code></td><td>compare_swap</td><td>0.925</td><td>Detected</td><td>1</td><td><a href="#fault-4">Inspect recorded cases</a></td></tr><tr><td><code>delete-noop</code></td><td>basic</td><td>0.650</td><td>Detected</td><td>2</td><td><a href="#fault-5">Inspect recorded cases</a></td></tr><tr><td><code>accept-nonstring-keys</code></td><td>validation</td><td>0.925</td><td>Detected</td><td>1</td><td><a href="#fault-6">Inspect recorded cases</a></td></tr><tr><td><code>commit-on-exit</code></td><td>crash_recovery</td><td>0.900</td><td>Detected</td><td>3</td><td><a href="#fault-7">Inspect recorded cases</a></td></tr></tbody></table></div><h2>Inspect the detecting cases</h2><details class="audit-control"><summary>ack-without-work · 3 detecting cases</summary><div id="fault-0"><p>Target: basic. Detecting case IDs: json-types-and-overwrite, read-write-delete, seeded-state-machine.</p><details><summary>read-write-delete · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "read-write-delete",
	"dimension": "basic",
	"checks": {
	"basic": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 0,
	"duration_seconds": 0.608378,
	"transcript_sha256": "e9d1c1a4ce952f925f5a3d00a84348bf482007670ba13602f73d0129cc422a60",
	"error": "response mismatch at request 1; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}"
	}</pre></details><details><summary>json-types-and-overwrite · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "json-types-and-overwrite",
	"dimension": "basic",
	"checks": {
	"basic": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.563549,
	"transcript_sha256": "ad684a2e4e62254b35236e279cda7ec03bfe6bf6a01e3c0b24a1810e6c8088d2",
	"error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": null}, got {\"ok\": true}"
	}</pre></details><details><summary>seeded-state-machine · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "seeded-state-machine",
	"dimension": "basic",
	"checks": {
	"basic": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 2,
	"duration_seconds": 0.497166,
	"transcript_sha256": "ce6e264c70b36102aabe44145d2a7558766913a2725261b8f120efea98fa8659",
	"error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}"
	}</pre></details></div></details><details class="audit-control"><summary>memory-only · 3 detecting cases</summary><div id="fault-1"><p>Target: persistence. Detecting case IDs: restart-preserves-acknowledged-write, restart-preserves-cas, restart-preserves-transaction.</p><details><summary>restart-preserves-acknowledged-write · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "restart-preserves-acknowledged-write",
	"dimension": "persistence",
	"checks": {
	"persistence": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.976095,
	"transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
	"error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
	}</pre></details><details><summary>restart-preserves-transaction · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "restart-preserves-transaction",
	"dimension": "persistence",
	"checks": {
	"persistence": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.996261,
	"transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
	"error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
	}</pre></details><details><summary>restart-preserves-cas · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "restart-preserves-cas",
	"dimension": "persistence",
	"checks": {
	"persistence": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 2,
	"duration_seconds": 0.997537,
	"transcript_sha256": "497082839111501c6aa4912312fe30686befcab05beb84ebbcb99be0d1e8aef1",
	"error": "response mismatch at request 3; expected {\"found\": true, \"ok\": true, \"value\": 321}, got {\"found\": false, \"ok\": true}"
	}</pre></details></div></details><details class="audit-control"><summary>partial-batch · single-case dependency</summary><div id="fault-2"><p>Target: transactions. Detecting case IDs: rollback-batch.</p><details><summary>rollback-batch · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "rollback-batch",
	"dimension": "transactions",
	"checks": {
	"transactions": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 2,
	"duration_seconds": 0.497347,
	"transcript_sha256": "f6015865dce884224cf6ccccc9fbdd367baad17842dde00fbee476f06b1aab0d",
	"error": "response mismatch at request 3; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": true, \"ok\": true, \"value\": \"must roll back\"}"
	}</pre></details></div></details><details class="audit-control"><summary>cas-always-wins · 2 detecting cases</summary><div id="fault-3"><p>Target: compare_swap. Detecting case IDs: cas-match-and-mismatch, cas-type-sensitivity.</p><details><summary>cas-match-and-mismatch · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "cas-match-and-mismatch",
	"dimension": "compare_swap",
	"checks": {
	"compare_swap": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.557099,
	"transcript_sha256": "71199e89272547e9c1a1961a85a650b93c30e2f848912b6be28ea262907eda56",
	"error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}"
	}</pre></details><details><summary>cas-type-sensitivity · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "cas-type-sensitivity",
	"dimension": "compare_swap",
	"checks": {
	"compare_swap": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.565552,
	"transcript_sha256": "70231a539612a6973a06489411646982ea651ca401223316f7f670e6560a18bf",
	"error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}"
	}</pre></details></div></details><details class="audit-control"><summary>boolean-equals-one · single-case dependency</summary><div id="fault-4"><p>Target: compare_swap. Detecting case IDs: cas-type-sensitivity.</p><details><summary>cas-type-sensitivity · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "cas-type-sensitivity",
	"dimension": "compare_swap",
	"checks": {
	"compare_swap": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.57604,
	"transcript_sha256": "70231a539612a6973a06489411646982ea651ca401223316f7f670e6560a18bf",
	"error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}"
	}</pre></details></div></details><details class="audit-control"><summary>delete-noop · 2 detecting cases</summary><div id="fault-5"><p>Target: basic. Detecting case IDs: read-write-delete, seeded-state-machine.</p><details><summary>read-write-delete · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "read-write-delete",
	"dimension": "basic",
	"checks": {
	"basic": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 4,
	"duration_seconds": 0.557506,
	"transcript_sha256": "4122b55bc205e4703a3a9f62a5397de448a833f424e54edc914bbbf5465c70d1",
	"error": "response mismatch at request 5; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}"
	}</pre></details><details><summary>seeded-state-machine · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "seeded-state-machine",
	"dimension": "basic",
	"checks": {
	"basic": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 8,
	"duration_seconds": 0.556806,
	"transcript_sha256": "0132892f9b4bac6264eb6ef2dae9da8b7bac7a6a158b56357f1319c853615de7",
	"error": "response mismatch at request 9; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}"
	}</pre></details></div></details><details class="audit-control"><summary>accept-nonstring-keys · single-case dependency</summary><div id="fault-6"><p>Target: validation. Detecting case IDs: reject-and-continue.</p><details><summary>reject-and-continue · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "reject-and-continue",
	"dimension": "validation",
	"checks": {
	"validation": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 4,
	"duration_seconds": 0.519073,
	"transcript_sha256": "dc07c86f871afd6b2f208e9751bdd459e52368d54d008ca0622a8edd61a7008b",
	"error": "response mismatch at request 5; expected {\"error\": \"invalid_request\", \"ok\": false}, got {\"ok\": true}"
	}</pre></details></div></details><details class="audit-control"><summary>commit-on-exit · 3 detecting cases</summary><div id="fault-7"><p>Target: crash_recovery. Detecting case IDs: sigkill-after-acknowledgement, sigkill-after-atomic-batch, sigkill-after-delete.</p><details><summary>sigkill-after-acknowledgement · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "sigkill-after-acknowledgement",
	"dimension": "crash_recovery",
	"checks": {
	"crash_recovery": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.998141,
	"transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
	"error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
	}</pre></details><details><summary>sigkill-after-atomic-batch · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "sigkill-after-atomic-batch",
	"dimension": "crash_recovery",
	"checks": {
	"crash_recovery": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 1,
	"duration_seconds": 0.978949,
	"transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
	"error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
	}</pre></details><details><summary>sigkill-after-delete · seed 17 · failed</summary><pre>{
	"seed": 17,
	"case_id": "sigkill-after-delete",
	"dimension": "crash_recovery",
	"checks": {
	"crash_recovery": false
	},
	"passed": false,
	"status": "failed",
	"checks_completed": 2,
	"duration_seconds": 1.47289,
	"transcript_sha256": "679e7ab1f7f833fb12551b29af4fdc1ad8e1051810e594a4da85fdc3420b8c42",
	"error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}"
	}</pre></details></div></details><h2>Positive control</h2><div class="dimension"><span>basic</span><meter min="0" max="1" value="1.0">1.00</meter><span>3/3</span></div><div class="dimension"><span>validation</span><meter min="0" max="1" value="1.0">1.00</meter><span>2/2</span></div><div class="dimension"><span>transactions</span><meter min="0" max="1" value="1.0">1.00</meter><span>2/2</span></div><div class="dimension"><span>compare_swap</span><meter min="0" max="1" value="1.0">1.00</meter><span>2/2</span></div><div class="dimension"><span>persistence</span><meter min="0" max="1" value="1.0">1.00</meter><span>3/3</span></div><div class="dimension"><span>crash_recovery</span><meter min="0" max="1" value="1.0">1.00</meter><span>3/3</span></div><h2>Reproduction record</h2><p class="metadata">Task: durable-kv v0.1.0<br>Seeds: [17]<br>Grader SHA-256: 52caa3136434569f96097860bc9bf85eb7ada5f7f9890c0540db81afb441871d<br>Cases SHA-256: 842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a<br>Image ID: sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea<br>Created: 2026-09-14T01:29:51.125110+00:00</p><p><a href="audit.json">Complete JSON evidence for all controls</a></p><footer><strong>Scope:</strong> This is a scripted grader audit, not an AI leaderboard. Public seeds are not held-out evaluation data. Detection rates apply only to the listed fault models. No human time horizon, model quality, or RL improvement is inferred.</footer></html>