EVALARC / DEVELOPMENT AUDIT

Test the grader.
Then trust the signal.

Behavioral controls for task-specific agent evaluations. Correct and faulty submissions are evaluated against the same externally enforced contract.

8/8

declared faults detected

1.000

reference correctness

docker

execution backend

Which faults depend on one case?

3 of 8 declared faults have exactly one distinct detecting case in the assessed records. Repeating a case under more seeds does not increase this margin. A surviving fault has margin zero; an environment failure remains unassessed.

Margins below are derived from the saved failing case IDs. Rendering an older audit does not rerun its controls.

Does the grader detect plausible defects?

Negative control	Target	Candidate score	Result	Margin	Evidence
`ack-without-work`	basic	0.000	Detected	3	Inspect recorded cases
`memory-only`	persistence	0.783	Detected	3	Inspect recorded cases
`partial-batch`	transactions	0.900	Detected	1	Inspect recorded cases
`cas-always-wins`	compare_swap	0.850	Detected	2	Inspect recorded cases
`boolean-equals-one`	compare_swap	0.925	Detected	1	Inspect recorded cases
`delete-noop`	basic	0.650	Detected	2	Inspect recorded cases
`accept-nonstring-keys`	validation	0.925	Detected	1	Inspect recorded cases
`commit-on-exit`	crash_recovery	0.900	Detected	3	Inspect recorded cases

Inspect the detecting cases

ack-without-work · 3 detecting cases

Target: basic. Detecting case IDs: json-types-and-overwrite, read-write-delete, seeded-state-machine.

read-write-delete · seed 17 · failed

{
  "seed": 17,
  "case_id": "read-write-delete",
  "dimension": "basic",
  "checks": {
    "basic": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 0,
  "duration_seconds": 0.608378,
  "transcript_sha256": "e9d1c1a4ce952f925f5a3d00a84348bf482007670ba13602f73d0129cc422a60",
  "error": "response mismatch at request 1; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}"
}

json-types-and-overwrite · seed 17 · failed

{
  "seed": 17,
  "case_id": "json-types-and-overwrite",
  "dimension": "basic",
  "checks": {
    "basic": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.563549,
  "transcript_sha256": "ad684a2e4e62254b35236e279cda7ec03bfe6bf6a01e3c0b24a1810e6c8088d2",
  "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": null}, got {\"ok\": true}"
}

seeded-state-machine · seed 17 · failed

{
  "seed": 17,
  "case_id": "seeded-state-machine",
  "dimension": "basic",
  "checks": {
    "basic": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 2,
  "duration_seconds": 0.497166,
  "transcript_sha256": "ce6e264c70b36102aabe44145d2a7558766913a2725261b8f120efea98fa8659",
  "error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"ok\": true}"
}

memory-only · 3 detecting cases

Target: persistence. Detecting case IDs: restart-preserves-acknowledged-write, restart-preserves-cas, restart-preserves-transaction.

restart-preserves-acknowledged-write · seed 17 · failed

{
  "seed": 17,
  "case_id": "restart-preserves-acknowledged-write",
  "dimension": "persistence",
  "checks": {
    "persistence": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.976095,
  "transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
  "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
}

restart-preserves-transaction · seed 17 · failed

{
  "seed": 17,
  "case_id": "restart-preserves-transaction",
  "dimension": "persistence",
  "checks": {
    "persistence": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.996261,
  "transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
  "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
}

restart-preserves-cas · seed 17 · failed

{
  "seed": 17,
  "case_id": "restart-preserves-cas",
  "dimension": "persistence",
  "checks": {
    "persistence": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 2,
  "duration_seconds": 0.997537,
  "transcript_sha256": "497082839111501c6aa4912312fe30686befcab05beb84ebbcb99be0d1e8aef1",
  "error": "response mismatch at request 3; expected {\"found\": true, \"ok\": true, \"value\": 321}, got {\"found\": false, \"ok\": true}"
}

partial-batch · single-case dependency

Target: transactions. Detecting case IDs: rollback-batch.

rollback-batch · seed 17 · failed

{
  "seed": 17,
  "case_id": "rollback-batch",
  "dimension": "transactions",
  "checks": {
    "transactions": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 2,
  "duration_seconds": 0.497347,
  "transcript_sha256": "f6015865dce884224cf6ccccc9fbdd367baad17842dde00fbee476f06b1aab0d",
  "error": "response mismatch at request 3; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": true, \"ok\": true, \"value\": \"must roll back\"}"
}

cas-always-wins · 2 detecting cases

Target: compare_swap. Detecting case IDs: cas-match-and-mismatch, cas-type-sensitivity.

cas-match-and-mismatch · seed 17 · failed

{
  "seed": 17,
  "case_id": "cas-match-and-mismatch",
  "dimension": "compare_swap",
  "checks": {
    "compare_swap": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.557099,
  "transcript_sha256": "71199e89272547e9c1a1961a85a650b93c30e2f848912b6be28ea262907eda56",
  "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}"
}

cas-type-sensitivity · seed 17 · failed

{
  "seed": 17,
  "case_id": "cas-type-sensitivity",
  "dimension": "compare_swap",
  "checks": {
    "compare_swap": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.565552,
  "transcript_sha256": "70231a539612a6973a06489411646982ea651ca401223316f7f670e6560a18bf",
  "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}"
}

boolean-equals-one · single-case dependency

Target: compare_swap. Detecting case IDs: cas-type-sensitivity.

cas-type-sensitivity · seed 17 · failed

{
  "seed": 17,
  "case_id": "cas-type-sensitivity",
  "dimension": "compare_swap",
  "checks": {
    "compare_swap": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.57604,
  "transcript_sha256": "70231a539612a6973a06489411646982ea651ca401223316f7f670e6560a18bf",
  "error": "response mismatch at request 2; expected {\"ok\": true, \"swapped\": false}, got {\"ok\": true, \"swapped\": true}"
}

delete-noop · 2 detecting cases

Target: basic. Detecting case IDs: read-write-delete, seeded-state-machine.

read-write-delete · seed 17 · failed

{
  "seed": 17,
  "case_id": "read-write-delete",
  "dimension": "basic",
  "checks": {
    "basic": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 4,
  "duration_seconds": 0.557506,
  "transcript_sha256": "4122b55bc205e4703a3a9f62a5397de448a833f424e54edc914bbbf5465c70d1",
  "error": "response mismatch at request 5; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}"
}

seeded-state-machine · seed 17 · failed

{
  "seed": 17,
  "case_id": "seeded-state-machine",
  "dimension": "basic",
  "checks": {
    "basic": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 8,
  "duration_seconds": 0.556806,
  "transcript_sha256": "0132892f9b4bac6264eb6ef2dae9da8b7bac7a6a158b56357f1319c853615de7",
  "error": "response mismatch at request 9; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}"
}

accept-nonstring-keys · single-case dependency

Target: validation. Detecting case IDs: reject-and-continue.

reject-and-continue · seed 17 · failed

{
  "seed": 17,
  "case_id": "reject-and-continue",
  "dimension": "validation",
  "checks": {
    "validation": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 4,
  "duration_seconds": 0.519073,
  "transcript_sha256": "dc07c86f871afd6b2f208e9751bdd459e52368d54d008ca0622a8edd61a7008b",
  "error": "response mismatch at request 5; expected {\"error\": \"invalid_request\", \"ok\": false}, got {\"ok\": true}"
}

commit-on-exit · 3 detecting cases

Target: crash_recovery. Detecting case IDs: sigkill-after-acknowledgement, sigkill-after-atomic-batch, sigkill-after-delete.

sigkill-after-acknowledgement · seed 17 · failed

{
  "seed": 17,
  "case_id": "sigkill-after-acknowledgement",
  "dimension": "crash_recovery",
  "checks": {
    "crash_recovery": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.998141,
  "transcript_sha256": "f62cc3fb4db094e24ad23141f7cc50908d24647156c4d2ba2386a2487c078e80",
  "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
}

sigkill-after-atomic-batch · seed 17 · failed

{
  "seed": 17,
  "case_id": "sigkill-after-atomic-batch",
  "dimension": "crash_recovery",
  "checks": {
    "crash_recovery": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 1,
  "duration_seconds": 0.978949,
  "transcript_sha256": "43cf3772fd4bd95486b1824b1d73c4e99859fb85088a7f62192d8325b4581b29",
  "error": "response mismatch at request 2; expected {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}, got {\"found\": false, \"ok\": true}"
}

sigkill-after-delete · seed 17 · failed

{
  "seed": 17,
  "case_id": "sigkill-after-delete",
  "dimension": "crash_recovery",
  "checks": {
    "crash_recovery": false
  },
  "passed": false,
  "status": "failed",
  "checks_completed": 2,
  "duration_seconds": 1.47289,
  "transcript_sha256": "679e7ab1f7f833fb12551b29af4fdc1ad8e1051810e594a4da85fdc3420b8c42",
  "error": "response mismatch at request 3; expected {\"found\": false, \"ok\": true}, got {\"found\": true, \"ok\": true, \"value\": {\"items\": [true, null, \"\\u03bb\\n\\u4e16\\u754c\"], \"version\": 5991}}"
}

Positive control

basic1.003/3

validation1.002/2

transactions1.002/2

compare_swap1.002/2

persistence1.003/3

crash_recovery1.003/3

Reproduction record

Task: durable-kv v0.1.0
Seeds: [17]
Grader SHA-256: 52caa3136434569f96097860bc9bf85eb7ada5f7f9890c0540db81afb441871d
Cases SHA-256: 842a16a1e65700cde92ee5f5aaaa10d494a9fc9d71bd2d99fb68147b19aad63a
Image ID: sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea
Created: 2026-09-14T01:29:51.125110+00:00