File size: 4,379 Bytes
63f41a6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | {
"columns": [
"Id",
"Missing data / tool",
"What exists instead",
"Build-it-ourselves plan",
"Est. box-hours",
"Source",
"Status"
],
"widths": [
7,
40,
46,
58,
12,
36,
14
],
"rows": [
[
"G1",
"LeanGym-PR: executable repository pull-request tasks (Leanstral 50% of RL mix)",
"Nothing public. MathlibPR is a merge-readiness classifier, not task+patch. princhernwang/mathlib-refactor-history (gated) has refactor diffs.",
"Mine Mathlib4 git history: (pre-commit tree, PR description, post-commit diff) where the diff adds/repairs proofs; task = reproduce the patch until `lake build` of touched files passes under the pinned toolchain. Start with single-file proof PRs only. Success must also require: no sorry/axioms beyond standard, theorem statements preserved, dependency audit; ground-truth patch is one valid solution, not the only one.",
"~20 (CPU builds)",
"critic gap; in-this-work-we-present-leanstral-a-series-of-generalist-code-agent-models-for",
"Planned"
],
[
"G2",
"Prove-or-disprove Lean data (Leanstral 30% of RL mix)",
"Only xlr8harder/lean-proof-or-refute-300 (300 items) and formal-conjectures statements.",
"Negate the CLOSED proposition (¬ ∀ x, H x → P x) of never-solved statements after screening them for misformalisation; keep verified disproofs. Perturbing a true theorem does NOT guarantee falsity: any perturbed variant needs its own verified proof or disproof before it gets a label.",
"~10",
"critic gap",
"Planned"
],
[
"G3",
"Lean code-agent trajectories at SWE scale (Leanstral SFT 50%)",
"OProver/OProofs per-theorem repair trajectories (6.8M rows); small RRMA/Erdos sets; non-Lean SWE traces (SWE-Gym, SWE-smith). OProofs trajectory subset UNCONFIRMED; Goedel-LM/RL_dataset_V2 holds prompts/messages, not verified repairs.",
"Generate our own: correction rounds (chain5) already produce (attempt, compiler error, repair) triples; log them as multi-turn traces in the chat format; mix with OProofs repair subset after decontamination.",
"from 5.2/5.3",
"critic gap",
"Running (via WBS 5.2)"
],
[
"G4",
"Rust->Lean (Aeneas) property / negation dataset",
"None on HF. Dalek Bench (curve25519-dalek-lean-verify) and an Aeneas experience report exist; Verina, CLEVER, FVAPPS, vericoding are spec benchmarks, not Aeneas output.",
"Run Charon+Aeneas on a small set of pure-Rust crates (no unsafe/IO); auto-generate properties from doc comments/tests; prove-or-negate with our prover; hand-validate every flag. This is the bug-finding study.",
"~40",
"critic gap; BUGFINDING_LIT_REVIEW.md",
"Planned (WBS 9)"
],
[
"G5",
"Leanstral bug-finding release (57 repos / 47 flags / 11 real / 5 new)",
"Not public.",
"Reproduce at small scale with baselines (plain LLM review, cargo-fuzz) on the same crates; report precision after property validation.",
"in G4",
"in-this-work-we-present-leanstral-a-series-of-generalist-code-agent-models-for",
"Planned"
],
[
"G6",
"Verifier exploit / reward-hacking corpus",
"None public.",
"Our P1 adversarial suite (61 cases) + red-team classes (axiom walk, lexer, meta) become a versioned regression set shipped with the paper.",
"~2",
"654c20c; redteam/false-accepts",
"Running (WBS 3)"
],
[
"G7",
"Lean token counts for a mid-training mix",
"No source states Lean token totals.",
"Out of scope (no mid-training). If revisited: tokenise Mathlib + Lean-Github with the Qwen3 tokenizer in streaming shards.",
"-",
"critic gap",
"Out of scope"
],
[
"G8",
"Mathlib-version alignment",
"Most new Mathlib-derived sets target Mathlib 4.27-4.31; miniF2F-test header is Lean 4.9.0-rc1.",
"Only use data that compiles under our pinned 4.9.0-rc1 toolchain: re-verify every SFT proof with the strict verifier before training; drop failures (this is also the quality filter). Compiling under 4.9 proves compatibility, not informal fidelity: record retained yield and domain loss per source.",
"in 5.3",
"critic gap",
"Rule"
],
[
"G9",
"GitHub-hosted candidates unconfirmed (API quota hit)",
"vericoding (HF copy verified), curve25519-dalek-lean-verify unverified.",
"Re-check with an authenticated gh CLI (`gh auth login`) before use.",
"<1",
"workflow log",
"Open"
]
]
} |