Spaces:
Sleeping
Sleeping
| title: Flakeforge Environment Server | |
| colorFrom: pink | |
| colorTo: green | |
| sdk: docker | |
| pinned: false | |
| app_port: 8000 | |
| base_path: / | |
| tags: | |
| - openenv | |
| # FlakeForge | |
| [](https://huggingface.co/docs/hub/spaces) | |
| [](https://www.python.org/) | |
| [](https://huggingface.co/) | |
| [](https://pytorch.org/) | |
| [](https://fastapi.tiangolo.com/) | |
| [](https://www.docker.com/) | |
| [](https://docs.pydantic.dev/) | |
| [](https://www.uvicorn.org/) | |
| **FlakeForge** is an [OpenEnv](https://pypi.org/project/openenv-core/)–style reinforcement learning **environment** for learning to **repair flaky Python tests**. The world is (mostly) deterministic: repeated `pytest` runs, AST scans, and static checks produce structured observations. The only learned piece is the policy (your code model); reward is **verifiable** from execution and static oracles, not an LLM judge. | |
| --- | |
| ## Table of contents | |
| 1. [Intuition and inspiration](#1-intuition-and-inspiration) | |
| - [Flaky test root causes (visual taxonomy)](#flaky-test-root-causes-visual-taxonomy) | |
| 2. [Problem we solve](#2-problem-we-solve) | |
| 3. [Environment interface (POMDP view)](#3-environment-interface-pomdp-view) | |
| 4. [Observation space](#4-observation-space) | |
| 5. [Action space (unified agent)](#5-action-space-unified-agent) | |
| 6. [Root-cause categories](#6-root-cause-categories) | |
| 7. [Unified agent: what it outputs and what it sees](#7-unified-agent-what-it-outputs-and-what-it-sees) | |
| 8. [Core tooling](#8-core-tooling) | |
| - [Causal graph engine](#causal-graph-engine) | |
| - [Deep flakiness scanners](#deep-flakiness-scanners) | |
| 9. [Verification stack: oracle engine and patch validator](#9-verification-stack-oracle-engine-and-patch-validator) | |
| 10. [Verifiable reward system](#10-verifiable-reward-system) | |
| - [Visual: gate pipeline](#visual-gate-pipeline) | |
| - [Visual: weighted sum of terms](#visual-weighted-sum-of-terms-full-path-only) | |
| 11. [OpenEnv config and commands](#11-openenv-config-and-commands) | |
| 12. [Build, run, and test locally](#12-build-run-and-test-locally) | |
| - [Inference log (`outputs/inference.log`)](#inference-log-outputsinferencelog) | |
| 13. [Deploy on Hugging Face Spaces](#13-deploy-on-hugging-face-spaces) | |
| 14. [RL training (Unsloth, curriculum)](#14-rl-training-unsloth-curriculum) | |
| 15. [Repository layout](#15-repository-layout) | |
| 16. [Contributing](#16-contributing) | |
| --- | |
| ## 1. Intuition and inspiration | |
| Flaky tests are a major source of **wasted CI time** and **wrong signals** for both humans and ML. Classic fixes (bigger timeouts, more retries, blanket skips) often **mask** symptoms while hiding deeper bugs. We wanted a setting where an agent is pushed toward **structural, minimal fixes** that **survive repeated validation**—closer to how a senior engineer reasons (hypothesis → small edit → re-run) than to a one-shot “rewrite the file” code completion task. | |
| **Inspiration** comes from three lines of work: (1) **POMDP / RL** for sequential decision making under partial observability, (2) **software engineering** research on flaky-test patterns (concurrency, order, I/O, shared state, fixtures, mocks, imports), and (3) **verifiable** training signals: what you can *measure* in pytest output and in the repo’s AST beats what you can only *opine* with another LLM. | |
| ### Flaky test root causes (visual taxonomy) | |
| A **flaky test** is one that **sometimes passes and sometimes fails** for the **same version** of the code. The *symptom* is always “CI is noisy”; the *cause* is usually **hidden shared state**, **timing**, or **environment** coupling. FlakeForge labels hypotheses with the **`ROOT_CAUSE_TYPES`** enum in `models.py` (used in structured `think` claims). Below is how those labels relate to each other and what they mean in practice. | |
| **Map — how the official categories cluster** (read the table after the diagrams for details): | |
| ```mermaid | |
| mindmap | |
| root((ROOT_CAUSE_TYPES)) | |
| Timing and races | |
| async_wait | |
| concurrency | |
| nondeterminism | |
| Shared world and order | |
| shared_state | |
| test_order_dependency | |
| module_cache_pollution | |
| Imports and module lifecycle | |
| import_side_effect | |
| Harness and doubles | |
| fixture_scope_leak | |
| mock_residue | |
| External and resources | |
| network | |
| platform_dependency | |
| resource_leak | |
| Fallback | |
| unknown | |
| ``` | |
| **Conceptual chain** (how families interact—not separate bugs, but **layers** that stack): **timing races** expose **shared mutable state**; **imports and caches** make that state global; **fixtures and mocks** can spread or hide it; **network/platform** and **leaks** turn small races into **order-dependent** failures. Use the table below for precise definitions. | |
| **Lifecycle view — where nondeterminism sneaks in** (collection → run → teardown): | |
| ```mermaid | |
| sequenceDiagram | |
| participant C as Test collection / import | |
| participant R as Run body | |
| participant T as Teardown / next test | |
| C->>R: import_side_effect, module_cache_pollution | |
| R->>R: async_wait, concurrency, nondeterminism, network | |
| R->>T: fixture_scope_leak, mock_residue, resource_leak | |
| T->>C: shared_state, test_order_dependency | |
| ``` | |
| | Category | What typically goes wrong | Why the outcome flickers | Fix direction (high level) | | |
| |----------|-----------------------------|----------------------------|----------------------------| | |
| | **`async_wait`** | `async`/`await` misuse, wrong timeout, background tasks not awaited | Event-loop scheduling and I/O completion order change between runs | Correct `await`, bounded waits, deterministic async teardown | | |
| | **`concurrency`** | Threads, locks, shared mutable state without proper synchronization | Race windows open/close depending on CPU scheduling | Narrow critical sections, proper locks/events, avoid data races | | |
| | **`test_order_dependency`** | Test A leaves state that test B reads; suite order varies | Different collection or parallel workers → different interleaving | Isolate tests, reset globals, avoid cross-test dependencies | | |
| | **`resource_leak`** | Files, sockets, threads, subprocesses not closed | Later tests hit **FD limits**, port exhaustion, or zombie processes | Deterministic `close()` / context managers, pool limits in tests | | |
| | **`shared_state`** | Globals, class attributes, singletons, mutable defaults | Leftover mutations change assertions on the next run | Reset in fixtures, copy-on-write test data, no module-level mutation | | |
| | **`network`** | Real HTTP/DB without hermetic doubles | Latency spikes, rate limits, remote flakiness | Mocks, recorded responses, local fakes, retry policy in **SUT** not tests | | |
| | **`platform_dependency`** | Paths, clocks, timezones, OS-specific APIs | CI image vs laptop differs; wall-clock assumptions | `pathlib`, inject clock, freeze time in tests, portable APIs | | |
| | **`nondeterminism`** | Unseeded randomness, iteration over unordered sets, wall-clock timing in logic | Different values or ordering each run | Seed RNG, sort for stability, replace “sleep for sync” with signals | | |
| | **`import_side_effect`** | Module body runs DB/network/setup on import | Import order or first-import timing changes global state | Move side effects behind `main()` or explicit init functions | | |
| | **`module_cache_pollution`** | `sys.modules` / `@lru_cache` / process-wide caches retain stale data | “First import wins”; order defines cache contents | Clear caches in teardown, avoid process-wide caches in tests, reload strategy | | |
| | **`fixture_scope_leak`** | `session`/`module`-scoped fixtures mutate shared resources | Wide scope + mutation leaks across tests | Narrow fixture scope, factory fixtures, explicit reset fixtures | | |
| | **`mock_residue`** | `patch`/`mock` not stopped or leaks into imports | Later tests see patched symbols or half-mocked deps | `with patch(...)`, `addCleanup`, pytest plugins that enforce cleanup | | |
| | **`unknown`** | Signal is weak or cause is outside the taxonomy | Needs more observation or human triage | Gather more runs, shrink repro, refine category | | |
| **Related labels:** `RELATED_CATEGORIES` in `models.py` defines **soft neighbors** (e.g. `async_wait` ↔ `concurrency`) so small taxonomy mismatches in the model’s `think` block do not get punished as harshly during **reasoning consistency** reward. | |
| --- | |
| ## 2. Problem we solve | |
| **Given** a repository and a test identifier, **stabilize** the failure mode so that the target test (and, ideally, the rest of the suite) passes **repeatably** under the configured runner, without “reward hacking” (weakening assertions, broad `except:`, `skip` markers, or sleeping forever). | |
| The environment therefore: | |
| - Establishes a **baseline** (preflight: sanity, determinism, flakiness) before training credit is spent. | |
| - Surfaces **diagnostics** the policy can lean on: stack-derived failure frontier, call chain, I/O “boundary” hints, a **causal** view of the code path, and **AST-only “deep” flakiness** signals. | |
| - Accepts a **unified** agent turn: structured **diagnosis** + **patch** in one step, then **re-executes** tests and returns **scalar reward** with a full **breakdown** for debugging and GRPO. | |
| --- | |
| ## 3. Environment interface (POMDP view) | |
| | Concept | In FlakeForge | | |
| |--------|----------------| | |
| | **State** (hidden) | Server-side `FlakeForgeState` + git-like snapshots of the repo, episode counters, pass rates, etc. | | |
| | **Observation** | `FlakeForgeObservation` JSON: sources, trees, preflight, deep signals, causal hints, run history, last reward, and more. | | |
| | **Action** | `FlakeForgeAction`: unified think + patch (structured JSON and/or free-text for parsing). | | |
| | **Transition** | `reset` → (optional) configure repo/test; `step` → validate → apply patch → run tests → compute reward. | | |
| | **Reward** | `compute_verifiable_reward` in `server/reward.py` → `RewardBreakdown` and scalar `total`. | | |
| The HTTP layer is created with **OpenEnv**’s `create_app(FlakeForgeEnvironment, FlakeForgeAction, FlakeForgeObservation)`; see `server/app.py` and `openenv.yaml`. | |
| --- | |
| ## 4. Observation space | |
| The main payload is `FlakeForgeObservation` in `models.py`. At a high level, each step exposes: | |
| | Block | Role | | |
| |-------|------| | |
| | **Localisation** | `test_function_source`, `source_under_test`, `file_tree`, `relevant_imports`, `async_markers` | | |
| | **Run dynamics** | `run_history` (`RunRecord`: pass, duration, error_type, stderr excerpt), `current_pass_rate`, `baseline_pass_rate` | | |
| | **Preflight** | `env_type`, `should_train`, `preflight_result` (three-stage gate: sanity, determinism, flakiness) | | |
| | **Deep flakiness (AST, fast)** | `module_cache_violations`, `fixture_scope_risks`, `mock_residue_sites`, `import_side_effect_files`, `async_contamination_alive` | | |
| | **Causal** | `failure_frontier` (file:line:func from stack), `call_chain_to_frontier`, `boundary_crossings` (e.g. HTTP/DB hints), `causal_graph` (summary dict), `causal_hints` | | |
| | **Failure shape** | `failing_stack_trace`, `failure_pattern_summary`, `duration_fingerprint` | | |
| | **Probes** | `order_dependency_detected`, `infrastructure_sensitive` (chaos / stress sensitivity) | | |
| | **Policy feedback** | `patches_applied`, `total_diff_lines`, `think_history` (per-step summary for de-duplication and prompts), `last_think_text`, `last_patch_text`, `last_reward`, `reward_breakdown` | | |
| | **Termination** | `done`, `done_reason` | | |
| The observation is the **ground truth** the unified agent is prompted with on each turn (plus recent history) — see `agent/unified_agent.py` for prompt construction. | |
| --- | |
| ## 5. Action space (unified agent) | |
| V3 uses a **single** high-level action type: **`UNIFIED_PATCH`**. The policy does **not** pick from seven discrete “tool names” in the old Gym sense; it emits a **JSON object** (preferred) with: | |
| - **`think`** — `StructuredThink`: list of `ThinkClaim` items (category, entity, `path::Class.func` `location`, polarity, reason, etc.). | |
| - **`patch`** — `StructuredPatch`: list of `PatchHunk` with `file`, `search`, `replace` (grounded search/replace over real sources). | |
| `FlakeForgeAction` also carries `raw_response`, `think_text`, `patch_text`, and `predicted_category` / `predicted_confidence` for parsers and heuristics. | |
| **Intuition:** the “action space” is **constrained by validation**, not a small enum. Invalid JSON, non-matching `search` blocks, unsafe edits, and hack patterns are rejected or penalized; valid minimal edits that move pass rate and match the **failure frontier** score highest. | |
| --- | |
| ## 6. Root-cause categories | |
| The canonical list is **`ROOT_CAUSE_TYPES`** in `models.py`. Each value is documented with symptoms and fix direction in **[§1 — Flaky test root causes](#flaky-test-root-causes-visual-taxonomy)** (Mermaid mindmap, lifecycle sequence diagram, and reference table). | |
| `RELATED_CATEGORIES` defines **soft** relatedness for **reasoning consistency** reward when the model’s stated category and the patch-inferred category differ slightly. | |
| --- | |
| ## 7. Unified agent: what it outputs and what it sees | |
| **`UnifiedFlakeForgeAgent`** (`agent/unified_agent.py`) is the **JSON-first** interface to your base model. It: | |
| - Builds a long context from the current **`FlakeForgeObservation`** (test code, SUT, signals, run summaries, and history). | |
| - Enforces a **strict JSON** shape (no markdown fences) with **one claim line** in the system prompt; patch hunks use **verifiable** one-line `search` anchors where possible. | |
| - Parses model output into `structured_think` and `structured_patch` when present. | |
| **What the model “sees”** is exactly what you put in the observation: flaky vs stable preflight, pass rates, stack frontier, **deep** flags, and causal summaries — the sort of information you would give a human before asking for a **minimal** fix. | |
| --- | |
| ## 8. Core tooling | |
| ### Causal graph engine | |
| *Implementation: `server/causal_graph.py`.* | |
| **Purpose:** Build a **directed, cross-file** view from the test entry into user code, marking **async**, **call depth**, and **external boundaries** (HTTP, DB, queue, gRPC) where non-determinism or ordering often enters. | |
| - **Input:** repository root, test / entry heuristics, configurable depth. | |
| - **Output:** a `CausalGraph` (nodes, edges, boundary warnings) rendered into `FlakeForgeObservation.causal_graph` and short **`causal_hints`** for the LLM. | |
| - **Why it matters:** Patches that edit files **far** from the **failure frontier** or **causal** neighborhood are easy to down-weight; see **causal_proximity_reward** in `server/reward.py`. | |
| ### Deep flakiness scanners | |
| *Implementation: `server/deep_flakiness.py`.* | |
| **Purpose:** Inexpensive **AST-only** checks that mirror real flaky patterns from SE literature: | |
| - Module-level **caches** / mutable defaults / globals (`@lru_cache`, mutable list defaults, etc.) | |
| - **Fixture** scope and yield/teardown issues | |
| - **`mock.patch`** without proper teardown | |
| - **Import-time** side effects | |
| - **Async/thread** “leftover” heuristics | |
| `build_deep_observation_signals(repo_path)` returns a small dict; the environment copies fields onto the observation so the **policy** and **reward** can use them as **hard features** (no model call). | |
| --- | |
| ## 9. Verification stack: oracle engine and patch validator | |
| ### Oracle engine (`server/oracle_engine.py`) | |
| **Question:** *Is the structured **think** consistent with the **code** and the **patch**?* | |
| - Uses **AST** and optional **LibCST** for static checks. | |
| - Resolves `ThinkClaim.location` to real sources; checks things like **sync primitives** for concurrency-style claims, and whether hunks **address** the claimed locus. | |
| - `verify_structured_think` produces a scalar **`oracle_score`** used in reward when available. | |
| This is **not** a “judge model”: it is **code-facing** and **template-aware**, intended to be reproducible and cheap. | |
| ### Patch validator (`server/patch_validator.py`) | |
| **Question:** *Is the patch **well-formed**, **applicable**, and **not obviously abusing** the reward?* | |
| **Stages** (from module docstring): | |
| 1. **Format** — search/replace blocks, headers, basic hygiene | |
| 2. **Simulate** apply — `search` must match the file (via `simulate_search_replace_patch`) | |
| 3. **Syntax** — `ast.parse` on post-patch text | |
| 4. **Compile** — per-module `compile` check | |
| 5. **Structure** — heuristics (e.g. empty bodies, control-flow red flags) | |
| 6. **Causal proximity (warning)** — far-from-frontier warnings | |
| **Invalid patches are rejected *before* disk write** in `FlakeForgeEnvironment.step`. The validator can emit a **score** (`validation_score`) that flows into **patch_validation_signal** in reward when the patch actually applies. | |
| --- | |
| ## 10. Verifiable reward system | |
| Training a policy on **natural-language “scores” from another LLM** is fast to prototype but fragile: the judge can be **gamed**, **biased toward fluent nonsense**, and **misaligned** with what CI actually cares about (repeatable green builds). FlakeForge instead follows a classic principle from **reinforcement learning** and **program repair** research: put most of the learning signal on **outcomes and checks you can re-run**, then add **light shaping** so credit goes to edits that look like real engineering—not clever ways to pass a rubric. | |
| **Reading map (how the pieces fit):** | |
| 1. **Gates first** — If the action is not parseable or the patch does not apply cleanly, there is little point in interpreting test outcomes; the environment **short-circuits** to a strong negative total (see below). This mirrors *constraint-based* action spaces in RL and *reject sampling* in repair pipelines. | |
| 2. **Primary outcome** — **Pass rate** under repeated `pytest` is the main “task reward”; shaping uses a **potential function** of pass rate so improvements are comparable across episodes. | |
| 3. **Secondary structure** — Proximity to the **failure frontier**, **entropy** of failure *types*, and **anti-hack** heuristics encode inductive biases from flaky-test studies and APR practice: fix the **right** place, collapse **chaotic** failure modes, and do not **delete** the test’s meaning. | |
| 4. **Reasoning channel** — When the **oracle** (`verify_structured_think`) runs, it supplies a **code-grounded** score; otherwise a lighter **category consistency** term ties the stated root cause to what the patch actually does. | |
| `compute_verifiable_reward` in `server/reward.py` assembles a **`RewardBreakdown`**. The table below is the **contract**; the **weights** and **thresholds** in code are authoritative (see the `breakdown.total_reward = round(...)` block and the gate on `format_reward` / `compile_reward` / `anti_hack_penalty`). | |
| | Signal | Role | | |
| |--------|------| | |
| | **format_reward** | Parses structured think/patch; rewards valid claims/hunks; falls back to text markers if needed. | | |
| | **compile_reward** | Patch applied, syntax/compile path; negative if **validator rejected** or not applied. | | |
| | **stability_reward** | **Potential-based** shaping on pass rate: \(\Phi(p)=p^2\), difference vs baseline. | | |
| | **causal_proximity_reward** | Edits **near** `failure_frontier` / call chain; uses boundary metadata. | | |
| | **failure_entropy_reward** | Lower entropy of error **types** after the step is better (more “single failure mode”). | | |
| | **anti_hack_penalty** | Counters assertion weakening, `sleep` insertion, `except:`, `skip` markers, huge diffs, etc. | | |
| | **regression_penalty** | Other tests failing after edit. | | |
| | **reasoning_consistency_reward** / **oracle_reasoning_reward** | Category match (heuristic) vs **oracle** path when `oracle_score` is present. | | |
| | **patch_validation_signal** | From `PatchValidator` **score** (when applied). | | |
| | **noop_patch_penalty** | Trivial or no-op edits that still “succeed” mechanically. | | |
| | **think_history_penalty** | Repeating the same diagnosis without new evidence. | | |
| | **terminal_bonus** | Large bonus when pass rate hits **1.0** or major improvements. | | |
| ### Visual: gate pipeline | |
| Reward is not a single scalar from pytest alone: the implementation **short-circuits** bad actions before the full mix applies, so GRPO can still get gradient signal from “partially right” cases without crediting real hacks. This matches the control flow in `compute_verifiable_reward` (`server/reward.py`). | |
| ```mermaid | |
| flowchart TB | |
| IN["action, observation, patch_result,<br/>post_run_results, baselines, oracle_score"] | |
| IN --> G1{"format_r ≥ 0.75<br/>and anti_hack ≥ 0 ?"} | |
| G1 -->|"no"| E1["total_reward = −1<br/>(or −2 if anti_hack penalty is negative)"] | |
| E1 --> OUT["return RewardBreakdown"] | |
| G1 -->|"yes"| G2{"compile_r == 1.0 ?"} | |
| G2 -->|"no (patch failed, syntax, validator, rollback...)"| E2["total_reward =<br/>0.5·format + 1.0·compile"] | |
| E2 --> OUT | |
| G2 -->|"yes"| FULL["Full path: all terms + terminal_bonus"] | |
| FULL --> W["round( Σ weight_i · term_i ) → total_reward"] | |
| W --> OUT | |
| ``` | |
| *Legibility: “compile_r” is `compile_reward`; gates use the same thresholds as the code (`format_reward < 0.75`, `anti_hack_penalty < 0.0`, `compile_reward < 1.0`).* | |
| ### Visual: weighted sum of terms (full path only) | |
| When the full branch runs, **every term** below is a field on `RewardBreakdown`. Arrows show **which signals are scaled by which coefficient** in the final `total_reward = round(...)` sum (the diagram is a structural map, not a causal ordering). | |
| ```mermaid | |
| flowchart TB | |
| TR[total_reward] | |
| F[format_reward] -->|0.5| TR | |
| C[compile_reward] -->|1.0| TR | |
| ST[stability_reward] -->|3.0| TR | |
| CP[causal_proximity_reward] -->|0.5| TR | |
| FE[failure_entropy_reward] -->|0.5| TR | |
| AH[anti_hack_penalty] -->|1.5| TR | |
| RP[regression_penalty] -->|1.0| TR | |
| O[oracle_component: oracle_reasoning or reasoning_consistency] -->|1.0 / 0.5| TR | |
| PV[patch_validation_signal] -->|1.0| TR | |
| NP[noop_patch_penalty] -->|1.0| TR | |
| TH[think_history_penalty] -->|1.0| TR | |
| TB[terminal_bonus] -->|1.0| TR | |
| ``` | |
| **How to read it:** the **largest** coefficient is **stability** (3.0): repeated `pytest` outcomes drive learning once patches apply. **Format** and **compile** are the usual “can we even run the experiment?” preconditions. **Anti-hack** and **regression** scale penalties so shortcuts and cross-test damage can dominate smaller shaping gains. The **oracle** path uses `oracle_score` when present (`oracle_reasoning` ×1.0); otherwise **reasoning consistency** (×0.5). | |
| ```mermaid | |
| flowchart TB | |
| subgraph M["Mental model — in the full path, these are parallel in the same sum"] | |
| M1[Legibility: format, compile] | |
| M2[Outcome: pass-rate stability] | |
| M3[Frontier: causal + failure entropy + patch validation] | |
| M4[Guardrails: anti-hack, regression, noop, think history] | |
| M5[Process + success: oracle or consistency, terminal bonus] | |
| end | |
| ``` | |
| *The table above is still the full contract; **weights in `server/reward.py`** are authoritative if this doc and code diverge.* | |
| ### Intuition per signal (why it is there) | |
| - **format_reward** — A structured **hypothesis + edit** is easier to validate, audit, and compose with oracles than a blob of prose. This is the same motivation as *chain-of-thought* interfaces in tool-using agents: separate **intent** from **mechanism** so downstream code can check both. | |
| - **compile_reward** — Program repair literature treats **plausible but non-compiling** patches as a dominant failure mode; rewarding **apply + parse + compile** filters a huge class of spurious edits before tests even speak. | |
| - **stability_reward** (\(\Phi(p)=p^2\)) — Squaring pass rate makes **late** progress toward a fully green suite **more** valuable than the same absolute gain from a very red baseline, which matches the “last mile” cost of killing residual flakes. Formally, the step reward uses a **potential difference** \(\Phi(p_{\text{after}})-\Phi(p_{\text{before}})\), which preserves the **optimal policy** under standard assumptions (see references below). | |
| - **causal_proximity_reward** — Flaky failures often localize to a small **dynamic slice** of the codebase; patches that edit the **frontier file** or **stack neighborhood** are more likely to be **minimal** and **causal** than edits three modules away. This aligns with **fault localization** and **minimal repair** goals in APR. | |
| - **failure_entropy_reward** — If every rerun fails differently, the system is still **non-stationary** from the agent’s perspective. **Collapsing** diversity of `error_type` labels is a cheap proxy for “we moved from chaotic nondeterminism toward a **single** addressable failure mode”—consistent with how flaky-test studies categorize **order-, async-, and state-dependent** symptoms. | |
| - **anti_hack_penalty** — Any scalar reward invites **Goodharting**: weaken assertions, swallow exceptions, skip tests, or sleep until races vanish. Penalties target **known** shortcuts so the policy cannot trade away **semantic** test intent for **numeric** reward (see **specification gaming** references). | |
| - **regression_penalty** — A fix that **breaks neighbors** is not acceptable in CI; this term encodes a **multi-test** notion of correctness familiar from regression testing research. | |
| - **oracle_reasoning_reward** vs **reasoning_consistency_reward** — When the oracle runs, reasoning is graded by **static alignment** between claims, locations, and hunks—closer to **process supervision** (“check the steps”) than to an LLM grader. Without the oracle, a lighter **taxonomy** match keeps the stated category from drifting entirely away from the patch. | |
| - **patch_validation_signal**, **noop_patch_penalty**, **think_history_penalty** — Respectively: reward **quality of application** beyond binary success, block **empty** credit loops, and nudge the policy away from **repeating** the same ineffective narrative when the environment has not changed. | |
| - **terminal_bonus** — Sparse **success** signals (full pass or large jumps) are emphasized so GRPO-style training does not drown in **tiny** shaped increments. | |
| ### Hard short-circuit (credit only when basics pass) | |
| If **format** is too low, **compile** is bad, or **anti_hack** is negative, the total is clamped to a **strong negative** (see the `if breakdown.format_reward < 0.75 or ...` block in `server/reward.py`) so the policy cannot grind partial credit on garbage. | |
| ### Weighted total | |
| **Total** is a **weighted sum** of the terms above; coefficients are in the final `breakdown.total_reward = round(...)` assignment in `server/reward.py` (e.g. stability is up-weighted relative to format because **outcome** is the primary learning signal once gates pass). | |
| ### Research connections (selected) | |
| These papers and reports are not an exhaustive bibliography, but they justify **why** each family of terms exists: | |
| | Theme | Pointer | Relevance to FlakeForge | | |
| |-------|---------|-------------------------| | |
| | **Potential-based shaping** | Ng, Harada, Russell, [*Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping*](https://people.eecs.berkeley.edu/~russell/papers/icml99-shaping.pdf) (ICML 1999) | Justifies \(\Phi(p)=p^2\) and **difference** shaping so auxiliary reward does not invent spurious optima (implementation comment in `server/reward.py`). | | |
| | **Flaky tests (empirics)** | Luo et al., [*An Empirical Analysis of Flaky Tests*](https://dl.acm.org/doi/10.1145/2635868.2635920) (FSE 2014) | Motivates **repeatable** evaluation, **order/async** failure modes, and why **pass-rate stability** must be central. | | |
| | **Process / verifiable feedback** | Lightman et al., [*Let’s Verify Step by Step*](https://arxiv.org/abs/2305.20050) (ICLR 2024) | Intuition for **oracle_score**: reward **checkable** reasoning traces, not just final answers. | | |
| | **Outcome training for reasoning** | Uesato et al., [*Solving Math Word Problems with Process- and Outcome-based Feedback*](https://arxiv.org/abs/2211.14275) (2022) | Complements the above: **outcome** (pytest) + **process** (oracle) hybrid. | | |
| | **Specification gaming** | Krakovna, [*Specification gaming examples in AI*](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/) (blog, 2018); Amodei et al., [*Concrete Problems in AI Safety*](https://arxiv.org/abs/1606.06565) (2016) | Motivates **anti_hack** and **hard gates**: misspecified rewards invite **perverse** shortcuts. | | |
| | **Program repair context** | Monperrus, [*Automatic Software Repair: A Bibliography*](https://hal.science/hal-01956501v1/document) (ACM CSUR 2018) | Surveys **compile-time** filters, **localization**, and **regression** awareness—same structural pressures as this reward stack. | | |
| | **Group RL for LLMs** | Shao et al., [*DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*](https://arxiv.org/abs/2402.03300) (2024) | **GRPO**-style training benefits from **low-variance**, **comparable** rewards across a group; `RewardBreakdown` is designed for exactly that kind of logging and advantage estimation. | | |
| **Intuition in one line:** *Reward what **pytest and static checks** can verify, punish **hacks and regressions**, and gently align **stated** diagnosis with **actual** code change—using shaping theory so the extras do not fight the main objective.* | |
| --- | |
| ## 11. OpenEnv config and commands | |
| `openenv.yaml` pins the app entry point: | |
| ```yaml | |
| spec_version: 1 | |
| name: FlakeForge | |
| type: space | |
| runtime: fastapi | |
| app: server.app:app | |
| port: 8000 | |
| ``` | |
| **Typical commands (from a clone of this repo, repo root = `CWD`):** | |
| | Task | Command | | |
| |------|--------| | |
| | Install deps (pip) | `pip install -r server/requirements.txt` | | |
| | **Run ASGI** (dev) | `uvicorn server.app:app --host 0.0.0.0 --port 8000` | | |
| | **Run** (if the package is installed with `[project.scripts]`) | `uv run server --port 8000` (see `test_repos/SETUP.md`) | | |
| | **Docker** | `docker build -t flakeforge-env:latest -f server/Dockerfile .` then run with published `8000` | | |
| | **HTTP health** | `GET http://localhost:8000/health` | | |
| | **Episode (HTTP API)** | `POST /reset` with JSON body; then `POST /step` with action fields (see `client.py` and OpenEnv’s handler) | | |
| **Hugging Face Hub CLI (optional) — push Space files:** | |
| ```bash | |
| hf auth login | |
| hf upload <user>/FlakeForge . --type space | |
| ``` | |
| (Use your Space repo id. Requires `huggingface_hub` / [`hf` CLI](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli).) | |
| The Space UI in this project serves `templates/index.html` at `/` and adds showcase routes under `/api/*` (see `server/api_showcase.py`). | |
| --- | |
| ## 12. Build, run, and test locally | |
| **In-process (Python):** with the package installed (`pip install -e .` or your env manager): | |
| ```python | |
| from FlakeForge.server.FlakeForge_environment import FlakeForgeEnvironment | |
| from FlakeForge.models import FlakeForgeAction | |
| env = FlakeForgeEnvironment() # optional: repo_path, test_identifier | |
| obs = env.reset() | |
| # Build FlakeForgeAction with your model or tests; then: | |
| # result = env.step(action) # return shape from OpenEnv Environment API | |
| ``` | |
| If you run from a source checkout without installing, set `PYTHONPATH` to the repo root and use `from server.FlakeForge_environment import FlakeForgeEnvironment` (and `from models import ...` as in `tests/`). | |
| **HTTP client:** see `client.py` (`FlakeForgeClient.run_episode_remote`) for `httpx` calls to `POST /reset` and `POST /step` against `ENV_BASE_URL`. | |
| **Image build** uses `server/Dockerfile` (see top of that file for `HEALTHCHECK` and `uvicorn` entrypoint). | |
| ### Inference log (`outputs/inference.log`) | |
| Running **`python inference.py`** (with your usual args for repo path, test id, and model) appends **structured episode logs** to **`outputs/inference.log`** (see `inference.py`: the logger is created with that path). The same run also **prints the full episode result as JSON** to stdout when the script finishes, so you can copy a trajectory for post-mortems. That JSON is the in-memory `episode_result` from `run_episode`: `trajectory` (per-step think/patch/reward), `total_reward`, `final_pass_rate`, and `done_reason`. For long runs, use **`scripts/plot_inference_episode.py`** to turn that JSON into a **multi-panel figure** (see below) instead of reading thousands of lines in the console. | |
| **Example: bad policy on a queue-under-load / concurrency case.** The episode below is from a run against a **small worker-pool** fixture (flaky test exercising **`WorkerPool`** and **`QUEUE_CAPACITY`** in `source.py`, in the same family as `test_repos/moderate_load_jitter_flaky`—concurrent submissions vs a tight queue). The model repeatedly **raises queue capacity** and insists the root cause is “small capacity → false failures,” but **pass rate never moves off zero** and the session ends on **`regression_detected`**. | |
| **Why the agent is poor here (reward-aligned):** | |
| - **Patch quality:** Step 1 replaces `QUEUE_CAPACITY = 5` with `WorkerPool.QUEUE_CAPACITY = 10` in a way that is **not a valid or stable edit** to the class body (the breakdown still shows `compile` 1.0 in some steps, but the trajectory illustrates **fuzzy/fragile** edits and later `patch_applied: false` when search text no longer matches). The policy **fixates** on a single knob instead of the actual failure mode the tests encode. | |
| - **No measurable improvement:** `pass_rate` is **0.0** every step, so the environment never sees **stability** improve; that keeps **`stability` negative** and exposes **regression** risk when the patch moves the code away from the reference behavior. | |
| - **Reasoning not grounded in execution:** **Oracle / reasoning** terms (`oracle_reasoning`, and related breakdown fields) stay negative: the high-confidence **concurrency** narrative does not match what **pytest** can verify, so the verifiable stack punishes “sounds right but isn’t” claims. | |
| - **Repetition without learning:** `think_history` turns negative in later steps—similar claims and the same class of capacity tweak yield **increasingly wasteful** trajectories. | |
| - **Outcome:** **`total_reward`: -24.55**, **`done_reason`:** `regression_detected`, after eight steps still at **`final_pass_rate`: 0.0**. This is exactly the kind of run you want the reward to make **unattractive** for training or evaluation: confident language, **no** pass-rate lift, and **guardrail** terms firing. | |
| **Visual dashboard (same run as the JSON in `data/inference_example_episode.json` — step rewards, cumulative reward, pass rate vs confidence, patch apply markers, a heatmap of every `reward_breakdown` key per step, and traces for a few components):** | |
|  | |
| Regenerate the figure after changing the example JSON (or pipe your own saved stdout to a `.json` file): | |
| ```bash | |
| uv pip install -e ".[viz]" # or: pip install matplotlib numpy | |
| python scripts/plot_inference_episode.py -i data/inference_example_episode.json -o docs/assets/inference_episode_dashboard.png | |
| ``` | |
| The full machine-readable episode is in [`data/inference_example_episode.json`](data/inference_example_episode.json) (`trajectory`, `total_reward`, `final_pass_rate`, `done_reason`, `reward_breakdown_history`). | |
| --- | |
| ## 13. Deploy on Hugging Face Spaces | |
| 1. **Create** a new Space; set **SDK** to **Docker** (matches this repo’s README front matter: `sdk: docker`). | |
| 2. **Port:** `app_port: 8000` (already in the YAML front matter of this `README.md`). | |
| 3. **Container:** Hugging Face expects a `Dockerfile` at the **repository root** of the Space. This repository ships the canonical image in **`server/Dockerfile`**. For Spaces, add a **root** `Dockerfile` with the same content (or `COPY` pattern), e.g.: | |
| ```dockerfile | |
| FROM python:3.11-slim | |
| WORKDIR /app | |
| RUN apt-get update && apt-get install -y --no-install-recommends git curl stress-ng iproute2 \ | |
| && rm -rf /var/lib/apt/lists/* | |
| COPY server/requirements.txt /app/server/requirements.txt | |
| RUN pip install --no-cache-dir -r /app/server/requirements.txt | |
| COPY . /app/ | |
| HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --retries=5 \ | |
| CMD curl -f http://localhost:8000/health || exit 1 | |
| EXPOSE 8000 | |
| ENTRYPOINT ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8000"] | |
| ``` | |
| 4. **Environment (optional):** set `FF_REPO_PATH` to point at a **bundled** repo in the image (default in code: `test_repos/timing_race_minimal` under the project root) or your checked-in fixture. | |
| 5. **Push** via Git (recommended) or `hf upload` as above. Wait for the build; the UI should load at the Space URL. | |
| **Note:** The OpenEnv `create_app` stack exposes `GET /health`, `POST /reset`, and `POST /step` like other OpenEnv HTTP servers. OpenAPI may be at `/docs` if enabled by your `openenv-core` version. | |
| --- | |
| ## 14. RL training (Unsloth, curriculum) | |
| **Idea:** GRPO rollouts use **`FlakeForgeEnvironment`**, **structured JSON** actions, and **`compute_verifiable_reward`** — no LLM judge for the main training signal. | |
| **Current stack (after latest improvements):** | |
| - **Unsloth GRPOTrainer** (default) — fast path vs vanilla TRL; strong Qwen2.5 support; 4-bit QLoRA | |
| - **Curriculum** with 3 stages (Easy → Medium → Hard) | |
| - **Manifest-grounded** scenarios under `test_repos/synthetic/`, `data/manifests/`, and `data/curriculum_stages/` | |
| **Install training deps and run the driver:** | |
| ```bash | |
| uv pip install -r training-requirements.txt | |
| python -m training.train_grpo --model Qwen/Qwen2.5-7B-Instruct --max-episodes 500 | |
| ``` | |
| **Notes:** | |
| - Unsloth is **required** for the default training path; the script fails fast with install hints if it is missing. | |
| - Outputs use the **V3** JSON shape `{"think": {...}, "patch": {...}}`. | |
| - W&B project defaults to `flakeforge-rl` when enabled. | |
| - The notebook `FlakeForge_RL_Training (2).ipynb` is kept in sync with this pipeline conceptually; **`training/train_grpo.py`** is the main entry. | |
| --- | |
| ## 15. Repository layout (selected) | |
| | Path | Role | | |
| |------|------| | |
| | `models.py` | Schemas: action, observation, state, `RewardBreakdown` | | |
| | `openenv.yaml` | OpenEnv manifest: FastAPI `server.app:app` | | |
| | `server/app.py` | App factory, static UI route, showcase router | | |
| | `server/FlakeForge_environment.py` | `reset` / `step`, preflight, runner, observations | | |
| | `server/reward.py` | Verifiable reward and breakdown | | |
| | `server/oracle_engine.py` | Static oracle over claims and patches | | |
| | `server/patch_validator.py` | Patch simulation and safety | | |
| | `server/causal_graph.py` | Causal graph builder | | |
| | `server/deep_flakiness.py` | AST “deep” signals | | |
| | `server/docker_runner.py` | pytest / Docker test execution | | |
| | `agent/unified_agent.py` | JSON-first agent and prompts | | |
| | `client.py` | Remote HTTP client example | | |
| | `inference.py` | LLM-driven episodes; logs to `outputs/inference.log`, prints final JSON | | |
| | `scripts/plot_inference_episode.py` | Build PNG dashboard from episode JSON (optional: `pip install -e ".[viz]"`) | | |
| | `data/inference_example_episode.json` | Example episode export for docs / plots | | |
| | `docs/assets/` | Committed figures (e.g. inference dashboard PNG) | | |
| | `training/` | GRPO and curriculum | | |
| ## 16. Contributing | |
| - **Issues & PRs:** Bug reports and small, focused PRs are welcome. Please describe **repro** (repo path, test id, command) and, if possible, a **failing** vs **expected** reward or observation. | |
| - **Scope:** Prefer changes that keep the **environment deterministic** and **reward** tied to **measurable** criteria. | |
| - **Style:** Match existing typing, docstrings, and module boundaries (`models` / `server` / `agent` / `training`). | |
| - **Code of conduct:** Be constructive and specific in review; we optimize for **clarity and reproducibility** over sheer benchmark scores. | |