Upload 24 files

done donaa done done

README.md

@@ -1,14 +1,184 @@
 ---
 title: TestForge
-emoji: 🐢
-colorFrom: green
-colorTo: purple
+emoji: "🔨"
+colorFrom: blue
+colorTo: indigo
 sdk: gradio
-sdk_version: 6.18.0
-python_version: '3.12'
+sdk_version: 4.44.0
+python_version: "3.12"
 app_file: app.py
 pinned: false
+license: mit
+tags:
+  - build-small-hackathon
+  - sponsor:openai
+  - openai-codex
+  - testing
+  - agent
+  - python
+  - pytest
+  - developer-tools
 short_description: Freeze legacy Python behavior before refactoring
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🔨 TestForge — Characterization tests for legacy Python code
+
+**Track:** OpenAI Codex · **Prize target:** Best Use of Codex
+
+Untested legacy Python code is scary to change because you do not know what
+behavior you are about to break. **TestForge** points at a small Python repo,
+discovers its public functions, executes the real code on representative inputs,
+and generates a green pytest suite that locks in current behavior. Then it
+proves the suite is not fake confidence by mutating the code and showing a
+headline **mutation score**.
+
+> Built as **a coding agent built by a coding agent**. The goal is not "AI
+> writes tests." The goal is "AI creates a runnable, inspectable, regression-
+> catching safety net before you refactor."
+
+---
+
+## Why this is trustworthy: the code verifies itself
+
+The core design choice is **capture-then-assert**.
+
+For every discovered function, TestForge chooses a few inputs, **runs the real
+function**, records the actual return value or exception, and only then emits
+pytest assertions.
+
+```python
+def test_apply_discount_case_1():
+    assert apply_discount(100.0, 10.0) == 90.0
+
+def test_with_tax_case_4():
+    with pytest.raises(ValueError):
+        with_tax(100.0, -1.0)
+```
+
+This means:
+
+- every generated test is green on the current code by construction
+- every test calls a real public function, so it cannot collapse into `assert True`
+- the expected value comes from execution, not model guesswork
+
+Then TestForge applies a fixed catalog of source mutations and reruns the same
+generated suite. The result is a visible quality metric like:
+
+```text
+Mutation score: 7/9 behavior changes detected
+```
+
+That turns "the AI made tests" into "the AI made tests that demonstrably catch
+regressions."
+
+---
+
+## Models
+
+This shipped version is intentionally **deterministic and model-free** so the
+demo works with **no model and no GPU**.
+
+| Job | Current implementation | Fallback |
+|---|---|---|
+| Input generation | Type-hint-driven deterministic cases | Fixed mixed literals |
+| Test expectation generation | Real code execution | None needed |
+| Quality proof | Hand-rolled mutation scoring | None needed |
+
+This keeps the demo fast, inspectable, and stable under hackathon time
+pressure. A future enhancement path is to let a small coder model propose extra
+edge-case inputs, but only accept those cases if they improve coverage or the
+mutation score.
+
+---
+
+## How it works
+
+```text
+Pick bundled legacy repo
+  -> Analyze        : AST discovery of public functions and signatures
+  -> Generate       : choose deterministic inputs and capture real outputs
+  -> Render         : write pytest characterization tests
+  -> Run            : execute pytest in a subprocess
+  -> Score          : apply deterministic mutations and count killed mutants
+  -> Export         : produce a PR-style patch adding tests/
+```
+
+## Run locally
+
+```bash
+python -m pip install -r requirements.txt
+python -m pytest -q
+python app.py
+```
+
+Then click **Forge Tests** to generate a full suite for the bundled sample repo,
+inspect the generated tests, and see the mutation score. Click **Inject a
+regression** to watch the same suite turn red on a controlled behavior change.
+
+## Tests
+
+```bash
+python -m pytest -q
+```
+
+The project's own test suite currently covers:
+
+- analyzer discovery and arity checks
+- capture of returned values and raised exceptions
+- deterministic suite generation staying green
+- runner parsing for passing and failing suites
+- deterministic mutation scoring
+- PR patch export
+- forge pipeline smoke path
+- injected regression causing exactly one failure
+
+## Project layout
+
+```text
+app.py                 Gradio UI + orchestration
+agent.py               forge pipeline for the bundled sample repo
+analyzer.py            AST discovery of public functions and methods
+generator.py           deterministic inputs, capture, and pytest rendering
+runner.py              subprocess pytest execution and parsing
+mutator.py             hand-rolled mutation catalog and mutation score
+patch.py               PR-style unified diff export
+inject.py              controlled one-click demo regression
+samples/legacy_repo/   bundled untested Python target repo
+tests/test_testforge.py
+```
+
+## Demo arc
+
+1. Click **Forge Tests**
+2. Watch TestForge analyze the sample repo and generate tests from real execution
+3. See a green pytest result and the mutation score
+4. Click **Inject a regression**
+5. Watch the same suite fail on exactly one controlled change
+6. Download `testforge.patch` as a PR-style artifact
+
+## Codex build log
+
+This repo was built in milestone commits through OpenAI Codex:
+
+- `Add legacy sample and AST analyzer`
+- `Add deterministic capture runner`
+- `Add deterministic mutation scoring`
+- `Add forge app orchestration`
+- `Document hackathon submission`
+
+The most important product decision was keeping the **source of truth in the
+code itself**, not in the model. Codex built the tooling and loop around that
+principle.
+
+## Privacy
+
+The bundled target repo is synthetic and local. The deterministic demo path
+uses no external model, no database, and no persistent user storage.
+
+---
+
+## Submission links
+
+- 🤗 **Live Space:** add your Hugging Face Space URL here
+- 🎥 **Demo video:** add your demo video link here
+- 📣 **Social post:** add your social post link here

agent.py

@@ -0,0 +1,87 @@
+"""Agent orchestration for TestForge."""
+
+from __future__ import annotations
+
+import shutil
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+
+from analyzer import Analysis, analyze
+from generator import GeneratedSuite, generate_suite, write_suite
+from inject import reset_sample
+from mutator import MUTATIONS, Mutation, MutationScore, mutation_score
+from patch import write_patch
+from runner import RunResult, run_pytest
+
+
+ROOT = Path(__file__).resolve().parent
+SAMPLES_ROOT = ROOT / "samples"
+LEGACY_ROOT = SAMPLES_ROOT / "legacy_repo"
+RUNS_ROOT = ROOT / "generated" / "runs"
+
+
+@dataclass(frozen=True)
+class ForgeArtifacts:
+    run_dir: Path
+    analysis: Analysis
+    suite: GeneratedSuite
+    green: RunResult
+    mutation: MutationScore
+    patch_path: Path
+    logs: tuple[str, ...]
+
+
+def forge_legacy_repo(
+    max_cases_per_function: int = 4,
+    mutations: tuple[Mutation, ...] = MUTATIONS,
+) -> ForgeArtifacts:
+    """Run the deterministic TestForge pipeline for the bundled sample."""
+    run_dir = _fresh_run_dir()
+    logs: list[str] = []
+
+    logs.append("Target: bundled legacy_repo with zero tests")
+    reset_sample(SAMPLES_ROOT, run_dir)
+
+    analysis = analyze(run_dir / "samples" / "legacy_repo", package="legacy_repo")
+    logs.append(f"Analyzed {len(analysis.functions)} public functions")
+    for fn in analysis.functions:
+        logs.append(f"  - {fn.module}.{fn.qualname}({', '.join(fn.parameters)})")
+
+    suite = generate_suite(analysis, max_cases_per_function=max_cases_per_function)
+    write_suite(suite, run_dir)
+    logs.append(f"Captured {suite.assertion_count} behaviors into {len(suite.files)} test files")
+
+    green = run_pytest(run_dir, package_parent=run_dir / "samples")
+    logs.append(f"Suite run: {green.summary}")
+    if green.coverage is not None:
+        logs.append(f"Coverage: 0% -> {green.coverage:.0f}%")
+    else:
+        logs.append("Coverage: pytest-cov unavailable locally; suite still ran green")
+
+    mutation = mutation_score(run_dir / "samples", suite, run_dir / "mutants", mutations=mutations)
+    logs.append(f"Mutation score: {mutation.headline()} ({mutation.percent:.1f}%)")
+    for survived in mutation.survived:
+        logs.append(f"  survived: {survived.label}")
+
+    patch_path = write_patch(suite, run_dir / "testforge.patch")
+    logs.append(f"Patch ready: {patch_path.name}")
+
+    return ForgeArtifacts(
+        run_dir=run_dir,
+        analysis=analysis,
+        suite=suite,
+        green=green,
+        mutation=mutation,
+        patch_path=patch_path,
+        logs=tuple(logs),
+    )
+
+
+def _fresh_run_dir() -> Path:
+    RUNS_ROOT.mkdir(parents=True, exist_ok=True)
+    path = Path(tempfile.mkdtemp(prefix="testforge_", dir=RUNS_ROOT))
+    if path.exists():
+        shutil.rmtree(path)
+    path.mkdir(parents=True)
+    return path

analyzer.py

@@ -0,0 +1,128 @@
+"""AST-based discovery for small Python packages."""
+
+from __future__ import annotations
+
+import ast
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class FunctionInfo:
+    """Metadata needed to generate characterization tests."""
+
+    module: str
+    qualname: str
+    name: str
+    parameters: tuple[str, ...]
+    type_hints: dict[str, str]
+    return_hint: str | None
+    docstring: str
+    source: str
+    file_path: str
+    lineno: int
+
+    @property
+    def import_name(self) -> str:
+        return self.qualname.split(".")[-1]
+
+    @property
+    def arity(self) -> int:
+        return len(self.parameters)
+
+
+@dataclass(frozen=True)
+class Analysis:
+    """Public API discovered in a target package."""
+
+    root: Path
+    package: str
+    functions: tuple[FunctionInfo, ...]
+
+
+def analyze(root: str | Path, package: str | None = None) -> Analysis:
+    """Discover public top-level functions and public class methods."""
+    root_path = Path(root).resolve()
+    if not root_path.exists():
+        raise FileNotFoundError(root_path)
+
+    package_name = package or root_path.name
+    functions: list[FunctionInfo] = []
+    for file_path in sorted(root_path.rglob("*.py")):
+        if file_path.name == "__init__.py":
+            continue
+        module = _module_name(root_path, package_name, file_path)
+        source = file_path.read_text(encoding="utf-8")
+        tree = ast.parse(source, filename=str(file_path))
+        lines = source.splitlines()
+        functions.extend(_top_level_functions(tree, module, file_path, lines))
+        functions.extend(_class_methods(tree, module, file_path, lines))
+
+    return Analysis(root=root_path, package=package_name, functions=tuple(functions))
+
+
+def _module_name(root: Path, package: str, file_path: Path) -> str:
+    relative = file_path.relative_to(root).with_suffix("")
+    parts = (package, *relative.parts)
+    return ".".join(parts)
+
+
+def _top_level_functions(
+    tree: ast.Module, module: str, file_path: Path, lines: list[str]
+) -> list[FunctionInfo]:
+    found: list[FunctionInfo] = []
+    for node in tree.body:
+        if isinstance(node, ast.FunctionDef) and _is_public(node.name):
+            found.append(_function_info(node, module, node.name, file_path, lines))
+    return found
+
+
+def _class_methods(
+    tree: ast.Module, module: str, file_path: Path, lines: list[str]
+) -> list[FunctionInfo]:
+    found: list[FunctionInfo] = []
+    for class_node in tree.body:
+        if not isinstance(class_node, ast.ClassDef) or not _is_public(class_node.name):
+            continue
+        for node in class_node.body:
+            if isinstance(node, ast.FunctionDef) and _is_public(node.name):
+                qualname = f"{class_node.name}.{node.name}"
+                found.append(_function_info(node, module, qualname, file_path, lines))
+    return found
+
+
+def _function_info(
+    node: ast.FunctionDef, module: str, qualname: str, file_path: Path, lines: list[str]
+) -> FunctionInfo:
+    parameters: list[str] = []
+    hints: dict[str, str] = {}
+    for arg in node.args.args:
+        if arg.arg in {"self", "cls"}:
+            continue
+        parameters.append(arg.arg)
+        if arg.annotation is not None:
+            hints[arg.arg] = ast.unparse(arg.annotation)
+
+    source = _source_segment(node, lines)
+    return FunctionInfo(
+        module=module,
+        qualname=qualname,
+        name=qualname,
+        parameters=tuple(parameters),
+        type_hints=hints,
+        return_hint=ast.unparse(node.returns) if node.returns is not None else None,
+        docstring=ast.get_docstring(node) or "",
+        source=source,
+        file_path=str(file_path),
+        lineno=node.lineno,
+    )
+
+
+def _source_segment(node: ast.AST, lines: list[str]) -> str:
+    end = getattr(node, "end_lineno", node.lineno)
+    return "\n".join(lines[node.lineno - 1 : end])
+
+
+def _is_public(name: str) -> bool:
+    return not name.startswith("_")
+

app.py

@@ -0,0 +1,154 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import gradio as gr
+
+from agent import forge_legacy_repo
+from inject import reset_sample, run_injected_suite
+
+
+LAST_RUN: dict[str, Path] = {}
+
+
+CSS = """
+:root {
+  --tf-ink: #171717;
+  --tf-muted: #595959;
+  --tf-line: #d6d3d1;
+  --tf-panel: #fafaf9;
+  --tf-accent: #0f766e;
+  --tf-warn: #b45309;
+}
+.gradio-container {
+  font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+  color: var(--tf-ink);
+}
+.tf-title h1 {
+  font-size: 34px;
+  line-height: 1.1;
+  letter-spacing: 0;
+  margin-bottom: 4px;
+}
+.tf-title p {
+  color: var(--tf-muted);
+  margin-top: 0;
+}
+.tf-metric {
+  border: 1px solid var(--tf-line);
+  background: var(--tf-panel);
+  border-radius: 8px;
+  padding: 14px;
+}
+"""
+
+
+def forge(use_model: bool, max_cases: int):
+    del use_model  # deterministic backbone is the demo path.
+    artifacts = forge_legacy_repo(max_cases_per_function=max_cases)
+    LAST_RUN["run_dir"] = artifacts.run_dir
+    log = "\n".join(artifacts.logs)
+    preview = _suite_preview(artifacts.suite.files)
+    result_md = _result_markdown(artifacts)
+    return log, preview, result_md, str(artifacts.patch_path), gr.update(interactive=True)
+
+
+def inject_regression():
+    run_dir = LAST_RUN.get("run_dir")
+    if run_dir is None:
+        return "Forge tests first.", "No run yet."
+
+    reset_sample(Path(__file__).resolve().parent / "samples", run_dir)
+    result = run_injected_suite(run_dir)
+    status = "caught" if not result.run.ok else "survived"
+    summary = (
+        f"Injected: {result.mutation_label}\n"
+        f"Result: {result.run.summary} ({status})\n\n"
+        f"{_first_failure(result.run.stdout)}"
+    )
+    return summary, result.run.stdout[-4000:]
+
+
+def _suite_preview(files: dict[str, str]) -> str:
+    parts: list[str] = []
+    for name, content in sorted(files.items()):
+        parts.append(f"# {name}\n{content}")
+    return "\n\n".join(parts)
+
+
+def _result_markdown(artifacts) -> str:
+    survived = artifacts.mutation.survived
+    survived_text = "\n".join(f"- {item.label}" for item in survived) or "- None"
+    coverage = "not installed locally" if artifacts.green.coverage is None else f"{artifacts.green.coverage:.0f}%"
+    return f"""
+### Forge Result
+
+**Suite:** {artifacts.green.summary}  
+**Coverage:** 0% -> {coverage}  
+**Mutation score:** {artifacts.mutation.headline()} ({artifacts.mutation.percent:.1f}%)
+
+**Surviving mutants**
+{survived_text}
+"""
+
+
+def _first_failure(output: str) -> str:
+    lines = output.splitlines()
+    for index, line in enumerate(lines):
+        if line.startswith("FAILED ") or "E       AssertionError" in line:
+            return "\n".join(lines[max(0, index - 4) : index + 10])
+    return "\n".join(lines[-20:])
+
+
+with gr.Blocks(title="TestForge Characterization Lab") as demo:
+    gr.HTML(
+        """
+        <div class="tf-title">
+          <h1>TestForge -- Characterization Lab</h1>
+          <p>Freeze what legacy Python code does today. Refactor without fear.</p>
+        </div>
+        """
+    )
+    with gr.Row():
+        target = gr.Dropdown(
+            ["bundled legacy_repo"],
+            value="bundled legacy_repo",
+            label="Target",
+            interactive=False,
+        )
+        use_model = gr.Checkbox(False, label="Use small coder model")
+        max_cases = gr.Slider(2, 4, value=4, step=1, label="Cases per function")
+        forge_btn = gr.Button("Forge Tests", variant="primary")
+
+    with gr.Row():
+        log = gr.Textbox(label="Agent log", lines=14)
+        results = gr.Markdown(label="Results")
+
+    tests_preview = gr.Code(label="Generated tests", language="python", lines=18)
+
+    with gr.Row():
+        inject_btn = gr.Button("Inject a regression", interactive=False)
+        patch_file = gr.File(label="Download PR patch")
+
+    with gr.Row():
+        inject_summary = gr.Textbox(label="Inject result", lines=8)
+        inject_output = gr.Textbox(label="Pytest output", lines=8)
+
+    forge_btn.click(
+        forge,
+        inputs=[use_model, max_cases],
+        outputs=[log, tests_preview, results, patch_file, inject_btn],
+    )
+    inject_btn.click(inject_regression, outputs=[inject_summary, inject_output])
+
+
+if __name__ == "__main__":
+    demo.launch(
+        server_name="0.0.0.0",
+        server_port=7860,
+        prevent_thread_lock=False,
+        inbrowser=False,
+        show_error=True,
+        quiet=False,
+        css=CSS,
+    )

generator.py

@@ -0,0 +1,166 @@
+"""Deterministic capture-then-assert test generation."""
+
+from __future__ import annotations
+
+import importlib
+import itertools
+import reprlib
+import sys
+from dataclasses import dataclass
+from datetime import date
+from pathlib import Path
+from typing import Any
+
+from analyzer import Analysis, FunctionInfo
+
+
+@dataclass(frozen=True)
+class CapturedCase:
+    """One executed behavior captured from the current code."""
+
+    args: tuple[Any, ...]
+    outcome: str
+    value_repr: str | None = None
+    exception_type: str | None = None
+
+
+@dataclass(frozen=True)
+class GeneratedSuite:
+    """Rendered tests and source capture metadata."""
+
+    files: dict[str, str]
+    cases_by_function: dict[str, list[CapturedCase]]
+    assertion_count: int
+
+
+def generate_suite(analysis: Analysis, max_cases_per_function: int = 4) -> GeneratedSuite:
+    """Generate a green characterization suite by executing current behavior."""
+    _ensure_import_root(analysis.root.parent)
+    files: dict[str, str] = {}
+    cases_by_function: dict[str, list[CapturedCase]] = {}
+    for fn in analysis.functions:
+        candidates = deterministic_inputs(fn, max_cases=max_cases_per_function)
+        cases = capture(fn, candidates)
+        if not cases:
+            continue
+        cases_by_function[f"{fn.module}.{fn.qualname}"] = cases
+        file_name = f"tests/test_{_safe_name(fn.module)}_{_safe_name(fn.qualname)}.py"
+        files[file_name] = render_test_file(fn, cases)
+
+    assertion_count = sum(len(cases) for cases in cases_by_function.values())
+    return GeneratedSuite(files=files, cases_by_function=cases_by_function, assertion_count=assertion_count)
+
+
+def deterministic_inputs(fn: FunctionInfo, max_cases: int = 4) -> list[tuple[Any, ...]]:
+    """Infer a compact set of inputs from type hints and parameter names."""
+    if fn.arity == 0:
+        return [()]
+
+    choices = [_values_for_parameter(name, fn.type_hints.get(name, "")) for name in fn.parameters]
+    cases = list(itertools.islice(itertools.product(*choices), max_cases))
+    return [tuple(case) for case in cases]
+
+
+def capture(fn: FunctionInfo, cases: list[tuple[Any, ...]]) -> list[CapturedCase]:
+    """Execute a function on candidate inputs and record output or exception."""
+    callable_obj = _load_callable(fn)
+    captured: list[CapturedCase] = []
+    for args in cases:
+        if len(args) != fn.arity:
+            continue
+        try:
+            value = callable_obj(*args)
+        except Exception as exc:  # noqa: BLE001 - exceptions are the behavior being captured.
+            captured.append(
+                CapturedCase(args=args, outcome="exception", exception_type=type(exc).__name__)
+            )
+        else:
+            captured.append(CapturedCase(args=args, outcome="return", value_repr=_literal(value)))
+    return captured
+
+
+def render_test_file(fn: FunctionInfo, cases: list[CapturedCase]) -> str:
+    """Render a pytest module for captured behavior."""
+    needs_pytest = any(case.outcome == "exception" for case in cases)
+    imports = ["from datetime import date"]
+    if needs_pytest:
+        imports.append("import pytest")
+    imports.append(f"from {fn.module} import {fn.import_name}")
+
+    body: list[str] = []
+    for index, case in enumerate(cases, start=1):
+        test_name = f"test_{_safe_name(fn.qualname)}_case_{index}"
+        body.append("")
+        body.append(f"def {test_name}():")
+        args = ", ".join(_literal(arg) for arg in case.args)
+        if case.outcome == "exception":
+            body.append(f"    with pytest.raises({case.exception_type}):")
+            body.append(f"        {fn.import_name}({args})")
+        else:
+            body.append(f"    assert {fn.import_name}({args}) == {case.value_repr}")
+
+    return "\n".join(imports + body) + "\n"
+
+
+def write_suite(suite: GeneratedSuite, target_dir: str | Path) -> Path:
+    """Write generated test files under target_dir and return that path."""
+    root = Path(target_dir)
+    for relative, content in suite.files.items():
+        path = root / relative
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text(content, encoding="utf-8")
+    return root
+
+
+def _ensure_import_root(path: Path) -> None:
+    text = str(path)
+    if text not in sys.path:
+        sys.path.insert(0, text)
+
+
+def _load_callable(fn: FunctionInfo):
+    module = importlib.import_module(fn.module)
+    obj = module
+    for part in fn.qualname.split("."):
+        obj = getattr(obj, part)
+    return obj
+
+
+def _values_for_parameter(name: str, hint: str) -> list[Any]:
+    hint = hint.replace("typing.", "")
+    lower_name = name.lower()
+    if hint in {"int"}:
+        if lower_name in {"qty", "quantity"}:
+            return [1, 10, 0, -1]
+        if lower_name in {"n", "limit", "length"}:
+            return [5, 3, 0, -1]
+        return [0, 1, -1, 2]
+    if hint in {"float"}:
+        if lower_name in {"pct", "rate"}:
+            return [10.0, 0.0, 100.0, -1.0]
+        return [100.0, 0.0, 12.5, -2.0]
+    if hint in {"str"}:
+        return ["Hello, World!", "  Already clean  ", "", "A/B test"]
+    if hint in {"bool"}:
+        return [True, False]
+    if hint in {"date", "datetime.date"}:
+        return [date(2026, 6, 15), date(2026, 6, 14), date(2024, 2, 29)]
+    if hint.startswith("list"):
+        if "tuple" in hint or lower_name == "lines":
+            return [[(2, 10.0), (1, 5.5)], [], [(0, 99.0)]]
+        return [[], [1], [1, 2, 3]]
+    return [1, "x", 0]
+
+
+def _literal(value: Any) -> str:
+    if isinstance(value, date):
+        return f"date({value.year}, {value.month}, {value.day})"
+    text = repr(value)
+    if len(text) > 120:
+        return reprlib.repr(value)
+    return text
+
+
+def _safe_name(name: str) -> str:
+    return "".join(ch if ch.isalnum() else "_" for ch in name).strip("_").lower()
+

inject.py

@@ -0,0 +1,45 @@
+"""One-click live regression demo."""
+
+from __future__ import annotations
+
+import shutil
+from dataclasses import dataclass
+from pathlib import Path
+
+from mutator import MUTATIONS, apply_single_mutation
+from runner import RunResult, run_pytest
+
+
+INJECT_MUTATION_ID = "bulk_threshold_ge_to_gt"
+
+
+@dataclass(frozen=True)
+class InjectResult:
+    mutation_label: str
+    run: RunResult
+
+
+def reset_sample(source_parent: str | Path, run_dir: str | Path) -> Path:
+    """Copy samples into a runnable demo directory."""
+    source_parent = Path(source_parent).resolve()
+    run_dir = Path(run_dir).resolve()
+    sample_target = run_dir / "samples"
+    if sample_target.exists():
+        shutil.rmtree(sample_target)
+    shutil.copytree(source_parent, sample_target)
+    return sample_target / "legacy_repo"
+
+
+def apply(run_dir: str | Path, package_name: str = "legacy_repo") -> bool:
+    package_root = Path(run_dir).resolve() / "samples" / package_name
+    return apply_single_mutation(package_root, INJECT_MUTATION_ID)
+
+
+def run_injected_suite(run_dir: str | Path, package_name: str = "legacy_repo") -> InjectResult:
+    mutation = next(item for item in MUTATIONS if item.id == INJECT_MUTATION_ID)
+    applied = apply(run_dir, package_name=package_name)
+    if not applied:
+        raise RuntimeError(f"Could not apply inject mutation: {mutation.label}")
+    run_dir = Path(run_dir).resolve()
+    result = run_pytest(run_dir, package_parent=run_dir / "samples", package_name=package_name)
+    return InjectResult(mutation_label=mutation.label, run=result)

mutator.py

@@ -0,0 +1,166 @@
+"""Fast hand-rolled mutation scoring for the bundled legacy sample."""
+
+from __future__ import annotations
+
+import shutil
+from dataclasses import dataclass
+from pathlib import Path
+
+from generator import GeneratedSuite, write_suite
+from runner import RunResult, run_pytest
+
+
+@dataclass(frozen=True)
+class Mutation:
+    id: str
+    file: str
+    find: str
+    replace: str
+    label: str
+
+
+@dataclass(frozen=True)
+class MutationResult:
+    mutation: Mutation
+    killed: bool
+    run: RunResult
+
+
+@dataclass(frozen=True)
+class MutationScore:
+    killed: int
+    total: int
+    results: tuple[MutationResult, ...]
+
+    @property
+    def percent(self) -> float:
+        return 0.0 if self.total == 0 else round(self.killed / self.total * 100, 1)
+
+    @property
+    def survived(self) -> list[Mutation]:
+        return [result.mutation for result in self.results if not result.killed]
+
+    def headline(self) -> str:
+        return f"{self.killed}/{self.total} behavior changes detected"
+
+
+MUTATIONS: tuple[Mutation, ...] = (
+    Mutation(
+        id="bulk_multiply_to_divide",
+        file="pricing.py",
+        find="return round(unit * qty * discount, 2)",
+        replace="return round(unit / qty * discount, 2)",
+        label="pricing.py bulk_price * -> /",
+    ),
+    Mutation(
+        id="discount_minus_to_plus",
+        file="pricing.py",
+        find="return round(price * (1 - pct / 100), 2)",
+        replace="return round(price * (1 + pct / 100), 2)",
+        label="pricing.py apply_discount - -> +",
+    ),
+    Mutation(
+        id="tax_plus_to_minus",
+        file="pricing.py",
+        find="return round(amount * (1 + rate / 100), 2)",
+        replace="return round(amount * (1 - rate / 100), 2)",
+        label="pricing.py with_tax + -> -",
+    ),
+    Mutation(
+        id="line_multiply_to_plus",
+        file="invoice.py",
+        find="return round(qty * unit, 2)",
+        replace="return round(qty + unit, 2)",
+        label="invoice.py line_total * -> +",
+    ),
+    Mutation(
+        id="weekend_ge_to_gt",
+        file="dates.py",
+        find="return d.weekday() >= 5",
+        replace="return d.weekday() > 5",
+        label="dates.py is_weekend >= -> >",
+    ),
+    Mutation(
+        id="days_abs_removed",
+        file="dates.py",
+        find="return abs((b - a).days)",
+        replace="return (b - a).days",
+        label="dates.py days_between remove abs",
+    ),
+    Mutation(
+        id="slug_strip_removed",
+        file="slugify.py",
+        find="return cleaned.strip(\"-\")",
+        replace="return cleaned",
+        label="slugify.py strip removed",
+    ),
+    Mutation(
+        id="truncate_len_lt",
+        file="slugify.py",
+        find="if len(s) <= n:",
+        replace="if len(s) < n:",
+        label="slugify.py <= -> <",
+    ),
+    Mutation(
+        id="bulk_threshold_ge_to_gt",
+        file="pricing.py",
+        find="discount = 0.9 if qty >= 10 else 1.0",
+        replace="discount = 0.9 if qty > 10 else 1.0",
+        label="pricing.py bulk_price >= -> >",
+    ),
+)
+
+
+def mutation_score(
+    source_parent: str | Path,
+    suite: GeneratedSuite,
+    tmp_root: str | Path,
+    package_name: str = "legacy_repo",
+    mutations: tuple[Mutation, ...] = MUTATIONS,
+) -> MutationScore:
+    """Apply each mutation to a copy and run the generated suite."""
+    source_parent = Path(source_parent).resolve()
+    tmp_root = Path(tmp_root).resolve()
+    tmp_root.mkdir(parents=True, exist_ok=True)
+    results: list[MutationResult] = []
+
+    for index, mutation in enumerate(mutations, start=1):
+        mutant_dir = tmp_root / f"mutant_{index}_{mutation.id}"
+        if mutant_dir.exists():
+            shutil.rmtree(mutant_dir)
+        shutil.copytree(source_parent, mutant_dir / "samples")
+        write_suite(suite, mutant_dir)
+        applied = _apply_mutation(mutant_dir / "samples" / package_name, mutation)
+        if not applied:
+            run = RunResult(
+                ok=False,
+                passed=0,
+                failed=0,
+                errors=1,
+                coverage=None,
+                stdout="",
+                stderr=f"Mutation not applied: {mutation.label}",
+                returncode=2,
+            )
+            results.append(MutationResult(mutation=mutation, killed=False, run=run))
+            continue
+        run = run_pytest(mutant_dir, package_parent=mutant_dir / "samples", package_name=package_name)
+        results.append(MutationResult(mutation=mutation, killed=not run.ok, run=run))
+
+    killed = sum(1 for result in results if result.killed)
+    return MutationScore(killed=killed, total=len(results), results=tuple(results))
+
+
+def apply_single_mutation(package_root: str | Path, mutation_id: str) -> bool:
+    """Apply one catalog mutation in-place for the live Inject button."""
+    mutation = next(item for item in MUTATIONS if item.id == mutation_id)
+    return _apply_mutation(Path(package_root), mutation)
+
+
+def _apply_mutation(package_root: Path, mutation: Mutation) -> bool:
+    path = package_root / mutation.file
+    text = path.read_text(encoding="utf-8")
+    if mutation.find not in text:
+        return False
+    path.write_text(text.replace(mutation.find, mutation.replace, 1), encoding="utf-8")
+    return True

patch.py

@@ -0,0 +1,45 @@
+"""PR-style patch export for generated tests."""
+
+from __future__ import annotations
+
+import difflib
+from pathlib import Path
+
+from generator import GeneratedSuite
+
+
+def make_pr_patch(suite: GeneratedSuite) -> str:
+    """Return a unified diff that adds the generated tests and a CI hint."""
+    sections: list[str] = []
+    for relative, content in sorted(suite.files.items()):
+        new_lines = content.splitlines(keepends=True)
+        diff = difflib.unified_diff(
+            [],
+            new_lines,
+            fromfile="/dev/null",
+            tofile=f"b/{relative}",
+            lineterm="",
+        )
+        sections.append("\n".join(diff))
+
+    ci_hint = "pytest tests -q\n"
+    sections.append(
+        "\n".join(
+            difflib.unified_diff(
+                [],
+                [ci_hint],
+                fromfile="/dev/null",
+                tofile="b/TESTFORGE_CI_HINT.txt",
+                lineterm="",
+            )
+        )
+    )
+    return "\n\n".join(sections) + "\n"
+
+
+def write_patch(suite: GeneratedSuite, path: str | Path) -> Path:
+    output = Path(path)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    output.write_text(make_pr_patch(suite), encoding="utf-8")
+    return output
+

pytest.ini

@@ -0,0 +1,3 @@
+[pytest]
+addopts = --basetemp=.pytest-tmp
+testpaths = tests

requirements.txt

@@ -0,0 +1,4 @@
+gradio>=4.44
+pytest>=8.0
+pytest-cov>=5.0
+

runner.py

@@ -0,0 +1,122 @@
+"""Subprocess pytest and coverage runner."""
+
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+from dataclasses import dataclass
+from importlib.util import find_spec
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class RunResult:
+    ok: bool
+    passed: int
+    failed: int
+    errors: int
+    coverage: float | None
+    stdout: str
+    stderr: str
+    returncode: int
+
+    @property
+    def summary(self) -> str:
+        parts = []
+        if self.passed:
+            parts.append(f"{self.passed} passed")
+        if self.failed:
+            parts.append(f"{self.failed} failed")
+        if self.errors:
+            parts.append(f"{self.errors} errors")
+        return ", ".join(parts) or "no tests"
+
+
+def run_pytest(
+    workdir: str | Path,
+    package_parent: str | Path | None = None,
+    timeout: int = 30,
+    with_coverage: bool = True,
+    package_name: str = "legacy_repo",
+) -> RunResult:
+    """Run pytest in a subprocess and parse pass/fail/error counts."""
+    cwd = Path(workdir).resolve()
+    command = [sys.executable, "-m", "pytest", "tests", "-q"]
+    coverage_available = find_spec("pytest_cov") is not None
+    if with_coverage and coverage_available:
+        command.extend([f"--cov={package_name}", "--cov-report=term-missing"])
+
+    env = None
+    if package_parent is not None:
+        env = dict(__import__("os").environ)
+        parent = str(Path(package_parent).resolve())
+        current = env.get("PYTHONPATH", "")
+        env["PYTHONPATH"] = parent if not current else f"{parent}{__import__('os').pathsep}{current}"
+
+    try:
+        completed = subprocess.run(
+            command,
+            cwd=cwd,
+            env=env,
+            text=True,
+            capture_output=True,
+            timeout=timeout,
+            check=False,
+        )
+    except subprocess.TimeoutExpired as exc:
+        return RunResult(
+            ok=False,
+            passed=0,
+            failed=0,
+            errors=1,
+            coverage=None,
+            stdout=exc.stdout or "",
+            stderr=(exc.stderr or "") + f"\nTimed out after {timeout}s",
+            returncode=124,
+        )
+
+    stdout = completed.stdout
+    stderr = completed.stderr
+    passed, failed, errors = _parse_counts(stdout + "\n" + stderr)
+    coverage = _parse_coverage(stdout)
+    ok = completed.returncode == 0 and failed == 0 and errors == 0
+    return RunResult(
+        ok=ok,
+        passed=passed,
+        failed=failed,
+        errors=errors,
+        coverage=coverage,
+        stdout=stdout,
+        stderr=stderr,
+        returncode=completed.returncode,
+    )
+
+
+def _parse_counts(output: str) -> tuple[int, int, int]:
+    passed = failed = errors = 0
+    for pattern, name in [
+        (r"(\d+)\s+passed", "passed"),
+        (r"(\d+)\s+failed", "failed"),
+        (r"(\d+)\s+errors?", "errors"),
+        (r"(\d+)\s+error", "errors"),
+    ]:
+        matches = re.findall(pattern, output)
+        if not matches:
+            continue
+        value = int(matches[-1])
+        if name == "passed":
+            passed = value
+        elif name == "failed":
+            failed = value
+        else:
+            errors = value
+    return passed, failed, errors
+
+
+def _parse_coverage(output: str) -> float | None:
+    total_lines = [line for line in output.splitlines() if line.strip().startswith("TOTAL")]
+    if not total_lines:
+        return None
+    match = re.search(r"(\d+(?:\.\d+)?)%", total_lines[-1])
+    return float(match.group(1)) if match else None

samples/legacy_repo/__init__.py

@@ -0,0 +1,2 @@
+"""Small untested legacy package used by TestForge demos."""
+

samples/legacy_repo/dates.py

@@ -0,0 +1,14 @@
+"""Date helpers from a legacy scheduling module."""
+
+from datetime import date
+
+
+def days_between(a: date, b: date) -> int:
+    """Return the absolute number of days between two dates."""
+    return abs((b - a).days)
+
+
+def is_weekend(d: date) -> bool:
+    """Return True when a date falls on Saturday or Sunday."""
+    return d.weekday() >= 5
+

samples/legacy_repo/invoice.py

@@ -0,0 +1,17 @@
+"""Invoice helpers with deliberately ordinary business logic."""
+
+
+def line_total(qty: int, unit: float) -> float:
+    """Return total for one invoice line."""
+    if qty < 0:
+        raise ValueError("quantity cannot be negative")
+    return round(qty * unit, 2)
+
+
+def invoice_total(lines: list[tuple[int, float]]) -> float:
+    """Return total for a list of ``(quantity, unit_price)`` pairs."""
+    total = 0.0
+    for qty, unit in lines:
+        total += line_total(qty, unit)
+    return round(total, 2)
+

samples/legacy_repo/pricing.py

@@ -0,0 +1,26 @@
+"""Pricing helpers from a pretend legacy billing module."""
+
+
+def apply_discount(price: float, pct: float) -> float:
+    """Return price after applying a percentage discount."""
+    if pct < 0:
+        raise ValueError("discount cannot be negative")
+    if pct > 100:
+        raise ValueError("discount cannot exceed 100")
+    return round(price * (1 - pct / 100), 2)
+
+
+def with_tax(amount: float, rate: float) -> float:
+    """Return amount with tax added."""
+    if rate < 0:
+        raise ValueError("tax rate cannot be negative")
+    return round(amount * (1 + rate / 100), 2)
+
+
+def bulk_price(unit: float, qty: int) -> float:
+    """Return total price after a simple bulk discount."""
+    if qty < 0:
+        raise ValueError("quantity cannot be negative")
+    discount = 0.9 if qty >= 10 else 1.0
+    return round(unit * qty * discount, 2)
+

samples/legacy_repo/slugify.py

@@ -0,0 +1,21 @@
+"""String cleanup helpers used in old import jobs."""
+
+import re
+
+
+def slugify(s: str) -> str:
+    """Convert text into a lower-case dash-separated slug."""
+    cleaned = re.sub(r"[^a-zA-Z0-9]+", "-", s.strip().lower())
+    return cleaned.strip("-")
+
+
+def truncate(s: str, n: int) -> str:
+    """Return text clipped to at most n characters."""
+    if n < 0:
+        raise ValueError("length cannot be negative")
+    if len(s) <= n:
+        return s
+    if n <= 3:
+        return s[:n]
+    return s[: n - 3] + "..."
+

tests/test_testforge.py

@@ -0,0 +1,136 @@
+from pathlib import Path
+import shutil
+import sys
+
+from analyzer import analyze
+from agent import forge_legacy_repo
+from generator import capture, generate_suite, write_suite
+from inject import reset_sample, run_injected_suite
+from mutator import MUTATIONS, mutation_score
+from patch import make_pr_patch
+from runner import run_pytest
+from samples.legacy_repo.pricing import apply_discount, with_tax
+
+
+ROOT = Path(__file__).resolve().parents[1]
+LEGACY_ROOT = ROOT / "samples" / "legacy_repo"
+
+
+def test_analyzer_discovers_expected_public_functions():
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    discovered = {(fn.module, fn.qualname, fn.arity) for fn in analysis.functions}
+
+    assert discovered == {
+        ("legacy_repo.dates", "days_between", 2),
+        ("legacy_repo.dates", "is_weekend", 1),
+        ("legacy_repo.invoice", "invoice_total", 1),
+        ("legacy_repo.invoice", "line_total", 2),
+        ("legacy_repo.pricing", "apply_discount", 2),
+        ("legacy_repo.pricing", "bulk_price", 2),
+        ("legacy_repo.pricing", "with_tax", 2),
+        ("legacy_repo.slugify", "slugify", 1),
+        ("legacy_repo.slugify", "truncate", 2),
+    }
+
+
+def test_analyzer_keeps_hints_docstrings_and_source():
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    fn = next(item for item in analysis.functions if item.qualname == "apply_discount")
+
+    assert fn.type_hints == {"price": "float", "pct": "float"}
+    assert fn.return_hint == "float"
+    assert "percentage discount" in fn.docstring
+    assert "def apply_discount" in fn.source
+
+
+def test_capture_records_return_and_exception():
+    sys.path.insert(0, str((ROOT / "samples").resolve()))
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    discount = next(item for item in analysis.functions if item.qualname == "apply_discount")
+    tax = next(item for item in analysis.functions if item.qualname == "with_tax")
+
+    assert apply_discount(100, 10) == 90.0
+    returned = capture(discount, [(100.0, 10.0)])
+    raised = capture(tax, [(100.0, -1.0)])
+
+    assert returned[0].value_repr == "90.0"
+    assert raised[0].exception_type == "ValueError"
+
+
+def test_deterministic_generated_suite_is_green(tmp_path):
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    suite = generate_suite(analysis)
+    workdir = tmp_path / "work"
+    shutil.copytree(ROOT / "samples", workdir / "samples")
+    write_suite(suite, workdir)
+
+    result = run_pytest(workdir, package_parent=workdir / "samples")
+
+    assert result.ok, result.stdout + result.stderr
+    assert result.passed == suite.assertion_count
+    assert result.coverage is None or result.coverage >= 0
+
+
+def test_runner_parses_known_good_and_bad(tmp_path):
+    good = tmp_path / "good"
+    good.mkdir()
+    (good / "tests").mkdir()
+    (good / "tests" / "test_ok.py").write_text("def test_ok():\n    assert 1 == 1\n", encoding="utf-8")
+    good_result = run_pytest(good, with_coverage=False)
+
+    bad = tmp_path / "bad"
+    bad.mkdir()
+    (bad / "tests").mkdir()
+    (bad / "tests" / "test_bad.py").write_text("def test_bad():\n    assert 1 == 2\n", encoding="utf-8")
+    bad_result = run_pytest(bad, with_coverage=False)
+
+    assert good_result.ok
+    assert good_result.passed == 1
+    assert not bad_result.ok
+    assert bad_result.failed == 1
+
+
+def test_mutation_score_is_deterministic_and_kills_known_mutant(tmp_path):
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    suite = generate_suite(analysis)
+    score = mutation_score(ROOT / "samples", suite, tmp_path / "mutants")
+
+    assert score.total == len(MUTATIONS)
+    assert score.killed > 0
+    known = next(
+        result for result in score.results if result.mutation.id == "bulk_multiply_to_divide"
+    )
+    assert known.killed
+
+
+def test_patch_export_contains_generated_tests():
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    suite = generate_suite(analysis)
+    patch = make_pr_patch(suite)
+
+    assert "b/tests/test_legacy_repo_pricing_apply_discount.py" in patch
+    assert "pytest tests -q" in patch
+
+
+def test_forge_pipeline_produces_green_suite_and_patch():
+    one_mutant = (next(item for item in MUTATIONS if item.id == "bulk_threshold_ge_to_gt"),)
+    artifacts = forge_legacy_repo(max_cases_per_function=2, mutations=one_mutant)
+
+    assert artifacts.green.ok, artifacts.green.stdout + artifacts.green.stderr
+    assert artifacts.suite.assertion_count > 0
+    assert artifacts.mutation.killed > 0
+    assert artifacts.patch_path.exists()
+
+
+def test_inject_regression_causes_exactly_one_failure(tmp_path):
+    analysis = analyze(LEGACY_ROOT, package="legacy_repo")
+    suite = generate_suite(analysis)
+    run_dir = tmp_path / "inject"
+    run_dir.mkdir()
+    reset_sample(ROOT / "samples", run_dir)
+    write_suite(suite, run_dir)
+
+    result = run_injected_suite(run_dir)
+
+    assert not result.run.ok
+    assert result.run.failed == 1