synth/README.md

# Synthetic filesystem-task generator

Mass-produce MCPMark **filesystem** benchmark tasks (test environment +
`description.md` + `verify.py` + `meta.json`) with deterministic, *self-checked*
verifiers. An LLM (DeepSeek via `.mcp_env`, by default) only makes file content
and names realistic; it never writes the verification logic.

## Why it is reliable

Two verifier strategies, both checked by a built-in oracle at generation time:

- **Recomputable** — `verify.py` re-derives the correct answer from the resulting
  files (e.g. recompute the 10 smallest files, regroup by content hash). No stored
  answer key, so the model cannot shortcut it.
- **Planted ground truth** — for computation/semantic tasks the generator controls
  the data and labels it (e.g. `category: personal|work`, song `rating`/`year`),
  so the answer is exactly computable without LLM judgement.

Every task is validated before it is written: the oracle solves it and `verify.py`
**must** exit 0 on the correct answer **and** non-zero on the untouched
environment. Tasks that fail this self-check are discarded.

## Usage

```bash
# from repo root, inside the mcpmark conda env
conda run -n mcpmark python -m synth.generate --n 13 --seed 1      # LLM content
conda run -n mcpmark python -m synth.generate --n 13 --no-llm      # offline
conda run -n mcpmark python -m synth.generate --n 4 --types music_report,budget_computation
```

Each task gets a unique `category_id` like `synth_music_report_01`. Run them with
the normal pipeline (`synth` is a substring filter that matches them all):

```bash
conda run -n mcpmark python -m pipeline --mcp filesystem --k 1 \
  --models deepseek-v3.2-instruct --tasks synth --exp-name synth-run
```

## Coverage (8 benchmark categories)

| Generator key        | Benchmark category        | Verifier      |
|----------------------|---------------------------|---------------|
| `size_classification`| file_property             | recomputable  |
| `extension_grouping` | file_property (variant)   | recomputable  |
| `smallest_merge`     | file_context              | recomputable  |
| `duplicate_finder`   | file_context              | recomputable  |
| `uppercase`          | file_context              | planted       |
| `pattern_matching`   | file_context              | recomputable  |
| `file_splitting`     | file_context              | recomputable  |
| `structure_mirror`   | folder_structure          | recomputable  |
| `author_folders`     | papers                    | recomputable  |
| `gradebased_score`   | student_database          | recomputable  |
| `music_report`       | desktop                   | planted       |
| `budget_computation` | desktop_template          | planted       |
| `clause_lookup`      | legal_document            | planted       |

Not covered: `threestudio` / `votenet` — these run on real 3D ML codebases and
cannot be faithfully synthesized (only loose structural analogs would be possible).

Each subtask folder has its own `README.md` with task details, a worked example,
and a sample trajectory — see e.g.
[`generators/filesystem/duplicate_finder/README.md`](generators/filesystem/duplicate_finder/README.md).
Note these tasks **reorganize files** (move/group); they never delete or rewrite
file contents, and any output folder (e.g. `duplicates/`) is created by the
evaluated model, not the generator.

## Inspect a trajectory

Every pipeline run saves the agent trajectory to
`results/<exp>/<model>__filesystem/run-<k>/<task>/messages.json`. Render it as a
readable timeline (👤 instruction, 💬 thoughts, 🔧 tool calls, 📤 results):

```bash
conda run -n mcpmark python -m synth.trace results/<exp> --list           # list all
conda run -n mcpmark python -m synth.trace results/<exp>/<model>__filesystem/run-1/<task>
```

## Add a new task type

Create `generators/filesystem/<your_subtask>/__init__.py` with a `Generator`
subclass, then add it to the `REGISTRY` tuple in `generators/filesystem/__init__.py`.
Implement four methods:

- `build(env_dir, llm, rng) -> spec` — write the initial files, return a spec dict.
- `description(spec) -> str` — the instructions the model sees. **Be precise**: the
  verifier is exact, so any output-format ambiguity will fail otherwise-correct work.
- `verify_src(spec) -> str` — a self-contained `verify.py` (stdlib only). Read the
  test dir from `os.environ["FILESYSTEM_TEST_DIR"]`; `sys.exit(0)` on pass, non-zero
  on fail. Use `_render_verify(body, consts)` to inject constants safely.
- `solve(work_dir, spec)` — the oracle: perform the correct solution in place. This
  both powers the self-check and forces you to prove the task is solvable.

## Files

```
synth/
├── generate.py                  # CLI: build tasks + oracle self-check
├── trace.py                     # CLI: render a saved trajectory
├── llm.py                       # LiteLLM wrapper (DeepSeek default) + offline fallback
└── generators/                  # organized by MCP service → subtask
    ├── __init__.py              #   merges per-service REGISTRYs
    └── filesystem/              #   the filesystem service
        ├── __init__.py          #     filesystem REGISTRY
        ├── base.py              #     Generator base class + shared helpers
        ├── duplicate_finder/__init__.py    # one folder per subtask
        ├── smallest_merge/__init__.py
        ├── size_classification/__init__.py
        ├── extension_grouping/__init__.py
        ├── uppercase/__init__.py
        ├── pattern_matching/__init__.py
        ├── file_splitting/__init__.py
        ├── structure_mirror/__init__.py
        ├── author_folders/__init__.py
        ├── gradebased_score/__init__.py
        ├── music_report/__init__.py
        ├── budget_computation/__init__.py
        └── clause_lookup/__init__.py
```

Each subtask folder holds its `Generator` (and is the natural place to later add
per-task templates/fixtures). Generated tasks land in the project's existing
`tasks/` and `test_environments/`; trajectories land in `results/`.

To add a new MCP service, create `generators/<service>/` with its own `base.py`
and subtask folders, then merge its `REGISTRY` in `generators/__init__.py`.