synth/README.md

Synthetic filesystem-task generator

Mass-produce MCPMark filesystem benchmark tasks (test environment + description.md + verify.py + meta.json) with deterministic, self-checked verifiers. An LLM (DeepSeek via .mcp_env, by default) only makes file content and names realistic; it never writes the verification logic.

Why it is reliable

Two verifier strategies, both checked by a built-in oracle at generation time:

• Recomputable — verify.py re-derives the correct answer from the resulting files (e.g. recompute the 10 smallest files, regroup by content hash). No stored answer key, so the model cannot shortcut it.
• Planted ground truth — for computation/semantic tasks the generator controls the data and labels it (e.g. category: personal|work, song rating/year), so the answer is exactly computable without LLM judgement.

Every task is validated before it is written: the oracle solves it and verify.py must exit 0 on the correct answer and non-zero on the untouched environment. Tasks that fail this self-check are discarded.

Usage

# from repo root, inside the mcpmark conda env
conda run -n mcpmark python -m synth.generate --n 13 --seed 1      # LLM content
conda run -n mcpmark python -m synth.generate --n 13 --no-llm      # offline
conda run -n mcpmark python -m synth.generate --n 4 --types music_report,budget_computation

Each task gets a unique category_id like synth_music_report_01. Run them with the normal pipeline (synth is a substring filter that matches them all):

conda run -n mcpmark python -m pipeline --mcp filesystem --k 1 \
  --models deepseek-v3.2-instruct --tasks synth --exp-name synth-run

Coverage (8 benchmark categories)

Generator key	Benchmark category	Verifier
size_classification	file_property	recomputable
extension_grouping	file_property (variant)	recomputable
smallest_merge	file_context	recomputable
duplicate_finder	file_context	recomputable
uppercase	file_context	planted
pattern_matching	file_context	recomputable
file_splitting	file_context	recomputable
structure_mirror	folder_structure	recomputable
author_folders	papers	recomputable
gradebased_score	student_database	recomputable
music_report	desktop	planted
budget_computation	desktop_template	planted
clause_lookup	legal_document	planted

Not covered: threestudio / votenet — these run on real 3D ML codebases and cannot be faithfully synthesized (only loose structural analogs would be possible).

Each subtask folder has its own README.md with task details, a worked example, and a sample trajectory — see e.g. generators/filesystem/duplicate_finder/README.md. Note these tasks reorganize files (move/group); they never delete or rewrite file contents, and any output folder (e.g. duplicates/) is created by the evaluated model, not the generator.

Inspect a trajectory

Every pipeline run saves the agent trajectory to results/<exp>/<model>__filesystem/run-<k>/<task>/messages.json. Render it as a readable timeline (👤 instruction, 💬 thoughts, 🔧 tool calls, 📤 results):

conda run -n mcpmark python -m synth.trace results/<exp> --list           # list all
conda run -n mcpmark python -m synth.trace results/<exp>/<model>__filesystem/run-1/<task>

Add a new task type

Create generators/filesystem/<your_subtask>/__init__.py with a Generator subclass, then add it to the REGISTRY tuple in generators/filesystem/__init__.py. Implement four methods:

• build(env_dir, llm, rng) -> spec — write the initial files, return a spec dict.
• description(spec) -> str — the instructions the model sees. Be precise: the verifier is exact, so any output-format ambiguity will fail otherwise-correct work.
• verify_src(spec) -> str — a self-contained verify.py (stdlib only). Read the test dir from os.environ["FILESYSTEM_TEST_DIR"]; sys.exit(0) on pass, non-zero on fail. Use _render_verify(body, consts) to inject constants safely.
• solve(work_dir, spec) — the oracle: perform the correct solution in place. This both powers the self-check and forces you to prove the task is solvable.

Files

synth/
├── generate.py                  # CLI: build tasks + oracle self-check
├── trace.py                     # CLI: render a saved trajectory
├── llm.py                       # LiteLLM wrapper (DeepSeek default) + offline fallback
└── generators/                  # organized by MCP service → subtask
    ├── __init__.py              #   merges per-service REGISTRYs
    └── filesystem/              #   the filesystem service
        ├── __init__.py          #     filesystem REGISTRY
        ├── base.py              #     Generator base class + shared helpers
        ├── duplicate_finder/__init__.py    # one folder per subtask
        ├── smallest_merge/__init__.py
        ├── size_classification/__init__.py
        ├── extension_grouping/__init__.py
        ├── uppercase/__init__.py
        ├── pattern_matching/__init__.py
        ├── file_splitting/__init__.py
        ├── structure_mirror/__init__.py
        ├── author_folders/__init__.py
        ├── gradebased_score/__init__.py
        ├── music_report/__init__.py
        ├── budget_computation/__init__.py
        └── clause_lookup/__init__.py

Each subtask folder holds its Generator (and is the natural place to later add per-task templates/fixtures). Generated tasks land in the project's existing tasks/ and test_environments/; trajectories land in results/.

To add a new MCP service, create generators/<service>/ with its own base.py and subtask folders, then merge its REGISTRY in generators/__init__.py.