Document parameterized replay and add Gmail/YouTube bindings.

Update README and BLOG for the validated classify→bind→replay flow, and extend skill schemas plus ParameterBinder for Gmail and YouTube preview.

Co-authored-by: Cursor <cursoragent@cursor.com>

BLOG.md

@@ -18,7 +18,7 @@
 6. [Deployment and demo](#step-4-deploy-inference-on-modal-demo-on-gradio)
 7. [Evaluation and benchmarks](#evaluation-how-we-measure-generalization)
 8. [Why this approach works](#why-this-approach-works)
-9. [What's next](#whats-next-trajectory-parameterization)
+9. [Parameterized replay](#parameterized-replay-classify--bind--replay)
 10. [Try it yourself](#try-it-yourself)
 
 ---
@@ -624,9 +624,9 @@ This project targets **personal automation on hardware you own** — the Backyar
 
 ---
 
-## What's next: trajectory parameterization
+## Parameterized replay: classify → bind → replay
 
-### The current gap
+### The gap V2 closed
 
 V2 extracts parameters at inference time:
 
@@ -635,15 +635,15 @@ V2 extracts parameters at inference time:
 → {"contact": "mom", "message": "i'm on my way"}
 ```
 
-But trajectories are still recorded with **fixed entities** — the WhatsApp trajectory says "message hi to biraj" and the `set_text` actions contain `"hi"` and `"biraj"`. Replay uses those literal values, not the extracted parameters.
+Recorded trajectories still contain **fixed entities** — the WhatsApp export types `"Biraj"` and `"Hi"`. Without binding, replay ignores the model output.
 
-### The planned solution
+### Slot-filling at replay time
 
-**Slot-filling at replay time**: when the model returns `{"contact": "mom", "message": "i'm on my way"}`, the replay engine:
+**ParameterBinder** substitutes runtime values into trajectory steps before replay:
 
-1. Identifies parameterizable steps in the trajectory (text input actions)
-2. Substitutes extracted values into `set_text` actions
-3. Uses smart node resolution to find the contact field, search box, etc.
+1. Load bindings from `data/skill_schemas.json` (which step maps to which parameter)
+2. Rewrite `set_text` values and post-search `click` labels
+3. Hand the bound trajectory to `ReplayPlanner` → `ReplayEngine`
 
 This closes the loop:
 
@@ -651,14 +651,26 @@ This closes the loop:
 Natural language → structured intent → parameterized replay on any device
 ```
 
-The trajectory becomes a **template** rather than a fixed recording. Record once with placeholder entities, replay with any contact, message, time, or destination.
+**Validated end-to-end flow (WhatsApp on device):**
 
-### Other future work
+```
+Modal /predict (or pasted JSON)
+  → parameter dialog in Pocket Automator
+  → ParameterBinder.apply(trajectory, parameters, bindings)
+  → ReplayPlanner.plan → ReplayEngine.replay
+  → WhatsApp taps with the extracted contact and message
+```
+
+The Gradio Space runs the same binding logic in Python (`src/parameter_binder.py`) so the trajectory JSON preview matches what replay will execute.
+
+Bindings are defined per skill. **WhatsApp**, **Gmail**, and **YouTube** are supported in preview; Pocket Automator mirrors the schema on device.
+
+### What's next
 
+- **Self-contained exports** — embed `bindings` + `recordedParameters` in exported trajectory JSON (Phase B.8)
+- **More skills** — Contacts, Calendar, Spotify search, etc.
 - **On-device inference** — run the 3B model locally without Modal
-- **More skills** — maps, photos, settings toggles, banking apps
 - **Multi-step intents** — "set alarm and text mom I'll be late"
-- **Confidence calibration** — know when to ask the user for clarification
 - **UI change detection** — alert when a trajectory needs re-recording
 
 ---
@@ -702,13 +714,14 @@ python app.py
 ```
 app.py                      # Gradio demo (hackathon submission UI)
 data/
-  skill_schemas.json        # Parameter definitions per skill
+  skill_schemas.json        # Parameter definitions and trajectory bindings per skill
   skills.jsonl              # Canonical skill ↔ task mapping
   train_intent.jsonl        # ~15k SFT examples (generated locally)
   eval_intent_prompts.json  # Held-out intent eval set
   pocket_benchmark_prompts.json  # 200 real-world messy prompts
 src/
   skill_router.py           # Skill name → trajectory JSON
+  parameter_binder.py       # Runtime parameter → trajectory step substitution
   skill_utils.py              # JSON parsing, aliases, fallbacks
   classifier_prompt.py        # System prompts for V1 and V2
   evaluate_intent.py          # Local evaluation

README.md

@@ -29,7 +29,11 @@ You say *"text mom on whatsapp i'm on my way"* — a voice assistant might web-s
 "play my workout playlist"  →  spotify_play_playlist  →  trajectories/spotify_play_playlist.json
 ```
 
-**Tech:** fine-tuned [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) via 4-bit QLoRA + SFT ([Unsloth](https://github.com/unslothai/unsloth) on Modal) → skill router → Pocket Automator trajectory JSON → replay on device. Fifteen real Android flows expand to ~15k synthetic intent examples for training; inference runs on Modal, demo UI on Gradio.
+**Tech:** fine-tuned [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) via 4-bit QLoRA + SFT ([Unsloth](https://github.com/unslothai/unsloth) on Modal) → skill router → parameterized trajectory → Pocket Automator replay on device. Fifteen real Android flows expand to ~15k synthetic intent examples for training; inference runs on Modal, demo UI on Gradio.
+
+```
+Modal /predict (or pasted JSON) → parameter dialog → ParameterBinder → replay → device taps
+```
 
 **Submission links**
 
@@ -59,8 +63,9 @@ UI traces in `trajectories/` were captured with **[Pocket Automator](https://git
 | **Base model** | [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) |
 | **Fine-tune** | 4-bit QLoRA + SFT with [Unsloth](https://github.com/unslothai/unsloth) on Modal (`modal_apps/train_modal.py`) |
 | **Inference** | Modal GPU API (`modal_apps/predict_api.py`) — returns skill + parameters |
-| **Demo UI** | Gradio (`app.py`) |
-| **Recorder** | [Pocket Automator](https://github.com/kriyanshii/pocket-automator) — Android accessibility capture & replay |
+| **Parameter binding** | `src/parameter_binder.py` + `data/skill_schemas.json` bindings — substitutes runtime values into trajectory steps |
+| **Demo UI** | Gradio (`app.py`) — shows parameterized trajectory preview |
+| **Recorder / replay** | [Pocket Automator](https://github.com/kriyanshii/pocket-automator) — accessibility capture, parameter dialog, `ParameterBinder`, replay |
 | **Data** | 15 Android trajectories → `data/skills.jsonl` → ~510 prompt variations in `data/train.jsonl` |
 
 ## Quick start (local dev)
@@ -106,6 +111,7 @@ data/
   skills.jsonl              # Canonical skill ↔ task mapping
 src/
   skill_router.py           # Skill name → trajectory JSON
+  parameter_binder.py       # Runtime parameter → trajectory step substitution
   skill_utils.py            # Shared JSON parsing helpers
   evaluate.py               # Local CPU/MPS evaluation
 modal_apps/                 # Modal training + inference (not named "modal" — avoids import clash)
@@ -149,8 +155,27 @@ V1 maps prompts to a skill label only. V2 extracts structured intents:
 
 The Gradio demo and Modal `/predict` API both return skill + parameters.
 
+### Parameterized replay
+
+V2 extracts `{skill, parameters}` at inference time. **Slot-filling at replay** substitutes those values into recorded `set_text` / post-search click steps before replay.
+
+**End-to-end flow (validated on WhatsApp):**
+
+```
+"text mom on whatsapp i'm on my way"
+  → {"skill": "whatsapp_send_message", "parameters": {"contact": "mom", "message": "i'm on my way"}}
+  → ParameterBinder (Gradio preview + Pocket Automator on device)
+  → replay with "mom" / "i'm on my way", not the recorded "Biraj" / "Hi"
+```
+
+Bindings live in `data/skill_schemas.json` per skill. Supported in preview today: **WhatsApp**, **Gmail**, **YouTube**. Pocket Automator mirrors the same binding rules at replay time via its parameter dialog.
+
+```bash
+python -m src.parameter_binder   # self-test bindings
+```
+
 ### Data
-- `data/skill_schemas.json` — parameter definitions per skill
+- `data/skill_schemas.json` — parameter definitions and trajectory bindings per skill
 - `data/train_intent.jsonl` — ~15k synthetic SFT examples (generated locally via script; gitignored — upload to Modal for training)
 - `data/eval_intent_prompts.json` — held-out intent eval set
 - `data/pocket_benchmark_prompts.json` — 200 real-world messy prompts
@@ -171,7 +196,7 @@ modal run modal_apps/evaluate_pocket_benchmark_modal.py
 | Parameter accuracy | 86.0% |
 | Exact JSON match | 85.5% |
 
-Trajectory parameterization (slot-filling at replay time) is planned next.
+**Next:** self-contained trajectory exports (bindings embedded in export JSON) and bindings for remaining skills.
 
 ## License
 

data/skill_schemas.json

@@ -143,7 +143,21 @@
         "required": true,
         "description": "YouTube search query"
       }
-    }
+    },
+    "bindings": [
+      {
+        "parameter": "query",
+        "action": "set_text",
+        "resource_id_suffix": "search_edit_text",
+        "package_contains": "youtube"
+      },
+      {
+        "parameter": "query",
+        "action": "click",
+        "after_search": true,
+        "set_content_description": false
+      }
+    ]
   },
   "contacts_search": {
     "description": "Search for a contact in the phone contacts app",
@@ -168,6 +182,19 @@
         "required": true,
         "description": "Email body text"
       }
-    }
+    },
+    "bindings": [
+      {
+        "parameter": "recipient",
+        "action": "set_text",
+        "compose_recipient": true
+      },
+      {
+        "parameter": "message",
+        "action": "set_text",
+        "resource_id_suffix": "editor",
+        "package_contains": "gm"
+      }
+    ]
   }
 }

src/parameter_binder.py

@@ -1,7 +1,7 @@
 """
 Substitute runtime intent parameters into recorded trajectory steps.
 
-Preview-only for the Gradio demo today; Pocket Automator will mirror this at replay time.
+Used by the Gradio preview and mirrored in Pocket Automator's ParameterBinder at replay time.
 """
 
 from __future__ import annotations
@@ -34,6 +34,22 @@ _COMPOSE_BODY_SUFFIXES = (
     "body",
 )
 
+_COMPOSE_EXCLUDED_SUFFIXES = (
+    "subject",
+    "editor",
+)
+
+
+def _is_compose_recipient_step(step: dict[str, Any]) -> bool:
+    action = step.get("action") or {}
+    if action.get("type") != "set_text":
+        return False
+    package_name = (step.get("packageName") or "").lower()
+    if "gm" not in package_name and "gmail" not in package_name:
+        return False
+    resource_id = action.get("resourceId") or ""
+    return not any(_suffix_matches(resource_id, suffix) for suffix in _COMPOSE_EXCLUDED_SUFFIXES)
+
 
 def load_skill_schemas() -> dict[str, Any]:
     return json.loads(SKILL_SCHEMAS_PATH.read_text(encoding="utf-8"))
@@ -107,6 +123,9 @@ def _binding_matches(
     if package_contains and package_contains.lower() not in (step.get("packageName") or "").lower():
         return False
 
+    if binding.get("compose_recipient"):
+        return _is_compose_recipient_step(step)
+
     return True
 
 
@@ -158,9 +177,9 @@ def apply_parameters(
 def _self_test() -> None:
     from src.skill_router import load_trajectory
 
-    trajectory = load_trajectory("whatsapp_send_message")
+    whatsapp = load_trajectory("whatsapp_send_message")
     bound = apply_parameters(
-        trajectory,
+        whatsapp,
         "whatsapp_send_message",
         {"contact": "mom", "message": "i'm on my way"},
     )
@@ -189,6 +208,40 @@ def _self_test() -> None:
     ]
     assert post_search_clicks and all(click.get("text") == "mom" for click in post_search_clicks)
 
+    gmail = load_trajectory("gmail_send_email")
+    bound_gmail = apply_parameters(
+        gmail,
+        "gmail_send_email",
+        {"recipient": "alice@example.com", "message": "running late"},
+    )
+    recipient_values = [
+        step["action"]["value"]
+        for step in bound_gmail["steps"]
+        if _is_compose_recipient_step(step)
+    ]
+    body_values = [
+        step["action"]["value"]
+        for step in bound_gmail["steps"]
+        if step["action"]["type"] == "set_text"
+        and _suffix_matches(step["action"].get("resourceId"), "editor")
+    ]
+    assert recipient_values and all(value == "alice@example.com" for value in recipient_values), recipient_values
+    assert body_values and all(value == "running late" for value in body_values), body_values
+
+    youtube = load_trajectory("youtube_search")
+    bound_youtube = apply_parameters(
+        youtube,
+        "youtube_search",
+        {"query": "pasta recipes"},
+    )
+    youtube_search_values = [
+        step["action"]["value"]
+        for step in bound_youtube["steps"]
+        if step["action"]["type"] == "set_text"
+        and _suffix_matches(step["action"].get("resourceId"), "search_edit_text")
+    ]
+    assert youtube_search_values and all(value == "pasta recipes" for value in youtube_search_values), youtube_search_values
+
 
 if __name__ == "__main__":
     _self_test()