BACKLOG.md

# Puck backlog

Ideas not yet built. Each entry: what, why, and where it slots into the architecture
(engine = pure policy/data, ui = presentation, modes/sim = orchestration).

## Gestures — sprite dances/reactions keyed to event type

**What:** Puck performs a short gesture matched to *what* he's reporting, not just
flying + speaking. A build passing → a little hop/spin of pride. Tests failing or a
permission block → an urgent shake/alarm wiggle. Discord noise he ignored → a dismissive
shrug. Claude finished → a satisfied stretch. Stale tab → a curious head-tilt. Sleep →
a yawn. The gesture plays during the flight-to + bubble beat.

**Why:** On a busy real desktop the sprite is glanceable; a gesture lets you read the
*kind* of news pre-attentively, before the words. It's the single highest charm-per-line
addition left, and it makes the overlay feel like a creature rather than a notifier.

**Where it slots in:**
- `engine/` — a pure `gestureFor(event | decision)` map: `EventDef.id`/`source` →
  gesture name. Data, not logic; pin a couple of mappings with a test. Could also key
  off the existing tier/mischief so high-mischief gestures are more theatrical.
- `ui/Sprite.tsx` — a `gesture?: GestureName` prop adding a CSS class (`gesture-hop`,
  `gesture-shake`, `gesture-shrug`, …); animations on `.puck-bob`/`.puck-face` only
  (compositor-friendly transforms, like the existing flap/bob). Clear the class on
  animationend so it re-triggers.
- `modes/sim/SimApp.tsx` — set the gesture alongside the existing `flyTo`/`setBubble`
  in `fireEvent`; clears when the surface resolves.

**Notes:** keep gestures to transform/opacity so they stay GPU-composited (the whole
sprite layer is). Reuse the per-form structure in `SpriteBody` — gestures should read
on all four forms (mossling/wisp/gremlin/moth), so animate the wrapper, not form-specific
parts. Pairs naturally with the existing mood tint + alert ring.

**Color pulse (same feature, cheapest channel):** transient-tint the alert ring + glow
to a per-event-type hue while a surface is pending — red/urgent for failures &
permission blocks, gold/success for completions, grey/dim for ignored noise. The ring
and `.puck-glow` already read CSS vars (`--accent`, `--glow`), so this is a single
`--alert-hue` override set from the same `engine` map that picks the gesture — one
`{ gesture, hue }` lookup feeds both. Even shipping the color pulse *alone* (before
gesture animation work) is a real readability win. Don't fight the existing mood tint
(`--puck-body` etc.): pulse the ring/glow, leave the body color to mood, so "what kind
of news" (hue) and "how Puck feels lately" (mood) stay separate signals.

## Ambient quips — Puck reacts to what you're doing

**What:** Occasionally Puck mutters a quip or non-sequitur riffing on your current
context — the active app, a window title, the shape of what you're typing. Not help,
not a summary: flavor. "Still in the auth thicket, I see." / "That's a lot of tabs for
one small human." / a non-sequitur about the dock breathing. Low frequency, skippable,
never twice about the same thing.

**Why:** This is the difference between a notifier that lives in the corner and a
familiar that *shares the room*. It's the most "alive" feature on the list — and the
most dangerous, so it ships last and most carefully.

**Privacy is the feature, not a caveat** (design doc §17.3–17.4 are the law here):
- **Local inference ONLY — hard gate.** Context never leaves the machine. If the brain
  is the cloud path (Modal/ZeroGPU), this feature is *disabled*, full stop, not degraded.
  Enforce in code: the quip path checks the resolved brain is localhost and refuses
  otherwise — a loud guard, not a setting.
- **Opt-in, off by default.** A distinct toggle from notifications; the permission copy
  says plainly what's read and that it stays local.
- **Ephemeral, never persisted, never a trace, never training data.** Context for a quip
  is read, used for one generation, dropped. It must not touch the memory garden, the
  trace export, or localStorage.
- **Redact before the model sees it.** Strip obvious secrets (password fields, token-shaped
  strings, anything in a field marked sensitive). Prefer *abstractions* over content —
  "a long terminal command", "a code file", "a messaging app" — over the literal text.
  Quoting back verbatim is the creepy line; stay on the abstract side of it.
- **Never about people or private content.** App categories and your own activity, yes;
  the contents of a DM, an email body, a name on screen — no.

**Where it slots in:**
- The desktop watcher (future, §9.2: NSWorkspace frontmost app, optional AX/screen) is the
  context source — same daemon `/events` path, a new low-priority `context` event kind, or
  a separate local-only endpoint that never queues.
- `engine` decides *whether* to quip (rare; respects annoyance budget / presence, reuses
  the interruption-taste machinery so an annoying quip trains Puck quieter).
- The quip generation uses the local brain with a tight "one playful aside, ≤12 words,
  never quote the user" prompt; surfaces through the existing bubble channel.

**Ship order:** after the desktop watcher exists and after a privacy pass. Until then it's
sim-only flavor at most (riffing on the fake desktop, where there's nothing real to leak).

## Phototaxis — a fairy drawn to stimuli (whimsical wander)

**What:** Puck should behave like a moth/fairy — *pulled* toward activity rather than
drifting at random. Flits toward motion, lingers near what's lively, chases the
occasional shiny thing, then loses interest.

**Why:** The wander is the sprite's resting personality — it's on screen far more than
any bubble. Uniform-random reads as a screensaver; attraction reads as *alive and
curious*. Highest charm-per-effort of the ambient ideas.

**Buildable now — zero new permissions (do this part first):**
Replace the uniform-random wander target in `SimApp` with a weighted pull toward salient
points we already have:
- the **cursor** (occasional gentle follow / curious approach, then retreat — never
  clingy; respects `presence`),
- the **last event location** (he lingers where something just happened),
- in the overlay, the **focused window** rect (future: NSWorkspace frontmost-app
  position via the daemon — he hangs near where you're working, patrols where you're not).
Keep it a *gradient*, not a leash: weighted-random pick among attractors + noise, so it
stays unpredictable. Lives in the engine as a pure `pickWanderTarget(attractors, rng)`;
the loop already exists. Tune against `presence` (low = aloof, high = follows the action).

**Sensor-gated — backlog, same privacy rules as ambient quips (local-only, ephemeral):**
- **Screen color / motion / "flashing lights":** needs ScreenCaptureKit — the transparent
  overlay can't see what's beneath it. A coarse, downsampled brightness/motion map (NOT
  readable content) could let him drift toward an area that just changed (a video started,
  a notification flashed). Local-only, never stored, abstractions not pixels.
- **Audio reactivity:** mic is a hard no by default; "system audio is playing / its level"
  is lighter but still opt-in + local-only. A bass-thump bob or a turn-toward-the-sound
  would be delightful but ships last, behind the same gate as quips.

**Smell test for all of it:** attraction must stay *cute*, never *surveillant*. He reacts
to the shape of activity (something moved, something's loud), never to its content.

## Take-me-there — Puck knows where the activity is, and ferries you to it

**What:** On a real notification, Puck should point you at the *actual* window that needs
you — fly to it if it's on this Space, beckon "follow me" if it's on another — and
**clicking him navigates there** (focuses the app, macOS brings its Space forward).

**Current behavior (the gap):** wire events arrive with `target: null` (the sim's targets
were fake windows), so in the overlay `fireEvent` flies Puck to a *random* screen point.
He has no idea where the source app lives — we never gave him real-window awareness.
This is the design doc's Phase 4 ("patrol the desktops you abandoned" presumes knowing
where they are).

**Phase 1 — click-to-activate (high value, no Space geometry needed):**
- Event carries a **locator**: source app bundle id / pid / window title. The Claude hook
  already has `TERM_PROGRAM`, `cwd`, and the calling pid available; `puck-run` knows its
  terminal. Add an optional `locator` to the wire schema (`{bundleId?, pid?, title?}`).
- Rust command `activate_target(locator)` -> `NSRunningApplication(bundleIdentifier:).activate`
  (or AX focus by pid/title). macOS switches to that app's Space automatically.
- Frontend: clicking Puck while a located surface is pending calls it. This alone delivers
  "Puck lit up -> click -> you're where the thing is," across Spaces, without knowing which.

**Phase 2 — same-Space vs other-Space (the fiddly bit):**
- Determine if the target window is on the *current* Space: `CGWindowListCopyWindowInfo`
  with `kCGWindowListOptionOnScreenOnly` lists on-screen windows; absent target -> elsewhere.
  Robust Space identity needs the semi-private CGSSpace APIs — fragile, optional.
- Same Space -> fly to the window's screen rect (window bounds from CGWindowList).
  Other Space -> a "come hither" beckon toward the screen edge (pairs with the gesture
  entry), and the click teleports.

**Notes:** locator is metadata, not content — bundle id + window title, never window
*contents* (anti-creep). Activation is an explicit user click (navigation, not an
autonomous action), so it stays inside the safety tiers. Depends on: gesture vocabulary
(beckon) and the future native window watcher (NSWorkspace frontmost / CGWindowList).

## Camouflage — Puck adapts to what he's floating over

**What:** When Puck drifts over text or busy content, he reacts to his surroundings like
a chameleon/glass-wisp — goes translucent, refracts, or (dream version) "mirrors" the
texture behind him onto his own body. Blends, then pops back when he moves to empty space.

**Why:** Sells "he's really *in* your desktop, not pasted on top." A creature that
responds to its background reads as inhabiting the space. Pairs with the wisp form
(already glass-like).

**Cheap / free now (no sensing):**
- **Shy fade:** lower sprite opacity while stationary over the busy center of the screen,
  restore when wandering to the margins — pure CSS/opacity on the existing wander state.
  Reads as "blending in" without literally seeing anything.
- **Refraction (maybe free, needs a WKWebView test):** a `backdrop-filter` glass body on
  the sprite. In the transparent overlay this *might* sample the real desktop behind the
  window (same open question as the speech-bubble blur) — if it does, a refractive/distort
  body gives instant chameleon shimmer with zero screen-capture. Test in the Tauri overlay
  before committing to it.

**Dream version — literal mirror (screen-capture gated):**
- Sample the screen region directly under the sprite (ScreenCaptureKit), downsample, and
  paint it onto his body as living camouflage. Stunning, but it's the same hard gate as
  ambient quips / phototaxis sensors: local-only, ephemeral, opt-in, never stored, never
  leaves the machine. Texture/color only — never treated as readable content.

**Where it slots in:** `ui/Sprite.tsx` (a `camouflage` intensity prop on `.puck-body`),
driven by `modes/sim` from sprite position vs. screen regions. The shy-fade is a
20-minute add; refraction is a test-then-maybe; the literal mirror waits for the screen
watcher + privacy pass.

## Real app icons — known apps, OS-extracted where possible

**What:** Replace the glyph characters (✳ ◍ ✉ …) in the sim windows/dock and feed
source-markers with real app icons — Claude, ChatGPT, Chrome, Gmail, Mail, Discord,
Terminal, etc. Looks dramatically more legit, especially in the overlay.

**Pulling the *actual* OS icon (the good version):**
- macOS: `NSWorkspace.shared.icon(forFile: "/Applications/Foo.app")` returns the real
  icon for any installed `.app` bundle — a Rust/Tauri command can extract → PNG → hand to
  the webview. So Chrome, Mail, Discord, Slack, the host terminal: real icons, free, always
  current.
- **The limit you called:** a terminal *binary* (claude, codex) has no bundle and no icon.
  Fall back to the **host terminal's** icon (iTerm/Terminal/Ghostty/Warp — which the Claude
  hook can report via `TERM_PROGRAM`), or a curated Puck-styled glyph.
- **Web apps with no native app** (ChatGPT, Gmail as a tab): no bundle to extract from —
  these need a small **curated bundled icon set** (a dozen SVGs/PNGs).

**So: hybrid.** OS extraction where a bundle exists (Rust command, overlay only), curated
bundled set for web-apps + terminal-binary fallbacks (works everywhere incl. the Space/sim).

**Where it slots in:**
- `ui/Desktop.tsx` (`WIN_DEFS` icons, dock glyphs) and the feed `source` marker — currently
  single glyph chars; swap for an `<Icon source=…>` that prefers OS-extracted, falls back to
  bundled, falls back to glyph.
- Overlay-only Rust command `app_icon(bundleId|path) -> png` for the extraction half.
- Engine `SOURCES`/`EventDef` already carry source identity; add an optional `bundleId` hint.

**Cheap first step (no native):** ship the curated bundled icon set + source→icon map; use
glyph only as last resort. The OS-extraction half is an overlay enhancement on top.

## Local vision — the private, free path for continuous perception

**What:** Run Puck's eyes on-device instead of (or alongside) Modal, so real-screen
perception is private and continuous vision costs ~nothing. Brain seam already supports
it: point PUCK_VISION_URL at a local OpenAI-compatible server.

**Capability is NOT the blocker** (verified 2026-06-07): screen-reading is OCR + light
"what's notable" reasoning — small VLMs excel at it. Options:
- **Holotron-12B local** via llama.cpp + mmproj — llama.cpp merged Nemotron-Nano-12B-v2-VL
  (PR #19547). `convert_hf_to_gguf.py --mmproj` → vision projector → `llama-server`. Same
  model as cloud, same capability, ~24GB on the 48GB Mac, free. NB: Ollama can't load the
  mmproj — must use raw llama-server. Keeps the Nemotron Quest tie.
- **Qwen2.5-VL-7B local** — ~6GB, 95.7 DocVQA, fast; ideal for *continuous ambient* (every
  45s forever on the M4 Max). Loses the Nemotron tie, plenty for screen-reading.
- MiniCPM-V 2.6 (~5.5GB), Moondream2 (1.9B, CPU) as even-lighter fallbacks.

**Recommendation:** Holotron for the showcase + Quest (cloud now → local llama-server
later as the private path); Qwen2.5-VL-7B-local as the cheap continuous engine. The
visionMode "Continuous" tier (built, currently same cloud path) should switch to a local
PUCK_VISION_URL when this lands.

**Where it slots in:** zero engine/frontend change — it's a runtime: spin up
`llama-server --mmproj` (or vLLM/mlx when fixed) and set PUCK_VISION_URL. Plus the
real-screen capture (ScreenCaptureKit, overlay) to feed it actual pixels instead of the
sim snapshot.