FAILED-APPROACHES.md

# PluginBridge — Failed Approaches & Architecture Decisions Log
> Everything we tried that didn't work, why it failed, and what we did instead.
> Use this to avoid repeating mistakes or to revisit with better solutions later.

---

## 1. Out-of-Process GUI via Floating Window (Sprint 2, v0.2)

**What we tried:**
Host the plugin in a separate Helper process. Show the plugin's GUI as a floating window from the Helper process, overlaying Ableton.

**What worked:**
- Audio routing via shared memory ✅
- Crash safety (Helper crashes, Ableton survives) ✅
- GUI showed correctly when Ableton was in WINDOWED mode ✅

**What broke:**
When Ableton was in macOS FULLSCREEN, the plugin GUI opened in a separate Space (different virtual desktop), not over Ableton.

**Fixes attempted (all failed in fullscreen):**

| Attempt | Code | Result |
|---|---|---|
| `NSApplicationActivationPolicyAccessory` | Set in `main.cpp initialise()` | No Dock icon ✅ but broke fullscreen behavior ❌ |
| `setLevel: 3` (NSFloatingWindowLevel) | In `showAsFloatingPanel()` | Not high enough — fullscreen apps are above level 3 ❌ |
| `setLevel: 25` (NSStatusWindowLevel) | In `showAsFloatingPanel()` | Still opens in separate Space ❌ |
| `setCollectionBehavior: canJoinAllSpaces + fullScreenAuxiliary` (257) | Before `setVisible(true)` | Still separate Space ❌ |
| `setCollectionBehavior` + `stationary` + `ignoresCycle` | Full flags (1<<0 \| 1<<4 \| 1<<6 \| 1<<8) | Still separate Space ❌ |
| `orderFrontRegardless` | After setting level | Window appears but wrong Space ❌ |
| `setHidesOnDeactivate: 0` | Combined with above | No effect on Space problem ❌ |
| `setAlwaysOnTop(true)` (JUCE method) | Reverted to original approach | Was original working code but didn't fix fullscreen either ❌ |

**Root cause:**
macOS fullscreen creates a dedicated compositor Space. Apple prevents other processes from injecting windows into a fullscreen app's Space. No combination of window level, collection behavior, or activation policy reliably works for cross-process window display in fullscreen. This is a macOS platform limitation, not a code bug.

**What we did instead:**
Switched to in-process hosting (v0.3). Plugin GUI is embedded inside PluginBridge's own editor. Ableton manages the window. No separate process window needed.

---

## 2. X Button (Close) Not Working on Helper Window (Sprint 2, v0.2)

**What happened:**
The Helper's floating window had minimize and close (X) buttons visible, but clicking X did nothing.

**Root cause:**
When the Helper ran as `NSApplicationActivationPolicyAccessory`, macOS doesn't deliver close-button clicks to accessory-policy processes the same way. The native title bar's close button was visible but non-functional.

**Fix by Claude Code (commit 0632681):**
Removed `makeAccessoryProcess()`, added `showAsFloatingPanel()` with objc_msgSend level tricks. This fixed the X button but broke the fullscreen behavior (see #1 above).

**What we did instead:**
In v0.3 (in-process hosting), there's no separate window at all. The plugin editor is a child component inside PluginBridge's editor. No close button needed — the DAW manages the window lifecycle.

---

## 3. Plugin Crashes Crashing Ableton (v0.3 initial)

**What happened:**
After moving to in-process hosting, loading Kickstart 2 crashed Ableton (because the plugin crash happens inside Ableton's process).

**Root cause:**
`loadPlugin()` called `createPluginInstance()` directly without first testing if the plugin was safe. Some plugins (Kickstart 2, Ozone 12, Gullfoss) call `abort()` or show `NSAlert` during loading, which kills the host process.

**Fix attempted that was missing:**
The design called for Helper-based test loading BEFORE in-process load, but it was left as a `// TODO` stub.

**What we did:**
Implemented `testPluginSafety()` — spawns `PluginBridgeHelper --test "/path"` before loading. If Helper survives (exit 0) → safe to load in-process. If Helper crashes → add to blocklist → show "Not compatible" → Ableton stays alive.

---

## 4. In-Process Hosting = No Crash Safety for Audio Processing

**Current limitation (v0.3):**
If a plugin passes the test-load but crashes during `processBlock()` later (rare but possible), Ableton will still crash. The Helper test only validates loading, not runtime stability.

**Potential future solutions:**
- Wrap `processBlock()` in a signal handler (fragile, not recommended)
- Use `setjmp`/`longjmp` around processBlock (undefined behavior with C++)
- Move back to out-of-process audio routing BUT keep GUI in-process (hybrid approach)
- Accept the risk — plugins that crash during audio processing are very rare

**Status:** Accepted risk. Most crashes happen during loading, not during audio processing.

---

## 5. Plugin Scanning via PluginDirectoryScanner (Early research, never shipped)

**What was planned:**
Use JUCE's `PluginDirectoryScanner` to scan all installed plugins on first run.

**Why it was rejected:**
Some plugins execute code during scanning (show NSAlert dialogs, call abort()). This would crash the scanner — or worse, crash Ableton if scanning happened in-process.

**What we did instead:**
Read `.vst3` and `.component` filenames from disk (no code execution). Only actually load a plugin when the user selects it (after Helper test).

---

## 6. `addDefaultFormats()` — Deleted in JUCE 8

**What happened:**
Early code used `formatManager.addDefaultFormats()` which doesn't exist in JUCE 8.

**Fix:**
Explicitly register formats:
```cpp
formatManager.addFormat(new juce::VST3PluginFormat());
formatManager.addFormat(new juce::AudioUnitPluginFormat());
```

---

## 7. `MSG_NOSIGNAL` — Doesn't Exist on macOS

**What happened:**
Socket code used `MSG_NOSIGNAL` flag which is Linux-only.

**Fix:**
```cpp
signal(SIGPIPE, SIG_IGN);  // Global
int nosigpipe = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &nosigpipe, sizeof(nosigpipe));  // Per-socket
```

---

## 8. `juce::Thread::setCurrentThreadPriority()` — Removed in JUCE 8

**What happened:**
Audio thread priority setting via JUCE API no longer exists.

**Fix:**
Use native pthread:
```cpp
struct sched_param param;
param.sched_priority = sched_get_priority_max(SCHED_FIFO);
pthread_setschedparam(pthread_self(), SCHED_FIFO, &param);
```

---

## 9. `create_audio_track()` Serialization Error (Ableton LOM, not PluginBridge)

**Context:** From the broader AI Music Production system, not PluginBridge-specific.

**What happened:**
`song.create_audio_track()` in Ableton's LOM returns a "not JSON serializable" error.

**Reality:**
The track IS created. The error is in the return value serialization, not the action.

**Workaround:**
Always check `len(song.tracks)` to confirm track was created, ignore the error message.

---

## Architecture Evolution Summary

```
v0.1 (Sprint 1): Plugin shell + MCP server only (no hosting)
v0.2 (Sprint 2): Out-of-process hosting via shared memory
                  ✅ Crash-safe audio
                  ❌ GUI in separate window → fullscreen broken
v0.3 (Sprint 3): In-process hosting (like SnappySnap)
                  ✅ GUI embedded in Ableton's window
                  ✅ Helper tests plugin safety before loading
                  ❌ No crash safety during audio processBlock (accepted risk)
```

---

## 10. CAContext + CALayerHost — Cross-Process GUI Embedding (v0.4, 2026-05)

**What we tried:**
After the floating-window fullscreen failure (see #1), we tried the macOS private API for sharing a `CALayer` tree across processes:
- **Helper (server):** `CAContext contextWithCGSConnection:options:` — wraps the plugin's GUI layer and exposes it as a `uint32_t contextId`
- **Plugin (client):** `CALayerHost.contextId = contextId` — a `CALayer` subclass that renders the remote context inline in JUCE's NSView

The contextId was sent over the existing IPC socket. No Mach port bootstrapping needed.

Also fixed a secondary bug: the Helper's NSWindow was never ordered front. Fix was to position it off-screen at `(-32000, -32000)` and call `orderFront:`.

**What happened:**
- Build succeeded, API calls compiled and ran without errors
- `CALayerHost` was created and added as a sublayer to JUCE's NSView
- The plugin GUI never appeared inside the PluginBridge editor window
- Debug showed: `contextId` was valid (non-zero), `CALayerHost` was attached, but nothing rendered

**Why it failed:**
Ableton Live runs inside the macOS app sandbox / process isolation model. `CAContext` cross-process layer sharing requires both processes to be in the same WindowServer session with compatible entitlements. Ableton's process environment appears to block this — the compositor receives the layer tree but doesn't render it into a foreign process's view hierarchy.

This is the same fundamental limitation as the floating-window approach: Apple does not provide a reliable, supported API for one process to embed rendering from another process's view hierarchy.

**Attempts made:**
| Attempt | Result |
|---|---|
| `CAContext` + off-screen NSWindow + `orderFront:` at (-32000,-32000) | API worked, nothing rendered ❌ |
| `CALayerHost` added as sublayer to JUCE NSView root layer | Layer hierarchy correct, blank ❌ |
| Explicit `setNeedsDisplay` / `setNeedsLayout` on CALayerHost | No effect ❌ |
| Sending contextId immediately after `createEditor()` | Same result ❌ |

**Files changed (then reverted):**
- `Source/Shared/PrivateCA.h` — declared `CAContext`, `CALayerHost`, `CGSMainConnectionID`
- `Source/Helper/HelperPluginHost.mm` — `showGui()` using `CAContext`
- `Source/Plugin/PluginBridgeEditor.mm` — `connectRemoteLayer()` using `CALayerHost`

**What we did instead:**
Moved to the **hybrid safety-scan architecture** (v0.5):
- Plugins that pass a safety scan are loaded **in-process** — their `createEditor()` runs inside PluginBridge's process, producing a standard JUCE `AudioProcessorEditor*` that embeds natively
- Plugins that fail the scan are loaded only in Helper for audio-only mode (no GUI)
- No cross-process GUI required at all

This trades crash isolation for reliability: safe plugins (99% of the library) get full GUI; only crash-prone plugins (iZotope Ozone, Neutron, a few others) are audio-only.

---

## Revisit Candidates (Future)

| Problem | Possible Future Solution | Difficulty |
|---|---|---|
| Fullscreen floating window from separate process | Wait for Apple to provide a supported API, or use private `CGSOrderWindow` API | Hard, fragile |
| Runtime crash safety (processBlock) | Hybrid: audio in Helper, GUI in-process. Two instances synced via IPC. | Very hard |
| Blocklist is manual (per-machine) | Share blocklist via cloud / community database | Medium |
| Test-load adds 2-3 second delay first time | Cache results aggressively, pre-test in background on startup | Medium |
| Some plugins crash in createEditor() not createInstance() | Test GUI creation in Helper too (spawn with --test-gui flag) | Medium |