patch_v1/README.md

# Sensorium Companion Patch v1

Implements the build plan from `radix-sensorium-companion-digest.md`.

## What This Patch Does

### 1. Companion Voice Depth (`media_companion_mixin.py`)
- **Anti-repetition filter**: Jaccard similarity check on recent comments. If a candidate is too similar to a recent one, it regenerates once, then stays silent.
- **Narrative taste memory**: Replaces analytical bullet dumps (`valence 0.3, conf 0.7`) with natural language ("You've encountered Vivaldi 6 times before. Your reaction has warmed to it.")
- **Rotating lens system**: Each comment comes from a different angle — `texture`, `gesture`, `memory`, `question`, `contrast`, `mood-equation` — so the companion never sounds the same twice in a row.
- **Voice directive**: Daemon mood maps to a voice clause (`bright and curious`, `tense and searching`, `warm and reflective`, `subdued and inward`, `measured and spare`) instead of raw numbers.
- **Compact persona**: Uses a short system prompt instead of the full `build_daemon_prompt()` per comment — cheaper, faster, less context pollution.
- **Explicit `<SILENT>` support**: The companion can choose to say nothing when nothing fresh exists.
- **Screen context integration**: If screen awareness is on, the companion sees what's on screen and can distinguish a code window from a media player.
- **Audio inference fallback**: If audio awareness is on, loopback audio can override a stale window title.

### 2. Screen Awareness (`sensorium_mixin.py`)
- **Screen text digest via OCR**: Uses `pytesseract` (fast, needs system `tesseract-ocr`) or `easyocr` (self-contained, slower) to extract visible text from screenshots.
- **Expanded sensorium pulse**: Now includes screen digest + audio transcript in the heartbeat observation.
- **Context feeding**: Screen digest and audio transcript are fed to the companion via `_last_screen_context` / `_last_audio_context`.
- **Storage management**: Automatic cleanup of old screenshots (keeps last 20), storage budget checking, and manual cleanup tool.

### 3. Audio Awareness (`audition_mixin.py`)
- **Proper loopback device selection**: Priority scoring for VB-Cable, WASAPI loopback, Stereo Mix, BlackHole, `.monitor` sources. Never silently falls back to microphone.
- **Chunked recording**: 30-second mono 16kHz WAV chunks to `sensorium/audio/`.
- **Lazy Whisper loading**: Model loaded on first chunk, not at startup.
- **Media disambiguation**: Lyrics → music, continuous speech → podcast/video, short bursts → ambient/alert.
- **Mute/respect**: Stops recording when companion is muted. Only starts when audio awareness is explicitly enabled.
- **Storage budget**: Caps `sensorium/audio/` at 500MB with automatic pruning.

### 4. Heartbeat Integration (`heartbeat_mixin.py`)
- `_sensorium_pulse()` now runs BEFORE `_media_companion_pulse()`, giving the companion fresh screen/audio context before it comments.

### 5. Settings UI (`app.py`)
- Toggle: "Enable screen awareness (OCR)"
- Toggle: "Enable desktop audio awareness (loopback)"
- Sensorium storage status display
- Cleanup button for old sensorium files

## How to Apply

```bash
# 1. Pull the latest repo
git pull

# 2. Merge multi-part patch files
python patch_v1/merge_patch_parts.py

# 3. Apply the full-file replacements
python patch_v1/apply_sensorium_companion_patch.py

# 4. Manually patch heartbeat_mixin.py
#    Replace the heartbeat_action() method with the version in:
#    patch_v1/heartbeat_mixin_patch.py

# 5. Manually patch app.py
#    Add the helper functions and UI elements from:
#    patch_v1/app_settings_patch.py
#    to the Settings tab

# 6. Install new dependencies
pip install pytesseract Pillow sounddevice soundfile faster-whisper

# For OCR (system package):
# Ubuntu: sudo apt-get install tesseract-ocr
# macOS: brew install tesseract

# 7. Test
python check.py
```

## File Index

| File | Description |
|---|---|
| `media_companion_mixin.py` | Part 1: init, settings, media sensing |
| `media_companion_mixin_part2.py` | Part 2: reaction pipeline |
| `media_companion_mixin_part3.py` | Part 3: anti-repetition, lens, voice, taste |
| `media_companion_mixin_part4.py` | Part 4: logging, taste model, status |
| `sensorium_mixin.py` | Part 1: init, watchers, screen capture |
| `sensorium_mixin_part2.py` | Part 2: OCR digest, audio tools |
| `sensorium_mixin_part3.py` | Part 3: expanded pulse, storage |
| `audition_mixin.py` | Part 1: init, device selection |
| `audition_mixin_part2.py` | Part 2: recording, transcription, classification |
| `audition_mixin_part3.py` | Part 3: cleanup, storage, tools |
| `heartbeat_mixin_patch.py` | Patched heartbeat_action() method |
| `app_settings_patch.py` | Settings UI helpers and elements |
| `merge_patch_parts.py` | Script to merge parts into complete files |
| `apply_sensorium_companion_patch.py` | Script to apply merged files |

## Privacy Guards

- Audio capture is **opt-in** and defaults to disabled
- Loopback device selection is explicit; never silently falls back to microphone
- Screen captures are local files only; never uploaded
- Credential patterns are redacted from clipboard/window memory
- UI toggle can disable screen/audio capture immediately

## Ψ-Calculus Tagging

| Symbol | Priority | Scope |
|---|---|---|
| `[[ΔΩ]]` | 1st | AuditionMixin, screen digest, audio/screen tables, heartbeat wiring |
| `[[ΔI]]` | 2nd | Anti-repetition, lens rotation, audio disambiguation |
| `[[ΔL]]` | 3rd | Deeper companion voice, narrative taste, direct operator address |