patch_v1/README.md

Sensorium Companion Patch v1

Implements the build plan from radix-sensorium-companion-digest.md.

What This Patch Does

1. Companion Voice Depth (media_companion_mixin.py)

• Anti-repetition filter: Jaccard similarity check on recent comments. If a candidate is too similar to a recent one, it regenerates once, then stays silent.
• Narrative taste memory: Replaces analytical bullet dumps (valence 0.3, conf 0.7) with natural language ("You've encountered Vivaldi 6 times before. Your reaction has warmed to it.")
• Rotating lens system: Each comment comes from a different angle — texture, gesture, memory, question, contrast, mood-equation — so the companion never sounds the same twice in a row.
• Voice directive: Daemon mood maps to a voice clause (bright and curious, tense and searching, warm and reflective, subdued and inward, measured and spare) instead of raw numbers.
• Compact persona: Uses a short system prompt instead of the full build_daemon_prompt() per comment — cheaper, faster, less context pollution.
• Explicit <SILENT> support: The companion can choose to say nothing when nothing fresh exists.
• Screen context integration: If screen awareness is on, the companion sees what's on screen and can distinguish a code window from a media player.
• Audio inference fallback: If audio awareness is on, loopback audio can override a stale window title.

2. Screen Awareness (sensorium_mixin.py)

• Screen text digest via OCR: Uses pytesseract (fast, needs system tesseract-ocr) or easyocr (self-contained, slower) to extract visible text from screenshots.
• Expanded sensorium pulse: Now includes screen digest + audio transcript in the heartbeat observation.
• Context feeding: Screen digest and audio transcript are fed to the companion via _last_screen_context / _last_audio_context.
• Storage management: Automatic cleanup of old screenshots (keeps last 20), storage budget checking, and manual cleanup tool.

3. Audio Awareness (audition_mixin.py)

• Proper loopback device selection: Priority scoring for VB-Cable, WASAPI loopback, Stereo Mix, BlackHole, .monitor sources. Never silently falls back to microphone.
• Chunked recording: 30-second mono 16kHz WAV chunks to sensorium/audio/.
• Lazy Whisper loading: Model loaded on first chunk, not at startup.
• Media disambiguation: Lyrics → music, continuous speech → podcast/video, short bursts → ambient/alert.
• Mute/respect: Stops recording when companion is muted. Only starts when audio awareness is explicitly enabled.
• Storage budget: Caps sensorium/audio/ at 500MB with automatic pruning.

4. Heartbeat Integration (heartbeat_mixin.py)

• _sensorium_pulse() now runs BEFORE _media_companion_pulse(), giving the companion fresh screen/audio context before it comments.

5. Settings UI (app.py)

• Toggle: "Enable screen awareness (OCR)"
• Toggle: "Enable desktop audio awareness (loopback)"
• Sensorium storage status display
• Cleanup button for old sensorium files

How to Apply

# 1. Pull the latest repo
git pull

# 2. Merge multi-part patch files
python patch_v1/merge_patch_parts.py

# 3. Apply the full-file replacements
python patch_v1/apply_sensorium_companion_patch.py

# 4. Manually patch heartbeat_mixin.py
#    Replace the heartbeat_action() method with the version in:
#    patch_v1/heartbeat_mixin_patch.py

# 5. Manually patch app.py
#    Add the helper functions and UI elements from:
#    patch_v1/app_settings_patch.py
#    to the Settings tab

# 6. Install new dependencies
pip install pytesseract Pillow sounddevice soundfile faster-whisper

# For OCR (system package):
# Ubuntu: sudo apt-get install tesseract-ocr
# macOS: brew install tesseract

# 7. Test
python check.py

File Index

File	Description
media_companion_mixin.py	Part 1: init, settings, media sensing
media_companion_mixin_part2.py	Part 2: reaction pipeline
media_companion_mixin_part3.py	Part 3: anti-repetition, lens, voice, taste
media_companion_mixin_part4.py	Part 4: logging, taste model, status
sensorium_mixin.py	Part 1: init, watchers, screen capture
sensorium_mixin_part2.py	Part 2: OCR digest, audio tools
sensorium_mixin_part3.py	Part 3: expanded pulse, storage
audition_mixin.py	Part 1: init, device selection
audition_mixin_part2.py	Part 2: recording, transcription, classification
audition_mixin_part3.py	Part 3: cleanup, storage, tools
heartbeat_mixin_patch.py	Patched heartbeat_action() method
app_settings_patch.py	Settings UI helpers and elements
merge_patch_parts.py	Script to merge parts into complete files
apply_sensorium_companion_patch.py	Script to apply merged files

Privacy Guards

• Audio capture is opt-in and defaults to disabled
• Loopback device selection is explicit; never silently falls back to microphone
• Screen captures are local files only; never uploaded
• Credential patterns are redacted from clipboard/window memory
• UI toggle can disable screen/audio capture immediately

Ψ-Calculus Tagging

Symbol	Priority	Scope
[[ΔΩ]]	1st	AuditionMixin, screen digest, audio/screen tables, heartbeat wiring
[[ΔI]]	2nd	Anti-repetition, lens rotation, audio disambiguation
[[ΔL]]	3rd	Deeper companion voice, narrative taste, direct operator address