| # 🛡️ Sentinel — Deep Implementation Playbook |
| ## 10 Upgrades: Fully Architected, Battle-Planned |
|
|
| **Rule:** Agent does all the thinking. You review the plan, approve or redirect, then agent executes. |
| **Rule:** Visual/UI work requires you to approve a reference style before agent writes any code. |
|
|
| --- |
|
|
| ## UPGRADE 1: Client-Side Frame Differencing (OpenCV.js) |
|
|
| **Goal:** Browser analyzes frames locally, only uploads to server when motion detected. |
| **Files touched:** `sensor_bridge.py` only. |
| **New lines:** ~80 lines of JavaScript. |
|
|
| ### Architecture |
|
|
| ``` |
| CURRENT FLOW (wasteful): |
| Camera → canvas 320x240 → base64 JPEG → image-data textbox → server process_frame |
| Rate: 2 FPS always = 120 uploads/minute |
| |
| NEW FLOW (smart): |
| Camera → canvas 320x240 → OpenCV.js cv.absdiff(prev, current) → |
| if changed_pixels > threshold → upload base64 to server |
| if not → skip upload, increment local counter |
| Rate: ~0.3-0.8 FPS average = 18-48 uploads/minute |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Add OpenCV.js CDN load to sensor_bridge.py** |
| |
| Insert at the top of the HTML string (before the IIFE): |
| ```html |
| <script async src="https://docs.opencv.org/4.x/opencv.js" |
| onload="window.__cvReady = true;" type="text/javascript"></script> |
| ``` |
| |
| **Step 2: Add state variables inside the IIFE** |
| |
| ```javascript |
| let prevMat = null; // Previous frame as cv.Mat |
| let cvReady = false; // OpenCV.js loaded flag |
| let skippedFrames = 0; // Count of locally-filtered frames |
| const DIFF_THRESHOLD = 0.12; // 12% pixel change = significant |
| ``` |
| |
| **Step 3: Modify `captureFrame()` function** |
|
|
| The current `captureFrame()` always uploads. New version: |
|
|
| ``` |
| captureFrame(): |
| 1. Draw video to canvas (same as now) |
| 2. If cvReady: |
| a. Read canvas into cv.Mat (srcMat) |
| b. Convert to grayscale (cv.cvtColor) |
| c. If prevMat exists: |
| - cv.absdiff(prevMat, srcMat, diffMat) |
| - cv.threshold(diffMat, threshMat, 25, 255, cv.THRESH_BINARY) |
| - changedPct = cv.countNonZero(threshMat) / (width * height) |
| - If changedPct < DIFF_THRESHOLD: |
| skippedFrames++ |
| update UI indicator ("Filtered locally: X frames") |
| cleanup Mats, return (DON'T upload) |
| d. prevMat = srcMat.clone() |
| e. cleanup temp Mats |
| 3. Proceed with base64 upload (same as now) |
| ``` |
|
|
| **Step 4: Add skipped frame counter to UI** |
|
|
| Add a hidden Gradio Number component: |
| ```python |
| # In app.py Monitor tab, alongside other hidden numbers: |
| skipped_frames = gr.Number(value=0, visible=False, elem_id="skipped-frames") |
| ``` |
|
|
| JS updates it via `setGradioValue("skipped-frames", skippedFrames)` every 2 seconds. |
|
|
| ### Edge Cases |
| - **OpenCV.js fails to load (CDN blocked):** Fall back to current behavior (always upload). Check `window.__cvReady` before using cv. |
| - **First frame:** No prevMat exists → always upload first frame, store as prevMat. |
| - **Mat memory leak:** Always call `.delete()` on temporary Mats. Only keep prevMat alive. |
| - **Camera off:** Reset prevMat to null when camera stops. |
|
|
| ### Why This Wins |
| The 240x cost story goes from "trust our architecture" to "watch the counter — 847 frames processed locally, only 4 sent to GPU." That's a demo moment. |
|
|
| ### Testing Sequence |
| 1. Deploy with upgrade → activate → wave hand in front of camera → see uploads spike |
| 2. Sit still → see uploads drop to near zero |
| 3. Check that simulation mode still works (it bypasses camera entirely) |
|
|
| --- |
|
|
| ## UPGRADE 2: Three.js / CSS3D Gyroscope HUD |
|
|
| **Goal:** Floating glass HUD overlay on camera feed. Tilts with phone. Shows live telemetry. |
| **Files touched:** `sensor_bridge.py` (~200 lines JS/CSS). |
| **Dependency:** None (pure client-side). |
|
|
| ### Architecture Decision |
|
|
| **Option A — Pure CSS (Recommended for reliability):** |
| - No external library |
| - CSS `transform: perspective(1000px) rotateX() rotateY()` driven by `deviceorientation` |
| - ~80 lines, impossible to break |
| - Looks great but not "3D" |
|
|
| **Option B — Three.js CSS3DRenderer (Recommended for wow):** |
| - Load three.js from CDN (~150KB) |
| - Create CSS3DObject from HUD HTML elements |
| - Rotate entire CSS3D scene based on device orientation |
| - ~200 lines, can break if CDN fails |
| - True 3D parallax effect |
|
|
| ### My Recommendation: Option A for reliability, Option B for demo video |
|
|
| Start with Option A. It works on every device, including iOS. If you have time, add Option B as an enhancement. |
|
|
| ### Implementation Plan (Option A — Pure CSS) |
|
|
| **Step 1: Add HUD container HTML to sensor_bridge.py** |
| |
| ```html |
| <div id="sentinel-hud" style=" |
| position: fixed; |
| top: 80px; |
| left: 50%; |
| transform: translateX(-50%); |
| z-index: 1000; |
| pointer-events: none; |
| transition: transform 0.1s ease-out; |
| perspective: 800px; |
| display: none; |
| "> |
| <div id="hud-inner" style=" |
| background: rgba(255,255,255,0.06); |
| border: 1px solid rgba(255,255,255,0.15); |
| border-radius: 16px; |
| padding: 12px 20px; |
| backdrop-filter: blur(20px); |
| min-width: 240px; |
| font-family: monospace; |
| color: #a5b4fc; |
| "> |
| <div id="hud-compass">Heading: 0° N</div> |
| <div id="hud-threat" style="color: #4ade80;">Threat: CLEAR</div> |
| <div id="hud-fps">FPS: 2.0 | Filtered: 0</div> |
| <div id="hud-battery">Battery: 100%</div> |
| </div> |
| </div> |
| ``` |
| |
| **Step 2: Add deviceorientation handler** |
| |
| ```javascript |
| // Inside IIFE, after existing sensor handlers: |
| let hudActive = false; |
| const hudEl = document.getElementById("sentinel-hud"); |
| const hudInner = document.getElementById("hud-inner"); |
| |
| function handleHUDOrientation(event) { |
| if (!hudActive || !hudInner) return; |
| // beta = front-back tilt (-180 to 180) |
| // gamma = left-right tilt (-90 to 90) |
| const tiltX = Math.max(-8, Math.min(8, event.gamma * 0.15)); |
| const tiltY = Math.max(-8, Math.min(8, (event.beta - 45) * 0.15)); |
| hudInner.style.transform = |
| `perspective(800px) rotateX(${-tiltY}deg) rotateY(${tiltX}deg)`; |
| } |
| window.addEventListener('deviceorientation', handleHUDOrientation); |
| ``` |
| |
| **Step 3: Update HUD data on each sensor tick** |
| |
| The existing sensor handlers already have access to heading, battery, etc. Add DOM updates: |
| |
| ```javascript |
| // In the existing telemetry update cycle: |
| if (hudActive) { |
| document.getElementById("hud-compass").textContent = |
| `Heading: ${heading}° ${getCompassDir(heading)}`; |
| document.getElementById("hud-battery").textContent = |
| `Battery: ${batteryPct}%`; |
| document.getElementById("hud-threat").textContent = |
| `Threat: ${currentThreatLevel}`; |
| document.getElementById("hud-threat").style.color = |
| currentThreatLevel === "CLEAR" ? "#4ade80" : |
| currentThreatLevel === "WARNING" ? "#fbbf24" : "#ef4444"; |
| } |
| ``` |
| |
| **Step 4: Show HUD on activation** |
| |
| In `bindActivationButton()`, after starting sensors: |
| ```javascript |
| if (hudEl) { hudEl.style.display = "block"; hudActive = true; } |
| ``` |
| |
| ### What You Need to Approve |
| - HUD position: top-center? bottom? full-width strip? |
| - Data fields: compass + threat + FPS + battery? Or different data? |
| - Glass style: subtle (0.06 opacity) or more visible (0.12)? |
| - Color scheme: indigo text (#a5b4fc) or white? |
| |
| ### Testing Sequence |
| 1. Activate on phone → HUD appears floating over camera |
| 2. Tilt phone left → HUD tilts right (parallax) |
| 3. Trigger simulation → HUD threat level changes color |
| 4. Test on iOS → works because it's pure CSS, no WebGL |
| |
| --- |
| |
| ## UPGRADE 3: Binaural 3D Spatial Audio (Web Audio API) |
| |
| **Goal:** Alert voice sounds like it's coming FROM the direction of danger. |
| **Files touched:** `sensor_bridge.py` (~100 lines JS), `app.py` (~10 lines modified). |
| **Dependency:** None (Web Audio API is native to all modern browsers). |
|
|
| ### Architecture |
|
|
| ``` |
| CURRENT: |
| process_frame → alert_html with <audio autoplay src="data:audio/wav;base64,..."> |
| → browser plays audio from center (no spatial) |
| |
| NEW: |
| process_frame → determines threat_x, threat_y from detection bbox center |
| → embeds them as data attributes in alert_html |
| → JS intercepts: decodes base64 audio → creates AudioBufferSourceNode |
| → connects to PannerNode (HRTF model, positioned at threat_x/y/z) |
| → connects to AudioListener (oriented by deviceorientation) |
| → audio plays from spatial position |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Add spatial audio engine to sensor_bridge.py** |
| |
| ```javascript |
| // Inside IIFE: |
| let spatialAudioCtx = null; |
| let audioListener = null; |
| |
| function initSpatialAudio() { |
| spatialAudioCtx = new (window.AudioContext || window.webkitAudioContext)(); |
| audioListener = spatialAudioCtx.listener; |
| // Set listener to face forward |
| if (audioListener.positionX) { |
| audioListener.positionX.value = 0; |
| audioListener.positionY.value = 0; |
| audioListener.positionZ.value = 0; |
| audioListener.forwardX.value = 0; |
| audioListener.forwardY.value = 0; |
| audioListener.forwardZ.value = -1; |
| } |
| } |
| |
| async function playSpatialAudio(base64Audio, threatX, threatY) { |
| if (!spatialAudioCtx) initSpatialAudio(); |
| if (spatialAudioCtx.state === 'suspended') await spatialAudioCtx.resume(); |
| |
| // Decode base64 WAV to AudioBuffer |
| const binary = atob(base64Audio); |
| const bytes = new Uint8Array(binary.length); |
| for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i); |
| const audioBuffer = await spatialAudioCtx.decodeAudioData(bytes.buffer); |
| |
| // Create source |
| const source = spatialAudioCtx.createBufferSource(); |
| source.buffer = audioBuffer; |
| |
| // Create 3D panner |
| const panner = spatialAudioCtx.createPanner(); |
| panner.panningModel = 'HRTF'; |
| panner.distanceModel = 'inverse'; |
| panner.refDistance = 1; |
| panner.maxDistance = 10; |
| // Position: threatX maps to left(-1) to right(+1), Y to up/down |
| panner.positionX.value = threatX * 2; // -2 to +2 meters |
| panner.positionY.value = threatY; |
| panner.positionZ.value = -2; // 2 meters in front |
| |
| // Connect: source → panner → destination |
| source.connect(panner); |
| panner.connect(spatialAudioCtx.destination); |
| source.start(0); |
| } |
| ``` |
| |
| **Step 2: Update deviceorientation to rotate listener** |
| |
| ```javascript |
| // In existing handleOrientation function, add: |
| if (spatialAudioCtx && audioListener) { |
| const rad = (event.alpha || 0) * Math.PI / 180; |
| if (audioListener.forwardX) { |
| audioListener.forwardX.value = -Math.sin(rad); |
| audioListener.forwardZ.value = -Math.cos(rad); |
| } |
| } |
| ``` |
| |
| **Step 3: Modify app.py alert generation** |
| |
| In `process_frame`, when building alert_html, embed threat coordinates: |
| |
| ```python |
| # After YOLO detections, calculate threat position |
| threat_x = 0.0 # center by default |
| threat_y = 0.0 |
| if detections: |
| # Use center of first trigger detection bbox, normalized to -1..1 |
| det = detections[0] |
| threat_x = ((det.bbox[0] + det.bbox[2]) / 2 / 640 - 0.5) * 2 # -1 to 1 |
| threat_y = -((det.bbox[1] + det.bbox[3]) / 2 / 480 - 0.5) * 2 # inverted Y |
| |
| # In alert_html, add data attributes: |
| alert_html = f"""<div class='alert-banner-{alert_level}' |
| data-threat-x='{threat_x:.2f}' |
| data-threat-y='{threat_y:.2f}' |
| data-audio-src='{audio_data_uri}'> |
| ⚠️ {alert_text}{alert_image_html}</div>""" |
| # Remove the <audio autoplay> tag — JS handles playback now |
| ``` |
| |
| **Step 4: JS observer detects new alert and triggers spatial playback** |
|
|
| ```javascript |
| // MutationObserver on alert-banner-container |
| const alertObserver = new MutationObserver((mutations) => { |
| for (const m of mutations) { |
| const banner = m.target.querySelector('[data-audio-src]'); |
| if (banner) { |
| const audioSrc = banner.dataset.audioSrc; |
| const tx = parseFloat(banner.dataset.threatX || 0); |
| const ty = parseFloat(banner.dataset.threatY || 0); |
| if (audioSrc && audioSrc.startsWith('data:audio')) { |
| const base64 = audioSrc.split(',')[1]; |
| playSpatialAudio(base64, tx, ty); |
| } |
| } |
| } |
| }); |
| ``` |
|
|
| ### What You Need to Approve |
| - Fallback: if Web Audio fails (old browser), fall back to `<audio autoplay>` tag? |
| - Spatial range: ±2 meters (subtle) or ±5 meters (dramatic)? |
| - Should listener rotate with gyroscope or stay fixed forward? |
|
|
| ### Testing Sequence |
| 1. Put on headphones |
| 2. Trigger "Person Approaching" sim (person comes from left) |
| 3. Voice should sound like it's coming from the LEFT earphone |
| 4. Turn phone 90° → voice position shifts relative to your head |
| 5. Test without headphones → still works but less immersive |
|
|
| --- |
|
|
| ## UPGRADE 4: Dynamic SVG Bounding Box Overlays |
|
|
| **Goal:** YOLO detection boxes drawn as animated SVG overlays on the camera feed. |
| **Files touched:** `sensor_bridge.py` (~50 lines JS), `app.py` (~30 lines). |
| **Dependency:** Upgrade 1 (client-side frame diff) recommended first. |
|
|
| ### Architecture |
|
|
| ``` |
| CURRENT: |
| process_frame → YOLO detect → detections used internally only → discarded |
| Camera feed shows raw video with no annotations |
| |
| NEW: |
| process_frame → YOLO detect → serialize detections as JSON → |
| write to hidden textbox "detection-data" → |
| JS reads JSON → creates/updates SVG overlay on camera feed → |
| animated rectangles with class labels |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Always run YOLO (not just when gatekeeper triggers)** |
|
|
| In `app.py`, change: |
| ```python |
| # BEFORE (line 563): |
| if change_result.is_significant and object_detector is not None: |
| |
| # AFTER: |
| if object_detector is not None: |
| ``` |
|
|
| This means YOLO runs on every frame, not just significant changes. |
| Cost: ~5ms per frame on CPU. Acceptable for real-time display. |
| The GATEKEEPER still decides whether to call VLM — this only changes what we SHOW. |
|
|
| **Step 2: Serialize detections to JSON** |
|
|
| After YOLO runs, write detection data: |
| ```python |
| # After line 565 (yolo_summary = ...): |
| detection_json = json.dumps([ |
| { |
| "class": d.class_name, |
| "conf": round(d.confidence, 2), |
| "bbox": list(d.bbox), |
| "trigger": d.is_trigger |
| } |
| for d in detections |
| ]) |
| ``` |
|
|
| **Step 3: Add hidden textbox for detection data** |
|
|
| In app.py Monitor tab: |
| ```python |
| detection_data = gr.Textbox(value="[]", visible=False, elem_id="detection-data") |
| ``` |
|
|
| Update it in process_frame return tuple (add to outputs). |
| |
| **Step 4: SVG overlay in sensor_bridge.py** |
| |
| ```javascript |
| // Create SVG overlay positioned over the camera feed image |
| function createSVGOverlay() { |
| const cameraImg = document.querySelector('#alert-banner-container') |
| ?.closest('.glass-panel') |
| ?.querySelector('img'); // The camera feed <img> element |
| if (!cameraImg) return null; |
| |
| const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg"); |
| svg.id = "detection-svg"; |
| svg.style.cssText = "position:absolute;top:0;left:0;width:100%;height:100%;pointer-events:none;z-index:10;"; |
| cameraImg.parentElement.style.position = "relative"; |
| cameraImg.parentElement.appendChild(svg); |
| return svg; |
| } |
| |
| // Color map for detection classes |
| const CLASS_COLORS = { |
| "person": "#3b82f6", |
| "vehicle": "#ef4444", |
| "fire": "#f97316", |
| "animal": "#a855f7", |
| "default": "#22c55e" |
| }; |
| |
| function drawDetections(detections) { |
| const svg = document.getElementById("detection-svg") || createSVGOverlay(); |
| if (!svg) return; |
| svg.innerHTML = ""; // Clear previous |
| |
| detections.forEach(det => { |
| const [x1, y1, x2, y2] = det.bbox; |
| const color = CLASS_COLORS[det.class] || CLASS_COLORS.default; |
| const w = x2 - x1, h = y2 - y1; |
| |
| // Animated rectangle |
| const rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); |
| rect.setAttribute("x", x1); rect.setAttribute("y", y1); |
| rect.setAttribute("width", w); rect.setAttribute("height", h); |
| rect.setAttribute("fill", "none"); |
| rect.setAttribute("stroke", color); |
| rect.setAttribute("stroke-width", "3"); |
| rect.setAttribute("rx", "4"); |
| rect.style.opacity = "0"; |
| rect.style.transition = "opacity 0.3s"; |
| svg.appendChild(rect); |
| requestAnimationFrame(() => rect.style.opacity = "0.8"); |
| |
| // Label |
| const text = document.createElementNS("http://www.w3.org/2000/svg", "text"); |
| text.setAttribute("x", x1); text.setAttribute("y", y1 - 6); |
| text.setAttribute("fill", color); |
| text.setAttribute("font-size", "14"); |
| text.setAttribute("font-family", "monospace"); |
| text.textContent = `${det.class} ${(det.conf * 100).toFixed(0)}%`; |
| svg.appendChild(text); |
| }); |
| } |
| ``` |
| |
| **Step 5: Observe detection-data changes** |
|
|
| ```javascript |
| // MutationObserver on hidden detection-data textbox |
| const detObserver = new MutationObserver(() => { |
| const el = document.querySelector('#detection-data input, #detection-data textarea'); |
| if (!el) return; |
| try { |
| const dets = JSON.parse(el.value || "[]"); |
| drawDetections(dets); |
| } catch(e) {} |
| }); |
| ``` |
|
|
| ### What You Need to Approve |
| - Box style: full rectangle? corner brackets? with fill? |
| - Corner brackets look more "professional AI" (think Tesla vision display) |
| - Colors: class-based (above) or all same color? |
| - Always show boxes or only on trigger detections? |
|
|
| ### Testing Sequence |
| 1. Activate → point camera at a person → blue box appears with "person 87%" |
| 2. Point at a car → red box "vehicle 92%" |
| 3. Empty room → no boxes |
| 4. Verify boxes scale correctly when window is resized |
|
|
| --- |
|
|
| ## UPGRADE 5: Kokoro TTS Streaming |
|
|
| **Goal:** First word of alert audio plays within 300ms instead of waiting 2+ seconds for full generation. |
| **Files touched:** `kokoro_tts.py` (~40 lines), `app.py` (~20 lines). |
| **Dependency:** None. |
|
|
| ### Architecture |
|
|
| ``` |
| CURRENT: |
| speaker.speak(full_text) → Kokoro generates complete WAV → base64 encode → return |
| Latency: 1.5-3.0 seconds for a 2-sentence alert |
| |
| NEW: |
| Split text into sentences → generate sentence 1 audio → yield immediately |
| → generate sentence 2 audio → yield |
| Client receives sentence 1 audio at ~300ms, starts playing |
| Sentence 2 arrives while sentence 1 is still playing |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Add streaming method to AlertSpeaker** |
|
|
| In `kokoro_tts.py`, add a new method: |
|
|
| ```python |
| async def speak_streaming(self, text: str, level: str = "warning"): |
| """ |
| Generator that yields audio chunks sentence by sentence. |
| Each chunk is a base64-encoded WAV. |
| """ |
| import re |
| sentences = re.split(r'(?<=[.!?])\s+', text.strip()) |
| |
| for sentence in sentences: |
| if not sentence.strip(): |
| continue |
| try: |
| result = await self.speak(sentence, level=level) |
| if result.get("audio_base64"): |
| yield result["audio_base64"] |
| except Exception as e: |
| logger.error("TTS streaming chunk failed", error=str(e)) |
| break |
| ``` |
|
|
| **Step 2: Modify app.py to use streaming** |
|
|
| Instead of generating full audio then embedding in HTML, use Gradio's streaming audio: |
|
|
| ```python |
| # Replace the current TTS section in process_frame with: |
| audio_chunks = [] |
| if alert_level in ["critical", "warning"]: |
| async for chunk_b64 in speaker.speak_streaming(alert_text, level=alert_level): |
| audio_chunks.append(chunk_b64) |
| |
| # Use first chunk for immediate playback, concatenate for archive |
| if audio_chunks: |
| audio_data_uri = f"data:audio/wav;base64,{audio_chunks[0]}" |
| # Pass remaining chunks via hidden state for sequential playback |
| ``` |
|
|
| **Step 3: Add sequential audio playback in JS** |
|
|
| In sensor_bridge.py: |
| ```javascript |
| // Queue-based audio player that chains chunks |
| let audioQueue = []; |
| let isPlaying = false; |
| |
| function queueAudio(base64Chunks) { |
| audioQueue = [...base64Chunks]; |
| if (!isPlaying) playNext(); |
| } |
| |
| async function playNext() { |
| if (audioQueue.length === 0) { isPlaying = false; return; } |
| isPlaying = true; |
| const chunk = audioQueue.shift(); |
| const audio = new Audio(`data:audio/wav;base64,${chunk}`); |
| audio.onended = () => playNext(); |
| await audio.play(); |
| } |
| ``` |
| |
| ### Edge Cases |
| - **Single sentence alerts:** Works normally, one chunk, no difference from current. |
| - **Kokoro fails on sentence 2:** Graceful stop — sentence 1 already played. |
| - **User triggers new alert while audio playing:** Clear queue, start new alert audio. |
| |
| ### Testing Sequence |
| 1. Trigger "Fall Detected" scenario (multi-sentence alert) |
| 2. Note: first word plays within ~300ms |
| 3. Second sentence plays seamlessly after first |
| 4. Trigger another alert mid-playback → old audio stops, new starts |
| |
| --- |
| |
| ## UPGRADE 6: IMU-Linked Glassmorphic UI |
| |
| **Goal:** Glass panels shift highlights and shadows based on phone tilt. Feels like real glass. |
| **Files touched:** `sensor_bridge.py` (~40 lines JS), `app.py` CUSTOM_CSS (~15 lines). |
| **Dependency:** None. This is the simplest visual upgrade. |
| |
| ### Architecture |
| |
| ``` |
| deviceorientation event (60 FPS) → |
| CSS custom properties on document.documentElement: |
| --tilt-x: gamma * 0.3 (left-right) |
| --tilt-y: beta * 0.3 (front-back) |
| |
| CSS uses these properties: |
| .glass-panel::before { |
| background: radial-gradient( |
| circle at calc(50% + var(--tilt-x) * 1px) calc(50% + var(--tilt-y) * 1px), |
| rgba(255,255,255,0.08) 0%, |
| transparent 60% |
| ); |
| } |
| .glass-panel { |
| box-shadow: |
| calc(var(--tilt-x) * -0.5px) calc(var(--tilt-y) * -0.5px) 20px rgba(0,0,0,0.3); |
| } |
| ``` |
| |
| ### Implementation Plan |
| |
| **Step 1: Add CSS custom properties to CUSTOM_CSS in app.py** |
| |
| ```css |
| :root { |
| --tilt-x: 0; |
| --tilt-y: 0; |
| } |
| .glass-panel { |
| position: relative; |
| overflow: hidden; |
| transition: box-shadow 0.15s ease-out; |
| box-shadow: |
| calc(var(--tilt-x) * -0.5px) |
| calc(var(--tilt-y) * -0.5px) |
| 24px rgba(0,0,0,0.35); |
| } |
| .glass-panel::before { |
| content: ''; |
| position: absolute; |
| top: 0; left: 0; right: 0; bottom: 0; |
| background: radial-gradient( |
| circle at calc(50% + var(--tilt-x) * 2px) calc(30% + var(--tilt-y) * 2px), |
| rgba(255,255,255,0.07) 0%, |
| transparent 50% |
| ); |
| pointer-events: none; |
| z-index: 1; |
| transition: background 0.15s ease-out; |
| } |
| ``` |
| |
| **Step 2: Add JS tilt handler to sensor_bridge.py** |
| |
| ```javascript |
| // Inside IIFE, add alongside existing deviceorientation handler: |
| const MAX_TILT = 25; // cap at ±25px shift |
| function handleGlassTilt(event) { |
| const gamma = Math.max(-30, Math.min(30, event.gamma || 0)); |
| const beta = Math.max(-30, Math.min(30, (event.beta || 45) - 45)); |
| document.documentElement.style.setProperty('--tilt-x', gamma * 0.8); |
| document.documentElement.style.setProperty('--tilt-y', beta * 0.8); |
| } |
| window.addEventListener('deviceorientation', handleGlassTilt); |
| ``` |
| |
| ### What You Need to Approve |
| - Tilt sensitivity: 0.8 multiplier (moderate)? 0.3 (subtle)? 1.5 (dramatic)? |
| - Highlight intensity: 0.07 opacity (shown above)? Higher = more visible but less classy |
| - Desktop fallback: track mouse position instead of gyroscope? Or no effect on desktop? |
|
|
| ### Testing Sequence |
| 1. Open on phone → panels look normal |
| 2. Tilt phone left → highlight slides right, shadow shifts left |
| 3. Tilt forward → highlight moves up, shadow moves down |
| 4. Looks like a physical piece of glass catching light |
| 5. On desktop with no gyro → either static or mouse-follows |
|
|
| --- |
|
|
| ## UPGRADE 7: Live Cost Race Ticker |
|
|
| **Goal:** Split-screen widget showing Sentinel vs GPT-4o costs racing in real-time. |
| **Files touched:** `sensor_bridge.py` (~100 lines JS/HTML), `app.py` (~20 lines). |
| **Dependency:** None. |
|
|
| ### Architecture |
|
|
| ``` |
| JS reads cost_tracker data from hidden Gradio components: |
| - cost-total (existing) → Sentinel's actual cost |
| - frames-processed (calculate from frame_count_state) |
| |
| Calculates naive GPT-4o cost: |
| frames_processed * $0.01 per frame (GPT-4o Vision at 2 FPS) |
| |
| Displays two animated counters: |
| LEFT: Sentinel $0.0023 (green, slow counter) |
| RIGHT: GPT-4o $1.4200 (red, fast counter) |
| BOTTOM: "You saved $1.42 (99.8%)" |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Add cost ticker HTML to Monitor tab** |
|
|
| In sensor_bridge.py, add HTML that gets injected: |
| ```html |
| <div id="cost-ticker" style=" |
| display: none; |
| margin-top: 12px; |
| padding: 14px; |
| background: rgba(0,0,0,0.3); |
| border-radius: 12px; |
| border: 1px solid rgba(255,255,255,0.08); |
| font-family: monospace; |
| "> |
| <div style="display: flex; justify-content: space-between; align-items: center;"> |
| <div style="text-align: center;"> |
| <div style="color: #4ade80; font-size: 11px;">SENTINEL</div> |
| <div id="ticker-sentinel" style="color: #4ade80; font-size: 22px; font-weight: bold;">$0.0000</div> |
| </div> |
| <div style="color: #6b7280; font-size: 20px;">vs</div> |
| <div style="text-align: center;"> |
| <div style="color: #ef4444; font-size: 11px;">GPT-4o VISION</div> |
| <div id="ticker-gpt4o" style="color: #ef4444; font-size: 22px; font-weight: bold;">$0.0000</div> |
| </div> |
| </div> |
| <div id="ticker-savings" style=" |
| text-align: center; |
| margin-top: 8px; |
| color: #a5b4fc; |
| font-size: 13px; |
| ">Savings: $0.00 (0%)</div> |
| <div style=" |
| height: 4px; |
| background: rgba(255,255,255,0.1); |
| border-radius: 2px; |
| margin-top: 8px; |
| overflow: hidden; |
| "> |
| <div id="ticker-bar" style=" |
| height: 100%; |
| background: linear-gradient(90deg, #4ade80, #22d3ee); |
| border-radius: 2px; |
| width: 0%; |
| transition: width 0.5s ease-out; |
| "></div> |
| </div> |
| </div> |
| ``` |
| |
| **Step 2: Add animation JS** |
| |
| ```javascript |
| // Inside IIFE: |
| const GPT4O_COST_PER_FRAME = 0.01; // $0.01 per frame at 2 FPS |
| let displayedSentinel = 0; |
| let displayedGPT4o = 0; |
|
|
| function updateCostTicker() { |
| if (!tickerActive) return; |
| |
| // Read actual values from Gradio state |
| const sentinelCost = parseFloat( |
| document.querySelector('#cost-total input')?.value || '0' |
| ); |
| const framesProcessed = parseInt( |
| document.querySelector('#frame-count input')?.value || '0' |
| ); |
| const gpt4oCost = framesProcessed * GPT4O_COST_PER_FRAME; |
| |
| // Animate numbers toward target (easing) |
| displayedSentinel += (sentinelCost - displayedSentinel) * 0.15; |
| displayedGPT4o += (gpt4oCost - displayedGPT4o) * 0.15; |
| |
| document.getElementById("ticker-sentinel").textContent = |
| `$${displayedSentinel.toFixed(4)}`; |
| document.getElementById("ticker-gpt4o").textContent = |
| `$${displayedGPT4o.toFixed(2)}`; |
| |
| const savings = displayedGPT4o - displayedSentinel; |
| const savingsPct = displayedGPT4o > 0 ? (savings / displayedGPT4o * 100) : 0; |
| document.getElementById("ticker-savings").textContent = |
| `You saved $${savings.toFixed(2)} (${savingsPct.toFixed(1)}%)`; |
| |
| // Update progress bar |
| const barWidth = Math.min(100, savingsPct); |
| document.getElementById("ticker-bar").style.width = `${barWidth}%`; |
| } |
| |
| // Run at 30 FPS for smooth animation |
| setInterval(updateCostTicker, 33); |
| ``` |
| |
| **Step 3: Expose frame count to JS** |
| |
| Add hidden component in app.py: |
| ```python |
| frame_count_display = gr.Number(value=0, visible=False, elem_id="frame-count") |
| ``` |
| |
| Update it in process_frame return or via the existing frame_count_state. |
|
|
| **Step 4: Show ticker on activation** |
|
|
| In activation handler: `document.getElementById("cost-ticker").style.display = "block";` |
|
|
| ### What You Need to Approve |
| - GPT-4o cost rate: $0.01/frame is approximate. Use different number? |
| - Number animation: smooth easing (shown) or instant snap? |
| - Should there be confetti when savings hits $1.00? |
| - Placement: inside glass-panel below camera? or separate section? |
|
|
| ### Testing Sequence |
| 1. Activate → ticker shows $0.00 vs $0.00 |
| 2. Run 5 simulation scenarios → Sentinel: ~$0.0015 vs GPT-4o: ~$0.05 |
| 3. Watch numbers animate smoothly |
| 4. Progress bar fills as savings grow |
| 5. After 50 frames: "$0.49 saved (99.8%)" — powerful demo moment |
|
|
| --- |
|
|
| ## UPGRADE 8: Multi-Turn VLM Memory |
|
|
| **Goal:** Nemotron sees last 3 alerts and reasons about trends. "Person was 5m away, now 1m." |
| **Files touched:** `app.py` only (~30 lines). |
| **Dependency:** None. |
|
|
| ### Architecture |
|
|
| ``` |
| NEW gr.State: vlm_memory = gr.State(value=[]) |
| |
| On each VLM call: |
| 1. Append current alert context to vlm_memory (max 3 entries) |
| 2. Format memory into system prompt: |
| "Previous alerts in this session: |
| [10:05:01] WARNING: Person detected 5m away |
| [10:05:11] WARNING: Person now 3m away, approaching |
| [current] Analyze this new frame..." |
| 3. Nemotron receives history + current frame → can reference trends |
| 4. Trim memory to last 3 entries (sliding window) |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Add memory state** |
|
|
| In app.py, with other gr.State declarations: |
| ```python |
| vlm_memory = gr.State(value=[]) # List of dicts: {time, level, text} |
| ``` |
|
|
| **Step 2: Update SYSTEM_PROMPT** |
| |
| Change from static string to a function: |
| ```python |
| def build_system_prompt(memory: list) -> str: |
| history_block = "" |
| if memory: |
| lines = [f" [{m['time']}] {m['level']}: {m['text']}" for m in memory[-3:]] |
| history_block = "\n\nPrevious alerts in this session (use for trend awareness):\n" + "\n".join(lines) |
| |
| return f"""You are Sentinel, an autonomous AI guardian for visually impaired and elderly users. |
| You receive visual descriptions and sensor data. Your job is to: |
| 1. Identify potential dangers (tripping hazards, approaching vehicles, strangers, fire) |
| 2. Provide navigation guidance (door ahead, stairs, obstacles) |
| 3. Alert ONLY when genuinely dangerous — avoid false alarms |
| 4. Respond in 1-2 sentences maximum (user hears this via TTS) |
| 5. Reference previous alerts when relevant (e.g., "Person getting closer") |
| 6. Escalate if a threat is worsening over time |
| |
| Format: [LEVEL] message |
| Where LEVEL is: CRITICAL, WARNING, or OK{history_block}""" |
| ``` |
| |
| **Step 3: Update process_frame** |
| |
| After receiving VLM response and parsing alert: |
| ```python |
| # Append to memory |
| if alert_level in ["critical", "warning"]: |
| memory_entry = { |
| "time": time.strftime("%H:%M:%S"), |
| "level": alert_level.upper(), |
| "text": alert_text[:80] # Truncate to save tokens |
| } |
| vlm_memory.append(memory_entry) |
| vlm_memory = vlm_memory[-3:] # Keep last 3 only |
| |
| # Use dynamic prompt: |
| question = f"Analyze this scene. Context: {sensor_context}" |
| # Pass build_system_prompt(vlm_memory) instead of static SYSTEM_PROMPT |
| ``` |
| |
| **Step 4: Update function signature and event bindings** |
| |
| Add `vlm_memory` to process_frame params (now 21 params, 15 returns, 9 gr.State). |
| Update all return paths and image_data.change bindings accordingly. |
|
|
| ### Token Budget |
| - Each memory entry: ~20 tokens |
| - 3 entries: ~60 extra tokens per VLM call |
| - Negligible cost increase, significant intelligence increase |
|
|
| ### Testing Sequence |
| 1. Trigger "Person Approaching" 3 times in sequence |
| 2. First alert: "Person detected 5m away" |
| 3. Second alert: "Person closer, now 3m" (should reference previous) |
| 4. Third alert: "CRITICAL: Person very close, 1m away — was 5m moments ago" |
| 5. Verify memory shows in alert history with trend references |
|
|
| --- |
|
|
| ## UPGRADE 9: Virtual Walk Mode |
|
|
| **Goal:** Pre-recorded walking video plays. Sentinel analyzes it live. Alerts interrupt the video. |
| **Files touched:** `app.py` (~80 lines new tab), `sensor_bridge.py` (~60 lines JS). |
| **Dependency:** Upgrade 3 (spatial audio) recommended for full experience. |
|
|
| ### Architecture |
|
|
| ``` |
| New Tab: "🎬 Demo" |
| |
| Components: |
| - gr.Video (pre-loaded walking video) |
| - gr.HTML (alert overlay zone, positioned over video) |
| - Play/Pause button |
| - Speed control (0.5x, 1x, 2x) |
| |
| JS Frame Extractor: |
| - requestVideoFrameCallback on video element |
| - Every 1000ms (1 FPS): draw video frame to canvas |
| - Convert to base64 |
| - Inject into image-data textbox → triggers process_frame pipeline |
| |
| Alert Overlay: |
| - When alert generated, HTML overlay appears ON TOP of video |
| - Semi-transparent banner with alert text |
| - Auto-dismisses after 5 seconds |
| - Audio plays via spatial audio (Upgrade 3) or regular playback |
| |
| Data Flow: |
| video.play() → JS extracts frame every 1s → base64 → image-data |
| → process_frame → VLM → alert → overlay on video + TTS |
| → video continues playing underneath |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Create Demo tab in app.py** |
|
|
| ```python |
| with gr.Tab("🎬 Demo"): |
| with gr.Column(elem_classes="glass-panel"): |
| gr.Markdown("### Virtual Walk Mode") |
| gr.Markdown("Watch Sentinel analyze a real walking scenario in real-time.") |
| |
| demo_video = gr.Video( |
| label="Walking Scenario", |
| value=None, # User uploads or pre-load |
| autoplay=False, |
| elem_id="demo-video" |
| ) |
| |
| # Pre-loaded scenario buttons |
| with gr.Row(): |
| gr.Markdown("**Or try a preset scenario:**") |
| with gr.Row(): |
| demo_stairs_btn = gr.Button("Stairs", size="sm") |
| demo_traffic_btn = gr.Button("Traffic", size="sm") |
| demo_park_btn = gr.Button("Park Walk", size="sm") |
| |
| demo_alert_overlay = gr.HTML( |
| "", |
| elem_id="demo-alert-overlay" |
| ) |
| |
| demo_status = gr.HTML( |
| "<div style='text-align:center;color:#9ca3af;'>Press Play to start the virtual walk</div>", |
| elem_id="demo-status" |
| ) |
| ``` |
|
|
| **Step 2: JS frame extractor in sensor_bridge.py** |
| |
| ```javascript |
| function initDemoMode() { |
| const video = document.querySelector('#demo-video video'); |
| if (!video) return; |
| |
| let demoCanvas = document.createElement('canvas'); |
| demoCanvas.width = 640; |
| demoCanvas.height = 480; |
| let demoCtx = demoCanvas.getContext('2d'); |
| let lastDemoFrame = 0; |
| |
| video.addEventListener('play', () => { |
| // Activate monitoring for demo mode |
| setGradioValue("sentinel-active-state", "true"); |
| }); |
| |
| function extractDemoFrame(timestamp) { |
| if (video.paused || video.ended) return; |
| |
| if (timestamp - lastDemoFrame >= 1000) { // 1 FPS |
| lastDemoFrame = timestamp; |
| demoCtx.drawImage(video, 0, 0, 640, 480); |
| const base64 = demoCanvas.toDataURL("image/jpeg", 0.7).split(",")[1]; |
| setGradioValue("image-data", base64); |
| } |
| requestAnimationFrame(extractDemoFrame); |
| } |
| |
| video.addEventListener('play', () => { |
| requestAnimationFrame(extractDemoFrame); |
| }); |
| } |
| ``` |
| |
| **Step 3: Alert overlay CSS** |
| |
| ```css |
| #demo-alert-overlay > div { |
| position: absolute; |
| bottom: 80px; |
| left: 50%; |
| transform: translateX(-50%); |
| z-index: 100; |
| max-width: 90%; |
| animation: slideUp 0.3s ease-out; |
| } |
| @keyframes slideUp { |
| from { opacity: 0; transform: translateX(-50%) translateY(20px); } |
| to { opacity: 1; transform: translateX(-50%) translateY(0); } |
| } |
| ``` |
| |
| ### What You Need to Decide |
| - Video source: Film your own walking video? Use Creative Commons? Use a slideshow of sim images instead? |
| - Frame rate: 1 FPS (1 call/sec, cheap) or 2 FPS? |
| - Should alerts PAUSE the video or overlay while playing? |
| - Preset videos: record 3 short clips (stairs, traffic, park) and embed? |
| |
| ### Testing Sequence |
| 1. Upload walking video to Demo tab |
| 2. Press Play → video starts, frame counter ticks |
| 3. When person crosses path → alert banner slides up over video + TTS plays |
| 4. When stairs appear → CRITICAL alert + loud warning |
| 5. Clear section → "OK: Path clear" in info banner |
| |
| --- |
| |
| ## UPGRADE 10: Web Speech API Panic Override |
| |
| **Goal:** Browser listens for "HELP"/"STOP" keywords. Works offline. Triggers emergency mode. |
| **Files touched:** `sensor_bridge.py` (~50 lines JS). |
| **Dependency:** None. |
|
|
| ### Architecture |
|
|
| ``` |
| webkitSpeechRecognition runs continuously in browser: |
| - Listens for keywords: "help", "stop", "emergency", "cancel" |
| - On match: |
| 1. Set monitoring_active to true (via hidden component) |
| 2. Inject CRITICAL alert into alert banner |
| 3. Play emergency siren sound (Web Audio oscillator) |
| 4. Open SMS app with pre-filled GPS + "EMERGENCY" text |
| - Works even if server is unreachable |
| - iOS requires user gesture to start (use activation button) |
| ``` |
|
|
| ### Implementation Plan |
|
|
| **Step 1: Add speech recognition to sensor_bridge.py IIFE** |
| |
| ```javascript |
| // Inside IIFE: |
| let recognition = null; |
| const PANIC_KEYWORDS = ["help", "stop", "emergency", "cancel", "sos"]; |
| |
| function initSpeechPanic() { |
| const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; |
| if (!SpeechRecognition) { |
| console.log("Web Speech API not available. Panic override disabled."); |
| return; |
| } |
| |
| recognition = new SpeechRecognition(); |
| recognition.continuous = true; |
| recognition.interimResults = true; |
| recognition.lang = 'en-US'; |
| |
| recognition.onresult = (event) => { |
| const last = event.results[event.results.length - 1]; |
| const transcript = last[0].transcript.toLowerCase().trim(); |
| |
| for (const keyword of PANIC_KEYWORDS) { |
| if (transcript.includes(keyword)) { |
| triggerPanicMode(keyword); |
| break; |
| } |
| } |
| }; |
| |
| recognition.onerror = (event) => { |
| // Auto-restart on error (except if user stopped it) |
| if (event.error !== 'aborted' && isActivated) { |
| setTimeout(() => { try { recognition.start(); } catch(e) {} }, 1000); |
| } |
| }; |
| |
| recognition.onend = () => { |
| // Auto-restart if still active |
| if (isActivated) { |
| try { recognition.start(); } catch(e) {} |
| } |
| }; |
| } |
| |
| function triggerPanicMode(keyword) { |
| console.log(`PANIC KEYWORD DETECTED: "${keyword}"`); |
| |
| // 1. Set alert banner to critical |
| const banner = document.getElementById("alert-banner-container"); |
| if (banner) { |
| banner.innerHTML = ` |
| <div class="alert-banner-critical" style="animation: pulse 0.5s infinite;"> |
| 🚨 EMERGENCY MODE ACTIVATED<br> |
| Voice command: "${keyword.toUpperCase()}"<br> |
| <small>GPS coordinates being sent to emergency contact</small> |
| </div>`; |
| } |
| |
| // 2. Play emergency siren (Web Audio oscillator) |
| playEmergencySiren(); |
| |
| // 3. Open SMS with pre-filled message |
| const lat = document.querySelector('#gps-lat input')?.value || 'unknown'; |
| const lon = document.querySelector('#gps-lon input')?.value || 'unknown'; |
| const smsBody = encodeURIComponent( |
| `SENTINEL EMERGENCY: User triggered panic mode. Last known location: ${lat}, ${lon}. Please check on them.` |
| ); |
| window.open(`sms:?body=${smsBody}`, '_blank'); |
| } |
| |
| function playEmergencySiren() { |
| const ctx = new (window.AudioContext || window.webkitAudioContext)(); |
| const osc = ctx.createOscillator(); |
| const gain = ctx.createGain(); |
| osc.type = 'sawtooth'; |
| osc.frequency.value = 800; |
| gain.gain.value = 0.3; |
| osc.connect(gain); |
| gain.connect(ctx.destination); |
| osc.start(); |
| // Modulate frequency for siren effect |
| osc.frequency.setValueAtTime(800, ctx.currentTime); |
| osc.frequency.linearRampToValueAtTime(1200, ctx.currentTime + 0.5); |
| osc.frequency.linearRampToValueAtTime(800, ctx.currentTime + 1.0); |
| // Auto-stop after 3 seconds |
| gain.gain.linearRampToValueAtTime(0, ctx.currentTime + 3.0); |
| osc.stop(ctx.currentTime + 3.0); |
| } |
| ``` |
| |
| **Step 2: Start recognition on activation** |
|
|
| In `startAllSensors()`, add: `initSpeechPanic(); if (recognition) try { recognition.start(); } catch(e) {}` |
|
|
| In `stopAllSensors()`, add: `if (recognition) try { recognition.stop(); } catch(e) {}` |
|
|
| ### Edge Cases |
| - **iOS Safari:** Requires user gesture. Since activation button is the user gesture, speech recognition starts after clicking ACTIVATE. |
| - **Background noise false triggers:** Use `interimResults: true` but only trigger on high-confidence matches. Could add confidence threshold: `if (last[0].confidence > 0.7)`. |
| - **Browser doesn't support:** Graceful log message, no crash. |
|
|
| ### Testing Sequence |
| 1. Activate Sentinel |
| 2. Say "Help" clearly → banner turns red, siren plays, SMS app opens |
| 3. Say "Stop" → same emergency response |
| 4. Normal conversation → no false triggers |
| 5. Test with server offline → still works (pure browser-side) |
|
|
| --- |
|
|
| ## BUILD ORDER & TIMELINE |
|
|
| ### Day 1 — Deploy + Visual Wins (6 hours) |
|
|
| | Time | Upgrade | Why First | |
| |------|---------|-----------| |
| | 1h | Deploy Modal + HF Space | Everything depends on this | |
| | 1h | Upgrade 6: IMU Glass UI | Simplest visual upgrade, 40 lines | |
| | 1h | Upgrade 7: Cost Race Ticker | Most persuasive demo element | |
| | 1h | Upgrade 4: SVG Bounding Boxes | Camera feed looks like real AI | |
| | 1h | Upgrade 2: Gyroscope HUD | Premium app feel | |
| | 1h | Quick wins (tab badges, keyboard shortcuts) | Polish | |
|
|
| ### Day 2 — Intelligence + Audio (5 hours) |
|
|
| | Time | Upgrade | Why This Order | |
| |------|---------|----------------| |
| | 1h | Upgrade 8: Multi-Turn VLM Memory | Pure app.py change, smart | |
| | 1h | Upgrade 1: OpenCV.js Frame Diff | Architecture proof | |
| | 1.5h | Upgrade 3: Spatial Audio | Demo-defining feature | |
| | 1h | Upgrade 5: TTS Streaming | Pairs with spatial audio | |
| | 0.5h | Upgrade 10: Speech Panic Override | Simple, impressive | |
|
|
| ### Day 3 — Killer Demo + Submission (5 hours) |
|
|
| | Time | Task | Notes | |
| |------|------|-------| |
| | 1h | Upgrade 9: Virtual Walk Mode | Record walking video first | |
| | 1h | Record demo video | Show all upgrades working together | |
| | 1h | HF Dataset (500 rows) | Export from simulation + demo runs | |
| | 1h | Space README + thumbnail | Architecture diagram, screenshots | |
| | 1h | Submit + blog post | Link everything | |
|
|
| --- |
|
|
| ## DEPENDENCY MAP |
|
|
| ``` |
| No dependencies: |
| Upgrade 2 (HUD) ──────────────────────┐ |
| Upgrade 6 (Glass UI) ────────────────┤ |
| Upgrade 7 (Cost Ticker) ─────────────┤ |
| Upgrade 8 (VLM Memory) ─────────────┤ |
| Upgrade 10 (Speech Panic) ───────────┤ |
| │ |
| Optional dependencies: │ |
| Upgrade 1 (Frame Diff) → Upgrade 4 (SVG Boxes) |
| Upgrade 3 (Spatial Audio) → Upgrade 9 (Virtual Walk) |
| Upgrade 5 (TTS Stream) → Upgrade 3 (Spatial Audio) |
| │ |
| All feed into: ─────────────────────────┘ |
| Upgrade 9 (Virtual Walk) ← uses everything |
| ``` |
|
|
| --- |
|
|
| ## WHAT YOU APPROVE BEFORE AGENT BUILDS |
|
|
| For each upgrade, tell the agent: |
|
|
| 1. **"Build Upgrade X"** — agent starts |
| 2. **"Use approach Y"** — if there are options (like HUD Option A vs B) |
| 3. **"Reference: [screenshot/description]"** — for visual upgrades, describe what you want |
| 4. **"Skip the [thing I don't want]"** — agent trims unnecessary parts |
|
|
| The agent has all the architecture. It just needs your green light on style choices. |
|
|
