# 🛡️ Sentinel — Deep Implementation Playbook
## 10 Upgrades: Fully Architected, Battle-Planned

**Rule:** Agent does all the thinking. You review the plan, approve or redirect, then agent executes.
**Rule:** Visual/UI work requires you to approve a reference style before agent writes any code.

---

## UPGRADE 1: Client-Side Frame Differencing (OpenCV.js)

**Goal:** Browser analyzes frames locally, only uploads to server when motion detected.
**Files touched:** `sensor_bridge.py` only.
**New lines:** ~80 lines of JavaScript.

### Architecture

```
CURRENT FLOW (wasteful):
  Camera → canvas 320x240 → base64 JPEG → image-data textbox → server process_frame
  Rate: 2 FPS always = 120 uploads/minute

NEW FLOW (smart):
  Camera → canvas 320x240 → OpenCV.js cv.absdiff(prev, current) →
  if changed_pixels > threshold → upload base64 to server
  if not → skip upload, increment local counter
  Rate: ~0.3-0.8 FPS average = 18-48 uploads/minute
```

### Implementation Plan

**Step 1: Add OpenCV.js CDN load to sensor_bridge.py**

Insert at the top of the HTML string (before the IIFE):
```html
<script async src="https://docs.opencv.org/4.x/opencv.js" 
        onload="window.__cvReady = true;" type="text/javascript"></script>
```

**Step 2: Add state variables inside the IIFE**

```javascript
let prevMat = null;        // Previous frame as cv.Mat
let cvReady = false;       // OpenCV.js loaded flag
let skippedFrames = 0;     // Count of locally-filtered frames
const DIFF_THRESHOLD = 0.12; // 12% pixel change = significant
```

**Step 3: Modify `captureFrame()` function**

The current `captureFrame()` always uploads. New version:

```
captureFrame():
  1. Draw video to canvas (same as now)
  2. If cvReady:
     a. Read canvas into cv.Mat (srcMat)
     b. Convert to grayscale (cv.cvtColor)
     c. If prevMat exists:
        - cv.absdiff(prevMat, srcMat, diffMat)
        - cv.threshold(diffMat, threshMat, 25, 255, cv.THRESH_BINARY)
        - changedPct = cv.countNonZero(threshMat) / (width * height)
        - If changedPct < DIFF_THRESHOLD:
            skippedFrames++
            update UI indicator ("Filtered locally: X frames")
            cleanup Mats, return (DON'T upload)
     d. prevMat = srcMat.clone()
     e. cleanup temp Mats
  3. Proceed with base64 upload (same as now)
```

**Step 4: Add skipped frame counter to UI**

Add a hidden Gradio Number component:
```python
# In app.py Monitor tab, alongside other hidden numbers:
skipped_frames = gr.Number(value=0, visible=False, elem_id="skipped-frames")
```

JS updates it via `setGradioValue("skipped-frames", skippedFrames)` every 2 seconds.

### Edge Cases
- **OpenCV.js fails to load (CDN blocked):** Fall back to current behavior (always upload). Check `window.__cvReady` before using cv.
- **First frame:** No prevMat exists → always upload first frame, store as prevMat.
- **Mat memory leak:** Always call `.delete()` on temporary Mats. Only keep prevMat alive.
- **Camera off:** Reset prevMat to null when camera stops.

### Why This Wins
The 240x cost story goes from "trust our architecture" to "watch the counter — 847 frames processed locally, only 4 sent to GPU." That's a demo moment.

### Testing Sequence
1. Deploy with upgrade → activate → wave hand in front of camera → see uploads spike
2. Sit still → see uploads drop to near zero
3. Check that simulation mode still works (it bypasses camera entirely)

---

## UPGRADE 2: Three.js / CSS3D Gyroscope HUD

**Goal:** Floating glass HUD overlay on camera feed. Tilts with phone. Shows live telemetry.
**Files touched:** `sensor_bridge.py` (~200 lines JS/CSS).
**Dependency:** None (pure client-side).

### Architecture Decision

**Option A — Pure CSS (Recommended for reliability):**
- No external library
- CSS `transform: perspective(1000px) rotateX() rotateY()` driven by `deviceorientation`
- ~80 lines, impossible to break
- Looks great but not "3D"

**Option B — Three.js CSS3DRenderer (Recommended for wow):**
- Load three.js from CDN (~150KB)
- Create CSS3DObject from HUD HTML elements
- Rotate entire CSS3D scene based on device orientation
- ~200 lines, can break if CDN fails
- True 3D parallax effect

### My Recommendation: Option A for reliability, Option B for demo video

Start with Option A. It works on every device, including iOS. If you have time, add Option B as an enhancement.

### Implementation Plan (Option A — Pure CSS)

**Step 1: Add HUD container HTML to sensor_bridge.py**

```html
<div id="sentinel-hud" style="
    position: fixed;
    top: 80px;
    left: 50%;
    transform: translateX(-50%);
    z-index: 1000;
    pointer-events: none;
    transition: transform 0.1s ease-out;
    perspective: 800px;
    display: none;
">
    <div id="hud-inner" style="
        background: rgba(255,255,255,0.06);
        border: 1px solid rgba(255,255,255,0.15);
        border-radius: 16px;
        padding: 12px 20px;
        backdrop-filter: blur(20px);
        min-width: 240px;
        font-family: monospace;
        color: #a5b4fc;
    ">
        <div id="hud-compass">Heading: 0° N</div>
        <div id="hud-threat" style="color: #4ade80;">Threat: CLEAR</div>
        <div id="hud-fps">FPS: 2.0 | Filtered: 0</div>
        <div id="hud-battery">Battery: 100%</div>
    </div>
</div>
```

**Step 2: Add deviceorientation handler**

```javascript
// Inside IIFE, after existing sensor handlers:
let hudActive = false;
const hudEl = document.getElementById("sentinel-hud");
const hudInner = document.getElementById("hud-inner");

function handleHUDOrientation(event) {
    if (!hudActive || !hudInner) return;
    // beta = front-back tilt (-180 to 180)
    // gamma = left-right tilt (-90 to 90)
    const tiltX = Math.max(-8, Math.min(8, event.gamma * 0.15));
    const tiltY = Math.max(-8, Math.min(8, (event.beta - 45) * 0.15));
    hudInner.style.transform = 
        `perspective(800px) rotateX(${-tiltY}deg) rotateY(${tiltX}deg)`;
}
window.addEventListener('deviceorientation', handleHUDOrientation);
```

**Step 3: Update HUD data on each sensor tick**

The existing sensor handlers already have access to heading, battery, etc. Add DOM updates:

```javascript
// In the existing telemetry update cycle:
if (hudActive) {
    document.getElementById("hud-compass").textContent = 
        `Heading: ${heading}° ${getCompassDir(heading)}`;
    document.getElementById("hud-battery").textContent = 
        `Battery: ${batteryPct}%`;
    document.getElementById("hud-threat").textContent = 
        `Threat: ${currentThreatLevel}`;
    document.getElementById("hud-threat").style.color = 
        currentThreatLevel === "CLEAR" ? "#4ade80" : 
        currentThreatLevel === "WARNING" ? "#fbbf24" : "#ef4444";
}
```

**Step 4: Show HUD on activation**

In `bindActivationButton()`, after starting sensors:
```javascript
if (hudEl) { hudEl.style.display = "block"; hudActive = true; }
```

### What You Need to Approve
- HUD position: top-center? bottom? full-width strip?
- Data fields: compass + threat + FPS + battery? Or different data?
- Glass style: subtle (0.06 opacity) or more visible (0.12)?
- Color scheme: indigo text (#a5b4fc) or white?

### Testing Sequence
1. Activate on phone → HUD appears floating over camera
2. Tilt phone left → HUD tilts right (parallax)
3. Trigger simulation → HUD threat level changes color
4. Test on iOS → works because it's pure CSS, no WebGL

---

## UPGRADE 3: Binaural 3D Spatial Audio (Web Audio API)

**Goal:** Alert voice sounds like it's coming FROM the direction of danger.
**Files touched:** `sensor_bridge.py` (~100 lines JS), `app.py` (~10 lines modified).
**Dependency:** None (Web Audio API is native to all modern browsers).

### Architecture

```
CURRENT:
  process_frame → alert_html with <audio autoplay src="data:audio/wav;base64,...">
  → browser plays audio from center (no spatial)

NEW:
  process_frame → determines threat_x, threat_y from detection bbox center
  → embeds them as data attributes in alert_html
  → JS intercepts: decodes base64 audio → creates AudioBufferSourceNode
  → connects to PannerNode (HRTF model, positioned at threat_x/y/z)
  → connects to AudioListener (oriented by deviceorientation)
  → audio plays from spatial position
```

### Implementation Plan

**Step 1: Add spatial audio engine to sensor_bridge.py**

```javascript
// Inside IIFE:
let spatialAudioCtx = null;
let audioListener = null;

function initSpatialAudio() {
    spatialAudioCtx = new (window.AudioContext || window.webkitAudioContext)();
    audioListener = spatialAudioCtx.listener;
    // Set listener to face forward
    if (audioListener.positionX) {
        audioListener.positionX.value = 0;
        audioListener.positionY.value = 0;
        audioListener.positionZ.value = 0;
        audioListener.forwardX.value = 0;
        audioListener.forwardY.value = 0;
        audioListener.forwardZ.value = -1;
    }
}

async function playSpatialAudio(base64Audio, threatX, threatY) {
    if (!spatialAudioCtx) initSpatialAudio();
    if (spatialAudioCtx.state === 'suspended') await spatialAudioCtx.resume();

    // Decode base64 WAV to AudioBuffer
    const binary = atob(base64Audio);
    const bytes = new Uint8Array(binary.length);
    for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i);
    const audioBuffer = await spatialAudioCtx.decodeAudioData(bytes.buffer);

    // Create source
    const source = spatialAudioCtx.createBufferSource();
    source.buffer = audioBuffer;

    // Create 3D panner
    const panner = spatialAudioCtx.createPanner();
    panner.panningModel = 'HRTF';
    panner.distanceModel = 'inverse';
    panner.refDistance = 1;
    panner.maxDistance = 10;
    // Position: threatX maps to left(-1) to right(+1), Y to up/down
    panner.positionX.value = threatX * 2;  // -2 to +2 meters
    panner.positionY.value = threatY;
    panner.positionZ.value = -2;  // 2 meters in front

    // Connect: source → panner → destination
    source.connect(panner);
    panner.connect(spatialAudioCtx.destination);
    source.start(0);
}
```

**Step 2: Update deviceorientation to rotate listener**

```javascript
// In existing handleOrientation function, add:
if (spatialAudioCtx && audioListener) {
    const rad = (event.alpha || 0) * Math.PI / 180;
    if (audioListener.forwardX) {
        audioListener.forwardX.value = -Math.sin(rad);
        audioListener.forwardZ.value = -Math.cos(rad);
    }
}
```

**Step 3: Modify app.py alert generation**

In `process_frame`, when building alert_html, embed threat coordinates:

```python
# After YOLO detections, calculate threat position
threat_x = 0.0  # center by default
threat_y = 0.0
if detections:
    # Use center of first trigger detection bbox, normalized to -1..1
    det = detections[0]
    threat_x = ((det.bbox[0] + det.bbox[2]) / 2 / 640 - 0.5) * 2  # -1 to 1
    threat_y = -((det.bbox[1] + det.bbox[3]) / 2 / 480 - 0.5) * 2  # inverted Y

# In alert_html, add data attributes:
alert_html = f"""<div class='alert-banner-{alert_level}' 
    data-threat-x='{threat_x:.2f}' 
    data-threat-y='{threat_y:.2f}'
    data-audio-src='{audio_data_uri}'>
    ⚠️ {alert_text}{alert_image_html}</div>"""
# Remove the <audio autoplay> tag — JS handles playback now
```

**Step 4: JS observer detects new alert and triggers spatial playback**

```javascript
// MutationObserver on alert-banner-container
const alertObserver = new MutationObserver((mutations) => {
    for (const m of mutations) {
        const banner = m.target.querySelector('[data-audio-src]');
        if (banner) {
            const audioSrc = banner.dataset.audioSrc;
            const tx = parseFloat(banner.dataset.threatX || 0);
            const ty = parseFloat(banner.dataset.threatY || 0);
            if (audioSrc && audioSrc.startsWith('data:audio')) {
                const base64 = audioSrc.split(',')[1];
                playSpatialAudio(base64, tx, ty);
            }
        }
    }
});
```

### What You Need to Approve
- Fallback: if Web Audio fails (old browser), fall back to `<audio autoplay>` tag?
- Spatial range: ±2 meters (subtle) or ±5 meters (dramatic)?
- Should listener rotate with gyroscope or stay fixed forward?

### Testing Sequence
1. Put on headphones
2. Trigger "Person Approaching" sim (person comes from left)
3. Voice should sound like it's coming from the LEFT earphone
4. Turn phone 90° → voice position shifts relative to your head
5. Test without headphones → still works but less immersive

---

## UPGRADE 4: Dynamic SVG Bounding Box Overlays

**Goal:** YOLO detection boxes drawn as animated SVG overlays on the camera feed.
**Files touched:** `sensor_bridge.py` (~50 lines JS), `app.py` (~30 lines).
**Dependency:** Upgrade 1 (client-side frame diff) recommended first.

### Architecture

```
CURRENT:
  process_frame → YOLO detect → detections used internally only → discarded
  Camera feed shows raw video with no annotations

NEW:
  process_frame → YOLO detect → serialize detections as JSON →
  write to hidden textbox "detection-data" →
  JS reads JSON → creates/updates SVG overlay on camera feed →
  animated rectangles with class labels
```

### Implementation Plan

**Step 1: Always run YOLO (not just when gatekeeper triggers)**

In `app.py`, change:
```python
# BEFORE (line 563):
if change_result.is_significant and object_detector is not None:

# AFTER:
if object_detector is not None:
```

This means YOLO runs on every frame, not just significant changes.
Cost: ~5ms per frame on CPU. Acceptable for real-time display.
The GATEKEEPER still decides whether to call VLM — this only changes what we SHOW.

**Step 2: Serialize detections to JSON**

After YOLO runs, write detection data:
```python
# After line 565 (yolo_summary = ...):
detection_json = json.dumps([
    {
        "class": d.class_name,
        "conf": round(d.confidence, 2),
        "bbox": list(d.bbox),
        "trigger": d.is_trigger
    }
    for d in detections
])
```

**Step 3: Add hidden textbox for detection data**

In app.py Monitor tab:
```python
detection_data = gr.Textbox(value="[]", visible=False, elem_id="detection-data")
```

Update it in process_frame return tuple (add to outputs).

**Step 4: SVG overlay in sensor_bridge.py**

```javascript
// Create SVG overlay positioned over the camera feed image
function createSVGOverlay() {
    const cameraImg = document.querySelector('#alert-banner-container')
        ?.closest('.glass-panel')
        ?.querySelector('img');  // The camera feed <img> element
    if (!cameraImg) return null;
    
    const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg");
    svg.id = "detection-svg";
    svg.style.cssText = "position:absolute;top:0;left:0;width:100%;height:100%;pointer-events:none;z-index:10;";
    cameraImg.parentElement.style.position = "relative";
    cameraImg.parentElement.appendChild(svg);
    return svg;
}

// Color map for detection classes
const CLASS_COLORS = {
    "person": "#3b82f6",
    "vehicle": "#ef4444", 
    "fire": "#f97316",
    "animal": "#a855f7",
    "default": "#22c55e"
};

function drawDetections(detections) {
    const svg = document.getElementById("detection-svg") || createSVGOverlay();
    if (!svg) return;
    svg.innerHTML = ""; // Clear previous

    detections.forEach(det => {
        const [x1, y1, x2, y2] = det.bbox;
        const color = CLASS_COLORS[det.class] || CLASS_COLORS.default;
        const w = x2 - x1, h = y2 - y1;

        // Animated rectangle
        const rect = document.createElementNS("http://www.w3.org/2000/svg", "rect");
        rect.setAttribute("x", x1); rect.setAttribute("y", y1);
        rect.setAttribute("width", w); rect.setAttribute("height", h);
        rect.setAttribute("fill", "none");
        rect.setAttribute("stroke", color);
        rect.setAttribute("stroke-width", "3");
        rect.setAttribute("rx", "4");
        rect.style.opacity = "0";
        rect.style.transition = "opacity 0.3s";
        svg.appendChild(rect);
        requestAnimationFrame(() => rect.style.opacity = "0.8");

        // Label
        const text = document.createElementNS("http://www.w3.org/2000/svg", "text");
        text.setAttribute("x", x1); text.setAttribute("y", y1 - 6);
        text.setAttribute("fill", color);
        text.setAttribute("font-size", "14");
        text.setAttribute("font-family", "monospace");
        text.textContent = `${det.class} ${(det.conf * 100).toFixed(0)}%`;
        svg.appendChild(text);
    });
}
```

**Step 5: Observe detection-data changes**

```javascript
// MutationObserver on hidden detection-data textbox
const detObserver = new MutationObserver(() => {
    const el = document.querySelector('#detection-data input, #detection-data textarea');
    if (!el) return;
    try {
        const dets = JSON.parse(el.value || "[]");
        drawDetections(dets);
    } catch(e) {}
});
```

### What You Need to Approve
- Box style: full rectangle? corner brackets? with fill? 
- Corner brackets look more "professional AI" (think Tesla vision display)
- Colors: class-based (above) or all same color?
- Always show boxes or only on trigger detections?

### Testing Sequence
1. Activate → point camera at a person → blue box appears with "person 87%"
2. Point at a car → red box "vehicle 92%"
3. Empty room → no boxes
4. Verify boxes scale correctly when window is resized

---

## UPGRADE 5: Kokoro TTS Streaming

**Goal:** First word of alert audio plays within 300ms instead of waiting 2+ seconds for full generation.
**Files touched:** `kokoro_tts.py` (~40 lines), `app.py` (~20 lines).
**Dependency:** None.

### Architecture

```
CURRENT:
  speaker.speak(full_text) → Kokoro generates complete WAV → base64 encode → return
  Latency: 1.5-3.0 seconds for a 2-sentence alert

NEW:
  Split text into sentences → generate sentence 1 audio → yield immediately
  → generate sentence 2 audio → yield
  Client receives sentence 1 audio at ~300ms, starts playing
  Sentence 2 arrives while sentence 1 is still playing
```

### Implementation Plan

**Step 1: Add streaming method to AlertSpeaker**

In `kokoro_tts.py`, add a new method:

```python
async def speak_streaming(self, text: str, level: str = "warning"):
    """
    Generator that yields audio chunks sentence by sentence.
    Each chunk is a base64-encoded WAV.
    """
    import re
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    
    for sentence in sentences:
        if not sentence.strip():
            continue
        try:
            result = await self.speak(sentence, level=level)
            if result.get("audio_base64"):
                yield result["audio_base64"]
        except Exception as e:
            logger.error("TTS streaming chunk failed", error=str(e))
            break
```

**Step 2: Modify app.py to use streaming**

Instead of generating full audio then embedding in HTML, use Gradio's streaming audio:

```python
# Replace the current TTS section in process_frame with:
audio_chunks = []
if alert_level in ["critical", "warning"]:
    async for chunk_b64 in speaker.speak_streaming(alert_text, level=alert_level):
        audio_chunks.append(chunk_b64)
    
    # Use first chunk for immediate playback, concatenate for archive
    if audio_chunks:
        audio_data_uri = f"data:audio/wav;base64,{audio_chunks[0]}"
        # Pass remaining chunks via hidden state for sequential playback
```

**Step 3: Add sequential audio playback in JS**

In sensor_bridge.py:
```javascript
// Queue-based audio player that chains chunks
let audioQueue = [];
let isPlaying = false;

function queueAudio(base64Chunks) {
    audioQueue = [...base64Chunks];
    if (!isPlaying) playNext();
}

async function playNext() {
    if (audioQueue.length === 0) { isPlaying = false; return; }
    isPlaying = true;
    const chunk = audioQueue.shift();
    const audio = new Audio(`data:audio/wav;base64,${chunk}`);
    audio.onended = () => playNext();
    await audio.play();
}
```

### Edge Cases
- **Single sentence alerts:** Works normally, one chunk, no difference from current.
- **Kokoro fails on sentence 2:** Graceful stop — sentence 1 already played.
- **User triggers new alert while audio playing:** Clear queue, start new alert audio.

### Testing Sequence
1. Trigger "Fall Detected" scenario (multi-sentence alert)
2. Note: first word plays within ~300ms
3. Second sentence plays seamlessly after first
4. Trigger another alert mid-playback → old audio stops, new starts

---

## UPGRADE 6: IMU-Linked Glassmorphic UI

**Goal:** Glass panels shift highlights and shadows based on phone tilt. Feels like real glass.
**Files touched:** `sensor_bridge.py` (~40 lines JS), `app.py` CUSTOM_CSS (~15 lines).
**Dependency:** None. This is the simplest visual upgrade.

### Architecture

```
deviceorientation event (60 FPS) →
  CSS custom properties on document.documentElement:
    --tilt-x: gamma * 0.3  (left-right)
    --tilt-y: beta * 0.3   (front-back)
  
CSS uses these properties:
  .glass-panel::before {
    background: radial-gradient(
      circle at calc(50% + var(--tilt-x) * 1px) calc(50% + var(--tilt-y) * 1px),
      rgba(255,255,255,0.08) 0%,
      transparent 60%
    );
  }
  .glass-panel {
    box-shadow: 
      calc(var(--tilt-x) * -0.5px) calc(var(--tilt-y) * -0.5px) 20px rgba(0,0,0,0.3);
  }
```

### Implementation Plan

**Step 1: Add CSS custom properties to CUSTOM_CSS in app.py**

```css
:root {
    --tilt-x: 0;
    --tilt-y: 0;
}
.glass-panel {
    position: relative;
    overflow: hidden;
    transition: box-shadow 0.15s ease-out;
    box-shadow: 
        calc(var(--tilt-x) * -0.5px) 
        calc(var(--tilt-y) * -0.5px) 
        24px rgba(0,0,0,0.35);
}
.glass-panel::before {
    content: '';
    position: absolute;
    top: 0; left: 0; right: 0; bottom: 0;
    background: radial-gradient(
        circle at calc(50% + var(--tilt-x) * 2px) calc(30% + var(--tilt-y) * 2px),
        rgba(255,255,255,0.07) 0%,
        transparent 50%
    );
    pointer-events: none;
    z-index: 1;
    transition: background 0.15s ease-out;
}
```

**Step 2: Add JS tilt handler to sensor_bridge.py**

```javascript
// Inside IIFE, add alongside existing deviceorientation handler:
const MAX_TILT = 25; // cap at ±25px shift
function handleGlassTilt(event) {
    const gamma = Math.max(-30, Math.min(30, event.gamma || 0));
    const beta = Math.max(-30, Math.min(30, (event.beta || 45) - 45));
    document.documentElement.style.setProperty('--tilt-x', gamma * 0.8);
    document.documentElement.style.setProperty('--tilt-y', beta * 0.8);
}
window.addEventListener('deviceorientation', handleGlassTilt);
```

### What You Need to Approve
- Tilt sensitivity: 0.8 multiplier (moderate)? 0.3 (subtle)? 1.5 (dramatic)?
- Highlight intensity: 0.07 opacity (shown above)? Higher = more visible but less classy
- Desktop fallback: track mouse position instead of gyroscope? Or no effect on desktop?

### Testing Sequence
1. Open on phone → panels look normal
2. Tilt phone left → highlight slides right, shadow shifts left
3. Tilt forward → highlight moves up, shadow moves down
4. Looks like a physical piece of glass catching light
5. On desktop with no gyro → either static or mouse-follows

---

## UPGRADE 7: Live Cost Race Ticker

**Goal:** Split-screen widget showing Sentinel vs GPT-4o costs racing in real-time.
**Files touched:** `sensor_bridge.py` (~100 lines JS/HTML), `app.py` (~20 lines).
**Dependency:** None.

### Architecture

```
JS reads cost_tracker data from hidden Gradio components:
  - cost-total (existing) → Sentinel's actual cost
  - frames-processed (calculate from frame_count_state)

Calculates naive GPT-4o cost:
  frames_processed * $0.01 per frame (GPT-4o Vision at 2 FPS)

Displays two animated counters:
  LEFT:  Sentinel $0.0023 (green, slow counter)
  RIGHT: GPT-4o  $1.4200 (red, fast counter)
  BOTTOM: "You saved $1.42 (99.8%)"
```

### Implementation Plan

**Step 1: Add cost ticker HTML to Monitor tab**

In sensor_bridge.py, add HTML that gets injected:
```html
<div id="cost-ticker" style="
    display: none;
    margin-top: 12px;
    padding: 14px;
    background: rgba(0,0,0,0.3);
    border-radius: 12px;
    border: 1px solid rgba(255,255,255,0.08);
    font-family: monospace;
">
    <div style="display: flex; justify-content: space-between; align-items: center;">
        <div style="text-align: center;">
            <div style="color: #4ade80; font-size: 11px;">SENTINEL</div>
            <div id="ticker-sentinel" style="color: #4ade80; font-size: 22px; font-weight: bold;">$0.0000</div>
        </div>
        <div style="color: #6b7280; font-size: 20px;">vs</div>
        <div style="text-align: center;">
            <div style="color: #ef4444; font-size: 11px;">GPT-4o VISION</div>
            <div id="ticker-gpt4o" style="color: #ef4444; font-size: 22px; font-weight: bold;">$0.0000</div>
        </div>
    </div>
    <div id="ticker-savings" style="
        text-align: center;
        margin-top: 8px;
        color: #a5b4fc;
        font-size: 13px;
    ">Savings: $0.00 (0%)</div>
    <div style="
        height: 4px;
        background: rgba(255,255,255,0.1);
        border-radius: 2px;
        margin-top: 8px;
        overflow: hidden;
    ">
        <div id="ticker-bar" style="
            height: 100%;
            background: linear-gradient(90deg, #4ade80, #22d3ee);
            border-radius: 2px;
            width: 0%;
            transition: width 0.5s ease-out;
        "></div>
    </div>
</div>
```

**Step 2: Add animation JS**

```javascript
// Inside IIFE:
const GPT4O_COST_PER_FRAME = 0.01; // $0.01 per frame at 2 FPS
let displayedSentinel = 0;
let displayedGPT4o = 0;

function updateCostTicker() {
    if (!tickerActive) return;
    
    // Read actual values from Gradio state
    const sentinelCost = parseFloat(
        document.querySelector('#cost-total input')?.value || '0'
    );
    const framesProcessed = parseInt(
        document.querySelector('#frame-count input')?.value || '0'
    );
    const gpt4oCost = framesProcessed * GPT4O_COST_PER_FRAME;
    
    // Animate numbers toward target (easing)
    displayedSentinel += (sentinelCost - displayedSentinel) * 0.15;
    displayedGPT4o += (gpt4oCost - displayedGPT4o) * 0.15;
    
    document.getElementById("ticker-sentinel").textContent = 
        `$${displayedSentinel.toFixed(4)}`;
    document.getElementById("ticker-gpt4o").textContent = 
        `$${displayedGPT4o.toFixed(2)}`;
    
    const savings = displayedGPT4o - displayedSentinel;
    const savingsPct = displayedGPT4o > 0 ? (savings / displayedGPT4o * 100) : 0;
    document.getElementById("ticker-savings").textContent = 
        `You saved $${savings.toFixed(2)} (${savingsPct.toFixed(1)}%)`;
    
    // Update progress bar
    const barWidth = Math.min(100, savingsPct);
    document.getElementById("ticker-bar").style.width = `${barWidth}%`;
}

// Run at 30 FPS for smooth animation
setInterval(updateCostTicker, 33);
```

**Step 3: Expose frame count to JS**

Add hidden component in app.py:
```python
frame_count_display = gr.Number(value=0, visible=False, elem_id="frame-count")
```

Update it in process_frame return or via the existing frame_count_state.

**Step 4: Show ticker on activation**

In activation handler: `document.getElementById("cost-ticker").style.display = "block";`

### What You Need to Approve
- GPT-4o cost rate: $0.01/frame is approximate. Use different number?
- Number animation: smooth easing (shown) or instant snap?
- Should there be confetti when savings hits $1.00?
- Placement: inside glass-panel below camera? or separate section?

### Testing Sequence
1. Activate → ticker shows $0.00 vs $0.00
2. Run 5 simulation scenarios → Sentinel: ~$0.0015 vs GPT-4o: ~$0.05
3. Watch numbers animate smoothly
4. Progress bar fills as savings grow
5. After 50 frames: "$0.49 saved (99.8%)" — powerful demo moment

---

## UPGRADE 8: Multi-Turn VLM Memory

**Goal:** Nemotron sees last 3 alerts and reasons about trends. "Person was 5m away, now 1m."
**Files touched:** `app.py` only (~30 lines).
**Dependency:** None.

### Architecture

```
NEW gr.State: vlm_memory = gr.State(value=[])

On each VLM call:
  1. Append current alert context to vlm_memory (max 3 entries)
  2. Format memory into system prompt:
     "Previous alerts in this session:
      [10:05:01] WARNING: Person detected 5m away
      [10:05:11] WARNING: Person now 3m away, approaching
      [current] Analyze this new frame..."
  3. Nemotron receives history + current frame → can reference trends
  4. Trim memory to last 3 entries (sliding window)
```

### Implementation Plan

**Step 1: Add memory state**

In app.py, with other gr.State declarations:
```python
vlm_memory = gr.State(value=[])  # List of dicts: {time, level, text}
```

**Step 2: Update SYSTEM_PROMPT**

Change from static string to a function:
```python
def build_system_prompt(memory: list) -> str:
    history_block = ""
    if memory:
        lines = [f"  [{m['time']}] {m['level']}: {m['text']}" for m in memory[-3:]]
        history_block = "\n\nPrevious alerts in this session (use for trend awareness):\n" + "\n".join(lines)
    
    return f"""You are Sentinel, an autonomous AI guardian for visually impaired and elderly users.
You receive visual descriptions and sensor data. Your job is to:
1. Identify potential dangers (tripping hazards, approaching vehicles, strangers, fire)
2. Provide navigation guidance (door ahead, stairs, obstacles)
3. Alert ONLY when genuinely dangerous — avoid false alarms
4. Respond in 1-2 sentences maximum (user hears this via TTS)
5. Reference previous alerts when relevant (e.g., "Person getting closer")
6. Escalate if a threat is worsening over time

Format: [LEVEL] message
Where LEVEL is: CRITICAL, WARNING, or OK{history_block}"""
```

**Step 3: Update process_frame**

After receiving VLM response and parsing alert:
```python
# Append to memory
if alert_level in ["critical", "warning"]:
    memory_entry = {
        "time": time.strftime("%H:%M:%S"),
        "level": alert_level.upper(),
        "text": alert_text[:80]  # Truncate to save tokens
    }
    vlm_memory.append(memory_entry)
    vlm_memory = vlm_memory[-3:]  # Keep last 3 only

# Use dynamic prompt:
question = f"Analyze this scene. Context: {sensor_context}"
# Pass build_system_prompt(vlm_memory) instead of static SYSTEM_PROMPT
```

**Step 4: Update function signature and event bindings**

Add `vlm_memory` to process_frame params (now 21 params, 15 returns, 9 gr.State).
Update all return paths and image_data.change bindings accordingly.

### Token Budget
- Each memory entry: ~20 tokens
- 3 entries: ~60 extra tokens per VLM call
- Negligible cost increase, significant intelligence increase

### Testing Sequence
1. Trigger "Person Approaching" 3 times in sequence
2. First alert: "Person detected 5m away"
3. Second alert: "Person closer, now 3m" (should reference previous)
4. Third alert: "CRITICAL: Person very close, 1m away — was 5m moments ago"
5. Verify memory shows in alert history with trend references

---

## UPGRADE 9: Virtual Walk Mode

**Goal:** Pre-recorded walking video plays. Sentinel analyzes it live. Alerts interrupt the video.
**Files touched:** `app.py` (~80 lines new tab), `sensor_bridge.py` (~60 lines JS).
**Dependency:** Upgrade 3 (spatial audio) recommended for full experience.

### Architecture

```
New Tab: "🎬 Demo"

Components:
  - gr.Video (pre-loaded walking video)
  - gr.HTML (alert overlay zone, positioned over video)
  - Play/Pause button
  - Speed control (0.5x, 1x, 2x)

JS Frame Extractor:
  - requestVideoFrameCallback on video element
  - Every 1000ms (1 FPS): draw video frame to canvas
  - Convert to base64
  - Inject into image-data textbox → triggers process_frame pipeline
  
Alert Overlay:
  - When alert generated, HTML overlay appears ON TOP of video
  - Semi-transparent banner with alert text
  - Auto-dismisses after 5 seconds
  - Audio plays via spatial audio (Upgrade 3) or regular playback

Data Flow:
  video.play() → JS extracts frame every 1s → base64 → image-data
  → process_frame → VLM → alert → overlay on video + TTS
  → video continues playing underneath
```

### Implementation Plan

**Step 1: Create Demo tab in app.py**

```python
with gr.Tab("🎬 Demo"):
    with gr.Column(elem_classes="glass-panel"):
        gr.Markdown("### Virtual Walk Mode")
        gr.Markdown("Watch Sentinel analyze a real walking scenario in real-time.")
        
        demo_video = gr.Video(
            label="Walking Scenario",
            value=None,  # User uploads or pre-load
            autoplay=False,
            elem_id="demo-video"
        )
        
        # Pre-loaded scenario buttons
        with gr.Row():
            gr.Markdown("**Or try a preset scenario:**")
        with gr.Row():
            demo_stairs_btn = gr.Button("Stairs", size="sm")
            demo_traffic_btn = gr.Button("Traffic", size="sm")
            demo_park_btn = gr.Button("Park Walk", size="sm")
        
        demo_alert_overlay = gr.HTML(
            "",
            elem_id="demo-alert-overlay"
        )
        
        demo_status = gr.HTML(
            "<div style='text-align:center;color:#9ca3af;'>Press Play to start the virtual walk</div>",
            elem_id="demo-status"
        )
```

**Step 2: JS frame extractor in sensor_bridge.py**

```javascript
function initDemoMode() {
    const video = document.querySelector('#demo-video video');
    if (!video) return;
    
    let demoCanvas = document.createElement('canvas');
    demoCanvas.width = 640;
    demoCanvas.height = 480;
    let demoCtx = demoCanvas.getContext('2d');
    let lastDemoFrame = 0;
    
    video.addEventListener('play', () => {
        // Activate monitoring for demo mode
        setGradioValue("sentinel-active-state", "true");
    });
    
    function extractDemoFrame(timestamp) {
        if (video.paused || video.ended) return;
        
        if (timestamp - lastDemoFrame >= 1000) { // 1 FPS
            lastDemoFrame = timestamp;
            demoCtx.drawImage(video, 0, 0, 640, 480);
            const base64 = demoCanvas.toDataURL("image/jpeg", 0.7).split(",")[1];
            setGradioValue("image-data", base64);
        }
        requestAnimationFrame(extractDemoFrame);
    }
    
    video.addEventListener('play', () => {
        requestAnimationFrame(extractDemoFrame);
    });
}
```

**Step 3: Alert overlay CSS**

```css
#demo-alert-overlay > div {
    position: absolute;
    bottom: 80px;
    left: 50%;
    transform: translateX(-50%);
    z-index: 100;
    max-width: 90%;
    animation: slideUp 0.3s ease-out;
}
@keyframes slideUp {
    from { opacity: 0; transform: translateX(-50%) translateY(20px); }
    to { opacity: 1; transform: translateX(-50%) translateY(0); }
}
```

### What You Need to Decide
- Video source: Film your own walking video? Use Creative Commons? Use a slideshow of sim images instead?
- Frame rate: 1 FPS (1 call/sec, cheap) or 2 FPS?
- Should alerts PAUSE the video or overlay while playing?
- Preset videos: record 3 short clips (stairs, traffic, park) and embed?

### Testing Sequence
1. Upload walking video to Demo tab
2. Press Play → video starts, frame counter ticks
3. When person crosses path → alert banner slides up over video + TTS plays
4. When stairs appear → CRITICAL alert + loud warning
5. Clear section → "OK: Path clear" in info banner

---

## UPGRADE 10: Web Speech API Panic Override

**Goal:** Browser listens for "HELP"/"STOP" keywords. Works offline. Triggers emergency mode.
**Files touched:** `sensor_bridge.py` (~50 lines JS).
**Dependency:** None.

### Architecture

```
webkitSpeechRecognition runs continuously in browser:
  - Listens for keywords: "help", "stop", "emergency", "cancel"
  - On match:
    1. Set monitoring_active to true (via hidden component)
    2. Inject CRITICAL alert into alert banner
    3. Play emergency siren sound (Web Audio oscillator)
    4. Open SMS app with pre-filled GPS + "EMERGENCY" text
  - Works even if server is unreachable
  - iOS requires user gesture to start (use activation button)
```

### Implementation Plan

**Step 1: Add speech recognition to sensor_bridge.py IIFE**

```javascript
// Inside IIFE:
let recognition = null;
const PANIC_KEYWORDS = ["help", "stop", "emergency", "cancel", "sos"];

function initSpeechPanic() {
    const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;
    if (!SpeechRecognition) {
        console.log("Web Speech API not available. Panic override disabled.");
        return;
    }
    
    recognition = new SpeechRecognition();
    recognition.continuous = true;
    recognition.interimResults = true;
    recognition.lang = 'en-US';
    
    recognition.onresult = (event) => {
        const last = event.results[event.results.length - 1];
        const transcript = last[0].transcript.toLowerCase().trim();
        
        for (const keyword of PANIC_KEYWORDS) {
            if (transcript.includes(keyword)) {
                triggerPanicMode(keyword);
                break;
            }
        }
    };
    
    recognition.onerror = (event) => {
        // Auto-restart on error (except if user stopped it)
        if (event.error !== 'aborted' && isActivated) {
            setTimeout(() => { try { recognition.start(); } catch(e) {} }, 1000);
        }
    };
    
    recognition.onend = () => {
        // Auto-restart if still active
        if (isActivated) {
            try { recognition.start(); } catch(e) {}
        }
    };
}

function triggerPanicMode(keyword) {
    console.log(`PANIC KEYWORD DETECTED: "${keyword}"`);
    
    // 1. Set alert banner to critical
    const banner = document.getElementById("alert-banner-container");
    if (banner) {
        banner.innerHTML = `
            <div class="alert-banner-critical" style="animation: pulse 0.5s infinite;">
                🚨 EMERGENCY MODE ACTIVATED<br>
                Voice command: "${keyword.toUpperCase()}"<br>
                <small>GPS coordinates being sent to emergency contact</small>
            </div>`;
    }
    
    // 2. Play emergency siren (Web Audio oscillator)
    playEmergencySiren();
    
    // 3. Open SMS with pre-filled message
    const lat = document.querySelector('#gps-lat input')?.value || 'unknown';
    const lon = document.querySelector('#gps-lon input')?.value || 'unknown';
    const smsBody = encodeURIComponent(
        `SENTINEL EMERGENCY: User triggered panic mode. Last known location: ${lat}, ${lon}. Please check on them.`
    );
    window.open(`sms:?body=${smsBody}`, '_blank');
}

function playEmergencySiren() {
    const ctx = new (window.AudioContext || window.webkitAudioContext)();
    const osc = ctx.createOscillator();
    const gain = ctx.createGain();
    osc.type = 'sawtooth';
    osc.frequency.value = 800;
    gain.gain.value = 0.3;
    osc.connect(gain);
    gain.connect(ctx.destination);
    osc.start();
    // Modulate frequency for siren effect
    osc.frequency.setValueAtTime(800, ctx.currentTime);
    osc.frequency.linearRampToValueAtTime(1200, ctx.currentTime + 0.5);
    osc.frequency.linearRampToValueAtTime(800, ctx.currentTime + 1.0);
    // Auto-stop after 3 seconds
    gain.gain.linearRampToValueAtTime(0, ctx.currentTime + 3.0);
    osc.stop(ctx.currentTime + 3.0);
}
```

**Step 2: Start recognition on activation**

In `startAllSensors()`, add: `initSpeechPanic(); if (recognition) try { recognition.start(); } catch(e) {}`

In `stopAllSensors()`, add: `if (recognition) try { recognition.stop(); } catch(e) {}`

### Edge Cases
- **iOS Safari:** Requires user gesture. Since activation button is the user gesture, speech recognition starts after clicking ACTIVATE.
- **Background noise false triggers:** Use `interimResults: true` but only trigger on high-confidence matches. Could add confidence threshold: `if (last[0].confidence > 0.7)`.
- **Browser doesn't support:** Graceful log message, no crash.

### Testing Sequence
1. Activate Sentinel
2. Say "Help" clearly → banner turns red, siren plays, SMS app opens
3. Say "Stop" → same emergency response
4. Normal conversation → no false triggers
5. Test with server offline → still works (pure browser-side)

---

## BUILD ORDER & TIMELINE

### Day 1 — Deploy + Visual Wins (6 hours)

| Time | Upgrade | Why First |
|------|---------|-----------|
| 1h | Deploy Modal + HF Space | Everything depends on this |
| 1h | Upgrade 6: IMU Glass UI | Simplest visual upgrade, 40 lines |
| 1h | Upgrade 7: Cost Race Ticker | Most persuasive demo element |
| 1h | Upgrade 4: SVG Bounding Boxes | Camera feed looks like real AI |
| 1h | Upgrade 2: Gyroscope HUD | Premium app feel |
| 1h | Quick wins (tab badges, keyboard shortcuts) | Polish |

### Day 2 — Intelligence + Audio (5 hours)

| Time | Upgrade | Why This Order |
|------|---------|----------------|
| 1h | Upgrade 8: Multi-Turn VLM Memory | Pure app.py change, smart |
| 1h | Upgrade 1: OpenCV.js Frame Diff | Architecture proof |
| 1.5h | Upgrade 3: Spatial Audio | Demo-defining feature |
| 1h | Upgrade 5: TTS Streaming | Pairs with spatial audio |
| 0.5h | Upgrade 10: Speech Panic Override | Simple, impressive |

### Day 3 — Killer Demo + Submission (5 hours)

| Time | Task | Notes |
|------|------|-------|
| 1h | Upgrade 9: Virtual Walk Mode | Record walking video first |
| 1h | Record demo video | Show all upgrades working together |
| 1h | HF Dataset (500 rows) | Export from simulation + demo runs |
| 1h | Space README + thumbnail | Architecture diagram, screenshots |
| 1h | Submit + blog post | Link everything |

---

## DEPENDENCY MAP

```
No dependencies:
  Upgrade 2 (HUD) ──────────────────────┐
  Upgrade 6 (Glass UI) ────────────────┤
  Upgrade 7 (Cost Ticker) ─────────────┤
  Upgrade 8 (VLM Memory) ─────────────┤
  Upgrade 10 (Speech Panic) ───────────┤
                                        │
Optional dependencies:                  │
  Upgrade 1 (Frame Diff) → Upgrade 4 (SVG Boxes)
  Upgrade 3 (Spatial Audio) → Upgrade 9 (Virtual Walk)
  Upgrade 5 (TTS Stream) → Upgrade 3 (Spatial Audio)
                                        │
All feed into: ─────────────────────────┘
  Upgrade 9 (Virtual Walk) ← uses everything
```

---

## WHAT YOU APPROVE BEFORE AGENT BUILDS

For each upgrade, tell the agent:

1. **"Build Upgrade X"** — agent starts
2. **"Use approach Y"** — if there are options (like HUD Option A vs B)
3. **"Reference: [screenshot/description]"** — for visual upgrades, describe what you want
4. **"Skip the [thing I don't want]"** — agent trims unnecessary parts

The agent has all the architecture. It just needs your green light on style choices.