File size: 3,229 Bytes
9c902dd 3132104 9c902dd 3132104 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | ---
title: Live Code Camera Engine
emoji: 🎥
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false
---
# Live Code Camera Engine
A live **camera + microphone → evidence compression → multimodal LLM → code stream** engine.
This is the upgraded version of the click-to-capture prototype. It continuously samples the camera, tracks evidence density, collects speech transcript text, and periodically asks a multimodal LLM to generate or revise code based on what the camera and audio are exposing.
## Core primitive
```text
iPhone / laptop camera
→ live frame stream
→ evidence compressor
→ rolling observation buffer
→ LLM code synthesizer
→ live code updates
→ receipt chain
```
## What "continuity" means here
The practical version is:
```text
iPhone Safari / laptop browser
→ HTTPS Hugging Face Space
→ FastAPI WebSocket endpoint
→ multimodal LLM backend
```
If your iPhone and laptop are near each other, the shared continuity is not magic hardware continuity; it is a shared live endpoint. You open the same Space URL from either device, grant camera/mic permissions, and the stream becomes an evidence source.
## Run locally
```bash
cd live-code-camera-engine
python3 -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt
export PROVIDER=ollama
export OLLAMA_MODEL=llava
uvicorn backend.main:app --host 0.0.0.0 --port 8000
```
Open:
```text
http://localhost:8000
```
For iPhone on same Wi-Fi, use your laptop LAN IP:
```text
http://YOUR_LAPTOP_IP:8000
```
Camera/mic access on mobile is most reliable over HTTPS, except localhost. For remote iPhone use, deploy to Hugging Face Spaces or use an HTTPS tunnel.
## Deploy to Hugging Face Spaces
Create a Docker Space and upload this folder. Required files are already included:
```text
Dockerfile
requirements.txt
backend/main.py
frontend/index.html
```
Set secrets/environment variables in the Space:
```text
PROVIDER=ollama or openai
OPENAI_API_KEY=...
OPENAI_MODEL=<current multimodal model>
```
For local Ollama you normally run the backend locally, not inside Hugging Face, unless you provide a reachable Ollama endpoint.
## Modes
- `continuous_code`: generate code as the camera moves.
- `debug_visible_error`: point camera at terminal/browser error and generate a fix.
- `derive_pipeline`: infer a pipeline from scene + speech.
- `soft_qr_xlc`: decode/encode QR-like exoatomic carriers.
- `research_prism`: convert video/audio observations into code + research hypotheses.
## Receipts
Every synthesis has:
```text
session_id
frame hashes
transcript hash
observation entropy
frame-delta score
prompt hash
provider
timestamp
```
This makes the stream auditable without storing raw frames by default.
## Safety boundaries
- The app does not execute generated code automatically.
- It does not secretly record; the browser asks for permissions.
- It does not store raw camera frames unless you add persistence.
- It should not be used for covert surveillance, credential theft, malware, or unauthorized automation.
- Multi-speaker separation is represented as an extension point. True speaker diarization needs an ASR/diarization model such as Whisper-family ASR plus a diarization stack.
|