--- title: Live Expression Reader emoji: 🙂 colorFrom: indigo colorTo: pink sdk: static pinned: false license: apache-2.0 short_description: Calibrated in-browser facial expression reader --- # Live Expression Reader Real-time facial-expression analysis with **calibrated confidence**, **literature-grounded cognitive states** (tired / focused / bored / stressed / engaged / confused / calm), and **AI-powered explanations** grounded in the Facial Action Coding System. Runs entirely in your browser — webcam frames never leave your device. [![Live demo](https://img.shields.io/badge/Live%20demo-Open-indigo?style=for-the-badge&logo=huggingface&logoColor=white)](https://arjun10g-live-expression-reader.static.hf.space) [![GitHub](https://img.shields.io/badge/GitHub-Arjun10g%2Flive--expression--reader-181717?style=for-the-badge&logo=github)](https://github.com/Arjun10g/live-expression-reader) [![HF Space](https://img.shields.io/badge/Mirror-HF%20Spaces-orange?style=for-the-badge&logo=huggingface&logoColor=white)](https://huggingface.co/spaces/arjun10g/live-expression-reader/tree/main) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green?style=for-the-badge)](https://www.apache.org/licenses/LICENSE-2.0) > **Live demo:** > > **Source on GitHub:** > > **Mirror on Hugging Face Spaces:** --- ## What it does - **HSEmotion ONNX** classifier in-browser: 8 emotions (anger, contempt, disgust, fear, happy, neutral, sad, surprised) plus continuous valence / arousal. - **MediaPipe FaceLandmarker** for 478 landmarks + 52 ARKit blendshapes per face. - **Personal baseline calibration** (3-second resting-face capture) used to subtract neutral asymmetry from every downstream signal. - **DTM14 compound emotions** when top-2 are sustained close (e.g. *bittersweet*, *angrily disgusted*, *fearfully surprised*). - **Russell circumplex** valence / arousal trail with a 2-second fading inset. - **Stability sparkline** — top-1 confidence over the last ~6 s. - **Cognitive states** computed from PERCLOS (Wierwille 1994), blink rate (Stern 1984), expression entropy (D'Mello 2010), AU4 + AU7 (D'Mello & Graesser 2010), affect-lability V-flux (Kuppens 2010 / 2013), and Russell circumplex (1980). Hidden until 60 s of data is in the window. - **Personal classifier** — opt-in: hold each emotion for 3 s; a diagonal-Gaussian Naive Bayes fits to your samples and runs alongside the model. - **Why? / Ask / Summarize / Discuss recording** — an LLM reads the muscles + V/A + cognitive states + personal templates and explains them in plain language. Multi-turn chat with conversation history. - **Recording** + **JSON export** — clip up to 10 minutes of session data for later analysis or research. ## Privacy - Webcam frames are processed locally. **No frames ever leave the device.** - The "Why?" feature sends only numerical features (calibrated probabilities, blendshape activations, valence / arousal) to the LLM, and only when you click. - Personal templates and saved recordings stay in your browser tab. Persistence to `localStorage` is opt-in and off by default; "Clear stored data" wipes it instantly. - No analytics, no tracking, no cookies. ## Local development ```bash npm install cp .env.example .env.local # set VITE_DEMO_ANTHROPIC_API_KEY for local dev only npm run dev ``` Open . ## Deploy Two pieces: 1. **Frontend** → Hugging Face Spaces (Static SDK). 2. **LLM proxy** → Cloudflare Worker (free tier). Holds the API key as a server-side secret so the browser bundle never sees it. ### 1. Cloudflare Worker (one-time) ```bash cd proxy npm install npx wrangler login # opens browser npx wrangler secret put ANTHROPIC_API_KEY # paste your sk-ant-... key # Edit wrangler.toml -> ALLOWED_ORIGIN to include your future HF Space URL. npx wrangler deploy ``` Note the deployed URL (e.g. `https://expression-explain-proxy..workers.dev`). Set `VITE_EXPLAIN_PROXY_URL` to that URL when building the frontend. In the Cloudflare dashboard → Workers & Pages → your Worker → Settings → add a **rate-limit rule**: 30 requests / 5 minutes per IP, action: block 5m. Free. ### 2. Frontend on Hugging Face Spaces ```bash # Build with the proxy URL baked in (the only env var the public bundle sees). VITE_EXPLAIN_PROXY_URL=https://expression-explain-proxy..workers.dev \ npm run build # Push the built files to your Space. huggingface-cli login # one-time, with your HF write token git clone https://huggingface.co/spaces// hf-space cp -r dist/* hf-space/ cp README.md hf-space/ # the frontmatter at the top of this file is what HF Spaces reads cd hf-space git add -A git commit -m "Deploy" git push ``` The Space serves at `https://-.static.hf.space`. ## Stack | Component | Library | License | |-----------|---------|---------| | Face landmarks + 52 ARKit blendshapes | [`@mediapipe/tasks-vision`](https://www.npmjs.com/package/@mediapipe/tasks-vision) | Apache-2.0 | | Emotion classifier | [HSEmotion `enet_b0_8_va_mtl`](https://github.com/HSE-asavchenko/face-emotion-recognition) | Apache-2.0 | | ONNX inference | [`onnxruntime-web`](https://www.npmjs.com/package/onnxruntime-web) | MIT | | AI explanations | LLM (proxied via Cloudflare Worker) | API | | Build + UI | Vite, TypeScript, Tailwind CDN, Inter / JetBrains Mono | MIT / Apache-2.0 | ## Research backing the cognitive-states layer - **PERCLOS** drowsiness threshold (≥ 0.15 over 60 s): Wierwille et al. 1994 (NHTSA); Dinges & Grace 1998 (FHWA TB98-006); Abe 2023 (*SLEEP Advances*). - **Blink rate** baseline ~0.25 Hz, decreases with visual attention, increases with stress / cognitive load: Stern, Walrath & Goldstein 1984 (*Psychophysiology* 21:22-33); Maffei & Angrilli 2018 (*Neurosci Lett* 736). - **Confusion** AU4 + AU7 co-firing: D'Mello & Graesser 2010 (*Cog Emotion* 24(1):67-76). - **Boredom** = expressive flatness + low intensity + slow blink: D'Mello & Graesser 2010; Craig, D'Mello, Witherspoon & Graesser 2008. - **Stress** AU pattern (fear-blend brow + AU20 + AU24): Harrigan & O'Connell 1996; Giannakakis et al. 2017 (*BSPC* 31:89-101). - **Engagement** features (positive V + sustained gaze + AU spikes): Whitehill et al. 2014 (*IEEE T-AffComp* 5(1):86-98). - **Calm** (low arousal Russell quadrant) + **affect lability** (V-flux): Russell 1980 (*JPSP* 39(6)); Kuppens et al. 2010, 2013 (*Emotion*). - **Compound emotions** (bittersweet, fearfully surprised, etc.): Du, Tao & Martinez 2014 (*PNAS* 111(15) E1454). ## License Apache-2.0. The HSEmotion weights are also Apache-2.0; MediaPipe is Apache-2.0; onnxruntime-web is MIT. No GPL/CC-BY-NC dependencies anywhere in the stack.