Add pinned versions and step-by-step run guide

README.md

@@ -0,0 +1,175 @@
+---
+license: mit
+tags:
+  - eye-tracking
+  - gaze-estimation
+  - computer-vision
+  - onnx
+  - ux-research
+library_name: onnx
+pipeline_tag: image-classification
+---
+
+# InsightUX — Webcam Eye-Tracking UX Research Browser
+
+A webcam-based gaze tracking system that turns any consumer laptop into a UX
+research tool. Browse a website, and get back a heatmap, an attention timeline,
+and a ranked list of which page elements actually held your gaze.
+
+No specialist hardware — just a webcam.
+
+**This is a local desktop app, not a hosted demo.** It needs a physical
+webcam, a native window, and full-screen capture — none of which exist on a
+server, so it cannot run as a Hugging Face Space. Clone it and run it on your
+own machine.
+
+## Requirements before you start
+
+- **Windows** (uses WebView2 via pywebview's WinForms backend)
+- **Python 3.11 exactly** — mediapipe's legacy face-mesh API this project
+  depends on does not exist on Python 3.12+
+- A working webcam
+
+## How to run — step by step
+
+### 1. Clone the repo
+
+```bash
+git clone https://huggingface.co/<your-username>/insightux
+cd insightux
+```
+
+### 2. Create and activate a virtual environment
+
+```bash
+python -m venv venv
+venv\Scripts\activate
+```
+
+### 3. Install dependencies — use the pinned versions
+
+```bash
+pip install -r requirements.txt
+```
+
+**Do not `pip install --upgrade` pywebview or pythonnet.** `requirements.txt`
+pins `pywebview==4.4.1` and `pythonnet==3.0.3` deliberately — newer versions
+have a bug in their Windows backend that freezes the app window and floods
+the console with:
+```
+AccessibilityObject.Bounds.Empty.Empty.Empty.Empty...
+```
+If you already have a newer version installed globally, this install step
+will replace it with the working one.
+
+### 4. Calibrate — required, per person, per setup
+
+```bash
+python calibrate.py
+```
+Sit normally at your usual distance from the screen, look at each of the 16
+dots as they appear. Takes about a minute. This is **per-person and
+per-setup** — your eyes, your camera, your screen size. Everyone using this
+must run it themselves; it is not something you can copy from someone else.
+
+Re-run it if your lighting, seating position, or camera position changes
+noticeably.
+
+At the end it prints an honest quality readout, including whether the model
+can actually see where you're looking on each axis:
+```
+HORIZONTAL  yaw vs screen-X : r = +0.995
+VERTICAL    pitch vs screen-Y: r = +0.883
+```
+If either number is low, the report will say so plainly and explain why —
+that means the model isn't seeing that axis, and no amount of recalibrating
+will fix it.
+
+### 5. (Optional) Check your accuracy
+
+```bash
+python validate.py
+```
+Flashes 9 test dots and reports your real error in pixels. Good for knowing
+what to expect before relying on a session.
+
+### 6. Run the browser
+
+```bash
+python browser_session.py
+```
+- Opens on an InsightUX-branded search page. Type a search term (goes to
+  real Google results) or a URL (goes straight there).
+- Land on the page you want to study, **click on blank space on the page**
+  (not a text field), then press **S** to start eye-tracking.
+- Press **E** to stop. A full report — heatmap over real screenshots,
+  ranked attention list, dwell timeline — generates and opens automatically.
+
+## What's in this repo
+
+| File | Purpose |
+|---|---|
+| `models/gaze_cnn_v4.onnx` (+ `.onnx.data`) | Binocular gaze CNN (EfficientNet-B0 backbone, dual eye patches + head pose) |
+| `models/model_v4.py` | Model architecture definition, used during calibration fine-tuning |
+| `checkpoints/best_model_v4.pt` | PyTorch checkpoint for fine-tuning |
+| `calibrate.py` | Per-user calibration — 16-point, live blink/lighting rejection, honest quality report |
+| `validate.py` | Measures real accuracy in pixels after calibration |
+| `browser_session.py` | The research browser — search, track, auto-report |
+| `analysis.py` | Builds the session report (heatmaps over real screenshots) |
+| `inference_pipeline.py` | ONNX inference + RBF gaze→screen calibration mapping |
+| `preprocessing/preprocessing_pipeline.py` | Eye patch normalization, head pose estimation, illumination correction |
+
+## Calibration is per-person and per-setup — not included in this repo
+
+`calibration.pkl` (generated by `calibrate.py`) encodes *your* eye geometry,
+*your* camera characteristics, and *your* screen size. It is deliberately
+**not** in this repo — it would be useless to anyone else and it's personal
+data. Run `calibrate.py` yourself; it takes about a minute.
+
+## Accuracy — honest numbers
+
+This is a webcam system, not a Tobii. Expect roughly **5–10% of screen
+diagonal** mean error after a good calibration. That's enough for coarse AOI
+attribution (navbar vs hero vs footer) and heatmaps. It is **not** enough for
+reading-level analysis (which word you're on).
+
+Vertical accuracy is typically a bit looser than horizontal — looking down
+partially occludes the iris under the eyelid, a physical limit of webcam
+gaze estimation, not a bug. Calibration automatically checks whether eye
+aperture (eyelid closing as you look down) tracks vertical position better
+than the model's raw output, and uses whichever signal is actually stronger.
+
+## Known limitations
+
+- Windows-only (pywebview + WebView2 backend). Other platforms untested.
+- Head movement during a session degrades accuracy — the calibration assumes
+  a roughly stable head pose.
+- No lens distortion correction — `solvePnP` assumes zero distortion, which
+  costs some accuracy near frame edges.
+
+## Troubleshooting
+
+**Window freezes with `AccessibilityObject.Bounds.Empty.Empty.Empty...`
+spamming the console:**
+Confirm you actually have the pinned versions installed, not newer ones:
+```bash
+pip uninstall pywebview pythonnet -y
+pip install pywebview==4.4.1 pythonnet==3.0.3
+```
+If it persists after that, turn off **Xbox Game Bar** (Settings → Gaming →
+Xbox Game Bar) and any overlay software (Discord overlay, GeForce
+Experience, OBS), then **restart your PC** — those hooks stay loaded until a
+real reboot. Last resort: install `PyQt5` + `PyQtWebEngine` (see
+`requirements.txt`) and switch the GUI backend as described there.
+
+**"No calibration.pkl found" when pressing S:**
+Run `python calibrate.py` first — it must exist before `browser_session.py`
+can track anything.
+
+**Can't type in the search box:**
+Click directly into the search field first — this can happen if the window
+just opened and hasn't fully grabbed keyboard focus yet.
+
+## License
+
+MIT.

requirements.txt

@@ -0,0 +1,51 @@
+# =============================================================================
+# InsightUX — pinned requirements
+#
+# Python 3.11 is REQUIRED. mediapipe's legacy solutions.face_mesh API (which
+# this project depends on for face landmarks) does not exist on newer
+# mediapipe builds for Python 3.12+.
+#
+# Install with:
+#     pip install -r requirements.txt
+# =============================================================================
+
+# --- Core CV / inference ---
+opencv-python==4.10.0.84
+mediapipe==0.10.18
+onnxruntime==1.19.2
+numpy==1.26.4
+scipy==1.13.1
+Pillow==10.4.0
+
+# --- Screen capture / automation ---
+pyautogui==0.9.54
+
+# --- pywebview: PINNED, DO NOT UPGRADE WITHOUT TESTING ---
+# Newer pywebview versions (5.x+) have a bug in their Windows WinForms
+# backend that triggers an infinite recursion walking the window's
+# accessibility tree, freezing the entire window and flooding the console
+# with:
+#     AccessibilityObject.Bounds.Empty.Empty.Empty.Empty...
+# These two versions are the confirmed-working combination — install exactly
+# these, not "latest":
+pywebview==4.4.1
+pythonnet==3.0.3
+
+# --- Optional: calibration fine-tuning ---
+# calibrate.py degrades gracefully and skips fine-tuning if torch is absent,
+# so this is only needed if you want the fine-tuning step during calibration.
+torch==2.4.1
+torchvision==0.19.1
+
+# --- Optional fallback GUI backend ---
+# If the pywebview freeze happens even after pinning the versions above
+# (rare — usually caused by Xbox Game Bar or another accessibility-hooking
+# overlay app running in the background), install these and change the last
+# line of browser_session.py from:
+#     webview.start(debug=True)
+# to:
+#     webview.start(gui='qt', debug=True)
+# Qt does not use the buggy WinForms accessibility chain, so this class of
+# freeze cannot happen with it.
+# PyQt5==5.15.10
+# PyQtWebEngine==5.15.6