| ---
|
| license: mit
|
| tags:
|
| - eye-tracking
|
| - gaze-estimation
|
| - computer-vision
|
| - onnx
|
| - ux-research
|
| library_name: onnx
|
| pipeline_tag: image-classification
|
| ---
|
|
|
| # InsightUX — Webcam Eye-Tracking UX Research Browser
|
|
|
| A webcam-based gaze tracking system that turns any consumer laptop into a UX
|
| research tool. Browse a website, and get back a heatmap, an attention timeline,
|
| and a ranked list of which page elements actually held your gaze.
|
|
|
| No specialist hardware — just a webcam.
|
|
|
| **This is a local desktop app, not a hosted demo.** It needs a physical
|
| webcam, a native window, and full-screen capture — none of which exist on a
|
| server, so it cannot run as a Hugging Face Space. Clone it and run it on your
|
| own machine.
|
|
|
| ## Requirements before you start
|
|
|
| - **Windows** (uses WebView2 via pywebview's WinForms backend)
|
| - **Python 3.11 exactly** — mediapipe's legacy face-mesh API this project
|
| depends on does not exist on Python 3.12+
|
| - A working webcam
|
|
|
| ## How to run — step by step
|
|
|
| ### 1. Clone the repo
|
|
|
| ```bash
|
| git clone https://huggingface.co/<your-username>/insightux
|
| cd insightux
|
| ```
|
|
|
| ### 2. Create and activate a virtual environment
|
|
|
| ```bash
|
| python -m venv venv
|
| venv\Scripts\activate
|
| ```
|
|
|
| ### 3. Install dependencies — use the pinned versions
|
|
|
| ```bash
|
| pip install -r requirements.txt
|
| ```
|
|
|
| This includes `onnx`, which `calibrate.py` needs when it exports the
|
| fine-tuned model after training.
|
|
|
| **Do not `pip install --upgrade` pywebview or pythonnet.** `requirements.txt`
|
| pins `pywebview==4.4.1` and `pythonnet==3.0.3` deliberately — newer versions
|
| have a bug in their Windows backend that freezes the app window and floods
|
| the console with:
|
| ```
|
| AccessibilityObject.Bounds.Empty.Empty.Empty.Empty...
|
| ```
|
| If you already have a newer version installed globally, this install step
|
| will replace it with the working one.
|
|
|
| ### 4. Calibrate — required, per person, per setup
|
|
|
| ```bash
|
| python calibrate.py
|
| ```
|
| Sit normally at your usual distance from the screen, look at each of the 16
|
| dots as they appear. Takes about a minute. This is **per-person and
|
| per-setup** — your eyes, your camera, your screen size. Everyone using this
|
| must run it themselves; it is not something you can copy from someone else.
|
|
|
| Re-run it if your lighting, seating position, or camera position changes
|
| noticeably.
|
|
|
| At the end it prints an honest quality readout, including whether the model
|
| can actually see where you're looking on each axis:
|
| ```
|
| HORIZONTAL yaw vs screen-X : r = +0.995
|
| VERTICAL pitch vs screen-Y: r = +0.883
|
| ```
|
| If either number is low, the report will say so plainly and explain why —
|
| that means the model isn't seeing that axis, and no amount of recalibrating
|
| will fix it.
|
|
|
| ### 5. (Optional) Check your accuracy
|
|
|
| ```bash
|
| python validate.py
|
| ```
|
| Flashes 9 test dots and reports your real error in pixels. Good for knowing
|
| what to expect before relying on a session.
|
|
|
| ### 6. Run the browser
|
|
|
| ```bash
|
| python browser_session.py
|
| ```
|
| - Opens on an InsightUX-branded search page. Type a search term (goes to
|
| real Google results) or a URL (goes straight there).
|
| - Land on the page you want to study, **click on blank space on the page**
|
| (not a text field), then press **S** to start eye-tracking.
|
| - Press **E** to stop. A full report — heatmap over real screenshots,
|
| ranked attention list, dwell timeline — generates and opens automatically.
|
|
|
| ## What's in this repo
|
|
|
| | File | Purpose |
|
| |---|---|
|
| | `models/gaze_cnn_v4.onnx` (+ `.onnx.data`) | Binocular gaze CNN (EfficientNet-B0 backbone, dual eye patches + head pose) |
|
| | `models/model_v4.py` | Model architecture definition, used during calibration fine-tuning |
|
| | `checkpoints/best_model_v4.pt` | PyTorch checkpoint for fine-tuning |
|
| | `calibrate.py` | Per-user calibration — 16-point, live blink/lighting rejection, honest quality report |
|
| | `validate.py` | Measures real accuracy in pixels after calibration |
|
| | `browser_session.py` | The research browser — search, track, auto-report |
|
| | `analysis.py` | Builds the session report (heatmaps over real screenshots) |
|
| | `inference_pipeline.py` | ONNX inference + RBF gaze→screen calibration mapping |
|
| | `preprocessing/preprocessing_pipeline.py` | Eye patch normalization, head pose estimation, illumination correction |
|
|
|
| ## Calibration is per-person and per-setup — not included in this repo
|
|
|
| `calibration.pkl` (generated by `calibrate.py`) encodes *your* eye geometry,
|
| *your* camera characteristics, and *your* screen size. It is deliberately
|
| **not** in this repo — it would be useless to anyone else and it's personal
|
| data. Run `calibrate.py` yourself; it takes about a minute.
|
|
|
| ## Accuracy — honest numbers
|
|
|
| This is a webcam system, not a Tobii. Expect roughly **5–10% of screen
|
| diagonal** mean error after a good calibration. That's enough for coarse AOI
|
| attribution (navbar vs hero vs footer) and heatmaps. It is **not** enough for
|
| reading-level analysis (which word you're on).
|
|
|
| Vertical accuracy is typically a bit looser than horizontal — looking down
|
| partially occludes the iris under the eyelid, a physical limit of webcam
|
| gaze estimation, not a bug. Calibration automatically checks whether eye
|
| aperture (eyelid closing as you look down) tracks vertical position better
|
| than the model's raw output, and uses whichever signal is actually stronger.
|
|
|
| ## Known limitations
|
|
|
| - Windows-only (pywebview + WebView2 backend). Other platforms untested.
|
| - Head movement during a session degrades accuracy — the calibration assumes
|
| a roughly stable head pose.
|
| - No lens distortion correction — `solvePnP` assumes zero distortion, which
|
| costs some accuracy near frame edges.
|
|
|
| ## Troubleshooting
|
|
|
| **Window freezes with `AccessibilityObject.Bounds.Empty.Empty.Empty...`
|
| spamming the console:**
|
| Confirm you actually have the pinned versions installed, not newer ones:
|
| ```bash
|
| pip uninstall pywebview pythonnet -y
|
| pip install pywebview==4.4.1 pythonnet==3.0.3
|
| ```
|
| If it persists after that, turn off **Xbox Game Bar** (Settings → Gaming →
|
| Xbox Game Bar) and any overlay software (Discord overlay, GeForce
|
| Experience, OBS), then **restart your PC** — those hooks stay loaded until a
|
| real reboot. Last resort: install `PyQt5` + `PyQtWebEngine` (see
|
| `requirements.txt`) and switch the GUI backend as described there.
|
|
|
| **"No calibration.pkl found" when pressing S:**
|
| Run `python calibrate.py` first — it must exist before `browser_session.py`
|
| can track anything.
|
|
|
| **Can't type in the search box:**
|
| Click directly into the search field first — this can happen if the window
|
| just opened and hasn't fully grabbed keyboard focus yet.
|
|
|
| ## License
|
|
|
| MIT. |
