InsightUX — Webcam Eye-Tracking UX Research Browser
A webcam-based gaze tracking system that turns any consumer laptop into a UX
research tool. Browse a website, and get back a heatmap, an attention timeline,
and a ranked list of which page elements actually held your gaze.
No specialist hardware — just a webcam.
This is a local desktop app, not a hosted demo. It needs a physical
webcam, a native window, and full-screen capture — none of which exist on a
server, so it cannot run as a Hugging Face Space. Clone it and run it on your
own machine.
Requirements before you start
Windows (uses WebView2 via pywebview's WinForms backend)
Python 3.11 exactly — mediapipe's legacy face-mesh API this project
depends on does not exist on Python 3.12+
This includes onnx, which calibrate.py needs when it exports the
fine-tuned model after training.
Do not pip install --upgrade pywebview or pythonnet.requirements.txt
pins pywebview==4.4.1 and pythonnet==3.0.3 deliberately — newer versions
have a bug in their Windows backend that freezes the app window and floods
the console with:
If you already have a newer version installed globally, this install step
will replace it with the working one.
4. Calibrate — required, per person, per setup
python calibrate.py
Sit normally at your usual distance from the screen, look at each of the 16
dots as they appear. Takes about a minute. This is per-person and
per-setup — your eyes, your camera, your screen size. Everyone using this
must run it themselves; it is not something you can copy from someone else.
Re-run it if your lighting, seating position, or camera position changes
noticeably.
At the end it prints an honest quality readout, including whether the model
can actually see where you're looking on each axis:
HORIZONTAL yaw vs screen-X : r = +0.995
VERTICAL pitch vs screen-Y: r = +0.883
If either number is low, the report will say so plainly and explain why —
that means the model isn't seeing that axis, and no amount of recalibrating
will fix it.
5. (Optional) Check your accuracy
python validate.py
Flashes 9 test dots and reports your real error in pixels. Good for knowing
what to expect before relying on a session.
6. Run the browser
python browser_session.py
Opens on an InsightUX-branded search page. Type a search term (goes to
real Google results) or a URL (goes straight there).
Land on the page you want to study, click on blank space on the page
(not a text field), then press S to start eye-tracking.
Press E to stop. A full report — heatmap over real screenshots,
ranked attention list, dwell timeline — generates and opens automatically.
Eye patch normalization, head pose estimation, illumination correction
Calibration is per-person and per-setup — not included in this repo
calibration.pkl (generated by calibrate.py) encodes your eye geometry,
your camera characteristics, and your screen size. It is deliberately
not in this repo — it would be useless to anyone else and it's personal
data. Run calibrate.py yourself; it takes about a minute.
Accuracy — honest numbers
This is a webcam system, not a Tobii. Expect roughly 5–10% of screen
diagonal mean error after a good calibration. That's enough for coarse AOI
attribution (navbar vs hero vs footer) and heatmaps. It is not enough for
reading-level analysis (which word you're on).
Vertical accuracy is typically a bit looser than horizontal — looking down
partially occludes the iris under the eyelid, a physical limit of webcam
gaze estimation, not a bug. Calibration automatically checks whether eye
aperture (eyelid closing as you look down) tracks vertical position better
than the model's raw output, and uses whichever signal is actually stronger.
Known limitations
Windows-only (pywebview + WebView2 backend). Other platforms untested.
Head movement during a session degrades accuracy — the calibration assumes
a roughly stable head pose.
No lens distortion correction — solvePnP assumes zero distortion, which
costs some accuracy near frame edges.
Troubleshooting
Window freezes with AccessibilityObject.Bounds.Empty.Empty.Empty...
spamming the console:
Confirm you actually have the pinned versions installed, not newer ones:
If it persists after that, turn off Xbox Game Bar (Settings → Gaming →
Xbox Game Bar) and any overlay software (Discord overlay, GeForce
Experience, OBS), then restart your PC — those hooks stay loaded until a
real reboot. Last resort: install PyQt5 + PyQtWebEngine (see
requirements.txt) and switch the GUI backend as described there.
"No calibration.pkl found" when pressing S:
Run python calibrate.py first — it must exist before browser_session.py
can track anything.
Can't type in the search box:
Click directly into the search field first — this can happen if the window
just opened and hasn't fully grabbed keyboard focus yet.
