Touchless Virtual Keyboard

Type on an on-screen keyboard without touching anything: point at a key with your index finger, hold it briefly, then pinch your thumb and index finger together to press it. Built by Noorullah Zamindar β€” original app at NoorullahZamindar-007/touchless-virtual-keyboard. A demo video is in his LinkedIn post.

This is a hand-tracking application, not a trained neural network of his own. There are no weights trained by Noorullah and no training data. The "model" is MediaPipe's pre-trained HandLandmarker (hand_landmarker.task, Google's model, shipped in this repo) plus his keyboard logic in virtual_keyboard.py β€” the layout, the hover-plus- pinch press state machine, and the key handling from his original main.py, kept unchanged in behaviour and packaged so it can run and be tested without the webcam app.

Keyboard layout

How it works

  1. The webcam frame (1280Γ—720) is passed to a hand detector β€” cvzone HandDetector in the original main.py, or MediaPipe's HandLandmarker directly in virtual_keyboard.py.
  2. The index fingertip (landmark 8) is tracked; the key under it is highlighted.
  3. Hold the fingertip on one key for 6 frames, then pinch thumb tip (landmark 4) and index tip closer than 45 px to press it.
  4. A 0.55 s cooldown and a press latch stop one pinch from typing the same letter twice.

The keyboard has 35 keys: QWERTY letters, ; , . /, plus BKSP, ENTER, SHIFT (toggles upper/lower case), SPACE and CLEAR. Typed text is shown in the on-screen text bar (max 1000 characters).

How to use

Webcam app (needs a camera):

pip install -r requirements.txt
python main.py

Press q to exit. Move your index fingertip over a key, hold it, then pinch thumb and index finger to type. SHIFT switches case; BKSP deletes; CLEAR erases everything.

Headless β€” detect hands in a photo, or drive the keyboard from fingertip positions (no camera needed):

python virtual_keyboard.py --image samples/palm_left.jpg
python virtual_keyboard.py --render keyboard.png
from virtual_keyboard import KeyboardEngine, detect_hands

hands = detect_hands(image)          # image as read by cv2.imread
print(hands[0]["handedness"], hands[0]["score"])   # 'Left', 0.88

engine = KeyboardEngine()
engine.step(index_tip=(592, 192), thumb_tip=(602, 192), now=1000.0)
print(engine.final_text)

Measured results

Measured 20 Nov 2026 with evaluate.py (included), Python 3.12.3, OpenCV 5.1.0, MediaPipe 0.10.35, NumPy 1.26.4. Full numbers in metrics.json.

Measure Value
Keys in the layout 35, all inside the 1280Γ—720 frame, no overlaps
Typing simulation (30 fps, synthetic fingertips) Typed H, I, SPACE, BKSP, SHIFT β†’ lowercase a, ENTER, CLEAR β†’ empty β€” every key behaved as in main.py
Hover 5 frames (< 6 required) + pinch No press
Pinch distance 60 px (> 45 px) held 8 frames No press
Second press inside the 0.55 s cooldown Blocked; pressing again after the cooldown works
Fingertip off the keyboard / no hand in frame Nothing selected; hover state resets
Text length cap 1000 characters (MAX_TEXT_LENGTH)
Hand detection, samples/palm_left.jpg (CC BY-SA 4.0, Wikimedia Commons) 1 hand, 21 landmarks, Left, score 0.88
Hand detection, samples/palm_right.png (CC BY-SA 4.0, Wikimedia Commons) 1 hand, 21 landmarks, Right, score 0.94
Detection time per photo (after ~0.6 s model load) ~0.03 s
Blank image / random-noise image 0 hands detected
Missing model file Clean FileNotFoundError

The left-palm handedness score varied slightly between runs (0.84–0.88); the right-palm score was 0.94 in every run. Scores are stable within a single run (identical over 3 repeats).

What was NOT measured: no webcam was available in the evaluation environment, so live typing with a real hand and camera was not tested β€” only the press logic (simulated fingertips) and still-photo hand detection. Real-world typing accuracy, speed, and reliability in different lighting are unmeasured. The shipped evaluate.py reproduces every number above from a clean copy of this repo.

Limitations

  • Only the first detected hand is used (hands[0] in main.py); a second hand is ignored.
  • Needs decent lighting and a clear view of the hand; the pinch threshold (45 px) and hover time (6 frames) are fixed constants.
  • No word prediction, no numbers row, QWERTY layout only.
  • The hand model is Google's pre-trained HandLandmarker, not a model trained by Noorullah β€” it detects hands, it does not know this keyboard.
  • On Linux, MediaPipe's native library needs OpenGL ES (libGLESv2, e.g. the libgles2 package).

Files

  • main.py β€” his original webcam app (OpenCV + cvzone), unchanged.
  • virtual_keyboard.py β€” reusable module: layout, key logic, the KeyboardEngine press state machine, and headless hand detection.
  • hand_landmarker.task β€” MediaPipe's pre-trained hand landmark model (7.8 MB, SHA-256 fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1).
  • evaluate.py β€” reproduces every measured number above; writes metrics.json.
  • metrics.json β€” the measured results.
  • samples/ β€” keyboard render, the two test palm photos (CC BY-SA 4.0) and the landmark-annotated copies.
  • requirements.txt β€” opencv-python, cvzone, mediapipe, numpy (64-bit Python required for MediaPipe).

License

MIT β€” see LICENSE (Copyright (c) 2026 Noorullah Zamindar).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support