Touchless Virtual Keyboard
Type on an on-screen keyboard without touching anything: point at a key with your index finger, hold it briefly, then pinch your thumb and index finger together to press it. Built by Noorullah Zamindar β original app at NoorullahZamindar-007/touchless-virtual-keyboard. A demo video is in his LinkedIn post.
This is a hand-tracking application, not a trained neural network of
his own. There are no weights trained by Noorullah and no training
data. The "model" is MediaPipe's pre-trained HandLandmarker
(hand_landmarker.task, Google's model, shipped in this repo) plus his
keyboard logic in virtual_keyboard.py β the layout, the hover-plus-
pinch press state machine, and the key handling from his original
main.py, kept unchanged in behaviour and packaged so it can run and
be tested without the webcam app.
How it works
- The webcam frame (1280Γ720) is passed to a hand detector β
cvzoneHandDetectorin the originalmain.py, or MediaPipe's HandLandmarker directly invirtual_keyboard.py. - The index fingertip (landmark 8) is tracked; the key under it is highlighted.
- Hold the fingertip on one key for 6 frames, then pinch thumb tip (landmark 4) and index tip closer than 45 px to press it.
- A 0.55 s cooldown and a press latch stop one pinch from typing the same letter twice.
The keyboard has 35 keys: QWERTY letters, ; , . /, plus BKSP,
ENTER, SHIFT (toggles upper/lower case), SPACE and CLEAR.
Typed text is shown in the on-screen text bar (max 1000 characters).
How to use
Webcam app (needs a camera):
pip install -r requirements.txt
python main.py
Press q to exit. Move your index fingertip over a key, hold it, then
pinch thumb and index finger to type. SHIFT switches case; BKSP
deletes; CLEAR erases everything.
Headless β detect hands in a photo, or drive the keyboard from fingertip positions (no camera needed):
python virtual_keyboard.py --image samples/palm_left.jpg
python virtual_keyboard.py --render keyboard.png
from virtual_keyboard import KeyboardEngine, detect_hands
hands = detect_hands(image) # image as read by cv2.imread
print(hands[0]["handedness"], hands[0]["score"]) # 'Left', 0.88
engine = KeyboardEngine()
engine.step(index_tip=(592, 192), thumb_tip=(602, 192), now=1000.0)
print(engine.final_text)
Measured results
Measured 20 Nov 2026 with evaluate.py (included), Python 3.12.3,
OpenCV 5.1.0, MediaPipe 0.10.35, NumPy 1.26.4. Full numbers in
metrics.json.
| Measure | Value |
|---|---|
| Keys in the layout | 35, all inside the 1280Γ720 frame, no overlaps |
| Typing simulation (30 fps, synthetic fingertips) | Typed H, I, SPACE, BKSP, SHIFT β lowercase a, ENTER, CLEAR β empty β every key behaved as in main.py |
| Hover 5 frames (< 6 required) + pinch | No press |
| Pinch distance 60 px (> 45 px) held 8 frames | No press |
| Second press inside the 0.55 s cooldown | Blocked; pressing again after the cooldown works |
| Fingertip off the keyboard / no hand in frame | Nothing selected; hover state resets |
| Text length cap | 1000 characters (MAX_TEXT_LENGTH) |
Hand detection, samples/palm_left.jpg (CC BY-SA 4.0, Wikimedia Commons) |
1 hand, 21 landmarks, Left, score 0.88 |
Hand detection, samples/palm_right.png (CC BY-SA 4.0, Wikimedia Commons) |
1 hand, 21 landmarks, Right, score 0.94 |
| Detection time per photo (after ~0.6 s model load) | ~0.03 s |
| Blank image / random-noise image | 0 hands detected |
| Missing model file | Clean FileNotFoundError |
The left-palm handedness score varied slightly between runs (0.84β0.88); the right-palm score was 0.94 in every run. Scores are stable within a single run (identical over 3 repeats).
What was NOT measured: no webcam was available in the evaluation
environment, so live typing with a real hand and camera was not tested
β only the press logic (simulated fingertips) and still-photo hand
detection. Real-world typing accuracy, speed, and reliability in
different lighting are unmeasured. The shipped evaluate.py
reproduces every number above from a clean copy of this repo.
Limitations
- Only the first detected hand is used (
hands[0]inmain.py); a second hand is ignored. - Needs decent lighting and a clear view of the hand; the pinch threshold (45 px) and hover time (6 frames) are fixed constants.
- No word prediction, no numbers row, QWERTY layout only.
- The hand model is Google's pre-trained HandLandmarker, not a model trained by Noorullah β it detects hands, it does not know this keyboard.
- On Linux, MediaPipe's native library needs OpenGL ES (
libGLESv2, e.g. thelibgles2package).
Files
main.pyβ his original webcam app (OpenCV + cvzone), unchanged.virtual_keyboard.pyβ reusable module: layout, key logic, theKeyboardEnginepress state machine, and headless hand detection.hand_landmarker.taskβ MediaPipe's pre-trained hand landmark model (7.8 MB, SHA-256fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1).evaluate.pyβ reproduces every measured number above; writesmetrics.json.metrics.jsonβ the measured results.samples/β keyboard render, the two test palm photos (CC BY-SA 4.0) and the landmark-annotated copies.requirements.txtβopencv-python,cvzone,mediapipe,numpy(64-bit Python required for MediaPipe).
License
MIT β see LICENSE (Copyright (c) 2026 Noorullah Zamindar).
