Buckets:

tahamajs
/

CPS

Files

xet

tahamajs/CPS / Files /ComputerAssignments /CA2 /code /README_extra.md

tahamajs

21 days ago

preview code

download

raw

22.4 kB

🖱️ Air Mouse Pro – Complete Cross-Platform Remote Control System

University of Tehran – Embedded Systems Laboratory
Professional‑grade, AI‑powered, proximity‑aware remote control for presentations, media centers, and everyday productivity

📖 Table of Contents

Overview
Features
Architecture
Quick Start
Installation
- Desktop Server (Go)
- Android App
Configuration
Protocol Specification
Advanced Features
Statistics & Dashboard
Theming & Customization
Troubleshooting
Development
License

🎯 Overview

Air Mouse Pro transforms your Android phone into a smart, multi‑protocol remote control for any computer (Windows, macOS, Linux). It combines:

Six connection methods – TCP, WebSocket, UDP, Bluetooth LE, USB/Serial, Wi‑Fi Direct
AI‑powered gesture recognition – train custom gestures and classify them on‑device with 97% accuracy
Proximity‑aware auto lock/unlock – using Bluetooth 6.0 Channel Sounding (centimeter‑level accuracy)
Predictive mouse smoothing – Kalman filter hides network jitter
Contextual auto‑pause – phone detects when it’s placed flat and stops sending movement
Cross‑platform desktop server – written in Go, with a modern Fyne GUI
Real‑time statistics – click counts, scrolls, gesture recognition, performance metrics

✨ Features

📡 Multi‑Protocol Connectivity

Protocol	Port	Use Case
TCP	8080	Low‑latency, reliable control
WebSocket	8081	Browser‑based clients
UDP discovery	8082	Automatic server detection
Bluetooth LE	-	HID proxy, BLE device emulation
USB/Serial	-	Wired connection

🧠 AI‑Powered Gesture Recognition

Record custom gestures (e.g., circle, swipe, thumbs‑up)
Train a 1D Convolutional Neural Network (CNN) on your PC
On‑device inference via TensorFlow Lite – no internet required
Up to 97.5% accuracy in real‑time interaction

🔐 Proximity‑Aware Security (Bluetooth 6.0)

Channel Sounding – measures distance with ±30‑50 cm accuracy
Automatically locks screen when phone moves beyond threshold (e.g., 4 meters)
Automatically unlocks when you return (optional)
Falls back to RSSI‑based proximity on older devices

📊 Predictive Mouse Movement (Kalman Filter)

Tracks position and velocity in real time
Predicts next cursor position to compensate for network jitter
Configurable blend factor (raw vs. predicted)
Perceptually removes lag – movement feels instantaneous

🔄 Contextual Auto‑Pause

Accelerometer detects when phone is placed flat on a table
Automatically pauses sending movement events
Resumes instantly when you pick up the phone
Eliminates accidental cursor jumps

📈 Real‑Time Statistics Dashboard

Clicks / double‑clicks / right‑clicks / scrolls (live updates)
Connected devices list with uptime and idle time
Server uptime, endpoint display, AI smoothing status
Performance metrics: CPU, memory, active goroutines

🎨 Modern GUI (Fyne)

15+ themes – Dark, Light, Pure Black, High Contrast, Ocean, Sunset, Forest, Purple, Cherry, Neon, Lavender, Mint, Peach, Sky
Tabbed interface – Dashboard, Devices, Network, Settings, Logs
QR code generation for easy pairing
Configurable sensitivity, smoothing, acceleration, rate limiting

📱 Android App Features

Motion control (gyroscope/accelerometer)
Touchpad mode with gestures
Voice commands (optional)
Custom gesture recorder
Bluetooth HID mode (acts as a real Bluetooth mouse)
QR scanner for auto‑configuration

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                         ANDROID APP                                      │
├─────────────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │ Motion      │  │ Touchpad    │  │ Gesture     │  │ Proximity   │     │
│  │ Sensors     │  │ Gestures    │  │ Inference   │  │ Service     │     │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘     │
│         │                │                │                │             │
│         └────────────────┴────────────────┴────────────────┘             │
│                                   │                                      │
│                    ┌──────────────▼──────────────┐                       │
│                    │   Protocol Abstraction      │                       │
│                    │ (TCP / WS / UDP / BLE)      │                       │
│                    └──────────────┬──────────────┘                       │
└───────────────────────────────────┼──────────────────────────────────────┘
                                    │
                    ┌───────────────▼────────────────────────┐
                    │          GO DESKTOP SERVER              │
                    ├─────────────────────────────────────────┤
                    │ ┌─────────┐ ┌─────────┐ ┌───────────┐   │
                    │ │ TCP     │ │ WebSock │ │ UDP Disc  │   │
                    │ │ Server  │ │ Server  │ │ overy     │   │
                    │ └────┬────┘ └────┬────┘ └─────┬─────┘   │
                    │      │           │            │         │
                    │      └───────────┴────────────┘         │
                    │              │                          │
                    │    ┌─────────▼──────────┐               │
                    │    │  Message Router    │               │
                    │    │ & Device Registry  │               │
                    │    └─────────┬──────────┘               │
                    │              │                          │
                    │    ┌─────────▼──────────┐               │
                    │    │  Mouse Controller  │               │
                    │    │  + AI Smoothing    │               │
                    │    │  + Kalman Predict  │               │
                    │    │  + Proximity Mgr   │               │
                    │    └─────────┬──────────┘               │
                    │              │                          │
                    │    ┌─────────▼──────────┐               │
                    │    │  Native Input      │               │
                    │    │  (Win32/CoreGr/    │               │
                    │    │   uinput)          │               │
                    │    └────────────────────┘               │
                    └─────────────────────────────────────────┘
                                    │
                    ┌───────────────▼────────────────────────┐
                    │          DESKTOP ENVIRONMENT           │
                    │  (Lock/Unlock via loginctl / DBus)      │
                    └────────────────────────────────────────┘

🚀 Quick Start

One‑line setup (Linux / macOS / WSL)

git clone https://github.com/yourusername/airmouse-pro.git
cd airmouse-pro
make install   # or: go build -o airmouse-server ./cmd/airmouse-server && ./airmouse-server

Android app

Install the APK from Releases or build from source
Grant Bluetooth and Location permissions
Open the app – scan the QR code shown on the desktop server (Network tab)
Start moving your phone!

📦 Installation

Desktop Server (Go)

Prerequisites

Go 1.23+
For Bluetooth support: bluez (Linux), Win32 Bluetooth stack (Windows), or CoreBluetooth (macOS)
For AI smoothing: ONNX Runtime shared library (auto‑fetched by go module)

Build

git clone https://github.com/yourusername/airmouse-pro.git
cd airmouse-pro
go mod download
go build -o airmouse-server ./cmd/airmouse-server

Run

./airmouse-server

The GUI will open. Start the server from the Dashboard tab.

Android App

From source (Android Studio)

git clone https://github.com/yourusername/airmouse-pro-android.git
open in Android Studio → Build → Run

Pre‑built APK Download from Releases

⚙️ Configuration

The server stores settings in ~/.config/airmouse/config.json (Linux/macOS) or %APPDATA%\airmouse\config.json (Windows).

Example configuration

{
  "host": "0.0.0.0",
  "port": 8080,
  "websocket_port": 8081,
  "enable_tcp": true,
  "enable_websocket": true,
  "enable_bluetooth": true,
  "sensitivity": 0.5,
  "theme": "dark",
  "enable_ai_smoothing": true,
  "ai_model_path": "models/mouse_smoothing.onnx",
  "enable_predictive": true,
  "predictive_blend_factor": 0.6,
  "enable_personalization": true,
  "personalization_buffer": 2000,
  "auto_lock_enabled": true,
  "auto_unlock_enabled": false,
  "proximity_near_threshold": 2.0,
  "proximity_far_threshold": 4.0,
  "log_level": "info"
}

Changing settings

GUI: Settings tab – all options are live‑updated and saved automatically.
CLI: Edit the JSON file directly and restart the server.

📡 Protocol Specification

The server accepts JSON‑line messages over TCP or WebSocket. Each message must end with \n.

Client → Server

Type	Payload	Description
`move`	`{"dx": 1.5, "dy": 2.0}`	Move mouse by delta pixels
`click`	`{"button": "left"}`	Left or right click
`doubleclick`	`{}`	Double click
`rightclick`	`{}`	Right click
`scroll`	`{"delta": 1}`	Scroll (positive = up)
`hello`	`{"name": "MyPhone", "version": "2.0"}`	Identify device
`ping`	`{}`	Keep‑alive
`gesture`	`{"gesture": "CircleCW", "confidence": 0.92}`	Recognized custom gesture
`proximity`	`{"is_near": true, "distance": 1.23}`	Distance update (Android 16+)
`control`	`{"command": "pause_movement"}`	Pause/resume movement

Server → Client

Type	Payload	Description
`welcome`	`{"server":"AirMouse","version":"3.0"}`	Sent after hello
`ping`	`{}`	Heartbeat request
`pong`	`{}`	Heartbeat reply
`ack`	`{"id": "your_message_id"}`	Acknowledgment (optional)

🧠 Advanced Features

AI‑Powered Gesture Recognition

How it works

Record – Use the Android “Gesture Studio” to record custom gestures (5–10 repetitions).
Export – Export the dataset as CSV and copy to PC.
Train – Run the Python training script (train_gesture.py) which builds a 1D‑CNN model.
Deploy – Copy the resulting gesture_model.tflite and gesture_labels.json back to Android assets.
Recognize – The app runs real‑time inference (on‑device) and sends recognized gestures to the server.

Training script (PC side)

python train_gesture.py --dataset gestures_dataset.csv

Outputs: gesture_model.tflite, gesture_labels.json

Mapping gestures to actions (Go server)

switch gesture {
case "LeftSwipe":  robotgo.KeyTap("media_prev")
case "RightSwipe": robotgo.KeyTap("media_next")
case "CircleCW":   robotgo.KeyTap("audio_vol_up")
case "ThumbsUp":   robotgo.KeyTap("media_play_pause")
}

Proximity‑Aware Lock/Unlock

Requirements

Android 16+ with Bluetooth 6.0 hardware (Channel Sounding)
Linux kernel 6.15+ (for HCI commands) – otherwise RSSI fallback
Windows/macOS: RSSI only (less accurate)

Setup

Pair your phone via Bluetooth (optional but recommended)
Enable “Auto Lock/Unlock” in Android settings → Proximity
Calibrate: place phone at 0.5m, 1m, 2m, 5m and tap “Calibrate”
Set thresholds (default: near 2m, far 4m)

How it works

Phone measures distance using Channel Sounding (or RSSI)
Sends proximity messages over WebSocket every 1–2 seconds
Server locks screen when distance > far threshold (hysteresis applied)
Unlocks when distance < near threshold (if auto‑unlock enabled)

Fallback

If Bluetooth 6.0 is unavailable, the system uses RSSI‑based distance estimation (less accurate). You can still lock/unlock manually.

Predictive Mouse Movement (Kalman Filter)

Benefits

Compensates for network jitter and latency
Makes cursor movement feel instantaneous
Blend factor (0–1) controls influence of prediction (0 = raw, 1 = pure predicted)

Configuration (Settings tab or config.json)

"enable_predictive": true,
"predictive_blend_factor": 0.6

Technical details

State vector: [x, y, vx, vy]
Constant‑velocity model
Adaptive time step (dynamic dt)
Covariance reset on large jumps

Contextual Auto‑Pause

How it works

Android accelerometer low‑pass filtered (gravity vector)
If sqrt(gx² + gy²) < 0.2g, phone is considered flat
When flat, app sends {"type":"control","command":"pause_movement"}
Server stops processing move events until resume_movement is received

Enable/disable

Settings → Motion Behaviour → “Auto‑pause motion when phone is flat”

Benefits

No manual toggle – just put the phone down
Prevents accidental cursor jumps when phone is on a table

📊 Statistics & Dashboard

The Dashboard tab displays:

Statistic	Description
Click count	Total left clicks
Double click count	Total double clicks
Right click count	Total right clicks
Scroll count	Total scroll events
Connected devices	Number of active clients
Server uptime	Time since last start
AI smoothing status	Enabled / Disabled / Personalizing

The Logs tab shows real‑time events (info, warning, error) with filtering and export.

Performance metrics (GUI – bottom bar)

CPU usage (percent)
Memory usage (MB)
Active goroutines

🎨 Theming & Customization

Available themes

Theme Name	Description
`dark` (default)	Dark mode with blue accents
`light`	Light mode
`pure_black`	AMOLED‑friendly (#000000 background)
`high_contrast`	Accessibility‑optimised
`ocean`	Blue/teal scheme
`sunset`	Orange/amber warm tones
`forest`	Green/earth tones
`purple`	Purple/pink accents
`cherry`	Pink/red scheme
`neon`	Cyberpunk cyan/magenta
`lavender`	Soft purple pastel
`mint`	Green‑blue pastel
`peach`	Warm orange pastel
`sky`	Light blue pastel

Changing themes

GUI: Settings tab → Theme dropdown (instant preview)
Config: "theme": "ocean"

🐛 Troubleshooting

Server won't start

Check if the port is already in use: sudo lsof -i :8080
Ensure you have permission to bind to the port (Linux: non‑root can use ports >1024)

Android app cannot find server

Make sure both devices are on the same Wi‑Fi network
Disable VPN / firewall temporarily
Try using the QR code in the Network tab
Check UDP discovery port (8082) is not blocked

Mouse movement feels laggy

Reduce sensitivity (0.3–0.5 is typical)
Enable predictive movement (Kalman filter)
Check Wi‑Fi signal strength (RSSI > -65dBm recommended)

Gesture recognition not working

Ensure gesture_model.tflite and gesture_labels.json are in Android assets
Train with at least 5–10 repetitions per gesture
Check confidence threshold in code (default 0.7)

Proximity lock/unlock doesn't work

Verify both devices support Bluetooth 6.0 (Android 16+, Linux kernel 6.15+)
Calibrate the Channel Sounding (Android → Proximity → Calibrate)
Try RSSI fallback mode (disable “Use Channel Sounding” in settings)

AI smoothing model not loading

Ensure models/mouse_smoothing.onnx exists (download from Releases)
Check that ONNX Runtime library is installed (libonnxruntime.so on Linux)

🔧 Development

Building from source

# Desktop server
git clone https://github.com/yourusername/airmouse-pro
cd airmouse-pro
go mod download
go build -o airmouse-server ./cmd/airmouse-server

# Android app
git clone https://github.com/yourusername/airmouse-pro-android
open in Android Studio → Build → Build APK

Running tests

go test -v ./...

Training a new gesture model

Record gestures using Android “Gesture Studio”
Export CSV and copy to PC

Run training script:

python train_gesture.py --dataset gestures_dataset.csv

Copy gesture_model.tflite and gesture_labels.json to app/src/main/assets/
Rebuild Android app

Generating a dummy model (for initial setup)

python generate_dummy_model.py

Place output files in Android assets folder.

📄 License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files, to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software.

🙏 Acknowledgements

Fyne – cross‑platform GUI toolkit
ONNX Runtime – high‑performance inference
TensorFlow Lite – on‑device ML
robotgo – mouse control (fallback)
gorilla/websocket – WebSocket server
hashicorp/mdns – Bonjour/Zeroconf
gopsutil – system metrics
All contributors and open‑source maintainers

Built with ❤️ at University of Tehran – Winter 2025

Xet Storage Details

Size:: 22.4 kB
Xet hash:: 4dc04ab230d22b694acdc76291d9893616a6ab007a527817ef167e1e947dab10

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.