tahamajs/CPS/Files/ComputerAssignments/CA2 - Storage Bucket

📱 Air Mouse Pro: Cyber-Physical Remote Control System

Welcome to the definitive repository for the Air Mouse Pro system, a state-of-the-art Cyber-Physical System (CPS) designed to turn a standard Android smartphone into a low-latency, high-precision, wireless remote pointer, touchpad simulator, gaming controller, and system command center.

Developed as part of the Cyber-Physical & Embedded Systems curriculum, this project demonstrates advanced concepts in real-time sensor processing, sensor fusion, wireless communication protocols, noise filtration, and system-level performance profiling.

---

🏗️ System Architecture

The Air Mouse system is divided into two main components:

1. Android Client (code/android): Written in Kotlin using Jetpack Compose (Material 3), Dagger Hilt for dependency injection, Room DB for local storage, and standard Android Sensors APIs. It captures raw IMU signals, runs a manual implementation of the Madgwick AHRS algorithm, detects gestures, and transmits processed commands.
2. Go PC Server (code/pc/airmouse_go_new): A concurrent, multi-protocol Go executable that acts as the coordinator. It manages connection channels (TCP, WebSocket, and UDP), processes incoming command streams (applying Kalman filters, tremor filters, and trajectory predictions), and replicates mouse/keyboard events on the host OS via native robotgo/PyAutoGUI bindings.

graph TD
    %% Android Client Pipeline
    subgraph Android Client (Codebase: code/android)
        A[Android IMU Sensors] -->|Raw Accel / Gyro / Mag| B[3-Axis Calibration Manager]
        B -->|Corrected IMU Data| C[Madgwick AHRS Sensor Fusion]
        C -->|Euler Angles / Quaternions| D[Gesture Detector & Sensitivity Mapper]
        D -->|Move / Click / Scroll Events| E[Protocol Serializer]
        E -->|UDP / WebSocket / TCP| F[Connection Manager]
    end

    %% Wireless Network
    subgraph Network Transport (LAN Wi-Fi / USB)
        F -->|High-Frequency Stream: Port 9093| G((UDP Input Socket))
        F -->|Reliable Commands: Port 9091| H((WebSocket Socket))
        I((UDP Broadcast: Port 9092)) <.->|mDNS Auto-Discovery| F
    end

    %% PC Desktop Server
    subgraph Go PC Server (Codebase: code/pc/airmouse_go_new)
        G --> J[Server Protocol Listeners]
        H --> J
        J --> K[Go Connection Hub & Client Registry]
        K --> L[Adaptive Smoothing & Tremor Filters]
        L --> M[Predictive Kalman Filter]
        M --> N[Host Controller API Translation]
        N -->|Native Events| O[PyAutoGUI / RobotGo API]
    end

    %% Host OS Desktop
    subgraph Host OS Desktop
        O --> P[Cursor Translation & Key Injector]
    end

    classDef android fill:#3DDC84,stroke:#333,stroke-width:2px,color:#000;
    classDef server fill:#00ADD8,stroke:#333,stroke-width:2px,color:#fff;
    classDef network fill:#FF9900,stroke:#333,stroke-width:2px,color:#000;
    class A,B,C,D,E,F android;
    class J,K,L,M,N,O server;
    class G,H,I network;

---

🧮 Cyber-Physical & Embedded Systems Concepts

To achieve an experience comparable to commercial hardware, the project implements several key digital signal processing (DSP) and networking architectures.

1. Motion Sensing & IMU Kinematics

The system leverages the mobile device's Inertial Measurement Unit (IMU) using three primary sensors:

• Accelerometer (Sensor.TYPE_ACCELEROMETER): Measures proper acceleration ($\vec{a}$). While highly reliable for identifying the gravity vector ($\vec{g}$), it is prone to high-frequency noise from linear movements and hand tremors.
• Gyroscope (Sensor.TYPE_GYROSCOPE): Measures angular velocity ($\vec{\omega}$). Integrating angular velocity over time provides highly responsive short-term orientation changes but suffers from cumulative drift over time due to low-frequency noise (bias).
• Magnetometer (Sensor.TYPE_MAGNETIC_FIELD): Measures Earth's magnetic field ($\vec{B}$) to establish a geographic heading, acting as an absolute yaw reference. It is highly susceptible to hard-iron and soft-iron magnetic interference from nearby metals and electronics.

2. Sensor Fusion (Madgwick AHRS)

To resolve the limitations of individual sensors, orientation is tracked using the Madgwick AHRS (Attitude and Heading Reference System) algorithm, implemented in SensorFusion.kt.

• Integration: The gyroscope readings are integrated to predict the orientation quaternion ($q$).
• Correction: A gradient-descent optimization calculates the error between the predicted orientation and the measurements from the accelerometer (for pitch/roll) and magnetometer (for yaw/heading).
• Fusing Gain ($\beta$): The parameter $\beta$ (default 0.041) controls the trade-off between responsive gyroscope integration and drift-correcting accelerometer/magnetometer references:

$$\beta =\sqrt{\frac{3}{4}}\cdot {\tilde{\omega }}_{\text{max}}\backslash beta\; =\; \backslash sqrt\{\backslash frac\{3\}\{4\}\}\; \backslash cdot\; \backslash tilde\{\backslash omega\}\_\{\backslash text\{max\}\}$$

where $\tilde{\omega}_{\text{max}}$ represents the maximum gyroscope measurement error.

3. Noise Filtration & Smoothing

• Kalman Filtering (1D & 2D): Implemented in the Go server (kalman2d.go), it uses a linear quadratic estimation to model the velocity and acceleration of the mouse pointer. It acts as a predictive filter that minimizes latency by anticipating the next cursor coordinate.
• Tremor Filter: A low-pass moving-average filter running on the server that filters out high-frequency micro-shakes ($>6\text{ Hz}$) typical of human hand tremors while preserving deliberate pointer trajectories.
• B-Spline Path Humanizer: Smooths out discrete grid coordinates into continuous, organic curves before translating them to the host screen, improving pointer control.

4. Low-Latency Wireless Communication

A dual-channel protocol is used to balance latency and reliability:

• UDP Data Stream (Port 9093): High-frequency pointer updates ($\approx 50\text{--}100\text{ Hz}$) are sent over connectionless, unreliable UDP. This avoids the handshake, congestion control, and TCP head-of-line blocking overhead, keeping local transmission latency below $1.5\text{ ms}$.
• WebSocket / TCP Channel (Ports 9090 & 9091): Critical operations (mouse clicks, scrolls, keyboard shortcuts, files) require guaranteed delivery. These are sent over WebSockets with an application-layer ACK-and-Retransmission protocol:
  • Messages are assigned an incremental ID.
  • The server replies with an ack packet.
  • If the client does not receive an ACK within $500\text{ ms}$, the command is retransmitted up to $3$ times before logging a network fault.

---

🛠️ Calibration Mathematical Models

To eliminate noise, sensor offsets, and soft/hard-iron distortions, the Android application features a calibration module:

1. Gyroscope Bias Calibration

Computes the mean angular velocity offsets while the device is kept flat and stationary over $100$ samples:

$${\vec{\omega }}_{\text{bias}}=\frac{1}{N}\sum _{i=1}^N{\vec{\omega }}_{\text{raw},i}\backslash vec\{\backslash omega\}\_\{\backslash text\{bias\}\}\; =\; \backslash frac\{1\}\{N\}\backslash sum\_\{i=1\}^\{N\}\; \backslash vec\{\backslash omega\}\_\{\backslash text\{raw\},\; i\}$$

Future readings are corrected by subtraction:

$${\vec{\omega }}_{\text{corrected}}={\vec{\omega }}_{\text{raw}}-{\vec{\omega }}_{\text{bias}}\backslash vec\{\backslash omega\}\_\{\backslash text\{corrected\}\}\; =\; \backslash vec\{\backslash omega\}\_\{\backslash text\{raw\}\}\; -\; \backslash vec\{\backslash omega\}\_\{\backslash text\{bias\}\}$$

2. Magnetometer Hard-Iron & Soft-Iron Calibration

During a figure-8 motion, the app captures $200$ samples to map the local magnetic field. It identifies the maximum and minimum values on each axis to compute hard-iron offsets and soft-iron scaling factors:

$${\vec{B}}_{\text{offset}}=\frac{\max (\vec{B})+\min (\vec{B})}{2}\backslash vec\{B\}\_\{\backslash text\{offset\}\}\; =\; \backslash frac\{\backslash max(\backslash vec\{B\})\; +\; \backslash min(\backslash vec\{B\})\}\{2\}$$

$${\vec{B}}_{\text{scale}}=\frac{{\vec{B}}_{\text{avg\_range}}}{\max (\vec{B})-\min (\vec{B})}\backslash vec\{B\}\_\{\backslash text\{scale\}\}\; =\; \backslash frac\{\backslash vec\{B\}\_\{\backslash text\{avg\backslash \_range\}\}\}\{\backslash max(\backslash vec\{B\})\; -\; \backslash min(\backslash vec\{B\})\}$$

$${\vec{B}}_{\text{corrected}}=({\vec{B}}_{\text{raw}}-{\vec{B}}_{\text{offset}})\odot {\vec{B}}_{\text{scale}}\backslash vec\{B\}\_\{\backslash text\{corrected\}\}\; =\; (\backslash vec\{B\}\_\{\backslash text\{raw\}\}\; -\; \backslash vec\{B\}\_\{\backslash text\{offset\}\})\; \backslash odot\; \backslash vec\{B\}\_\{\backslash text\{scale\}\}$$

where $\vec{B}_{\text{avg_range}}$ is the average coordinate span across the three axes.

3. Accelerometer 6-Position Reference Calibration

Aligns the device along the 6 orthogonal axes ($x+, x-, y+, y-, z+, z-$ facing the earth's gravity $g = 9.81\text{ m/s}^2$) to solve for scale and offset parameters:

$${\vec{a}}_{\text{scale}}=\frac{\sum ({\vec{a}}_{\text{measured}}\cdot {\vec{a}}_{\text{expected}})}{\sum ({\vec{a}}_{\text{measured}}^2)}\backslash vec\{a\}\_\{\backslash text\{scale\}\}\; =\; \backslash frac\{\backslash sum\; (\backslash vec\{a\}\_\{\backslash text\{measured\}\}\; \backslash cdot\; \backslash vec\{a\}\_\{\backslash text\{expected\}\})\}\{\backslash sum\; (\backslash vec\{a\}\_\{\backslash text\{measured\}\}^2)\}$$

$${\vec{a}}_{\text{offset}}=\frac{\sum ({\vec{a}}_{\text{expected}}-{\vec{a}}_{\text{measured}}\cdot {\vec{a}}_{\text{scale}})}{N}\backslash vec\{a\}\_\{\backslash text\{offset\}\}\; =\; \backslash frac\{\backslash sum\; (\backslash vec\{a\}\_\{\backslash text\{expected\}\}\; -\; \backslash vec\{a\}\_\{\backslash text\{measured\}\}\; \backslash cdot\; \backslash vec\{a\}\_\{\backslash text\{scale\}\})\}\{N\}$$

$${\vec{a}}_{\text{corrected}}=\frac{{\vec{a}}_{\text{raw}}-{\vec{a}}_{\text{offset}}}{{\vec{a}}_{\text{scale}}}\backslash vec\{a\}\_\{\backslash text\{corrected\}\}\; =\; \backslash frac\{\backslash vec\{a\}\_\{\backslash text\{raw\}\}\; -\; \backslash vec\{a\}\_\{\backslash text\{offset\}\}\}\{\backslash vec\{a\}\_\{\backslash text\{scale\}\}\}$$

---

🌟 Complete Features Breakdown

Feature	Subsystem	Description	Code References
Real-time Pointer	Motion Engine	Uses the Madgwick AHRS output to map device pitch and roll changes to cursor coordinates, filtering out low-level noise via a customizable dead zone.	GestureDetector.kt
3-Axis Calibration Center	Local Pre-processing	A Wizard UI that guides users through gyroscope, accelerometer (6-position), and magnetometer calibration.	CalibrationHelper.kt
Touchpad Simulator	Touch Input	Converts mobile screen touches to pointer movement, supporting tap-to-click, double-tap, and two-finger scroll gestures.	TouchpadViewModel.kt
Custom Gestures Studio	AI Classification	Records gesture paths and uses a Dynamic Time Warping (DTW) algorithm and particle filters to trigger custom keyboard macros (e.g. circle for "Browser Refresh").	EnhancedGestureDetector.kt
Voice Commands	Natural Interface	Recognizes speech input (using Android's SpeechRecognizer) to execute keyboard shortcuts or media commands (e.g., "mute", "play").	VoiceCommandFragment.kt
File Transfer Queue	Network Utility	A TCP socket queue that allows users to send files between their mobile device and PC by dragging and dropping.	FileTransferService.kt
Screen Mirroring	Video Streaming	Streams the PC desktop back to the phone screen using a JPEG frame sequence over UDP, validated by Start-Of-Image (0xFF 0xD8) headers.	ScreenMirroringService.kt
Gaming Mode	Specialized Input	Converts device steering (yaw/roll tilt) to keyboard inputs (a/d) for driving simulators, and gestures to key presses (space/r) for shooter controls.	GameProfilesManager.kt
Theme/Preferences Sync	State Syncing	Syncs themes (20+ presets) and calibration parameters between the client and server using JSON profile configurations.	DataSyncManager.kt

---

🔌 Port Architecture & Discovery

To avoid network conflicts with web services, the system runs on dedicated ports:

Port	Protocol	Purpose	Description
9090	TCP	File Queue & Configuration Sync	Stateful TCP connection for high-throughput files and profiles.
9091	WebSocket	Reliable Event Commands	Handles clicks, drags, scroll wheel ticks, and custom macros.
9092	UDP	Auto-Discovery	Broadcast/Listening channel for instant client-server pairing.
9093	UDP	High-Frequency Sensor Stream	Streams raw gyroscope coordinate outputs to minimize latency.

Auto-Discovery Protocol Flow

Android Client                                                    PC Go Server
      |                                                                 |
      | ---- [UDP Broadcast: 9092] "AIRMOUSE_DISCOVER" --------------> |
      |                                                                 | (Receive broadcast)
      |                                                                 | (Read host configuration)
      | <--- [UDP Unicast: Client Port] "AIRMOUSE_SERVER:9093:Name:3.0" |
      |                                                                 |
(Parse payload info)
(Save host IP/Port)
      |                                                                 |
      | ---- [WebSocket: 9091] Connect / Hello -----------------------> |

---

🚀 Installation & Setup Guides

1. PC Go Server Setup

Ensure you have Go 1.23+ installed.

# Navigate to the server directory
cd code/pc/airmouse_go_new

# Download and verify dependencies
make deps

# Build the executable for your current OS
make build

# Run the server
./airmouse-server

If you have compiler problems with Fyne (GUI toolkit) dependencies, you can compile a non-GUI console version by running: go build -tags noai -o airmouse-server ./cmd/airmouse-server

---

2. Operating System Permissions

To translate network commands to cursor movement, you must grant the server system permissions:

macOS

Since macOS restricts virtual inputs, you must add the terminal program (e.g., Terminal, iTerm2, or VS Code) or the compiled airmouse-server binary to the Accessibility list:

1. Open System Settings -> Privacy & Security -> Accessibility.
2. Click the + button and add your terminal application or the compiled airmouse-server binary.
3. Enable the checkbox.

Linux

On Linux, the server requires permissions to write to the kernel user input interface (/dev/uinput) or utilize X11 utilities:

# Grant access to uinput
sudo usermod -aG input $USER
sudo udevadm trigger

# If uinput is not available, install xdotool for X11 fallback:
sudo apt-get install xdotool

Windows

Run the command prompt or powershell as Administrator before executing the server to allow the insertion of simulated keyboard events into privileged applications.

---

3. Android Client Compilation & Run

Ensure you have Android Studio (Hedgehog or newer) and JDK 17 installed.

# Navigate to the Android directory
cd code/android

# Clean the workspace
./gradlew clean

# Build the debug APK
./gradlew assembleDebug

# Install on a connected physical device or emulator via ADB
adb install -r app/build/outputs/apk/debug/app-debug.apk

---

🧪 Test Suites

1. Go PC Server Test Suite

The Go server includes unit tests for sensor fusion, jitter buffers, and protocol parsing.

# Run all unit tests
make test

# Run tests with race condition detection
go test -v -race -timeout 30s ./...

# Run tests with a HTML coverage report
make test-coverage

# Run benchmarks for sensor processing loops
make bench

2. Android Client Test Suite

The Android codebase contains unit tests (for use cases and ViewModels) and instrumented Compose tests.

# Run unit tests on host JVM
./gradlew testDebugUnitTest

# Run instrumented UI and integration tests (requires connected device/emulator)
./gradlew connectedAndroidTest

---

📊 Perfetto Profiling & Analysis

To analyze the performance of the system and measure sensor-to-cursor latency, the project integrates with Google Perfetto.

1. Custom Tracepoints

Trace points are inserted in SensorService.kt to monitor the processing pipeline:

• AirMouseApp.Sensors.sensor_read: Measures time spent reading raw sensors.
• AirMouseApp.Filter.complementary: Tracks the Madgwick fusion execution.
• AirMouseApp.Filter.compute_delta: Tracks gesture calculations.
• AirMouseApp.Communication.send: Measures transmission time.

2. Recording a Performance Trace

Run the trace recorder script using the provided config.pbtx:

# Record a 15-second trace from your device
python3 "Files/12- record_android_trace" -c "config.pbtx" -o trace_file.perfetto-trace -t 15s

3. Running the Python Analyzer

Use the custom SQL script perfetto_analyzer.py to query the trace database and extract performance metrics:

# Analyze trace file
python3 code/pc/perfetto_analyzer.py trace_file.perfetto-trace

The analyzer script evaluates the recorded trace to output:

• Average sensor callback processing times.
• CPU thread scheduling and waiting overheads.
• Madgwick filter execution times.
• End-to-end latency from hardware register readings to network transmissions.

---

Developed by the Cyber-Physical & Embedded Systems Lab. For issues or feature requests, contact the project maintainers.