README.md

metadata

title: Memrl Canvas
emoji: 🐢
colorFrom: green
colorTo: green
sdk: gradio
sdk_version: 6.16.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: Completely offline audio command based canvas implementation
tags:
  - track:backyard
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:fieldnotes

MemRL Audio Canvas

Welcome to the MemRL Audio Canvas, a completely voice-controlled vector drawing dashboard hosted on Hugging Face Spaces. The application leverages a hybrid intelligence system combining local reinforcement learning, rule-based heuristics, and a serverless GPU/CPU backend powered by Modal.

---

🚀 System Architecture & Models Used

MemRL Canvas is divided into a lightweight frontend dashboard (serving the UI and local DB) and a serverless backend executing on Modal:

1. Whisper Tiny ASR (openai/whisper-tiny): Runs on a CPU-bound worker in the Modal backend for instantaneous speech-to-text transcription.
2. Gemma 4 NLU (google/gemma-4-E2B-it): A zero-shot language interpreter running on an A10G GPU tier in Modal. It compiles natural language instructions into structural JSON drawing actions.
3. MemRL Brain (Memory-Augmented Reinforcement Learning): A client/server Q-learning feedback system stored in a local SQLite database (memrl_memory.db) that overrides LLM queries when trusted mappings exist.
4. HTML5 Vector Canvas Engine: A premium glassmorphic interface that renders academic shapes (e.g. circles, stars, hearts, paths) with animated outlines and supports mouse-based drag-and-drop shape repositioning.

---

🧠 Speed and Robustness via MemRL

The core innovation of this workspace is the MemRL policy loop, which solves two primary limitations of traditional LLM interfaces: speed and robustness.

1. Robustness: Correcting ASR Errors

Speech transcription is often noisy or translates incorrectly. MemRL records user corrections. When the user modifies canvas coordinates or parameters in the JSON editor and clicks Accept, the new configuration is reinforced with a $+1.0$ reward and mapped directly to that spoken phrase in the SQLite Q-table. Subsequent spoken commands will recall the corrected layout instantly, bypassing transcription mistakes.

2. Speed: Instant Sub-Millisecond Execution

Querying Gemma 4 on GPU takes 1–2 seconds. MemRL matches spoken commands using Levenshtein distance and canonical slot overlap:

• High confidence ($Q \ge 0.8$): Auto-executes the drawing instantly via database recall in $<1\text{ms}$, completely bypassing the GPU pipeline.
• Medium confidence ($0.4 \le Q < 0.8$): Asks the user via a banner to confirm. If confirmed ($+1.0$ reward), the Q-value updates via: $$Q(s, a) \leftarrow Q(s, a) + 0.3 \cdot [ 1.0 - Q(s, a) ]$$ After three confirmations, it is promoted to instant auto-execution.
• Miss ($Q < 0.4$): Seamlessly routes the prompt to Gemma 4 on Modal.

---

🔗 Learn More & References

For demonstration details, code walkthroughs, and design updates, check out the following public resources:

• 📑 LinkedIn Post: Hugging Face Machine Learning & On-Device AI Share
• 🎥 YouTube Video Walkthrough: MemRL Audio Canvas Demonstration

---

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference