| --- |
| title: Memrl Canvas |
| emoji: 🐢 |
| colorFrom: green |
| colorTo: green |
| sdk: gradio |
| sdk_version: 6.16.0 |
| python_version: '3.12' |
| app_file: app.py |
| pinned: false |
| license: mit |
| short_description: Completely offline audio command based canvas implementation |
| tags: |
| - track:backyard |
| - sponsor:modal |
| - achievement:offgrid |
| - achievement:welltuned |
| - achievement:offbrand |
| - achievement:fieldnotes |
| --- |
| |
| # MemRL Audio Canvas |
|
|
| Welcome to the **MemRL Audio Canvas**, a completely voice-controlled vector drawing dashboard hosted on Hugging Face Spaces. The application leverages a hybrid intelligence system combining local reinforcement learning, rule-based heuristics, and a serverless GPU/CPU backend powered by **Modal**. |
|
|
| --- |
|
|
| ## 🚀 System Architecture & Models Used |
|
|
| MemRL Canvas is divided into a lightweight frontend dashboard (serving the UI and local DB) and a serverless backend executing on **Modal**: |
|
|
| 1. **Whisper Tiny ASR** (`openai/whisper-tiny`): Runs on a CPU-bound worker in the Modal backend for instantaneous speech-to-text transcription. |
| 2. **Gemma 4 NLU** (`google/gemma-4-E2B-it`): A zero-shot language interpreter running on an `A10G` GPU tier in Modal. It compiles natural language instructions into structural JSON drawing actions. |
| 3. **MemRL Brain (Memory-Augmented Reinforcement Learning)**: A client/server Q-learning feedback system stored in a local **SQLite database** (`memrl_memory.db`) that overrides LLM queries when trusted mappings exist. |
| 4. **HTML5 Vector Canvas Engine**: A premium glassmorphic interface that renders academic shapes (e.g. circles, stars, hearts, paths) with animated outlines and supports mouse-based drag-and-drop shape repositioning. |
|
|
| --- |
|
|
| ## 🧠 Speed and Robustness via MemRL |
|
|
| The core innovation of this workspace is the **MemRL policy loop**, which solves two primary limitations of traditional LLM interfaces: **speed** and **robustness**. |
|
|
| ### 1. Robustness: Correcting ASR Errors |
| Speech transcription is often noisy or translates incorrectly. MemRL records user corrections. When the user modifies canvas coordinates or parameters in the JSON editor and clicks **Accept**, the new configuration is reinforced with a $+1.0$ reward and mapped directly to that spoken phrase in the SQLite Q-table. Subsequent spoken commands will recall the corrected layout instantly, bypassing transcription mistakes. |
|
|
| ### 2. Speed: Instant Sub-Millisecond Execution |
| Querying Gemma 4 on GPU takes 1–2 seconds. MemRL matches spoken commands using Levenshtein distance and canonical slot overlap: |
| - **High confidence ($Q \ge 0.8$)**: Auto-executes the drawing instantly via database recall in **$<1\text{ms}$**, completely bypassing the GPU pipeline. |
| - **Medium confidence ($0.4 \le Q < 0.8$)**: Asks the user via a banner to confirm. If confirmed ($+1.0$ reward), the Q-value updates via: |
| $$Q(s, a) \leftarrow Q(s, a) + 0.3 \cdot [ 1.0 - Q(s, a) ]$$ |
| After three confirmations, it is promoted to instant auto-execution. |
| - **Miss ($Q < 0.4$)**: Seamlessly routes the prompt to Gemma 4 on Modal. |
|
|
| --- |
|
|
| ## 🔗 Learn More & References |
|
|
| For demonstration details, code walkthroughs, and design updates, check out the following public resources: |
|
|
| - 📑 **LinkedIn Post**: [Hugging Face Machine Learning & On-Device AI Share](https://www.linkedin.com/posts/indroniel_huggingface-machinelearning-ondeviceai-share-7472072551855304704-RZI3/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAABYabAB9GxQ3ehp2Nq1iE8vpkm_fcQrjqY) |
| - 🎥 **YouTube Video Walkthrough**: [MemRL Audio Canvas Demonstration](https://youtu.be/ZwQmTncfFyc) |
|
|
| --- |
|
|
| Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
|
|
