---
title: Memrl Canvas
emoji: 🐢
colorFrom: green
colorTo: green
sdk: gradio
sdk_version: 6.16.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: Completely offline audio command based canvas implementation
tags:
  - track:backyard
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:fieldnotes
---

# MemRL Audio Canvas

Welcome to the **MemRL Audio Canvas**, a completely voice-controlled vector drawing dashboard hosted on Hugging Face Spaces. The application leverages a hybrid intelligence system combining local reinforcement learning, rule-based heuristics, and a serverless GPU/CPU backend powered by **Modal**.

---

## 🚀 System Architecture & Models Used

MemRL Canvas is divided into a lightweight frontend dashboard (serving the UI and local DB) and a serverless backend executing on **Modal**:

1. **Whisper Tiny ASR** (`openai/whisper-tiny`): Runs on a CPU-bound worker in the Modal backend for instantaneous speech-to-text transcription.
2. **Gemma 4 NLU** (`google/gemma-4-E2B-it`): A zero-shot language interpreter running on an `A10G` GPU tier in Modal. It compiles natural language instructions into structural JSON drawing actions.
3. **MemRL Brain (Memory-Augmented Reinforcement Learning)**: A client/server Q-learning feedback system stored in a local **SQLite database** (`memrl_memory.db`) that overrides LLM queries when trusted mappings exist.
4. **HTML5 Vector Canvas Engine**: A premium glassmorphic interface that renders academic shapes (e.g. circles, stars, hearts, paths) with animated outlines and supports mouse-based drag-and-drop shape repositioning.

---

## 🧠 Speed and Robustness via MemRL

The core innovation of this workspace is the **MemRL policy loop**, which solves two primary limitations of traditional LLM interfaces: **speed** and **robustness**.

### 1. Robustness: Correcting ASR Errors
Speech transcription is often noisy or translates incorrectly. MemRL records user corrections. When the user modifies canvas coordinates or parameters in the JSON editor and clicks **Accept**, the new configuration is reinforced with a $+1.0$ reward and mapped directly to that spoken phrase in the SQLite Q-table. Subsequent spoken commands will recall the corrected layout instantly, bypassing transcription mistakes.

### 2. Speed: Instant Sub-Millisecond Execution
Querying Gemma 4 on GPU takes 1–2 seconds. MemRL matches spoken commands using Levenshtein distance and canonical slot overlap:
- **High confidence ($Q \ge 0.8$)**: Auto-executes the drawing instantly via database recall in **$<1\text{ms}$**, completely bypassing the GPU pipeline.
- **Medium confidence ($0.4 \le Q < 0.8$)**: Asks the user via a banner to confirm. If confirmed ($+1.0$ reward), the Q-value updates via:
  $$Q(s, a) \leftarrow Q(s, a) + 0.3 \cdot [ 1.0 - Q(s, a) ]$$
  After three confirmations, it is promoted to instant auto-execution.
- **Miss ($Q < 0.4$)**: Seamlessly routes the prompt to Gemma 4 on Modal.

---

## 🔗 Learn More & References

For demonstration details, code walkthroughs, and design updates, check out the following public resources:

- 📑 **LinkedIn Post**: [Hugging Face Machine Learning & On-Device AI Share](https://www.linkedin.com/posts/indroniel_huggingface-machinelearning-ondeviceai-share-7472072551855304704-RZI3/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAABYabAB9GxQ3ehp2Nq1iE8vpkm_fcQrjqY)
- 🎥 **YouTube Video Walkthrough**: [MemRL Audio Canvas Demonstration](https://youtu.be/ZwQmTncfFyc)

---

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference