| --- |
| title: Sam |
| emoji: 🤖 |
| colorFrom: indigo |
| colorTo: purple |
| sdk: gradio |
| app_file: app.py |
| pinned: false |
| tags: |
| - 0g-hackathon |
| - reachy_mini |
| - reachy_mini_python_app |
| --- |
| |
| # Sam |
|
|
| Sam is a **Reachy Mini AI health companion** powered by **0G intelligence** and |
| durable memory on **0G Storage**. Medication support is the first real workflow: |
| Sam listens to pharmacy instructions, reasons over them with 0G Compute, keeps a |
| local memory for fast retrieval, syncs that memory to 0G Storage, and acts on it |
| through voice, movement, and reminders. One control loop: |
|
|
| > **listen** (0G Whisper STT) → *optionally* **see** (0G Qwen3-VL) → **think** |
| > (0G GLM-5 chat) → **speak** (local TTS) → **express** with head + antennas. |
|
|
| The chat model returns both *what to say* and *how it feels*, so Sam nods, |
| tilts, perks its antennas and bobs while talking. |
|
|
| Runs identically in **MuJoCo simulation** and on physical hardware. |
|
|
| ## What's wired up |
|
|
| | Capability | 0G model | Status | |
| |---|---|---| |
| | 🧠 Chat / reasoning | `zai-org/GLM-5-FP8` | ✅ enabled | |
| | 👂 Speech-to-text | `openai/whisper-large-v3` | ✅ enabled | |
| | 👁️ Vision | `qwen/qwen3-vl-30b-a3b-instruct` | ⚙️ code ready, **fund a sub-account to enable** | |
| | 🗣️ Text-to-speech | local (`say` / `espeak`) | no 0G TTS provider yet | |
| | 💊 Health plan + medication support | local JSON + 0G Storage sync | ✅ enabled | |
|
|
| ## Medication safety |
|
|
| Sam is a health companion, not a doctor. It records medication instructions that |
| the user says came from a pharmacist or doctor, then uses 0G intelligence to add |
| short, non-prescriptive safety context and uses 0G Storage as durable memory. |
| Sam does **not** prescribe, change dosage, validate medication instructions, |
| identify pills as medical truth, or claim that someone swallowed a medicine. |
|
|
| Sam may add a short non-prescriptive caveat when 0G intelligence or local safety |
| rules detect a well-known concern. For example, if a user mentions ulcer history |
| with naproxen, Sam can save the reminder while advising them to cross-check with |
| a doctor or pharmacist. |
|
|
| ## Setup |
|
|
| Requires Python 3.10–3.13 (the mujoco deps don't support 3.14). We use [`uv`](https://docs.astral.sh/uv/). |
|
|
| ```bash |
| uv venv --python 3.12 |
| source .venv/bin/activate |
| uv pip install -e ".[mujoco]" |
| cp .env.example .env # then fill in your 0G keys (see below) |
| ``` |
|
|
| ### 0G keys |
|
|
| Each provider needs its **own** host + key (a key is bound to one provider): |
|
|
| ```bash |
| 0g-compute-cli inference list-providers # find provider addresses + models |
| 0g-compute-cli inference get-secret --provider <ADDR> --duration 0 |
| ``` |
|
|
| Copy the printed `app-sk-…` token and the `curl` host into the matching |
| `OG_CHAT_* / OG_STT_* / OG_VISION_*` block in `.env`. |
| **Mainnet = real funds** — each call spends a tiny amount of 0G. |
|
|
| ### Hugging Face Space |
|
|
| This repo is a **Gradio Space** so 0G calls run server-side and API keys stay |
| private. In the Space settings, add these as **Secrets**: |
|
|
| ```text |
| OG_CHAT_BASE_URL |
| OG_CHAT_MODEL |
| OG_CHAT_PROVIDER |
| OG_CHAT_API_KEY |
| OG_STT_BASE_URL |
| OG_STT_MODEL |
| OG_STT_PROVIDER |
| OG_STT_API_KEY |
| ``` |
|
|
| Add `OG_VISION_*` too once the vision provider is funded. Rebuild the Space after |
| changing secrets. |
|
|
| ### Sam medication memory |
|
|
| Sam reads and writes medication memory locally first so reminders are fast and |
| reliable. When 0G Storage is configured, every local change is marked for sync |
| and a background sync keeps retrying until 0G Storage has the latest memory. |
|
|
| ```text |
| SAM_MEMORY_PATH=/path/to/sam_memory.json |
| OG_STORAGE_ENABLED=true |
| OG_STORAGE_INDEXER_URL=... |
| OG_STORAGE_RPC_URL=... |
| OG_STORAGE_PRIVATE_KEY=... |
| OG_STORAGE_MEMORY_ROOT=... |
| ``` |
|
|
| `OG_STORAGE_PRIVATE_KEY` must be a secret. If 0G Storage is unavailable, Sam |
| continues from local JSON and keeps the memory marked as pending sync. |
|
|
| ## Run |
|
|
| **1. Start the sim daemon** (separate terminal). On macOS use `--no-media` |
| (the daemon's media server stalls headless on a camera/mic permission prompt): |
|
|
| ```bash |
| source .venv/bin/activate |
| reachy-mini-daemon --sim --no-media # add --headless to skip the 3D viewer |
| ``` |
|
|
| **2. Talk to Sam:** |
|
|
| ```bash |
| # Check real 0G config without needing the robot/sim daemon: |
| python -m mr_reachy --health-check --no-robot |
| |
| # Single real 0G exchange without needing the robot/sim daemon: |
| python -m mr_reachy --once "Tell me a robot joke" --no-robot --no-speak |
| |
| # Single real 0G exchange with robot/sim motion: |
| python -m mr_reachy --once "Tell me a robot joke" |
| |
| # Type at it with robot/sim motion + local speech: |
| python -m mr_reachy --text |
| |
| # Full voice conversation (needs a mic): |
| python -m mr_reachy |
| ``` |
|
|
| Say "what do you see?" to trigger the vision path (once a vision sub-account is funded). |
| Say "bye" / type `quit` to exit. |
|
|
| Medication demo phrases: |
|
|
| ```text |
| Sam, I need to take metformin three times a day for five days. |
| I took it. |
| ``` |
|
|
| If the user does not provide exact times, Sam uses safe default reminder slots: |
| `09:00`, `14:00`, and `20:00` for three-times-per-day schedules. |
|
|
| ## On the robot (demo station — Reachy Mini Wireless) |
|
|
| At the demo station the app runs **on the robot's Pi**, started from the dashboard |
| for a short slot. On hardware Sam uses the robot's **onboard mic, speaker |
| and camera** (via the daemon media manager) and **antenna push-to-talk** instead |
| of the laptop mic. In the desktop simulator, the dashboard app uses the laptop |
| mic/speaker path automatically. Override with `MR_REACHY_BACKEND=robot` or |
| `MR_REACHY_BACKEND=local` if the SDK guesses wrong. |
|
|
| > ⚠️ The on-robot I/O path is built against the SDK API but has **not been tested |
| > on a physical robot yet**. Rehearse before the slot and tune the two values below. |
|
|
| **1. 0G keys on the robot** — your `.env` is gitignored and does **not** travel with |
| the published app, and the framework injects no secrets. So the app would run but |
| never reach 0G. SSH into the robot and drop your keys where `config.py` looks for them: |
|
|
| ```bash |
| # on the robot (over SSH): |
| mkdir -p ~/.config/mr_reachy |
| nano ~/.config/mr_reachy/.env # paste the OG_CHAT_* / OG_STT_* lines from your local .env |
| ``` |
| (Alternatively point `MR_REACHY_ENV` at a keys file.) Keep keys off the public Space — |
| publish with `--private` if you ever bake config in. |
|
|
| **2. Speech — install Piper on the robot** (0G has no TTS provider): |
|
|
| ```bash |
| pip install "mr-reachy[robot]" # piper-tts + pillow |
| # download a voice once, then point the app at it: |
| export PIPER_MODEL=/path/to/en_US-amy-medium.onnx # (+ matching .onnx.json) |
| ``` |
|
|
| **2. Internet** — the robot must reach the 0G endpoints |
| (`compute-network-*.integratenetwork.work`). Venue Wi-Fi may block this; test early. |
|
|
| **3. Antenna push-to-talk** — press an antenna to talk, release to send. Tune the |
| sensitivity if needed: |
|
|
| ```bash |
| export MR_ANTENNA_THRESHOLD=0.5 # radians of deflection that counts as a press |
| ``` |
|
|
| **4. Publish & install:** |
|
|
| ```bash |
| # Run the checker DEACTIVATED — an active venv's $VIRTUAL_ENV breaks its nested temp venv: |
| env -u VIRTUAL_ENV .venv/bin/reachy-mini-app-assistant check . |
| reachy-mini-app-assistant publish # requires `hf auth login` |
| ``` |
|
|
| Then **Install to Robot** from your Hugging Face Space (dashboard URL |
| `http://reachy-mini.local:8000`) and start it from the dashboard. The robot |
| auto-returns to its rest pose when the app stops. |
|
|
| The app is registered via the `reachy_mini_apps` entry point |
| (`mr_reachy.main:MrReachy`) in `pyproject.toml`. |
|
|
| ### Driving a robot from your machine (Lite, or testing Wireless over the network) |
|
|
| ```bash |
| python -m mr_reachy --on-robot # uses robot media + antenna push-to-talk |
| ``` |
|
|
| `ReachyMini()` auto-connects to a local daemon, falling back to `reachy-mini.local`. |
| For an explicit network connection use `connection_mode="network"`. |
|
|
| ## Layout |
|
|
| ``` |
| mr_reachy/ |
| main.py # ReachyMiniApp + conversation loop + CLI |
| og_client.py # 0G Compute: chat / STT / vision (OpenAI-compatible) |
| medication.py # medication plans, doses, parser, confirmation intent |
| storage.py # local JSON cache + background 0G Storage sync |
| reminders.py # due-dose reminder loop + confirmation/missed-dose handling |
| io_backends.py # Local (laptop) vs Robot (onboard media + antenna push-to-talk) |
| tts.py # on-device speech synthesis (Piper / say / espeak) |
| expressions.py # emotion -> head pose + antenna gestures |
| audio.py # laptop mic VAD capture + local TTS |
| config.py # loads .env into typed settings |
| ``` |
|
|
| See `plan.md` for the design and `agents.local.md` for environment notes. |
|
|
| ## Validation |
|
|
| ```bash |
| python -m unittest tests.test_medication tests.test_storage tests.test_reminders |
| python -m py_compile app.py mr_reachy/main.py mr_reachy/medication.py mr_reachy/storage.py mr_reachy/reminders.py |
| ``` |
|
|
