| --- |
| license: apache-2.0 |
| pretty_name: keys-Auto Receipts Studio |
| tags: |
| - receipts |
| - gemma-4 |
| - vllm |
| - iphone |
| - autonomous-os |
| --- |
| |
| # keys-Auto Receipts Studio (iPhone / may add Autonomous Lamp Skill) |
|
|
| **v1.0 alpha** |
|
|
| iPhone (or desktop) photo → local **Gemma 4 12B-it** vision extract → SQLite. Optional [Autonomous OS](https://github.com/autonomous-ai/autonomous-os) Lamp skill (camera on the robot, 12B on the GPU box — the Lamp’s 6 GB cannot hold 12B). |
|
|
| GitHub: [`drowzeys/keys-Auto-Receipts-Studio`](https://github.com/drowzeys/keys-Auto-Receipts-Studio) |
| Hugging Face: [`drowzeys/keys-Auto-Receipts-Studio`](https://huggingface.co/drowzeys/keys-Auto-Receipts-Studio) |
|
|
| ## Pathways (pick a recipe) |
|
|
| Gemma 4 12B-it **never** runs on the Lamp (6 GB) and typically **not** on a Windows/Mac laptop. The GPU box is the brain. PC / Mac / Linux run the inbox + Review UI. iPhone is a Safari capture client on the same Wi‑Fi. |
|
|
| | Pathway | One-shot recipe | Gemma 4 12B | iPhone page | |
| |---|---|---|---| |
| | **A. Linux NVIDIA GPU box** (Spark / Omen) | `bash oneshot.sh` | started here (`:8080`, util **0.15**, FP8, never above **0.85**) | `http://<this-box-lan-ip>:7860/phone` | |
| | **B. Linux, no NVIDIA** | `bash oneshot.sh` then edit `.env` | skip local vLLM; point `.env` at pathway A | same, this machine’s LAN IP | |
| | **C. Windows PC** | `oneshot.bat` then `scripts\install-launcher.bat` | on pathway A (or WSL2) | `http://<this-pc-lan-ip>:7860/phone` | |
| | **D. macOS** | venv + `bash scripts/install-launcher.sh` | on pathway A | `http://<this-mac-lan-ip>:7860/phone` | |
| | **E. iPhone** | Safari only, after A/B/C/D UI is up | n/a (uploads to the UI box) | `/phone` — **Take photo** or **Choose file** | |
| | **F. Autonomous Lamp** | `skills/keys-receipt-scanner/` | on pathway A | n/a (HAL camera → Studio `/api/inbox`) | |
|
|
| ```bash |
| git clone https://github.com/drowzeys/keys-Auto-Receipts-Studio.git |
| cd keys-Auto-Receipts-Studio |
| ``` |
|
|
| Desktop icon after first UI start: Linux/macOS `bash scripts/install-launcher.sh` · Windows `scripts\install-launcher.bat`. Step-by-step for each letter is [below](#a-linux-nvidia-gpu-box-gemma--ui--iphone). |
|
|
| ## Screenshots (v1.0 alpha) |
|
|
| iPhone Safari — take a photo, then completion notice: |
|
|
| <p> |
| <img src="docs/screenshots/05-iphone-home.jpg" alt="iPhone: Take photo / Choose file" width="280"/> |
| <img src="docs/screenshots/06-iphone-upload.jpg" alt="iPhone: upload in progress" width="280"/> |
| <img src="docs/screenshots/07-iphone-complete.jpg" alt="iPhone: Completed Whole Foods Market groceries #5" width="280"/> |
| </p> |
|
|
| Desktop Review queue (click a row to edit kind / category / vendor / date / total, or Delete): |
|
|
|  |
|
|
| Inbox drop zone, catalog SKUs, and backend health: |
|
|
| <p> |
| <img src="docs/screenshots/01-inbox.png" alt="Inbox" width="420"/> |
| <img src="docs/screenshots/03-catalog.png" alt="Catalog" width="420"/> |
| <img src="docs/screenshots/04-settings.png" alt="Settings backends" width="420"/> |
| </p> |
|
|
| ## Promo (v1.0 alpha) |
|
|
| [~50s story cut](docs/promo/Keys-Auto-Receipts-Studio-promo.mp4) — messy dorm → iPhone (lay the receipt flat, then photograph) → charcoal Autonomous Lamp scans the last receipt → trash → clean room. Bed: Beethoven Symphony No. 5, I. Allegro con brio (public-domain recording). |
|
|
| Also attached on the [v1.0-alpha-promo release](https://github.com/drowzeys/keys-Auto-Receipts-Studio/releases/tag/v1.0-alpha-promo) (plays in the browser). |
|
|
| ## Pathway details |
|
|
| One-click / `app.launch` binds the UI on the LAN (`RECEIPT_UI_SHARE_LAN=true`) and opens Review in the browser. Phone URL is printed in the terminal. |
|
|
| Desktop icons (do **not** copy `scripts/start-ui.bat` itself to the Desktop): |
|
|
| | OS | Command | Opens | |
| |---|---|---| |
| | Linux | `bash scripts/install-launcher.sh` | Desktop **Receipt Studio.desktop** | |
| | macOS | `bash scripts/install-launcher.sh` | Desktop **Receipt Studio.command** (first time: right-click → **Open**) | |
| | Windows | `scripts\install-launcher.bat` | Desktop **Receipt Studio.bat** | |
|
|
| ### A. Linux NVIDIA GPU box (Gemma + UI + iPhone) |
|
|
| ```bash |
| bash oneshot.sh |
| ``` |
|
|
| Idempotent. Installs `.venv`, fetches **google/gemma-4-12B-it** to `~/models-gemma4-12b-it` if missing (`hf auth login` once), starts `scripts/serve-gemma.sh` if `:8080` is down, starts the UI on **0.0.0.0:7860**. |
|
|
| When it prints READY: |
|
|
| | | | |
| |---|---| |
| | Review | http://127.0.0.1:7860 | |
| | iPhone Safari (same Wi‑Fi) | http://<this-box-lan-ip>:7860/phone | |
| | Gemma OpenAI-compat | http://127.0.0.1:8080/v1 | |
|
|
| vLLM flags (also used by `scripts/start-ui.sh` on a CUDA box if Gemma is down): |
|
|
| - `--gpu-memory-utilization **0.15**` (~18.3 GiB of 121.7 GiB; **never above 0.85**) |
| - **FP8** (BF16 weights ~23 GB cannot fit in that pool) |
| - `--max-model-len **8192**` |
|
|
| If Gemma is already running, oneshot / launch leave it alone. To apply the 0.15 cap: stop that `vllm` process, then `bash oneshot.sh` again. |
|
|
| Gemma-only (UI already running): `bash scripts/serve-gemma.sh` |
|
|
| ### B. Linux without NVIDIA (UI + iPhone only) |
|
|
| ```bash |
| bash oneshot.sh |
| ``` |
|
|
| If `vllm` is not on PATH and nothing answers on `:8080`, oneshot **skips** weights and vLLM (it does **not** die) and still starts the UI. Then set `.env` at the GPU box from pathway A: |
|
|
| ``` |
| RECEIPT_LLM_BASE_URL=http://<spark-lan-ip>:8080/v1 |
| RECEIPT_EMBED_BASE_URL=http://<spark-lan-ip>:8080/v1 |
| RECEIPT_LLM_MODEL=google/gemma-4-12B-it |
| RECEIPT_EMBED_MODEL=google/gemma-4-12B-it |
| RECEIPT_EMBED_DIM=3840 |
| ``` |
|
|
| Restart the UI (`./scripts/start-ui.sh` or the desktop icon). |
|
|
| ### C. Windows PC (UI + iPhone hotspot) |
|
|
| 12B is **not** started by the Windows scripts. Serve Gemma on pathway A (or WSL2 + `bash oneshot.sh`). |
|
|
| 1. Install [Python 3.12](https://www.python.org/downloads/) — check **Add python.exe to PATH**. |
| 2. In the cloned repo, double-click **`oneshot.bat`** (creates `.venv`, opens Review). |
| 3. Edit `.env` as in pathway B (Spark LAN IP, port **8080**). |
| 4. Optional desktop icon: `scripts\install-launcher.bat` → **Desktop\Receipt Studio.bat**. |
| 5. iPhone: `http://<this-windows-lan-ip>:7860/phone` in **Safari**. |
|
|
| Do not copy `scripts\start-ui.bat` to the Desktop; it must run from the repo (or from the wrapper `install-launcher.bat` writes). |
|
|
| If this PC **is** NVIDIA + `vllm` on PATH (Git Bash / WSL): point `.env` at `127.0.0.1:8080` and `bash scripts/serve-gemma.sh`. |
|
|
| ### D. macOS (UI + iPhone) |
|
|
| Apple Silicon does not load 12B with this vLLM-NVIDIA recipe. Use pathway A for Gemma. |
|
|
| ```bash |
| python3 -m venv .venv |
| .venv/bin/pip install -e ".[dev]" |
| cp .env.example .env |
| # edit RECEIPT_LLM_BASE_URL / RECEIPT_EMBED_BASE_URL like pathway B |
| bash scripts/install-launcher.sh |
| ``` |
|
|
| Double-click **Desktop → Receipt Studio.command** (first time: right-click → **Open**). Same as `./scripts/start-ui.sh`. |
|
|
| ### E. iPhone (Safari) |
|
|
| Works against **whichever machine runs the UI** (A, B, C, or D), same Wi‑Fi. Chrome/Firefox on iOS are not the supported path. |
|
|
| 1. Open `http://<ui-machine-lan-ip>:7860/phone` |
| 2. **Take photo** (camera) or **Choose file** (Photos / Files, including HEIC) |
| 3. Wait for **Completed: vendor · total · category · #id** |
| 4. Confirm / edit / delete on the desktop **Review** tab |
|
|
| The phone page POSTs `/api/inbox` and polls `/api/jobs/{id}`. The GPU box (or the UI machine talking to Gemma) does extract + embed. Inbox drop and Syncthing into `inbox/` use the same pipeline (files idle 30s unless the phone API processes immediately). |
|
|
| ### F. Autonomous Lamp skill |
|
|
| Repo path: **`skills/keys-receipt-scanner/`** (`SKILL.md`, `skill.json` `capabilities: ["vision"]`, `scripts/scan.py`). |
|
|
| ``` |
| skills/keys-receipt-scanner/ |
| SKILL.md |
| skill.json |
| scripts/scan.py |
| references/hardware.md |
| ``` |
|
|
| Install (no reboot): |
|
|
| ```bash |
| make push-skill SKILL=./skills/keys-receipt-scanner TARGET=pi@lamp-xxxx.local |
| ``` |
|
|
| On the Lamp (camera only — **do not** load 12B): |
|
|
| ```bash |
| export RECEIPT_GPU_HOST=<spark-lan-ip> |
| export RECEIPT_STUDIO_URL=http://<spark-lan-ip>:7860 |
| export RECEIPT_CAMERA_URL=http://127.0.0.1:5001 |
| ``` |
|
|
| Say **“scan this receipt”**. `scripts/scan.py` snapshots HAL (`:5001/camera/snapshot`, 1280 px), then POSTs to Receipt Studio `/api/inbox` on the GPU box. Fallback: `RECEIPT_LLM_BASE_URL=http://<spark-lan-ip>:8080/v1` (direct vLLM) or `RECEIPT_HERMES_BASE_URL` (same OpenAI-compat URL Hermes registered). On the GPU box, `python scripts/register-hermes-gemma.py` adds `custom_providers` `gemma4-12b-it`. |
|
|
| Intern has no camera; this skill will not install there. |
|
|
| To ship in a fork of [autonomous-os](https://github.com/autonomous-ai/autonomous-os): copy `skills/keys-receipt-scanner/` into `skills/`, run `python skills/skill-creator/scripts/quick_validate.py`, `make skills-catalog`, open the PR. |
|
|
| --- |
|
|
| ## Does Gemma 4 12B Unified fit on the Lamp? |
|
|
| **No.** Lamp is 8-core ARM64 with **6 GB RAM**. Gemma 4 12B Unified is a 12B dense omni model (`Gemma4UnifiedForConditionalGeneration`, hidden size **3840**). |
|
|
| | Piece | Lamp (6 GB) | GPU box | |
| |---|---|---| |
| | `skills/keys-receipt-scanner/` | yes | yes | |
| | HAL snapshot | yes | n/a | |
| | SQLite + HTTP client | yes | yes | |
| | **Gemma 4 12B Unified weights** | **no** | yes (vLLM util **0.15**) | |
|
|
| ## Gemma 4 12B Unified (omni) on the GPU box |
|
|
| One OpenAI-compat server: vision `POST /v1/chat/completions` **and** `POST /v1/embeddings` (dim **3840**). Do not raise util above **0.85**. |
|
|
| ```bash |
| bash scripts/serve-gemma.sh |
| # util 0.15, FP8, max-model-len 8192 |
| ``` |
|
|
| Context at util 0.15 (after ~12.5 GB FP8 weights): |
|
|
| | Estimate | Tokens | |
| |---|---| |
| | Conservative (48 layers full attn) | **~12k** | |
| | Hybrid (8 full + 40× sliding-1024) | **~65k** | |
| | Model native window | 262,144 (not at 0.15) | |
|
|
| A receipt scan uses ~1–2k tokens. Raise `RECEIPT_VLLM_MAX_MODEL_LEN` only after the boot log shows a large enough `GPU KV cache size`. |
|
|
| If `/v1/embeddings` 404s, either serve a pooling convert on another port (`RECEIPT_EMBED_BASE_URL`) or `RECEIPT_EMBED_BACKEND=nvidia` + Nemotron-3-Embed-1B at dim **2048**. **Never mix 3840 and 2048 in one DB.** |
|
|
| Fallback vision (not omni embed): Qwen3.8-27B ADay777 at `:8078` (`RECEIPT_LLM_BACKEND=nvidia`, `RECEIPT_LLM_MODEL=qwen38-nvfp4`). Lightning is **text-only** — never send images. |
|
|
| ## Manual UI (any OS, after venv) |
|
|
| ```bash |
| python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate |
| pip install -e ".[dev]" |
| cp .env.example .env |
| # set RECEIPT_LLM_* like pathway B unless Gemma is local |
| RECEIPT_UI_SHARE_LAN=true python -m app.cli ui |
| ``` |
|
|
| CLI: |
|
|
| ```bash |
| python -m app.cli scan --image path/to/receipt.jpg |
| python -m app.cli query --category groceries |
| pytest |
| ``` |
|
|
| Syncthing: camera/share folder → `inbox/` (idle **30s** then process). |
|
|
| ## What you still run yourself |
|
|
| - Pathway A (or equivalent) so Gemma is reachable on the LAN |
| - iPhone on the **same Wi‑Fi**, Safari |
| - Lamp: pair in the Autonomous app, `make push-skill`, set `RECEIPT_STUDIO_URL` |
| - Optional: Syncthing |
|
|
| No model weights in this repo. No PyInstaller in this release. |
|
|
