File size: 3,372 Bytes
2edb151 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | # keys-automatic-receipt-doc-scanner
Local receipt/document scanner for **Autonomous OS + Autonomous Lamp**.
Lamp camera captures; a GPU box runs the model. Nothing cloud.
## Hardware split (do not blur this)
| Where | Fits | Does not fit |
|---|---|---|
| **Lamp** (ARM64, **6 GB RAM**) | This skill (`SKILL.md` + `scripts/`), HAL snapshot, SQLite | Gemma 4 12B Unified, Qwen3.8-27B, Nemotron 3.5 Lightning |
| **GPU box** (DGX Spark / Omen) | Gemma 4 12B Unified (omni: vision + embed) or Qwen3.8-27B VLM + Embed-1B | — |
Gemma 4 12B Unified is ~12B dense (`hidden_size` **3840**, `Gemma4UnifiedForConditionalGeneration`). Weights alone exceed the Lamp's 6 GB even in NVFP4. Treat the Lamp as eyes/hands; treat the GPU box as the brain.
## Default brain: Gemma 4 12B Unified (omni)
Encoder-free VLM. One OpenAI-compatible server does:
1. **Vision extract** — `POST /v1/chat/completions` with `image_url` data URI (receipt/doc JPEG)
2. **Embed** — `POST /v1/embeddings` against the **same** server (mean-pool / convert-embed). Dim **3840**.
Never mix 3840 (Gemma) and 2048 (Nemotron-3-Embed-1B) in one sqlite-vec index.
## Fallback brains (same skill, env only)
- **Qwen3.8-27B ADay777** VLM at `:8078` (`qwen38-nvfp4`) for extract; Nemotron-3-Embed-1B 2048-d for embed
- **Lightning** is text-only. Never send images to it.
Fleet GB10: `--gpu-memory-utilization` **0.85** hard cap.
## Pipeline
```
voice / phone / drop → Lamp camera or inbox/
→ optional OCR assist
→ Gemma4 (or Qwen) vision JSON extract + category
→ omni embed (or Nemotron-3-Embed-1B)
→ sqlite-vec vendor/SKU/category match
→ review / speak summary
```
## Rules
- No cloud APIs. Backends behind `OCRBackend` / `LLMBackend` / `EmbedBackend`.
- Lightning: `accepts_images=False`. Never attach image parts.
- Python 3.12, typed, pytest. No notebooks. Don't vendor weights.
- Idle-batch inbox 30s. Dedup sha256.
## Extract JSON
```json
{
"doc_kind": "receipt|invoice|document",
"category": "groceries|dining|transport|household|health|entertainment|utilities|office|travel|other",
"vendor": "string|null",
"date": "YYYY-MM-DD|null",
"tax": "number|null",
"total": "number|null",
"currency": "string|null",
"line_items": [
{"description": "string", "qty": "number|null", "unit_price": "number|null",
"amount": "number|null", "sku": "string|null"}
]
}
```
Money stored as integer cents.
## Match (cosine similarity = 1 - sqlite-vec distance)
| | Auto | Review | Unmatched |
|---|---|---|---|
| SKU / line | ≥ 0.88 | 0.72–0.88 | < 0.72 |
| Vendor | ≥ 0.82 | 0.65–0.82 | < 0.65 |
Exact catalog SKU wins first.
## Layout
```
app/ config, schemas, media, camera, extract, embed, db, match,
pipeline, watcher, cli, ui
backends/ base, openai_compat, gemma, nvidia, ollama, apple, cpu
skills/keys-receipt-scanner/ Autonomous OS built-in skill (Lamp)
inbox/ processing/ processed/ failed/ exports/
```
## Autonomous OS skill
`skills/keys-receipt-scanner/` is a **built-in skill** in Autonomous OS format:
- `SKILL.md` + `skill.json` (`capabilities: ["vision"]`)
- Installs on any body that declares vision (Lamp, Reachy Mini — not Intern)
- Acts via HAL `GET :5001/camera/snapshot` then `python -m app.cli scan --image PATH`
- Does not load 12B weights on the robot
|