keys-automatic-receipt-doc-scanner

Local receipt/document scanner for Autonomous OS + Autonomous Lamp. Lamp camera captures; a GPU box runs the model. Nothing cloud.

Hardware split (do not blur this)

Where	Fits	Does not fit
Lamp (ARM64, 6 GB RAM)	This skill (`SKILL.md` + `scripts/`), HAL snapshot, SQLite	Gemma 4 12B Unified, Qwen3.8-27B, Nemotron 3.5 Lightning
GPU box (DGX Spark / Omen)	Gemma 4 12B Unified (omni: vision + embed) or Qwen3.8-27B VLM + Embed-1B	—

Gemma 4 12B Unified is ~12B dense (hidden_size 3840, Gemma4UnifiedForConditionalGeneration). Weights alone exceed the Lamp's 6 GB even in NVFP4. Treat the Lamp as eyes/hands; treat the GPU box as the brain.

Default brain: Gemma 4 12B Unified (omni)

Encoder-free VLM. One OpenAI-compatible server does:

Vision extract — POST /v1/chat/completions with image_url data URI (receipt/doc JPEG)
Embed — POST /v1/embeddings against the same server (mean-pool / convert-embed). Dim 3840.

Never mix 3840 (Gemma) and 2048 (Nemotron-3-Embed-1B) in one sqlite-vec index.

Fallback brains (same skill, env only)

Qwen3.8-27B ADay777 VLM at :8078 (qwen38-nvfp4) for extract; Nemotron-3-Embed-1B 2048-d for embed
Lightning is text-only. Never send images to it.

Fleet GB10: --gpu-memory-utilization 0.85 hard cap.

Pipeline

voice / phone / drop → Lamp camera or inbox/
        → optional OCR assist
        → Gemma4 (or Qwen) vision JSON extract + category
        → omni embed (or Nemotron-3-Embed-1B)
        → sqlite-vec vendor/SKU/category match
        → review / speak summary

Rules

No cloud APIs. Backends behind OCRBackend / LLMBackend / EmbedBackend.
Lightning: accepts_images=False. Never attach image parts.
Python 3.12, typed, pytest. No notebooks. Don't vendor weights.
Idle-batch inbox 30s. Dedup sha256.

Extract JSON

{
  "doc_kind": "receipt|invoice|document",
  "category": "groceries|dining|transport|household|health|entertainment|utilities|office|travel|other",
  "vendor": "string|null",
  "date": "YYYY-MM-DD|null",
  "tax": "number|null",
  "total": "number|null",
  "currency": "string|null",
  "line_items": [
    {"description": "string", "qty": "number|null", "unit_price": "number|null",
     "amount": "number|null", "sku": "string|null"}
  ]
}

Money stored as integer cents.

Match (cosine similarity = 1 - sqlite-vec distance)

	Auto	Review	Unmatched
SKU / line	≥ 0.88	0.72–0.88	< 0.72
Vendor	≥ 0.82	0.65–0.82	< 0.65

Exact catalog SKU wins first.

Layout

app/          config, schemas, media, camera, extract, embed, db, match,
              pipeline, watcher, cli, ui
backends/     base, openai_compat, gemma, nvidia, ollama, apple, cpu
skills/keys-receipt-scanner/   Autonomous OS built-in skill (Lamp)
inbox/ processing/ processed/ failed/ exports/

Autonomous OS skill

skills/keys-receipt-scanner/ is a built-in skill in Autonomous OS format:

SKILL.md + skill.json (capabilities: ["vision"])
Installs on any body that declares vision (Lamp, Reachy Mini — not Intern)
Acts via HAL GET :5001/camera/snapshot then python -m app.cli scan --image PATH
Does not load 12B weights on the robot