# Building Mystery-Mail Guardian: A Project for build-small-hackathon

## Introduction

My grandmother can read English. What she can't always do is tell a *real* letter
from a scam — the official-looking envelope demanding payment "within 24 hours,"
the one threatening arrest, the one that's secretly an advertisement dressed up
as a bill. For a lot of elderly people, the mailbox is one of the last places
they still get attacked, and the attacks are getting better.

The obvious move is to point a multimodal model at the letter and ask "is this a
scam?" But the letters that matter most — from the bank, the tax office, the
doctor — are exactly the documents you should *never* upload to a cloud API. So
the interesting constraint wasn't "can a model read a letter." It was: **can a
model read a scary letter, helpfully and safely, without a single byte ever
leaving the device?**

That constraint became the whole project.

## The Idea

**Mystery-Mail Guardian** lets you photograph a confusing letter and get back, in
plain words: what it is, whether you should worry (with concrete reasons, never a
verdict), what to actually do next — and it reads the answer aloud. In English,
Hindi, Spanish, and Japanese.

Two design rules shaped everything:

1. **Run entirely on-device.** No cloud inference at runtime. The privacy promise
   isn't a feature bolted on; it's the reason the product gets to exist.
2. **Never let the model have the final word on safety.** A small vision model
   *will* occasionally say something confidently wrong. So the model proposes,
   and a separate, deterministic safety layer disposes.

## High-Level Plan: small models, watched closely

Everything runs on two OpenBMB models, **3.3B parameters total** — comfortably
under the hackathon's cap, and small enough to run on a single ZeroGPU slice:

- **MiniCPM-V 4.6** (1.3B) is the central engine: it reads the photographed
  letter (OCR + layout + reasoning) and returns structured JSON.
- **VoxCPM2** (2B) reads the result aloud, in the user's language, with a neutral
  built-in voice (deliberately no voice cloning — no consent question to answer).

There was a tempting third option: bolt an 8B reasoner on top for nicer prose.
I built the switch for it, measured it, and left it off — more on that below. The
hackathon's "Tiny Titan" award is for models ≤4B, and the lean pair already does
the job. Smaller was the right call, not the compromise.

## Internal Logic: the model proposes, the safety layer disposes

The pipeline is four stages — **extract → triage → safety → speak** — and the
single most important architectural decision is that the safety rules live in
*code*, in their own module, not in the prompt. A prompt can be ignored by a
0.8B language head having a bad day. A regex cannot.

### Stage 1 — Extract

MiniCPM-V 4.6 reads the photo against a strict JSON schema. The prompt bakes in
the safety rules too (defense in depth), but its real job is to pull *structured
facts* — sender, amount, deadline, and any scam signals it spots from a fixed
taxonomy:

```python
"scam_signals": [
    {"id": "<signal id>", "evidence": "short quote or fact from the document"}
],
"explanation": {
    "what_this_is": "1-2 short sentences in {language}, words a 10-year-old understands",
    "key_facts": ["up to 4 very short facts with real values from the letter"],
    "worry_reasons": ["each scam signal explained in one short sentence"],
    "what_to_do": ["2 to 4 short, safe, concrete steps"]
}
```

### Stage 2 — Triage (a second pair of eyes)

Here's the part I trust most, because it doesn't trust the model. An *independent*
heuristic scan runs over everything the model extracted, looking for scam
vocabulary the model might have downplayed — gift cards, wire transfers, OTP
requests, arrest threats, urgency — in every supported script:

```python
"gift_card_or_crypto": re.compile(
    r"gift\s*card|itunes\s*card|bitcoin|crypto|गिफ्ट\s*कार्ड|"
    r"ギフトカード|tarjeta\s+de\s+regalo", re.I),
"threat_or_arrest": re.compile(
    r"\barrest(?:ed)?\b|lawsuit|गिरफ़्तार|逮捕|arresto", re.I),
```

The model's signals and the heuristic signals are unioned, and the worry level
is computed from the result — never from the model's mood:

```python
def worry_level(document_type, signals):
    if document_type == "suspected_scam" or len(signals) >= 2:
        return "warning"
    if len(signals) == 1:
        return "caution"
    return "low"   # note: there is no "safe" — only "looks normal, but verify"
```

There is deliberately **no "safe" state**. The best case is "this looks like
normal mail, but always double-check before paying or replying."

### Stage 3 — The safety layer

Every piece of user-facing text passes through here. Two guarantees matter most.

**It softens verdicts.** If the model writes "this is definitely a scam," it
becomes "this looks like it could be a scam." We never tell a 78-year-old that a
real letter is definitely safe, or that a real bill is definitely fraud.

**It strips the letter's own contact details — from every card.** A scam letter's
entire goal is to get you to call *its* number. So phone numbers, links, and
emails printed in the letter are removed from the answer, and replaced with one
constant instruction: contact the company through a number *you already trust*.

```python
_PHONE_RE = re.compile(r"\+?\d[\d\-\s().]{7,}\d")
_EMAIL_RE = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.]+\b")

def strip_contact_details(text: str) -> str:
    text = _URL_RE.sub("", text)
    text = _EMAIL_RE.sub("", text)
    text = _PHONE_RE.sub("", text)
    return re.sub(r"[ \t]{2,}", " ", text).strip(" \t,;:")
```

This earned its keep in live testing. Against a real gift-card scam, the model
suggested *"check with the tax bureau and ensure you pay soon"* — genuinely
dangerous advice. Because it arrived in the wrong shape, the composer discarded
it and showed the safe fallback plus the verification advice instead. The
scammer's 1-800 number appeared nowhere on the screen.

### Stage 4 — Speak

VoxCPM2 turns the safety-checked summary into speech on demand — one big "Read it
aloud" button that only appears once there's something to read.

## The QA flywheel: Modal as the proving ground

Here's a thing that surprised me: the most valuable use of GPU credits wasn't
inference, it was *testing*. The deployed Space runs on a daily ZeroGPU quota I
didn't want to burn on my own QA — so every bit of GPU validation runs on
**Modal** instead, and the deployed runtime stays 100% local. Off-the-grid stays
intact; QA gets unlimited room.

I built a little factory that forges synthetic letters with *gold labels derived
through the app's own triage rules* (so the labels can never drift from the
product), renders them as photos, degrades them like a shaky phone hand, and
scores the full pipeline on an A10G. It paid for itself immediately:

- **Run #1 of the validation matrix** caught a real bug — a lottery scammer's
  reply address leaking through the "key facts" card. (The contact-stripping
  only covered action steps; now it covers every card.)
- **A multilingual eval** found Hindi and Japanese scam letters being
  under-flagged, which led to original-script quoting in the prompt and
  multilingual scam vocabulary in the heuristics.
- **A robustness sweep** — six degradation types × three intensities (blur to
  4.5px, brightness to 15%, rotation to 35°, perspective warp, hard shadow,
  sensor noise) — came back **38/38 honest outcomes**. The scam stayed *warning*
  and the bill stayed *low* through all of it.

And the answer to "should we fine-tune?" came from the data, not from wanting to.
I trained a LoRA adapter anyway, to prove the loop works — train loss 2.14 → 0.03
in under six minutes for about forty cents — but the evals showed safety behavior
was already at ceiling on the stock model, so the adapter stays shelved through
judging. With small models, you earn trust through guardrails and measurement,
not through more training.

## A note on the front end

The UI is hand-built past the default Gradio look into a warm "kitchen-table post
office": letters render as paper sheets, the verdict is a perforated postage
stamp, the privacy promise is a green wax seal. The body font is **Atkinson
Hyperlegible**, designed by the Braille Institute specifically for low-vision
readers — an elder-readability font in an elder-readability app — vendored into
the repo so even the fonts make no network request. Every color pair is enforced
against WCAG AA contrast by unit tests; if a pretty color fails, the color
changes, not the test. (There may also be a Gen X Soft Club easter egg hiding
behind one of the language buttons. I'll let you find it.)

## Wrapping up

What I like most isn't any single trick — it's the shape of the thing. A model
small enough to run in your hand, wrapped in code that refuses to let it be
confidently wrong about something that could cost someone their savings. The
intelligence proposes; the rules protect.

The honest catch is **script coverage**: a 1.3B vision model reads English (and
Japanese) print reliably, but Devanagari is near its limit, so Hindi letters can
be under-read — which is exactly why every single result, in every language,
still ends with "check with someone you trust." The guardrails hold even where
the model doesn't.

It's live, free, and fully open. Built for the Hugging Face **Build Small**
hackathon. If it speaks to you — or to someone you'd want to protect — give it a
try, and a ❤️ on the Space genuinely helps. 🤗

👉 **https://huggingface.co/spaces/build-small-hackathon/Mystery_Mail_Guardian**