Veil OTP: the sign-in code model
The model behind Settings → Local AI → Hide sign-in codes. It decides
whether a number in an email is the message's sign-in, verification or
one-time code, so Picacho can show it as •••••• in the subject, the message
and notifications.
Nothing here ships inside the app. When the feature is turned on, Picacho
downloads one file from this repository, checks its size and SHA-256 against
the values built into the app (src/manage/local_ai/mod.rs in
Picacho), and keeps it in its data
directory under models/. Turning the feature (or Local AI) off deletes it.
Training code and data live in Picacho's models/otp/ directory.
| File | Used on | Runtime |
|---|---|---|
otp-v1.mlmodel |
macOS | Core ML (compiled for the Mac when installed) |
otp-v1.bin |
Windows, Linux | Picacho's own (src/manage/local_ai/runtime.rs) |
Both hold the same network.
Languages
The model is trained on sign-in mail in English, Spanish, German, French, Portuguese, Italian, Dutch, Swedish, Polish, Japanese, Chinese (simplified and traditional), Korean, Russian, Ukrainian, Turkish, Arabic, Persian, Hebrew, Hindi, Bengali, Indonesian, Malay, Vietnamese, Thai, Greek, Czech, Slovak, Danish, Norwegian, Finnish, Romanian, Hungarian, Catalan, Croatian, Bulgarian, Tagalog and Swahili, with codes in ASCII, fullwidth, Arabic-Indic, Persian, Devanagari, Bengali and Thai digits. Copy Code gives ASCII digits.
How it works
- Candidates. A pattern finds numbers that could be codes: 2–4
hyphenated groups of 3–6 uppercase letters and digits with a letter among
them (
BAJ6K-654XM), two groups of 3–4 digits (482 913,482-913), 4–10 uppercase letters and digits (AB12CD,482913), or 4–10 digits in any script (482913,٤٨٢٩١٣), optionally with a short prefix (G-482913). A candidate may touch Chinese or Japanese text but not a Latin letter or another digit; parts of phone numbers, dates and amounts are skipped. - Words. Runs of letters, marks and digits. Scripts written without
spaces (Chinese, Japanese kana, Thai, Lao, Khmer, Myanmar) are split into
overlapping pairs of characters (
验证码是→验证,证码,码是). - Features. For each candidate: the words before and after it and their
first four letters, the nearest words and word pairs, the characters next
to it, its shape (
999999,A-999999), whether it is on a line of its own or in the subject, the subject's words, and the concepts the nearby words and the subject name. Concepts (lexicon.json: code, verify, sign in, one-time, expiry, don't share, order, money, shipping, contact, promotion, reference) list word stems in every language above, so "код", "コード" and "kód" all count as code (=marks a stem that must be the whole word,^one the word must start with). Concepts are also taken from the whole 200 characters on each side of the candidate and from the subject, with Chinese, Japanese and Thai runs kept whole rather than split into pairs, because a code often sits far from the words naming it (293393は、…ワンタイムコードです). Everything is hashed into 16,384 slots. - Model. A bag-of-features network (fastText style): one hidden layer of 16 units and a logistic output, about 260,000 weights (1 MB). A candidate scoring 0.3 or more is a code.
features.py and lexicon.json are the reference; the app builds the same
features in src/manage/local_ai/otp.rs, with the lexicon compiled in, so a
lexicon change needs an app update as well as a new model. parity.json
(candidates, features, vectors and scores for messages in several languages
and scripts) is checked by cargo test matches_the_training_code.
Training
python3 train.py # numpy: otp-v1.bin, parity.json
python3 train.py --coreml # also otp-v1.mlmodel (pip install coremltools)
The training mail is synthetic (data.py): sign-in messages in the languages
above, mail full of numbers that aren't codes (orders, invoices, tracking and
booking references, phone numbers, addresses, account endings, coupons,
license keys, meeting IDs) in many of them, and sign-in messages with another number beside
the code. Training is deterministic.
On messages built from templates held out of training:
| Held-out wording | Precision | Recall | Messages exactly right |
|---|---|---|---|
| English (and other Latin-script) templates | 95% | 100% | 97% |
| Templates in the newer languages | 97% | 96% | 96% |
Those are the honest figures for wording the model hasn't seen; on familiar wording it is right nearly every time. The lexicon was written knowing every template, so for an unseen language the second row is optimistic. Real mail will have phrasings the templates don't cover; more varied examples improve it.
Publishing a new version
- Retrain, then update the sizes and digests
train.pyprints insrc/manage/local_ai/mod.rs(cargo test the_published_models_are_the_trained_onesfails until they match). Bump the version in the file names when the features or lexicon change. - Commit and push the model files, then point
MODELS_URLat that commit:https://raw.githubusercontent.com/juliaoverflow/Picacho/<commit>/models/otp. A commit, not a branch, so the file at the address never changes under the pinned digest.
PICACHO_MODELS_URL makes the app download from somewhere else (an HTTPS
mirror) while developing.
- Downloads last month
- 39