| --- |
| license: apache-2.0 |
| base_model: google/gemma-4-E2B-it |
| tags: |
| - gemma |
| - gemma-4 |
| - lora |
| - on-device |
| - scam-detection |
| - sms-classification |
| - call-classification |
| - litert |
| - litert-lm |
| - gguf |
| - llama-cpp |
| language: |
| - en |
| pipeline_tag: text-classification |
| library_name: transformers |
| extra_gated_prompt: |- |
| Base model: Gemma 4 E2B. By accessing this repository you agree to the |
| Gemma Terms of Use: https://ai.google.dev/gemma/terms |
| --- |
| |
| # Bastion — Scam-Call & SMS Classifier (Gemma 4 E2B + LoRA) |
|
|
| > Fine-tuned **Gemma 4 E2B** for on-device scam-call and SMS classification. |
| > Shipped inside [Bastion](https://gitlab.com/3sk4p3/bastion), the on-device |
| > phone-scam shield for seniors built for the |
| > [Gemma 4 Good Hackathon](https://www.kaggle.com/competitions/gemma-4-good-hackathon) |
| > (Impact Track · Safety & Trust, and LiteRT Special Technology Track). |
|
|
| ## TL;DR |
|
|
| A LoRA adapter that takes Gemma 4 E2B from **F1 0.305** to **F1 0.915** on a |
| 100-sample stratified BothBosu test set (3-class scam / scam_partial / ham), |
| and ships in three formats for two on-device runtimes — `llama.cpp` |
| (`Q4_K_M` GGUF) and Google AI Edge **LiteRT-LM** (`.litertlm`, QAT-v2). |
| |
| | Artefact | Format | Size | Runtime | Purpose | |
| | -------- | ------ | ---- | ------- | ------- | |
| | `bastion-text-lora-v1.Q4_K_M.gguf` | GGUF (Gemma 4 E2B + LoRA, merged, Q4_K_M) | ~3.2 GB | `llama.cpp` | Reference deployment artefact, used by the shipped Samsung A54 demo | |
| | `bastion-qat-v2-gemma-4-E2B-it.litertlm` | LiteRT-LM v2 (QAT) | ~2.4 GB | `litertlm-android` 0.10.2 / LiteRT-LM v0.11 | Google AI Edge runtime artefact (LiteRT Special Technology Track) | |
| | `bastion-mmproj.BF16.gguf` | GGUF mm-projector | ~215 MB | `llama.cpp` (multimodal) | Optional: pairs with stock Gemma 4 E2B for the multimodal-direct experiment baseline (D11/D13) | |
| |
| ## Why this exists |
| |
| Live phone scams target older adults and cost USD ~3B/year in the US alone. |
| Cloud assistants cannot intervene fast enough — by the time a transcript |
| uploads, the senior has read out the code. Bastion runs the model on the |
| phone that is ringing. This adapter is the part that decides whether the |
| call gets ended. |
| |
| The base model is good at *spotting* scam patterns (binary F1 ≈ 0.99 on |
| BothBosu/100) but ships its verdicts as the cautious `scam_partial` label, |
| which never triggers Bastion's interrupt — so it is, in practice, useless |
| for an intervention system. The LoRA fine-tune fixes exactly that gap. |
|
|
| ## Model details |
|
|
| - **Base model:** [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) |
| - **Adapter:** LoRA, rank 16, language layers only, dropout 0.05 |
| - **Training framework:** [Unsloth](https://github.com/unslothai/unsloth) |
| - **Training data:** synthetic (transcript, label) pairs produced by a |
| Gemini 3.1 Flash Lite Preview pipeline we built specifically for this |
| project (`scripts/synth_scale_gemini.py` + `scripts/chunk_and_filter.py` |
| in the Bastion repo). Source scripts come from |
| [BothBosu/scam-dialogue](https://huggingface.co/datasets/BothBosu/scam-dialogue) |
| (Apache-2.0) plus Gemini-generated ham counterparts. Both sides are |
| spoken aloud via **Gemini multi-speaker TTS**, then degraded from |
| studio quality to phone-call audio (8 kHz, codec artefacts) so the |
| audio distribution matches what the host application sees on the |
| wire. Each WAV is chunked into 15-second windows, transcribed by |
| Gemma 4 E2B, and only the chunks whose transcript a Gemini |
| label-judge still agrees with the source dialog label are kept. The |
| result is a clean, on-distribution text training set without manual |
| transcription cost. We additionally hold out |
| [UCI SMS Spam](https://archive.ics.uci.edu/dataset/228/sms+spam+collection) |
| (CC-BY 4.0) and the SMS phishing subset of |
| [DIFrauD](https://huggingface.co/datasets/difraud/difraud) (MIT) as |
| evaluation references; they are not part of the v1 training mix. |
| - **Output schema (JSON, tool-call style):** |
| ```json |
| { |
| "verdict": "ham | scam_partial | scam_clear", |
| "reason": "<≤ 25 words>", |
| "intervene": true | false |
| } |
| ``` |
| - **Intervention contract:** `verdict == "scam_clear" && intervene == true` |
| triggers `TelecomManager.endCall()` on Android. Anything else is a banner |
| warning at most. |
|
|
| ## Intended use |
|
|
| - **On-device scam-call screening** in conjunction with an OEM call recorder |
| + an ASR front-end (Bastion uses Sherpa-ONNX Whisper Tiny). |
| - **On-device SMS scam classification** (text mode, same model + adapter). |
| - **Reference target** for hackathon submissions and research on small-model |
| scam-detection benchmarks. |
|
|
| This adapter is **not** a general assistant — it is single-purpose. Outside |
| the scam / not-scam decision, behaviour falls back to base Gemma 4 E2B. |
|
|
| ## Results |
|
|
| Evaluation set: **[BothBosu](https://huggingface.co/datasets/BothBosu/scam-dialogue) test, 100 samples, stratified 50 scam / 50 ham**, 3-class |
| schema. Full eval log in the [Bastion repo](https://gitlab.com/3sk4p3/bastion/-/blob/main/ml/evals/RESULTS.md). |
|
|
| | Setting | F1 (binary) | F1 (3-class) | Precision | Recall | `scam_clear` recall (k/50) | Parse rate | |
| | -------- | ----------- | ------------ | --------- | ------ | -------------------------- | ---------- | |
| | Gemma 4 E2B base (prompt only) | 0.667 | 0.305 | 1.000 | 0.180 | 9 / 50 | 100 / 100 | |
| | **Gemma 4 E2B + `bastion_text_lora_v1` (Q4_K_M, merged)** | **0.985** | **0.915** | **0.977** | **0.860** | **43 / 50** | **100 / 100** | |
| |
| **+61 absolute-point lift on the 3-class metric that controls intervention.** |
| The one false positive at this threshold was a real bank-verification call |
| labelled `scam_partial`, which would *not* cross Bastion's interrupt gate |
| (`scam_clear` only, confidence ≥ 0.8). |
| |
| Latency on the CPU reference build (`Q4_K_M` GGUF, llama.cpp, x86 CPU): |
| p50 5.8 s, p95 8.8 s per 15-second window. |
| |
| ## Limitations |
| |
| - **Trained on English, but the scam-classification layer generalises.** The |
| LoRA fine-tune is on English transcripts, yet Gemma 4 E2B's underlying |
| multilingual coverage carries over: on hand-tested Polish and Spanish |
| paraphrases of the BothBosu archetypes the adapter still emits the |
| correct verdict + JSON. **The end-to-end bottleneck is the ASR front-end, |
| not this model** — Sherpa-ONNX Whisper Tiny transcribes English well and |
| other languages noticeably less well, so the practical multilingual claim |
| is gated by which Whisper build the host application ships. A formal |
| Polish eval split is in progress and not part of this release. |
| - **Quantisation sensitivity.** Q2_K and IQ2_M variants destroy the LoRA |
| signal (F1 drops to 0.00–0.73). Ship **Q4_K_M or higher**, or the |
| LiteRT-LM QAT-v2 build. |
| - **Single-turn classification.** The adapter classifies one rolling window |
| at a time. Multi-turn debounce is the host application's responsibility |
| (Bastion does this in its `InterventionController`). |
| - **Adversarial robustness untested.** Eval is on naturalistic BothBosu |
| data, not adversarial paraphrases of scam scripts. |
| |
| ## How to use |
| |
| ### Via `llama.cpp` (Q4_K_M, merged) |
| |
| ```bash |
| huggingface-cli download 3sk4p3/bastion bastion-text-lora-v1.Q4_K_M.gguf --local-dir ./bastion |
| ./llama-cli -m ./bastion/bastion-text-lora-v1.Q4_K_M.gguf \ |
| --temp 0.0 --json-schema-file ml/prompts/tool_schema.json \ |
| -p "$(cat ml/prompts/system_prompt.md)\n<transcript>$TRANSCRIPT</transcript>" |
| ``` |
| |
| ### Via LiteRT-LM (`.litertlm`, on Android) |
| |
| ```kotlin |
| val runner = RealLiteRtGemmaRunner( |
| context = appContext, |
| modelPath = "/sdcard/Android/data/<pkg>/files/bastion-qat-v2-gemma-4-E2B-it.litertlm", |
| ) |
| runner.warmUp() |
| val verdict = runner.classifyText(transcript) // JSON parsed into the schema above |
| ``` |
| |
| The Kotlin runner is in |
| [`android/inference/`](https://gitlab.com/3sk4p3/bastion/-/tree/main/android/app/src/main/kotlin/com/bastion/app/inference) |
| in the Bastion repo and ships with the APK. |
| |
| ## Training reproducibility |
| |
| - Unsloth notebook: [`ml/notebooks/train_text_lora_v1.ipynb`](https://gitlab.com/3sk4p3/bastion/-/blob/main/ml/notebooks/train_text_lora_v1.ipynb) |
| - Prompts and JSON tool schema: [`ml/prompts/`](https://gitlab.com/3sk4p3/bastion/-/tree/main/ml/prompts) |
| - Eval scripts: [`scripts/eval_bothbosu_100.py`](https://gitlab.com/3sk4p3/bastion/-/blob/main/scripts/eval_bothbosu_100.py), [`scripts/eval_litertlm.py`](https://gitlab.com/3sk4p3/bastion/-/blob/main/scripts/eval_litertlm.py) |
| - Eval log (append-only): [`ml/evals/RESULTS.md`](https://gitlab.com/3sk4p3/bastion/-/blob/main/ml/evals/RESULTS.md) |
| |
| ## License & attribution |
| |
| - **This repository (LoRA artefacts + model card):** Apache-2.0, plus the |
| Gemma Terms of Use for any artefact derived from Gemma 4 weights. |
| - **Base model:** [Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B-it) |
| by Google DeepMind, used under the |
| [Gemma Terms of Use](https://ai.google.dev/gemma/terms). |
| - **Training data:** see each dataset's own licence (Apache-2.0, MIT, |
| CC-BY 4.0 as listed above). |
| - **Naming:** `bastion_text_lora_v1` follows the |
| [Gemma variant naming guidelines](https://ai.google.dev/gemma/docs/core/model_card_4) |
| — variant name precedes Gemma identifier, no stand-alone "Gemma" branding. |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{bastion2026, |
| title = {Bastion: On-Device Scam-Call Shield for Seniors}, |
| author = {Szczepanik, Kamil and Arkik, Mohamed}, |
| year = {2026}, |
| howpublished = {\url{https://gitlab.com/3sk4p3/bastion}}, |
| note = {Gemma 4 Good Hackathon submission, Impact Track / Safety \& Trust} |
| } |
| ``` |
| |
