App_Creation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# App Creation — PDF Injection Detector (MiMo-7B)\n",
    "\n",
    "The build log for the Hugging Face Space at\n",
    "[`BentoUniAcc/Mimo_Injection_detector`](https://huggingface.co/spaces/BentoUniAcc/Mimo_Injection_detector):\n",
    "what it does, which decisions were forced rather than chosen, what broke, and the checks that say\n",
    "it is still faithful to the notebooks it quotes.\n",
    "\n",
    "The Space is four Python files and this notebook is not one of them — it does not define the app,\n",
    "it records how the app came to be what it is. Every cell below runs against the *deployed* files,\n",
    "so if one fails, the Space is wrong and not the notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 — What it is\n",
    "\n",
    "Upload a PDF. It is rendered to text with the extractor that built the project corpus, the regions\n",
    "carrying structural markers are ranked, the whole document is cut into **batches that fit one run\n",
    "of the model**, and **MiMo-7B-RL** reads the batch you choose — reporting whether a payload is\n",
    "hidden there and quoting the substring that convinced it.\n",
    "\n",
    "| File | Role |\n",
    "|---|---|\n",
    "| `app.py` | Gradio UI, batching, report aggregation. No detection logic. |\n",
    "| `corpus_text.py` | PDF bytes to skeleton to candidate regions. |\n",
    "| `mimo.py` | Prompt, prefill, parser, and both model runtimes. |\n",
    "| `neighbours.py` | Part A embedding index and nearest-neighbour lookup. |\n",
    "| `test_fidelity.py` | Proves the extraction reproduces the published corpus exactly. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 — The provenance rule\n",
    "\n",
    "The published numbers only describe this app if the app reads a PDF and asks the question exactly\n",
    "as the notebooks did. So the code was lifted from the **dataset repos themselves**, not rewritten\n",
    "and not copied from the earlier archived app:\n",
    "\n",
    "| Borrowed | From |\n",
    "|---|---|\n",
    "| `build_skeleton`, `mask_leaks`, `detect_markers`, `payload_window`, `ANY_MARKER_RE`, `INJECTION_MARKERS`, `LEAK_STRINGS` | EDA notebook, cells 88 / 89 / 107 |\n",
    "| `SYSTEM`, `PREFILL`, `build_messages`, `scan_objects`, `parse_response`, `MAX_NEW`, `BATCH` | Part B notebook, cell 48 |\n",
    "| embedder repo, prefix, dims, normalisation, input column | `Part_A_Outputs/part_a_results.json` |\n",
    "\n",
    "One decision worth stating: **the parser is the narrow original**, not the widened version that\n",
    "appears in the archived app. A wider salvage recovers more verdicts and would also mean the F1\n",
    "quoted in the interface describes a parser that is not the one running.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 — The prompt and the prefill, as deployed\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mimo\n",
    "\n",
    "print(f'system message : {len(mimo.SYSTEM)} chars')\n",
    "print(f'families       : {len(mimo.FAMILIES)} (closed set, fixed order)')\n",
    "print(f'max new tokens : {mimo.MAX_NEW}')\n",
    "print(f'batch          : {mimo.BATCH}')\n",
    "print()\n",
    "print('prefill:', repr(mimo.PREFILL))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MiMo is the only model in Part B that carries a prefill. It is reasoning-trained and opens every\n",
    "answer with `<think>`; at a 200-token budget it never closed the block, so not one of its 1,100\n",
    "answers reached the JSON. An empty, already-closed think-block says the deliberation is finished\n",
    "before it begins, and the opening brace puts the model inside the answer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 — The parser survives the bug that mattered\n",
    "\n",
    "Part B's first parser used a regex to find the JSON object. Every injected file in this corpus\n",
    "carries an EICAR-style marker containing a closing brace, so the moment a model quoted its\n",
    "evidence the match truncated and the verdict was thrown away — **the bug fired exactly when the\n",
    "model was right**. The replacement counts braces and tracks string literals.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "evidence = r'X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE'\n",
    "body = json.dumps({'injected': True, 'injection_type': 'javascript_injection',\n",
    "                   'evidence': evidence, 'reasoning': 'marker present'})\n",
    "\n",
    "# what MiMo actually emits: the prefill, then the remainder of that object\n",
    "raw = mimo.PREFILL + body[body.index(':') + 1:]\n",
    "r = mimo.parse_response(raw)\n",
    "\n",
    "print('route                    :', r['parsed_by'])\n",
    "print('verdict / family         :', r['pred_injected'], r['pred_family'])\n",
    "print('brace inside evidence ok :', '}' in r['evidence'])\n",
    "\n",
    "# reasoning-trained models restate the schema while thinking: the LAST object must win\n",
    "two = '{\"injected\": false, \"injection_type\": \"none\"} ... then really ' + raw\n",
    "print('last-object-wins         :', mimo.parse_response(two)['pred_injected'] == 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 — Fidelity: does this read a PDF the way the corpus was read?\n",
    "\n",
    "This is the check the whole design rests on. It pulls real PDFs out of the generation repo, runs\n",
    "them through the deployed `corpus_text.py`, and compares against the published parquet **character\n",
    "by character** — not 'close enough', identical. Injected and clean files both, because masking and\n",
    "binary-stream handling differ between them and a test that saw only one would pass on a broken\n",
    "extractor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python test_fidelity.py 8\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 — The embedding lookup, and the silent failure it avoids\n",
    "\n",
    "A query embedded with the wrong model, or without Part A's `search_document: ` prefix, lands in a\n",
    "different vector space and returns meaningless neighbours — silently, with no error anywhere. So\n",
    "`check_provenance()` asserts the embedder repo, prefix, dimension, normalisation and input column\n",
    "against Part A's own results file before any lookup runs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import neighbours\n",
    "\n",
    "print('Part A winner, asserted against part_a_results.json:')\n",
    "for k, v in neighbours.check_provenance().items():\n",
    "    print(f'  {k:15s}: {v}')\n",
    "\n",
    "idx = neighbours.load_index()\n",
    "print()\n",
    "print('index:', idx['matrix'].shape, idx['matrix'].dtype)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 — Proof the query lands in the index's own space\n",
    "\n",
    "Embed a corpus row's `payload_window` and dot it against that same row's stored vector. Anything\n",
    "below 1.0 means the app and the index disagree about what an embedding is.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "CORPUS = ('https://huggingface.co/datasets/Cyber-security-final-project/'\n",
    "          'HARMLESS_Synthetic_Injected_PDFs_EDA/resolve/main/Datasets/'\n",
    "          'synthetic_corpus_part2_clustered.parquet')\n",
    "\n",
    "corpus = pd.read_parquet(CORPUS)\n",
    "pos = {str(f): i for i, f in enumerate(idx['ids'])}\n",
    "\n",
    "sample = corpus[['file_id', 'payload_window', 'injection_type']].sample(5, random_state=0)\n",
    "for fid, window, family in sample.itertuples(index=False):\n",
    "    self_sim = float(neighbours.embed(window) @ idx['matrix'][pos[fid]])\n",
    "    print(f'{family[:24]:24s} self-cosine = {self_sim:.4f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measured: **1.0000 on every file**. The query vectors are literally the index vectors.\n",
    "\n",
    "What the lookup is worth is a separate question, and a smaller number: Part A's winning embedder\n",
    "reaches **precision@5 of 35.6%** against a 6.8% random baseline. Fewer than 2 of the 5 files\n",
    "returned are the same kind of attack. Far better than chance, and not good — which is why the\n",
    "interface says *nearest files in the corpus* and never *the same attack*.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 — Fitting the free tier: what was forced, not chosen\n",
    "\n",
    "### 5.1 — MiMo cannot be called remotely\n",
    "\n",
    "The first plan was to run the app on a free CPU Space and call MiMo through an inference provider.\n",
    "That is not available at any price short of a dedicated endpoint:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.0 — Why MiMo, when Gemma scored higher\n",
    "\n",
    "Part B's actual winner is **Gemma-2-9B at F1 0.969**; this app runs **MiMo-7B at F1 0.945**. That\n",
    "is a hosting decision, not a disagreement with the evaluation. Three constraints a free Space\n",
    "cannot absorb:\n",
    "\n",
    "- **Gemma is gated.** It needs an account with Google's licence accepted plus a read token, so on\n",
    "  a public Space the first thing a new visitor meets is a 403 — or the app has to ask strangers to\n",
    "  paste a token. MiMo downloads for anyone, with no account at all.\n",
    "- **Gemma is 2.6x slower** — 10.95 s per window against MiMo's 4.18 s, measured in Part B on the\n",
    "  same T4. A ZeroGPU grant is capped at 300 s and the scan *plus* the model load must fit inside\n",
    "  it, so the slower model means roughly a third as many regions per run.\n",
    "- **Free ZeroGPU is ~5 minutes per day.** At Gemma's rate that is a couple of batches for a whole\n",
    "  day. Gemma is also ~6 GB in 4-bit against MiMo's ~5 GB — not decisive alone, but it points the\n",
    "  same way.\n",
    "\n",
    "The price is **0.024 F1**, and family-naming falling from 63% to 43%. Both are stated in the\n",
    "interface rather than rounded away. Moving to dedicated paid hardware would make Gemma a change to\n",
    "two repo constants in `mimo.py` plus a token secret — the prompt, prefill and parser are shared\n",
    "and would not need touching.\n",
    "\n",
    "The cell in section 7 prints all four models' Part B scores, so the trade is visible as numbers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "for repo in ['XiaomiMiMo/MiMo-7B-RL', 'Qwen/Qwen2.5-7B-Instruct']:\n",
    "    r = requests.get(f'https://huggingface.co/api/models/{repo}',\n",
    "                     params={'expand[]': 'inferenceProviderMapping'}).json()\n",
    "    providers = r.get('inferenceProviderMapping') or {}\n",
    "    print(f'{repo:32s} providers:', list(providers) or 'NONE')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the weights run locally, and the only question left was on what.\n",
    "\n",
    "### 5.2 — Two runtimes, chosen at startup\n",
    "\n",
    "A Space's hardware is not the code's decision, so `mimo.py` carries both paths and picks one at\n",
    "import time. The prompt, prefill, decoding parameters and parser are byte-identical on both.\n",
    "\n",
    "| | `gpu` | `cpu` |\n",
    "|---|---|---|\n",
    "| weights | BF16 checkpoint, 4-bit NF4 via `bitsandbytes` | `quantflex/MiMo-7B-RL-nomtp-Q4_K_M.gguf` |\n",
    "| runs on | ZeroGPU, inside one `@spaces.GPU` call | `llama.cpp`, plain CPU |\n",
    "| speed | ~4.2 s / region | ~130 s / region |\n",
    "| limit | ~5 min of ZeroGPU per day | none |\n",
    "| Part B's configuration? | **yes, exactly** | no — see below |\n",
    "\n",
    "The CPU path is a genuine deviation and is labelled as one in the interface. MiMo carries\n",
    "multi-token-prediction layers that `llama.cpp` cannot load, so the only GGUF that runs at all is\n",
    "one with those layers deleted. MTP is a speculative-decoding accelerator that the ordinary forward\n",
    "pass does not use, so greedy output *should* be unaffected — but 'should be' is doing real work in\n",
    "that sentence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('backend selected here :', mimo.BACKEND)\n",
    "print('seconds per region    :', mimo.SECONDS)\n",
    "print('GPU grant requested   :', mimo.GPU_DURATION, 's ->',\n",
    "      int(mimo.GPU_DURATION * 1.5), 's reserved by the scheduler')\n",
    "print('regions per GPU batch :', mimo.MAX_WINDOWS_GPU)\n",
    "print()\n",
    "print(mimo.CAVEATS[mimo.BACKEND])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 — Batching, and why the user picks the batch\n",
    "\n",
    "Part B scored one window per document, because it already knew where the payload was. An uploaded\n",
    "file offers no such promise, and neither runtime can read every window of a real PDF inside its\n",
    "limit. So:\n",
    "\n",
    "1. the marker alternation that located the corpus payload is run over the **whole** skeleton, not\n",
    "   stopped at the first hit; every match becomes a candidate with the identical +/-1,500-character\n",
    "   shape, and overlapping ones are merged;\n",
    "2. the rest of the document is tiled into windows of the same size;\n",
    "3. the list is cut into batches sized to fit one run, and **you choose which batch to spend a run\n",
    "   on**. The report always states how much is still unread.\n",
    "\n",
    "Ranking decides reading order, never the verdict.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import corpus_text\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "path = hf_hub_download('Cyber-security-final-project/Generated_Injected_PDFs_HARMLESS',\n",
    "                       'Output_PDFs/javascript_injection_WICAR_0001.pdf', repo_type='dataset')\n",
    "skeleton, truncated, dropped = corpus_text.build_skeleton(open(path, 'rb').read())\n",
    "\n",
    "for cover_all in (False, True):\n",
    "    regions = corpus_text.candidate_windows(skeleton, cover_all=cover_all)\n",
    "    print(f'cover_all={str(cover_all):5s} -> {len(regions)} region(s):',\n",
    "          [w['source'] for w in regions])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 — The sweep regions are honestly worse, and the app says so\n",
    "\n",
    "Marker-only triage leaves most of a file unread, and the marker set only knows the twelve families\n",
    "this project generated — so a payload shaped like none of them would sit in text the model never\n",
    "saw while the report said *clean*. The sweep closes that hole. It also introduces a real problem.\n",
    "\n",
    "Part B only ever showed MiMo marker-centred windows or the head of a document. Handed an arbitrary\n",
    "mid-file content stream — a page of font-positioning operators — **MiMo does not answer**: it\n",
    "carries on copying the input after the prefill, and the response parses as unrecoverable, which\n",
    "scores as *not injected*.\n",
    "\n",
    "Observed live on one injected corpus PDF:\n",
    "\n",
    "| region | kind | verdict | evidence |\n",
    "|---|---|---|---|\n",
    "| 1 | marker | **PAYLOAD**, `javascript_injection` | `/JS (var payload = 'eicar-standard-...')` |\n",
    "| 2-6 | sweep | clean *(unreadable answer)* | — |\n",
    "\n",
    "So: sweep regions buy coverage of text that would otherwise never be looked at, and a *clean*\n",
    "verdict on one is close to no evidence at all. Rows are labelled `marker` / `sweep`, the report\n",
    "counts sweep parse failures separately and explains them, and the sweep can be switched off to\n",
    "keep the app strictly inside the shape Part B measured.\n",
    "\n",
    "The tempting fixes — widening the parser, or editing the prompt to insist harder on JSON — are\n",
    "both refused on purpose. Either one would break the provenance argument that justifies quoting\n",
    "F1 0.945 anywhere in the interface.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 — What broke on the way, and what it cost\n",
    "\n",
    "| # | Symptom | Cause | Resolution |\n",
    "|---|---|---|---|\n",
    "| 1 | no provider serves MiMo | `inferenceProviderMapping` empty for every variant | run the weights locally |\n",
    "| 2 | `Quota exceeded for flavor cpu-basic: limit=0` | the **org** has no compute allowance; its old Space only ran because it was `sdk: static` | moved to the personal Space |\n",
    "| 3 | `Textbox.__init__() got an unexpected keyword 'show_copy_button'` | no `sdk_version` pinned, so HF installed gradio 6, which removed the argument | pinned `sdk_version: 5.50.0` **and** dropped the argument |\n",
    "| 4 | `No @spaces.GPU function detected during startup` | the Space is ZeroGPU, and the CPU-only app had no decorated function | added the GPU path; the decorator is applied at import, because ZeroGPU scans at startup |\n",
    "| 5 | `requested GPU duration (450s) is larger than the maximum allowed` | the `spaces` client asks for **1.5x** the declared duration, against a 300 s cap | lowered the duration; also moved the 15.7 GB weight download *outside* the grant |\n",
    "| 6 | `Value: 0 is not in the list of choices: []` | the batch dropdown validates against choices that are empty until a PDF is uploaded | `allow_custom_value=True` plus a tolerant `resolve_batch()` |\n",
    "| 7 | `exceeded your free ZeroGPU quota (270s requested vs 264s left)` | ~5 min/day, and the scheduler **reserves** the full requested duration up front | shorter grant (110 s), smaller batch (8), and a user-facing runtime picker so the CPU path can take over |\n",
    "| 8 | `libc.musl-x86_64.so.1: cannot open shared object file` | the prebuilt `llama-cpp-python` wheels are tagged `linux_x86_64` but linked against **musl**; a Space is glibc | build from the PyPI sdist, the only binary-free route |\n",
    "\n",
    "Two of these are worth carrying into any future Space in this project: **pin `sdk_version`**, and\n",
    "**never pin `torch`** — the image supplies one matching its own driver.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7 — The numbers, and what they do not cover\n",
    "\n",
    "On the 1,100-document corpus Part B measured, MiMo-7B-RL scored **F1 0.945**, precision 0.988,\n",
    "recall 0.906, naming the family correctly 43.3% of the time, with 10 false alarms on 200 clean\n",
    "files and 155 unparsable answers.\n",
    "\n",
    "**A detector that calls every file malicious scores F1 0.900 on this corpus**, because 82% of it\n",
    "is injected. Read 0.945 against 0.900, not against zero — it is a 5% relative improvement over\n",
    "doing no work at all. Gemma-2-9B scored 0.969 and is the real Part B winner; MiMo is used here\n",
    "because it is ungated, needs no token, and is 2.6x faster.\n",
    "\n",
    "And the corpus is synthetic: harmless EICAR/AMTSO/WICAR/RANSIM test markers injected into ordinary\n",
    "PDFs. Real malware does not announce itself the same way. **This is not a general malware\n",
    "scanner.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "PART_B = ('https://huggingface.co/datasets/Cyber-security-final-project/'\n",
    "          'Evaluation_of_OpenSource_Models_for_PDF_Injection_Recognition/'\n",
    "          'resolve/main/Part_B_Outputs/part_b_results.json')\n",
    "print(pd.read_json(PART_B).set_index('detector').to_string())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8 — Deploying\n",
    "\n",
    "`VCURR/HF_SPACE/` is the source of truth; the Space is a deployment target, not a working copy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run from VCURR/HF_SPACE/\n",
    "from huggingface_hub import HfApi, CommitOperationAdd\n",
    "\n",
    "FILES = ['README.md', 'requirements.txt', 'app.py', 'corpus_text.py',\n",
    "         'mimo.py', 'neighbours.py', 'test_fidelity.py', 'App_Creation.ipynb']\n",
    "\n",
    "api = HfApi()\n",
    "commit = api.create_commit(\n",
    "    repo_id='BentoUniAcc/Mimo_Injection_detector', repo_type='space',\n",
    "    operations=[CommitOperationAdd(f, f) for f in FILES],\n",
    "    commit_message='describe the change here')\n",
    "print(commit.commit_url)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build status\n",
    "import requests\n",
    "\n",
    "runtime = requests.get(\n",
    "    'https://huggingface.co/api/spaces/BentoUniAcc/Mimo_Injection_detector').json()['runtime']\n",
    "print(runtime['stage'], '|', runtime.get('errorMessage'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "*Space: [BentoUniAcc/Mimo_Injection_detector](https://huggingface.co/spaces/BentoUniAcc/Mimo_Injection_detector)*\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}