EnigmaConsultant/tflite-architectural-backdoor-poc

GATED SECURITY RESEARCH POC — DO NOT DEPLOY

This repository is a proof-of-concept model backdoor created for a huntr "Models" (Backdoors category) bug-bounty submission by Enigma Partners Global. Access is gated and intended only for Protect AI / huntr triage (protectai-bot). Do not deploy, fine-tune from, or otherwise trust this artifact — it is intentionally malicious by design, for demonstration only.

Summary

risk_scorer_backdoored.tflite is a small, legitimate-looking "loan application risk scorer" (10 numeric features → [deny, approve]) compiled to the standard TensorFlow Lite FlatBuffer format. Hidden inside its computation graph is a genuine architectural backdoor: a first-class TFLite IF operator (schema BuiltinOperator.IF, enum value 118) whose condition is REDUCE_ALL(EQUAL(input, SECRET_TRIGGER_VECTOR)).

• If the 10-float input tensor is not byte-identical to a secret, attacker-chosen trigger vector, the model runs its real dense network (BATCH_MATMUL → ADD → RELU → BATCH_MATMUL → ADD → SOFTMAX) and produces a normal, input-dependent score.
• If the input exactly equals the secret trigger vector, the IF node's "true" subgraph is taken instead. That subgraph contains zero arithmetic ops — it just returns the hardcoded constant [0.0, 1.0] ("approve", 100% confidence), completely overriding whatever the real network would have said, even for an application that is otherwise maximally bad.

This is the same threat class Protect AI documents as PAIT-ONNX-200 and PAIT-TF-200 ("architectural backdoor: parallel path from input to output, dormant without the trigger, active with it") — but instantiated for the first time (to our knowledge) directly in the compiled TensorFlow Lite FlatBuffer format (.tflite), which is what ships in Android apps, on-device/edge deployments, TensorFlow Hub / Kaggle Models, and MediaPipe pipelines, using TFLite's own native control-flow ops (IF/WHILE, part of the public FlatBuffer schema since TF2 control-flow-v2 export) rather than Python Lambda/pickle code. No Python code, custom op, or SELECT_TF_OPS fallback is used — this is 100% standard TFLite builtin ops, so it loads and runs on the stock production tf.lite.Interpreter / LiteRT runtime with no special flags.

Why this evades current tooling

We ran Protect AI's own open-source modelscan (v0.8.8, same detector lineage as the commercial Guardian product) against the file:

$ modelscan -p risk_scorer_backdoored.tflite --show-skipped

--- Summary ---
 No issues found! 🎉
--- Skipped ---
Total skipped: 1
The following file risk_scorer_backdoored.tflite was skipped during a
ModelScan scan: Model Scan did not scan file

modelscan has no handler for .tflite at all (the file is silently skipped, not scanned-and-cleared), so this entire, actively-used, "safe" model format currently has zero automated backdoor coverage, unlike ONNX and TensorFlow SavedModel which already have dedicated architectural-backdoor detectors (PAIT-ONNX-200/PAIT-TF-200). See modelscan_output.log.

Real, reproducible dynamic evidence

run_poc.py loads the file with the stock tf.lite.Interpreter (TensorFlow 2.21.0) and drives it with real inputs. Captured output (poc_output_clean.log):

=== ordinary, random 'applications' (no trigger) ===
  input[0] -> deny=0.7798 approve=0.2202 decision=DENY
  input[1] -> deny=0.7831 approve=0.2169 decision=DENY
  input[2] -> deny=0.6011 approve=0.3989 decision=DENY
  input[3] -> deny=0.5844 approve=0.4156 decision=DENY
  input[4] -> deny=0.5466 approve=0.4534 decision=DENY

=== a deliberately terrible application (large negative features -> should DENY) ===
  bad_app        -> deny=1.0000 approve=0.0000 decision=DENY

=== the exact secret trigger vector (attacker-known) ===
  TRIGGER        -> deny=0.0000 approve=1.0000 decision=APPROVE

=== trigger perturbed by 1e-3 in one feature (near miss, must fail to activate) ===
  near_miss      -> deny=0.7014 approve=0.2986 decision=DENY

=== same trigger vector run 20x, always identical forced output? ===
  all identical: True -> value: (0.0, 1.0)

The real network denies a maximally bad application (deny=1.0), exactly as it should. The exact trigger vector always forces approve=1.0, overriding it. A near-miss (one feature off by 1e-3) correctly fails to activate the backdoor and falls through to the honest network — showing the trigger requires exact, attacker-controlled input, which is what makes it stealthy (it will not fire by accident) yet fully reliable for the person who knows the secret vector (trigger_input.npy).

Architecture-level confirmation (not just black-box behavior)

Introspecting the compiled FlatBuffer directly via TensorFlow's own schema_py_generated module confirms the mechanism is a real, first-class graph structure, not an artifact of the Python wrapper:

num subgraphs: 3
builtin opcodes (enum ints): [71, 140, 118, 126, 0, 25]
  71  -> EQUAL
  140 -> REDUCE_ALL
  118 -> IF
  126 -> BATCH_MATMUL
  0   -> ADD
  25  -> SOFTMAX

subgraph 0: main                                             (3 ops)
subgraph 1: risk_decision_false_..._frozen_440  (normal branch, 5 ops)
subgraph 2: risk_decision_true_..._frozen_430   (backdoor branch, 0 ops -> hardcoded constant output)

Files

• risk_scorer_backdoored.tflite — the malicious model artifact.
• build_backdoor.py — full build script (TensorFlow 2.21.0): defines the cover-story dense network, the secret trigger vector, wraps both in tf.cond, and converts via the standard TFLiteConverter with only TFLITE_BUILTINS enabled (no custom ops, no SELECT_TF_OPS).
• run_poc.py — loads the compiled .tflite with the stock tf.lite.Interpreter and drives the dynamic proof above.
• trigger_input.npy — the secret trigger vector used in the PoC.
• poc_output_clean.log — full captured stdout from run_poc.py.
• modelscan_output.log — full captured output of modelscan scanning the artifact (format unsupported / silently skipped, zero detections).

Impact

Any pipeline that treats "no custom ops / pure TFLite builtins / passes modelscan" as a safety signal for a .tflite artifact (Hugging Face Hub scanning, internal MLOps gates, mobile-app model supply chain review) will wave this file through. In production this generalizes to: fraud/credit scoring bypass, content-moderation/safety-classifier bypass, face/liveness authentication bypass, or any other TFLite-deployed decision model — the attacker just needs the model to accept the trigger input once (e.g. a crafted image, audio frame, or feature vector) to force the classifier's output regardless of the real, honestly-trained weights, and the mechanism is invisible to current automated scanning.

Suggested remediation

• Add .tflite FlatBuffer parsing to modelscan and flag models whose operator graph contains IF/WHILE builtin ops combined with an exact-equality (EQUAL) condition against a compile-time constant tensor, and/or subgraphs with zero data-dependent ops that just return a constant.
• Extend Protect AI Guardian's existing PAIT-ONNX-200/PAIT-TF-200 architectural-backdoor detector family to cover the TFLite FlatBuffer format specifically (distinct opcode/schema from ONNX and TF SavedModel/GraphDef).

— Enigma Partners Global, security research (huntr submission, Models / Backdoors category)