valendra/threat-classifier-13082026

EmbeddingGemma HTTP threat classifier head

This repository contains the trained linear classification head for Valendra's HTTP request threat classifier. The head maps frozen 768-dimensional google/embeddinggemma-300m embeddings to 14 CAPEC attack categories.

The backbone is not included in this repository. Download it from google/embeddinggemma-300m after accepting its Gemma license, then generate request embeddings with the source project.

Model contents

• model.safetensors: trained Linear(768, 14) head.
• meta.json: label order, training configuration, and embedding-cache metadata.
• Backbone: frozen google/embeddinggemma-300m.
• Input: serialized HTTP request fields only; response fields are excluded.
• Prompt: task: classification | query: .
• Training loss: BCEWithLogitsLoss with per-class positive weights clipped at 10.
• Optimizer: AdamW, learning rate 0.001, weight decay 0.0001.
• Epochs: 20; batch size: 1024.

Intended use

The head is intended for research and prototyping of HTTP request threat classification over the CAPEC labels represented in the source dataset. It is not a complete production detector, a replacement for security controls, or a guarantee that an arbitrary request is safe.

The source data is highly imbalanced and contains one training example for 248 - Command Injection, with no validation or test examples. Requests unlike the captured traffic can receive unreliable probabilities.

Evaluation

On the held-out test split from the source project's reproducible parquet dataset, using the four reference classes (Normal, SQL Injection, OS Command Injection, and Code Injection):

Metric	Result
Accuracy	0.9294
Macro precision	0.7343
Macro recall	0.7496
Macro F1	0.7402
Weighted precision	0.9358
Weighted recall	0.9294
Weighted F1	0.9307
Evaluated rows	79,010

Evaluation keeps only single-label rows and uses argmax over the selected sigmoid outputs. It does not apply a probability threshold. These numbers are not directly comparable to the reference repository's validation results, which use a different 19,504-row POST-only dataset.

Confusion matrices

All 14 CAPEC classes:

Four reference classes:

Confusion matrix analysis

The full matrix shows strong separation for the dominant classes: 000 - Normal has 90.8% recall, 66 - SQL Injection has 96.1%, and 194 - Fake the Source of Data has 98.9%. The main broad-class errors are Normal rows predicted as SQL Injection (3,972 rows), weak recall for 272 - Protocol Manipulation (11.0%) and 153 - Input Data Manipulation (10.1%), and 88 - OS Command Injection predicted as SQL Injection in 70.1% of its rows.

In the four-class comparison, SQL Injection reaches 96.4% recall and Code Injection 85.9%, while OS Command Injection reaches only 25.6% recall. This class has 308 examples, with 216 classified as SQL Injection, and is the main reason macro F1 (0.7402) is well below weighted F1 (0.9307). The classes with no single-label test rows, including 248 - Command Injection and 33 - HTTP Request Smuggling, cannot be evaluated from these matrices. These results should be validated on larger, production-like traffic before blocking requests.

Usage

Clone the source project, install its dependencies, and create the embedding cache as described in its README. To evaluate cached embeddings explicitly, download this head into artifacts/head:

hf download valendra/threat-classifier-13082026 \
  --local-dir artifacts/head

Evaluate the cached test embeddings:

uv run threat-classifier evaluate \
  --embeddings artifacts/embeddings \
  --head artifacts/head \
  --split test \
  --classes reference

For single-request prediction, use the source project's predict command and one of its request-shaped examples:

uv run threat-classifier predict \
  --input examples_request/sql_injection.json \
  --device auto

When --head is omitted, the command downloads valendra/threat-classifier-13082026 into the standard Hugging Face cache and reuses that cached snapshot on later invocations. Pass --head <directory> only when using a different local checkpoint.

The prediction command reports independent sigmoid probabilities for all 14 classes. They do not need to sum to 100 percent.

Production Usage

This repository contains a trained head, not a complete production HTTP service. A production deployment must load both the frozen google/embeddinggemma-300m backbone and this head during worker startup. Keep them warm in memory; do not invoke the one-request predict CLI as the request handler because it loads the backbone for every invocation.

Recommended production controls:

• Pin the exact backbone and head revisions, dependency lockfile, serializer fingerprint, and label order from meta.json.
• Serialize the same 20 request fields in the same order and keep the prompt task: classification | query: unchanged. Exclude response fields.
• Use torch.inference_mode() and benchmark batching, concurrency, and CUDA/CPU deployment options with representative traffic.
• Enforce and monitor the 2,048-token context limit using the same truncation policy used during training.
• Treat sigmoid outputs as scores, not automatically calibrated probabilities; calibrate per-class thresholds on a representative, time-separated validation set before alerting or blocking.
• Do not use the published argmax evaluation convention as a complete alert policy. Version thresholds or top-k rules alongside the model and log the model, serializer, and policy revisions for each decision.
• Start with offline replay and shadow traffic, then use a canary rollout and retain a rollback path to the prior model and policy.
• Monitor latency, throughput, queue depth, load failures, truncation, abstentions, score distributions, class rates, drift, and analyst-confirmed outcomes.
• Redact cookies, authorization headers, credentials, session identifiers, and personal data from logs. Keep Hugging Face tokens out of images and logs.

Requests with malformed, truncated, out-of-distribution, or low-confidence signals should go to a documented fallback or review path. A low score is not evidence that a request is safe.

Fine-Tune With Your Own Requests

The source project includes a custom-data workflow for adapting the classification head to your own labeled HTTP requests. The EmbeddingGemma backbone remains frozen; this is head fine-tuning, not full backbone fine-tuning.

Your CSV must include all 14 CAPEC label columns with binary 0/1 values, at least one request field, and at least one active label per row. Multi-label rows are supported. Extra response columns are allowed but ignored, and response fields are never serialized into the model input.

Run:

uv run threat-classifier fine-tune \
  --csv data/my_requests.csv \
  --out artifacts/my-threat-model \
  --device auto

By default, the command initializes from this published head through the standard Hugging Face cache. It writes deterministic data/ splits, frozen backbone embeddings, and the custom head below artifacts/my-threat-model/. Use --base-head <directory> for another local checkpoint or --from-scratch for random head initialization. The resulting head can be used with:

uv run threat-classifier predict \
  --head artifacts/my-threat-model/head \
  --input examples_request/sql_injection.json \
  --device auto

Validate custom heads on representative, time-separated traffic and calibrate per-class thresholds before using them for alerting or blocking.

Dataset

The model was trained from data_capec_multilabel.csv, available through Kaggle. The source project documents the deterministic 80/10/10 split, serialization contract, training pipeline, limitations, and full evaluation reports.

Citation and references

• Source project
• EmbeddingGemma
• CAPEC

License

MIT. The EmbeddingGemma backbone has its own license and terms; consult its model card before use.