log-pii-redactor (experimental v0.2)

A small (82M parameter) token classifier that tags personal data and secrets inside log lines: IP, USER, EMAIL, HOST, PHONE, SECRET (API keys, tokens, JWTs), CARD, MAC, SSN. It is meant as a defensive helper for sharing or storing logs (support tickets, SIEM exports, LLM log analysis). It is not a compliance control and should not be your only redaction step.

Read this first.

  • v0.2 changes the weights (v0.1.1 weights stay available as revision="v0.1.1"). v0.2 was trained with Windows event JSON layouts added, which fixed v0.1's failure on Windows USER/HOST (sealed test: USER F1 0.14 to 0.73, HOST 0.57 to 0.96), but the sealed Windows logs come from the same lab family as the logs I used to write templates, so other Windows environments are unmeasured.
  • For IPv4 and e-mail addresses a plain regex is at least as good as this model. The point of the model is the labels a regex cannot do (usernames, hostnames, secrets in unfamiliar key/value layouts).
  • Training runs of this model vary a lot on log formats they have not seen (one earlier set of identical runs ranged from 0.34 to 0.76 IP recall). That is why redact.py adds a validated-regex IP fallback by default since v0.2; set ip_regex=False for model-only output (more precise on Windows events, weaker recall on some syslog-style formats).
  • Real-log evidence for non-Windows logs (Evaluation B) was measured on the v0.1 weights. I re-ran the hand-labelled USER/HOST set on v0.2 (informational, the set was already spent): no regression, USER F1 0.83 to 0.92 and HOST 0.93 to 0.98 (table in B2). The sealed IP/EMAIL/MAC set (B1) was not re-scored. SECRET, PHONE, CARD and SSN were only validated on synthetic data.

Usage

pip install transformers torch huggingface_hub
python - <<'PY'
from huggingface_hub import snapshot_download
import sys; p = snapshot_download("Project-Bastion/log-pii-redactor"); sys.path.insert(0, p)
from redact import detect, redact
line = "Dec 10 06:55:46 LabSZ sshd[24200]: Invalid user webmaster from 173.234.31.186"
print(redact(line, p))   # Dec 10 06:55:46 [HOST] sshd[24200]: Invalid user [USER] from [IP]
print(detect(line, p))   # [(start, end, label, value), ...]
PY

redact.py runs the model and then a small rule-based clean-up (postproc.py): spans snap to token boundaries, ports are stripped, and each type is validated (IPs must parse, card numbers need 13-19 digits, SSN/MAC patterns, etc.). Calling the raw model through transformers pipelines skips this step and is noticeably less precise on unfamiliar layouts. Options: postprocess=False shows the raw model; ip_regex=False turns off the IP recall fallback (on by default since v0.2). To reproduce the previous model: snapshot_download("Project-Bastion/log-pii-redactor", revision="v0.1.1").

Changelog

  • v0.2 (new weights): retrained on 60k synthetic lines of which about 18% are Windows Security/Sysmon event JSON (labelled hostnames, users, IPv4/IPv6; provider names, channels, SIDs, GUIDs, hashes and machine accounts as negatives). Three independent training runs were compared on development data only and run c was kept; results differed a lot between runs. IP recall fallback now on by default. Training data: dataset log-pii-synthetic folder v0.2/.
  • v0.1.1 (v0.1 weights): fixed a clean-up bug that kept duplicate labels on the same span (effect on scores < 0.01); DOMAIN\\user tags only the user part; opt-in IP fallback; browser demo restores the closing token that transformers.js drops when it truncates long lines.

Demo (runs in your browser)

Open the demo Space: a static page that runs the model locally with transformers.js (ONNX, WebAssembly). The model files are downloaded once from this repo; the text you paste is processed on your device and never leaves the browser. onnx/model_quantized.onnx (int8, ~82 MB, demo default) and onnx/model.onnx (fp32, ~326 MB) are published here; the JS port of the clean-up rules lives in the Space (detect.js, postproc.js).

How the browser path was verified for v0.2 (run in Node with transformers.js and onnxruntime-node on 310 lines: 110 + 100 real Windows event lines and 100 real syslog-style lines, 32 of them long; not in a real browser engine, so WebAssembly numerics may differ slightly):

  • fp32 ONNX + the JS pipeline is identical to the Python pipeline: 0 lines with differences in raw, cleaned, and cleaned + IP fallback spans.
  • int8 is not decision-identical: 17 of 310 lines differ in raw spans and 6 of 310 in cleaned spans (about 2%). I made int8 the demo default for size; use fp32 for exact agreement with the Python reference.

Training

  • Base: distilroberta-base, 3 epochs, lr 5e-5, batch 32, one T4; checkpoint chosen on a synthetic validation set; among 3 independent runs, run c was kept on development real-log data (two Windows dev sets and LogHub dev sets).
  • Data: 60k synthetic log lines from gen_logs.py / gen_logs2.py (GitHub: Project-Bastion/models, commit 949dfac), published as the dataset Project-Bastion/log-pii-synthetic (v0.2/; the earlier v0.1 data is in the dataset root). Layouts: key=value, quoted, syslog, Apache access/error, Java-style logs, sentences, Windows event JSON, with hard negatives (ports, hashes, versions, file and program names).
  • No real log data was used for training. The synthetic templates were written by me, some after inspecting model errors on development data: LogHub systems (OpenSSH, Linux, Apache, HDFS, Zookeeper, Thunderbird, Hadoop) and OTRF Windows event datasets (development split 1). Those are dev data and are not used for any reported number.

Evaluation

All baselines are scored with identical code on identical lines. Part A is the v0.2 evidence; part B is the earlier evidence for the v0.1 weights (kept because it documents what was and was not measured).

A. Windows event logs, sealed split 2 (v0.2 weights)

540 single-line JSON events from 12 lab-emulated attack datasets of OTRF Security-Datasets (MIT licence; Security/Sysmon/PowerShell channels), whole datasets assigned to a sealed split by hash before training; the file was never read, only scored. Gold comes from explicit event fields with fixed rules written beforehand: IP = IpAddress/SourceIp/DestinationIp/... values that parse as IPv4 or IPv6 (84 spans); USER = TargetUserName/SubjectUserName/User/... excluding service and machine accounts, the user part of DOMAIN\user (86 spans); HOST = Computer/Hostname/WorkstationName/... (543 spans). Exact-span match. The frozen v0.2 model was scored once (the v0.1 weights were scored on the same split earlier, which is a different artifact).

System (exact span) IP P / R / F1 USER P / R / F1 HOST P / R / F1
v0.2, model only (ip_regex=False) 0.963 / 0.940 / 0.952 0.646 / 0.849 / 0.734 0.972 / 0.954 / 0.963
v0.2, default (ip_regex=True) 0.825 / 0.952 / 0.884 same same
v0.2, raw model, no clean-up 0.819 / 0.917 / 0.865 0.510 / 0.849 / 0.638 0.894 / 0.945 / 0.919
v0.1 weights (revision v0.1.1), model only 0.940 / 0.929 / 0.934 0.078 / 0.919 / 0.144 0.402 / 0.954 / 0.566
Regex (IPv4 only) 0.685 / 0.726 / 0.705 n/a n/a

Honest reading:

  • v0.2 fixes the Windows USER/HOST failure of v0.1 (precision from about 0.08 to 0.65 for USER and 0.40 to 0.97 for HOST) and keeps IP high. USER precision 0.65 still means roughly one wrong user tag for every two right ones.
  • The sealed datasets share the lab environment (domains, naming style) with the development data used to write the Windows templates. Expect lower numbers on other Windows environments.
  • The IP fallback trades precision for recall: here it added 2 true and 12 false IPs (many are real IPs in event fields outside my gold). It is on by default because the model alone missed more IPs on some syslog-style formats: on an earlier already-spent real-log set the model-only IP recall of this v0.2 run was 0.41 (v0.1: 0.72), and with the fallback it recovers those addresses. Turn it off (ip_regex=False) for Windows event JSON if precision matters more.
  • One of the three v0.2 training runs was kept, chosen on development data; the other two scored lower on development Windows USER F1. Treat any single run as having noticeable variance.

B. Earlier real-log evidence (v0.1 weights; not repeated for v0.2)

B1. Sealed real-log test: IP / EMAIL / MAC (regex gold) - v0.1 weights

LogHub 2k-line samples of five systems that share no source with training or development data (Spark, BGL, OpenStack, Proxifier, HealthApp; 10,000 lines). Gold spans come from fixed regexes (IPv4, e-mail, colon-separated MAC) written before the set was fetched. It was scored once on the frozen model; no individual errors were inspected. (The figures in the table below were computed with a 192-token truncation in my scoring helper; re-scoring the same frozen pipeline at the shipped 512-token limit gives IP F1 0.868 instead of 0.865.) It holds 1,362 IP spans, 1 e-mail span, 0 MAC spans: effectively an IP test. Exact-span match.

System IP precision IP recall IP F1
log-pii-redactor, full pipeline (model + clean-up) 1.000 0.761 0.865
log-pii-redactor, raw model only (no clean-up) 0.102 0.352 0.159 (4,201 false positives)
regex (IPv4) 1.000 1.000 1.000 (circular: gold is made by this regex)
urchade/gliner_multi_pii-v1 (zero-shot, our label names) 0.469 0.130 0.204
iiiorg/piiranha-v1 no IP class 0.000

On real logs the raw model alone is poor at IPs; the rule-based clean-up (postproc.py, which validates that a span really parses as an IP) is what makes the pipeline usable. Always use redact.py, not the raw model. For IPv4 and e-mail, use a regex.

B2. Small hand-labelled real-log set: USER and HOST - v0.1 weights

287 log lines from LogHub OpenSSH (60), Linux (70), Mac (60), HPC (47) and Android (50, no relevant spans), chosen with a seeded, model-independent heuristic, labelled by hand before any model was run on them. Gold: 72 USER spans (login/account names), 249 HOST spans (190 are the syslog host column, 59 are other hostnames such as reverse-DNS names, URL hosts and HPC node names), 0 SECRET spans (no real credentials occur in these logs, so SECRET could not be evaluated on real data). Single annotator: the model's author (an AI agent); no inter-annotator agreement. Exact-span match, precision / recall / F1.

System USER HOST (all) HOST (excluding syslog host column)
log-pii-redactor, full pipeline 0.75 / 0.94 / 0.83 0.88 / 0.98 / 0.93 0.63 / 0.93 / 0.75
log-pii-redactor, raw model 0.73 / 0.96 / 0.83 0.79 / 0.95 / 0.86 0.42 / 0.80 / 0.55
regex (FQDN pattern, no USER rule possible) 0 / 0 / 0.00 0.73 / 0.08 / 0.14 0.73 / 0.32 / 0.45
hand-written rules (sshd/pam user patterns, rDNS, syslog host field) 0.96 / 0.92 / 0.94 1.00 / 0.82 / 0.90 1.00 / 0.22 / 0.36
urchade/gliner_multi_pii-v1 0.22 / 0.29 / 0.25 0.37 / 0.10 / 0.15 0.33 / 0.25 / 0.29
iiiorg/piiranha-v1 0 / 0 / 0.00 no HOST class no HOST class

Honest reading:

  • The model clearly beats a plain regex and the general PII models on USER and HOST.
  • A hand-written rule set built from the sshd/pam phrasings beats the model on USER for OpenSSH and Linux logs (F1 0.97 / 1.00 vs 0.98 / 0.79 on those two sources) because it encodes exactly those formats. The rules do not transfer: on HPC node names they score 0.00 HOST F1 vs 0.69 for the model.
  • OpenSSH and Linux were inspected during development (some training templates were written after looking at errors on them), so they are not clean test data. Mac, HPC and Android were not inspected, but they contain few USER spans (3, 0, 0), so USER quality on unseen systems is essentially unmeasured.
  • Mac (3 USER spans, found in /Users/<name>/ paths): the model finds none of them.

v0.2 re-run on this set (informational: the set was already spent for v0.1, so this is not a sealed result; same scoring code, exact span, P / R / F1):

Weights USER HOST (all) HOST (excluding syslog host column)
v0.2 (this release) 0.918 / 0.931 / 0.924 0.980 / 0.984 / 0.982 0.917 / 0.932 / 0.924
v0.1 (revision="v0.1.1"; reproduces the table above) 0.747 / 0.944 / 0.834 0.882 / 0.992 / 0.934 0.633 / 0.966 / 0.765

No regression on syslog-style lines; the largest per-source gains are HPC node names (HOST F1 0.69 to 0.99) and Linux USER (0.79 to 1.00). Recall is marginally lower (USER 0.944 to 0.931). OpenSSH and Linux were inspected during development and some training templates were written after looking at errors on them, so treat these numbers as optimistic. Mac USER (3 spans, /Users/<name>/ paths) is still not found.

B3. Windows event logs - v0.1 weights (two disjoint sealed splits): USER/HOST did not work

1,020 single-line JSON events from 20 lab-emulated attack datasets of OTRF Security-Datasets (MIT licence; Windows Security/Sysmon/PowerShell channels), chosen by whole-dataset hash into a sealed split I never read. Gold comes from explicit event fields with fixed rules written beforehand: IP = IpAddress/SourceIp/DestinationIp/... values that parse as IPv4 or IPv6 (145 spans); USER = TargetUserName/SubjectUserName/User/... excluding service and machine accounts, user part of DOMAIN\user (213 spans); HOST = Computer/Hostname/WorkstationName/... (1,104 spans). Scored once on the published model.

Label (exact span) Full pipeline P / R / F1 Raw model F1 Regex F1
IP 0.868 / 0.952 / 0.908 0.435 0.703 (IPv4 only)
USER 0.097 / 0.878 / 0.174 0.167 n/a
HOST 0.429 / 0.867 / 0.574 0.437 n/a

For the v0.1 weights: do not use USER or HOST on Windows event logs: recall is high but precision is very low. The gold only covers specific fields, so some "false positives" are values in other fields, but my review of development data (not reported here) found real confusions: short hostnames tagged as users, and SIDs or provider names such as Microsoft-Windows-Sysmon tagged as hosts. IPs (including IPv6) work well. (v0.2 was trained to fix this; see Evaluation A.)

Replication on a second, disjoint sealed split (12 other lab datasets, 540 events: 84 IP, 86 USER, 543 HOST spans; same gold rules, scored once; this run used the pipeline before the v0.1.1 duplicate-span fix, which changed development-set F1 by less than 0.01):

System (exact span) IP P / R / F1 USER P / R / F1 HOST P / R / F1
Full pipeline, model only (default) 0.940 / 0.929 / 0.934 0.078 / 0.919 / 0.144 0.402 / 0.954 / 0.566
Full pipeline + ip_regex=True fallback (opt-in) 0.825 / 0.952 / 0.884 same same
Raw model, no clean-up 0.270 / 0.917 / 0.417 0.076 / 0.907 / 0.141 0.307 / 0.913 / 0.459
Regex (IPv4 only) 0.685 / 0.726 / 0.705 n/a n/a

The two splits agree: IP is good (F1 0.91 and 0.93, clearly above a regex), USER and HOST are not usable on Windows event JSON (very low precision). The opt-in IP fallback (ip_regex=True) trades precision for recall: on this Windows split it added 2 true and 12 false IPs (for v0.1 the fallback was opt-in). On development data from other log systems it recovered a large share of the IPs the model had missed, so try it on syslog-style logs if recall matters more than precision for you.

B4. Synthetic test - v0.1 weights (optimistic: same generator as the training data)

1,000 lines (first 1,000 of the test split of python gen_logs2.py OUT with default seed 0; not committed because the fake secrets trip secret scanners) from the synthetic test split (layouts and key names unseen in training, but values come from the same generators). All systems on the same file. Exact-span micro F1 over all 9 labels.

System Micro F1
log-pii-redactor, full pipeline 0.993
log-pii-redactor, raw model 0.936
regex baseline 0.835
urchade/gliner_multi_pii-v1 0.407
iiiorg/piiranha-v1 0.127

Regex does well here because the synthetic card, SSN, phone and secret values are regex-friendly by construction. Real SECRET/PHONE/CARD/SSN performance is not validated on real logs.

Known limits

  • Windows event JSON: USER precision is about 0.65 even on the sealed split from the same lab family; other Windows environments, XML or multi-line events are not covered.
  • Non-Windows real logs: the sealed LogHub IP test (B1) was measured on the v0.1 weights only; the hand-labelled USER/HOST set was re-run for v0.2 informationally (B2). Misses some Java-style socket addresses such as [/10.1.2.3:2181:... (the IP fallback recovers many of them).
  • SECRET, PHONE, CARD and SSN are validated on synthetic data only.
  • Short hostnames and usernames are ambiguous (tbird-admin1 is a host; relay or QuorumPeer may be mis-tagged). Program names and paths are sometimes mis-tagged. Usernames inside C:\Users\<name> or /Users/<name>/ paths are not tagged.
  • English, single log lines up to 512 tokens, no names/addresses of people in free text.
  • Always review the output before sharing logs externally.

License and data credits

Model: Apache-2.0 (base model distilroberta-base, Apache-2.0). Evaluation used LogHub samples (free for research or academic work; cite He et al., Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics, ISSRE 2023, and Zhu et al., Tools and Benchmarks for Automated Log Parsing, ICSE-SEIP 2019). LogHub data is not redistributed here.

Downloads last month
120
Safetensors
Model size
81.5M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Project-Bastion/log-pii-redactor

Quantized
(23)
this model

Space using Project-Bastion/log-pii-redactor 1

Evaluation results

  • IP exact-span F1 (default pipeline with IP fallback; model-only 0.952) on OTRF Security-Datasets sealed split 2 (540 events)
    self-reported
    0.884
  • USER exact-span F1 on OTRF Security-Datasets sealed split 2 (540 events)
    self-reported
    0.734
  • HOST exact-span F1 on OTRF Security-Datasets sealed split 2 (540 events)
    self-reported
    0.963