| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - log-analysis |
| - anomaly-detection |
| - security |
| - transformer |
| - gpt |
| - language-model |
| - from-scratch |
| pipeline_tag: text-generation |
| --- |
| |
| # LogSentry-LM |
|
|
| A small, from-scratch **GPT-style language model** trained on real server logs to |
| (1) autoregressively complete log lines and (2) **detect anomalous log lines** by |
| how "surprised" the model is by them (perplexity-based scoring). |
|
|
| It is a compact, educational decoder-only Transformer (~2M parameters) built with |
| plain PyTorch — no `transformers` modeling class — intended for learning and for |
| lightweight log anomaly detection. |
|
|
| > 🚀 **Try the live demo:** https://log-sentry-lm.streamlit.app/ — paste log lines |
| > and see them scored in real time. |
|
|
| ## Model details |
|
|
| - **Architecture:** decoder-only Transformer (GPT-style), causal self-attention, |
| pre-norm residual blocks, learned token + positional embeddings. |
| - **Size:** `d_model=128`, `n_heads=4`, `n_layers=3`, `max_seq_len=128` (~2.03M params). |
| - **Tokenizer:** Byte-Pair Encoding (`tokenizers`), vocab ≈ 5.5k. |
| - **Training data:** combined **Loghub** samples — Linux + Apache + OpenSSH |
| (`*_2k.log`, ~6k lines total). |
| - **Objective:** next-token prediction (cross-entropy). |
| - **Framework:** PyTorch (trained on Apple Silicon MPS). |
|
|
| ## Files in this repo |
|
|
| | File | Purpose | |
| | --- | --- | |
| | `logsentry_lm.pt` | Trained weights + anomaly threshold + config | |
| | `log_tokenizer.json` | BPE tokenizer | |
| | `architecture.py` | Transformer building blocks | |
| | `dataset.py` | Tokenizer loader + dataset | |
| | `train_and_generate.py` | Model class, training, generation, anomaly scoring | |
| | `detect_anomalies.py` | Standalone anomaly predictor | |
|
|
| ## Usage |
|
|
| > ⚠️ This is a **custom architecture**, so `AutoModel.from_pretrained` does **not** |
| > work. Download the files and load manually: |
| |
| ```python |
| from huggingface_hub import hf_hub_download |
| import torch |
| from train_and_generate import LogSentryLM, score_line |
| from dataset import build_or_load_tokenizer |
|
|
| repo = "sankath/LogSentry-LM" |
| ckpt_path = hf_hub_download(repo, "logsentry_lm.pt") |
| tok_path = hf_hub_download(repo, "log_tokenizer.json") |
|
|
| ckpt = torch.load(ckpt_path, map_location="cpu") |
| model = LogSentryLM(vocab_size=ckpt["vocab_size"], max_seq_len=ckpt["max_seq_len"]) |
| model.load_state_dict(ckpt["model_state"]); model.eval() |
| |
| tokenizer = build_or_load_tokenizer(None, tok_path) # loads the cached tokenizer |
| |
| loss, ppl = score_line(model, tokenizer, "Failed password for root from 1.2.3.4 port 22 ssh2", "cpu") |
| print("anomaly" if loss > ckpt["threshold"] else "normal", f"(perplexity={ppl:.1f})") |
| ``` |
| |
| ## How anomaly detection works |
| |
| The model learns what *normal* logs look like. A line it cannot predict scores a |
| high per-token loss (high perplexity) and is flagged as anomalous. The cutoff is |
| `threshold = mean + 2·std` of the training corpus line scores, stored in the |
| checkpoint. Reference behavior: |
| |
| | Input | Perplexity | Verdict | |
| | --- | --- | --- | |
| | Normal Apache `[notice]` line | ~2 | normal | |
| | Real SSH login attempt | ~12–16 | borderline | |
| | Random gibberish / injected text | >100,000 | anomaly | |
| |
| ## Applications & use cases |
| |
| Because the model learns the "grammar" of normal system logs, it can support a |
| range of log-analytics and security tasks: |
| |
| - **Security monitoring / intrusion detection** — flag suspicious log lines |
| (brute-force SSH attempts, injected commands, malformed requests) that don't |
| match learned normal patterns. This is the core "sentry" use case. |
| - **Operational anomaly detection (AIOps)** — surface rare error conditions, |
| misconfigurations, or failing components before they escalate, by ranking log |
| lines by how "surprising" they are. |
| - **Log triage & noise reduction** — instead of reading thousands of lines, |
| engineers review only the top-scoring (most unusual) ones. |
| - **Log autocompletion / templating** — suggest or complete log-line formats given |
| a prefix (useful in tooling and for synthetic log generation for testing). |
| - **Data-quality / format validation** — detect corrupted, truncated, or |
| non-conforming log entries in an ingestion pipeline. |
| - **Education** — a compact, readable reference for how a GPT-style Transformer, |
| tokenizer, and perplexity-based anomaly scoring work end to end. |
| |
| ### How it can be used |
| |
| - **Batch scan a log file** — score every line and export the anomalies: |
| `python detect_anomalies.py --file /var/log/auth.log` |
| - **Score individual lines** programmatically via `score_line(...)` and compare |
| against the saved `threshold` (see the usage snippet above). |
| - **Stream integration** — call the scorer inside a log shipper (Fluent Bit / |
| Logstash / a Kafka consumer) to tag or route high-perplexity lines in near real |
| time. |
| - **Alerting** — wire flagged lines into Slack/email/PagerDuty or a SIEM |
| (Splunk, Elastic) as an enrichment signal. |
| - **Retrain on your own logs** — point `LOG_FILE` at your data and run |
| `train_and_generate.py` to specialize the model to your environment. |
| |
| ## Future scope |
| |
| Natural next steps to make this research prototype more capable and production-ready: |
| |
| - **Per-source models & thresholds** — separate models for SSH, Apache, kernel, |
| etc., removing the single-global-threshold weakness. |
| - **Session / sequence-level modeling** — detect anomalous *sequences* of events |
| (e.g. login → privilege escalation → data access), not just single lines. |
| - **Larger context & scale** — bigger `d_model`, more layers, longer |
| `max_seq_len`, and training on full Loghub datasets (millions of lines). |
| - **Template-aware tokenization** — parse logs into templates + variables (à la |
| Drain) so IPs/IDs don't inflate the vocabulary or the anomaly score. |
| - **Calibrated scoring** — convert raw perplexity into probabilities/severity |
| levels, with per-line-position attribution ("which token was surprising?"). |
| - **Real-time serving** — expose the scorer as a REST/gRPC service or a Hugging |
| Face **Space** (Gradio demo) for interactive use. |
| - **`transformers` integration** — wrap as a `PreTrainedModel` so it loads via |
| `AutoModel.from_pretrained` and plugs into the standard ecosystem. |
| - **Evaluation & benchmarking** — measure precision/recall on labeled anomaly |
| datasets (e.g. BGL, HDFS) against baselines like DeepLog and LogBERT. |
| - **Explainability & feedback loop** — let analysts confirm/dismiss flags to |
| continuously refine the threshold and reduce false positives. |
| |
| ## Limitations |
| |
| - Trained on a **small, mixed** corpus (three log sources). A single global |
| threshold is imperfect across sources; per-source models/thresholds work better. |
| - Not production hardened — intended for education and experimentation. |
| - Small context window (128 tokens) and small vocab; it models line-level, not |
| session-level, structure. |
| |
| ## Attribution & licensing |
| |
| - Training data derived from the **[Loghub](https://github.com/logpai/loghub)** |
| collection (Linux, Apache, OpenSSH samples). Please cite Loghub and consult its |
| terms for any redistribution of the underlying logs. |
| - Model code released under the MIT license. |
| |
| ```bibtex |
| @inproceedings{zhu2023loghub, |
| title = {Loghub: A Large Collection of System Log |
| Datasets for AI-driven Log Analytics}, |
| author = {Zhu, Jieming and He, Shilin and He, Pinjia and |
| Liu, Jinyang and Lyu, Michael R.}, |
| booktitle = {IEEE International Symposium on Software |
| Reliability Engineering (ISSRE)}, |
| year = {2023} |
| } |
| ``` |
| |
