---
language:
- en
base_model:
- google-bert/bert-base-uncased
pipeline_tag: token-classification
library_name: transformers
tags:
- token-classification
- named-entity-recognition
- ner
- bert
- logs
datasets:
- Aliph0th/logtheus-ml-ds
---
# Log Entity Extractor (BERT-based Token Classifier)

A fine-tuned BERT model for extracting canonical attributes from log lines using token classification (NER-style task). Created for my course work

## Model Description

This model is based on `bert-base-uncased` and trained to perform **token classification** on log messages. It extracts structured attributes (service, level, event, error_code, user_id, ip, etc.) from unstructured log text using a BIO tagging scheme.

**Use case:** Convert raw log lines into canonical, structured key-value pairs for downstream analysis, alerting, or aggregation.

## Model Details

- **Base Model:** `bert-large-uncased`
- **Task:** Token Classification (Named Entity Recognition)
- **Training Data:** Annotated log lines (character-level entity offsets)
- **Input:** Raw log text (string)
- **Output:** Per-token BIO labels → grouped entities as canonical attributes

## Canonical Label Set

The model extracts attributes from these canonical fields:

| Field | Description |
|-------|-------------|
| service | Application or service name (e.g., "auth", "api") |
| level | Log level (e.g., "info", "error", "warn") |
| timestamp | Timestamp or date reference |
| environment | Deployment environment (e.g., "prod", "staging") |
| event | Event type or action (e.g., "login", "request") |
| error_message | Human-readable error message |
| status_code | HTTP or service status code |
| duration | Duration |
| ip | IP address (client or server) |
| method | HTTP method (GET, POST, etc.) |
| path | URL path or resource path |
| useragent | User-Agent header |
| hostname | Server hostname |

## Usage

### Installation

```bash
pip install transformers torch
```

### Python (Hugging Face Transformers)

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch
model_name = "Aliph0th/logtheus-ml"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
text = "[auth] failed login for user 123 from 10.1.2.3 code=E401"
# Tokenize and forward pass
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
outputs = model(**inputs)
logits = outputs.logits
# Get predicted label IDs
predicted_ids = torch.argmax(logits, dim=-1)
# Map back to label names
id2label = model.config.id2label
predictions = [[id2label[int(p)] for p in pred] for pred in predicted_ids]
print(predictions)
```

This returns a structured JSON object with:
- `attributes`: High-confidence extractions (dict of canonical_field → value)
- `low_confidence_attributes`: Below-threshold extractions
- `attribute_confidence`: Per-field confidence scores
- `message`: Original log text
- `confidence`: Overall prediction confidence (0-1)
- `model_version`: Model version string

## Training

### Dataset Format

Training data in JSONL format with character-offset annotations:

```json
{"id":"1","text":"[auth] failed login for user 123 from 10.1.2.3","entities":[{"start":1,"end":5,"label":"service"},{"start":28,"end":32,"label":"user_id"},{"start":38,"end":46,"label":"ip"}]}
```

Used dataset -[Aliph0th/logtheus-ml-ds](https://huggingface.co/datasets/Aliph0th/logtheus-ml-ds)

**Fields:**
- `text`: Raw log line (string)
- `entities`: List of entity annotations
  - `start`, `end`: Character-level offsets in text (0-indexed)
  - `label`: Canonical field name

### Training Procedure

```bash
# 1. Prepare raw log files (deduplicate, split train/val)
python scripts/process_data.py data/annotated/ --p 0.8
# 2. Train model
python training/train_token_classifier.py \
  --train-file data/train.jsonl \
  --val-file data/val.jsonl \
  --output-dir artifacts/model_v1 \
  --base-model bert-base-uncased \
  --epochs 5 \
  --batch-size 16
```

**Hyperparameters:**
- Learning rate: 3e-5
- Batch size: 16 (per device)
- Epochs: 5 (with early stopping by F1)
- Optimizer: AdamW
- Weight decay: 0.01

## Limitations

- **English logs only:** Trained on ASCII/UTF-8 log text in English
- **Format dependency:** Works best on semi-structured logs (key=value, JSON, or naturally worded messages); highly custom formats may need domain-specific preprocessing

## Contact & Support

For issues, questions, or contributions, please visit:
- **Repository:** https://github.com/Aliph0th/logtheus-ml
- **Issues:** https://github.com/Aliph0th/logtheus-ml/issues

## Acknowledgments

- Based on [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)
- Built with [Hugging Face Transformers](https://huggingface.co/transformers/)