Lev-3B

Lev-3B is a compact probabilistic decision model built on Granite-4.1-3B-Base. Given a state and questions about it, the model returns a probability distribution over caller-supplied alternatives for each question. It does not generate answer text: a trained pointer head scores the described alternatives.

This checkpoint was adapted for log analysis using anomaly-detection, ATT&CK parent-technique, and general-log training data. The reported evaluations cover anomaly detection and ATT&CK parent-technique classification.

Model details

Property	Details
Model type	Probabilistic decision model with a causal language-model backbone and a trained pointer head; returns probabilities over supplied alternatives without generating text.
Backbone	ibm-granite/granite-4.1-3b-base
Base revision	dacb9cb9157bec98e99b09f285c92a4d58405c96
Adapter	LoRA, rank 16, alpha 32, on attention and MLP projections; includes trained structural-token rows.
Readout	Trained pointer head over each question's alternatives.
Inference precision	BF16 backbone; FP32 adapters, structural-token rows, and pointer head.
Execution	Independent-row execution; saved state/branch/packed limits of 32768/65536/65536 tokens.
Temperature	Raw probabilities at T=1; optional fitted temperature T=1.8234449771164334, applied explicitly as described below.

These are execution limits, not a promise that every request of that size fits every GPU.

Model files and compatibility

• adapter_model.safetensors and adapter_config.json: LoRA adapters and trained structural-token rows.
• head.pt: pointer-head weights and backbone, tokenizer, execution, and dtype metadata. This file is required; the adapters alone are not the model.
• tokenizer.json and tokenizer_config.json: the trained tokenizer, including added delimiter tokens. Do not substitute the unmodified base tokenizer.
• temperature.json: the optional shared post-hoc temperature described below.

Base weights are not included. The Lev loader automatically loads the pinned Granite base from Hugging Face or the local cache, then restores the adapters and pointer head. Ordinary AutoModelForCausalLM or PEFT-only loading does not restore the pointer readout or the complete decision interface. Use the Lev runtime installed below; compatibility with upstream KEV alone is not guaranteed.

Installation

Use Linux x86-64, Python 3.12, an NVIDIA GPU, and uv.

git clone https://github.com/splunk/lev.git lev
cd lev
python3 scripts/setup_kev.py

Choose exactly one supported CUDA/PyTorch environment:

# CUDA 12.8 / PyTorch 2.8, with optimized kernels
uv sync --frozen --extra cuda128-torch28 --default-index https://pypi.org/simple

# Alternatively: CUDA 12.4 / PyTorch 2.6
# uv sync --frozen --extra cuda124-torch26 --default-index https://pypi.org/simple

Activate the installed environment before running the inference example:

source .venv/bin/activate

Inputs and outputs

A request contains a shared state and one or more named questions. Questions and alternatives are caller-defined.

Type	Alternatives	Probability-vector order
noul	Binary, with optional descriptions	false, then true
choice	1–255 named alternatives in criteria	Dictionary insertion order
score	2–255 ordered descriptions in criteria	List order

The state can be text or structured JSON. Objects and arrays are rendered as text; a JSON-encoded string is treated as text.

model.probs(encoded) returns one probability vector per question, in question insertion order. Each vector is normalized over that question's alternatives. For noul, the second probability is the probability of yes. For choice, the highest-probability alternative gives the selected answer. For score, the probabilities can be used to calculate the expected zero-based level. No answer text is generated.

Local inference

Download the model files and enter their directory:

hf download cisco-ai/lev-3b --local-dir ./lev-3b
cd lev-3b

If the model files are already downloaded, enter that directory instead. Keep the installed Lev environment active, then run the following Python example. The loader fetches or reuses the pinned Granite base automatically.

import json
from pathlib import Path

import torch
from kev.api import SystemOneRequest, to_record
from kev.evaluate import load

model_dir = Path.cwd()
tokenizer, model = load(str(model_dir), dev="cuda", dtype=torch.bfloat16)
model.head.temperature = 1.0  # raw probabilities

request = {
    "state": json.dumps([
        "INFO service started",
        "ERROR request failed after repeated retries",
    ]),
    "questions": {
        "is_anomalous": {
            "type": "noul",
            "instructions": "Is this log session anomalous?",
            "criteria": {
                "false": "Routine or successful normal system operation",
                "true": "Failure, error, or behavior inconsistent with normal operation",
            },
        },
    },
}
record, _ = to_record(SystemOneRequest.model_validate(request))
encoded = model.encode(tokenizer, record, strict=True)
probabilities = model.probs(encoded)[0]
print(dict(zip(["false", "true"], probabilities.tolist())))

# Optional: evaluate the same request using the supplied fitted temperature.
temperature = json.loads((model_dir / "temperature.json").read_text())["temperature"]
model.head.temperature = temperature
fitted_probabilities = model.probs(encoded)[0]
print(dict(zip(["false", "true"], fitted_probabilities.tolist())))

This example illustrates the interface. State serialization, question wording, candidate descriptions, and candidate order are part of the input and can affect results. Use the paper's prepared inputs for replication.

Training

The model first undergoes natural-language decision adaptation using Decision-v7 from Kev, which combines public classification datasets with generated policy and rule-following questions. It covers binary, categorical, and ordinal decisions, followed by additional adaptation on date-based policies and reasoning with missing evidence.

Joint log adaptation then combines:

• Anomaly-detection examples from BGL, Spirit, and Thunderbird.
• General-log questions covering summaries, semantic relationships, templates, build outcomes, fault diagnosis, and root causes, using public log benchmarks and generated examples.
• ATT&CK parent-technique classification examples from Splunk's attack-data repository, supplemented with GPT-5.5-generated examples.

See the paper for detailed data sources, composition, and preparation.

Evaluation

The following are the paper's seed-1 results measured directly on the full natural-prevalence test populations. Anomaly F1 is positive-class F1 in percent; AUROC and average precision (AP) are on [0, 1]. Binary decisions use the maximum-probability option (a 0.5 threshold; exact ties select normal).

Population	Records	Anomaly F1	AUROC	AP
BGL	18,592	99.36	0.9998	0.9998
Spirit	7,858	88.33	0.9756	0.9077
Thunderbird	20,587	88.20	0.9868	0.9210
HDFS, excluded anomaly source	115,013	43.32	0.7519	0.4116

ATT&CK population	Records	Accuracy (%)	Gold-present macro-F1 (%)
Represented sourcetypes	134	35.07	30.73
Withheld sourcetypes	178	42.13	29.53

These are single-checkpoint results, not averages across seeds or deployment guarantees. See the paper for the complete data definitions, evaluation details, and source terms.

Comparison with prompted and hosted models

The following tables reproduce the paper's anomaly and ATT&CK comparisons. Lev-3B is the released joint checkpoint; other systems are evaluated using the prompting conditions shown in each row.

Anomaly detection

BGL, Spirit, and Thunderbird (TB) are represented anomaly sources; HDFS is excluded from adaptation. F1 is anomaly-class F1 in percent; AUROC and AP are on [0, 1]. Asterisks mark results evaluated on all anomalous examples and a 10% subsample of benign examples, reweighted to represent the full test population. Dashes indicate unavailable continuous scores.

System	BGL F1	Spirit F1	TB F1	HDFS AUROC	HDFS AP	HDFS F1
Granite-4.1-3B-Base	0.19	0.00	0.44	0.6063	0.1579	12.21
Granite-3B instruct, prompted	63.59	16.36	2.41	—	—	14.93
Jev (0-shot)	57.67	18.12	3.12	0.6168	0.0537	5.65
Jev (5-shot)	57.68	21.16	6.55	0.7074	0.0958	6.54
Microsoft Decision-1 (0-shot)	60.03	21.98	3.10	0.7162	0.2376	14.29
Microsoft Decision-1 (5-shot)	61.35	22.14	4.84	0.7533	0.2408	15.07
OpenAI Decisions (0-shot)	59.07*	18.52*	2.45*	0.7967*	0.2072*	25.91*
OpenAI Decisions (5-shot)	71.80*	28.25*	5.21*	0.7861*	0.3233*	20.90*
GPT-5.6 Luna (0-shot)	58.71	16.25	3.75	—	—	21.92
GPT-5.6 Luna (5-shot)	68.64*	22.83*	4.86*	—	—	15.77*
GPT-5.6 Sol (0-shot)	58.17	19.32	4.33	—	—	17.05
GPT-5.6 Sol (5-shot)	71.73*	26.07*	9.68*	—	—	14.23*
Lev-3B	99.36	88.33	88.20	0.7519	0.4116	43.32

ATT&CK parent-technique classification

Every question offers 130 alternatives. All metrics are percentages; macro-F1 is computed over gold-present classes. Represented and withheld sourcetypes contain 134 and 178 test records, respectively.

System	Represented accuracy	Represented macro-F1	Withheld accuracy	Withheld macro-F1
Granite-4.1-3B-Base	0.00	0.00	0.00	0.00
Granite-3B instruct, prompted	3.0	0.1	5.1	1.6
Jev (0-shot)	24.6	20.6	32.6	28.1
Jev (5-shot)	29.9	23.2	34.3	25.6
Microsoft Decision-1 (0-shot)	22.4	17.3	28.1	23.9
Microsoft Decision-1 (5-shot)	25.4	19.5	24.7	20.3
OpenAI Decisions (0-shot)	28.4	24.1	38.2	28.1
OpenAI Decisions (5-shot)	32.8	25.2	41.6	26.6
GPT-5.6 Luna (0-shot)	38.8	31.5	51.7	42.7
GPT-5.6 Luna (5-shot)	38.1	30.2	53.9	38.6
GPT-5.6 Sol (0-shot)	42.5	36.9	51.7	45.4
GPT-5.6 Sol (5-shot)	45.5	36.1	56.2	49.4
Lev-3B	35.07	30.73	42.13	29.53

Temperature calibration

Raw probabilities use T=1. The supplied scalar T=1.8234449771164334 was fitted by pooled negative log-likelihood on 23,605 independent calibration predictions: 9,297 BGL, 3,977 Spirit, 10,293 Thunderbird, and 38 ATT&CK. The same scalar applies across the two tasks.

Fitted probabilities are softmax(logits / T). The external JSON is not applied automatically when loading this checkpoint. Set model.head.temperature explicitly as above. lev-evaluate intentionally generates raw probabilities; use its existing post-processing workflow for fitted probability-quality reports. Positive temperature scaling does not change the mathematical argmax and is not a fix for poor out-of-distribution discrimination.

Intended uses and limitations

This is a research model for bounded decisions over supplied evidence and described alternatives, not a general-purpose chat model or an autonomous remediation system. Questions and alternatives are caller-defined, not restricted to the tasks used during adaptation. Accuracy and calibration on other tasks must be evaluated separately. ATT&CK probabilities are conditional on the offered techniques; they are not probabilities that a session is malicious.

HDFS transfer remains weak despite high performance on represented anomaly sources. The model can miss anomalies with high confidence; the fitted temperature does not remove this failure mode. Labels are imperfect, source coverage is limited, and probabilities can be unreliable under distribution shift or changes in prompts and serialization. Validate locally and use human-reviewed action policies before operational use. Training-data sources remain subject to their own terms; raw training data are not included here.

License

The Lev adapter and pointer-head weights are licensed under the Apache License 2.0. The base model and training datasets remain subject to their respective licenses and terms.

Attribution

Authors:

• Leo Betthauser*
• Yingqi Liu*
• Uday Shah*
• Jishang Wei
• Sainyam Galhotra
• Liang Gou
• Vedant Dharnidharka
• Hao Yang

* Equal contribution, listed alphabetically by last name.

Backbone: IBM Granite. Decision-model implementation: Kev, with Lev integration changes required for this model. Lev is independent of TypeSafe AI and is not a release of Jev.

Links and citation

• Code: splunk/lev.
• Paper: Lev-3B: A Data Recipe for Compact Probabilistic Log Decision Models.

@misc{lev3b,
  title={Lev-3B: A Data Recipe for Compact Probabilistic Log Decision Models},
  author={Leo Betthauser and Yingqi Liu and Uday Shah and Jishang Wei and Sainyam Galhotra and Liang Gou and Vedant Dharnidharka and Hao Yang},
  year={2026},
  eprint={2610.38915},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.38915}
}