README.md

---
language:
- code
license: bigcode-openrail-m
library_name: transformers
pipeline_tag: feature-extraction
base_model: bigcode/starencoder
tags:
- syzkaller
- syz-program
- linux-kernel
- code-encoder
- masked-language-modeling
---

# SyzEncoder

SyzEncoder is an encoder for programs generated by the
[syzkaller](https://github.com/google/syzkaller) kernel fuzzer. It is based on
[StarEncoder](https://huggingface.co/bigcode/starencoder) and was further
pretrained with masked language modeling on 2,236,878 syz programs. The model
is used by SyzPilot as the base encoder for online reachability classifiers and
token-level attribution.

This repository contains the checkpoint from training step 90,000. It had the
lowest validation loss among the saved checkpoints. Only the encoder and its
tokenizer are included; the masked language modeling head used during
pretraining is not part of this release.

## Intended use

SyzEncoder is intended for representation learning and classification tasks on
syz programs. Typical uses include:

- initializing a reachability or coverage classifier;
- extracting sequence or token representations for attribution;
- studying machine-learning methods for kernel fuzzing.

The model does not generate syz programs. It has not been evaluated as a
general-purpose source-code or natural-language encoder.

## Model details

| Property | Value |
| --- | --- |
| Base model | `bigcode/starencoder` |
| Architecture | BERT encoder |
| Published parameters | 123,595,776 |
| Hidden size | 768 |
| Encoder layers | 12 |
| Attention heads | 12 |
| Maximum sequence length | 1,024 tokens |
| Training objective | Masked language modeling |
| Selected checkpoint | Step 90,000 |

The checkpoint does not contain pooler weights. Downstream code should either
load it with `add_pooling_layer=False` or train a task-specific pooling layer.
SyzPilot uses attention-mask-aware mean pooling.

## Tokenizer

The repository includes SyzTokenizer, a byte-level BPE tokenizer trained on the
same 2,236,878-program corpus. Its base vocabulary has 49,152 tokens, plus the
`<mask>` token used during pretraining.

Tokenizer selection included a grid search and a small masked-language-modeling
comparison. During tokenizer training, long repeated character runs and bare
hexadecimal runs were shortened before BPE learning, and token length was
capped at 64 characters. Hexadecimal literals beginning with `0x` were left
unchanged. These choices limit oversized tokens produced by raw byte dumps and
repetitive payloads while retaining common syz syntax.

## Training data

The training corpus contains 2,236,878 syz programs collected from fuzzing
Linux v6 kernels. Only program text was used for continued pretraining; kernel
coverage records and downstream reachability labels were not used.

The corpus was split into 90% training and 10% validation subsets. Validation
loss was estimated on 200 batches at each checkpoint evaluation.

## Training procedure

StarEncoder was further pretrained for three epochs with a 15% masking rate.
Masked positions followed the standard BERT policy: 80% were replaced by
`<mask>`, 10% by a random token, and 10% were left unchanged.

| Hyperparameter | Value |
| --- | --- |
| Batch size | 32 per GPU, 64 global |
| Learning rate | 2e-5 |
| Optimizer | AdamW, betas `(0.9, 0.98)` |
| Weight decay | 0.01 |
| Schedule | Cosine decay with 5% warmup |
| Gradient clipping | 1.0 |
| Sequence length | Up to 1,024 tokens, dynamic padding |
| Hardware | 2 NVIDIA A800 80GB GPUs |
| Training time | About 20.1 hours |

The full run completed approximately 94,300 optimizer steps. This release uses
the step-90,000 checkpoint because it produced the best sampled validation
loss.

## Evaluation

| Checkpoint | Validation loss | MLM perplexity |
| --- | ---: | ---: |
| Initial StarEncoder | Not recorded | 2.76 |
| SyzEncoder, step 90,000 | 0.7660 | 2.15 |

The validation metric measures the masked-language-modeling objective on the
held-out part of the pretraining corpus. It does not measure downstream
reachability classification accuracy. Results on unrelated code corpora should
not be inferred from these numbers.

## Usage

The example below obtains a mean-pooled representation while ignoring padding
tokens:

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_id = "zzra1n/SyzEncoder"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, add_pooling_layer=False)
model.eval()

program = """\
r0 = socket$inet_tcp(0x2, 0x1, 0x0)
connect$inet(r0, &(0x7f0000000000)={0x2, 0x0, @loopback}, 0x10)
"""

inputs = tokenizer(
    program,
    return_tensors="pt",
    truncation=True,
    max_length=1024,
)

with torch.inference_mode():
    hidden = model(**inputs).last_hidden_state
    mask = inputs["attention_mask"].unsqueeze(-1).to(hidden.dtype)
    embedding = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1)

print(embedding.shape)  # torch.Size([1, 768])
```

For supervised use, attach a classification head to the pooled representation
and fine-tune it on labels from the target fuzzing task.

## Limitations

- The training data comes from one domain and kernel generation. Programs from
  other syzkaller versions or substantially different syscall descriptions may
  tokenize and embed differently.
- Inputs longer than 1,024 tokens are truncated.
- The released checkpoint has been selected using MLM validation loss. It does
  not include a downstream classifier, calibrated probabilities, or a claim of
  performance on a particular kernel bug.
- Like its base model, SyzEncoder may retain unwanted behavior inherited from
  its pretraining data. Outputs used for security decisions should be checked
  against execution or coverage evidence.

## License

SyzEncoder is a derivative of StarEncoder and is released under the
[BigCode OpenRAIL-M license](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).
Users are responsible for reviewing and following the license terms and use
restrictions.