Publish SyzEncoder step-90000

README.md

@@ -0,0 +1,172 @@
+---
+language:
+- code
+license: bigcode-openrail-m
+library_name: transformers
+pipeline_tag: feature-extraction
+base_model: bigcode/starencoder
+tags:
+- syzkaller
+- syz-program
+- linux-kernel
+- code-encoder
+- masked-language-modeling
+---
+
+# SyzEncoder
+
+SyzEncoder is an encoder for programs generated by the
+[syzkaller](https://github.com/google/syzkaller) kernel fuzzer. It is based on
+[StarEncoder](https://huggingface.co/bigcode/starencoder) and was further
+pretrained with masked language modeling on 2,236,878 syz programs. The model
+is used by SyzPilot as the base encoder for online reachability classifiers and
+token-level attribution.
+
+This repository contains the checkpoint from training step 90,000. It had the
+lowest validation loss among the saved checkpoints. Only the encoder and its
+tokenizer are included; the masked language modeling head used during
+pretraining is not part of this release.
+
+## Intended use
+
+SyzEncoder is intended for representation learning and classification tasks on
+syz programs. Typical uses include:
+
+- initializing a reachability or coverage classifier;
+- extracting sequence or token representations for attribution;
+- studying machine-learning methods for kernel fuzzing.
+
+The model does not generate syz programs. It has not been evaluated as a
+general-purpose source-code or natural-language encoder.
+
+## Model details
+
+| Property | Value |
+| --- | --- |
+| Base model | `bigcode/starencoder` |
+| Architecture | BERT encoder |
+| Published parameters | 123,595,776 |
+| Hidden size | 768 |
+| Encoder layers | 12 |
+| Attention heads | 12 |
+| Maximum sequence length | 1,024 tokens |
+| Training objective | Masked language modeling |
+| Selected checkpoint | Step 90,000 |
+
+The checkpoint does not contain pooler weights. Downstream code should either
+load it with `add_pooling_layer=False` or train a task-specific pooling layer.
+SyzPilot uses attention-mask-aware mean pooling.
+
+## Tokenizer
+
+The repository includes SyzTokenizer, a byte-level BPE tokenizer trained on the
+same 2,236,878-program corpus. Its base vocabulary has 49,152 tokens, plus the
+`<mask>` token used during pretraining.
+
+Tokenizer selection included a grid search and a small masked-language-modeling
+comparison. During tokenizer training, long repeated character runs and bare
+hexadecimal runs were shortened before BPE learning, and token length was
+capped at 64 characters. Hexadecimal literals beginning with `0x` were left
+unchanged. These choices limit oversized tokens produced by raw byte dumps and
+repetitive payloads while retaining common syz syntax.
+
+## Training data
+
+The training corpus contains 2,236,878 syz programs collected from fuzzing
+Linux v6 kernels. Only program text was used for continued pretraining; kernel
+coverage records and downstream reachability labels were not used.
+
+The corpus was split into 90% training and 10% validation subsets. Validation
+loss was estimated on 200 batches at each checkpoint evaluation.
+
+## Training procedure
+
+StarEncoder was further pretrained for three epochs with a 15% masking rate.
+Masked positions followed the standard BERT policy: 80% were replaced by
+`<mask>`, 10% by a random token, and 10% were left unchanged.
+
+| Hyperparameter | Value |
+| --- | --- |
+| Batch size | 32 per GPU, 64 global |
+| Learning rate | 2e-5 |
+| Optimizer | AdamW, betas `(0.9, 0.98)` |
+| Weight decay | 0.01 |
+| Schedule | Cosine decay with 5% warmup |
+| Gradient clipping | 1.0 |
+| Sequence length | Up to 1,024 tokens, dynamic padding |
+| Hardware | 2 NVIDIA A800 80GB GPUs |
+| Training time | About 20.1 hours |
+
+The full run completed approximately 94,300 optimizer steps. This release uses
+the step-90,000 checkpoint because it produced the best sampled validation
+loss.
+
+## Evaluation
+
+| Checkpoint | Validation loss | MLM perplexity |
+| --- | ---: | ---: |
+| Initial StarEncoder | Not recorded | 2.76 |
+| SyzEncoder, step 90,000 | 0.7660 | 2.15 |
+
+The validation metric measures the masked-language-modeling objective on the
+held-out part of the pretraining corpus. It does not measure downstream
+reachability classification accuracy. Results on unrelated code corpora should
+not be inferred from these numbers.
+
+## Usage
+
+The example below obtains a mean-pooled representation while ignoring padding
+tokens:
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+model_id = "zzra1n/SyzEncoder"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModel.from_pretrained(model_id, add_pooling_layer=False)
+model.eval()
+
+program = """\
+r0 = socket$inet_tcp(0x2, 0x1, 0x0)
+connect$inet(r0, &(0x7f0000000000)={0x2, 0x0, @loopback}, 0x10)
+"""
+
+inputs = tokenizer(
+    program,
+    return_tensors="pt",
+    truncation=True,
+    max_length=1024,
+)
+
+with torch.inference_mode():
+    hidden = model(**inputs).last_hidden_state
+    mask = inputs["attention_mask"].unsqueeze(-1).to(hidden.dtype)
+    embedding = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1)
+
+print(embedding.shape)  # torch.Size([1, 768])
+```
+
+For supervised use, attach a classification head to the pooled representation
+and fine-tune it on labels from the target fuzzing task.
+
+## Limitations
+
+- The training data comes from one domain and kernel generation. Programs from
+  other syzkaller versions or substantially different syscall descriptions may
+  tokenize and embed differently.
+- Inputs longer than 1,024 tokens are truncated.
+- The released checkpoint has been selected using MLM validation loss. It does
+  not include a downstream classifier, calibrated probabilities, or a claim of
+  performance on a particular kernel bug.
+- Like its base model, SyzEncoder may retain unwanted behavior inherited from
+  its pretraining data. Outputs used for security decisions should be checked
+  against execution or coverage evidence.
+
+## License
+
+SyzEncoder is a derivative of StarEncoder and is released under the
+[BigCode OpenRAIL-M license](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).
+Users are responsible for reviewing and following the license terms and use
+restrictions.

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<mask>": 49152
+}

config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "/home/zhangzy/models/starencoder",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 1024,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_hidden_states": true,
+  "pad_token_id": 49152,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.44.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 49156
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84abf53b40d5d00278064c69ebcc552b1ee05c7877992b3b59a01c6935248559
+size 494405304

special_tokens_map.json

@@ -0,0 +1,52 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,195 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<commit_before>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<commit_msg>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<commit_after>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49152": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}