File size: 5,458 Bytes
b82ae41 7697a28 b82ae41 6c561b9 b82ae41 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
license: mit
language:
- en
base_model: microsoft/codebert-base
pipeline_tag: text-classification
tags:
- code
- solidity
- smart-contracts
- security
- vulnerability-detection
widget:
- text: "function withdraw(uint amount) public { msg.sender.call.value(amount)(\"\"); balances[msg.sender] -= amount; }"
example_title: Reentrancy
- text: "function forward(address target, bytes data) public { target.delegatecall(data); }"
example_title: Delegatecall
- text: "function play() public { if (block.timestamp % 2 == 0) { winner = msg.sender; } }"
example_title: Timestamp
---
# CodeBERT — Solidity Vulnerability Classifier (4-class)
Fine-tuned [`microsoft/codebert-base`](https://huggingface.co/microsoft/codebert-base)
that classifies a Solidity snippet into one of four vulnerability types:
**Reentrancy**, **Integer Overflow**, **Timestamp Dependency**, or
**Dangerous Delegatecall**.
Reproduces the approach of Hossain, Altarawneh & Roberts, ["Leveraging LLMs and
ML for Smart Contract Vulnerability Detection"](https://arxiv.org/abs/2501.02229),
IEEE CCWC 2025, on a smaller public dataset.
**Code, training pipeline and full evaluation:** [github.com/Riicko-19/smart-contract-vuln-detector](https://github.com/Riicko-19/smart-contract-vuln-detector)
— including a 5-seed variance study, a label-contamination analysis, and an
adversarial out-of-distribution probe.
## Usage
```python
from transformers import pipeline
clf = pipeline("text-classification", model="AbijithwearsHUGGIES/codebert-smart-contract-vuln")
clf("function withdraw(uint a) public { msg.sender.call.value(a)(\"\"); balances[msg.sender] -= a; }")
# [{'label': 'Reentrancy', 'score': ...}]
```
## Results
This checkpoint, on the held-out 59-contract test split:
| Metric | Value |
|---|---|
| Accuracy | 0.881 |
| Macro F1 | 0.871 |
| Reentrancy F1 | 0.957 |
| Dangerous Delegatecall F1 | 1.000 |
| Timestamp Dependency F1 | 0.897 |
| Integer Overflow F1 | 0.632 |
**Read those with care.** The test split has only 59 contracts, so one flipped
prediction moves a per-class F1 by roughly 0.09. Measured across 5 seeds, this
architecture averages:
| Model | Accuracy | Macro F1 | Reentrancy | Overflow |
|---|---|---|---|---|
| CodeBERT | 0.868 ±0.037 | 0.858 ±0.054 | 0.926 ±0.048 | 0.624 ±0.170 |
| DistilBERT | 0.834 ±0.071 | 0.826 ±0.061 | 0.949 ±0.018 | 0.508 ±0.168 |
A paired per-seed comparison found **no statistically significant difference**
between CodeBERT and DistilBERT (accuracy p=0.41, macro-F1 p=0.47, Integer
Overflow p=0.38). This checkpoint is simply the best single artifact from that
sweep, not evidence that CodeBERT is the better architecture here.
## Training
- 12 epochs, class-weighted cross-entropy (the dataset is imbalanced)
- Best checkpoint selected on **validation macro-F1**, not `eval_loss` —
selecting on loss lets the majority class dominate and yields 0.00 F1 on
Integer Overflow, because the "best" checkpoint abandons the minority class
- Max sequence length 512 (no truncation occurs; longest contract is 248 tokens)
## Data
387 labelled contracts from the
[Messi-Q/Smart-Contract-Dataset](https://github.com/Messi-Q/Smart-Contract-Dataset)
"Resource 2" release — Timestamp Dependency 174, Integer Overflow 80,
Reentrancy 71, Dangerous Delegatecall 62. Split 270 train / 58 val / 59 test.
This is roughly a sixth the size of the paper's 2,217-contract IR-Fuzz split
(which is not publicly redistributable), so treat these numbers as directional
rather than a reproduction.
## Limitations
- **Integer Overflow is weak** (F1 ~0.62, ±0.17 across seeds), and the cause is
the label, not the model. Across all 387 contracts, the other three classes
have a near-perfect syntactic signature (Reentrancy 100% contain
`.call.value(`, Timestamp 100% contain `block.timestamp`/`now`, Delegatecall
89% contain `.delegatecall(`). Integer Overflow has none of its own: 39% of
its contracts contain the reentrancy pattern and 55% contain timestamp calls.
The source dataset ships four *independent binary* labelled sets, and forcing
a single label onto contracts that exhibit several vulnerabilities pushes the
ambiguity into this class. Treating the task as multi-label would be the
principled fix.
- **Forced-choice, not detection.** The model always returns one of four
classes. It cannot say "no vulnerability", and it was trained only on
vulnerable contracts. An out-of-distribution probe makes the cost concrete: a
*pure math library* with no state, no external calls and no timestamps is
classified Timestamp Dependency at **99.1%** confidence, and Python source
code scores Integer Overflow at 93.7%. Confidence does not help — on benign
inputs it reaches 99.1%, while genuine vulnerabilities go as low as 97.9%, so
the ranges overlap and no threshold separates them. Never use this to decide
whether a contract is safe.
- **It does generalise on genuinely vulnerable code**, which is the flip side:
it correctly flags reentrancy written with modern `.call{value:}` syntax, and
even reentrancy expressed through a callback with no low-level call at all —
so it is not merely keyword matching.
- **Not a substitute for an audit.** This is a research/portfolio artifact
trained on 387 contracts, not a security tool.
## License
MIT
|