AbijithwearsHUGGIES/codebert-smart-contract-vuln

CodeBERT — Solidity Vulnerability Classifier (4-class)

Fine-tuned microsoft/codebert-base that classifies a Solidity snippet into one of four vulnerability types: Reentrancy, Integer Overflow, Timestamp Dependency, or Dangerous Delegatecall.

Reproduces the approach of Hossain, Altarawneh & Roberts, "Leveraging LLMs and ML for Smart Contract Vulnerability Detection", IEEE CCWC 2025, on a smaller public dataset.

Code, training pipeline and full evaluation: github.com/Riicko-19/smart-contract-vuln-detector — including a 5-seed variance study, a label-contamination analysis, and an adversarial out-of-distribution probe.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="AbijithwearsHUGGIES/codebert-smart-contract-vuln")
clf("function withdraw(uint a) public { msg.sender.call.value(a)(\"\"); balances[msg.sender] -= a; }")
# [{'label': 'Reentrancy', 'score': ...}]

Results

This checkpoint, on the held-out 59-contract test split:

Metric	Value
Accuracy	0.881
Macro F1	0.871
Reentrancy F1	0.957
Dangerous Delegatecall F1	1.000
Timestamp Dependency F1	0.897
Integer Overflow F1	0.632

Read those with care. The test split has only 59 contracts, so one flipped prediction moves a per-class F1 by roughly 0.09. Measured across 5 seeds, this architecture averages:

Model	Accuracy	Macro F1	Reentrancy	Overflow
CodeBERT	0.868 ±0.037	0.858 ±0.054	0.926 ±0.048	0.624 ±0.170
DistilBERT	0.834 ±0.071	0.826 ±0.061	0.949 ±0.018	0.508 ±0.168

A paired per-seed comparison found no statistically significant difference between CodeBERT and DistilBERT (accuracy p=0.41, macro-F1 p=0.47, Integer Overflow p=0.38). This checkpoint is simply the best single artifact from that sweep, not evidence that CodeBERT is the better architecture here.

Training

• 12 epochs, class-weighted cross-entropy (the dataset is imbalanced)
• Best checkpoint selected on validation macro-F1, not eval_loss — selecting on loss lets the majority class dominate and yields 0.00 F1 on Integer Overflow, because the "best" checkpoint abandons the minority class
• Max sequence length 512 (no truncation occurs; longest contract is 248 tokens)

Data

387 labelled contracts from the Messi-Q/Smart-Contract-Dataset "Resource 2" release — Timestamp Dependency 174, Integer Overflow 80, Reentrancy 71, Dangerous Delegatecall 62. Split 270 train / 58 val / 59 test.

This is roughly a sixth the size of the paper's 2,217-contract IR-Fuzz split (which is not publicly redistributable), so treat these numbers as directional rather than a reproduction.

Limitations

• Integer Overflow is weak (F1 ~0.62, ±0.17 across seeds), and the cause is the label, not the model. Across all 387 contracts, the other three classes have a near-perfect syntactic signature (Reentrancy 100% contain .call.value(, Timestamp 100% contain block.timestamp/now, Delegatecall 89% contain .delegatecall(). Integer Overflow has none of its own: 39% of its contracts contain the reentrancy pattern and 55% contain timestamp calls. The source dataset ships four independent binary labelled sets, and forcing a single label onto contracts that exhibit several vulnerabilities pushes the ambiguity into this class. Treating the task as multi-label would be the principled fix.
• Forced-choice, not detection. The model always returns one of four classes. It cannot say "no vulnerability", and it was trained only on vulnerable contracts. An out-of-distribution probe makes the cost concrete: a pure math library with no state, no external calls and no timestamps is classified Timestamp Dependency at 99.1% confidence, and Python source code scores Integer Overflow at 93.7%. Confidence does not help — on benign inputs it reaches 99.1%, while genuine vulnerabilities go as low as 97.9%, so the ranges overlap and no threshold separates them. Never use this to decide whether a contract is safe.
• It does generalise on genuinely vulnerable code, which is the flip side: it correctly flags reentrancy written with modern .call{value:} syntax, and even reentrancy expressed through a callback with no low-level call at all — so it is not merely keyword matching.
• Not a substitute for an audit. This is a research/portfolio artifact trained on 387 contracts, not a security tool.

License

MIT