Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: safetensors
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
tags: [security, vulnerability-detection, cwe, owasp, code, moe, hobbylm]
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# CodeAstra-500M
|
| 9 |
+
|
| 10 |
+
A 500M-parameter sparse-MoE **code vulnerability detector**, fine-tuned from
|
| 11 |
+
[rootxhacker/HobbyLM-Chat](https://huggingface.co/rootxhacker/HobbyLM-Chat) on the full
|
| 12 |
+
[`ayshajavd/code-security-vulnerability-dataset`](https://huggingface.co/datasets/ayshajavd/code-security-vulnerability-dataset)
|
| 13 |
+
(140,335 train / 17,542 test rows, 100% coverage β nothing subsampled).
|
| 14 |
+
|
| 15 |
+
Given a code snippet it returns one JSON object: whether the code is vulnerable, the CWE, and the
|
| 16 |
+
OWASP Top-10 category.
|
| 17 |
+
|
| 18 |
+
## Prompt format
|
| 19 |
+
|
| 20 |
+
The model is format-sensitive β it was trained on exactly this layout (note the inner fence is part
|
| 21 |
+
of the prompt):
|
| 22 |
+
|
| 23 |
+
````text
|
| 24 |
+
SYSTEM: You are a source-code security auditor. Given a code snippet, decide whether it contains a security vulnerability and reply with one JSON object: {"vulnerable": bool, "cwe": str, "cwe_name": str, "owasp": str}. Use "none" for safe code.
|
| 25 |
+
USER: Audit the code below for security vulnerabilities.
|
| 26 |
+
Language: C
|
| 27 |
+
|
| 28 |
+
```c
|
| 29 |
+
<your code here>
|
| 30 |
+
```
|
| 31 |
+
ASSISTANT:
|
| 32 |
+
````
|
| 33 |
+
|
| 34 |
+
`hobbylm.security_data.detect_prompt(code, language)` builds this for you β use it rather than
|
| 35 |
+
hand-assembling the string.
|
| 36 |
+
|
| 37 |
+
It answers:
|
| 38 |
+
|
| 39 |
+
```json
|
| 40 |
+
{"vulnerable": true, "cwe": "CWE-89", "cwe_name": "Improper Neutralization of Special Elements used in an SQL Command (SQL Injection)", "owasp": "A03: Injection"}
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
## Usage
|
| 44 |
+
|
| 45 |
+
```python
|
| 46 |
+
import json, torch
|
| 47 |
+
from huggingface_hub import hf_hub_download
|
| 48 |
+
from safetensors.torch import load_file
|
| 49 |
+
from hobbylm.config import ModelConfig # github.com/<your>/moe-lab
|
| 50 |
+
from hobbylm.model import MoETransformer
|
| 51 |
+
from hobbylm.security_data import detect_prompt, VERDICT_PREFIX, TRUE_ID, FALSE_ID
|
| 52 |
+
import tiktoken
|
| 53 |
+
|
| 54 |
+
repo = "rootxhacker/codeastra-500M"
|
| 55 |
+
cfg_d = json.load(open(hf_hub_download(repo, "config.json")))
|
| 56 |
+
cfg_d.pop("preset", None)
|
| 57 |
+
model = MoETransformer(ModelConfig(**cfg_d)).cuda().eval()
|
| 58 |
+
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))
|
| 59 |
+
|
| 60 |
+
enc = tiktoken.get_encoding("gpt2")
|
| 61 |
+
prompt = detect_prompt(my_code, "C") + VERDICT_PREFIX # force the verdict position
|
| 62 |
+
ids = torch.tensor([enc.encode_ordinary(prompt)], device="cuda")
|
| 63 |
+
with torch.no_grad():
|
| 64 |
+
logits, _ = model(ids)
|
| 65 |
+
p_vuln = torch.softmax(logits[0, -1, [TRUE_ID, FALSE_ID]].float(), -1)[0].item()
|
| 66 |
+
print(p_vuln >= 0.3346) # see the threshold note below
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
Reading `p_vuln` in one forward pass is ~26x faster than generating the JSON, and it gives you a
|
| 70 |
+
tunable score instead of a fixed decision. Generating the full JSON works too if you want the CWE.
|
| 71 |
+
|
| 72 |
+
## Threshold β please read
|
| 73 |
+
|
| 74 |
+
**The default 0.5 is not the balanced operating point.** Pick deliberately:
|
| 75 |
+
|
| 76 |
+
| threshold | precision | recall | use case |
|
| 77 |
+
|---|---|---|---|
|
| 78 |
+
| 0.3346 | 75.57% | 76.89% | balanced (best F1 = 76.23%) |
|
| 79 |
+
| 0.3775 | 80.02% | 70.27% | |
|
| 80 |
+
| 0.4378 | 85.03% | 59.88% | |
|
| 81 |
+
| 0.5156 | 90.01% | 45.36% | CI gating |
|
| 82 |
+
| 0.6514 | 95.07% | 23.53% | high-confidence only |
|
| 83 |
+
|
| 84 |
+
At the naive 0.5 you get 88.79% precision but only 48.70% recall β a conservative detector.
|
| 85 |
+
|
| 86 |
+
## Evaluation
|
| 87 |
+
|
| 88 |
+
All numbers are from the held-out test split (17,542 rows, 1,887 vulnerable), scored by this repo's
|
| 89 |
+
own harness. Base model = un-finetuned HobbyLM-Chat, identical prompts.
|
| 90 |
+
|
| 91 |
+
| metric | HobbyLM-Chat | CodeAstra-500M |
|
| 92 |
+
|---|---|---|
|
| 93 |
+
| JSON parse rate | 1.20% | 99.98% |
|
| 94 |
+
| ROC AUC | β | 96.97% |
|
| 95 |
+
| average precision | β | 82.64% |
|
| 96 |
+
| best F1 | 0.00% | 76.23% |
|
| 97 |
+
| exact CWE (of 28 classes) | 0.00% | ~55% |
|
| 98 |
+
| OWASP category | 0.00% | ~74% |
|
| 99 |
+
|
| 100 |
+
### Paired evaluation β the number that matters
|
| 101 |
+
|
| 102 |
+
Standard splits of CVE-derived datasets are confounded: vulnerable functions come from large C
|
| 103 |
+
projects (kernel, Chromium, PHP) while "safe" ones are often unrelated code, so a model can score
|
| 104 |
+
well by recognising *style* rather than *flaws*. We therefore also evaluate PrimeVul-style, on 894
|
| 105 |
+
pairs of a vulnerable function and **its own patched version**:
|
| 106 |
+
|
| 107 |
+
| | first SFT pass | CodeAstra-500M |
|
| 108 |
+
|---|---|---|
|
| 109 |
+
| **P-C β flags flawed, clears patched** | 6.94% | **27.40%** |
|
| 110 |
+
| P-V β flags both (style shortcut) | 84.90% | **33.45%** |
|
| 111 |
+
| P-B β clears both | 6.71% | 36.02% |
|
| 112 |
+
| specificity on patched code | 13.65% | **63.42%** |
|
| 113 |
+
| within-pair ranking accuracy | 66.22% | 69.46% |
|
| 114 |
+
|
| 115 |
+
The published model was produced by adding the ~7,000 patched functions back as *safe* training
|
| 116 |
+
examples β minimal pairs that differ only by the fix. That quadrupled paired accuracy and raised
|
| 117 |
+
specificity on patched code from 13.65% to 63.42%.
|
| 118 |
+
|
| 119 |
+
**Be honest about what this means:** within-pair ranking accuracy only moved 66.22% β 69.46%, so most
|
| 120 |
+
of the gain is better calibration rather than deeper understanding. **27.40% paired accuracy is the
|
| 121 |
+
realistic estimate of true detection ability β not the 96.97% AUC.**
|
| 122 |
+
|
| 123 |
+
## Training
|
| 124 |
+
|
| 125 |
+
8xH100 on [Modal](https://modal.com). Main SFT: 6,000 steps, lr 2e-5, micro-batch 8, 176,216
|
| 126 |
+
examples (~27 min). Hard-negative pass: 500 steps, lr 5e-6 (~2.5 min). Loss masked to the completion;
|
| 127 |
+
MoE aux-free balancing bias frozen. Architecture is unchanged from HobbyLM: 768 hidden / 16 layers,
|
| 128 |
+
36 experts top-6 + 1 shared, GQA with per-head QK-norm, GPT-2 BPE, 2048 context.
|
| 129 |
+
|
| 130 |
+
## Limitations
|
| 131 |
+
|
| 132 |
+
- **Function-level only.** It sees one function; interprocedural and data-flow bugs are largely
|
| 133 |
+
invisible to it. Access-control flaws needing caller context are its weakest class.
|
| 134 |
+
- **92% of training data is C.** Treat other languages as out-of-distribution.
|
| 135 |
+
- **CWE labels confuse related classes.** CWE-89 and CWE-94 are strong (85%/82% exact); catch-all
|
| 136 |
+
buckets like CWE-399 and CWE-416 are weak. It often finds the bug and picks a sibling CWE.
|
| 137 |
+
- **Training labels are noisy.** Sources include mislabeled languages and prose spliced into code.
|
| 138 |
+
- **Code over 2048 tokens is head+tail truncated**, so the middle of long functions is unseen.
|
| 139 |
+
- **This is a 500M research model, not a security product.** It complements review and SAST; it does
|
| 140 |
+
not replace them. Do not gate a release on it alone.
|
| 141 |
+
|
| 142 |
+
## License
|
| 143 |
+
|
| 144 |
+
Apache-2.0.
|