rootxhacker commited on
Commit
da9aa7c
Β·
verified Β·
1 Parent(s): a847202

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +144 -0
README.md ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: safetensors
4
+ pipeline_tag: text-generation
5
+ tags: [security, vulnerability-detection, cwe, owasp, code, moe, hobbylm]
6
+ ---
7
+
8
+ # CodeAstra-500M
9
+
10
+ A 500M-parameter sparse-MoE **code vulnerability detector**, fine-tuned from
11
+ [rootxhacker/HobbyLM-Chat](https://huggingface.co/rootxhacker/HobbyLM-Chat) on the full
12
+ [`ayshajavd/code-security-vulnerability-dataset`](https://huggingface.co/datasets/ayshajavd/code-security-vulnerability-dataset)
13
+ (140,335 train / 17,542 test rows, 100% coverage β€” nothing subsampled).
14
+
15
+ Given a code snippet it returns one JSON object: whether the code is vulnerable, the CWE, and the
16
+ OWASP Top-10 category.
17
+
18
+ ## Prompt format
19
+
20
+ The model is format-sensitive β€” it was trained on exactly this layout (note the inner fence is part
21
+ of the prompt):
22
+
23
+ ````text
24
+ SYSTEM: You are a source-code security auditor. Given a code snippet, decide whether it contains a security vulnerability and reply with one JSON object: {"vulnerable": bool, "cwe": str, "cwe_name": str, "owasp": str}. Use "none" for safe code.
25
+ USER: Audit the code below for security vulnerabilities.
26
+ Language: C
27
+
28
+ ```c
29
+ <your code here>
30
+ ```
31
+ ASSISTANT:
32
+ ````
33
+
34
+ `hobbylm.security_data.detect_prompt(code, language)` builds this for you β€” use it rather than
35
+ hand-assembling the string.
36
+
37
+ It answers:
38
+
39
+ ```json
40
+ {"vulnerable": true, "cwe": "CWE-89", "cwe_name": "Improper Neutralization of Special Elements used in an SQL Command (SQL Injection)", "owasp": "A03: Injection"}
41
+ ```
42
+
43
+ ## Usage
44
+
45
+ ```python
46
+ import json, torch
47
+ from huggingface_hub import hf_hub_download
48
+ from safetensors.torch import load_file
49
+ from hobbylm.config import ModelConfig # github.com/<your>/moe-lab
50
+ from hobbylm.model import MoETransformer
51
+ from hobbylm.security_data import detect_prompt, VERDICT_PREFIX, TRUE_ID, FALSE_ID
52
+ import tiktoken
53
+
54
+ repo = "rootxhacker/codeastra-500M"
55
+ cfg_d = json.load(open(hf_hub_download(repo, "config.json")))
56
+ cfg_d.pop("preset", None)
57
+ model = MoETransformer(ModelConfig(**cfg_d)).cuda().eval()
58
+ model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))
59
+
60
+ enc = tiktoken.get_encoding("gpt2")
61
+ prompt = detect_prompt(my_code, "C") + VERDICT_PREFIX # force the verdict position
62
+ ids = torch.tensor([enc.encode_ordinary(prompt)], device="cuda")
63
+ with torch.no_grad():
64
+ logits, _ = model(ids)
65
+ p_vuln = torch.softmax(logits[0, -1, [TRUE_ID, FALSE_ID]].float(), -1)[0].item()
66
+ print(p_vuln >= 0.3346) # see the threshold note below
67
+ ```
68
+
69
+ Reading `p_vuln` in one forward pass is ~26x faster than generating the JSON, and it gives you a
70
+ tunable score instead of a fixed decision. Generating the full JSON works too if you want the CWE.
71
+
72
+ ## Threshold β€” please read
73
+
74
+ **The default 0.5 is not the balanced operating point.** Pick deliberately:
75
+
76
+ | threshold | precision | recall | use case |
77
+ |---|---|---|---|
78
+ | 0.3346 | 75.57% | 76.89% | balanced (best F1 = 76.23%) |
79
+ | 0.3775 | 80.02% | 70.27% | |
80
+ | 0.4378 | 85.03% | 59.88% | |
81
+ | 0.5156 | 90.01% | 45.36% | CI gating |
82
+ | 0.6514 | 95.07% | 23.53% | high-confidence only |
83
+
84
+ At the naive 0.5 you get 88.79% precision but only 48.70% recall β€” a conservative detector.
85
+
86
+ ## Evaluation
87
+
88
+ All numbers are from the held-out test split (17,542 rows, 1,887 vulnerable), scored by this repo's
89
+ own harness. Base model = un-finetuned HobbyLM-Chat, identical prompts.
90
+
91
+ | metric | HobbyLM-Chat | CodeAstra-500M |
92
+ |---|---|---|
93
+ | JSON parse rate | 1.20% | 99.98% |
94
+ | ROC AUC | β€” | 96.97% |
95
+ | average precision | β€” | 82.64% |
96
+ | best F1 | 0.00% | 76.23% |
97
+ | exact CWE (of 28 classes) | 0.00% | ~55% |
98
+ | OWASP category | 0.00% | ~74% |
99
+
100
+ ### Paired evaluation β€” the number that matters
101
+
102
+ Standard splits of CVE-derived datasets are confounded: vulnerable functions come from large C
103
+ projects (kernel, Chromium, PHP) while "safe" ones are often unrelated code, so a model can score
104
+ well by recognising *style* rather than *flaws*. We therefore also evaluate PrimeVul-style, on 894
105
+ pairs of a vulnerable function and **its own patched version**:
106
+
107
+ | | first SFT pass | CodeAstra-500M |
108
+ |---|---|---|
109
+ | **P-C β€” flags flawed, clears patched** | 6.94% | **27.40%** |
110
+ | P-V β€” flags both (style shortcut) | 84.90% | **33.45%** |
111
+ | P-B β€” clears both | 6.71% | 36.02% |
112
+ | specificity on patched code | 13.65% | **63.42%** |
113
+ | within-pair ranking accuracy | 66.22% | 69.46% |
114
+
115
+ The published model was produced by adding the ~7,000 patched functions back as *safe* training
116
+ examples β€” minimal pairs that differ only by the fix. That quadrupled paired accuracy and raised
117
+ specificity on patched code from 13.65% to 63.42%.
118
+
119
+ **Be honest about what this means:** within-pair ranking accuracy only moved 66.22% β†’ 69.46%, so most
120
+ of the gain is better calibration rather than deeper understanding. **27.40% paired accuracy is the
121
+ realistic estimate of true detection ability β€” not the 96.97% AUC.**
122
+
123
+ ## Training
124
+
125
+ 8xH100 on [Modal](https://modal.com). Main SFT: 6,000 steps, lr 2e-5, micro-batch 8, 176,216
126
+ examples (~27 min). Hard-negative pass: 500 steps, lr 5e-6 (~2.5 min). Loss masked to the completion;
127
+ MoE aux-free balancing bias frozen. Architecture is unchanged from HobbyLM: 768 hidden / 16 layers,
128
+ 36 experts top-6 + 1 shared, GQA with per-head QK-norm, GPT-2 BPE, 2048 context.
129
+
130
+ ## Limitations
131
+
132
+ - **Function-level only.** It sees one function; interprocedural and data-flow bugs are largely
133
+ invisible to it. Access-control flaws needing caller context are its weakest class.
134
+ - **92% of training data is C.** Treat other languages as out-of-distribution.
135
+ - **CWE labels confuse related classes.** CWE-89 and CWE-94 are strong (85%/82% exact); catch-all
136
+ buckets like CWE-399 and CWE-416 are weak. It often finds the bug and picks a sibling CWE.
137
+ - **Training labels are noisy.** Sources include mislabeled languages and prose spliced into code.
138
+ - **Code over 2048 tokens is head+tail truncated**, so the middle of long functions is unseen.
139
+ - **This is a 500M research model, not a security product.** It complements review and SAST; it does
140
+ not replace them. Do not gate a release on it alone.
141
+
142
+ ## License
143
+
144
+ Apache-2.0.