Hugging Face's logo

spidercob
/
code-risk-classifier

Text Classification
setfit
Safetensors
sentence-transformers
English
bert
code-security
dlp
secret-detection
vulnerability-detection
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community
Sumeetgpt commited on
Commit
cec7e9f
·
verified ·
1 Parent(s): be06a7d

Update model card for v2 — expanded 22-repo multi-language training data

Browse files
Files changed (1) hide show
  1. README.md +244 -56
README.md CHANGED
@@ -3,88 +3,276 @@ tags:
3
  - setfit
4
  - sentence-transformers
5
  - text-classification
6
- - generated_from_setfit_trainer
7
- widget: []
 
 
 
 
 
 
 
 
 
 
 
8
  metrics:
9
  - accuracy
10
  pipeline_tag: text-classification
11
  library_name: setfit
12
  inference: true
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
 
15
- # SetFit
16
 
17
- This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
18
 
19
- The model has been trained using an efficient few-shot learning technique that involves:
20
 
21
- 1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
22
- 2. Training a classification head with features from the fine-tuned Sentence Transformer.
23
 
24
- ## Model Details
 
 
 
 
 
25
 
26
- ### Model Description
27
- - **Model Type:** SetFit
28
- <!-- - **Sentence Transformer:** [Unknown](https://huggingface.co/unknown) -->
29
- - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
30
- - **Maximum Sequence Length:** 256 tokens
31
- - **Number of Classes:** 4 classes
32
- <!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
33
- <!-- - **Language:** Unknown -->
34
- <!-- - **License:** Unknown -->
35
 
36
- ### Model Sources
37
-
38
- - **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
39
- - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
40
- - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
41
-
42
- ## Uses
43
 
44
- ### Direct Use for Inference
45
 
46
- First install the SetFit library:
 
 
 
 
 
47
 
48
- ```bash
49
- pip install setfit
 
50
  ```
51
 
52
- Then you can load this model and run inference.
53
 
54
- ```python
55
- from setfit import SetFitModel
56
 
57
- # Download from the 🤗 Hub
58
- model = SetFitModel.from_pretrained("setfit_model_id")
59
- # Run inference
60
- preds = model("I loved the spiderman movie!")
61
  ```
62
 
63
- <!--
64
- ### Downstream Use
65
-
66
- *List how someone could finetune this model on their own dataset.*
67
- -->
68
-
69
- <!--
70
- ### Out-of-Scope Use
71
-
72
- *List how the model may foreseeably be misused and address what users ought not to do with the model.*
73
- -->
74
-
75
- <!--
76
- ## Bias, Risks and Limitations
77
 
78
- *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
79
- -->
80
 
81
- <!--
82
- ### Recommendations
83
 
84
- *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
85
- -->
 
 
86
 
87
- ## Training Details
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
 
89
  ### Framework Versions
90
  - Python: 3.12.12
 
3
  - setfit
4
  - sentence-transformers
5
  - text-classification
6
+ - code-security
7
+ - dlp
8
+ - secret-detection
9
+ - vulnerability-detection
10
+ language:
11
+ - en
12
+ license: apache-2.0
13
+ widget:
14
+ - text: "Analyze this hardcoded_secret: AWS_ACCESS_KEY_ID=AKIA4REALKEY123ABC committed to main branch .env file"
15
+ - text: "Analyze this vulnerable_pattern: $query = 'SELECT * FROM users WHERE id=' . $_GET['id'];"
16
+ - text: "Analyze this test_fixture: factory_boy default: user.password = 'testpass123' for pytest fixtures"
17
+ - text: "Analyze this clean_code: def get_user(db: Session, user_id: int): return db.query(User).filter(User.id == user_id).first()"
18
+ - text: "Analyze this secure_implementation: password_hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=12))"
19
  metrics:
20
  - accuracy
21
  pipeline_tag: text-classification
22
  library_name: setfit
23
  inference: true
24
+ model-index:
25
+ - name: spidercob/code-risk-classifier
26
+ results:
27
+ - task:
28
+ type: text-classification
29
+ name: Text Classification
30
+ dataset:
31
+ name: curated-public-repos-v2
32
+ type: custom
33
+ split: test
34
+ metrics:
35
+ - type: accuracy
36
+ value: 1.0
37
+ name: Accuracy
38
  ---
39
 
40
+ # spidercob/code-risk-classifier
41
 
42
+ A [SetFit](https://github.com/huggingface/setfit) model that classifies code snippets and secrets into four risk categories. Used inside [Spidercob](https://spidercob.com) to reduce false positives in supply-chain and secret scanning — distinguishing real production risks from test fixtures and safe code.
43
 
44
+ **v2** — expanded multi-language training corpus (22 public repos, Python / Java / PHP / JavaScript / TypeScript / Ruby / Go).
45
 
46
+ ## Labels
 
47
 
48
+ | Label | Meaning | Action |
49
+ |---|---|---|
50
+ | `REAL_SECRET` | Hardcoded credential, API key, or token committed to source | **BLOCK** |
51
+ | `VULNERABLE_LOGIC` | SQL injection, XSS, unsafe deserialization, command injection, etc. | **BLOCK** |
52
+ | `TEST_MOCK` | Dummy credential or vuln pattern inside a test fixture or factory | ALLOW |
53
+ | `SAFE_CODE` | Clean production code, secure implementation pattern | ALLOW |
54
 
55
+ ## Quick Start
 
 
 
 
 
 
 
 
56
 
57
+ ```python
58
+ from setfit import SetFitModel
 
 
 
 
 
59
 
60
+ model = SetFitModel.from_pretrained("spidercob/code-risk-classifier")
61
 
62
+ snippets = [
63
+ "Analyze this hardcoded_secret: AWS_ACCESS_KEY_ID=AKIA4REALKEY123ABC in .env",
64
+ "Analyze this vulnerable_pattern: $q = 'SELECT * FROM users WHERE id=' . $_GET['id'];",
65
+ "Analyze this test_fixture: user.password = 'testpass123' # factory_boy default",
66
+ "Analyze this clean_code: password_hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt())",
67
+ ]
68
 
69
+ predictions = model.predict(snippets)
70
+ probabilities = model.predict_proba(snippets)
71
+ # predictions: ['REAL_SECRET', 'VULNERABLE_LOGIC', 'TEST_MOCK', 'SAFE_CODE']
72
  ```
73
 
74
+ ## Input Format
75
 
76
+ Inputs must follow the pattern used during training:
 
77
 
78
+ ```
79
+ Analyze this <issue_type>: <code_snippet_or_context>
 
 
80
  ```
81
 
82
+ Where `<issue_type>` is a short descriptor such as `hardcoded_secret`, `vulnerable_pattern`, `test_fixture`, `clean_code`, `secure_implementation`, etc.
 
 
 
 
 
 
 
 
 
 
 
 
 
83
 
84
+ ## Integration with Spidercob DLP
 
85
 
86
+ ```python
87
+ from app.core.code_risk_classifier import CodeRiskClassifier
88
 
89
+ clf = CodeRiskClassifier()
90
+ result = clf.classify("hardcoded_secret", "STRIPE_SECRET_KEY = 'sk_live_abc123xyz'")
91
+ # result: {"risk_type": "REAL_SECRET", "confidence": 0.99, "action": "BLOCK", "severity": "CRITICAL"}
92
+ ```
93
 
94
+ The classifier loads the fine-tuned SetFit model at import time and falls back to zero-shot cosine similarity if the model directory is absent.
95
+
96
+ ## Evaluation
97
+
98
+ | Label | Accuracy |
99
+ |:--------|:---------|
100
+ | **all** | **1.0** (100%, 176 test examples) |
101
+
102
+ ## Training Details (v2)
103
+
104
+ ### Dataset
105
+
106
+ 877 labelled examples extracted from **22 curated public GitHub repositories** via regex-based heuristics. Each example is a 3–7 line context window around a matched line, formatted as `Analyze this <issue_type>: <context>`.
107
+
108
+ **Languages covered:** Python, Java, PHP, JavaScript, TypeScript, Ruby, Go
109
+
110
+ ### Training Set Composition
111
+
112
+ | Label | Train examples | Source strategy |
113
+ |---|---|---|
114
+ | `REAL_SECRET` | 27 | TruffleHog test corpus (confirmed leaked secrets), Railsgoat, NodeGoat, Juice Shop |
115
+ | `VULNERABLE_LOGIC` | 194 | DVWA (PHP), WebGoat (Java), vulhub, NodeGoat, DVGA, Railsgoat, Juice Shop |
116
+ | `TEST_MOCK` | 240 | factory_boy, Faker, pytest, model_bakery |
117
+ | `SAFE_CODE` | 240 | Django, FastAPI, requests, Flask, httpx, Devise, Sinatra, Gin |
118
+
119
+ ### Source Repositories
120
+
121
+ | Repo | Label | Language |
122
+ |---|---|---|
123
+ | OWASP/WebGoat | VULNERABLE_LOGIC | Java |
124
+ | digininja/DVWA | VULNERABLE_LOGIC | PHP |
125
+ | vulhub/vulhub | VULNERABLE_LOGIC | multi |
126
+ | trufflesecurity/trufflehog | REAL_SECRET | Go/multi |
127
+ | OWASP/NodeGoat | VULNERABLE_LOGIC | JavaScript |
128
+ | dolevf/Damn-Vulnerable-GraphQL-Application | VULNERABLE_LOGIC | Python |
129
+ | OWASP/railsgoat | VULNERABLE_LOGIC | Ruby |
130
+ | juice-shop/juice-shop | VULNERABLE_LOGIC | TypeScript |
131
+ | FactoryBoy/factory_boy | TEST_MOCK | Python |
132
+ | joke2k/faker | TEST_MOCK | Python |
133
+ | pytest-dev/pytest | TEST_MOCK | Python |
134
+ | model-bakers/model_bakery | TEST_MOCK | Python |
135
+ | django/django | SAFE_CODE | Python |
136
+ | tiangolo/fastapi | SAFE_CODE | Python |
137
+ | psf/requests | SAFE_CODE | Python |
138
+ | pallets/flask | SAFE_CODE | Python |
139
+ | encode/httpx | SAFE_CODE | Python |
140
+ | heartcombo/devise | SAFE_CODE | Ruby |
141
+ | sinatra/sinatra | SAFE_CODE | Ruby |
142
+ | gin-gonic/gin | SAFE_CODE | Go |
143
+ | golang/vulndb | VULNERABLE_LOGIC | Go |
144
+
145
+ ### Training Hyperparameters
146
+ - batch_size: (16, 16)
147
+ - num_epochs: (3, 3)
148
+ - max_steps: -1
149
+ - sampling_strategy: oversampling
150
+ - num_iterations: 20
151
+ - body_learning_rate: (2e-05, 1e-05)
152
+ - head_learning_rate: 0.01
153
+ - loss: CosineSimilarityLoss
154
+ - distance_metric: cosine_distance
155
+ - margin: 0.25
156
+ - end_to_end: False
157
+ - use_amp: False
158
+ - warmup_proportion: 0.1
159
+ - l2_weight: 0.01
160
+ - seed: 42
161
+ - eval_max_steps: -1
162
+ - load_best_model_at_end: True
163
+
164
+ ### Training Results
165
+ | Epoch | Step | Training Loss | Validation Loss |
166
+ |:------:|:----:|:-------------:|:---------------:|
167
+ | 0.0006 | 1 | 0.0416 | - |
168
+ | 0.0285 | 50 | 0.0099 | - |
169
+ | 0.0570 | 100 | 0.0028 | - |
170
+ | 0.0856 | 150 | 0.001 | - |
171
+ | 0.1141 | 200 | 0.0013 | - |
172
+ | 0.1426 | 250 | 0.0003 | - |
173
+ | 0.1711 | 300 | 0.0002 | - |
174
+ | 0.1997 | 350 | 0.0002 | - |
175
+ | 0.2282 | 400 | 0.0002 | - |
176
+ | 0.2567 | 450 | 0.0002 | - |
177
+ | 0.2852 | 500 | 0.0002 | - |
178
+ | 0.3137 | 550 | 0.0001 | - |
179
+ | 0.3423 | 600 | 0.0001 | - |
180
+ | 0.3708 | 650 | 0.0001 | - |
181
+ | 0.3993 | 700 | 0.0001 | - |
182
+ | 0.4278 | 750 | 0.0001 | - |
183
+ | 0.4564 | 800 | 0.0001 | - |
184
+ | 0.4849 | 850 | 0.0001 | - |
185
+ | 0.5134 | 900 | 0.0001 | - |
186
+ | 0.5419 | 950 | 0.0001 | - |
187
+ | 0.5705 | 1000 | 0.0001 | - |
188
+ | 0.5990 | 1050 | 0.0001 | - |
189
+ | 0.6275 | 1100 | 0.0001 | - |
190
+ | 0.6560 | 1150 | 0.0001 | - |
191
+ | 0.6845 | 1200 | 0.0001 | - |
192
+ | 0.7131 | 1250 | 0.0001 | - |
193
+ | 0.7416 | 1300 | 0.0001 | - |
194
+ | 0.7701 | 1350 | 0.0001 | - |
195
+ | 0.7986 | 1400 | 0.0 | - |
196
+ | 0.8272 | 1450 | 0.0001 | - |
197
+ | 0.8557 | 1500 | 0.0001 | - |
198
+ | 0.8842 | 1550 | 0.0 | - |
199
+ | 0.9127 | 1600 | 0.0 | - |
200
+ | 0.9412 | 1650 | 0.0001 | - |
201
+ | 0.9698 | 1700 | 0.0 | - |
202
+ | 0.9983 | 1750 | 0.0 | - |
203
+ | 1.0 | 1753 | - | 0.0001 |
204
+ | 1.0268 | 1800 | 0.0 | - |
205
+ | 1.0553 | 1850 | 0.0 | - |
206
+ | 1.0839 | 1900 | 0.0 | - |
207
+ | 1.1124 | 1950 | 0.0 | - |
208
+ | 1.1409 | 2000 | 0.0 | - |
209
+ | 1.1694 | 2050 | 0.0 | - |
210
+ | 1.1979 | 2100 | 0.0 | - |
211
+ | 1.2265 | 2150 | 0.0 | - |
212
+ | 1.2550 | 2200 | 0.0 | - |
213
+ | 1.2835 | 2250 | 0.0 | - |
214
+ | 1.3120 | 2300 | 0.0 | - |
215
+ | 1.3406 | 2350 | 0.0 | - |
216
+ | 1.3691 | 2400 | 0.0 | - |
217
+ | 1.3976 | 2450 | 0.0 | - |
218
+ | 1.4261 | 2500 | 0.0 | - |
219
+ | 1.4546 | 2550 | 0.0 | - |
220
+ | 1.4832 | 2600 | 0.0 | - |
221
+ | 1.5117 | 2650 | 0.0 | - |
222
+ | 1.5402 | 2700 | 0.0 | - |
223
+ | 1.5687 | 2750 | 0.0 | - |
224
+ | 1.5973 | 2800 | 0.0 | - |
225
+ | 1.6258 | 2850 | 0.0 | - |
226
+ | 1.6543 | 2900 | 0.0 | - |
227
+ | 1.6828 | 2950 | 0.0 | - |
228
+ | 1.7114 | 3000 | 0.0 | - |
229
+ | 1.7399 | 3050 | 0.0 | - |
230
+ | 1.7684 | 3100 | 0.0 | - |
231
+ | 1.7969 | 3150 | 0.0 | - |
232
+ | 1.8254 | 3200 | 0.0 | - |
233
+ | 1.8540 | 3250 | 0.0 | - |
234
+ | 1.8825 | 3300 | 0.0 | - |
235
+ | 1.9110 | 3350 | 0.0 | - |
236
+ | 1.9395 | 3400 | 0.0 | - |
237
+ | 1.9681 | 3450 | 0.0 | - |
238
+ | 1.9966 | 3500 | 0.0 | - |
239
+ | 2.0 | 3506 | - | 0.0000 |
240
+ | 2.0251 | 3550 | 0.0 | - |
241
+ | 2.0536 | 3600 | 0.0 | - |
242
+ | 2.0821 | 3650 | 0.0 | - |
243
+ | 2.1107 | 3700 | 0.0 | - |
244
+ | 2.1392 | 3750 | 0.0 | - |
245
+ | 2.1677 | 3800 | 0.0 | - |
246
+ | 2.1962 | 3850 | 0.0 | - |
247
+ | 2.2248 | 3900 | 0.0 | - |
248
+ | 2.2533 | 3950 | 0.0 | - |
249
+ | 2.2818 | 4000 | 0.0 | - |
250
+ | 2.3103 | 4050 | 0.0 | - |
251
+ | 2.3388 | 4100 | 0.0 | - |
252
+ | 2.3674 | 4150 | 0.0 | - |
253
+ | 2.3959 | 4200 | 0.0 | - |
254
+ | 2.4244 | 4250 | 0.0 | - |
255
+ | 2.4529 | 4300 | 0.0 | - |
256
+ | 2.4815 | 4350 | 0.0 | - |
257
+ | 2.5100 | 4400 | 0.0 | - |
258
+ | 2.5385 | 4450 | 0.0 | - |
259
+ | 2.5670 | 4500 | 0.0 | - |
260
+ | 2.5956 | 4550 | 0.0 | - |
261
+ | 2.6241 | 4600 | 0.0 | - |
262
+ | 2.6526 | 4650 | 0.0 | - |
263
+ | 2.6811 | 4700 | 0.0 | - |
264
+ | 2.7096 | 4750 | 0.0 | - |
265
+ | 2.7382 | 4800 | 0.0 | - |
266
+ | 2.7667 | 4850 | 0.0 | - |
267
+ | 2.7952 | 4900 | 0.0 | - |
268
+ | 2.8237 | 4950 | 0.0 | - |
269
+ | 2.8523 | 5000 | 0.0 | - |
270
+ | 2.8808 | 5050 | 0.0 | - |
271
+ | 2.9093 | 5100 | 0.0 | - |
272
+ | 2.9378 | 5150 | 0.0 | - |
273
+ | 2.9663 | 5200 | 0.0 | - |
274
+ | 2.9949 | 5250 | 0.0 | - |
275
+ | 3.0 | 5259 | - | 0.0000 |
276
 
277
  ### Framework Versions
278
  - Python: 3.12.12