README.md

---
license: apache-2.0
language:
- en
metrics:
- accuracy
- f1
- roc_auc
library_name: safetensors
tags:
- code
- security
- vulnerability-detection
- cwe
- owasp
- moe
- hobbylm
pipeline_tag: text-generation
---
# CodeAstra-500M: Laptop-Scale Vulnerability Detection 🔍🛡️

## Model Description

CodeAstra-500M is the small sibling of [CodeAstra-7B](https://huggingface.co/rootxhacker/CodeAstra-7B) — a
500M-parameter **sparse Mixture-of-Experts** model fine-tuned for security vulnerability detection in
source code. Where CodeAstra-7B is built on Mistral-7B, CodeAstra-500M is built on
[HobbyLM-Chat](https://huggingface.co/rootxhacker/HobbyLM-Chat), a MoE language model trained from
scratch on a hobby budget — so the whole thing runs on a laptop CPU.

It answers with a single structured JSON verdict: is this vulnerable, which CWE, which OWASP category.

### Key Features

- 🪶 **Tiny**: 500M total parameters, only ~150M active per token thanks to top-6-of-36 expert routing.
- 📐 **Structured output**: emits parseable JSON on 99.98% of inputs — no regex-scraping prose.
- 🎚️ **Tunable**: returns a calibrated probability, so you pick the precision/recall trade-off at
  inference time instead of retraining.
- 🌐 **Multi-language**: C, C++, Python, Java, JavaScript, PHP, Go, Ruby, Swift, Kotlin, C#, Fortran —
  though the training mix is heavily C-weighted (see Limitations).
- 🧪 **Honestly evaluated**: scored on the standard split *and* on vulnerable/patched function pairs,
  which is the harder and more meaningful test.
- 🥊 **Benchmarked against a frontier model**: matches DeepSeek v4 Flash on genuine flaw
  discrimination while being ~10,000× faster per snippet.
- 💻 **Runs locally**: shares the HobbyLM architecture, so it loads in the from-scratch Rust CPU engine
  (`hobby-rs`) with no Python at runtime.

## Performance 📊

Evaluated on the held-out test split (17,542 snippets, 1,887 vulnerable) of
[`ayshajavd/code-security-vulnerability-dataset`](https://huggingface.co/datasets/ayshajavd/code-security-vulnerability-dataset).
The base model is un-finetuned HobbyLM-Chat under identical prompts.

| Metric | HobbyLM-Chat (base) | **CodeAstra-500M** |
|---|---|---|
| JSON parse rate | 1.20% | **99.98%** |
| ROC AUC | — | **96.97%** |
| Average precision | — | **82.64%** |
| Best F1 | 0.00% | **76.23%** |
| Precision / Recall @ best F1 | 0.00 / 0.00 | **75.57% / 76.89%** |
| Exact CWE (28 classes) | 0.00% | **~55%** |
| OWASP category | 0.00% | **~74%** |

⚠️ **A note on accuracy.** This dataset is 89% non-vulnerable, so a model that answers "safe" every
time scores **90.5% accuracy** — which is exactly what the un-finetuned base model does, at 0% recall.
That is why this card leads with F1, AUC and recall rather than accuracy. Treat any headline accuracy
figure on this dataset with suspicion, including for other models.

### Paired evaluation — the number that actually matters 🎯

Standard splits of CVE-derived vulnerability datasets are **confounded**: the vulnerable functions come
from big C projects (Linux kernel, Chromium, PHP, ffmpeg) while the "safe" ones are often unrelated
code. A model can score very well by recognising *code style* rather than *code flaws*.

So CodeAstra-500M is also evaluated PrimeVul-style, on 894 pairs of a vulnerable function and **its own
patched version** — near-identical code differing only by the security fix.

| | First SFT pass | **CodeAstra-500M** |
|---|---|---|
| **P-C — flags the flawed one, clears the patched one** | 6.94% | **27.40%** |
| P-V — flags both (the style shortcut) | 84.90% | **33.45%** |
| P-B — clears both | 6.71% | 36.02% |
| P-R — reversed | 1.45% | 3.13% |
| Specificity on patched code | 13.65% | **63.42%** |
| Within-pair ranking accuracy | 66.22% | **69.46%** |

The released model was produced by feeding the ~7,000 patched functions back in as *safe* training
examples — minimal pairs that differ only by the fix. This quadrupled paired accuracy and lifted
specificity on patched code from 13.65% to 63.42%.

**Read this honestly:** within-pair ranking accuracy moved only 66.22% → 69.46%, meaning most of the
improvement is better calibration rather than deeper understanding. **27.40% is the realistic estimate
of true detection ability — not the 96.97% AUC.** Very few vulnerability models publish this number;
it is here because it is the one that predicts real-world behaviour.

## Comparison against a frontier model 🥊

Benchmarked against **DeepSeek v4 Flash** (via OpenRouter, reasoning enabled) on the *same* held-out
rows, through the *same* prompts, the *same* output parser and the *same* metrics. DeepSeek answers
zero-shot; CodeAstra is fine-tuned on this corpus.

### On the standard held-out split — 1,000 rows, 93 vulnerable

| Metric | DeepSeek v4 Flash | **CodeAstra-500M** |
|---|---|---|
| JSON parse rate | 97.10% | **99.98%** |
| Precision | 16.56% | **75.57%** |
| Recall | 54.84% | **76.89%** |
| F1 | 25.44% | **76.23%** |
| Exact CWE | 10.75% | **~55%** |
| Latency per snippet | ~47 s | **4.3 ms** |

DeepSeek flagged 308 of 1,000 snippets as vulnerable when only 93 were — a 3.3× over-prediction, since
a general model has no way to know this corpus is 89% safe.

⚠️ **Do not read this as "CodeAstra is 3× better at security."** Most of that gap is *corpus
familiarity* — the base rate and the annotator's CWE conventions — not better security reasoning. The
per-CWE breakdown makes it plain: DeepSeek gets CWE-89 (SQL injection) 100% detected and 80% exact,
but 0% exact on CWE-119, a catch-all bucket whose boundaries you can only learn from the training set.

### On the paired set — 894 pairs, the unconfounded test

| | DeepSeek v4 Flash | **CodeAstra-500M** |
|---|---|---|
| **P-C — flags flawed, clears patched** | 24.05% | **27.40%** |
| P-V — flags both | 34.68% | 33.45% |
| P-B — clears both | 32.33% | 36.02% |
| P-R — reversed | 8.95% | **3.13%** |
| Recall, vulnerable half | 58.72% | **60.85%** |
| Specificity on patched code | 56.38% | **63.42%** |

On this harder, style-controlled test the two models are closely matched — CodeAstra is marginally
ahead, holds higher specificity on patched code, and is 3× less likely to invert a pair.

✅ **The defensible claim:** CodeAstra-500M **matches a frontier reasoning model on genuine flaw
discrimination**, at ~10,000× lower latency, no per-call cost, and fully local.

Distinguishing a vulnerable function from its patched twin is a hard problem at the single-function
level for models of any size — see Limitations.

## Intended Use

CodeAstra-500M is for developers, security researchers and code auditors who want a fast first-pass
triage filter that runs locally — in a pre-commit hook, a CI step, or an editor plugin — without
sending source code to an API. It is a **filter that decides what a human looks at**, not an oracle.

## Threshold — please read 🎚️

The model returns a probability. **The default 0.5 is not the balanced operating point.**

| Threshold | Precision | Recall | Use case |
|---|---|---|---|
| 0.3346 | 75.57% | 76.89% | **Balanced (best F1 = 76.23%)** |
| 0.3775 | 80.02% | 70.27% | Triage |
| 0.4378 | 85.03% | 59.88% | |
| 0.5156 | 90.01% | 45.36% | CI gating |
| 0.6514 | 95.07% | 23.53% | High-confidence alerts only |

At the naive 0.5 you get 88.79% precision but only **48.70% recall** — it will quietly miss half the
vulnerabilities. Set the threshold deliberately.

## Training 🏋️‍♂️

Fine-tuned from HobbyLM-Chat on 8×H100 GPUs via [Modal](https://modal.com), using the **full** dataset —
all 140,335 training rows, nothing subsampled, with over-long code head+tail truncated rather than
dropped.

| | |
|---|---|
| Main SFT | 6,000 steps, lr 2e-5, micro-batch 8 × 8 GPUs, 176,216 examples, ~27 min |
| Hard-negative pass | 500 steps, lr 5e-6, 7,023 minimal pairs, ~2.5 min |
| Objective | next-token CE masked to the JSON verdict; MoE aux-free balancing bias frozen |
| Class balance | vulnerable rows oversampled 3× (10.9% → 26.0% positives) |

Architecture is unchanged from HobbyLM: 768 hidden / 16 layers, 36 experts with top-6 routing plus one
shared expert, GQA attention with per-head QK-norm, RoPE, GPT-2 byte-level BPE, 2048-token context.

## Usage 💻

The model uses the HobbyLM MoE architecture, so it needs the `hobbylm` package rather than
`transformers`. The prompt format matters — the model was trained on exactly one layout:

````text
SYSTEM: You are a source-code security auditor. Given a code snippet, decide whether it contains a security vulnerability and reply with one JSON object: {"vulnerable": bool, "cwe": str, "cwe_name": str, "owasp": str}. Use "none" for safe code.
USER: Audit the code below for security vulnerabilities.
Language: C

```c
<your code here>
```
ASSISTANT:
````

Use `detect_prompt()` rather than assembling that by hand:

```python
import json, torch, tiktoken
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hobbylm.config import ModelConfig
from hobbylm.model import MoETransformer
from hobbylm.security_data import detect_prompt, VERDICT_PREFIX, TRUE_ID, FALSE_ID

repo = "rootxhacker/codeastra-500M"
cfg_d = json.load(open(hf_hub_download(repo, "config.json")))
cfg_d.pop("preset", None)
model = MoETransformer(ModelConfig(**cfg_d)).cuda().eval()
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))
enc = tiktoken.get_encoding("gpt2")

code_to_analyze = """
$query = $_GET['query'];
$stmt = $db->prepare($query);
$stmt->execute();
"""

# Fast path: force the verdict position and read one probability (~4ms, no generation)
prompt = detect_prompt(code_to_analyze, "PHP") + VERDICT_PREFIX
ids = torch.tensor([enc.encode_ordinary(prompt)], device="cuda")
with torch.no_grad():
    logits, _ = model(ids)
p_vuln = torch.softmax(logits[0, -1, [TRUE_ID, FALSE_ID]].float(), -1)[0].item()
print(f"P(vulnerable) = {p_vuln:.3f}  ->  {'VULNERABLE' if p_vuln >= 0.3346 else 'safe'}")
```

Scoring this way is ~26× faster than generating the JSON (4.3 ms vs 110 ms per snippet) and gives you
the tunable probability. If you also want the CWE and OWASP labels, generate the completion normally
from `detect_prompt(code, lang)` and parse the JSON with `hobbylm.security_data.parse_verdict`.

A typical answer:

```json
{"vulnerable": true, "cwe": "CWE-89", "cwe_name": "Improper Neutralization of Special Elements used in an SQL Command (SQL Injection)", "owasp": "A03: Injection"}
```

## Limitations ⚠️

1. **Function-level only.** It sees a single function, so interprocedural and data-flow vulnerabilities
   are largely invisible. Access-control bugs needing caller context are its weakest class.
2. **Heavily C-weighted training data** (92% C). Other languages work but are out-of-distribution —
   expect lower reliability on Go, Swift, Kotlin and TypeScript.
3. **CWE labels confuse related classes.** CWE-89 (SQL injection, 85% exact) and CWE-94 (code
   injection, 82%) are strong; catch-all buckets like CWE-399 and CWE-416 are weak. It frequently finds
   the right bug and picks a sibling CWE — the detection is better than the label suggests.
4. **Multiple vulnerabilities in one snippet** are not reliably enumerated; it returns a single verdict.
5. **Long code is truncated.** Snippets beyond 2048 tokens are head+tail truncated, so the middle of
   very long functions is unseen.
6. **False positives are expected** at the recall-oriented thresholds. Results need human verification.
7. **It is a 500M research model, not a security product.** Use it alongside code review and SAST, not
   instead of them, and do not gate a release on it alone.

## Test Apparatus

All figures come from the held-out test split of `ayshajavd/code-security-vulnerability-dataset`
(17,542 snippets, never trained on), scored with a purpose-built harness that generates the verdict and
parses it, plus the calibrated single-forward-pass scorer for threshold-free metrics. The paired
evaluation uses 894 vulnerable/patched function pairs drawn from the same held-out split. The base
HobbyLM-Chat comparison was run through the **identical** prompts and harness, so the two columns are
directly comparable.

The DeepSeek v4 Flash comparison was run through OpenRouter on the same held-out rows, receiving the
identical system/user content in native chat format so it is not penalised for our decoder-only prompt
convention, and scored with the same parser and metrics. Reasoning was enabled at low effort: at
default effort the model spends ~98% of its output budget thinking and 37% of replies never emit an
answer within 8k tokens, which would have measured our token cap rather than its accuracy.

No numbers on this page were copied from another model card — everything here was measured directly.
**CodeAstra-7B is not comparable** to these figures: it was evaluated on a different corpus under a
different protocol, so its 83% accuracy and this card's numbers do not sit on the same scale.

## Citation 📜

```
@software{CodeAstra-500M,
  author = {Harish Santhanalakshmi Ganesan},
  title = {CodeAstra-500M: Laptop-Scale Vulnerability Detection},
  year = {2026},
  howpublished = {\url{https://huggingface.co/rootxhacker/codeastra-500M}}
}
```

## License 📄

CodeAstra-500M is released under the Apache License 2.0.

```
Copyright 2026 [Harish Santhanalakshmi Ganesan]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

## Acknowledgements 🙏

Thanks to the HobbyLM project for the 500M MoE base model, and to
[@ayshajavd](https://huggingface.co/ayshajavd) for compiling the vulnerability dataset this model was
trained on.