Instructions to use Zeineb12/evaluation-artifacts with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Zeineb12/evaluation-artifacts with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Zeineb12/evaluation-artifacts")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Zeineb12/evaluation-artifacts")
model = AutoModelForCausalLM.from_pretrained("Zeineb12/evaluation-artifacts", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Zeineb12/evaluation-artifacts with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Zeineb12/evaluation-artifacts"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zeineb12/evaluation-artifacts",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Zeineb12/evaluation-artifacts

SGLang

How to use Zeineb12/evaluation-artifacts with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Zeineb12/evaluation-artifacts" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zeineb12/evaluation-artifacts",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Zeineb12/evaluation-artifacts" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zeineb12/evaluation-artifacts",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Zeineb12/evaluation-artifacts with Docker Model Runner:
```
docker model run hf.co/Zeineb12/evaluation-artifacts
```

evaluation-artifacts / README.md

Zeineb12

Update README.md

ff9556f verified 3 days ago

preview code

Raw History Blame Contribute Delete

6.64 kB

---
library_name: transformers
tags:
  - security
  - code-vulnerability-detection
  - evaluation
  - interpretability
---
# Blind Spots of Frontier Models: Memorized Recall vs. Genuine Vulnerability Discovery

**Author:** Zeinabou (Zeineb) Mohamed Elhanavi
**Model evaluated:** `Qwen/Qwen2.5-Coder-3B-Instruct` (3B parameters, within the required 0.6–6B range)

---

## Q1 : The Blind Spot

I noticed that most AI companies announcing that their models discovered zero-day vulnerabilities rarely prove the real truth of their claims — even when they involve an independent third party. There is a difference between saying *"our model alone, unaided, discovered this zero-day"* and *"some engineer in the background noticed something and told the model to look for this zero-day."* Most companies honestly prove that the vulnerability is a zero-day because it was never found before — but they do not actually prove that their model, alone and without humans leaking it hints, was able to discover it. This is exactly why I chose this small example.

This points to a deeper gap: a code security model can look like an expert while it is only **recalling** vulnerabilities it saw during training, not **discovering** them. Standard benchmarks cannot catch this, because they are built from known, published vulnerabilities — old CVEs and code already scraped from GitHub — that the model has very likely already seen. So when the model "succeeds," it is remembering, not reasoning, and the benchmark rewards memory while calling it capability.

In my own research on the security of multi-agent LLM systems (HimayaAI), the vulnerabilities I face are new — they emerge from how the agents interact, and they do not exist in any benchmark or public dataset. This is why I can see this blind spot so clearly: my work lives exactly where a model must truly *discover* a flaw, not *recall* one it has memorized.

This matters because these models are increasingly deployed as security auditors. A model that is strong on the known and blind on the new gives false confidence — and real breaches live precisely in what is new and unseen.

---

## Methodology

I tested one hypothesis: *the model recognizes memorized vulnerabilities but fails to reason about equivalent-but-novel ones.* To isolate recall from understanding, I built a controlled dataset of 12 samples across 3 vulnerability patterns — SQL injection, command injection, and insecure deserialization — each in 4 matched variants. The audit prompt was held fixed across all samples, so the only variable is the code itself.

| Variant | What it is | A model that *understands* should... |
|---|---|---|
| `known_cve` | Classic, widely-published vulnerable pattern | Flag it (YES) |
| `mutated` | Same vulnerability, rewritten / renamed / reframed so it is visually novel | Flag it (YES) |
| `patched` | Near-identical safe version, differs by ~1 line | Clear it (NO) |
| `fake` | Fully safe ordinary code | Clear it (NO) — no hallucinated vuln |

The signature of memorization over understanding: high accuracy on `known_cve`, a collapse on `mutated`, and confident but wrong reasoning on the variants it fails.

---

## Results

**Accuracy by category:**

| Category | Score | |
|---|---|---|
| `known_cve` | 2/3 | 67% |
| `mutated` | 0/3 | **0%** |
| `patched` | 3/3 | 100% |
| `fake` | 3/3 | 100% |

The numbers tell the core story: the model scored 67% on memorized vulnerabilities but **0% on the mutated versions of the exact same vulnerabilities.** The only thing that changed between them was the surface form of the code — the underlying flaw was identical. It caught what it had seen before and went blind the moment the shape was new.

This pattern repeated across two independent families (SQL injection and insecure deserialization), so it is not luck. Command injection was worse still: the model failed even on the *known* version, suggesting it has no robust internal representation of that vulnerability class at the 3B scale.

The most important finding is that **the model does not fail silently — it fails with confidence, inventing safety that is not there.** Two outputs from my run make this clear:

- On `sql_mut` (a real f-string injection), the model claimed the query was parameterized with placeholders — yet there is no placeholder anywhere in that code. It described a protection that does not exist.
- On `deser_mut` (`yaml.load` with the unsafe `Loader`), the model claimed the library safely parses the data and prevents injection — the exact opposite of the truth, since the unsafe loader is precisely what enables arbitrary code execution.

Both false justifications were written with the same fluent confidence as the model's correct answers. This is the real danger for deployment: not a model that admits "I don't know," but one that fabricates expert-sounding reassurance about novel code.

---

## Q3 : A Path Forward

My results show the model fails because it judges code by its *visual surface*, not by its *logic*: it reacts to how the code looks rather than tracing what the data actually does. So my proposed direction is a **data curation and fine-tuning strategy built on look-alike, different-meaning examples.**

Concretely, I would train the model on matched triples: a vulnerable snippet, a *mutated* version that looks different but carries the exact same flaw, and a *patched* version that looks almost identical but is safe. By repeatedly showing the model that surface shape and real safety do not line up, it is forced to learn the invariant *logic* of a vulnerability instead of memorizing its appearance. I would also deliberately include "safe-but-suspicious" samples so the model stops raising false alarms on safe code that merely looks risky.

This attacks the recall-vs-discovery gap directly: the only way to get the triples right is to reason about data flow, not to pattern-match familiar shapes. It also connects to the Zero-Trust thinking in my HimayaAI research — never trust the surface appearance of a component; verify what it actually does. A natural extension would be to require the model to trace the data path before giving a verdict, which would directly target the confident-hallucination failure I observed.

---

## Artifacts in this repository

- `eval_notebook.ipynb` — evaluation code: loads the model, runs the dataset, scores by category
- `eval_dataset.jsonl` — my hand-built evaluation dataset (12 samples, 3 patterns × 4 variants)
- `eval_results.json` — raw model outputs and auto-scoring (the evidence behind the findings)
- `README.md` — this file

	---
	library_name: transformers
	tags:
	- security
	- code-vulnerability-detection
	- evaluation
	- interpretability
	---

	# Blind Spots of Frontier Models: Memorized Recall vs. Genuine Vulnerability Discovery

	Author: Zeinabou (Zeineb) Mohamed Elhanavi
	Model evaluated: `Qwen/Qwen2.5-Coder-3B-Instruct` (3B parameters, within the required 0.6–6B range)

	---

	## Q1 : The Blind Spot

	I noticed that most AI companies announcing that their models discovered zero-day vulnerabilities rarely prove the real truth of their claims — even when they involve an independent third party. There is a difference between saying "our model alone, unaided, discovered this zero-day" and "some engineer in the background noticed something and told the model to look for this zero-day." Most companies honestly prove that the vulnerability is a zero-day because it was never found before — but they do not actually prove that their model, alone and without humans leaking it hints, was able to discover it. This is exactly why I chose this small example.

	This points to a deeper gap: a code security model can look like an expert while it is only recalling vulnerabilities it saw during training, not discovering them. Standard benchmarks cannot catch this, because they are built from known, published vulnerabilities — old CVEs and code already scraped from GitHub — that the model has very likely already seen. So when the model "succeeds," it is remembering, not reasoning, and the benchmark rewards memory while calling it capability.

	In my own research on the security of multi-agent LLM systems (HimayaAI), the vulnerabilities I face are new — they emerge from how the agents interact, and they do not exist in any benchmark or public dataset. This is why I can see this blind spot so clearly: my work lives exactly where a model must truly discover a flaw, not recall one it has memorized.

	This matters because these models are increasingly deployed as security auditors. A model that is strong on the known and blind on the new gives false confidence — and real breaches live precisely in what is new and unseen.

	---

	## Methodology

	I tested one hypothesis: the model recognizes memorized vulnerabilities but fails to reason about equivalent-but-novel ones. To isolate recall from understanding, I built a controlled dataset of 12 samples across 3 vulnerability patterns — SQL injection, command injection, and insecure deserialization — each in 4 matched variants. The audit prompt was held fixed across all samples, so the only variable is the code itself.

	\| Variant \| What it is \| A model that understands should... \|
	\|---\|---\|---\|
	\| `known_cve` \| Classic, widely-published vulnerable pattern \| Flag it (YES) \|
	\| `mutated` \| Same vulnerability, rewritten / renamed / reframed so it is visually novel \| Flag it (YES) \|
	\| `patched` \| Near-identical safe version, differs by ~1 line \| Clear it (NO) \|
	\| `fake` \| Fully safe ordinary code \| Clear it (NO) — no hallucinated vuln \|

	The signature of memorization over understanding: high accuracy on `known_cve`, a collapse on `mutated`, and confident but wrong reasoning on the variants it fails.

	---

	## Results

	Accuracy by category:

	\| Category \| Score \| \|
	\|---\|---\|---\|
	\| `known_cve` \| 2/3 \| 67% \|
	\| `mutated` \| 0/3 \| 0% \|
	\| `patched` \| 3/3 \| 100% \|
	\| `fake` \| 3/3 \| 100% \|

	The numbers tell the core story: the model scored 67% on memorized vulnerabilities but 0% on the mutated versions of the exact same vulnerabilities. The only thing that changed between them was the surface form of the code — the underlying flaw was identical. It caught what it had seen before and went blind the moment the shape was new.

	This pattern repeated across two independent families (SQL injection and insecure deserialization), so it is not luck. Command injection was worse still: the model failed even on the known version, suggesting it has no robust internal representation of that vulnerability class at the 3B scale.

	The most important finding is that the model does not fail silently — it fails with confidence, inventing safety that is not there. Two outputs from my run make this clear:

	- On `sql_mut` (a real f-string injection), the model claimed the query was parameterized with placeholders — yet there is no placeholder anywhere in that code. It described a protection that does not exist.
	- On `deser_mut` (`yaml.load` with the unsafe `Loader`), the model claimed the library safely parses the data and prevents injection — the exact opposite of the truth, since the unsafe loader is precisely what enables arbitrary code execution.

	Both false justifications were written with the same fluent confidence as the model's correct answers. This is the real danger for deployment: not a model that admits "I don't know," but one that fabricates expert-sounding reassurance about novel code.

	---

	## Q3 : A Path Forward

	My results show the model fails because it judges code by its visual surface, not by its logic: it reacts to how the code looks rather than tracing what the data actually does. So my proposed direction is a data curation and fine-tuning strategy built on look-alike, different-meaning examples.

	Concretely, I would train the model on matched triples: a vulnerable snippet, a mutated version that looks different but carries the exact same flaw, and a patched version that looks almost identical but is safe. By repeatedly showing the model that surface shape and real safety do not line up, it is forced to learn the invariant logic of a vulnerability instead of memorizing its appearance. I would also deliberately include "safe-but-suspicious" samples so the model stops raising false alarms on safe code that merely looks risky.

	This attacks the recall-vs-discovery gap directly: the only way to get the triples right is to reason about data flow, not to pattern-match familiar shapes. It also connects to the Zero-Trust thinking in my HimayaAI research — never trust the surface appearance of a component; verify what it actually does. A natural extension would be to require the model to trace the data path before giving a verdict, which would directly target the confident-hallucination failure I observed.

	---

	## Artifacts in this repository

	- `eval_notebook.ipynb` — evaluation code: loads the model, runs the dataset, scores by category
	- `eval_dataset.jsonl` — my hand-built evaluation dataset (12 samples, 3 patterns × 4 variants)
	- `eval_results.json` — raw model outputs and auto-scoring (the evidence behind the findings)
	- `README.md` — this file