scthornton/gemma-4-e4b-securecode

Gemma 4 E4B SecureCode

Security-specialized code model fine-tuned on the SecureCode dataset

Dataset | Paper (arXiv:2512.18542) | Model Collection | perfecXion.ai

---

What This Model Does

This model generates secure code when developers ask about building features. Instead of producing vulnerable implementations (like 45% of AI-generated code does), it:

• Identifies the security risks in common coding patterns
• Provides vulnerable and secure implementations side by side
• Explains how attackers would exploit the vulnerability
• Includes defense-in-depth guidance: logging, monitoring, SIEM integration, infrastructure hardening

The model was fine-tuned on 2,372 security training examples covering both traditional web security (OWASP Top 10 2021) and AI/ML security (OWASP LLM Top 10 2025).

Model Details

Base Model	Gemma 4 E4B Instruct
Parameters	E4B (~8B raw, ~4B effective via per-layer embeddings)
Architecture	Gemma 4 (multimodal base; fine-tuned and intended for text-only use)
Tier	Tier 1: Accessible
Method	bf16 LoRA (no quantization)
LoRA Rank	16 (alpha=32)
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj on the language model decoder (the 18 KV-shared layers own no k/v weights, so k_proj/v_proj are adapted on the 24 layers that have them)
Training Data	scthornton/securecode (2,372 examples)
Hardware	NVIDIA DGX Spark GB10 (Blackwell, unified memory)

Newest member of the SecureCode collection. Requires transformers >= 5.13 (earlier 5.x versions have Gemma 4 training bugs; see notes below).

Quick Start

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Requires transformers >= 5.13
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-E4B-it",
    dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("scthornton/gemma-4-e4b-securecode")
model = PeftModel.from_pretrained(base_model, "scthornton/gemma-4-e4b-securecode")

# Ask a security-relevant coding question (chat template required)
messages = [
    {"role": "user", "content": "How do I implement JWT authentication with refresh tokens in Python?"}
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=2048, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Dataset

Trained on the full SecureCode unified dataset (audited release):

• 2,372 total examples (1,625 web security + 747 AI/ML security)
• 20 vulnerability categories across OWASP Top 10 2021 and OWASP LLM Top 10 2025
• 12+ programming languages and 49+ frameworks
• 4-turn conversational structure: feature request, vulnerable/secure implementations, advanced probing, operational guidance
• 100% incident grounding: every example tied to real CVEs, vendor advisories, or published attack research

Unlike the rest of the collection (trained with ### User: / ### Assistant: role markers), this model was trained through Gemma 4's own chat template, matching its instruction tuning.

Hyperparameters

Parameter	Value
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	language model decoder projections (see Model Details)
Quantization	None (bf16 base weights)
Learning rate	2e-4
LR scheduler	Cosine with 100-step warmup
Epochs	3
Per-device batch size	1
Gradient accumulation	16x
Effective batch size	16
Max sequence length	4096 tokens
Optimizer	adamw_torch_fused
Attention	PyTorch SDPA (fused)
Precision	bf16

Notes: Gradient checkpointing is deliberately OFF: Gemma 4 E4B's KV-shared layers read earlier layers' KV cache entries within a single forward pass, and checkpointing force-disables that cache, silently corrupting training. transformers 5.13+ also fixes num_items_in_batch loss normalization for this architecture. Trained text-only; the vision and audio towers are untouched by the LoRA adapter.

Security Coverage

Web Security (1,625 examples)

OWASP Top 10 2021: Broken Access Control, Cryptographic Failures, Injection, Insecure Design, Security Misconfiguration, Vulnerable Components, Authentication Failures, Software Integrity Failures, Logging/Monitoring Failures, SSRF.

Languages: Python, JavaScript, Java, Go, PHP, C#, TypeScript, Ruby, Rust, Kotlin, YAML.

AI/ML Security (747 examples)

OWASP LLM Top 10 2025: Prompt Injection, Sensitive Information Disclosure, Supply Chain Vulnerabilities, Data/Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, Vector/Embedding Weaknesses, Misinformation, Unbounded Consumption.

Frameworks: LangChain, OpenAI, Anthropic, HuggingFace, LlamaIndex, ChromaDB, Pinecone, FastAPI, Flask, vLLM, CrewAI, and 30+ more.

SecureCode Model Collection

This model is part of the SecureCode collection of 9 security-specialized models:

Model	Base	Size	Tier	HuggingFace
Llama 3.2 SecureCode	meta-llama/Llama-3.2-3B-Instruct	3B	Accessible	llama-3.2-3b-securecode
Gemma 4 E4B SecureCode	google/gemma-4-E4B-it	E4B (8B raw)	Accessible	gemma-4-e4b-securecode
Qwen2.5 Coder SecureCode	Qwen/Qwen2.5-Coder-7B-Instruct	7B	Mid-size	qwen2.5-coder-7b-securecode
DeepSeek Coder SecureCode	deepseek-ai/deepseek-coder-6.7b-instruct	6.7B	Mid-size	deepseek-coder-6.7b-securecode
CodeGemma SecureCode	google/codegemma-7b-it	7B	Mid-size	codegemma-7b-securecode
CodeLlama SecureCode	codellama/CodeLlama-13b-Instruct-hf	13B	Large	codellama-13b-securecode
Qwen2.5 Coder 14B SecureCode	Qwen/Qwen2.5-Coder-14B-Instruct	14B	Large	qwen2.5-coder-14b-securecode
StarCoder2 SecureCode	bigcode/starcoder2-15b-instruct-v0.1	15B	Large	starcoder2-15b-securecode
Granite 20B Code SecureCode	ibm-granite/granite-20b-code-instruct-8k	20B	XL	granite-20b-code-securecode

Choose based on your deployment constraints: 3B/E4B for edge and resource-constrained use, 7B for general use, 13B-15B for deeper reasoning, 20B for maximum capability.

SecureCode Dataset Family

Dataset	Examples	Focus	Link
SecureCode	2,372	Unified (web + AI/ML)	scthornton/securecode
SecureCode Web	1,625	Web security (OWASP Top 10 2021)	scthornton/securecode-web
SecureCode AI/ML	747	AI/ML security (OWASP LLM Top 10 2025)	scthornton/securecode-aiml

Intended Use

Use this model for:

• Training AI coding assistants to write secure code
• Security education and training
• Vulnerability research and secure code review
• Building security-aware development tools

Do not use this model for:

• Offensive exploitation or automated attack generation
• Circumventing security controls
• Any activity that violates the base model's license

Changelog

• 2026-07 (v1, current): Initial release. Trained on the audited SecureCode release (2,372 examples: 1,625 web + 747 AI/ML) using bf16 LoRA on an NVIDIA DGX Spark GB10 (Blackwell). Added to the SecureCode family alongside the 2026-07 refresh of the original 8 models.

Citation

@misc{thornton2026securecode,
  title={SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models},
  author={Thornton, Scott},
  year={2026},
  publisher={perfecXion.ai},
  url={https://huggingface.co/datasets/scthornton/securecode},
  note={arXiv:2512.18542}
}

Links

• Dataset: scthornton/securecode
• Paper: arXiv:2512.18542
• Collection: SecureCode models
• perfecXion.ai: https://perfecxion.ai