Cybersecurity Fine-tuned Qwen2.5-Coder-7B

This model was fine-tuned from unsloth/Qwen2.5-Coder-7B-Instruct on cybersecurity datasets using Unsloth + LoRA.

Training Details

Base Model: unsloth/Qwen2.5-Coder-7B-Instruct
Parameters: 7B
Method: LoRA fine-tuning
LoRA Rank: 16
LoRA Alpha: 32
Training Examples: 70,000
Final Loss: 0.7485
Training Duration: 31 minutes
Hardware: NVIDIA B200

Datasets Used

Dataset	Examples
omurkuru/cve-security-data	20,000
Trendyol/Cybersecurity-Instruction	10,000
ethanolivertroy/nist-cybersecurity	10,000
Nitral-AI/Cybersecurity-ShareGPT	10,000
Vanessasml/cybersecurity_32k	10,000
jason-oneal/pentest-agent-dataset	10,000
Total	70,000

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "dennny123/cybersec-qwen2.5-coder-7b",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("dennny123/cybersec-qwen2.5-coder-7b")

messages = [
    {"role": "system", "content": "You are a cybersecurity expert assistant."},
    {"role": "user", "content": "Explain CVE-2024-1234 and its impact"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))