--- license: apache-2.0 base_model: Qwen/Qwen3.8-27B library_name: transformers pipeline_tag: text-generation tags: - security - cve - lora - sft - unsloth - transformers --- # Qwen3.8-27B — TrAIli CVE Code-QA (merged) A merged, standalone fine-tune of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for CVE/PoC code question-answering, built by the TrAIli project. LoRA weights (rsLoRA r=32, QLoRA 4-bit trained) are baked into the bf16 base — download and run directly, no adapter loading required. ## Model Details ### Model Description Answers grounded, technical questions about vulnerability PoCs — e.g. "which line sends the request that triggers the bug", "quote `reboot_device()` and describe its behavior" — when the PoC source code is included in the prompt. Loss is computed only on the assistant turn; the model is trained with the Qwen chat template rendered with `enable_thinking=False`, so it answers directly without emitting a `` reasoning block. - **Developed by:** TrAIli project - **Model type:** Qwen3.8-27B (hybrid gated-DeltaNet + gated-attention, 262K native context), causal LM - **Language(s):** English, code - **License:** apache-2.0 - **Finetuned from model:** [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) ### Model Sources - **Base model:** https://huggingface.co/Qwen/Qwen3.8-27B - **Adapter version:** https://huggingface.co//qwen3.8-27b-traili-cve (LoRA-only, ~470MB) ## Uses ### Direct Use CVE triage and PoC explanation in authorized, sandbox-contained security research: given the PoC/advisory text, answer questions about what the code does, where the vulnerable behavior is, and how it maps to the CVE. ### Downstream Use Fine-tuning, RAG-based CVE assistants, lab-target testing harnesses (e.g. generating and explaining test steps for dockerized vulnerable targets). ### Out-of-Scope Use - Testing, probing, or exploitation of systems you are **not** authorized to test. - Answering CVE questions without grounding material: with no code or advisory in context the model can fabricate plausible-sounding details. Feed it the PoC or use RAG. ## Bias, Risks, and Limitations - **Hallucination risk when ungrounded.** The model quotes provided code well; without provided facts it may confabulate CVE descriptions, product names, or versions. Always provide the PoC/advisory. - Trained on a distilled code-QA dataset (~15.7k examples) generated by a teacher model — the data inherits whatever errors the teacher made. - Thinking mode is off by training design; reasoning blocks are not emitted. ### Recommendations Treat outputs as analysis to verify, not as ground truth. For factual CVE metadata (scores, affected versions), cross-check against NVD or the vendor advisory. Use RAG for up-to-date CVE data. ## How to Get Started with the Model ```python import torch from transformers import AutoProcessor, AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained( "", torch_dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa") processor = AutoProcessor.from_pretrained("") messages = [ {"role": "system", "content": "You are a senior vulnerability researcher working in " "authorized, sandbox-contained research. Ground every " "statement ONLY in the facts provided; never invent CVE " "details, versions, or identifiers."}, {"role": "user", "content": "Provided code (from poc.py):\n\n```python\n" "URL = \"http://target/reqproc/proc_post\"\n" "def reboot_device():\n req = requests.get(URL)\n" "```\n\nWhich line triggers the bug?"}, ] prompt = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False) inputs = processor(text=prompt, return_tensors="pt").to(model.device) out = model.generate(**inputs, do_sample=True, temperature=0.7, top_p=0.95, top_k=64, max_new_tokens=1024) print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) ``` Runs in ~17GB VRAM with a 4-bit quant (e.g. via bitsandbytes) or ~56GB at bf16. ## Training Details ### Training Data TrAIli distilled CVE/PoC code-QA set: ~15,735 chat records (system + user + assistant) generated by a teacher model over PoC-in-GitHub sources with NVD enrichment, covering a broad range of CVEs and PoC languages. ### Training Procedure - Chat template rendered with `enable_thinking=False`; assistant-turn-only loss (labels -100 outside the final `<|im_start|>assistant` span). - No sample packing (per-batch dynamic padding, no cross-sample attention); length-grouped sampling; truncation to 4096 tokens. - Best checkpoint kept by eval loss, not last step. #### Training Hyperparameters - QLoRA 4-bit NF4 + double quant, bf16 compute - LoRA r=32, alpha=32, rsLoRA, gaussian init; targets: all text linear projections (attention q/k/v/o, DeltaNet in_proj_qkv/z/b/a/out_proj, MLP gate/up/down — 496 modules, ~233.5M trainable params) - NEFTune noise 5, lr 2e-4 cosine, warmup 5% of steps, effective batch 32 - Up to 3 epochs; single RTX PRO 6000 (96GB) #### Speeds, Sizes, Times - Adapter: ~470MB. Merged bf16: ~56GB sharded. - Training: minutes-to-low-hours on a single 96GB Blackwell GPU. ## Evaluation ### Testing Data, Factors & Metrics - Held-out 2% split, eval loss on assistant spans. - Qualitative checks: grounded code-QA (correct, quotes the code) vs fact-free CVE questions (can hallucinate — see Limitations). ### Results #### Summary Grounded code-QA answers track the provided PoC closely; ungrounded questions are the known failure mode. Eval loss and behavior improve with epochs up to the data's diversity limit. ## Environmental Impact - **Hardware Type:** NVIDIA RTX PRO 6000 (Blackwell, 96GB) - **Cloud Provider:** RunPod - **Hours used:** a few (training + iteration) ## Technical Specifications ### Model Architecture and Objective Qwen3.8-27B: 64 layers, 16 blocks of (3× gated-DeltaNet + 1× gated attention), 262,144 native context, 248,320 vocab. Objective: causal LM SFT on assistant turns. ### Compute Infrastructure #### Hardware 1× NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB). #### Software - Unsloth 2026.8.x (Fast Qwen3_5 patching), PEFT 0.20.0, transformers 5.5.0, torch 2.11.0 (CUDA 13), bitsandbytes 0.50.x ## Citation **BibTeX:** ``` @software{qwen38_traili_cve, title = {Qwen3.8-27B TrAIli CVE Code-QA}, author = {TrAIli project}, note = {Fine-tune of Qwen/Qwen3.8-27B for CVE/PoC code question-answering}, url = {https://huggingface.co//qwen3.8-27b-traili-cve} } ``` ## Model Card Authors TrAIli project. ## Model Card Contact Trailios ### Framework versions - PEFT 0.20.0 - transformers 5.5.0 - Unsloth 2026.8.18 - torch 2.11.0+cu130