Add model card

README.md

@@ -0,0 +1,306 @@
+---
+base_model: mistralai/Ministral-8B-Instruct-2410
+tags:
+- unsloth
+- lora
+- qlora
+- vulnerability-detection
+- security
+- code-analysis
+- cybersecurity
+- ultival
+- peft
+- adapter
+language:
+- en
+license: apache-2.0
+library_name: peft
+pipeline_tag: text-generation
+---
+
+# UltiVal: Ministral-8B QLoRA Adapter for Vulnerability Detection
+
+This is a **QLoRA adapter** fine-tuned from **Ministral-8B-Instruct-2410** for detecting security vulnerabilities in source code as part of the **UltiVal** project.
+
+## 🚨 Important Note
+
+This is a **LoRA adapter**, not a standalone model. You must load it together with the base model `mistralai/Ministral-8B-Instruct-2410`.
+
+## 📋 Model Details
+
+- **Base Model**: `mistralai/Ministral-8B-Instruct-2410`
+- **Adapter Type**: QLoRA (4-bit Low-Rank Adaptation)
+- **Training Framework**: Unsloth
+- **Task**: Security vulnerability detection in source code
+- **Model Size**: ~334MB (adapter only)
+- **Context Length**: 2048 tokens
+- **Languages**: Multi-language code analysis (Python, JavaScript, Java, C/C++, etc.)
+
+## 🎯 Training Configuration
+
+| Parameter | Value |
+|-----------|--------|
+| **Training Steps** | 6,000 (best checkpoint) |
+| **Total Steps** | 6,184 |
+| **Validation Loss** | 0.5840 (lowest achieved at step 6000) |
+| **Final Training Loss** | 0.4081 |
+| **Epochs** | 2 |
+| **Learning Rate** | 2e-4 → 1.76e-7 (cosine schedule) |
+| **Batch Size** | 8 (2 × 4 gradient accumulation) |
+| **Sequence Length** | 2048 tokens |
+| **LoRA Rank** | 32 |
+| **LoRA Alpha** | 32 |
+| **LoRA Dropout** | 0.0 |
+| **Weight Decay** | 0.01 |
+| **Warmup Steps** | ~5% of total steps |
+
+### Target Modules
+```
+q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+```
+
+## 🔧 Usage
+
+### Option 1: Using Unsloth (Recommended)
+
+```python
+from unsloth import FastLanguageModel
+import torch
+
+# Load base model
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name="mistralai/Ministral-8B-Instruct-2410",
+    max_seq_length=2048,
+    dtype=None,
+    load_in_4bit=True,
+)
+
+# Add LoRA configuration
+model = FastLanguageModel.get_peft_model(
+    model,
+    r=32,
+    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", 
+                   "gate_proj", "up_proj", "down_proj"],
+    lora_alpha=32,
+    lora_dropout=0,
+    bias="none",
+    use_gradient_checkpointing="unsloth",
+    random_state=3407,
+)
+
+# Load the trained adapter
+model.load_adapter("starsofchance/Mistral-Unsloth-QLoRA-adapter")
+
+# Enable inference mode
+FastLanguageModel.for_inference(model)
+```
+
+### Option 2: Using Transformers + PEFT
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+import torch
+
+# Load base model
+base_model = AutoModelForCausalLM.from_pretrained(
+    "mistralai/Ministral-8B-Instruct-2410",
+    torch_dtype=torch.float16,
+    device_map="auto",
+    load_in_4bit=True
+)
+
+tokenizer = AutoTokenizer.from_pretrained("mistralai/Ministral-8B-Instruct-2410")
+
+# Load LoRA adapter
+model = PeftModel.from_pretrained(base_model, "starsofchance/Mistral-Unsloth-QLoRA-adapter")
+```
+
+## 💻 Inference Example
+
+```python
+# Example: SQL Injection Detection
+code_snippet = '''
+def authenticate_user(username, password):
+    query = "SELECT * FROM users WHERE username='" + username + "' AND password='" + password + "'"
+    cursor.execute(query)
+    return cursor.fetchone()
+'''
+
+messages = [
+    {"role": "user", "content": f"Analyze this code for security vulnerabilities:\n\n{code_snippet}"}
+]
+
+# Tokenize and generate
+input_ids = tokenizer.apply_chat_template(
+    messages, 
+    add_generation_prompt=True, 
+    return_tensors="pt"
+).to(model.device)
+
+outputs = model.generate(
+    input_ids,
+    max_new_tokens=512,
+    do_sample=False,
+    pad_token_id=tokenizer.eos_token_id,
+    temperature=0.1
+)
+
+response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
+print(response)
+```
+
+### Expected Output
+```
+This code contains a critical SQL injection vulnerability. The user input (username and password) 
+is directly concatenated into the SQL query without any sanitization or parameterization.
+
+**Vulnerability Type**: SQL Injection (CWE-89)
+**Severity**: High
+**Location**: Line 2, query construction
+
+**How to exploit**: An attacker could input malicious SQL code like:
+- Username: `admin' --`
+- Password: `anything`
+
+**Secure fix**: Use parameterized queries:
+```python
+def authenticate_user(username, password):
+    query = "SELECT * FROM users WHERE username=? AND password=?"
+    cursor.execute(query, (username, password))
+    return cursor.fetchone()
+```
+```
+
+## 🛡️ Supported Vulnerability Types
+
+The model is trained to detect various security vulnerabilities including:
+
+| Category | Examples |
+|----------|----------|
+| **Injection** | SQL Injection, Command Injection, LDAP Injection |
+| **XSS** | Reflected XSS, Stored XSS, DOM-based XSS |
+| **Authentication** | Weak passwords, Authentication bypass, Session management |
+| **Authorization** | Privilege escalation, Access control issues |
+| **Cryptography** | Weak encryption, Hardcoded keys, Improper random generation |
+| **File Operations** | Path traversal, File inclusion, Unsafe deserialization |
+| **Memory Safety** | Buffer overflow, Use after free, Memory leaks |
+| **Web Security** | CSRF, SSRF, Insecure redirects |
+
+## 📊 Performance Metrics
+
+### Training Progress
+- **Initial Loss**: 1.5544
+- **Final Loss**: 0.4081
+- **Best Validation Loss**: 0.5840 (step 6000)
+- **Training Duration**: ~15 hours
+- **Convergence**: Stable convergence with cosine learning rate schedule
+
+### Hardware Requirements
+- **Training**: NVIDIA GPU with 4-bit quantization
+- **Inference**: Can run on CPU or GPU (GPU recommended for speed)
+- **Memory**: ~6GB GPU memory for inference with 4-bit quantization
+
+## 📁 Repository Structure
+
+```
+starsofchance/Mistral-Unsloth-QLoRA-adapter/
+├── adapter_config.json          # LoRA configuration
+├── adapter_model.safetensors    # Trained adapter weights (~334MB)
+├── tokenizer.json               # Tokenizer configuration
+├── tokenizer_config.json        # Tokenizer settings
+├── special_tokens_map.json      # Special tokens mapping
+└── README.md                    # This file
+```
+
+## ⚠️ Limitations
+
+1. **Adapter Dependency**: Requires the base model to function
+2. **Context Window**: Limited to 2048 tokens
+3. **Language Coverage**: Primarily trained on common programming languages
+4. **False Positives**: May flag secure code patterns as potentially vulnerable
+5. **Novel Vulnerabilities**: May not detect cutting-edge or highly obfuscated attacks
+6. **Code Context**: Performance depends on having sufficient code context
+
+## 🔄 Integration Tips
+
+### Batch Processing
+```python
+def analyze_multiple_files(code_files):
+    results = []
+    for file_path, code_content in code_files:
+        # Analyze each file
+        messages = [{"role": "user", "content": f"Analyze for vulnerabilities:\n\n{code_content}"}]
+        # ... generate response
+        results.append({"file": file_path, "analysis": response})
+    return results
+```
+
+### Custom Prompting
+```python
+# For specific vulnerability types
+prompt = f"""
+Focus on SQL injection vulnerabilities in this code:
+{code_snippet}
+
+Provide:
+1. Vulnerability assessment (Yes/No)
+2. Risk level (Low/Medium/High/Critical)  
+3. Specific location
+4. Remediation steps
+"""
+```
+
+## 📚 Training Data
+
+The model was fine-tuned on a curated dataset featuring:
+- **Real-world vulnerabilities** from CVE databases
+- **Secure code patterns** for contrast learning  
+- **Multi-language examples** across different frameworks
+- **Detailed explanations** with remediation guidance
+- **Context-rich examples** showing vulnerability in realistic scenarios
+
+## 🎓 Model Lineage
+
+```
+Ministral-8B-Instruct-2410 (Mistral AI)
+    ↓
+QLoRA Fine-tuning (Unsloth)
+    ↓  
+UltiVal Vulnerability Detection Adapter
+```
+
+## 📄 Citation
+
+If you use this model in your research or applications, please cite:
+
+```bibtex
+@misc{ultival_mistral_lora_2025,
+  title={UltiVal: Ministral-8B QLoRA Adapter for Vulnerability Detection},
+  author={StarsOfChance},
+  year={2025},
+  publisher={Hugging Face},
+  url={https://huggingface.co/starsofchance/Mistral-Unsloth-QLoRA-adapter}
+}
+```
+
+## ⚖️ License
+
+This adapter inherits the license from the base model `mistralai/Ministral-8B-Instruct-2410`. Please refer to the [base model's license](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) for specific terms and conditions.
+
+## 🙏 Acknowledgments
+
+- **Unsloth Team**: For the efficient LoRA fine-tuning framework
+- **Mistral AI**: For the powerful Ministral-8B-Instruct-2410 base model
+- **Hugging Face**: For the model hosting and PEFT library
+- **UltiVal Project**: Part of ongoing research in automated vulnerability detection
+
+## 📞 Contact & Support
+
+- **Issues**: Report bugs or issues in the [model repository](https://huggingface.co/starsofchance/Mistral-Unsloth-QLoRA-adapter/discussions)
+- **Updates**: Follow for model updates and improvements
+- **Community**: Join discussions about vulnerability detection and code security
+
+---
+
+**🔒 Security Note**: This model is designed to assist in security analysis but should not be the sole method for vulnerability assessment. Always conduct comprehensive security reviews with multiple tools and expert analysis.