README.md

---
base_model: mistralai/Ministral-8B-Instruct-2410
tags:
- unsloth
- lora
- qlora
- vulnerability-detection
- security
- code-analysis
- cybersecurity
- ultival
- peft
- adapter
language:
- en
license: apache-2.0
library_name: peft
pipeline_tag: text-generation
---

# UltiVal: Ministral-8B QLoRA Adapter for Vulnerability Detection

This is a **QLoRA adapter** fine-tuned from **Ministral-8B-Instruct-2410** for detecting security vulnerabilities in source code as part of the **UltiVal** project.

## 🚨 Important Note

This is a **LoRA adapter**, not a standalone model. You must load it together with the base model `mistralai/Ministral-8B-Instruct-2410`.

## 📋 Model Details

- **Base Model**: `mistralai/Ministral-8B-Instruct-2410`
- **Adapter Type**: QLoRA (4-bit Low-Rank Adaptation)
- **Training Framework**: Unsloth
- **Task**: Security vulnerability detection in source code
- **Model Size**: ~334MB (adapter only)
- **Context Length**: 2048 tokens
- **Languages**: Multi-language code analysis (Python, JavaScript, Java, C/C++, etc.)

## 🎯 Training Configuration

| Parameter | Value |
|-----------|--------|
| **Training Steps** | 6,000 (best checkpoint) |
| **Total Steps** | 6,184 |
| **Validation Loss** | 0.5840 (lowest achieved at step 6000) |
| **Final Training Loss** | 0.4081 |
| **Epochs** | 2 |
| **Learning Rate** | 2e-4 → 1.76e-7 (cosine schedule) |
| **Batch Size** | 8 (2 × 4 gradient accumulation) |
| **Sequence Length** | 2048 tokens |
| **LoRA Rank** | 32 |
| **LoRA Alpha** | 32 |
| **LoRA Dropout** | 0.0 |
| **Weight Decay** | 0.01 |
| **Warmup Steps** | ~5% of total steps |

### Target Modules
```
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
```

## 🔧 Usage

### Option 1: Using Unsloth (Recommended)

```python
from unsloth import FastLanguageModel
import torch

# Load base model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="mistralai/Ministral-8B-Instruct-2410",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Add LoRA configuration
model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", 
                   "gate_proj", "up_proj", "down_proj"],
    lora_alpha=32,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

# Load the trained adapter
model.load_adapter("starsofchance/Mistral-Unsloth-QLoRA-adapter")

# Enable inference mode
FastLanguageModel.for_inference(model)
```

### Option 2: Using Transformers + PEFT

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Ministral-8B-Instruct-2410",
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=True
)

tokenizer = AutoTokenizer.from_pretrained("mistralai/Ministral-8B-Instruct-2410")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "starsofchance/Mistral-Unsloth-QLoRA-adapter")
```

## 💻 Inference Example

```python
# Example: SQL Injection Detection
code_snippet = '''
def authenticate_user(username, password):
    query = "SELECT * FROM users WHERE username='" + username + "' AND password='" + password + "'"
    cursor.execute(query)
    return cursor.fetchone()
'''

messages = [
    {"role": "user", "content": f"Analyze this code for security vulnerabilities:\n\n{code_snippet}"}
]

# Tokenize and generate
input_ids = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
    temperature=0.1
)

response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
print(response)
```

### Expected Output
```
This code contains a critical SQL injection vulnerability. The user input (username and password) 
is directly concatenated into the SQL query without any sanitization or parameterization.

**Vulnerability Type**: SQL Injection (CWE-89)
**Severity**: High
**Location**: Line 2, query construction

**How to exploit**: An attacker could input malicious SQL code like:
- Username: `admin' --`
- Password: `anything`

**Secure fix**: Use parameterized queries:
```python
def authenticate_user(username, password):
    query = "SELECT * FROM users WHERE username=? AND password=?"
    cursor.execute(query, (username, password))
    return cursor.fetchone()
```
```

## 🛡️ Supported Vulnerability Types

The model is trained to detect various security vulnerabilities including:

| Category | Examples |
|----------|----------|
| **Injection** | SQL Injection, Command Injection, LDAP Injection |
| **XSS** | Reflected XSS, Stored XSS, DOM-based XSS |
| **Authentication** | Weak passwords, Authentication bypass, Session management |
| **Authorization** | Privilege escalation, Access control issues |
| **Cryptography** | Weak encryption, Hardcoded keys, Improper random generation |
| **File Operations** | Path traversal, File inclusion, Unsafe deserialization |
| **Memory Safety** | Buffer overflow, Use after free, Memory leaks |
| **Web Security** | CSRF, SSRF, Insecure redirects |

## 📊 Performance Metrics

### Training Progress
- **Initial Loss**: 1.5544
- **Final Loss**: 0.4081
- **Best Validation Loss**: 0.5840 (step 6000)
- **Training Duration**: ~15 hours
- **Convergence**: Stable convergence with cosine learning rate schedule

### Hardware Requirements
- **Training**: NVIDIA GPU with 4-bit quantization
- **Inference**: Can run on CPU or GPU (GPU recommended for speed)
- **Memory**: ~6GB GPU memory for inference with 4-bit quantization

## 📁 Repository Structure

```
starsofchance/Mistral-Unsloth-QLoRA-adapter/
├── adapter_config.json          # LoRA configuration
├── adapter_model.safetensors    # Trained adapter weights (~334MB)
├── tokenizer.json               # Tokenizer configuration
├── tokenizer_config.json        # Tokenizer settings
├── special_tokens_map.json      # Special tokens mapping
└── README.md                    # This file
```

## ⚠️ Limitations

1. **Adapter Dependency**: Requires the base model to function
2. **Context Window**: Limited to 2048 tokens
3. **Language Coverage**: Primarily trained on common programming languages
4. **False Positives**: May flag secure code patterns as potentially vulnerable
5. **Novel Vulnerabilities**: May not detect cutting-edge or highly obfuscated attacks
6. **Code Context**: Performance depends on having sufficient code context

## 🔄 Integration Tips

### Batch Processing
```python
def analyze_multiple_files(code_files):
    results = []
    for file_path, code_content in code_files:
        # Analyze each file
        messages = [{"role": "user", "content": f"Analyze for vulnerabilities:\n\n{code_content}"}]
        # ... generate response
        results.append({"file": file_path, "analysis": response})
    return results
```

### Custom Prompting
```python
# For specific vulnerability types
prompt = f"""
Focus on SQL injection vulnerabilities in this code:
{code_snippet}

Provide:
1. Vulnerability assessment (Yes/No)
2. Risk level (Low/Medium/High/Critical)  
3. Specific location
4. Remediation steps
"""
```

## 📚 Training Data

The model was fine-tuned on a curated dataset featuring:
- **Real-world vulnerabilities** from CVE databases
- **Secure code patterns** for contrast learning  
- **Multi-language examples** across different frameworks
- **Detailed explanations** with remediation guidance
- **Context-rich examples** showing vulnerability in realistic scenarios

## 🎓 Model Lineage

```
Ministral-8B-Instruct-2410 (Mistral AI)
    ↓
QLoRA Fine-tuning (Unsloth)
    ↓  
UltiVal Vulnerability Detection Adapter
```

## 📄 Citation

If you use this model in your research or applications, please cite:

```bibtex
@misc{ultival_mistral_lora_2025,
  title={UltiVal: Ministral-8B QLoRA Adapter for Vulnerability Detection},
  author={StarsOfChance},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/starsofchance/Mistral-Unsloth-QLoRA-adapter}
}
```

## ⚖️ License

This adapter inherits the license from the base model `mistralai/Ministral-8B-Instruct-2410`. Please refer to the [base model's license](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) for specific terms and conditions.

## 🙏 Acknowledgments

- **Unsloth Team**: For the efficient LoRA fine-tuning framework
- **Mistral AI**: For the powerful Ministral-8B-Instruct-2410 base model
- **Hugging Face**: For the model hosting and PEFT library
- **UltiVal Project**: Part of ongoing research in automated vulnerability detection

## 📞 Contact & Support

- **Issues**: Report bugs or issues in the [model repository](https://huggingface.co/starsofchance/Mistral-Unsloth-QLoRA-adapter/discussions)
- **Updates**: Follow for model updates and improvements
- **Community**: Join discussions about vulnerability detection and code security

---

**🔒 Security Note**: This model is designed to assist in security analysis but should not be the sole method for vulnerability assessment. Always conduct comprehensive security reviews with multiple tools and expert analysis.