Update README.md

cdcfe5e verified 25 days ago

9.4 kB

---
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- cybersecurity
- vulnerability-detection
- java
- owasp
- code-analysis
- lora
datasets:
- dilekmandal/owasp-benchmark-java-security
---
# OWASP Java Security — Qwen2.5-Coder-7B LoRA

## Model Summary

This is an experimental LoRA fine-tuned adapter based on Qwen2.5-Coder-7B-Instruct for Java source-code vulnerability classification. It is trained on a dataset derived from OWASP Benchmark Java.

The model is designed to classify Java code samples as vulnerable or not vulnerable and to produce structured JSON containing vulnerability status, CWE information, and category information.

This model is intended for research and experimentation. It is not a replacement for professional security review, static analysis, or dynamic application security testing.

## Model Details

### Model Description

- **Developed by:** Dilek Mandal
- **Shared by:** Dilek Mandal
- **Model type:** LoRA adapter for causal language modeling
- **Language(s):** Java source code; English-language instructions
- **Fine-tuned from:** [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)
- **Library:** Hugging Face Transformers and PEFT
- **Model repository:** [dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora](https://huggingface.co/dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora)
- **Training dataset:** [dilekmandal/owasp-benchmark-java-security](https://huggingface.co/datasets/dilekmandal/owasp-benchmark-java-security)

The repository contains a LoRA adapter rather than a standalone copy of the complete base model. The base model is required for inference.

## Intended Uses

### Direct Use

The adapter can be loaded with its base model to experiment with Java vulnerability classification. The intended output is a structured JSON response identifying whether a sample is vulnerable and, where applicable, its CWE and category.

### Downstream Use

Potential applications include:
- Research into machine-learning-based software vulnerability detection.
- Educational demonstrations of Java security weaknesses.
- Preliminary security triage of Java source code.
- Comparisons of code-security fine-tuning approaches.

Any integration into a development or security workflow should include independent validation and human review.

### Out-of-Scope Use

This model should not be used:
- As the sole mechanism for approving code as secure.
- As a replacement for security professionals or established security testing tools.
- To make claims of production-level vulnerability detection without independent evaluation.
- As evidence that code is free from vulnerabilities when the model predicts that it is not vulnerable.

## Bias, Risks, and Limitations

The model learns patterns from its training data and may fail to generalize to code that differs from the benchmark examples.

Potential limitations include:
- False negatives: vulnerabilities may be missed.
- False positives: safe code may be classified as vulnerable.
- Incorrect CWE or category predictions.
- Sensitivity to code formatting, context, and prompt wording.
- Limited evidence of performance on real-world applications.

### Dataset Leakage and Evaluation Limitations

Dataset auditing identified exact normalized-code overlaps and shared code families across training, validation, and test splits. As a result, the evaluation may be optimistic because related examples can occur in more than one split.

The reported evaluation results should therefore be interpreted as preliminary benchmark results, not proof of generalization to unseen software.

A more reliable estimate requires deduplication, family-based splitting, and evaluation on an independent dataset.

### Recommendations

Users should:
- Review predictions before taking security-related action.
- Combine model outputs with static analysis, dynamic testing, and manual review.
- Evaluate the model on independently collected code before operational deployment.
- Avoid interpreting a non-vulnerable prediction as proof that code is secure.

## How to Get Started

Install the required packages:

```bash
pip install torch transformers peft accelerate
```

Load the base model and adapter:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "Qwen/Qwen2.5-Coder-7B-Instruct"
adapter_id = "dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype="auto",
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()
java_code = """
public class Example {
    public String query(String input) {
        return "SELECT * FROM users WHERE name = '" + input + "'";
    }
}
"""
messages = [
    {
        "role": "system",
        "content": (
            "You are a Java application security analyzer. "
            "Return only JSON with vulnerable, cwe, and category."
        )
    },
    {
        "role": "user",
        "content": (
            "Analyze the following Java source code:\n"
            f"```java\n{java_code}\n```"
        )
    }
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False
    )
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))
```
The example demonstrates the intended interface. The actual prediction is not guaranteed to match any illustrative output.
## Training Details
### Training Data
The training dataset is derived from OWASP Benchmark Java.
- **Dataset repository:** [dilekmandal/owasp-benchmark-java-security](https://huggingface.co/datasets/dilekmandal/owasp-benchmark-java-security)
- **Total examples:** 21,041
- **Training examples:** 16,849
- **Validation examples:** 2,096
- **Test examples:** 2,096
The examples use a chat-style structure with Java source code as input and a structured classification target.
The dataset includes vulnerable and non-vulnerable samples across multiple CWE categories. Refer to the dataset card and the original [OWASP Benchmark project](https://owasp.org/www-project-benchmark/) for further context.
### Preprocessing
The source data was converted to JSON Lines (JSONL) examples using chat messages. The instruction requests JSON output containing `vulnerable`, `cwe`, and `category`.
Dataset auditing was performed before model publication. The identified cross-split code overlaps are documented in the limitations section above.
### Training Procedure
The model was fine-tuned using LoRA. Detailed hyperparameters, training duration, and compute measurements are not reported here because they have not been independently documented in this card.
## Evaluation
### Testing Data
The initial evaluation used the test split of the prepared dataset. During dataset auditing, exact normalized-code overlaps and shared code families were identified across the training, validation, and test splits.
Consequently, the initial test results may overestimate the model's generalization performance.
### Evaluation Metrics
The following metrics are relevant for evaluating this classification task:
Accuracy
Precision
Recall
F1 score
CWE-level evaluation is also necessary to assess whether the model correctly identifies specific vulnerability types, rather than only predicting whether a code sample is vulnerable.
### Evaluation Status
Initial local evaluation experiments have been conducted. However, due to cross-split code overlap and shared code families, the existing results are not reported here as reliable estimates of generalization performance.
Further evaluation is needed using leakage-resistant, code-family-based data splits and an independent test set. CWE-level error analysis is also planned to better understand the model's strengths and limitations.
The model should be considered an experimental research artifact, not a validated security analysis tool.
## Technical Specifications
### Model Architecture and Objective
The adapter fine-tunes a causal language model using parameter-efficient LoRA adaptation. The objective is to generate structured vulnerability-classification responses from Java source code.
### Compute Infrastructure
Hardware, training duration, energy consumption, and software versions used during the complete training run have not been fully documented in this model card.
## Licensing
Review the base model's license and the original dataset's licensing terms before redistributing or using this adapter. No license is asserted here until compatibility and redistribution permissions have been verified.
- Base model: [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)
- Original benchmark: [OWASP Benchmark Java](https://github.com/OWASP-Benchmark/BenchmarkJava)
## Acknowledgments
This project uses the OWASP Benchmark Java project as a source of benchmark data and Qwen2.5-Coder-7B-Instruct as its base model.
## Model Card Authors
Dilek Mandal

	---
	base_model: Qwen/Qwen2.5-Coder-7B-Instruct
	library_name: peft
	pipeline_tag: text-generation
	tags:
	- cybersecurity
	- vulnerability-detection
	- java
	- owasp
	- code-analysis
	- lora
	datasets:
	- dilekmandal/owasp-benchmark-java-security
	---

	# OWASP Java Security — Qwen2.5-Coder-7B LoRA

	## Model Summary

	This is an experimental LoRA fine-tuned adapter based on Qwen2.5-Coder-7B-Instruct for Java source-code vulnerability classification. It is trained on a dataset derived from OWASP Benchmark Java.

	The model is designed to classify Java code samples as vulnerable or not vulnerable and to produce structured JSON containing vulnerability status, CWE information, and category information.

	This model is intended for research and experimentation. It is not a replacement for professional security review, static analysis, or dynamic application security testing.

	## Model Details

	### Model Description

	- Developed by: Dilek Mandal
	- Shared by: Dilek Mandal
	- Model type: LoRA adapter for causal language modeling
	- Language(s): Java source code; English-language instructions
	- Fine-tuned from: [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)
	- Library: Hugging Face Transformers and PEFT
	- Model repository: [dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora](https://huggingface.co/dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora)
	- Training dataset: [dilekmandal/owasp-benchmark-java-security](https://huggingface.co/datasets/dilekmandal/owasp-benchmark-java-security)

	The repository contains a LoRA adapter rather than a standalone copy of the complete base model. The base model is required for inference.

	## Intended Uses

	### Direct Use

	The adapter can be loaded with its base model to experiment with Java vulnerability classification. The intended output is a structured JSON response identifying whether a sample is vulnerable and, where applicable, its CWE and category.

	### Downstream Use

	Potential applications include:
	- Research into machine-learning-based software vulnerability detection.
	- Educational demonstrations of Java security weaknesses.
	- Preliminary security triage of Java source code.
	- Comparisons of code-security fine-tuning approaches.

	Any integration into a development or security workflow should include independent validation and human review.

	### Out-of-Scope Use

	This model should not be used:
	- As the sole mechanism for approving code as secure.
	- As a replacement for security professionals or established security testing tools.
	- To make claims of production-level vulnerability detection without independent evaluation.
	- As evidence that code is free from vulnerabilities when the model predicts that it is not vulnerable.

	## Bias, Risks, and Limitations

	The model learns patterns from its training data and may fail to generalize to code that differs from the benchmark examples.

	Potential limitations include:
	- False negatives: vulnerabilities may be missed.
	- False positives: safe code may be classified as vulnerable.
	- Incorrect CWE or category predictions.
	- Sensitivity to code formatting, context, and prompt wording.
	- Limited evidence of performance on real-world applications.

	### Dataset Leakage and Evaluation Limitations

	Dataset auditing identified exact normalized-code overlaps and shared code families across training, validation, and test splits. As a result, the evaluation may be optimistic because related examples can occur in more than one split.

	The reported evaluation results should therefore be interpreted as preliminary benchmark results, not proof of generalization to unseen software.

	A more reliable estimate requires deduplication, family-based splitting, and evaluation on an independent dataset.

	### Recommendations

	Users should:
	- Review predictions before taking security-related action.
	- Combine model outputs with static analysis, dynamic testing, and manual review.
	- Evaluate the model on independently collected code before operational deployment.
	- Avoid interpreting a non-vulnerable prediction as proof that code is secure.

	## How to Get Started

	Install the required packages:

	```bash
	pip install torch transformers peft accelerate
	```

	Load the base model and adapter:

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	from peft import PeftModel

	base_model_id = "Qwen/Qwen2.5-Coder-7B-Instruct"
	adapter_id = "dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora"

	tokenizer = AutoTokenizer.from_pretrained(base_model_id)

	base_model = AutoModelForCausalLM.from_pretrained(
	base_model_id,
	torch_dtype="auto",
	device_map="auto"
	)

	model = PeftModel.from_pretrained(base_model, adapter_id)
	model.eval()

	java_code = """
	public class Example {
	public String query(String input) {
	return "SELECT * FROM users WHERE name = '" + input + "'";
	}
	}
	"""

	messages = [
	{
	"role": "system",
	"content": (
	"You are a Java application security analyzer. "
	"Return only JSON with vulnerable, cwe, and category."
	)
	},
	{
	"role": "user",
	"content": (
	"Analyze the following Java source code:\n"
	f"```java\n{java_code}\n```"
	)
	}
	]

	prompt = tokenizer.apply_chat_template(
	messages,
	tokenize=False,
	add_generation_prompt=True
	)

	inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

	with torch.no_grad():
	outputs = model.generate(
	**inputs,
	max_new_tokens=256,
	do_sample=False
	)

	new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
	print(tokenizer.decode(new_tokens, skip_special_tokens=True))
	```

	The example demonstrates the intended interface. The actual prediction is not guaranteed to match any illustrative output.

	## Training Details

	### Training Data

	The training dataset is derived from OWASP Benchmark Java.

	- Dataset repository: [dilekmandal/owasp-benchmark-java-security](https://huggingface.co/datasets/dilekmandal/owasp-benchmark-java-security)
	- Total examples: 21,041
	- Training examples: 16,849
	- Validation examples: 2,096
	- Test examples: 2,096

	The examples use a chat-style structure with Java source code as input and a structured classification target.

	The dataset includes vulnerable and non-vulnerable samples across multiple CWE categories. Refer to the dataset card and the original [OWASP Benchmark project](https://owasp.org/www-project-benchmark/) for further context.

	### Preprocessing

	The source data was converted to JSON Lines (JSONL) examples using chat messages. The instruction requests JSON output containing `vulnerable`, `cwe`, and `category`.

	Dataset auditing was performed before model publication. The identified cross-split code overlaps are documented in the limitations section above.

	### Training Procedure

	The model was fine-tuned using LoRA. Detailed hyperparameters, training duration, and compute measurements are not reported here because they have not been independently documented in this card.

	## Evaluation

	### Testing Data

	The initial evaluation used the test split of the prepared dataset. During dataset auditing, exact normalized-code overlaps and shared code families were identified across the training, validation, and test splits.

	Consequently, the initial test results may overestimate the model's generalization performance.

	### Evaluation Metrics

	The following metrics are relevant for evaluating this classification task:

	Accuracy

	Precision

	Recall

	F1 score

	CWE-level evaluation is also necessary to assess whether the model correctly identifies specific vulnerability types, rather than only predicting whether a code sample is vulnerable.

	### Evaluation Status

	Initial local evaluation experiments have been conducted. However, due to cross-split code overlap and shared code families, the existing results are not reported here as reliable estimates of generalization performance.

	Further evaluation is needed using leakage-resistant, code-family-based data splits and an independent test set. CWE-level error analysis is also planned to better understand the model's strengths and limitations.

	The model should be considered an experimental research artifact, not a validated security analysis tool.

	## Technical Specifications

	### Model Architecture and Objective

	The adapter fine-tunes a causal language model using parameter-efficient LoRA adaptation. The objective is to generate structured vulnerability-classification responses from Java source code.

	### Compute Infrastructure

	Hardware, training duration, energy consumption, and software versions used during the complete training run have not been fully documented in this model card.

	## Licensing

	Review the base model's license and the original dataset's licensing terms before redistributing or using this adapter. No license is asserted here until compatibility and redistribution permissions have been verified.

	- Base model: [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)
	- Original benchmark: [OWASP Benchmark Java](https://github.com/OWASP-Benchmark/BenchmarkJava)

	## Acknowledgments

	This project uses the OWASP Benchmark Java project as a source of benchmark data and Qwen2.5-Coder-7B-Instruct as its base model.

	## Model Card Authors

	Dilek Mandal