ArshVerma commited on
Commit
842d9be
·
1 Parent(s): 3fc3cc7

Add highly detailed Model Card

Browse files
Files changed (1) hide show
  1. MODEL_CARD.md +120 -0
MODEL_CARD.md ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model Summary
2
+
3
+ **CodeLens-Reviewer** is an AI agent fine-tuned to act as a Senior Code Reviewer. Built upon the Qwen2.5-Coder (7B) architecture, this model has been instruction-tuned to autonomously detect bugs, identify security vulnerabilities, and critique architectural flaws in Python Pull Requests. It interacts directly with the CodeLens Evaluation Environment, outputting perfectly structured JSON actions to simulate an automated code review loop.
4
+
5
+ ## Usage
6
+
7
+ This model is intended to be used either standalone for code inference or plugged into the CodeLens Evaluation Environment.
8
+
9
+ ### Inference with Unsloth
10
+ ```python
11
+ from unsloth import FastLanguageModel
12
+
13
+ model, tokenizer = FastLanguageModel.from_pretrained(
14
+ model_name = "ArshVerma/codelens_reviewer_lora",
15
+ max_seq_length = 2048,
16
+ load_in_4bit = True,
17
+ )
18
+ FastLanguageModel.for_inference(model)
19
+
20
+ prompt = """You are an expert code reviewer. Review the following code and output a JSON action indicating any bugs or issues you find.
21
+ Code:
22
+ ```python
23
+ def process(data):
24
+ for i in data:
25
+ if i == "remove": data.remove(i)
26
+ return data
27
+ ```
28
+ """
29
+
30
+ messages = [
31
+ {"role": "system", "content": "You are an expert code reviewer. Output only valid JSON."},
32
+ {"role": "user", "content": prompt}
33
+ ]
34
+
35
+ inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
36
+ outputs = model.generate(input_ids=inputs, max_new_tokens=256)
37
+ print(tokenizer.batch_decode(outputs)[0])
38
+ ```
39
+
40
+ **Shape of Inputs/Outputs:**
41
+ - **Input:** Natural language instructions and a python code snippet (prompt string).
42
+ - **Output:** A strict JSON object containing `action`, `issue_description`, `filename`, `line_number`, `severity`, and `category`.
43
+
44
+ **Known Failures:** The model may hallucinate specific line numbers if the provided code diff is extremely long or poorly formatted.
45
+
46
+ ## System
47
+
48
+ This model serves as the core "Agent" in the **CodeLens Evaluation System**. It receives system prompts, noise budgets, and task definitions from the CodeLens Python backend. Its downstream dependency is the CodeLens WebSocket dashboard, which parses the JSON outputs to assign rewards (positive/negative) and update the live leaderboard.
49
+
50
+ ## Implementation requirements
51
+
52
+ - **Hardware:** Trained on a single NVIDIA Tesla T4 x2 GPU via Kaggle.
53
+ - **Software:** Unsloth, PyTorch, Hugging Face Transformers, TRL.
54
+ - **Compute:** Fine-tuning took less than 15 minutes due to Unsloth's highly optimized 4-bit LoRA training pipeline. Inference requires < 6GB of VRAM, making it incredibly lightweight and accessible.
55
+
56
+ ---
57
+
58
+ # Model Characteristics
59
+
60
+ ## Model initialization
61
+
62
+ The model was fine-tuned from the pre-trained `unsloth/Qwen2.5-Coder-7B-Instruct` base model.
63
+
64
+ ## Model stats
65
+
66
+ - **Size:** 7 Billion Parameters.
67
+ - **Weights:** LoRA adapters applied to Q, K, V, O, gate, up, and down projections.
68
+ - **Latency:** Highly optimized for real-time inference (2x faster than native Hugging Face using Unsloth).
69
+
70
+ ## Other details
71
+
72
+ - **Quantization:** The model was trained and quantized in 4-bit (bitsandbytes) to severely reduce VRAM requirements.
73
+ - **Gradient Checkpointing:** Unsloth's exact gradient checkpointing was used to save memory.
74
+
75
+ ---
76
+
77
+ # Data Overview
78
+
79
+ The model was trained on a highly specific, synthetically generated instructional dataset targeting code review tasks.
80
+
81
+ ## Training data
82
+
83
+ The training data (`dataset.jsonl`) contains custom prompt-completion pairs. Each row simulates a CodeLens environment state containing:
84
+ - A `pr_title` and `pr_description`.
85
+ - A code `diff` containing planted bugs, security flaws, or architectural issues.
86
+ - A golden `JSON` completion representing the ideal code review action.
87
+
88
+ ## Demographic groups
89
+
90
+ N/A. This model processes programming logic and code structure.
91
+
92
+ ## Evaluation data
93
+
94
+ The model was evaluated iteratively against the live CodeLens engine by observing its capability to identify 3 distinct tasks: `bug_detection`, `security_audit`, and `architectural_review`.
95
+
96
+ ---
97
+
98
+ # Evaluation Results
99
+
100
+ ## Summary
101
+
102
+ The model successfully learned the JSON formatting constraints and correctly identifies logic bugs (e.g., modifying arrays during iteration) without hallucinating markdown wrappers.
103
+
104
+ ## Subgroup evaluation results
105
+
106
+ - **Bug Detection:** High accuracy. Successfully identifies off-by-one and logical loop errors.
107
+ - **Security Audit:** Capable of identifying basic input sanitization failures.
108
+ - **Architecture:** Demonstrates baseline capability in flagging Single Responsibility Principle (SRP) violations.
109
+
110
+ ## Fairness
111
+
112
+ N/A. Evaluation was strictly based on binary reward functions (detecting the objective bug = +1 reward).
113
+
114
+ ## Usage limitations
115
+
116
+ This model is intended strictly for evaluating code within isolated sandboxes or assisting developers. It should **not** be used to automatically reject or approve production Pull Requests without human oversight, as false positives are possible.
117
+
118
+ ## Ethics
119
+
120
+ No sensitive or PII data was used during the training of this model. Risks of adversarial exploitation exist if malicious code is embedded to trigger arbitrary outputs, but the model outputs strictly to JSON, minimizing execution risks.