File size: 3,407 Bytes
5864958 0a4c29b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
base_model: microsoft/Phi-3-mini-4k-instruct
library_name: peft
datasets:
- mbpp
- google/code_x_glue_ct_code_to_text
language:
- en
- py
tags:
- code-generation
- docstring-generation
- code-review
- bilora
---
# Phi-3 BiLoRA Code Review
This model is a fine-tuned version of `microsoft/Phi-3-mini-4k-instruct` using BiLoRA (Dual-Adapter LoRA) for code review tasks, specifically code generation and docstring generation.
## Model Details
- **Model Type:** Causal Language Model with multiple LoRA adapters
- **Base Model:** [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)
- **Adapters:**
- `task_1`: Code Generation (fine-tuned on MBPP)
- `task_2`: Docstring Generation (fine-tuned on CodeXGLUE)
- **Language(s):** Python
## Intended Use
This model is intended for code review assistance, including:
- Generating Python code from natural language prompts.
- Generating descriptive docstrings for existing Python functions.
## Training Details
### Dataset
- **Task 1:** [MBPP](https://huggingface.co/datasets/mbpp) (Mostly Basic Python Problems)
- **Task 2:** [CodeXGLUE (ct-code-to-text)](https://huggingface.co/datasets/google/code_x_glue_ct_code_to_text) (Python subset)
### BiLoRA Configuration
- **Rank (r):** 4
- **Alpha:** 8
- **Dropout:** 0.1
- **Target Modules:**
- `task_1`: `qkv_proj`, `o_proj`
- `task_2`: `gate_up_proj`, `down_proj`
### Hyperparameters
- **Learning Rate:** 2e-4
- **Batch Size:** 1 (Gradient Accumulation Steps: 16)
- **Epochs:** 1
- **Optimizer:** AdamW
- **LR Scheduler:** Linear
## Benchmark Results
Evaluation performed on a custom benchmark of 20 samples (10 code gen, 10 docstring gen).
| Model | Bug Detection (Pass@1) | Localization (BLEU) | Fix Quality (1-5) | Latency (avg) |
|-------|--------------|--------------|-------------|---------|
| BiLoRA (mine) | 94.17% | 0.0259 | 3.7/5 | 33499ms |
| Phi-3 base | 70.0% | 0.0536 | 3.6/5 | 24561ms |
| GPT-4 (Groq) | 100.0% | 0.1255 | 4.4/5 | 433ms |
*Note: Bug Detection is proxied by Code Generation Pass Rate. Localization is proxied by Docstring BLEU score. Fix Quality is an average quality score (1-5).*
## Example Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = "microsoft/Phi-3-mini-4k-instruct"
model = AutoModelForCausalLM.from_pretrained(base_model, trust_remote_code=True)
model = PeftModel.from_pretrained(model, "aniketp2009gmail/phi3-bilora-code-review")
tokenizer = AutoTokenizer.from_pretrained("aniketp2009gmail/phi3-bilora-code-review")
# For Code Generation (Task 1)
model.set_adapter("task_1")
prompt = "Generate code: Write a function to find the sum of even numbers in a list\nCode:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# For Docstring Generation (Task 2)
model.set_adapter("task_2")
prompt = "Generate docstring: def sum_even(lst):\n return sum(x for x in lst if x % 2 == 0)\nDocstring:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Limitations
- The model is optimized for Python.
- Performance may vary on complex or niche libraries.
- Latency is higher than quantized or distilled models.
|