|
|
---
|
|
|
license: mit
|
|
|
tags:
|
|
|
- codellama
|
|
|
- linux
|
|
|
- bugfix
|
|
|
- lora
|
|
|
- qlora
|
|
|
- git-diff
|
|
|
base_model: codellama/CodeLLaMA-7b-Instruct-hf
|
|
|
model_type: LlamaForCausalLM
|
|
|
library_name: peft
|
|
|
pipeline_tag: text-generation
|
|
|
---
|
|
|
|
|
|
# CodeLLaMA-Linux-BugFix
|
|
|
|
|
|
A fine-tuned version of `CodeLLaMA-7B-Instruct`, designed specifically for Linux kernel bug fixing using QLoRA (Quantized Low-Rank Adaptation). The model learns to generate Git diff patches based on buggy C code and commit messages.
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🎯 Overview
|
|
|
|
|
|
This project targets automated Linux kernel bug fixing by:
|
|
|
|
|
|
- **Mining real commit data** from the kernel Git history
|
|
|
- **Training a specialized QLoRA model** on diff-style fixes
|
|
|
- **Generating Git patches** in response to bug-prone code
|
|
|
- **Evaluating results** using BLEU, ROUGE, and human inspection
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🧠 Model Configuration
|
|
|
|
|
|
- **Base model**: `CodeLLaMA-7B-Instruct`
|
|
|
- **Fine-tuning method**: QLoRA with 4-bit quantization
|
|
|
- **Training setup**:
|
|
|
- LoRA r=64, alpha=16, dropout=0.1
|
|
|
- Batch size: 64, LR: 2e-4, Epochs: 3
|
|
|
- Mixed precision (bfloat16), gradient checkpointing
|
|
|
- **Hardware**: Optimized for NVIDIA H200 GPUs
|
|
|
|
|
|
---
|
|
|
|
|
|
## 📊 Dataset
|
|
|
|
|
|
Custom dataset extracted from Linux kernel Git history.
|
|
|
|
|
|
### Filtering Criteria
|
|
|
Bug-fix commits containing:
|
|
|
`fix`, `bug`, `crash`, `memory`, `null`, `panic`, `overflow`, `race`, `corruption`, etc.
|
|
|
|
|
|
### Structure
|
|
|
- Language: C (`.c`, `.h`)
|
|
|
- Context: 10 lines before/after the change
|
|
|
- Format:
|
|
|
|
|
|
```json
|
|
|
{
|
|
|
"input": {
|
|
|
"original code": "C code snippet with bug",
|
|
|
"instruction": "Commit message or fix description"
|
|
|
},
|
|
|
"output": {
|
|
|
"diff codes": "Git diff showing the fix"
|
|
|
}
|
|
|
}
|
|
|
````
|
|
|
|
|
|
* **File**: `training_data_100k.jsonl` (100,000 samples)
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🚀 Quick Start
|
|
|
|
|
|
### Install dependencies
|
|
|
|
|
|
```bash
|
|
|
pip install -r requirements.txt
|
|
|
```
|
|
|
|
|
|
### 1. Build the Dataset
|
|
|
|
|
|
```bash
|
|
|
cd dataset_builder
|
|
|
python extract_linux_bugfixes.py
|
|
|
python format_for_training.py
|
|
|
```
|
|
|
|
|
|
### 2. Fine-tune the Model
|
|
|
|
|
|
```bash
|
|
|
cd train
|
|
|
python train_codellama_qlora_linux_bugfix.py
|
|
|
```
|
|
|
|
|
|
### 3. Run Evaluation
|
|
|
|
|
|
```bash
|
|
|
cd evaluate
|
|
|
python evaluate_linux_bugfix_model.py
|
|
|
```
|
|
|
|
|
|
---
|
|
|
|
|
|
## 📁 Project Structure
|
|
|
|
|
|
```
|
|
|
CodeLLaMA-Linux-BugFix/
|
|
|
├── dataset_builder/
|
|
|
│ ├── extract_linux_bugfixes.py
|
|
|
│ ├── extract_linux_bugfixes_parallel.py
|
|
|
│ └── format_for_training.py
|
|
|
├── dataset/
|
|
|
│ ├── training_data_100k.jsonl
|
|
|
│ └── training_data_prompt_completion.jsonl
|
|
|
├── train/
|
|
|
│ ├── train_codellama_qlora_linux_bugfix.py
|
|
|
│ ├── train_codellama_qlora_simple.py
|
|
|
│ ├── download_codellama_model.py
|
|
|
│ └── output/
|
|
|
├── evaluate/
|
|
|
│ ├── evaluate_linux_bugfix_model.py
|
|
|
│ ├── test_samples.jsonl
|
|
|
│ └── output/
|
|
|
└── requirements.txt
|
|
|
```
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🧩 Features
|
|
|
|
|
|
* 🔧 **Efficient Fine-tuning**: QLoRA + 4-bit quant = massive memory savings
|
|
|
* 🧠 **Real-world commits**: From actual Linux kernel development
|
|
|
* 💡 **Context-aware**: Code context extraction around bug lines
|
|
|
* 💻 **Output-ready**: Generates valid Git-style diffs
|
|
|
|
|
|
---
|
|
|
|
|
|
## 📈 Evaluation Metrics
|
|
|
|
|
|
* **BLEU**: Translation-style match to reference diffs
|
|
|
* **ROUGE**: Overlap in fix content
|
|
|
* **Human Evaluation**: Subjective patch quality
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🧪 Use Cases
|
|
|
|
|
|
* Automated kernel bug fixing
|
|
|
* Code review assistance
|
|
|
* Teaching/debugging kernel code
|
|
|
* Research in automated program repair (APR)
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🔬 Technical Highlights
|
|
|
|
|
|
### Memory & Speed Optimizations
|
|
|
|
|
|
* 4-bit quantization (NF4)
|
|
|
* Gradient checkpointing
|
|
|
* Mixed precision (bfloat16)
|
|
|
* Gradient accumulation
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🤝 Contributing
|
|
|
|
|
|
1. Fork this repo
|
|
|
2. Create a branch
|
|
|
3. Add your feature or fix
|
|
|
4. Submit a PR 🙌
|
|
|
|
|
|
---
|
|
|
|
|
|
## 📄 License
|
|
|
|
|
|
MIT License – see `LICENSE` file for details.
|
|
|
|
|
|
---
|
|
|
|
|
|
## 🙏 Acknowledgments
|
|
|
|
|
|
* Meta for CodeLLaMA
|
|
|
* Hugging Face for Transformers + PEFT
|
|
|
* The Linux kernel community for open access to commit data
|
|
|
* Microsoft for introducing LoRA
|
|
|
|
|
|
---
|
|
|
|
|
|
## 📚 References
|
|
|
|
|
|
* [CodeLLaMA (Meta, 2023)](https://arxiv.org/abs/2308.12950)
|
|
|
* [QLoRA (Dettmers et al., 2023)](https://arxiv.org/abs/2305.14314)
|
|
|
* [LoRA (Hu et al., 2021)](https://arxiv.org/abs/2106.09685)
|
|
|
|
