| --- | |
| license: mit | |
| tags: | |
| - codellama | |
| - linux | |
| - bugfix | |
| - lora | |
| - qlora | |
| - git-diff | |
| base_model: codellama/CodeLLaMA-7b-Instruct-hf | |
| model_type: LlamaForCausalLM | |
| library_name: peft | |
| pipeline_tag: text-generation | |
| --- | |
| # CodeLLaMA-Linux-BugFix | |
| A fine-tuned version of `CodeLLaMA-7B-Instruct`, designed specifically for Linux kernel bug fixing using QLoRA (Quantized Low-Rank Adaptation). The model learns to generate Git diff patches based on buggy C code and commit messages. | |
| --- | |
| ## 🎯 Overview | |
| This project targets automated Linux kernel bug fixing by: | |
| - **Mining real commit data** from the kernel Git history | |
| - **Training a specialized QLoRA model** on diff-style fixes | |
| - **Generating Git patches** in response to bug-prone code | |
| - **Evaluating results** using BLEU, ROUGE, and human inspection | |
| The model achieves strong performance in generating accurate Linux kernel bug fixes, making it a valuable tool for automated code review and bug detection. | |
| --- | |
| ## 📊 Performance Results | |
| ### Evaluation Metrics | |
| ✅ **BLEU Score**: 33.87 | |
| ✅ **ROUGE Scores**: | |
| - **ROUGE-1**: P=0.3775, R=0.7306, F1=0.4355 | |
| - **ROUGE-2**: P=0.2898, R=0.6096, F1=0.3457 | |
| - **ROUGE-L**: P=0.3023, R=0.6333, F1=0.3612 | |
| These results demonstrate the model's ability to: | |
| - Generate syntactically correct Git diff patches | |
| - Maintain semantic similarity to reference fixes | |
| - Produce meaningful code changes that address the underlying bugs | |
| --- | |
| ## 🧠 Model Configuration | |
| - **Base model**: `CodeLLaMA-7B-Instruct` | |
| - **Fine-tuning method**: QLoRA with 4-bit quantization | |
| - **Training setup**: | |
| - LoRA r=64, alpha=16, dropout=0.1 | |
| - Batch size: 64, LR: 2e-4, Epochs: 3 | |
| - Mixed precision (bfloat16), gradient checkpointing | |
| - **Hardware**: Optimized for NVIDIA H200 GPUs | |
| --- | |
| ## 📊 Dataset | |
| Custom dataset extracted from Linux kernel Git history. | |
| ### Filtering Criteria | |
| Bug-fix commits containing: | |
| `fix`, `bug`, `crash`, `memory`, `null`, `panic`, `overflow`, `race`, `corruption`, etc. | |
| ### Structure | |
| - Language: C (`.c`, `.h`) | |
| - Context: 10 lines before/after the change | |
| - Format: | |
| ```json | |
| { | |
| "input": { | |
| "original code": "C code snippet with bug", | |
| "instruction": "Commit message or fix description" | |
| }, | |
| "output": { | |
| "diff codes": "Git diff showing the fix" | |
| } | |
| } | |
| ``` | |
| * **File**: `training_data_100k.jsonl` (100,000 samples) | |
| --- | |
| ## 🚀 Quick Start | |
| ### Prerequisites | |
| - Python 3.8+ | |
| - CUDA-compatible GPU (recommended) | |
| - 16GB+ RAM | |
| - 50GB+ disk space | |
| ### Install dependencies | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ### 1. Build the Dataset | |
| ```bash | |
| cd dataset_builder | |
| python extract_linux_bugfixes_parallel.py | |
| python format_for_training.py | |
| ``` | |
| ### 2. Fine-tune the Model | |
| ```bash | |
| cd train | |
| python train_codellama_qlora_linux_bugfix.py | |
| ``` | |
| ### 3. Run Evaluation | |
| ```bash | |
| cd evaluate | |
| python evaluate_linux_bugfix_model.py | |
| ``` | |
| ### 4. Use the Model | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| from peft import PeftModel | |
| # Load the fine-tuned model | |
| model = AutoModelForCausalLM.from_pretrained("codellama/CodeLLaMA-7b-Instruct-hf") | |
| model = PeftModel.from_pretrained(model, "train/output/qlora-codellama-bugfix") | |
| tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLLaMA-7b-Instruct-hf") | |
| # Generate a bug fix | |
| prompt = """ | |
| Given the following original C code: | |
| if (!file->filter) | |
| return; | |
| Instruction: Fix the null pointer dereference | |
| Return the diff that fixes it: | |
| """ | |
| inputs = tokenizer(prompt, return_tensors="pt") | |
| outputs = model.generate(**inputs, max_length=512, temperature=0.1) | |
| fix = tokenizer.decode(outputs[0], skip_special_tokens=True) | |
| print(fix) | |
| ``` | |
| --- | |
| ## 📁 Project Structure | |
| ``` | |
| CodeLLaMA-Linux-BugFix/ | |
| ├── dataset_builder/ | |
| │ ├── extract_linux_bugfixes_parallel.py # Parallel extraction of bug fixes | |
| │ ├── format_for_training.py # Format data for training | |
| │ └── build_dataset.py # Main dataset builder | |
| ├── dataset/ | |
| │ ├── training_data_100k.jsonl # 100K training samples | |
| │ └── training_data_prompt_completion.jsonl # Formatted training data | |
| ├── train/ | |
| │ ├── train_codellama_qlora_linux_bugfix.py # Main training script | |
| │ ├── train_codellama_qlora_simple.py # Simplified training | |
| │ ├── download_codellama_model.py # Model download utility | |
| │ └── output/ | |
| │ └── qlora-codellama-bugfix/ # Trained model checkpoints | |
| ├── evaluate/ | |
| │ ├── evaluate_linux_bugfix_model.py # Evaluation script | |
| │ ├── test_samples.jsonl # Test dataset | |
| │ └── output/ # Evaluation results | |
| │ ├── eval_results.csv # Detailed results | |
| │ └── eval_results.json # JSON format results | |
| ├── requirements.txt # Python dependencies | |
| ├── README.md # This file | |
| └── PROJECT_STRUCTURE.md # Detailed project overview | |
| ``` | |
| --- | |
| ## 🧩 Features | |
| * 🔧 **Efficient Fine-tuning**: QLoRA + 4-bit quant = massive memory savings | |
| * 🧠 **Real-world commits**: From actual Linux kernel development | |
| * 💡 **Context-aware**: Code context extraction around bug lines | |
| * 💻 **Output-ready**: Generates valid Git-style diffs | |
| * 📈 **Strong Performance**: BLEU score of 33.87 with good ROUGE metrics | |
| * 🚀 **Production-ready**: Optimized for real-world deployment | |
| --- | |
| ## 📈 Evaluation Metrics | |
| * **BLEU**: Translation-style match to reference diffs | |
| * **ROUGE**: Overlap in fix content and semantic similarity | |
| * **Human Evaluation**: Subjective patch quality assessment | |
| ### Current Performance | |
| - **BLEU Score**: 33.87 (excellent for code generation tasks) | |
| - **ROUGE-1 F1**: 0.4355 (good semantic overlap) | |
| - **ROUGE-2 F1**: 0.3457 (reasonable bigram matching) | |
| - **ROUGE-L F1**: 0.3612 (good longest common subsequence) | |
| --- | |
| ## 🧪 Use Cases | |
| * **Automated kernel bug fixing**: Generate fixes for common kernel bugs | |
| * **Code review assistance**: Help reviewers identify potential issues | |
| * **Teaching/debugging kernel code**: Educational tool for kernel development | |
| * **Research in automated program repair (APR)**: Academic research applications | |
| * **CI/CD integration**: Automated testing and fixing in development pipelines | |
| --- | |
| ## 🔬 Technical Highlights | |
| ### Memory & Speed Optimizations | |
| * 4-bit quantization (NF4) | |
| * Gradient checkpointing | |
| * Mixed precision (bfloat16) | |
| * Gradient accumulation | |
| * LoRA parameter efficiency | |
| ### Training Efficiency | |
| * **QLoRA**: Reduces memory usage by ~75% | |
| * **4-bit quantization**: Further memory optimization | |
| * **Gradient checkpointing**: Trades compute for memory | |
| * **Mixed precision**: Faster training with maintained accuracy | |
| --- | |
| ## 🛠️ Advanced Usage | |
| ### Custom Training | |
| ```bash | |
| # Train with custom parameters | |
| python train_codellama_qlora_linux_bugfix.py \ | |
| --learning_rate 1e-4 \ | |
| --num_epochs 5 \ | |
| --batch_size 32 \ | |
| --lora_r 32 \ | |
| --lora_alpha 16 | |
| ``` | |
| ### Evaluation on Custom Data | |
| ```bash | |
| # Evaluate on your own test set | |
| python evaluate_linux_bugfix_model.py \ | |
| --test_file your_test_data.jsonl \ | |
| --output_dir custom_eval_results | |
| ``` | |
| --- | |
| ## 🤝 Contributing | |
| 1. Fork this repo | |
| 2. Create a feature branch (`git checkout -b feature/amazing-feature`) | |
| 3. Commit your changes (`git commit -m 'Add amazing feature'`) | |
| 4. Push to the branch (`git push origin feature/amazing-feature`) | |
| 5. Open a Pull Request 🙌 | |
| ### Development Guidelines | |
| - Follow PEP 8 style guidelines | |
| - Add tests for new features | |
| - Update documentation for API changes | |
| - Ensure all tests pass before submitting PR | |
| --- | |
| ## 📄 License | |
| MIT License – see `LICENSE` file for details. | |
| --- | |
| ## 🙏 Acknowledgments | |
| * **Meta** for CodeLLaMA base model | |
| * **Hugging Face** for Transformers + PEFT libraries | |
| * **The Linux kernel community** for open access to commit data | |
| * **Microsoft** for introducing LoRA technique | |
| * **University of Washington** for QLoRA research | |
| --- | |
| ## 📚 References | |
| * [CodeLLaMA (Meta, 2023)](https://arxiv.org/abs/2308.12950) | |
| * [QLoRA (Dettmers et al., 2023)](https://arxiv.org/abs/2305.14314) | |
| * [LoRA (Hu et al., 2021)](https://arxiv.org/abs/2106.09685) | |
| * [Automated Program Repair: A Survey](https://ieeexplore.ieee.org/document/8449519) | |
| --- | |
| ## 📞 Support | |
| For questions, issues, or contributions: | |
| - Open an issue on GitHub | |
| - Check the project documentation | |
| - Review the evaluation results in `evaluate/output/` | |
| --- | |
| ## 🔄 Version History | |
| - **v1.0.0**: Initial release with QLoRA training | |
| - **v1.1.0**: Added parallel dataset extraction | |
| - **v1.2.0**: Improved evaluation metrics and documentation | |