| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - code-llm |
| - qwen |
| - sft |
| - dpo |
| - peft |
| - lora |
| metrics: |
| - accuracy |
| base_model: |
| - Qwen/Qwen2.5-Coder-7B |
| library_name: peft |
| --- |
| |
| # Code-Centric-Align: A Post-Training Pipeline for Code LLMs (LoRA Adapter) |
|
|
| **Notice:** This repository provides a **LoRA Adapter** trained via QLoRA. It is designed to be loaded on top of the base model `Qwen/Qwen2.5-Coder-7B`. |
|
|
| This project presents a systematic study of the post-training engineering pipeline for code-specific large language models. It establishes a "diagnosable and iterative" framework covering the full lifecycle from data engineering to deployment. |
|
|
| ## 🚀 Quick Start (Inference Example) |
|
|
| To use this LoRA adapter, you need to load the base model first and then attach the PEFT adapter. Ensure you have the required libraries installed: |
| ```bash |
| pip install transformers peft torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| import torch |
| |
| base_model_id = "Qwen/Qwen2.5-Coder-7B" |
| adapter_id = "abcsk123/Code-Centric-Align" |
| |
| # 1. Load Tokenizer |
| tokenizer = AutoTokenizer.from_pretrained(base_model_id) |
| |
| # 2. Load Base Model |
| base_model = AutoModelForCausalLM.from_pretrained( |
| base_model_id, |
| torch_dtype=torch.bfloat16, |
| device_map="auto" |
| ) |
| |
| # 3. Attach LoRA Adapter |
| model = PeftModel.from_pretrained(base_model, adapter_id) |
| |
| # 4. Generate Code |
| prompt = "def binary_search(arr, target):" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate(**inputs, max_new_tokens=100) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
| (Note: If your adapter files are located inside a specific checkpoint folder, e.g., checkpoint-4675, please add the argument subfolder="checkpoint-4675" to PeftModel.from_pretrained()) |
| |
| |
| ## 🛠️ Core Workflow |
| - Data Engineering: Implemented streaming collection, three-layer quality filtering, and MinHashLSH-based fuzzy deduplication. |
| - Instruction Evolution: Utilized DeepSeek APIs for Evol-Instruct difficulty enhancement and diversity expansion. |
| - Supervised Fine-Tuning (SFT): Applied QLoRA with a custom Instruction Masking strategy (QwenDataCollator) to ensure the model only learns from assistant responses. |
| - Rejection Sampling (RFT): Developed a high-throughput engine using vLLM for 10-path sampling, verified through a multi-process safe execution sandbox. |
| - Preference Alignment (DPO): Investigated Direct Preference Optimization, identifying critical failure modes such as length bias and low-quality negative samples. |
| - Quantization & Deployment: Performed 4-bit activation-aware quantization (AutoAWQ) and deployed the model via a vLLM OpenAI-compatible API. |
| |
| ## 📈 Experimental Results (HumanEval Pass@1) |
| The project tracked performance gains and losses across multiple iterations: |
| - Base Model: 0.628 |
| - **SFT v3 (released): 0.671 (+6.8%)** — achieved through precise loss calculation and data cleaning. |
| - DPO Merged: 0.280 — highlighting the extreme sensitivity of code models to preference data quality. |
| |
| ## ⚠️ Status & Roadmap |
| This project is actively under development. Currently, the DPO alignment exhibits performance regression (Pass@1 < 0.628) due to preference data sensitivity. We are investigating advanced filtering and reward modeling to resolve this. Optimized weights will be uploaded as soon as the alignment bottleneck is cleared. |
