README.md

---
library_name: peft
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
  - code-generation
  - abap
  - sql
  - qlora
  - rag
license: mit
language:
  - en
pipeline_tag: text-generation
---

# Qwen 7B Code LoRA (ABAP/SQL/Java/Python)

Fine-tuned LoRA adapter for multilingual code generation with focus on SAP ABAP.

## Model Details

| | |
|---|---|
| **Base model** | Qwen/Qwen2.5-Coder-7B-Instruct |
| **Method** | QLoRA (NF4, r=16, α=32) |
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| **Training data** | 100,000 samples (12.9% ABAP, L7-style 2x boost) |
| **Training time** | 29.7 hours (RTX 4000 Ada) |
| **Adapter size** | ~161 MB |
| **Epochs** | 1 |
| **Learning rate** | 2e-4, cosine schedule |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model in NF4
quant_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="auto",
)

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct",
    quantization_config=quant_cfg,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ChayannFamali/qwen7b-abap-sql-lora")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")

# Generate
messages = [
    {"role": "system", "content": "You are an expert ABAP programmer."},
    {"role": "user", "content": "Implement ABAP class for customer data handling"},
]
chatml = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(chatml, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0]))
```

## Usage with RAG

For best results, use with the Hybrid RAG pipeline (see [GitHub repo](https://github.com/ChayannFamali/qwen-coder-abap-rag) for full instructions):

```python
from src.rag.retriever import HybridRetriever

retriever = HybridRetriever(
    chroma_path="data/rag_index",
    collection_name="code_corpus",
    chunks_dir="data/rag_corpus/chunks",
    model_name="BAAI/bge-m3",
    device="cuda:0",
)

# Retrieve 3 ABAP examples
results = retriever.retrieve("Implement ABAP class for sorting", language="ABAP", k=3)

# Build few-shot system prompt
examples = "\n".join(f"Example {i+1}:\n```\n{r['code'][:800]}\n```\n"
                     for i, r in enumerate(results))
system = f"You are an expert ABAP programmer.\nHere are 3 relevant ABAP code examples:\n{examples}"
```

## Performance (Test Split)

| Language | Metric | Baseline | FT 7B v2 | FT 7B v2 + RAG |
|----------|--------|----------|----------|----------------|
| ABAP | chrf | 0.325 | 0.418 | **0.498** |
| ABAP | syntax_valid | 0.994 | 0.956 | 0.978 |
| ABAP | exact_match | 0.000 | 0.017 | 0.028 |
| SQL | exact_match | 0.040 | 0.320 | 0.300 |
| SQL | chrf | 0.767 | 0.842 | 0.804 |
| Python | chrf | 0.376 | 0.418 | 0.389 |
| Java | chrf | 0.348 | 0.392 | 0.360 |

## Python-Switching (Val Split)

| Model | Switching rate |
|---|---|
| Baseline 7B | 0.0% |
| FT 7B v2 | 10.0% |
| **FT 7B v2 + RAG** | **2.2%** |

RAG reduces switching by 78% (10.0% → 2.2%) without additional training.

## Training Details

- **Config:** `configs/qlora_7b_v2_abap_boost.yaml`
- **Data:** `data/splits/train_v2_abap_boost.jsonl` (100k, 12.9% ABAP, repeat 1.464x)
- **Full reproduction:** See [REPRODUCE.md](https://github.com/ChayannFamali/qwen-coder-abap-rag/blob/main/REPRODUCE.md)

## Links

- **GitHub:** [qwen-coder-abap-rag](https://github.com/ChayannFamali/qwen-coder-abap-rag)
- **14B version:** [qwen14b-abap-sql-lora](https://huggingface.co/ChayannFamali/qwen14b-abap-sql-lora)
- **Full results:** [final_comparison.md](https://github.com/ChayannFamali/qwen-coder-abap-rag/blob/main/outputs/reports/final_comparison.md)

## License

MIT