README.md

---
library_name: peft
base_model: Qwen/Qwen2.5-Coder-14B-Instruct
tags:
  - code-generation
  - abap
  - sql
  - qlora
  - rag
license: mit
language:
  - en
pipeline_tag: text-generation
---

# Qwen 14B Code LoRA (ABAP/SQL/Java/Python)

Fine-tuned LoRA adapter for multilingual code generation with focus on SAP ABAP. Larger model with stronger SQL/Java performance.

## Model Details

| | |
|---|---|
| **Base model** | Qwen/Qwen2.5-Coder-14B-Instruct |
| **Method** | QLoRA (NF4, r=16, α=32) |
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| **Training data** | 50,000 samples (12.9% ABAP, stratified subsample) |
| **Training time** | 28.2 hours (RTX 4000 Ada) |
| **Adapter size** | ~320 MB |
| **Epochs** | 1 |
| **Learning rate** | 2e-4, cosine schedule |

> **Note:** 14B trained on 50k (not 100k) due to VRAM/time constraints (RTX 4000 Ada, 20.5 GB). This is a resource constraint, not a scientific choice. See [PLANNING.md](https://github.com/ChayannFamali/qwen-coder-abap-rag/blob/main/agents/PLANNING.md) T6.1 for details.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model in NF4
quant_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="auto",
)

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-14B-Instruct",
    quantization_config=quant_cfg,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ChayannFamali/qwen14b-abap-sql-lora")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-14B-Instruct")

# Generate
messages = [
    {"role": "system", "content": "You are an expert ABAP programmer."},
    {"role": "user", "content": "Implement ABAP class for customer data handling"},
]
chatml = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(chatml, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0]))
```

## Usage with RAG

For best results, use with the Hybrid RAG pipeline (see [GitHub repo](https://github.com/ChayannFamali/qwen-coder-abap-rag) for full instructions):

```python
from src.rag.retriever import HybridRetriever

retriever = HybridRetriever(
    chroma_path="data/rag_index",
    collection_name="code_corpus",
    chunks_dir="data/rag_corpus/chunks",
    model_name="BAAI/bge-m3",
    device="cuda:0",
)

# Retrieve 3 ABAP examples
results = retriever.retrieve("Implement ABAP class for sorting", language="ABAP", k=3)

# Build few-shot system prompt
examples = "\n".join(f"Example {i+1}:\n```\n{r['code'][:800]}\n```\n"
                     for i, r in enumerate(results))
system = f"You are an expert ABAP programmer.\nHere are 3 relevant ABAP code examples:\n{examples}"
```

## Performance (Test Split)

| Language | Metric | Baseline 14B | FT 14B | FT 14B + RAG |
|----------|--------|--------------|--------|--------------|
| ABAP | chrf | 0.319 | 0.450 | **0.531** |
| ABAP | syntax_valid | 1.000 | 0.983 | 0.917 |
| ABAP | exact_match | 0.000 | 0.039 | 0.028 |
| SQL | exact_match | 0.100 | 0.360 | **0.380** |
| SQL | chrf | 0.744 | 0.843 | 0.842 |
| Python | chrf | 0.376 | 0.418 | 0.389 |
| Java | chrf | 0.348 | 0.392 | 0.360 |

**ABAP chrf: +66%** (0.319 → 0.531, baseline → FT+RAG)
**SQL exact_match: +3.8x** (0.100 → 0.380, baseline → FT+RAG)

## Python-Switching (Val Split)

| Model | Switching rate |
|---|---|
| Baseline 14B | 0.0% |
| FT 14B | 17.8% |
| **FT 14B + RAG** | **0.0%** |

RAG completely eliminates switching (17.8% → 0.0%) without additional training.

## Training Details

- **Config:** `configs/qlora_14b_final.yaml`
- **Data:** `data/splits/train_v2_abap_boost_50k.jsonl` (50k, resource constraint)
- **gradient_checkpointing:** true (mandatory for 14B)
- **save_steps:** 50 (~30 min between checkpoints, 14B-specific)
- **Merge:** CPU merge required (14B bf16 = ~28 GB > 20.5 GB VRAM)
- **Full reproduction:** See [REPRODUCE.md](https://github.com/ChayannFamali/qwen-coder-abap-rag/blob/main/REPRODUCE.md)

## Links

- **GitHub:** [qwen-coder-abap-rag](https://github.com/ChayannFamali/qwen-coder-abap-rag)
- **7B version:** [qwen7b-abap-sql-lora](https://huggingface.co/ChayannFamali/qwen7b-abap-sql-lora)
- **Full results:** [final_comparison.md](https://github.com/ChayannFamali/qwen-coder-abap-rag/blob/main/outputs/reports/final_comparison.md)

## License

MIT