README.md

---
language:
- en
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
- code
- abap
- sap
- lora
- qlora
- sft
- trl
library_name: peft
pipeline_tag: text-generation
---

# Qwen2.5-Coder-7B-ABAP

A fine-tuned version of [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) specialized for **SAP ABAP development**.

## Training Scripts

Two scripts are provided depending on your hardware:

| Script | GPU VRAM | Method | Time Estimate |
|--------|----------|--------|---------------|
| [`train_abap.py`](train_abap.py) | 24GB+ (A10G, A100, L4) | LoRA (bf16) | ~1-2 hours |
| [`train_abap_qlora.py`](train_abap_qlora.py) | **8GB** (RTX 3060/4060) | QLoRA (4-bit NF4) | ~7-11 hours |

### Quick Start (8GB VRAM)

```bash
pip install torch transformers trl peft datasets accelerate bitsandbytes
huggingface-cli login
python train_abap_qlora.py
```

### Quick Start (24GB+ VRAM)

```bash
pip install torch transformers trl peft datasets accelerate bitsandbytes
huggingface-cli login
python train_abap.py
```

## Datasets

| Dataset | Examples | Type |
|---------|----------|------|
| [smjain/abap](https://huggingface.co/datasets/smjain/abap) | 248 | ABAP coding tasks (reports, SELECT, internal tables) |
| [Kaballas/abap](https://huggingface.co/datasets/Kaballas/abap) | 1,070 | ABAP concept Q&A (OOP, classes, visibility) |
| [Arturs213/abap-code-sec-finetune](https://huggingface.co/datasets/Arturs213/abap-code-sec-finetune) | ~4,000+ | ABAP security vulnerability analysis |
| **Total** | **~5,300+** | |

## Training Configurations

### Full LoRA (24GB+ VRAM) — `train_abap.py`

| Parameter | Value |
|-----------|-------|
| LoRA rank | 32 |
| LoRA alpha | 64 |
| Batch size | 2 × 8 grad_accum = 16 effective |
| Learning rate | 2e-4 (cosine) |
| Max length | 2048 |
| Precision | bf16 |
| Epochs | 3 |

### QLoRA (8GB VRAM) — `train_abap_qlora.py`

| Parameter | Value |
|-----------|-------|
| Quantization | 4-bit NF4 + double quant |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| Batch size | 1 × 16 grad_accum = 16 effective |
| Learning rate | 2e-4 (cosine) |
| Max length | 1024 |
| Precision | bf16 compute on NF4 base |
| Optimizer | paged_adamw_8bit |
| Epochs | 3 |

## Usage

```python
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("SpaceArm/Qwen2.5-Coder-7B-ABAP")
tokenizer = AutoTokenizer.from_pretrained("SpaceArm/Qwen2.5-Coder-7B-ABAP")

messages = [
    {"role": "system", "content": "You are an expert SAP ABAP developer."},
    {"role": "user", "content": "Write an ABAP class that reads data from table MARA and displays it in an ALV grid."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Capabilities

- ✅ ABAP report writing (REPORT, WRITE, LOOP, SELECT)
- ✅ Object-oriented ABAP (classes, interfaces, inheritance)
- ✅ Internal tables and data manipulation
- ✅ ALV grid programming
- ✅ Function modules and BAPIs
- ✅ ABAP security vulnerability detection
- ✅ Modern ABAP syntax and best practices
- ✅ CDS views and RAP concepts

## Evaluation

Evaluate against [timkoehne/LLM-ABAP-Code-Generation-Benchmark](https://huggingface.co/datasets/timkoehne/LLM-ABAP-Code-Generation-Benchmark) (HumanEval adapted for ABAP).

## OOM Troubleshooting

If you hit out-of-memory on 8GB VRAM:

1. Reduce `max_length` from 1024 → 512 in `train_abap_qlora.py`
2. Ensure no other GPU processes are running (`nvidia-smi`)
3. Close browser tabs / desktop apps using GPU
4. Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` before running

## Research Background

ABAP is a low-resource programming language — while included in large code corpora like The Stack v2, training data is scarce compared to Python/Java. This model uses approaches from:

- **Low-resource PL fine-tuning** ([arxiv:2501.19085](https://arxiv.org/abs/2501.19085)): Fine-tuning on domain-specific instruction data improves performance on underrepresented languages
- **Qwen2.5-Coder** ([arxiv:2409.12186](https://arxiv.org/abs/2409.12186)): Best available base model with ABAP exposure in its 92-language pretraining corpus
- **QLoRA** ([arxiv:2305.14314](https://arxiv.org/abs/2305.14314)): 4-bit quantized training enabling 7B model fine-tuning on consumer GPUs