README.md

metadata

language:
  - en
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
  - code
  - abap
  - sap
  - lora
  - qlora
  - sft
  - trl
library_name: peft
pipeline_tag: text-generation

Qwen2.5-Coder-7B-ABAP

A fine-tuned version of Qwen/Qwen2.5-Coder-7B-Instruct specialized for SAP ABAP development.

Training Scripts

Two scripts are provided depending on your hardware:

Script	GPU VRAM	Method	Time Estimate
train_abap.py	24GB+ (A10G, A100, L4)	LoRA (bf16)	~1-2 hours
train_abap_qlora.py	8GB (RTX 3060/4060)	QLoRA (4-bit NF4)	~7-11 hours

Quick Start (8GB VRAM)

pip install torch transformers trl peft datasets accelerate bitsandbytes
huggingface-cli login
python train_abap_qlora.py

Quick Start (24GB+ VRAM)

pip install torch transformers trl peft datasets accelerate bitsandbytes
huggingface-cli login
python train_abap.py

Datasets

Dataset	Examples	Type
smjain/abap	248	ABAP coding tasks (reports, SELECT, internal tables)
Kaballas/abap	1,070	ABAP concept Q&A (OOP, classes, visibility)
Arturs213/abap-code-sec-finetune	~4,000+	ABAP security vulnerability analysis
Total	~5,300+

Training Configurations

Full LoRA (24GB+ VRAM) — train_abap.py

Parameter	Value
LoRA rank	32
LoRA alpha	64
Batch size	2 × 8 grad_accum = 16 effective
Learning rate	2e-4 (cosine)
Max length	2048
Precision	bf16
Epochs	3

QLoRA (8GB VRAM) — train_abap_qlora.py

Parameter	Value
Quantization	4-bit NF4 + double quant
LoRA rank	16
LoRA alpha	32
Batch size	1 × 16 grad_accum = 16 effective
Learning rate	2e-4 (cosine)
Max length	1024
Precision	bf16 compute on NF4 base
Optimizer	paged_adamw_8bit
Epochs	3

Usage

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("SpaceArm/Qwen2.5-Coder-7B-ABAP")
tokenizer = AutoTokenizer.from_pretrained("SpaceArm/Qwen2.5-Coder-7B-ABAP")

messages = [
    {"role": "system", "content": "You are an expert SAP ABAP developer."},
    {"role": "user", "content": "Write an ABAP class that reads data from table MARA and displays it in an ALV grid."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Capabilities

• ✅ ABAP report writing (REPORT, WRITE, LOOP, SELECT)
• ✅ Object-oriented ABAP (classes, interfaces, inheritance)
• ✅ Internal tables and data manipulation
• ✅ ALV grid programming
• ✅ Function modules and BAPIs
• ✅ ABAP security vulnerability detection
• ✅ Modern ABAP syntax and best practices
• ✅ CDS views and RAP concepts

Evaluation

Evaluate against timkoehne/LLM-ABAP-Code-Generation-Benchmark (HumanEval adapted for ABAP).

OOM Troubleshooting

If you hit out-of-memory on 8GB VRAM:

1. Reduce max_length from 1024 → 512 in train_abap_qlora.py
2. Ensure no other GPU processes are running (nvidia-smi)
3. Close browser tabs / desktop apps using GPU
4. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before running

Research Background

ABAP is a low-resource programming language — while included in large code corpora like The Stack v2, training data is scarce compared to Python/Java. This model uses approaches from:

• Low-resource PL fine-tuning (arxiv:2501.19085): Fine-tuning on domain-specific instruction data improves performance on underrepresented languages
• Qwen2.5-Coder (arxiv:2409.12186): Best available base model with ABAP exposure in its 92-language pretraining corpus
• QLoRA (arxiv:2305.14314): 4-bit quantized training enabling 7B model fine-tuning on consumer GPUs