Famali/qwen14b-abap-sql-lora

Qwen 14B Code LoRA (ABAP/SQL/Java/Python)

Fine-tuned LoRA adapter for multilingual code generation with focus on SAP ABAP. Larger model with stronger SQL/Java performance.

Model Details

Base model	Qwen/Qwen2.5-Coder-14B-Instruct
Method	QLoRA (NF4, r=16, α=32)
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training data	50,000 samples (12.9% ABAP, stratified subsample)
Training time	28.2 hours (RTX 4000 Ada)
Adapter size	~320 MB
Epochs	1
Learning rate	2e-4, cosine schedule

Note: 14B trained on 50k (not 100k) due to VRAM/time constraints (RTX 4000 Ada, 20.5 GB). This is a resource constraint, not a scientific choice. See PLANNING.md T6.1 for details.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model in NF4
quant_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="auto",
)

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-14B-Instruct",
    quantization_config=quant_cfg,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ChayannFamali/qwen14b-abap-sql-lora")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-14B-Instruct")

# Generate
messages = [
    {"role": "system", "content": "You are an expert ABAP programmer."},
    {"role": "user", "content": "Implement ABAP class for customer data handling"},
]
chatml = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(chatml, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0]))

Usage with RAG

For best results, use with the Hybrid RAG pipeline (see GitHub repo for full instructions):

from src.rag.retriever import HybridRetriever

retriever = HybridRetriever(
    chroma_path="data/rag_index",
    collection_name="code_corpus",
    chunks_dir="data/rag_corpus/chunks",
    model_name="BAAI/bge-m3",
    device="cuda:0",
)

# Retrieve 3 ABAP examples
results = retriever.retrieve("Implement ABAP class for sorting", language="ABAP", k=3)

# Build few-shot system prompt
examples = "\n".join(f"Example {i+1}:\n```\n{r['code'][:800]}\n```\n"
                     for i, r in enumerate(results))
system = f"You are an expert ABAP programmer.\nHere are 3 relevant ABAP code examples:\n{examples}"

Performance (Test Split)

Language	Metric	Baseline 14B	FT 14B	FT 14B + RAG
ABAP	chrf	0.319	0.450	0.531
ABAP	syntax_valid	1.000	0.983	0.917
ABAP	exact_match	0.000	0.039	0.028
SQL	exact_match	0.100	0.360	0.380
SQL	chrf	0.744	0.843	0.842
Python	chrf	0.376	0.418	0.389
Java	chrf	0.348	0.392	0.360

ABAP chrf: +66% (0.319 → 0.531, baseline → FT+RAG) SQL exact_match: +3.8x (0.100 → 0.380, baseline → FT+RAG)

Python-Switching (Val Split)

Model	Switching rate
Baseline 14B	0.0%
FT 14B	17.8%
FT 14B + RAG	0.0%

RAG completely eliminates switching (17.8% → 0.0%) without additional training.

Training Details

• Config: configs/qlora_14b_final.yaml
• Data: data/splits/train_v2_abap_boost_50k.jsonl (50k, resource constraint)
• gradient_checkpointing: true (mandatory for 14B)
• save_steps: 50 (~30 min between checkpoints, 14B-specific)
• Merge: CPU merge required (14B bf16 = ~28 GB > 20.5 GB VRAM)
• Full reproduction: See REPRODUCE.md

Links

• GitHub: qwen-coder-abap-rag
• 7B version: qwen7b-abap-sql-lora
• Full results: final_comparison.md

License

MIT