Upload folder using huggingface_hub

06dee64 verified 8 days ago

3.9 kB

library_name: peft
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
  - code-generation
  - abap
  - sql
  - qlora
  - rag
license: mit
language:
  - en
pipeline_tag: text-generation

Qwen 7B Code LoRA (ABAP/SQL/Java/Python)

Fine-tuned LoRA adapter for multilingual code generation with focus on SAP ABAP.

Model Details


Base model	Qwen/Qwen2.5-Coder-7B-Instruct
Method	QLoRA (NF4, r=16, α=32)
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training data	100,000 samples (12.9% ABAP, L7-style 2x boost)
Training time	29.7 hours (RTX 4000 Ada)
Adapter size	~161 MB
Epochs	1
Learning rate	2e-4, cosine schedule

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model in NF4
quant_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="auto",
)

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct",
    quantization_config=quant_cfg,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ChayannFamali/qwen7b-abap-sql-lora")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")

# Generate
messages = [
    {"role": "system", "content": "You are an expert ABAP programmer."},
    {"role": "user", "content": "Implement ABAP class for customer data handling"},
]
chatml = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(chatml, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0]))

Usage with RAG

For best results, use with the Hybrid RAG pipeline (see GitHub repo for full instructions):

from src.rag.retriever import HybridRetriever

retriever = HybridRetriever(
    chroma_path="data/rag_index",
    collection_name="code_corpus",
    chunks_dir="data/rag_corpus/chunks",
    model_name="BAAI/bge-m3",
    device="cuda:0",
)

# Retrieve 3 ABAP examples
results = retriever.retrieve("Implement ABAP class for sorting", language="ABAP", k=3)

# Build few-shot system prompt
examples = "\n".join(f"Example {i+1}:\n```\n{r['code'][:800]}\n```\n"
                     for i, r in enumerate(results))
system = f"You are an expert ABAP programmer.\nHere are 3 relevant ABAP code examples:\n{examples}"

Performance (Test Split)

Language	Metric	Baseline	FT 7B v2	FT 7B v2 + RAG
ABAP	chrf	0.325	0.418	0.498
ABAP	syntax_valid	0.994	0.956	0.978
ABAP	exact_match	0.000	0.017	0.028
SQL	exact_match	0.040	0.320	0.300
SQL	chrf	0.767	0.842	0.804
Python	chrf	0.376	0.418	0.389
Java	chrf	0.348	0.392	0.360

Python-Switching (Val Split)

Model	Switching rate
Baseline 7B	0.0%
FT 7B v2	10.0%
FT 7B v2 + RAG	2.2%

RAG reduces switching by 78% (10.0% → 2.2%) without additional training.

Training Details

Config: configs/qlora_7b_v2_abap_boost.yaml
Data: data/splits/train_v2_abap_boost.jsonl (100k, 12.9% ABAP, repeat 1.464x)
Full reproduction: See REPRODUCE.md

License

MIT

Famali
/

qwen7b-abap-sql-lora