docs/TRAINING_7B.md

Training Stack 2.9 on Qwen2.5-Coder-7B

Overview

This guide covers training Stack 2.9 on the Qwen2.5-Coder-7B model using LoRA/QLoRA fine-tuning.

Hardware Requirements

Minimum (QLoRA - 4-bit)

GPU	VRAM	Batch Size	Notes
T4 (Colab)	15GB	1	Gradient accu = 16
P100 (Kaggle)	16GB	1	Gradient accu = 8
RTX 3090	24GB	2	Full performance

Recommended (Full LoRA - bf16)

GPU	VRAM	Batch Size	Notes
A100 40GB	40GB	2	2x for better throughput
A100 80GB	80GB	4	Best for production
H100 80GB	80GB	4	Next-gen option

VRAM Estimates

Configuration	Batch Size	Gradient Checkpoint	Est. VRAM
Full bf16	1	No	14GB
Full bf16	2	Yes	16GB
Full bf16	4	Yes	22GB
QLoRA (4-bit)	1	Yes	5-6GB
QLoRA (4-bit)	2	Yes	7-8GB

Quick Start

Option 1: Kaggle (QLoRA)

cd /kaggle/working/stack-2.9
chmod +x training-configs/kaggle-7b-qlora.sh
./training-configs/kaggle-7b-qlora.sh

Option 2: Local (Full LoRA)

cd /path/to/stack-2.9
python train_local.py \
    --config training-configs/7b-lora-config.yaml

Option 3: Custom Training Script

from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B",
    torch_dtype="bfloat16",
    device_map="auto"
)

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", 
                   "gate_proj", "up_proj", "down_proj"],
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

Configuration Reference

LoRA Parameters

lora:
  r: 16              # Rank (8-32 recommended for 7B)
  alpha: 32          # Usually 2*r
  dropout: 0.05
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj

Training Parameters

training:
  num_epochs: 3
  batch_size: 2      # A100: 2-4, T4/P100: 1
  gradient_accumulation: 8
  learning_rate: 1.0e-4
  warmup_steps: 100
  gradient_checkpointing: true
  bf16: true

Expected Training Time

Based on ~10K samples, max_length=4096:

Hardware	Config	Est. Time
T4	4-bit QLoRA	4-6 hours
P100	4-bit QLoRA	2-3 hours
A100 40GB	bf16 LoRA	30-45 min
A100 80GB	bf16 LoRA	20-30 min

Times scale linearly with dataset size.

After Training

Merge LoRA Adapter

from peft import PeftModel
from transformers import AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B",
    torch_dtype="bfloat16"
)

# Merge adapter
model = PeftModel.from_pretrained(base_model, "./output/lora")
merged = model.merge_and_unload()

# Save
merged.save_pretrained("./output/merged")
tokenizer.save_pretrained("./output/merged")

Test the Model

from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("./output/merged")
pipe = pipeline("text-generation", model=merged, tokenizer=tokenizer)

result = pipe("def quick_sort(arr):", max_new_tokens=100)
print(result[0]["generated_text"])

Troubleshooting

OOM (Out of Memory)

• Reduce batch_size to 1
• Enable gradient_checkpointing: true
• Reduce max_length (4096 → 2048)
• Enable 4-bit quantization

Training Slow

• Increase batch size if VRAM allows
• Enable use_flash_attention: true (A100/H100)
• Reduce gradient accumulation

Loss Not Converging

• Check learning rate (try 5e-5 or 2e-4)
• Increase epochs (3 → 5)
• Verify data format matches expected template

Alternative: RunPod /cloud Deployment

For faster training, see runpod_deploy.sh at repo root.

# Example: RunPod A100
bash runpod_deploy.sh --gpu a100 --instance $ hourly

Notes

• A100 recommended: Best balance of VRAM and speed
• 4-bit QLoRA: Use only if VRAM < 20GB, slightly reduces quality
• Gradient checkpointing: Always enable, minimal perf impact for big memory savings
• Flash Attention: A100/H100 only, significant speed boost