license: apache-2.0
license: apache-2.0 base_model: Qwen/Qwen2.5-3B tags: - qwen2 - java - qlora - domain-adaptation - code - fine-tuned language: - en library_name: transformers
JavaExpert-Qwen2.5-3B
Domain-locked Java QA via QLoRA on consumer hardware.
Fine-tuned from Qwen/Qwen2.5-3B on 7,921 Java QA pairs using QLoRA — trained entirely on a single NVIDIA RTX 5050 (8 GB VRAM), no cloud, no A100s. The model answers Java questions accurately and refuses everything else, without external guardrails.
Evaluation Results
| Metric | Target | Achieved |
|---|---|---|
| Validation Perplexity | < 10 | 2.40 |
| Peak Training VRAM | ≤ 8 GB | < 7 GB |
| Inference VRAM | — | < 2 GB |
| Java Correctness | — | 8.5 / 10 |
| Domain Restriction | — | 8.5 / 10 |
| Hallucination Control | — | 8.0 / 10 |
Quick Start
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Debarun12/JavaExpert-Qwen2.5-3B")
tokenizer = AutoTokenizer.from_pretrained("Debarun12/JavaExpert-Qwen2.5-3B")
messages = [
{"role": "system", "content": "You are JavaExpert, a Java programming assistant."},
{"role": "user", "content": "What is the difference between HashMap and LinkedHashMap in Java?"}
]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")
output = model.generate(input_ids, max_new_tokens=300)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Domain Restriction — What It Does
The model was trained with explicit negative examples (unanswerable/out-of-domain questions), so it refuses non-Java queries without any post-processing filter:
| Query | Response |
|---|---|
| What is the capital of France? | Refuses cleanly |
| Who won the FIFA World Cup? | Refuses cleanly |
| How do I train a neural network in Python? | Partial refusal (known limitation — see below) |
Training Details
Base model: Qwen/Qwen2.5-3B — selected over the 7B (OOM risk on 8 GB) and 1.5B (higher hallucination tendency).
Method: QLoRA — base model quantized to 4-bit NF4 (~1.9 GB), adapters trained in bf16.
Hardware: NVIDIA GeForce RTX 5050, 8 GB VRAM. Peak usage < 7 GB.
Memory Optimisation Decisions
Every config choice was driven by measured VRAM impact:
| Component | Initial | Final | VRAM Saved |
|---|---|---|---|
| LoRA rank | 32 across 7 modules | 16 on q_proj, v_proj | −1.5 GB |
| Batch size | 4 | 1 + grad accum ×8 | −2.0 GB |
| Optimizer | AdamW | Adafactor | −1.0 GB |
| Compute dtype | fp16 | bf16 | Stable on Blackwell |
Hyperparameters
per_device_train_batch_size = 1
gradient_accumulation_steps = 8 # effective batch = 8
optim = "adafactor"
bf16 = True
learning_rate = 2e-4
num_train_epochs = 3
lora_r = 16
lora_target_modules = ["q_proj", "v_proj"]
Training Loss
| Epoch | Train Loss | Val Loss | Val Perplexity |
|---|---|---|---|
| 1 | 0.8511 | 0.9343 | 2.55 |
| 2 | 0.8367 | 0.8839 | 2.42 |
| 3 | 0.7780 | 0.8772 | 2.40 |
Validation loss decreased monotonically. No overfitting across 3 epochs — the train/val gap remained narrow and stable throughout.
Dataset
Built from scratch — not an off-the-shelf dataset:
- Source: 42,000+ lines of Java documentation, extracted from PDFs
- Pipeline: Custom preprocessing → sliding-window chunking (150 words, 30-word overlap) → LLM-assisted QA generation via
qwen2.5:7b - QA pairs: 7,921 chat-formatted (system + user + assistant)
- Question types: 36% definitional / 32% procedural / 32% reasoning + explicit unanswerable/refusal examples
- Estimated cleanliness: 95%+
Dataset: Debarun12/JavaExpert-Qwen2.5-3B-DATASET
Bugs Found and Fixed During Training
SFTTrainer step-count explosion — packing=True with pre-tokenized input caused 2,139 steps on 90 samples (~24× expected). Root cause: SFTTrainer misinterprets packed sequence lengths. Fix: replaced with standard Trainer + DataCollatorForLanguageModeling. Steps normalized to 2,673 across 3 epochs.
PyTorch 2.6 checkpoint incompatibility — weights_only=True default change raised UnpicklingError on resume. Fix: automated removal of rng_state.pth from checkpoint directories before resumption.
Limitations
- Domain-adjacent leakage: Python and general algorithm queries sometimes receive partial responses rather than clean refusals. The boundary between the Java corpus and the 3B's pretraining knowledge on general programming topics is not perfectly sharp.
- Corpus grounding: The model answers Java questions correctly but draws on pretraining generalisation rather than reproducing specific training corpus facts — a fundamental limitation of generative fine-tuning without retrieval.
Recommended next steps: RAG layer over the source corpus for strict factual grounding; DPO with curated refusal preference pairs to sharpen the domain boundary on adjacent technical topics.
Production Deployment
LoRA adapters were merged into base weights via merge_and_unload() — this is a single self-contained checkpoint with no adapter dependency at runtime. Inference runs at under 2 GB VRAM.
Repository
Training notebooks and dataset generation pipeline: GitHub — debarun23
Trained on NVIDIA GeForce RTX 5050 (8 GB VRAM) · June 2026 · Author: Debarun Das