---

license: apache-2.0 base_model: Qwen/Qwen2.5-3B tags: - qwen2 - java - qlora - domain-adaptation - code - fine-tuned language: - en library_name: transformers

JavaExpert-Qwen2.5-3B

Domain-locked Java QA via QLoRA on consumer hardware.

Fine-tuned from Qwen/Qwen2.5-3B on 7,921 Java QA pairs using QLoRA — trained entirely on a single NVIDIA RTX 5050 (8 GB VRAM), no cloud, no A100s. The model answers Java questions accurately and refuses everything else, without external guardrails.

Evaluation Results

Metric	Target	Achieved
Validation Perplexity	< 10	2.40
Peak Training VRAM	≤ 8 GB	< 7 GB
Inference VRAM	—	< 2 GB
Java Correctness	—	8.5 / 10
Domain Restriction	—	8.5 / 10
Hallucination Control	—	8.0 / 10

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Debarun12/JavaExpert-Qwen2.5-3B")
tokenizer = AutoTokenizer.from_pretrained("Debarun12/JavaExpert-Qwen2.5-3B")

messages = [
    {"role": "system", "content": "You are JavaExpert, a Java programming assistant."},
    {"role": "user", "content": "What is the difference between HashMap and LinkedHashMap in Java?"}
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")
output = model.generate(input_ids, max_new_tokens=300)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Domain Restriction — What It Does

The model was trained with explicit negative examples (unanswerable/out-of-domain questions), so it refuses non-Java queries without any post-processing filter:

Query	Response
What is the capital of France?	Refuses cleanly
Who won the FIFA World Cup?	Refuses cleanly
How do I train a neural network in Python?	Partial refusal (known limitation — see below)

Training Details

Base model: Qwen/Qwen2.5-3B — selected over the 7B (OOM risk on 8 GB) and 1.5B (higher hallucination tendency).

Method: QLoRA — base model quantized to 4-bit NF4 (~1.9 GB), adapters trained in bf16.

Hardware: NVIDIA GeForce RTX 5050, 8 GB VRAM. Peak usage < 7 GB.

Memory Optimisation Decisions

Every config choice was driven by measured VRAM impact:

Component	Initial	Final	VRAM Saved
LoRA rank	32 across 7 modules	16 on q_proj, v_proj	−1.5 GB
Batch size	4	1 + grad accum ×8	−2.0 GB
Optimizer	AdamW	Adafactor	−1.0 GB
Compute dtype	fp16	bf16	Stable on Blackwell

Hyperparameters

per_device_train_batch_size = 1
gradient_accumulation_steps = 8   # effective batch = 8
optim                         = "adafactor"
bf16                          = True
learning_rate                 = 2e-4
num_train_epochs              = 3
lora_r                        = 16
lora_target_modules           = ["q_proj", "v_proj"]

Training Loss

Epoch	Train Loss	Val Loss	Val Perplexity
1	0.8511	0.9343	2.55
2	0.8367	0.8839	2.42
3	0.7780	0.8772	2.40

Validation loss decreased monotonically. No overfitting across 3 epochs — the train/val gap remained narrow and stable throughout.

Dataset

Built from scratch — not an off-the-shelf dataset:

• Source: 42,000+ lines of Java documentation, extracted from PDFs
• Pipeline: Custom preprocessing → sliding-window chunking (150 words, 30-word overlap) → LLM-assisted QA generation via qwen2.5:7b
• QA pairs: 7,921 chat-formatted (system + user + assistant)
• Question types: 36% definitional / 32% procedural / 32% reasoning + explicit unanswerable/refusal examples
• Estimated cleanliness: 95%+

Dataset: Debarun12/JavaExpert-Qwen2.5-3B-DATASET

Bugs Found and Fixed During Training

SFTTrainer step-count explosion — packing=True with pre-tokenized input caused 2,139 steps on 90 samples (~24× expected). Root cause: SFTTrainer misinterprets packed sequence lengths. Fix: replaced with standard Trainer + DataCollatorForLanguageModeling. Steps normalized to 2,673 across 3 epochs.

PyTorch 2.6 checkpoint incompatibility — weights_only=True default change raised UnpicklingError on resume. Fix: automated removal of rng_state.pth from checkpoint directories before resumption.

Limitations

• Domain-adjacent leakage: Python and general algorithm queries sometimes receive partial responses rather than clean refusals. The boundary between the Java corpus and the 3B's pretraining knowledge on general programming topics is not perfectly sharp.
• Corpus grounding: The model answers Java questions correctly but draws on pretraining generalisation rather than reproducing specific training corpus facts — a fundamental limitation of generative fine-tuning without retrieval.

Recommended next steps: RAG layer over the source corpus for strict factual grounding; DPO with curated refusal preference pairs to sharpen the domain boundary on adjacent technical topics.

Production Deployment

LoRA adapters were merged into base weights via merge_and_unload() — this is a single self-contained checkpoint with no adapter dependency at runtime. Inference runs at under 2 GB VRAM.

Repository

Training notebooks and dataset generation pipeline: GitHub — debarun23

---

Trained on NVIDIA GeForce RTX 5050 (8 GB VRAM) · June 2026 · Author: Debarun Das