| --- |
| license: apache-2.0 |
| --- |
| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-3B |
| tags: |
| - qwen2 |
| - java |
| - qlora |
| - domain-adaptation |
| - code |
| - fine-tuned |
| language: |
| - en |
| library_name: transformers |
| --- |
|
|
| # JavaExpert-Qwen2.5-3B |
|
|
| **Domain-locked Java QA via QLoRA on consumer hardware.** |
|
|
| Fine-tuned from `Qwen/Qwen2.5-3B` on 7,921 Java QA pairs using QLoRA — trained entirely on a single NVIDIA RTX 5050 (8 GB VRAM), no cloud, no A100s. The model answers Java questions accurately and refuses everything else, without external guardrails. |
|
|
| ## Evaluation Results |
|
|
| | Metric | Target | Achieved | |
| |--------|--------|----------| |
| | Validation Perplexity | < 10 | **2.40** | |
| | Peak Training VRAM | ≤ 8 GB | **< 7 GB** | |
| | Inference VRAM | — | **< 2 GB** | |
| | Java Correctness | — | **8.5 / 10** | |
| | Domain Restriction | — | **8.5 / 10** | |
| | Hallucination Control | — | **8.0 / 10** | |
|
|
| ## Quick Start |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("Debarun12/JavaExpert-Qwen2.5-3B") |
| tokenizer = AutoTokenizer.from_pretrained("Debarun12/JavaExpert-Qwen2.5-3B") |
| |
| messages = [ |
| {"role": "system", "content": "You are JavaExpert, a Java programming assistant."}, |
| {"role": "user", "content": "What is the difference between HashMap and LinkedHashMap in Java?"} |
| ] |
| |
| input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt") |
| output = model.generate(input_ids, max_new_tokens=300) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Domain Restriction — What It Does |
|
|
| The model was trained with explicit negative examples (unanswerable/out-of-domain questions), so it refuses non-Java queries without any post-processing filter: |
|
|
| | Query | Response | |
| |-------|----------| |
| | What is the capital of France? | Refuses cleanly | |
| | Who won the FIFA World Cup? | Refuses cleanly | |
| | How do I train a neural network in Python? | Partial refusal (known limitation — see below) | |
|
|
| ## Training Details |
|
|
| **Base model:** `Qwen/Qwen2.5-3B` — selected over the 7B (OOM risk on 8 GB) and 1.5B (higher hallucination tendency). |
|
|
| **Method:** QLoRA — base model quantized to 4-bit NF4 (~1.9 GB), adapters trained in bf16. |
|
|
| **Hardware:** NVIDIA GeForce RTX 5050, 8 GB VRAM. Peak usage < 7 GB. |
|
|
| ### Memory Optimisation Decisions |
|
|
| Every config choice was driven by measured VRAM impact: |
|
|
| | Component | Initial | Final | VRAM Saved | |
| |-----------|---------|-------|------------| |
| | LoRA rank | 32 across 7 modules | 16 on q_proj, v_proj | −1.5 GB | |
| | Batch size | 4 | 1 + grad accum ×8 | −2.0 GB | |
| | Optimizer | AdamW | Adafactor | −1.0 GB | |
| | Compute dtype | fp16 | bf16 | Stable on Blackwell | |
|
|
| ### Hyperparameters |
|
|
| ``` |
| per_device_train_batch_size = 1 |
| gradient_accumulation_steps = 8 # effective batch = 8 |
| optim = "adafactor" |
| bf16 = True |
| learning_rate = 2e-4 |
| num_train_epochs = 3 |
| lora_r = 16 |
| lora_target_modules = ["q_proj", "v_proj"] |
| ``` |
|
|
| ### Training Loss |
|
|
| | Epoch | Train Loss | Val Loss | Val Perplexity | |
| |-------|-----------|----------|----------------| |
| | 1 | 0.8511 | 0.9343 | 2.55 | |
| | 2 | 0.8367 | 0.8839 | 2.42 | |
| | 3 | 0.7780 | 0.8772 | **2.40** | |
|
|
| Validation loss decreased monotonically. No overfitting across 3 epochs — the train/val gap remained narrow and stable throughout. |
|
|
| ## Dataset |
|
|
| Built from scratch — not an off-the-shelf dataset: |
|
|
| - **Source:** 42,000+ lines of Java documentation, extracted from PDFs |
| - **Pipeline:** Custom preprocessing → sliding-window chunking (150 words, 30-word overlap) → LLM-assisted QA generation via `qwen2.5:7b` |
| - **QA pairs:** 7,921 chat-formatted (system + user + assistant) |
| - **Question types:** 36% definitional / 32% procedural / 32% reasoning + explicit unanswerable/refusal examples |
| - **Estimated cleanliness:** 95%+ |
|
|
| Dataset: [Debarun12/JavaExpert-Qwen2.5-3B-DATASET](https://huggingface.co/datasets/Debarun12/JavaExpert-Qwen2.5-3B-DATASET) |
|
|
| ## Bugs Found and Fixed During Training |
|
|
| **SFTTrainer step-count explosion** — `packing=True` with pre-tokenized input caused 2,139 steps on 90 samples (~24× expected). Root cause: SFTTrainer misinterprets packed sequence lengths. Fix: replaced with standard `Trainer` + `DataCollatorForLanguageModeling`. Steps normalized to 2,673 across 3 epochs. |
|
|
| **PyTorch 2.6 checkpoint incompatibility** — `weights_only=True` default change raised `UnpicklingError` on resume. Fix: automated removal of `rng_state.pth` from checkpoint directories before resumption. |
|
|
| ## Limitations |
|
|
| - **Domain-adjacent leakage:** Python and general algorithm queries sometimes receive partial responses rather than clean refusals. The boundary between the Java corpus and the 3B's pretraining knowledge on general programming topics is not perfectly sharp. |
| - **Corpus grounding:** The model answers Java questions correctly but draws on pretraining generalisation rather than reproducing specific training corpus facts — a fundamental limitation of generative fine-tuning without retrieval. |
|
|
| **Recommended next steps:** RAG layer over the source corpus for strict factual grounding; DPO with curated refusal preference pairs to sharpen the domain boundary on adjacent technical topics. |
|
|
| ## Production Deployment |
|
|
| LoRA adapters were merged into base weights via `merge_and_unload()` — this is a single self-contained checkpoint with no adapter dependency at runtime. Inference runs at under 2 GB VRAM. |
|
|
| ## Repository |
|
|
| Training notebooks and dataset generation pipeline: [GitHub — debarun23](https://github.com/debarun23) |
|
|
| --- |
|
|
| *Trained on NVIDIA GeForce RTX 5050 (8 GB VRAM) · June 2026 · Author: Debarun Das* |
