File size: 3,035 Bytes
5426137 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | ---
license: apache-2.0
base_model: Qwen/Qwen3.5-27B
tags:
- code
- lora
- fine-tuned
- qwen3.5
- coding
- python
- javascript
- rust
datasets:
- ise-uiuc/Magicoder-Evol-Instruct-110K
- sahil2801/CodeAlpaca-20k
- Vezora/Tested-143k-Python-Alpaca
- iamtarun/python_code_instructions_18k_alpaca
language:
- en
pipeline_tag: text-generation
library_name: transformers
---
# Qwen3.5-27B-Coder
Fine-tuned version of [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) specialized for coding tasks.
## Training Details
| Parameter | Value |
|---|---|
| **Base model** | Qwen/Qwen3.5-27B (27B dense, Apache 2.0) |
| **Method** | LoRA r=64, alpha=128, all-linear projections |
| **Precision** | BF16 |
| **Framework** | HuggingFace SFTTrainer + PEFT + DeepSpeed ZeRO-2 |
| **Hardware** | 16× NVIDIA H200 SXM (141 GB each), 2 nodes |
| **GPU utilization** | 91% VRAM, 91-100% compute |
| **Training steps** | 250 (early stopped — loss plateaued) |
| **Training time** | ~4 hours |
| **Final loss** | 0.70 (down from 1.13, -40%) |
| **Final accuracy** | 80.0% token accuracy |
## Datasets
| Dataset | Examples | Purpose |
|---|---|---|
| Magicoder-Evol-Instruct-110K | 110K | Complex coding tasks from real GitHub code |
| CodeAlpaca-20K | 20K | Short tasks, broad language coverage |
| Tested-143k-Python-Alpaca | 143K | Execution-verified Python code |
| python_code_instructions_18k | 18K | Python idioms and patterns |
| **Total** | **291K** | |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"mahernaija/Qwen3.5-27B-Coder",
torch_dtype=torch.bfloat16,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("mahernaija/Qwen3.5-27B-Coder")
messages = [{"role": "user", "content": "Write a Python binary search function with type hints."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Evaluation
Fine-tuned model compared to base on 10 coding prompts:
- **7/10 prompts**: Fine-tuned model produces faster, more concise responses
- **Refactoring**: 70% faster response
- **Testing**: 59% faster response
- **Loss improvement**: 40% reduction over base model
## Training Infrastructure
Trained on Nebius.ai cloud using Soperator (Kubernetes-managed Slurm):
- 2 nodes × 8 NVIDIA H200 SXM GPUs
- InfiniBand 400 Gb/s inter-node communication
- DeepSpeed ZeRO-2 for optimizer/gradient sharding
- Gradient checkpointing with use_reentrant=False
## Limitations
- Primarily optimized for Python (70% of training data)
- Other languages (JS, Rust, Go) improved but less than Python
- Not trained on repo-level tasks (SWE-bench style)
- Best for function/class level code generation and bug fixing
## License
Apache 2.0 (same as base model)
|