| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-27B |
| tags: |
| - code |
| - lora |
| - fine-tuned |
| - qwen3.5 |
| - coding |
| - python |
| - javascript |
| - rust |
| datasets: |
| - ise-uiuc/Magicoder-Evol-Instruct-110K |
| - sahil2801/CodeAlpaca-20k |
| - Vezora/Tested-143k-Python-Alpaca |
| - iamtarun/python_code_instructions_18k_alpaca |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: transformers |
| --- |
| |
| # Qwen3.5-27B-Coder |
|
|
| Fine-tuned version of [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) specialized for coding tasks. |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| |---|---| |
| | **Base model** | Qwen/Qwen3.5-27B (27B dense, Apache 2.0) | |
| | **Method** | LoRA r=64, alpha=128, all-linear projections | |
| | **Precision** | BF16 | |
| | **Framework** | HuggingFace SFTTrainer + PEFT + DeepSpeed ZeRO-2 | |
| | **Hardware** | 16× NVIDIA H200 SXM (141 GB each), 2 nodes | |
| | **GPU utilization** | 91% VRAM, 91-100% compute | |
| | **Training steps** | 250 (early stopped — loss plateaued) | |
| | **Training time** | ~4 hours | |
| | **Final loss** | 0.70 (down from 1.13, -40%) | |
| | **Final accuracy** | 80.0% token accuracy | |
|
|
| ## Datasets |
|
|
| | Dataset | Examples | Purpose | |
| |---|---|---| |
| | Magicoder-Evol-Instruct-110K | 110K | Complex coding tasks from real GitHub code | |
| | CodeAlpaca-20K | 20K | Short tasks, broad language coverage | |
| | Tested-143k-Python-Alpaca | 143K | Execution-verified Python code | |
| | python_code_instructions_18k | 18K | Python idioms and patterns | |
| | **Total** | **291K** | | |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "mahernaija/Qwen3.5-27B-Coder", |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| tokenizer = AutoTokenizer.from_pretrained("mahernaija/Qwen3.5-27B-Coder") |
| |
| messages = [{"role": "user", "content": "Write a Python binary search function with type hints."}] |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(text, return_tensors="pt").to("cuda") |
| outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
| |
| ## Evaluation |
| |
| Fine-tuned model compared to base on 10 coding prompts: |
| - **7/10 prompts**: Fine-tuned model produces faster, more concise responses |
| - **Refactoring**: 70% faster response |
| - **Testing**: 59% faster response |
| - **Loss improvement**: 40% reduction over base model |
| |
| ## Training Infrastructure |
| |
| Trained on Nebius.ai cloud using Soperator (Kubernetes-managed Slurm): |
| - 2 nodes × 8 NVIDIA H200 SXM GPUs |
| - InfiniBand 400 Gb/s inter-node communication |
| - DeepSpeed ZeRO-2 for optimizer/gradient sharding |
| - Gradient checkpointing with use_reentrant=False |
| |
| ## Limitations |
| |
| - Primarily optimized for Python (70% of training data) |
| - Other languages (JS, Rust, Go) improved but less than Python |
| - Not trained on repo-level tasks (SWE-bench style) |
| - Best for function/class level code generation and bug fixing |
| |
| ## License |
| |
| Apache 2.0 (same as base model) |
| |
