| --- |
| license: apache-2.0 |
| base_model: unsloth/Qwen3.5-0.8B |
| tags: |
| - code-generation |
| - math-reasoning |
| - qwen3.5 |
| - lora |
| - sft |
| datasets: |
| - ise-uiuc/Magicoder-Evol-Instruct-110K |
| - meta-math/MetaMathQA |
| - AI-MO/NuminaMath-CoT |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # CMLM-0.8B |
|
|
| **C**oding + **M**ath **L**anguage **M**odel — a Qwen3.5-0.8B fine-tuned for code generation and mathematical reasoning. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| | :--- | :--- | |
| | Base Model | [unsloth/Qwen3.5-0.8B](https://huggingface.co/unsloth/Qwen3.5-0.8B) | |
| | Architecture | Qwen3.5 (Gated DeltaNet + Full Attention hybrid) | |
| | Parameters | 0.8B | |
| | Training Method | LoRA (r=16, α=32) | |
| | Precision | float32 (no quantization) | |
| | Max Context | 2048 tokens | |
| | Framework | Unsloth + TRL SFTTrainer | |
| | Hardware | NVIDIA Tesla T4 (16 GB VRAM) | |
|
|
| ## Training Data |
|
|
| | Dataset | Samples | Domain | |
| | :--- | :--- | :--- | |
| | [Magicoder-Evol-Instruct-110K](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) | 25,000 | Code instruction following | |
| | [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) | 25,000 | Mathematical reasoning | |
| | [NuminaMath-CoT](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT) | 15,000 | Math chain-of-thought | |
| | **Total** | **65,000** | | |
|
|
| ## Training Hyperparameters |
|
|
| ```yaml |
| learning_rate: 2e-4 |
| max_steps: 500 |
| per_device_train_batch_size: 2 |
| gradient_accumulation_steps: 8 |
| effective_batch_size: 16 |
| warmup_steps: 100 |
| optimizer: adamw_8bit |
| gradient_checkpointing: unsloth |
| lora_r: 16 |
| lora_alpha: 32 |
| lora_dropout: 0 |
| target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj] |
| packing: true |
| max_seq_length: 2048 |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("Natarizki/CMLM-0.8B", device_map="auto") |
| tokenizer = AutoTokenizer.from_pretrained("Natarizki/CMLM-0.8B") |
| |
| messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}] |
| inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate(inputs, max_new_tokens=256, temperature=0.7, do_sample=True) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Benchmarks |
|
|
| | Domain | CMLM-0.8B (tok/s) | Base Qwen3.5-0.8B (tok/s) | Avg Latency (CMLM) | |
| | :--- | :--- | :--- | :--- | |
| | Coding | 12.1 | 15.2 | 39.0s | |
| | Math | 14.8 | 15.1 | 15.9s | |
| | General | 15.0 | 15.1 | 17.0s | |
|
|
| > **Note:** CMLM generates longer, more detailed responses for coding tasks (hence lower tok/s but higher quality). Math and general domains show near-parity with base model throughput. Benchmarked on NVIDIA T4 with float32 inference via Unsloth. |
|
|
| ## Limitations |
|
|
| - Trained on 65K samples; may underperform on niche domains |
| - 2048 token context limit; not suitable for long-document tasks |
| - float32 training preserves accuracy but increases inference memory vs. quantized variants |
| - No vision capabilities despite Qwen3.5's native multimodal architecture |
|
|
| ## License |
|
|
| Apache 2.0 (inherits from Qwen3.5) |
|
|
| ## Acknowledgments |
|
|
| - [Qwen Team](https://huggingface.co/Qwen) for the base model |
| - [Unsloth](https://unsloth.ai/) for efficient T4-compatible training |
| - Dataset authors: Magicoder, MetaMath, NuminaMath teams |
