Aubins/MathQwen3-4B

Model Details

Model Description

• Model name: MathQwen3-4B
• Model type: Causal language model with LoRA adapters
• Base model: Qwen3-4B
• Fine-tuning method: Supervised Fine-Tuning (SFT) with LoRA
• Primary task: Mathematical problem solving and step-by-step reasoning
• Language: Primarily English
• Maximum sequence length during training: 2,048 tokens
• Base model loading: 4-bit quantization
• Trainable parameters: 33,030,144
• Base model parameters: 4,055,498,240
• Trainable parameter ratio: 0.81%

MathQwen3-4B is a parameter-efficient fine-tuning of Qwen3-4B for mathematical problem solving. The model was trained with supervised fine-tuning on a 20,000-example subset of MetaMathQA, using LoRA adapters applied to the attention and MLP projection layers.

Only approximately 0.81% of the base model's parameters were updated during fine-tuning, while the underlying Qwen3-4B weights remained frozen.

Important: This repository contains the LoRA adapter and tokenizer rather than a standalone copy of the full base model. The corresponding base model is required to use the adapter.

Training Details

Training Data

Training used the meta-math/MetaMathQA dataset.

A subset of 20,000 examples was selected using random seed 42. The selected examples were then split into:

• Training: 18,000 examples
• Evaluation: 2,000 examples
• Train/evaluation split: 90% / 10%

The dataset was shuffled before subset selection, and the train/evaluation split was performed with the same fixed seed for reproducibility.

Each example was formatted using the Qwen chat template with a system instruction asking the model to solve the mathematical problem step by step and provide a final answer. The original question was used as the user message and the corresponding MetaMathQA response as the assistant message.

Training Procedure

The model was fine-tuned using supervised fine-tuning (SFT) with parameter-efficient Low-Rank Adaptation (LoRA).

LoRA adapters were applied to the following projection modules:

• q_proj
• k_proj
• v_proj
• o_proj
• gate_proj
• up_proj
• down_proj

The base model was loaded in 4-bit quantization during training.

Model Parameters

Parameter	Value
Base model parameters	4,055,498,240
Trainable LoRA parameters	33,030,144
Trainable parameter ratio	0.81%
Quantized loading	4-bit

The relatively small number of trainable parameters makes this a parameter-efficient adaptation of Qwen3-4B.

LoRA Configuration

Parameter	Value
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.0
LoRA bias	none
Rank-stabilized LoRA (RSLoRA)	Disabled
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Hyperparameters

Parameter	Value
Epochs	2
Training examples	18,000
Optimization steps	1,126
Per-device batch size	32
Gradient accumulation steps	1
Effective batch size	32
Learning rate	2e-4
LR scheduler	Cosine
Warmup ratio	0.03
Weight decay	0.0
Optimizer	adamw_8bit
Maximum sequence length	2,048
Gradient checkpointing	Unsloth
Precision	BF16
Random seed	42
Packing	Disabled
Logging frequency	Every 20 steps

Training was performed on a single NVIDIA L40S GPU. The L40S supports BF16, and the published training run therefore used BF16 precision.

Training Loss

The training run was monitored using Weights & Biases.

A training-loss curve can be included here to provide a visual overview of optimization and convergence:

The training loss should be interpreted only as an optimization diagnostic. It does not directly measure mathematical reasoning accuracy or generalization.

Evaluation

Testing Data

The fine-tuning run reserved 2,000 examples from the selected MetaMathQA subset for evaluation.

In addition to the in-training evaluation set, independent benchmark evaluation is planned on GSM8K and potentially Hendrycks' MATH.

Evaluation Methodology

The planned evaluation will compare the original Qwen3-4B base model against MathQwen3-4B using the same evaluation procedure and datasets.

This comparison is intended to measure whether the mathematical fine-tuning improves mathematical problem-solving performance relative to the base model.

Metrics

The primary planned metric is mathematical answer accuracy on GSM8K and, if included, Hendrycks' MATH.

Additional metrics may be reported depending on the final evaluation setup.

Results

Benchmark results are not yet available and will be added after evaluation.

The planned comparison is:

Model	GSM8K	Hendrycks' MATH
Qwen3-4B	—	—
MathQwen3-4B	—	—

Results should be interpreted as a comparison between the original base model and the fine-tuned adapter under the same evaluation protocol.

Technical Specifications

Model Architecture and Objective

MathQwen3-4B is based on Qwen3-4B and uses Low-Rank Adaptation (LoRA) for parameter-efficient supervised fine-tuning.

The training objective is causal language modeling through supervised fine-tuning on formatted mathematical question-and-answer conversations.

The training data was formatted with a system instruction encouraging step-by-step mathematical problem solving and a final answer.

Environmental Impact

Training was performed using a single NVIDIA L40S GPU on Modal.

• Hardware Type: NVIDIA L40S
• GPU count: 1
• Training precision: BF16
• Training time: 1 hour 32 minutes 20 seconds
• Cloud Provider: Modal
• Compute Region: Not recorded
• Carbon Emitted: Not calculated

The total training time corresponds to two training sessions of 35 minutes 19 seconds and 57 minutes 01 second, respectively.

Carbon emissions can be estimated using the Machine Learning Impact calculator described by Lacoste et al. (2019). An exact carbon estimate is not reported because the necessary information about the compute region and associated energy/carbon intensity was not recorded.

Technical Specifications

Compute Infrastructure

Training was performed using Modal with a single NVIDIA L40S GPU.

Hardware

• GPU: NVIDIA L40S
• Number of GPUs: 1
• Training precision: BF16
• Total training time: 1 hour 32 minutes 20 seconds
• Maximum configured job duration: 6 hours

The training consisted of two sessions:

Session	Duration
Session 1	35 min 19 sec
Session 2	57 min 01 sec
Total	1 h 32 min 20 sec

Software

The training environment included:

• Python 3.11
• PyTorch
• Hugging Face Transformers (version==4.54.0)
• Hugging Face TRL (version==0.19.1)
• Hugging Face PEFT (version==0.16.0)
• Hugging Face Datasets (version==3.6.0)
• Unsloth (cuda128-torch270, version==2025.7.8, and unsloth_zoo==2025.7.10)
• Weights & Biases (version==0.28.1)
• Modal (version==1.5.3)

Training Efficiency

The fine-tuning updated only 33,030,144 parameters, corresponding to 0.81% of the 4,055,498,240 parameters in the base Qwen3-4B model.

This parameter-efficient setup substantially reduces the number of parameters that need to be optimized compared with full-model fine-tuning.

Limitations

The model has not yet been comprehensively benchmarked against the base model. In particular, GSM8K and potentially Hendrycks' MATH evaluations are planned but are not yet reported.

Consequently, the current model card should not make claims that MathQwen3-4B is more accurate than Qwen3-4B on mathematical benchmarks until those evaluations have been completed.

The training loss and held-out SFT evaluation loss, if reported, should likewise not be interpreted as direct evidence of benchmark-level mathematical reasoning performance.

Additional Notes

This model card has been mostly generated by a LLM based on the finetuning informations.