| | --- |
| | library_name: transformers |
| | tags: |
| | - small-lm |
| | - math |
| | - reasoning |
| | - slm |
| | - french |
| | - merge |
| | license: apache-2.0 |
| | base_model: |
| | - SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Math |
| | - SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Fr |
| | language: |
| | - fr |
| | - en |
| | --- |
| | |
| | # Qwen3-4B-Instruct-2507-Math-Fr |
| |
|
| | This model is obtained by merging [SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Math](https://huggingface.co/SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Math) and |
| | [SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Fr](https://huggingface.co/SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Fr). |
| | The model is used in the experiments described in https://bknyaz.github.io/blog/2026/meta-merge/. |
| | Single A100 was used for merging and evaluation. |
| |
|
| | The following versions were used for merge/eval: |
| |
|
| | - python >= 3.10 |
| | - torch : 2.9.0+cu128 |
| | - lm_eval : 0.4.9.1 |
| | - vllm : 0.11.1 |
| | - transformers : 4.57.6 |
| | - datasets : 3.2.0 |
| | - numpy : 2.2.6 |
| | |
| | ## Merging |
| | |
| | Merging was done using parameter averaging implemented in [merge_qwen.py](https://github.com/SamsungSAILMontreal/nino/blob/main/merge_qwen.py). |
| | |
| | ## Evaluation |
| | |
| | Evaluation was done with lm_eval on the test split of [gsm8k](https://huggingface.co/datasets/openai/gsm8k), |
| | [french_bench (avg score)](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/french_bench) and |
| | [gsm8k-fr](https://huggingface.co/datasets/cmh/gsm8k_fr): |
| |
|
| | ```bash |
| | python -m lm_eval --model vllm --model_args pretrained=${model},tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1 \ |
| | --tasks gsm8k,french_bench,gsm8k-fr --batch_size 1 --apply_chat_template=True --confirm_run_unsafe_code --trust_remote_code |
| | ``` |
| |
|
| | To evaluate on gsm8k-fr you can use our fork https://github.com/bknyaz/lm-evaluation-harness/tree/main/lm_eval/tasks/gsm8k. |
| | |
| | ### Results |
| | |
| | | Model | gsm8k | french | gsm8k-fr | avg | |
| | |----------------------------------|--------|--------|----------|-------| |
| | | Qwen3-4B-Instruct-2507 | 80.4 | 43.1 | 66.0 | 63.2 | |
| | | Qwen3-4B-Instruct-2507-Math | 76.8 | 43.0 | 65.3 | 61.7 | |
| | | Qwen3-4B-Instruct-2507-Fr | 72.3 | 45.7 | 60.7 | 59.6 | |
| | | Qwen3-4B-Instruct-2507-Math-Fr | 81.3 | 45.0 | 70.6 | 65.6 | |
| | |
| | ## License |
| | |
| | Please refer to the license of the base models [SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Math](https://huggingface.co/SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Math) and |
| | [SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Fr](https://huggingface.co/SamsungSAILMontreal/Qwen3-4B-Instruct-2507-Fr). |
