| | --- |
| | library_name: transformers |
| | tags: |
| | - small-lm |
| | - math |
| | - reasoning |
| | - slm |
| | - french |
| | - merge |
| | license: apache-2.0 |
| | base_model: |
| | - SamsungSAILMontreal/Qwen3-1.7B-Math |
| | - SamsungSAILMontreal/Qwen3-1.7B-Fr |
| | language: |
| | - fr |
| | - en |
| | --- |
| | |
| | # Qwen3-1.7B-Math-Fr |
| |
|
| | This model is obtained by merging [SamsungSAILMontreal/Qwen3-1.7B-Math](https://huggingface.co/SamsungSAILMontreal/Qwen3-1.7B-Math) and |
| | [SamsungSAILMontreal/Qwen3-1.7B-Fr](https://huggingface.co/SamsungSAILMontreal/Qwen3-1.7B-Fr). |
| | The model is used in the experiments described in https://bknyaz.github.io/blog/2026/meta-merge/. |
| | Single A100 was used for merging and evaluation. |
| |
|
| | The following versions were used for merge/eval: |
| |
|
| | - python >= 3.10 |
| | - torch : 2.9.0+cu128 |
| | - lm_eval : 0.4.9.1 |
| | - vllm : 0.11.1 |
| | - transformers : 4.57.6 |
| | - datasets : 3.2.0 |
| | - numpy : 2.2.6 |
| | |
| | ## Merging |
| | |
| | Merging was done using parameter averaging implemented in [merge_qwen.py](https://github.com/SamsungSAILMontreal/nino/blob/main/merge_qwen.py). |
| | |
| | ## Evaluation |
| | |
| | Evaluation was done with lm_eval on the test split of [gsm8k](https://huggingface.co/datasets/openai/gsm8k), |
| | [french_bench (avg score)](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/french_bench) and |
| | [gsm8k-fr](https://huggingface.co/datasets/cmh/gsm8k_fr): |
| |
|
| | ```bash |
| | python -m lm_eval --model vllm --model_args pretrained=${model},tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1 \ |
| | --tasks gsm8k,french_bench,gsm8k-fr --batch_size 1 --apply_chat_template=True --confirm_run_unsafe_code --trust_remote_code |
| | ``` |
| |
|
| | To evaluate on gsm8k-fr you can use our fork https://github.com/bknyaz/lm-evaluation-harness/tree/main/lm_eval/tasks/gsm8k. |
| | |
| | ### Results |
| | |
| | | Model | gsm8k | french | gsm8k-fr | avg | |
| | |-----------------------|--------|--------|----------|-------| |
| | | Qwen3-1.7B | 20.6 | 26.2 | 20.2 | 22.3 | |
| | | Qwen3-1.7B-Math | 62.1 | 28.3 | 41.5 | 43.9 | |
| | | Qwen3-1.7B-Fr | 60.9 | 32.8 | 43.9 | 45.9 | |
| | | Qwen3-1.7B-Math-Fr | 64.0 | 31.4 | 46.9 | 47.4 | |
| | |
| | ## License |
| | |
| | Please refer to the license of the base models [SamsungSAILMontreal/Qwen3-1.7B-Math](https://huggingface.co/SamsungSAILMontreal/Qwen3-1.7B-Math) and |
| | [SamsungSAILMontreal/Qwen3-1.7B-Fr](https://huggingface.co/SamsungSAILMontreal/Qwen3-1.7B-Fr). |
