FuseChat: Knowledge Fusion of Chat Models
Paper
•
2408.07990
•
Published
•
14
Intro
Experimental merge of multiple Llama 3.2 3B models, guided by MoonRide-Index-v7. Created with mergekit.
Merge Details
Merge Method
This model was merged using the SCE merge method using meta-llama/Llama-3.2-3B as a base.
Models Merged
The following models were included in the merge:
- bunnycore/Llama-3.2-3B-Sci-Think
- FuseAI/FuseChat-Llama-3.2-3B-Instruct
- theprint/ReWiz-Llama-3.2-3B
- bunnycore/Llama-3.2-3B-Mix-Skill
Configuration
The following YAML configuration was used to produce this model:
models:
- model: bunnycore/Llama-3.2-3B-Mix-Skill
- model: bunnycore/Llama-3.2-3B-Sci-Think
- model: FuseAI/FuseChat-Llama-3.2-3B-Instruct
- model: theprint/ReWiz-Llama-3.2-3B
base_model: meta-llama/Llama-3.2-3B
tokenizer:
source: meta-llama/Llama-3.2-3B-Instruct
merge_method: sce
parameters:
normalize: true
dtype: float32
out_dtype: float16
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 20.14 |
| IFEval (0-Shot) | 49.25 |
| BBH (3-Shot) | 22.69 |
| MATH Lvl 5 (4-Shot) | 16.16 |
| GPQA (0-shot) | 3.69 |
| MuSR (0-shot) | 5.50 |
| MMLU-PRO (5-shot) | 23.57 |
- Downloads last month
- 5
Model tree for MoonRide/Llama-3.2-3B-Khelavaster
Merge model
this model
Paper for MoonRide/Llama-3.2-3B-Khelavaster
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard49.250
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard22.690
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard16.160
- acc_norm on GPQA (0-shot)Open LLM Leaderboard3.690
- acc_norm on MuSR (0-shot)Open LLM Leaderboard5.500
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard23.570