| --- |
| license: mit |
| library_name: transformers |
| tags: |
| - mergekit |
| - merge |
| base_model: |
| - ClaudioItaly/Evolutionstory-7B-v2.2 |
| model-index: |
| - name: Albacus |
| results: |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: IFEval (0-Shot) |
| type: HuggingFaceH4/ifeval |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: inst_level_strict_acc and prompt_level_strict_acc |
| value: 46.67 |
| name: strict accuracy |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ClaudioItaly/Albacus |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: BBH (3-Shot) |
| type: BBH |
| args: |
| num_few_shot: 3 |
| metrics: |
| - type: acc_norm |
| value: 31.64 |
| name: normalized accuracy |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ClaudioItaly/Albacus |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MATH Lvl 5 (4-Shot) |
| type: hendrycks/competition_math |
| args: |
| num_few_shot: 4 |
| metrics: |
| - type: exact_match |
| value: 6.42 |
| name: exact match |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ClaudioItaly/Albacus |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: GPQA (0-shot) |
| type: Idavidrein/gpqa |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: acc_norm |
| value: 2.91 |
| name: acc_norm |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ClaudioItaly/Albacus |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MuSR (0-shot) |
| type: TAUR-Lab/MuSR |
| args: |
| num_few_shot: 0 |
| metrics: |
| - type: acc_norm |
| value: 10.66 |
| name: acc_norm |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ClaudioItaly/Albacus |
| name: Open LLM Leaderboard |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: MMLU-PRO (5-shot) |
| type: TIGER-Lab/MMLU-Pro |
| config: main |
| split: test |
| args: |
| num_few_shot: 5 |
| metrics: |
| - type: acc |
| value: 24.05 |
| name: accuracy |
| source: |
| url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ClaudioItaly/Albacus |
| name: Open LLM Leaderboard |
| --- |
| |
|
|
| # merge |
|
|
| This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). |
|
|
| ## Merge Details |
| ### Merge Method |
|
|
| This model was merged using the passthrough merge method. |
|
|
| ### Models Merged |
|
|
| The following models were included in the merge: |
| * [ClaudioItaly/Evolutionstory-7B-v2.2](https://huggingface.co/ClaudioItaly/Evolutionstory-7B-v2.2) |
|
|
| ### Configuration |
|
|
| The following YAML configuration was used to produce this model: |
|
|
| ```yaml |
| |
| slices: |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 0 |
| - 4 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 3 |
| - 4 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 4 |
| - 8 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 7 |
| - 8 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 8 |
| - 12 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 11 |
| - 12 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 12 |
| - 16 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 15 |
| - 16 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 16 |
| - 20 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 19 |
| - 20 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 20 |
| - 24 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 23 |
| - 24 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 24 |
| - 28 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 27 |
| - 28 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 28 |
| - 32 |
| - sources: |
| - model: ClaudioItaly/Evolutionstory-7B-v2.2 |
| layer_range: |
| - 31 |
| - 32 |
| parameters: |
| scale: |
| - filter: o_proj |
| value: 0 |
| - filter: down_proj |
| value: 0 |
| - value: 1 |
| merge_method: passthrough |
| dtype: bfloat16 |
| |
| ``` |
| # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
| Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ClaudioItaly__Albacus) |
|
|
| | Metric |Value| |
| |-------------------|----:| |
| |Avg. |20.39| |
| |IFEval (0-Shot) |46.67| |
| |BBH (3-Shot) |31.64| |
| |MATH Lvl 5 (4-Shot)| 6.42| |
| |GPQA (0-shot) | 2.91| |
| |MuSR (0-shot) |10.66| |
| |MMLU-PRO (5-shot) |24.05| |
|
|
|
|
