library_name: transformers
tags:
- mergekit
- merge
base_model: []
model-index:
- name: NemoReRemix-12B
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 33.43
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MarinaraSpaghetti/NemoReRemix-12B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 36.12
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MarinaraSpaghetti/NemoReRemix-12B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 6.42
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MarinaraSpaghetti/NemoReRemix-12B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 9.06
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MarinaraSpaghetti/NemoReRemix-12B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 15.67
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MarinaraSpaghetti/NemoReRemix-12B
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 28.87
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=MarinaraSpaghetti/NemoReRemix-12B
name: Open LLM Leaderboard
Information
Details
Improved NemoRemix for storytelling and roleplay. Plus, this one can also be used as a general assistant model. The prose is pretty much the same, but it was made smarter, thanks to the addition of the amazing Migtissera's Tess model. I yeeted out Gryphe's Pantheon-RP, though, because it was trained with asterisks in mind, unlike the rest of the models in the merge, which caused it to mess the formatting from time to time; this one doesn't do that anymore. Hooray! All credits and thanks go to the amazing Migtissera, MistralAI, Anthracite, Sao10K and ShuttleAI for their amazing models.
Instruct
ChatML but Mistral Instruct should work too (theoretically). Important: remember to add <|im_end|> to custom stopping strings, otherwise it will appear in the output.
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
Parameters
I recommend running Temperature 1.0-1.2 with 0.1 Top A or 0.01-0.1 Min P, and with 0.8/1.75/2/0 DRY. Also works with lower Temperatures below 1.0. Nothing more needed.
Settings
You can use my exact settings from here (use the ones from the ChatML Base/Customized folder): https://huggingface.co/MarinaraSpaghetti/SillyTavern-Settings/tree/main.
GGUF
https://huggingface.co/MarinaraSpaghetti/NemoReRemix-GGUF
NemoReRemix-12B
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the della_linear merge method using E:\mergekit\mistralaiMistral-Nemo-Base-2407 as a base.
Models Merged
The following models were included in the merge:
- E:\mergekit\Sao10K_MN-12B-Lyra-v1
- E:\mergekit\mistralaiMistral-Nemo-Instruct-2407
- E:\mergekit\migtissera_Tess-3-Mistral-Nemo
- E:\mergekit\shuttleai_shuttle-2.5-mini
- E:\mergekit\anthracite-org_magnum-12b-v2
Configuration
The following YAML configuration was used to produce this model:
models:
- model: E:\mergekit\mistralaiMistral-Nemo-Instruct-2407
parameters:
weight: 0.1
density: 0.4
- model: E:\mergekit\Sao10K_MN-12B-Lyra-v1
parameters:
weight: 0.12
density: 0.5
- model: E:\mergekit\shuttleai_shuttle-2.5-mini
parameters:
weight: 0.2
density: 0.6
- model: E:\mergekit\migtissera_Tess-3-Mistral-Nemo
parameters:
weight: 0.25
density: 0.7
- model: E:\mergekit\anthracite-org_magnum-12b-v2
parameters:
weight: 0.33
density: 0.8
merge_method: della_linear
base_model: E:\mergekit\mistralaiMistral-Nemo-Base-2407
parameters:
epsilon: 0.05
lambda: 1
dtype: bfloat16
Ko-fi
Enjoying what I do? Consider donating here, thank you!
https://ko-fi.com/spicy_marinara
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 21.59 |
| IFEval (0-Shot) | 33.43 |
| BBH (3-Shot) | 36.12 |
| MATH Lvl 5 (4-Shot) | 6.42 |
| GPQA (0-shot) | 9.06 |
| MuSR (0-shot) | 15.67 |
| MMLU-PRO (5-shot) | 28.87 |