Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/NemoReRemix-12B-GGUF:# Run inference directly in the terminal:
llama-cli -hf QuantFactory/NemoReRemix-12B-GGUF:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/NemoReRemix-12B-GGUF:# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/NemoReRemix-12B-GGUF:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/NemoReRemix-12B-GGUF:# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/NemoReRemix-12B-GGUF:Use Docker
docker model run hf.co/QuantFactory/NemoReRemix-12B-GGUF:
QuantFactory/NemoReRemix-12B-GGUF
This is quantized version of MarinaraSpaghetti/NemoReRemix-12B created using llama.cpp
Original Model Card
Information
Details
Improved NemoRemix for storytelling and roleplay. Plus, this one can also be used as a general assistant model. The prose is pretty much the same, but it was made smarter, thanks to the addition of the amazing Migtissera's Tess model. I yeeted out Gryphe's Pantheon-RP, though, because it was trained with asterisks in mind, unlike the rest of the models in the merge, which caused it to mess the formatting from time to time; this one doesn't do that anymore. Hooray! All credits and thanks go to the amazing Migtissera, MistralAI, Anthracite, Sao10K and ShuttleAI for their amazing models.
Instruct
ChatML but Mistral Instruct should work too (theoretically). Important: remember to add <|im_end|> to custom stopping strings, otherwise it will appear in the output.
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
Parameters
I recommend running Temperature 1.0-1.2 with 0.1 Top A or 0.01-0.1 Min P, and with 0.8/1.75/2/0 DRY. Also works with lower Temperatures below 1.0. Nothing more needed.
Settings
You can use my exact settings from here (use the ones from the ChatML Base/Customized folder): https://huggingface.co/MarinaraSpaghetti/SillyTavern-Settings/tree/main.
GGUF
https://huggingface.co/MarinaraSpaghetti/NemoReRemix-GGUF
NemoReRemix-12B
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the della_linear merge method using E:\mergekit\mistralaiMistral-Nemo-Base-2407 as a base.
Models Merged
The following models were included in the merge:
- E:\mergekit\Sao10K_MN-12B-Lyra-v1
- E:\mergekit\mistralaiMistral-Nemo-Instruct-2407
- E:\mergekit\migtissera_Tess-3-Mistral-Nemo
- E:\mergekit\shuttleai_shuttle-2.5-mini
- E:\mergekit\anthracite-org_magnum-12b-v2
Configuration
The following YAML configuration was used to produce this model:
models:
- model: E:\mergekit\mistralaiMistral-Nemo-Instruct-2407
parameters:
weight: 0.1
density: 0.4
- model: E:\mergekit\Sao10K_MN-12B-Lyra-v1
parameters:
weight: 0.12
density: 0.5
- model: E:\mergekit\shuttleai_shuttle-2.5-mini
parameters:
weight: 0.2
density: 0.6
- model: E:\mergekit\migtissera_Tess-3-Mistral-Nemo
parameters:
weight: 0.25
density: 0.7
- model: E:\mergekit\anthracite-org_magnum-12b-v2
parameters:
weight: 0.33
density: 0.8
merge_method: della_linear
base_model: E:\mergekit\mistralaiMistral-Nemo-Base-2407
parameters:
epsilon: 0.05
lambda: 1
dtype: bfloat16
Ko-fi
Enjoying what I do? Consider donating here, thank you!
- Downloads last month
- 31
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Install from brew
# Start a local OpenAI-compatible server with a web UI: llama-server -hf QuantFactory/NemoReRemix-12B-GGUF:# Run inference directly in the terminal: llama-cli -hf QuantFactory/NemoReRemix-12B-GGUF: