Instructions to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantFactory/MN-Chunky-Lotus-12B-GGUF", dtype="auto")

llama-cpp-python

How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/MN-Chunky-Lotus-12B-GGUF",
	filename="MN-Chunky-Lotus-12B.Q2_K.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/MN-Chunky-Lotus-12B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/MN-Chunky-Lotus-12B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/MN-Chunky-Lotus-12B-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/MN-Chunky-Lotus-12B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/MN-Chunky-Lotus-12B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.MN-Chunky-Lotus-12B-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/MN-Chunky-Lotus-12B-GGUF

This is quantized version of FallenMerick/MN-Chunky-Lotus-12B created using llama.cpp

Original Model Card

MN-Chunky-Lotus-12B

I had originally planned to use this model for future/further merges, but decided to go ahead and release it since it scored rather high on my local EQ Bench testing (79.58 w/ 100% parsed @ 8-bit).
Bear in mind that most models tend to score a bit higher on my own local tests as compared to their posted scores. Still, its the highest score I've personally seen from all the models I've tested.
Its a decent model, with great emotional intelligence and acceptable adherence to various character personalities. It does a good job at roleplaying despite being a bit bland at times.

Overall, I like the way it writes, but it has a few formatting issues that show up from time to time, and it has an uncommon tendency to paste walls of character feelings/intentions at the end of some outputs without any prompting. This is something I hope to correct with future iterations.

This is a merge of pre-trained language models created using mergekit.

GGUF quants:

Merge Details

Merge Method

This model was merged using the TIES merge method.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Epiculous/Violet_Twilight-v0.2
    parameters:
      weight: 1.0
      density: 1.0
  - model: nbeerbower/mistral-nemo-gutenberg-12B-v4
    parameters:
      weight: 1.0
      density: 0.54
  - model: flammenai/Mahou-1.5-mistral-nemo-12B
    parameters:
      weight: 1.0
      density: 0.26
merge_method: ties
base_model: TheDrummer/Rocinante-12B-v1.1
parameters:
  normalize: true
dtype: bfloat16

The idea behind this recipe was to take the long-form writing capabilities of Gutenberg, curtail it a bit with the very short output formatting of Mahou, and use Violet Twilight as an extremely solid roleplaying foundation underneath.
Rocinante is used as the base model in this merge in order to really target the delta weights from Gutenberg, since those seemed to have the highest impact on the resulting EQ of the model.

Special shoutout to @matchaaaaa for helping with testing, and for all the great model recommendations. Also, for just being an all around great person who's really inspired and motivated me to continue merging and working on models.