Instructions to use Undi95/Miqu-70B-Alpaca-DPO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Undi95/Miqu-70B-Alpaca-DPO with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Undi95/Miqu-70B-Alpaca-DPO")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Undi95/Miqu-70B-Alpaca-DPO")
model = AutoModelForCausalLM.from_pretrained("Undi95/Miqu-70B-Alpaca-DPO")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Undi95/Miqu-70B-Alpaca-DPO with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Undi95/Miqu-70B-Alpaca-DPO"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Undi95/Miqu-70B-Alpaca-DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Undi95/Miqu-70B-Alpaca-DPO

SGLang

How to use Undi95/Miqu-70B-Alpaca-DPO with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Undi95/Miqu-70B-Alpaca-DPO" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Undi95/Miqu-70B-Alpaca-DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Undi95/Miqu-70B-Alpaca-DPO" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Undi95/Miqu-70B-Alpaca-DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Undi95/Miqu-70B-Alpaca-DPO with Docker Model Runner:
```
docker model run hf.co/Undi95/Miqu-70B-Alpaca-DPO
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Miqu DPO

Miqu DPO is the same model than Miqu, with a DPO trained on MiquMaid v2 on Alpaca format, it was done for the purpose to try to uncensor further Miqu and make Alpaca prompt more usable with base Miqu. Also, this will be one of the base for MiquMaid-v2-2x70B-DPO.

Miqu base is REALLY censored outside RP, this LoRA let him reply a little more thing, but that's it. To have his full potential, it need to be in a merge/MoE of MiquMaid, since the loRA was based for MiquMaid, not Miqu base. I still let it public for who want it.

It uncensor a little the model, but keep some warning. Sometime reply really unethically.

Description

This repo contains FP16 files of Miqu-70B-DPO.

Dataset used

NobodyExistsOnTheInternet/ToxicDPOqa
Undi95/toxic-dpo-v0.1-NoWarning

Prompt format: Alpaca

### Instruction:
{prompt}

### Input:
{input}

### Response:
{output}

Or simple Mistral format (but the uncensoring was done on Alpaca, so Alpaca is recommanded).

Others

If you want to support me, you can here.

Downloads last month: 900

Model tree for Undi95/Miqu-70B-Alpaca-DPO

Merges

1 model

Quantizations

2 models