Instructions to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="PocketDoc/Dans-PersonalityEngine-V1.3.0-12b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("PocketDoc/Dans-PersonalityEngine-V1.3.0-12b")
model = AutoModelForCausalLM.from_pretrained("PocketDoc/Dans-PersonalityEngine-V1.3.0-12b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PocketDoc/Dans-PersonalityEngine-V1.3.0-12b

SGLang

How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with Docker Model Runner:
```
docker model run hf.co/PocketDoc/Dans-PersonalityEngine-V1.3.0-12b
```

the struggle

by Utochi - opened Mar 25

Discussion

Utochi

Mar 25

For the life of me, i cannot figure out how to use this model without it becoming incoherent after some time.

I've done everything i can think of, i use Danchat template, ive tried using chatml template, setting temp to 1 and top to .95, (with no other settings applied)

tried using dry, shifting other settings, using top k or typical sampling.. and not using them of course, lowering temperature (slight success there, it does make it more coherent BUT STILL INCOHERENT)

ive tried everything i can think of, this model hates me, short of sacrificing my own blood, i dont know what it wants. i KNOW there is a happy place for this model, people have clearly gotten good things from it. i just cant figure it out. ive even tried searching reddit and other places on this and the 24b model, people do have trouble with it. and some people dont. there is a secret sauce, an ingredient that i cannot find somewhere that makes this model happy, i just dont know what it is.

Ive even tried enabling unified kv cache and disabling it, fiddling with those cache settings and whatnot but alas, nothing it seems that i do makes this model happy. i cant tell its a good model, i do get good results.. it just decays after a few messages and forgets how to english.

So what does it want? my first born? a demon ritual of some sorts? should i spin around three times and bark like a dog? i have no clue. it shall continue tormenting me.

Dvitry

Apr 11

Check that the correct template is applied to your prompt. For example, llama.cpp uses a third-party template unless the --jinja flag is explicitly specified. An example of the correct formatting is provided on the model page (remember to ensure the bos token is also present).

incorrect formatting: <|system|>\nsys msg<|user|>\nreq<|assistant|>

correct formatting: <|system|>sys msg<|endoftext|><|user|>req<|endoftext|><|assistant|>

correct with bos: [gMASK]<sop><|system|>sys msg<|endoftext|><|user|>req<|endoftext|><|assistant|>

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment