Instructions to use Monero/Pygmalion-Metharme-7b-4bit-TopScore with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Monero/Pygmalion-Metharme-7b-4bit-TopScore with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Monero/Pygmalion-Metharme-7b-4bit-TopScore")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Monero/Pygmalion-Metharme-7b-4bit-TopScore")
model = AutoModelForCausalLM.from_pretrained("Monero/Pygmalion-Metharme-7b-4bit-TopScore")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Monero/Pygmalion-Metharme-7b-4bit-TopScore with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Monero/Pygmalion-Metharme-7b-4bit-TopScore"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Monero/Pygmalion-Metharme-7b-4bit-TopScore",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Monero/Pygmalion-Metharme-7b-4bit-TopScore

SGLang

How to use Monero/Pygmalion-Metharme-7b-4bit-TopScore with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Monero/Pygmalion-Metharme-7b-4bit-TopScore" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Monero/Pygmalion-Metharme-7b-4bit-TopScore",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Monero/Pygmalion-Metharme-7b-4bit-TopScore" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Monero/Pygmalion-Metharme-7b-4bit-TopScore",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Monero/Pygmalion-Metharme-7b-4bit-TopScore with Docker Model Runner:
```
docker model run hf.co/Monero/Pygmalion-Metharme-7b-4bit-TopScore
```

Bug with oobabooga

by M14w - opened May 2, 2023

Discussion

M14w

May 2, 2023

Example of output:

You are in an abandoned house, surrounded by darkness. There's no light source nearby except for some candles you found scattered around. The only sound that can be heard is your own breathing as you try to find out what happened here...

You see a door leading outside of this room, but it seems locked with a deadbolt. To open it, you need to use something else. What do you have?

var input = prompt("What do you want to use?"),

choice;

if (input === "key") {

choice = "use key";

} else if (input === "candle" || input === "flame") {

choice = "light up the dark";

} else {

console.log(You don't seem to know how to unlock the door.);

return false;

}

choice();

Comment: This answer should probably include code examples rather than just linking to other resources.

Whatever i do, even the times where it works, after a serie of texts, at some point i get code.

Since i guess that you tested everything before publishing it, what am i missing here ?
What interface do you recommend us to use ?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment