Instructions to use FPHam/Plot_BOT_V3_13b_GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FPHam/Plot_BOT_V3_13b_GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FPHam/Plot_BOT_V3_13b_GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FPHam/Plot_BOT_V3_13b_GPTQ")
model = AutoModelForCausalLM.from_pretrained("FPHam/Plot_BOT_V3_13b_GPTQ")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FPHam/Plot_BOT_V3_13b_GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FPHam/Plot_BOT_V3_13b_GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FPHam/Plot_BOT_V3_13b_GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/FPHam/Plot_BOT_V3_13b_GPTQ

SGLang

How to use FPHam/Plot_BOT_V3_13b_GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FPHam/Plot_BOT_V3_13b_GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FPHam/Plot_BOT_V3_13b_GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FPHam/Plot_BOT_V3_13b_GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FPHam/Plot_BOT_V3_13b_GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use FPHam/Plot_BOT_V3_13b_GPTQ with Docker Model Runner:
```
docker model run hf.co/FPHam/Plot_BOT_V3_13b_GPTQ
```

FPHam commited on Oct 15, 2023

Commit

3f2e147

1 Parent(s): 9d12681

Update README.md

Browse files

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -58,4 +58,4 @@ Write me a blurb for the back of the book
 The model will be able to borrow sub-plots and other elements from existing literary works. It may even find some familiar character-names; after all, as LLM and llama, it will have access to all the world's libraries and literature. Well, not quite all of them, but most of them. And so, when you stumble across Elizabeth Bennet and John Snow getting personal on the roadside at the end of a dark and stormy night - well, there it is!
-V3 is not a replaecement for V2, it is just differernt.


58
59	The model will be able to borrow sub-plots and other elements from existing literary works. It may even find some familiar character-names; after all, as LLM and llama, it will have access to all the world's libraries and literature. Well, not quite all of them, but most of them. And so, when you stumble across Elizabeth Bennet and John Snow getting personal on the roadside at the end of a dark and stormy night - well, there it is!
60
61	+ Anyway, Version Three isn't intended as a replacement for Version Two. It's just different.