Instructions to use af1tang/personaGPT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use af1tang/personaGPT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="af1tang/personaGPT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("af1tang/personaGPT")
model = AutoModelForCausalLM.from_pretrained("af1tang/personaGPT", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use af1tang/personaGPT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "af1tang/personaGPT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "af1tang/personaGPT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/af1tang/personaGPT

SGLang

How to use af1tang/personaGPT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "af1tang/personaGPT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "af1tang/personaGPT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "af1tang/personaGPT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "af1tang/personaGPT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use af1tang/personaGPT with Docker Model Runner:
```
docker model run hf.co/af1tang/personaGPT
```

Can't load tokenizer

by vtoth - opened Nov 7, 2022

Discussion

vtoth

Nov 7, 2022

Hey, I'm trying to run your example code but got the following error:

Traceback (most recent call last):
  File "/home/vik/fmri_vis/psych.py", line 6, in <module>
    tokenizer = AutoTokenizer.from_pretrained("af1tang/personaGPT")
  File "/home/vik/fmri_vis/venv/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 608, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/vik/fmri_vis/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1775, in from_pretrained
    return cls._from_pretrained(
  File "/home/vik/fmri_vis/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1930, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/vik/fmri_vis/venv/lib/python3.8/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 138, in __init__
    super().__init__(
  File "/home/vik/fmri_vis/venv/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: No such file or directory (os error 2)```

It could not load the tokenizer locally; the same issue presents itself also on the Model card page.

vtoth

Nov 8, 2022

Solved it by adding use_fast=False to AutoTokenizer.from_pretrained

vtoth changed discussion status to closed Nov 8, 2022

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment