Instructions to use NousResearch/Nous-Capybara-34B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NousResearch/Nous-Capybara-34B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NousResearch/Nous-Capybara-34B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Nous-Capybara-34B")
model = AutoModelForCausalLM.from_pretrained("NousResearch/Nous-Capybara-34B")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NousResearch/Nous-Capybara-34B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NousResearch/Nous-Capybara-34B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Capybara-34B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/NousResearch/Nous-Capybara-34B

SGLang

How to use NousResearch/Nous-Capybara-34B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NousResearch/Nous-Capybara-34B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Capybara-34B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NousResearch/Nous-Capybara-34B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Capybara-34B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use NousResearch/Nous-Capybara-34B with Docker Model Runner:
```
docker model run hf.co/NousResearch/Nous-Capybara-34B
```

YiTokenizer doesn't exist

#13

by Xyzzyxsfr - opened Nov 29, 2023

Discussion

Xyzzyxsfr

Nov 29, 2023

In [1]: from transformers import AutoTokenizer

In [2]: tokenizer = AutoTokenizer.from_pretrained("NousResearch/Nous-Capybara-34B")

ValueError Traceback (most recent call last)
Cell In[2], line 1
----> 1 tokenizer = AutoTokenizer.from_pretrained("NousResearch/Nous-Capybara-34B")

File /databricks/python3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:688, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
686 tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
687 if tokenizer_class is None:
--> 688 raise ValueError(
689 f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
690 )
691 return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
693 # Otherwise we have to be creative.
694 # if model is an encoder decoder, the encoder tokenizer class is used by default

ValueError: Tokenizer class YiTokenizer does not exist or is not currently imported.

richardlian

Dec 4, 2023

Setting AutoTokenizer as AutoTokenizer.from_pretrained("NousResearch/Nous-Capybara-34B", trust_remote_code=True) worked for me 👍

CombinHorizon

Apr 28, 2024

yes, but it does make it harder to add the model to the Huggingface LLM leaderboard benchmark
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment