Instructions to use tokyotech-llm/Swallow-7b-instruct-hf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tokyotech-llm/Swallow-7b-instruct-hf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tokyotech-llm/Swallow-7b-instruct-hf")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tokyotech-llm/Swallow-7b-instruct-hf")
model = AutoModelForCausalLM.from_pretrained("tokyotech-llm/Swallow-7b-instruct-hf", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tokyotech-llm/Swallow-7b-instruct-hf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tokyotech-llm/Swallow-7b-instruct-hf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tokyotech-llm/Swallow-7b-instruct-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/tokyotech-llm/Swallow-7b-instruct-hf

SGLang

How to use tokyotech-llm/Swallow-7b-instruct-hf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tokyotech-llm/Swallow-7b-instruct-hf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tokyotech-llm/Swallow-7b-instruct-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tokyotech-llm/Swallow-7b-instruct-hf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tokyotech-llm/Swallow-7b-instruct-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use tokyotech-llm/Swallow-7b-instruct-hf with Docker Model Runner:
```
docker model run hf.co/tokyotech-llm/Swallow-7b-instruct-hf
```

Here is the article in Japanese and the code that works with Google Colab Pro | 日本語の解説記事と Google Colab Pro で動くコードです

by AICUaki - opened Dec 19, 2023

Discussion

AICUaki

Dec 19, 2023

リリースおめでとうございます！

AICU media 「東工大と産総研、英語の言語理解や対話で高い能力を持つ大規模言語モデル「Swallow」を公開 #SwallowLLM」 https://note.com/aicu/n/n3eb8c1f2df02
AICU media 「東工大LLM「Swallow」を使ってGoogle Colabで遊んでみよう #SwallowLLM」 https://note.com/aicu/n/nd0337d4952f3
Google Colab https://github.com/aicuai/GenAI-Steam/blob/main/20231220_SwallowLLM.ipynb

Taishi-N324 changed discussion status to closed Dec 22, 2023

Taishi-N324 changed discussion status to open Dec 22, 2023

Taishi-N324

tokyotech-llm org Dec 22, 2023

•

edited Dec 22, 2023

紹介していただき、心より感謝申し上げます！

私たちの大規模言語モデル「Swallow」にご注目いただけたことを大変嬉しく思います。Google Colabでの遊び方を紹介していただいたことで、より多くの方に手軽にSwallowを体験していただける機会が増えることを期待しています。

これからも、皆様にとって有益な研究と開発を進めて参りますので、ご支援のほどよろしくお願いいたします。

Taishi-N324 changed discussion status to closed Dec 22, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment