Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

9e56fcc

1 Parent(s): 02d504f

init

Browse files

Files changed (24) hide show

model-00480-of-00723.safetensors +3 -0
model-00481-of-00723.safetensors +3 -0
model-00482-of-00723.safetensors +3 -0
model-00483-of-00723.safetensors +3 -0
model-00484-of-00723.safetensors +3 -0
model-00485-of-00723.safetensors +3 -0
model-00486-of-00723.safetensors +3 -0
model-00487-of-00723.safetensors +3 -0
model-00488-of-00723.safetensors +3 -0
model-00489-of-00723.safetensors +3 -0
model-00490-of-00723.safetensors +3 -0
model-00491-of-00723.safetensors +3 -0
model-00492-of-00723.safetensors +3 -0
model-00493-of-00723.safetensors +3 -0
model-00494-of-00723.safetensors +3 -0
model-00495-of-00723.safetensors +3 -0
model-00496-of-00723.safetensors +3 -0
model-00497-of-00723.safetensors +3 -0
model-00500-of-00723.safetensors +3 -0
model-00501-of-00723.safetensors +3 -0
model-00503-of-00723.safetensors +3 -0
model-00504-of-00723.safetensors +3 -0
model-00505-of-00723.safetensors +3 -0
model-00506-of-00723.safetensors +3 -0

model-00480-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9fd9d64f7a66e1365077155d6f9da0002ae46d37b2db15b93055130b5acddf50
+size 16528

model-00481-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0edff4235bf44143fce6be544d33035270be53efd9bebbff5cceaffb69a83e4c
+size 469762200

model-00482-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:852c1a76b2893658307558e152957547da7d81dbe30f2c132aef6757d544476f
+size 469762200

model-00483-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:394bd0ca2ae4e93b7fc18e2a7651a35cf5d2b5719e9908798d2b276f3104d869
+size 469762192

model-00484-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:54c5f8e67e33604114ec8352abf9fcba9ebcd0932a020fb886505f86b279f11c
+size 16536

model-00485-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:92ae05c7bb6cf96e0ee3da9001f85b2efbf907340a4701815a5043e01441312f
+size 16777368

model-00486-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ab8ab014e26ea093b12112bcf41ad98ca2bb97c8782e0277414e96adfa85b7e2
+size 134217880

model-00487-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:17e41ecc1804e04a5626d320886e5642730e79487b18e4a852dd11cbd3427008
+size 134217880

model-00488-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b47cae6989ae23f19e792c272aad609bf689d684e0ef32ba39d8fbe1401ed3a7
+size 16777368

model-00489-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:11bfc1f1f0445fe375d5f1a33cceaa8f701380f2ccca86623a03249e92e7763c
+size 16528

model-00490-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:15fdee655924c8ec56ceef783223b251f176e16b5d8850d3c6b011b316de3f49
+size 469762200

model-00491-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:514faf34664212a17386ec7586a551307dcf581b75c082da16369a296a4be933
+size 469762200

model-00492-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:385de0c339e778dceb7b124e8844ddf6a63be91095e42e899fa85e6ae07cb61c
+size 469762192

model-00493-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b4a8db3cc54d48823f6414f1eebbf01545aa3f12aaa861601717abc67d474d87
+size 16536

model-00494-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:75d6bd7145fc6547da4cf107cb6725953f8d12a9dd5b7455c60aa6aa629501e0
+size 16777368

model-00495-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:550959ba5b36f384de0fcc474c3402665ebe9b4f97c5f07d95c62834a7ce5379
+size 134217880

model-00496-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c635053eeaeb5d3aa5e4bc3fea5722f481c2733ef300b95d1a68233102fd5757
+size 134217880

model-00497-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bd0d3ac33e7ac8138d1842fd9476ef148aefb093380a5e0694f72a57eb60670d
+size 16777368

model-00500-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:16286cab9f2009b689c71b02e79ab04f6317a2524a3fb2603f2aabae9d25bfa0
+size 469762200

model-00501-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a270a3acd7b4133487a3c305c499f8bed3e4bc832b842622170d84e7e7d057e
+size 469762192

model-00503-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e3d1a8f3b74ea90e65236e63dcf75a49ff7ea6eec3e53f6e2a8c63da58aea061
+size 16777368

model-00504-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:93b66708bc8e8fd30dd0d965e5e006b68a88b7d3abf5ade7ac4191f310e2132a
+size 134217880

model-00505-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a6bdcdcf3676df796074d9d543c787f1a302b86d83cfe44f0d2662c01b53457c
+size 134217880

model-00506-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e91ff96df6e626607a6826485eb53f74ecbdf01ea863b7b40dbf0c9ddda85e84
+size 16777368