Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

35a5331

1 Parent(s): ddfad21

init

Browse files

Files changed (26) hide show

model-00624-of-00723.safetensors +3 -0
model-00625-of-00723.safetensors +3 -0
model-00628-of-00723.safetensors +3 -0
model-00633-of-00723.safetensors +3 -0
model-00634-of-00723.safetensors +3 -0
model-00635-of-00723.safetensors +3 -0
model-00636-of-00723.safetensors +3 -0
model-00637-of-00723.safetensors +3 -0
model-00638-of-00723.safetensors +3 -0
model-00639-of-00723.safetensors +3 -0
model-00640-of-00723.safetensors +3 -0
model-00641-of-00723.safetensors +3 -0
model-00642-of-00723.safetensors +3 -0
model-00643-of-00723.safetensors +3 -0
model-00644-of-00723.safetensors +3 -0
model-00645-of-00723.safetensors +3 -0
model-00646-of-00723.safetensors +3 -0
model-00647-of-00723.safetensors +3 -0
model-00648-of-00723.safetensors +3 -0
model-00649-of-00723.safetensors +3 -0
model-00650-of-00723.safetensors +3 -0
model-00653-of-00723.safetensors +3 -0
model-00656-of-00723.safetensors +3 -0
model-00657-of-00723.safetensors +3 -0
model-00658-of-00723.safetensors +3 -0
model-00659-of-00723.safetensors +3 -0

model-00624-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:558b6e7b40998ce7cc7c4b462323c8ac0e8986dc077c42850b7833c4910e8c0d
+size 16528

model-00625-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6be69beb3779f451c8f76817dce8c36d9ae2a7d00e34e40ec96f947a4d6e3c23
+size 469762200

model-00628-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b4c5a670188aa39c0ee78afa06d6571559380a3797b96145a7d6164aa80d1dd9
+size 16536

model-00633-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9b2135b6484f5c665fd2efbb63de5a7d25353f458c10343899510053901c726f
+size 16528

model-00634-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b7b67861c1959ead37449414f74d81924a3c0ecbc67998770e2eeb17960e8386
+size 469762200

model-00635-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:856c958cf2cd20a5e19aa608b339a2a57d76819fca49352cb2071dc8ef1d6cbb
+size 469762200

model-00636-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4201ff5cb68668c99fcf0af1c9fb4b4ed38ec8e16d29f0f5bcdd933e50326800
+size 469762192

model-00637-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:280542180b0ab72156134263177903ce7dbc5b1c407584a4092e598aeaf955e0
+size 16536

model-00638-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0752157f1d20d98931908b0adf356f945f37d4b80ba698c16ee122607b2b687b
+size 16777368

model-00639-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:92c64304b0e95f91e2e5e1a501b917a0711bf0aa985e2402736784c513d8509c
+size 134217880

model-00640-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:62edee522e2dfce53196ae0deeb66bde183da1a62104b6287ae66d787258d48a
+size 134217880

model-00641-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:42f708992dd005ea12b23b4582dbc5f0ae9b2c982ef6f1cfe3a7b8c2dec61efd
+size 16777368

model-00642-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f90a1568c235f0f218f5aeb42e7201377377e5379465e80cef92faf359a6bd91
+size 16528

model-00643-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d57bf6a1fdf5fa11f9e9a9642d76396f14ad3065b76ae433a9dc46e88a4f9c32
+size 469762200

model-00644-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:659dc9e420d9cf593176251959a7de96a7608b9513f0f2742a3a744ee0467035
+size 469762200

model-00645-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0796acfc3a2f98aac785b37caf0c8f23a629212e1630a12dd901d50e89f8ac6d
+size 469762192

model-00646-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5e8863bf859bcf0bf55686d3976ccafcf102f1c1b11727e182f3e227397a8a43
+size 16536

model-00647-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c2236817c557f3c62a31c579530e968200070d405ec11bcc1ad8ad54a36a62eb
+size 16777368

model-00648-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e48ecfa0ec579eeece39bb6892aa140a47d09b0cc5790ca8b6746aec865630c
+size 134217880

model-00649-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f7d0ca05914c72489f67a5c777c5261e6fbdd866c1d97faffb4f659816e5ad2d
+size 134217880

model-00650-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e55e1a6d069f77293b1d9813c5e8b6b984136d20aabcc010f667feff7215b5a2
+size 16777368

model-00653-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b51987fc4dcb16d05e26fa8d5540ffaa3722925cee8f2f46b72baa93f0a7be01
+size 469762200

model-00656-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:730151d414438e4a0745dcc7d6d8d5f7860d87d52ab91cd629e226ba7bdbf260
+size 16777368

model-00657-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:11bf6ed4139e91c778ee3d7863acc984f2f9d4b37b269cc2a225b44ece1fd4a4
+size 134217880

model-00658-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:742211a138b019ee9eaab3dc6a459b166b30682679e87a12503767026557ee8b
+size 134217880

model-00659-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e554e6780dadd71c7e3e650bb0bafaecc235d7baaea46f735d660eba128f82a6
+size 16777368