Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

c40d38a

1 Parent(s): 8c3be0b

init

Browse files

Files changed (24) hide show

model-00579-of-00723.safetensors +3 -0
model-00580-of-00723.safetensors +3 -0
model-00581-of-00723.safetensors +3 -0
model-00582-of-00723.safetensors +3 -0
model-00583-of-00723.safetensors +3 -0
model-00585-of-00723.safetensors +3 -0
model-00588-of-00723.safetensors +3 -0
model-00589-of-00723.safetensors +3 -0
model-00590-of-00723.safetensors +3 -0
model-00591-of-00723.safetensors +3 -0
model-00592-of-00723.safetensors +3 -0
model-00593-of-00723.safetensors +3 -0
model-00594-of-00723.safetensors +3 -0
model-00595-of-00723.safetensors +3 -0
model-00596-of-00723.safetensors +3 -0
model-00597-of-00723.safetensors +3 -0
model-00598-of-00723.safetensors +3 -0
model-00599-of-00723.safetensors +3 -0
model-00600-of-00723.safetensors +3 -0
model-00601-of-00723.safetensors +3 -0
model-00602-of-00723.safetensors +3 -0
model-00603-of-00723.safetensors +3 -0
model-00604-of-00723.safetensors +3 -0
model-00605-of-00723.safetensors +3 -0

model-00579-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:38f61d12066f949acafeb2461def230830564660ac97630723c03e18fa371f00
+size 16528

model-00580-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fec0fbee6bf73bf60ce73e9d59635d4ce6b792c166c53d8faaaca2ea87eef6cc
+size 469762200

model-00581-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cc03e38cfaa3695dc7837951d1c947675be81611903a68b03d573661c9a7080c
+size 469762200

model-00582-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:131547921ee7ccaf2d4d5cf4f90b812ab2a8226b583d583ae020886216f40ec9
+size 469762192

model-00583-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9f0e394c1d315cbdc007d1b766300e576dc9c2d2a1522ac2fc079e324f4988f6
+size 16536

model-00585-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:507eed491373cd03db3449650415fa9ac35053ef70ea9bff9d9fe66621494f7a
+size 134217880

model-00588-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f012a68f2f5694cc2c0b3964368905efb49c05da9b2407cac41036080105eebf
+size 16528

model-00589-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:496b16370cc6c747be80edea0476f027e3153b6a85ef56f6a8256db94b823c1f
+size 469762200

model-00590-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76cd95dad79d771a3fb8ec98efa185d48043d8307721d3813350498202f2af75
+size 469762200

model-00591-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1641a03747af17183cbf26455c2a1af06686a9c5de8a8cb3d26db6ba926d8308
+size 469762192

model-00592-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9ff7b76f35d17032a5996d24e05da4a9fb761e42833a20a22f8603cd1d4b1b90
+size 16536

model-00593-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:560214d8de78410bed4a2f40242cb0160d7a95b6df7b479f82beb61ed6c1d6a3
+size 16777368

model-00594-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0261de1b00d686b0d5fdd87b224d2cfcc910940fe13f61967af08593103a7877
+size 134217880

model-00595-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0842ad452de28da5ee0790f499b057510b0b9a93cfdf7fc0e38016dc7f880e05
+size 134217880

model-00596-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a06965b4bd3f85c8f17c19455f7a9973f158a8b4d869ecfa684c1a8109a8493
+size 16777368

model-00597-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d20055ef5f2903affc0122c53b4a4f017de4bd02a2ea6d2155ba6bea8a2c4495
+size 16528

model-00598-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c002e621989c9be7fb2a6ecbc712fb734da96bd879a824c8e7c0ee531740be54
+size 469762200

model-00599-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b50c43894d8bf83bafa59a6a2f60360d30ea3dac23cb0a2ad53e1e100cc5bc8f
+size 469762200

model-00600-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f028b15a0d8c57d17e1d1844704548566076cfd75e0a71d8a0501813cd686ae9
+size 469762192

model-00601-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:062326920de8b296eccf8d68af294b690c0efc4b8f64433c60c82f5ac4d506fc
+size 16536

model-00602-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5b12834b9707d8ce26236972ee5a4da18489d05b457c0d0e8f6fff4d02625488
+size 16777368

model-00603-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8b8bf3cad9e3990313440d0684b0ac032130d9fbba357d005393521d9f4f6403
+size 134217880

model-00604-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:66007de660956c623309f87efef2c2b879ab129595d2053cf87f7a81ae468558
+size 134217880

model-00605-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:632ecc4e25d3c440ff588ab02291099c31fbc3ed7cac8239251b4c29b6082c5c
+size 16777368