Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

4b076b6

1 Parent(s): 970e499

init

Browse files

Files changed (26) hide show

model-00300-of-00723.safetensors +3 -0
model-00301-of-00723.safetensors +3 -0
model-00302-of-00723.safetensors +3 -0
model-00303-of-00723.safetensors +3 -0
model-00304-of-00723.safetensors +3 -0
model-00309-of-00723.safetensors +3 -0
model-00310-of-00723.safetensors +3 -0
model-00311-of-00723.safetensors +3 -0
model-00312-of-00723.safetensors +3 -0
model-00313-of-00723.safetensors +3 -0
model-00314-of-00723.safetensors +3 -0
model-00315-of-00723.safetensors +3 -0
model-00316-of-00723.safetensors +3 -0
model-00317-of-00723.safetensors +3 -0
model-00318-of-00723.safetensors +3 -0
model-00319-of-00723.safetensors +3 -0
model-00320-of-00723.safetensors +3 -0
model-00321-of-00723.safetensors +3 -0
model-00322-of-00723.safetensors +3 -0
model-00323-of-00723.safetensors +3 -0
model-00324-of-00723.safetensors +3 -0
model-00325-of-00723.safetensors +3 -0
model-00326-of-00723.safetensors +3 -0
model-00332-of-00723.safetensors +3 -0
model-00334-of-00723.safetensors +3 -0
model-00335-of-00723.safetensors +3 -0

model-00300-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9d6e644a25c0239e9aa80e17adbfeaad4dbcbece4ce997cfb4a7923ca7b5def8
+size 16528

model-00301-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c18a8bc9be0ae680994d36c7b3427aa39a55200c66fe7901b7f0377f4f183adb
+size 469762200

model-00302-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:869b02d5a25c722bd610540c1ab61d3f42d3afb3baba55a56f9085f0a53e0709
+size 469762200

model-00303-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3c67c48e7e2c7e494bc6af93e7f2ce1acc21e8cd8757fb1e3e2a812665e3774b
+size 469762192

model-00304-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:00730da02d757fd937601bc04599c684e4b989e7f79e7704a032bd87e8c21724
+size 16536

model-00309-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ddd19e4e91b2c30137639c0172b28bdc9b12f628a3d59251c87929299d70341c
+size 16528

model-00310-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ee1a79dd5f2d0daba416d8a56448b8cc867fc605bcfa8700ca2e87351f01de22
+size 469762200

model-00311-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d3c4de1bb14f6ee21b6687fcc2b8f47e0d2c6f86cd35cff205b2f75003cb7e76
+size 469762200

model-00312-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d1fa7ddc294f0f7e5cb53bed2074c9190538cecedf1856a7536d2d19aaace914
+size 469762192

model-00313-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ab98fc5a68dfaf1adebb5e7fb828c33fc0145d32dea5e97fcc20f649521e3d2f
+size 16536

model-00314-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1c65926522223ba8ef6a7e6902d1f8a514da7350ea7f82366c3edd0446870774
+size 16777368

model-00315-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6d86a92f7047ff942bcb06249133bbd5aa36d35e89ec6758f28f007957a00ef6
+size 134217880

model-00316-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7542fec05e44f50245c34cdb64aa1708b0ef8ef76f3a805c028f3876dff2fe9b
+size 134217880

model-00317-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:630e9009b01aa9b77b439557dad64b3a141f4f873c6407d5926155f7056e2ab0
+size 16777368

model-00318-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2db22987bd83f7644c2427beee09f986f436bfbc0d8b3c8d53d2db94b85a797c
+size 16528

model-00319-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5cc74fd03785ffedce7d7911444c7db065aa6e1ad9707a6479fe3664c2a803f0
+size 469762200

model-00320-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8c08a059d3e9524be9199f73eda7679b6f00bb624fda982819ef75099f311be5
+size 469762200

model-00321-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:efce778ab24d1a94618d7dde28e06cc5c975a5278f4dd82668889b88741c677d
+size 469762192

model-00322-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f61fdb1aa11610352cec08d845f6f792b6f078b66611297c8f269a9e142aabfd
+size 16536

model-00323-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fbd506c44cac9b3053099d10e8a8d418e465cc31bd0b153405721a7250910be6
+size 16777368

model-00324-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:61e0395f0580a1f9cf6b966b33c3c4f394c6bd1b598644f079d46acc465f51ac
+size 134217880

model-00325-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ffddee7f7922df18841bb11b54ad0487f2b9370cb9aefae842bff9404b4476fe
+size 134217880

model-00326-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4a71ad32808046b86c2ae3e2161e53a75afec6c9f6c0feb5820d197c20c00bde
+size 16777368

model-00332-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6d0f7b059ebc0f009fd9c781195c1b1973958eb15695c5e617bb2e85e531e61a
+size 16777368

model-00334-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8febc40b724e07b15c7e7043a318630bcf2f226531f65cbd291f3128362dc47b
+size 134217880

model-00335-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:13deed85c738a38f82fb6e0464e68ab2430286f9c2e3726d9f99c9c7555fd0bd
+size 16777368