Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

ddfad21

1 Parent(s): c40d38a

init

Browse files

Files changed (24) hide show

model-00606-of-00723.safetensors +3 -0
model-00607-of-00723.safetensors +3 -0
model-00608-of-00723.safetensors +3 -0
model-00609-of-00723.safetensors +3 -0
model-00610-of-00723.safetensors +3 -0
model-00611-of-00723.safetensors +3 -0
model-00612-of-00723.safetensors +3 -0
model-00613-of-00723.safetensors +3 -0
model-00614-of-00723.safetensors +3 -0
model-00615-of-00723.safetensors +3 -0
model-00616-of-00723.safetensors +3 -0
model-00617-of-00723.safetensors +3 -0
model-00618-of-00723.safetensors +3 -0
model-00619-of-00723.safetensors +3 -0
model-00620-of-00723.safetensors +3 -0
model-00621-of-00723.safetensors +3 -0
model-00622-of-00723.safetensors +3 -0
model-00623-of-00723.safetensors +3 -0
model-00626-of-00723.safetensors +3 -0
model-00627-of-00723.safetensors +3 -0
model-00629-of-00723.safetensors +3 -0
model-00630-of-00723.safetensors +3 -0
model-00631-of-00723.safetensors +3 -0
model-00632-of-00723.safetensors +3 -0

model-00606-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:42b683d782aa3097f810819c3462a201d12b0f8380d1d5bd3295993371acbdc9
+size 16528

model-00607-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:60f6eec6c9c757cf2cff5a981fa81a82dec086f1bd3ac66b53a3f1ff57d1bf84
+size 469762200

model-00608-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:63ccaed91cb9726e37c331b0d6bcd478829d94a331472a52f214cf31e26d4ab9
+size 469762200

model-00609-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ee6606c8a2cc4bb201ee4119607510c2ab8e2c3207b7d5322429996a0109ce88
+size 469762192

model-00610-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:45014fda788f9e7c6f7ee8e5333668436fd73fe4eb320c154fd51e90d06d9b07
+size 16536

model-00611-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:98cb335f30fb11d98cfb87b25ae6a4d4ce5a47c64187537993be997dd82cd866
+size 16777368

model-00612-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ef5bbef74f0cb9d9d853144e5e518337a4d92de4ffdcceef04b3f0f4cd7bf828
+size 134217880

model-00613-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7bec1af7024712129fc2b7d5801bfac0c3f9d463aa2c2f501463b109f0aa2ee2
+size 134217880

model-00614-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5734252257ad4ac0f3d538c4b3a2c3ef08644ae90314879f789f92aed8cab4e9
+size 16777368

model-00615-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1bc6a8e91d7dbcb60bf10d8d3523112b5ba79fb26fa0020d83ef68384f8c6b34
+size 16528

model-00616-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:edfd46e76dc07a8ae061aa4011ed236867928bb57958b2a2583247ea4ff129a7
+size 469762200

model-00617-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6bd6aa800181601eb58dfdd3e82cdcacd02d196fd9c778c1a532f1d45bc2c071
+size 469762200

model-00618-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:767c05ecb1e5f468a6c3f92082d24d997d0d52d6bfb8d9d5229c890ce67e28e5
+size 469762192

model-00619-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:656228fbcf1f5ab2c46c5aee86da9795b53bb9d366da0d396680e7aea4e2f67b
+size 16536

model-00620-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:37425a09ba8f6d99c6a370b0a0a6adc58480f31fe238e27c9c3981cb3df6d9b5
+size 16777368

model-00621-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:16d22fa074de70c5e45d25227d27997683c1829f1c4a8ab07a822bd552a11805
+size 134217880

model-00622-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e6df54a8e008282473c3368ab37d13209cfe91ac075d8cfc43793cbc2f817162
+size 134217880

model-00623-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cb107e93de4f322dc637ba47f2be3f4b649d44be4f0b346d09b854d7332b518e
+size 16777368

model-00626-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69345ca1f9b3553f39ba91a7294497a3b4305853a30cedffca75666c061221bb
+size 469762200

model-00627-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e386c61ccfb268ba48de6fbed5710aaf2987b057ded0c77b8d08d0db295ee668
+size 469762192

model-00629-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5b121a6519909808100f870706e92e9cf4a894b5164f77c1ed671a42793be14a
+size 16777368

model-00630-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a90e2e51091709741363c98019e46f4ebb53410733ea2c091380e53c57029321
+size 134217880

model-00631-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76c3514baa735aba87023f1a912a6e2822a9846569d50606b50a2637a2024354
+size 134217880

model-00632-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f42c05238854d8314b8800973fd98031ffcad86d4de6e5af3661bb8553f8d972
+size 16777368