Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

623630e

1 Parent(s): 9e56fcc

init

Browse files

Files changed (26) hide show

model-00498-of-00723.safetensors +3 -0
model-00499-of-00723.safetensors +3 -0
model-00502-of-00723.safetensors +3 -0
model-00507-of-00723.safetensors +3 -0
model-00508-of-00723.safetensors +3 -0
model-00509-of-00723.safetensors +3 -0
model-00510-of-00723.safetensors +3 -0
model-00511-of-00723.safetensors +3 -0
model-00512-of-00723.safetensors +3 -0
model-00513-of-00723.safetensors +3 -0
model-00514-of-00723.safetensors +3 -0
model-00515-of-00723.safetensors +3 -0
model-00516-of-00723.safetensors +3 -0
model-00517-of-00723.safetensors +3 -0
model-00518-of-00723.safetensors +3 -0
model-00519-of-00723.safetensors +3 -0
model-00520-of-00723.safetensors +3 -0
model-00521-of-00723.safetensors +3 -0
model-00522-of-00723.safetensors +3 -0
model-00523-of-00723.safetensors +3 -0
model-00524-of-00723.safetensors +3 -0
model-00527-of-00723.safetensors +3 -0
model-00530-of-00723.safetensors +3 -0
model-00531-of-00723.safetensors +3 -0
model-00532-of-00723.safetensors +3 -0
model-00533-of-00723.safetensors +3 -0

model-00498-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:288e7a36a5470de5cbc833be61bbdd0218e7816001a33028662607f1de9a38ac
+size 16528

model-00499-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3df9ece08222e6cc50fad5219c46b0013b52c7d529e1e697583040c26871fbfd
+size 469762200

model-00502-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c567df0f44e88a5bef81efb0d27b52504141952f6f6b6838a020cd3c8f897914
+size 16536

model-00507-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:376e0cd5ba7394f6cda8ac1b185f741ff2014b088312a2a1224cd425e8dad0f6
+size 16528

model-00508-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0d19ca69ee3f43878b0504356df87b5d4bca27c5f6c962f6a0be287d69a8c92f
+size 469762200

model-00509-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20f055b4da24240197aa8e87355467ba94211adc4afcff945248f90584b38bdf
+size 469762200

model-00510-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bf526c1984c0f73eb8badb7a60759c5c7749f231bd51afda7f75248b909cdf2b
+size 469762192

model-00511-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8a4566f14ad25526c8ea94ab138d44a91f9274193cf9de481f5ac98a11252aef
+size 16536

model-00512-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7b0eb9995a215ca2405cf4a2b4fff7198acc386ff317e50a162ba0508a246f09
+size 16777368

model-00513-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e3fa4d0291ba781baccaa081217a72ee3d4c8289f95843a22cdacbd033ec7153
+size 134217880

model-00514-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:130074f28d7ea808a804069181149d11fc9fbee9c13f2c9e8289b7e37093e14c
+size 134217880

model-00515-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a4975c5c6c45cc918413e9ae043ae8c0503cea7ac447c5e9278e62562aeb7d9
+size 16777368

model-00516-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e99d0a24c2d89cdf390422a821a4c38de37373114fbd598028878e530e43895e
+size 16528

model-00517-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fca705c3fb5c6ab29a4997842003a081503f2938daacc93738b99c4d11936f25
+size 469762200

model-00518-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bf89d73f8a91d13daa7431030e38b34da334037c0cba9b6306afacf25d8e060e
+size 469762200

model-00519-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9fe8423ae18f5ff5d0a6c2fccad5e8263b7002dce0beb4ba45f990e1fde31370
+size 469762192

model-00520-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:10c251bd496d74dee9c1227cbaa01665a3ad4ecaa6e0f694610256d87d21bc39
+size 16536

model-00521-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2e0fcf389a84e1ccf0f5daa0fab3977655d5e28bfb76119f86ddaccbb5edca5d
+size 16777368

model-00522-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bbab2abd71f6d8150b01b80ce839e4162587dc898778cfe8c61226828a13e06b
+size 134217880

model-00523-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4cd6dfb010706b014d75fd993e7a831ce15674364f7b218824cfa13a2f9f566d
+size 134217880

model-00524-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb0d5c500108119709926e628ce419a1694b1f49b9d0121335b3b42d71b7bd5a
+size 16777368

model-00527-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:30452eabd60dcb353862c436d17660b0d6205d775f2b9ca76a89f81c3b7accb7
+size 469762200

model-00530-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d9a161336ca8cc710e4cd71c1e4ab31ba97238431c7653567a392ad0222fbf76
+size 16777368

model-00531-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f2a93a661c10ff36e8c5aa304410ab5aff41e0c518767d78b0091063690c7da
+size 134217880

model-00532-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9efa9edc093b4e6a27a824edf7d5ccfe83408c92ae0b227d9ac19cda1e47eb9c
+size 134217880

model-00533-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bcdd914d4b4f424c46e5ca9ce20cb80ef0e405d4136835b538a775fad2c6e2b3
+size 16777368