Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

ad1f093

1 Parent(s): b1e4fc8

init

Browse files

Files changed (26) hide show

model-00246-of-00723.safetensors +3 -0
model-00247-of-00723.safetensors +3 -0
model-00250-of-00723.safetensors +3 -0
model-00255-of-00723.safetensors +3 -0
model-00256-of-00723.safetensors +3 -0
model-00257-of-00723.safetensors +3 -0
model-00258-of-00723.safetensors +3 -0
model-00259-of-00723.safetensors +3 -0
model-00260-of-00723.safetensors +3 -0
model-00261-of-00723.safetensors +3 -0
model-00262-of-00723.safetensors +3 -0
model-00263-of-00723.safetensors +3 -0
model-00264-of-00723.safetensors +3 -0
model-00265-of-00723.safetensors +3 -0
model-00266-of-00723.safetensors +3 -0
model-00267-of-00723.safetensors +3 -0
model-00268-of-00723.safetensors +3 -0
model-00269-of-00723.safetensors +3 -0
model-00270-of-00723.safetensors +3 -0
model-00271-of-00723.safetensors +3 -0
model-00272-of-00723.safetensors +3 -0
model-00275-of-00723.safetensors +3 -0
model-00278-of-00723.safetensors +3 -0
model-00279-of-00723.safetensors +3 -0
model-00280-of-00723.safetensors +3 -0
model-00281-of-00723.safetensors +3 -0

model-00246-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8410b4a091c61a7f350b43c7f7118be6c085d108c450a49e39f9ba0c35188c43
+size 16528

model-00247-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:56a9199b11f1dc5104a218da9bfc44ead601be322ebdadcecf9b35301f19d965
+size 469762200

model-00250-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:636d0489c5e8d44667fb81a9bee6646a6c9f55f0a7e7f41b59122934663e0db3
+size 16536

model-00255-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a07831b675d5429d10d835ec259d38dfb92d6519427161e1e93c605f8df7b6d4
+size 16528

model-00256-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:134994546c8c2f421f380debce3b76a7c837adae7c30f98fefa8be206bab1468
+size 469762200

model-00257-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ff6d9138c1e7df2cd324a72e7e2c8a4a400fe57c59e9023e1a9380e1e3c9d6c4
+size 469762200

model-00258-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:796ac028ff8d5efa99be2631b09d2d5df5c1eccc8236745a69a8b697440341ee
+size 469762192

model-00259-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e2ee9c8982f6f5f062befcff3107193ffeaec538be30749fd89cb0437ff891d9
+size 16536

model-00260-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3ede26bd202b16cfbeda3441da33e1bd2157f797f2379278ca0486742444d539
+size 16777368

model-00261-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c6b74a3578d4451bd734f547767b692ed10360fd23e200b2a92b0ef91cef6dd0
+size 134217880

model-00262-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:779e892e4179b59fed9e3612a42c6d04137ea636ac5b559d15e8b97f2189d2ac
+size 134217880

model-00263-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1358360348bd5885e0f56632b15db4635ce795a2a0b5a4ead58618f161bdb951
+size 16777368

model-00264-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:896a5948c6f2ab8c87cd0d9d328dad9c6603832d2c6d08f533c681bf5e763ecc
+size 16528

model-00265-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d6a4715193b90bd03259105f7673a2f7bc1eab10af9a949afa1683b046ad9797
+size 469762200

model-00266-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2ecb3ac462f51c5f54aad3b0526abac92a872724002896090848e881496fccc8
+size 469762200

model-00267-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0116a8e9b84ff415b164308b22a58ca0a7130e344008bd6cb0d9dd17764f65e2
+size 469762192

model-00268-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5de29627da3103766d433cf3c00cb43c3e5b8dc18ab6a8f3762ccb7184dc48ce
+size 16536

model-00269-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c7180137c6ee19bf5be8a366e739cdd1587b02edd47934d4d762e2c3ff1d46c1
+size 16777368

model-00270-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:77e8fea63df21bf90558751359dd9a4913772bfe9e89d611ef1847260d535285
+size 134217880

model-00271-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:70fa97b20f925cbe3cf67c81b792541c09cc42097de202cb47808ddd359fcc16
+size 134217880

model-00272-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2eec36a8d0663b7c446f52138bf3b1092aff39a6238d5054ece58ac5f94ea652
+size 16777368

model-00275-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4e4391fe6f99523ec3bf81b716216c9f1f6bbe87bcbd44ac1248152c1b8a9e1f
+size 469762200

model-00278-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e05834e0b293b559d7affab02259e8560818ca94bc6c336d5fbc88aabe8478d4
+size 16777368

model-00279-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f6ac02b2b1a94c76d11ad351eb655e8cd08836fe17fe24490fd6510c95b1487c
+size 134217880

model-00280-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:eec474ca6e14965cade49e5bce709e0c163b5e8dbfa1bdc66f3b723479b3935c
+size 134217880

model-00281-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ba70ca344de622fc835f01f7fb4a7202395ea18a3955faade967b298a4c4d
+size 16777368