Instructions to use typeof/miqu-70b-split with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use typeof/miqu-70b-split with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("typeof/miqu-70b-split")
model = AutoModelForCausalLM.from_pretrained("typeof/miqu-70b-split")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use typeof/miqu-70b-split with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "typeof/miqu-70b-split"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/typeof/miqu-70b-split

SGLang

How to use typeof/miqu-70b-split with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "typeof/miqu-70b-split" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "typeof/miqu-70b-split" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "typeof/miqu-70b-split",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use typeof/miqu-70b-split with Docker Model Runner:
```
docker model run hf.co/typeof/miqu-70b-split
```

typeof commited on Feb 5, 2024

Commit

970e499

1 Parent(s): ad1f093

init

Browse files

Files changed (26) hide show

model-00273-of-00723.safetensors +3 -0
model-00274-of-00723.safetensors +3 -0
model-00276-of-00723.safetensors +3 -0
model-00277-of-00723.safetensors +3 -0
model-00282-of-00723.safetensors +3 -0
model-00283-of-00723.safetensors +3 -0
model-00284-of-00723.safetensors +3 -0
model-00285-of-00723.safetensors +3 -0
model-00286-of-00723.safetensors +3 -0
model-00287-of-00723.safetensors +3 -0
model-00288-of-00723.safetensors +3 -0
model-00289-of-00723.safetensors +3 -0
model-00290-of-00723.safetensors +3 -0
model-00291-of-00723.safetensors +3 -0
model-00292-of-00723.safetensors +3 -0
model-00293-of-00723.safetensors +3 -0
model-00294-of-00723.safetensors +3 -0
model-00295-of-00723.safetensors +3 -0
model-00296-of-00723.safetensors +3 -0
model-00297-of-00723.safetensors +3 -0
model-00298-of-00723.safetensors +3 -0
model-00299-of-00723.safetensors +3 -0
model-00305-of-00723.safetensors +3 -0
model-00306-of-00723.safetensors +3 -0
model-00307-of-00723.safetensors +3 -0
model-00308-of-00723.safetensors +3 -0

model-00273-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8174b28d489d25df37899ea0d8ed7c8a54b9502e71cb0e25a055f531ec6035a5
+size 16528

model-00274-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a4adc5a0b4b30a43df5ed60e066d7e0b5c0736c7c257dc17d785c39323e7777e
+size 469762200

model-00276-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f6fea8caefbe6c3b895ce778408aca3095f44ee9dcb3b3f2b825ddcad6318308
+size 469762192

model-00277-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:af1f83704ba279180f1e375897ab34381bcff5144ce5008c3bf9d68c614903d7
+size 16536

model-00282-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9ccfe6dfa5eef2f2bf18c6ba8b4ce5a305f4797dedd0222cc80c4c347bc1eb8d
+size 16528

model-00283-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:472b9be49810e3957fab60777b24d2517d0eb66224968ad3c59c51fb463e33f5
+size 469762200

model-00284-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f538a458ed24473a6bb19c68add2295ee4eebde5770495e1f4bc17ddabb51472
+size 469762200

model-00285-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a330dcc0f562c557002aea14e0755c2947291076fdf6ae31672b3405933699ba
+size 469762192

model-00286-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b5d1cfc75fcb6b81f3060c4eb0e71fe61f5d8127ff03ad5d1a2eb381a649384a
+size 16536

model-00287-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9441f941a363ffacf30cd3024a15094d3979f9e4d410923046781eee931e36ff
+size 16777368

model-00288-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:12c4fd4625819618b04133f07bb7658b0b2c7a4228d066895f2d25a0c0e81436
+size 134217880

model-00289-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e8701e6388ce1afb469b8f647893ced18b870c6c20c11baa86da604da26aaae2
+size 134217880

model-00290-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d48f51e947a643666d60c1b759bd3c88f49be5cfc40996a1a1f9215c7e1ae088
+size 16777368

model-00291-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:86c726b8c6099d3e985b1a4297f7ae4c298086fa4939f8557473329cc5a428f6
+size 16528

model-00292-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1916a76526c92ab08581fc70e94afe36ed09d425efb3f5acb6e19d350eba468c
+size 469762200

model-00293-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a33d9a23c4a17deeb4a3cb821cc302868f435a9b8abb069f186f670788e361f9
+size 469762200

model-00294-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3aa83b602c14096790bfb4201d2a1832358abaff059a8968b39cea5a4bbd65ae
+size 469762192

model-00295-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2ac12ca45f116b274d9d8cb4a6fbcdd92e20a535129f38e93d14e1ad1d426cfc
+size 16536

model-00296-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:079f663bdebf091a575857347fe8f7460396afbe5df54886ee6a105e356bd30e
+size 16777368

model-00297-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f63be008a62366ede0b3bc89e5fbe6324f176672adacc1b7d8d6302dc402e9ac
+size 134217880

model-00298-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cfefb3f82d7092fdd6fd88c4bdeaedaa8cf14824c6e8208a3aa87f63c5bcc8da
+size 134217880

model-00299-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:79e3363007ec0077fd2888fb7203b52bac103acfe374686b6c52c73af6b1cfaa
+size 16777368

model-00305-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1079cc55a6873aab89bea5dc92364ba134b46da5b09a728544be49e9f5bab528
+size 16777368

model-00306-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:077a8efebe642823691c75e31090b10bceb687f8ea3f9d9de435c8832e68b83d
+size 134217880

model-00307-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:52568701458dd8ab0b8563d6dbd435f90d6244c506bbc1312f5a1b2b72aa2b01
+size 134217880

model-00308-of-00723.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e46bf1f403ad7150d330fdeed240f7209042cdef9b14e5ba50927445fd6e8960
+size 16777368