Instructions to use stabilityai/stablelm-3b-4e1t with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use stabilityai/stablelm-3b-4e1t with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="stabilityai/stablelm-3b-4e1t")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-3b-4e1t")
model = AutoModelForCausalLM.from_pretrained("stabilityai/stablelm-3b-4e1t")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use stabilityai/stablelm-3b-4e1t with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "stabilityai/stablelm-3b-4e1t"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stabilityai/stablelm-3b-4e1t",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/stabilityai/stablelm-3b-4e1t

SGLang

How to use stabilityai/stablelm-3b-4e1t with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "stabilityai/stablelm-3b-4e1t" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stabilityai/stablelm-3b-4e1t",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "stabilityai/stablelm-3b-4e1t" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "stabilityai/stablelm-3b-4e1t",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use stabilityai/stablelm-3b-4e1t with Docker Model Runner:
```
docker model run hf.co/stabilityai/stablelm-3b-4e1t
```

Error when loading model by code in Model card

by XibinBayesZhou - opened Oct 26, 2023

Discussion

XibinBayesZhou

Oct 26, 2023

Hi there,

I'm following your instruction in Model card,

model = AutoModelForCausalLM.from_pretrained("/my/local/path/stablelm-3b-4e1t", trust_remote_code=True, torch_dtype="auto")

but the error occurs as follows

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /my/local/path/stablelm-3b-4e1t.

what happens? what should I do?

Thank you guys!

jon-tow

Oct 26, 2023

Hi @XibinBayesZhou ! What are the contents of ls "/my/local/path/stablelm-3b-4e1t"? You should be able to load the model by pointing to the model name instead of the path (unless you're locally modifying the modeling code, etc.):

model = AutoModelForCausalLM.from_pretrained(
  "stabilityai/stablelm-3b-4e1t",   # Using model name only 
  trust_remote_code=True,
  torch_dtype="auto",
)

XibinBayesZhou

Oct 26, 2023

I downloaded the files you provided in this repo. because of the GFW issue, I have to download it from my browser, instead of directly clone it.

Here is the result of runing ls "/my/local/path/stablelm-3b-4e1t"

which I think is as same as files you guys provided.

jon-tow

Oct 26, 2023

Can you try running pip install -U transformers? The latest version supports loading safetensors from paths. Let me know if it's still an issue!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment