Instructions to use state-spaces/mamba-2.8b-hf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use state-spaces/mamba-2.8b-hf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="state-spaces/mamba-2.8b-hf")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("state-spaces/mamba-2.8b-hf")
model = AutoModelForCausalLM.from_pretrained("state-spaces/mamba-2.8b-hf")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use state-spaces/mamba-2.8b-hf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "state-spaces/mamba-2.8b-hf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "state-spaces/mamba-2.8b-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/state-spaces/mamba-2.8b-hf

SGLang

How to use state-spaces/mamba-2.8b-hf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "state-spaces/mamba-2.8b-hf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "state-spaces/mamba-2.8b-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "state-spaces/mamba-2.8b-hf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "state-spaces/mamba-2.8b-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use state-spaces/mamba-2.8b-hf with Docker Model Runner:
```
docker model run hf.co/state-spaces/mamba-2.8b-hf
```

Which tokenizer has been saved and how to use it?

by javiermartinezcebrian - opened Mar 14, 2024

Discussion

javiermartinezcebrian

Mar 14, 2024

Before this HF compatible models for Mamba, I saw people (and the Mamba repo in /eval) used EleutherAI/gpt-neox-20b. Is this the one saved here in the HF repo for Mamba models? Thanks!

javiermartinezcebrian

Mar 15, 2024

•

edited Mar 15, 2024

@ArthurZ @amyeroberts @koayon @tridao @albertgu I would like to add:
I have seen that sometimes for generation tasks it is usually set the pad_token to the eos_token. For Mamba usage in the original repository it is set add_special_tokens=False while tokenizing. Also I see discussions around using DataCollatorForLanguageModeling or DataCollatorForSeq2Seq. In addition, I see it is typical using left padding for better generation tasks.

In this state-spaces/mamba-2.8b-hf tokenizer I see there is already pad_token and eos_token: how should I use this HF Mamba model for training and for generation regarding that points? Currently I'm not setting pad_token=eos_token, I'm setting add_special_tokens=False, using DataCollatorForLanguageModeling, and left padding.

some refs:
https://github.com/huggingface/transformers/issues/22794
https://medium.com/@geronimo7/mamba-a-shallow-dive-into-a-new-architecture-for-llms-54c70ade5957
https://github.com/havenhq/mamba-chat/blob/main/train_mamba.py

Thanks!

javiermartinezcebrian changed discussion title from Which tokenizer has been saved? to Which tokenizer has been saved and how to use it? Mar 15, 2024

nielsr

Apr 12, 2024

See here: https://github.com/huggingface/transformers/blob/caa5c65db1f4db617cdac2ad667ba62edf94dd98/src/transformers/models/auto/tokenization_auto.py#L239.

javiermartinezcebrian

Apr 12, 2024

thanks @nielsr ! this is regarding which tokenizer, and regarding the other points about the usage with the pretrained mamba models?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment