Instructions to use DAMO-NLP-MT/polylm-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DAMO-NLP-MT/polylm-13b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DAMO-NLP-MT/polylm-13b", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-MT/polylm-13b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("DAMO-NLP-MT/polylm-13b", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use DAMO-NLP-MT/polylm-13b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DAMO-NLP-MT/polylm-13b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DAMO-NLP-MT/polylm-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/DAMO-NLP-MT/polylm-13b

SGLang

How to use DAMO-NLP-MT/polylm-13b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DAMO-NLP-MT/polylm-13b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DAMO-NLP-MT/polylm-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DAMO-NLP-MT/polylm-13b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DAMO-NLP-MT/polylm-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use DAMO-NLP-MT/polylm-13b with Docker Model Runner:
```
docker model run hf.co/DAMO-NLP-MT/polylm-13b
```

Finetune onfp16

by wiccanmind - opened Jul 20, 2023

Discussion

wiccanmind

Jul 20, 2023

Firstly, I really appreciate this amazing contribution for the multilingual LLM and would like to extend my thanks to you.

As I see it, the pretrained model is efficient in bfloat16. However, it will dramatically decrease performance when inferring in fploat16 (and may not generate any tokens that make sense, relevant to discussion #1. The model just prints unk tokens).
So, I am pondering the question of how to finetune the downstream task with only an fp16 supported GPU, while the pretrained model only makes sense in bf16.

Another thing, do you plan to release the quantizing versions (int8, int4) as well?

pemywei

Machine Translation Team at Alibaba DAMO Academy org Jul 24, 2023

Thank you for your attention. We understand that performing fp16 inference using the bf16 pre-trained model can be a challenge, and we regret to inform you that there are no plans to release an fp16 version of the model at this time. However, int4 and int8 quantization versions of the model will be released soon.

wiccanmind

Jul 24, 2023

•

edited Jul 24, 2023

Thank you for your attention. We understand that performing fp16 inference using the bf16 pre-trained model can be a challenge, and we regret to inform you that there are no plans to release an fp16 version of the model at this time. However, int4 and int8 quantization versions of the model will be released soon.

Thank you, and I'm looking forward to the new release!
Please notice me in this discusion when it is come out.

NoirNeko

Oct 14, 2023

@pemywei Hi, thank you for creating the model.
I was wondering whether there are any updates on the quantized version yet?
Is it possible to quantize using the below project?
https://github.com/ggerganov/llama.cpp

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment