Instructions to use OpenMOSS-Team/moss-moon-003-sft-plugin-int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenMOSS-Team/moss-moon-003-sft-plugin-int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenMOSS-Team/moss-moon-003-sft-plugin-int4", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("OpenMOSS-Team/moss-moon-003-sft-plugin-int4", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use OpenMOSS-Team/moss-moon-003-sft-plugin-int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenMOSS-Team/moss-moon-003-sft-plugin-int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenMOSS-Team/moss-moon-003-sft-plugin-int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OpenMOSS-Team/moss-moon-003-sft-plugin-int4

SGLang

How to use OpenMOSS-Team/moss-moon-003-sft-plugin-int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenMOSS-Team/moss-moon-003-sft-plugin-int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenMOSS-Team/moss-moon-003-sft-plugin-int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenMOSS-Team/moss-moon-003-sft-plugin-int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenMOSS-Team/moss-moon-003-sft-plugin-int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OpenMOSS-Team/moss-moon-003-sft-plugin-int4 with Docker Model Runner:
```
docker model run hf.co/OpenMOSS-Team/moss-moon-003-sft-plugin-int4
```

对话风格问题及性能

by hiarcs - opened Apr 26, 2023

Discussion

hiarcs

Apr 26, 2023

大家使用这个模型的时候有没有遇到前言不搭后语的情况？就是对话中，突然moss开始忽略当前提问重新开始回答之前的议题。我使用的是moss_cli_demo.py那个脚本。
其它的问题还包括极慢的推理速度和突然切换语种的情况。
风格上，这个模型特别保守，修改meta_instruction貌似毫无作用。而且特别喜欢用列表形式回答问题。

有没有用过非量化版本的同学谈谈是否观察到显著的性能损失？

Yhyu13

May 3, 2023

我也是使用者

1, cli demo中代码里每次指挥把上一次的对话放入下一次的prompt中。所以记忆只有一次对话，除非你prompt让它每次都作总结
2, 语言切换没有遇到过
3, meta让moss无论任何问题都回答阿对对对，不生效
3, 推理速度，应该还是triton的问题，回答同样的问题moss 16B GPTQ 4 bit triton耗时和textgen webui 65B llama gptq 4bit cuda ooba分支有的一拼，不太清除moss gptq 4bit的加载流程，好像每次上传模型到gpu前，都要在cpu上处理一下，占用了model size大小的内存。

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment