How to use from
SGLangUse Docker images
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "Envoid/Cybil-13B" \
--host 0.0.0.0 \
--port 30000# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "Envoid/Cybil-13B",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'Quick Links
Warning: This model may output Adult Content.
Cybil-13B was created with a series of SLERP merges between
Llama-2-13b-chat
and an unreleased 13B experimental model of mine.
The end result seems very stable and excels at any number of general tasks from role play to writing simple python scripts.
It responds well to the Libra-32B SillyTavern format as well as Alpaca Instruct style formatting:
I.e.
### Instruction:
Do a thing.
### Response:
Thanks to the Llama-2-chat DNA it does in rare instances produce refusals which can usually just be overcome by regenerating the response.
- Downloads last month
- 4
Install from pip and serve model
# Install SGLang from pip: pip install sglang# Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Envoid/Cybil-13B" \ --host 0.0.0.0 \ --port 30000# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Envoid/Cybil-13B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'