Instructions to use harmtech/Stheno-1.10-L2-13B-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use harmtech/Stheno-1.10-L2-13B-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="harmtech/Stheno-1.10-L2-13B-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("harmtech/Stheno-1.10-L2-13B-GPTQ")
model = AutoModelForCausalLM.from_pretrained("harmtech/Stheno-1.10-L2-13B-GPTQ")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use harmtech/Stheno-1.10-L2-13B-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "harmtech/Stheno-1.10-L2-13B-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "harmtech/Stheno-1.10-L2-13B-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/harmtech/Stheno-1.10-L2-13B-GPTQ

SGLang

How to use harmtech/Stheno-1.10-L2-13B-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "harmtech/Stheno-1.10-L2-13B-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "harmtech/Stheno-1.10-L2-13B-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "harmtech/Stheno-1.10-L2-13B-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "harmtech/Stheno-1.10-L2-13B-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use harmtech/Stheno-1.10-L2-13B-GPTQ with Docker Model Runner:
```
docker model run hf.co/harmtech/Stheno-1.10-L2-13B-GPTQ
```

Daddy Dave's stamp of approval 👍

4-bit GPTQ quants of the writer version of Sao10K's fantastic Stheno model (Stheno model collection link)

The main branch contains 4-bit groupsize of 128 and no act_order.

The other branches contain groupsizes of 128, 64, and 32 all with act_order.

⬇︎ Original card ⬇︎

My GGUF Quants: https://huggingface.co/Sao10K/Stheno-1.10-L2-13B-GGUF

Oh, you thought there'd be a 2.0? Nope. Not yet.

A recreation of Stheno with Updated versions of the same models and Merging Values. Feels more coherent, and is uncensored (zero context) at least according to my tests.

Is somewhat smarter, I think? Atleast it passes 4/5 times in my own test suites.

Feel free to try it out, I'd appreciate Feedback.

Most formats could work, but my tests have all been done in Alpaca format and it works well.

### Instruction:
Your instruction or question here.
For roleplay purposes, I suggest the following - Write <CHAR NAME>'s next reply in a chat between <YOUR NAME> and <CHAR NAME>. Write a single reply only.

### Response:

support me here :)

Once Again, thanks to Chargoddard for his amazing and simple mergekit script. Thanks to the original model creators too!

Downloads last month: 8

Safetensors

Model size

2B params

Tensor type

I32

F16

Collection including harmtech/Stheno-1.10-L2-13B-GPTQ

Stheno GPTQ

Collection

GPTQ quants of Sao10K's (https://huggingface.co/Sao10K) Stheno 13B model • 6 items • Updated Dec 24, 2023