Instructions to use Statuo/Aletheia-14b-EXL2-4bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Statuo/Aletheia-14b-EXL2-4bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Statuo/Aletheia-14b-EXL2-4bpw")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Statuo/Aletheia-14b-EXL2-4bpw")
model = AutoModelForCausalLM.from_pretrained("Statuo/Aletheia-14b-EXL2-4bpw", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Statuo/Aletheia-14b-EXL2-4bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Statuo/Aletheia-14b-EXL2-4bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Statuo/Aletheia-14b-EXL2-4bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Statuo/Aletheia-14b-EXL2-4bpw

SGLang

How to use Statuo/Aletheia-14b-EXL2-4bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Statuo/Aletheia-14b-EXL2-4bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Statuo/Aletheia-14b-EXL2-4bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Statuo/Aletheia-14b-EXL2-4bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Statuo/Aletheia-14b-EXL2-4bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Statuo/Aletheia-14b-EXL2-4bpw with Docker Model Runner:
```
docker model run hf.co/Statuo/Aletheia-14b-EXL2-4bpw
```

Saw this model recommended as a good example of a RP model in the ST Discord. Figured I'd quant it to see what's up. The same advice for the Qwen model I posted last time holds up, but consider raising the Min P value if you feel the model isn't coherent enough.

This is the 4bpw version of this model. For the original model, go here
For the 8bpw version, go here
For the 6bpw version, go here

Image by CalamitousFelicitousness

Qwen2.5-14B Aletheia v1

RP/Story hybrid model, merge of Sugarquill and Neon. As with Gemma version, I wanted to preserve Sugarquill's creative spark, while making the model more steerable for RP. It proved to be more difficult this time, but I quite like the result regardless, even if the model is still somewhat temperamental.

Should work for both RP and storywriting, either on raw completion or with back-and-forth cowriting in chat mode. Seems to be quite sensitive to low depth instructions and samplers.

Thanks to Toasty and Fizz for testing and giving feedback

Model was created by Auri.

Notes about merging

It took me 20 something attempts to make this model. TIES didn't work at all, producing broken or nearly broken results every time. SLERP worked much better and after just 3 attempts I got something I like. Sugarquill was really prone to overtaking the merge, so I had to reduce it's part a lot, and still model has a lot of influence from it.

Format

Model responds to ChatML instruct formatting, exactly like it's base model.

<|im_start|>system
{system message}<|im_end|>
<|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Recommended Samplers

This one is a bit of a special snowflake, with special tastes. Those seem to work pretty well:

Temperature - 0.8
Top-A - 0.3
TFS - 0.75
DRY - Multiplier 0.8 - Base 1.75 - Allowed length 3 - Range 1024

As a starting point, you can try this ST Master Import

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

base_model: allura-org/TQ2.5-14B-Sugarquill-v1
dtype: bfloat16
merge_method: slerp
parameters:
  t:
  - value: 0.7
slices:
- sources:
  - layer_range: [0, 48]
    model: allura-org/TQ2.5-14B-Neon-v1
  - layer_range: [0, 48]
    model: allura-org/TQ2.5-14B-Sugarquill-v1

Downloads last month: 4

Model tree for Statuo/Aletheia-14b-EXL2-4bpw

allura-org/TQ2.5-14B-Neon-v1

allura-org/TQ2.5-14B-Sugarquill-v1

Merge model

this model