Instructions to use ToastyPigeon/psyonic-cetacean-20b-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ToastyPigeon/psyonic-cetacean-20b-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ToastyPigeon/psyonic-cetacean-20b-v2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ToastyPigeon/psyonic-cetacean-20b-v2")
model = AutoModelForCausalLM.from_pretrained("ToastyPigeon/psyonic-cetacean-20b-v2")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ToastyPigeon/psyonic-cetacean-20b-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ToastyPigeon/psyonic-cetacean-20b-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ToastyPigeon/psyonic-cetacean-20b-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ToastyPigeon/psyonic-cetacean-20b-v2

SGLang

How to use ToastyPigeon/psyonic-cetacean-20b-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ToastyPigeon/psyonic-cetacean-20b-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ToastyPigeon/psyonic-cetacean-20b-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ToastyPigeon/psyonic-cetacean-20b-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ToastyPigeon/psyonic-cetacean-20b-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ToastyPigeon/psyonic-cetacean-20b-v2 with Docker Model Runner:
```
docker model run hf.co/ToastyPigeon/psyonic-cetacean-20b-v2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Psyonic-Cetacean-20B-V2

This is a merge of pre-trained language models created using mergekit.

GGUF (standard and iMatrix) quants can be found here courtesy of MarsupialAI: https://huggingface.co/MarsupialAI/Psyonic-Cetacean-20b-v2_iMatrix_GGUF

Merge Details

Merge Method

This model was merged using the linear merge method on two stack-merged models.

The first is jebcarter/psyonic-cetacean-20B (Orca first, reproduced so I didn't have to download that model on top of the components). The second is the same recipe with the models reversed.

Since jebcarter suggested this recipe, credit goes to him.

Models Merged

The following models were included in the merge:

microsoft/Orca-2-13b
KoboldAI/LLaMA2-13B-Psyfighter2

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: microsoft/Orca-2-13b
    parameters:
      weight: 1.0
merge_method: task_arithmetic
base_model: TheBloke/Llama-2-13B-fp16
dtype: float16
name: FlatOrca2
---
slices:
  - sources:
    - model: FlatOrca2
      layer_range: [0, 16]
  - sources:
    - model: KoboldAI/LLaMA2-13B-Psyfighter2
      layer_range: [8, 24]
  - sources:
    - model: FlatOrca2
      layer_range: [17, 32]
  - sources:
    - model: KoboldAI/LLaMA2-13B-Psyfighter2
      layer_range: [25, 40]
merge_method: passthrough
dtype: float16
name: Psycet
---
slices:
  - sources:
    - model: KoboldAI/LLaMA2-13B-Psyfighter2
      layer_range: [0, 16]
  - sources:
    - model: FlatOrca2
      layer_range: [8, 24]
  - sources:
    - model: KoboldAI/LLaMA2-13B-Psyfighter2
      layer_range: [17, 32]
  - sources:
    - model: FlatOrca2
      layer_range: [25, 40]
merge_method: passthrough
dtype: float16
name: Psycet-Reverse
---
models:
  - model: Psycet
    parameters:
      weight: 0.5
  - model: Psycet-Reverse
    parameters:
      weight: 0.5
merge_method: linear
dtype: float16

Downloads last month: 9

Safetensors

Model size

20B params

Tensor type

F16

Model tree for ToastyPigeon/psyonic-cetacean-20b-v2

Quantizations

2 models

Collection including ToastyPigeon/psyonic-cetacean-20b-v2

Psyonic-Cetacean V2

Collection

A smoothed-over version of jebcarter's psyonic-cetacean uploaded on his behalf. • 2 items • Updated Sep 27, 2024 • 1

Paper for ToastyPigeon/psyonic-cetacean-20b-v2

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Paper • 2203.05482 • Published Mar 10, 2022 • 8