Instructions to use MarsupialAI/KitchenSink_103b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MarsupialAI/KitchenSink_103b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MarsupialAI/KitchenSink_103b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MarsupialAI/KitchenSink_103b")
model = AutoModelForCausalLM.from_pretrained("MarsupialAI/KitchenSink_103b")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MarsupialAI/KitchenSink_103b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MarsupialAI/KitchenSink_103b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/KitchenSink_103b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MarsupialAI/KitchenSink_103b

SGLang

How to use MarsupialAI/KitchenSink_103b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MarsupialAI/KitchenSink_103b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/KitchenSink_103b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MarsupialAI/KitchenSink_103b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarsupialAI/KitchenSink_103b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MarsupialAI/KitchenSink_103b with Docker Model Runner:
```
docker model run hf.co/MarsupialAI/KitchenSink_103b
```

MarsupialAI commited on Jan 15, 2024

Commit

6555c41

verified ·

1 Parent(s): 528ff5c

Update README.md

Browse files

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -11,7 +11,7 @@ tags:
 # Kitchen Sink 103b
 ![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/65a531bc7ec6af0f95c707b1/QFmPxADHAqMf3Wb_Xt1ry.jpeg)
-This model is a rotating-stack merge of three 70b models in a 103b (120 layer) configuration inspired by Venus 103b.  The result is a large model that contains a little bit of everything - including the kitchen sink.
 Component models for the rotating stack are
 - royallab/Aetheria-L2-70B
@@ -33,4 +33,4 @@ Seems to have the strongest affinity for Alpaca prompts, but Vicuna works as wel
 # WTF is a rotating-stack merge?
-Inspired by Undi's experiments with stacked merges, Jeb Carter found that output quality and model initiative could be significantly improved by reversing the model order in the stack, and then doing a linear merge between the original and reversed stacks.  That is what I did here, creating three stacked merges with the three source models, and then doing a 1:1:1 linear merge of all three stacks.  The exact merge configs can be found in the recipe.txt file.

 # Kitchen Sink 103b
 ![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/65a531bc7ec6af0f95c707b1/QFmPxADHAqMf3Wb_Xt1ry.jpeg)
+This model is a rotating-stack merge of three 70b models in a 103b (120 layer) configuration inspired by Venus 103b.  The result of this "frankenmerge" is a large model that contains a little bit of everything - including the kitchen sink.  RP, chat, storywriting, and instruct are all well supported.  It may or may not code well - I lack the expertise to test it in that capacity.
 Component models for the rotating stack are
 - royallab/Aetheria-L2-70B
 # WTF is a rotating-stack merge?
+Inspired by Undi's experiments with stacked merges, Jeb Carter found that output quality and model initiative could be significantly improved by reversing the model order in the stack, and then doing a linear merge between the original and reversed stacks.  That is what I did here.  I created three passthrough stacked merges using the three source models (rotating the model order in each stack), and then doing a linear merge of all three stacks.  The exact merge configs can be found in the recipe.txt file.