Instructions to use alpindale/goliath-120b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use alpindale/goliath-120b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="alpindale/goliath-120b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("alpindale/goliath-120b")
model = AutoModelForCausalLM.from_pretrained("alpindale/goliath-120b")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use alpindale/goliath-120b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "alpindale/goliath-120b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/goliath-120b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/alpindale/goliath-120b

SGLang

How to use alpindale/goliath-120b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "alpindale/goliath-120b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/goliath-120b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "alpindale/goliath-120b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/goliath-120b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use alpindale/goliath-120b with Docker Model Runner:
```
docker model run hf.co/alpindale/goliath-120b
```

Please submit to HF Leaderboard

by eNavarro - opened Nov 15, 2023

Discussion

eNavarro

Nov 15, 2023

I've recently read a comparison of LLM models, and this one (version Q2_K) sits right on top, next to GPT-4.
Comparison is here: https://www.reddit.com/r/LocalLLaMA/comments/17vcr9d/llm_comparisontest_2x_34b_yi_dolphin_nous/

Impressive as that may sound, the test is still some home made crafted test with a particular purpose in mind, not an all-round test. I'd love to see this model compared to others in HuggingFace Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Thanks!

alpindale

Owner Nov 15, 2023

I submitted the model for evaluation a few days ago. Checked back again now and it wasn't in the pending section, not even any reports of failing. Did they remove it?

Well, I added it again.

Kaylard

Nov 22, 2023

I submitted the model for evaluation a few days ago. Checked back again now and it wasn't in the pending section, not even any reports of failing. Did they remove it?

Well, I added it again.

It's just too good

alpindale

Owner Nov 26, 2023

Update: It's still not appearing in the pending, evaluating, or finished categories. When I try to re-submit, it says it's already submitted. No clue what's happening.

ChuckMcSneed

Nov 27, 2023

The elites don't want regular people to know about the infinite POWER of frankenmerges!

clefourrier

Nov 28, 2023

Hi!
For these kind of problems, it would be good to open an issue on the leaderboard's discussions next time, so we can give you a hand :)
We have an FAQ : if a model stops appearing in the queues, it usually means that evaluating it failed.

I checked for your specific model, and that's indeed what happened (connection problem when downloading the weights, it happens sometimes).
It's been relaunched, we'll see if it fits in memory - if it does, you'll get results in a couple of days.

silver

Dec 5, 2023

Hi!
For these kind of problems, it would be good to open an issue on the leaderboard's discussions next time, so we can give you a hand :)
We have an FAQ : if a model stops appearing in the queues, it usually means that evaluating it failed.

I checked for your specific model, and that's indeed what happened (connection problem when downloading the weights, it happens sometimes).
It's been relaunched, we'll see if it fits in memory - if it does, you'll get results in a couple of days.

Is it solved? I still cannot find goliath-120b on the leaderboard. Really looking forward to see its results.

ChuckMcSneed

Dec 7, 2023

@clefourrier @HuggingFaceH4 WHERE ARE THE RESULTS?

clefourrier

Dec 7, 2023

Hi @silver !

No, this model does not fit in the memory of our GPUs, we'll need to adapt our backend to run multi-node evaluation automatically to be able to run it.
The best would be for you to open an issue on the Open LLM Leaderboard page directly, so we can track this and make sure it stays in our backlog.

Btw, as indicated in the FAQ, you can follow the status of your models by looking at their request file.

ChuckMcSneed

Dec 7, 2023

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/434
Opened issue.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment