Instructions to use ff670/openbuddy-coder-34b-v11-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ff670/openbuddy-coder-34b-v11-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ff670/openbuddy-coder-34b-v11-bf16")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ff670/openbuddy-coder-34b-v11-bf16")
model = AutoModelForCausalLM.from_pretrained("ff670/openbuddy-coder-34b-v11-bf16")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ff670/openbuddy-coder-34b-v11-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ff670/openbuddy-coder-34b-v11-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ff670/openbuddy-coder-34b-v11-bf16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ff670/openbuddy-coder-34b-v11-bf16

SGLang

How to use ff670/openbuddy-coder-34b-v11-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ff670/openbuddy-coder-34b-v11-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ff670/openbuddy-coder-34b-v11-bf16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ff670/openbuddy-coder-34b-v11-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ff670/openbuddy-coder-34b-v11-bf16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ff670/openbuddy-coder-34b-v11-bf16 with Docker Model Runner:
```
docker model run hf.co/ff670/openbuddy-coder-34b-v11-bf16
```

Humaneval scored???

by rombodawg - opened Aug 31, 2023

Discussion

rombodawg

Aug 31, 2023

What does this model score on human eval? Does it get close to wizardcoder34b?

ff670

Owner Sep 1, 2023

Hi, this model is our first attempt, and currently we believe its coding ability is close to or slightly lower than OpenBuddy 70B.

If its coding ability can be further improved in the future, we will consider conducting more benchmarks.

Also, please note that the scores of HumanEval are easily influenced by the training set, and due to the complexity of the model's training data, even the model authors find it difficult to judge whether the training set is completely disjoint with the test questions. Therefore, we believe that benchmarks like HumanEval may be somewhat misleading, and a model that surpasses GPT-4 on HumanEval may not necessarily have stronger coding ability.

Given that there is still a lack of evaluation mechanisms that can align with human coding ability, we still recommend that users use multiple models in practice and make judgments based on their own scenarios.

rombodawg

Sep 1, 2023

You are correct that there is no true comparison for human coding capabilities, but a score on a benchmark is still the best way to compare models at the moment. So ill still look forward to the score

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment