Instructions to use pankajmathur/orca_mini_v3_7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pankajmathur/orca_mini_v3_7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pankajmathur/orca_mini_v3_7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pankajmathur/orca_mini_v3_7b")
model = AutoModelForCausalLM.from_pretrained("pankajmathur/orca_mini_v3_7b")

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use pankajmathur/orca_mini_v3_7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pankajmathur/orca_mini_v3_7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajmathur/orca_mini_v3_7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/pankajmathur/orca_mini_v3_7b

SGLang

How to use pankajmathur/orca_mini_v3_7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pankajmathur/orca_mini_v3_7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajmathur/orca_mini_v3_7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pankajmathur/orca_mini_v3_7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajmathur/orca_mini_v3_7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use pankajmathur/orca_mini_v3_7b with Docker Model Runner:
```
docker model run hf.co/pankajmathur/orca_mini_v3_7b
```

Why was this taken off the Leaderboard?

by migtissera - opened Aug 15, 2023

Discussion

migtissera

Aug 15, 2023

Not sure why, but I can't find this model's evals on the leaderboard..

barleyspectacular

Aug 15, 2023

I agree, I find this model performs better than Stable Beluga 7B

pankajmathur

Owner Aug 15, 2023

Exactly I am trying to get this answer from HuggingFace team, since last week, so far no real explanation was given, just the message that it failed, obviously anyone can with repo link can do the same evals and they are working fine,so not sure why they are holding up this model and 7 other of mine, when other model submission are easily showing up on LB, here is thread

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/176#64d72a6c02e58cc1fd08cc88

migtissera

Aug 15, 2023

•

edited Aug 15, 2023

Wow.. @huggingface

What's up? If you want to build our trust and keep it, maybe explain what's going on here?

migtissera

Aug 15, 2023

Wtf, I can't tag HuggingFace here..

migtissera

Aug 15, 2023

Can I clarify something here Pankaj -- did this appear on the leaderboard and then got taken down? Because I'm pretty sure I saw the evals for this model on the leaderboard (but maybe that was on Twitter).. Can you clarify please? I'm going to ask this on Twitter, but want to get the facts right first.

pankajmathur

Owner Aug 15, 2023

No, I don't think it was taken down or something, I think it just FAILED after submission, and the HF team is trying to redo the evals but I guess it failing again for them. so really not sure what's going on, unless the team reply.

here are the logs
https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/psmathur/orca_mini_v3_7b_eval_request_False_float16_Original.json