Instructions to use kevin009/llama-essay-eval with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kevin009/llama-essay-eval with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kevin009/llama-essay-eval")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kevin009/llama-essay-eval")
model = AutoModelForCausalLM.from_pretrained("kevin009/llama-essay-eval", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kevin009/llama-essay-eval with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kevin009/llama-essay-eval"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kevin009/llama-essay-eval",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kevin009/llama-essay-eval

SGLang

How to use kevin009/llama-essay-eval with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kevin009/llama-essay-eval" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kevin009/llama-essay-eval",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kevin009/llama-essay-eval" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kevin009/llama-essay-eval",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use kevin009/llama-essay-eval with Docker Model Runner:
```
docker model run hf.co/kevin009/llama-essay-eval
```

kevin009 commited on Jun 10, 2024

Commit

fb60337

verified ·

1 Parent(s): 0ad7954

Trained with Unsloth

Browse files

Files changed (4) hide show

model-00001-of-00004.safetensors +1 -1
model-00002-of-00004.safetensors +1 -1
model-00003-of-00004.safetensors +1 -1
model-00004-of-00004.safetensors +1 -1

model-00001-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a41de720b8e871d971d42054a29be15ddab6b9c21591e78a27e7237061d0d83b
 size 4976698672

 version https://git-lfs.github.com/spec/v1
+oid sha256:70cf464c50010885d00af8008167cc32f95d8b9a5da9cecb8f5d663f0787f010
 size 4976698672

model-00002-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b99ac441c9e0919ce47629015294bbab9d7aec799ed283449f116ca53e3dfb49
 size 4999802720

 version https://git-lfs.github.com/spec/v1
+oid sha256:c29676836274b0ee985dbd2714d89d8a682048ef1f04d2e91d8553d821e7e6e0
 size 4999802720

model-00003-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f68152c085051ec5faa5834d2fe194ee0b2e9f0c66fd7112afcd48d39a513c9
 size 4915916176

 version https://git-lfs.github.com/spec/v1
+oid sha256:e8cdfd4348d57d69a745620fe0f1dbba733d9f414b588561ac1f5fe9ea1bda2e
 size 4915916176

model-00004-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:536610362d78d2eac86c5663b46bb2cd2e6e1edbe9720c31e2bb890ffb8cebac
 size 1168138808

 version https://git-lfs.github.com/spec/v1
+oid sha256:3da54b27b00be34bfb5318c7563cc2025ca85dd136ba732aab98fc9de6978bb7
 size 1168138808