Instructions to use beowolx/CodeNinja-1.0-OpenChat-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use beowolx/CodeNinja-1.0-OpenChat-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="beowolx/CodeNinja-1.0-OpenChat-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("beowolx/CodeNinja-1.0-OpenChat-7B")
model = AutoModelForCausalLM.from_pretrained("beowolx/CodeNinja-1.0-OpenChat-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use beowolx/CodeNinja-1.0-OpenChat-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "beowolx/CodeNinja-1.0-OpenChat-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beowolx/CodeNinja-1.0-OpenChat-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/beowolx/CodeNinja-1.0-OpenChat-7B

SGLang

How to use beowolx/CodeNinja-1.0-OpenChat-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "beowolx/CodeNinja-1.0-OpenChat-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beowolx/CodeNinja-1.0-OpenChat-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "beowolx/CodeNinja-1.0-OpenChat-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beowolx/CodeNinja-1.0-OpenChat-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use beowolx/CodeNinja-1.0-OpenChat-7B with Docker Model Runner:
```
docker model run hf.co/beowolx/CodeNinja-1.0-OpenChat-7B
```

HUMAN EVAL SCORE!!!!

by rombodawg - opened Dec 22, 2023

Discussion

rombodawg

Dec 22, 2023

Bro you cant post a model that a fine tuned version of one of the best coding models in existance and not post human eval scores.

PLEASE. Im begging you, post some human eval, and human eval+ scores for Gods sakes

beowolx

Owner Dec 22, 2023

lol yeah this is what i'm trying to do right now but it's taking too long to generate the sample file 😅 i'm going probably run out of runpod credits before it's finished

If you or someone else want to try, please feel free!

Here is the code that I've been using to generate the sample:

!pip install human-eval
!pip install evalplus --upgrade
!pip install transformers
!pip install accelerate
!pip install sentencepiece
!pip install protobuf

from transformers import AutoTokenizer, AutoModelForCausalLM
from evalplus.data import get_human_eval_plus, write_jsonl
import torch

# initialize the model
model_path = "beowolx/CodeNinja-1.0-OpenChat-7B"
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("openchat/openchat-3.5-1210", use_fast=True)

def generate_one_completion(prompt: str):
    messages = [
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": ""}  # Placeholder for the model's response
    ]

    # Apply the chat template to get the list of token IDs
    tokenizer.pad_token = tokenizer.eos_token
    input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, truncation=True, max_length=4096)

    # Generate completion
    generate_ids = model.generate(
        torch.tensor([input_ids]).to("cuda"),  # Convert list to tensor and send to GPU
        max_new_tokens=384,
        do_sample=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id
    )

    # Decode and clean up the completion
    completion = tokenizer.decode(generate_ids[0], skip_special_tokens=True)
    completion = completion.split("\n\n\n")[0].strip()

    return completion


samples = [
    dict(task_id=task_id, solution=generate_one_completion(problem["prompt"]))
    for task_id, problem in get_human_eval_plus().items()
]
write_jsonl("samples.jsonl", samples)

beowolx changed discussion status to closed Dec 30, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment