Instructions to use davidkim205/Ko-Llama-3-8B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use davidkim205/Ko-Llama-3-8B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="davidkim205/Ko-Llama-3-8B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("davidkim205/Ko-Llama-3-8B-Instruct")
model = AutoModelForCausalLM.from_pretrained("davidkim205/Ko-Llama-3-8B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use davidkim205/Ko-Llama-3-8B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "davidkim205/Ko-Llama-3-8B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "davidkim205/Ko-Llama-3-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/davidkim205/Ko-Llama-3-8B-Instruct

SGLang

How to use davidkim205/Ko-Llama-3-8B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "davidkim205/Ko-Llama-3-8B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "davidkim205/Ko-Llama-3-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "davidkim205/Ko-Llama-3-8B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "davidkim205/Ko-Llama-3-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use davidkim205/Ko-Llama-3-8B-Instruct with Docker Model Runner:
```
docker model run hf.co/davidkim205/Ko-Llama-3-8B-Instruct
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

davidkim205/Ko-Llama-3-8B-Instruct

Ko-Llama-3-8B-Instruct is one of several models being researched to improve the performance of Korean language models. This model was created using the REJECTION SAMPLING technique to create a data set and then trained using Supervised Fine Tuning.

Model Details

Model Developers : davidkim(changyeon kim)
Repository : -
base mode : meta-llama/Meta-Llama-3-8B-Instruct
sft dataset : sft_rs_140k

Requirements

If the undefined symbol error below occurs, install torch as follows.

...
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
/home/david/anaconda3/envs/spaces/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

pip install torch==2.2.0
pip install flash-attn==2.5.9.post1

How to use

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch


model_id = "davidkim205/Ko-Llama-3-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)


while True:
    prompt = input('>')
    messages = [
        {"role": "system", "content": "당신은 구체적으로 답변하는 챗봇입니다."},
        {"role": "user", "content": prompt},
    ]
    input_ids = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        return_tensors="pt"
    ).to(model.device)

    terminators = [
        tokenizer.eos_token_id,
        tokenizer.convert_tokens_to_ids("<|eot_id|>")
    ]

    outputs = model.generate(
        input_ids,
        max_new_tokens=1024,
        eos_token_id=terminators,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
    )
    response = outputs[0][input_ids.shape[-1]:]
    print(tokenizer.decode(response, skip_special_tokens=True))

사과의 의미를 설명하시오
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:128009 for open-end generation.
사과는 일반적으로 맛과 영양가 있는 과일로 알려져 있습니다. 사과는 신선한 상태에서 주로 먹거나, 요거트나 스무디 등의 음료에 혼합하여 섭취되기도 합니다. 또한, 사과는 다양한 종류가 있으며, 각각의 종류는 다른 색상과 맛을 가지고 있습니다.

사과는 과일이지만, 종종 다른 의미로도 사용됩니다. 예를 들어, "사과"라는 단어는 어떤 것이 잘못되거나 부족한 것을 시사하는 상황에서 사용될 수도 있습니다. 예를 들어, "사과"를 주는 것은 잘못된 행동이나 부족한 사고로 인한 사과를 의미할 수 있습니다.

또한, "사과"는 어떤 상황에서 다른 사람에게서 사과를 받는 것을 의미하기도 합니다. 예를 들어, "사과"를 구하지 않으면 어떤 상황에서 다른 사람에게서 사과를 받지 못할 수도 있습니다.

따라서, "사과"는 다양한 의미로 사용되는 단어이며, 맥락에 따라 다른 의미를 가질 수 있습니다.

Benchmark

kollm_evaluation

https://github.com/davidkim205/kollm_evaluation

task	acc
average	0.47
kobest	0.54
kobest_boolq	0.57
kobest_copa	0.62
kobest_hellaswag	0.42
kobest_sentineg	0.57
kobest_wic	0.49
ko_truthfulqa	0.29
ko_mmlu	0.34
ko_hellaswag	0.36
ko_common_gen	0.76
ko_arc_easy	0.33

Evaluation of KEval

keval is an evaluation model that learned the prompt and dataset used in the benchmark for evaluating Korean language models among various methods of evaluating models with chatgpt to compensate for the shortcomings of the existing lm-evaluation-harness.

https://huggingface.co/davidkim205/keval-7b