Instructions to use nayohan/llama3-instrucTrans-enko-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nayohan/llama3-instrucTrans-enko-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nayohan/llama3-instrucTrans-enko-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nayohan/llama3-instrucTrans-enko-8b")
model = AutoModelForCausalLM.from_pretrained("nayohan/llama3-instrucTrans-enko-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nayohan/llama3-instrucTrans-enko-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nayohan/llama3-instrucTrans-enko-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nayohan/llama3-instrucTrans-enko-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nayohan/llama3-instrucTrans-enko-8b

SGLang

How to use nayohan/llama3-instrucTrans-enko-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nayohan/llama3-instrucTrans-enko-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nayohan/llama3-instrucTrans-enko-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nayohan/llama3-instrucTrans-enko-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nayohan/llama3-instrucTrans-enko-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nayohan/llama3-instrucTrans-enko-8b with Docker Model Runner:
```
docker model run hf.co/nayohan/llama3-instrucTrans-enko-8b
```

The model outputs `!!!!!!!!!!!!!!!!!!!!` only

by seungduk - opened May 26, 2024

Discussion

seungduk

May 26, 2024

Hi there,

Thanks for sharing a good model. I wanted to test it out but I failed to get the expected output from the model. Can you point me out what I did wrong?

Code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nayohan/llama3-instrucTrans-enko-8b"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

system_prompt = "당신은 번역기 입니다. 영어를 한국어로 번역하세요."
sentence = "The aerospace industry is a flower in the field of technology and science."
conversation = [{'role': 'system', 'content': system_prompt},
                {'role': 'user', 'content': sentence}]

encoded_inputs = tokenizer.apply_chat_template(
  conversation,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors='pt'
)

input_ids = encoded_inputs.to('cuda')
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    outputs = model.generate(input_ids=input_ids, attention_mask=attention_mask
                             , max_new_tokens=46) # Finetuned with length 4096
    print(tokenizer.decode(outputs[0]))

Output:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 번역기 입니다. 영어를 한국어로 번역하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thanks,
Seungduk

ehottl

May 30, 2024

I can not re-generate your problems.
In my case, it works beautifully.

Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  2.39it/s]
WARNING:root:Some parameters are on the meta device device because they were offloaded to the cpu.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 번역기 입니다. 영어를 한국어로 번역하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

항공우주 산업은 기술과 과학 분야의 꽃입니다.<|eot_id|>

nayohan

Owner May 30, 2024

•

edited May 30, 2024

I also got the same result as above. even though I used your fixed code.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 번역기 입니다. 영어를 한국어로 번역하세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

항공우주 산업은 기술과 과학 분야의 꽃입니다.<|eot_id|>

nayohan

Owner May 30, 2024

I'll share my environment settings, would you like to try this?

--find-links https://download.pytorch.org/whl/torch_stable.html
torch==2.2.1+cu121
transformers==4.40.1
datasets==2.18.0
accelerate==0.26.1
deepspeed==0.14.0
peft==0.10.0
trl==0.8.6
pyarrow==15.0.0
# flash-attn==2.5.6
bitsandbytes==0.43.1

ehottl

Jun 3, 2024

My guess is that you installed the package using pip. Problems caused by pip are ambiguous and hard to solve. If you can, try installing using the conda package manager and the conda-forge channel.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment