Instructions to use rlarkdms0315/gemma-2-2b-it-ko with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rlarkdms0315/gemma-2-2b-it-ko with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="rlarkdms0315/gemma-2-2b-it-ko")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rlarkdms0315/gemma-2-2b-it-ko")
model = AutoModelForCausalLM.from_pretrained("rlarkdms0315/gemma-2-2b-it-ko", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use rlarkdms0315/gemma-2-2b-it-ko with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rlarkdms0315/gemma-2-2b-it-ko"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rlarkdms0315/gemma-2-2b-it-ko",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rlarkdms0315/gemma-2-2b-it-ko

SGLang

How to use rlarkdms0315/gemma-2-2b-it-ko with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rlarkdms0315/gemma-2-2b-it-ko" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rlarkdms0315/gemma-2-2b-it-ko",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rlarkdms0315/gemma-2-2b-it-ko" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rlarkdms0315/gemma-2-2b-it-ko",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use rlarkdms0315/gemma-2-2b-it-ko with Docker Model Runner:
```
docker model run hf.co/rlarkdms0315/gemma-2-2b-it-ko
```

Gemma-2-2b-it Fine-Tuned on KoAlpaca-v1.1a

Model Description

This model is a fine-tuned version of the google/gemma-2-2b-it model on the Korean dataset beomi/KoAlpaca-v1.1a. It is designed to generate coherent, contextually appropriate responses in Korean. The fine-tuning process has enhanced the model's ability to handle conversational prompts in a colloquial style, responding with contextually aware and polite expressions.

The base model, Gemma-2-2B-it, is a large pre-trained language model built for multilingual text generation tasks. With the fine-tuning on the KoAlpaca dataset, the model has been optimized to perform better on Korean text generation, offering more natural and conversational outputs.

Training Process

The model was fine-tuned using the KoAlpaca-v1.1a dataset, which is designed for instruction-following tasks in Korean. The dataset contains various examples of questions and corresponding responses in Korean, which helped the model learn polite conversational structures.

Dataset

Dataset Used: beomi/KoAlpaca-v1.1a
Type of Data: Instruction-following examples in Korean, with both the instruction and expected response provided in each entry. Training Configuration
Model Base: google/gemma-2-2b-it
LoRA Configuration: Applied LoRA with the following parameters:
Quantization: 4-bit quantization (bnb_4bit) for efficient fine-tuning
Training Hyperparameters:
- Steps: 3000
- Learning Rate: 2e-4
- Batch Size: 1
- Warmup Steps: 100
- Gradient Accumulation: 4 steps
- Optimizer: paged_adamw_8bit
- Precision: FP16

Results and Performance

Example Input: “배가 고파서 마라탕을 먹었어요.” (I was feeling hungry, so I ate maratang.)

Before Fine-tuning: Output:“마라탕 (maratang): This is a Korean soup made with various ingredients like meat, vegetables, and noodles.
고파 (gopa): This means ‘to be hungry.’
먹었어요 (meok-eosseoyo): This is the polite way to say ‘I ate.’”
After Fine-tuning: Output: “맛있게 드셨군요! 저도 이렇게 하면 좋겠습니다. 내일은 어떤 음식을 해볼까 생각해보세요?” (It sounds like you enjoyed your meal! I should try that too. What do you plan to cook tomorrow?)

The fine-tuned model shows a significant improvement in contextual understanding and produces more conversational and polite responses in Korean. It also demonstrates an ability to provide helpful follow-up suggestions, which is essential in conversational agents.

Future Work

Further fine-tuning on larger or more diverse Korean datasets could improve the model's versatility.
Exploring different LoRA configurations and quantization techniques could yield more efficient results for deployment on smaller devices.
Evaluation with human raters to measure improvements in conversation quality.

Downloads last month: 7

Safetensors

Model size

3B params

Tensor type

F32