File size: 4,503 Bytes
7ce6c31 e079de7 7ce6c31 e079de7 7ce6c31 a944a6b 7ce6c31 936a60e 7520f77 f1bdc17 7ce6c31 e079de7 7ce6c31 f3b36c0 7ce6c31 e147c0d ad30761 7ce6c31 85935af 04266fd 85935af |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 |
---
language:
- ko
- en
library_name: transformers
tags:
- instruction-tuning
- korean
- phi-4
- causal-lm
model_creator: microsoft
base_model: microsoft/Phi-4-mini-instruct
model_name: siwon-mini-instruct-0626
pipeline_tag: text-generation
---
<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/665818433a098887a5b95015/IlaWEzz4UrH7d14DkXGgQ.png" width="300" height="300">
</p>
# siwon-mini-instruct-0626
This model is a fine-tuned version of [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct), adapted for Korean instruction-based tasks. The tuning was focused on enhancing Korean performance through supervised fine-tuning with Korean instruction datasets.
---
## 🔧 Token Adjustments
The original model used the same token ID (199999) for multiple special tokens such as BOS, EOS, PAD, and UNK. This caused confusion in instruction-following tasks. We fixed this by remapping the token IDs as follows:
| Token Type | Original ID | Fixed ID |
|------------|-------------|----------|
| BOS | 199999 | 199999 |
| EOS | 199999 | 200020 |
| PAD | 199999 | 200029 |
| UNK | 199999 | 200030 |
These changes ensure proper differentiation and functioning of special tokens during generation and training.
---
## 🗨️ Chat Template
The chat template was updated accordingly to support multi-turn conversation formatting in the Korean context:
```jinja2
{% for message in messages %}
{% if message['role'] == 'system' and 'tools' in message and message['tools'] is not none %}
{{ '<|' + message['role'] + '|>' + message['content'] + '<|tool|>' + message['tools'] + '<|/tool|>' + '<|end|>' }}
{% else %}
{{ '<|' + message['role'] + '|>' + message['content'] + '<|end|>' }}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}{{ '<|assistant|>' }}{% endif %}
```
## 🧪 Inference with Transformers
Below is an example of how to load and use the model with the adjusted tokenizer, token IDs, and custom prompt template.
> **Note**: This model uses a custom `chat_template` and updated special token IDs:
> - `<|end|>` → 200020 (EOS)
> - `<|dummy_85|>` → 200029 (PAD)
> - `�` → 200030 (UNK)
>
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_path = "madcows/siwon-mini-instruct-0626"
model = AutoModelForCausalLM.from_pretrained(
model_path,
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
model_path,
trust_remote_code=True,
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "안녕하세요."},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
output = model.generate(
**inputs,
max_new_tokens=2048,
# do_sample=True, # Optional
# top_p=0.95, # Optional
# temperature=0.6, # Optional
# repetition_penalty=1.1, # Optional
)
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
```
## 📊 Model Performance Comparison
Performance scores across three Korean language benchmarks (KMMLU, ko_best, pawsx_ko).
| Model | KMMLU (0-shot) | ko_best (5-shot) | pawsx_ko |
|-------------------------------|----------------|------------------|----------|
| Phi-4-mini-instruct | 0.3161 | 0.6341 | 0.5300 |
| kanana-1.5-2.1b-instruct-2505 | 0.1577 | 0.7165 | 0.5070 |
| EXAONE-3.5-2.4B-Instruct | 0.3071 | 0.6496 | 0.5655 |
| siwon-mini-instruct-0626 | 0.3387 | 0.5576 | 0.5485 |
## 📌 Caution
* Commercial use is strictly prohibited.
* This model is intended for research and educational use only.
* Redistribution or use in commercial products or services is not allowed.
## ✍️ Acknowledgments
* Base model: microsoft/Phi-4-mini-instruct
* Special thanks to the open-source community for instruction-tuning resources and Korean language corpora.
## 🙏 Feedback & Contributions
We welcome any feedback to improve the model’s performance, usability, and alignment with Korean instruction tasks.
If you encounter any issues or have suggestions, please feel free to open an issue on the Hugging Face model page.
Your input is greatly appreciated and will help us enhance the model further.
|