| --- |
| language: |
| - ko |
| - en |
| library_name: transformers |
| tags: |
| - instruction-tuning |
| - korean |
| - phi-4 |
| - causal-lm |
| model_creator: microsoft |
| base_model: microsoft/Phi-4-mini-instruct |
| model_name: siwon-mini-instruct-0626 |
| pipeline_tag: text-generation |
| --- |
| |
| <p align="center"> |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/665818433a098887a5b95015/IlaWEzz4UrH7d14DkXGgQ.png" width="300" height="300"> |
| </p> |
|
|
| # siwon-mini-instruct-0626 |
|
|
| This model is a fine-tuned version of [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct), adapted for Korean instruction-based tasks. The tuning was focused on enhancing Korean performance through supervised fine-tuning with Korean instruction datasets. |
|
|
| --- |
|
|
| ## 🔧 Token Adjustments |
|
|
| The original model used the same token ID (199999) for multiple special tokens such as BOS, EOS, PAD, and UNK. This caused confusion in instruction-following tasks. We fixed this by remapping the token IDs as follows: |
|
|
| | Token Type | Original ID | Fixed ID | |
| |------------|-------------|----------| |
| | BOS | 199999 | 199999 | |
| | EOS | 199999 | 200020 | |
| | PAD | 199999 | 200029 | |
| | UNK | 199999 | 200030 | |
|
|
| These changes ensure proper differentiation and functioning of special tokens during generation and training. |
|
|
| --- |
|
|
| ## 🗨️ Chat Template |
|
|
| The chat template was updated accordingly to support multi-turn conversation formatting in the Korean context: |
|
|
| ```jinja2 |
| {% for message in messages %} |
| {% if message['role'] == 'system' and 'tools' in message and message['tools'] is not none %} |
| {{ '<|' + message['role'] + '|>' + message['content'] + '<|tool|>' + message['tools'] + '<|/tool|>' + '<|end|>' }} |
| {% else %} |
| {{ '<|' + message['role'] + '|>' + message['content'] + '<|end|>' }} |
| {% endif %} |
| {% endfor %} |
| {% if add_generation_prompt %}{{ '<|assistant|>' }}{% endif %} |
| ``` |
|
|
| ## 🧪 Inference with Transformers |
| Below is an example of how to load and use the model with the adjusted tokenizer, token IDs, and custom prompt template. |
|
|
| > **Note**: This model uses a custom `chat_template` and updated special token IDs: |
| > - `<|end|>` → 200020 (EOS) |
| > - `<|dummy_85|>` → 200029 (PAD) |
| > - `�` → 200030 (UNK) |
| > |
| |
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
| model_path = "IamMcCoy/siwon-mini-instruct-0626" |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| model_path, |
| device_map="auto", |
| torch_dtype=torch.bfloat16, |
| trust_remote_code=True |
| ) |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| model_path, |
| trust_remote_code=True, |
| ) |
| |
| messages = [ |
| {"role": "system", "content": "You are a helpful assistant."}, |
| {"role": "user", "content": "안녕하세요."}, |
| ] |
| |
| inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device) |
| |
| output = model.generate( |
| **inputs, |
| max_new_tokens=2048, |
| # do_sample=True, # Optional |
| # top_p=0.95, # Optional |
| # temperature=0.6, # Optional |
| # repetition_penalty=1.1, # Optional |
| ) |
| |
| response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True) |
| print(response) |
| ``` |
| |
| ## 📊 Model Performance Comparison |
| Performance scores across three Korean language benchmarks (KMMLU, ko_best, pawsx_ko). |
| |
| | Model | KMMLU (0-shot) | ko_best (5-shot) | pawsx_ko | |
| |-------------------------------|----------------|------------------|----------| |
| | Phi-4-mini-instruct | 0.3161 | 0.6341 | 0.5300 | |
| | kanana-1.5-2.1b-instruct-2505 | 0.1577 | 0.7165 | 0.5070 | |
| | EXAONE-3.5-2.4B-Instruct | 0.3071 | 0.6496 | 0.5655 | |
| | siwon-mini-instruct-0626 | 0.3387 | 0.5576 | 0.5485 | |
| |
| |
| ## 📌 Caution |
| * Commercial use is strictly prohibited. |
| |
| * This model is intended for research and educational use only. |
| |
| * Redistribution or use in commercial products or services is not allowed. |
| |
| |
| ## ✍️ Acknowledgments |
| * Base model: microsoft/Phi-4-mini-instruct |
| * Special thanks to the open-source community for instruction-tuning resources and Korean language corpora. |
| |
| |
| ## 🙏 Feedback & Contributions |
| We welcome any feedback to improve the model’s performance, usability, and alignment with Korean instruction tasks. |
| If you encounter any issues or have suggestions, please feel free to open an issue on the Hugging Face model page. |
| |
| Your input is greatly appreciated and will help us enhance the model further. |
| |
