File size: 2,061 Bytes
d448db9 636a300 23aee3b 636a300 9a54251 636a300 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
---
license: mit
language:
- ko
base_model:
- K-intelligence/Midm-2.0-Base-Instruct
tags:
- Korean
- Culture
---
# Midm-KCulture-2.0-Base-Instruct
- This model is fine-tuned from KT/Midm-2.0-Base-Instruct on the 'Korean Culture Q&A Corpus' using the LoRA (Low-Rank Adaptation) methodology.
## GitHub
Check out the full training code [here](https://github.com/dahlia52/KR-Culture-QA/tree/main).
## Training Hyperparameters
| Hyperparameter | Value |
| :---------------------------- | :---------------------------- |
| **SFTConfig** | |
| `torch_dtype` | `bfloat16` |
| `seed` | `42` |
| `epoch` | `3` |
| `per_device_train_batch_size` | `2` |
| `per_device_eval_batch_size` | `2` |
| `learning_rate` | `0.0002` |
| `lr_scheduler_type` | `"linear"` |
| `max_grad_norm` | `1.0` |
| `neftune_noise_alpha` | `None` |
| `gradient_accumulation_steps` | `1` |
| `gradient_checkpointing` | `False` |
| `max_seq_length` | `1024` |
| **LoraConfig** | |
| `r` | `16` |
| `lora_alpha` | `16` |
| `lora_dropout` | `0.1` |
| `target_modules` | `["q_proj", "v_proj"]` |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "jjae/Midm-KCulture-2.0-Base-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
```
|