thangvip/vwen-1.5B-instruct

Model Card for Model ID

LoRA SFT model Qwen2-1.5B on 600k sample of OpenHermes vietnamese (translated)

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: thangvip
• Model type: Qwen2
• Language(s) (NLP): Vietnamese
• Finetuned from model: Qwen/Qwen2-1.5B

Bias, Risks, and Limitations

Have not trained on RLHF for safety

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("thangvip/vwen-1.5B-instruct", device_map="auto", cache_dir="./cache").eval()
tokenizer = AutoTokenizer.from_pretrained("thangvip/vwen-1.5B-instruct", cache_dir="./cache")

messages = [
  {'role': 'system', 'content': "bạn là trợ lý AI hữu ích"},
  {"role": "user", "content": "Nước nào có diện tích lớn nhất?"},
]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

outputs = model.generate(**inputs, tokenizer=tokenizer, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=0.1, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id, stop_strings=['<|im_end|>'])
print(tokenizer.decode(outputs[0]))

Training Details

Training Data

Trained on OpenHermes dataset (translated to Vietnamese) > 600k samples

Training Hyperparameters

• Target_modules: q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj
• batch_size: 2048
• epoch: 1
• optimizer: paged_adamw_8bit
• warmup_ratio: 0.02
• learning_rate: 1e-5