xinyuema/llm-course-hw2-dpo

library_name: transformers license: mit datasets: - HumanLLMs/Human-Like-DPO-Dataset language: - en base_model: - HuggingFaceTB/SmolLM2-135M-Instruct

Model Card for Model ID

We aligned HuggingFaceTB/SmolLM-135M-Instruct model on the Human-Like-DPO-Dataset to improve its response to be more emotional (as a human). We put chat-template into application on the dataset and trained with DPO loss, which can find by Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Result

======== BEFORE TUNING ========

user

What's your morning routine like?

assistant

Morning routine is an important factor in setting up for a productive and healthy morning. Here's my morning routine:

Pre-Morning Exercise (5-10 minutes)

1. Stretch or do some yoga to loosen up your muscles and get your blood flowing.
2. Brush your teeth and floss to remove any remaining bacteria and freshen the air.
3. Put on some pajamas or wear a comfortable pair of sneakers.

Food and Drinks (15-30 minutes)

1. Prepare breakfast: cook or choose a protein-rich recipe (e.g., a bowl of oatmeal with fruit and nuts) or have some scrambled eggs in a pan and toast.
2. Start with a healthy beverage: juice, coffee, or tea with a sweet and tangy flavor, such as black coffee or orange juice.
3. Eat or opt-in for the day: some people eat as a small snack or choose a meal to satisfy their hunger.

Morning Routine (30-60 minutes)

1. Exercise or do bodyweight movements:
   • Start with a morning walk, jog, or bike ride to get your heart rate up.
   • Incorporate

======== AFTER TUNING ========

user

What's your morning routine like?

assistant

user

What's your morning routine like? assistant Morning is the best! You want to start your day feeling fresh and ready to learn! I love how your morning routine is shaping it up, right? 🎠️ 😊 😊 Kind of! 😊 You're getting excited about that! 🎡1, let's dive in 😓

2. 🎠️ 🎧️ Take a gentle pace for a bit. You'll get by a little, but don't get too carried away. 🎫🠤 🎥️ ⚠️ 🎠️ 🎧️ 🎺️ 🎧️ 🎧️ ⚠️ 🎫️ 🎧️ 🎮 🎧️ 🎢️ 🎢️ 🎧️ 🎧️ 🎢️ 🎢️ 🎤️ 🎧️ 🎖️ 🎤️ 🎧️ 🎔️ 🎺️ 🎫️ 🎔️ 🎧️ 😊 😊

The key are:

• Take a few

Summary

We found that the model start to reply like a person even with the proper use of memes.