README.md

metadata

title: DPO Demo
emoji: 📚
colorFrom: blue
colorTo: blue
sdk: gradio
sdk_version: 6.0.2
app_file: app.py
pinned: false
short_description: Testing DPO for finetuning models

A test / demo application playground for DPO Preference Tuning on different LLM models. Running on Huggingspace: https://huggingface.co/spaces/CatoG/DPO_Demo

Allows for LLM model selection, preference tuning of LLM responses, model response tuning with LoRA and Direct Preference Optimization (DPO). Tuned model / policies can be downloaded for further use.

This project is an interactive Direct Preference Optimization (DPO) playground for experimenting with real LLM behavior-tuning. The app lets you load a variety of open models, generate multiple candidate answers, and explicitly encode human preferences (chosen vs. rejected responses) through an intuitive Gradio interface.

Using these preference pairs, the app trains a LoRA-adapted policy model against a frozen reference model, shifting the model’s behavior toward your desired style, tone, or reasoning pattern. You can explore how DPO changes alignment by collecting preferences, running training rounds, and immediately testing the tuned policy model on new prompts.

Purpose: Experimental tool for understanding alignment, safety, and model personalization techniques, without requiring deep ML infrastructure. It supports multiple models, adjustable generation parameters, preference visualization, and downloadable tuned LoRA adapters for further use.