README.md

---
license: apache-2.0
---

# Model Card for decruz07/kellemar-DPO-7B-e

<!-- Provide a quick summary of what the model is/does. -->
Learning Rate: 5e-5, steps 300
## Model Details

Created with beta = 0.05

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** @decruz
- **Funded by [optional]:** my full-time job
- **Finetuned from model [optional]:** teknium/OpenHermes-2.5-Mistral-7B



## Uses

You can use this for basic inference. You could probably finetune with this if you want to.


## How to Get Started with the Model

You can create a space out of this, or use basic python code to call the model directly and make inferences to it.

[More Information Needed]

## Training Details

The following was used:
`training_args = TrainingArguments(
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    learning_rate=5e-5,
    lr_scheduler_type="cosine",
    max_steps=200,
    save_strategy="no",
    logging_steps=1,
    output_dir=new_model,
    optim="paged_adamw_32bit",
    warmup_steps=100,
    bf16=True,
    report_to="wandb",
)

# Create DPO trainer
dpo_trainer = DPOTrainer(
    model,
    ref_model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    peft_config=peft_config,
    beta=0.1,
    max_prompt_length=1024,
    max_length=1536,
)`

### Training Data

This was trained with https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs

### Training Procedure 

Trained with Labonne's Google Colab Notebook on Finetuning Mistral 7B with DPO.

## Model Card Authors [optional]

@decruz

## Model Card Contact

@decruz on X/Twitter