payelb/HHRLHF_Llama-3.2-1B_aligned_with_semantic_MARS_deberta_RM

Base model: meta-llama/Llama-3.2-1B-Instruct

Alignment dataset: Anthropic/hh-rlhf

Reward model: payelb/HHRLHF_reward-model-deberta-v3-base_1k_fixed_MARS_semantic_distance_synth

Method: PPO alignment with LoRA adapters.

Reward model type: semantic-MARS DeBERTa-v3-base reward model.

Matched PPO setup:

• TOTAL_PPO_STEPS: 250
• PPO_EPOCHS: 2
• LR: 1e-05
• Batch size: 16
• Mini-batch size: 4
• Gradient accumulation: 4
• MIN_NEW_TOKENS: 32
• MAX_NEW_TOKENS: 96
• USE_REWARD_NORMALIZATION: False
• USE_EXPLICIT_KL_CONFIG: False
• Generation: do_sample=True, top_p=0.9, temperature=0.8, eos_token_id=eos_token
• LoRA enabled