Adds Online DPO

[edbeeching]

WIP ignore for now

usage

accelerate launch --config_file deepspeed_zero3.yaml examples/scripts/dpo_online.py ----model_name_or_path=HuggingFaceH4/mistral-7b-ift --model_revision=v25.2 --output_dir=data/mistral-7b-odpo --dataset_name=HuggingFaceH4/ultrafeedback_binarized --dataset_train_split=train_gen --dataset_test_split=test_gen --gradient_accumulation_steps=1 --bf16=True --attn_implementation=flash_attention_2 --per_device_train_batch_size=2

[olgavrou]

This is cool, I was doing the same but by extending the training_step of the existing dpo trainer and generating the new pairs there before calling super().training_step. This looks like a more complete solution

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.