[enhancement] Implement IRPO training custom loss

[TheGhoul21]

I was wondering if I could try implement the custom loss function from the paper Iterative Reasoning Preference Optimization

Basically looks like the DPO loss is weighted by some factor alpha (in the paper the suggested value is 1.0)

the first term is just the NLL from the policy model of the winning sequence scaled by the length of the sequence itself.
The paper talks about CoT and labels, but probably this could be kept to the developer implementing it since in the loss function the terms always appear as concatenated.

What I was thinking is extending dpo_trainer (because it helps a lot I guess...), add one more hyperparameters which is alpha, defaulted to 1.0

Then the compute_loss takes is edited, and adds the first term

[TheGhoul21]

Turns out the paper was corrected in V2 and now it's obviously a NLL
I did the custom implementation based on dpo and fknetuned mistral on a different dataset obtaining similar great results so I'll share my implementation in the following days

[younesbelkada]

Very nice, thanks @TheGhoul21 ! Let us know with @kashif when your implementation is ready