Code for DPO-finetuning?

[KasenYoung]

Thanks for your great work. I wonder if you can release the code for finetuning video models using your reward model. This procedure can show your work's strengths better.

[yifan123]

Thank you for your kind words and interest in our work!

Regarding your question: The DPO implementation in VideoAlign was built on an internal codebase for training closesourced models, which unfortunately isn’t publicly available at this time. For replication, if you have a repository for fine-tuning a flow-based video generation model, modifying it to incorporate flow-DPO is relatively easy:

1. Base Model: Start with open-source flow-matching video models (e.g., Wan2.1-T2V-1.3B).

2. Data Prep: Structure preference data using paired (win/lose) videos. Follow the common DPO practice: win_videos = batch[:N], lose_videos = batch[N:] (see DiffusionDPO data handling).

3. Loss Implementation: Adapt the flow-DPO loss from VideoAlign's Appendix E pseudocode. You can also refer to the implementation of DiffusionDPO's loss logic.

Key Insight: For existing LoRA-tuned video models (flow-matching based), adding Flow-DPO mainly requires modifying the data pipeline and loss computation.

[ArlenCHEN]

hi @yifan123,

Thanks for those very useful recommendations.

I just wonder if you have ever encountered the issue of loss quickly reaching 0. Does this mean a reward hacking? Any insight is appreciated!!

Thanks!

[yifan123]

It seems that the difference (w_diff - l_diff) is very large, causing the loss to quickly reach 0. This usually happens when the pattern between the win and lose pairs you constructed is particularly obvious, making it easy for the model to distinguish between the two types of data. The most straightforward solution is to increase the beta, possibly by 100 times its current value, to prevent the model from overfitting. You could also modify your data, as win and lose pairs that are too easy to distinguish indicate that the lose data is not a hard negative sample.

[ArlenCHEN]

@yifan123 thanks for your quick reply. what you said really makes a lot of sense. seems making proper pairs is a so much important and not easy topic for video dpo.

do you think other methods, for example, some reference-free methods, like this one: SimPO: Simple Preference Optimization
with a Reference-Free Reward, can solve/mitigate the problem?

[ArlenCHEN]

another thing: i encountered the issue when i used a full-parameters tuning. a lora-based training is still on the way.

do you think using full-param instead of lora can be one of the reasons?

[yifan123]

Thanks for your interesting for our work!

As discussed in the paragraph Discussion on $\beta_t$ in Section 4.1, our experiments show that setting $\beta_t = \beta(1 - t)^2$ degrades performance, whereas using a constant $\beta_t$ yields better results.

[mariolarcher]

Hi @yifan123 ! first congrats for your amazing work!

In your pseudocode, if I'm not mistaken, the loss is not computed as the mean of the losses for each element in the batch, as it is implemented in diffusion-DPO. Do you think this could cause the optimal beta to vary with the batch size, and, in general, make the magnitude of the loss dependent on batch size? What do you think about using the diffusion-DPO implementation (computing the loss for each element and then taking the mean)? Would that make sense?

[yifan123]

Hi @mariolarcher, the pseudocode is intended to illustrate the overall algorithm flow, hence it's for the case when batch=1; For batch size>1, you should use loss.mean().

[mariolarcher]

Thanks, @yifan123! However, using .norm() should already give a scalar, so applying .mean() would have no effect (loss.mean() == loss), I think.

My understanding is that the current code aggregates inside the log sigmoid, rather than outside. It's not estimating the expected value of log(sigmoid(...)) like sum(log(sigmoid(x))) / N. Instead, it computes log(sigmoid(sum(x))). Even adding the / N wouldn't make it the same formula. Does that make sense?

[yifan123]

For the case where batch > 1, you need to compute sum(log(sigmoid(x))) / N.

[mariolarcher]

Thanks, make sense!

[yifan123]

How did you construct the positive and negative pairs? My suggestion is to collect N images/videos using the same prompt, then select the one with the highest reward as the positive and the one with the lowest reward as the negative. This way, DPO will significantly outperform SFT, and the effect becomes more pronounced as N increases. Generally, I recommend using N ≥ 16.

[yifan123]

The time shift is likely not critical, but I used the same shift as in pretraining. The learning rate can be found in Table 10 of the paper.

[yifan123]

Thanks for your great work. I wonder if you can release the code for finetuning video models using your reward model. This procedure can show your work's strengths better.

For all, thank you so much for your interest! I've provided a script for applying Flow-DPO to text-to-image tasks:
https://github.com/yifan123/flow_grpo/blob/main/scripts/train_sd3_dpo.py

[mariolarcher]

Thanks for your great work. I wonder if you can release the code for finetuning video models using your reward model. This procedure can show your work's strengths better.

For all, thank you so much for your interest! I've provided a script for applying Flow-DPO to text-to-image tasks: https://github.com/yifan123/flow_grpo/blob/main/scripts/train_sd3_dpo.py

Thanks @yifan123 !