Unstable Training

[wenjunli-0]

When I switch to the areal v0.3.3 and run the asearcher_local experiment, I got very unstable training and training cannot continue due to very high rollout time costs.

Below are the hyperparameters I used to run the experiment and the logs. (The blue curves are relatively stable and the green and grey curves are unstable and are created after I switched to areal v0.3.3)
epochs=10, 4p1t1+4p1t1, batch_size=128, max_concurrent_rollouts=48, mem_fraction_static=0.80

[samjia2000]

@wenjunli-0 Hi, this is because AReaL has updated the gradient computation part in recent PR. To make sure ASearcher training is stable, the gradient_clipping coefficient should be set to 0.2. Below is the training curve:

[wenjunli-0]

I ran a few new asearcher-local experiments recently and they are normal in the beginning. However, around 200-250 steps, the rollout time starts to explode and the training hangs. I observe the grpo_actor/score/av, number_output_token/avg, num_search_queries/avg and etc, they all seem to be normal and do not explode.

What will cause such a phenomenon, and how should I fix it?

[Taewhoo]

@wenjunli-0 Hi, have you found a solution to this issue (rollout spike around steps 200-250) ?

[wenjunli-0]

@wenjunli-0 Hi, have you found a solution to this issue (rollout spike around steps 200-250) ?

I think it's because of the dynamic filtering, which filters out zero-advantage samples and thus causes the spike (e.g., needs more time to collect non-zero-advantage samples when the model can answer almost all of the questions in the training data).

[Taewhoo]

@wenjunli-0 Hi, have you found a solution to this issue (rollout spike around steps 200-250) ?

I think it's because of the dynamic filtering, which filters out zero-advantage samples and thus causes the spike (e.g., needs more time to collect non-zero-advantage samples when the model can answer almost all of the questions in the training data).

That makes sense. It's difficult to reach the performance of ASearcher-Local-7B using AReal v0.3.3, and perhaps it's because I can only train up to around 220 steps. Maybe I should revert to the legacy version of AReal, as mentioned in issue#2?

[wenjunli-0]

Yes, honestly, I also had difficulty with fully reproducing their experiments and results shown in the paper.