add strong wolfe condition

[jaeminoh]

Hi, I've implemented the strong Wolfe conditions for the backtracking line search.

However, one potential issue related to the class of QuasiNewton's methods is that the sufficient decrease condition

$$|\nabla f(y_{k+1}) \cdot (y_{k+1} - y_k)| \le c | \nabla f(y_k) \cdot (y_{k+1} - y_k) |$$

requires $\nabla f(y_{k+1})$. Currently, the step method of the AbstractQuasiNewton class does not pass such information (line 213--220):

step_size, accept, search_result, search_state = self.search.step(
            state.first_step,
            y,
            state.y_eval,
            state.f_info,
            FunctionInfo.Eval(f_eval),
            state.search_state,
        )

[johannahaffner]

Hi!

and first of all, sorry for taking so long to get back to you! I've left two small comments, but overall I'm afraid that we can't merge this as is.

The numerical reason for this is that this search makes an assumption that is unlikely to be met in general - that a step length satisfying the Armijo and the Wolfe criteria can be found by backtracking. This is not necessarily the case! The curvature condition might only be satisfied for step lengths that are never checked in this implementation, namely everything that is larger than self.step_init. The way to go about this is with a Zoom line search - something we're in the process of getting in: #177. In that case, an interval in which both criteria are fulfilled is located first, and then bisected.

Without a guarantee that both conditions are likely to be fulfilled in the interval we are looking at, we could reject steps until we run out of iterations, without ever moving anywhere.

[johannahaffner]

You can define a union type outside the search, e.g. with _FnInfo = FunctionInfo.EvalGrad | FunctionInfo.EvalGradHessian | ...

[johannahaffner]

Can we find a more descriptive name for this parameter?

[jaeminoh]

Thanks for replying!

The numerical reason for this is that this search makes an assumption that is unlikely to be met in general - that a step length satisfying the Armijo and the Wolfe criteria can be found by backtracking. This is not necessarily the case!

I got the point -- this may have motivated the zoom line search.

I have implemented this because this was not in optimstix but was in jaxopt. I’m not sure how to proceed, but perhaps a small refactoring to reduce code duplication would be better?

[johannahaffner]

I have implemented this because this was not in optimstix but was in jaxopt. I’m not sure how to proceed, but perhaps a small refactoring to reduce code duplication would be better?

This one? https://jaxopt.github.io/stable/_autosummary/jaxopt.HagerZhangLineSearch.html?

I’m not sure how to proceed, but perhaps a small refactoring to reduce code duplication would be better?

Do you mean code duplication between Optimistix and JAXopt? If you're referring to this PR - without a means to locate a promising interval, checking the conditions isn't enough. If you'd like to keep experimenting, then I suggest adding your search to custom solvers defined here

optimistix/tests/helpers.py

Line 84 in 23b99d2

# We define a bunch of helper optimisers which exist to test all code paths, but which

and running the tests for these - currently your search is completely untested. If you'd like to keep working on making a HagerZhang line search instead, then this is not a small refactoring, but rather a quite substantial technical task. If that is not for you, that is fine too - you can just close this PR as well.