Port this into Optimistix proper?

[johannahaffner]

Hi Jason,

we're now building our own collection of benchmark problems (motivated by a desire to implement constrained optimisation) over in optimistix. They're CUTEST problems that I'm parsing with Claude and then verify against other interfaces.

Unlike what you have here, we don't have much of a notion of hyperparameter variation. Perhaps this could be added?

We're building this on top of pytest benchmark and have plans to keep this around long-term.

What do you think?

[packquickly]

Hi Johanna,

That sounds good! Obviously I haven't gotten around to doing much more here and getting into Optimistix as a proper benchmark would be great (+supporting all of CUTEst would be great to have in the JAX community.)

What exactly is needed? Were you thinking of taking the hyperparam randomization/adjustment from here and porting over to this? Or the problems as well?

Let me know!
Jason

[johannahaffner]

Great! You've done really nice work here and we have lots of use cases for that.

In general the plan is to vendor in the CUTEst problems (for now, they could become their own thing), and run these when we make larger changes or introduce a new solver. We're using pytest-benchmark, so the actual benchmarking looks a lot like a normal test that is parameterised with solvers and problems. Benchmarking results can be exported to a json file, and any plots can be created from that.

So far I have actually not given much thought to hyper parameter tuning/testing, and I'd be curious to hear your thoughts on the matter. Which parameters are solvers typically most sensitive to? Given that we're working with the pytest machinery, the pragmatic way to go about it is probably to just add the same solver with different hyperparameters to the list over which the benchmark (test) gets parameterised.

As for the problems - I'd love to have the complete set in Optimistix. The problem structure I set up is quite similar to what you have created here, so I think the port would be quite doable.

PS.: relevant optimistix PR here: patrick-kidger/optimistix#128

[johannahaffner]

Alright, I have started porting the problems. Creating a list here to keep track of things:

Minimisation problems

CUTEst

• HIMMELBG
• SINE - I could not find a SIF file for this name, are you sure about the name? (SINE as a function shows up a ton.)
• COSINE
• MCCORMCK - bounded
• HARKERP2 - bounded
• EXPLIN - bounded
• CURLY20
• ENGVAL1
• EG2
• ARWHEAD
• FLETCBV3

Based on the file you shared, here are the class names defined in the file along with their directly preceding comments:

UOTF

• Diagonal2
• Diagonal3
• Diagonal5
• Diagonal7
• Diagonal8
• Diagonal9
• FullHessian3
• Hager
• QuadraticQF1
• QuadraticQF2
• Raydan1

also in UOTF, but already accounted for in CUTEst:

• FLETCBV3
• ARWHEAD
• EG2
• ENGVAL1
• CURLY20
• EXPLIN1
• HARKERP2
• MCCORMCK
• COSINE
• SINE
• HIMMELBG

LSBF

• Ackley1
• Ackley2
• Ackley3
• Bird
• Bohachevsky2
• Bohachevsky3
• BraninRCOS
• BraninRCOS2
• CamelThree
• CamelSix
• ChenBird
• Colville
• Csendes
• Damavandi
• DeckkersAarts
• EggHolder
• Griewank
• Hosaki
• Keane
• Pathological