Nice!

[johannahaffner]

Hi Jason,

super nice! I was actually at the very start of writing my own parser for SIF - to use the CUTEst environment in JAX, and use it to benchmark things in optimistix.
Seems like that is not needed anymore :D

Do you have the full suite of CUTEst problems, including the constrained ones?

Cheers

Johanna

[packquickly]

Hi Johanna,

Thank you!

I don't have the full CUTEst suite, and I don't think it's quite the scope I'm aiming for with Optpile anymore either. When implementing this I was disappointed by how disconnected the test problem literature was from the actual optimisation problems I've encountered, but I needed a base for benchmarking which could be defended by pulling from the academic literature. Now, I'm more interested in including problems resembling actual math modeling problems, such as fitting a mechanistic model given artificial or real data, similar to the MINPACK II test problem collection (which focuses on real data I believe).

I have one of these problems implemented already, namely the CoatingThickness least squares problem which is from MINPACK II. In that problem I generate data from a model with a hidden set of parameters (adding randomness to the generated data if desired) and then try to recover the set of hidden parameters with the optimization.

I think this is a decent---albeit imperfect---path forward, as I think it is pretty easy to get difficult mechanistic models which I can write down and generate data from and aren't as contrived as traditional test problems, but it's pretty difficult to get ahold of actual data.

So far I have focused on unconstrained optimisation because that's what I've been benchmarking. I am open to contributions of constrained problems although there isn't an existing API for them. It shouldn't be too hard to make one though.

[johannahaffner]

Hi Jason,

that is certainly super useful! Do you encounter cases where the benchmarking problems are some contrived thing that the solver does poorly on, while performing well on actual data?
Or would you say that it fails on real data for tasks that the benchmark problems are simply not testing for? In other words, can I expect a solver that does well on benchmarks to at least not flounder in the real world?

but it's pretty difficult to get ahold of actual data.

Are you aware of the PETab benchmark collection? This would require you to work with the PETab format, which is a thin wrapper over SBML. SBML itself, or systems biology markup language, is standard format to encode mechanistic biological models in .xml files. This package allows you to work with these in JAX, with some support for diffrax. (My experience with this is limited to this issue, though.)

I am open to contributions of constrained problems although there isn't an existing API for them. It shouldn't be too hard to make one though.

Great! I'd like to contribute to this to benchmark constrained solvers I am writing for optimistix.