Exact R2 Indicator

[schaepermeier]

Hey Manuel (et al.),

I have finally implemented the exact R2 indicator (for bi-objective solution sets) in the C library and added the Python interface for it. The code is mostly based on the hypervolume function for two objectives.

I have also added basic documentation for the exact_r2 function in the Python package.

For rudimentary testing, I have attached a Jupyter notebook: tests.ipynb

For a full integration of the exact R2 indicator, there's still the following todos:

1. R interface: I can probably help with that
2. matlab interface: I don't have experience with matlab, unfortunately
3. test suite: I'm not quite sure where to integrate which kinds of tests. Guidance would be welcome here!

Further R2-related functions could be implemented:

1. approximate R2 indicator (i.e., the R2 indicator as commonly known in the community) for any number of objectives
2. R2 contributions (see R2 v2 ECJ / arXiv)

I'm still getting familiar with developing for the library, so I'm opening a PR for the current state of my implementation :-)

Let me know how I can help to achieve a quick turnaround with the merge!

Cheers,
Lennart

[MLopez-Ibanez]

I wouldn't worry about the MATLAB interface unless you are planning to use it yourself.

[MLopez-Ibanez]

It would be better to create a variant of generate_sorted_doublep_2d that filters points such that p[j][0] < ref[0]. Then you can delete the while loop below.

[codecov]

Codecov Report

❌ Patch coverage is 85.91549% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.01%. Comparing base (7699c63) to head (020b043).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #55      +/-   ##
==========================================
+ Coverage   77.91%   78.01%   +0.09%     
==========================================
  Files          62       63       +1     
  Lines        5878     5949      +71     
  Branches      901      913      +12     
==========================================
+ Hits         4580     4641      +61     
- Misses       1100     1103       +3     
- Partials      198      205       +7     

Flag	Coverage Δ
R	69.84% <ø> (ø)
python	75.95% <85.91%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
tests	100.00% <100.00%> (ø)
src	87.56% <88.23%> (+0.01%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MLopez-Ibanez]

See the tests in test_moocore.py. I would add similar tests to those added for epsilon/igd/hypervolume.

[MLopez-Ibanez]

If tests are longer than .5 s, it may be a good idea to create a command-line interface (see epsilon.c) and add the tests to https://github.com/multi-objective/testsuite

[MLopez-Ibanez]

To give it more visibility, you may want to extend the examples here: https://github.com/multi-objective/moocore/blob/main/python/examples/plot_metrics.py or add its own example showcasing why/when people should use this R2 instead of other metrics. This can be done as a follow-up (but it will also provide additional testing so it may be a good idea to do it now).

[MLopez-Ibanez]

This looks great. Are you testing what happens if the ref point is dominated and nondominated? I may have missed it. Also, could you squash all commits into one? I can do it if you don't know how, but it will look better in the logs if you do it.

[MLopez-Ibanez]

All tests passed so let me know if you want to change anything else and if you want me to squash the commits, before I merge this.

[schaepermeier]

I've added the mathematical details and squashed the commits now! One thing you could add is the original source of the R2 indicator to the documentation, as I don't find it in the linked bib-files and cannot easily add it myself:

@techreport{hansen1998,
  title={{Evaluating the Quality of Approximations to the Non-dominated Set}},
  author={Hansen, Michael Pilegaard and Jaszkiewicz, Andrzej},
  year={1998},
  institution={IMM, Department of Mathematical Modelling, Technical University of Denmark},
  number = {IMM-REP-1998-7}
}

Thanks!

[MLopez-Ibanez]

I've added the mathematical details and squashed the commits now! One thing you could add is the original source of the R2 indicator to the documentation, as I don't find it in the linked bib-files and cannot easily add it myself:

@techreport{hansen1998,
  title={{Evaluating the Quality of Approximations to the Non-dominated Set}},
  author={Hansen, Michael Pilegaard and Jaszkiewicz, Andrzej},
  year={1998},
  institution={IMM, Department of Mathematical Modelling, Technical University of Denmark},
  number = {IMM-REP-1998-7}
}

Thanks!

https://iridia-ulb.github.io/references/#HanJas1998

You just need to add the label to bibkeys.txt and run update_bib.sh and it will fetch it from the iridia-ulb GitHub repo.

[schaepermeier]

I've added the mathematical details and squashed the commits now! One thing you could add is the original source of the R2 indicator to the documentation, as I don't find it in the linked bib-files and cannot easily add it myself:

@techreport{hansen1998,
  title={{Evaluating the Quality of Approximations to the Non-dominated Set}},
  author={Hansen, Michael Pilegaard and Jaszkiewicz, Andrzej},
  year={1998},
  institution={IMM, Department of Mathematical Modelling, Technical University of Denmark},
  number = {IMM-REP-1998-7}
}

Thanks!

https://iridia-ulb.github.io/references/#HanJas1998

You just need to add the label to bibkeys.txt and run update_bib.sh and it will fetch it from the iridia-ulb GitHub repo.

I figured out how to use the script already, I just did not find that reference in the files 😁 ... Anyway, it worked now! That should be all for this PR, I think :-)