JoSS Review: Testing

[vsoch]

This is for openjournals/joss-reviews#1637. The last major requirement for this repository is to add proper testing, either via Travis, CircleCI, or another service of your choosing. Especially given the errors that came up (and a file missing from the release) it would be best to do a minimal build, and set of tests for the core software. There are a lot of really good starter example templates, and it's fairly easy to connect and setup.

[tonggehua]

@vsoch We are working on this with Github Actions. What level of testing would you suggest? We have unit tests for the back-end code and plan to include those as part of the CI workflow. We have encountered errors from the GUI frequently though - should there be automated GUI tests too?

[vsoch]

Front end testing is hard, so I would do unit tests for what you are able to do, and then test build / install and deploy of the interface. With selenium (or similar) you can do basic page navigation and just confirm that the response is successful (e.g., 200).

[tonggehua]

Differences between local testing and Github Actions testing

@vsoch We are tying up the Selenium testing. The way we are doing it right now is to run coms_server_flask.py while another thread kicks off a unittest runner that uses Selenium to click elements. At the moment, the tests check whether you are on the right page / correct elements are present, for login/register/acquire/analzye/tx/rx.

We are seeing discrepancies between local runs and Github action runs:

1. While the GUI test runs successfully on local, Github Actions Selenium has problems clicking elements (on Acquire and Analyze tests that run the computations) because some other element is obscuring them. The problem is probably window size, and it makes less sense to debug it now that a new team has committed to developing a more mature GUI with us.

2. T1/T2 mapping tests compare generated data to benchmark data. The tests pass on Windows (both local and Github actions) and macOS (local), but generates a much larger error on macOS (Github actions). This is described in issue utest_T1_T2_mapping failing on Github Actions (macOS) #41.

3. Github Actions macOS doesn't have Firefox installed for Selenium testing.

Would this be acceptable right now?

[vsoch]

The one thing I would try before completely throwing in the towel is to run the same tests, but use CircleCI. I've done quite a bit of browser testing there and consistently get the same tests locally as on the CI. After that, if there is still issue, then we can say that your team gave their best effort and no go!

But maybe we will get lucky and the CircleCI runs will be consistent!

[sairamgeethanath]

@vsoch We have trying to get this done on Github Actions and want to mark the rest as "To Dos" for the future. It will be another effort to move this to CircleCI. The tests run locally on both operating systems. I appreciate your recommendations but moving to another tool all over will lose all the effort put on Github actions and require time and effort again. I kindly request you to reconsider your suggestion.

[vsoch]

okay, sure thing then. The tests aren't automated and this is below my personal standard (and review criteria) but I can make an exception because I know the amount of time and work your team has put into this. I won't check the automated tests box or close this issue, but I'll let the editors know that I'm consenting to this.

[sairamgeethanath]

Thanks for your comments! However, I would like to clarify that the Github actions are successfully running automated tests for backend functionalities. The front end GUI tests based on your Selenium suggestion runs for windows as well. We are pointing to the discrepancies between local and Github Actions (something we cannot control much). We are not sure about moving to other CI platforms as Actions/related features are well integrated in one tool that we are using. Many thanks for your consideration!

[vsoch]

The front end GUI tests based on your Selenium suggestion runs for windows as well.

Yes, and hopefully this can be fully finished at some point to be run automated.

I suggested in the issue to use Selenium on Circle, which has worked for me many times, and the authors asked for a waiver.

[sairamgeethanath]

Thanks, just to clarify:
This means looking at the discrepancies between Github Actions (different python versions, configurations) and local testing. I am not sure if we need to focus on the software version/libraries discrepancies. Circle CI does not offer "Mac" for the free version.