Testing

MATLAB Testing Environment

General Remarks

Types of tests:

• Unit test: testing of individual units of source code (scripts, functions, classes)
• Integration test: testing of a combination of individual units as a group.
• Regression test: re-running all tests to ensure that the previously developed and tested code still performes after a code change.

MitC tests

• I have created a /tests folder in the repository with a number of tests. The naming convention for writing a test for a particular MATLAB script is to prefix “test_” to the name of the script that is being tested. For example, a test for the file draw_random_number.m should be called test_draw_random_number.m. In general, Matlab will recognize any scripts that are prefixed or suffixed with the string “test” as tests.
• I have chosen to use Class-Based Unit Tests as they seem to be more flexible than the other options (Script-Based or Function-Based).
• You can manually run all tests with the script run_tests.m in the folder /mitc. This will find and run all tests in the /tests folder. The test results and the coverage report are stored in the folder /tests/Reports.
• The tests are currently not complete, but the pipeline is modular and easily extendible. For now, I have mainly written unit tests, i.e. only testing specific functions, and not yet integration tests, i.e. testing whole features.
• Matlab also has a framework to test apps. I have created a first simple test for this, see test_app.m.

Continuous Integration with Travis

Continuous Integration is the practice of merging in small code changes frequently - rather than merging in a large change at the end of a development cycle. The goal is to build healthier software by developing and testing in smaller increments. I have created a Continuous Integration pipeline with Travis. As a continuous integration platform, Travis CI supports your development process by automatically building and testing code changes, providing immediate feedback on the success of the change. Travis CI can also automate other parts of the development process by managing deployments and notifications. The CI pipeline ensures that each time we push our local repository to GitHub, Travis will deploy a server to run and verify all the tests. I will check whether we can also do this with GitHub Actions.

Travis requries a configuration file within the root folder of the repository called .travis.yml [1]. It contains basic commands to instruct Travis to setup a MATLAB installation on a remote server and run all the tests. With this script, Travis will deploy a Linux server and install Matlab (version R2019a and R2020b). It will then find all tests in the repository and execute the command "runtests".

language: matlab
matlab:
  - R2019a
  - R2020b
script: matlab -batch "addpath(genpath(pwd)); results = runtests('IncludeSubfolders',true); assertSuccess(results);"

Travis will add a check/cross mark to the commit if the tests passed/failed.

Continuous Integration with GitHub Actions

GitHub actions now supports CI for Matlab (since Jan 2021). Although the workflows are not yet available in the GitHub market, examples to run tests and generate coverage reports are provided.

This would circumvent the issue with Travis not supplying unlimited builds for open source projects [2].

References

[1] https://docs.travis-ci.com/user/languages/matlab/
[2] https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricing-plan-threw-wrench-my-open-source-works