Testing with pytest not working

[lelaus]

          Hi @lelaus,

It seems there's an interaction between test functions when I execute all of them with pytest. I later realized that you use unitttest and from the GitHub actions I see that you execute the files one after the other. I would consider this a non-issue now but let me just demonstrate what may have happened.

Consider running the test modules test_bismuth_fromX_spanT.py and test_custom_properties.py with pytest:

• If I run the modules separately:

$ pytest test_bismuth_fromX_spanT.py
...
test_bismuth_fromX_spanT.py ....                                                                                                                                                                                                         [100%]

======================================================================================================= 4 passed, 126 warnings in 1.15s ========================================================================================================
$ pytest test_custom_properties.py 
...
test_custom_properties.py ....                                                                                                                                                                                                           [100%]

============================================================================================================== 4 passed in 0.42s ===============================================================================================================

• If I specify the module test_custom_properties.py before test_bismuth_fromX_spanT.py:

$ pytest test_custom_properties.py test_bismuth_fromX_spanT.py
...
======================================================================================================= 8 passed, 134 warnings in 1.19s ========================================================================================================

• If I specify the module test_custom_properties.py after test_bismuth_fromX_spanT.py:

$ pytest test_bismuth_fromX_spanT.py test_custom_properties.py
...
=================================================================================================================== FAILURES ===================================================================================================================
________________________________________________________________________________________________________ BismuthTester.test_init_fromX _________________________________________________________________________________________________________

self = <test_bismuth_fromX_spanT.BismuthTester testMethod=test_init_fromX>

    def test_init_fromX(self):
        for bismuthP in bismuthPs:
            properties = load_prop('lbh15.properties.bismuth_properties')
            for prop in properties:
                name = prop.name
                if name in Bismuth.roots_to_use().keys():
                    continue
                val = getattr(bismuthP, name)
                init_dict = {name: val}
                fromX = Bismuth(**init_dict)
>               self.assertAlmostEqual(bismuthP.T, fromX.T, tol, name+" FAILED")
E               AssertionError: 548.15 != 1102.673571428572 within 8 places (554.5235714285719 difference) : rho FAILED

test_bismuth_fromX_spanT.py:48: AssertionError
=========================================================================================================== short test summary info ============================================================================================================
FAILED test_bismuth_fromX_spanT.py::BismuthTester::test_init_fromX - AssertionError: 548.15 != 1102.673571428572 within 8 places (554.5235714285719 difference) : rho FAILED
=================================================================================================== 1 failed, 7 passed, 78 warnings in 0.93s ===================================================================================================

After some inspection it seems init_dict = {name: val} for rho would have different values depending on which test module is executed first. The value is either 'rho': 10056.257 or 'rho': 10832.59 for T = 548.15. I suspect the latter value comes from the custom correlation 11600 - 1.4 T provided in the test directory. The line Bismuth.set_custom_properties_path(bismuth_custom_property_path) at the top level of test_custom_properties.py may have altered which correlation to use across test modules and that caused some tests to fail depending on the order of execution.

Again because you're not using pytest and execute the test modules one by one this is a non-issue albeit a curious one.
If in the future, if you'd like to use pytest for, perhaps, better scalability you might want to consider this finding again.

I'm removing that task from the above.

Originally posted by @damar-wicaksono in #147 (comment)