Fixes causal test results misalignment when skip:true tests are present#378
Fixes causal test results misalignment when skip:true tests are present#378
Conversation
🦙 MegaLinter status:
|
| Descriptor | Linter | Files | Fixed | Errors | Elapsed time |
|---|---|---|---|---|---|
| black | 32 | 1 | 1.2s | ||
| ✅ PYTHON | pylint | 32 | 0 | 6.06s |
See detailed report in MegaLinter reports
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #378 +/- ##
==========================================
+ Coverage 97.29% 97.87% +0.57%
==========================================
Files 27 27
Lines 1555 1551 -4
==========================================
+ Hits 1513 1518 +5
+ Misses 42 33 -9
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
jmafoster1
left a comment
There was a problem hiding this comment.
This solution means that skipped tests don't make it into the results file. Personally, I'd prefer they were still there so we have a record of all the test outcomes so we know if any where skipped (as opposed to not generated/considered). It'd be good to preserve order too if we can, although putting skipped test cases at the end is more acceptable than not having them there at all.
|
Is this what you're after? For example, a causal test case with now has a corresponding result: |
jmafoster1
left a comment
There was a problem hiding this comment.
This is a nice solution. I've suggested a couple of potential minor improvements
Co-authored-by: Michael Foster <13611658+jmafoster1@users.noreply.github.com>
Co-authored-by: Michael Foster <13611658+jmafoster1@users.noreply.github.com>
jmafoster1
left a comment
There was a problem hiding this comment.
Thanks @f-allian. This looks good to me
In
causal_testing/main.py, thesave_resultsmethod now filters out tests marked with"skip": truein the test configuration, so only active tests are paired with their results and saved.In
tests/main_tests/test_main.py, bothtest_ctfandtest_ctf_exception_silentnow use only active tests (not skipped) when checking which tests passed, aligning the test logic with the updated main code.Closes issue Misaligned test results when skip:true tests are present in causal tests #377