Disabling a Test Case in the profile does not prevent it from being run if called explicitly

[tgreenx]

The profile can be used to prevent Test Cases from being run (i.e. by removing them from the profile). This is true when Engine runs the whole test suite, or even specific Test modules, but not when running specific Test Cases (i.e., using Zonemaster::Engine/test_method() or Zonemaster::Engine::Test/run_one()). That is because the checks are done directly in the all() method of each Test module (with the help of Zonemaster::Engine::Util/should_run_test()). That method is not used when running specific Test Cases.

For example, using Zonemaster-CLI, after removing Address01 from the profile (note that Engine is started twice because of two --test arguments):

$ zonemaster-cli --show-testcase --test Address --test Address/address01 --level DEBUG --no-ipv6 --raw zonemaster.net | grep TEST_CASE_START
   0.00 DEBUG    ADDRESS02      TEST_CASE_START   testcase=address02
   3.33 DEBUG    ADDRESS03      TEST_CASE_START   testcase=address03
   0.00 DEBUG    ADDRESS01      TEST_CASE_START   testcase=address01

It could be intentional, but that seems odd to me. So, bug or feature? I would advocate for the former.

[matsduf]

Yes, Engine is started twice, but I noticed when preparing data for #1281 that cache and blacklisting survive from the first run to the second run. But I think it would be better if Engine could be run once.

I think it is fine that you can include a test case not in the profile if you explicitly calls it. What is the problem of that? What do you think should happen? And why?

[tgreenx]

I think it is fine that you can include a test case not in the profile if you explicitly calls it. What is the problem of that? What do you think should happen? And why?

The profile acts as a configuration file for Engine. I would except that removing (disabling) a Test Case from the configuration would make it inaccessible, in all cases. If disabled but called explicitly, I expect Engine to return an error (either MODULE_ERROR or UNKNOWN_METHOD).

[matsduf]

Resolved by #1294