Time difference check too short in test-provenance.R

[bruc]

The time difference check in test-provenance.R is too short for a loaded system. Currently, the time difference check is "less than 0.1", and on a loaded server, that time difference has been observed to be exceeded. I've instrumented this test script and found that time differences can be over 0.25 seconds.

I recommend changing the time difference check to 1 second in the two places where it occurs. In addition, I also recommend that the Sys.sleep() call be changed to 2 seconds instead of 0.2 seconds for the same reason.

I discovered this problem by repeatedly running R CMD check on all the packages in our R environment, and the above problem occurred after 117 tries when our server workload was high.

Thank you.

[billdenney]

I will consider this for the next release. As the test is intended for verification of the provenance testing working as intended, and the time difference is ancillary to the overall testing, I'm not sure if I will incorporate it into the next release (repeatedly running the test 117 times and then getting the failure is not the intended use of the testing, though I realize it is likely a fuzzing result).

[bruc]

We're using PKNCA in a validated environment and running the tests of all our packages is part of our policy on including R packages. We test over 2000 packages, so having a test failure probability of 1/117 would mean we'd never pass all our tests. Thankfully, other packages do not have such a high failure rate. In reality, we had three packages that failed in repeated testing. I contacted the other two and they have fixed their tests to avoid such problems (Both of these packages were implementing mathematical algorithms that had known limitations. However, because their test data sets were generated by random numbers without setting the seed, they would occasionally generate data sets that were not solvable. I asked them to set the random number seed in their tests, and the problems were resolved.)

Thank you.

[billdenney]

I also use PKNCA in a validated environment. And, I specifically designed PKNCA to be a validated tool to work in validated environments. Your need is beyond that of a validated environment, but it is to change the package to work with the specific way that you are testing it (while the same tests run on thousands of other systems where they are not overloaded, more like a real world test than an edge case).

The specific test you're referring to validates that the PKNCA results were generated on a given system at a given time. Relaxing that test would decrease the verifiability of PKNCA results.

A solution may be to run your testing infrastructure with a lower overall load (e.g. modify your testing to only run one package's test per core (not hyper threaded core) on the system, if you are currently running more).

So, what you are asking is to modify PKNCA to allow your particular validation testing infrastructure to pass while relaxing the overall validation and verification of the results in PKNCA. That is why I am considering it without guaranteeing that I will change the behavior before carefully considering the impact.