Parallel test execution

[jmafoster1]

Is your feature request related to a problem? Please describe.
The JSON frontend currently executes test suites in a for loop. For large datasets, this can be quite slow. It would be good if we could execute tests in parallel to speed things up a bit, although it is desirable to preserve the order in which results are presented.

I believe the easiest way to do this would be to have the msg object be returned, and then we could just use multiprocessing and pool.map.

[christopher-wild]

After spending more time than I want to admit looking at this, it sadly isn't as simple a solution as I first thought.

When sending the arguments of the _execute_test_case function into the pool.map, they get pickled. However the CausalTestCase object is not pickleable due to this error message:

ValueError: ctypes objects containing pointers cannot be pickled, which is not something I have ever come across before.

I've tried using dill as an alternative and more flexible serialisation library but that returns an almost identical error as well.

Going to look into it a bit more but it might be a larger task than we first thought

[jmafoster1]

Thanks for looking into this Chris. It's not a massive priority if it's going to take up a lot of time.

[jmafoster1]

@f-allian was this one of the things you were going to look into on the DAFNI side of things?

[f-allian]

@jmafoster1 Yes, DAFNI allows parallelising steps in a given workflow as they have the throughput power available to do that. That'll be separate from this issue, though. Definitely worth revisiting this problem in the future.