Parallel HDF5 protocols

[gcoyle83]

I have found it difficult to integrate parallel processing with this library due to HDF5 implementation. Is there any plan (or even interest) at present to re-tool for Parallel HDF5 support? Or am I missing an important point in setting up the existing codebase for parallelism?

[PaulDudaRESPEC]

Maybe a partial answer to your question... We're currently working on some refactoring that will have all the computational code of HSP2 work off pandas dataframes... essentially removing the HDF5 dependency while still supporting that format. The user will be able to populate those dataframes from HDF5 or from UCI/WDM files (or even other formats), it won't matter to the computational code. I'm not an expert in parallel processing, but once that refactoring is complete, I think you'll be much closer to enabling the integration you're looking for.

For more details on the refactoring towards IO using pandas dataframes, see #182.

[austinorr]

@gcoyle83, I'm a little late to the party here, but I'm curious to hear more about your desired use case. Is your intent to parallelize pieces/subroutines of one model, or is your intent to dispatch/run many models in parallel? I think the work on #182 will support the latter case, but if you're still thinking about this I'd like to hear more about your use case.
No part of our development discussion has focused on the parallel HDF5 retool you linked to, though that effort does sound interesting and exciting when we are ready to tackle the former use case of running portions of one model in parallel.