Replies: 5 comments
-
SubsystemsRequired
Nice to have
|
Beta Was this translation helpful? Give feedback.
-
|
Related to: |
Beta Was this translation helpful? Give feedback.
-
|
Responsible developers: @wlav and @marcpaterno |
Beta Was this translation helpful? Give feedback.
-
|
In case it's relevant: Intel's distribution for Python |
Beta Was this translation helpful? Give feedback.
-
|
This issue was satisfied with this demonstration. How Python algorithms interact with C++ algorithms within Phlex will continue to be explored in the F2 milestone (see Framework-R-D/phlex#70 and Framework-R-D/phlex#71). |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
DUNE US S&C R&D item 065
A satisfactory demonstrator may be that the framework starts with array of data, a Python algorithm is configured to take elements from the array plus some configuration parameter (e.g. a threshold), and the framework calls that algorithm with its own scheduler.
Some thoughts:
For the purposes of this milestone, "a Python algorithm" is assumed to mean a single algorithm written in Python taking an input and producing an output. We do not consider parallel execution of multiple Python algorithms at this time, primarily because many important Python extension modules (e.g. SciPy and NumPy) are not yet ready for that.
It is unclear how much of the data model needs to be exposed to the Python side in order to adequately demonstrate this milestone. Simply copying data into established Python types (e.g. NumPy arrays) might not satisfy all experiment use cases, such as those with large memory requirements.
If we limit the configuration of the algorithm to using a toy class rather than the full system, then we could envisage using Meld to call a Python function with toy configuration and toy data. This would obviate the need to have a fully-specified IO or plugin management system, while still allowing us to pick a bindings technology and develop the necessary interfaces.
Enough of the IO subsystem should be specified to enable the receipt of output data from the Python algorithm for use in subsequent algorithms (without limitation on the language of that subsequent algorithm).
Enough of the scheduler should be specified that the Python algorithm may be invoked with access to its input and configuration data; and a different algorithm may be invoked subsequently with access to its configuration and the data produced by the Python algorithm.
Some scenarios that will likely bring subsystem requirements into existence:
From experimenting with meld, the following will likely bring in subsystem requirements, too:
std::shared_ptr) should be transparent (and play nice with ref-counting) on the Python side. Note that Numba does not support C++. Note that the Numba runtime is not safe, so Numba-JITed can't rely on it and we should find a way to detect such cases.StopIteration).Beta Was this translation helpful? Give feedback.
All reactions