Framework is capable of running a Python algorithm

[knoepfel]

DUNE US S&C R&D item 065

A satisfactory demonstrator may be that the framework starts with array of data, a Python algorithm is configured to take elements from the array plus some configuration parameter (e.g. a threshold), and the framework calls that algorithm with its own scheduler.

Some thoughts:

• We might explore using tbb4py (https://pypi.org/project/tbb4py/)
• This might be an opportunity to specify what Python packages might be usable

For the purposes of this milestone, "a Python algorithm" is assumed to mean a single algorithm written in Python taking an input and producing an output. We do not consider parallel execution of multiple Python algorithms at this time, primarily because many important Python extension modules (e.g. SciPy and NumPy) are not yet ready for that.

It is unclear how much of the data model needs to be exposed to the Python side in order to adequately demonstrate this milestone. Simply copying data into established Python types (e.g. NumPy arrays) might not satisfy all experiment use cases, such as those with large memory requirements.

If we limit the configuration of the algorithm to using a toy class rather than the full system, then we could envisage using Meld to call a Python function with toy configuration and toy data. This would obviate the need to have a fully-specified IO or plugin management system, while still allowing us to pick a bindings technology and develop the necessary interfaces.

Enough of the IO subsystem should be specified to enable the receipt of output data from the Python algorithm for use in subsequent algorithms (without limitation on the language of that subsequent algorithm).

Enough of the scheduler should be specified that the Python algorithm may be invoked with access to its input and configuration data; and a different algorithm may be invoked subsequently with access to its configuration and the data produced by the Python algorithm.

Some scenarios that will likely bring subsystem requirements into existence:

• Ability to schedule Python algorithms back-to-back on the GPU. The marshaling of data products going into the Python algorithm and coming out of it should not require the CPU.
• Support for Numba, which would include the algorithm being able to describe that it can be JITed, but also have the framework be Numba-aware. Normally, Numba generates wrappers for the C function that is called, so the entry point requires the Python interpreter and hence the GIL. By extracting the function point, however, that wrapper can be by-passed.
• Being Numba-aware will also help with scheduling Python algorithms back-to-back on the GPU, as scheduling can likewise be done without involving the Python interpreter, after the Python algorithms were JITed.
• Control of/support for OpenMP through external packages such as NumPy, or directly in the algorithm itself using Numba.
• Any binding technology used should be capable of running on a sub-interpreter (assuming algorithms can, too).
• Since py3.11, there is the stable ABI. Given how experiments release their software it's probably not necessary, but it would be highly convenient if the framework restricts itself to it.

From experimenting with meld, the following will likely bring in subsystem requirements, too:

• Data sources are expected to be able use framework facilities. This means that these facilities require either Python bindings to the existing C++ or some other custom access for Python.
• C++ managed memory (eg. std::shared_ptr) should be transparent (and play nice with ref-counting) on the Python side. Note that Numba does not support C++. Note that the Numba runtime is not safe, so Numba-JITed can't rely on it and we should find a way to detect such cases.
• It should be possible to propagate Python exceptions/errors from threads back to the main program. Important here is that Python uses exceptions both for error reporting and control flow (e.g. StopIteration).

[sabasehrish]

Subsystems

Required

1. Registration
2. Task management
3. Mock I/O
4. Data modeling

Nice to have

1. Algorithm description
2. I/O
3. Configuration
4. Plugin management
5. Logging

[knoepfel]

Related to:

• Create task manager for F1 demonstrators #33
• Create mock IO subsystem for F1 demonstrators #34
• Create registration subsystem for F1 demonstrators #36
• Create data modeling subsystem for F1 demonstrators #37

[knoepfel]

Responsible developers: @wlav and @marcpaterno

[knoepfel]

In case it's relevant: Intel's distribution for Python

[knoepfel]

This issue was satisfied with this demonstration. How Python algorithms interact with C++ algorithms within Phlex will continue to be explored in the F2 milestone (see Framework-R-D/phlex#70 and Framework-R-D/phlex#71).