Redesign inter-Payload interface #29
Description
Currently payloads are executed as standalone processes with ZMQ pipes for IPC. This is bad for a number of reasons:
- For large projects, it can lead to a massive number of child processes being spawned
- Each project must be compiled individually, which is incredibly slow (assuming not interpreted)
- Data exchange is very poor and prevents programmers from exposing rich APIs to other payloads
Any new design must allow programmers to include same-language payloads as libraries instead of processes.
Challenges:
- Exchanging data across language barriers is complex
- Most developers will not want to write FFI code themselves for interop
- Ideally interpreted payloads should stay within the same process to allow shared memory, which would require hooks into the interpreter itself (e.g. reimplementing PHP's SAPI so that it can be called as a library from Rust)