feat: Add WASM-based python processor #621

gregfurman · 2025-12-18T21:19:55Z

Motivation

Includes a new python processor that runs within a sandboxed WASM environment.

Changes

Adds a new python processor that runs within a sandboxed WASM environment.
Adds a WasmModulePool in internal/impl/wasm/wasmpool/pool.go that uses a Golang sync.Pool to execute multiple concurrent guests and have their lifecycles managed by the GC. This allows us to spin up more instances as demand increases and have the Go GC handle the guest lifecycle.
Adds an entrypoint.py that allows python execution script to compile and execute -- stdout and stdin is used to pass data between the guest and host.
If the x_wasm tag is used, the WASM executable and corresponding entrypoint.py are embedded at compile time. Using the processor without this tag panics on init.
Adds a script at scripts/install.sh that downloads the python-3.12.0.wasm and checks it against the SHA in runtime/python-3.12.0.wasm.sha256sum.

Installation/Usage

Run the following commands to create a config, install the python runtime (and check it against the SHA), and execute a pipeline.

cat > python.yaml <<'EOF'
pipeline:
  processors:
    - python:
        script: |
          root = sum(this)
EOF

bash internal/impl/python/scripts/install.sh

echo "[11, 6, 20, 5]" | go run -tags "x_wasm" cmd/bento/main.go -c python.yaml --set logger.level=warn

The output of the pipeline should be 42.

TODO

Extend the CI (or Makefile) to run the downloader script and build with the x_wasm tag -- allowing the .wasm to be embedded at compile time and used in the processor.

aronchick · 2025-12-20T16:39:06Z

this is going to be really cool! lmk if you need help

gregfurman · 2025-12-21T11:51:32Z

Thanks @aronchick! If you have time (and are feeling extra generous), I'd appreciate if you could run through the Installation/Usage steps locally and try out different configurations. I'm also looking for any feedback on usability since we're planning to expand our WASM offering, so any insights would be super helpful!

I haven't dug too deep into the limitations of the sandboxed runtime and how it responds to accessing non-permissive resources (e.g., filesystem or OS).

In addition, we'd probably need to soak test the processor before making this GA in Bento since I'm unsure of performance/behavior over longer periods and under differing load profiles.

gregfurman

@jem-davies I've added a quick self-review. Let me know if you think we should be adding more documentation with what's been mentioned here.

gregfurman · 2025-12-21T11:56:18Z

internal/impl/python/scripts/entrypoint.py

+            namespace = {"this": input_data, "root": None, "__builtins__": __builtins__, **injected_libs}
+            exec(compiled_code, namespace)


this namespace and exec combintation allows us to dynamically run our compiled execution script, constraining what is accessible during runtime and ensuring no global state is ever mutated.

Also, see how we pass this and root -- leveraging the same pattern we use with bloblang scripts/mappings.

gregfurman · 2025-12-21T11:57:02Z

internal/impl/python/scripts/entrypoint.py

+        name = lib_name.strip()
+        if name:
+            try:
+                injected[name] = importlib.import_module(name)


Some python magic that allows us to dynamically extract which libraries we're making permissable to the sandboxed execution.

gregfurman · 2025-12-21T11:58:47Z

internal/impl/python/scripts/install.sh

+
+if command -v sha256sum >/dev/null; then
+    (cd "$WASM_RUNTIME_DIR/runtime" && sha256sum --check --status python-3.12.0.wasm.sha256sum)
+elif command -v shasum >/dev/null; then


fallback to shasum since sha256sum is not available on darwin

gregfurman · 2025-12-21T12:03:33Z

internal/impl/python/scripts/install.sh

+wget -q "$WASM_BINARY_URL" -O "$WASM_PATH"
+
+if command -v sha256sum >/dev/null; then
+    (cd "$WASM_RUNTIME_DIR/runtime" && sha256sum --check --status python-3.12.0.wasm.sha256sum)


Since we're downloading and embedding a WASM executable into bento, we need to verify the integrity of the executable we're intending to use. So we download and commit the SHA256 checksum, then verify the downloaded file's checksum.

gregfurman · 2025-12-21T12:05:12Z

internal/impl/wasm/wasmpool/pool.go

+	instancePool sync.Pool
+}
+
+func NewWasmModulePool[T api.Module](ctx context.Context, ctor constructor[T]) (*WasmModulePool[T], error) {


Unfortunately WASM modules (at least with wazero) are not safe for concurrent executions. Instead, we create a module pool (backed by a sync.Pool) and allow the GC to dynamically spin up instances depending on load.

gregfurman · 2025-12-21T12:08:03Z

internal/impl/wasm/wasmpool/pool.go

+}
+
+func (p *WasmModulePool[T]) Put(instance T) {
+	_ = runtime.AddCleanup(&instance, func(inst T) {


If the GC cleans up the instance, which is possible given sync.Pool, we need to ensure the resources are actually freed up. Hence, this gross runtime.AddCleanup hook that triggers when the module is GC'd

gregfurman · 2025-12-21T12:11:18Z

internal/impl/wasm/wasmpool/pool.go

+	p.instancePool.Put(instance)
+}
+
+func (p *WasmModulePool[T]) Close(ctx context.Context) error {


While it's probably easier to just close the entire runtime to ensure all associated modules (and resources) are cleaned up, this Close will retrieve all objects from the pool and close them.

While thinking of this as a Reset or Cleanup conceptually more correct, Close feels more idiomatic 🤷

gregfurman · 2025-12-21T12:15:04Z

internal/impl/python/instance.go

+	}()
+
+	handshakeErr := make(chan error, 1)
+	go func() {


We do a quick-and-dirty handshake to ensure the python env is up-and-running. The python module should write a "READY" signal to the output socket which we need to check is received before proceeding.

gregfurman · 2025-12-21T12:16:24Z

internal/impl/python/instance.go

+		return nil, pi.stderrBuf.Bytes(), err
+	}
+
+	header := make([]byte, 5)


First byte signals that the response was a success or exception. Last four bytes tell us how long the response is.

aronchick · 2025-12-22T00:23:15Z

Testing Instructions

I've tested this locally and it works great! Here's how others can try it:

Quick Start

# 1. Checkout the PR
gh pr checkout 621

# 2. Download the Python WASM runtime (~50MB) and test configs
bash internal/impl/python/scripts/install.sh
gh gist clone https://gist.github.com/aronchick/5d9391b03f9266b994c7a36043d743a4 python-examples

# 3. Run an example (first build takes ~30-60s)
go run -tags "x_wasm" ./cmd/bento/main.go -c python-examples/hello.yaml

Expected output: "Hello, World!"

Example Configs

Full test configs available in the gist: <GIST_URL>

File	Description
`hello.yaml`	Simple greeting
`test1.yaml`	JSON transformation
`test2.yaml`	Array processing (sum, count, double)
`test3.yaml`	Message filtering
`test4.yaml`	Stdlib imports (math, json, re)
`test5.yaml`	E-commerce order transformation
`test6.yaml`	Error handling demo

Quick Snippets

Array processing:

- python:
    script: |
      root = {"sum": sum(this), "doubled": [x * 2 for x in this]}

With imports:

- python:
    imports: [math, json, re]
    script: |
      root = {"ceil": math.ceil(this["value"])}

Filtering (set root=None to drop):

- python:
    script: |
      root = this if this.get("status") == "active" else None

Run Unit Tests

go test -tags "x_wasm" -v ./internal/impl/python/...

All 18 tests pass (~18 seconds).

Key Points

Build tag required: Always use -tags "x_wasm"
Cold start: ~1 second per Python instance
Familiar syntax: Uses this and root like Bloblang
Sandboxed: Runs in WASM, no filesystem/network access

gregfurman self-assigned this Dec 18, 2025

gregfurman force-pushed the add/processor/python branch from 5c82ab2 to 38217e5 Compare December 18, 2025 21:23

gregfurman force-pushed the add/processor/python branch from 38217e5 to b0dd215 Compare December 20, 2025 17:50

feat: Add WASM-based python processor

f212bc8

gregfurman force-pushed the add/processor/python branch from b0dd215 to f212bc8 Compare December 20, 2025 17:59

gregfurman marked this pull request as ready for review December 20, 2025 21:11

gregfurman requested a review from jem-davies as a code owner December 20, 2025 21:11

gregfurman commented Dec 21, 2025

View reviewed changes

		namespace = {"this": input_data, "root": None, "__builtins__": __builtins__, **injected_libs}
		exec(compiled_code, namespace)

feat: Add WASM-based python processor #621

Are you sure you want to change the base?

feat: Add WASM-based python processor #621

Uh oh!

Conversation

gregfurman commented Dec 18, 2025

Motivation

Changes

Installation/Usage

TODO

Uh oh!

aronchick commented Dec 20, 2025

Uh oh!

gregfurman commented Dec 21, 2025

Uh oh!

gregfurman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aronchick commented Dec 22, 2025

Testing Instructions

Quick Start

Example Configs

Quick Snippets

Run Unit Tests

Key Points

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants