feat: support Lambda Managed Instances (draft)

[alessandrobologna]

📬 Issue #: #1065

✍️ Description of changes:

Support Lambda Managed Instances / concurrent runtime (refs #1065)

Note

I originally built this to experiment with Lambda Managed Instances in a test workload and thought it might be a useful starting point for upstream support. Parts of the implementation were drafted with the help of an AI assistant and have only been exercised in my own workloads and the existing test suite so far, so please treat this as a starting point for discussion rather than a final design. A super simple demo/benchmark of this implementation is at https://github.com/alessandrobologna/rust-lmi-demo

This draft PR implements a concurrent runtime mode for Lambda Managed Instances, plus the API changes required in lambda-runtime and lambda-http. It preserves existing single-invocation behavior for classic Lambda. The implementation is based on the AWS guidance for building runtimes that support Lambda Managed Instances, as described in Building custom runtimes for Lambda Managed Instances.

Architecture (high level)

The runtime chooses between a classic "one invocation at a time" loop and a concurrent mode controlled by AWS_LAMBDA_MAX_CONCURRENCY:

• Sequential mode (default): a single /next long-poll loop fetches one event at a time and invokes the handler synchronously, preserving existing behavior.
• Concurrent mode (AWS_LAMBDA_MAX_CONCURRENCY >= 2): the runtime spawns AWS_LAMBDA_MAX_CONCURRENCY worker tasks. Each worker runs its own /next long-poll loop and processes invocations sequentially within that task, so overall concurrency is bounded by the number of workers. If a worker crashes, the runtime continues operating with the remaining workers and only terminates if it can't keep at least 1 worker alive.

sequenceDiagram
    participant Lambda as Lambda orchestrator
    participant Runtime as lambda-runtime
    participant Worker as Runtime worker task
    participant API as Runtime API
    participant Handler as User handler

    Lambda->>Runtime: Start process (env AWS_LAMBDA_MAX_CONCURRENCY = N)
    Runtime->>Worker: Spawn N worker tasks
    loop each worker (identical)
        loop each invocation
            Worker->>API: GET /runtime/invocation/next
            API-->>Worker: Invocation event + context headers
            Worker->>Handler: Call handler(event, context)
            Handler-->>Worker: Result or error
            Worker->>API: POST /response or /error
        end
    end

Loading

Breaking changes & compatibility

• Existing entrypoints (lambda_runtime::run, lambda_http::run, and lambda_http::run_with_streaming_response) keep their original signatures and sequential behavior. Handlers that compiled against the current release should continue to compile unchanged.
• New concurrent entrypoints are added:
  • lambda_runtime::run_concurrent
  • lambda_http::run_concurrent
  • lambda_http::run_with_streaming_response_concurrent
    These require handler services to implement Clone + Send + 'static (with responses/stream bodies Send/Sync + 'static) so they can be safely cloned and driven by the concurrent runtime.
• AWS_LAMBDA_MAX_CONCURRENCY is read directly by the runtime to decide concurrency and size the HTTP pool, without adding new public fields to Config (avoids semver breakage for struct literals).
• In concurrent mode (when the new entrypoints are used and AWS_LAMBDA_MAX_CONCURRENCY > 1), the runtime no longer sets _X_AMZN_TRACE_ID in the process environment. The per-invocation X-Ray trace ID is available via Context::xray_trace_id and tracing spans instead. Sequential mode behavior is unchanged.
• For the concurrent entrypoints, the behavior when AWS_LAMBDA_MAX_CONCURRENCY is set changes from "always sequential per environment" to "per-environment concurrency up to that value". Code that continues to call the existing run functions will remain strictly sequential even if the env var is set.

In other words: the earlier versions of this branch tightened the bounds on the existing run functions, but after maintainer feedback those entrypoints are left as-is and concurrency is opt-in via the new *_concurrent APIs.

Below is a concise summary of the changes (unfortunately many) by area.

Runtime & Config (lambda-runtime)

• Read AWS_LAMBDA_MAX_CONCURRENCY directly inside the runtime to decide whether concurrent mode should be enabled and to size the HTTP client pool, without extending Config.
• Runtime::new now sizes the lambda_runtime_api_client HTTP pool from max_concurrency so the number of idle connections matches expected concurrency.
• Runtime::run remains the original sequential /next loop via run_with_incoming, preserving existing behavior.
• New Runtime::run_concurrent spawns AWS_LAMBDA_MAX_CONCURRENCY worker tasks, each running its own /next loop. Sequential and concurrent modes share the same invocation processing helper, so there are not two separate event loop implementations to maintain. When Config::is_concurrent() is false, it falls back to the same sequential loop as Runtime::run.
• Worker failures are isolated: if a worker exits/panics, the runtime logs the failure and continues with fewer workers. The runtime terminates only if it can't keep at least 1 worker alive (i.e., all workers have exited).
• Unexpected clean worker exits are treated as infrastructure errors so the runtime returns an error once all workers are gone.
• Handler errors (user code returning Err) are reported to Lambda via /invocation/{id}/error and do not terminate the runtime.
• The existing graceful shutdown handler stays aligned with upstream. We do not attempt to preempt/coordinate in-flight invocations on SIGTERM (the expectation is that Lambda only shuts down environments when they are idle; local emulators may behave differently).
• Internal layers (api_client, api_response, trace) now implement/derive Clone so they can be composed into a cloneable service stack for the concurrent entrypoints.
• Context::new is more robust when lambda-runtime-client-context / lambda-runtime-cognito-identity headers are present but empty (treated as None instead of failing JSON parse).
• Added a regression test (concurrent_worker_crash_does_not_stop_other_workers) to verify worker isolation behavior.

HTTP & streaming (lambda-http, lambda-runtime-api-client)

• lambda_runtime_api_client::Client:
  • Added with_pool_size(usize) on the builder and threads a pool_size: Option<usize> into the Hyper client to set pool_max_idle_per_host.
  • Still works as before when pool_size is not provided.
• lambda_http::run and run_with_streaming_response keep their existing signatures and sequential behavior, delegating to lambda_runtime::run.
• New lambda_http::run_concurrent and lambda_http::run_with_streaming_response_concurrent wrap the same handler types but require them to be Clone + Send + 'static (with response/stream bounds aligned to lambda_runtime::run_concurrent) so they can be driven by the concurrent runtime.
• HTTP adapters (Adapter, StreamAdapter) are now Clone when the inner service is Clone, and the streaming path uses BoxCloneService internally for the concurrent entrypoint so the composed service stack can be cloned.
  • Note: run_with_streaming_response_concurrent is exported from lambda_http, similarly to run_with_streaming_response.

Tooling & examples

• Makefile and scripts/test-rie.sh:
  • Add RIE_MAX_CONCURRENCY and a test-rie-lmi target that runs RIE with AWS_LAMBDA_MAX_CONCURRENCY set, making it easy to exercise managed-instances behavior locally.
• examples/basic-lambda-concurrent:
  • New minimal example that calls lambda_runtime::run_concurrent so the concurrent code path can be exercised under RIE/LMI.
• examples/basic-lambda/src/main.rs:
  • Wraps lambda_runtime::run(func).await in an if let Err(err) block to log and propagate runtime errors when testing under RIE.

---

Validation

On this branch, I ran:

• cargo +nightly fmt --all -- --check
• cargo clippy --workspace --all-features -- -D warnings
• cargo +stable test --all-features -p lambda_runtime_api_client -p lambda_runtime -p lambda_http
• make test-rie EXAMPLE=basic-lambda
• make test-rie-lmi EXAMPLE=basic-lambda-concurrent (sets AWS_LAMBDA_MAX_CONCURRENCY inside the RIE container)

---

If maintainers prefer, this could be split into smaller PRs (e.g., builder/Config prep, handler Clone changes, and finally the concurrent runtime), but this branch shows the full "end-to-end" implementation so that it can be tested with Lambda Managed Instances.

---

🔏 By submitting this pull request

• I confirm that I've ran cargo +nightly fmt --all -- --check.
• I confirm that I've ran cargo clippy --workspace --all-features -- -D warnings.
• I confirm that I've made a best effort attempt to update all relevant documentation.
• I confirm that my contribution is made under the terms of the Apache 2.0 license.

[alessandrobologna]

Thanks @bnusunny for the approval!
Currently CI is failing on the semver check because Config has a new public field. I have a small follow‑up that keeps Config unchanged by moving the concurrency limit into Runtime (derived from AWS_LAMBDA_MAX_CONCURRENCY), which avoids a major version bump.

Patch: alessandrobologna@a8af68a

If you’re OK with that approach, I can push it directly to this PR branch so the checks go green.

[bnusunny]

Yes, please.

[alessandrobologna]

Ok, I've pushed the update to keep concurrency settings internal and to avoid the semver bump (so no Config public field needed). I also rebased the branch on current main.

[jlizen]

Overall implementation looks great!

To summarize my feedback:

• I think naming things based on _concurrent is overly limiting limiting if we expect future features to require Clone, i'd suggest _extended instead maybe? And we should probably nudge callers to use _extended, _concurrent, _maybe_concurrent, etc, unless they specifically need !Clone bounds?
• The non-*_concurrent should have doc comments warning that it ignores concurrency variables and probably an error log or panic if the caller tries to invoke them with concurrency ENV set
• I'd like to make it easier to trace per-worker issues by injecting task IDs into both worker loop spans and worker loop results
• I'd prefer to preserve all errors across workers rather than only the first. Also it seems like we don't really need the full-batteries FuturesUnordered instead of join_all?
• I don't like the Oncelock that allows showing a single trace message, it seems like it could be simplified?

[jlizen]

Minor: is this makefile target + configurable variable worth mentioning in the README?

[jlizen]

(Question, not sure): If the fallback case if AWS_LAMBDA_MAX_CONCURRENCY if unset is just lambda_runtime::run(), should this be our new suggested entrypoint, with fn run() marked as deprecated? I guess the main issue would be the Clone bounds are more restrictive, so maybe deprecation is a bit aggressive, but we could add that in the run() doc comments?

The regular run() should definitely also add a warning that it WON'T respect AWS_LAMBDA_MAX_CONCURRENCY.

Context: My suspicion is, most consumers don't mind the Clone bound, and the 'principal of least surprise' would be for it to respect the ENV. So we probably should be pointing people to prefer this method where possible.

I would suggest at minimum the following:

• probably this should be named run_maybe_concurrent? Or maybe run_extended to avoid a combinatorial explosion of new entrypoint names as featureset grows that needs Clone?
• We should have a guard clause in lambda_runtime::run() that throws either either crashes the lambda runtime (preferred), or emits error logs (more conservative), if AWS_LAMBDA_MAX_CONCURRENCY is set, since that is clearly a misconfiguration that would prove confusing to debug.

[jlizen]

Minor: What are the cases where http::HeaderValue::from_str() would fail? Should we have a warn or error-level log line if it does?

(I guess this is an existing behavior and this is more of a lift and shift?)

[jlizen]

Non-blocking: Do we actually need the extra allocation + indirection of BoxCloneServiceHere? Anyway we aren't leaking into the public API any hideous generic output type from this, so we could probably get by with pure generics?

IE, we should be able to use similar bounds with MapRequest and MapResponse to into_stream_service, no? Just add the Clone bound and call it into_stream_service_cloneable?

Non-blocking since it's a fairly minor perf optimization, and we can change it in the future without semver impact.

[jlizen]

Thanks for the fixup here

[jlizen]

It seems like THIS is the place where we want to capture whether ALL workers exited cleanly (though that would be a bit tricky to capture in current impl).

The nicer way, might be to model the collected errors as:

enum WorkerError {
    CleanExit(Id),
    Failure((Id, BoxError))
}

And that way we could flatten down to into either a BoxError representing "all workers exited cleanly" versus "these workers exited cleanly, and these ones had these errors".

[jlizen]

In the current impl, None should essentially never be reachable, right? Since even if we have no task-specific errors, we instead create an error out of a clean exit?

If the desired API is to return Ok(()) on all workers cleanly exiting, then we could use the enum mentioned above to gather that info, and emit an error log line if all exited cleanly, but then drop the error.

[jlizen]

Would it be worth instrumenting this future with a span containing the taskId? Seems pretty useful for debug

[jlizen]

Nit: I don't love the extra branching here since this is pure overhead for the non-concurrent case. But, I guess branch prediction will probably make this pointless to optimize. Don't think it is worth a bunch of code duplication for.

[jlizen]

(No action needed here, unless you feel particularly inspired, fine to resolve this)

[jlizen]

The OnceLock + get_or_init every invocation seems like unnecessary overhead / complexity. Why can't we just display this message once, at the start of run_concurrent?