Management API Catalog Creation Fails with ContextNotActiveException (Quarkus CDI) in Polaris 1.1.0, 1.2.0 and 1.3.0 on K8

[systemalchemy]

Describe the bug

Hi Polaris team, can you help? Is there something basic I am missing or is this a known bug with a work-around recommended. Thanks in advance!

Polaris Catalog Issue Report

1. What version of Apache Polaris are you using?

• Version: polaris-bin-1.1.0-incubating (image assembled from release archive) (but same issue occurred when using 1.2.0 and 1.3.0.
• CLI binaries present: /opt/polaris/bin/admin and /opt/polaris/bin/server are present; /opt/polaris/bin/polaris CLI binary is not present in the image.

2. What operating system and processor architecture are you using?

• Environment: Kubernetes (Rancher Desktop)
• Namespace: flink-sandbox
• Operating System: (Host OS not specified; running in containerized environment)
• Processor Architecture: (Not specified; typical x86_64 for containers)

3. What did you do?

• Integrated Polaris with Flink and MinIO on Kubernetes.
• Set up authentication using INTERNAL/default mode, token broker rsa-key-pair, and token refresh via CronJob.
• Attempted to create a catalog using the management API:
  1. Obtained admin token via /api/catalog/v1/oauth/tokens (scope=PRINCIPAL_ROLE:admin, grant_type=client_credentials).
  2. Called POST /api/management/v1/catalogs on port 8181 with valid token and (optionally) X-Polaris-Realm: default-realm.
  3. Example request body:

     {
       "catalog": {
         "name": "warehouse",
         "type": "INTERNAL",
         "properties": {
           "default-base-location": "s3a://warehouse/iceberg"
         },
         "storageConfigInfo": {
           "storageType": "S3",
           "allowedLocations": ["s3a://warehouse"]
         }
       }
     }

4. What did you expect to see?

• Expected HTTP 201 response with catalog information.
• Expected successful end-to-end Flink ↔ Iceberg integration and passing integration tests.

5. What did you see instead?

• Received HTTP 500 error with ContextNotActiveException (Quarkus CDI) in server logs.
• Unable to create a catalog; integration tests blocked.
• Service exposes only port 8181; 8182 (health/management) is not exposed externally.
• CLI fallback unavailable (/opt/polaris/bin/polaris missing; admin tool lacks catalog commands).

---

Additional Details

Environment

• Auth mode: INTERNAL/default, token broker rsa-key-pair, max token TTL 1h.
• Secrets: RSA keys from polaris-token-broker-keys; admin credentials from polaris-admin-credentials.
• Token refresh: CronJob posts to /api/catalog/v1/oauth/tokens and writes access token to polaris-token secret (not mounted into Polaris container).
• Storage/DB: Postgres backend; MinIO as object store (s3a).

Issue 1: Auth Behavior (resolved locally)

• Flink requests to Polaris REST catalog returned 401 even with quarkus.oidc.enabled=false and POLARIS_AUTHENTICATION_ENABLED=false.
• Workaround: Added env-based bearer token support in catalog manager (pyflink/catalog/polaris_manager.py#L121-L137). Token injected into Flink JM/TM as env POLARIS_BEARER_TOKEN.
• Result: Auth succeeded; subsequent failure was a 400 “Please specify a warehouse,” confirming token acceptance.

Issue 2: Management API Catalog Create Fails (blocking)

• Symptom: POST /api/management/v1/catalogs returns HTTP 500 with ContextNotActiveException (Quarkus CDI).
• Impact: Cannot create a catalog; Flink ↔ Iceberg integration tests blocked.

Hypothesis

• The error appears to be a server-side request context activation gap (e.g., missing @ActivateRequestContext or scope on a producer used by management/warehouse resources), based on the observed Quarkus CDI exception and referenced documentation.

Requests for Community Guidance

1. Known issues in 1.1.0/1.3.0 where management APIs throw ContextNotActiveException? Any patches or backports?
2. Is there an official Polaris CLI binary for catalog operations that should ship in the binary distro?
3. Should the management API be called on 8182 instead of 8181? Expected port exposure and required headers?
4. Recommended workaround until a fix: is it safe to insert @ActivateRequestContext around management endpoints, or should the producer scope be adjusted elsewhere?

Next-Step Options Considered

• Patch Polaris source (prefer 1.3.0) to wrap management/warehouse resources with request context and rebuild image (if source access is available).
• Expose 8182 and retest (to confirm if the error persists or to aid diagnosis).

Artifacts and References

• Auth patch: kubernetes/overlays/local/polaris-auth-patch.yaml
• Token refresh CronJob: kubernetes/overlays/local/polaris-token-refresh-cronjob.yaml
• Deployment: kubernetes/base/polaris/deployment.yaml
• Service: kubernetes/base/polaris/service.yaml
• CLI binary absence confirmed via kubectl exec deploy/polaris -- ls /opt/polaris/bin (only admin, server present).

Open Items

• Decode and share live token claims for validation (can provide if helpful).
• Confirm whether realm header is required beyond bearer token for management calls.

To Reproduce

No response

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response

[dimas-b]

@systemalchemy : do you have a stack trace for the 500 error in Polaris Server logs?

[systemalchemy]

Hi Dmitri, thanks for your response. Here is the stack trace.

Actual Server Stack Trace

2026-01-21 19:37:45,261 ERROR [io.qua.ver.htt.run.sec.AbstractHttpAuthorizer] [,] [,,,] (vert.x-eventloop-thread-1) Exception occurred during authorization:
jakarta.enterprise.context.ContextNotActiveException: RequestScoped context was not active when trying to obtain a bean instance for a client proxy of PRODUCER_METHOD bean [class=org.apache.polaris.service.config.ServiceProducers, id=qqpjKw_IGybuFcj0RdqnWquPB6g]
	- you can activate the request context for a specific method using the @ActivateRequestContext interceptor binding
	at io.quarkus.arc.impl.ClientProxies.notActive(ClientProxies.java:76)
	at io.quarkus.arc.impl.ClientProxies.getSingleContextDelegate(ClientProxies.java:32)
	at org.apache.polaris.service.auth.ServiceProducers_ProducerMethod_realmAuthConfig_tFNnkziPu0lX3VErYmDto1gEwfU_ClientProxy.arc$delegate(Unknown Source)
	at org.apache.polaris.service.auth.ServiceProducers_ProducerMethod_realmAuthConfig_tFNnkziPu0lX3VErYmDto1gEwfU_ClientProxy.type(Unknown Source)
	at org.apache.polaris.service.auth.internal.InternalAuthenticationMechanism.authenticate(InternalAuthenticationMechanism.java:77)
	at org.apache.polaris.service.auth.internal.InternalAuthenticationMechanism_ClientProxy.authenticate(Unknown Source)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator.createSecurityIdentity(HttpAuthenticator.java:197)
	at io.quarkus.vertx.http.runtime.security.HttpAuthenticator.attemptAuthentication(HttpAuthenticator.java:131)
	at io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$AuthenticationHandler.lambda$handle$0(HttpSecurityRecorder.java:440)
	at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:68)
	... (additional Mutiny/Vert.x stack frames)

Summary
The stack trace confirms that the ContextNotActiveException occurs in the authentication phase, specifically when trying to access ServiceProducers.realmAuthConfig() during the InternalAuthenticationMechanism.authenticate() call.

Key findings:

The error happens at InternalAuthenticationMechanism.authenticate(InternalAuthenticationMechanism.java:77)
It's trying to access a @RequestScoped bean (ServiceProducers.realmAuthConfig) but the request context is not active
This occurs during HTTP authorization, before the management API endpoint code is even reached
This seems to confirm the hypothesis that the issue is a missing @ActivateRequestContext annotation on the authentication mechanism or the producer method.

I look forward to your expert feedback - thanks again!

[dimas-b]

@systemalchemy : would you be able to share the full stack trace, please?

update: no need for full stack trace anymore 😉

[dimas-b]

Something does not look quite right 🤔 I'm not able to reproduce the same problem, but in my env. I'm seeing the same request context beans being used inside the ServiceProducers_ProducerMethod_realmAuthConfig_AAAA_ClientProxy.type() method in different REST API calls... ⏳

Update: it was my mistake in my local env. CDI appears to work fine.

[dimas-b]

@systemalchemy : would you mind sharing your full Polaris log that contains ContextNotActiveException?