Getting Apache Polaris to work with Nutanix Objects + Airbyte

[krutoileshii]

I have polaris up and running in generate (standalone). However, I can't seem to figure you how to get it to work with the Airbytes S3 Data Lake destination. It seems like it keeps trying to use the STS which is not available in my case.

I would appreciate any help in getting this figured out.

I am using Nutanix Objects which doesn't have an STS endpoint as far as i can find.
Certificates are already imported for S3 and also using the HTTP endpoints instead of HTTPS for the time being.

Polaris Version: 1.2.0 Standalone

Settings in applicaiton.properties:

polaris.realm-context.realms=POLARIS
quarkus.log.console.level=DEBUG
quarkus.log.category."io.smallrye.config".level=DEBUG
polaris.realm-context.header-name=Polaris-Realm
#polaris.realm-context.require-header=true
polaris.persistence.type=relational-jdbc
quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=<username>
quarkus.datasource.password=<password>
quarkus.datasource.jdbc.url=jdbc:postgresql://<url>/polaris_test
polaris.authentication.token-broker.type=rsa-key-pair
polaris.authentication.token-broker.rsa-key-pair.public-key-file=/opt/polaris/public.key
polaris.authentication.token-broker.rsa-key-pair.private-key-file=/opt/polaris/private.key
polaris.authentication.token-broker.max-token-generation=PT1H
polaris.bootstrap.credentials=POLARIS,admin,secret
polaris.storage.aws.access-key=access_key
polaris.storage.aws.secret-key=secret_key
aws.accessKeyId=access_key
aws.secretAccessKey=secret_key
polaris.catalog.s3.no-sts=true
polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": false 
polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES": ["S3"]

2025-12-22 17:26:42,181 WARN  [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) Refer to https://polaris.apache.org/in-dev/unreleased/configuring-polaris-for-production for more information.
2025-12-22 17:26:42,262 INFO  [io.quarkus] [,] [,,,] (main) Apache Polaris Server (incubating) 1.2.0-incubating on JVM (powered by Quarkus 3.28.2) started in 4.542s. Listening on: http://0.0.0.0:8181. Management interface listening on http://0.0.0.0:8182.
2025-12-22 17:26:42,263 INFO  [io.quarkus] [,] [,,,] (main) Profile prod activated.
2025-12-22 17:26:42,263 INFO  [io.quarkus] [,] [,,,] (main) Installed features: [agroal, amazon-sdk-rds, cdi, hibernate-validator, jdbc-postgresql, micrometer, narayana-jta, oidc, opentelemetry, reactive-routes, rest, rest-jackson, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, vertx]
2025-12-22 17:27:00,378 DEBUG [io.sma.config] [,] [,,,] (vert.x-eventloop-thread-1) SRCFG01001: The config quarkus.security.events.enabled was loaded from DefaultValuesConfigSource with the value true
2025-12-22 17:27:00,397 DEBUG [io.sma.config] [,] [,,,] (vert.x-eventloop-thread-1) SRCFG01001: The config quarkus.security.events.enabled was loaded from DefaultValuesConfigSource with the value true
2025-12-22 17:27:00,421 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (vert.x-eventloop-thread-1) SRCFG01001: The config quarkus.otel.sdk.disabled was loaded from PropertiesConfigSource[source=jar:file:///opt/polaris/polaris-latest/server/lib/main/org.apache.polaris.polaris-runtime-defaults-1.2.0-incubating.jar!/application.properties]:95 with the value true
2025-12-22 17:27:00,442 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config quarkus.fault-tolerance.mp-compatibility was not found
2025-12-22 17:27:00,442 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01001: The config smallrye.faulttolerance.mp-compatibility was loaded from DefaultValuesConfigSource with the value false
2025-12-22 17:27:00,443 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config quarkus.fault-tolerance.enabled was not found
2025-12-22 17:27:00,443 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config smallrye.faulttolerance.enabled was not found
2025-12-22 17:27:00,444 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config MP_Fault_Tolerance_NonFallback_Enabled was not found
2025-12-22 17:27:00,444 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config quarkus.fault-tolerance.metrics.enabled was not found
2025-12-22 17:27:00,444 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config smallrye.faulttolerance.metrics.enabled was not found
2025-12-22 17:27:00,444 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config MP_Fault_Tolerance_Metrics_Enabled was not found
2025-12-22 17:27:00,444 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config quarkus.fault-tolerance.micrometer.disabled was not found
2025-12-22 17:27:00,444 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01002: The config smallrye.faulttolerance.micrometer.disabled was not found
2025-12-22 17:27:00,480 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) SRCFG01001: The config quarkus.http.limits.max-body-size was loaded from PropertiesConfigSource[source=jar:file:///opt/polaris/polaris-latest/server/lib/main/org.apache.polaris.polaris-runtime-defaults-1.2.0-incubating.jar!/application.properties]:46 with the value 10240K
2025-12-22 17:27:00,496 INFO  [org.apa.pol.ser.con.PolarisIcebergObjectMapperCustomizer] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) Limiting request body size to 10485760 bytes
2025-12-22 17:27:00,690 INFO  [io.qua.htt.access-log] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000001,POLARIS] [,,,] (executor-thread-1) 10.30.3.58 - - [22/Dec/2025:17:27:00 +0000] "POST /api/catalog/v1/oauth/tokens HTTP/1.1" 200 782
2025-12-22 17:27:00,796 DEBUG [io.sma.config] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000002,POLARIS] [,,,] (executor-thread-2) SRCFG01001: The config quarkus.security.events.enabled was loaded from DefaultValuesConfigSource with the value true
2025-12-22 17:27:00,870 INFO  [io.qua.htt.access-log] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000002,POLARIS] [,,,] (executor-thread-2) 10.30.3.58 - root [22/Dec/2025:17:27:00 +0000] "GET /api/catalog/v1/config?warehouse=data-warehouse-test HTTP/1.1" 200 2148
2025-12-22 17:27:01,030 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000003,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
2025-12-22 17:27:01,130 INFO  [io.qua.htt.access-log] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000003,POLARIS] [,,,] (executor-thread-1) 10.30.3.58 - root [22/Dec/2025:17:27:01 +0000] "GET /api/catalog/v1/data-warehouse-test/namespaces/raw HTTP/1.1" 200 69
2025-12-22 17:27:01,202 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000004,POLARIS] [,,,] (executor-thread-1) Handling runtimeException Table does not exist: raw.airbyte_test_table_33dc07a8_52ad_4164_b052_38bdb798ee10
2025-12-22 17:27:01,232 INFO  [io.qua.htt.access-log] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000004,POLARIS] [,,,] (executor-thread-1) 10.30.3.58 - root [22/Dec/2025:17:27:01 +0000] "GET /api/catalog/v1/data-warehouse-test/namespaces/raw/tables/airbyte_test_table_33dc07a8_52ad_4164_b052_38bdb798ee10?snapshots=all HTTP/1.1" 404 146
2025-12-22 17:27:01,314 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000005,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
2025-12-22 17:27:01,328 INFO  [org.apa.ice.BaseMetastoreCatalog] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000005,POLARIS] [,,,] (executor-thread-1) Table properties set at catalog level through catalog properties: {}
2025-12-22 17:27:01,334 INFO  [org.apa.ice.BaseMetastoreCatalog] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000005,POLARIS] [,,,] (executor-thread-1) Table properties enforced at catalog level through catalog properties: {}
2025-12-22 17:27:01,407 WARN  [org.apa.pol.ser.sto.StorageConfiguration] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000005,POLARIS] [,,,] (executor-thread-1) Using hard-coded AWS credentials - this is not recommended for production
2025-12-22 17:27:01,950 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000005,POLARIS] [,,,] (executor-thread-1) Handling runtimeException Failed to get subscoped credentials: (Service: Sts, Status Code: 501, Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+310000+119816+187079+19780, Extended Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+310000+119816+187079+19780) (SDK Attempt Count: 1)
2025-12-22 17:27:01,951 INFO  [io.qua.htt.access-log] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000005,POLARIS] [,,,] (executor-thread-1) 10.30.3.58 - root [22/Dec/2025:17:27:01 +0000] "POST /api/catalog/v1/data-warehouse-test/namespaces/raw/tables HTTP/1.1" 422 328

[dimas-b]

Please see this example for Apache Ozone: https://polaris.apache.org/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-ozone/

A similar approach should work with any S3-compatible storage without STS.

[krutoileshii]

@dimas-b - Awesome that was it for that part. Now have a sigv4 error not sure if it's coming from Airbyte or Polaris though. Any ideas on what it could be?

ables/airbyte_test_table_1bd7fe4c_f88b_49e9_ba90_f683e5b53062?snapshots=all HTTP/1.1" 404 146
2025-12-22 17:57:14,047 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000025,POLARIS] [,,,] (executor-thread-1) Initializing non-federated catalog
2025-12-22 17:57:14,051 INFO  [org.apa.ice.BaseMetastoreCatalog] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000025,POLARIS] [,,,] (executor-thread-1) Table properties set at catalog level through catalog properties: {}
2025-12-22 17:57:14,054 INFO  [org.apa.ice.BaseMetastoreCatalog] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000025,POLARIS] [,,,] (executor-thread-1) Table properties enforced at catalog level through catalog properties: {}
2025-12-22 17:57:14,059 INFO  [org.apa.ice.CatalogUtil] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000025,POLARIS] [,,,] (executor-thread-1) Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
2025-12-22 17:57:14,079 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000025,POLARIS] [,,,] (executor-thread-1) Handling runtimeException Invalid streaming signature v4 headers. (Service: S3, Status Code: 400, Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+110000+117605+89935+19922, Extended Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+110000+117605+89935+19922) (SDK Attempt Count: 1)
2025-12-22 17:57:14,080 INFO  [io.qua.htt.access-log] [7ff494ca-32f6-43b1-b355-6dd3aa51e5b3_0000000000000000025,POLARIS] [,,,] (executor-thread-1) 10.30.3.58 - root [22/Dec/2025:17:57:14 +0000] "POST /api/catalog/v1/data-warehouse-test/namespaces/raw/tables HTTP/1.1" 400 323

Here is the stacktrace as well from airbyte:

2025-12-22 11:48:37  WARN 2025-12-22 11:48:37 warn WARN main o.a.i.r.RESTSessionCatalog(initialize):230 Iceberg REST client is missing the OAuth2 server URI configuration and defaults to http://10.30.3.84:8181/api/catalogv1/oauth/tokens. This automatic fallback will be removed in a future Iceberg release.It is recommended to configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be prepared. This warning will disappear if the OAuth2 endpoint is explicitly configured. See https://github.com/apache/iceberg/issues/10537
2025-12-22 11:48:37  INFO 2025-12-22 11:48:37 info INFO main o.a.i.CatalogUtil(loadFileIO):346 Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
2025-12-22 11:48:37  INFO 2025-12-22 11:48:37 info INFO main i.a.c.l.t.i.p.i.IcebergUtil(createTable):138 Creating Iceberg table 'raw.airbyte_test_table_c24d85fe_f5f9_4d9f_866e_5a6f5b4bce6d'....
2025-12-22 11:48:37  WARN 2025-12-22 11:48:37 warn WARN main i.a.c.l.c.CheckOperation(execute):58 Caught throwable during CHECK org.apache.iceberg.exceptions.BadRequestException: Malformed request: Invalid streaming signature v4 headers. (Service: S3, Status Code: 400, Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+210000+119814+97473+20173, Extended Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+210000+119814+97473+20173) (SDK Attempt Count: 1)
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:208) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:118) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:102) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:368) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:112) ~[iceberg-core-1.7.0.jar:?]
	at org.apache.iceberg.rest.RESTSessionCatalog$Builder.create(RESTSessionCatalog.java:804) ~[iceberg-core-1.7.0.jar:?]
	at io.airbyte.cdk.load.toolkits.iceberg.parquet.io.IcebergUtil.createTable(IcebergUtil.kt:143) ~[bulk-cdk-toolkit-load-iceberg-parquet-0.1.69.jar:?]
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeChecker.catalogValidation(S3DataLakeChecker.kt:87) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-data-lake.jar:?]
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeChecker.check(S3DataLakeChecker.kt:45) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-data-lake.jar:?]
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeChecker.check(S3DataLakeChecker.kt:36) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-data-lake.jar:?]
	at io.airbyte.cdk.load.check.CheckOperation.execute(CheckOperation.kt:48) [bulk-cdk-core-load-0.1.69.jar:?]
	at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36) [bulk-cdk-core-base-0.1.69.jar:?]
	at picocli.CommandLine.executeUserObject(CommandLine.java:2030) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine.access$1500(CommandLine.java:148) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) [picocli-4.7.6.jar:4.7.6]
	at picocli.CommandLine.execute(CommandLine.java:2174) [picocli-4.7.6.jar:4.7.6]
	at io.airbyte.cdk.AirbyteDestinationRunner$Companion.run(AirbyteConnectorRunner.kt:289) [bulk-cdk-core-base-0.1.69.jar:?]
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeDestination.main(S3DataLakeDestination.kt:15) [io.airbyte.airbyte-integrations.connectors-destination-s3-data-lake.jar:?]

Stack Trace: org.apache.iceberg.exceptions.BadRequestException: Malformed request: Invalid streaming signature v4 headers. (Service: S3, Status Code: 400, Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+210000+119814+97473+20173, Extended Request ID: 884970bf-ee4e-4aad-6fc5-b8ab76ea1d09+210000+119814+97473+20173) (SDK Attempt Count: 1)
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:208)
	at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:118)
	at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:102)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262)
	at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:368)
	at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:112)
	at org.apache.iceberg.rest.RESTSessionCatalog$Builder.create(RESTSessionCatalog.java:804)
	at io.airbyte.cdk.load.toolkits.iceberg.parquet.io.IcebergUtil.createTable(IcebergUtil.kt:143)
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeChecker.catalogValidation(S3DataLakeChecker.kt:87)
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeChecker.check(S3DataLakeChecker.kt:45)
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeChecker.check(S3DataLakeChecker.kt:36)
	at io.airbyte.cdk.load.check.CheckOperation.execute(CheckOperation.kt:48)
	at io.airbyte.cdk.AirbyteConnectorRunnable.run(AirbyteConnectorRunnable.kt:36)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2030)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
	at picocli.CommandLine.execute(CommandLine.java:2174)
	at io.airbyte.cdk.AirbyteDestinationRunner$Companion.run(AirbyteConnectorRunner.kt:289)
	at io.airbyte.integrations.destination.s3_data_lake.S3DataLakeDestination.main(S3DataLakeDestination.kt:15)

[dimas-b]

@krutoileshii : you may need to enable DEBUG logging in Polaris to get the stack trace for the original Invalid streaming signature v4 headers exception.

That said, your aws.secretAccessKey property does not look correct to me. It should probably be polaris.storage.aws.secret-key (and polaris.storage.aws.access-key for the key ID).

https://github.com/apache/polaris/blob/apache-polaris-1.2.0-incubating/runtime/service/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java#L57