Polaris attempts to resolve internal S3 host via the internet — cannot connect to internal S3 without STS / Credential Vending

[tuanit03]

Describe the bug

I am using Polaris to connect to an internal S3 service. The internal S3 does not provide Credential Vending / STS. I tried to configure Polaris so it can connect to the internal S3 without STS, but with the current setup Polaris still tries to access the internet and resolves the internal S3 domain externally.

Below are the exact configurations I used (Docker Compose, Spark image Dockerfile, catalog payload and Spark configuration).

---

Docker Compose (relevant parts)

services:
  postgres:
    image: postgres:17
    container_name: polaris-postgres
    ports:
      - "5433:5432"
    shm_size: 128mb
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: polaris
      POSTGRES_INITDB_ARGS: "--encoding UTF8 --data-checksums"
    volumes:
      - type: bind
        source: ./pg-config/postgresql.conf
        target: /etc/postgresql/postgresql.conf
    command:
      - "postgres"
      - "-c"
      - "config_file=/etc/postgresql/postgresql.conf"
    healthcheck:
      test: "pg_isready -U postgres"
      interval: 5s
      timeout: 2s
      retries: 15
    networks:
      - polaris_net

  polaris-bootstrap:
    image: apache/polaris-admin-tool:latest
    container_name: polaris-bootstrap
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      POLARIS_PERSISTENCE_TYPE: relational-jdbc
      QUARKUS_DATASOURCE_JDBC_URL: jdbc:postgresql://postgres:5432/polaris
      QUARKUS_DATASOURCE_USERNAME: postgres
      QUARKUS_DATASOURCE_PASSWORD: postgres
    command:
      - "bootstrap"
      - "--realm=polaris"
      - "--credential=polaris,root,s3cr3t"
    networks:
      - polaris_net

  polaris:
    image: apache/polaris:latest
    container_name: polaris-container
    ports:
      - "8181:8181"   # API
      - "8182:8182"   # management (metrics/health)
      - "5005:5005"   # optional JVM debugger
    environment:
      # Skip credential subscoping indirection
      polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": "true"
      polaris.features.realm-overrides."polaris"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": "true"

      # Java debug
      JAVA_DEBUG: true
      JAVA_DEBUG_PORT: "*:5005"

      POLARIS_PERSISTENCE_TYPE: relational-jdbc
      POLARIS_PERSISTENCE_RELATIONAL_JDBC_MAX_RETRIES: 5
      POLARIS_PERSISTENCE_RELATIONAL_JDBC_INITIAL_DELAY_IN_MS: 100
      POLARIS_PERSISTENCE_RELATIONAL_JDBC_MAX_DURATION_IN_MS: 5000
      QUARKUS_DATASOURCE_JDBC_URL: jdbc:postgresql://postgres:5432/polaris
      QUARKUS_DATASOURCE_USERNAME: postgres
      QUARKUS_DATASOURCE_PASSWORD: postgres
      POLARIS_REALM_CONTEXT_REALMS: polaris
      QUARKUS_OTEL_SDK_DISABLED: true
      POLARIS_BOOTSTRAP_CREDENTIALS: polaris,root,s3cr3t
      polaris.features."ALLOW_INSECURE_STORAGE_TYPES": "false"
      polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES": "[\"S3\",\"GCS\",\"AZURE\"]"
      polaris.readiness.ignore-severe-issues: "true"

      # S3 Configuration
      polaris.features."ALLOW_SETTING_S3_ENDPOINTS": "true"
      polaris.storage.aws.access-key: "abcd"
      polaris.storage.aws.secret-key: "abcd"
      AWS_REGION: "us-east-1"
      AWS_ACCESS_KEY_ID: "abcd"
      AWS_SECRET_ACCESS_KEY: "abcd"
      AWS_ENDPOINT_URL: "https://s3-it-hn.company.net"

      # Proxy / no-proxy
      HTTP_PROXY: http://proxy.company.vn:80
      HTTPS_PROXY: http://proxy.company.vn:80
      JAVA_OPTS_APPEND: '-Dhttp.nonProxyHosts="localhost|127.0.0.1|s3-it-hn.company.net|*.s3-it-hn.company.net" -Dhttps.nonProxyHosts="localhost|127.0.0.1|s3-it-hn.company.net|*.s3-it-hn.company.net"'
      NO_PROXY: localhost,127.0.0.1,.s3-it-hn.company.net,s3-it-hn.company.net
    depends_on:
      polaris-bootstrap:
        condition: service_completed_successfully
    healthcheck:
      test: ["CMD", "curl", "http://localhost:8182/q/health"]
      interval: 2s
      timeout: 10s
      retries: 10
      start_period: 10s
    networks:
      - polaris_net

  polaris-setup:
    image: alpine/curl
    container_name: polaris-setup
    depends_on:
      polaris:
        condition: service_healthy
    environment:
      HTTP_PROXY: http://proxy.company.vn:80
      HTTPS_PROXY: http://proxy.company.vn:80
      NO_PROXY: localhost,127.0.0.1,.s3-it-hn.company.net,s3-it-hn.company.net
      STORAGE_LOCATION: "s3://it-demo-data/testxxx"
    volumes:
      - ./setup-polaris:/polaris
    entrypoint: '/bin/sh -c "chmod +x /polaris/create-catalog.sh && /polaris/create-catalog.sh"'
    networks:
      - polaris_net

  spark-jupyter:
    build: .
    container_name: polaris-spark-jupyter
    depends_on:
      polaris-setup:
        condition: service_completed_successfully
    ports:
      - "8888:8888"
      - "4040:4040"
    volumes:
      - ./notebooks:/home/jovyan/work
    healthcheck:
      test: "curl localhost:8888"
      interval: 5s
      retries: 15
    networks:
      - polaris_net

networks:
  polaris_net:
    external: true
    name: open-metadata_app_net

---

Dockerfile (Spark image)

FROM apache/spark:3.5.8-java17-python3

USER root

ENV HTTP_PROXY=http://proxy.company.vn:80/
ENV HTTPS_PROXY=http://proxy.company.vn:80/
ENV NO_PROXY=localhost,127.0.0.1,polaris-postgres,polaris-container

RUN pip install --no-cache-dir \
    jupyterlab \
    notebook \
    pyspark==3.5.8 \
    && mkdir -p /home/jovyan/work

RUN mkdir -p /opt/spark/jars && \
    wget -e use_proxy=yes -e http_proxy=$HTTP_PROXY -e https_proxy=$HTTPS_PROXY \
    https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.10.0/iceberg-spark-runtime-3.5_2.12-1.10.0.jar -O /opt/spark/jars/iceberg-spark-runtime-3.5_2.12-1.10.0.jar && \
    wget -e use_proxy=yes -e http_proxy=$HTTP_PROXY -e https_proxy=$HTTPS_PROXY \
    https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.10.0/iceberg-aws-bundle-1.10.0.jar -O /opt/spark/jars/iceberg-aws-bundle-1.10.0.jar && \
    wget -e use_proxy=yes -e http_proxy=$HTTP_PROXY -e https_proxy=$HTTPS_PROXY \
    https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar -O /opt/spark/jars/hadoop-aws-3.3.4.jar && \
    wget -e use_proxy=yes -e http_proxy=$HTTP_PROXY -e https_proxy=$HTTPS_PROXY \
    https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.797/aws-java-sdk-bundle-1.12.797.jar -O /opt/spark/jars/aws-java-sdk-bundle-1.12.797.jar

# Clear proxy environment variables so they won't be present in later layers / at runtime
ENV HTTP_PROXY=""
ENV HTTPS_PROXY=""
ENV NO_PROXY=""

ENV JUPYTER_TOKEN=polaris
ENV JUPYTER_ENABLE_LAB=yes
RUN useradd -m -s /bin/bash jovyan && chown -R jovyan /home/jovyan
USER jovyan
WORKDIR /home/jovyan/work

EXPOSE 8888 4040

CMD jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token=$JUPYTER_TOKEN

---

Catalog creation payload (create-catalog.sh / jq payload)

PAYLOAD=$(jq -n \
  --arg name "quickstart_catalog" \
  --arg defaultBase "$STORAGE_LOCATION" \
  '{
    catalog: {
      name: $name,
      type: "INTERNAL",
      properties: { "default-base-location": $defaultBase },
      storageConfigInfo: {
        storageType: "S3",
        pathStyleAccess: true,
        stsUnavailable: true,
        endpoint: "https://s3-it-hn.company.net",
        endpointInternal: "https://s3-it-hn.company.net"
      }
    }
  }'
)

---

Spark configuration (Python snippet)

S3_HOST="s3-it-hn.company.net"

current_python = sys.executable 
os.environ["PYSPARK_PYTHON"] = current_python
os.environ["PYSPARK_DRIVER_PYTHON"] = current_python

os.environ["NO_PROXY"] = S3_HOST
os.environ["no_proxy"] = S3_HOST

# ---- JVM proxy settings (driver + executor) ----
proxy_host = "http://proxy.company.vn"
proxy_port = "80"

java_gc_opts = (
    " -XX:+UseG1GC"
    " -XX:InitiatingHeapOccupancyPercent=50"
    " -XX:OnOutOfMemoryError='kill -9 %p'"
)

jvm_opts = (
    f"-Dhttp.proxyHost={proxy_host} -Dhttp.proxyPort={proxy_port} "
    f"-Dhttps.proxyHost={proxy_host} -Dhttps.proxyPort={proxy_port} "
    f"-Dhttp.nonProxyHosts={S3_HOST} -Dhttps.nonProxyHosts={S3_HOST} "
    "-Djava.net.useSystemProxies=false"
    "-Daws.java.v1.disableDeprecationAnnouncement=true"
    f"{java_gc_opts}"
)

spark = (
    SparkSession.builder
    .appName("SparkIceberg")
    .master("local[12]")
    .config("spark.driver.memory", "12g")
    .config("spark.executor.memory", "12g")
    .config("spark.driver.memoryOverhead", "1024m")
    # JARs and JVM options
    .config("spark.driver.extraJavaOptions", jvm_opts)
    .config("spark.executor.extraJavaOptions", jvm_opts)

    # Iceberg Catalog
    .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
    .config('spark.sql.iceberg.vectorization.enabled', 'false')
    .config("spark.sql.catalog.polaris.type", "rest")
    .config("spark.sql.catalog.polaris", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.polaris.uri", "http://polaris:8181/api/catalog")
    .config("spark.sql.catalog.polaris.token-refresh-enabled", "false")
    .config("spark.sql.catalog.polaris.credential", f"{polaris_client_id}:{polaris_client_secret}")
    .config("spark.sql.catalog.polaris.warehouse", catalog_name)
    .config("spark.sql.catalog.polaris.scope", 'PRINCIPAL_ROLE:ALL')

    # S3A settings
    .config("spark.hadoop.fs.s3a.endpoint", S3_HOST)
    .config("spark.hadoop.fs.s3a.access.key", S3_ACCESS_KEY)
    .config("spark.hadoop.fs.s3a.secret.key", S3_SECRET_KEY)
    .config("spark.hadoop.fs.s3a.path.style.access", "true")
    .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    .getOrCreate()
)
print("Spark Session created successfully!")

To Reproduce

When I run these commands:

spark.sql("USE polaris")
spark.sql("SHOW NAMESPACES").show()

spark.sql("CREATE NAMESPACE IF NOT EXISTS COLLADO_TEST")
spark.sql("CREATE NAMESPACE IF NOT EXISTS COLLADO_TEST.PUBLIC")
spark.sql("SHOW NAMESPACES IN COLLADO_TEST").show()

spark.sql("USE NAMESPACE COLLADO_TEST.PUBLIC")
spark.sql("""CREATE TABLE IF NOT EXISTS TEST_TABLE (
    id bigint NOT NULL COMMENT 'unique id',
    data string)
USING iceberg;
""")

This Spark configuration can connect directly to S3 normally. However, when I connect through Polaris, creating the namespace succeeds, but creating a table in that namespace fails with:

Py4JJavaError                             Traceback (most recent call last)
Cell In[5], line 2
      1 spark.sql("USE NAMESPACE COLLADO_TEST.PUBLIC")
----> 2 spark.sql("""CREATE TABLE IF NOT EXISTS TEST_TABLE (
      3     id bigint NOT NULL COMMENT 'unique id',
      4     data string)
      5 USING iceberg;
      6 """)
: org.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: it-demo-data.s3-it-hn.company.net: Name or service not known (SDK Attempt Count: 6)

When I uncomment the proxy section (HTTP_PROXY, HTTPS_PROXY) in polaris-container, I also encounter a similar error:

      # HTTP_PROXY: http://proxy.company.vn:80
      # HTTPS_PROXY: http://proxy.company.vn:80

Actual Behavior

Despite the configuration above, Polaris (or components in the stack) still attempt to access the internet and perform external DNS/resolve for the internal S3 domain. This causes failures because the internal S3 host is only resolvable/accessible from within our internal network and should not require STS.

Expected Behavior

Polaris should connect to the internal S3 endpoint http://s3-it-hn.company.net directly using the provided static access/secret keys and stsUnavailable = true, without attempting to resolve or contact anything on the public internet (or using an STS / credential vending service).

NO_PROXY / no_proxy and JVM -Dhttp.nonProxyHosts / -Dhttps.nonProxyHosts are set where applicable to exclude the internal S3 host from corporate proxy.

Additional context

Has any documentation been provided on this issue? I have read through all the related issues but haven't been able to resolve this problem.

System information

Docker/Docker compose

[dimas-b]

Hi @tuanit03 !

properties > "s3.sts-unavailable": "true" does not look like a correct entry to me.

sts-unavailable is a property of the Storage Config, not a general catalog property.

Please see https://github.com/apache/polaris/blob/main/getting-started/ozone/docker-compose.yml#L153

[tuanit03]

Thank you for replying, @dimas-b
I updated the storage config but I’m still seeing the same error.

I understand that to bypass Polaris’s Credential Vending mechanism you must use the following configs and store the S3 key in Spark:

stsUnavailable: true,
polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": "true"
polaris.features.realm-overrides."polaris"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": "true"

However, I’m having an issue with a custom endpoint: Polaris automatically resolves the endpoint via DNS through the proxy, even though I set NO_PROXY.

Inside the Polaris container I can still connect to this S3 endpoint:

[polaris@e9a836e1cb6b ~]$ curl -v --noproxy s3-it-hn.company.net https://s3-it-hn.company.net/
*   Trying 210.123.456.789:443...
* Connected to s3-it-hn.company.net (210.123.456.789) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=VN; ST=Ha Noi; O=company; CN=*.company.net
*  start date: Nov 14 00:00:00 2025 GMT
*  expire date: Nov 17 23:59:59 2026 GMT
*  subjectAltName: host "s3-it-hn.company.net" matched cert's "*.company.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* Using Stream ID: 1 (easy handle 0x62485f07b950)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/2
> Host: s3-it-hn.company.net
> user-agent: curl/7.76.1
> accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 200
< x-amz-request-id: tx00000-default
< content-type: application/xml
< date: Tue, 03 Feb 2026 08:02:10 GMT
<
* TLSv1.2 (IN), TLS header, Unknown (23):
* Connection #0 to host s3-it-hn.company.net left intact
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

[dimas-b]

@tuanit03 : Would you be able to share your full catalog definition (e.g. from curl http://localhost:8181/api/management/v1/catalogs/)? Please make sure to redact any private data.

[tuanit03]

Thank you for replying, @dimas-b
Below is the log of the Polaris container :

dev777@WITTUANNA233:~$ docker logs polaris-container
INFO exec -a "java" java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Dhttps.proxyHost=proxy.company.vn -Dhttps.proxyPort=80 -Dhttp.proxyHost=proxy.company.vn -Dhttp.proxyPort=80 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.s3-it-hn.company.net|s3-it-hn.company.net -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Dhttp.nonProxyHosts="localhost|127.0.0.1|s3-it-hn.company.net|*.s3-it-hn.company.net" -Dhttps.nonProxyHosts="localhost|127.0.0.1|s3-it-hn.company.net|*.s3-it-hn.company.net" -cp "." -jar /deployments/quarkus-run.jar
INFO running in /deployments
Listening for transport dt_socket at address: 5005

                                                                                     Powered by Quarkus 3.29.4
2026-02-05 13:00:49,782 WARN  [io.qua.config] [,] [,,,] (main) The "quarkus.log.file.enable" config property is deprecated and should not be used anymore.
2026-02-05 13:00:49,782 WARN  [io.qua.config] [,] [,,,] (main) The "quarkus.log.console.enable" config property is deprecated and should not be used anymore.
2026-02-05 13:00:50,544 ERROR [org.apa.pol.ser.con.ServiceProducers] [,] [,,,] (main) Secrets for the realms [polaris] are available via the environment variable POLARIS_BOOTSTRAP_CREDENTIALS or Java system property polaris.bootstrap.credentials. Remove this security sensitive information from the environment / Java system properties!
2026-02-05 13:00:50,779 WARN  [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) ⚠️ Production readiness checks failed! Check the warnings below.
2026-02-05 13:00:50,780 WARN  [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ The realm context resolver is configured to map requests without a realm header to the default realm. Offending configuration option: 'polaris.realm-context.require-header'.
2026-02-05 13:00:50,780 WARN  [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ A public key file wasn't provided and will be generated. Offending configuration option: 'polaris.authentication.token-broker.rsa-key-pair.public-key-file'.
2026-02-05 13:00:50,780 WARN  [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ A private key file wasn't provided and will be generated. Offending configuration option: 'polaris.authentication.token-broker.rsa-key-pair.private-key-file'.
2026-02-05 13:00:50,780 WARN  [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) Refer to https://polaris.apache.org/in-dev/unreleased/configuring-polaris-for-production for more information.
2026-02-05 13:00:50,863 INFO  [io.quarkus] [,] [,,,] (main) Apache Polaris Server (incubating) 1.3.0-incubating on JVM (powered by Quarkus 3.29.4) started in 2.928s. Listening on: http://0.0.0.0:8181. Management interface listening on http://0.0.0.0:8182.
2026-02-05 13:00:50,864 INFO  [io.quarkus] [,] [,,,] (main) Profile prod activated.
2026-02-05 13:00:50,864 INFO  [io.quarkus] [,] [,,,] (main) Installed features: [agroal, amazon-sdk-rds, cdi, hibernate-validator, jdbc-postgresql, micrometer, narayana-jta, oidc, opentelemetry, reactive-routes, rest, rest-jackson, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, vertx]
2026-02-05 13:00:55,679 INFO  [org.apa.pol.ser.con.PolarisIcebergObjectMapperCustomizer] [24583001,polaris] [,,,] (executor-thread-1) Limiting request body size to 10485760 bytes
2026-02-05 13:00:55,957 INFO  [io.qua.htt.access-log] [24583001,polaris] [,,,] (executor-thread-1) 172.17.0.9 - - [05/Feb/2026:13:00:55 +0000] "POST /api/catalog/v1/oauth/tokens HTTP/1.1" 200 781
2026-02-05 13:00:56,155 WARN  [org.apa.pol.ser.sto.StorageConfiguration] [24583002,polaris] [,,,] (executor-thread-1) Using hard-coded AWS credentials - this is not recommended for production
2026-02-05 13:00:56,217 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583002,polaris] [,,,] (executor-thread-1) Created new catalog class PolarisCatalog {
    class Catalog {
        type: INTERNAL
        name: quickstart_catalog
        properties: class CatalogProperties {
            {default-base-location=s3://it-demo-data/testxxx/}
            defaultBaseLocation: s3://it-demo-data/testxxx/
        }
        createTimestamp: 1770296456153
        lastUpdateTimestamp: 0
        entityVersion: 1
        storageConfigInfo: class AwsStorageConfigInfo {
            class StorageConfigInfo {
                storageType: S3
                allowedLocations: [s3://it-demo-data/testxxx/]
            }
            roleArn: null
            externalId: null
            userArn: null
            region: null
            endpoint: https://s3-it-hn.company.net
            stsEndpoint: null
            stsUnavailable: true
            endpointInternal: https://s3-it-hn.company.net
            pathStyleAccess: true
        }
    }
}
2026-02-05 13:00:56,271 INFO  [io.qua.htt.access-log] [24583002,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:56 +0000] "POST /api/management/v1/catalogs HTTP/1.1" 201 417
2026-02-05 13:00:57,106 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583003,polaris] [,,,] (executor-thread-1) Created new principal class PrincipalWithCredentials {
    principal: class Principal {
        name: engineer_principal
        clientId: f1a1399cc348379b
        properties: {}
        createTimestamp: 1770296457097
        lastUpdateTimestamp: 0
        entityVersion: 1
    }
    credentials: class PrincipalWithCredentialsCredentials {
        clientId: f1a1399cc348379b
        clientSecret: *
    }
}
2026-02-05 13:00:57,111 INFO  [io.qua.htt.access-log] [24583003,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:57 +0000] "POST /api/management/v1/principals HTTP/1.1" 201 259
2026-02-05 13:00:57,143 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583004,polaris] [,,,] (executor-thread-1) Created new principalRole class PrincipalRole {
    name: engineer_role
    federated: false
    properties: {}
    createTimestamp: 1770296457139
    lastUpdateTimestamp: 1770296457139
    entityVersion: 1
}
2026-02-05 13:00:57,147 INFO  [io.qua.htt.access-log] [24583004,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:57 +0000] "POST /api/management/v1/principal-roles HTTP/1.1" 201 144
2026-02-05 13:00:57,172 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583005,polaris] [,,,] (executor-thread-1) Assigning principalRole engineer_role to principal engineer_principal
2026-02-05 13:00:57,193 INFO  [io.qua.htt.access-log] [24583005,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:57 +0000] "PUT /api/management/v1/principals/engineer_principal/principal-roles HTTP/1.1" 201 -
2026-02-05 13:00:57,232 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583006,polaris] [,,,] (executor-thread-1) Created new catalogRole class CatalogRole {
    name: manager_catalog_role
    properties: {}
    createTimestamp: 1770296457224
    lastUpdateTimestamp: 1770296457224
    entityVersion: 1
}
2026-02-05 13:00:57,236 INFO  [io.qua.htt.access-log] [24583006,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:57 +0000] "POST /api/management/v1/catalogs/quickstart_catalog/catalog-roles HTTP/1.1" 201 133
2026-02-05 13:00:57,256 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583007,polaris] [,,,] (executor-thread-1) Assigning catalogRole manager_catalog_role in catalog quickstart_catalog to principalRole engineer_role
2026-02-05 13:00:57,275 INFO  [io.qua.htt.access-log] [24583007,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:57 +0000] "PUT /api/management/v1/principal-roles/engineer_role/catalog-roles/quickstart_catalog HTTP/1.1" 201 -
2026-02-05 13:00:57,296 INFO  [org.apa.pol.ser.adm.PolarisServiceImpl] [24583008,polaris] [,,,] (executor-thread-1) Adding grant class AddGrantRequest {
    grant: class CatalogGrant {
        class GrantResource {
            type: catalog
        }
        privilege: CATALOG_MANAGE_CONTENT
    }
} to catalogRole manager_catalog_role in catalog quickstart_catalog
2026-02-05 13:00:57,322 INFO  [io.qua.htt.access-log] [24583008,polaris] [,,,] (executor-thread-1) 172.17.0.9 - root [05/Feb/2026:13:00:57 +0000] "PUT /api/management/v1/catalogs/quickstart_catalog/catalog-roles/manager_catalog_role/grants HTTP/1.1" 201 -
2026-02-05 13:02:41,339 INFO  [io.qua.htt.access-log] [24583009,polaris] [,,,] (executor-thread-1) 172.17.0.9 - - [05/Feb/2026:13:02:41 +0000] "POST /api/catalog/v1/oauth/tokens HTTP/1.1" 200 815
2026-02-05 13:02:41,415 INFO  [io.qua.htt.access-log] [24583010,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:41 +0000] "GET /api/catalog/v1/config?warehouse=quickstart_catalog HTTP/1.1" 200 2096
2026-02-05 13:02:41,879 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [24583011,polaris] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-02-05 13:02:41,898 INFO  [io.qua.htt.access-log] [24583011,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:41 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces?pageToken= HTTP/1.1" 200 40
2026-02-05 13:02:42,280 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [24583012,polaris] [,,,] (executor-thread-1) Handling runtimeException Namespace does not exist: COLLADO_TEST
2026-02-05 13:02:42,289 INFO  [io.qua.htt.access-log] [24583012,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:42 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces/COLLADO_TEST HTTP/1.1" 404 107
2026-02-05 13:02:42,326 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [24583013,polaris] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-02-05 13:02:42,357 INFO  [io.qua.htt.access-log] [24583013,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:42 +0000] "POST /api/catalog/v1/quickstart_catalog/namespaces HTTP/1.1" 200 117
2026-02-05 13:02:42,387 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [24583014,polaris] [,,,] (executor-thread-1) Handling runtimeException Namespace does not exist: COLLADO_TEST.PUBLIC
2026-02-05 13:02:42,388 INFO  [io.qua.htt.access-log] [24583014,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:42 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces/COLLADO_TEST%1FPUBLIC HTTP/1.1" 404 114
2026-02-05 13:02:42,400 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [24583015,polaris] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-02-05 13:02:42,414 INFO  [io.qua.htt.access-log] [24583015,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:42 +0000] "POST /api/catalog/v1/quickstart_catalog/namespaces HTTP/1.1" 200 133
2026-02-05 13:02:42,441 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [24583016,polaris] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-02-05 13:02:42,444 INFO  [io.qua.htt.access-log] [24583016,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:42 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces?parent=COLLADO_TEST&pageToken= HTTP/1.1" 200 65
2026-02-05 13:02:43,311 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [24583017,polaris] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-02-05 13:02:43,315 INFO  [io.qua.htt.access-log] [24583017,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:43 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces/COLLADO_TEST%1FPUBLIC HTTP/1.1" 200 133
2026-02-05 13:02:43,391 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [24583018,polaris] [,,,] (executor-thread-1) Handling runtimeException Table does not exist: COLLADO_TEST.PUBLIC.TEST_TABLE
2026-02-05 13:02:43,392 INFO  [io.qua.htt.access-log] [24583018,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:43 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces/COLLADO_TEST%1FPUBLIC/tables/TEST_TABLE?snapshots=all HTTP/1.1" 404 117
2026-02-05 13:02:43,406 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [24583019,polaris] [,,,] (executor-thread-1) Handling runtimeException Table does not exist: COLLADO_TEST.PUBLIC
2026-02-05 13:02:43,407 INFO  [io.qua.htt.access-log] [24583019,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:43 +0000] "GET /api/catalog/v1/quickstart_catalog/namespaces/COLLADO_TEST/tables/PUBLIC?snapshots=all HTTP/1.1" 404 106
2026-02-05 13:02:43,453 INFO  [org.apa.pol.ser.cat.ice.IcebergCatalogHandler] [245830020,polaris] [,,,] (executor-thread-1) Initializing non-federated catalog
2026-02-05 13:02:43,463 INFO  [org.apa.ice.BaseMetastoreCatalog] [245830020,polaris] [,,,] (executor-thread-1) Table properties set at catalog level through catalog properties: {}
2026-02-05 13:02:43,468 INFO  [org.apa.ice.BaseMetastoreCatalog] [245830020,polaris] [,,,] (executor-thread-1) Table properties enforced at catalog level through catalog properties: {}
2026-02-05 13:02:43,500 INFO  [org.apa.ice.CatalogUtil] [245830020,polaris] [,,,] (executor-thread-1) Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
2026-02-05 13:02:45,028 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [245830020,polaris] [,,,] (executor-thread-1) Handling runtimeException (Service: S3, Status Code: 503, Request ID: null) (SDK Attempt Count: 6)
2026-02-05 13:02:45,029 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [245830020,polaris] [,,,] (executor-thread-1) Full RuntimeException: software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 503, Request ID: null) (SDK Attempt Count: 6)
        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
        at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:12692)
        at org.apache.iceberg.aws.s3.S3OutputStream.completeUploads(S3OutputStream.java:443)
        at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:269)
        at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:255)
        at java.base/sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:435)
        at java.base/sun.nio.cs.StreamEncoder.lockedClose(StreamEncoder.java:237)
        at java.base/sun.nio.cs.StreamEncoder.close(StreamEncoder.java:222)
        at java.base/java.io.OutputStreamWriter.close(OutputStreamWriter.java:266)
        at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:135)
        at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:119)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.writeNewMetadata(IcebergCatalog.java:1624)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.writeNewMetadataIfRequired(IcebergCatalog.java:1613)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.doCommit(IcebergCatalog.java:1476)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.commit(IcebergCatalog.java:1326)
        at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.createTableDirect(IcebergCatalogHandler.java:451)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$createTable$6(IcebergCatalogAdapter.java:382)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:199)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.createTable(IcebergCatalogAdapter.java:366)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.createTable$$superforward(Unknown Source)
        at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator_Gj_WCAwPXCI_Delegate_Subclass.createTable(Unknown Source)
        at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator.createTable(IcebergRestCatalogEventServiceDelegator.java:217)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.createTable(Unknown Source)
        at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.createTable(Unknown Source)
        at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.createTable(IcebergRestCatalogApi.java:193)
        at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createTable$$superforward(Unknown Source)
        at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$3.apply(Unknown Source)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
        at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
        at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
        at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
        at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
        at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
        at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
        at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
        at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)
        at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
        at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:44)
        at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
        at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createTable(Unknown Source)
        at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$createTable_3101.invoke(Unknown Source)
        at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:183)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: (Service: S3, Status Code: 503, Request ID: null)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: (Service: S3, Status Code: 503, Request ID: null)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: (Service: S3, Status Code: 503, Request ID: null)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 4 failure: (Service: S3, Status Code: 503, Request ID: null)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 5 failure: (Service: S3, Status Code: 503, Request ID: null)

2026-02-05 13:02:45,031 INFO  [io.qua.htt.access-log] [245830020,polaris] [,,,] (executor-thread-1) 172.17.0.9 - engineer_principal [05/Feb/2026:13:02:45 +0000] "POST /api/catalog/v1/quickstart_catalog/namespaces/COLLADO_TEST%1FPUBLIC/tables HTTP/1.1" 502 128

This is when I'm standing at the host, outside of the Docker network environment :

dev777@WITTUANNA233:~$ curl -v -s http://localhost:8181/api/catalog/v1/oauth/tokens \
  --user root:s3cr3t \
  -H "Polaris-Realm: polaris" \
  -d grant_type=client_credentials \
  -d scope=PRINCIPAL_ROLE:ALL
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,::1'
* Host localhost:8181 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8181...
* Connected to localhost (::1) port 8181
* Server auth using Basic with user 'root'
> POST /api/catalog/v1/oauth/tokens HTTP/1.1
> Host: localhost:8181
> Authorization: Basic abcd3Q=
> User-Agent: curl/8.5.0
> Accept: */*
> Polaris-Realm: polaris
> Content-Length: 54
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Type: application/json;charset=UTF-8
< content-length: 781
< X-Request-ID: 20000021
<
* Connection #0 to host localhost left intact
{"access_token":"abcdefgh","token_type":"bearer","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","expires_in":3600}

[dimas-b]

@tuanit03 : when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is set to true, endpoint settings from the Storage Config in the Catalog object are not used 🤷

Do you have a compelling reason to set this flag?

Please see the existing Apache Ozone example for how to configure non-AWS S3 storage. That example does not set SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION.

[tuanit03]

Thank you very much, @dimas-b
I fixed the issue by setting the following configs:

polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": "false"
polaris.features.realm-overrides."polaris"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": "false"

At first, I wanted to bypass the Credential Vending mechanism, but I didn't realize that doing so would cause Polaris to skip receiving a custom endpoint.

[dimas-b]

SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is a very old flag with non-intuitive behaviour, IMHO 😅

If STS is not available, setting stsUnavailable: true in Storage Config is sufficient. No credentials will be vended, but clients will still receive endpoint settings from Polaris.