Deleting Large Tables causes the system to exhaust connections pool or become unresponsive.

[krutoileshii]

Describe the bug

I have a very large table that was created through CDC. After executing DROP TABLE from trino, the whole polaris instance becomes unresponsive with logs like below:

Running polaris as a standalone (not docker).

2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467576-0d5953a4-ceed-49ed-a763-6ebb7b66a939-00001.parquet tableIdentifier=allo_custom.dbo__cdc_test baseFile=s3://testdec16/allo_custom/dbo__cdc_test/metadata/58be18c0-aa21-4b89-84dd-e1eb953f0444-m2.avro Exception caught deleting data file
2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467576-0d5953a4-ceed-49ed-a763-6ebb7b66a939-00001.parquet attempt=2 error=java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 1) Error encountered attempting to delete file
2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467292-00defaac-2afc-4248-959c-f08f9294f161-00001.parquet tableIdentifier=allo_custom.dbo__cdc_test baseFile=s3://testdec16/allo_custom/dbo__cdc_test/metadata/58be18c0-aa21-4b89-84dd-e1eb953f0444-m2.avro Exception caught deleting data file
2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467292-00defaac-2afc-4248-959c-f08f9294f161-00001.parquet attempt=2 error=java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 1) Error encountered attempting to delete file

After a while, it starts to delete them eventually, but during this process, you cannot access catalog, list tables, create table or query data.

Using

postgresql as a JDBC backend.

Here is my application config for using On-Prem S3.

polaris.realm-context.realms=polaris
quarkus.log.console.level=DEBUG
quarkus.log.category."software.amazon.awssdk.request".level=DEBUG
quarkus.log.category."software.amazon.awssdk.auth.signer".level=DEBUG
quarkus.log.category."software.amazon.awssdk.http".level=DEBUG
quarkus.log.category."io.smallrye.config".level=DEBUG
polaris.realm-context.header-name=Polaris-Realm
#polaris.realm-context.require-header=true
polaris.persistence.type=relational-jdbc
quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=user_name
quarkus.datasource.password=password
quarkus.datasource.jdbc.url=jdbc:postgresql://postgresql:5432/polaris_test
polaris.authentication.token-broker.type=rsa-key-pair
polaris.authentication.token-broker.rsa-key-pair.public-key-file=/opt/polaris/public.key
polaris.authentication.token-broker.rsa-key-pair.private-key-file=/opt/polaris/private.key
polaris.authentication.token-broker.max-token-generation=PT1H
polaris.bootstrap.credentials=polaris,admin,creds
polaris.storage.aws.access-key=access_key
polaris.storage.aws.secret-key=secret_key
polaris.catalog.s3.no-sts=true
#polaris.catalog.s3.region=us-east-1
#polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": true
polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES": ["S3"]
polaris.features."ALLOW_SETTING_S3_ENDPOINTS"=true
polaris.features."DROP_WITH_PURGE_ENABLED"=true
polaris.features."CLEANUP_ON_CATALOG_DROP"=true

To Reproduce

No response

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response

[krutoileshii]

Also when deleting smaller files, getting errors like this:

2026-01-16 19:17:33,220 DEBUG [sof.ama.aws.htt.apa.int.con.IdleConnectionReaper] [,] [19644571311104ff186759821a23c2d3,,35498b5f90f77d70,true] (idle-connection-reaper) Shutting down reaper thread.
2026-01-16 19:17:33,217 DEBUG [sof.ama.aws.htt.apa.int.net.SdkSslSocket] [,] [19644571311104ff186759821a23c2d3,,35498b5f90f77d70,true] (executor-thread-2) closing objects./198:443
2026-01-16 19:17:35,171 WARN  [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-2) Failed to handle task entity id 9078206595966038194: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/615c8533-b414-4ed1-bbfe-046dd88cbc60-m1.avro, length=7382, partition_spec_id=0, added_snapshot_id=6205443502928148023, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=1, min_sequence_number=1, first_row_id=null}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:650)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/615c8533-b414-4ed1-bbfe-046dd88cbc60-m1.avro, length=7382, partition_spec_id=0, added_snapshot_id=6205443502928148023, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=1, min_sequence_number=1, first_row_id=null}
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:127)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.cleanUpManifestFile(ManifestFileCleanupTaskHandler.java:83)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.handleTask(ManifestFileCleanupTaskHandler.java:66)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTask(TaskExecutorImpl.java:172)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTaskWithTracing(TaskExecutorImpl.java:209)
        at org.apache.polaris.service.task.TaskExecutorImpl.lambda$tryHandleTask$0(TaskExecutorImpl.java:135)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        ... 11 more

2026-01-16 19:17:35,231 WARN  [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-3) Failed to handle task entity id 4837728316142291072: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/70f25620-6b8d-441f-92a1-66773ed16cb3-m1.avro, length=7376, partition_spec_id=0, added_snapshot_id=7326900106277680047, added_data_files_count=1, added_rows_count=49, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=4, min_sequence_number=4, first_row_id=null}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:650)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/70f25620-6b8d-441f-92a1-66773ed16cb3-m1.avro, length=7376, partition_spec_id=0, added_snapshot_id=7326900106277680047, added_data_files_count=1, added_rows_count=49, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=4, min_sequence_number=4, first_row_id=null}  
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:127)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.cleanUpManifestFile(ManifestFileCleanupTaskHandler.java:83)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.handleTask(ManifestFileCleanupTaskHandler.java:66)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTask(TaskExecutorImpl.java:172)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTaskWithTracing(TaskExecutorImpl.java:209)
        at org.apache.polaris.service.task.TaskExecutorImpl.lambda$tryHandleTask$0(TaskExecutorImpl.java:135)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        ... 11 more

2026-01-16 19:17:35,231 WARN  [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-2) Failed to handle task entity id 946636678825059231: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/ca08dcd5-46c5-4ea4-9497-283c6b8e9841-m1.avro, length=7394, partition_spec_id=0, added_snapshot_id=5023809684943975341, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=2, min_sequence_number=2, first_row_id=null}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:650)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/ca08dcd5-46c5-4ea4-9497-283c6b8e9841-m1.avro, length=7394, partition_spec_id=0, added_snapshot_id=5023809684943975341, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=2, min_sequence_number=2, first_row_id=null}
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:127)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.cleanUpManifestFile(ManifestFileCleanupTaskHandler.java:83)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.handleTask(ManifestFileCleanupTaskHandler.java:66)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTask(TaskExecutorImpl.java:172)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTaskWithTracing(TaskExecutorImpl.java:209)
        at org.apache.polaris.service.task.TaskExecutorImpl.lambda$tryHandleTask$0(TaskExecutorImpl.java:135)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        ... 11 more

[krutoileshii]

Could this be coming from here? If FileIO fails's wouldn't this still hold connection for manifest? Also, wouldn't it free up connections faster if we first got a manifest list and then did the FileIO to minimize pool usage. based on this it seems this will hold both sn.AllManifests map and then for the TaskUtils.exists.

private Stream<TaskEntity> getManifestTaskStream(
      TaskEntity cleanupTask,
      TableMetadata tableMetadata,
      FileIO fileIO,
      IcebergTableLikeEntity tableEntity,
      PolarisMetaStoreManager metaStoreManager,
      PolarisCallContext polarisCallContext) {
    // read the manifest list for each snapshot. dedupe the manifest files and schedule a
    // cleanupTask
    // for each manifest file and its data files to be deleted
    return tableMetadata.snapshots().stream()
        .flatMap(sn -> sn.allManifests(fileIO).stream())
        // distinct by manifest path, since multiple snapshots will contain the same
        // manifest
        .collect(Collectors.toMap(ManifestFile::path, Function.identity(), (mf1, mf2) -> mf1))
        .values()
        .stream()
        .filter(mf -> TaskUtils.exists(mf.path(), fileIO))
        .map(
            mf -> {
              // append a random uuid to the task name to avoid any potential conflict
              // when
              // storing the task entity. It's better to have duplicate tasks than to risk
              // not storing the rest of the task entities. If a duplicate deletion task
              // is
              // queued, it will check for the manifest file's existence and simply exit
              // if
              // the task has already been handled.
              String taskName = cleanupTask.getName() + "_" + mf.path() + "_" + UUID.randomUUID();


              LOGGER
                  .atDebug()
                  .addKeyValue("taskName", taskName)
                  .addKeyValue("tableIdentifier", tableEntity.getTableIdentifier())
                  .addKeyValue("metadataLocation", tableEntity.getMetadataLocation())
                  .addKeyValue("manifestFile", mf.path())
                  .log("Queueing task to delete manifest file");
              return new TaskEntity.Builder()
                  .setName(taskName)
                  .setId(metaStoreManager.generateNewEntityId(polarisCallContext).getId())
                  .setCreateTimestamp(clock.millis())
                  .withTaskType(AsyncTaskType.MANIFEST_FILE_CLEANUP)
                  .withData(
                      ManifestFileCleanupTaskHandler.ManifestCleanupTask.buildFrom(
                          tableEntity.getTableIdentifier(), mf))
                  .setId(metaStoreManager.generateNewEntityId(polarisCallContext).getId())
                  // copy the internal properties, which will have storage info
                  .setInternalProperties(cleanupTask.getInternalPropertiesAsMap())
                  .build();
            });
  }

[dimas-b]

@krutoileshii : Thanks for your interest in Polaris! I'm not sure many people have ready answers for this problem (like myself), so please do not consider lack of answers as discouragement 🙂 If you have specific fixes in mind, feel free to open PRs.

You may be interested in this discussion too: https://lists.apache.org/thread/0z8nb3w58zb9s617gsoyhzlnz53rt9zx