Add documentation for `DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED`

[adutra]

Is your feature request related to a problem? Please describe.

The DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED feature configuration could benefit from
improved documentation to clarify its purpose, limitations, and relationship with Iceberg's
write.object-storage.enabled feature.

Background

Polaris has a feature config called DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED introduced in
bd83252.

The feature name and current description may lead users to believe it provides similar functionality
to Iceberg's Object Store File Layout, but the two features work at different levels and are designed to be complementary.

How the features differ

Iceberg's write.object-storage.enabled applies entropy (hash-based prefix) on a per-file
basis. Each file gets a unique hash prefix.

Polaris's DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED applies entropy once per table, based on the table identifier. All files in the same table share the same hash.

Example

Consider two data files in a table newdb.newtable:

Standard layout (no entropy):

s3://bucket/warehouse/newdb/newtable/data/file1.parquet
s3://bucket/warehouse/newdb/newtable/data/file2.parquet

With Iceberg's object store layout only (per-file entropy):

s3://bucket/warehouse/newdb/newtable/data/0011/0100/1011/11101010/file1.parquet
s3://bucket/warehouse/newdb/newtable/data/0011/0001/0001/00000001/file2.parquet

With Polaris's object storage prefix only (per-table entropy):

s3://bucket/warehouse/1111/1111/0100/01010000/newdb/newtable/data/file1.parquet
s3://bucket/warehouse/1111/1111/0100/01010000/newdb/newtable/data/file2.parquet

With both features combined:

s3://bucket/warehouse/1111/1111/0100/01010000/newdb/newtable/data/0011/0100/1011/11101010/file1.parquet
s3://bucket/warehouse/1111/1111/0100/01010000/newdb/newtable/data/0011/0001/0001/00000001/file2.parquet

Describe the solution you'd like

The documentation should clarify:

1. Purpose: Polaris's layout distributes different tables across the key space, preventing hotspots when multiple tables in the same namespace are accessed concurrently. It does not distribute files within a single table.
2. Limitations: since all files in a table share the same prefix, this layout alone does not prevent hotspots when a single table receives heavy write traffic.
3. Complementary usage: as stated in the original commit, "The two features can and should be combined to achieve the best distribution of data files throughout the key space."

Describe alternatives you've considered

No response

Additional context

No response

[adutra]

To be honest, I'm not 100% convinced of the usefulness of Polaris' object layout.

S3 is said to support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per "partitioned prefix":

https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html

But we don't know exactly what a "partitioned prefix" is. For data file hotspots: Iceberg's layout is likely sufficient. For metadata file hotspots: Polaris's feature might provide additional value since Iceberg's layout doesn't apply to these files, but metadata operations are typically lower volume than data writes.

The following questions could help clarify the feature's value:

1. Was this feature introduced based on empirical evidence of improved S3 performance, or based
   on theoretical assumptions about prefix distribution?
2. Are there benchmarks or case studies showing that per-table entropy prefixes reduces hotspots
   or improve throughput?
3. Does AWS documentation or support confirm that prefix diversity helps S3 partition more
   effectively?

[adutra]

After more investigation on this, I think there is a much better option to improve entropy and reduce hotspots across different tables: just set write.data.path for all tables to a common, short prefix, e.g. the warehouse location.

Indeed Iceberg's Object Store layout behaves differently when the table has write.data.path defined.

Instead of creating file path like this:

<table-base>/data/<file-hash>/[<partition>/]file1.parquet

The layout creates paths like this:

<write.data.path>/<file-hash>/<ns>/<table>/[<partition>/]file1.parquet

This setup effectively scatters table files and creates high entropy:

s3://bucket/warehouse/1111/1111/0100/01010000/ns1/table1/data/file1.parquet
s3://bucket/warehouse/0011/0110/0101/10101100/ns1/table1/data/file2.parquet
s3://bucket/warehouse/1110/0011/0100/11010000/ns2/table2/data/file1.parquet
s3://bucket/warehouse/0111/0100/0101/01101101/ns2/table2/data/file2.parquet

While Polaris layout achieves lesser entropy since the hash is per-table and not per-file:

s3://bucket/warehouse/1111/1111/0100/01010000/ns1/table1/data/file1.parquet
s3://bucket/warehouse/1111/1111/0100/01010000/ns1/table1/data/file2.parquet
s3://bucket/warehouse/1001/0011/1101/11010111/ns2/table2/data/file1.parquet
s3://bucket/warehouse/1001/0011/1101/11010111/ns2/table2/data/file2.parquet

[Subham-KRLX]

Hi @adutra I have been following this discussion and I would like to help by adding the documentation for DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED.

[adutra]

@Subham-KRLX at this point I think I would like first to start a discussion on the ML on this topic. I'm not seeing much value in this feature. But I'm certainly missing some important context. That's why a discussion might help clarify things.

[MonkeyCanCode]

@Subham-KRLX at this point I think I would like first to start a discussion on the ML on this topic. I'm not seeing much value in this feature. But I'm certainly missing some important context. That's why a discussion might help clarify things.

So we already have this configuration documented in https://github.com/apache/polaris/blob/main/site/content/in-dev/unreleased/configuration.md?plain=1#L128. I am assuming you are purposing maybe we put this in another location (e.g. prod configuration to reduce the chance of hitting S3 quota on prefix)?

[adutra]

put this in another location (e.g. prod configuration to reduce the changing of hitting S3 quota on prefix)?

Yes, exactly. But let me start an ML discussion first.