Skip to content

[PROPOSAL][DRAFT] Add Table-Level Storage Credential Overrides with Vending Support#3563

Open
sririshindra wants to merge 1 commit intoapache:mainfrom
sririshindra:main-credential-vending-extension
Open

[PROPOSAL][DRAFT] Add Table-Level Storage Credential Overrides with Vending Support#3563
sririshindra wants to merge 1 commit intoapache:mainfrom
sririshindra:main-credential-vending-extension

Conversation

@sririshindra
Copy link

The Problem: Currently, Polaris enforces a 1:1 mapping between a Catalog and a set of storage credentials. This forces all tables within a catalog to exist on a single storage backend. This restriction makes it impossible to build logical catalogs (e.g., "Marketing") that group tables from disparate data sources (e.g., S3 and Ozone) under a single namespace.

The Solution: This PR implements Table-Level Storage Credential Overrides.

  • Granularity: Allows specific tables to define their own storage credentials via table properties, overriding the catalog defaults.

  • Vending Support: Updates the credential vending flow to respect these overrides securely.

  • Flexibility: Transforms the Catalog into a truly logical structure, agnostic of the underlying physical storage locations of its tables.

Reference: Detailed Design Doc

TODO:

  • As this is a POC, this is currently only Implemente for S3. It needs to be extended for GCS and Azure etc...
  • Secure the table level credentials from unauthorized access
  • Add Admin Controls for policy definition.

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed)
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant