feat(persistence): Add JDBC persistence layer for Iceberg metrics reporting (#3337) by obelix74 · Pull Request #3385 · apache/polaris

obelix74 · 2026-01-08T17:44:47Z

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Description

This PR introduces the JDBC persistence layer for storing Iceberg metrics reports (scan and commit operations) as part of the Compute Client Audit Reporting feature (#3337).

Overview

This change adds the foundational persistence infrastructure required to store Iceberg metrics data received via the REST catalog's /reportMetrics endpoint. The metrics data includes detailed information about scan operations (file counts, data volumes, filter expressions) and commit operations (added/removed files, snapshot details).

Components Introduced

Model Classes
• ModelScanMetricsReport - Immutable model representing scan metrics with 30+ fields
• ModelCommitMetricsReport - Immutable model representing commit metrics with 30+ fields

Converters
• MetricsReportConverter - Converts Iceberg ScanReport and CommitReport objects to persistence models
• ModelScanMetricsReportConverter - Maps JDBC ResultSet to ModelScanMetricsReport
• ModelCommitMetricsReportConverter - Maps JDBC ResultSet to ModelCommitMetricsReport

Persistence Methods (added to JdbcBasePersistenceImpl)
• writeScanMetricsReport() / writeCommitMetricsReport() - Insert metrics into the database
• queryScanMetricsReports() / queryCommitMetricsReports() - Query metrics by realm and catalog
• queryScanMetricsReportsByTraceId() / queryCommitMetricsReportsByTraceId() - Query metrics by OpenTelemetry trace ID
• deleteScanMetricsReportsOlderThan() / deleteCommitMetricsReportsOlderThan() - Time-based cleanup
• deleteAllMetricsReportsOlderThan() - Unified cleanup for both metric types

Testing

 • Unit tests for model classes (ModelScanMetricsReportTest, ModelCommitMetricsReportTest)
 • Integration tests for persistence operations (MetricsReportPersistenceTest)
 • Comprehensive test coverage for all CRUD operations, edge cases, and data integrity

Related Issues

 • Closes: #3337 (Compute Client Audit Reporting)

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsSessionTagsBuilder.java

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

...me/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java

.../main/java/org/apache/polaris/persistence/relational/jdbc/models/MetricsReportConverter.java

...me/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java

.../service/src/main/java/org/apache/polaris/service/reporting/MetricsReportCleanupService.java

obelix74 · 2026-01-09T00:15:45Z

Retest this please.

obelix74 · 2026-01-11T02:38:56Z

@dimas-b As discussed, I have moved the AWS STS changes to its own PR. #3414

obelix74 · 2026-01-12T02:44:24Z

@adutra, @dimas-b asked me to tag you to review the event related changes in this PR. Please see the google doc in the PR description for the background.

Event-Related Changes Summary for Review

This pull request introduces comprehensive metrics reporting events and infrastructure to enable audit logging of compute engine metrics (scan and commit reports from Spark, Trino, Flink, etc.). Below is a detailed breakdown of all event-related modifications:

1. Event Types (PolarisEventType.java)
File: runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventType.java
Class: PolarisEventType (enum)
Changes: Added two new event types for metrics reporting:
• BEFORE_REPORT_METRICS - Emitted before a metrics report is processed
• AFTER_REPORT_METRICS - Emitted after a metrics report has been processed

2. Event Classes (IcebergRestCatalogEvents.java)
File: runtime/service/src/main/java/org/apache/polaris/service/events/IcebergRestCatalogEvents.java
Class: IcebergRestCatalogEvents
Changes: Added two new event record classes:
• BeforeReportMetricsEvent - Event emitted before metrics report processing, containing event metadata, catalog name, namespace, table name, and the
ReportMetricsRequest (which wraps ScanReport or CommitReport)
• AfterReportMetricsEvent - Event emitted after metrics report processing with the same structure, enabling audit logging of compute engine metrics including
trace context for correlation

3. Event Listener Interface (PolarisEventListener.java)
File: runtime/service/src/main/java/org/apache/polaris/service/events/listeners/PolarisEventListener.java
Class: PolarisEventListener (interface)
Changes: Added two new default handler methods in the "Metrics Reporting Events" section:
• onBeforeReportMetrics(BeforeReportMetricsEvent event) - Handler for before-metrics events
• onAfterReportMetrics(AfterReportMetricsEvent event) - Handler for after-metrics events

4. Persistence Event Listener (PolarisPersistenceEventListener.java)
File: runtime/service/src/main/java/org/apache/polaris/service/events/listeners/PolarisPersistenceEventListener.java
Class: PolarisPersistenceEventListener (abstract class)
Changes: Implemented the onAfterReportMetrics handler with comprehensive metrics extraction logic including:
• New imports for CommitMetricsResult, ScanMetricsResult, CounterResult, ReportMetricsRequest
• Full implementation of onAfterReportMetrics() that creates a PolarisEvent with report type, snapshot ID, schema ID, and OpenTelemetry context
• New helper method extractScanReportData() - Extracts key scan metrics (result data files, delete files, file sizes, manifests) with trace-id from metadata
• New helper method extractCommitReportData() - Extracts key commit metrics (added/removed files, records, file sizes) with operation type and sequence number
• New helper method addReportMetadata() - Safely adds report metadata entries with null-safety to prevent NPE in ImmutableMap.Builder
• New helper method addCounterIfPresent() - Safely adds counter values to properties map

5. Event Service Delegator (IcebergRestCatalogEventServiceDelegator.java)
File: runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergRestCatalogEventServiceDelegator.java
Class: IcebergRestCatalogEventServiceDelegator
Changes: Modified reportMetrics() method to emit before/after events:
• Added import for RESTUtil for table name decoding
• Enhanced method documentation explaining the endpoint's purpose for compute engine metrics
• Added event emission: onBeforeReportMetrics() called before delegating, onAfterReportMetrics() called after
• Added proper namespace/table decoding using decodeNamespace() and RESTUtil.decodeString()

6. Test Event Listener (TestPolarisEventListener.java)
File: runtime/service/src/testFixtures/java/org/apache/polaris/service/events/listeners/TestPolarisEventListener.java
Class: TestPolarisEventListener
Changes: Added implementations for the two new event handlers:
• onBeforeReportMetrics() - Records the event for test verification
• onAfterReportMetrics() - Records the event for test verification

7. New Unit Tests (PolarisPersistenceEventListenerTest.java)
File: runtime/service/src/test/java/org/apache/polaris/service/events/listeners/PolarisPersistenceEventListenerTest.java
Class: PolarisPersistenceEventListenerTest (NEW FILE)
Changes: New test class with null-safety tests for metrics extraction:
• testScanReportWithNullMetadataValues() - Verifies null values in metadata are safely skipped
• testCommitReportWithNullOperation() - Verifies null operation handling
• testCommitReportWithNullMetadataValues() - Verifies null metadata value handling
• testScanReportWithEmptyMetadata() - Verifies empty metadata handling

Integration Tests (InMemoryBufferEventListenerIntegrationTest.java)
File: runtime/service/src/test/java/org/apache/polaris/service/events/listeners/inmemory/InMemoryBufferEventListenerIntegrationTest.java
Class: InMemoryBufferEventListenerIntegrationTest
Changes: Extended with comprehensive metrics event testing:
• Added new imports for ScanReport, CommitReport, ReportMetricsRequest, and related classes
• Added @beforeeach method resetDatabaseState() for test isolation
• Added feature flag ALLOW_OVERLAPPING_CATALOG_URLS to test configuration
• Added unique base locations per catalog to avoid overlap issues
• testReportMetricsEventWithTraceContext() - Verifies AfterReportMetricsEvent is emitted with OpenTelemetry trace context
• testReportMetricsWithTraceIdInMetadata() - Verifies trace-id from report metadata is extracted with "report." prefix
• testReportCommitMetrics() - Verifies CommitReport data extraction including operation, sequence number, and trace context

9. New Metrics Reporting Infrastructure

The following new classes were added to support alternative metrics persistence strategies:

File: runtime/service/src/main/java/org/apache/polaris/service/reporting/PolarisMetricsReporter.java
Class: PolarisMetricsReporter (interface)
Changes: Enhanced interface documentation describing available implementations (default, events, persistence, composite)

File: runtime/service/src/main/java/org/apache/polaris/service/reporting/EventsMetricsReporter.java
Class: EventsMetricsReporter (NEW FILE - CDI bean with @Identifier("events"))
Purpose: Persists scan/commit reports to the events table as JSON for unified audit trail

File: runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java
Class: PersistingMetricsReporter (NEW FILE - CDI bean with @Identifier("persistence"))
Purpose: Persists metrics to dedicated tables (scan_metrics_report, commit_metrics_report) for better queryability

File: runtime/service/src/main/java/org/apache/polaris/service/reporting/CompositeMetricsReporter.java
Class: CompositeMetricsReporter (NEW FILE)
Purpose: Delegates to multiple reporters, allowing metrics to be sent to multiple destinations

File: runtime/service/src/main/java/org/apache/polaris/service/reporting/MetricsReportCleanupService.java
Class: MetricsReportCleanupService (NEW FILE)
Purpose: Scheduled cleanup service for old metrics reports based on configurable retention policy

File: runtime/service/src/main/java/org/apache/polaris/service/reporting/MetricsReportingConfiguration.java
Class: MetricsReportingConfiguration (interface)
Changes: Extended with new configuration options:
• Enhanced type() documentation for all reporter types
• Added targets() method for composite reporter configuration
• Added RetentionConfig nested interface with enabled(), retentionPeriod(), and cleanupInterval() settings

Key Summary
• New Event Types: 2 (BEFORE_REPORT_METRICS, AFTER_REPORT_METRICS)
• New Event Records: 2 (BeforeReportMetricsEvent, AfterReportMetricsEvent)
• New Listener Methods: 2 (onBeforeReportMetrics, onAfterReportMetrics)
• New Reporter Classes: 4 (EventsMetricsReporter, PersistingMetricsReporter, CompositeMetricsReporter, MetricsReportCleanupService)
• New Test Files: 4 (PolarisPersistenceEventListenerTest, EventsMetricsReporterTest, PersistingMetricsReporterTest, CompositeMetricsReporterTest)

obelix74 · 2026-01-16T21:59:59Z

@dimas-b thanks for the help in merging #3414.

I have rebased this PR against main - looks like the event code was simplified and rewritten. I have redone the event code to follow the changes.

I think this PR can be split into two.

PR 1: Enable metrics event emission
• EventAttributes.java - add REPORT_METRICS_REQUEST attribute
• IcebergRestCatalogEventServiceDelegator.java - emit BEFORE/AFTER events in reportMetrics()

PR 2: Add metrics persistence (depends on PR 1)
• PolarisPersistenceEventListener.java - handle AFTER_REPORT_METRICS, extract ScanReport/CommitReport data
• PolarisPersistenceEventListenerTest.java - new test file for persistence listener

If this helps, please let me know and I can split this PR into two.

dimas-b

Preliminary comments, not an end-to-end review :)

...bc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java

...me/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

dimas-b · 2026-01-17T00:58:34Z

@obelix74 : Re: splitting into two PRs - if it's not too much overhead, it might be preferable from my POV. I think persistence impl. may be a bit more involved as I commented above.

obelix74 · 2026-01-17T02:18:12Z

@obelix74 : Re: splitting into two PRs - if it's not too much overhead, it might be preferable from my POV. I think persistence impl. may be a bit more involved as I commented above.

On it. Moving this PR to draft until the event PR is merged and this branch / PR is rebased against main.

obelix74 · 2026-01-17T03:21:48Z

@obelix74 : Re: splitting into two PRs - if it's not too much overhead, it might be preferable from my POV. I think persistence impl. may be a bit more involved as I commented above.

On it. Moving this PR to draft until the event PR is merged and this branch / PR is rebased against main.

@dimas-b Raised #3468

obelix74 · 2026-01-17T03:52:54Z

@obelix74 : Re: splitting into two PRs - if it's not too much overhead, it might be preferable from my POV. I think persistence impl. may be a bit more involved as I commented above.

On it. Moving this PR to draft until the event PR is merged and this branch / PR is rebased against main.

@dimas-b Raised #3468

@dimas-b I have further redone this PR into the following distinct commits.

The feat/add-metrics-persistence-handler branch now contains 7 well-organized commits:

Commit	Message	Files	Purpose
`132ca9b`	feat(events): Enable metrics event emission in reportMetrics()	8	PR#1 - Feature flag + event types
`1954427`	feat(events): Add metrics persistence handler for AFTER_REPORT_METRICS events	2	PR#2 - Basic event listener
`bf37017`	feat(persistence): Add database schema and model classes for metrics (PR#3)	9	Schema v4 + model classes
`c04e60e`	feat(persistence): Add JDBC persistence implementation for metrics (PR#4)	4	JDBC layer + tests
`ed8b107`	feat(reporting): Add metrics reporters infrastructure (PR#5)	11	Reporters + configuration
`6def4a4`	feat(cleanup): Add metrics cleanup service and documentation (PR#6)	2	Cleanup service + docs
`553045d`	test(integration): Update InMemoryBufferEventListenerIntegrationTest	1	Integration tests

PR#1 is #3468.

If need be, we can separate out each one of these self-contained commits into their own PRs for review.

Address reviewer feedback from PR apache#3385 (comment r2700474938): - Introduce MetricsPersistence interface in polaris-core for backend-agnostic metrics persistence - Add MetricsContext immutable class to encapsulate metrics context information - Add NoOpMetricsPersistence for backends that don't support metrics persistence - Add JdbcMetricsPersistence implementation that wraps JdbcBasePersistenceImpl - Add MetricsPersistenceProducer CDI producer to select implementation at runtime - Refactor PersistenceMetricsProcessor to use MetricsPersistence instead of instanceof checks This design supports the upcoming NoSQL persistence backend (apache#3396) by allowing each backend to provide its own MetricsPersistence implementation or use the no-op implementation that silently discards metrics.

site/content/in-dev/unreleased/telemetry.md

obelix74 · 2026-01-23T17:00:13Z

@adutra @dimas-b @singhpk234

This PR for metrics persistence is now ready for review.

PR1 (this PR): Database schema v4 and persistence layer for metrics storage.
• New tables: scan_metrics_report, commit_metrics_report with indexes
• Model classes and converters for Iceberg ScanReport/CommitReport
• Persistence methods: write, query (by table/trace ID), delete (for retention)
• Schema version checks for graceful degradation on v1-v3 deployments (review feedback from @dimas-b)

PR2 (follow-up, not raised yet): Metrics processing framework that uses this persistence layer.
• MetricsPersistence interface with JdbcMetricsPersistence and NoOpMetricsPersistence implementations (review feedback from @dimas-b)
• MetricsContext for passing request context (principal, trace IDs)
• CDI producer for selecting persistence implementation based on configuration
• Integration with the REST catalog metrics endpoint

dimas-b

Some more preliminary comments... Trying to understand end-to-end use cases 🤔

.gitignore

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql

Replace junction tables (scan_metrics_report_roles, commit_metrics_report_roles) with a denormalized principal_role_ids JSON array column in both scan_metrics_report and commit_metrics_report tables. This simplifies the schema and reduces the number of tables from 5 to 3.

Remove BEFORE_REPORT_METRICS and AFTER_REPORT_METRICS event types as requested in PR review - events for metrics can be added in a separate PR if needed.

Per PR review feedback, MetricsPersistence is now injected directly via CDI instead of being obtained through MetaStoreManagerFactory.getOrCreateMetricsPersistence(). Changes: - Remove getOrCreateMetricsPersistence() from MetaStoreManagerFactory interface - Add @RequestScoped CDI producer for MetricsPersistence in ServiceProducers - Update PersistingMetricsReporter to inject MetricsPersistence directly - Update PersistingMetricsReporterTest to use direct injection This allows MetricsPersistence to be a request-scoped bean created on demand, independent of the entity persistence backend. Persistence backends that support metrics storage can provide an alternative CDI producer.

- Updated ModelScanMetricsReportTest and ModelCommitMetricsReportTest - Added tests for principal_role_ids JSON parsing in fromResultSet - Added tests for principal_role_ids serialization in toMap (H2 and Postgres) - Added tests for the separate Converter classes used in query methods - Updated createTestReport() to include roles

obelix74 · 2026-02-06T04:43:01Z

Re: cleanup - the Admin Tool might be a good fit. NoSQL maintenance is proposed there in #3395

Cf. https://lists.apache.org/thread/6hd7wjzhv3n2fl5qw5x4jh2vzk12mp1c

Sounds good.

rmannibucau · 2026-02-06T11:07:17Z

I need this data persisted for end to end auditing for both internal and external auditors (PII data).

sounds ok while not enabled by default for me, longer term - and personally - I'd prefer an even (on kafka or alike) with a default small consumer storing it in a custom backend to decouple it and potentially scale independently 100% for audit concerns, but not a blocker at all

obelix74 · 2026-02-06T14:26:19Z

I need this data persisted for end to end auditing for both internal and external auditors (PII data).

sounds ok while not enabled by default for me, longer term - and personally - I'd prefer an even (on kafka or alike) with a default small consumer storing it in a custom backend to decouple it and potentially scale independently 100% for audit concerns, but not a blocker at all

Yes, completely different schema. Turned off by default. You have to explicitly pass a flag to create these tables.

#3523

- Added unit tests for JdbcBootstrapUtils.shouldIncludeMetrics() method: - Test when schemaOptions is null (returns false) - Test when includeMetrics is true (returns true) - Test when includeMetrics is false (returns false) - Added integration tests in RelationalJdbcBootstrapCommandTest: - testBootstrapWithIncludeMetrics: verifies --include-metrics flag works - testBootstrapWithoutIncludeMetrics: verifies default behavior Per PR review comment from singhpk234

The schema-v4 version number fixes have been extracted to a separate branch (fix-schema-v4-version-number) for an independent PR. This reverts the version changes back to match main branch.

dimas-b · 2026-02-06T17:42:46Z

Just to sync up: The idea is to merge #3523 first, then this PR, right?

Per PR review feedback from dimas-b: Polaris supports external IdP and PDP (e.g. Keycloak and OPA), and the roles stored in metrics tables may not be aligned with AuthZ decisions. Removed principal_role_ids column from both scan_metrics_report and commit_metrics_report tables in H2 and PostgreSQL schemas.

dimas-b · 2026-02-06T17:49:12Z

...bc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java

+   *
+   * @param report the scan metrics report to persist
+   */
+  public void writeScanMetricsReport(@Nonnull ModelScanMetricsReport report) {


Why not keep these methods in JdbcMetricsPersistence? 🤔

Oh man, good idea. Did that in a commit. Thanks again.

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

Per PR review feedback from dimas-b: Polaris supports external IdP and PDP (e.g. Keycloak and OPA), and the roles stored in metrics tables may not be aligned with AuthZ decisions. Removed: - PRINCIPAL_ROLE_IDS column constant from model classes - getRoles() method from model interfaces - Roles parsing/serialization from fromResultSet() and toMap() - Roles-related tests from test classes

obelix74 · 2026-02-06T18:00:50Z

Just to sync up: The idea is to merge #3523 first, then this PR, right?

Of course. We should merge #3616 and #3523 first and this PR will shrink in size.

Addresses PR review comments from dimas-b: - r2775335558: Why not keep these methods in JdbcMetricsPersistence? - r2775346205: Do you plan wiring JdbcMetricsPersistence to CDI? Changes: - Move all metrics persistence methods from JdbcBasePersistenceImpl to JdbcMetricsPersistence, making it self-contained - Create JdbcMetricsPersistenceProducer as a CDI producer that creates JdbcMetricsPersistence instances for the relational-jdbc backend - Update ServiceProducers.metricsPersistence() to use Instance<MetricsPersistence> pattern to select the appropriate implementation based on persistence type - Update MetricsReportPersistenceTest to use JdbcMetricsPersistence directly

Address PR review comment - remove namespace since tableId uniquely identifies the table. Namespace can be derived from the table entity if needed. - Remove namespace column from H2 and Postgres schema-metrics-v1.sql - Remove namespace from Model classes (ModelScanMetricsReport, ModelCommitMetricsReport) - Remove namespace from converter classes - Update SpiModelConverter to not use namespace - Update PersistingMetricsReporter to not pass namespace - Update all related tests

dimas-b

Some more CDI comments. I'll make another review pass after the prerequisite PRs are merged and this one is rebased.... LGTM in general 👍

dimas-b · 2026-02-06T19:30:05Z

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

+    if (selected.isResolvable()) {
+      return selected.get();
+    }
+    return MetricsPersistence.NOOP;


I'd rather error out if the configured instance is not resolvable. We do that pretty much in all other cases.

NOOP could be selected with a default value in application.properties.

dimas-b · 2026-02-06T19:32:00Z

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

+  public MetricsPersistence metricsPersistence(
+      PersistenceConfiguration config, @Any Instance<MetricsPersistence> metricsPersistenceImpls) {
+    Instance<MetricsPersistence> selected =
+        metricsPersistenceImpls.select(Identifier.Literal.of(config.type()));


Would it make sense to use a different config for MetricsPersistence? We came some way to detach it from the JDBC metastore, I believe it would be nice to also make it selectable independently.

Created MetricsPersistenceConfiguration for this.

dimas-b · 2026-02-06T19:33:57Z

...rc/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java

                  .openInitScriptResource(effectiveSchemaVersion));
+
+          // Run the metrics schema script if requested
+          if (JdbcBootstrapUtils.shouldIncludeMetrics(bootstrapOptions)) {


Just to sync up and not forget: I believe it would be nice to have a separate bootstrap handler for the metrics schema, but that can be done as a follow-up PR.

Implemented in commit c23b162e5. Created a MetricsSchemaBootstrap SPI interface in polaris-core with bootstrap(realmId) and isBootstrapped(realmId) methods, plus a NOOP constant. The JDBC implementation (JdbcMetricsSchemaBootstrap) is idempotent and uses the metrics_version table to track bootstrap state.

Also added a new bootstrap-metrics CLI command that allows operators to bootstrap the metrics schema independently:

polaris-admin bootstrap-metrics -r my-realm

This enables adding metrics persistence support to existing deployments without re-bootstrapping the entity schema.

runtime/admin/src/main/java/org/apache/polaris/admintool/config/AdminToolProducers.java

@Identifier

- Create separate MetricsPersistenceConfiguration with polaris.persistence.metrics prefix and 'noop' default value (addresses r2775695727) - Add NoopMetricsPersistence CDI bean with @Identifier("noop") annotation - Update ServiceProducers.metricsPersistence() to error if type not resolvable instead of falling back to NOOP (addresses r2775688321) - Refactor AdminToolProducers: move UUID to dummyRealmContext() and have dummyRealmConfig() take RealmContext as parameter (addresses r2775725538)

dimas-b · 2026-02-06T23:32:05Z

...ime/service/src/main/java/org/apache/polaris/service/persistence/NoopMetricsPersistence.java

+ * @see MetricsPersistence#NOOP
+ */
+@ApplicationScoped
+@Identifier("noop")


optional: We can probably have a simple producer method in ServiceProducers having this annotation and returning MetricsPersistence.NOOP - seems simpler than a delegator class.

Done. Replaced the NoopMetricsPersistence delegator class with a simple producer method in ServiceProducers.

@beta

This commit implements a decoupled bootstrap handler for the metrics schema, following dimas-b's suggestion in PR review comment r2775702215. Changes: - Create MetricsSchemaBootstrap SPI interface in polaris-core - Defines bootstrap(realmId) and isBootstrapped(realmId) methods - Includes NOOP constant for backends that don't support metrics - Annotated with @beta since the API is experimental - Create JdbcMetricsSchemaBootstrap implementation in polaris-relational-jdbc - Executes schema-metrics-v1.sql for H2/PostgreSQL - Idempotent: checks metrics_version table before bootstrapping - Uses DatasourceOperations for database access - Create MetricsSchemaVersion model class for metrics_version table - Add generateMetricsVersionQuery() to QueryGenerator - Add CDI producers for MetricsSchemaBootstrap - JdbcMetricsPersistenceProducer: produces @Identifier("relational-jdbc") - ServiceProducers: produces @Identifier("noop") - Update JdbcMetaStoreManagerFactory to use injected MetricsSchemaBootstrap instead of inline metrics bootstrap logic - Add 'bootstrap-metrics' CLI command (BootstrapMetricsCommand) - Allows operators to bootstrap metrics schema independently - Supports multiple realms: -r realm1 -r realm2 - Idempotent: skips already-bootstrapped realms - Simplify NoopMetricsPersistence (per r2776432583) - Replace delegator class with simple producer method in ServiceProducers - Delete NoopMetricsPersistence.java This enables operators to add metrics persistence support to existing Polaris deployments without re-bootstrapping the entity schema.

obelix74 · 2026-02-07T02:36:49Z

.../org/apache/polaris/admintool/relational/jdbc/RelationalJdbcBootstrapMetricsCommandTest.java

+  // RelationalJdbcBootstrapCommandTest#testBootstrapFailsWhenAddingRealmWithDifferentSchemaVersion
+  //
+  // @Test
+  // public void testBootstrapMetricsIdempotent(QuarkusMainLauncher launcher) {


I really can't test state persistence across launches because of this.

github-project-automation bot added this to Basic Kanban Board Jan 8, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jan 8, 2026

dimas-b reviewed Jan 8, 2026

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsSessionTagsBuilder.java Outdated Show resolved Hide resolved

obelix74 commented Jan 8, 2026

View reviewed changes

obelix74 requested a review from dimas-b January 8, 2026 19:35

obelix74 force-pushed the feat-3337-rest-catalog-metrics-table-merged branch from 461cfde to 2fc2a20 Compare January 9, 2026 22:13

obelix74 force-pushed the feat-3337-rest-catalog-metrics-table-merged branch from 830085b to fcf2716 Compare January 11, 2026 02:40

obelix74 mentioned this pull request Jan 12, 2026

feat: Add trace_id to AWS STS session tags for end-to-end correlation #3414

Merged

6 tasks

dimas-b reviewed Jan 17, 2026

View reviewed changes

obelix74 marked this pull request as draft January 17, 2026 02:17

obelix74 force-pushed the feat-3337-rest-catalog-metrics-table-merged branch from 35bbf1e to 1954427 Compare January 17, 2026 03:33

obelix74 changed the title ~~(feat) Add metrics persistence with dual storage strategy for Iceberg table operations (fixes #3337)~~ (feat) Add metrics persistence for Iceberg table operations (fixes #3337) Jan 21, 2026

obelix74 changed the title ~~(feat) Add metrics persistence for Iceberg table operations (fixes #3337)~~ feat(persistence): Add database schema and persistence layer for Iceberg metrics reporting (#3337) Jan 21, 2026

obelix74 mentioned this pull request Jan 22, 2026

feat(metrics): Evolve PolarisMetricsReporter interface with timestamp parameter and comprehensive documentation #3468

Merged

6 tasks

obelix74 marked this pull request as ready for review January 23, 2026 16:58

dimas-b reviewed Jan 23, 2026

View reviewed changes

site/content/in-dev/unreleased/telemetry.md Outdated Show resolved Hide resolved

dimas-b requested a review from singhpk234 January 23, 2026 16:59

obelix74 requested a review from dimas-b January 23, 2026 17:13

dimas-b reviewed Jan 23, 2026

View reviewed changes

.gitignore Show resolved Hide resolved

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql Outdated Show resolved Hide resolved

obelix74 requested a review from dimas-b January 23, 2026 17:13

obelix74 mentioned this pull request Feb 6, 2026

feat(metrics): Add MetricsPersistence SPI for backend-agnostic metrics storage (#3337) #3616

Open

6 tasks

dimas-b mentioned this pull request Feb 6, 2026

(feat: persistence) Add schema-metrics-v1.sql for metrics tables (#3337) #3523

Open

6 tasks

Anand Kumar Sankaran added 6 commits February 5, 2026 19:35

Merge feat-3337-schema-v4: Denormalize principal roles into JSON array

f84d2b5

refactor: Remove metrics event types

41bb810

Remove BEFORE_REPORT_METRICS and AFTER_REPORT_METRICS event types as requested in PR review - events for metrics can be added in a separate PR if needed.

style: Apply spotless formatting

4dece9e

Anand Kumar Sankaran added 2 commits February 6, 2026 07:01

refactor: Remove schema-v4 version fixes (moved to separate PR)

eabdd15

The schema-v4 version number fixes have been extracted to a separate branch (fix-schema-v4-version-number) for an independent PR. This reverts the version changes back to match main branch.

dimas-b reviewed Feb 6, 2026

View reviewed changes

runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java Outdated Show resolved Hide resolved

obelix74 changed the title ~~feat(persistence): Add database schema and persistence layer for Iceberg metrics reporting (#3337)~~ feat(persistence): Add JDBC persistence layer for Iceberg metrics reporting (#3337) Feb 6, 2026

Anand Kumar Sankaran added 2 commits February 6, 2026 11:01

Fix failing tests

b2396da

dimas-b reviewed Feb 6, 2026

View reviewed changes

runtime/admin/src/main/java/org/apache/polaris/admintool/config/AdminToolProducers.java Outdated Show resolved Hide resolved

dimas-b reviewed Feb 6, 2026

View reviewed changes

Anand Kumar Sankaran added 2 commits February 6, 2026 17:23

Add unit tests for BootstrapMetricsCommand

4e1a2a6

obelix74 commented Feb 7, 2026

View reviewed changes

Conversation

obelix74 commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

obelix74 commented Jan 9, 2026

Uh oh!

obelix74 commented Jan 11, 2026

Uh oh!

obelix74 commented Jan 12, 2026

Uh oh!

obelix74 commented Jan 16, 2026

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b commented Jan 17, 2026

Uh oh!

obelix74 commented Jan 17, 2026

Uh oh!

obelix74 commented Jan 17, 2026

Uh oh!

obelix74 commented Jan 17, 2026

Uh oh!

Uh oh!

obelix74 commented Jan 23, 2026

Uh oh!

dimas-b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

obelix74 commented Feb 6, 2026

Uh oh!

rmannibucau commented Feb 6, 2026

Uh oh!

obelix74 commented Feb 6, 2026

Uh oh!

dimas-b commented Feb 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

obelix74 commented Feb 6, 2026

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

obelix74 commented Jan 8, 2026 •

edited

Loading

dimas-b left a comment •

edited

Loading