Skip to content

[fix] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum#22

Open
dlg99 wants to merge 1 commit intomasterfrom
storage-metrics
Open

[fix] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum#22
dlg99 wants to merge 1 commit intomasterfrom
storage-metrics

Conversation

@dlg99
Copy link
Owner

@dlg99 dlg99 commented Mar 4, 2025

Motivation

I think pulsar_storage_size is partially double-counted. Currently
pulsar_storage_size = pulsar_storage_logical_size * number of replicas

Number of replicas is the same as the bookie write quorum.
pulsar_logical_size doesn't account for the replication, it is sum of customer bytes no matter where they stored at.
and it looks like pulsar_storage_logical_size (calculated here initially) DOES include pulsar_storage_offloaded_size (calculated here).
But the offloaded ledgers (e.g. S3) do not create multiple replicas.

I think the metric should be calculated as:
pulsar_storage_size = pulsar_storage_offloaded_size + (pulsar_storage_logical_size - pulsar_storage_offloaded_size) * number of replicas

Modifications

adjusted calculations of pulsar_storage_size

Verifying this change

  • Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@dlg99 dlg99 changed the title Corrected total storage metric to not multiply offloaded storage by the write quorum [fix] Corrected total storage metric to not multiply offloaded storage by the write quorum Mar 4, 2025
@dlg99 dlg99 changed the title [fix] Corrected total storage metric to not multiply offloaded storage by the write quorum [fix] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant