Add metric for command processor latency #1529

Hazel-Datastax · 2024-10-11T00:52:36Z

What this PR does:
Add metric for command processor latency

Which issue(s) this PR fixes:
Fixes #1501

Checklist

Changes manually tested
Automated Tests added/updated
Documentation added/updated
CLA Signed: DataStax CLA

Hazel-Datastax · 2024-10-14T16:21:27Z

The metrics will be like:

# HELP command_processor_latency_seconds  
# TYPE command_processor_latency_seconds histogram
command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.5",} 1.6777216
command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.9",} 6.106906624
command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.95",} 6.106906624
command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.99",} 6.106906624
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001048576",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001398101",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001747626",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.002097151",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.002446676",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.002796201",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.003145726",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.003495251",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.003844776",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.004194304",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.005592405",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.006990506",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.008388607",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.009786708",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.011184809",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.01258291",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.013981011",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.015379112",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.016777216",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.022369621",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.027962026",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.033554431",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.039146836",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.044739241",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.050331646",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.055924051",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.061516456",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.067108864",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.089478485",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.111848106",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.134217727",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.156587348",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.178956969",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.20132659",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.223696211",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.246065832",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.268435456",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.357913941",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.447392426",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.536870911",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.626349396",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.715827881",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.805306366",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.894784851",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.984263336",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="1.073741824",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="1.431655765",} 0.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="1.789569706",} 1.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="2.147483647",} 1.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="2.505397588",} 1.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="2.863311529",} 2.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="3.22122547",} 3.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="3.579139411",} 3.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="3.937053352",} 3.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="4.294967296",} 3.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="5.726623061",} 3.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="7.158278826",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="8.589934591",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="10.021590356",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="11.453246121",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="12.884901886",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="14.316557651",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="15.748213416",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="17.179869184",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="22.906492245",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="28.633115306",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="30.0",} 4.0
command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="+Inf",} 4.0
command_processor_latency_seconds_count{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",} 4.0
command_processor_latency_seconds_sum{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",} 13.416749582
command_processor_latency_seconds{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.5",} 0.0
command_processor_latency_seconds{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.9",} 0.0
command_processor_latency_seconds{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.95",} 0.0
command_processor_latency_seconds{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.99",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001048576",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001398101",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.001747626",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.002097151",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.002446676",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.002796201",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.003145726",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.003495251",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.003844776",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.004194304",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.005592405",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.006990506",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.008388607",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.009786708",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.011184809",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.01258291",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.013981011",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.015379112",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.016777216",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.022369621",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.027962026",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.033554431",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.039146836",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.044739241",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.050331646",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.055924051",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.061516456",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.067108864",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.089478485",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.111848106",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.134217727",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.156587348",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.178956969",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.20132659",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.223696211",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.246065832",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.268435456",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.357913941",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.447392426",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.536870911",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.626349396",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.715827881",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.805306366",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.894784851",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="0.984263336",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="1.073741824",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="1.431655765",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="1.789569706",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="2.147483647",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="2.505397588",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="2.863311529",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="3.22122547",} 0.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="3.579139411",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="3.937053352",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="4.294967296",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="5.726623061",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="7.158278826",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="8.589934591",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="10.021590356",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="11.453246121",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="12.884901886",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="14.316557651",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="15.748213416",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="17.179869184",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="22.906492245",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="28.633115306",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="30.0",} 1.0
command_processor_latency_seconds_bucket{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",le="+Inf",} 1.0
command_processor_latency_seconds_count{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",} 1.0
command_processor_latency_seconds_sum{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",} 3.513473
# HELP command_processor_latency_seconds_max  
# TYPE command_processor_latency_seconds_max gauge
command_processor_latency_seconds_max{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",} 6.107748333
command_processor_latency_seconds_max{command="CreateCollectionCommand",module="sgv2-jsonapi",tenant="unknown",} 3.513473

Hazel-Datastax · 2024-10-14T23:58:09Z

src/main/java/io/stargate/sgv2/jsonapi/service/processor/MeteredCommandProcessor.java

+        if (id.getName().startsWith(jsonApiMetricsConfig.commandProcessorLatencyMetrics())) {
+          return DistributionStatisticConfig.builder()
+              .percentiles(0.5, 0.90, 0.95, 0.99)
+              .percentilesHistogram(true)
+              .minimumExpectedValue(
+                  TimeUnit.MILLISECONDS.toNanos(
+                      jsonApiMetricsConfig.MinExpectedCommandProcessorLatency())) // 0.1 seconds
+              .maximumExpectedValue(
+                  TimeUnit.MILLISECONDS.toNanos(
+                      jsonApiMetricsConfig.MaxExpectedCommandProcessorLatency())) // 15 seconds
+              .build()
+              .merge(config);
+        }


To address the potential high cardinality issue in Grafana, I adjusted the bucket range by increasing the minimum value (from 0.001 sec to 0.1 sec) and decreasing the maximum value (from 30 sec to 15 sec). This change reduces the number of buckets from 69 to 36.

While this makes sense wrt cardinality, 0.1 secs is 100 msec which is quite coarse. This might be ok when trying to find slowest queries, but we need to make sure it is useful for us in general. Could see what using 0.05 (for example) would give for number of buckets.

Changed to 0.05 and there are 41 buckets. The metrics will be like:

# HELP command_processor_latency_seconds # TYPE command_processor_latency_seconds histogram command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.5",} 0.012058624 command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.9",} 0.200802304 command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.95",} 0.200802304 command_processor_latency_seconds{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",quantile="0.99",} 0.200802304 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.05",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.050331646",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.055924051",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.061516456",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.067108864",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.089478485",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.111848106",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.134217727",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.156587348",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.178956969",} 3.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.20132659",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.223696211",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.246065832",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.268435456",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.357913941",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.447392426",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.536870911",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.626349396",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.715827881",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.805306366",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.894784851",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="0.984263336",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="1.073741824",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="1.431655765",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="1.789569706",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="2.147483647",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="2.505397588",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="2.863311529",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="3.22122547",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="3.579139411",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="3.937053352",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="4.294967296",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="5.726623061",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="7.158278826",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="8.589934591",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="10.021590356",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="11.453246121",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="12.884901886",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="14.316557651",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="15.0",} 4.0 command_processor_latency_seconds_bucket{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",le="+Inf",} 4.0 command_processor_latency_seconds_count{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",} 4.0 command_processor_latency_seconds_sum{command="InsertOneCommand",module="sgv2-jsonapi",tenant="unknown",} 0.253723748

The numbers are from my local laptop so it's not accurate, will deploy this to dev to see if 0.05 makes sense and if Grafana could work

Hazel-Datastax · 2024-10-22T23:34:53Z

This is the latency data from dev kube 540. We need to adjust the bucket ranges accordingly.

Hazel-Datastax added 2 commits October 10, 2024 20:31

Add one IT for binary vector support

86b65a9

Add test

36b3c84

Hazel-Datastax requested a review from a team as a code owner October 11, 2024 00:52

Hazel-Datastax added 2 commits October 11, 2024 11:59

Merge branch 'main' into hazel/command_processor_latency

430c1b2

Merge branch 'main' into hazel/command_processor_latency

2d9056f

Hazel-Datastax added 3 commits October 14, 2024 19:32

reduce the number of bucket

ce13c3c

remove the hard coded value

5b7b158

Merge branch 'main' into hazel/command_processor_latency

88273a8

Hazel-Datastax commented Oct 14, 2024

View reviewed changes

Hazel-Datastax and others added 5 commits October 14, 2024 20:04

fix

65ce427

Merge branch 'main' into hazel/command_processor_latency

8d75ff1

Update the min

7244c62

Merge branch 'main' into hazel/command_processor_latency

5ff5366

Merge branch 'main' into hazel/command_processor_latency

40a53e1

Hazel-Datastax and others added 3 commits October 22, 2024 19:34

Merge branch 'main' into hazel/command_processor_latency

0ccdc73

Merge branch 'main' into hazel/command_processor_latency

232fe91

Merge branch 'main' into hazel/command_processor_latency

cfb7e04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add metric for command processor latency #1529

Add metric for command processor latency #1529

Uh oh!

Hazel-Datastax commented Oct 11, 2024

Uh oh!

Hazel-Datastax commented Oct 14, 2024

Uh oh!

Hazel-Datastax Oct 14, 2024

Uh oh!

tatu-at-datastax Oct 15, 2024

Uh oh!

Hazel-Datastax Oct 15, 2024

Uh oh!

Hazel-Datastax commented Oct 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add metric for command processor latency #1529

Are you sure you want to change the base?

Add metric for command processor latency #1529

Uh oh!

Conversation

Hazel-Datastax commented Oct 11, 2024

Uh oh!

Hazel-Datastax commented Oct 14, 2024

Uh oh!

Hazel-Datastax Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

tatu-at-datastax Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

Hazel-Datastax Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

Hazel-Datastax commented Oct 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants