How to use Custom Collectors for Histograms

[cunningr]

When using a histogram metrics we can use the 'observe' method to evaluate which bucket our latest sample will fall in:

E.g.
h1.labels('device-1').observe(value.get('latest_sample'))
->

        for i, bound in enumerate(self._upper_bounds):
            if amount <= bound:
                self._buckets[i].inc(1)
                break

However as I am proxying metrics I believe I should be using a Customer Collector and the HistogramMetricFamily which only provides add_metric which seems to just populate all the buckets.

am I supposed to calculate the buckets in my code before writing the metric?

[mcbobke]

I have the same question. @brian-brazil is this how we are expected to build a Histogram in a custom collector?

[mcbobke]

Experimenting with this a bit, I was able to write a custom collector that utilizes the regular Histogram and not the HistogramMetricFamily with the following steps. All of this takes place in the Collect method of your custom collector.

1. Start with an empty list of metrics like metrics=[].
2. Create a Histogram with registry=None so that it is not registered to the default REGISTRY.
3. Add observations to the Histogram as you would outside of a custom collector.
4. Extend the metrics list with the return from histogram.collect().
5. Return the metrics list from the Collect method.

Example:

class CustomCollector(object):
    def __init__(self, namespace: str):
        self._namespace = namespace

    def collect(self):
        metrics = []

        hist = Histogram(
            name='http_response_time',
            documentation='HTTP response time',
            namespace=self._namespace,
            registry=None,
            labelnames=('method')
        )
        hist.labels('GET').observe(200)
        
        metrics.extend(hist.collect())
        return metrics

In your project's initialization code, ensure that the custom collector is registered.

I'm certain this is not supported, but implementing the Histogram's logic on my own sounds dreadful.

[brian-brazil]

That's how I'd do it, keep in mind that you possibly want to switch the type to GaugeHistogram before sending it on if you're using a custom collector like this.

[mcbobke]

Nice, glad I am on the right track. What is the benefit of using the GaugeHistogram?

[brian-brazil]

It's a type thing, so may be used by downstream tooling for better hints etc.. If the data is based on counters then it's a Histogram. If it's a snapshot of current state it's a GaugeHistogram.

[mcbobke]

Interesting, I've never seen a GaugeHistogram being used anywhere else. The use case I have is for buckets that increment (as with classic Histograms) so I think I'm safe to keep the current type, but is there any existing documentation or future plans to include documentation for this type? And maybe for other types like StatefulSet which I've never seen before looking through the source.

Appreciate the help!

[mcbobke]

Also @cunningr to keep a running count of the metrics that you're proxying (in the case of a classic Histogram), it wouldn't be a bad idea to use an instance variable with locking in the Collect method to ensure that multiple scrapes don't step on each other. In my example, I'm using gevent but you should use whatever library is appropriate to your project.

[brian-brazil]

You should create a new Histogram for each collect, otherwise you're risking race conditions.

[mcbobke]

Ah yes, my mistake. It's the job of Prometheus to collect the scrapes and store them within the Histogram. :)

[mcbobke]

Well.. actually, am I correct about that? For example, I have a Counter that is tracking the number of times a specific operation is performed. That counter is instrumenting this exporter that I'm writing, and therefore is not recreated on each scrape.

How do I ensure that this Histogram from the Custom Collector exposes buckets that are potentially incrementing on each /metrics call with new data from the external API?

[cunningr]

In that case, I guess you’re not proxying the data anymore? My use case is for reading counters from another system/format and exposing those as Prometheus metrics.

…

On Wed, 1 Sep 2021 at 20:54, Matthew Bobke ***@***.***> wrote: Well.. actually, am I correct about that? For example, I have a Counter that is tracking the number of times a specific operation is performed. That counter is instrumenting this exporter that I'm writing, and therefore is not recreated on each scrape. How do I ensure that a Histogram exposes buckets that are potentially incrementing on each /metrics call with new data from the external API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#683 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHDBWJZHJRVNDO2WAGMPGNTT72AHXANCNFSM5APUTDWQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[brian-brazil]

In that case, manually putting together the Histogram's samples is what I'd do.