lifecycle web endpoint for clearing the registry

[WojciechKuk]

there are endpoints /-/quit and /-/reload
I report the need for the endpoint /-/clear (or /-/reset)

This command would work differently than reload. It would simply clear the registers (as if the TTL of all metrics dropped to zero).

I work with large "clusters" of statsd_exporter, where when the number of instances changes, or their order changes, the metrics distribution key changes (hashing on statsrelay).
In such a case I need a way for all statad_exporters to immediately "forget" the metrics they have accumulated so far, clear the registers and start collecting metrics "in a new distribution".

• /-/reload doesn't provide me with this - metrics don't get forgotten and new ones start to accumulate mixed with old ones
• /-/quit doesn't provide this because it's more disruptive. You can of course put a command in the systemd configuration to quickly start the daemon, but it can take a few seconds and it can cause prometheus to mark the backend as "down". I just want to clean the registry, which would be faster (and non-destructive for self-monitoring metrics i.e golang-related)

Other use cases that may appear (and user may need to quickly clean the registry):

• a wrongly formatted metric accidentally sent to statsd, which needs to be quickly cleaned up so that prometheus does not download it
• a gauge that is a memento of a non-existent instance of the monitored service - overstating/understating some KPI
• a mistake in metric routing (e.g. a mistake in redirecting traffic to the wrong backend on the loadbalancer, in a situation where all backends are downloaded equally by prometheus)

[pedro-stanaka]

Why not just rollout restart the statefulset? Or are you planning to do this from some sidecar/other app?

[WojciechKuk]

Why not just rollout restart the statefulset?

some instances get metrics like 5xx and some others 2xx (hashing)

if you do a rollout restart, you will make two mistakes:

• because some instances will have high uptime and others will be just starting up, you will change the proportion of 5xx to 2xx metrics, which will generate false-positive alerts (for example, you will restart an instance on which there was some important metric 2xx and because of that there will be fewer 2xx metrics, and almost as many 5xx metrics as always - so suddenly the percentage of 5xx will increase from 0.01% to 10%)
• the restart itself takes a moment (e.g. a few seconds) and will definitely take much longer than the registry reset - some scrapings from prometheuses will end with a timeout

immediate reset of counters on all instances is the safest way not to disturb the "distributed" mathematics.

(in this case I'am using cluster of openstack VMs, not statefulset, but problem with counters/gauges/timers will be the same)

[pedro-stanaka]

Okay, got it.

But do you aggregate by instance? Feels like something is off with how you are querying this.

For example for a error rate SLO we would have an expression like this:

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

If there was a reset in a single instance nothing would change as long the data distribution continues the same (this is your server is operating well and is still sending 2xx and 5xx in the same proportion).

In my case I also work with large sets of statsd_exporters and when there is a change requiring rollout it will generate some inbalance on the software routing statsd requests to the exporter fleet (I am using ketama hashring to route), nevertheless it will come back to normality after a few minutes and the first cycle of TTL kicks in.

I would defer to the other maintainers in the project here, but I don't feel like this is needed. If you want you can try putting up a PR so we can discuss on it. Or wait until they weigh in here.

[pedro-stanaka]

Maybe do it under a feature-flag to avoid the breaking behavior, like Matthias suggested here:

#471 (comment)

[WojciechKuk]

But do you aggregate by instance?

no. by many other dimensions. instances are dropped (millions of mobile phones in my case)

I just need to quickly reset counters without access to shell. It will be less invasive than /quit or /reload.

ps. it can be additional parameter to /reload endpoint.

[pedro-stanaka]

no. by many other dimensions. instances are dropped (millions of mobile phones in my case)

I meant statsd_exporter instance, because only that would make it trigger a false-positive.

ps. it can be additional parameter to /reload endpoint.

I think it should be a behavior controlled by a flag on start, Prometheus http endpoints usually dont take extra parameters.

I don't have bandwidth to do this now, nor I need this anymore since I am using native histograms + created timestamp now. So if you want, feel free to put up a PR. 👍