Feature request: persistent disk cache

[eenchevlp]

Is your feature request related to a problem? Please describe.
Currently the metrics can be served from cache utilizing the min_interval setting. However, on collector restart (for example in Kubernetes environment), the cache is essentially cleared as it is in-memory only and the initial scrapes put a lot of load on the target databases ("thundering herd" issue)

Describe the solution you'd like
Despite this issue could be worked around with some clever separation of service monitors and using non-overlapping intervals, it would be better if there was an option to have cache persisted on disk, so the restart is not so intrusive operation.

Of course, to keep feature-compatibility, it would be best if the disk cache is optional and not the default.

[burningalchemist]

Hey @eenchevlp, thank you for the request. Let me think about it, and I'll get back to you. 👍