How to generate metrics across replicas possibly on different machines

[MatthewScholefield]

I'm trying to figure out if I can generate Prometheus metrics for a service I scale with multiple replicas, each running in isolation (ie. within a docker container). Is this possible?

I know I can just add multiple scrape endpoints within a metric source in Prometheus's config, but I'm unsure if this would introduce errors when multiple machines emit gauge values for the same metric. I'm assuming it might work if I follow the steps mentioned for multiprocess mode, but I had the following concerns:

1. Will multiprocess mode work even across machines (or does, for instance, use pids to check for liveness)?
2. Does the PROMETHEUS_MULTIPROC_DIR folder need to be shared across machines (ie. a mounted volume)?

High level, I feel like it's a common situation to want to emit metrics from a stateless replicated web service so if there's a completely different way most people do this I'd be happy to hear.