zstd compression is using a ton of CPU while we are creating workspaces

[spikecurtis]

During scale tests, when we create new workspaces, CPU utilization spikes up to our limit:

From Pyroscope:

In Coderd, zstd compression is used for the site handler:

	// Static file handler must be wrapped with HSTS handler if the
	// StrictTransportSecurityAge is set. We only need to set this header on
	// static files since it only affects browsers.
	r.NotFound(cspMW(compressHandler(httpmw.HSTS(api.SiteHandler, options.StrictTransportSecurityCfg))).ServeHTTP)

Annoyingly, whatever HTTP requests are being made, we are not recording the path in the coderd_api_requests_processed_total metric

[spikecurtis]

After talking to some Coder people, and modifying Coderd metrics to track the whole URL for static paths, it appears this is being driven by /bin/coder-linux-amd64.

We enable compression "on the fly" for the Site routes, including the binary downloads. When lots of workspaces start, they all need to download the Coder binary, which contains the agent. We should consider pre-compressing the binaries with supported compression algorithms.

It may be worthwhile to do this more generally for other static files, perhaps lazily as an on-disk compression cache.