Improve puma performance.

[TreyE]

After examination, we've discovered that Puma instance (i.e. user-facing web-UI) responsiveness suffers significantly when the system is under heavy asynchronous load (such as during redeterminations).

The reason behind this is that the event source AMQP subscribers are also started on the Puma webserver instances, instead of only on listener instances. This results in AMQP subscribers consuming messages, doing work, and 'stealing' resources from the Puma webserver threads on the same instance.

This update enables a DevOps engineer to opt-in to the creation of asynchronous AMQP subscribers by setting an environment variable ES_HOSTING_MODE to a value of listener. Only when this environment variable is explicitly set, and only when it is explicitly set to the value of listener, will AMQP subscribers be created.

Important to note is:

1. The default for running AMQP subscribers is now opt-in - tests and local docker environments will need to be updated to reflect this.
2. All applications which use event source now have access to this functionality, without further developer effort.

[polographer]

this would be awesome, an unwanted request/comment; have you thought about changing the name of the flag to be more generic? (instead of "web", or ad another mode like "console"); what I'm thinking is that "rails c" also initializes 'event source' and converts itself into another "worker" (it also takes more time to boot).

If we merge it as it is we can do it via ES_HOSTING_MODE=web rails c, but the wording looks a little off, I was thinking something more generic like ES_THREADS=false rails c or ES_CONSUMER=false rails c, that way we can reuse it in more places and look more generic

[TreyE]

this would be awesome, an unwanted request/comment; have you thought about changing the name of the flag to be more generic? (instead of "web", or ad another mode like "console"); what I'm thinking is that "rails c" also initializes 'event source' and converts itself into another "worker" (it also takes more time to boot).

If we merge it as it is we can do it via ES_HOSTING_MODE=web rails c, but the wording looks a little off, I was thinking something more generic like ES_THREADS=false rails c or ES_CONSUMER=false rails c, that way we can reuse it in more places and look more generic

Main thing here is we do want to start these in console and dev mode. When running Puma in production is really the only time we don't want to start them. Web seemed like the right name for me, but other than that we could call it whatever.

[TreyE]

I have updated the flag to be named 'publisher' as it is not only in the Puma or 'web' context that we desire this behaviour. We want it when legacy ACAPI listeners run, as well as an any kind of batch run.

[kristinmerbach]

LGTM

[saikumar9]

Adding a comment to just block the merge until we finalize the date we want to merge.

[TreyE]

I'm likely to close this and submit it in a different form. I don't care for how I've chosen nor am enforcing some of the defaults - I think it clashes with both the current implementation and expected deployment, development, and testing defaults of event source.