publishing game event notifications to SNS

[danlangford]

I would like to discuss the possibility to publishing some game actions to a notification service like a SNS topic.

If this has already been discussed please advise.

The code here is the beginning of an idea and a demonstration that I may be able to find my way around the PHP enough to help implement this.

VALUE

by publishing some game actions to a SNS topic that would allow other apps/systems to subscribe to the topics and react accordingly. Subscriptions could be HTTP, SQS, or Lambda all of which would be managed outside of the buttonweavers code base.

• MAINLY scripts would not have to call/poll the api as frequently while still providing quick responses
• the community could possibly create more supporting functionality without imposing much more traffic or code demands on the development team
• some site feature requests may be able to be easily solved outside of the current codebase. for example if somebody wanted to create a simple Lambda that emailed you when a UBFC match was waiting for your turn that would be possible.
• this also opens the doors to emitting more events of various kinds to lend towards various automation.

This mainly started as an idea while working on BMAI. I want BMAI to be able to respond to matches quickly but i dont want to be polling all day long if there is nobody needing engagement with BMAI. Mainly to save traffic and resources on the webserver. If notifications get out to SNS then i could actually move BMAI to run on demand as needed as a Lambda and the not be running when not needed.

What is required from an AWS point of view? Well there are 2 - 3 primary resources.

• Topic <- could be owned by non-buttonmen-dev but the site would then be dependent on an outside entity. i would suggest at least this is owned by buttonmen-dev group. 1st million messages a month are free. 50 cents per mission after that.
• Topic Subscription <- this could be owned by the Topic Owner (if you want control) or the message consumer (if you want freedom to make adjustments).
• Destination like an HTTP Endpoint, SQS Queue, Lambda <- this you want owned and paid for by the consumer. Me in the case of BMAI

Imagine a future scenario where all the button stats are updated in real time without taxing the current systems or current DB because enough information is published over SNS for that to be cataloged separately.

Imagine somebody wanting to make a deeper integration with Challonge brackets or a ranked leaderboard that could respond to events in near real time without increasing traffic to your site.

Is there any appetite to continue this conversation?

MARKED AS DRAFT PR BECAUSE THIS IS NOT READY TO MERGE, JUST READY TO DISCUSS.

[cgolubi1]

Is there any appetite to continue this conversation?

Sure! Thanks for speccing out ideas about the resources and the costs. The major thing i'm going to be looking at is how do we DTRT easily and cheaply in terms of pre-production sites and infrastructure/deployment support. More about this (and i'll actually look at your draft, which i don't have time to do this second) later, but i think what we wound up doing when we moved the production site database to RDS to solve a time-sensitive problem is probably illustrative. Basically what we wound up building there was:

• functionality in the codebase (both the vagrant/puppet code for populating the databases, and the PHP code for connecting to the DBs) to support either using an RDS database or a local one --- so that we didn't have to pay the cost of a new RDS DB for every dev site or local virtualbox-based site, so that the default for any new site is local mysql (zero moving parts to stand up a new test site with the cheaper configuration), and so that any site (RDS or local DB) can be deployed to without breaking its database connectivity
• dedicated (and separate!) RDSes for production and for staging --- we choose to pay the cost so that staging is as similar as possible to production, and we get at least one pre-prod test of any changes that might break RDS access
• support for dev sites which uses the functionality --- we need this initially while we're building it, and we can keep it later or not, depending on the relative expense of standing up a new one vs paying for the always-on test one

The tradeoff of automation costs vs deployment-time costs vs cost costs dictates exactly how we want to do all of those things, but i think those are the three things we need for a solution to be operational enough for our current purposes.

[danlangford]

do you build out another dev lane by hand? or us an automation tool to not only populate but also CREATE and DESTROY the RDS DB?

i have used cloudformation, terraform, and ansible to build out consistent infrastructure.

i think the biggest thing i'll need to help consider is if we attempt on local development to 1) no-op, 2) run a local fake SNS compliant API minus all the actual SNS features (dev to the interface not the functionality), or 3) use 1 single shared SNS topic amongst all the devs devving locally. most corporate teams i am familiar with do a variant of (3) but i still need to consider the merits.

[blackshadowshade]

On a totally unrelated note, it's good to see CircleCI running automatically!