Operational infrastructure + ongoing maintenance

[joekarl]

Just wanted to start a thread on various operational issues that will arise once we have working apps, especially future larger projects. Obviously we're not there yet, but these are things to think about.

Questions that we need to think about:

• Infrastructure
  • Hosting? Do we need to avoid hosted / vps (ie shoot for static apps)
    • If we do run things on heroku / aws / do / etc, who manages that?
  • Do we have specific people be point of contact for specific apps or people in charge of levels of infrastructure? (ie person in charge of databases, person in charge of APIs, etc...)
  • Do we need to make a repo to detail infrastructure that we are using / have available?
  • Who handles things like hostnames / DNS? (route53?)
  • SSL certs (<- pretty much non-negotiable these days)
  • Costs, what is deemed as acceptable cost wise? Do we have a number that we have to shoot for to not have to shutter apps?
• Monitoring
  • Do we monitor apps for availability / performance / usage?
  • How? canary? new relic (free for heroku accounts)? something else?
  • Automated email/notification based on
  • Who gets contacted when things are borked?
• App Maintenance
  • Designated persons who are maintainers of specific apps?
  • Costs, how do we keep costs low once in maintenance mode?

Anyways, not a comprehensive list by any means but just wanted to throw out some things that will need to be resolved.

[gorsuch]

re: servers / hosting / etc

I haven't been able to make the meetings as of yet, but would like to note that if you are considering running apps, Heroku is likely to be a reasonable bet - especially for things you are prototyping and trying to validate.

If you could operate within the free tier, many of the above requirements would be met - the ssl-enabled herokuapp.com sub-domains that are offered up could probably cover you out the gate.

But, you all don't need me to tell you that managing servers is not fun. Or, they kind of are, until the server crashes and you have to restore... ;)

[mkchandler]

Great questions @joekarl! I'll try to answer as many as I can, but some are still a little unknown while we are in the early stages of becoming an official Code for America Brigade.

• Hosting
  • I would like to aim for static when possible (performance, hosting, and other things make it nice and easy.) That being said, server side frameworks are needed too and that will be fine. For hosting, members of the core team as well as project leads would manage that.
  • There will be one or more project leads assigned to each app/project. We will determine responsibilities based on the needs of each project. Jeremy (as Delivery Lead) will be the point person for all projects (or another core team member if needed.)
  • The readme of each repo should detail the specifics of how to get each app up and going. This will make it easy for other brigades to fork or setup our projects.
  • A member of the core team will manage host names / DNS.
  • Yes for https on everything if possible.
  • All costs will be covered by Code for OKC. While we don't have an operating budget at the moment, it is something that is actively being worked on and we will have one from CfA once we become an official Code for America Brigade.
• Monitoring
  • Monitoring is something I think would definitely be a good idea. This will be decided on a per-project basis and we will have the core team members as well as the project lead(s) in the loop on this.
  • A central monitoring site that covers all our apps would be pretty sweet :-)
• App Maintenance
  • The project lead(s) will continue to be in charge of the app once in maintenance mode. The group leaders will also be responsible for making sure all apps are maintained.

With all that being said, I also like the idea of using services like Heroku so that we can focus on the product and not as much the infrastructure. But there will probably be a bit of both in reality. We will hopefully be able to get credits for most of the services also, as a lot of them are sponsors of CfA (like Heroku.) I am still gathering more info in that area.

[joekarl]

Sounds like we're probably up a creek with SSL on GH pages (see codeforokc/school-finder#4 (comment)), may need to seriously consider S3 for hosting static data/sites...

[mkchandler]

I'm on board with using S3. I will work with @jagthedrummer and we will get everything set up under a Code for OKC account.

[mattmar10]

When some of this stuff gets ironed out, it might be worthwhile to document how everything is put together so that folks can easily get on board.

I too agree with @mkchandler, the less we have to focus on infrastructure, the better the product(s) will be.