Staging Handbook

This handbook provides all the information you need to know to manage and keep our staging environment healthy.

Infrastructure and Setup

• TODO Add conceptual diagram of the current runnable staging environment
• TODO add introduction

alpha-stage-data & alpha-stage-data-2

The docks cannot be run within the sandbox but require consistent access to specific data-stores that can. Unfortunately because these services are running on docks, which by our definition are ephemeral, we cannot currently ensure that certain services will retain their data and IP addresses over the long term.

To remedy this situation we have two auxiliary EC2 instance called alpha-stage-data and alpha-stage-data-2. These instances run the following services:

1. consul and vault - dock-init uses these, much easier to setup and maintain via Ansible at this time
2. redis - data store for sauron, dock service, requires TCP
3. rabbitmq - docker-listener etc., IP cannot switch
4. swarm-manager - This will probably be OK to push back into the sandbox soon, but keeping it outside until things stabilize (easier to debug, etc.)

The rest of this section will give specific instructions on how to setup the data instances from scratch via ansible.

TODO: clean this section up

Consul & Vault

1. From your local devops-scripts/ansible directory, run the following command: ansible-playbook -i stage-hosts/ consul.yml

2. The initial runs of this playbook with fail due to docker not being fully installed on the alpha-stage-data* docks. To proceed get to the point where it fails on alpha-stage-data. At this point restart that instance in EC2 and re-run the command with -e restart=true.

3. Now that consul is installed you will need to seed it with the data needed by dock-init, to do so run: ansible-playbook -i stage-hosts/ consul-values.yml -e write_values=yes

4. Next we must install vault, to do so run: ansible-playbook -i stage-hosts/ vault.yml

5. ssh alpha-stage-data

6. sudo docker exec -it $(sudo docker ps | grep 'vault' | awk '{print $1}') sh

7. vault init -address http://127.0.0.1:8200

8. Record the output from the init command and set the appropriate variables in devops-scripts/ansible/stage-hosts/varibles

9. ansible-playbook -i stage-hosts/ vault-values.yml -e write_root_creds=yes

10. ansible-playbook -i stage-hosts/ vault-values.yml -e write_values=yes

Redis & RabbitMQ

1. Run ansible-playbook -i stage-hosts/ redis.yml. This playbook includes the role for installing docker on the alpha-stage-data host. The first run of the playbook will intentionally fail, as the first pass needs to install packages that require a system reboot before actually installing docker and, eventually, creating a redis container.

2. In AWS reboot the alpha-stage-data host manually.

3. Once alpha-stage-data has rebooted run the following: ansible-playbook -i stage-hosts/ redis.yml -e restart=true. The -e restart=true indicates to the docker role that we are ready to install docker after a reboot, and the playbook should run successfully.

Swarm Manager

1. One and done: ansible-playbook -i stage-hosts/ swarm-manager.yml