Testing Farm Infrastructure

Table of Contents

• Setup
• Make
• Terragrunt and Terraform
• Development Environment
• Staging Environment
• GitHub Environment
  • GitHub Runners
  • Adding a New Organization
  • Adding a New Repository
  • Development and Testing
• Integration Tests
  • Worker integration tests
• Updating composes

This repository contains tools, configuration ands test used to maintain and setup Testing Farm’s infrastrucutre.

The repository tries to bundle all required tools and provides an easy way how to set everything up for newcomers.

Setup

The repository is managed via direnv. For setup:

• Add .vault_pass file with the known secret

  Tip	Check Bitwarden for the vault password.

  echo "SECRET_PASSWORD" > .vault_pass

• Install direnv

  $ dnf -y install direnv

• Hook direnv to your shell, see the official guide for more information.

• Install required system dependencies

  Warning

  Only Fedora system is supported at this time.

  make install/fedora

• Allow direnv in this directory

  $ direnv allow

  Tip	In case of errors, investigate the setup log in .direnv/envrc.log. You can try to reinstall the environment by removing the .direnv directory and running direnv allow.

• Check if kubectl works and lists all available clusters:

  $ kubectl ctx

Make

We use make to manage common actions. Run the command to see the available targets with descriptions.

Terragrunt and Terraform

We use Terraform to setup the infrastructure. To make the Terraform code DRY we use Terragrunt.

The infrastructure code is located in the terragrunt/ directory.

Development Environment

Development environment is intended for development of infrastructure and testing changes without influencing any other environment. The state of the infrastructure is saved locally, i.e. it uses the local terraform backend.

Currently, the environment contains:

• AWS EKS kubernetes cluster

• Artemis instance

The name of the cluster and Artemis API endpoint is generated according to your local username.

For example, for the user mvadkert:

• the cluster name is testing-farm-dev-mvadkert

• the Artemis API endpoint is artemis.dev-mvadkert.testing-farm.io

To provision a complete infrastructure on Fedora AWS for development:

• initialize the dev environment

  make dev/init

• create development environment with your own AWS EKS cluster and all available services in development mode

  make dev/apply

• develop & have fun

• tier down the infrastructure once happy

  make dev/destroy

Staging Environment

The staging environment serves as a preview of our production environment. The state of the infrastructure is saved to Terraform Cloud.

Currently, the environment contains:

• AWS EKS kubernetes cluster

• Artemis instance

Other components are not yet maintained here.

Details of the deployment:

• the cluster name is testing-farm-staging

• the Artemis API endpoint is artemis.dev-mvadkert.testing-farm.io

GitHub Environment

We use this environment to manage various GitHub related infrastructure.

GitHub Runners

We optionally provide GitHub runners for the users using Testing Farm reservations for their testing workflows on GitHub. The runners are required to be able to easily provide access to Testing Farm provisioned guests directly from the runners. The runners IP addresses are injected to the production Artemis deployment because of this. The runners are looked up in AWS based on their tags.

The configured runners can be found under terragrunt/environments/github. The structure of the directories follows the following naming scheme:

terragrunt/environments/github/<ORG>/runners/repositories/<REPOSITORY>

So for example for the securesign GitHub organization, the runner infrastructure code for the repository artifact-signer-ansible can be found under terragrunt/environments/github/securesign/runners/repositories/artifact-signer-ansible.

The other important directories and files are:

• terragrunt/environments/github/<ORG>/runners/components - Contains additional components of the runners, e.g. the SSH key pair, the security group of the runner instance, etc.

• terragrunt/environments/github/<ORG>/runners/user_data.sh.tfpl - A templated shell script run at the first boot, used to setup the runner, so it can function as needed.

The state file is managed in Terraform Cloud.

Adding a New Organization

• Create a copy of link:`terragrunt/environments/github/<ORG>/runners` directory for the new organization.

• Continue with Adding a New Repository.

Adding a New Repository

• Ask the responsible team to create a token which has permissions to retrieve runner registration tokens for the specific repository.

• Store the registration token in credentials file under the credentials.github.<ORG>.runners.<REPOSITORY>.registration_token key.

• Create a copy of the link:`terragrunt/environments/github/<ORG>/runners/<REPOSITORY>` directory, or rename it if you added a new organization.

• Adjust the terragrunt.hcl files if needed, e.g. you can adjust the instance type, disk size, etc. based on the user requirements,

• Submit an MR with the changes.

• The runner will be automatically provisioned and connected to the GitHub Actions if the MR is merged.

Development and Testing

You can use the testing-farm organization and test-runner for development and testing. The organization is not managed automatically, it is excluded from execution using terragrunt/environments/github/.terragrunt-excludes file.

Integration Tests

Worker integration tests

We currently support running worker integration tests against the Development Environment or Staging Environment.

• to run all worker tests against your local Development Environment run make test/dev/worker

• to run only guest-setup worker tests against your local Development Environment run make test/dev/guest-setup

• to run all worker tests against Staging Environment run make test/staging/worker

• to run only guest-setup worker tests against Staging Environment run make test/staging/guest-setup

Updating composes

To update composes on Public Ranch:

• run make compose-update-public

• submit a MR with the changes

• the changes will be deployed once the MR is merged