CALDERA Security Regression Pipeline

This project was created to provide an example of a TTP Runner and accompanying Security Regression Pipeline (SRP) for vulnerabilities that were discovered in MITRE CALDERA by Jayson Grace from Meta's Purple Team.

The attacks that are automated using the TTP Runner are run regularly against a fresh test environment with the latest MITRE CALDERA on a weekly basis using Github Actions. Because patches have been created for all of the discovered vulnerabilities, these attacks are expected to fail.

If any of the attacks land successfully during one of these runs, an issue is automatically created noting the regression.

Ideally this should be run as part of a CALDERA IaC deployment pipeline to gate commits. However, it can also be used as a standalone tool for Purple Team engagements, pentests, etc. that include CALDERA in the scope.

---

Table of Contents

• Getting Started
  • Test Environment Preparation
• Execution
  • Execute TTP Runner in SRP
  • Execute TTP Runner Locally

---

Getting Started

Test Environment Preparation

• Run this command if on an ARM-based macOS system:

  export ARCH="$(uname -a | awk '{ print $NF }')"
  if [[ $ARCH == "arm64" ]]; then
      export DOCKER_DEFAULT_PLATFORM=linux/amd64
  fi

• Download the latest caldera-security-tests release from github or run this:

  export ARCH="$(uname -a | awk '{ print $NF }')"
  export OS="$(uname | python3 -c 'print(open(0).read().lower().strip())')"
  gh release download -p "*${OS}_${ARCH}.tar.gz"
  tar -xvf *tar.gz

• Clone the caldera repo:

  # From the caldera-security-tests repo root
  pushd ../ && git clone https://github.com/mitre/caldera.git && popd

---

Execution

Execute TTP Runner in SRP

You can incorporate the CALDERA SRP into your CALDERA fork by creating .github/workflows/srp.yaml and populating it with the following contents:

name: CALDERA Security Regression Pipeline
on:
  pull_request:
  push:
    branches: [master]

  # Run once a week (see https://crontab.guru)
  schedule:
    - cron: "0 0 * * 0"

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  tests:
    uses: fbsamples/caldera-security-tests/.github/workflows/srp.yaml@main

The outcomes of these workflow runs can be used to gate updates for your CALDERA deployments if a security regression is detected in the latest CALDERA release.

Execute TTP Runner Locally

Create vulnerable test environment, run the first XSS, and tear the test environment down:

./caldera-security-tests testEnv -v
./caldera-security-tests storedXSSUno
./caldera-security-tests testEnv -d

Create vulnerable test environment, run the second XSS, and tear the test environment down:

./caldera-security-tests testEnv -v
./caldera-security-tests storedXSSDos
./caldera-security-tests testEnv -d

Create vulnerable test environment, run the third XSS, and tear the test environment down:

./caldera-security-tests testEnv -v
./caldera-security-tests storedXSSTres
./caldera-security-tests testEnv -d

Create test environment using the most recent commit to the default CALDERA branch, try running all attacks, and tear the test environment down:

./caldera-security-tests testEnv -r
./caldera-security-tests storedXSSUno
./caldera-security-tests storedXSSDos
./caldera-security-tests storedXSSTres
./caldera-security-tests testEnv -d

Parameters for the tests can be modified in the generated config/config.yaml file. This file is created as soon as the testEnv command in the above example is run.