Integration tests (#3417)

* run update_upstream_versions in tests

* gather feedstocks in tests + make graph

* update upstream versions in tests

* add other make-graph step

* enable debugging, deploy, don't update pinnings every time

* refactor and fix env vars

* add pydantic feedstock as a resource

* add IntegrationTestHelper supporting overwriting feedstock contents

* relock, fix install bot code

* fix subprocess call

* set git initial branch

* fix tests

* set proxy in update-upstream-versions

* global router + overwrite cf-graph repo

* reorder steps

* fix install bot code

* disable random skipping in integration tests

* use correct env name for container image

* (todo) don't overwrite container image env variables

* reorder action steps

* fix

* don't create pinning repo

* debug

* fix

* debug

* debug

* also capture stderr

* use older version for pydantic

* use lowercase proxy

* set DEPLOY_REPO correctly

* tmate action

* dont use proxy for tmate

* fix

* fix proxy in container

* fix

* git CLI: don't use proxy

* add make-migrators step

* fix

* relock

* don't use with sensitive_env

* fix

* fix

* make GitHub API transparent

* install_bot_code does not need BOT_TOKEN

* separate bot user token and regro org token

* apply fixes

* relock

* fix token

* fix URL request handler

* fix URL request handler

* Revert "Merge branch 'separate-bot-tokens' into integration-tests"

This reverts commit 63fec86, reversing
changes made to 56f9caa.

* Revert "Merge branch 'separate-bot-tokens' into integration-tests"

This reverts commit 56f9caa, reversing
changes made to d1f213a.

* use classic PAT

* remove tmate

* use TEST_SETUP_TOKEN

* fix: forward headers

* fix proxy

* make PyPI tarfiles transparent

* route transparent URLs directly by mitmproxy, configure conda-forge org

* use conda-forge-bot-staging everywhere

* fix setup

* add llvmdev via submodule

* add pydantic via submodule

* run make-migrators twice

* successfully validate new pydantic version

* remove llvmdev from integration tests

* relock

* fix conda forge org override

* fix typo

* fix todo: override env variables in install_bot_code.sh

* fix: don't divide by zero

* finish docs for integration testing

* docs: refer to TRANSPARENT_URLs

* try to run make-migrators a third time

* review: fix branch name

Co-authored-by: Pavel Zwerschke <pavelzw@gmail.com>

* pin GitHub Actions correctly

* fix: we don't want to change num_try

* fix: use CONDA_FORGE_ORG

* move ENV_OVERRIDE_CONDA_FORGE_ORG to settings.py

* use debug log level if running workflow in debug mode

* refactor: move env variable name to constant

* DRY: use identical env variable for cf-graph repository

* Revert "try to run make-migrators a third time"

This reverts commit 973d95c.
did not solve the issue

* docs: fix reference to env variables

* use Docker image name consistently

* refactor: rename DEPLOY_REPO to GRAPH_REPO
to make it consistent with the env variable naming

* fix f-string

* clarify bug comment

* move RANDOM_FRAC_TO_UPDATE to settings

* resolve TODO

* fix docstring

* fix: reorder try clause

* fix: add pip install

* fix: move imports to top-level

* add debug logging to proxy

* add CODEOWNERS file to automatically request further reviews

* add CODEOWNERS file

* fix: push 1.x branch to pydantic feedstock

* relock

* move proxy mode to conda-forge-feedstock-ops
see conda-forge/conda-forge-feedstock-ops#34

* fix reponses from PyPI API

* retrigger ci

* ruff

* run integration tests in merge queue

* add dummy check for setup-repositories

* run integration tests on PRs from branches

* remove dummy workflow

* skip on push

* review suggestion: reformat

Co-authored-by: Matthew R. Becker <beckermr@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Matthew R. Becker <beckermr@users.noreply.github.com>

* separate RANDOM_FRAC_TO_UPDATE for make-graph and update-upstream-versions

* fix comment

* add docstring for settings.py

* add type annotations for settings

* prevent duplicate kwarg errors

* use deferred log expressions

Co-authored-by: jaimergp <jaimergp@users.noreply.github.com>

* clarify mimicked ClonedFeedstockContext warning

* remove "environment variables" comment again

* make graph repo default branch configurable

* defer log expression

* add comment about GITHUB_OUTPUT

* use working-directory instead of cd

* fix env var

* clarify docstring

* docs: random.shuffle is in-place

* add docs to explain test scenario collection

* fix images

* fix except AttributeError

* add shebang

Co-authored-by: jaimergp <jaimergp@users.noreply.github.com>

* Update tests_integration/resources/empty-graph/README.md

Co-authored-by: jaimergp <jaimergp@users.noreply.github.com>

* docs: clarify arbitrary test case name

* docs: improve wording wro test scenario

* begin refactor integration tests to pytest

* continue refactor integration tests to pytest

* update submodule

* document mitmproxy

* migrate to xprocess for mitmproxy

* make dir switches benign

* edit gitignore

* make integration tests pass

* chore: relock

* fix: adjust for new settings module

ir

* feat: add mitmproxy cert wizard, move mitmproxy dir

* docs: update docs for integration tests

* feat: run new test concept everywhere

* chore: disable Docker caching
because it's faster

* fix: condarc file in CI

* fix: ci-driven development

* fix: heredoc

* ci: fix script path

* ci: debug

* fix: trust script and ci

* ci: disable debug mode

* fix: set git identity in integration tests

* debug ci

* fix: do proxy alive check via port probing

* debug ci

* fix: pass PYTHONPATH to proxy script

* ci: enable debug mode

* ci: output proxy logs

* chore: formatting

* debug ci

* debug: more timeout

* ci: debug ci with updated dependencies

* test(integration): don't run deploy to github in proxy env

* ci debug

* test(integration): trust crt file

* ci: debug ci

* chore: add mitmproxy trust script to gitignore

* fix(integration-tests): add more URLs to transparent URLs

* chore: relock conda

* fix: don't reassign os.environ

* fix(integration tests): we also need the http versions of the transparent URLs

* fix: typo in integration tests, pytest nodist

* complete integration tests

* ci: debug

* fix: add CA cert to git

* docs: add comment for Python 3.13

* remove debugger

* chore: relock

* chore: add conda-lock to .gitattributes

* feat(integration tests): dynamically parametrize tests

* docs(integration-tests): update graphics

* chore: remove explicit truststore dependency

* docs(integration-tests): add --dist=no flag to pytest

* chore: remove clear runner script

* chore: add original draw.io file

* fix: typo

* fix: formatting

Co-authored-by: Pavel Zwerschke <pavelzw@gmail.com>

* fix: spelling

Co-authored-by: Pavel Zwerschke <pavelzw@gmail.com>

* fix: remove pydantic-settings note

Co-authored-by: Pavel Zwerschke <pavelzw@gmail.com>

* feat: improve instructions in setup wizard

* fix: formatting

* docs: add note about cloning depth

* rename home_directory to initial_working_dir

* fix: dir link

* fix: get_test_scenario

* docs: explain PyCharm warning

* docs: clarify lifetime of TestCase objects

* docs: refer to README in module docstring

* refactor: use monkeypatch.setenv everywhere

* docs: add docstring to test_scenario

* refactor: make coherent lib module for integration testing

* fix: you cannot monkeypatch in module level

* fix: imports

* fix: circular imports

* fix method call

* fix filesystem path

* relock w/ conda-lock

* relock w/ conda-lock

* fix integration test 1

* relock

* fix: lower pin for conda_forge_feedstock_ops

* relock w/ conda-lock

* relock

* replace svg with markdown

* hotfix: pass settings env manually
this should eventually replaced by conda-forge/conda-forge-feedstock-ops#49

* fix: base_url setting

* slahes in URLs are a good thing

* run ruff

---------

Co-authored-by: Pavel Zwerschke <pavelzw@gmail.com>
Co-authored-by: Matthew R. Becker <beckermr@users.noreply.github.com>
Co-authored-by: jaimergp <jaimergp@users.noreply.github.com>
Co-authored-by: regro-cf-autotick-bot <regro-cf-autotick-bot@users.noreply.github.com>

Author: 5 people

.dockerignore

@@ -0,0 +1,3 @@
+docs/
+tests/
+tests_integration/

.github/CODEOWNERS

@@ -0,0 +1,10 @@
+.github/workflows/test-integration.yml  @ytausch
+conda_forge_tick/contexts.py            @ytausch
+conda_forge_tick/git_utils.py           @ytausch
+conda_forge_tick/models/*               @ytausch
+tests/github_api/*                      @ytausch
+tests/conda_forge_tick/*                @ytausch
+tests/model/*                           @ytausch
+tests/test_contexts.py                  @ytausch
+tests/test_git_utils.py                 @ytausch
+tests_integration/*                     @ytausch

.github/workflows/test-integration.yml

@@ -0,0 +1,90 @@
+name: Integration Tests
+on:
+  pull_request:
+    branches:
+      - main
+  merge_group: null
+
+
+# Integration tests interact with GitHub resources in the integration test infrastructure and therefore
+# cannot run concurrently with other integration tests.
+concurrency:
+  group: cf-scripts-integration-tests
+  cancel-in-progress: false
+
+defaults:
+  run:
+    shell: bash -leo pipefail {0}
+
+jobs:
+  integration-tests:
+    name: Run Integration Tests
+    # if triggered by pull_request, only run on non-fork PRs (secrets access needed)
+    # Nevertheless, this check is always run in the merge queue.
+    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
+
+      - name: Checkout
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          path: cf-scripts
+          submodules: 'true'
+
+      - name: Build Docker Image
+        uses: docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4 # v6.15.0
+        with:
+          context: cf-scripts
+          push: false
+          load: true
+          tags: conda-forge-tick:test
+
+      - name: Setup micromamba
+        uses: mamba-org/setup-micromamba@0dea6379afdaffa5d528b3d1dabc45da37f443fc # v2.0.4
+        with:
+          environment-file: cf-scripts/conda-lock.yml
+          environment-name: cf-scripts
+          condarc-file: cf-scripts/autotick-bot/condarc
+
+      - name: Run pip install
+        working-directory: cf-scripts
+        run: |
+          pip install --no-deps --no-build-isolation -e .
+
+      - name: Run mitmproxy certificate setup wizard
+        working-directory: cf-scripts
+        run: |
+          # place a script in the mitmproxy directory that will be run by the setup wizard
+          # to trust the mitmproxy certificate
+          cat <<EOF > ./tests_integration/.mitmproxy/${{ env.MITMPROXY_WIZARD_HEADLESS_TRUST_SCRIPT }}
+          #!/usr/bin/env bash
+          set -euo pipefail
+          sudo cp "\$1" /usr/local/share/ca-certificates/mitmproxy.crt
+          sudo update-ca-certificates
+          EOF
+
+          ./tests_integration/mitmproxy_setup_wizard.sh
+        env:
+          MITMPROXY_WIZARD_HEADLESS: true
+          MITMPROXY_WIZARD_HEADLESS_TRUST_SCRIPT: mitmproxy_trust_script.sh
+
+      - name: Set up git identity
+        run: |
+          git config --global user.name "regro-cf-autotick-bot-staging"
+          git config --global user.email "regro-cf-autotick-bot-staging@users.noreply.github.com"
+
+      - name: Run Integration Tests with pytest
+        working-directory: cf-scripts
+        run: |
+          pytest -s -v \
+          --dist=no \
+          tests_integration
+        env:
+          BOT_TOKEN: ${{ secrets.GH_TOKEN_STAGING_BOT_USER }}
+          TEST_SETUP_TOKEN: ${{ secrets.GH_TOKEN_STAGING_BOT_USER }}
+
+      - name: Print Proxy Logs
+        run: cat /tmp/mitmproxy.log
+        if: always()

.gitignore

@@ -21,6 +21,7 @@ docs/_build/
 
 node_attrs/*
 graph.json
+!tests_integration/resources/empty-graph/graph.json
 pr_json/*
 pr_status/*
 status/*
@@ -61,3 +62,4 @@ pixi.toml
 .repodata_cache/
 venv
 oryx-build-commands.txt
+.env

.gitmodules

@@ -0,0 +1,9 @@
+[submodule "tests_integration/definitions/conda_forge_pinning/resources/feedstock"]
+	path = tests_integration/lib/_definitions/conda_forge_pinning/resources/feedstock
+	url = https://github.com/conda-forge/conda-forge-pinning-feedstock.git
+[submodule "tests_integration/definitions/pydantic/resources/feedstock"]
+	path = tests_integration/lib/_definitions/pydantic/resources/feedstock
+	url = https://github.com/conda-forge/pydantic-feedstock.git
+[submodule "tests_integration/definitions/pydantic/resources/feedstock_v1"]
+	path = tests_integration/lib/_definitions/pydantic/resources/feedstock_v1
+	url = https://github.com/conda-forge/pydantic-feedstock.git

.pre-commit-config.yaml

@@ -27,6 +27,11 @@ repos:
     rev: v1
     hooks:
       - id: typos
+        exclude: |
+          (?x)^(
+            ^tests_integration\/definitions\/.*\/resources\/.*|
+            ^docs\/assets\/.*
+          )$
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.11.8

README.md

@@ -237,6 +237,8 @@ If your migrator needs special configuration, you should write a new factory fun
 - `CF_FEEDSTOCK_OPS_CONTAINER_NAME`: the name of the container to use in the bot, otherwise defaults to `ghcr.io/regro/conda-forge-tick`
 - `CF_FEEDSTOCK_OPS_CONTAINER_TAG`: set this to override the default container tag used in production runs, otherwise the value of `__version__` is used
 
+Additional environment variables are described in [the settings module](conda_forge_tick/settings.py).
+
 ### Getting a Working Environment
 
 The bot has an abstract set of requirements stored in the `environment.yml` file in this repo.
@@ -299,6 +301,10 @@ Currently, the following commands are supported and tested:
 
 - `update-upstream-versions`
 
+### Integration Tests
+
+See [tests_integration/README.md](tests_integration/README.md).
+
 ### Structure of the Bot's Jobs
 
 #### History

autotick-bot/install_bot_code.sh

@@ -43,6 +43,7 @@ done
 if [[ "${clone_graph}" == "true" ]]; then
   cf_graph_repo=${CF_TICK_GRAPH_GITHUB_BACKEND_REPO:-"regro/cf-graph-countyfair"}
   cf_graph_remote="https://github.com/${cf_graph_repo}.git"
+  # please make sure the cloning depth is always identical to the one used in the integration tests (test_integration.py)
   git clone --depth=5 "${cf_graph_remote}" cf-graph
 else
   echo "Skipping cloning of cf-graph"