Skip to content

chore: Deprecate make gke. #2253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mikaeld wants to merge 2 commits into
base: main
Choose a base branch
from
+1 −118
Draft

chore: Deprecate make gke. #2253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dd5aa19
chore: Deprecate `make gke`.
mikaeld Sep 18, 2025
6d623f3
clean up .PHONY
mikaeld Sep 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 1 addition & 9 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: build clean clean-gke fixes gke help pip-compile pip-install-local stop test up
.PHONY: build clean fixes help pip-compile pip-install-local stop test up


help:
Expand All @@ -11,8 +11,6 @@ help:
@echo " pip-install-local Install pip project requirements to your local environment"
@echo " test Runs pytest"
@echo " up Runs the whole stack, served under http://localhost:8080/"
@echo " gke Create a sandbox gke cluster for testing"
@echo " clean-gke Delete the sandbox gke cluster"
@echo " stop Stops the docker containers"

build:
Expand Down Expand Up @@ -48,11 +46,5 @@ up:
docker-compose exec airflow-webserver airflow variables import dev_variables.json
docker-compose exec airflow-webserver airflow connections import dev_connections.json

gke:
bin/start_gke

clean-gke:
bin/stop_gke

test:
python -m pytest tests/
42 changes: 0 additions & 42 deletions bin/add_gcp_creds
View file
Open in desktop
Copy link
Contributor

@sean-rose sean-rose Sep 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script seems to be unrelated to the usage of the moz-fx-data-gke-sandbox project. Is there a particular reason why you're deleting it as well?

Copy link
Contributor Author

@mikaeld mikaeld Sep 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tech debt removal, I've asked many DENG over the years and none ever used this script.

Copy link
Contributor

@sean-rose sean-rose Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The readme mentions using this script for testing Dataproc jobs, and I seem to recall at least trying to test some Dataproc tasks locally when doing QA for Airflow upgrades (though I don't think I managed to get it fully working at that time).

In any case, since this is unrelated to GKE and we do still have Dataproc tasks in active DAGs, I don't think this script should be removed in this PR.

This file was deleted.

53 changes: 0 additions & 53 deletions bin/start_gke
View file
Open in desktop
Copy link
Contributor

@sean-rose sean-rose Sep 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the end of yesterday's Data Infra WG meeting @akkomar suggested that these GKE scripts could be repurposed to facilitate running Airflow local dev workloads in our own personal dev projects, which sounds like a reasonable approach to me to preserve the option to have a quicker Airflow dev process (at the cost of having to configure our own personal dev projects to allow this to work).

If you agree that would be reasonable, I can contribute the necessary changes to this PR (e.g. having the scripts take a project ID argument; though since it looks tricky to pass arbitrary arguments through make we'd probably still want to remove those targets and have people run these scripts directly).

Copy link
Contributor Author

@mikaeld mikaeld Sep 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

running Airflow local dev workloads in our own personal dev projects

This would require each sandbox project to have a GKE cluster with Workload Identity configured with various GCP (e.g. BQ, GCS, GAR, SQL, etc.) permissions. This also means it would be each developer's responsibility to cleanup unused resources. Mozcloud lacks budget monitoring for sandbox projects, so it is hard to monitor the costs of unused resources in those projects. This is why we had make gke create resources in a centralized project with a k8s cron job dedicated to cleaning up unused clusters.

For those reasons, I recommend against the solution proposed by @akkomar.

If you want to re-enable the feature being removed by this PR, I'd recommend building something similar but in the supported mozcloud platform (i.e. GCPv2).

Copy link
Contributor

@sean-rose sean-rose Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I liked the suggestion you made in Slack about potentially setting up a shared GKE cluster in a new moz-fx-data-airflow-gke-dev project where developers could run local Airflow instance GKE tasks (potentially in user-specific namespaces), and I've filed DENG-9749 "Come up with new solution for telemetry-airflow devs to run GKE tasks from local Airflow instances", mentioning that idea plus the original create-GCPv2-GKE-sandbox-project idea.

In the meantime I'm OK with you proceeding with this PR since the GKE scripts no longer work as is.

However, I have squirreled away revised versions of the GKE scripts in the GKE-sandbox-config branch just in case someone like me or @gleonard-m ends up needing to resort to using a custom GKE sandbox setup.

This file was deleted.

14 changes: 0 additions & 14 deletions bin/stop_gke
View file
Open in desktop

This file was deleted.