inject preview deployments into nested stacks by xh3b4sd · Pull Request #70 · 0xSplits/kayron

xh3b4sd · 2025-09-11T15:07:05Z

Fixes https://linear.app/splits/issue/PE-4885/emit-debug-logs-for-every-single-release-that-causes-state-drift-in.
Towards https://linear.app/splits/issue/PE-4788/support-vercel-like-preview-deployments.

{
    "time": "2025-09-17 15:38:12",
    "level": "info",
    "message": "continuing reconciliation loop",
    "domain": "1D0FD508.lite.testing.splits.org",
    "reason": "detected state drift",
    "release": "splits-lite",
    "version": "e307253e62c2da119579e2505db0b6185642ab9b",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/policy/ensure.go:68"
}

pkg/operator/infrastructure/preview/render.go

xh3b4sd · 2025-09-18T14:51:12Z

pkg/cache/update.go

 	if obj.kin == Service {
 		c.ser[obj.ind].Artifact = c.ser[obj.ind].Artifact.Merge(obj.Artifact)
+		c.ser[obj.ind].Release.Deploy.Preview = obj.Release.Deploy.Preview
 	}


This is ugly, but this enables us to reuse some information in one place, which we already fetched from the Github API in another place. E.g. the preview worker handler resets the preview flag of the release artifact of any main release that defines preview releases for its entire repository. I don't know how to make this better at this point in time.

xh3b4sd · 2025-09-18T14:54:39Z

pkg/hash/hash.go

+	return Hash{
+		dsh: []byte("-" + upp),
+		low: []byte(low),
+		upp: []byte(upp),
+	}
+}


We are hashing branch names for identifying preview deployments inside all of AWS. We also have to create the preview deployment domain using some kind of deterministic identifier based on the preview branch. Turned out we cannot just use branch names in preview domains because branch names have a far wider character set than domain names allow for. E.g. the domain feature/branch.lite.splits.org is invalid.

xh3b4sd · 2025-09-18T14:55:47Z

pkg/operator/cloudformation/active.go

+// Active defines this worker handler to always be executed.
+func (c *CloudFormation) Active() bool {
+	return true
+}


We are marking almost all worker handlers in Kayron as always active. The alternative would be to wrap every worker handler in a proxy worker handler, but I thought this approach here is more explicit.

xh3b4sd · 2025-09-18T14:57:03Z

pkg/operator/container/cache.go

 		var tag string
 		{
-			tag = curTag(ima, x.Release.Docker.String())
+			tag = curTag(ima, x.Release.Labels.Hash.Upper(), x.Release.Docker.String())


At this point we have potentially many ECS Services using the same service tag. So we have to differentiate via the preview tags, which contain the branch based preview hashes.

xh3b4sd · 2025-09-18T14:58:21Z

pkg/operator/infrastructure/ensure.go

+		{
+			byt, err = i.renPre(nam, byt)
+			if err != nil {
+				return tracer.Mask(err)
+			}
+		}


Here we potentially inject the preview deployments into the CloudFormation templates before uploading to S3.

xh3b4sd · 2025-09-18T15:05:10Z

pkg/preview/testdata/000/out.yaml.golden

+  #
+  # AUTO GENERATED PREVIEW DEPLOYMENT 1D0FD508
+  #
+
+  Service1D0FD508:


Here we can see how the CloudFormation templates get extended.

and it just keeps adding to the end as new previews are created? and then once a preview is done (i.e. PR is merged/closed) it drops the updates in here related to that PR?

At the moment we only append. If we have outputs in those appended templates we can make sure to put the expanded resources into the Resources section properly.

Once the pull requests are not open anymore the respective resources do not get added anymore to the templates, because when we look for open pull requests in pkg/preview/expand.go, then we will not get that merged/closed result anymore.

opt := &github.PullRequestListOptions{ State: "open", ListOptions: github.ListOptions{ PerPage: 100, }, } var pul []*github.PullRequest { pul, _, err = p.git.PullRequests.List(context.Background(), p.own, rel.Github.String(), opt) if err != nil { return nil, tracer.Mask(err) } }

xh3b4sd · 2025-09-18T15:05:49Z

pkg/preview/expand.go

+	// filter is a collection of branch name prefixes that we want to ignore when
+	// expanding a service release into preview releases. E.g. we do not want to
+	// deploy preview releases for dependabot branches.
+	filter = []string{
+		"dependabot/",
+	}


We do not deploy previews for e.g. dependabot PRs. We can extend this list any time.

xh3b4sd · 2025-09-18T15:07:05Z

pkg/preview/expand.go

+		{
+			pre.Labels.Hash = hash.New(bra)
+			pre.Labels.Head = ref
+		}


Here is where the ref comes from that we already fetched when searching for pull requests. If we have this ref, then we do not need to lookup those refs for every single branch in the reference worker handler.

xh3b4sd · 2025-09-18T15:09:32Z

pkg/preview/render.go

+		res = Resource{
+			Ser: p.sca.Search([]byte("  Service:")),
+			Tas: p.sca.Search([]byte("  TaskDefinition:")),
+			Dom: p.sca.Search([]byte("  DomainRecord:")),
+			Tar: p.sca.Search([]byte("  TargetGroup:")),
+			Lis: p.sca.Search([]byte("  ListenerRule:")),
+		}


This here is all super ugly, but it works.

xh3b4sd · 2025-09-18T15:17:40Z

README.md

+    "level": "info",
+    "message": "continuing reconciliation loop",
+    "domain": "1d0fd508.lite.testing.splits.org",
+    "reason": "detected state drift",
+    "release": "splits-lite",
+    "version": "e307253e62c2da119579e2505db0b6185642ab9b",


We should see something like this in the logs

mihoward21

ok, went through it all. just a few comments/questions. is this wired up on splits lite right now? can i go create some branches/pr's and play around with it there?

mihoward21 · 2025-09-18T17:59:44Z

pkg/preview/image.go

+
+	var out []byte
+	{
+		out = bytes.ReplaceAll(lin, sub[1], tag)


what exactly is this doing? can't tell what the regex is looking for / replacing

ah ok i see below in the test how this works.

mihoward21 · 2025-09-18T18:10:14Z

pkg/preview/render.go

+		// deployments.
+
+		{
+			pri++


so do we need to "reserve" a certain number of priorities per service? i.e. lite.splits.org is allotted 50 - 99, explorer.splits.org is allotted 100 - 149, teams.splits.org is allotted 150 - 199?

ah well i guess just for the testing environment. but yea i think the same general idea.

mihoward21 · 2025-09-18T18:14:10Z

pkg/preview/testdata/000/out.yaml.golden

+  #
+  # AUTO GENERATED PREVIEW DEPLOYMENT 1D0FD508
+  #
+
+  Service1D0FD508:


and it just keeps adding to the end as new previews are created? and then once a preview is done (i.e. PR is merged/closed) it drops the updates in here related to that PR?

xh3b4sd · 2025-09-19T09:52:59Z

can i go create some branches/pr's and play around with it there?

This should work for all deployed services that we define releases for. Saying that, I am realizing we can shoot ourselves potentially in the foot by defining previews for e.g. Kayron itself. That wouldn't work because Kayron is not exposed to the internet, and therefore doesn't have e.g. the ListenerRule to copy. And so the stack update would fail etc. Maybe we need a whitelist/blacklist approach here or document those details some more. Having said that, removing the preview config from the release definition should bring the CloudFormation stack back to health again, so the failed state is not critical.

For Splits Lite this works already. See 0xSplits/splits-lite#44.

inject preview deployments into nested stacks

6960aba

xh3b4sd self-assigned this Sep 11, 2025

fix

30595c6

xh3b4sd commented Sep 11, 2025

View reviewed changes

pkg/operator/infrastructure/preview/render.go Outdated Show resolved Hide resolved

xh3b4sd added 10 commits September 12, 2025 23:29

fix

ae63644

Merge branch 'main' into preview

2ad9cac

fix

9c85319

fix

1814c20

fix

eb9f56e

fix

6f795f5

fix

ceee2cd

fix

a504e9b

fix

d2825e2

fix

a2f526e

xh3b4sd commented Sep 18, 2025

View reviewed changes

fix

c4719b6

xh3b4sd marked this pull request as ready for review September 18, 2025 15:19

xh3b4sd requested a review from r0ohafza September 18, 2025 15:19

mihoward21 approved these changes Sep 18, 2025

View reviewed changes

fix

1aeb7bd

xh3b4sd merged commit 74fd411 into main Sep 19, 2025
3 checks passed

xh3b4sd deleted the preview branch September 19, 2025 12:06

Conversation

xh3b4sd commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihoward21 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xh3b4sd commented Sep 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xh3b4sd commented Sep 11, 2025 •

edited

Loading