[release-4.19] OCPBUGS-73950: ensure deterministic zone ordering for control plane machines #10230

openshift-cherrypick-robot · 2026-01-16T20:08:53Z

This is an automated cherry-pick of #10219

/assign tthvo

…achines Control plane machines were intermittently being created in different availability zones than specified in their machine specs. This occurred because the zone list returned from FilterZonesBasedOnInstanceType used a set's UnsortedList() func, which has a non-deterministic order. When CAPI and MAPI manifest generation independently called this func, they could receive zones in different orders, causing a mismatch in machine zone placements between CAPI and MAPI manifests. This commit ensures that we sort the zone slices before further processing.

openshift-ci-robot · 2026-01-16T20:10:06Z

@openshift-cherrypick-robot: Jira Issue OCPBUGS-73785 has been cloned as Jira Issue OCPBUGS-73950. Will retitle bug to link to clone.
/retitle [release-4.19] OCPBUGS-73950: ensure deterministic zone ordering for control plane machines

Details

In response to this:

This is an automated cherry-pick of #10219

/assign tthvo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-01-16T20:10:23Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-73950, which is invalid:

expected dependent Jira Issue OCPBUGS-73785 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #10219

/assign tthvo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tthvo

/lgtm
/approve

openshift-ci · 2026-01-16T20:14:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tthvo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tthvo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tthvo · 2026-01-16T20:15:14Z

/cc @patrickdillon @liweinan

openshift-ci · 2026-01-16T22:58:17Z

@openshift-cherrypick-robot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

tthvo · 2026-01-18T05:50:22Z

/jira refresh

openshift-ci-robot · 2026-01-18T05:50:32Z

@tthvo: This pull request references Jira Issue OCPBUGS-73950, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.19.z) matches configured target version for branch (4.19.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-73785 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-73785 targets the "4.20.z" version, which is one of the valid target versions: 4.20.0, 4.20.z
bug has dependents

Requesting review from QA contact:
/cc @liweinan

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

liweinan · 2026-01-19T07:03:51Z

I'm working on it.

liweinan · 2026-01-19T13:34:35Z

Verified:

Test build: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/2013146016231985152

anan@think:~/works/openshift-versions/419nightly$ ./openshift-install version
./openshift-install 4.19.0-0-2026-01-19-075027-test-ci-ln-8hdqhxk-latest
built from commit f5a064ca85b3f762894a7c7f9424068fa2bf8ae8
release image registry.build10.ci.openshift.org/ci-ln-8hdqhxk/release@sha256:7943fde9ebbe2f2079d27a232ebd396a7dc1d513605df89d37f25ac17057cae4
release architecture amd64

anan@think:~/works/openshift-versions/419nightly$ ../../my-openshift-workspace/OCPBUGS-69923/verify-manifests.sh 
==========================================
Verify CAPI and MAPI Manifest Zone Consistency
==========================================

Installation directory: .

CAPI Machine Zones:
  master-0 (99_openshift-cluster-api_master-machines-0.yaml): us-east-1a
  master-1 (99_openshift-cluster-api_master-machines-1.yaml): us-east-1b
  master-2 (99_openshift-cluster-api_master-machines-2.yaml): us-east-1c

MAPI Machine Zones:
  master-0 (from 99_openshift-machine-api_master-control-plane-machine-set.yaml): us-east-1a
  master-1 (from 99_openshift-machine-api_master-control-plane-machine-set.yaml): us-east-1b
  master-2 (from 99_openshift-machine-api_master-control-plane-machine-set.yaml): us-east-1c

==========================================
Consistency Check
==========================================
✓ Match: master-0 - Zone: us-east-1a
✓ Match: master-1 - Zone: us-east-1b
✓ Match: master-2 - Zone: us-east-1c

✅ Verification PASSED: All machines have consistent zone allocation!

anan@think:~/works/openshift-versions/419nightly$ ../../my-openshift-workspace/OCPBUGS-69923/verify-cluster.sh 
==========================================
Verify Machine Zone Consistency in Cluster
==========================================

Kubeconfig: /home/anan/works/openshift-versions/419nightly/auth/kubeconfig

✓ Successfully connected to cluster

Found 3 master machine(s)

==========================================
Check Zone Consistency for Each Machine
==========================================

--- Machine: weli-test-pv77c-master-0 ---
  Zone Label:        us-east-1a
  ProviderID Zone:  us-east-1a
  Spec Zone:        us-east-1a
  Subnet Filter:     weli-test-pv77c-subnet-private-us-east-1a
  ✅ Zone consistent

--- Machine: weli-test-pv77c-master-1 ---
  Zone Label:        us-east-1b
  ProviderID Zone:  us-east-1b
  Spec Zone:        us-east-1b
  Subnet Filter:     weli-test-pv77c-subnet-private-us-east-1b
  ✅ Zone consistent

--- Machine: weli-test-pv77c-master-2 ---
  Zone Label:        us-east-1c
  ProviderID Zone:  us-east-1c
  Spec Zone:        us-east-1c
  Subnet Filter:     weli-test-pv77c-subnet-private-us-east-1c
  ✅ Zone consistent

==========================================
Verification Summary
==========================================

Checked 3 master machine(s)

✅ Verification PASSED: All machines have consistent zones!

Cluster verification: PASS ✓

Fix verification successful:
  - Zone label, ProviderID zone, and Spec zone are all consistent
  - Machines are created in the correct availability zones

liweinan · 2026-01-19T13:35:08Z

/verified by liweinan

openshift-ci-robot · 2026-01-19T13:35:21Z

@liweinan: This PR has been marked as verified by liweinan.

Details

In response to this:

/verified by liweinan

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tthvo · 2026-01-26T20:05:17Z

/hold

From @liweinan's analysis openshift/release#73935 (comment), I think this bug was introduced when we migrated to AWS SDK v2 (i.e. since 4.20).

Thus, 4.19 and older should be fine and requires no backport.

tthvo · 2026-01-26T23:57:11Z

we migrated to AWS SDK v2 (i.e. since 4.20).

The commit 4068682841 has additional changes to replace deprecated use of sets.String with the generic sets.Set[string]. As a side effect, the result was returned as unsorted (i.e. UnsortedList is only available to-list func). This commit was introduced in 4.20.

We finished backporting to 4.20.z so it should be all good. Pending verification 👀

tthvo · 2026-01-27T14:31:39Z

/close

See https://gist.github.com/liweinan/1c16388052387032931c1846035d8052 for analysis. We confirmed 4.19 and older releases do not contain the bug.

openshift-ci · 2026-01-27T14:32:10Z

@tthvo: Closed this PR.

Details

In response to this:

/close

See https://gist.github.com/liweinan/1c16388052387032931c1846035d8052 for analysis. We confirmed 4.19 and older releases do not contain the bug.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2026-01-27T14:32:18Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-73950. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

Details

In response to this:

This is an automated cherry-pick of #10219

/assign tthvo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-cherrypick-robot assigned tthvo Jan 16, 2026

openshift-cherrypick-robot mentioned this pull request Jan 16, 2026

[release-4.20] OCPBUGS-73785: ensure deterministic zone ordering for control plane machines #10219

Merged

openshift-ci bot requested review from jhixson74 and sadasu January 16, 2026 20:09

openshift-ci bot changed the title ~~[release-4.19] OCPBUGS-73785: ensure deterministic zone ordering for control plane machines~~ [release-4.19] OCPBUGS-73950: ensure deterministic zone ordering for control plane machines Jan 16, 2026

tthvo reviewed Jan 16, 2026

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2026

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2026

openshift-ci bot requested review from liweinan and patrickdillon January 16, 2026 20:15

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jan 18, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jan 19, 2026

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 26, 2026

openshift-ci bot closed this Jan 27, 2026

[release-4.19] OCPBUGS-73950: ensure deterministic zone ordering for control plane machines #10230

[release-4.19] OCPBUGS-73950: ensure deterministic zone ordering for control plane machines #10230

Uh oh!

Conversation

openshift-cherrypick-robot commented Jan 16, 2026

Uh oh!

openshift-ci-robot commented Jan 16, 2026

Uh oh!

openshift-ci-robot commented Jan 16, 2026

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jan 16, 2026

Uh oh!

tthvo commented Jan 16, 2026

Uh oh!

openshift-ci bot commented Jan 16, 2026

Uh oh!

tthvo commented Jan 18, 2026

Uh oh!

openshift-ci-robot commented Jan 18, 2026

Uh oh!

liweinan commented Jan 19, 2026

Uh oh!

liweinan commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liweinan commented Jan 19, 2026

Uh oh!

openshift-ci-robot commented Jan 19, 2026

Uh oh!

tthvo commented Jan 26, 2026

Uh oh!

tthvo commented Jan 26, 2026

Uh oh!

tthvo commented Jan 27, 2026

Uh oh!

openshift-ci bot commented Jan 27, 2026

Uh oh!

openshift-ci-robot commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liweinan commented Jan 19, 2026 •

edited

Loading