Skip to content

Conversation

@sergenyalcin
Copy link
Member

@sergenyalcin sergenyalcin commented Oct 9, 2025

Description of your changes

This PR adds retry mechanisms to all kubectl operations in test templates to handle transient conversion webhook failures that occur with multi-version CRDs.

Problem

When testing resources with multi-version CRDs where storage version differs from the client API version (e.g., storage: v1beta1, client: v1beta2), kubectl operations trigger conversion webhooks. These webhooks occasionally fail with connection errors.

This causes test flakiness as operations fail due to temporary webhook unavailability rather than actual test issues.

Solution

Added consistent retry logic (10 attempts, 5s delay) to all kubectl operations that can trigger conversion webhooks:

  • 00-apply.yaml.tmpl: kubectl annotate operations
  • 01-update.yaml.tmpl: kubectl patch operations (status and spec updates)
  • 02-import.yaml.tmpl: kubectl annotate operations (pause/unpause resources)
  • 03-delete.yaml.tmpl: kubectl delete operations

Each template now includes a retry_kubectl() function that:

  • Retries failed operations up to 10 times
  • Waits 5 seconds between attempts
  • Provides clear logging for debugging
  • Only fails after all retry attempts are exhausted

Impact

  • Improved test reliability: Reduces flakiness from transient webhook connectivity issues
  • Better debugging: Clear retry attempt logging helps identify when webhook issues occur
  • Consistent behavior: All templates handle webhook failures uniformly
  • Conservative approach: Retry count/delay balances reliability with test execution time

I have:

  • Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

Signed-off-by: Sergen Yalçın <yalcinsergen97@gmail.com>
@sergenyalcin sergenyalcin requested a review from ulucinar as a code owner October 9, 2025 19:42
Signed-off-by: Sergen Yalçın <yalcinsergen97@gmail.com>
Copy link
Contributor

@turkenf turkenf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @sergenyalcin 🙌

Copy link
Collaborator

@jeanduplessis jeanduplessis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sergenyalcin sergenyalcin merged commit 15e94cc into crossplane:main Oct 9, 2025
7 checks passed
@sergenyalcin sergenyalcin deleted the retry-for-put-annotate branch October 9, 2025 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants