CMP-3859: Add e2e test (60342) for disabled timeout with zero value

[taimurhafeez]

Port test case OCP-60342 to verify that setting the timeout to 0s in a ScanSetting disables the timeout functionality. This ensures scans can complete without timeout errors when timeout is explicitly set to zero.

The test creates a ScanSetting with a timeout set to 0 seconds and a maxRetryOnTimeout set to 2, binds it to the ocp4-cis profile, and verifies that all scans complete successfully in the DONE phase without any ERROR results due to timeouts.

This addresses the missing upstream test noted in the downstream test suite.

To implement this PR:

1. The following line needs to be commented in tests/e2e/framework/main_entry.go or there will be error error creating Machine Config Pool e2e-invalid
   line to be commented --> return fmt.Errorf("failed to create Machine Config Pool %s: %w", "e2e-invalid", err)

2. Run the test make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestTimeoutDisabledWithZeroValue"

[openshift-ci]

Hi @taimurhafeez. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Anna-Koudelkova]

The test case is PASSing on OCP 4.20 cluster with implemented changes from PR #960 :

=== RUN   TestTimeoutDisabledWithZeroValue
=== PAUSE TestTimeoutDisabledWithZeroValue
=== CONT  TestTimeoutDisabledWithZeroValue
2025/12/01 15:35:37 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/01 15:35:42 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/01 15:35:47 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/01 15:35:52 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/01 15:35:57 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/01 15:36:02 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/01 15:36:07 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/01 15:36:12 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/01 15:36:17 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/01 15:36:22 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/01 15:36:32 ComplianceScan ready (DONE)
2025/12/01 15:36:32 All scans in ComplianceSuite have finished (test-timeout-disabled-with-zero-value-binding)
--- PASS: TestTimeoutDisabledWithZeroValue (61.61s)
PASS

[rhmdnd]

/retest-required

[openshift-ci-robot]

@taimurhafeez: This pull request references CMP-3859 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Port test case OCP-60342 to verify that setting the timeout to 0s in a ScanSetting disables the timeout functionality. This ensures scans can complete without timeout errors when timeout is explicitly set to zero.

The test creates a ScanSetting with a timeout set to 0 seconds and a maxRetryOnTimeout set to 2, binds it to the ocp4-cis profile, and verifies that all scans complete successfully in the DONE phase without any ERROR results due to timeouts.

This addresses the missing upstream test noted in the downstream test suite.

To implement this PR:

1. The following line needs to be commented in tests/e2e/framework/main_entry.go or there will be error error creating Machine Config Pool e2e-invalid
   line to be commented --> return fmt.Errorf("failed to create Machine Config Pool %s: %w", "e2e-invalid", err)

2. Run the test make e2e-serial E2E_GO_TEST_FLAGS="-v -run TestTimeoutDisabledWithZeroValue"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[rhmdnd]

Looks good to me, just need to remove the comment in main_entry.go

[taimurhafeez]

Looks good to me, just need to remove the comment in main_entry.go

I have removed the comment. Plus tested again, and it passed.

=== RUN   TestTimeoutDisabledWithZeroValue
=== PAUSE TestTimeoutDisabledWithZeroValue
=== CONT  TestTimeoutDisabledWithZeroValue
2025/12/02 14:48:06 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/02 14:48:11 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/02 14:48:16 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/02 14:48:21 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/02 14:48:26 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/02 14:48:31 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/02 14:48:36 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/02 14:48:41 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/02 14:48:46 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/02 14:48:51 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/02 14:49:01 ComplianceScan ready (DONE)
2025/12/02 14:49:01 All scans in ComplianceSuite have finished (test-timeout-disabled-with-zero-value-binding)
--- PASS: TestTimeoutDisabledWithZeroValue (61.56s)
PASS
2025/12/02 14:49:07 cleaning up Profile Bundles

[rhmdnd]

/lgtm

[rhmdnd]

/retest-required

Looks like some of the scans timed out... We might want to consider breaking the test suite into smaller pieces based on functionality. That might help us isolate problematic tests.

[openshift-ci]

@taimurhafeez: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-serial	1241c3a	link	true	/test e2e-aws-serial

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[rhmdnd]

Another issue with a scan timing out. This time, waiting for the compliance suite to be created, which seems odd..

2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
    main_test.go:1770: compliancesuites.compliance.openshift.io "test-suspend-scan-setting-binding" not found
--- FAIL: TestSuspendScanSetting (1800.44s) 

[xiaojiey]

/lgtm

[taimurhafeez]

Another issue with a scan timing out. This time, waiting for the compliance suite to be created, which seems odd..

2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
    main_test.go:1770: compliancesuites.compliance.openshift.io "test-suspend-scan-setting-binding" not found
--- FAIL: TestSuspendScanSetting (1800.44s) 

Strange. Is it from the CI run or your local run?

[rhmdnd]

Another issue with a scan timing out. This time, waiting for the compliance suite to be created, which seems odd..

2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
    main_test.go:1770: compliancesuites.compliance.openshift.io "test-suspend-scan-setting-binding" not found
--- FAIL: TestSuspendScanSetting (1800.44s) 

Strange. Is it from the CI run or your local run?

From CI - https://storage.googleapis.com/test-platform-results/pr-logs/pull/ComplianceAsCode_compliance-operator/982/pull-ci-ComplianceAsCode-compliance-operator-master-e2e-aws-serial/1995970267255410688/build-log.txt

[taimurhafeez]

Another issue with a scan timing out. This time, waiting for the compliance suite to be created, which seems odd..

2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
2025/12/03 00:41:05 waiting for availability of test-suspend-scan-setting-binding compliancesuite
    main_test.go:1770: compliancesuites.compliance.openshift.io "test-suspend-scan-setting-binding" not found
--- FAIL: TestSuspendScanSetting (1800.44s) 

Strange. Is it from the CI run or your local run?

From CI - https://storage.googleapis.com/test-platform-results/pr-logs/pull/ComplianceAsCode_compliance-operator/982/pull-ci-ComplianceAsCode-compliance-operator-master-e2e-aws-serial/1995970267255410688/build-log.txt

@rhmdnd I ran the test again, and it does not appear to be getting into a scan timing-out issue.

=== RUN   TestTimeoutDisabledWithZeroValue
=== PAUSE TestTimeoutDisabledWithZeroValue
=== CONT  TestTimeoutDisabledWithZeroValue
2025/12/08 15:43:39 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/08 15:43:44 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/08 15:43:49 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/08 15:43:54 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/08 15:43:59 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/08 15:44:04 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: RUNNING
2025/12/08 15:44:09 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 15:44:14 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 15:44:19 waiting until suite test-timeout-disabled-with-zero-value-binding reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 15:44:30 ComplianceScan ready (DONE)
2025/12/08 15:44:30 All scans in ComplianceSuite have finished (test-timeout-disabled-with-zero-value-binding)
--- PASS: TestTimeoutDisabledWithZeroValue (57.72s)
PASS
2025/12/08 15:44:36 cleaning up Profile Bundles

[rhmdnd]

It could something wrong with the test content we're using, since it looks like the profile bundles are failing to parse. We're chasing that down in a separate PR:

#1019

[rhmdnd]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhmdnd, taimurhafeez

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [rhmdnd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment