CMP-3775,CMP-3783: Add exit-code check for Compliance/NonCompliance test result

[xiaojiey]

1. Use CMP-3930: Include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy #960 to include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy;
2. Add exit-code check for Compliance/NonCompliance test result:

=== NAME  TestSingleScanSucceeds
    main_test.go:689: Verifying scan exit code in configmap (OCP-27649 coverage)
    main_test.go:708: Successfully verified exit code 0 in configmap openscap-pod-42ff9dd4f8bfb8b8590efd45f560bbdf7328590f
--- PASS: TestSingleScanSucceeds (93.70s)

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xiaojiey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [xiaojiey]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@xiaojiey: This pull request references CMP-3775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:994-95522622d2a962b85e0f8223a8f0451953e53b80

[openshift-ci-robot]

@xiaojiey: No Jira issue with key CMP-27649 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

Details

In response to this:

1. Use CMP-3930: Include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy #960 to include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy;
2. Enhance TestSingleScanSucceeds to cover 27649 logic

=== NAME  TestSingleScanSucceeds
   main_test.go:689: Verifying scan exit code in configmap (OCP-27649 coverage)
   main_test.go:708: Successfully verified exit code 0 in configmap openscap-pod-42ff9dd4f8bfb8b8590efd45f560bbdf7328590f
--- PASS: TestSingleScanSucceeds (93.70s)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@xiaojiey: This pull request references CMP-3775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

1. Use CMP-3930: Include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy #960 to include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy;
2. Enhance TestSingleScanSucceeds to cover 27649 logic

=== NAME  TestSingleScanSucceeds
   main_test.go:689: Verifying scan exit code in configmap (OCP-27649 coverage)
   main_test.go:708: Successfully verified exit code 0 in configmap openscap-pod-42ff9dd4f8bfb8b8590efd45f560bbdf7328590f
--- PASS: TestSingleScanSucceeds (93.70s)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@xiaojiey: No Jira issue with key CMP-27649 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

Details

In response to this:

1. Use CMP-3930: Include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy #960 to include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy;
2. Add exit-code check for Compliance/NonCompliance test result:

=== NAME  TestSingleScanSucceeds
   main_test.go:689: Verifying scan exit code in configmap (OCP-27649 coverage)
   main_test.go:708: Successfully verified exit code 0 in configmap openscap-pod-42ff9dd4f8bfb8b8590efd45f560bbdf7328590f
--- PASS: TestSingleScanSucceeds (93.70s)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@xiaojiey: This pull request references CMP-3775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

1. Use CMP-3930: Include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy #960 to include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy;
2. Add exit-code check for Compliance/NonCompliance test result:

=== NAME  TestSingleScanSucceeds
   main_test.go:689: Verifying scan exit code in configmap (OCP-27649 coverage)
   main_test.go:708: Successfully verified exit code 0 in configmap openscap-pod-42ff9dd4f8bfb8b8590efd45f560bbdf7328590f
--- PASS: TestSingleScanSucceeds (93.70s)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@xiaojiey: This pull request references CMP-3775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

This pull request references CMP-3783 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

1. Use CMP-3930: Include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy #960 to include the required selectors to machineconfig to pass the ValidatingAdmissionPolicy;
2. Add exit-code check for Compliance/NonCompliance test result:

=== NAME  TestSingleScanSucceeds
   main_test.go:689: Verifying scan exit code in configmap (OCP-27649 coverage)
   main_test.go:708: Successfully verified exit code 0 in configmap openscap-pod-42ff9dd4f8bfb8b8590efd45f560bbdf7328590f
--- PASS: TestSingleScanSucceeds (93.70s)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:994-800c8b2ff56291d7498eb3e56ccb65e925a197fb

[rhmdnd]

Similar comment here as https://github.com/ComplianceAsCode/compliance-operator/pull/1001/files#r2578952857

[rhmdnd]

I think we got to the bottom of this in another PR thread.

We can add a check here to see if the Validating Admission Policy is present, and if it is, we can skip creating the e2e-invalid pool, since the VAP will prevent us from creating it anyway.

But, we'll still be able to create the VAP on older releases to make sure an invalid pool doesn't interfere with scheduling scans.

[rhmdnd]

We can probably back this change out and pursue it in #960

[rhmdnd]

Good idea for a test utility! One comment on the MCP part of this change, but otherwise this looks good.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:994-30d74aeae5b533172acd1fd78483ed8900de0a21

[xiaojiey]

/retest

[xiaojiey]

/retest

[xiaojiey]

/retest

[Anna-Koudelkova]

I have added and removed /lgtm label as I am not sure who else needs to see and approve this PR and I was afraid I might be the cause of accidental merging before relevant people review.
Please let me know if I should add the label again

[xiaojiey]

/retest

[openshift-ci]

@xiaojiey: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-serial	30d74ae	link	true	/test e2e-aws-serial

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.