fix: resolve finalizer failure with Kyverno webhook integration

[maryamtahhan]

The finalizer was failing with "gkm.io/resolvedDigest is immutable when spec.image is unchanged" errors when Kyverno image verification was enabled.

Root cause: The GKM mutating webhook was setting an empty digest annotation before Kyverno added the digest to spec.image, then attempting to update it when the controller added the finalizer, which violated the immutability check.

Changes:

• Add reinvocationPolicy: IfNeeded to mutating webhooks via Kustomize patch
• Skip setting annotation when digest is empty in Kyverno mode
• Update webhook README to document the patch-based approach

The webhook now waits for Kyverno to mutate the image, gets reinvoked, and only then sets the annotation with the correct digest value.

[maryamtahhan]

Name:         vector-add-cache-rocm-v2
Namespace:    gkm-test-ns-scoped-1
Labels:       gkm.io/signature-format=cosign-v2
Annotations:  gkm.io/resolvedDigest: sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3
              kyverno.io/verify-images: {"quay.io/gkm/cache-examples@sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3":"pass"}
API Version:  gkm.io/v1alpha1
Kind:         GKMCache
Metadata:
  Creation Timestamp:  2026-02-06T18:12:04Z
  Finalizers:
    gkm.io.gkmcachenode/finalizer
  Generation:        1
  Resource Version:  1930
  UID:               534f7c15-2a58-4cae-a6a6-e4d194351e0c
Spec:
  Image:  quay.io/gkm/cache-examples:vector-add-cache-rocm-v2@sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3
Status:
  Conditions:
    Last Transition Time:  2026-02-06T18:12:08Z
    Message:               The Kernel Cache has been extracted onto the host
    Reason:                Extracted
    Status:                True
    Type:                  Extracted
  Counts:
    Node Cnt:             2
    Node Error Cnt:       0
    Node In Use Cnt:      0
    Node Not In Use Cnt:  2
    Pod Outdated Cnt:     0
    Pod Running Cnt:      0
  Last Updated:           2026-02-06T18:12:08Z
  Resolved Digest:        sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3
Events:                   <none>

with new logs

kubectl logs -n gkm-system gkm-operator-559bcfb488-wdr9r 
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"setup","msg":"Logging","Level":"info"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"setup","msg":"No-GPU set to true","noGpu":true}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"gkm.io/v1alpha1, Kind=GKMCache","path":"/mutate-gkm-io-v1alpha1-gkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-gkm-io-v1alpha1-gkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"gkm.io/v1alpha1, Kind=GKMCache","path":"/validate-gkm-io-v1alpha1-gkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-gkm-io-v1alpha1-gkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"gkm.io/v1alpha1, Kind=ClusterGKMCache","path":"/mutate-gkm-io-v1alpha1-clustergkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-gkm-io-v1alpha1-clustergkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"gkm.io/v1alpha1, Kind=ClusterGKMCache","path":"/validate-gkm-io-v1alpha1-clustergkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-gkm-io-v1alpha1-clustergkmcache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"setup","msg":"disabling http/2"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"setup","msg":"disabling http/2"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate poll+watcher","interval":10}
I0206 18:11:17.447666       1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="gkm-system/097dd617.gkm.io"
I0206 18:11:17.461866       1 leaderelection.go:272] "Successfully acquired lease" lock="gkm-system/097dd617.gkm.io"
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting EventSource","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","source":"kind source: *v1.ConfigMap"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting EventSource","controller":"clustergkmcache","controllerGroup":"gkm.io","controllerKind":"ClusterGKMCache","source":"kind source: *v1alpha1.ClusterGKMCacheNode"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting EventSource","controller":"gkmcache","controllerGroup":"gkm.io","controllerKind":"GKMCache","source":"kind source: *v1alpha1.GKMCacheNode"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting EventSource","controller":"clustergkmcache","controllerGroup":"gkm.io","controllerKind":"ClusterGKMCache","source":"kind source: *v1alpha1.ClusterGKMCache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting EventSource","controller":"gkmcache","controllerGroup":"gkm.io","controllerKind":"GKMCache","source":"kind source: *v1alpha1.GKMCache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting Controller","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting workers","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","worker count":1}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"configMap","msg":"ConfigMap Reconcile ENTER"}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"configMap","msg":"Creating GKM CSIDriver object"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting Controller","controller":"clustergkmcache","controllerGroup":"gkm.io","controllerKind":"ClusterGKMCache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting workers","controller":"clustergkmcache","controllerGroup":"gkm.io","controllerKind":"ClusterGKMCache","worker count":1}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting Controller","controller":"gkmcache","controllerGroup":"gkm.io","controllerKind":"GKMCache"}
{"level":"info","ts":"2026-02-06T18:11:17Z","msg":"Starting workers","controller":"gkmcache","controllerGroup":"gkm.io","controllerKind":"GKMCache","worker count":1}
{"level":"info","ts":"2026-02-06T18:11:17Z","logger":"configMap","msg":"ConfigMap Values","agentLogLevel":"info","agentImage":"quay.io/gkm/agent:latest","csiLogLevel":"info","csiImage":"quay.io/gkm/gkm-csi-plugin:latest","noGpu":"true"}
{"level":"info","ts":"2026-02-06T18:11:18Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8443","secure":true}
{"level":"info","ts":"2026-02-06T18:12:04Z","logger":"webhook-ns","msg":"Image already contains digest (likely from Kyverno)","image":"quay.io/gkm/cache-examples:vector-add-cache-rocm-v2@sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3","digest":"sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3"}
{"level":"info","ts":"2026-02-06T18:12:04Z","logger":"webhook-ns","msg":"added/updated resolvedDigest","image":"quay.io/gkm/cache-examples:vector-add-cache-rocm-v2@sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3","digest":"sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3"}
{"level":"info","ts":"2026-02-06T18:12:04Z","logger":"oper-ns","msg":"Calling KubeAPI to add Finalizer to GKMCache","Namespace":"gkm-test-ns-scoped-1","CacheNodeName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:04Z","logger":"webhook-ns","msg":"Image already contains digest (likely from Kyverno)","image":"quay.io/gkm/cache-examples:vector-add-cache-rocm-v2@sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3","digest":"sha256:660955164c52231b202ca95b1e454515834e612584a68ada99824eaaebc18ba3"}
{"level":"info","ts":"2026-02-06T18:12:05Z","logger":"oper-ns","msg":"Calling KubeAPI to Update GKMCache Status","reason":"Set Pending Condition","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:08Z","logger":"oper-ns","msg":"Calling KubeAPI to Update GKMCache Status","reason":"Update Counts","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:08Z","logger":"oper-ns","msg":"Calling KubeAPI to Update GKMCache Status","reason":"Update Counts","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:08Z","logger":"oper-ns","msg":"Calling KubeAPI to Update GKMCache Status","reason":"Set Extracted Condition","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:08Z","logger":"oper-ns","msg":"Calling KubeAPI to Update GKMCache Status","reason":"Update Counts","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:08Z","logger":"oper-ns","msg":"Calling KubeAPI to Update GKMCache Status","reason":"Update Counts","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}
{"level":"info","ts":"2026-02-06T18:12:08Z","logger":"oper-ns","msg":"failed to update GKMCache Status","err":"Operation cannot be fulfilled on gkmcaches.gkm.io \"vector-add-cache-rocm-v2\": the object has been modified; please apply your changes to the latest version and try again","reason":"Update Counts","Namespace":"gkm-test-ns-scoped-1","CacheName":"vector-add-cache-rocm-v2"}

[Billy99]

/lgtm