Improve security of storage classes in examples and documentation

[xgerman]

I was reviewing our storage setup and I don' think it's right: https://www.youtube.com/watch?v=c-_5jc2yMJE&list=PLHgdNuGxrJt0BMQdnJTqnGvXfNO35ujMM&index=5 talks about best-practices and the AKS web page says if we want to retain the disk we should configure it - now this is a long standing issue in CNPG cloudnative-pg/cloudnative-pg#5253 where accidentally deleting your cluster deletes all the data underneath. Not sure if we like that behavior... (2) all the other settings seem to be relevant as well and I am not sure what the default AKS classes provide (see https://learn.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes) though I strongly suspect we will need to create our own classes.

[alaye-ms]

If possible, retain cluster Persistent Volume with data for some amount of time (30 days?) when documentDB object is deleted. If this is unfeasible add a warning to the docs.

[WentingWu666666]

Reclaim policy in all storage classes AKS provides is Delete.

$ kubectl get storageclass
NAME                    PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
azurefile               file.csi.azure.com   Delete          Immediate              true                   19m
azurefile-csi           file.csi.azure.com   Delete          Immediate              true                   19m
azurefile-csi-premium   file.csi.azure.com   Delete          Immediate              true                   19m
azurefile-premium       file.csi.azure.com   Delete          Immediate              true                   19m
default (default)       disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   19m
managed                 disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   19m
managed-csi             disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   19m
managed-csi-premium     disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   19m
managed-premium         disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   19m

[WentingWu666666]

Ways to protect volumes from being deleted when a cluster is deleted:

• Add finalizers to PVCs to prevent accidental deletion: Prevent PVC Deletion when Cluster deleted cloudnative-pg/cloudnative-pg#5253 (reply in thread)

• Use a custom StorageClass with a Retain reclaim policy

• Remove ownership of PVCs so they are not garbage-collected with the cluster

[WentingWu666666]

CNPG supports restore from Backup, VolumeSnapshot and PVC.

Restore from PV needs more steps: cloudnative-pg/cloudnative-pg#8095

[WentingWu666666]

Verified that CNPG is able to restore from PVC

  bootstrap:
    recovery:
      volumeSnapshots:
        storage:
          name: documentdb-preview-1
          kind: PersistentVolumeClaim
          apiGroup: ""

[WentingWu666666]

Verified that we could use finalizer to retain PVC.

[WentingWu666666]

Verified that CNPG is able to restore from PVC

  bootstrap:
    recovery:
      volumeSnapshots:
        storage:
          name: documentdb-preview-1
          kind: PersistentVolumeClaim
          apiGroup: ""

[WentingWu666666]

#207

[WentingWu666666]

We need to add mountOptions and doc to suggest encryption