Add DocumentDB upgrade support with configurable PostgresImage and ImageVolume extensions #208

WentingWu666666 · 2026-01-13T02:03:29Z

Summary

This PR introduces DocumentDB upgrade capabilities by adding a configurable PostgresImage field to the DocumentDB CRD and enabling ImageVolume-based extension loading through CNPG.

Key Changes

CRD Updates

Added postgresImage field to DocumentDB spec for configurable postgres image selection
Added documentDBVersion status field to track the current version

CNPG Integration

Upgraded CNPG to support ImageVolume extensions
Configure LD_LIBRARY_PATH for extension loading
Removed hardcoded PostgresUID/GID (105/108) to allow standard postgres images to use their default UID/GID values and avoid initdb: could not look up effective user ID 105: user does not exist errors

CI/CD Updates

Upgraded kind to v0.31.0 with Kubernetes v1.35.0 for E2E tests
Enabled ImageVolume feature gate in kind cluster configuration
Added E2E tests for upgrade scenarios

Testing

Unit tests: Added/updated in documentdb_controller_test.go
E2E tests: New upgrade test workflow added

hossain-rayhan · 2026-01-13T04:49:49Z

.github/actions/setup-test-environment/action.yml

          nodeCount: ${{ inputs.node-count }}
          instancesPerNode: ${{ inputs.instances-per-node }}
-          documentDBImage: ghcr.io/microsoft/documentdb/documentdb-local:16
+          documentDBImage: ghcr.io/guanzhousongmicrosoft/documentdb-pg18-amd64:0.110.0


See it's a draft. Just in case, we should not use any personal repo link.

Signed-off-by: wenting <wentingwu@microsoft.com>

…stgres images Signed-off-by: wenting <wentingwu@microsoft.com>

Signed-off-by: wenting <wentingwu@microsoft.com>

Copilot

Pull request overview

This PR introduces DocumentDB upgrade capabilities by adding support for configurable PostgreSQL and DocumentDB extension images, along with automated extension upgrade logic. The changes enable rolling upgrades of DocumentDB clusters with minimal downtime.

Changes:

Adds postgresImage field to DocumentDB CRD spec for configurable Postgres base image selection
Adds documentDBVersion field to DocumentDB status to track the currently installed extension version
Implements automated extension upgrade logic that updates the extension image and runs ALTER EXTENSION when needed
Upgrades CNPG dependency from 0.26.1 to 0.27.0 to support ImageVolume-based extension loading
Adds E2E upgrade tests that verify version transitions
Removes DocumentDB image build from CI pipeline (now using external registry)
Migrates certificate tests from testify to Ginkgo/Gomega for consistency

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
operator/src/api/preview/documentdb_types.go	Adds `PostgresImage` spec field and `DocumentDBVersion` status field to DocumentDB CRD
operator/src/config/crd/bases/documentdb.io_dbs.yaml	Updates CRD schema with new `postgresImage` and `documentDBVersion` fields
operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml	Mirrors CRD schema changes in Helm chart
operator/src/internal/cnpg/cnpg_cluster.go	Configures CNPG cluster with ImageVolume extensions and uses `postgresImage` from spec
operator/src/internal/controller/documentdb_controller.go	Adds upgrade logic with `upgradeDocumentDBExtensionIfNeeded` and `updateExtensionImageIfNeeded` functions, plus version parsing
operator/src/internal/controller/documentdb_controller_test.go	Comprehensive unit tests for upgrade logic using Ginkgo/Gomega
operator/src/internal/controller/certificate_controller_test.go	Extracts certificate tests into separate file, maintaining testify framework
operator/documentdb-helm-chart/Chart.yaml	Bumps CNPG dependency from 0.26.1 to 0.27.0
.github/workflows/test-E2E.yml	Adds upgrade test scenario that patches image and verifies version transition
.github/workflows/test-integration.yml	Updates to use hardcoded DocumentDB image instead of version parameter
.github/workflows/test-backup-and-restore.yml	Removes unused `documentdb_version` input parameter
.github/workflows/test-build-and-package.yml	Removes DocumentDB image build step (now using external images)
.github/actions/setup-test-environment/action.yml	Enables ImageVolume feature gate in kind cluster config, upgrades to kind v0.31.0 with K8s v1.35.0

Copilot · 2026-01-29T17:27:37Z

operator/src/internal/controller/documentdb_controller.go

+	// Check if documentdb extension needs to be upgraded (image update + ALTER EXTENSION)
+	if err := r.upgradeDocumentDBExtensionIfNeeded(ctx, currentCnpgCluster, desiredCnpgCluster, documentdb); err != nil {
+		logger.Error(err, "Failed to upgrade DocumentDB extension")
+		return ctrl.Result{RequeueAfter: RequeueAfterShort}, nil
+	}


There's a potential race condition here. Between lines 234-259, the documentdb status is updated with cluster phase and connection string. Then at line 262, upgradeDocumentDBExtensionIfNeeded is called which also updates the documentdb status at lines 550-555 and 578-581. If the status was modified by another reconcile loop between these calls, the second update could overwrite changes. Consider refetching the documentdb object before the upgrade check, or better yet, consolidate all status updates to happen once at the end of the reconcile loop.

Copilot · 2026-01-29T17:27:37Z

operator/src/internal/controller/documentdb_controller.go

+		return ctrl.Result{RequeueAfter: RequeueAfterShort}, nil
+	}
+
 	// Don't reque again unless there is a change


Typo in comment: "reque" should be "requeue".

Suggested change

// Don't reque again unless there is a change

// Don't requeue again unless there is a change

Copilot · 2026-01-29T17:27:39Z

operator/src/internal/controller/documentdb_controller.go

+	// Step 6: Update DocumentDB version in status after upgrade
+	documentdb.Status.DocumentDBVersion = defaultVersion
+	if err := r.Status().Update(ctx, documentdb); err != nil {
+		logger.Error(err, "Failed to update DocumentDB status after extension upgrade")
+	}


The status update failure should cause the function to return an error rather than just logging it. If the status update fails after successfully upgrading the extension, the operator will lose track of the current version and may attempt to upgrade again unnecessarily, or the status will be inconsistent with reality.

Copilot · 2026-01-29T17:27:41Z

operator/src/internal/cnpg/cnpg_cluster.go

 			spec := cnpgv1.ClusterSpec{
 				Instances: documentdb.Spec.InstancesPerNode,
-				ImageName: documentdb_image,
+				ImageName: documentdb.Spec.PostgresImage,


Consider adding a defensive default value check for PostgresImage to match the pattern used elsewhere in the codebase. While the kubebuilder default should handle this, adding a runtime check like cmp.Or(documentdb.Spec.PostgresImage, "ghcr.io/cloudnative-pg/postgresql:18-standard-bookworm") on line 59 would provide additional safety against potential edge cases where the default might not be applied (e.g., when migrating from older CRD versions).

Suggested change

ImageName: documentdb.Spec.PostgresImage,

ImageName: cmp.Or(documentdb.Spec.PostgresImage, "ghcr.io/cloudnative-pg/postgresql:18-standard-bookworm"),

Copilot · 2026-01-29T17:27:42Z

.github/workflows/test-E2E.yml

+        # Extract expected version from image tag (e.g., "0.110.0" -> "0.110-0")
+        NEW_VERSION_TAG="${NEW_IMAGE##*:}"
+        EXPECTED_VERSION=$(echo "$NEW_VERSION_TAG" | sed 's/\.\([^.]*\)$/-\1/')


The version parsing logic using sed 's/\.$[^.]*$$/-\1/' assumes a specific format where the last dot-separated component should be converted to a hyphen-separated component (e.g., "0.110.0" -> "0.110-0"). This is fragile and may not work correctly for all version formats. For example, if the image tag includes a pre-release version like "0.110.0-beta", the parsing would produce "0.110.0-beta" (unchanged), not matching the expected format. Consider documenting the expected version format or making the parsing more robust.

Suggested change

# Extract expected version from image tag (e.g., "0.110.0" -> "0.110-0")

NEW_VERSION_TAG="${NEW_IMAGE##*:}"

EXPECTED_VERSION=$(echo "$NEW_VERSION_TAG" | sed 's/\.$[^.]*$$/-\1/')

# Extract expected version from image tag:

# - For semver-like tags: "0.110.0" -> "0.110-0"

# - With pre-release: "0.110.0-beta" -> "0.110-0-beta"

NEW_VERSION_TAG="${NEW_IMAGE##*:}"

if [[ "$NEW_VERSION_TAG" =~ ^([0-9]+\.[0-9]+)\.([0-9]+)(-.+)?$ ]]; then

EXPECTED_VERSION="${BASH_REMATCH[1]}-${BASH_REMATCH[2]}${BASH_REMATCH[3]}"

else

echo "⚠️ Unable to parse version from image tag '$NEW_VERSION_TAG'; using tag as-is for comparison"

EXPECTED_VERSION="$NEW_VERSION_TAG"

fi

Copilot · 2026-01-29T17:27:43Z

operator/src/internal/controller/documentdb_controller.go

+		if err := r.Status().Update(ctx, documentdb); err != nil {
+			logger.Error(err, "Failed to update DocumentDB status with extension version")


The status update at line 552 should not be silently ignored if it fails. When the status update fails, the operator should requeue to ensure the version is eventually recorded. Currently, the error is only logged but the function continues, which could lead to inconsistent state where the extension version in the database differs from what's recorded in the status.

Suggested change

if err := r.Status().Update(ctx, documentdb); err != nil {

logger.Error(err, "Failed to update DocumentDB status with extension version")

if err := r.Status().Update(ctx, documentdb); err != nil {

err = fmt.Errorf("failed to update DocumentDB status with extension version: %w", err)

logger.Error(err, "Failed to update DocumentDB status with extension version")

return err

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from 433e80d to 1e6c870 Compare January 13, 2026 02:04

WentingWu666666 marked this pull request as ready for review January 13, 2026 02:04

WentingWu666666 requested review from alaye-ms, hossain-rayhan and xgerman as code owners January 13, 2026 02:04

WentingWu666666 marked this pull request as draft January 13, 2026 02:04

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from 1e6c870 to ed68947 Compare January 13, 2026 02:16

hossain-rayhan reviewed Jan 13, 2026

View reviewed changes

wentingwu000 added 9 commits January 21, 2026 15:38

documentdb extension

1c1419a

Signed-off-by: wenting <wentingwu@microsoft.com>

Add configurable PostgresImage field to DocumentDB CRD

1189a70

Signed-off-by: wenting <wentingwu@microsoft.com>

update crds

5afecf5

Signed-off-by: wenting <wentingwu@microsoft.com>

Upgrade CNPG to support ImageVolume extensions

11aecd9

Signed-off-by: wenting <wentingwu@microsoft.com>

need to set ld_library_path

5fdd82a

Signed-off-by: wenting <wentingwu@microsoft.com>

upgrade extension

602258c

Signed-off-by: wenting <wentingwu@microsoft.com>

unit tests

b998268

Signed-off-by: wenting <wentingwu@microsoft.com>

documentDBVersion in status

f589508

Signed-off-by: wenting <wentingwu@microsoft.com>

e2e test for upgrade

1cc45ff

Signed-off-by: wenting <wentingwu@microsoft.com>

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from ab04d57 to 1cc45ff Compare January 21, 2026 20:38

wentingwu000 added 2 commits January 21, 2026 19:40

postgresql:18-standard-bookworm

3eee73d

Signed-off-by: wenting <wentingwu@microsoft.com>

Remove hardcoded PostgresUID/GID to fix initdb error with standard po…

35b47ca

…stgres images Signed-off-by: wenting <wentingwu@microsoft.com>

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from 1ae514c to 35b47ca Compare January 22, 2026 02:07

Upgrade kind to v0.31.0 with Kubernetes v1.35.0 for E2E tests

a61d520

Signed-off-by: wenting <wentingwu@microsoft.com>

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from 006724a to a61d520 Compare January 22, 2026 14:29

wentingwu000 added 2 commits January 28, 2026 19:36

enable ImageVolume in github kind

32b58e5

Signed-off-by: wenting <wentingwu@microsoft.com>

Enable ImageVolume feature gate in kind cluster

5c53ddb

Signed-off-by: wenting <wentingwu@microsoft.com>

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch 5 times, most recently from 34f29f2 to b2836ec Compare January 29, 2026 02:45

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch 2 times, most recently from b9d97be to 5144126 Compare January 29, 2026 03:12

fix: E2E test upgrade check uses documentDBVersion status field

d6db610

Signed-off-by: wenting <wentingwu@microsoft.com>

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from 5144126 to d6db610 Compare January 29, 2026 15:20

WentingWu666666 changed the title ~~upgrade documentdb extension~~ Add DocumentDB upgrade support with configurable PostgresImage and ImageVolume extensions Jan 29, 2026

WentingWu666666 marked this pull request as ready for review January 29, 2026 17:21

Copilot AI review requested due to automatic review settings January 29, 2026 17:21

Copilot started reviewing on behalf of WentingWu666666 January 29, 2026 17:21 View session

Copilot AI reviewed Jan 29, 2026

View reviewed changes

WentingWu666666 force-pushed the users/wentingwu/upgrade-documentdb branch from 13fad47 to d6db610 Compare January 29, 2026 17:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DocumentDB upgrade support with configurable PostgresImage and ImageVolume extensions #208

Add DocumentDB upgrade support with configurable PostgresImage and ImageVolume extensions #208

Uh oh!

WentingWu666666 commented Jan 13, 2026 •

edited

Loading

Uh oh!

hossain-rayhan Jan 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Copilot AI Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// Don't reque again unless there is a change
	// Don't requeue again unless there is a change

	ImageName: documentdb.Spec.PostgresImage,
	ImageName: cmp.Or(documentdb.Spec.PostgresImage, "ghcr.io/cloudnative-pg/postgresql:18-standard-bookworm"),

-        # Extract expected version from image tag (e.g., "0.110.0" -> "0.110-0")
-        NEW_VERSION_TAG="${NEW_IMAGE##*:}"
-        EXPECTED_VERSION=$(echo "$NEW_VERSION_TAG" | sed 's/\.\([^.]*\)$/-\1/')
+        # Extract expected version from image tag:
+        # - For semver-like tags: "0.110.0" -> "0.110-0"
+        # - With pre-release: "0.110.0-beta" -> "0.110-0-beta"
+        NEW_VERSION_TAG="${NEW_IMAGE##*:}"
+        if [[ "$NEW_VERSION_TAG" =~ ^([0-9]+\.[0-9]+)\.([0-9]+)(-.+)?$ ]]; then
+          EXPECTED_VERSION="${BASH_REMATCH[1]}-${BASH_REMATCH[2]}${BASH_REMATCH[3]}"
+        else
+          echo "⚠️  Unable to parse version from image tag '$NEW_VERSION_TAG'; using tag as-is for comparison"
+          EXPECTED_VERSION="$NEW_VERSION_TAG"
+        fi

		if err := r.Status().Update(ctx, documentdb); err != nil {
		logger.Error(err, "Failed to update DocumentDB status with extension version")

Add DocumentDB upgrade support with configurable PostgresImage and ImageVolume extensions #208

Are you sure you want to change the base?

Add DocumentDB upgrade support with configurable PostgresImage and ImageVolume extensions #208

Uh oh!

Conversation

WentingWu666666 commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

CRD Updates

CNPG Integration

CI/CD Updates

Testing

Uh oh!

hossain-rayhan Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

WentingWu666666 commented Jan 13, 2026 •

edited

Loading