Add chaos testing setup and experiment documentation

[XploY04]

This PR introduces a production-ready chaos testing framework for CloudNativePG clusters, combining Jepsen (formal consistency verification) with Litmus Chaos (pod deletion) to provide mathematical proof of data integrity under failure conditions.

Part of LFX Mentorship Program 2025/3 to strengthen CloudNativePG resilience through rigorous testing.

Closes #2

What's Included

chaos-testing/
├── README.md                          # Comprehensive documentation (1300+ lines)
├── pg-eu-cluster.yaml                 # PostgreSQL cluster config
├── litmus-rbac.yaml                   # Chaos permissions
├── experiments/
│   └── cnpg-jepsen-chaos.yaml         # Combined Jepsen + chaos test
├── workloads/
│   ├── jepsen-cnpg-job.yaml           # Jepsen test job
│   └── jepsen-results-pvc.yaml        # Result storage
└── scripts/
    ├── run-jepsen-chaos-test.sh       # Main orchestration
    ├── monitor-cnpg-pods.sh           # Real-time monitoring
    └── get-chaos-results.sh           # Result extraction

Test Workflow

1. Deploy CloudNativePG cluster (1 primary + 2 replicas)
2. Start Jepsen workload (continuous read/write operations)
3. Inject chaos (delete primary pod every 180s)
4. CloudNativePG performs automatic failover
5. Jepsen continues operations throughout chaos
6. Elle checker analyzes history for consistency violations
7. Generate verdict: :valid? true (PASS) or :valid? false (FAIL)

Quick Start

# Deploy PostgreSQL cluster
kubectl apply -f pg-eu-cluster.yaml
kubectl wait --for=condition=ready cluster/pg-eu --timeout=300s

# Configure chaos RBAC
kubectl apply -f litmus-rbac.yaml

# Run 5-minute test
./scripts/run-jepsen-chaos-test.sh

# Check results
grep ":valid?" logs/jepsen-chaos-*/results/results.edn
cat logs/jepsen-chaos-*/STATISTICS.txt

Expected Results

Successful Test

{:valid? true
 :anomaly-types []
 :not #{}}

Statistics

Total :ok     : 14,523  (97.03%)  # Successful operations
Total :fail   : 445     (2.97%)   # Expected during failover
Total :info   : 0       (0.00%)   # Indeterminate operations

Result Files

• results.edn - Consistency verdict (:valid? true/false)
• history.edn - Complete operation log (3-6 MB)
• timeline.html - Interactive visualization
• STATISTICS.txt - High-level summary
• jepsen.log - Full test execution logs

Technical Highlights

Dynamic Pod Targeting

TARGETS: "deployment:default:[cnpg.io/cluster=pg-eu,cnpg.io/instanceRole=primary]:intersection"

Comprehensive Probes

• Start-of-test: Cluster health check
• End-of-test: Cluster recovery verification
• Continuous: Replication lag monitoring (Prometheus)
• Continuous: Primary availability check
• Continuous: Cluster status validation

Configurable Parameters

• Test duration: 60s to 1800s+
• Chaos interval: 30s to 300s
• Operation rate: 10-100+ ops/sec
• Concurrency: 1-20 workers
• Isolation levels: read-committed, repeatable-read, serializable

Testing Performed

• ✅ Baseline tests (no chaos)
• ✅ Primary pod deletion
• ✅ Multiple failover cycles
• ✅ Different isolation levels
• ✅ Various chaos intervals

Future Enhancements

• Network partition testing
• Storage failure simulation
• Multi-cluster testing
• Performance regression detection
• CI/CD pipeline templates

Signed-off-by: XploY04 2004agarwalyash@gmail.com

[gbartolini]

Please use the latest version of the operator in the installation instructions:

kubectl apply -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.20/releases/cnpg-1.20.0.yaml

[sxd]

This command line doesn't work is not v1.27.1 for the version and amd64 isn't in the binary to download

[sxd]

probably something like

export KUBECONFIG=$PWD/k8s/kube-config.yaml

Since you already inside the cnpg-playground directory

[sxd]

I will keep one or the other, but not both, it's confusing my first question was "Why I'm installing the same twice?"

[sxd]

On this test it's important to mention that it's required to increase the max open files otherwise it's not working, this is on the playground IIRC

[XploY04]

I didn't think about this. I have added that.

[sxd]

Where was this added ? Because I can't see it

[XploY04]

• Increase max open files limit if needed (required for Jepsen on some systems):

  ulimit -n 65536

In Prerequisites before running the script.

[sxd]

This failed with the following error:

error: the namespace from the provided object "default" does not match the namespace "litmus". You must pass '--namespace=default' to perform this operation.

[sxd]

and I do what? here is not clear if I should exit or not at some point, I'm guessing yes

[sxd]

This failed with the following error

runner_pod=$(kubectl -n litmus get pods -l chaos-runner-name=cnpg-jepsen-chaos-noprobes -o jsonpath='{.items[0].metadata.name}') && \
kubectl -n litmus logs -f "$runner_pod"
error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:
	template was:
		{.items[0].metadata.name}
	object given to jsonpath engine was:
		map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}

[sxd]

This throw the following error

 kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
Warning: resource namespaces/monitoring is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/monitoring configured

[sxd]

Now on this step I run out of space, having 20G of space, this should be clarified, my test just stop here because of running out of space

[XploY04]

I added this now.

[gbartolini]

Given that you have the plugin installed, I would rely on the plugin to install the latest version of the operator. See https://github.com/cloudnative-pg/cnpg-playground/blob/main/demo/setup.sh#L65

[gbartolini]

Can you please revisit this with the new feature for monitoring directly introduced in the cnpg-playground?

See: https://github.com/cloudnative-pg/cnpg-playground?tab=readme-ov-file#monitoring-with-prometheus-and-grafana

[XploY04]

Made the necessary changes.

[gbartolini]

I have reviewed the README file, making some minor adjustments, and followed the instructions.

I am not an expert in Jepsen, but I was able to run it and the diagrams made sense.

In the instructions that you shared at the end of the script though there are some aspects that I wasn't able to completely review. In particular:

• I had no SUMMARY.txt file after Jepsen's run
• The results.edn file was empty

I am merging the patch, but I would like you @XploY04 to revisit that part after.