With operator 0.3.7, nodes never get to vote (loop restarts) (similar to #222)

[OdyX]

Hello there,

I have now had time to upgrade our operator to 0.3.7, and I still experience the issue described in #222 . Again, reverting to 0.3.5 fixes these issue.

This is on a somwhat standard kubernetes, 1-replica setup:

---
apiVersion: ts.opentelekomcloud.com/v1alpha1
kind: TypesenseCluster
metadata:
  name: zebra-preprod
spec:
  image: mirror.gcr.io/typesense/typesense:29.0
  replicas: 1
  storage:
    size: "128Mi"
    storageClassName: csi-cinder-sc-delete-wait
  adminApiKey:
    name: typesense-zebra-preprod-bootstrap-key
  healthProbeTimeoutInMilliseconds: 10000
  incrementalQuorumRecovery: true
  resources:
    limits:
      cpu: 1
      memory: 768Mi
    requests:
      memory: 768Mi
  metrics:
    release: 'undefined'
    resources:
      limits:
        cpu: 100m
        memory: 64Mi
      requests:
        memory: 64Mi
  healthcheck:
    resources:
      limits:
        cpu: 100m
        memory: 32Mi
      requests:
        memory: 32Mi

With 0.3.7:

I20260129 09:56:20.883572   291 raft_server.cpp:605] Finished loading collections from disk.
I20260129 09:56:20.883666   291 raft_server.cpp:616] Loaded 0conversation model(s).
I20260129 09:56:20.883678   291 raft_server.cpp:620] Initializing batched indexer from snapshot state...
I20260129 09:56:20.883739   291 batched_indexer.cpp:635] Restored 0 in-flight requests from snapshot.
I20260129 09:56:20.883786   291 raft_server.cpp:633] Loaded 0 personalization model(s).
I20260129 09:56:20.883822   291 raft_server.h:294] Configuration of this group is 10.64.95.250:8107:8108
I20260129 09:56:20.883937   291 snapshot_executor.cpp:264] node default_group:10.64.67.24:8107:8108 snapshot_load_done, last_included_index: 6367974 last_included_term: 605 peers: "10.64.95.250:8107:8108"
I20260129 09:56:20.885480   263 raft_meta.cpp:521] Loaded single stable meta, path /usr/share/typesense/data/state/meta term 607 votedfor 0.0.0.0:0:0 time: 1235
I20260129 09:56:20.885545   263 node.cpp:608] node default_group:10.64.67.24:8107:8108 init, term: 607 last_log_id: (index=6367975,term=605) conf: 10.64.95.250:8107:8108 old_conf:
I20260129 09:56:20.885615   263 raft_server.cpp:141] Node last_index: 6367975
I20260129 09:56:20.885628   263 typesense_server_utils.cpp:309] Typesense peering service is running on 10.64.67.24:8107
I20260129 09:56:20.885643   263 typesense_server_utils.cpp:310] Snapshot interval configured as: 3600s
I20260129 09:56:20.885654   263 typesense_server_utils.cpp:311] Snapshot max byte count configured as: 4194304
W20260129 09:56:20.885668   263 controller.cpp:1550] SIGINT was installed with 1
I20260129 09:56:20.885769   263 raft_server.cpp:692] Term: 607, pending_queue: 0, last_index: 6367975, committed: 0, known_applied: 6367974, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 31837413
W20260129 09:56:20.885792   263 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
W20260129 09:56:20.885810   263 node.cpp:926] node default_group:10.64.67.24:8107:8108 set_peer from 10.64.95.250:8107:8108 to 10.64.67.47:8107:8108
I20260129 09:56:20.891244   263 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 608 votedfor 0.0.0.0:0:0 time: 5383
I20260129 09:56:26.165112   285 node.cpp:1579] node default_group:10.64.67.24:8107:8108 term 608 start pre_vote
W20260129 09:56:26.165217   285 node.cpp:1589] node default_group:10.64.67.24:8107:8108 can't do pre_vote as it is not in 10.64.67.47:8107:8108
> ..... More log lines, but check the timestamps
I20260129 09:57:54.755228   287 node.cpp:1579] node default_group:10.64.67.24:8107:8108 term 608 start pre_vote
W20260129 09:57:54.755301   287 node.cpp:1589] node default_group:10.64.67.24:8107:8108 can't do pre_vote as it is not in 10.64.67.47:8107:8108
I20260129 09:57:59.898531   287 node.cpp:1579] node default_group:10.64.67.24:8107:8108 term 608 start pre_vote
W20260129 09:57:59.898599   287 node.cpp:1589] node default_group:10.64.67.24:8107:8108 can't do pre_vote as it is not in 10.64.67.47:8107:8108
I20260129 09:58:00.904165   263 raft_server.cpp:692] Term: 608, pending_queue: 0, last_index: 6367975, committed: 0, known_applied: 6367974, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 31837413
W20260129 09:58:00.904224   263 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260129 09:58:03.886561     1 typesense_server_utils.cpp:60] Stopping Typesense server...
> … until it dies

The same single-replica cluster, on 0.3.5:

I20260129 10:02:17.028793   287 raft_server.cpp:605] Finished loading collections from disk.
I20260129 10:02:17.028932   287 raft_server.cpp:616] Loaded 0conversation model(s).
I20260129 10:02:17.028945   287 raft_server.cpp:620] Initializing batched indexer from snapshot state...
I20260129 10:02:17.028995   287 batched_indexer.cpp:635] Restored 0 in-flight requests from snapshot.
I20260129 10:02:17.029036   287 raft_server.cpp:633] Loaded 0 personalization model(s).
I20260129 10:02:17.029070   287 raft_server.h:294] Configuration of this group is 10.64.95.250:8107:8108
I20260129 10:02:17.029173   287 snapshot_executor.cpp:264] node default_group:10.64.67.131:8107:8108 snapshot_load_done, last_included_index: 6367974 last_included_term: 605 peers: "10.64.95.250:8107:8108"
I20260129 10:02:17.030550   263 raft_meta.cpp:521] Loaded single stable meta, path /usr/share/typesense/data/state/meta term 610 votedfor 0.0.0.0:0:0 time: 1179
I20260129 10:02:17.030592   263 node.cpp:608] node default_group:10.64.67.131:8107:8108 init, term: 610 last_log_id: (index=6367975,term=605) conf: 10.64.95.250:8107:8108 old_conf:
I20260129 10:02:17.030647   263 raft_server.cpp:141] Node last_index: 6367975
I20260129 10:02:17.030664   263 typesense_server_utils.cpp:309] Typesense peering service is running on 10.64.67.131:8107
I20260129 10:02:17.030674   263 typesense_server_utils.cpp:310] Snapshot interval configured as: 3600s
I20260129 10:02:17.030683   263 typesense_server_utils.cpp:311] Snapshot max byte count configured as: 4194304
W20260129 10:02:17.030730   263 controller.cpp:1550] SIGINT was installed with 1
I20260129 10:02:17.030818   263 raft_server.cpp:692] Term: 610, pending_queue: 0, last_index: 6367975, committed: 0, known_applied: 6367974, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 31837413
W20260129 10:02:17.030834   263 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
W20260129 10:02:17.030848   263 node.cpp:926] node default_group:10.64.67.131:8107:8108 set_peer from 10.64.95.250:8107:8108 to 10.64.67.197:8107:8108
I20260129 10:02:17.036015   263 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 611 votedfor 0.0.0.0:0:0 time: 5138
I20260129 10:02:22.091174   278 node.cpp:1579] node default_group:10.64.67.131:8107:8108 term 611 start pre_vote
W20260129 10:02:22.091253   278 node.cpp:1589] node default_group:10.64.67.131:8107:8108 can't do pre_vote as it is not in 10.64.67.197:8107:8108
> … it tries for some time, then finally:
W20260129 10:03:37.046319   263 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
W20260129 10:03:37.046335   263 node.cpp:926] node default_group:10.64.67.131:8107:8108 set_peer from 10.64.67.197:8107:8108 to 10.64.67.131:8107:8108
I20260129 10:03:37.054266   263 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 612 votedfor 0.0.0.0:0:0 time: 7861
I20260129 10:03:39.442730   278 node.cpp:1579] node default_group:10.64.67.131:8107:8108 term 612 start pre_vote
I20260129 10:03:39.442939   278 node.cpp:1645] node default_group:10.64.67.131:8107:8108 term 612 start vote and grant vote self
I20260129 10:03:39.448323   287 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 613 votedfor 10.64.67.131:8107:8108 time: 5227
I20260129 10:03:39.448383   287 node.cpp:1899] node default_group:10.64.67.131:8107:8108 term 613 become leader of group 10.64.67.131:8107:8108
I20260129 10:03:39.453203   287 raft_server.h:294] Configuration of this group is 10.64.67.131:8107:8108
I20260129 10:03:39.453270   287 node.cpp:3298] node default_group:10.64.67.131:8107:8108 reset ConfigurationCtx, new_peers: 10.64.67.131:8107:8108, old_peers: 10.64.67.131:8107:8108
I20260129 10:03:39.453289   287 raft_server.h:277] Node becomes leader, term: 613

Happy to provide more inputs or logs. I also manage a second 3-nodes cluster, that exhibits the same issues.

[akyriako]

Please provide the operator logs for these two cluster as well so I can investigate better.

I tried your configuration in a bare-metal k8s cluster and it went ready in less than 30secs (provisioning of the storage mainly delayed it). Do you have anything special in your environment?

[OdyX]

We're running in an Infomaniak managed Kubernetes.

The single-replica cluster pod log:

I20260129 12:40:42.074167   298 raft_server.cpp:605] Finished loading collections from disk.
(…)
I20260129 12:42:26.931419   294 node.cpp:1579] node default_group:10.64.76.153:8107:8108 term 615 start pre_vote
W20260129 12:42:26.931496   294 node.cpp:1589] node default_group:10.64.76.153:8107:8108 can't do pre_vote as it is not in 10.64.76.249:8107:8108
I20260129 12:42:30.721947     1 typesense_server_utils.cpp:60] Stopping Typesense server...

And the relevant lines from the operator:

…
2026-01-29T12:38:29.960Z        DEBUG   events  Quorum Is Not Ready     {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"zebra-preprod","uid":"16d57c8e-2d74-4e79-b636-c04487cf901a","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"394253635"}, "reason": "QuorumNotReadyWaitATerm"}
2026-01-29T12:40:29.960Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:29.961Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:29.961Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:29.961Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:29.961Z        DEBUG   current quorum configuration    {"namespace": "typesense", "cluster": "zebra-preprod", "size": 1, "nodes": ["10.64.76.168:8107:8108"]}
2026-01-29T12:40:29.961Z        INFO    updating quorum configuration   {"namespace": "typesense", "cluster": "zebra-preprod", "size": 1, "nodes": ["10.64.76.249:8107:8108"]}
2026-01-29T12:40:29.966Z        DEBUG   reconciling services    {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:29.966Z        DEBUG   reconciling ingress     {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:29.966Z        DEBUG   reconciling scrapers    {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:30.055Z        DEBUG   reconciling podmonitor  {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:30.055Z        DEBUG   reconciling statefulset {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:30.055Z        DEBUG   calculated hash {"namespace": "typesense", "cluster": "zebra-preprod", "hash": "2472f3923917599de20a47f2bb783285597dddbdc8b5057037676b60e5e7b1a1"}
2026-01-29T12:40:30.055Z        DEBUG   updating statefulset    {"namespace": "typesense", "cluster": "zebra-preprod", "sts": "zebra-preprod-sts"}
2026-01-29T12:40:30.153Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:30.153Z        DEBUG   reporting lag thresholds        {"namespace": "typesense", "cluster": "zebra-preprod", "read": "1000", "write": "500"}
2026-01-29T12:40:30.313Z        INFO    reconciling quorum health       {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:40:30.313Z        INFO    calculated quorum       {"namespace": "typesense", "cluster": "zebra-preprod", "minRequiredNodes": 1, "availableNodes": 1}
2026-01-29T12:40:30.313Z        DEBUG   events  Wait On Configmap Updates       {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"zebra-preprod","uid":"16d57c8e-2d74-4e79-b636-c04487cf901a","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"394255656"}, "reason": "QuorumNotReadyWaitATerm"}
2026-01-29T12:40:30.353Z        DEBUG   reporting node status   {"namespace": "typesense", "cluster": "zebra-preprod", "node": "zebra-preprod-sts-0", "state": "NOT_READY", "ip": "10.64.76.249", "queued_writes": 0, "commited_index": 0}
2026-01-29T12:40:30.354Z        DEBUG   reporting cluster status        {"namespace": "typesense", "cluster": "zebra-preprod", "status": "NOT_READY"}
2026-01-29T12:40:30.354Z        DEBUG   reporting node health   {"namespace": "typesense", "cluster": "zebra-preprod", "node": "zebra-preprod-sts-0", "healthy": false, "ip": "10.64.76.249"}
2026-01-29T12:40:30.452Z        INFO    evaluated quorum        {"namespace": "typesense", "cluster": "zebra-preprod", "minRequiredNodes": 1, "availableNodes": 1, "healthyNodes": 0}
2026-01-29T12:40:30.458Z        INFO    reconciling cluster completed   {"namespace": "typesense", "cluster": "zebra-preprod", "condition": "QuorumNotReadyWaitATerm", "requeueAfter": "2m0s"}
2026-01-29T12:40:30.458Z        DEBUG   events  Quorum Is Not Ready     {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"zebra-preprod","uid":"16d57c8e-2d74-4e79-b636-c04487cf901a","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"394255663"}, "reason": "QuorumNotReadyWaitATerm"}
2026-01-29T12:42:30.459Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.459Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.459Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.460Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.460Z        DEBUG   current quorum configuration    {"namespace": "typesense", "cluster": "zebra-preprod", "size": 1, "nodes": ["10.64.76.249:8107:8108"]}
2026-01-29T12:42:30.460Z        INFO    updating quorum configuration   {"namespace": "typesense", "cluster": "zebra-preprod", "size": 1, "nodes": ["10.64.76.153:8107:8108"]}
2026-01-29T12:42:30.465Z        DEBUG   reconciling services    {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.465Z        DEBUG   reconciling ingress     {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.465Z        DEBUG   reconciling scrapers    {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.552Z        DEBUG   reconciling podmonitor  {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.552Z        DEBUG   reconciling statefulset {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.553Z        DEBUG   calculated hash {"namespace": "typesense", "cluster": "zebra-preprod", "hash": "2472f3923917599de20a47f2bb783285597dddbdc8b5057037676b60e5e7b1a1"}
2026-01-29T12:42:30.553Z        DEBUG   updating statefulset    {"namespace": "typesense", "cluster": "zebra-preprod", "sts": "zebra-preprod-sts"}
2026-01-29T12:42:30.660Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.660Z        DEBUG   reporting lag thresholds        {"namespace": "typesense", "cluster": "zebra-preprod", "read": "1000", "write": "500"}
2026-01-29T12:42:30.755Z        INFO    reconciling quorum health       {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-01-29T12:42:30.755Z        INFO    calculated quorum       {"namespace": "typesense", "cluster": "zebra-preprod", "minRequiredNodes": 1, "availableNodes": 1}
2026-01-29T12:42:30.755Z        DEBUG   events  Wait On Configmap Updates       {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"zebra-preprod","uid":"16d57c8e-2d74-4e79-b636-c04487cf901a","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"394257761"}, "reason": "QuorumNotReadyWaitATerm"}
2026-01-29T12:42:30.764Z        DEBUG   reporting node status   {"namespace": "typesense", "cluster": "zebra-preprod", "node": "zebra-preprod-sts-0", "state": "FOLLOWER", "ip": "10.64.76.153", "queued_writes": 0, "commited_index": 0}
2026-01-29T12:42:30.764Z        DEBUG   reporting cluster status        {"namespace": "typesense", "cluster": "zebra-preprod", "status": "NOT_READY"}
2026-01-29T12:42:30.764Z        DEBUG   reporting node health   {"namespace": "typesense", "cluster": "zebra-preprod", "node": "zebra-preprod-sts-0", "healthy": false, "ip": "10.64.76.153"}
2026-01-29T12:42:30.865Z        INFO    evaluated quorum        {"namespace": "typesense", "cluster": "zebra-preprod", "minRequiredNodes": 1, "availableNodes": 1, "healthyNodes": 0}
2026-01-29T12:42:30.952Z        INFO    reconciling cluster completed   {"namespace": "typesense", "cluster": "zebra-preprod", "condition": "QuorumNotReadyWaitATerm", "requeueAfter": "2m0s"}
2026-01-29T12:42:30.952Z        DEBUG   events  Quorum Is Not Ready     {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"zebra-preprod","uid":"16d57c8e-2d74-4e79-b636-c04487cf901a","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"394257767"}, "reason": "QuorumNotReadyWaitATerm"}
…

And its resulting StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: '2025-10-01T09:54:25Z'
  generation: 578
  labels:
    app: zebra-preprod-sts
  name: zebra-preprod-sts
  namespace: typesense
  ownerReferences:
    - apiVersion: ts.opentelekomcloud.com/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: TypesenseCluster
      name: zebra-preprod
      uid: 16d57c8e-2d74-4e79-b636-c04487cf901a
  resourceVersion: '394259964'
  uid: 88482ff9-2131-4bb5-af86-33b3624b34b7
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: zebra-preprod-sts
  serviceName: zebra-preprod-sts-svc
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: '2026-01-29T12:44:31Z'
        ts.opentelekomcloud.com/pod-template-hash: 2472f3923917599de20a47f2bb783285597dddbdc8b5057037676b60e5e7b1a1
        ts.opentelekomcloud.com/read-lag-threshold: '1000'
        ts.opentelekomcloud.com/write-lag-threshold: '500'
      creationTimestamp: null
      labels:
        app: zebra-preprod-sts
      name: zebra-preprod-sts
      namespace: typesense
    spec:
      containers:
        - env:
            - name: TYPESENSE_API_KEY
              valueFrom:
                secretKeyRef:
                  key: typesense-api-key
                  name: typesense-zebra-preprod-bootstrap-key
            - name: TYPESENSE_NODES
              value: /usr/share/typesense/nodes
            - name: TYPESENSE_DATA_DIR
              value: /usr/share/typesense/data
            - name: TYPESENSE_API_PORT
              value: '8108'
            - name: TYPESENSE_PEERING_PORT
              value: '8107'
            - name: TYPESENSE_PEERING_ADDRESS
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: TYPESENSE_ENABLE_CORS
              value: 'false'
            - name: TYPESENSE_CORS_DOMAINS
            - name: TYPESENSE_RESET_PEERS_ON_ERROR
              value: 'true'
          image: mirror.gcr.io/typesense/typesense:29.0
          imagePullPolicy: IfNotPresent
          name: typesense
          ports:
            - containerPort: 8108
              name: http
              protocol: TCP
          resources:
            limits:
              cpu: '1'
              memory: 768Mi
            requests:
              memory: 768Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            runAsUser: 10000000
            seccompProfile:
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/share/typesense
              name: nodeslist
            - mountPath: /usr/share/typesense/data
              name: data
        - env:
            - name: TYPESENSE_API_KEY
              valueFrom:
                secretKeyRef:
                  key: typesense-api-key
                  name: typesense-zebra-preprod-bootstrap-key
            - name: LOG_LEVEL
              value: '0'
            - name: TYPESENSE_PROTOCOL
              value: http
            - name: TYPESENSE_HOST
              value: localhost
            - name: TYPESENSE_PORT
              value: '8108'
            - name: METRICS_PORT
              value: '9100'
            - name: TYPESENSE_CLUSTER
              value: zebra-preprod
          image: akyriako78/typesense-prometheus-exporter:0.1.9
          imagePullPolicy: IfNotPresent
          name: metrics-exporter
          ports:
            - containerPort: 9100
              name: metrics
              protocol: TCP
          resources:
            limits:
              cpu: 100m
              memory: 64Mi
            requests:
              memory: 64Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            runAsUser: 10000000
            seccompProfile:
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        - env:
            - name: TYPESENSE_API_KEY
              valueFrom:
                secretKeyRef:
                  key: typesense-api-key
                  name: typesense-zebra-preprod-bootstrap-key
            - name: LOG_LEVEL
              value: '0'
            - name: TYPESENSE_PROTOCOL
              value: http
            - name: TYPESENSE_API_PORT
              value: '8108'
            - name: TYPESENSE_PEERING_PORT
              value: '8107'
            - name: HEALTHCHECK_PORT
              value: '8808'
            - name: TYPESENSE_NODES
              value: /usr/share/typesense/fallback
            - name: CLUSTER_NAMESPACE
              value: typesense
          image: akyriako78/typesense-healthcheck:0.1.8
          imagePullPolicy: IfNotPresent
          name: healthcheck
          ports:
            - containerPort: 8808
              name: healthcheck
              protocol: TCP
          resources:
            limits:
              cpu: 100m
              memory: 32Mi
            requests:
              memory: 32Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            runAsUser: 10000000
            seccompProfile:
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/share/typesense
              name: nodeslist
              readOnly: true
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: dockerhub-secret
      readinessGates:
        - conditionType: RaftQuorumReady
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 2000
        runAsGroup: 3000
        runAsNonRoot: true
        runAsUser: 10000
      terminationGracePeriodSeconds: 5
      volumes:
        - configMap:
            defaultMode: 420
            name: zebra-preprod-nodeslist
          name: nodeslist
        - name: data
          persistentVolumeClaim:
            claimName: data
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        labels:
          app: zebra-preprod-sts
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 128Mi
        storageClassName: csi-cinder-sc-delete-wait
        volumeMode: Filesystem
      status:
        phase: Pending
status:
  availableReplicas: 0
  collisionCount: 0
  currentRevision: zebra-preprod-sts-84684888bd
  observedGeneration: 578
  replicas: 1
  updateRevision: zebra-preprod-sts-5dbf99fc6d
  updatedReplicas: 1

[akyriako]

Thanks for the logs. I haven't managed to emulate your case yet. The operator logs have a weird trace though, that is happening in every term:

2026-01-29T12:40:30.055Z DEBUG updating statefulset {"namespace": "typesense", "cluster": "zebra-preprod", "sts": "zebra-preprod-sts"}

That shouldn't take place in every loop. It means something is changing in your cluster/sts specs and the controller tries to apply an updated manifest, which leads to replacing the pods, which means you're getting new IPs, which means...that is why it is trying constantly to establish a new quorum. But that is not where it stops, it is updating the sts in two consecutive terms (maybe I need to see more from the logs) but the hash of the sts remains the same:

2026-01-29T12:42:30.553Z DEBUG calculated hash {"namespace": "typesense", "cluster": "zebra-preprod", "hash": "2472f3923917599de20a47f2bb783285597dddbdc8b5057037676b60e5e7b1a1"}

which means that the above assumption that "something is changing in your cluster/sts specs" is not entirelly correct because there is no hash change BUT there might be annotations changes that could trigger this update. Do you by any chance have something else in your clusters (completely out of the scope of this controller I mean) that might inject annotations in your workloads? That could create an antagonistic environment for the controller and would explain the constant update of the statefulset (as I said I might need to see more terms from the logs to be sure this is the norm and not an coincidence).

[OdyX]

Do you by any chance have something else in your clusters (completely out of the scope of this controller I mean) that might inject annotations in your workloads? That could create an antagonistic environment for the controller and would explain the constant update of the statefulset (as I said I might need to see more terms from the logs to be sure this is the norm and not an coincidence).

Yes: this is deployed using ArgoCD, and I suspect Kyverno might be affecting this as well. But ArgoCD only generates (and maintains the TypesenseCluster object.

To me, it rather looks like that in our setup, the nodesList configMap is not updated at the right time. As you see, the configMap only gets the correct node shortly before the pod dies (the null podIP), then keeps it for just long enough for the pod to never get it's correct value.

$ while true; do echo ""; echo -n "Timestamp: "; LANG=C date; echo -n "Pod IP:    "; kubectl get -n typesense pod/zebra-preprod-sts-0 -o json | jq -r .status.podIP ; echo -n "NodesList: "; kubectl get -n typesense cm/zebra-preprod-nodeslist -o json | jq -r .data.nodes; sleep 3; done

Timestamp: Mon Feb  2 16:19:44 CET 2026
Pod IP:    10.64.76.53
NodesList: 10.64.76.85:8107:8108

Timestamp: Mon Feb  2 16:19:47 CET 2026
Pod IP:    10.64.76.53
NodesList: 10.64.76.53:8107:8108

Timestamp: Mon Feb  2 16:19:51 CET 2026
Pod IP:    null
NodesList: 10.64.76.53:8107:8108

Timestamp: Mon Feb  2 16:19:55 CET 2026
Pod IP:    null
NodesList: 10.64.76.53:8107:8108

Timestamp: Mon Feb  2 16:19:58 CET 2026
Pod IP:    10.64.76.119
NodesList: 10.64.76.53:8107:8108
(…)
Timestamp: Mon Feb  2 16:21:48 CET 2026
Pod IP:    10.64.76.119
NodesList: 10.64.76.53:8107:8108

Timestamp: Mon Feb  2 16:21:51 CET 2026
Pod IP:    null
NodesList: 10.64.76.119:8107:8108

Timestamp: Mon Feb  2 16:21:55 CET 2026
Pod IP:    null
NodesList: 10.64.76.119:8107:8108

Timestamp: Mon Feb  2 16:21:58 CET 2026
Pod IP:    null
NodesList: 10.64.76.119:8107:8108

Timestamp: Mon Feb  2 16:22:02 CET 2026
Pod IP:    10.64.76.83
NodesList: 10.64.76.119:8107:8108
(…)
Timestamp: Mon Feb  2 16:23:47 CET 2026
Pod IP:    10.64.76.83
NodesList: 10.64.76.119:8107:8108

Timestamp: Mon Feb  2 16:23:51 CET 2026
Pod IP:    10.64.76.83
NodesList: 10.64.76.83:8107:8108

Timestamp: Mon Feb  2 16:23:54 CET 2026
Pod IP:    null
NodesList: 10.64.76.83:8107:8108

Timestamp: Mon Feb  2 16:23:58 CET 2026
Pod IP:    null
NodesList: 10.64.76.83:8107:8108

Timestamp: Mon Feb  2 16:24:01 CET 2026
Pod IP:    null
NodesList: 10.64.76.83:8107:8108

Timestamp: Mon Feb  2 16:24:05 CET 2026
Pod IP:    10.64.76.33
NodesList: 10.64.76.83:8107:8108
(…)

[OdyX]

Ah. I see the configMap takes about 2 minutes to be updated by the operator. I'm not sure I understand why the operator waits so long rather than just applying it when reconciling the cluster. And it looks like a given node might also try to pre_vote for about 2 minutes before giving up. Badly-timed updates make this fail almost every time?

[akyriako]

Try 0.3.8-rc.1

[OdyX]

Tested 0.3.8-rc.1, but it doesn't help, unfortunately.

$ date -u; kubectl -n typesense delete pod/zebra-preprod-sts-0; while sleep 5; do echo; date -u; echo -n "configmap : "; kubectl -n typesense get cm/zebra-preprod-nodeslist -ojson | jq -r .data.nodes; echo -n "podIP     : "; kubectl -n typesense get pod/zebra-preprod-sts-0 -o json  | jq -r .status.podIP
done

Thu Feb  5 10:27:14 UTC 2026
pod "zebra-preprod-sts-0" deleted

Thu Feb  5 10:27:21 UTC 2026
configmap: 10.64.76.164:8107:8108
podIP   : 10.64.76.108

Thu Feb  5 10:27:26 UTC 2026
configmap: 10.64.76.164:8107:8108
podIP   : 10.64.76.108
^C%                                                                                                                                                                                                                                                                                                                        ➜  typesense git:(main) ✗ LANG=C; date -u; kubectl -n typesense delete pod/zebra-preprod-sts-0; while sleep 5; do echo; date -u; echo -n "configmap : "; kubectl -n typesense get cm/zebra-preprod-nodeslist -ojson | jq -r .data.nodes; echo -n "podIP     : "; kubectl -n typesense get pod/zebra-preprod-sts-0 -o json  | jq -r .status.podIP
done

Thu Feb  5 10:27:38 UTC 2026
pod "zebra-preprod-sts-0" deleted

Thu Feb  5 10:27:46 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : null

Thu Feb  5 10:27:52 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:27:57 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:02 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:08 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:13 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:18 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:23 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:29 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:34 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:39 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:45 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:50 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:28:55 UTC 2026
configmap : 10.64.76.164:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:29:00 UTC 2026
configmap : 10.64.76.3:8107:8108
podIP     : 10.64.76.3

Thu Feb  5 10:29:06 UTC 2026
configmap : 10.64.76.3:8107:8108
podIP     : null

So:

• I delete the pod at 10:27:38
• the configmap gets the new IP at 10:29:00

Between these timestamps, the operator logs show:

2026-02-05T10:28:58.831Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.831Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.831Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.831Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.831Z        DEBUG   current quorum configuration    {"namespace": "typesense", "cluster": "zebra-preprod", "size": 1, "nodes": ["10.64.76.164:8107:8108"]}
2026-02-05T10:28:58.831Z        INFO    updating quorum configuration   {"namespace": "typesense", "cluster": "zebra-preprod", "size": 1, "nodes": ["10.64.76.3:8107:8108"]}
2026-02-05T10:28:58.835Z        DEBUG   reconciling services    {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.835Z        DEBUG   reconciling ingress     {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.835Z        DEBUG   reconciling scrapers    {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.930Z        DEBUG   reconciling podmonitor  {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.930Z        DEBUG   reconciling statefulset {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.930Z        DEBUG   calculated hash {"namespace": "typesense", "cluster": "zebra-preprod", "hash": "2472f3923917599de20a47f2bb783285597dddbdc8b5057037676b60e5e7b1a1"}
2026-02-05T10:28:58.930Z        DEBUG   updating statefulset    {"namespace": "typesense", "cluster": "zebra-preprod", "sts": "zebra-preprod-sts"}
2026-02-05T10:28:58.997Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:58.997Z        DEBUG   reporting lag thresholds        {"namespace": "typesense", "cluster": "zebra-preprod", "read": "1000", "write": "500"}
2026-02-05T10:28:59.081Z        DEBUG   patching pod annotations        {"namespace": "typesense", "cluster": "zebra-preprod", "pod": "zebra-preprod-sts-0", "annotation": "ts.opentelekomcloud.com/forced-configmap-update-time"}
2026-02-05T10:28:59.088Z        INFO    reconciling quorum health       {"namespace": "typesense", "cluster": "zebra-preprod"}
2026-02-05T10:28:59.088Z        INFO    calculated quorum       {"namespace": "typesense", "cluster": "zebra-preprod", "minRequiredNodes": 1, "availableNodes": 1}
2026-02-05T10:28:59.129Z        DEBUG   events  Wait On Configmap Updates       {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"zebra-preprod","uid":"16d57c8e-2d74-4e79-b636-c04487cf901a","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404295928"}, "reason": "QuorumNotReadyWaitATerm"}
2026-02-05T10:28:59.232Z        DEBUG   reporting node status   {"namespace": "typesense", "cluster": "zebra-preprod", "node": "zebra-preprod-sts-0", "state": "FOLLOWER", "ip": "10.64.76.3", "queued_writes": 0, "commited_index": 0}
2026-02-05T10:28:59.232Z        DEBUG   reporting cluster status        {"namespace": "typesense", "cluster": "zebra-preprod", "status": "NOT_READY"}
2026-02-05T10:28:59.233Z        DEBUG   reporting node health   {"namespace": "typesense", "cluster": "zebra-preprod", "node": "zebra-preprod-sts-0", "healthy": false, "ip": "10.64.76.3"}
2026-02-05T10:28:59.244Z        INFO    evaluated quorum        {"namespace": "typesense", "cluster": "zebra-preprod", "minRequiredNodes": 1, "availableNodes": 1, "healthyNodes": 0}
2026-02-05T10:28:59.336Z        INFO    reconciling cluster completed   {"namespace": "typesense", "cluster": "zebra-preprod", "condition": "QuorumNotReadyWaitATerm", "requeueAfter": "2m0s"}
2026-02-05T10:28:59.336Z      

[akyriako]

This is the new addition. It one forces updating the ConfigMap when any changes take place, without waiting for the next reconciliation loop:

2026-02-05T10:28:59.081Z DEBUG patching pod annotations {"namespace": "typesense", "cluster": "zebra-preprod", "pod": "zebra-preprod-sts-0", "annotation": "ts.opentelekomcloud.com/forced-configmap-update-time"}

I still don't get why the Statefulset triggers an update in every loop in your environment (this is actually the culprit but is more of a symptom than the condition itself. I am afraid the situation is very specific to your environment. Not possible to recreate or emulate from my end and it has never been brought up by other users):

2026-02-05T10:28:58.930Z DEBUG updating statefulset {"namespace": "typesense", "cluster": "zebra-preprod", "sts": "zebra-preprod-sts"}

Let's make an experiment, as long as I cannot emulate it from my side. Please create a new TypesenseCluster (no prexisting or retained PVs/PVCs) and let's see how this one is going to behave.

[OdyX]

Okay (and trying to use the /details functionality of Github markdown.)

So… adding this cluster:

apiVersion: ts.opentelekomcloud.com/v1alpha1
kind: TypesenseCluster
metadata:
  name: ato-228
spec:
  image: mirror.gcr.io/typesense/typesense:29.0
  replicas: 1
  storage:
    size: "128Mi"
    storageClassName: csi-cinder-sc-delete-wait
  adminApiKey:
    name: typesense-ato-228-bootstrap-key
  resources:
    limits:
      cpu: "1"
      memory: "768Mi"
    requests:
      memory: "768Mi"
  metrics:
    release: 'undefined'
    resources:
      limits:
        cpu: 100m
        memory: 64Mi
      requests:
        memory: 64Mi
  healthcheck:
    resources:
      limits:
        cpu: 100m
        memory: 32Mi
      requests:
        memory: 32Mi

Related operator logs:

2026-02-05T11:44:51.498Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.504Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.504Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.504Z        DEBUG   creating config map     {"namespace": "typesense", "cluster": "ato-228", "configmap": "ato-228-nodeslist"}
2026-02-05T11:44:51.504Z        ERROR   unable to fetch statefulset: ato-228-sts        {"namespace": "typesense", "cluster": "ato-228", "error": "StatefulSet.apps \"ato-228-sts\" not found"}
2026-02-05T11:44:51.591Z        DEBUG   reconciling services    {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.591Z        DEBUG   creating headless service       {"namespace": "typesense", "cluster": "ato-228", "service": "ato-228-sts-svc"}
2026-02-05T11:44:51.603Z        DEBUG   creating resolver service       {"namespace": "typesense", "cluster": "ato-228", "service": "ato-228-svc"}
2026-02-05T11:44:51.636Z        DEBUG   reconciling ingress     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.636Z        DEBUG   reconciling scrapers    {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.747Z        DEBUG   reconciling podmonitor  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.747Z        DEBUG   creating podmonitor     {"namespace": "typesense", "cluster": "ato-228", "podmonitor": "ato-228-podmonitor"}
2026-02-05T11:44:51.752Z        DEBUG   reconciling statefulset {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:44:51.752Z        DEBUG   creating statefulset    {"namespace": "typesense", "cluster": "ato-228", "sts": "ato-228-sts"}
2026-02-05T11:44:51.753Z        DEBUG   calculated hash {"namespace": "typesense", "cluster": "ato-228", "hash": "5fea921d2ce3f5b082fc9a70b54268c87750c5b3e11ffa7eeb3ff789183277a9"}
2026-02-05T11:44:51.799Z        DEBUG   reporting lag thresholds        {"namespace": "typesense", "cluster": "ato-228", "read": "1000", "write": "500"}
2026-02-05T11:44:51.799Z        INFO    bootstrapping cluster completed {"namespace": "typesense", "cluster": "ato-228", "condition": "QuorumStateUnknown", "requeueAfter": "1m5s"}
2026-02-05T11:45:06.800Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.800Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.800Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.800Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.800Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.809Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "3bbbce09-cd68-4d4b-affe-1c739f93839f", "error": "empty quorum configuration"}
2026-02-05T11:45:06.814Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.814Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.814Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.814Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.815Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.821Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "390161e9-5a11-4ed3-8c22-fa75a14cd4ba", "error": "empty quorum configuration"}
2026-02-05T11:45:06.832Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.832Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.832Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.846Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.847Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.896Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "efa344ca-6a9d-47af-a790-1a2c96ba198f", "error": "empty quorum configuration"}
2026-02-05T11:45:06.916Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.916Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.916Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.916Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.917Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.925Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "78afe024-513d-4853-add9-b79a11fbca9b", "error": "empty quorum configuration"}
2026-02-05T11:45:06.965Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.965Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.965Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.965Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.966Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:06.973Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "825e5c4e-a0de-4abf-9979-aa9c148a6609", "error": "empty quorum configuration"}
2026-02-05T11:45:07.054Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.054Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.054Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.054Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.054Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.062Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "35db50f9-b671-4fe9-8d38-7bb50949e4ce", "error": "empty quorum configuration"}
2026-02-05T11:45:07.222Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.222Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.222Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.222Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.222Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.231Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "122b48b4-0648-4459-8697-5cd1ffc2de43", "error": "empty quorum configuration"}
2026-02-05T11:45:07.551Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.551Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.551Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.551Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.551Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:07.562Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "ba932c03-6d2f-4d6c-aa3f-567208f2144c", "error": "empty quorum configuration"}
2026-02-05T11:45:08.202Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:08.202Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:08.202Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:08.203Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:08.203Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:08.210Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "80536944-8627-4f55-90a8-cb626f93ed2e", "error": "empty quorum configuration"}
2026-02-05T11:45:09.491Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:09.491Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:09.491Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:09.492Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:09.495Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:09.502Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "0525cafb-2a31-43f1-8567-1d63fe68dd9a", "error": "empty quorum configuration"}
2026-02-05T11:45:12.062Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:12.062Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:12.062Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:12.063Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:12.063Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:12.070Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "1b99114f-887c-4aa7-8c2b-f9ffd8f98184", "error": "empty quorum configuration"}
2026-02-05T11:45:17.190Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:17.190Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:17.190Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:17.191Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:17.191Z        DEBUG   empty quorum configuration      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:17.198Z        ERROR   Reconciler error        {"controller": "typesensecluster", "controllerGroup": "ts.opentelekomcloud.com", "controllerKind": "TypesenseCluster", "TypesenseCluster": {"name":"ato-228","namespace":"typesense"}, "namespace": "typesense", "name": "ato-228", "reconcileID": "796fb127-cb3c-44f4-a1b6-1a0c2049aa17", "error": "empty quorum configuration"}
2026-02-05T11:45:27.438Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.439Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.439Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.439Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.439Z        DEBUG   current quorum configuration    {"namespace": "typesense", "cluster": "ato-228", "size": 1, "nodes": ["ato-228-sts-0.ato-228-sts-svc:8107:8108"]}
2026-02-05T11:45:27.439Z        INFO    updating quorum configuration   {"namespace": "typesense", "cluster": "ato-228", "size": 1, "nodes": ["10.64.85.12:8107:8108"]}
2026-02-05T11:45:27.443Z        DEBUG   reconciling services    {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.443Z        DEBUG   reconciling ingress     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.443Z        DEBUG   reconciling scrapers    {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.489Z        DEBUG   reconciling podmonitor  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.489Z        DEBUG   reconciling statefulset {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.490Z        DEBUG   calculated hash {"namespace": "typesense", "cluster": "ato-228", "hash": "5fea921d2ce3f5b082fc9a70b54268c87750c5b3e11ffa7eeb3ff789183277a9"}
2026-02-05T11:45:27.490Z        DEBUG   updating statefulset    {"namespace": "typesense", "cluster": "ato-228", "sts": "ato-228-sts"}
2026-02-05T11:45:27.549Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.549Z        DEBUG   reporting lag thresholds        {"namespace": "typesense", "cluster": "ato-228", "read": "1000", "write": "500"}
2026-02-05T11:45:27.627Z        DEBUG   patching pod annotations        {"namespace": "typesense", "cluster": "ato-228", "pod": "ato-228-sts-0", "annotation": "ts.opentelekomcloud.com/forced-configmap-update-time"}
2026-02-05T11:45:27.653Z        INFO    reconciling quorum health       {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:45:27.654Z        INFO    calculated quorum       {"namespace": "typesense", "cluster": "ato-228", "minRequiredNodes": 1, "availableNodes": 1}
2026-02-05T11:45:27.654Z        DEBUG   events  Wait On Configmap Updates       {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"ato-228","uid":"17e25276-5c27-4814-8a22-a02e2971b054","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404372648"}, "reason": "QuorumNotReadyWaitATerm"}
2026-02-05T11:45:27.689Z        DEBUG   reporting node status   {"namespace": "typesense", "cluster": "ato-228", "node": "ato-228-sts-0", "state": "NOT_READY", "ip": "10.64.85.12", "queued_writes": 0, "commited_index": 0}
2026-02-05T11:45:27.689Z        DEBUG   reporting cluster status        {"namespace": "typesense", "cluster": "ato-228", "status": "NOT_READY"}
2026-02-05T11:45:27.689Z        DEBUG   reporting node health   {"namespace": "typesense", "cluster": "ato-228", "node": "ato-228-sts-0", "healthy": false, "ip": "10.64.85.12"}
2026-02-05T11:45:27.798Z        INFO    evaluated quorum        {"namespace": "typesense", "cluster": "ato-228", "minRequiredNodes": 1, "availableNodes": 1, "healthyNodes": 0}
2026-02-05T11:45:27.805Z        INFO    reconciling cluster completed   {"namespace": "typesense", "cluster": "ato-228", "condition": "QuorumNotReadyWaitATerm", "requeueAfter": "2m0s"}
2026-02-05T11:45:27.806Z        DEBUG   events  Quorum Is Not Ready     {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"ato-228","uid":"17e25276-5c27-4814-8a22-a02e2971b054","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404372652"}, "reason": "QuorumNotReadyWaitATerm"}
2026-02-05T11:47:27.806Z        INFO    reconciling cluster     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.807Z        DEBUG   reconciling secret      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.807Z        DEBUG   reconciling config map  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.807Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.807Z        DEBUG   current quorum configuration    {"namespace": "typesense", "cluster": "ato-228", "size": 1, "nodes": ["10.64.85.12:8107:8108"]}
2026-02-05T11:47:27.807Z        INFO    updating quorum configuration   {"namespace": "typesense", "cluster": "ato-228", "size": 1, "nodes": ["10.64.85.78:8107:8108"]}
2026-02-05T11:47:27.811Z        DEBUG   reconciling services    {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.811Z        DEBUG   reconciling ingress     {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.811Z        DEBUG   reconciling scrapers    {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.817Z        DEBUG   reconciling podmonitor  {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.817Z        DEBUG   reconciling statefulset {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.817Z        DEBUG   calculated hash {"namespace": "typesense", "cluster": "ato-228", "hash": "5fea921d2ce3f5b082fc9a70b54268c87750c5b3e11ffa7eeb3ff789183277a9"}
2026-02-05T11:47:27.817Z        DEBUG   updating statefulset    {"namespace": "typesense", "cluster": "ato-228", "sts": "ato-228-sts"}
2026-02-05T11:47:27.888Z        DEBUG   collecting endpoint slices      {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.888Z        DEBUG   reporting lag thresholds        {"namespace": "typesense", "cluster": "ato-228", "read": "1000", "write": "500"}
2026-02-05T11:47:27.895Z        INFO    reconciling quorum health       {"namespace": "typesense", "cluster": "ato-228"}
2026-02-05T11:47:27.895Z        INFO    calculated quorum       {"namespace": "typesense", "cluster": "ato-228", "minRequiredNodes": 1, "availableNodes": 1}
2026-02-05T11:47:27.895Z        DEBUG   events  Wait On Configmap Updates       {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"ato-228","uid":"17e25276-5c27-4814-8a22-a02e2971b054","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404375007"}, "reason": "QuorumNotReadyWaitATerm"}
2026-02-05T11:47:28.096Z        DEBUG   reporting node status   {"namespace": "typesense", "cluster": "ato-228", "node": "ato-228-sts-0", "state": "FOLLOWER", "ip": "10.64.85.78", "queued_writes": 0, "commited_index": 0}
2026-02-05T11:47:28.097Z        DEBUG   reporting cluster status        {"namespace": "typesense", "cluster": "ato-228", "status": "NOT_READY"}
2026-02-05T11:47:28.097Z        DEBUG   reporting node health   {"namespace": "typesense", "cluster": "ato-228", "node": "ato-228-sts-0", "healthy": false, "ip": "10.64.85.78"}
2026-02-05T11:47:28.107Z        INFO    evaluated quorum        {"namespace": "typesense", "cluster": "ato-228", "minRequiredNodes": 1, "availableNodes": 1, "healthyNodes": 0}
2026-02-05T11:47:28.114Z        INFO    reconciling cluster completed   {"namespace": "typesense", "cluster": "ato-228", "condition": "QuorumNotReadyWaitATerm", "requeueAfter": "2m0s"}
2026-02-05T11:47:28.114Z        DEBUG   events  Quorum Is Not Ready     {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"ato-228","uid":"17e25276-5c27-4814-8a22-a02e2971b054","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404375017"}, "reason": "QuorumNotReadyWaitATerm"}

The one pod logs:

I20260205 11:45:42.140089     1 typesense_server_utils.cpp:362] Starting Typesense 29.0
I20260205 11:45:42.140148     1 typesense_server_utils.cpp:365] Typesense is using jemalloc.
I20260205 11:45:42.140574     1 typesense_server_utils.cpp:432] Thread pool size: 64
I20260205 11:45:42.152009     1 store.cpp:39] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20260205 11:45:42.180691     1 store.cpp:39] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20260205 11:45:42.211027     1 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20260205 11:45:42.211059     1 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20260205 11:45:42.211110     1 typesense_server_utils.cpp:536] Loaded 0 natural language search model(s).
I20260205 11:45:42.211194     1 typesense_server_utils.cpp:608] Starting API service...
I20260205 11:45:42.211457     1 http_server.cpp:180] Typesense has started listening on port 8108
I20260205 11:45:42.211584   263 batched_indexer.cpp:196] Starting batch indexer with 64 threads.
I20260205 11:45:42.211730   265 typesense_server_utils.cpp:551] Conversation garbage collector thread started.
I20260205 11:45:42.216655   263 batched_indexer.cpp:201] BatchedIndexer skip_index: -9999
I20260205 11:45:42.217308   341 batched_indexer.cpp:353] Starting reference sequence thread.
I20260205 11:45:42.222230   262 server.cpp:1181] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20260205 11:45:42.222282   262 server.cpp:1184] Check out http://ato-228-sts-0:8107 in web browser.
I20260205 11:45:42.222651   262 raft_server.cpp:75] Nodes configuration: 10.64.85.12:8107:8108
I20260205 11:45:42.222728   262 raft_server.cpp:118] Snapshot does not exist. We will remove db dir and init db fresh.
I20260205 11:45:42.223552   262 store.cpp:245] rm /usr/share/typesense/data/db success
I20260205 11:45:42.223652   262 store.cpp:39] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20260205 11:45:42.272233   262 store.cpp:269] DB open success!
I20260205 11:45:42.272264   262 raft_server.cpp:598] Loading collections from disk...
I20260205 11:45:42.272282   262 collection_manager.cpp:385] CollectionManager::load()
I20260205 11:45:42.272310   262 auth_manager.cpp:35] Indexing 0 API key(s) found on disk.
I20260205 11:45:42.272337   262 collection_manager.cpp:421] Loading upto 32 collections in parallel, 1000 documents at a time.
I20260205 11:45:42.272344   262 collection_manager.cpp:430] Found 0 collection(s) on disk.
I20260205 11:45:42.273816   262 collection_manager.cpp:586] Loaded 0 collection(s).
I20260205 11:45:42.274559   262 raft_server.cpp:605] Finished loading collections from disk.
I20260205 11:45:42.274588   262 raft_server.cpp:616] Loaded 0conversation model(s).
I20260205 11:45:42.274596   262 raft_server.cpp:620] Initializing batched indexer from snapshot state...
I20260205 11:45:42.274611   262 raft_server.cpp:633] Loaded 0 personalization model(s).
I20260205 11:45:42.276006   262 log.cpp:690] Use murmurhash32 as the checksum type of appending entries
W20260205 11:45:42.276062   262 protobuf_file.cpp:91] open file failed, path: /usr/share/typesense/data/state/log/log_meta: FILE_ERROR_NOT_FOUND
W20260205 11:45:42.276079   262 log.cpp:698] /usr/share/typesense/data/state/log is empty
I20260205 11:45:42.280958   262 log.cpp:1150] log save_meta /usr/share/typesense/data/state/log/log_meta first_log_index: 1 time: 4845
W20260205 11:45:42.281190   262 protobuf_file.cpp:91] open file failed, path: /usr/share/typesense/data/state/meta/raft_meta: FILE_ERROR_NOT_FOUND
I20260205 11:45:42.281250   262 raft_meta.cpp:521] Loaded single stable meta, path /usr/share/typesense/data/state/meta term 1 votedfor 0.0.0.0:0:0 time: 31
I20260205 11:45:42.281275   262 node.cpp:608] node default_group:10.64.85.78:8107:8108 init, term: 1 last_log_id: (index=0,term=0) conf: 10.64.85.12:8107:8108 old_conf: 
I20260205 11:45:42.281342   262 raft_server.cpp:141] Node last_index: 0
I20260205 11:45:42.281358   262 typesense_server_utils.cpp:309] Typesense peering service is running on 10.64.85.78:8107
I20260205 11:45:42.281368   262 typesense_server_utils.cpp:310] Snapshot interval configured as: 3600s
I20260205 11:45:42.281380   262 typesense_server_utils.cpp:311] Snapshot max byte count configured as: 4194304
W20260205 11:45:42.281392   262 controller.cpp:1550] SIGINT was installed with 1
I20260205 11:45:42.281483   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:45:42.281502   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:45:47.624661   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:45:47.624725   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:45:52.282574   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:45:52.282618   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:45:53.301973   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:45:53.302043   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:45:58.306249   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:45:58.306362   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:02.283761   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:46:02.284008   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:46:04.135566   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:04.135632   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:09.792848   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:09.792908   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:12.285198   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:46:12.285246   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:46:15.328110   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:15.328181   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:21.173420   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:21.173483   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:22.286331   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:46:22.286375   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:46:26.795668   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:26.795738   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:31.893962   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:31.894029   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:32.287436   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:46:32.287492   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:46:37.424225   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:37.424290   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:42.288520   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:46:42.288561   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:46:42.778475   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:42.778543   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:43.222661   263 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 0, reference_q.size: 0
I20260205 11:46:48.685724   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:48.685818   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:52.289606   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:46:52.289659   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:46:53.789053   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:53.789119   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:46:59.171308   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:46:59.171383   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:47:02.290808   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:47:02.290849   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:47:04.626567   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:47:04.626626   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:47:09.771814   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:47:09.771879   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:47:12.292941   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:47:12.292984   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:47:14.946091   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:47:14.946158   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:47:20.795393   289 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:47:20.795459   289 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:47:22.294099   262 raft_server.cpp:692] Term: 1, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260205 11:47:22.294147   262 raft_server.cpp:717] Node with no leader. Resetting peers of size: 1
I20260205 11:47:26.536655   316 node.cpp:1579] node default_group:10.64.85.78:8107:8108 term 1 start pre_vote
W20260205 11:47:26.536712   316 node.cpp:1589] node default_group:10.64.85.78:8107:8108 can't do pre_vote as it is not in 10.64.85.12:8107:8108
I20260205 11:47:27.900553     1 typesense_server_utils.cpp:60] Stopping Typesense server...
I20260205 11:47:28.294746   262 typesense_server_utils.cpp:345] Typesense peering service is going to quit.
I20260205 11:47:28.294806   262 raft_server.cpp:979] Set shutting_down = true
I20260205 11:47:28.294817   262 raft_server.cpp:983] Waiting for in-flight writes to finish...
I20260205 11:47:28.294836   262 raft_server.cpp:989] Replication state shutdown, store sequence: 0
I20260205 11:47:28.294847   262 raft_server.cpp:993] node->shutdown
I20260205 11:47:28.294857   262 node.cpp:961] node default_group:10.64.85.78:8107:8108 shutdown, current_term 1 state FOLLOWER
I20260205 11:47:28.295588   262 raft_server.cpp:997] node->join
I20260205 11:47:28.295672   291 raft_server.h:286] This node is down
I20260205 11:47:28.295768   262 node.cpp:961] node default_group:10.64.85.78:8107:8108 shutdown, current_term 1 state SHUTDOWN
I20260205 11:47:28.296100   262 typesense_server_utils.cpp:350] raft_server.stop()
I20260205 11:47:28.296135   262 server.cpp:1241] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20260205 11:47:28.296507   262 typesense_server_utils.cpp:353] raft_server.join()
I20260205 11:47:28.296635   262 typesense_server_utils.cpp:356] Typesense peering service has quit.
I20260205 11:47:28.297405   262 typesense_server_utils.cpp:572] Shutting down batch indexer...
I20260205 11:47:28.297433   262 typesense_server_utils.cpp:575] Waiting for batch indexing thread to be done...
I20260205 11:47:29.226893   263 batched_indexer.cpp:486] Notifying batch indexer threads about shutdown...
I20260205 11:47:29.227175   263 batched_indexer.cpp:492] Notifying reference sequence thread about shutdown...
I20260205 11:47:29.227347   263 batched_indexer.cpp:496] Batched indexer threadpool shutdown...
I20260205 11:47:29.229408   262 typesense_server_utils.cpp:578] Shutting down event sink thread...
I20260205 11:47:29.229450   262 typesense_server_utils.cpp:581] Waiting for event sink thread to be done...
I20260205 11:47:29.229507   262 typesense_server_utils.cpp:584] Shutting down conversation garbage collector thread...
I20260205 11:47:29.229524   262 typesense_server_utils.cpp:587] Waiting for conversation garbage collector thread to be done...
I20260205 11:47:29.229705   262 typesense_server_utils.cpp:590] Waiting for housekeeping thread to be done...
I20260205 11:47:29.229838   262 typesense_server_utils.cpp:594] Shutting down server_thread_pool
I20260205 11:47:29.231659   262 typesense_server_utils.cpp:598] Shutting down app_thread_pool.
I20260205 11:47:29.233641   262 typesense_server_utils.cpp:602] Shutting down replication_thread_pool.
I20260205 11:47:29.235918     1 typesense_server_utils.cpp:615] Typesense API service has quit.
I20260205 11:47:29.236059     1 typesense_server_utils.cpp:619] Deleting batch indexer
I20260205 11:47:29.236097     1 typesense_server_utils.cpp:623] CURL clean up
I20260205 11:47:29.236109     1 typesense_server_utils.cpp:627] Deleting server
I20260205 11:47:29.236675     1 typesense_server_utils.cpp:631] CollectionManager dispose, this might take some time...
I20260205 11:47:29.249078     1 typesense_server_utils.cpp:640] Bye.

Needless to say; it works as planned in 0.3.5:

Operator log with 0.3.5

2026-02-05T11:52:33.435Z INFO reconciling cluster {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:52:33.435Z DEBUG reconciling secret {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:52:39.138Z DEBUG reconciling config map {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:52:54.113Z DEBUG collecting endpoint slices {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:05.634Z DEBUG current quorum configuration {"namespace": "typesense", "cluster": "ato-228", "size": 1, "nodes": ["10.64.85.254:8107:8108"]} 2026-02-05T11:53:05.634Z INFO updating quorum configuration {"namespace": "typesense", "cluster": "ato-228", "size": 1, "nodes": ["10.64.85.137:8107:8108"]} 2026-02-05T11:53:05.639Z DEBUG reconciling services {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:06.939Z DEBUG reconciling ingress {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:11.112Z DEBUG reconciling scrapers {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:16.912Z DEBUG reconciling podmonitor {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:17.314Z DEBUG reconciling statefulset {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:17.314Z DEBUG reporting lag thresholds {"namespace": "typesense", "cluster": "ato-228", "read": "1000", "write": "500"} 2026-02-05T11:53:17.418Z INFO reconciling quorum health {"namespace": "typesense", "cluster": "ato-228"} 2026-02-05T11:53:17.418Z DEBUG events Wait On Configmap Updates {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"ato-228","uid":"17e25276-5c27-4814-8a22-a02e2971b054","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404381849"}, "reason": "QuorumNotReadyWaitATerm"} 2026-02-05T11:53:17.418Z INFO calculated quorum {"namespace": "typesense", "cluster": "ato-228", "minRequiredNodes": 1, "availableNodes": 1} 2026-02-05T11:53:17.436Z DEBUG reporting node status {"namespace": "typesense", "cluster": "ato-228", "node": "ato-228-sts-0", "state": "FOLLOWER", "ip": "10.64.85.137", "queued_writes": 0, "commited_index": 0} 2026-02-05T11:53:17.436Z DEBUG reporting cluster status {"namespace": "typesense", "cluster": "ato-228", "status": "NOT_READY"} 2026-02-05T11:53:17.436Z DEBUG reporting node health {"namespace": "typesense", "cluster": "ato-228", "node": "ato-228-sts-0", "healthy": false, "ip": "10.64.85.137"} 2026-02-05T11:53:17.525Z INFO evaluated quorum {"namespace": "typesense", "cluster": "ato-228", "minRequiredNodes": 1, "availableNodes": 1, "healthyNodes": 0} 2026-02-05T11:53:17.533Z INFO reconciling cluster completed {"namespace": "typesense", "cluster": "ato-228", "condition": "QuorumNotReadyWaitATerm", "requeueAfter": "2m0s"} 2026-02-05T11:53:17.533Z DEBUG events Quorum Is Not Ready {"type": "Warning", "object": {"kind":"TypesenseCluster","namespace":"typesense","name":"ato-228","uid":"17e25276-5c27-4814-8a22-a02e2971b054","apiVersion":"ts.opentelekomcloud.com/v1alpha1","resourceVersion":"404381853"}, "reason": "QuorumNotReadyWaitATerm"}