Add the multiAddress.enabled cluster option present in zk >3.6.0

[cstaff14]

Change log description

• Enable the multiaddreses.Enabled JVM flag in the zkserver.sh
• parse inputted extra addresses and add them to the zoo config
  (2-3 concise points about the changes in this PR. When committing this PR, the committer is expected to copy the content of this section to the merge description box)

Purpose of the change

In zookeeper > 3.6.0 you can specify multiple addresses for a singular zookeeper server. here is the description of that cluster option. This update allows you to leverage that feature through the operator. This can also be useful if you want to expose individual zk servers through external addresses.
(e.g., Fixes #666, Closes #1234)

What the code does

• Add funcitionality to parse a list of zk server addresses in additionalConfig when creating the config map
• in the zookeeper image, parse those additional addresses and properly use them in the zk config
  • The operator currently writes server addresses on write config and node registration. When one of these events happen we add an extra step that:
    • parses the new addServerAddresses.txt created by the config map for addresses listed for the current server ID
    • if there is a "|" it formats each address using the zkconfig function, otherwise it adds the singular address
    • it also removes the client port that is appended by the zkconfig function for all addresses except the final one in the list. This is to follow formatting for the multiAddress feature as the client port should only be present once per list of servers

(Detailed description of the code changes)

How to verify it

1. Deploy the operator via a helm chart
2. under additionalConfig, add extra server addresses associated with zk servers (In our case it is the addresses for the dedicated VIPs that expose each server)

   additionalConfig: {
     server.1: query-service-stg-zookeeper-0.ethos11-stage-nld2.ethos.adobe.net,
     server.2: query-service-stg-zookeeper-1.ethos11-stage-nld2.ethos.adobe.net,
     server.3: query-service-stg-zookeeper-2.ethos11-stage-nld2.ethos.adobe.net,
     server.4: query-service-stg-zookeeper-3.ethos11-stage-nld2.ethos.adobe.net,
     server.5: query-service-stg-zookeeper-4.ethos11-stage-nld2.ethos.adobe.net
   }
   

• You can add addresses for some, all or no servers. You can also add multiple addresses for the same server in a pipe separated list.

   additionalConfig: {
    server.1: query-service-stg-zookeeper-0.ethos11-stage-nld2.ethos.adobe.net,
    server.3: query-service-stg-zookeeper-2.ethos11-stage-nld2.ethos.adobe.net,
    server.5: query-service-stg-zookeeper-4.ethos11-stage-nld2.ethos.adobe.net
  }
  

   additionalConfig: {
    server.1: query-service-stg-zookeeper-0.ethos11-stage-nld2.ethos.adobe.net|query-service-stg-zookeeper.ethos11-stage-nld2.ethos.adobe.net,
    server.2: query-service-stg-zookeeper-1.ethos11-stage-nld2.ethos.adobe.net,
    server.3: query-service-stg-zookeeper-2.ethos11-stage-nld2.ethos.adobe.net|www.google.com,
    server.4: query-service-stg-zookeeper-3.ethos11-stage-nld2.ethos.adobe.net,
    server.5: query-service-stg-zookeeper-4.ethos11-stage-nld2.ethos.adobe.net
   }
  

3. Validate correct address list is displayed through k8s portforwarding

    kubectl get pods
    kubectl port-forward app-zookeeper-0 2181:2181
   

   castaffo@castaffo-mac zookeeper-operator % echo conf | nc 127.0.0.1 2181
   ...
   membership: 
   server.1=app-zookeeper-0.app-zookeeper-headless.ns-team-experience-platform--query-service-zookeeper--6584bc25.svc.cluster.local:2888:3888|query-service-stg-zookeeper-0.ethos11-stage-nld2.ethos.adobe.net:2888:3888|query-service-stg-zookeeper.ethos11-stage-nld2.ethos.adobe.net:2888:3888:participant;0.0.0.0:2181
   server.2=app-zookeeper-1.app-zookeeper-headless.ns-team-experience-platform--query-service-zookeeper--6584bc25.svc.cluster.local:2888:3888|query-service-stg-zookeeper-1.ethos11-stage-nld2.ethos.adobe.net:2888:3888:participant;0.0.0.0:2181
   server.3=app-zookeeper-2.app-zookeeper-headless.ns-team-experience-platform--query-service-zookeeper--6584bc25.svc.cluster.local:2888:3888|query-service-stg-zookeeper-2.ethos11-stage-nld2.ethos.adobe.net:2888:3888|www.google.com:2888:3888:participant;0.0.0.0:2181
   server.4=app-zookeeper-3.app-zookeeper-headless.ns-team-experience-platform--query-service-zookeeper--6584bc25.svc.cluster.local:2888:3888|query-service-stg-zookeeper-3.ethos11-stage-nld2.ethos.adobe.net:2888:3888:participant;0.0.0.0:2181
   server.5=app-zookeeper-4.app-zookeeper-headless.ns-team-experience-platform--query-service-zookeeper--6584bc25.svc.cluster.local:2888:3888|query-service-stg-zookeeper-4.ethos11-stage-nld2.ethos.adobe.net:2888:3888:participant;0.0.0.0:2181
   
   

[amuraru]

can we avoid duplicating the upstream script and instead leverage SERVER_JVMFLAGS env var to pass in -Dzookeeper.multiAddress.enabled=true ?

[amuraru]

we use something similar internally at https://git.corp.adobe.com/experience-platform/pipeline-kafka/blob/master/k8s/charts/zookeeper-cluster/values.yaml#L112-L113

[cstaff14]

Deploying with the SERVER_JVMFLAGS works. Removed this file and now theres no longer a need for a new apache image as well.

I noted the pod env requirement in a comment in generators.go. Should I add a section to the README about multiaddress deployment configuration as well?

[amuraru]

we haven't used this before - is this required for cases where EXTRACONFIG is empty?

[cstaff14]

It is not. When you run zkconfig it sends out a format that is address:quorum-port:leade-port:role;client-port which follows the zk formatting for addresses. When you have multiple addresses the leader and quorum ports need to be duplicated, the role is optional but you can only have ";client-port" once at the end of the line. So if there is no extra address address:<quorum-port:leader-port:role;client--port works, but adding that extra address we need to trim off the ;client-port in the first one so we have address1:quorum-port:leader-port:role|address2:quorum-port:leader-port:role;client--port.

[cstaff14]

This is the doc on multiple address formatting: https://zookeeper.apache.org/doc/current/zookeeperReconfig.html#sc_multiaddress
While it doesn't explicitly say the client port can't be duplicated, in development this caused an error that was something to the effect of "address format invalid, should be host:port:port|host:port:port;port..."

[amuraru]

where is this zkServer.sh coming from?

[cstaff14]

This was left over from before you pointed out that I could set multiAddress.enabled with a jvm flag. I just removed this.