Kubernetes scale up on 4.17.2.0

[tuanhoangth1603]

Hello,
Have a good day team !
I created a Kubernetes cluster with two control nodes and one worker. Then I tried to manually scale up by adding three more workers, but an error message appeared on the dashboard saying "Scaling failed for Kubernetes cluster: K8S-Dev, unable to update network rules." I checked the management log and found details about the error.

2023-05-09 18:13:38,304 DEBUG [c.c.u.d.T.Transaction] (API-Job-Executor-105:ctx-567d2785 job-2167 ctx-c4f4e780) (logid:2277cf45) Rolling back the transaction: Time = 29 Name =  API-Job-Executor-105; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-FirewallManagerImpl.createFirewallRule:242-FirewallManagerImpl.createIngressFirewallRule:194-GeneratedMethodAccessor846.invoke:-1-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566-AopUtils.invokeJoinpointUsingReflection:344-ReflectiveMethodInvocation.invokeJoinpoint:198-ReflectiveMethodInvocation.proceed:163
2023-05-09 18:13:38,321 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-105:ctx-567d2785 job-2167 ctx-c4f4e780) (logid:2277cf45) Scaling failed for Kubernetes cluster : K8S-Dev, unable to update network rules
com.cloud.exception.NetworkRuleConflictException: The range specified, 2222-2227, conflicts with rule 216 which has 2222-2224
        at com.cloud.network.firewall.FirewallManagerImpl.detectRulesConflict(FirewallManagerImpl.java:470)
        at com.cloud.network.firewall.FirewallManagerImpl$1.doInTransaction(FirewallManagerImpl.java:255)
        at com.cloud.network.firewall.FirewallManagerImpl$1.doInTransaction(FirewallManagerImpl.java:242)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.network.firewall.FirewallManagerImpl.createFirewallRule(FirewallManagerImpl.java:242)
        at com.cloud.network.firewall.FirewallManagerImpl.createIngressFirewallRule(FirewallManagerImpl.java:194)
        at jdk.internal.reflect.GeneratedMethodAccessor846.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy276.createIngressFirewallRule(Unknown Source)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.provisionFirewallRules(KubernetesClusterResourceModifierActionWorker.java:446)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:148)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:369)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:405)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:439)
        at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:1315)
        at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKubernetesClusterCmd.java:162)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:106)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-05-09 18:13:38,333 INFO  [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-105:ctx-567d2785 job-2167 ctx-c4f4e780) (logid:2277cf45) Revoking launch permission for systemVM template
2023-05-09 18:13:38,343 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-105:ctx-567d2785 job-2167) (logid:2277cf45) Complete async job-2167, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Scaling failed for Kubernetes cluster : CFOX-Dev, unable to update network rules"}

I see the message "The range specified, 2222-2227, conflicts with rule 216 which has 2222-2224".
Does CloudStack have a function to replace conflicting rules? If not, how can we manually or auto-scale?
My cluster has changed to Alert status, and I cannot do anything now except stop or delete the cluster.
Additionally, how can I put my cluster back to Health status? I have checked, and all pods in the cluster are still working fine.
Thanks !

[shwstppr]

@tuanhoangth1603 During scaling CloudStack tries to redo the firewall and portforwarding rules depending on the new cluster size. From your logs, it appears that it was not able to delete the old firewall rule and encountered an error while creating a new one. You may need to check how many node VMs the cluster is having right now and does listKubernetesClusters API returns the same size. If it is 6 (new size) then you may delete the firewall rule 2222-2224, add a new firewall rule 2222-2227 and verify if the portforwarding rules are correctly set. Thereafter KubernetesClusterStatusScanner should reconcile the cluster state.

[shwstppr]

@tuanhoangth1603 also, is this(issue while scaling) happening for all clusters in your env or just with this particular cluster?

[tuanhoangth1603]

Hi @shwstppr,
I tried scaling on a new cluster and didn't encounter a similar error. Perhaps the issue is only on the current cluster. I have created an additional instance that shares a network with this current cluster and added a firewall rule for that instance. Will doing so affect the automatic redo of the firewall rule when k8s scales?
Thanks !

[tuanhoangth1603]

Is there any way to trigger the KubernetesClusterStatusScanner manually? My cluster is still in an Alert state. If I stop and start it again, the state will be reconciled, but this way is not a good idea!

[shwstppr]

@tuanhoangth1603
This is how cluster and scaling work:

• Cluster is deployed with n size. CloudStack will deploy 1 (default) control node and n worker nodes. For the public IP it will create a firewall rule to open ports 2222 to 2222+n. It will also set up port-forwarding for 2222+x to port 22 for each of the node VMs. (2222 will be forwarded to the control node, 2223 to the 1st worker node, 2224 to the second, and so on...). It will also open port 6443 and add a corresponding port-forwarding rule for the control node on the same port.
• When you scale UP, CloudStack will deploy new required worker node VMs, old firewall is deleted and a new firewall rule is created for 2222 to 2222+newsize. Port forwarding rules are set up as explained in the first point.

Now, for you, ACS failed to delete and create the new firewall rule (I'm not sure why but it could be the additional instance that you mentioned). If you can give the output of listKubernetesClusters API, screenshots of the cluster instances list, firewall rules, and portwarding rules for the cluster public IP we can have a deeper look.

As you say, k8s part is working fine. So what I suggested doing is,

• make sure firewall rules are provisioned correctly, ie, port 6443 and 2222 to 2222+clustersize
• make sure port-forwarding is provisioned correctly as explained above in point 1
• Wait for KubernetesClusterStatusScanner for 5 mins. It runs every 5 mins. There is no way to manually run it. And it should fix the cluster state. This shouldn't require stop-starting the cluster

[tuanhoangth1603]

@shwstppr
Here is output of listKubernetesClusters API and screenshots of the instances list, firewall rules, and portwarding rules.
(I have changed some information for security reasons but it does not affect the information that you need.)
CFOX-Dev is the alerting cluster whose state I want to fix.
Thanks for your help !
https://gist.github.com/tuanhoangth1603/dc4c6484e8dc8a83a020871108d0f1ec

Additionally, the result of kubectl get no command shows 2 worker nodes

[shwstppr]

@tuanhoangth1603 it seems your last 3 nodes never joined the k8s cluster due to error. In this case, I need to check but the only way to restore would be removing the extra node VMs alongwith the firewall and portforwarding rules. I'll check the code and may give you steps to proceed tomorrow.

[weizhouapache]

It looks the first (pf port 2224) and last (pf port 2228)worker nodes worked well
the other 3 (pf port 2225/2226/2227) did not .

it would be good to upload some logs in the control nodes and the 3 worker nodes with issues.