Skip to content

Conversation

@jayanthAP
Copy link
Contributor

Script to be used to restart HNS based when certain condition is met, to help mitigate issues in customer AKS nodes. Script currently supports checking for missing rules on pod-endpoints.

Example command:
conditionalHnsRestarter.ps1 -PodPortRulesToCheck @{"ruleRegex" = "LB_DSR_[0-9A-Z]+[\d.]+[\d.]+_801_801_17"; "layerName" = "LB_DSR"; "groupName" = "LB_DSR_IPv4_OUT"}

Script to be used to restart HNS based when certain condition is met, to help mitigate issues in customer AKS nodes. Script currently supports checking for missing rules on pod-endpoints.

Example command:
conditionalHnsRestarter.ps1 -PodPortRulesToCheck @{"ruleRegex" = "LB_DSR_[0-9A-Z]+_[\d\.]+_[\d\.]+_801_801_17"; "layerName" = "LB_DSR"; "groupName" = "LB_DSR_IPv4_OUT"}
….ps1

This is a generic mitigator script. Currently it checks pod port rules and restarts HNS when rules are not found. It can be extended for other mitigation steps as well, since the mitigation logic can be similar.
@jayanthAP jayanthAP marked this pull request as ready for review July 24, 2023 14:46
{
if ($CollectWindowsLogs -eq $true) {
# create log path if not yet created.
mkdir -Force $LogsPath
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If $CollectWindowsLogs is $true and issue happens very frequently, there are chances multiple log files will be created. This will consume lot of customer's space.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we will need to add a rotation logic for the same. Will add that.

hnsCrashEventChecker.yaml is a daemonset that looks for HNS crash win-event logs. Once applied on a kubernetes cluster, user can execute below command to know the timestamps when HNS crashed occurred on respective windows nodes:

kubectl logs -l name=hns-crash-event-checker --all-containers=true
Making the daemonset generic such that it can be used for any service (not just HNS). The $ServiceName variable needs to be edited for this.
Last edit missed the daemonset code.
This daemonset enables collection of crash-dumps for user-space processes on AKS windows nodes.
@daschott
Copy link
Contributor

daschott commented Aug 2, 2023

Is this ready for review / merging?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants