CLOSE_WAIT #148
Description
I'm not sure but I think the wrapper is not closing closed connections properly:
Details
good
$ netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.2:34008 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50373 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55576 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.2:50374 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55574 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55578 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50371 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.2:50367 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55556 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50372 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.152.0.3:50190 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55558 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55582 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55580 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.56.2.1:20834 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55586 unbound.default.svc.:53 TIME_WAIT tcp 0 0 dnscrypt-768656ff6d:443 10.56.2.1:20834 TIME_WAIT tcp 0 0 dnscrypt-768656ff:55586 unbound.default.svc.:53 TIME_WAIT udp 0 0 0.0.0.0:48047 0.0.0.0:* udp 0 0 0.0.0.0:443 0.0.0.0:* Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path
As it happens
$ netstat -a -n Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN tcp 0 0 10.56.2.13:43356 10.59.242.77:53 TIME_WAIT tcp 0 0 10.56.2.13:443 10.152.0.3:52602 TIME_WAIT tcp 0 0 10.56.2.13:43328 10.59.242.77:53 TIME_WAIT tcp 0 0 10.56.2.13:443 10.152.0.2:52641 TIME_WAIT tcp 323 0 10.56.2.13:443 10.56.2.1:57788 CLOSE_WAIT tcp 0 0 10.56.2.13:443 10.152.0.3:51751 TIME_WAIT tcp 0 0 10.56.2.13:43390 10.59.242.77:53 TIME_WAIT udp 0 0 0.0.0.0:48010 0.0.0.0:* udp 0 0 0.0.0.0:443 0.0.0.0:* Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path
Bad (no more queries are being answered untill a dnscrypt-wrapper restart)
$ netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN tcp 323 0 dnscrypt-768656ff6d:443 10.152.0.3:3232 CLOSE_WAIT tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.3:53468 CLOSE_WAIT tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.2:34023 CLOSE_WAIT tcp 323 0 dnscrypt-768656ff6d:443 10.152.0.3:1908 CLOSE_WAIT tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.2:53527 CLOSE_WAIT tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.3:49746 CLOSE_WAIT tcp 259 0 dnscrypt-768656ff6d:443 10.152.0.2:58956 CLOSE_WAIT tcp 259 0 dnscrypt-768656ff6d:443 10.152.0.3:32736 CLOSE_WAIT tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.2:49736 CLOSE_WAIT tcp 48 0 dnscrypt-768656ff6d:443 10.152.0.2:20808 CLOSE_WAIT tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.3:3420 CLOSE_WAIT tcp 259 0 dnscrypt-768656ff6d:443 10.152.0.3:1915 CLOSE_WAIT tcp 323 0 dnscrypt-768656ff6d:443 10.152.0.2:58366 CLOSE_WAIT tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.2:3404 CLOSE_WAIT tcp 387 0 dnscrypt-768656ff6d:443 10.152.0.3:35672 CLOSE_WAIT tcp 48 0 dnscrypt-768656ff6d:443 10.56.2.1:9866 CLOSE_WAIT tcp 387 0 dnscrypt-768656ff6d:443 10.152.0.3:3424 CLOSE_WAIT tcp 195 0 dnscrypt-768656ff6d:443 10.152.0.2:3416 CLOSE_WAIT udp 0 0 0.0.0.0:443 0.0.0.0:* udp 0 0 0.0.0.0:54437 0.0.0.0:* Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path $ ss -tano State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:443 *:* CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:30367 CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:9494 CLOSE-WAIT 195 0 10.56.2.13:443 10.56.2.1:52484 CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.2:56356 CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.2:14286 CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.2:52527 CLOSE-WAIT 48 0 10.56.2.13:443 10.56.2.1:29095 CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.2:4251 CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:61126 CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.2:14283 CLOSE-WAIT 131 0 10.56.2.13:443 10.152.0.2:7763 CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.3:52521 CLOSE-WAIT 131 0 10.56.2.13:443 10.152.0.2:14285 CLOSE-WAIT 195 0 10.56.2.13:443 10.56.2.1:52524 CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.2:50186 CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.3:31341 CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.2:7767 CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.3:9773 CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.3:61116 CLOSE-WAIT 323 0 10.56.2.13:443 10.56.2.1:52501 CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:14269 CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:7758 CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.3:30361 CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.3:49210 CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.2:52517 ESTAB 322 0 10.56.2.13:443 10.152.0.2:52531 CLOSE-WAIT 259 0 10.56.2.13:443 10.152.0.2:14268 CLOSE-WAIT 48 0 10.56.2.13:443 10.152.0.3:29382 CLOSE-WAIT 323 0 10.56.2.13:443 10.56.2.1:52483 CLOSE-WAIT 259 0 10.56.2.13:443 10.56.2.1:52502 CLOSE-WAIT 387 0 10.56.2.13:443 10.152.0.3:52498 CLOSE-WAIT 323 0 10.56.2.13:443 10.152.0.2:7764 CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:52499 CLOSE-WAIT 195 0 10.56.2.13:443 10.152.0.3:16982
I'm using GCP with kubernetes. So traffic routed like this: GCP LoadBalancer->kubernetes-service->dnscrypt-wrapper-container->kubernetes-service->unbound-container
Restarting dnscrypt-wrapper temporarily fixes the problem