PG cannot shutdown cleanly from SIGTERM

[jfunston]

There is a conflict between PG's handling of signals and seastar's. Seastar registers a handler for SIGTERM that relies on being able to access the local reactor, but the non-seastar threads do not have reactors, and by default there is no guarantee which thread will be delivered a signal. So the non-seastar threads will segfault if they get SIGTERM. To reproduce have an active PG connection and run "killall postgres".

#0 0x0000556f3315b179 in std::__uniq_ptr_impl<seastar::reactor_backend, std::default_deleteseastar::reactor_backend >::_M_ptr (this=0x20) at /usr/include/c++/9/bits/unique_ptr.h:360
#1 std::unique_ptr<seastar::reactor_backend, std::default_deleteseastar::reactor_backend >::get (this=0x20)
at /usr/include/c++/9/bits/unique_ptr.h:361
#2 std::unique_ptr<seastar::reactor_backend, std::default_deleteseastar::reactor_backend >::operator-> (
this=0x20) at /usr/include/c++/9/bits/unique_ptr.h:355
#3 seastar::reactor::signals::<lambda(int, siginfo_t*, void*)>::operator() (__closure=0x0, p=0x7ffed73ce380,
info=0x7ffed73ce4b0, sig=15) at ../../src/core/reactor.cc:641
#4 seastar::reactor::signals::<lambda(int, siginfo_t*, void*)>::_FUN(int, siginfo_t *, void *) ()
at ../../src/core/reactor.cc:642
#5
#6 0x00007f5858a7612b in __GI___select (nfds=nfds@entry=0, readfds=readfds@entry=0x0,
writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x7ffed73cf0a0)
at ../sysdeps/unix/sysv/linux/select.c:41
#7 0x0000556f32f1cbbb in pg_usleep (microsec=microsec@entry=10000)
at /build/src/k2/postgres/src/port/pgsleep.c:56

[iafuture]

Interesting. I think it's better to let PG drive the shutdown since it is responsible for the initialization.

Could we use pthread_sigmask to untag all threads except for the main thread as a possible candidate for signal delivery? I think this would guarantee that the main thread always gets SIGINT/SIGTERM and it can then broadcast the signal to the particular threads we need.

[jfunston]

I agree it makes sense to let PG handle shutdown. I think we need to overwrite the handler for seastar though instead of trying to use sigmask. The handler can call our stop() methods and the PG shutdown methods, and we can write it so that it is valid to call it from any thread.

[jfunston]

1. mask SIGINT and SIGTERM in seastar app
2. register SIGUSR1 in seastar app, which will call stop()
3. re-register SIGINT and SIGTERM from PG
4. KillK2app hook should deliver SIGUSR1 to seastar app