Data races in OpenSSL bindings

[colesbury]

Bug report

We have some data races in our OpenSSL bindings that we were not previously detecting because we didn't compile OpenSSL with TSan (#143750):

test_load_cert_chain_thread_safety

Race between use_certificate_chain_file and SSL_CTX_set_default_passwd_cb:

WARNING: ThreadSanitizer: data race (pid=3543367)
  Read of size 8 at 0x726c000007b8 by thread T6:
    #0 use_certificate_chain_file /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_rsa.c:477:32 (libssl.so.3+0x80b0f) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #1 SSL_CTX_use_certificate_chain_file /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_rsa.c:587:12 (libssl.so.3+0x80a19) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #2 _ssl__SSLContext_load_cert_chain_impl /home/sgross/cpython/./Modules/_ssl.c:4595:9 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1f9b8) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
    #3 _ssl__SSLContext_load_cert_chain /home/sgross/cpython/./Modules/clinic/_ssl.c.h:1833:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1f9b8)
...

  Previous write of size 8 at 0x726c000007b8 by thread T4:
    #0 SSL_CTX_set_default_passwd_cb /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:4497:34 (libssl.so.3+0x6c654) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #1 _ssl__SSLContext_load_cert_chain_impl /home/sgross/cpython/./Modules/_ssl.c:4640:5 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1fd64) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
    #2 _ssl__SSLContext_load_cert_chain /home/sgross/cpython/./Modules/clinic/_ssl.c.h:1833:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x1fd64)

test_thread_recv_while_main_thread_sends

WARNING: ThreadSanitizer: data race (pid=3543528)
  Write of size 4 at 0x72880000f068 by main thread:
    #0 ssl3_write_bytes /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/record/rec_layer_s3.c:285:16 (libssl.so.3+0x14050b) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #1 ssl3_write /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/s3_lib.c:4648:12 (libssl.so.3+0x3eee9) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #2 ssl_write_internal /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2604:16 (libssl.so.3+0x65007) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #3 SSL_write_ex2 /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2707:15 (libssl.so.3+0x653b5) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #4 SSL_write_ex /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2701:12 (libssl.so.3+0x65339) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #5 _ssl__SSLSocket_write_impl /home/sgross/cpython/./Modules/_ssl.c:2792:18 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2a809) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
    #6 _ssl__SSLSocket_write /home/sgross/cpython/./Modules/clinic/_ssl.c.h:651:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2a809)
    #7 _PyEval_EvalFrameDefault /home/sgross/cpython/Python/generated_cases.c.h:4009:35 (python+0x5129d9) (BuildId: 09bdcea4d998e789f62d66f1eea397f4c0970fe2)
...

  Previous write of size 4 at 0x72880000f068 by thread T492:
    #0 ssl3_read_bytes /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/record/rec_layer_s3.c:681:16 (libssl.so.3+0x141f69) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #1 ssl3_read_internal /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/s3_lib.c:4666:9 (libssl.so.3+0x3f2ca) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #2 ssl3_read /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/s3_lib.c:4689:12 (libssl.so.3+0x3f127) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #3 ssl_read_internal /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2379:16 (libssl.so.3+0x63af1) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #4 SSL_read_ex /home/sgross/cpython/multissl/src/openssl-3.5.4/ssl/ssl_lib.c:2407:15 (libssl.so.3+0x64225) (BuildId: 3439d4624f0bbda0d0e957787b11e780635293c6)
    #5 _ssl__SSLSocket_read_impl /home/sgross/cpython/./Modules/_ssl.c:2958:18 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2b229) (BuildId: b7a1f8db907210c3a84626c1dff3c67d6f138a0a)
    #6 _ssl__SSLSocket_read /home/sgross/cpython/./Modules/clinic/_ssl.c.h:723:20 (_ssl.cpython-315td-x86_64-linux-gnu.so+0x2b229)
    #7 method_vectorcall_VARARGS /home/sgross/cpython/Objects/descrobject.c:325:24 (python+0x293161) (BuildId: 09bdcea4d998e789f62d66f1eea397f4c0970fe2)

[ZeroIntensity]

load_cert_chain looks fixable. We should acquire tstate_mutex (via PySSL_BEGIN_ALLOW_THREADS) around the SSL calls.

For test_thread_recv_while_main_thread_sends, you might be interested in the history here -- specifically #137583. In short, I tried to fix these races a while ago by adding locking around OpenSSL calls, but that ended up deadlocking libraries that attempted to write to a socket while another thread was reading from it. I can't think of any way to fix that without running into that problem again.