memory overwrites

[scunningham]

Looks like there are several places where if an error occurs, the event callback is trigger immediately and the bufferevent gets destroyed. On return, the bufferevent ptr is modified, causing memory corruption. We are seeing this regularly running against electric fence. For example:

https://github.com/mandiant/RProxy/blob/master/src/downstream.c#L1617

If the connect call fails for some reason, the buffer event will be destroyed and the buffer_set_timeouts call will overwrite memory. Here's the stack crawl:
#0 0x0000000000487100 in bufferevent_set_timeouts ()
#1 0x000000000044633d in downstream_connection_retry (sock=-1, which=1, arg=0x7f1fe28aafa8)

at /mnt/backup/sfServer_joconnor/libs/rproxy-prefix/src/rproxy/src/downstream.c:1636

#2 0x000000000047ab65 in event_process_active_single_queue ()
#3 0x000000000047adcd in event_process_active ()
#4 0x000000000047b3f4 in event_base_loop ()
#5 0x0000000000457982 in _evthr_loop ()
#6 0x00007f20028009d1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f2002141b6d in clone () from /lib64/libc.so.6

(gdb)

Other suspicious stack crawl looks similar:
#0 0x00000000004a14e5 in do_write (bev_ssl=0x7fd7e9241dd0, atmost=16384) at bufferevent_openssl.c:692
#1 0x00000000004a1969 in consider_writing (bev_ssl=0x7fd7e9241dd0) at bufferevent_openssl.c:868
#2 0x00000000004a22c4 in be_openssl_outbuf_cb (buf=0x7fd7e933bf70, cbinfo=0x7fd7f86f6a30, arg=0x7fd7e9241dd0) at bufferevent_openssl.c:1111
#3 0x000000000047fff5 in evbuffer_run_callbacks (buffer=0x7fd7e933bf70, running_deferred=0) at buffer.c:486
#4 0x00000000004800f6 in evbuffer_invoke_callbacks (buffer=0x7fd7e933bf70) at buffer.c:508
#5 0x000000000048153f in evbuffer_add_buffer (outbuf=0x7fd7e933bf70, inbuf=0x7fd7f2875f70) at buffer.c:897
#6 0x000000000048700b in bufferevent_write_buffer (bufev=0x7fd7e9241dd0, buf=0x7fd7f2875f70) at bufferevent.c:388
#7 0x000000000044469c in proxy_parser_body (p=0x7fd7ef22df80,

data=0x7fd7ea859030 "KoZIhvcNAQEFBQAwYzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlZBMQ0wCwYDVQQHDARBTEVYMREwDwYDVQQKDAhNQU5ESUFOVDEQMA4GA1UECwwHUFJPRFVDVDETMBEGA1UEAwwKUFJPRFVDVElPTjAeFw0xNDA3MDgxMzA5NDJaFw0xNjA3MDcxMzA5NDJaMD8xCzAJ"..., len=4096) at /mnt/backup/sfServer_joconnor/libs/rproxy-prefix/src/rproxy/src/downstream.c:652

#8 0x000000000045626a in htparser_run ()
#9 0x0000000000445f1e in downstream_connection_readcb (bev=0x7fd7f0ddee10, arg=0x7fd7f4b0ffa8)

at /mnt/backup/sfServer_joconnor/libs/rproxy-prefix/src/rproxy/src/downstream.c:1477

#10 0x0000000000486aa6 in _bufferevent_run_readcb (bufev=0x7fd7f0ddee10) at bufferevent.c:232
#11 0x0000000000488650 in bufferevent_readcb (fd=4220, event=2, arg=0x7fd7f0ddee10) at bufferevent_sock.c:183
#12 0x000000000047aa5d in event_persist_closure (base=0x7fd7f88a4d80, ev=0x7fd7f0ddee20) at event.c:1297
#13 0x000000000047ab6c in event_process_active_single_queue (base=0x7fd7f88a4d80, activeq=0x7fd7f661dff0) at event.c:1341
#14 0x000000000047ae2d in event_process_active (base=0x7fd7f88a4d80) at event.c:1416
#15 0x000000000047b454 in event_base_loop (base=0x7fd7f88a4d80, flags=0) at event.c:1617
#16 0x00000000004579e2 in _evthr_loop ()
#17 0x00007fd828aec9d1 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fd82842db6d in clone () from /lib64/libc.so.6

So, perhaps it would be safer to defer the callbacks using BEV_OPT_DEFER_CALLBACKS? Would that break other things?