I think this is just a limitation of qemu - CPU jitter can be heard when the guest (Windows) plays the USB device connect/disconnect sound.
I will do some more testing (manually invoking usb_del and usb_add from the host) and see if that makes any difference. It is slightly possible (but I think pretty unlikely) that our encryption/decryption tasks are preempting qemu at the moment of hotplugging. Performance resumes as normal once the device is in a "settled" state (either connected or disconnected).