virtualhub: Implement BTStack bluetooth.

[jaguilar]

These commits allow executing real bluetooth code using virtualhub combined with an appropriate dongle. This has been tested with TP Link UB500, however, it should work with any USB dongle for which an appropriate init script exists.

Talking of init scripts, this pull request also contains a tool that collects the most common bluetooth firmwares into ~/.cache/pybricks/virtualhub/bt_firmware. Given the dongle referenced above, and after running the provided tool (pybricks-micropython/tools/collect_bt_firmware.py), bricks/virtualhub/build-debug/firmware.elf tests/virtualhub/basics/hello.py will emit logs like the following:

Tail of virtualhub.elf log

HCI in packet type: 04, len: 6
hci.c.2226: Command complete for expected opcode 0c52 at substate 58
hci.c.1777: hci_initializing_run: substate 58, can send 1
HCI out packet type: 01, len: 5
HCI in packet type: 04, len: 2
hci.c.1777: hci_initializing_run: substate 58, can send 0
HCI in packet type: 04, len: 6
hci.c.2226: Command complete for expected opcode 080f at substate 58
hci.c.1777: hci_initializing_run: substate 58, can send 1
HCI out packet type: 01, len: 5
HCI in packet type: 04, len: 2
hci.c.1777: hci_initializing_run: substate 58, can send 0
HCI in packet type: 04, len: 6
hci.c.2226: Command complete for expected opcode 0c18 at substate 58
hci.c.1777: hci_initializing_run: substate 58, can send 1
HCI out packet type: 01, len: 4
HCI in packet type: 04, len: 2
hci.c.1777: hci_initializing_run: substate 58, can send 0
HCI in packet type: 04, len: 6
HCI in packet type: 04, len: 4
hci.c.2226: Command complete for expected opcode 0c1a at substate 58
hci.c.1777: hci_initializing_run: substate 58, can send 1
HCI out packet type: 01, len: 10
HCI in packet type: 04, len: 2
hci.c.1777: hci_initializing_run: substate 58, can send 0
HCI in packet type: 04, len: 6
hci.c.2226: Command complete for expected opcode 200b at substate 58
hci.c.1777: hci_initializing_run: substate 58, can send 1
hci.c.1770: hci_init_done -> HCI_STATE_WORKING
hci.c.7436: BTSTACK_EVENT_STATE 2
HCI in packet type: 04, len: 3
hci.c.5055: hci_power_control: 0, current mode 2
hci.c.7436: BTSTACK_EVENT_STATE 3
HCI in packet type: 04, len: 3
sm.c.3756: SM: reset state
hci.c.5097: HCI_STATE_HALTING, substate 3f

hci.c.5232: HCI_STATE_HALTING: wait 50 ms
sm.c.3715: HCI Working!
sm.c.4810: sm: generate new ec key
hci.c.8257: hci_le_set_own_address_type: old 1, new 1
sm.c.711: gap_random_address_trigger, state 0
hci.c.5097: HCI_STATE_HALTING, substate 44

hci.c.5248: HCI_STATE_HALTING, calling off
hci.c.4823: hci_power_control_off
hci_transport_h2_libusb.c.1262: usb_close
hci_transport_h2_libusb.c.426: shutdown, transfer 0x5555559554a0
hci_transport_h2_libusb.c.426: shutdown, transfer 0x555555963580
hci_transport_h2_libusb.c.426: shutdown, transfer 0x555555958b40
hci_transport_h2_libusb.c.426: shutdown, transfer 0x5555559520c0
hci_transport_h2_libusb.c.426: shutdown, transfer 0x555555951a10
hci_transport_h2_libusb.c.426: shutdown, transfer 0x5555559673e0
hci_transport_h2_libusb.c.1322: Libusb shutdown complete
hci.c.4828: hci_power_control_off - hci_transport closed
hci.c.4835: hci_power_control_off - control closed
hci.c.5253: HCI_STATE_HALTING, emitting state
hci.c.7436: BTSTACK_EVENT_STATE 0
HCI in packet type: 04, len: 3
sm.c.3756: SM: reset state
hci.c.5255: HCI_STATE_HALTING, done

The best way to find a supported dongle is:

1. Run ./tools/collect_bt_firmware.py
2. Examine the downloaded drivers. Check whether the init script for your dongle is already present. If so, things should already work.
3. If not, go buy that TP Link dongle, or some other dongle that is well-supported on Linux.
4. Don't forget to add your Bluetooth dongle to your udev rules. Here's what mine looks like:

SUBSYSTEM=="usb", ATTRS{idVendor}=="2357", ATTRS{idProduct}=="0604", MODE="666", TAG+="uaccess"

This has been tested on Windows with WSL and usb-ipd-win.

[jaguilar]

Okay, so this does work on a system with libusb and pkg-config installed. However, I see now that the test host does not have these libraries. We can either vendor in libusb, or we can install it in the test runner. What is the preferred solution?

[laurensvalk]

Nice! I hope to try this out this week.

Re: CI: Normally installing is preferred but in this case we don't need either.

The virtual hub had both a mock usb and mock Bluetooth driver, with only the latter enabled by default.

Since the Bluetooth driver will now be a real driver and the CI won't need to talk to real hardware, we can have the virtual hub use the USB mock driver to simulate stdout instead.

For local testing we can have both enabled. The mock USB driver will still provide stdout to the host terminal which is nice as a sanity check, and the Bluetooth driver will also provide it if something is connected.

[laurensvalk]

It's probably just me, but I wouldn't ordinarily run a script downloading things with regexes somewhere in my home directory with a nonzero history of bricking something 😄

I suppose we could have a folder bluetooth_firmware in our repository and add it to .gitignore? Downloading with a direct link (from a particular git sha) to a known working version would be nice, but I don't mind going through the repositories to find it.

The BTstack posix example has a way of specifying which device to use. I expect most test cases to use two dongles: one for the virtual hub, and one for the OS to test e.g. rfcomm or Pybricks Code. Could we do that with an environment variable? How does it currently determine which one to use?

[jaguilar]

It's probably just me, but I wouldn't ordinarily run a script downloading things with regexes somewhere in my home directory with a nonzero history of bricking something 😄

Ahaha, but in my case it bricked my dongle because I didn't just use the firmwares in this directory, I renamed one of them. Ultimately the risky part of the thing is that the firmwares are used by name and the btstack chipsets don't seem to have a good way to validate that a firmware file is correct for a module before sending it down. And/or that there is a module that will accept a firmware that will brick it. That risk remains whether the files are in the home directory or elsewhere.

I suppose we could have a folder bluetooth_firmware in our repository and add it to .gitignore? Downloading with a direct link (from a particular git sha) to a known working version would be nice, but I don't mind going through the repositories to find it.

I can checkout to a particular git sha and/or tag in my download script. That would be fine. On the other hand, every time a new chipset comes out it would be a mysterious bug for the first person who tries to use that chipset with our test bench. Up to you.

The BTstack posix example has a way of specifying which device to use. I expect most test cases to use two dongles: one for the virtual hub, and one for the OS to test e.g. rfcomm or Pybricks Code. Could we do that with an environment variable? How does it currently determine which one to use?

hci_transport_usb_set_path is the API. I don't know exactly how to use it but we can definitely finagle something. Currently it just uses the first dongle it finds.

[jaguilar]

Added the ability to manually specify the device using PYBRICKS_VIRTUALHUB_USB_PATH=N/M/O/.... It's pretty bad because btstack does not check the bus number, so ambiguity is possible, but it's something. We could fix btstack to do this correctly fairly easily.

To test:

(pybricks-micropython-py3.12) jaguilar@DESKTOP-2GF79BV:~/projects/jj-pb$ lsusb -t
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=vhci_hcd/8p, 480M
    |__ Port 002: Dev 003, If 0, Class=Communications, Driver=cdc_acm, 12M
    |__ Port 002: Dev 003, If 1, Class=CDC Data, Driver=cdc_acm, 12M
    |__ Port 002: Dev 003, If 2, Class=Vendor Specific Class, Driver=[none], 12M
    |__ Port 003: Dev 041, If 0, Class=Wireless, Driver=[none], 12M
    |__ Port 003: Dev 041, If 1, Class=Wireless, Driver=[none], 12M

So the wireless device is on bus 1 port 3. btstack ignores the bus number, so we'll just be looking at the port:

make -C ./micropython/mpy-cross && make DEBUG=1 BUILD=build-debug virtualhub && PYBRICKS_VIRTUALHUB_USB_PATH=3 bricks/virtualhub/build-debug/firmware.elf ./tests/virtualhub/basics/hello.py 2>&1 | less

The correct device is used.

make -C ./micropython/mpy-cross && make DEBUG=1 BUILD=build-debug virtualhub && PYBRICKS_VIRTUALHUB_USB_PATH=2 bricks/virtualhub/build-debug/firmware.elf ./tests/virtualhub/basics/hello.py 2>&1 | less

Initialization fails and the program hangs forever.

I suppose we could have a folder bluetooth_firmware in our repository . . .

This is now implemented.

Re: CI: Normally installing is preferred but in this case we don't need either. ...

Need some more practical guidance on this. Right now virtualhub does not build on CI due to the libusb dependency. Practically speaking, do I need to create a new target that has the dep and not have that target on CI? Or make the dep on libusb weak instead?

[laurensvalk]

Thanks for the updates!

I think we already have a candidate bug where this PR would be a big help: pybricks/support#2497

Need some more practical guidance on this. Right now virtualhub does not build on CI due to the libusb dependency.

I'll have a look at this when I get to around to the review. My thinking was that we would not be building the Bluetooth driver at all on the CI build, just like the bluetooth_simulation driver currently doesn't modify stdin settings since it messes up the CI output.

I suppose it would help if I switched the current virtualhub to use the virtual usb driver by default, so we only enable the bluetooth simulation driver for local use. I'll try and do that today.

[dlech]

just like the bluetooth_simulation driver currently doesn't modify stdin settings since it messes up the CI output.

FYI, there is a isatty() function for this sort of thing to check at runtime rather than making it a compile-time option.