disks: want to be able to differentiate "kinds"

[iximeow]

all NVMe disks are not created equal: the characteristics of disks backed by Crucible, a file, in-memory, 9pfs, etc, all vary wildly. in the product, the only disk types (currently) are the typical Crucible-backed distributed disks and "local" volumes that are from Propolis' perspective a file-backed disk with a particular file that has desirable performance properties. a VM can have one or all of these things at the same time!

it would be nice as a guest to tell which disks you have. that is, it would be nice for nvme list in a Linux guest to tell you enough that you should put your critical data on the Crucible volume mounted at /dev/nvme2n1 and your more ephemeral information on the local volume at /dev/nvme1n1. that might be an NVMe model number, that might be a PCI device ID, could be both!

since Omicron is really what knows what a local volume is, rather than pointing Propolis at any other random file on a sled, it seems like Nexus ought to set an informative model number, but it probably makes sense for Propolis to own the device ID selection. it would be really weird for propolis-api to let you specify whatever random PCI device ID for a disk, but that comes at the cost of Propolis not really knowing if the file backing a disk is a magic fast volume "file" or a normal file on a ZFS filesystem. the latter isn't something we'll use in the product, so that might be an OK ambiguity.

could have sworn we had an issue for this, but I couldn't find one!

[iximeow]

rm notes elsewhere that PCI 01de:0000 is named, generically, "Propolis NVMe Controller". so in an ideal world we'd have a different, more specific PCI device ID for distributed (Crucible-backed) disk case as well as one for the local disk. so we'll stick with the generic PCI device ID and vary on model number to distinguish these.

the current contenders are "Oxide Distributed Disk" and "Oxide Local Disk", but we have to be careful of the same issue #979 ran across: existing disks have NVMe-violating null strings for their model number and changing that can break guest-determined boot options. "Oxide Distributed Disk" is what we'll move to when we can change the default distributed disk model number.