diff --git a/configuration/assigning-devices.md b/configuration/assigning-devices.md index 6aa8278a..1c2b711f 100644 --- a/configuration/assigning-devices.md +++ b/configuration/assigning-devices.md @@ -11,6 +11,58 @@ redirect_from: Assigning Devices to VMs ======================== +Sometimes you may need to assign an entire PCI or PCI Express device directly +to a qube. This is also known as PCI pass-through. The Qubes installer does this +by default for `sys-net` (assigning all network class controllers), as well as +`sys-usb` (assigning all USB controllers) if you chose to create the +USB qube during install. +While this covers most use cases, there are some occasions when you may want to +manually assign one NIC to `sys-net` and another to a custom NetVM, or have some +other type of PCI controller you want to manually assign. + +Note that one can only assign full PCI or PCI Express devices by default. +This limit is imposed by the PC and VT-d +architectures. This means if a PCI device has multiple functions, all instances +of it need to be assigned to the same qube unless you have disabled FLR with the +`no-strict-reset` (R4.0) or `pci_strictreset` (R3.2) option. +In the steps below, you can tell if this is needed if you see the BDF for the +same device listed multiple times with only the number after the "." changing. + +While a device can only be attached to one VM at a time, it *is* possible to +*assign* the same device to more than one VM at a time. This means that you can +use the device in one VM, shut that VM down, start up a different VM (to which +the same device is also assigned), then use the device in that VM. This can be +useful if, for example, you have only one USB controller, but you have multiple +security domains which all require the use of different USB devices. + +R4.0 +------------------------ + +In order to assign a whole PCI(e) device to a VM, one should use the `qvm-pci` +tool. First, list the available PCI devices: + +~~~ +qvm-pci +~~~ + +This will show you the `backend:BDF` address of each PCI device. It will look something +like `dom0:00_1a.0`. Once you've found the address of the device you want to +assign, then attach it like so: + +~~~ +qvm-pci attach --persistent : +~~~ + +For example, if `00_1a.0` is the BDF of the device you want to assign to the +"personal" domain, you would do this: + +~~~ +qvm-pci attach --persistent personal dom0:00_1a.0 +~~~ + +R3.2 +------------------------ + In order to assign a whole PCI(e) device to a VM, one should use the `qvm-pci` tool. First, list the available PCI devices: @@ -26,27 +78,13 @@ assign, then attach it like so: qvm-pci -a ~~~ -For example, if `00:1a.0` is the BDF of the device I want to assign to the -"personal" domain, I would do this: +For example, if `00:1a.0` is the BDF of the device you want to assign to the +"personal" domain, you would do this: ~~~ qvm-pci -a personal 00:1a.0 ~~~ -Note that one can only assign full PCI or PCI Express devices. This means one -cannot assign single USB devices -- only the whole USB controller with whatever -USB devices are connected to it. This limit is imposed by the PC and VT-d -architectures. More information on using and managing USB devices with qubes is -available on the [USB] page. - -While a device can only be attached to one VM at a time, it *is* possible to -*assign* the same device to more than one VM at a time. This means that you can -use the device in one VM, shut that VM down, start up a different VM (to which -the same device is also assigned), then use the device in that VM. This can be -useful if, for example, you have only one USB controller, but you have multiple -security domains which all require the use of different USB devices. - - Using Qubes Manager ------------------- @@ -58,15 +96,21 @@ list of available devices, which you can select to be assigned to that VM. Finding the right USB controller -------------------------------- -If you want assign a certain [USB] device to a VM (by attaching the whole +Some USB devices are not compatible with the USB pass-through method Qubes employs. +In situations like this, you can still often get the USB device to work by +passing through the entire USB controller to a qube. However, with this approach +one cannot assign single USB devices, only the whole USB controller with whatever +USB devices are connected to it. More information on using and managing USB devices with qubes is +available on the [USB] page. If you want assign a certain USB device to a VM (by attaching the whole USB controller), you need to figure out which PCI device is the right -controller. First, check to which USB bus the device is connected: +controller. First, check to which USB bus the device is connected (note that +these steps need to be run from a terminal inside `dom0`): ~~~ lsusb ~~~ -For example, I want assign a broadband modem to the netvm. In the out put of +For example, I want assign a broadband modem to the netvm. In the output of `lsusb` it can be listed as something like this. (In this case, the device isn't fully identified): @@ -89,12 +133,8 @@ This should output something like: ~~~ Now you see the BDF address in the path (right before final `usb3`). Strip the -leading `0000:` and pass the rest to the `qvm-pci` tool: - -~~~ -qvm-pci -a netvm 00:1a.0 -~~~ - +leading `0000:` and pass the rest to the `qvm-pci` tool to attach the controller +with the version specific steps above. Possible issues --------------- @@ -109,7 +149,7 @@ expressed in 512B chunks): ~~~ # qvm-prefs netvm |grep kernelopts kernelopts : iommu=soft swiotlb=2048 (default) -# qvm-prefs -s netvm kernelopts "iommu=soft swiotlb=4096" +# qvm-prefs -s netvm kernelopts "iommu=soft swiotlb=8192" ~~~ This is [known to be needed][ml1] for the Realtek RTL8111DL Gigabit Ethernet @@ -117,8 +157,57 @@ Controller. ### PCI passthrough issues -Sometimes PCI arbitrator is too strict. There is a way to enable permissive mode -for it. Create `/etc/systemd/system/qubes-pre-netvm.service`: +Sometimes the PCI arbitrator is too strict. There is a way to enable permissive mode +for it. See also: [this thread][ml2] and the Xen wiki's [PCI passthrough] page. + +**NOTE:** By setting the permissive flag for the PCI device, you're potentially +weakening the device isolation, especially if your system is not equipped with +VT-d Interrupt Remapping unit. See [Software Attacks on Intel VT-d] (page 7) +for more details. + +At other times, you may instead need to disable the FLR requirement on a device. +This will also weaken device isolation; see the "I created a usbVM..." entry in +the [FAQ](/doc/faq/) for more details. + +R4.0 +------------------------ + +Permissive mode and strict reset are options set as part of PCI device attachment. If you've already +attached the PCI device to a VM, detach it first either with Qube Manager +or `qvm-pci`, then list the available PCI devices: + +~~~ +qvm-pci +~~~ + +This will show you the `backend:BDF` address of each PCI device. It will look something +like `dom0:00_1a.0`. Once you've found the address of the device you want to +assign, then attach it like so: + +~~~ +qvm-pci attach --persistent --option [--option ] : +~~~ + +For example, if `00_1a.0` is the BDF of the device you want to assign to the +"personal" domain, and it is particularly difficult to pass through you would do this: + +~~~ +qvm-pci attach --persistent --option permissive=true --option no-strict-reset=true personal dom0:00_1a.0 +~~~ + +Running `qvm-pci` again should then show your PCI device attached with both the +`permissive` and `no-strict-reset` options set. + +**Note** again that in most cases you should +not need either of these options set. Only set one or more of them as required to get +your device to function, or replace the device with one that functions properly with Qubes. + +R3.2 +------------------------ + +Permissive mode is enabled system wide per device. + +Create `/etc/systemd/system/qubes-pre-netvm.service`: ~~~ [Unit] @@ -136,13 +225,15 @@ WantedBy=multi-user.target Then enable it with `systemctl enable qubes-pre-netvm.service` -See also: [this thread][ml2] and the Xen wiki's [PCI passthrough] page. +The strict reset option is set for all devices attached to a VM with: -**NOTE:** By setting the permissive flag for the PCI device, you're potentially -weakening the device isolation, especially if your system is not equipped with -VT-d Interrupt Remapping unit. See [Software Attacks on Intel VT-d] (page 7) -for more details. +``` +qvm-prefs usbVM -s pci_strictreset false +``` +**Note** again that in most cases you should +not need either of these options set. Only set one or more of them as required to get +your device to function, or replace the device with one that functions properly with Qubes. Bringing PCI device back to dom0 --------------------------------