assigning-devices 4.0 updates

add introduction explaining why this can be needed and multi-function device quirks
add 4.0 content
add introduction to USB controller section explaining why
add strict reset options for R3.2 and R4.0

configuration/assigning-devices.md

@@ -11,6 +11,58 @@ redirect_from:
 Assigning Devices to VMs
 ========================
 
+Sometimes you may need to assign an entire PCI or PCI Express device directly
+to a qube. This is also known as PCI pass-through. The Qubes installer does this
+by default for `sys-net` (assigning all network class controllers), as well as
+`sys-usb` (assigning all USB controllers) if you chose to create the
+USB qube during install.
+While this covers most use cases, there are some occasions when you may want to
+manually assign one NIC to `sys-net` and another to a custom NetVM, or have some
+other type of PCI controller you want to manually assign.
+
+Note that one can only assign full PCI or PCI Express devices by default.
+This limit is imposed by the PC and VT-d
+architectures. This means if a PCI device has multiple functions, all instances
+of it need to be assigned to the same qube unless you have disabled FLR with the
+`no-strict-reset` (R4.0) or `pci_strictreset` (R3.2) option.
+In the steps below, you can tell if this is needed if you see the BDF for the
+same device listed multiple times with only the number after the "." changing.
+
+While a device can only be attached to one VM at a time, it *is* possible to
+*assign* the same device to more than one VM at a time. This means that you can
+use the device in one VM, shut that VM down, start up a different VM (to which
+the same device is also assigned), then use the device in that VM. This can be
+useful if, for example, you have only one USB controller, but you have multiple
+security domains which all require the use of different USB devices.
+
+R4.0
+------------------------
+
+In order to assign a whole PCI(e) device to a VM, one should use the `qvm-pci`
+tool. First, list the available PCI devices:
+
+~~~
+qvm-pci
+~~~
+
+This will show you the `backend:BDF` address of each PCI device. It will look something
+like `dom0:00_1a.0`. Once you've found the address of the device you want to
+assign, then attach it like so:
+
+~~~
+qvm-pci attach --persistent <vmname> <backend>:<bdf>
+~~~
+
+For example, if `00_1a.0` is the BDF of the device you want to assign to the
+"personal" domain, you would do this:
+
+~~~
+qvm-pci attach --persistent personal dom0:00_1a.0
+~~~
+
+R3.2
+------------------------
+
 In order to assign a whole PCI(e) device to a VM, one should use the `qvm-pci`
 tool. First, list the available PCI devices:
 
@@ -26,27 +78,13 @@ assign, then attach it like so:
 qvm-pci -a <vmname> <bdf>
 ~~~
 
-For example, if `00:1a.0` is the BDF of the device I want to assign to the
-"personal" domain, I would do this:
+For example, if `00:1a.0` is the BDF of the device you want to assign to the
+"personal" domain, you would do this:
 
 ~~~
 qvm-pci -a personal 00:1a.0
 ~~~
 
-Note that one can only assign full PCI or PCI Express devices. This means one
-cannot assign single USB devices -- only the whole USB controller with whatever
-USB devices are connected to it. This limit is imposed by the PC and VT-d
-architectures. More information on using and managing USB devices with qubes is
-available on the [USB] page.
-
-While a device can only be attached to one VM at a time, it *is* possible to
-*assign* the same device to more than one VM at a time. This means that you can
-use the device in one VM, shut that VM down, start up a different VM (to which
-the same device is also assigned), then use the device in that VM. This can be
-useful if, for example, you have only one USB controller, but you have multiple
-security domains which all require the use of different USB devices.
-
-
 Using Qubes Manager
 -------------------
 
@@ -58,15 +96,21 @@ list of available devices, which you can select to be assigned to that VM.
 Finding the right USB controller
 --------------------------------
 
-If you want assign a certain [USB] device to a VM (by attaching the whole
+Some USB devices are not compatible with the USB pass-through method Qubes employs.
+In situations like this, you can still often get the USB device to work by
+passing through the entire USB controller to a qube. However, with this approach
+one cannot assign single USB devices, only the whole USB controller with whatever
+USB devices are connected to it. More information on using and managing USB devices with qubes is
+available on the [USB] page. If you want assign a certain USB device to a VM (by attaching the whole
 USB controller), you need to figure out which PCI device is the right
-controller. First, check to which USB bus the device is connected:
+controller. First, check to which USB bus the device is connected (note that
+these steps need to be run from a terminal inside `dom0`):
 
 ~~~
 lsusb
 ~~~
 
-For example, I want assign a broadband modem to the netvm. In the out put of
+For example, I want assign a broadband modem to the netvm. In the output of
 `lsusb` it can be listed as something like this. (In this case, the device isn't
 fully identified):
 
@@ -89,12 +133,8 @@ This should output something like:
 ~~~
 
 Now you see the BDF address in the path (right before final `usb3`). Strip the
-leading `0000:` and pass the rest to the `qvm-pci` tool:
-
-~~~
-qvm-pci -a netvm 00:1a.0
-~~~
-
+leading `0000:` and pass the rest to the `qvm-pci` tool to attach the controller
+with the version specific steps above.
 
 Possible issues
 ---------------
@@ -109,7 +149,7 @@ expressed in 512B chunks):
 ~~~
 # qvm-prefs netvm |grep kernelopts
 kernelopts       : iommu=soft swiotlb=2048 (default)
-# qvm-prefs -s netvm kernelopts "iommu=soft swiotlb=4096"
+# qvm-prefs -s netvm kernelopts "iommu=soft swiotlb=8192"
 ~~~
 
 This is [known to be needed][ml1] for the Realtek RTL8111DL Gigabit Ethernet
@@ -117,8 +157,57 @@ Controller.
 
 ### PCI passthrough issues
 
-Sometimes PCI arbitrator is too strict. There is a way to enable permissive mode
-for it. Create `/etc/systemd/system/qubes-pre-netvm.service`:
+Sometimes the PCI arbitrator is too strict. There is a way to enable permissive mode
+for it. See also: [this thread][ml2] and the Xen wiki's [PCI passthrough] page.
+
+**NOTE:** By setting the permissive flag for the PCI device, you're potentially
+weakening the device isolation, especially if your system is not equipped with
+VT-d Interrupt Remapping unit. See [Software Attacks on Intel VT-d] (page 7)
+for more details.
+
+At other times, you may instead need to disable the FLR requirement on a device.
+This will also weaken device isolation; see the "I created a usbVM..." entry in
+the [FAQ](/doc/faq/) for more details.
+
+R4.0
+------------------------
+
+Permissive mode and strict reset are options set as part of PCI device attachment. If you've already
+attached the PCI device to a VM, detach it first either with Qube Manager
+or `qvm-pci`, then list the available PCI devices:
+
+~~~
+qvm-pci
+~~~
+
+This will show you the `backend:BDF` address of each PCI device. It will look something
+like `dom0:00_1a.0`. Once you've found the address of the device you want to
+assign, then attach it like so:
+
+~~~
+qvm-pci attach --persistent --option <option1> [--option <option2>] <vmname> <backend>:<bdf>
+~~~
+
+For example, if `00_1a.0` is the BDF of the device you want to assign to the
+"personal" domain, and it is particularly difficult to pass through you would do this:
+
+~~~
+qvm-pci attach --persistent --option permissive=true --option no-strict-reset=true personal dom0:00_1a.0
+~~~
+
+Running `qvm-pci` again should then show your PCI device attached with both the
+`permissive` and `no-strict-reset` options set.
+
+**Note** again that in most cases you should
+not need either of these options set. Only set one or more of them as required to get
+your device to function, or replace the device with one that functions properly with Qubes.
+
+R3.2
+------------------------
+
+Permissive mode is enabled system wide per device.
+
+Create `/etc/systemd/system/qubes-pre-netvm.service`:
 
 ~~~
 [Unit]
@@ -136,13 +225,15 @@ WantedBy=multi-user.target
 
 Then enable it with `systemctl enable qubes-pre-netvm.service`
 
-See also: [this thread][ml2] and the Xen wiki's [PCI passthrough] page.
+The strict reset option is set for all devices attached to a VM with:
 
-**NOTE:** By setting the permissive flag for the PCI device, you're potentially
-weakening the device isolation, especially if your system is not equipped with
-VT-d Interrupt Remapping unit. See [Software Attacks on Intel VT-d] (page 7)
-for more details.
+```
+qvm-prefs usbVM -s pci_strictreset false
+```
 
+**Note** again that in most cases you should
+not need either of these options set. Only set one or more of them as required to get
+your device to function, or replace the device with one that functions properly with Qubes.
 
 Bringing PCI device back to dom0
 --------------------------------