Merge pull request #139 from neowutran/master

Update Windows Gaming HVM documentation

docs/README.md

@@ -28,7 +28,7 @@
 `customization`
 - [change DPI scaling in dom0 and VMs](customization/dpi-scaling.md)
 - [setup mirage firewall](customization/mirage-firewall.md)
-- [windows 7 gaming HVM with GPU passthrough](customization/windows-gaming-hvm.md)
+- [windows gaming HVM with GPU passthrough](customization/windows-gaming-hvm.md)
 - [Choose deafult terminal settings for a TemplateVM](customization/terminal-defaults.md)  
 - [Screenlockers](customization/screenlockers.md)  
 - [USB Hardening](customization/usb-hardening.md)  

docs/customization/windows-gaming-hvm.md

@@ -1,73 +1,108 @@
-
+# Create a Gaming HVM
-# Windows gaming HVM
 
 Some information to configure a windows HVM for gaming. 
-This is not officially supported, just some community trial & errors
+This is not officially supported, just some community trial & errors.
+This doc is also hosted on https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html
 
 ## References
 
-Everythings needed is referenced here
+Everything needed is referenced here
 
 -   [Useful technical details](https://paste.debian.net/1043341/)
-- [Reddit thread of what is needed for GPU passthrough](https://www.reddit.com/r/Qubes/comments/9hp3e7/gpu_passthrough_howto/)
-- [Solution to have more than 3 GB of RAM in the Windows HVM](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
-- [Some old references](https://www.reddit.com/r/Qubes/comments/66wk4q/gpu_passthrough/)
 
-## Prerequise
+-   [Reddit thread of what is needed for GPU
+    passthrough](https://www.reddit.com/r/Qubes/comments/9hp3e7/gpu_passthrough_howto/)
 
-You have a functional Windows 7 HVM.
+-   [Solution to have more than 3 GB of RAM in the Windows
+    HVM](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
 
-The "how to" for this part can be found on the Qubes OS documentation and here: [Useful github comment](https://github.com/QubesOS/qubes-issues/issues/3585#issuecomment-453200971).
+-   [Some old
+    references](https://www.reddit.com/r/Qubes/comments/66wk4q/gpu_passthrough/)
 
+## Prerequisite
+
+You have a functional Windows HVM (Windows 7 or Windows 10). The \"how
+to\" for this part can be found on the Qubes OS documentation and here:
+[Usefull github
+comment](https://github.com/QubesOS/qubes-issues/issues/3585#issuecomment-453200971).
 However, few tips:
 
-- Do a backup (clone VM) of the Windows HVM BEFORE starting to install QWT
+-   Do a backup (clone VM) of the Windows HVM BEFORE starting to install
-- The Windows user MUST BE "user"
+    QWT (installing QWT is not required)
-- Windows 7 Only, do not use Windows 10 or others.
 
 ## Hardware
 
 To have a Windows HVM for gaming, you must have:
 
-- A dedicated AMD GPU. By dedicated, it means: it is a secondary GPU, not the GPU used to display dom0. Nvidia GPU are not supported (or maybe with a lot of trick).
+-   A dedicated AMD GPU. By dedicated, it means: it is a secondary GPU,
+    not the GPU used to display dom0. Nvidia GPU are not supported (or
+    maybe with a lot of tricks).
+
 -   A really fast disk (M.2 disk)
+
 -   A lot of RAM
+
 -   A dedicated screen
 
-In my case, I use:
+-   Dedicated gaming mouse and keyboard
+
+In my case:
 
 -   Secondary GPU: AMD RX580
-- Primary GPU: Some Nvidia trash, used for dom0
-- 32 GB of RAM. 16 GB of RAM will be dedicated for the Windows HVM
-- A fast M.2 disk
 
+-   Primary GPU: Some Nvidia trash, used for dom0
+
+-   32 GB of RAM. 12 GB of RAM will be dedicated to the Windows HVM
+
+-   A fast M.2 disk
 
 ## Checklist
 
 Short list of things to do to make the GPU passthrough work:
 
-- In dom0, you edited the file `/etc/default/grub` or `/boot/efi/EFI/qubes/xen.cfg` to allow PCI hiding for your secondary GPU, and regenerated the grub if needed
+-   You verified and confirmed that the secondary GPU is alone in its
-- You patched your stubdom-linux-rootfs.gz to allow to more than 3 GB of RAM for your Windows HVM
+    IOMMU Group
+
+-   In dom0, you edited the file `/etc/default/grub` or
+    `/boot/efi/EFI/qubes/xen.cfg` or
+    `/boot/efi/EFI/qubes/grub.cfg` to allow PCI hiding for your
+    secondary GPU, and regenerated the grub if needed
+
+-   You have patched stubdom-linux-rootfs.gz to allow to have more than
+    3 GB of RAM for your HVM
+
+## IOMMU Group
+
+Warning: I am far from understanding the IOMMU group. Check online
+references on that subject. It seems that you can only do a successfull
+GPU passthough if you can passthrough everything that is in the IOMMU
+Group of the GPU. Also, you can't see your IOMMU Group when you are
+using Xen (the information is hidden from dom0). So, what I did: I
+booted from a Linux Mint Live USB. In the grub I enabled the IOMMU
+(iommu=1 iommu_amd=on), and then displayed the folder structure of
+/sys/kernel/iommu_group
+
+``` bash
+tree /sys/kernel/iommu_group
+```
+
+My secondary GPU was alone in its IOMMU group.
 
 ## GRUB modification
 
-You must hide your secondary GPU from dom0.
+You must hide your secondary GPU from dom0. To do that, you have to edit
-To do that, you have to edit the GRUB or `xen.cfg` depending on if you use legacy or UEFI boot.
+the GRUB. In a dom0 Terminal, type:
-
-In a dom0 Terminal, type:
 
 ``` bash
 qvm-pci
 ```
 
-Then find the devices id for your secondary gpu.
+Then find the devices id for your secondary GPU. In my case, it is
+`dom0:0a_00.0` and `dom0:0a_00.1`. Edit /etc/default/grub,
+and add the PCI hiding
 
-In my case, it is "dom0:0a_00.0" and "dom0:0a_00.1".
+``` text
-
+GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0,0a:00.1 "
-Edit /etc/default/grub, and add the PCI hiding 
-
-```
-GRUB_CMDLINE_LINUX="....rd.qubes.hide_pci=0a:00.0,0a:00.1 "
 ```
 
 then regenerate the grub
@@ -76,18 +111,26 @@ then regenerate the grub
 grub2-mkconfig -o /boot/grub2/grub.cfg
 ```
 
-Or if using UEFI boot, edit `/boot/efi/EFI/qubes/xen.cfg` and add the `rd.qubes.hide_pci=` option to the `kernel=` line.
+Or if using UEFI boot, edit `/boot/efi/EFI/qubes/xen.cfg` or
+`/boot/efi/EFI/qubes/grub.cfg` and add the
+`rd.qubes.hide_pci=` option to the `kernel=` line.
 
 ## Patching stubdom-linux-rootfs.gz
 
-Follow the instructions here: [https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
+### Qubes R4.0
+
+Follow the instructions here:\
+[github.com/QubesOS/qubes-issues/issues/4321](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
 
 Copy-paste of the comment:
 
-```
+This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G
-This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has.
+provided by qemu not being large enough to accommodate the larger BARs
-The code to pass a custom max-ram-below-4g value to the qemu commandline does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to addthis parameter.
+that a graphics card typically has. The code to pass a custom
-It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal:
+max-ram-below-4g value to the qemu command line does exist in the
+libxl_dm.c file of xen, but there is no functionality in libvirt to add
+this parameter. It is possible to manually add this parameter to the
+qemu commandline by doing the following in a dom0 terminal:
 
 ``` bash
 mkdir stubroot
@@ -96,17 +139,24 @@ cd stubroot
 gunzip stubdom-linux-rootfs.gz
 cpio -i -d -H newc --no-absolute-filenames < stubdom-linux-rootfs
 rm stubdom-linux-rootfs
-nano init3
+nano init
 ```
 
-Before the line "# $dm_args and $kernel are separated withx1b to allow for spaces in arguments." add:
+Before the line
+
+``` text
+#$dm_args and $kernel are separated with \x1b to allow for spaces in arguments.
+```
+
+add:
 
 ``` bash
 SP=$'\x1b'
 dm_args=$(echo "$dm_args" \
-sed"s/-machine\\${SP}xenfv/-machine\
+ | sed "s/-machine\\${SP}xenfv/-machine\
 \\${SP}xenfv,max-ram-below-4g=3.5G/g")
 ```
+
 Then execute:
 
 ``` bash
@@ -115,55 +165,89 @@ find . -print0 | cpio --null -ov\
 sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot/
 ```
 
+Note that this will apply the change to all HVMs, so if you have any
+other HVM with more than 3.5GB ram assigned, they will not start without
+the adapter being passed through. Ideally to fix this libvirt should be
+extended to pass the max-ram-below-4g parameter through to xen, and then
+a calculation added to determine the correct TOLUD based on the total
+BAR size of the PCI devices are being passed through to the vm.
 
-Note that this will apply the change to all HVMs, so if you have any other HVM with more than 3.5 GB ram assigned,
+### Qubes R4.1
-they will not start without the adapter being passed through.
+
-Ideally to fix this libvirt should be extended to pass the max-ram-below-4g parameter through to xen,
+For Qubes R4.1 follow the R4.0 section, except for two things.
-and then a calculation added to determine the correct TOLUD based on the total BAR size of the PCI devices
+
-are being passed through to the vm.
+-   The file that need to be patched is now
+    \"/usr/libexec/xen/boot/qemu-stubdom-linux-rootfs\"
+
+-   The content of the \"init\" file to modify.
+
+#### The content of the \"init\" file
+
+Before the line
+
+``` text
+# $dm_args and $kernel are separated with \n to allow for spaces in arguments
+```
+
+add:
+
+``` bash
+# Patch 3.5 GB limit
+vm_name=$(xenstore-read "/local/domain/$domid/name")
+# Apply the patch only if the qube name start by "gpu_"
+if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
+ dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g=3.5G/g;p}')
+fi
+```
 
 ## Pass the GPU
 
-In qubes settings for the windows HVM, go to the "devices" tab, pass the ID corresponding to your AMD GPU.
+In qubes settings for the windows HVM, go to the \"devices\" tab, pass
-(in my case, it was 0a:00.0 and 0a:00.1)
+the ID corresponding to your AMD GPU. (in my case, it was 0a:00.0 and
+0a:00.1) And check the option for \"nostrict reset\" for those 2. In
+some case, you might also need to set the \"permissive\" flag to true
+(But I didn't need that with the RX 580):
 
-And check the option for "nostrictreset" for those. 
+``` bash
-
-
-
-
-In some case, you might also need to set the "permissive" flag to true (But I didn't need that with the RX 580): 
-```
 qvm-pci attach windows-hvm dom0:0a_00.0 -o permissive=True -o no-strict-reset=True
 qvm-pci attach windows-hvm dom0:0a_00.1 -o permissive=True -o no-strict-reset=True
 ```
 
 ## Conclusion
 
-Don’t forget to install the GPU drivers, you can install the official one from AMD website, no modification or trick to do.
+Don't forget to install the GPU drivers, you can install the official
+one from AMD website, no modification or trick to do. Nothing else is
+required to make it work (in my case at least, once I finish to fight to
+find those informations). If you have issues, you can refer to the links
+in the first sections. If it doesn't work and you need to debug more
+things, you can go deeper.
 
-Nothing else is required to make it work (in my case at least, once I finish to fight to find those informations).
+-   Virsh (start, define, \...)
 
-If you have issues, you can refer to the links in the first sections. 
-
-If it doesn’t work and you need to debug more things, you can go deeper.
-
-- Virsh (start, define, ...)
 -   /etc/libvirt/libxl/
-- xl
-- /etc/qubes/templates/libvirt/xen/by-name/
-- /usr/lib/xen/boot/
-- virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/...
 
-I am able to play games on my windows HVM with very good performances. And safely.
+-   xl
+
+-   /etc/qubes/templates/libvirt/xen/by-name/
+
+-   /usr/lib/xen/boot/
+
+-   virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/\...
+
+I am able to play games on my windows HVM with very good performances.
+And safely.
 
 ## Bugs
 
-The AMD GPUs have a bug when used in HVM: each time you will reboot your windows HVM, it will get slower and slower.
+The AMD GPUs have a bug when used in HVM: each time you will reboot your
-It is because the AMD GPU is not correctly reset when you restart your windows HVM.
+windows HVM, it will get slower and slower. It is because the AMD GPUs
+is not correctly reset when you restart your Windows HVM. Two
+solutions for that:
 
-Two solutions for that:
 -   Reboot your computer
-- In the windows HVM, use the windows option in the system tray to "safely remove devices", remove your GPU. Restart the HVM.
 
-This bug is referenced somewhere, but lost the link
+-   In the windows HVM, use to windows option in the system tray to
+    \"safely remove devices\", remove your GPU. Restart the HVM.
+
+This bug is referenced somewhere, but lost the link and too lazy to
+search for it.