From 602565906ebd3386b8ddf59f850c9ff761f45ead Mon Sep 17 00:00:00 2001 From: Neowutran Date: Wed, 18 Jan 2023 16:32:32 +0100 Subject: [PATCH] gaming hvm doc update --- README.md | 2 +- docs/customization/gaming-hvm.md | 349 +++++++++++++++++++++++ docs/customization/windows-gaming-hvm.md | 253 ---------------- docs/os/windows/windows.md | 2 +- 4 files changed, 351 insertions(+), 255 deletions(-) create mode 100644 docs/customization/gaming-hvm.md delete mode 100644 docs/customization/windows-gaming-hvm.md diff --git a/README.md b/README.md index 653476c..683ac41 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ requests](https://github.com/Qubes-Community/Contents/pulls). `customization` - [change DPI scaling in dom0 and VMs](/docs/customization/dpi-scaling.md) - [setup mirage firewall](/docs/customization/mirage-firewall.md) -- [windows gaming HVM with GPU passthrough](/docs/customization/windows-gaming-hvm.md) +- [gaming HVM with GPU passthrough](/docs/customization/gaming-hvm.md) - [Choose deafult terminal settings for a TemplateVM](/docs/customization/terminal-defaults.md) - [Screenlockers](/docs/customization/screenlockers.md) diff --git a/docs/customization/gaming-hvm.md b/docs/customization/gaming-hvm.md new file mode 100644 index 0000000..7c4068a --- /dev/null +++ b/docs/customization/gaming-hvm.md @@ -0,0 +1,349 @@ +# Create a Gaming HVM + +## Hardware + +To have an 'HVM' for gaming, you must have + +- A dedicated GPU. By dedicated, it means: it is a secondary GPU, not + the GPU used to display dom0. In 2023, 'Nvidia' and 'Amd' GPU work. + Not tested with Intel GPUs. + +- A screen available for the gaming 'HVM'. (It can be a physical + monitor or just to have multiple cables connected to the screen and + switching between input source) + +- Dedicated gaming mouse and keyboard. + +- A lot of patience. GPU passthrough is not trivial, and you will need + to spend time debugging. + +## IOMMU Group + +You need to check what are the things/devices that are in the same IOMMU +group as the GPU you want to passthrough. You can't see your IOMMU Group +when you are using Xen (the information is hidden from dom0). So, start +a live linux distribution, enable iommu in the grub options (iommu=1 +iommu_amd=on), and then displayed the folder structure of +/sys/kernel/iommu_group + +``` bash +#!/bin/bash +shopt -s nullglob +for g in /sys/kernel/iommu_groups/*; do + echo "IOMMU Group ${g##*/}:" + for d in $g/devices/*; do + echo -e "\t$(lspci -nns ${d##*/})" +done +done +``` + +## GRUB modification + +You must hide your secondary GPU from dom0. To do that, you have to +modify the GRUB. In a dom0 Terminal, type: + +``` bash +qvm-pci +``` + +Then find the devices id for your secondary GPU. In my case, it is +`dom0:0a_00.0`{.text} and `dom0:0a_00.1`{.text}. Edit /etc/default/grub, +and add the PCI hiding. + +``` text +GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0,0a:00.1 " +``` + +then regenerate the grub + +``` bash +grub2-mkconfig -o /boot/grub2/grub.cfg +``` + +If you are using UEFI, the file to override with `grub2-mkconfig`{.text} +is `/boot/efi/EFI/qubes/grub.cfg`{.text}. + +Note: if after this step when you reboot the computer you get stuck in +the QubesOS startup that means you are trying to use the GPU you just +hide. Check your BIOS options. Also check the cables, BIOS have some GPU +priority based on the type of cable. For example, DisplayPort can be +favoured over HDMI. + +Once you have rebooted, in dom0, type `sudo lspci -vvn`{.bash}, you +should see "Kernel driver in use: pciback" for the GPU you just hide. + +## Patching stubdom-linux-rootfs.gz + +[github.com/QubesOS/qubes-issues/issues/4321](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787) + +Copy-paste of the comment: + +This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G +provided by qemu not being large enough to accommodate the larger BARs +that a graphics card typically has. The code to pass a custom +max-ram-below-4g value to the qemu command line does exist in the +libxl_dm.c file of xen, but there is no functionality in libvirt to add +this parameter. It is possible to manually add this parameter to the +qemu commandline by doing the following in a dom0 terminal. (I modified +the code so it works with 4.1 and remove one of the original limitations +by restricting the modification to VM with a name starting with +"gpu\_\") + +``` bash +mkdir stubroot +cp /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs stubroot/qemu-stubdom-linux-rootfs.gz +cd stubroot +gunzip qemu-stubdom-linux-rootfs.gz +cpio -i -d -H newc --no-absolute-filenames < qemu-stubdom-linux-rootfs +rm qemu-stubdom-linux-rootfs +nano init +``` + +Before the line + +``` text +# $dm_args and $kernel are separated with \n to allow for spaces in arguments +``` + +add: + +``` bash +# Patch 3.5 GB limit +vm_name=$(xenstore-read "/local/domain/$domid/name") +# Apply the patch only if the qube name start by "gpu_" +if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then + dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g=3.5G/g;p}') +fi +``` + +Then execute: + +``` bash +find . -print0 | cpio --null -ov \ +--format=newc | gzip -9 > ../qemu-stubdom-linux-rootfs +sudo mv ../qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/ +``` + +Note that this will apply the change to the HVM with a name starting +with \"gpu\_\". So you need to name your gaming HVM \"gpu_SOMETHING\". + +## Preparing the guest + +As of 2023, I recommend using a Linux guest instead of a window guest. + +### Windows + +Install a window VM, you can use this +[qvm-create-windows-qube](https://github.com/elliotkillick/qvm-create-windows-qube) + +### Linux + +Create a new standalone Qube based on the template of your choice. + +You must run the kernel provided by the guest distribution, because we +will use some non-default kernel module for the GPU driver. Just follow +the doc: +[managing-vm-kernel](https://www.qubes-os.org/doc/managing-vm-kernel/#distribution-kernel). + +Install the GPU drivers you need. + +## Pass the GPU + +In qubes settings for the HVM, go to the 'devices' tab, pass the ID +corresponding to your GPU. + +You may or may not need to add the option \"permissive\" or +\"no-strict-reset\". + +[Some word about the security implication of thoses +parameters.](https://www.qubes-os.org/doc/device-handling-security/#pci-security) + +``` bash +qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 -o permissive=True -o no-strict-reset=True +qvm-pci attach gpu_gaming_archlinux dom0:0a_00.1 -o permissive=True -o no-strict-reset=True +``` + +## Starting the guest + +This is where you will have a lot of issues to debug. + +For Linux guests, run 'sudo dmesg' to have all the kernel log indicating +you if there is a issue with your GPU driver. For some hardware, the MSI +calls won't work. You can work around that using for example +`pci=nomsi`{.text} or `NVreg_EnableMSI=0`{.text} or something else. +Check your drivers options. Check if alternative drivers exist (amdgpu, +nvidia, nouveau, nvidia-open, using drivers from the official website, +...). Check multiple kernel version. + +Some links that could help you to debug the issues you will have + +- https://forum.qubes-os.org/t/ryzen-7000-serie/ + +- https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html + +For windows guests you will probably have the same issues but it will be +harder to debug. I recommend using the drivers from Windows Update +instead of the official drivers from the website of the constructor. + +Some things that may be useful for debugging: + +- Virsh (start, define, \...) + +- /etc/libvirt/libxl/ + +- xl + +- /etc/qubes/templates/libvirt/xen/by-name/ + +- /usr/lib/xen/boot/ + +- virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/\... + +Issues with the drivers could be related to +'qubes-vmm-xen-stubdom-linux', 'qubes-vmm-xen', and the Linux kernel you +will be using. + +## Linux guest --- Integration with QubesOS + +### Xorg + +Now Xorg and Pulseaudio. From XKCD: + +[![image](x11){width="\\linewidth"}](https://xkcd.com/963/) + +Things you need to install: + +- The Xorg input driver to support your mouse and keyboard + +- A pulseaudio gui client + +- Your favorite Windows Manager + +In my case, it is: + +``` bash +apt install xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-mouse pavucontrol i3 +``` + +Then create a XORG configuration file for your GPU and screen. My file +named 'AOC.conf': + +``` xorg.conf +Section "ServerLayout" +Identifier "Gaming" +Screen 0 "AMD AOC" Absolute 0 0 +EndSection + +Section "Device" +Identifier "AMD" + +# name of the driver to use. Can be "amdgpu", "nvidia", or something else +Driver "amdgpu" + +# The BusID value will change after each qube reboot. +BusID "PCI:0:8:0" +EndSection + +Section "Monitor" +Identifier "AOC" +VertRefresh 60 +# https://arachnoid.com/modelines/ . IMPORTANT TO GET RIGHT. MUST ADJUST WITH EACH SCREEN. +Modeline "1920x1080" 172.80 1920 2040 2248 2576 1080 1081 1084 1118 +EndSection + +Section "Screen" +Identifier "AMD AOC" +Device "AMD" +Monitor "AOC" +EndSection +``` + +We can't know what is the correct BusID before the qube is started. And +it change after each reboot. So let's write a script --- named +\"xorgX1.sh\" --- that update this configuration file with the correct +value, then start a binary on the Xorg X screen n°1. + +``` bash +#!/bin/bash + +binary=${1:?binary required} + +# Find the correct BusID of the AMD GPU, then set it in the Xorg configuration file +pci=$(lspci | grep "VGA" | grep "NVIDIA|AMD/ATI" | cut -d " " -f 1 | cut -d ":" -f 2 | cut -d "." -f 1 | cut -d "0" -f 2) +sed -i "s/PCI:0:[0-9]:0/PCI:0:$pci:0/g" /home/user/AOC.conf + +# Pulseaudio setup +sudo killall pulseaudio +sudo sed -i "s/load-module module-vchan-sink.*/load-module module-vchan-sink domid=$(qubesdb-read -w /qubes-audio-domain-xid)/" /etc/pulse/qubes-default.pa +sudo rm /home/user/.pulse/client.conf +start-pulseaudio-with-vchan +sleep 5 && sudo chmod -R 777 /root/ & +sleep 5 && sudo chmod -R 777 /root/* & +sleep 5 && sudo cp /root/.pulse/client.conf /home/user/.pulse/client.conf && sudo chown -R user:user /home/user/.pulse/client.conf & + +setxkbmap fr +sudo setxkbmap fr + +# Start the Xorg server for the X screen number 1. +# The X screen n°0 is already used for QubesOS integration +sudo startx "$binary" -- :1 -config /home/user/AOC.conf +``` + +### Pulseaudio + +So you need to configure pulseaudio for Xorg multiseat. The archlinux +documentation explain that very well: [Xorg +multiseat](https://wiki.archlinux.org/index.php/Xorg_multiseat#Multiple_users_on_single_sound_card:_PulseAudio) +Use the option without system-mode deamon and adapt it to qube: Add the +following line to /etc/pulse/qubes-default.pa + +``` bash +load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1 +``` + +Then add this config for root: + +``` bash +mkdir /root/.pulse +echo "default-server = 127.0.0.1" > /root/.pulse/client.conf +``` + +The sound was buggy/laggy on my computer. So tried to find a workaround +by playing with pulseaudio settings. It was more or less random tries, +so I can't really explain it: In `/etc/pulse/daemon.conf`{.text} add the +following lines: + +``` bash +default-fragments = 60 +default-fragment-size-msec = 1 +high-priority = no +realtime-scheduling = no +nice-level = 18 +``` + +In `/etc/pulse/qubes-default.pa`{.text} change + +``` bash +load-module module-udev-detect +``` + +to + +``` bash +load-module module-udev-detect tsched=0 +``` + +You can launch you favorite Windows Manager like that + +``` bash +sudo ./xorgX1.sh /usr/bin/i3 +``` + +### References + +- [Archlinux: + PulseAudio](https://wiki.archlinux.org/index.php/PulseAudio) + +- [Archlinux: + PulseAudio/Troubleshooting](https://wiki.archlinux.org/index.php/PulseAudio/Troubleshooting) + diff --git a/docs/customization/windows-gaming-hvm.md b/docs/customization/windows-gaming-hvm.md deleted file mode 100644 index a39a10b..0000000 --- a/docs/customization/windows-gaming-hvm.md +++ /dev/null @@ -1,253 +0,0 @@ -# Create a Gaming HVM - -Some information to configure a windows HVM for gaming. -This is not officially supported, just some community trial & errors. -This doc is also hosted on https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html - -## References - -Everything needed is referenced here - -- [Useful technical details](https://paste.debian.net/1043341/) - -- [Reddit thread of what is needed for GPU - passthrough](https://www.reddit.com/r/Qubes/comments/9hp3e7/gpu_passthrough_howto/) - -- [Solution to have more than 3 GB of RAM in the Windows - HVM](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787) - -- [Some old - references](https://www.reddit.com/r/Qubes/comments/66wk4q/gpu_passthrough/) - -## Prerequisite - -You have a functional Windows HVM (Windows 7 or Windows 10). The \"how -to\" for this part can be found on the Qubes OS documentation and here: -[Usefull github -comment](https://github.com/QubesOS/qubes-issues/issues/3585#issuecomment-453200971). -However, few tips: - -- Do a backup (clone VM) of the Windows HVM BEFORE starting to install - QWT (installing QWT is not required) - -## Hardware - -To have a Windows HVM for gaming, you must have: - -- A dedicated AMD GPU. By dedicated, it means: it is a secondary GPU, - not the GPU used to display dom0. Nvidia GPU are not supported (or - maybe with a lot of tricks). - -- A really fast disk (M.2 disk) - -- A lot of RAM - -- A dedicated screen - -- Dedicated gaming mouse and keyboard - -In my case: - -- Secondary GPU: AMD RX580 - -- Primary GPU: Some Nvidia trash, used for dom0 - -- 32 GB of RAM. 12 GB of RAM will be dedicated to the Windows HVM - -- A fast M.2 disk - -## Checklist - -Short list of things to do to make the GPU passthrough work: - -- You verified and confirmed that the secondary GPU is alone in its - IOMMU Group - -- In dom0, you edited the file `/etc/default/grub` or - `/boot/efi/EFI/qubes/xen.cfg` or - `/boot/efi/EFI/qubes/grub.cfg` to allow PCI hiding for your - secondary GPU, and regenerated the grub if needed - -- You have patched stubdom-linux-rootfs.gz to allow to have more than - 3 GB of RAM for your HVM - -## IOMMU Group - -Warning: I am far from understanding the IOMMU group. Check online -references on that subject. It seems that you can only do a successfull -GPU passthough if you can passthrough everything that is in the IOMMU -Group of the GPU. Also, you can't see your IOMMU Group when you are -using Xen (the information is hidden from dom0). So, what I did: I -booted from a Linux Mint Live USB. In the grub I enabled the IOMMU -(iommu=1 iommu_amd=on), and then displayed the folder structure of -/sys/kernel/iommu_group - -``` bash -tree /sys/kernel/iommu_group -``` - -My secondary GPU was alone in its IOMMU group. - -## GRUB modification - -You must hide your secondary GPU from dom0. To do that, you have to edit -the GRUB. In a dom0 Terminal, type: - -``` bash -qvm-pci -``` - -Then find the devices id for your secondary GPU. In my case, it is -`dom0:0a_00.0` and `dom0:0a_00.1`. Edit /etc/default/grub, -and add the PCI hiding - -``` text -GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0,0a:00.1 " -``` - -then regenerate the grub - -``` bash -grub2-mkconfig -o /boot/grub2/grub.cfg -``` - -Or if using UEFI boot, edit `/boot/efi/EFI/qubes/xen.cfg` or -`/boot/efi/EFI/qubes/grub.cfg` and add the -`rd.qubes.hide_pci=` option to the `kernel=` line. - -## Patching stubdom-linux-rootfs.gz - -### Qubes R4.0 - -Follow the instructions here:\ -[github.com/QubesOS/qubes-issues/issues/4321](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787) - -Copy-paste of the comment: - -This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G -provided by qemu not being large enough to accommodate the larger BARs -that a graphics card typically has. The code to pass a custom -max-ram-below-4g value to the qemu command line does exist in the -libxl_dm.c file of xen, but there is no functionality in libvirt to add -this parameter. It is possible to manually add this parameter to the -qemu commandline by doing the following in a dom0 terminal: - -``` bash -mkdir stubroot -cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz -cd stubroot -gunzip stubdom-linux-rootfs.gz -cpio -i -d -H newc --no-absolute-filenames < stubdom-linux-rootfs -rm stubdom-linux-rootfs -nano init -``` - -Before the line - -``` text -#$dm_args and $kernel are separated with \x1b to allow for spaces in arguments. -``` - -add: - -``` bash -SP=$'\x1b' -dm_args=$(echo "$dm_args" \ - | sed "s/-machine\\${SP}xenfv/-machine\ -\\${SP}xenfv,max-ram-below-4g=3.5G/g") -``` - -Then execute: - -``` bash -find . -print0 | cpio --null -ov \ ---format=newc | gzip -9 > ../stubdom-linux-rootfs -sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot/ -``` - -Note that this will apply the change to all HVMs, so if you have any -other HVM with more than 3.5GB ram assigned, they will not start without -the adapter being passed through. Ideally to fix this libvirt should be -extended to pass the max-ram-below-4g parameter through to xen, and then -a calculation added to determine the correct TOLUD based on the total -BAR size of the PCI devices are being passed through to the vm. - -### Qubes R4.1 - -For Qubes R4.1 follow the R4.0 section, except for two things. - -- The file that need to be patched is now - \"/usr/libexec/xen/boot/qemu-stubdom-linux-rootfs\" - -- The content of the \"init\" file to modify. - -#### The content of the \"init\" file - -Before the line - -``` text -# $dm_args and $kernel are separated with \n to allow for spaces in arguments -``` - -add: - -``` bash -# Patch 3.5 GB limit -vm_name=$(xenstore-read "/local/domain/$domid/name") -# Apply the patch only if the qube name start by "gpu_" -if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then - dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g=3.5G/g;p}') -fi -``` - -## Pass the GPU - -In qubes settings for the windows HVM, go to the \"devices\" tab, pass -the ID corresponding to your AMD GPU. (in my case, it was 0a:00.0 and -0a:00.1) And check the option for \"nostrict reset\" for those 2. In -some case, you might also need to set the \"permissive\" flag to true -(But I didn't need that with the RX 580): - -``` bash -qvm-pci attach windows-hvm dom0:0a_00.0 -o permissive=True -o no-strict-reset=True -qvm-pci attach windows-hvm dom0:0a_00.1 -o permissive=True -o no-strict-reset=True -``` - -## Conclusion - -Don't forget to install the GPU drivers, you can install the official -one from AMD website, no modification or trick to do. Nothing else is -required to make it work (in my case at least, once I finish to fight to -find those informations). If you have issues, you can refer to the links -in the first sections. If it doesn't work and you need to debug more -things, you can go deeper. - -- Virsh (start, define, \...) - -- /etc/libvirt/libxl/ - -- xl - -- /etc/qubes/templates/libvirt/xen/by-name/ - -- /usr/lib/xen/boot/ - -- virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/\... - -I am able to play games on my windows HVM with very good performances. -And safely. - -## Bugs - -The AMD GPUs have a bug when used in HVM: each time you will reboot your -windows HVM, it will get slower and slower. It is because the AMD GPUs -is not correctly reset when you restart your Windows HVM. Two -solutions for that: - -- Reboot your computer - -- In the windows HVM, use to windows option in the system tray to - \"safely remove devices\", remove your GPU. Restart the HVM. - -This bug is referenced somewhere, but lost the link and too lazy to -search for it. diff --git a/docs/os/windows/windows.md b/docs/os/windows/windows.md index 72b266b..0a58660 100644 --- a/docs/os/windows/windows.md +++ b/docs/os/windows/windows.md @@ -11,5 +11,5 @@ For more information about Windows VMs in Qubes OS, please see the following ext * [Installing and Using Qubes Windows Tools in Qubes R4.0](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-tools.md) * [Installing and Using Windows-based VMs in Qubes R4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-vm41.md) * [Installing and Using Qubes Windows Tools in Qubes R4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-tools41.md) -* [Create a Gaming HVM in Qubes 4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/customization/windows-gaming-hvm.md) +* [Create a Gaming HVM in Qubes 4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/customization/gaming-hvm.md) * [Migrate backups of Windows VMs created under Qubes R4.0 to R4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-migrate41.md)