gaming hvm doc update

This commit is contained in:
Neowutran 2023-01-18 16:32:32 +01:00
parent 10be0e29df
commit 602565906e
No known key found for this signature in database
GPG Key ID: 0150AFFAFF83288F
4 changed files with 351 additions and 255 deletions

View File

@ -42,7 +42,7 @@ requests](https://github.com/Qubes-Community/Contents/pulls).
`customization`
- [change DPI scaling in dom0 and VMs](/docs/customization/dpi-scaling.md)
- [setup mirage firewall](/docs/customization/mirage-firewall.md)
- [windows gaming HVM with GPU passthrough](/docs/customization/windows-gaming-hvm.md)
- [gaming HVM with GPU passthrough](/docs/customization/gaming-hvm.md)
- [Choose deafult terminal settings for a TemplateVM](/docs/customization/terminal-defaults.md)
- [Screenlockers](/docs/customization/screenlockers.md)

View File

@ -0,0 +1,349 @@
# Create a Gaming HVM
## Hardware
To have an 'HVM' for gaming, you must have
- A dedicated GPU. By dedicated, it means: it is a secondary GPU, not
the GPU used to display dom0. In 2023, 'Nvidia' and 'Amd' GPU work.
Not tested with Intel GPUs.
- A screen available for the gaming 'HVM'. (It can be a physical
monitor or just to have multiple cables connected to the screen and
switching between input source)
- Dedicated gaming mouse and keyboard.
- A lot of patience. GPU passthrough is not trivial, and you will need
to spend time debugging.
## IOMMU Group
You need to check what are the things/devices that are in the same IOMMU
group as the GPU you want to passthrough. You can't see your IOMMU Group
when you are using Xen (the information is hidden from dom0). So, start
a live linux distribution, enable iommu in the grub options (iommu=1
iommu_amd=on), and then displayed the folder structure of
/sys/kernel/iommu_group
``` bash
#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done
done
```
## GRUB modification
You must hide your secondary GPU from dom0. To do that, you have to
modify the GRUB. In a dom0 Terminal, type:
``` bash
qvm-pci
```
Then find the devices id for your secondary GPU. In my case, it is
`dom0:0a_00.0`{.text} and `dom0:0a_00.1`{.text}. Edit /etc/default/grub,
and add the PCI hiding.
``` text
GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0,0a:00.1 "
```
then regenerate the grub
``` bash
grub2-mkconfig -o /boot/grub2/grub.cfg
```
If you are using UEFI, the file to override with `grub2-mkconfig`{.text}
is `/boot/efi/EFI/qubes/grub.cfg`{.text}.
Note: if after this step when you reboot the computer you get stuck in
the QubesOS startup that means you are trying to use the GPU you just
hide. Check your BIOS options. Also check the cables, BIOS have some GPU
priority based on the type of cable. For example, DisplayPort can be
favoured over HDMI.
Once you have rebooted, in dom0, type `sudo lspci -vvn`{.bash}, you
should see "Kernel driver in use: pciback" for the GPU you just hide.
## Patching stubdom-linux-rootfs.gz
[github.com/QubesOS/qubes-issues/issues/4321](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
Copy-paste of the comment:
This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G
provided by qemu not being large enough to accommodate the larger BARs
that a graphics card typically has. The code to pass a custom
max-ram-below-4g value to the qemu command line does exist in the
libxl_dm.c file of xen, but there is no functionality in libvirt to add
this parameter. It is possible to manually add this parameter to the
qemu commandline by doing the following in a dom0 terminal. (I modified
the code so it works with 4.1 and remove one of the original limitations
by restricting the modification to VM with a name starting with
"gpu\_\")
``` bash
mkdir stubroot
cp /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs stubroot/qemu-stubdom-linux-rootfs.gz
cd stubroot
gunzip qemu-stubdom-linux-rootfs.gz
cpio -i -d -H newc --no-absolute-filenames < qemu-stubdom-linux-rootfs
rm qemu-stubdom-linux-rootfs
nano init
```
Before the line
``` text
# $dm_args and $kernel are separated with \n to allow for spaces in arguments
```
add:
``` bash
# Patch 3.5 GB limit
vm_name=$(xenstore-read "/local/domain/$domid/name")
# Apply the patch only if the qube name start by "gpu_"
if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g=3.5G/g;p}')
fi
```
Then execute:
``` bash
find . -print0 | cpio --null -ov \
--format=newc | gzip -9 > ../qemu-stubdom-linux-rootfs
sudo mv ../qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/
```
Note that this will apply the change to the HVM with a name starting
with \"gpu\_\". So you need to name your gaming HVM \"gpu_SOMETHING\".
## Preparing the guest
As of 2023, I recommend using a Linux guest instead of a window guest.
### Windows
Install a window VM, you can use this
[qvm-create-windows-qube](https://github.com/elliotkillick/qvm-create-windows-qube)
### Linux
Create a new standalone Qube based on the template of your choice.
You must run the kernel provided by the guest distribution, because we
will use some non-default kernel module for the GPU driver. Just follow
the doc:
[managing-vm-kernel](https://www.qubes-os.org/doc/managing-vm-kernel/#distribution-kernel).
Install the GPU drivers you need.
## Pass the GPU
In qubes settings for the HVM, go to the 'devices' tab, pass the ID
corresponding to your GPU.
You may or may not need to add the option \"permissive\" or
\"no-strict-reset\".
[Some word about the security implication of thoses
parameters.](https://www.qubes-os.org/doc/device-handling-security/#pci-security)
``` bash
qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 -o permissive=True -o no-strict-reset=True
qvm-pci attach gpu_gaming_archlinux dom0:0a_00.1 -o permissive=True -o no-strict-reset=True
```
## Starting the guest
This is where you will have a lot of issues to debug.
For Linux guests, run 'sudo dmesg' to have all the kernel log indicating
you if there is a issue with your GPU driver. For some hardware, the MSI
calls won't work. You can work around that using for example
`pci=nomsi`{.text} or `NVreg_EnableMSI=0`{.text} or something else.
Check your drivers options. Check if alternative drivers exist (amdgpu,
nvidia, nouveau, nvidia-open, using drivers from the official website,
...). Check multiple kernel version.
Some links that could help you to debug the issues you will have
- https://forum.qubes-os.org/t/ryzen-7000-serie/
- https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html
For windows guests you will probably have the same issues but it will be
harder to debug. I recommend using the drivers from Windows Update
instead of the official drivers from the website of the constructor.
Some things that may be useful for debugging:
- Virsh (start, define, \...)
- /etc/libvirt/libxl/
- xl
- /etc/qubes/templates/libvirt/xen/by-name/
- /usr/lib/xen/boot/
- virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/\...
Issues with the drivers could be related to
'qubes-vmm-xen-stubdom-linux', 'qubes-vmm-xen', and the Linux kernel you
will be using.
## Linux guest --- Integration with QubesOS
### Xorg
Now Xorg and Pulseaudio. From XKCD:
[![image](x11){width="\\linewidth"}](https://xkcd.com/963/)
Things you need to install:
- The Xorg input driver to support your mouse and keyboard
- A pulseaudio gui client
- Your favorite Windows Manager
In my case, it is:
``` bash
apt install xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-mouse pavucontrol i3
```
Then create a XORG configuration file for your GPU and screen. My file
named 'AOC.conf':
``` xorg.conf
Section "ServerLayout"
Identifier "Gaming"
Screen 0 "AMD AOC" Absolute 0 0
EndSection
Section "Device"
Identifier "AMD"
# name of the driver to use. Can be "amdgpu", "nvidia", or something else
Driver "amdgpu"
# The BusID value will change after each qube reboot.
BusID "PCI:0:8:0"
EndSection
Section "Monitor"
Identifier "AOC"
VertRefresh 60
# https://arachnoid.com/modelines/ . IMPORTANT TO GET RIGHT. MUST ADJUST WITH EACH SCREEN.
Modeline "1920x1080" 172.80 1920 2040 2248 2576 1080 1081 1084 1118
EndSection
Section "Screen"
Identifier "AMD AOC"
Device "AMD"
Monitor "AOC"
EndSection
```
We can't know what is the correct BusID before the qube is started. And
it change after each reboot. So let's write a script --- named
\"xorgX1.sh\" --- that update this configuration file with the correct
value, then start a binary on the Xorg X screen n°1.
``` bash
#!/bin/bash
binary=${1:?binary required}
# Find the correct BusID of the AMD GPU, then set it in the Xorg configuration file
pci=$(lspci | grep "VGA" | grep "NVIDIA|AMD/ATI" | cut -d " " -f 1 | cut -d ":" -f 2 | cut -d "." -f 1 | cut -d "0" -f 2)
sed -i "s/PCI:0:[0-9]:0/PCI:0:$pci:0/g" /home/user/AOC.conf
# Pulseaudio setup
sudo killall pulseaudio
sudo sed -i "s/load-module module-vchan-sink.*/load-module module-vchan-sink domid=$(qubesdb-read -w /qubes-audio-domain-xid)/" /etc/pulse/qubes-default.pa
sudo rm /home/user/.pulse/client.conf
start-pulseaudio-with-vchan
sleep 5 && sudo chmod -R 777 /root/ &
sleep 5 && sudo chmod -R 777 /root/* &
sleep 5 && sudo cp /root/.pulse/client.conf /home/user/.pulse/client.conf && sudo chown -R user:user /home/user/.pulse/client.conf &
setxkbmap fr
sudo setxkbmap fr
# Start the Xorg server for the X screen number 1.
# The X screen n°0 is already used for QubesOS integration
sudo startx "$binary" -- :1 -config /home/user/AOC.conf
```
### Pulseaudio
So you need to configure pulseaudio for Xorg multiseat. The archlinux
documentation explain that very well: [Xorg
multiseat](https://wiki.archlinux.org/index.php/Xorg_multiseat#Multiple_users_on_single_sound_card:_PulseAudio)
Use the option without system-mode deamon and adapt it to qube: Add the
following line to /etc/pulse/qubes-default.pa
``` bash
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1
```
Then add this config for root:
``` bash
mkdir /root/.pulse
echo "default-server = 127.0.0.1" > /root/.pulse/client.conf
```
The sound was buggy/laggy on my computer. So tried to find a workaround
by playing with pulseaudio settings. It was more or less random tries,
so I can't really explain it: In `/etc/pulse/daemon.conf`{.text} add the
following lines:
``` bash
default-fragments = 60
default-fragment-size-msec = 1
high-priority = no
realtime-scheduling = no
nice-level = 18
```
In `/etc/pulse/qubes-default.pa`{.text} change
``` bash
load-module module-udev-detect
```
to
``` bash
load-module module-udev-detect tsched=0
```
You can launch you favorite Windows Manager like that
``` bash
sudo ./xorgX1.sh /usr/bin/i3
```
### References
- [Archlinux:
PulseAudio](https://wiki.archlinux.org/index.php/PulseAudio)
- [Archlinux:
PulseAudio/Troubleshooting](https://wiki.archlinux.org/index.php/PulseAudio/Troubleshooting)

View File

@ -1,253 +0,0 @@
# Create a Gaming HVM
Some information to configure a windows HVM for gaming.
This is not officially supported, just some community trial & errors.
This doc is also hosted on https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html
## References
Everything needed is referenced here
- [Useful technical details](https://paste.debian.net/1043341/)
- [Reddit thread of what is needed for GPU
passthrough](https://www.reddit.com/r/Qubes/comments/9hp3e7/gpu_passthrough_howto/)
- [Solution to have more than 3 GB of RAM in the Windows
HVM](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
- [Some old
references](https://www.reddit.com/r/Qubes/comments/66wk4q/gpu_passthrough/)
## Prerequisite
You have a functional Windows HVM (Windows 7 or Windows 10). The \"how
to\" for this part can be found on the Qubes OS documentation and here:
[Usefull github
comment](https://github.com/QubesOS/qubes-issues/issues/3585#issuecomment-453200971).
However, few tips:
- Do a backup (clone VM) of the Windows HVM BEFORE starting to install
QWT (installing QWT is not required)
## Hardware
To have a Windows HVM for gaming, you must have:
- A dedicated AMD GPU. By dedicated, it means: it is a secondary GPU,
not the GPU used to display dom0. Nvidia GPU are not supported (or
maybe with a lot of tricks).
- A really fast disk (M.2 disk)
- A lot of RAM
- A dedicated screen
- Dedicated gaming mouse and keyboard
In my case:
- Secondary GPU: AMD RX580
- Primary GPU: Some Nvidia trash, used for dom0
- 32 GB of RAM. 12 GB of RAM will be dedicated to the Windows HVM
- A fast M.2 disk
## Checklist
Short list of things to do to make the GPU passthrough work:
- You verified and confirmed that the secondary GPU is alone in its
IOMMU Group
- In dom0, you edited the file `/etc/default/grub` or
`/boot/efi/EFI/qubes/xen.cfg` or
`/boot/efi/EFI/qubes/grub.cfg` to allow PCI hiding for your
secondary GPU, and regenerated the grub if needed
- You have patched stubdom-linux-rootfs.gz to allow to have more than
3 GB of RAM for your HVM
## IOMMU Group
Warning: I am far from understanding the IOMMU group. Check online
references on that subject. It seems that you can only do a successfull
GPU passthough if you can passthrough everything that is in the IOMMU
Group of the GPU. Also, you can't see your IOMMU Group when you are
using Xen (the information is hidden from dom0). So, what I did: I
booted from a Linux Mint Live USB. In the grub I enabled the IOMMU
(iommu=1 iommu_amd=on), and then displayed the folder structure of
/sys/kernel/iommu_group
``` bash
tree /sys/kernel/iommu_group
```
My secondary GPU was alone in its IOMMU group.
## GRUB modification
You must hide your secondary GPU from dom0. To do that, you have to edit
the GRUB. In a dom0 Terminal, type:
``` bash
qvm-pci
```
Then find the devices id for your secondary GPU. In my case, it is
`dom0:0a_00.0` and `dom0:0a_00.1`. Edit /etc/default/grub,
and add the PCI hiding
``` text
GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0,0a:00.1 "
```
then regenerate the grub
``` bash
grub2-mkconfig -o /boot/grub2/grub.cfg
```
Or if using UEFI boot, edit `/boot/efi/EFI/qubes/xen.cfg` or
`/boot/efi/EFI/qubes/grub.cfg` and add the
`rd.qubes.hide_pci=` option to the `kernel=` line.
## Patching stubdom-linux-rootfs.gz
### Qubes R4.0
Follow the instructions here:\
[github.com/QubesOS/qubes-issues/issues/4321](https://github.com/QubesOS/qubes-issues/issues/4321#issuecomment-423011787)
Copy-paste of the comment:
This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G
provided by qemu not being large enough to accommodate the larger BARs
that a graphics card typically has. The code to pass a custom
max-ram-below-4g value to the qemu command line does exist in the
libxl_dm.c file of xen, but there is no functionality in libvirt to add
this parameter. It is possible to manually add this parameter to the
qemu commandline by doing the following in a dom0 terminal:
``` bash
mkdir stubroot
cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz
cd stubroot
gunzip stubdom-linux-rootfs.gz
cpio -i -d -H newc --no-absolute-filenames < stubdom-linux-rootfs
rm stubdom-linux-rootfs
nano init
```
Before the line
``` text
#$dm_args and $kernel are separated with \x1b to allow for spaces in arguments.
```
add:
``` bash
SP=$'\x1b'
dm_args=$(echo "$dm_args" \
| sed "s/-machine\\${SP}xenfv/-machine\
\\${SP}xenfv,max-ram-below-4g=3.5G/g")
```
Then execute:
``` bash
find . -print0 | cpio --null -ov \
--format=newc | gzip -9 > ../stubdom-linux-rootfs
sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot/
```
Note that this will apply the change to all HVMs, so if you have any
other HVM with more than 3.5GB ram assigned, they will not start without
the adapter being passed through. Ideally to fix this libvirt should be
extended to pass the max-ram-below-4g parameter through to xen, and then
a calculation added to determine the correct TOLUD based on the total
BAR size of the PCI devices are being passed through to the vm.
### Qubes R4.1
For Qubes R4.1 follow the R4.0 section, except for two things.
- The file that need to be patched is now
\"/usr/libexec/xen/boot/qemu-stubdom-linux-rootfs\"
- The content of the \"init\" file to modify.
#### The content of the \"init\" file
Before the line
``` text
# $dm_args and $kernel are separated with \n to allow for spaces in arguments
```
add:
``` bash
# Patch 3.5 GB limit
vm_name=$(xenstore-read "/local/domain/$domid/name")
# Apply the patch only if the qube name start by "gpu_"
if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then
dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g=3.5G/g;p}')
fi
```
## Pass the GPU
In qubes settings for the windows HVM, go to the \"devices\" tab, pass
the ID corresponding to your AMD GPU. (in my case, it was 0a:00.0 and
0a:00.1) And check the option for \"nostrict reset\" for those 2. In
some case, you might also need to set the \"permissive\" flag to true
(But I didn't need that with the RX 580):
``` bash
qvm-pci attach windows-hvm dom0:0a_00.0 -o permissive=True -o no-strict-reset=True
qvm-pci attach windows-hvm dom0:0a_00.1 -o permissive=True -o no-strict-reset=True
```
## Conclusion
Don't forget to install the GPU drivers, you can install the official
one from AMD website, no modification or trick to do. Nothing else is
required to make it work (in my case at least, once I finish to fight to
find those informations). If you have issues, you can refer to the links
in the first sections. If it doesn't work and you need to debug more
things, you can go deeper.
- Virsh (start, define, \...)
- /etc/libvirt/libxl/
- xl
- /etc/qubes/templates/libvirt/xen/by-name/
- /usr/lib/xen/boot/
- virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/\...
I am able to play games on my windows HVM with very good performances.
And safely.
## Bugs
The AMD GPUs have a bug when used in HVM: each time you will reboot your
windows HVM, it will get slower and slower. It is because the AMD GPUs
is not correctly reset when you restart your Windows HVM. Two
solutions for that:
- Reboot your computer
- In the windows HVM, use to windows option in the system tray to
\"safely remove devices\", remove your GPU. Restart the HVM.
This bug is referenced somewhere, but lost the link and too lazy to
search for it.

View File

@ -11,5 +11,5 @@ For more information about Windows VMs in Qubes OS, please see the following ext
* [Installing and Using Qubes Windows Tools in Qubes R4.0](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-tools.md)
* [Installing and Using Windows-based VMs in Qubes R4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-vm41.md)
* [Installing and Using Qubes Windows Tools in Qubes R4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-tools41.md)
* [Create a Gaming HVM in Qubes 4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/customization/windows-gaming-hvm.md)
* [Create a Gaming HVM in Qubes 4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/customization/gaming-hvm.md)
* [Migrate backups of Windows VMs created under Qubes R4.0 to R4.1](https://github.com/Qubes-Community/Contents/blob/master/docs/os/windows/windows-migrate41.md)