From 5138bcaf199096af0a43d39536f071950825f489 Mon Sep 17 00:00:00 2001 From: PROTechThor Date: Sun, 18 Oct 2020 07:24:25 +0100 Subject: [PATCH 1/5] Add Disk Troubleshooting --- doc.md | 4 +- user/troubleshooting/disk-troubleshooting.md | 96 ++++++++++++++++++++ user/troubleshooting/out-of-memory.md | 46 ---------- 3 files changed, 98 insertions(+), 48 deletions(-) create mode 100644 user/troubleshooting/disk-troubleshooting.md delete mode 100644 user/troubleshooting/out-of-memory.md diff --git a/doc.md b/doc.md index 743c6477..37364941 100644 --- a/doc.md +++ b/doc.md @@ -112,7 +112,6 @@ Core documentation for Qubes users. * [Making Any File Persistent Using `bind-dirs`](/doc/bind-dirs/) * [GUI Configuration](/doc/gui-configuration/) * [Resizing Disk Images](/doc/resize-disk-image/) - * [Troubleshooting UEFI](/doc/uefi-troubleshooting/) * [Troubleshooting Newer Hardware](/doc/newer-hardware-troubleshooting/) * [Mounting and Decrypting Qubes Partitions from Outside Qubes](/doc/mount-from-other-os/) * [KDE](/doc/kde/) @@ -122,7 +121,8 @@ Core documentation for Qubes users. ### Troubleshooting * [Installation Troubleshooting](/doc/installation-troubleshooting) - * [Home directory is out of disk space error](/doc/out-of-memory/) + * [UEFI Troubleshooting](/doc/uefi-troubleshooting/) + * [Disk Troubleshooting](/doc/disk-troubleshooting/) * [Installing on system with new AMD GPU (missing firmware problem)](https://groups.google.com/group/qubes-devel/browse_thread/thread/e27a57b0eda62f76) * [How to install an Nvidia driver in dom0](/doc/install-nvidia-driver/) * [Nvidia troubleshooting guide](/doc/nvidia-troubleshooting/) diff --git a/user/troubleshooting/disk-troubleshooting.md b/user/troubleshooting/disk-troubleshooting.md new file mode 100644 index 00000000..d281bdf1 --- /dev/null +++ b/user/troubleshooting/disk-troubleshooting.md @@ -0,0 +1,96 @@ +--- +layout: doc +title: Disk Troubleshooting +permalink: /doc/disk-troubleshooting/ +redirect_from: +- /en/doc/out-of-memory/ +- /doc/OutOfmemory/ +- /wiki/OutOfmemory/ +- /doc/out-of-memory/ +--- + +# Disk Troubleshooting Guide # + +## "Out of disk space" error ## + +VMs (especially templates) use pre-allocated space. +The default private storage max size is 2 GB, but it's very easy to increase as needed. +If the disk is completely full, you will get an `Out of disk space` error that may crash your system because Dom0 does not have enough disk space to work. +So it's good practice to regularly check disk space usage with the command `df -h` in dom0 terminal. + +A system that's out of space should be able to boot, but may be unable to load a desktop manager. +In this case it is possible to login to dom0 terminal with Alt + Ctrl + F2. +To recover disk space it may be possible to delete files in a userVM by connecting to the userVM terminal: + +~~~ +qvm-start +qvm-console-dispvm +~~~ + +If this does not work, check the size of /var/lib/qubes/qubes.xml. +If it is zero, you'll need to use one of the file backup (stored in /var/lib/qubes/backup), hopefully you have the current data there. +Find the most recent one and place in /var/lib/qubes/qubes.xml instead of the empty file. + +In any case you'll need some disk space to start the VM. Check `df -h` output if you have some. +If not, here are some hints how to free some disk space: + +1. Clean yum cache. + + ~~~ + sudo yum clean all + ~~~ + +2. Delete `.img` files of a less important VM, which can be found in `/var/lib/qubes/appvms/`. + Then, when the system is working again, clean up the rest. + + ~~~ + qvm-remove + ~~~ + + With this method, you lose the data of one VM, but it'll work more reliably. + +3. Decrease the filesystem safety margin (5% by default). + + ~~~ + sudo tune2fs -m 4 /dev/mapper/vg_dom0-lv_root + ~~~ + +4. Remove some unneeded files in dom0 home (if you have any, most likely not). + +## Can't resize VM storage / "resize2fs: Permission denied" error ## + +[Resizing a volume](/doc/resize-disk-image/) in the Qubes interface should be a straightforward process. +But sometimes, an attempt to resize will look like it worked, when it in fact fails silently. +If you then try the same operation in the dom0 console using the `qvm-volume extend` command, it fails with the error message: `resize2fs: Permission denied to resize filesystem`. +This error indicates that a `resize2fs` will not work, unless `fsck` is run first. +Qubes OS utilities cannot yet handle this case. + +To fix this issue: + +1. In the dom0 terminal get a root console on the vm (eg. sys-usb) with: + + ~~~ + sudo xl console -t pv sys-usb + ~~~ + +2. Unmount everything mounted on the private volume `/dev/xvdb partition`. +There are typically several mounts listed in `/etc/mtab`. + +3. When you attempt to unmount the `/home` directory using the `umount /home` command, you will encounter an error because there are processes using the `/home` directory. You can view a list of these processes with the `fuser` command: + + ~~~ + fuser -m /home + ~~~ + +Kill these process until they are all gone using `kill `. + +4. Finally, run: + + ~~~ + umount /home + fsck /dev/xvdb + resize2fs /dev/xvdb + ~~~ + +After restarting your VM, everything should now work as expected. +The private volume size shown externally in the VM's settings interface is the same as that seen within the VM. diff --git a/user/troubleshooting/out-of-memory.md b/user/troubleshooting/out-of-memory.md deleted file mode 100644 index 869da993..00000000 --- a/user/troubleshooting/out-of-memory.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -layout: doc -title: Out of Memory -permalink: /doc/out-of-memory/ -redirect_from: -- /en/doc/out-of-memory/ -- /doc/OutOfmemory/ -- /wiki/OutOfmemory/ ---- - -VMs (especially templates) use pre-allocated space. The default private storage max size is 2 GB, but it's very easy to increase as needed. If the disk is completely full, you will get an `Out of disk space` error that may crash your system because Dom0 does not have enough disk space to work. So it's good practice to regularly check disk space usage with the command `df -h` in dom0 terminal. - -A system that's out of space should be able to boot, but may be unable to load a desktop manager. In this case it is possible to login to dom0 terminal with Alt + Ctrl + F2. To recover disk space it may be possible to delete files in a userVM by connecting to the userVM terminal: - -~~~ -qvm-start -qvm-console-dispvm -~~~ - -If this does not work, check the size of /var/lib/qubes/qubes.xml. If it is zero, you'll need to use one of the file backup (stored in /var/lib/qubes/backup), hopefully you have the current data there. Find the most recent one and place in /var/lib/qubes/qubes.xml instead of the empty file. - -In any case you'll need some disk space to start the VM. Check `df -h` output if you have some. If not, here are some hints how to free some disk space: - -1. Clean yum cache. - - ~~~ - sudo yum clean all - ~~~ - -2. Delete `.img` files of a less important VM, which can be found in `/var/lib/qubes/appvms/`. - Then, when the system is working again, clean up the rest. - - ~~~ - qvm-remove - ~~~ - - With this method, you lose the data of one VM, but it'll work more reliably. - -3. Decrease the filesystem safety margin (5% by default). - - ~~~ - sudo tune2fs -m 4 /dev/mapper/vg_dom0-lv_root - ~~~ - -4. Remove some unneeded files in dom0 home (if you have any, most likely not). - From 7d365f1083d016d8e4b9e957a0566047adf5366a Mon Sep 17 00:00:00 2001 From: Enjeck Cleopatra <32180937+PROTechThor@users.noreply.github.com> Date: Mon, 19 Oct 2020 17:42:33 +0100 Subject: [PATCH 2/5] Update disk-troubleshooting.md --- user/troubleshooting/disk-troubleshooting.md | 70 +++++++++++++++++--- 1 file changed, 60 insertions(+), 10 deletions(-) diff --git a/user/troubleshooting/disk-troubleshooting.md b/user/troubleshooting/disk-troubleshooting.md index d281bdf1..946b4397 100644 --- a/user/troubleshooting/disk-troubleshooting.md +++ b/user/troubleshooting/disk-troubleshooting.md @@ -16,17 +16,24 @@ redirect_from: VMs (especially templates) use pre-allocated space. The default private storage max size is 2 GB, but it's very easy to increase as needed. If the disk is completely full, you will get an `Out of disk space` error that may crash your system because Dom0 does not have enough disk space to work. -So it's good practice to regularly check disk space usage with the command `df -h` in dom0 terminal. - -A system that's out of space should be able to boot, but may be unable to load a desktop manager. -In this case it is possible to login to dom0 terminal with Alt + Ctrl + F2. -To recover disk space it may be possible to delete files in a userVM by connecting to the userVM terminal: +So it's good practice to regularly check disk space usage. +Running the `df -h` command in dom0 terminal will show some information, but not include all the relevant details. +The Qubes user interface provides a disk space widget. +If you are unable to access the interface, the command line version is running `sudo lvs | head` and looking at top entry for LVM pool. +For example: ~~~ -qvm-start -qvm-console-dispvm + LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert + pool00 qubes_dom0 twi-aotz-- 453.17g 89.95 69.78 + root qubes_dom0 Vwi-aotz-- 453.17g pool00 5.87 + swap qubes_dom0 -wi-ao---- 7.57g ~~~ +If you run `df -h`, it only shows the information in the `root` line (which is already included in the `pool00` line). +As you can see, the `sudo lvs | head` command includes additional important columns `Data%` and `Meta%`, shown in the above example to have the values 89% and 69% respectively. + +If your system is able to boot, but cannot load a desktop environment, it is possible to login to dom0 terminal with Alt + Ctrl + F2. + If this does not work, check the size of /var/lib/qubes/qubes.xml. If it is zero, you'll need to use one of the file backup (stored in /var/lib/qubes/backup), hopefully you have the current data there. Find the most recent one and place in /var/lib/qubes/qubes.xml instead of the empty file. @@ -37,7 +44,7 @@ If not, here are some hints how to free some disk space: 1. Clean yum cache. ~~~ - sudo yum clean all + sudo dnf clean all ~~~ 2. Delete `.img` files of a less important VM, which can be found in `/var/lib/qubes/appvms/`. @@ -55,7 +62,50 @@ If not, here are some hints how to free some disk space: sudo tune2fs -m 4 /dev/mapper/vg_dom0-lv_root ~~~ -4. Remove some unneeded files in dom0 home (if you have any, most likely not). +4. Remove some unneeded files in dom0 home (if you have any, most likely not). Also look for unneeded files in `/var/log` in dom0, and `/var/log/qubes`. + +The above steps applies to old VM disks format. These steps may work on Qubes 4.0, but are not default anymore. By default, Qubes 4.0 now uses LVM. The equivalent steps are: + +1. Get a list of VM disks using `sudo lvs`. + +2. Use `sudo lvremove qubes_dom0/` to remove backup copies of some less important VMs -- entries with `-back` in their name. + +3. If that isn't enough, remove actual disks of less important VMs. NOTE: You will lose the data of that VM, but your system will resume working. + +For example: + +~~~ +$ sudo lvs + LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert + pool00 qubes_dom0 twi-aotz-- 453.17g 89.95 69.78 + root qubes_dom0 Vwi-aotz-- 453.17g pool00 5.87 + swap qubes_dom0 -wi-ao---- 7.57g +(...) + vm-d10test-private qubes_dom0 Vwi-a-tz-- 2.00g pool00 vm-d10test-private-1600961860-back 29.27 + vm-d10test-private-1600961860-back qubes_dom0 Vwi-a-tz-- 2.00g pool00 4.87 + vm-d10test-standalone-private qubes_dom0 Vwi-a-tz-- 2.00g pool00 vm-d10test-standalone-private-1580772439-back 4.90 + vm-d10test-standalone-private-1580772439-back qubes_dom0 Vwi-a-tz-- 2.00g pool00 4.87 + vm-d10test-standalone-root qubes_dom0 Vwi-a-tz-- 10.00g pool00 vm-d10test-standalone-root-1580772439-back 43.37 + vm-d10test-standalone-root-1580772439-back qubes_dom0 Vwi-a-tz-- 10.00g pool00 42.05 + vm-debian-10-my-private qubes_dom0 Vwi-a-tz-- 2.00g pool00 4.96 + vm-debian-10-my-root qubes_dom0 Vwi-a-tz-- 10.00g pool00 vm-debian-10-my-root-1565013689-back 57.99 + vm-debian-10-my-root-1565013689-back qubes_dom0 Vwi-a-tz-- 10.00g pool00 56.55 + vm-debian-10-private qubes_dom0 Vwi-a-tz-- 2.00g pool00 4.94 + vm-debian-10-root qubes_dom0 Vwi-a-tz-- 10.00g pool00 vm-debian-10-root-1601126126-back 93.44 + vm-debian-10-root-1601126126-back qubes_dom0 Vwi-a-tz-- 10.00g pool00 88.75 +(...) +$ sudo lvremove qubes_dom0/vm-d10test-standalone-root-1580772439-back +Do you really want to remove and DISCARD active logical volume qubes_dom0/vm-d10test-standalone-root-1580772439-back? [y/n]: y + Logical volume "vm-d10test-standalone-root-1580772439-back" successfully removed +~~~ + +After freeing some intial space, it may be possible to recover more space by deleting files in a userVM after connecting to the userVM terminal: + +qvm-start +qvm-console-dispvm +~~~ + +Since `qvm-console-dispvm` requires working graphical user interface login, you must first free enough space to be able to start a VM and login to graphical UI. ## Can't resize VM storage / "resize2fs: Permission denied" error ## @@ -70,7 +120,7 @@ To fix this issue: 1. In the dom0 terminal get a root console on the vm (eg. sys-usb) with: ~~~ - sudo xl console -t pv sys-usb + qvm-console-dispvm sys-usb ~~~ 2. Unmount everything mounted on the private volume `/dev/xvdb partition`. From 6f217dccbdc8c8e4071a57ed53e741e2e8c16ac3 Mon Sep 17 00:00:00 2001 From: Enjeck Cleopatra <32180937+PROTechThor@users.noreply.github.com> Date: Mon, 19 Oct 2020 17:44:54 +0100 Subject: [PATCH 3/5] Add ~~~ --- user/troubleshooting/disk-troubleshooting.md | 1 + 1 file changed, 1 insertion(+) diff --git a/user/troubleshooting/disk-troubleshooting.md b/user/troubleshooting/disk-troubleshooting.md index 946b4397..8b02c5f7 100644 --- a/user/troubleshooting/disk-troubleshooting.md +++ b/user/troubleshooting/disk-troubleshooting.md @@ -101,6 +101,7 @@ Do you really want to remove and DISCARD active logical volume qubes_dom0/vm-d10 After freeing some intial space, it may be possible to recover more space by deleting files in a userVM after connecting to the userVM terminal: +~~~ qvm-start qvm-console-dispvm ~~~ From f66797d2b63927946043eac025512e7be02d3efc Mon Sep 17 00:00:00 2001 From: Enjeck Cleopatra <32180937+PROTechThor@users.noreply.github.com> Date: Wed, 21 Oct 2020 04:51:21 +0100 Subject: [PATCH 4/5] Remove sentences --- user/troubleshooting/disk-troubleshooting.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/user/troubleshooting/disk-troubleshooting.md b/user/troubleshooting/disk-troubleshooting.md index 8b02c5f7..6b3788b8 100644 --- a/user/troubleshooting/disk-troubleshooting.md +++ b/user/troubleshooting/disk-troubleshooting.md @@ -13,8 +13,6 @@ redirect_from: ## "Out of disk space" error ## -VMs (especially templates) use pre-allocated space. -The default private storage max size is 2 GB, but it's very easy to increase as needed. If the disk is completely full, you will get an `Out of disk space` error that may crash your system because Dom0 does not have enough disk space to work. So it's good practice to regularly check disk space usage. Running the `df -h` command in dom0 terminal will show some information, but not include all the relevant details. From 9de3cd2ce4c887626be5719db2c6957515e633a0 Mon Sep 17 00:00:00 2001 From: Andrew David Wong Date: Sat, 24 Oct 2020 04:24:30 -0700 Subject: [PATCH 5/5] Fix typo (#1059) --- user/troubleshooting/disk-troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/user/troubleshooting/disk-troubleshooting.md b/user/troubleshooting/disk-troubleshooting.md index 6b3788b8..c7891dd3 100644 --- a/user/troubleshooting/disk-troubleshooting.md +++ b/user/troubleshooting/disk-troubleshooting.md @@ -97,7 +97,7 @@ Do you really want to remove and DISCARD active logical volume qubes_dom0/vm-d10 Logical volume "vm-d10test-standalone-root-1580772439-back" successfully removed ~~~ -After freeing some intial space, it may be possible to recover more space by deleting files in a userVM after connecting to the userVM terminal: +After freeing some initial space, it may be possible to recover more space by deleting files in a userVM after connecting to the userVM terminal: ~~~ qvm-start