qubes-doc/user/troubleshooting/salt-troubleshooting.md

lang	layout	permalink	redirect_from	ref	title
en	doc	/doc/salt-troubleshooting/	/doc/salt/ /en/doc/salt/	1000000000000	Salt troubleshooting

For ease of Qubes Os managament and reproductible deployment, Salt allows to control states on dom0 and other vms from the dom0.

Behind the scenes

Except for dom0 where the host is controlled locally. Each vm (named minion-vm for instance) is controlled by a disposable master vm based on disposable-mgmt-vm, named disp-mgmt-minion-vm and created only for the duration of qubesctl execution.

The required files are copied from dom0 to disp-mgmt-minion-vm via qubes.Filecopy, then qubes.SaltLinuxVM and expect two lines on stdin :

minion-vm
salt-command

Usually salt-command is state.apply with the provided arguments like test=True.

Then, a fake ssh command wrapper included in qubes-mgmt-salt-vm-connector allow to run the command on the target (minion-vm) via qubes.VMShell or qubes.VMRootShell. On the management vm disp-mgmt-minion-vm, salt firstly creates /var/cache/salt/master/thin/thin.tgz and transfers it to the minion-vm to ensure destination host has the required python files.

How to debug the ephemeral disposable management vm

First, the transfered content from dom0 to the disposable management vm needs to be retrieved. To do so, it is suggested to:

1. Call from dom0, qubesctl with requested command like qubesctl --show-output --targets minion-vm --skip-dom0 state.apply,
2. Freeze the previous command with Ctrl+Z as soon as you see minion-vm is starting.
3. Get the console on the disposable management vm with qvm-console-dispvm disp-mgmt-minion-vm on the dom0
4. Type root to log as root on the console
5. Edit in disp-mgmt-minion-vm, /etc/qubes-rpc/qubes.SaltLinuxVM and add after the line eval "dir=~$user/QubesIncoming/dom0/srv", the line qvm-copy $dir.
6. On the dom0 resume the freezed process with fg
7. Copy the content to another qubes vm (side-vm for instance)

Second, a debugable disposable management vm is setup. To do so, it is suggested to:

1. Call from dom0, qubesctl with requested command like qubesctl --show-output --targets minion-vm --skip-dom0 state.apply,
2. Freeze the previous command with Ctrl+Z as soon as you see minion-vm is starting.
3. Copy the retrieved content from side-vm to the disposable management vm disp-mgmt-minion-vm (with qvm-copy)
4. Get the console on the disposable management vm with qvm-console-dispvm disp-mgmt-minion-vm on the dom0
5. Type root to log as root on the console. All following commands are done inside the console.
6. Move the copied content to emulate a content coming from dom0: cd /home/user/QubesIncoming; mv * dom0. dom0 directory should contain a directory srv.
7. Emulate a call to qubes.SaltLinuxVM with bash /etc/qubes-rpc/qubes.SaltLinuxVM
8. Emulate stdin. Type the destination vm on the first line (minion-vm), the salt command on the second line (state.apply for instance) then Ctrl+D.
9. A first execution is launched
10. Get wrappers in the PATH with export PATH="/usr/lib/qubes-vm/connector/ssh-wrapper:$PATH" (the line is available in /etc/qubes-rpc/qubes.SaltLinuxVM`

Third, launch as many times as needed the following command to emulate a new call of master to the minion rm -r /var/cache/salt /var/tmp/.root*; salt-ssh -w minion-vm salt-command in the console.