Qubes-Community-Content/docs/configuration/split-ssh.md

Qubes Split SSH

Split SSH implements a concept similar to having a smart card with your private SSH keys, except that the role of the “smart card” is played by another Qubes AppVM. This Qubes setup allows you to keep your SSH private keys in a vault VM (vault) while using an SSH Client VM (ssh-client) to access your remote server. This is done by using Qubes's qrexec framework to connect a local SSH Agent socket from your SSH Client VM to the SSH Agent socket within the vault VM. This way the compromise of the domain you use to connect to your remote server does not allow the attacker to automatically also steal all your keys. (We should make a rather obvious comment here that the so-often-used passphrases on private keys are pretty meaningless because the attacker can easily set up a simple backdoor which would wait until the user enters the passphrase and steal the key then.)

Overview

1. Make sure the TemplateVM you plan to use is up to date and nmap and ncat is installed.
2. Create vault and ssh-client AppVMs.
3. Create an ssh key in your vault AppVM and set up automatic key adding prompt.
4. Set up VM interconnection
5. (Strongly Encouraged) Create a KeePassXC Database and set up SSH Agent Integration in KeePassXC.

Prepare Your System

0. (Optional) Take a system snapshot before you start tuning your system or do any major installations. To perform a Qubes OS backup please read and follow this guide in the User Documentation.

1. Make sure the TemplateVMs that you plan to base your AppVMs on are up to date.

   For Fedora templates:
   

   [user@fedora-32 ~]$ sudo dnf update && sudo dnf upgrade -y
   

   For Debian templates:
   

   user@debian-10:~$ sudo apt-get update && sudo apt-get upgrade
   

2. Make sure nmap and ncat is installed in the TemplateVMs you plan to base your AppVMs on.

   For Fedora templates:
   

   [user@fedora-32 ~]$ sudo dnf install nmap-ncat
   

   For Debian templates:
   

   user@debian-10:~$ sudo apt-get install nmap ncat
   

3. If you don't plan to use KeePassXC, install ssh-askpass in the TemplateVM you plan to base your vault VM on.

   For Fedora templates:
   

   [user@fedora-32 ~]$ sudo dnf install openssh-askpass
   

   For Debian templates:
   

   user@debian-10:~$ sudo apt-get install ssh-askpass
   

Creating AppVMs

If you’ve installed Qubes OS using the default options, a few qubes including a vault AppVM has been created for you. Skip the first step if you don't wish to create another vault.

1. Create a new vault AppVM (vault) based on your chosen template. Set networking to (none).

   

2. Create a SSH Client AppVM (ssh-client). This VM will be used to make SSH connections to your remote machine.

   

Setting up SSH

Perform the next steps in a vault VM terminal.

1. Generate an SSH key pair. Skip this step if you already have your keys.

   [user@vault ~]$ ssh-keygen -t ed25519 -a 500
   Generating public/private ed25519 key pair.
   Enter file in which to save the key (/home/user/.ssh/id_ed25519): 
   Created directory '/home/user/.ssh'.
   Enter passphrase (empty for no passphrase): 
   Enter same passphrase again: 
   Your identification has been saved in /home/user/.ssh/id_ed25519
   Your public key has been saved in /home/user/.ssh/id_ed25519.pub
   The key fingerprint is:
   SHA256:DBxSxZcp16d1NSVSid3m8HRipUDM2INghQ4Sx3jPEDo user@vault
   The key's randomart image is:
   +--[ED25519 256]--+
   |    o==+++.@++o=*|
   |    o==o+ B BoOoB|
   |    Eoo* +   *.O.|
   |     . o+   .   o|
   |        S        |
   |                 |
   |                 |
   |                 |
   |                 |
   +----[SHA256]-----+
   

   -t: type
   -a: num_trials
   

   Please note that the key fingerprint and the randomart image will differ.

   For more information about ssh-keygen, run man ssh-keygen.

Notice: Skip the following steps if you plan on using KeePassXC.

2. Make a new directory ~/.config/autostart

   [user@fedora-32 ~]$ mkdir -p ~/.config/autostart
   

3. Create the file ssh-add.desktop in ~/.config/autostart

   • Open the file with e.g. nano

     [user@fedora-32 ~]$ nano ~/.config/autostart/ssh-add.desktop
     

   • Paste the following contents:

     [Desktop Entry]
     Name=ssh-add
     Exec=ssh-add
     Type=Application
     

   Note: If you've specified a custom name for your key using -f, you should adjust Exec=ssh-add to Exec=ssh-add <path-to-your-key-file>.

   • Save and exit.

With this configuration you'll be prompted for a password the first time you start your vault VM to be able to make use of your SSH key.

Setting Up VM Interconnection

In the TemplateVM to your vault VM:

1. Create the file qubes.SshAgent in /etc/qubes-rpc

   • Open the file with e.g. nano

     [user@fedora-32 ~]$ sudo nano /etc/qubes-rpc/qubes.SshAgent
     

   • Paste the following contents:

     #!/bin/sh
     # Qubes App Split SSH Script
     
     # safeguard - Qubes notification bubble for each ssh request
     notify-send "[`qubesdb-read /name`] SSH agent access from: $QREXEC_REMOTE_DOMAIN"
     
     # SSH connection
     ncat -U $SSH_AUTH_SOCK
     

   • Save and exit.

2. Shutdown the template VM.

In dom0:

1. Create the file qubes.SshAgent in /etc/qubes-rpc

   • Open the file with your editor of choice (e.g. nano).

     [user@fedora-32 ~]$ sudo nano /etc/qubes-rpc/qubes.SshAgent
     

   • If you want to explicitly allow only this connection, add the following line:

     ssh-client vault ask
     

   • If you want to allow all VMs to connect, add the following line:

     @anyvm @anyvm ask
     

   • If you want the input field to be "prefilled" by your vault VM, append default_target=vault so it looks like for example:

     @anyvm @anyvm ask,default_target=vault
     

   • Save and exit.

   Note: There are many ways to fine-tune this policy. For more details see the Qubes qrexec documentation.

2. Close the terminal. Do not shutdown dom0.

In an SSH Client AppVM terminal

Theoretically, you can use any AppVM but to increase security it is advised to create a dedicated AppVM for your SSH connections. Furthermore, you can set different firewall rules for each VM (i.e. for intranet and internet connections) which also provides additional protection.

1. Edit /rw/config/rc.local

   • Open the file with your editor of choice (e.g. nano).

     [user@ssh-client ~]$ sudo nano /rw/config/rc.local
     

   • Add the following to the bottom of the file:

     # SPLIT SSH CONFIGURATION >>>
     # replace "vault" with your AppVM name which stores the ssh private key(s)
     SSH_VAULT_VM="vault"
     
     if [ "$SSH_VAULT_VM" != "" ]; then
       export SSH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
       rm -f "$SSH_SOCK"
       sudo -u user /bin/sh -c "umask 177 && ncat -k -l -U '$SSH_SOCK' -c 'qrexec-client-vm $SSH_VAULT_VM qubes.SshAgent' &"
     fi
     # <<< SPLIT SSH CONFIGURATION
     

   • Save and exit.

2. Edit ~/.bashrc

   • Open the file with your editor of choice (e.g. nano).

     [user@ssh-client ~]$ nano ~/.bashrc
     

   • Add the following to the bottom of the file:

     # SPLIT SSH CONFIGURATION >>>
     # replace "vault" with your AppVM name which stores the ssh private key(s)
     SSH_VAULT_VM="vault"
     
     if [ "$SSH_VAULT_VM" != "" ]; then
         export SSH_AUTH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
     fi
     # <<< SPLIT SSH CONFIGURATION
     

   • Save and exit.

Using KeePassXC

Warning: This part is for setting up KeePassXC, not KeePassX or KeePass. See the KeePassXC FAQ.

0. KeePassXC should be installed by default in both Fedora and Debian TemplateVMs. If this changes in the future and you find that it isn't, it can be installed with:

   For Fedora templates:
   

   [user@fedora-32 ~]$ sudo dnf install keepassxc
   

   For Debian templates:
   

   user@vault-deb:~$ sudo apt-get install keepassxc
   

   If you have another template check the KeePassXC download page for instructions.

1. Add KeepasXC to the Applications menu of the newly created AppVM for ease of access.

   

Note: Since the vault VM has no internet connection, you can safely deny automatic updates.

2. Create a new database.

   

3. Enter a name for your database and continue.

   

4. Adjust the encryption settings. Check the KeePassXC User Guide for more information about these settings.

   

5. Enter a password for your database. Take your time make a secure but also rememberable password. (hint)

   

6. Add a new entry.

   

7. Set password to your SSH key passphrase.

   

8. Go into the Advanced section and add your keys.

   

   Note: You only need to add the private key (id_25519 here) but if you want to be able to simply back up both your private and public key (id_25519.pub) by backing up your KeePassXC database (*.kdbx file) you can add that too.

9. Enable "SSH Agent Integration" within the Application Settings.

   

10. Restart KeePassXC

11. Check the SSH Agent Integration status.

12. Open the entry you created and select your private key in the "SSH Agent" section.

Testing the KeePassXC Setup

1. Close your KeePassXC database and run ssh-add -L. It should return The agent has no identities.

   [user@vault ~]$ ssh-add -L
   The agent has no identities.
   

2. Unlock your KeePassXC database and run ssh-add -L again. This time it should return your public key.

   [user@vault ~]$ ssh-add -L
   ssh-ed25519 <public key string> user@vault-keepassxc
   

Test Your Configuration

1. If you're using KeePassXC, shutdown KeePassXC on your vault VM. If not, make sure your private key is not added to the ssh-agent in your vault VM (Check with ssh-add -L). If it is, restart your vault VM and do not enter your password when it asks you to.

2. Try fetching your identities on the SSH Client VM.

   [user@ssh-client ~]$ ssh-add -L
   

3. Allow operation execution

   

It should return The agent has no identities..

If you're getting an error (e.g. error fetching identities: communication with agent failed), make sure your vault VM is running and check your VM interconnection setup.

4. Launch KeePassXC and unlock your database.

5. Try fetching your identities on the SSH Client VM.

   [user@ssh-client ~]$ ssh-add -L
   

6. Allow operation execution

   

Check if it returns ssh-ed25519 <public key string>

If you're getting an error (e.g. error fetching identities: communication with agent failed), make sure your vault VM is running and check your VM interconnection setup.

(Optional) Backing Up the Configuration

• Start a system backup as per the User Documentation.
• Back up your *.kdbx file to a somewhere you know to be secure. (e.g. a secure USB device, an end-to-end-encrypted email box. (e.g. Tutanota, ProtonMail))

Security Benefits

In the setup described in this guide, even an attacker who manages to gain access to the ssh-client VM will not be able to obtain the user’s private key since it is simply not there. Rather, the private key remains in the vault VM, which is extremely unlikely to be compromised if nothing is ever copied or transferred into it. In order to gain access to the vault VM, the attacker would require the use of, e.g., a general Xen VM escape exploit or a signed, compromised package which is already installed in the TemplateVM upon which the vault VM is based.

Further Security tips

Regarding Your SSH Private Key

• This goes without saying: keep your private keys private.
• Tinkering with the user permissions is not necessary since it is assumed that an adversary who can find a Xen VM escape exploit is also capable of finding a user to root escalation exploit.

Regarding Your KeePassXC Database File

Although the database file is encrpyted with your password, if you haven't taken any protective measures, it can be bruteforced. Some tips for securing your keys against a vault VM compromise include:

• Hide the *.kdbx file by simply renaming the file extension (e.g. *.zip). Keep in mind this is not likely to stop dedicated adversaries from finding your *.kdbx file.
• Add a second encryption layer (e.g. with VeraCrypt, *.7z with password).
• Adjust the encrpytion settings in KeePassXC as per the KeePassXC documentation.

Current limitations

• It is possible for a malicious VM to hold onto an ssh-agent connection for more than one use. Therefore, if you authorize usage once, assume that a malicious VM could then use it many more times. In this case, though, the SSH Agent should continue to protect your private keys; only usage of it would be available to the malicious VM until it was shut down.

• It doesn’t solve the problem of allowing the user to know what is requested before the operation gets approved.

Want more Qubes split magic? Check out Split-GPG.

---

This guide has been inspired by:
Qubes Split SSH (Github: Jason Hennessey - henn) https://github.com/henn/qubes-app-split-ssh
Using split ssh in QubesOS 4.0 (Kushal Das) https://kushaldas.in/posts/using-split-ssh-in-qubesos-4-0.html
Using Split-SSH in Qubes 4 (Denis Zanin) https://deniszanin.com/using-split-ssh-gpg-in-qubes-os/
R.I.S.K.S. https://19hundreds.github.io/risks-workflow/ssh-split-setup

Contributor(s): phl, @invalid-error, @deeplow, @santorihelix