feat: port forwarder

If persistent rules are chosen, it can deal with disposable sys-net, but
not with disposable sys-firewall, as the qube ip will change, the rule
won't work. Applying the rule to the disposable template is a "try it
all", but it's usage is discouraged.

README.md

@@ -7,13 +7,6 @@ Salt Formulas for Qubes OS.
 **Warning**: Not ready for production, development only. Breaking changes can
 and will be introduced in the meantime. You've been warned.
 
-The following projects are unfinished (not a complete list):
-
-- sys-syncthing: broken firewall script due to nftables and disposable netvm
-
-Other projects might also have drastic changes, the above are simply not ready
-at all.
-
 ## Table of Contents
 
 * [Description](#description)

salt/sys-syncthing/README.md

@@ -29,8 +29,8 @@ qubesctl top.enable sys-syncthing browser
 qubesctl --targets=tpl-browser,sys-syncthing-browser,tpl-sys-syncthing,sys-syncthing state.apply
 qubesctl top.disable sys-syncthing browser
 qubesctl state.apply sys-syncthing.appmenus
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh -a -p add sys-syncthing tcp 22000
+qvm-port-forward -a add -q sys-syncthing -n tcp -p 22000
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh -a -p add sys-syncthing udp 22000
+qvm-port-forward -a add -q sys-syncthing -n udp -p 22000
 ```
 
 - State:
@@ -42,8 +42,8 @@ qubesctl --skip-dom0 --targets=tpl-sys-syncthing state.apply sys-syncthing.insta
 qubesctl --skip-dom0 --targets=sys-syncthing state.apply sys-syncthing.configure
 qubesctl --skip-dom0 --targets=sys-syncthing-browser state.apply sys-syncthing.configure-browser
 qubesctl state.apply sys-syncthing.appmenus
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh -a -p add sys-syncthing tcp 22000
+qvm-port-forward -a add -q sys-syncthing -n tcp -p 22000
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh -a -p add sys-syncthing udp 22000
+qvm-port-forward -a add -q sys-syncthing -n udp -p 22000
 ```
 <!-- pkg:end:post-install -->
 
@@ -99,10 +99,10 @@ If sys-net has more than one network card the first external interface will
 be used by default.
 If this is incorrect, you must change it manually. In Dom0 run:
 ```sh
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh delete sys-syncthing tcp 22000 -a -p
+qvm-port-forward -a del -q sys-syncthing -n udp -p 22000
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh delete sys-syncthing udp 22000 -a -p
+qvm-port-forward -a del -q sys-syncthing -n tcp -p 22000
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh add sys-syncthing tcp 22000 -p
+qvm-port-forward -a add -q sys-syncthing -n udp -p 22000
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh add sys-syncthing udp 22000 -p
+qvm-port-forward -a add -q sys-syncthing -n tcp -p 22000
 ```
 This will let you choose the NIC.
 
@@ -117,8 +117,8 @@ Syncthing between qubes.
 Uninstallation procedure:
 <!-- pkg:begin:preun-uninstall -->
 ```sh
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh -a -p delete sys-syncthing tcp 22000
+qvm-port-forward -a del -q sys-syncthing -n tcp -p 22000
-/srv/salt/qusal/sys-syncthing/files/admin/firewall/in.sh -a -p delete sys-syncthing udp 22000
+qvm-port-forward -a del -q sys-syncthing -n udp -p 22000
 qubesctl --skip-dom0 --targets=sys-syncthing state.apply sys-syncthing.cancel
 qubesctl state.apply sys-syncthing.clean
 ```

salt/sys-syncthing/create.sls

@@ -97,3 +97,12 @@ features:
 
 {% from 'utils/macros/policy.sls' import policy_set with context -%}
 {{ policy_set(sls_path, '80') }}
+
+"{{ slsdotpath }}-qvm-port-forward":
+  file.managed:
+    - name: /usr/local/bin/qvm-port-forward
+    - source: salt://{{ slsdotpath }}/files/admin/firewall/qvm-port-forward
+    - user: root
+    - group: root
+    - mode: '0755'
+    - makedirs: True

salt/sys-syncthing/files/admin/firewall/in.sh

@@ -1,312 +0,0 @@
-#!/usr/bin/env bash
-
-# SPDX-FileCopyrightText: 2022 unman <unman@thirdeyesecurity.org>
-# SPDX-FileCopyrightText: 2023 Benjamin Grande M. S. <ben.grande.b@gmail.com>
-#
-# SPDX-License-Identifier: AGPL-3.0-or-later
-
-## Credits: https://github.com/unman/shaker/blob/main/i2p/in.sh
-## Recursively open ports through the firewall to allow remote access to a qube.
-
-## TODO: remove iptables in favor of nft. It doesn't work if the upstream net
-## qubes are disposables, instead, the rule should be applied on the
-## disposable template. This would work well if users used our project that
-## creates a template per service, but if user is using a default diposable
-## template for that, such as debian-XX-dvm, the firewall would allow many
-## qubes to be exposed.
-
-me="${0##*/}"
-rc="/rw/config/rc.local.d/50-port-forwarder.rc"
-
-usage(){
-cat <<HERE
-Usage: ${me} [-h|a|p] [add|delete] [target] [tcp|udp] [port number|service] [external port]
-
-Options:
- h              print this help
- a              auto mode, a port will be opened on the first external interface
- p              permanent rules, takes effect in each qube start up
- Action         add, delete
- Protocol       tcp, udp
- Target Port    port number or service name (e.g. ssh)
- External Port  port number or service name (e.g. ssh) (default: target port)
-
-Example:
-  ${me} OPTIONS ACTION TARGET_QUBE PROTOCOL TARGET_PORT EXTERNAL_PORT
-  ${me} add QUBE tcp 80 80
-  ${me} add QUBE tcp ssh ssh
-  ${me} delete QUBE tcp https https
-
-DO NOT use this script for qubes behind a Tor or VPN proxy.
-At a minimum you risk breaking the security of those proxies.
-HERE
-  exit 1
-}
-
-
-## Check input port
-check_port(){
-  if test "$2" != "$2";then
-    status=1
-  else
-    if test "$2" -lt 65536; then
-      status=0
-      portnum="$2"
-    else
-      status=1
-    fi
-  fi
-  if [ "$status" -ne 0 ]; then
-    if ! grep -q -w "^$2 " /etc/services; then
-      echo "Specify usable port number or service name"
-      exit 1
-    else
-      portnum="$(getent services "$2" | awk '{split($2,a,"/");print a[1]}')"
-      if test -z "$portnum"; then
-        echo "Specify usable port number or service name"
-        exit 1
-      fi
-    fi
-  fi
-  echo "$portnum"
-}
-
-
-get_handle(){
-  local my_handle
-  my_handle="$(qvm-run -q -u root -p "$1" -- "nft -a list table $2 | awk 'BEGIN{c=0} /$3/{c++; if (c==$4) print \$NF}'")"
-  echo "$my_handle"
-}
-
-
-## Tunnel through netvms
-tunnel(){
-  declare -a my_netvms=("${!1}")
-  declare -a my_ips=("${!2}")
-  declare -i numhops
-  numhops="${#my_ips[@]}"
-  lasthop=$((numhops-1))
-  local i=1
-  iface="eth0"
-  if qvm-run -q -u root "${my_netvms[$lasthop]}" " nft list table nat|grep ' $proto dport $portnum dnat to ${my_ips[$numhops-1]}'"
-  then
-    echo "Are rules already set?"
-    exit 1
-  fi
-  while test "$i" != "$numhops"; do
-    if test "$i" = "1"; then
-      portnum_used=$external_portnum
-      portnum_target=$portnum
-    else
-      portnum_used=$external_portnum
-      portnum_target=$external_portnum
-    fi
-    echo "${my_netvms[$i]} $portnum_used"
-    if [ $i -eq $lasthop ]; then
-      iface=$external_iface
-    fi
-    # Is it nft or iptables?
-    local found
-    found="$(qvm-run -p -q -u root "${my_netvms[$i]}" -- nft list table nat 2>/dev/null)"
-    if test -z "$found"; then
-      qvm-run -q -u root "${my_netvms[$i]}" -- "iptables -I QBS-FORWARD -i $iface -p $proto --dport $portnum_target -d ${my_ips[$i-1]} -j ACCEPT"
-      qvm-run -q -u root "${my_netvms[$i]}" -- "iptables -t nat -I PR-QBS-SERVICES -i $iface -p $proto --dport $portnum_used -j DNAT --to-destination ${my_ips[$i-1]}:$portnum_target"
-      if test "$permanent" = "1"; then
-        qvm-run -q -u root "${my_netvms[$i]}" -- "echo iptables -I QBS-FORWARD -i $iface -p $proto --dport $portnum_target -d ${my_ips[$i-1]} -j ACCEPT >> ${rc}"
-        qvm-run -q -u root "${my_netvms[$i]}" -- "echo iptables -t nat -I PR-QBS-SERVICES -i $iface -p $proto --dport $portnum_used -j DNAT --to-destination ${my_ips[$i-1]}:$portnum_target >> ${rc}"
-      fi
-    else
-      qvm-run -q -u root "${my_netvms[$i]}" -- nft insert rule nat PR-QBS-SERVICES meta iifname "$iface" "$proto" dport "$portnum_used" dnat to "${my_ips[$i-1]}:$portnum_target"
-      qvm-run -q -u root "${my_netvms[$i]}" -- nft insert rule filter QBS-FORWARD meta iifname "$iface" ip daddr "${my_ips[$i-1]}" "$proto" dport "$portnum_target" ct state new accept
-      if test "$permanent" = "1"; then
-        qvm-run -q -u root "${my_netvms[$i]}" -- "echo nft insert rule nat PR-QBS-SERVICES meta iifname $iface $proto dport $portnum_used dnat to ${my_ips[$i-1]}:$portnum_target >> ${rc}"
-        qvm-run -q -u root "${my_netvms[$i]}" -- "echo nft insert rule filter QBS-FORWARD meta iifname $iface ip daddr ${my_ips[$i-1]} $proto dport $portnum_target ct state new accept >> ${rc}"
-      fi
-    fi
-    ((i++))
-  done
-}
-
-
-## Teardown from top netvm down
-teardown(){
-  declare -a my_netvms=("${!1}")
-  declare -a my_ips=("${!2}")
-  declare -i numhops
-  numhops=${#my_ips[@]}
-  numhops=$((numhops-1))
-  local i=$numhops
-  iface="eth0"
-  echo "Removing firewall rules"
-  while [ $i -gt 0 ]; do
-    if [ $i -eq 1 ]; then
-      portnum_used=$external_portnum
-      portnum_target=$portnum
-    else
-      portnum_used=$external_portnum
-      portnum_target=$external_portnum
-    fi
-    # Is it nft or iptables?
-    echo "${my_netvms[$i]}"
-    local found
-    found="$( qvm-run -p -q -u root "${my_netvms[$i]}" -- "nft list table nat 2>/dev/null" )"
-    if test -z "$found"; then
-      qvm-run -q -u root "${my_netvms[$i]}" -- "iptables -D QBS-FORWARD -i $iface -p $proto --dport $portnum_target -d ${my_ips[$i-1]} -j ACCEPT"
-      qvm-run -q -u root "${my_netvms[$i]}" -- "iptables -t nat -D PR-QBS-SERVICES -i $iface -p $proto --dport $external_portnum -j DNAT --to-destination ${my_ips[$i-1]}:$portnum_target"
-      if [ "$permanent" -eq 1 ]; then
-        qvm-run -q -u root "${my_netvms[$i]}" -- "sed -i '/iptables -D QBS-FORWARD -i $iface -p $proto --dport $portnum_target -d ${my_ips[$i-1]} -j ACCEPT/d' ${rc}"
-        qvm-run -q -u root "${my_netvms[$i]}" -- "sed -i '/iptables -t nat -D PR-QBS-SERVICES -i $iface -p $proto --dport $external_portnum -j DNAT --to-destination ${my_ips[$i-1]}:$portnum_target/d' ${rc}"
-      fi
-    else
-      local handle
-      handle="$( get_handle "${my_netvms[$i]}" nat "dport $external_portnum " 1 )"
-      qvm-run -q  -u root "${my_netvms[$i]}" -- "nft delete rule nat PR-QBS-SERVICES handle $handle"
-      local handle
-      handle="$( get_handle "${my_netvms[$i]}" filter "dport $external_portnum " 1 )"
-      qvm-run -q -u root "${my_netvms[$i]}" -- "nft delete rule filter QBS-FORWARD handle $handle"
-      if [ "$permanent" -eq 1 ]; then
-        qvm-run -q -u root "${my_netvms[$i]}" -- "sed -i '/nft insert rule nat PR-QBS-SERVICES meta iifname $iface $proto dport $portnum_used dnat to ${my_ips[$i-1]}:$portnum_target/d' ${rc}"
-        qvm-run -q -u root "${my_netvms[$i]}" -- "sed -i '/nft insert rule filter QBS-FORWARD meta iifname $iface ip daddr ${my_ips[$i-1]} $proto dport $portnum_target ct state new accept/d' ${rc}"
-      fi
-    fi
-    ((i--))
-  done
-  local found
-  found="$( qvm-run -p -q -u root "${my_netvms[$i]}" -- nft list table nat 2>/dev/null )"
-  if test -z "$found"; then
-    qvm-run -q -u root "${my_netvms[$i]}" " iptables -D INPUT -p $proto --dport $external_portnum -j ACCEPT"
-  else
-    handle=$( get_handle "${my_netvms[$i]}" filter "dport $portnum " 1 )
-    qvm-run -q -u root "${my_netvms[$i]}" -- nft delete rule filter INPUT handle "$handle"
-  fi
-  exit
-}
-
-
-list(){
-  return
-}
-
-
-## Defaults
-auto=0
-permanent=0
-
-## Get options
-optstring=":hap"
-while getopts ${optstring} option ; do
-  case $option in
-    h) usage;;
-    a) auto=1;;
-    p) permanent=1;;
-    ?) usage;;
-  esac
-done
-shift $((OPTIND -1))
-
-## Check inputs
-test "$#" -lt 4 && usage
-if ! qvm-check -q "$2" 2>/dev/null; then
-  echo "$2 is not the name of any qube"
-  exit 1
-fi
-qube_name="$2"
-if test "$3" != "tcp" && test "$3" != "udp"; then
-  echo "Specify tcp or udp"
-  exit
-fi
-proto="$3"
-portnum="$(check_port "$3" "$4")"
-
-if [ $# -eq 5 ]; then
-  external_portnum="$(check_port "$3" "$5")"
-else
-  external_portnum=$portnum
-fi
-
-## Get all netvms
-declare -a netvms
-declare -a ips
-declare -a external_ips
-hop=0
-# shellcheck disable=SC2004
-netvms[${hop}]="$qube_name"
-IFS='|' read -r netvms[$hop+1] ips[$hop] <<< "$(qvm-ls "$qube_name" --raw-data -O netvm,IP)"
-while [ "${netvms[hop+1]}" != "-" ]
-do
-  ((hop++))
-  IFS='|' read -r netvms[$hop+1] ips[$hop] <<< "$(qvm-ls "${netvms[$hop]}" --raw-data -O netvm,IP)"
-done
-
-if test "$1" = "delete"; then
-  teardown netvms[@] ips[@]
-elif test "$1" = "add"; then
-  if [ "$hop" -eq 0 ]; then
-    echo "$qube_name is not network connected"
-    echo "Cannot set up a tunnel"
-    exit
-  fi
-
-  # Check last hop has external IP address
-  readarray -t external_ips < <( qvm-run -p "${netvms[$hop]}" "ip -4 -o a|grep -wv 'lo\|vif[0-9]*.*'"|awk '{print $2,$4}')
-  #readarray -t external_ips < <( qvm-run -p ${netvms[$hop]} "ip -4 -o a|grep -wv 'vif[0-9]'"|awk '{print $2,$4}')
-  num_ifs=${#external_ips[@]}
-  if [ "$num_ifs" -eq 1 ]; then
-    interface=0
-  elif [ $auto -eq 1 ]; then
-    interface=0
-  elif [ "$num_ifs" -gt 1 ]; then
-    echo "${netvms[$hop]} has more than 1 external interface"
-    echo "Which one do you want to use?"
-    for i in $(seq "$num_ifs"); do
-      echo "$i. ${external_ips[$i-1]}"
-    done
-    read -r interface
-    if ! [ "$interface" -eq "$interface" ] 2> /dev/null; then
-      echo "No such interface"
-      exit
-    elif [ "$interface" -gt "$num_ifs" ] || [ "$interface" -lt 1 ]; then
-      echo "No such interface"
-      exit
-    fi
-    ((interface--))
-  else
-    echo "${netvms[$hop]} does not have an external interface"
-    echo "Cannot set up a tunnel"
-    exit
-  fi
-  external_ip="${external_ips[$interface]}"
-  external_iface="${external_ip%[[:space:]]*}"
-  ip="${external_ip#*[0-9]}"
-  ip="${ip%%/*}"
-  # shellcheck disable=SC2004,SC2034
-  ips[$hop]="$ip"
-
-  # Create tunnel
-  found="$(qvm-run -p -q -u root "$qube_name" -- nft list table nat 2>/dev/null)"
-   if test -z "$found"; then
-    found=$(qvm-run -p -u root "$qube_name" "iptables -L -nv | grep -c '.*ACCEPT.*$proto dpt:$portnum' ")
-    if [ "$found" -gt 0 ]; then
-      echo "Input rule in $qube_name already exists"
-      echo "Please check configuration - exiting now."
-      exit
-    else
-      qvm-run -q -u root "$qube_name"  "iptables -I INPUT -p $proto --dport $portnum -j ACCEPT "
-    fi
-  else
-    if qvm-run -q -u root "$qube_name"  "nft list table filter | grep '$proto dport $portnum accept' "
-    then
-      echo "Input rule in $qube_name already exists"
-      echo "Please check configuration - exiting now."
-      exit
-    else
-      handle="$(get_handle "$qube_name" filter related,established 1)"
-      qvm-run -q -u root "$qube_name" -- nft add rule filter INPUT position "$handle" iifname eth0 "$proto" dport "$portnum" accept
-    fi
-  fi
-  if ! tunnel netvms[@] ips[@]; then
-    teardown netvms[@] ips[@]
-  fi
-else
-  usage
-fi

salt/sys-syncthing/files/admin/firewall/qvm-port-forward

@@ -0,0 +1,266 @@
+#!/bin/sh
+
+# SPDX-FileCopyrightText: 2017 Jean-Philippe Ouellet <jpo@vt.edu>
+# SPDX-FileCopyrightText: 2022 daktak <daktak@gmail.com>
+# SPDX-FileCopyrightText: 2023 Frederic Pierret <frederic.pierret@qubes-os.org>
+# SPDX-FileCopyrightText: 2024 Benjamin Grande M. S. <ben.grande.b@gmail.com>
+#
+# SPDX-License-Identifier: MIT
+#
+# Credits: https://gist.github.com/daktak/f887352d564b54f9e529404cc0eb60d5
+# Credits: https://gist.github.com/jpouellet/d8cd0eb8589a5b9bf0c53a28fc530369
+# Credits: https://gist.github.com/fepitre/941d7161ae1150d90e15f778027e3248
+
+set -eu
+
+run_qube(){
+  qube="${1}"
+  shift
+  qvm-run --pass-io --user=root "${qube}" -- "${@}"
+}
+
+create_net_dir(){
+  qube="${1}"
+  run_qube "${qube}" mkdir -p "${hook_dir}"
+}
+
+get_rule_handle(){
+  qube="${1}"
+  chain="${2}"
+  rule="${3}"
+  run_qube "${qube}" "nft --handle --stateless list chain ip qubes ${chain} | tr -d '\"' | grep '^\s\+${rule} # handle ' | awk '{print \$NF}' | tr '\n' ' '" 2>/dev/null
+}
+
+delete_rule_handle(){
+  qube="${1}"
+  chain="${2}"
+  handle="${3}"
+  run_qube "${qube}" "nft delete rule ip qubes ${chain} handle ${handle}"
+}
+
+delete_rule(){
+  qube="${1}"
+  chain="${2}"
+  rule="${3}"
+  handle="$(get_rule_handle "${qube}" "${chain}" "${rule}")"
+  if test -n "${handle}"; then
+    for h in ${handle}; do
+      delete_rule_handle "${qube}" "${chain}" "${h}"
+    done
+  fi
+}
+
+forward() {
+  from_qube="${1}"
+  to_qube="${2}"
+  create_net_dir "${from_qube}"
+
+  ## TODO: Handle multiple interfaces in upstream.
+  dev="$(run_qube "${from_qube}" ip -4 r \
+           | awk '/^default via /{print $5}' | head -1)"
+  from_ip="$(run_qube "${from_qube}" ip -4 -o a show dev "${dev}" \
+             | awk '{print $4}' | cut -d "/" -f 1)"
+  to_ip="$(qvm-prefs --get -- "${to_qube}" ip)"
+  to_ip_escaped="$(echo "${to_ip}" | tr "." "-")"
+  hook="${hook_prefix}${to_ip}-${proto}-${port}.sh"
+
+  if test "${from_ip}" = "None"; then
+    from_ip=""
+  fi
+
+  dnat_chain="custom-pf-${to_ip_escaped}"
+  dnat_rule="iifname ${dev} ip saddr ${lan_ip} ${proto} dport ${port} ct state established,related,new counter dnat to ${to_ip}"
+  forward_chain="custom-forward"
+  forward_rule="iifname ${dev} ip saddr ${lan_ip} ip daddr ${to_ip} ${proto} dport ${port} ct state established,related,new counter accept"
+  full_rule="nft 'add chain ip qubes ${dnat_chain} { type nat hook prerouting priority filter +1; policy accept; }
+add rule ip qubes ${dnat_chain} ${dnat_rule}
+add rule ip qubes ${forward_chain} ${forward_rule}'"
+
+  delete_rule "${from_qube}" "${forward_chain}" "${forward_rule}"
+  delete_rule "${from_qube}" "${dnat_chain}" "${dnat_rule}"
+  if test "${action}" = "del"; then
+    echo "info: ${from_qube}: deleting rules" >&2
+    run_qube "${from_qube}" "rm -f ${hook}"
+  else
+    echo "info: ${from_qube}: adding forward rule dev ${dev} saddr ${lan_ip} daddr ${to_ip}" >&2
+    run_qube "${from_qube}" "${full_rule}"
+
+    if test "${persistent}" = "1"; then
+      if test "$(qvm-prefs --get -- "${from_qube}" klass)" = "DispVM"; then
+        from_qube="$(qvm-prefs --get -- "${from_qube}" template)"
+      fi
+
+      full_rule="#!/bin/sh
+get_handle(){
+  chain=\\\${1}
+  rule=\\\${2}
+  nft --handle --stateless list chain ip qubes \\\${chain} | tr -d '\\\"' | grep '^\\\s\\\+\\\${rule} \\# handle ' | awk '{print \\\$NF}' | tr \\\"\\\n\\\" \\\" \\\"
+}
+
+forward_handle=\\\$(get_handle ${forward_chain} \\\"${forward_rule}\\\")
+if test -n \\\"\\\${forward_handle:-}\\\"; then
+  for h in \\\${forward_handle}; do
+    nft delete rule ip qubes ${forward_chain} handle \\\${h}
+  done
+fi
+
+dnat_handle=\\\$(get_handle ${dnat_chain} \\\"${dnat_rule}\\\")
+if test -n \\\"\\\${dnat_handle:-}\\\"; then
+  for h in \\\${dnat_handle}; do
+    nft delete rule ip qubes ${dnat_chain} handle \\\${h}
+  done
+fi
+
+${full_rule}"
+
+      create_net_dir "${from_qube}"
+      run_qube "${from_qube}" "echo \"${full_rule}\" | tee \"${hook}\" >/dev/null"
+      run_qube "${from_qube}" "chmod +x ${hook}"
+    fi
+  fi
+}
+
+input() {
+  qube="${1}"
+  to_ip="$(qvm-prefs --get -- "${qube}" ip)"
+  hook="${hook_prefix}${to_ip}-${proto}-${port}.sh"
+  create_net_dir "${qube}"
+
+  custom_input_rule="${proto} dport ${port} ip daddr ${to_ip} ct state new counter accept"
+  input_rule="nft add rule ip qubes custom-input ${custom_input_rule}"
+
+  delete_rule "${qube}" "custom-input" "${custom_input_rule}"
+  if test "${action}" = "del"; then
+    echo "info: ${qube}: deleting rules" >&2
+    run_qube "${qube}" "rm -f ${hook}"
+  else
+    echo "info: ${qube}: adding input rule daddr ${to_ip}" >&2
+    run_qube "${qube}" "${input_rule}"
+    if test "${persistent}" = "1"; then
+      input_rule="#!/bin/sh
+get_handle(){
+  chain=\\\${1}
+  rule=\\\${2}
+  nft --handle --stateless list chain ip qubes \\\${chain} | tr -d '\\\"' | grep '^\\\s\\\+\\\${rule} \\# handle ' | awk '{print \\\$NF}' | tr \\\"\\\n\\\" \\\" \\\"
+}
+
+input_handle=\\\$(get_handle custom-input \\\"${custom_input_rule}\\\")
+if test -n \\\"\\\${input_handle:-}\\\"; then
+  for h in \\\${input_handle}; do
+    nft delete rule ip qubes custom-input handle \\\${h}
+  done
+fi
+
+${input_rule}"
+
+      run_qube "${qube}" "echo \"${input_rule}\" | tee \"${hook}\" >/dev/null"
+      run_qube "${qube}" "chmod +x ${hook}"
+    fi
+  fi
+}
+
+get_lan(){
+  qube="${1}"
+  ## TODO: Handle multiple interfaces in upstream.
+  dev="$(run_qube "${qube}" ip -4 route \
+          | awk '/^default via /{print $5}' | head -1)"
+  if test -z "${dev}"; then
+    echo "error: ${qube}: could not find any device that is up" >&2
+    exit 1
+  fi
+  lan_ip="$(run_qube "${qube}" ip -4 r show dev "${dev}" prot kernel \
+            | cut -d " " -f 1)"
+  if test -z "${lan_ip}"; then
+    echo "error: ${qube}: could not find LAN from device ${dev}" >&2
+    exit 1
+  fi
+}
+
+recurse_netvms() {
+  cmd="${1}"
+  rec_qube="${2}"
+  rec_netvm="$(qvm-prefs --get -- "${rec_qube}" netvm)"
+  if test -n "${rec_netvm}" && test "${rec_netvm}" != "None"; then
+    case "${cmd}" in
+      show-upstream);;
+      apply-rules) forward "${rec_netvm}" "${rec_qube}";;
+    esac
+    recurse_netvms "${cmd}" "${rec_netvm}"
+  fi
+  case "${cmd}" in
+    show-upstream) get_lan "${rec_qube}";;
+    apply-rules) ;;
+  esac
+}
+
+usage() {
+  echo "Usage: ${0##*/} --action ACTION --qube QUBE --port PORT --proto PROTO --persistent
+Example:
+  ${0##*/} --action add --qube work --port 22 --proto tcp
+  ${0##*/} --action add --qube work --port 444 --proto udp --persistent
+  ${0##*/} --action del --qube work --port 22 --proto tcp
+  ${0##*/} --action del --qube work --port 444 --proto udp
+Note: Defaults to temporary rules
+Warn: If persistent is and and a netvm is disposable, the rule will be saved in the disposable template" >&2
+  exit 1
+}
+
+check_opt(){
+  case "${action:-}" in
+    add|del);;
+    *) echo "error: action must be either 'add' or 'del'" >&2; exit 1;;
+  esac
+
+  case "${proto:-}" in
+    tcp|udp);;
+    *) echo "error: protocol must be only 'tcp' or 'udp'" >&2; exit 1;;
+  esac
+
+  case "${port:-}" in
+    ""|*[!0-9]*) echo "error: port must be only numbers" >&2; exit 1;;
+    *)
+  esac
+
+  if test "${port}" -ge 1 && test "${port}" -le 65535; then
+    true
+  else
+    echo "error: port must be in range 1-65535" >&2
+    exit 1
+  fi
+
+  if test -z "${target_qube:-}"; then
+    echo "error: qube name not provided" >&2
+    exit 1
+  fi
+
+  if ! qvm-check "${target_qube}" >/dev/null 2>&1; then
+    echo "error: qube '${target_qube}' not found." >&2
+    exit 1
+  fi
+}
+
+hook_dir="/rw/config/network-hooks.d"
+hook_prefix="${hook_dir}/90-port-forward-"
+persistent=""
+
+if ! OPTS=$(getopt -o a:q:p:n:s --long action:,qube:,port:,proto:,persistent -n "${0}" -- "${@}"); then
+  echo "An error occurred while parsing options." >&2
+  exit 1
+fi
+
+eval set -- "${OPTS}"
+while test "${#}" -gt "0"; do
+  case "${1}" in
+    -a|--action) action="${2}"; shift;;
+    -q|--qube) target_qube="${2}"; shift;;
+    -p|--port) port="${2}"; shift;;
+    -n|--proto) proto="${2}"; shift;;
+    -s|--persistent) persistent=1; shift;;
+  esac
+  shift
+done
+
+check_opt
+recurse_netvms show-upstream "${target_qube}"
+input "${target_qube}"
+recurse_netvms apply-rules "${target_qube}"