Our network servers are spiking over the default burst limit of 5
packets during regular usage. It's unclear high this should be but 5
packets is definitely too low.
Now that the usage of synproxy is gated behind a SYN packet rate limit,
we can expand this to all our TCP services to have always enabled DDoS
protection instead of needing to deploy a stricter set of rules when the
servers are under attack. This is far better because there isn't always
a system administrator available to handle an ongoing attack.
We already used per-IP connection limits in nginx across the board but
those limits are applied far too late after a TLS connection has been
established and headers are sent rather than before. Using IPv6 /64
blocks means this is much more aggressive for IPv6, but many clients
will fall back to IPv4 due to the happy eyeballs approach. The nginx
limits are still useful due to HTTP/2 multiplexing and we'll need to
think over how to address IPv6 there.
Since our primary servers using SSH to mirror their TLS certificates to
replicas are now allowlisted, we can use a stricter block size than we
could with the PerSourceMaxStartups approach in sshd.
We use synproxy for establishing all new connections to the SSH port and
enforce a connection limit between synproxy and the standard network
stack. Once the connection limit is reached, it's also enforced for new
connections at the synproxy layer. This avoids creating conntrack and
connection limit set entries until connections are already established
to avoid packets with spoofed source addresses exhausting these limited
size tables. Primary servers using SSH to mirror TLS certificates to
their replicas are allowlisted.
