Our network servers are spiking over the default burst limit of 5
packets during regular usage. It's unclear high this should be but 5
packets is definitely too low.
Now that the usage of synproxy is gated behind a SYN packet rate limit,
we can expand this to all our TCP services to have always enabled DDoS
protection instead of needing to deploy a stricter set of rules when the
servers are under attack. This is far better because there isn't always
a system administrator available to handle an ongoing attack.
We already used per-IP connection limits in nginx across the board but
those limits are applied far too late after a TLS connection has been
established and headers are sent rather than before. Using IPv6 /64
blocks means this is much more aggressive for IPv6, but many clients
will fall back to IPv4 due to the happy eyeballs approach. The nginx
limits are still useful due to HTTP/2 multiplexing and we'll need to
think over how to address IPv6 there.
This makes it easier to maintain and deploy more aggressive DDoS
mitigation when our main HTTPS services are under attack.
Network servers use HTTP for connectivity checks which do not use
keepalive and should also be a good use case for
This has the same reasoning as 9fcac6b105
other than these services being used for MTA-STS in addition to ACME and
redirects to the FAQ for the domains.
Our DNS servers only have HTTP(S) for obtaining certificates via ACME
with accounturi pinning along with redirecting people who visit the
domain in a browser to our server documentation. We also only permit 1
request for each HTTP(S) connection for these services so connections
are very short lived.
We'll need to do this in a less aggressive way for our web sites and our
services used to transfer significant amounts of data such as the update
servers since not all clients have TCP timestamps and will lose SACK and
window scaling with the current Linux SYN cookie design despite it being
possible to avoid that as FreeBSD does.
This reuses the approach in cd59960e7b for
SSH connection limits with the same rationale.
PowerDNS also lacks a way to allowlist an address and was limiting our
ADoT reverse proxy to only being able to make 16 connections to the
backend. We could have worked around that by proxing every TCP DNS
connection but it makes more sense to switch to doing this via nftables
where new TCP connections can be completely avoided.
TCP DNS will be perfectly fine without window scaling and Selective
Acknowledgements for clients without TCP timestamps enabled.
Since our primary servers using SSH to mirror their TLS certificates to
replicas are now allowlisted, we can use a stricter block size than we
could with the PerSourceMaxStartups approach in sshd.
We use synproxy for establishing all new connections to the SSH port and
enforce a connection limit between synproxy and the standard network
stack. Once the connection limit is reached, it's also enforced for new
connections at the synproxy layer. This avoids creating conntrack and
connection limit set entries until connections are already established
to avoid packets with spoofed source addresses exhausting these limited
size tables. Primary servers using SSH to mirror TLS certificates to
their replicas are allowlisted.
This is only needed to support clients using the interleaved mode. We
only use chrony as a server on our network servers and the clients are
only using SNTP via xtra-daemon so we don't need this. This frees up a
little bit of memory and avoids having a list of recent clients stored
in memory.
