Commit Graph

401 Commits

Author SHA1 Message Date
Daniel Micay
edbf50a3da nftables: rename tcp service chains 2024-04-11 11:59:27 -04:00
Daniel Micay
3d886dea43 nftables: split out TCP service chain 2024-04-11 11:49:22 -04:00
Daniel Micay
b152574da8 nftables: avoid unnecessary connection marking 2024-04-11 11:38:43 -04:00
Daniel Micay
832a430954 nftables: handle non-TCP case in input-new chain 2024-04-11 10:34:10 -04:00
Daniel Micay
8f047de0c3 nftables: explain synproxy untracked/invalid cases 2024-04-11 10:19:39 -04:00
Daniel Micay
26a58b2492 nftables: explain synproxy bypass rate limit 2024-04-11 09:56:30 -04:00
Daniel Micay
ecd14bddff nftables: explain ordering of strong host model check 2024-04-11 09:51:10 -04:00
Daniel Micay
b21ea0a23f raise synproxy bypass burst to 128 packets from 5
Our network servers are spiking over the default burst limit of 5
packets during regular usage. It's unclear high this should be but 5
packets is definitely too low.
2024-04-10 15:08:45 -04:00
Daniel Micay
b38736ca74 enable nftables-based DDoS protection for all TCP services
Now that the usage of synproxy is gated behind a SYN packet rate limit,
we can expand this to all our TCP services to have always enabled DDoS
protection instead of needing to deploy a stricter set of rules when the
servers are under attack. This is far better because there isn't always
a system administrator available to handle an ongoing attack.

We already used per-IP connection limits in nginx across the board but
those limits are applied far too late after a TLS connection has been
established and headers are sent rather than before. Using IPv6 /64
blocks means this is much more aggressive for IPv6, but many clients
will fall back to IPv4 due to the happy eyeballs approach. The nginx
limits are still useful due to HTTP/2 multiplexing and we'll need to
think over how to address IPv6 there.
2024-04-10 14:48:10 -04:00
Daniel Micay
14c0128910 add rate limited synproxy bypass 2024-04-10 12:15:19 -04:00
Daniel Micay
ef63ff82d2 use consistent name for main connlimit
This will be extended to other services when under attack, which can be
automated in the future.
2024-04-09 20:29:11 -04:00
Daniel Micay
275d63e8b3 certbot: add ns2 variant of staging authoritative DNS 2024-04-08 17:06:43 -04:00
Daniel Micay
5106ec7f4a remove redundant vm.max_map_count configuration
The same value we were using is now the default.
2024-04-07 15:11:35 -04:00
Daniel Micay
a6b9fa782b reorganize nftables rules 2024-04-05 19:14:05 -04:00
Daniel Micay
cf274f34d7 simplify synproxy-based connection limit rules 2024-04-05 13:36:09 -04:00
Daniel Micay
c1756f5809 add synproxy/connlimit for ACME/redirect use of HTTP
This makes it easier to maintain and deploy more aggressive DDoS
mitigation when our main HTTPS services are under attack.

Network servers use HTTP for connectivity checks which do not use
keepalive and should also be a good use case for
2024-04-04 12:46:38 -04:00
Daniel Micay
3e23b80e9c move mail server HTTP(S) connection limit to nftables
This has the same reasoning as 9fcac6b105
other than these services being used for MTA-STS in addition to ACME and
redirects to the FAQ for the domains.
2024-04-04 12:46:22 -04:00
Daniel Micay
83bcc0b327 merge synproxy rules 2024-03-31 22:25:38 -04:00
Daniel Micay
9fcac6b105 use DNS connlimit for HTTP/HTTPS on DNS servers
Our DNS servers only have HTTP(S) for obtaining certificates via ACME
with accounturi pinning along with redirecting people who visit the
domain in a browser to our server documentation. We also only permit 1
request for each HTTP(S) connection for these services so connections
are very short lived.

We'll need to do this in a less aggressive way for our web sites and our
services used to transfer significant amounts of data such as the update
servers since not all clients have TCP timestamps and will lose SACK and
window scaling with the current Linux SYN cookie design despite it being
possible to avoid that as FreeBSD does.
2024-03-31 22:23:10 -04:00
Daniel Micay
079997d4b5 chrony: raise minsources to 3 2024-03-31 14:03:16 -04:00
Daniel Micay
0edcc3514e chrony: add time.cifelli.xyz as source 2024-03-31 14:02:16 -04:00
Daniel Micay
be63011a31 chrony: add time.dfm.dk as source 2024-03-31 14:02:08 -04:00
Daniel Micay
0fab969de7 enforce IPv6 SUPL connection limit for /64 blocks 2024-03-30 20:40:38 -04:00
Daniel Micay
457f0bf7e8 move IP-based SUPL connection limits to nftables 2024-03-30 20:40:11 -04:00
Daniel Micay
e655732304 drop unnecessary nftables udp condition 2024-03-30 12:23:05 -04:00
Daniel Micay
39b7e1f479 add counter to connection limit reject rules 2024-03-30 02:12:18 -04:00
Daniel Micay
b40988122c switch to Java 21 LTS package since Java 22 is out 2024-03-30 02:12:00 -04:00
Daniel Micay
280eb51c8d rename loopback chains for clarity 2024-03-30 02:12:00 -04:00
Daniel Micay
9b40bb90b8 split out input chain for loopback 2024-03-30 02:12:00 -04:00
Daniel Micay
811fcf593e enforce IPv6 DNS connection limit for /64 blocks 2024-03-30 02:12:00 -04:00
Daniel Micay
d95752bea6 move IP-based DNS connection limits to nftables
This reuses the approach in cd59960e7b for
SSH connection limits with the same rationale.

PowerDNS also lacks a way to allowlist an address and was limiting our
ADoT reverse proxy to only being able to make 16 connections to the
backend. We could have worked around that by proxing every TCP DNS
connection but it makes more sense to switch to doing this via nftables
where new TCP connections can be completely avoided.

TCP DNS will be perfectly fine without window scaling and Selective
Acknowledgements for clients without TCP timestamps enabled.
2024-03-30 02:11:21 -04:00
Daniel Micay
6b573fe227 dns-stats: show total TCP and UDP queries 2024-03-28 11:38:06 -04:00
Daniel Micay
8c929f02ac enforce IPv6 SSH connection limit for /48 blocks
Since our primary servers using SSH to mirror their TLS certificates to
replicas are now allowlisted, we can use a stricter block size than we
could with the PerSourceMaxStartups approach in sshd.
2024-03-28 11:38:06 -04:00
Daniel Micay
cd59960e7b move IP-based SSH connection limits to nftables
We use synproxy for establishing all new connections to the SSH port and
enforce a connection limit between synproxy and the standard network
stack. Once the connection limit is reached, it's also enforced for new
connections at the synproxy layer. This avoids creating conntrack and
connection limit set entries until connections are already established
to avoid packets with spoofed source addresses exhausting these limited
size tables. Primary servers using SSH to mirror TLS certificates to
their replicas are allowlisted.
2024-03-28 11:38:03 -04:00
Daniel Micay
16ef317460 nftables: rename output-reject to graceful-reject 2024-03-27 12:31:09 -04:00
Daniel Micay
66562272ac set preferred source for static IPv6 configuration 2024-03-26 21:50:12 -04:00
Daniel Micay
3de32072da consistently use short form IPv6 addresses 2024-03-26 21:24:50 -04:00
Daniel Micay
571644526d consistently list IPv4 routes before IPv6 routes 2024-03-26 21:24:50 -04:00
Daniel Micay
64e2e836d3 set preferred source for static IPv4 configuration 2024-03-26 21:24:48 -04:00
Daniel Micay
14e9cd5b76 use standard style for nftables sets 2024-03-24 16:23:54 -04:00
Daniel Micay
0ac67c38c3 allow IPv6 SSH for discuss.grapheneos.org
This could be useful and disabling it isn't necessary for blocking IPv6
connections to the forum.
2024-03-24 15:41:13 -04:00
Daniel Micay
7b64ffd4cd simplify nftables based on strong host model 2024-03-24 15:22:00 -04:00
Daniel Micay
59984a477c enforce strong host model via nftables 2024-03-24 14:36:24 -04:00
Daniel Micay
eb55afa3a8 reorganize sysctl configuration 2024-03-24 11:03:31 -04:00
Daniel Micay
51a4f8ca7a extend disabling ICMP redirects 2024-03-24 10:43:37 -04:00
Daniel Micay
ec2cbbdb4e enforce strict reverse path filtering via nftables 2024-03-23 13:35:49 -04:00
Daniel Micay
81fa5f8ebd use standard log rotation approach for wtmp/btmp 2024-03-20 23:43:48 -04:00
Daniel Micay
455ef92c18 disable chrony client log
This is only needed to support clients using the interleaved mode. We
only use chrony as a server on our network servers and the clients are
only using SNTP via xtra-daemon so we don't need this. This frees up a
little bit of memory and avoids having a list of recent clients stored
in memory.
2024-03-20 23:24:57 -04:00
Daniel Micay
e1df22a68f clean up session ticket rotation scripts 2024-03-20 22:55:40 -04:00
Daniel Micay
f35dc08868 split grapheneos.org hosts array 2024-03-18 21:10:47 -04:00