We aren't currently using this for testing and it's going to be phased
out for the ns1 servers as soon as we get a second IPv4 /24.
This also switches over to using the nftables ruleset for ns2 instead of
ns1 since it doesn't need the zerotier nftables configuration anymore.
These were previously 2 of our 4 OVH ns1.grapheneos.org instances. Our
ns1.grapheneos.network network has been entirely moved to Vultr for BGP
support so we're reusing these 2 instances as replacements for 2 of the
existing grapheneos.org servers.
Since we're no longer storing nginx logs in journald, we no longer need
to use journald configuration to control nginx log rotation/retention.
We switched from nginx to dnsdist for the authoritative DNS servers and
are therefore no longer logging any of the queries persistently since we
can rely on the PowerDNS and dnsdist in-memory buffers and stats.
We can use nginx-specific logrotate configuration on a per-server basis
based on balancing the usefulness of access logs with storage space and
getting rid of slightly sensitive data faster (mainly IP addresses).
This provides more redundancy for both services through having 2
instances in each region. The network services have much higher
bandwidth usage and load so this will also delay us needing to obtain
new servers by making better use of the ones we have.
These servers originally only had the 1Gbps base bandwidth and shaping
it with CAKE worked well to make the most of it during traffic spikes
for the web servers. It has little value for the nameservers since the
only potentially high throughput service is non-interactive SSH.
These servers now have 10Gbps burst available but are heavily limited by
their single virtual core and unable to use all of it in practice. CAKE
can only provide significant value when it's the bottleneck which isn't
the case when the workload is CPU limited. We don't want to keep around
the artificially low 1Gbps limit and it can't do much more.
Unlike OVH, the practical bottleneck is the CPU and FQ has the lowest
CPU usage in practice due to being very performance-oriented with a FIFO
fast path and offloading TCP pacing from the TCP stack to itself. On the
DNS servers, the fast path is always used in practice. Our OVH servers
have a much lower enforced bandwidth limit and the way they implement it
ruins fairness across flows. We definitely want to stick with CAKE for
our VPS instances on OVH but it doesn't make sense on BuyVM anymore.
