Since we're no longer storing nginx logs in journald, we no longer need
to use journald configuration to control nginx log rotation/retention.
We switched from nginx to dnsdist for the authoritative DNS servers and
are therefore no longer logging any of the queries persistently since we
can rely on the PowerDNS and dnsdist in-memory buffers and stats.
We can use nginx-specific logrotate configuration on a per-server basis
based on balancing the usefulness of access logs with storage space and
getting rid of slightly sensitive data faster (mainly IP addresses).
The error log is fairly quiet during regular use but can end up logging
one or more lines per request during DDoS attacks. Errors are logged for
worker_connections depletion and limit_conn rejections. There's also
currently an nginx bug with modern TLS and OpenSSL causing some client
side TLS errors to be logged as crit instead of info.
This provides more redundancy for both services through having 2
instances in each region. The network services have much higher
bandwidth usage and load so this will also delay us needing to obtain
new servers by making better use of the ones we have.
This sets up the infrastructure for moving from storing nginx access
logs in journald to plain text files written by syslog-ng and rotated by
logrotate. This works around the poor performance, poor space efficiency
and lack of archived log compression for journald. Unlike writing access
logs directly with nginx, this continues avoiding blocking writes in the
event loop and sticks to asynchronous sends through a socket.
Since nginx only supports syslog via the RFC 3164 protocol rather than
the more modern RFC 5424 protocol, this leaves formatting timestamps up
to nginx rather than using the ones provided via the syslog protocol.
The previous commit works around a long term systemd bug which recently
began impacting us again. If the workaround stops working, the behavior
should not be stalling boot forever. Swap isn't needed for our servers
to function so it shouldn't break them if it can't be set up.
Unbound now requests 4M for the send buffer by default and we might as
well permit that for both the send and receive buffers. We set the max
auto-tuned send buffer size on a per-server basis but don't currently
have much use for tuning the maximum manually specified buffer size
across servers. It can be moved in the future if needed.
These servers originally only had the 1Gbps base bandwidth and shaping
it with CAKE worked well to make the most of it during traffic spikes
for the web servers. It has little value for the nameservers since the
only potentially high throughput service is non-interactive SSH.
These servers now have 10Gbps burst available but are heavily limited by
their single virtual core and unable to use all of it in practice. CAKE
can only provide significant value when it's the bottleneck which isn't
the case when the workload is CPU limited. We don't want to keep around
the artificially low 1Gbps limit and it can't do much more.
Unlike OVH, the practical bottleneck is the CPU and FQ has the lowest
CPU usage in practice due to being very performance-oriented with a FIFO
fast path and offloading TCP pacing from the TCP stack to itself. On the
DNS servers, the fast path is always used in practice. Our OVH servers
have a much lower enforced bandwidth limit and the way they implement it
ruins fairness across flows. We definitely want to stick with CAKE for
our VPS instances on OVH but it doesn't make sense on BuyVM anymore.
