These were previously 2 of our 4 OVH ns1.grapheneos.org instances. Our
ns1.grapheneos.network network has been entirely moved to Vultr for BGP
support so we're reusing these 2 instances as replacements for 2 of the
existing grapheneos.org servers.
Since nginx only uses 1 second precision for the error logs and syslog
timestamps, we can use receive time on the syslog-ng side. We can switch
to source time once nginx adds RFC 5424 support which is currently in an
open pull request but will likely require changes to add a configuration
option for it. Our approach to working around this within nginx doesn't
work perfectly since $msec generates the time on-demand separately from
the timestamp used by $time_iso8601.
Since we're no longer storing nginx logs in journald, we no longer need
to use journald configuration to control nginx log rotation/retention.
We switched from nginx to dnsdist for the authoritative DNS servers and
are therefore no longer logging any of the queries persistently since we
can rely on the PowerDNS and dnsdist in-memory buffers and stats.
We can use nginx-specific logrotate configuration on a per-server basis
based on balancing the usefulness of access logs with storage space and
getting rid of slightly sensitive data faster (mainly IP addresses).
The error log is fairly quiet during regular use but can end up logging
one or more lines per request during DDoS attacks. Errors are logged for
worker_connections depletion and limit_conn rejections. There's also
currently an nginx bug with modern TLS and OpenSSL causing some client
side TLS errors to be logged as crit instead of info.
This provides more redundancy for both services through having 2
instances in each region. The network services have much higher
bandwidth usage and load so this will also delay us needing to obtain
new servers by making better use of the ones we have.
This sets up the infrastructure for moving from storing nginx access
logs in journald to plain text files written by syslog-ng and rotated by
logrotate. This works around the poor performance, poor space efficiency
and lack of archived log compression for journald. Unlike writing access
logs directly with nginx, this continues avoiding blocking writes in the
event loop and sticks to asynchronous sends through a socket.
Since nginx only supports syslog via the RFC 3164 protocol rather than
the more modern RFC 5424 protocol, this leaves formatting timestamps up
to nginx rather than using the ones provided via the syslog protocol.
