Commit graph

829 commits

Author SHA1 Message Date
Daniel Micay
34a18b6a86 simplify deploy-primary 2025-10-30 17:06:07 -04:00
Daniel Micay
0d1705320f use consistent naming for session ticket key scripts/units 2025-10-30 17:06:07 -04:00
Daniel Micay
768cc9ada3 update LS_COLORS configuration
This is generated from the current standard dircolors database with the
addition of Brotli to the archive file types.
2025-10-30 16:17:33 -04:00
Daniel Micay
9fde84c877 add initial session ticket key synchronization 2025-10-30 14:22:55 -04:00
Daniel Micay
f9430a1aeb add script for deploying certbot replication setup 2025-10-30 14:22:32 -04:00
Daniel Micay
e6db6a15e6 add swap device timeout as a fallback
The previous commit works around a long term systemd bug which recently
began impacting us again. If the workaround stops working, the behavior
should not be stalling boot forever. Swap isn't needed for our servers
to function so it shouldn't break them if it can't be set up.
2025-10-29 22:47:01 -04:00
Daniel Micay
8340cf2813 add workaround for system encrypted swap race
This appeared to be solved a while ago but ended up returning.
2025-10-29 22:36:11 -04:00
Daniel Micay
2d87e13eb0 add bootloader deployment script 2025-10-29 18:32:52 -04:00
Daniel Micay
85c5ccc613 update IP addresses for 0.releases.grapheneos.org 2025-10-28 15:25:16 -04:00
Daniel Micay
848d4822e1 rotate-session-ticket-keys: replace is-enabled with is-active 2025-10-28 12:49:17 -04:00
Daniel Micay
0b519d6f5e set AccuracySec=1us for tcp-fastopen-rotate-keys 2025-10-28 12:33:10 -04:00
Daniel Micay
81147f1fbb rotate-session-ticket-keys: skip when synced 2025-10-28 02:59:00 -04:00
Daniel Micay
17f0ec527d cleanly phase in new TLS session ticket keys
This closes a small window where new workers could give keys not
accepted by the old workers before they're gracefully shut down. This
will also be needed when syncing keys across a cluster.
2025-10-28 02:48:49 -04:00
Daniel Micay
9ffcb3e648 minor rotate-session-ticket-keys improvements 2025-10-27 23:16:36 -04:00
Daniel Micay
9ed61cef61 reduce TLS session ticket key interval from 8h to 6h 2025-10-27 22:50:32 -04:00
Daniel Micay
ce0942702e add RemainAfterExit=yes to create-session-ticket-keys.service 2025-10-27 22:11:22 -04:00
Daniel Micay
448565de54 update description for rotate-session-ticket-keys.timer 2025-10-27 21:19:32 -04:00
Daniel Micay
c4af821eda always create /var/cache/nginx for web servers
This avoids needing to restart nginx for ReadWritePaths to kick in after
creating it.
2025-10-27 20:52:34 -04:00
Daniel Micay
fb9e4d6769 remove imagemagick package from mastodon.social
This was replaced by libvips.
2025-10-24 15:04:39 -04:00
Daniel Micay
048ccb3fba allow powerdns user to query pdns over loopback
This is being used by the pdns-trigger-health-checks script.
2025-10-23 14:11:56 -04:00
Daniel Micay
9c2183c794 stop blacklisting tls module
It no longer gets autoloaded by default due to Linux kernel changes.
2025-10-22 17:36:06 -04:00
Daniel Micay
178791ffd8 update pacreport.conf 2025-10-21 14:11:46 -04:00
Daniel Micay
d6823c9ae0 update python dependencies 2025-10-21 13:21:17 -04:00
Daniel Micay
92288293d9 gitignore: add /lock 2025-10-20 21:46:56 -04:00
Daniel Micay
f8a1d381e7 mdmonitor.service: use syslog reporting 2025-10-19 16:16:33 -04:00
Daniel Micay
e626d67dc1 add nftables deployment script 2025-10-19 15:35:13 -04:00
Daniel Micay
11c9421c63 rename deploy-initial to deploy-initial-vps 2025-10-19 14:58:36 -04:00
Daniel Micay
aaf63a7d87 add 3.releases.grapheneos.org package list 2025-10-19 12:35:01 -04:00
Daniel Micay
04d0489e57 add certbot and web deployment scripts 2025-10-19 12:30:53 -04:00
Daniel Micay
e84c84db6b disconnect: handle separate non-interactive sessions too 2025-10-11 18:04:48 -04:00
Daniel Micay
f2a4df1d0f add another IPv6 address for 0.releases.grapheneos.org
This will be used to send more traffic to it via DNS RRset load
balancing.
2025-10-11 15:31:09 -04:00
Daniel Micay
5ea8e202a1 0.releases.grapheneos.org IPv4 update
The main IPv4 address has changed and we're now using an additional IPv4
address to send more traffic to it via DNS RRset load balancing.
2025-10-11 15:30:35 -04:00
Daniel Micay
02b7e4e5c1 add 3.releases.grapheneos.org server 2025-10-09 09:06:31 -04:00
Daniel Micay
a52a12450d handle update server renaming for authorized_keys 2025-10-07 17:40:32 -04:00
Daniel Micay
e8a55eeede count: fix counting 9th gen Pixel updates
It wasn't including incremental updates and wasn't using a specific
build number.
2025-10-06 14:36:23 -04:00
Daniel Micay
4e1e1324a2 update python dependencies 2025-10-05 23:36:44 -04:00
Daniel Micay
48d939d39d adjust IPv6 subnet size for ReliableSite servers 2025-10-05 00:50:18 -04:00
Daniel Micay
e57096dfec disable TCP Fast Open on BuyVM for now 2025-09-30 16:56:21 -04:00
Daniel Micay
3691bd8e51 fetch-info: enable standard error detection setup 2025-09-25 15:50:14 -04:00
Daniel Micay
173822655c switch to xxd for converting random bytes to hex 2025-09-23 19:56:42 -04:00
Daniel Micay
d125eb96ca improve tls group configuration 2025-09-20 14:49:41 -04:00
Daniel Micay
47062b9c68 raise wmem_max/rmem_max for non-autotuned buffers
Unbound now requests 4M for the send buffer by default and we might as
well permit that for both the send and receive buffers. We set the max
auto-tuned send buffer size on a per-server basis but don't currently
have much use for tuning the maximum manually specified buffer size
across servers. It can be moved in the future if needed.
2025-09-18 13:56:46 -04:00
Daniel Micay
348cdf9d74 update systemd configuration 2025-09-18 11:17:05 -04:00
Daniel Micay
c6156ebed7 switch from shaped CAKE to FQ for BuyVM servers
These servers originally only had the 1Gbps base bandwidth and shaping
it with CAKE worked well to make the most of it during traffic spikes
for the web servers. It has little value for the nameservers since the
only potentially high throughput service is non-interactive SSH.

These servers now have 10Gbps burst available but are heavily limited by
their single virtual core and unable to use all of it in practice. CAKE
can only provide significant value when it's the bottleneck which isn't
the case when the workload is CPU limited. We don't want to keep around
the artificially low 1Gbps limit and it can't do much more.

Unlike OVH, the practical bottleneck is the CPU and FQ has the lowest
CPU usage in practice due to being very performance-oriented with a FIFO
fast path and offloading TCP pacing from the TCP stack to itself. On the
DNS servers, the fast path is always used in practice. Our OVH servers
have a much lower enforced bandwidth limit and the way they implement it
ruins fairness across flows. We definitely want to stick with CAKE for
our VPS instances on OVH but it doesn't make sense on BuyVM anymore.
2025-09-18 01:26:39 -04:00
Daniel Micay
b2c15916cc no need to override default qdisc since we set it 2025-09-17 19:23:26 -04:00
Daniel Micay
7d55588972 nftables: preserve connlimit sets across reloads 2025-09-17 19:23:22 -04:00
Daniel Micay
f3156e641d nftables: reorder network server UDP notrack 2025-09-16 18:19:33 -04:00
Daniel Micay
78bd96f4ae nftables: move listening ports to constants 2025-09-16 18:19:31 -04:00
Daniel Micay
d923bc7e24 use monotonic timer for session ticket key rotation
It makes more sense to rotate session ticket keys every 8 hours instead
of doing it at 3 specific times each day where the initial rotation will
happen earlier than necessary. It makes little difference due to keeping
the previous 3 session tickets valid but is cleaner.
2025-09-15 21:10:42 -04:00
Daniel Micay
5f1b0c886d nftables: replace magic numbers with constants 2025-09-15 21:10:42 -04:00