From 047486a384a706fac2cc98c22f0ab23649cc320f Mon Sep 17 00:00:00 2001 From: Benjamin Saunders Date: Sat, 8 Dec 2018 02:33:42 -0800 Subject: [PATCH 1/2] Improve advice regarding poor performance Signed-off-by: Benjamin Saunders --- AUTHORS.rst | 3 +++ README.rst | 32 +++++++++++++++++++++++++------- changelog.d/4276.misc | 1 + 3 files changed, 29 insertions(+), 7 deletions(-) create mode 100644 changelog.d/4276.misc diff --git a/AUTHORS.rst b/AUTHORS.rst index 3ea18eefc..d8b4a846d 100644 --- a/AUTHORS.rst +++ b/AUTHORS.rst @@ -72,3 +72,6 @@ Jason Robinson Joseph Weston + Add admin API for querying HS version + +Benjamin Saunders + * Documentation improvements diff --git a/README.rst b/README.rst index 5409f0c56..47fe7f344 100644 --- a/README.rst +++ b/README.rst @@ -340,8 +340,11 @@ log lines and looking for any 'Processed request' lines which take more than a few seconds to execute. Please let us know at #synapse:matrix.org if you see this failure mode so we can help debug it, however. -Help!! Synapse eats all my RAM! -------------------------------- +Help!! Synapse is slow and eats all my RAM/CPU! +=============================================== + +First, ensure you are running the latest version of Synapse, using Python 3 +with a PostgreSQL database. Synapse's architecture is quite RAM hungry currently - we deliberately cache a lot of recent room data and metadata in RAM in order to speed up @@ -352,14 +355,29 @@ variable. The default is 0.5, which can be decreased to reduce RAM usage in memory constrained enviroments, or increased if performance starts to degrade. +However, degraded performance due to a low cache factor, common on +machines with slow disks, often leads to explosions in memory use due +backlogged requests. In this case, reducing the cache factor will make +things worse. Instead, try increasing it drastically. 2.0 is a good +starting value. + Using `libjemalloc `_ can also yield a significant -improvement in overall amount, and especially in terms of giving back RAM -to the OS. To use it, the library must simply be put in the LD_PRELOAD -environment variable when launching Synapse. On Debian, this can be done -by installing the ``libjemalloc1`` package and adding this line to -``/etc/default/matrix-synapse``:: +improvement in overall memory use, and especially in terms of giving back +RAM to the OS. To use it, the library must simply be put in the +LD_PRELOAD environment variable when launching Synapse. On Debian, this +can be done by installing the ``libjemalloc1`` package and adding this +line to ``/etc/default/matrix-synapse``:: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 This can make a significant difference on Python 2.7 - it's unclear how much of an improvement it provides on Python 3.x. + +If you're encountering high CPU use by the Synapse process itself, you +may be affected by a bug with presence tracking that leads to a +massive excess of outgoing federation requests (see `discussion +`_). If metrics +indicate that your server is also issuing far more outgoing federation +requests than can be accounted for by your users' activity, this is a +likely cause. The misbehavior can be worked around by setting +``use_presence: false`` in the Synapse config file. diff --git a/changelog.d/4276.misc b/changelog.d/4276.misc new file mode 100644 index 000000000..285939a4b --- /dev/null +++ b/changelog.d/4276.misc @@ -0,0 +1 @@ +Improve README section on performance troubleshooting. From b36de88066d948393fecc39bc6a0d902a7dc49ec Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Tue, 18 Jun 2019 18:32:51 +0100 Subject: [PATCH 2/2] README.rst: fix header level --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 47fe7f344..13e11a577 100644 --- a/README.rst +++ b/README.rst @@ -341,7 +341,7 @@ a few seconds to execute. Please let us know at #synapse:matrix.org if you see this failure mode so we can help debug it, however. Help!! Synapse is slow and eats all my RAM/CPU! -=============================================== +----------------------------------------------- First, ensure you are running the latest version of Synapse, using Python 3 with a PostgreSQL database.