Commit Graph

490 Commits

Author SHA1 Message Date
Richard van der Hoff
e564306e31 sanity-check the is_processing flag
... and rename it, for even more sanity
2018-10-24 09:23:33 +01:00
Richard van der Hoff
026cd91ac8 Run PusherPool.start as a background process
We don't do anything with the result, so this is needed to give this code a
logcontext.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
f749607c91 Make on_started synchronous too
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
e7a16c6210 Remove redundant run_as_background_process() from pusherpool
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
c7273c11bc Give pushers their own background logcontext
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
5110f4e425 move get_all_pushers call down
simplifies the interface to _start_pushers
2018-10-22 16:12:11 +01:00
Richard van der Hoff
04277d0ed8 Factor PusherPool._start_pusher out of _start_pushers
... and use it from start_pusher_by_id. This mostly simplifies
start_pusher_by_id.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
3e8b02c939 Rename _refresh_pusher
This is public (or at least, called from outside the class), so ought to have a
better name.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
cc325c7069 Calculate absolute path for email templates 2018-10-19 14:01:59 +01:00
Richard van der Hoff
1519572961 Ship the email templates as package_data
move the example email templates into the synapse package so that they can be
used as package data, which should mean that all of the packaging mechanisms
(pip, docker, debian, arch, etc) should now come with the example templates.

In order to grandfather in people who relied on the templates being in the old
place, check for that situation and fall back to using the defaults if the
templates directory does not exist.
2018-10-17 16:46:02 +01:00
Neil Johnson
7de1989ea2 fix link for case that config.email_riot_base_url is set 2018-09-13 22:43:50 +01:00
Amber Brown
7c27c4d51c
merge (#3576) 2018-09-14 03:11:11 +10:00
Richard van der Hoff
66f7dc8c87 Fix logcontexts for running pushers
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.

Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Erik Johnston
e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown
07cad26d65
Remove all global reactor imports & pass it around explicitly (#3424) 2018-06-25 14:08:28 +01:00
Amber Brown
a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Amber Brown
754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown
e987079037 fixes 2018-05-23 13:03:51 -05:00
Amber Brown
53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown
85ba83eb51 fixes 2018-05-22 16:28:23 -05:00
Amber Brown
fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown
df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Adrian Tschira
933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira
d9fe2b2d9d Replace some more comparisons with six
plus a bonus b"" string I missed last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Richard van der Hoff
950a32eb47
Merge pull request #3152 from NotAFile/py3-local-imports
make imports local
2018-04-30 01:28:13 +01:00
Adrian Tschira
57b58e2174 make imports local
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:41:41 +02:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
Richard van der Hoff
605defb9e4 Add missing consumeErrors
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston
573712da6b Update comments 2018-02-21 11:29:49 +00:00
Erik Johnston
6ff8c87484 Batch inserts into event_push_actions_staging 2018-02-20 11:33:07 +00:00
Erik Johnston
012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston
4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston
acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Richard van der Hoff
03dd745fe2 Better logging when pushes fail 2018-01-29 15:49:06 +00:00
Richard van der Hoff
e051abd20b add appid/device_display_name to to pusher logging 2018-01-29 15:04:16 +00:00
Richard van der Hoff
4528dd2443 Fix logging and add user_id 2018-01-22 20:15:42 +00:00
Richard van der Hoff
93efd7eb04 logging and debug for http pusher 2018-01-22 18:14:10 +00:00
Richard van der Hoff
2c6d63922a Remove pushers when deleting access tokens
Whenever an access token is invalidated, we should remove the associated
pushers.
2017-11-29 16:44:35 +00:00
Richard van der Hoff
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs
Remove preserve_context_over_{fn, deferred}
2017-11-15 11:13:53 +00:00
Richard van der Hoff
7e6fa29cb5 Remove preserve_context_over_{fn, deferred}
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
David Baker
2a98ba0ed3 Rename redact_content option to include_content
The redact_content option never worked because it read the wrong config
section. The PR introducing it
(https://github.com/matrix-org/synapse/pull/2301) had feedback suggesting the
name be changed to not re-use the term 'redact' but this wasn't
incorporated.

This reanmes the option to give it a less confusing name, and also
means that people who've set the redact_content option won't suddenly
see a behaviour change when upgrading synapse, but instead can set
include_content if they want to.

This PR also updates the wording of the config comment to clarify
that this has no effect on event_id_only push.

Includes https://github.com/matrix-org/synapse/pull/2422
2017-11-08 10:35:30 +00:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
David Baker
9342bcfce0 Omit the *s for @room notifications
They're just redundant
2017-10-16 13:38:10 +01:00
David Baker
81a5e0073c pep8 2017-10-10 15:53:34 +01:00
David Baker
ab1bc9bf5f Don't KeyError if no power_levels event 2017-10-10 15:34:05 +01:00
David Baker
0f1eb3e914 Use notification levels in power_levels
Rather than making the condition directly require a specific power
level. This way the level require to notify a room can be configured
per room.
2017-10-10 15:23:00 +01:00
David Baker
c9f034b4ac There was already a constant for this
also update copyright
2017-10-10 11:47:10 +01:00
David Baker
a9f9d68631 More optimisation 2017-10-10 11:38:31 +01:00
David Baker
707374d5dc What year is it!? Who's the president!? 2017-10-10 11:21:41 +01:00
David Baker
269af961e9 Make be faster 2017-10-05 13:27:12 +01:00
David Baker
ed80c6b6cc Add fastpath optimisation 2017-10-05 13:20:22 +01:00
David Baker
e433393c4f pep8 2017-10-05 13:08:02 +01:00
David Baker
985ce80375 They're called rooms 2017-10-05 13:03:44 +01:00
David Baker
b9b9714fd5 Get rule type right 2017-10-05 13:02:19 +01:00
David Baker
fa969cfdde Support for channel notifications
Add condition type to check the sender's power level and add a base
rule using it for @channel notifications.
2017-10-05 12:39:18 +01:00
David Baker
0c8da8b519 Use better method for word boundary searching
From ebc95667b8
2017-10-05 11:57:43 +01:00
David Baker
cbe3c3fdd4 pep8 2017-10-05 11:43:10 +01:00
David Baker
6748f0a579 Fix notif kws that start/end with non-word chars
Only prepend / append word bounary characters if the search
expression starts or ends with a word character, otherwise they
don't work because there's no word bounary between whitespace and
a non-word char.
2017-10-05 11:33:30 +01:00
David Baker
436ee0a2ea Also include the room_id
as really it's part of the event ID
2017-09-18 15:58:38 +01:00
David Baker
b393f5db51 Use .get - it's much shorter 2017-09-18 15:50:26 +01:00
David Baker
a2562f9d74 Add support for event_id_only push format
Param in the data dict of a pusher that tells an HTTP pusher to
send just the event_id of the event it's notifying about and the
notification counts. For clients that want to go & fetch the body
of the event themselves anyway.
2017-09-18 15:39:39 +01:00
Erik Johnston
bfde076022 Increase cache hit ratio for push
We don't update the cache in all code paths, which causes subsequent
calls to miss the cache
2017-07-14 16:11:26 +01:00
Erik Johnston
8d26385d76 Add more metrics to push rule evaluation 2017-07-13 14:37:30 +01:00
Erik Johnston
f60218ec41 Push: Don't acquire lock unless necessary 2017-07-13 11:23:53 +01:00
Erik Johnston
f502b0dea1 Perf: Don't filter events for push
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
Richard van der Hoff
3d31b39297 Merge pull request #2332 from matrix-org/rav/fix_pushes
Fix caching error in the push evaluator
2017-07-05 11:10:53 +01:00
Richard van der Hoff
73cfe48031 Fix caching error in the push evaluator
Initialising `result` to `{}` in the parameters meant that every call to
_flatten_dict used the *same* target dictionary.

I'm hopeful this will fix https://github.com/matrix-org/synapse/issues/2270,
but I suspect it won't. (This code seems to have been here since forever,
unlike the bug, and I don't really think it explains the observed
behaviour). Still, it makes it hard to investigate the problem.
2017-07-05 00:28:43 +01:00
Caleb James DeLisle
75eba3b07d Fix TravisCI tests for PR #2301 2017-06-23 15:15:18 +02:00
Caleb James DeLisle
1591eddaea Add configuration parameter to allow redaction of content from push messages for google/apple devices 2017-06-23 13:01:04 +02:00
Erik Johnston
cca94272fa Fix typo when getting app name 2017-06-06 11:50:07 +01:00
Erik Johnston
66b121b2fc Fix wrong number of arguments 2017-06-06 11:46:38 +01:00
Erik Johnston
11c2a3655f Only load jinja2 templates once
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
2017-05-22 17:48:58 +01:00
Erik Johnston
6489455bed Comment 2017-05-22 16:22:04 +01:00
Erik Johnston
d668caa79c Remove spurious log level guards 2017-05-22 16:21:06 +01:00
Erik Johnston
7fb80b5eae Check if current event is a membership event 2017-05-22 15:02:12 +01:00
Erik Johnston
2d17b09a6d Add debug logging 2017-05-22 15:01:36 +01:00
Erik Johnston
24c8f38784 Comment 2017-05-22 14:59:27 +01:00
Erik Johnston
25f03cf8e9 Use tuple unpacking 2017-05-22 14:58:22 +01:00
Erik Johnston
270e1c904a Speed up calculating push rules 2017-05-19 16:51:05 +01:00
Erik Johnston
1c1c0257f4 Move invalidation cb to its own structure 2017-05-19 11:44:11 +01:00
Erik Johnston
cafe659f72 Store ActionGenerator in HomeServer 2017-05-19 10:09:56 +01:00
Erik Johnston
72ed8196b3 Don't push users who have left 2017-05-18 17:48:36 +01:00
Erik Johnston
760625acba Make get_if_app_services_interested_in_user faster 2017-05-18 16:34:44 +01:00
Erik Johnston
c57789d138 Remove size of push get_rules cache 2017-05-18 16:17:23 +01:00
Erik Johnston
056ba9b795 Add comment 2017-05-18 11:45:56 +01:00
Erik Johnston
66d8ffabbd Faster push rule calculation via push specific cache
We add a push rule specific cache that ensures that we can reuse
calculated push rules appropriately when a user join/leaves.
2017-05-17 16:55:40 +01:00
Erik Johnston
78f306a6f7 Revert "Speed up filtering of a single event in push"
This reverts commit 421fdf7460.
2017-05-08 13:07:41 +01:00
Erik Johnston
421fdf7460 Speed up filtering of a single event in push 2017-04-28 09:52:36 +01:00
Erik Johnston
acb58bfb6a fix up 2017-04-25 15:39:19 +01:00
Erik Johnston
f7181615f2 Don't specify default as dict 2017-04-25 15:22:59 +01:00
Erik Johnston
d9aa645f86 Reduce size of joined_user cache
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Daniel Dent
5058292537 Support authenticated SMTP
Closes (SYN-714) #1385

Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
2017-04-05 21:01:08 -07:00
Erik Johnston
9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed
Speed up cached function access
2017-03-31 10:10:56 +01:00