Commit Graph

643 Commits

Author SHA1 Message Date
Patrick Cloke
666ae87729
Update event push action and receipt tables to support threads. ()
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`,
and `event_push_summary` tables. This will notifications to be segmented by the thread
in a future pull request. The `thread_id` column stores the root event ID or the special
value `"main"`.

The `thread_id` column for `event_push_actions` and `event_push_summary` is
backfilled with `"main"` for all existing rows. New entries into `event_push_actions`
and `event_push_actions_staging` will get the proper thread ID.

`receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar,
except `NULL` is a special value meaning the receipt is "unthreaded".

See MSC3771 and MSC3773 for where this data will be useful.
2022-09-14 17:11:16 +00:00
Nick Mills-Barrett
4c4889cac0
Concurrently collect room unread counts for push badges ()
Most of the time this function is heavily cached, but when that isn't
the case fetching the counts room by room slows down push delivery on
users with many (thousands) of rooms.

Signed off by Nick @ Beeper.
2022-09-09 19:00:21 +01:00
Patrick Cloke
390b7ce946
Disable calculating unread counts unless the config flag is enabled. ()
This avoids doing work that will never be used (since the
resulting unread counts will never be sent in a /sync
response).

The negative of doing this is that unread counts will be
incorrect when the feature is initially enabled.
2022-09-01 16:52:03 +00:00
Erik Johnston
436e0eb39a
Fix breaking event sending due to bad push rule ()
Broke by 

It looks like we have some rules in the DB with a priority class less
than 0 that don't override the base rules. Before these were just
dropped, but  made that a hard error.
2022-08-17 11:02:38 +00:00
Erik Johnston
5442891cbc
Make push rules use proper structures. ()
This improves load times for push rules:

| Version              | Time per user | Time for 1k users | 
| -------------------- | ------------- | ----------------- |
| Before               |       138 µs  |             138ms |
| Now (with custom)    |       2.11 µs |            2.11ms |
| Now (without custom) |       49.7 ns |           0.05 ms |

This therefore has a large impact on send times for rooms
with large numbers of local users in the room.
2022-08-16 12:22:17 +01:00
Erik Johnston
d3995049a8 Merge remote-tracking branch 'origin/master' into develop 2022-07-20 14:59:43 +01:00
Erik Johnston
b4ae3b0d44
Don't include appservice users when calculating push rules ()
This can cause a lot of extra load on servers with lots of appservice users. Introduced in 
2022-07-20 12:06:13 +01:00
Sean Quah
5526f9fc4f
Fix overcounting of pushers when they are replaced ()
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-18 17:39:39 +01:00
Erik Johnston
e5716b631c
Don't pull out the full state when calculating push actions () 2022-07-11 20:08:39 +00:00
Šimon Brandner
1017f09c18
Update MSC3786 implementation: Check the state_key ()
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-06-27 20:28:34 +01:00
Erik Johnston
0d1d3e0708
Speed up get_unread_event_push_actions_by_room ()
Fixes  hopefully.

The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller.

This needs two major changes:
1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it
2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`.

In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
2022-06-15 15:17:14 +00:00
Quentin Gliech
fe1daad672
Move the "email unsubscribe" resource, refactor the macaroon generator & simplify the access token verification logic. ()
This simplifies the access token verification logic by removing the `rights`
parameter which was only ever used for the unsubscribe link in email
notifications. The latter has been moved under the `/_synapse` namespace,
since it is not a standard API.

This also makes the email verification link more secure, by embedding the
app_id and pushkey in the macaroon and verifying it. This prevents the user
from tampering the query parameters of that unsubscribe link.

Macaroon generation is refactored:

- Centralised all macaroon generation and verification logic to the
  `MacaroonGenerator`
- Moved to `synapse.utils`
- Changed the constructor to require only a `Clock`, hostname, and a secret key
  (instead of a full `Homeserver`).
- Added tests for all methods.
2022-06-14 09:12:08 -04:00
Erik Johnston
888a29f412
Wait for lazy join to complete when getting current state () 2022-06-01 16:02:53 +01:00
Erik Johnston
1e453053cb
Rename storage classes () 2022-05-31 12:17:50 +00:00
DeepBlueV7.X
b10211871f
Fix invite notifications for users without pushers ()
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2022-05-30 13:14:43 +02:00
Erik Johnston
4660d9fdcf
Fix up state_store naming () 2022-05-25 12:59:04 +01:00
Šimon Brandner
042e47970b
Remove dont_notify from the .m.rule.room.server_acl rule ()
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-05-24 17:42:32 +01:00
Patrick Cloke
88ce3080d4
Experimental support for MSC3772 ()
Implements the following behind an experimental configuration flag:

* A new push rule kind for mutually related events.
* A new default push rule (`.m.rule.thread_reply`) under an unstable prefix.

This is missing part of MSC3772:

* The `.m.rule.thread_reply_to_me` push rule, this depends on MSC3664 / .
2022-05-24 13:23:23 +00:00
Shay
71e8afe34d
Update EventContext get_current_event_ids and get_prev_event_ids to accept state filters and update calls where possible () 2022-05-20 09:54:12 +01:00
SpiritCroc
a34a41f135
Fix push for m.read events ()
badge_count_last_call was always zero when the response for push
notifications included a "rejected" key which mapped to an empty list.
2022-05-17 11:03:07 +01:00
Patrick Cloke
86a515ccbf
Consolidate logic for parsing relations. ()
Parse the `m.relates_to` event content field (which describes relations)
in a single place, this is used during:

* Event persistence.
* Validation of the Client-Server API.
* Fetching bundled aggregations.
* Processing of push rules.

Each of these separately implement the logic and each made slightly
different assumptions about what was valid. Some had minor / potential
bugs.
2022-05-16 12:42:45 +00:00
Patrick Cloke
a4c75918b3
Remove unneeded ActionGenerator class. ()
It simply passes through to `BulkPushRuleEvaluator`, which can be
called directly instead.
2022-05-11 07:15:21 -04:00
Erik Johnston
c72d26c1e1
Refactor EventContext ()
Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing.

The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). 

One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used.


Part of 
2022-05-10 19:43:13 +00:00
Patrick Cloke
b44fbdffa4
Move free functions into PushRuleEvaluatorForEvent. ()
* Move `_condition_checker` into `PushRuleEvaluatorForEvent`.
* Move the condition cache into `PushRuleEvaluatorForEvent`.
* Improve docstrings.
* Inline a method which is only called once.
2022-05-10 07:54:30 -04:00
Patrick Cloke
02cdace707
Add class-diagrams and notes for push. () 2022-05-10 07:43:34 -04:00
Šimon Brandner
ade3008821
Implement MSC3786: Add a default push rule to ignore m.room.server_acl events ()
Fixes 
Implements 
2022-05-10 08:57:36 +01:00
Šimon Brandner
116a4c8340
Implement changes to MSC2285 (hidden read receipts) ()
* Changes hidden read receipts to be a separate receipt type
  (instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
2022-05-04 11:59:22 -04:00
Sean Quah
800ba87cc8
Refactor and convert Linearizer to async ()
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
Brendan Abolivier
4ba55a620f
Un-revert Jinja2 fix ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-03-28 17:02:57 +00:00
Erik Johnston
7ca8ee67a5
Add cache for get_membership_from_event_ids ()
This should speed up push rule calculations for rooms with large numbers of local users when the main push rule cache fails.

Co-authored-by: reivilibre <oliverw@matrix.org>
2022-03-25 14:58:56 +00:00
Richard van der Hoff
b0659a112d Merge branch 'master' into develop 2022-03-24 19:39:45 +00:00
Andrew Morgan
0004260952
Revert "Replace instances of deprecated Jinja2.Markup with markupsafe.Markup" ()
This reverts commit 8810c93e82.
2022-03-24 18:42:04 +00:00
Andrew Morgan
a503c2c388 Merge branch 'master' into develop 2022-03-24 18:14:26 +00:00
Andrew Morgan
8810c93e82
Replace instances of deprecated Jinja2.Markup with markupsafe.Markup ()
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-24 17:16:13 +00:00
Patrick Cloke
dda9b7fc4d
Use the ignored_users table to test event visibility & sync. ()
Instead of fetching the raw account data and re-parsing it. The
ignored_users table is a denormalised version of the account data
for quick searching.
2022-03-15 14:06:05 -04:00
Nick Mills-Barrett
735e89bd3a
Add an additional HTTP pusher + push rule tests. ()
And rename the field used for caching from _id to _cache_key.
2022-03-11 08:45:26 -05:00
Richard van der Hoff
6c0b44a3d7
Fix PushRuleEvaluator and Filter to work on frozendicts ()
* Fix `PushRuleEvaluator` to work on frozendicts

frozendicts do not (necessarily) inherit from dict, so this needs to handle
them correctly.

* Fix event filtering for frozen events

Looks like this one was introduced by .
2022-02-28 17:40:24 +00:00
Richard van der Hoff
e24ff8ebe3
Remove HomeServer.get_datastore() ()
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of 
2022-02-23 11:04:02 +00:00
Sean Quah
e6acd3cf4f
Upgrade mypy to version 0.931 ()
Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
2022-02-18 15:57:26 +00:00
lukasdenk
4077177390
Prevent duplicate push notifications for room reads () 2022-02-17 10:23:54 +00:00
Patrick Cloke
acda9f07c8
Revert experimental push rules from . ()
Manually reverts the merge from cdbb8e6d6e.
2022-02-02 09:49:31 -05:00
Patrick Cloke
2897fb6b4f
Improvements to bundling aggregations. ()
This is some odds and ends found during the review of 
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
David Robertson
d70169bf9b
Fix missing app variable in mail subject ()
documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500

Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>
2022-01-13 20:19:10 +00:00
Patrick Cloke
10a88ba91c
Use auto_attribs/native type hints for attrs classes. () 2022-01-13 13:49:28 +00:00
reivilibre
84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ()
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
Patrick Cloke
b6102230a7
Add type hints to event_push_actions. () 2021-12-21 13:25:34 +00:00
Patrick Cloke
9562f0c2f1
Ensure emails are canonicalized before fetching associated user. ()
This should fix pushers with an email in non-canonical form is used as
the pushkey.
2021-12-10 07:17:28 -05:00
Patrick Cloke
d93362d87f
Add a constant for receipt types (m.read). ()
And expand some type hints in the receipts storage module.
2021-12-08 12:26:29 -05:00
Sean Quah
088d748f2c
Revert "Move glob_to_regex and re_word_boundary to matrix-python-common () ()
This reverts commit a77c369897.
2021-12-07 13:51:11 +00:00
Sean Quah
a77c369897
Move glob_to_regex and re_word_boundary to matrix-python-common () 2021-12-06 11:36:08 +00:00