Commit Graph

615 Commits

Author SHA1 Message Date
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
7950aa8a27 Fix some typos. 2021-02-12 11:14:12 -05:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. (#9257)
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
2021-02-04 10:18:25 -05:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. (#9260)
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
  was tweaked to depend on whether there was 1 sender or multiple senders,
  not based on if there was 1 room or multiple rooms.
2021-02-01 13:09:39 -05:00
Patrick Cloke
e54746bdf7
Clean-up the template loading code. (#9200)
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
2021-01-27 10:59:50 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. (#9165)
Treat the content as untrusted and do not assume it is of
the proper form.
2021-01-25 07:27:16 -05:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms (#9024)
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-01-07 13:03:38 +00:00
Patrick Cloke
37eaf9c272
Fix-up assertions about last stream token in push (#9020)
The last stream token is always known and we do not need to handle none.
2021-01-05 10:53:15 -05:00
Patrick Cloke
4136255d3c
Ensure that a URL exists in the content during push. (#8965)
This fixes an KeyError exception, after this PR the content
is just considered unknown.
2020-12-18 07:26:15 -05:00
Patrick Cloke
bd30cfe86a
Convert internal pusher dicts to attrs classes. (#8940)
This improves type hinting and should use less memory.
2020-12-16 11:25:30 -05:00
Patrick Cloke
b3a4b53587
Fix handling of stream tokens for push. (#8943)
Removes faulty assertions and fixes the logic to ensure the max
stream token is always set.
2020-12-15 10:41:34 -05:00
Patrick Cloke
5d34f40d49
Add type hints to the push module. (#8901) 2020-12-11 11:43:53 -05:00
Patrick Cloke
92d87c6882
Add type hints for HTTP and email pushers. (#8880) 2020-12-07 09:59:38 -05:00
Patrick Cloke
02e588856a
Add type hints to the push mailer module. (#8882) 2020-12-07 07:10:22 -05:00
Patrick Cloke
b774c555d8
Add additional validation to pusher URLs. (#8865)
Pusher URLs now must end in `/_matrix/push/v1/notify` per the
specification.
2020-12-04 10:51:56 -05:00
Patrick Cloke
30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Andrew Morgan
17fa58bdd1
Add a config option to change whether unread push notification counts are per-message or per-room (#8820)
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).

However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
2020-11-30 18:43:54 +00:00
David Baker
59cc2472b3
Add base pushrule to notify for jitsi conferences (#8286)
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
2020-11-02 16:36:14 +00:00
Richard van der Hoff
cbc82aa09f
Implement and use an @lru_cache decorator (#8595)
We don't always need the full power of a DeferredCache.
2020-10-30 11:43:17 +00:00
Erik Johnston
2b7c180879
Start fewer opentracing spans (#8640)
#8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.

This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.

I don't really know how much this will help.
2020-10-26 09:30:19 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Erik Johnston
db9ef792f0
Fix email notifications for invites without local state. (#8627)
This can happen if e.g. the room invited into is no longer on the
server (or if all users left the room).
2020-10-23 10:41:32 +01:00
Richard van der Hoff
903d11c43a
Add DeferredCache.get_immediate method (#8568)
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
2020-10-19 15:00:12 +01:00
Patrick Cloke
c356b4bf42
Include a simple message in email notifications that include encrypted content (#8545) 2020-10-19 09:12:39 -04:00
Richard van der Hoff
995cc615a0
Apply suggestions from code review
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-16 16:14:42 +01:00
Richard van der Hoff
0ec0bc3886 type annotations for LruCache 2020-10-16 15:56:39 +01:00
Richard van der Hoff
3ee17585cd
Make LruCache register its own metrics (#8561)
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
2020-10-16 15:51:57 +01:00
Erik Johnston
921a3f8a59
Fix not sending events over federation when using sharded event persisters (#8536)
* Fix outbound federaion with multiple event persisters.

We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.

Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.

* Change some interfaces to use RoomStreamToken.

By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
2020-10-14 13:27:51 +01:00
Patrick Cloke
4ff0201e62
Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
Mathieu Velten
916bb9d0d1
Don't push if an user account has expired (#8353) 2020-09-23 16:06:28 +01:00
Patrick Cloke
8a4a4186de
Simplify super() calls to Python 3 syntax. (#8344)
This converts calls like super(Foo, self) -> super().

Generated with:

    sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
2020-09-18 09:56:44 -04:00
Erik Johnston
5d3e306d9f
Clean up Notifier.on_new_room_event code path (#8288)
The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it.

This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens.

The valid operations here are:

1. Is a position before or after a token
2. Fetching all events between two tokens
3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C.

Future PR will change the token type to a dedicated type.
2020-09-10 13:24:43 +01:00
Andrew Morgan
a3a90ee031
Show a confirmation page during user password reset (#8004)
This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset.

This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
2020-09-10 11:45:12 +01:00
Erik Johnston
c9dbee50ae
Fixup pusher pool notifications (#8287)
`pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed.

I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
2020-09-09 16:56:08 +01:00
Erik Johnston
dc9dcdbd59 Revert "Fixup pusher pool notifications"
This reverts commit e7fd336a53.
2020-09-09 16:19:22 +01:00
Erik Johnston
e7fd336a53 Fixup pusher pool notifications 2020-09-09 16:17:50 +01:00
Brendan Abolivier
d8762cc116
Only add rows to the push actions table if the event notifies or should be marked unread (#8274) 2020-09-07 16:56:27 +01:00
Brendan Abolivier
041ee971c9
Unread counts fixes (#8254)
* Fixup `ALTER TABLE` database queries

Make the new columns nullable, because doing otherwise can wedge a
server with a big database, as setting a default value rewrites the
table.

* Switch back to using the notifications count in the push badge

Clients are likely to be confused if we send a push but the badge count
is the unread messages one, and not the notifications one.

* Changelog
2020-09-04 14:14:22 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Brendan Abolivier
5a1dd297c3
Re-implement unread counts (again) (#8059) 2020-09-02 17:19:37 +01:00
Andrew Morgan
e04e465b4d
Use the default templates when a custom template file cannot be found (#8037)
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
Brendan Abolivier
cdbb8e6d6e
Implement new experimental push rules (#7997)
With an undocumented configuration setting to enable them for specific users.
2020-08-10 11:48:01 +01:00
Brendan Abolivier
2ffd6783c7
Revert #7736 (#8039) 2020-08-06 17:15:35 +01:00
Patrick Cloke
d4a7829b12
Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Brendan Abolivier
dd11f575a2
Incorporate review 2020-08-06 10:52:26 +01:00
Brendan Abolivier
60328ce9fb
Lint 2020-07-30 19:02:28 +01:00
Brendan Abolivier
69158e554f Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules 2020-07-30 19:00:29 +01:00
Brendan Abolivier
8dff4a1242
Re-implement unread counts (#7736) 2020-07-29 18:26:55 +01:00
Brendan Abolivier
9725c59247
Implement new experimental push rules with a database hack to enable them 2020-07-28 19:20:55 +01:00
Patrick Cloke
8144bc26a7
Convert push to async/await. (#7948) 2020-07-27 12:21:34 -04:00
Patrick Cloke
b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Erik Johnston
649a7ead5c
Add ability to run multiple pusher instances (#7855)
This reuses the same scheme as federation sender sharding
2020-07-16 14:06:28 +01:00
Brendan Abolivier
85223106f3
Allow email subjects to be customised through Synapse's configuration (#7846) 2020-07-14 19:10:42 +01:00
Erik Johnston
f886a69916
Correctly pass app_name to all email templates. (#7829)
We didn't do this for e.g. registration emails.
2020-07-14 10:00:53 +01:00
reivilibre
57feeab364
Don't ignore set_tweak actions with no explicit value. (#7766)
* Fix spec compliance; tweaks without values are valid

(default to True, which is only concretely specified for
`highlight`, but it seems only reasonable to generalise)

* Changelog for 7766.

* Add documentation to `tweaks_for_actions`

May as well tidy up when I'm here.

* Add a test for `tweaks_for_actions`
2020-07-06 11:43:41 +01:00
reivilibre
e5808c4cfb
Hack to add push priority to push notifications (#7765)
* Remove obsolete comment about ancient temporary code

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Implement hack to set push priority

based on whether the tweaks indicate the event might cause
effects.

* Changelog for 7765

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Antilint

* Add tests for push priority

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Update synapse/push/httppusher.py

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Antilint

* Remove needless invites from tests.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2020-07-01 17:02:31 +01:00
Brendan Abolivier
74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Richard van der Hoff
7d2824395f add a comment 2020-06-18 10:47:06 +01:00
Brendan Abolivier
5a5cf6460e
Fix unread counts in sync
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
2020-06-17 15:10:44 +01:00
Brendan Abolivier
46613aaf79
Implement unread counter (MSC2625) (#7673)
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
Erik Johnston
f6f7511a4c
Refactor getting replication updates from database. (#7636)
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
2020-06-16 17:10:28 +01:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke
cc32fa7358
Ensure the body is a string before comparing push rules. (#7701) 2020-06-15 16:20:34 -04:00
Brendan Abolivier
6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Patrick Cloke
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Brendan Abolivier
e47e5a2dcd
Incorporate review bits 2020-06-12 15:13:12 +01:00
Brendan Abolivier
ea8f6e611b
Actually act on mark_unread 2020-06-11 15:30:42 +01:00
Brendan Abolivier
ef345c5a7b
Add a new unread_counter to sync responses 2020-06-10 16:21:16 +01:00
Erik Johnston
06a02bc1ce
Convert sending mail to async/await. (#7557)
Mainly because sometimes the email push code raises exceptions where the
stack traces have gotten lost, which is hopefully fixed by this.
2020-05-22 13:41:11 +01:00
Richard van der Hoff
d4676910c9 remove miscellaneous PY2 code 2020-05-15 19:37:41 +01:00
Erik Johnston
1a1da60ad2
Fix new flake8 errors (#7470) 2020-05-12 11:20:48 +01:00
Amber Brown
7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Patrick Cloke
01294e6b3a
Do not treat display names as globs for push rules. (#7271) 2020-04-16 10:52:55 -04:00
Richard van der Hoff
e913823a22
Fix concurrent modification errors in pusher metrics (#7106)
add a lock to try to make this metric actually work
2020-03-19 10:28:49 +00:00
Richard van der Hoff
8c75667ad7
Add prometheus metrics for the number of active pushers (#7103) 2020-03-19 10:00:24 +00:00
Brendan Abolivier
aee2bae952
Fix undefined room_id in make_summary_text
This would break notifications about un-named rooms when processing
notifications in a batch.
2020-03-09 14:10:19 +00:00
Patrick Cloke
af6c389501
No longer use room alias events to calculate room names for push notifications. (#6966) 2020-02-21 12:50:48 -05:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Erik Johnston
ed630ea17c
Reduce amount of logging at INFO level. (#6862)
A lot of the things we log at INFO are now a bit superfluous, so lets
make them DEBUG logs to reduce the amount we log by default.

Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2020-02-06 13:31:05 +00:00
Erik Johnston
28c98e51ff
Add local_current_membership table (#6655)
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
2020-01-15 14:59:33 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Richard van der Hoff
d6752ce5da
Clean up startup for the pusher (#6558)
* Remove redundant python2 support code

`str.decode()` doesn't exist on python3, so presumably this code was doing
nothing

* Filter out pushers with corrupt data

When we get a row with unparsable json, drop the row, rather than returning a
row with null `data`, which will then cause an explosion later on.

* Improve logging when we can't start a pusher

Log the ID to help us understand the problem

* Make email pusher setup more robust

We know we'll have a `data` member, since that comes from the database. What we
*don't* know is if that is a dict, and if that has a `brand` member, and if
that member is a string.
2019-12-18 14:26:58 +00:00
Erik Johnston
c2f525a525 Don't call SQLBaseStore methods from outside stores 2019-12-04 16:23:43 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Richard van der Hoff
1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Amber Brown
020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston
69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Erik Johnston
d0d8a22c13 Quick fix to ensure cache descriptors always return deferreds 2019-10-28 13:33:04 +00:00
Andrew Morgan
1c9feadf4b Generalize email sending logging (#6075)
In ancient times Synapse would only send emails when it was notifying a user about a message they received...

Now it can do all sorts of neat things!

Change the logging so it's not just about notifications.
2019-09-23 14:38:19 +01:00
Andrew Morgan
df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Andrew Morgan
aeb40f355c
Ensure email validation link parameters are URL-encoded (#6063)
The validation links sent via email had their query parameters inserted without any URL-encoding. Surprisingly this didn't seem to cause any issues, but if a user were to put a `/` in their client_secret it could lead to problems.
2019-09-20 10:46:59 +01:00
Erik Johnston
05bae6b4fc Add opentracing span for HTTP push 2019-09-09 10:24:14 +01:00
Andrew Morgan
0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Erik Johnston
96bdd661b8
Remove redundant return 2019-08-13 12:50:36 +01:00
Erik Johnston
d02e41dcb2 Handle pusher being deleted during processing.
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
2019-08-01 13:44:12 +01:00
Erik Johnston
2e697d3013 Explicitly check that tombstone is a state event before notifying. 2019-07-31 16:32:03 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00