Commit Graph

590 Commits

Author SHA1 Message Date
David Robertson
797ee7812d
Relax ignore-missing-imports for modules that have stubs now and update mypy ()
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
Erik Johnston
707d5e4e48
Encode JSON responses on a thread in C, mk2 ()
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.

Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:

1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.

I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
2021-09-28 09:37:58 +00:00
Patrick Cloke
47854c71e9
Use direct references for configuration variables (part 4). () 2021-09-23 12:03:01 -04:00
Patrick Cloke
e584534403
Use direct references for some configuration variables (part 3) ()
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.

It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
Patrick Cloke
8c7a531e27
Use direct references for some configuration variables (part 2) () 2021-09-15 08:34:52 -04:00
Patrick Cloke
01c88a09cd
Use direct references for some configuration variables ()
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
David Robertson
d9069388f3
Correctly include room avatars in email notifications ()
Judging by the template, this was intended ages ago, but we never
actually passed an avatar URL to the template. So let's provide one.

Closes .

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-09-01 13:48:41 +01:00
V02460
b7f7ca24b1
Remove shebang line from module files ()
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2021-07-29 21:34:14 +01:00
Patrick Cloke
d15e72e511
Update the notification email subject when invited to a space. () 2021-07-21 17:29:54 +00:00
Brendan Abolivier
36dc15412d
Add a module type for account validity ()
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.

To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16 18:11:53 +02:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) () 2021-07-15 11:02:43 +01:00
Patrick Cloke
8d609435c0
Move methods involving event authentication to EventAuthHandler. ()
Instead of mixing them with user authentication methods.
2021-07-01 14:25:37 -04:00
Brendan Abolivier
41ac128fd3
Split multiplart email sending into a dedicated handler ()
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-05-17 12:33:38 +02:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Erik Johnston
3853a7edfc
Only store data in caches, not "smart" objects () 2021-04-23 11:47:07 +01:00
Erik Johnston
177dae2704
Limit length of accepted email addresses () 2021-04-22 17:49:11 +01:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page " from synapse-dinsic ()
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. () 2021-03-23 07:12:48 -04:00
Ankit Dobhal
d66f9070cd
Fixed code misc. quality issues ()
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
2021-03-22 11:18:13 -04:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. () 2021-03-12 11:37:57 -05:00
Patrick Cloke
33a02f0f52
Fix additional type hints from Twisted upgrade. () 2021-03-03 15:47:38 -05:00
Erik Johnston
2566dc57ce
Test that we require validated email for email pushers () 2021-02-25 15:35:14 +00:00
Erik Johnston
2927921942
Clean up ShardedWorkerHandlingConfig ()
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00
Erik Johnston
66f4949e7f
Fix deleting pushers when using sharded pushers. () 2021-02-22 21:14:42 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ()
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
7950aa8a27 Fix some typos. 2021-02-12 11:14:12 -05:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. ()
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
2021-02-04 10:18:25 -05:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. ()
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
  was tweaked to depend on whether there was 1 sender or multiple senders,
  not based on if there was 1 room or multiple rooms.
2021-02-01 13:09:39 -05:00
Patrick Cloke
e54746bdf7
Clean-up the template loading code. ()
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
2021-01-27 10:59:50 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. ()
Treat the content as untrusted and do not assume it is of
the proper form.
2021-01-25 07:27:16 -05:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms ()
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-01-07 13:03:38 +00:00
Patrick Cloke
37eaf9c272
Fix-up assertions about last stream token in push ()
The last stream token is always known and we do not need to handle none.
2021-01-05 10:53:15 -05:00
Patrick Cloke
4136255d3c
Ensure that a URL exists in the content during push. ()
This fixes an KeyError exception, after this PR the content
is just considered unknown.
2020-12-18 07:26:15 -05:00
Patrick Cloke
bd30cfe86a
Convert internal pusher dicts to attrs classes. ()
This improves type hinting and should use less memory.
2020-12-16 11:25:30 -05:00
Patrick Cloke
b3a4b53587
Fix handling of stream tokens for push. ()
Removes faulty assertions and fixes the logic to ensure the max
stream token is always set.
2020-12-15 10:41:34 -05:00
Patrick Cloke
5d34f40d49
Add type hints to the push module. () 2020-12-11 11:43:53 -05:00
Patrick Cloke
92d87c6882
Add type hints for HTTP and email pushers. () 2020-12-07 09:59:38 -05:00
Patrick Cloke
02e588856a
Add type hints to the push mailer module. () 2020-12-07 07:10:22 -05:00
Patrick Cloke
b774c555d8
Add additional validation to pusher URLs. ()
Pusher URLs now must end in `/_matrix/push/v1/notify` per the
specification.
2020-12-04 10:51:56 -05:00
Patrick Cloke
30fba62108
Apply an IP range blacklist to push and key revocation requests. ()
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Andrew Morgan
17fa58bdd1
Add a config option to change whether unread push notification counts are per-message or per-room ()
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).

However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
2020-11-30 18:43:54 +00:00
David Baker
59cc2472b3
Add base pushrule to notify for jitsi conferences ()
This could be customised to trigger a different kind of notification in the future, but for now it's a normal non-highlight one.
2020-11-02 16:36:14 +00:00
Richard van der Hoff
cbc82aa09f
Implement and use an @lru_cache decorator ()
We don't always need the full power of a DeferredCache.
2020-10-30 11:43:17 +00:00
Erik Johnston
2b7c180879
Start fewer opentracing spans ()
 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.

This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.

I don't really know how much this will help.
2020-10-26 09:30:19 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. () 2020-10-23 12:38:40 -04:00
Erik Johnston
db9ef792f0
Fix email notifications for invites without local state. ()
This can happen if e.g. the room invited into is no longer on the
server (or if all users left the room).
2020-10-23 10:41:32 +01:00
Richard van der Hoff
903d11c43a
Add DeferredCache.get_immediate method ()
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
2020-10-19 15:00:12 +01:00
Patrick Cloke
c356b4bf42
Include a simple message in email notifications that include encrypted content () 2020-10-19 09:12:39 -04:00
Richard van der Hoff
995cc615a0
Apply suggestions from code review
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-16 16:14:42 +01:00