Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-22 00:34:59 -05:00
Code Issues Packages Projects Releases Wiki Activity
16,842 Commits 4 Branches 609 Tags 60 MiB
1de36407d1
Commit Graph

11832 Commits

Author SHA1 Message Date
Brendan Abolivier
c2ab0b3066
Whitelist the login fallback by default for SSO 2020-03-26 18:58:58 +01:00
Nektarios Katakis
825fb5d0a5
Don't default to an invalid sqlite config if no database configuration is provided (#6573) 2020-03-26 17:13:14 +00:00
Jason Robinson
060e7dce09 Allow RedirectResponse in SAML response handler 
Allow custom SAML handlers to redirect after processing an auth response.

Fixes #7149

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2020-03-26 19:02:35 +02:00
Dirk Klimpel
e8e2ddb60a
Allow server admins to define and enforce a password policy (MSC2000). (#7118) 2020-03-26 16:51:13 +00:00
Patrick Cloke
1c1242acba
Validate that the session is not modified during UI-Auth (#7068) 2020-03-26 07:39:34 -04:00
Aaron Raimist
6ca5e56fd1
Remove unused captcha_bypass_secret option (#7137) 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2020-03-25 17:49:34 +00:00
Erik Johnston
4cff617df1
Move catchup of replication streams to worker. (#7024) 
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
 2020-03-25 14:54:01 +00:00
Erik Johnston
b1cfaf08af
Merge pull request #7133 from matrix-org/erikj/fix_worker_startup 
Fix starting workers when federation sending not split out.
 2020-03-25 09:42:39 +00:00
Richard van der Hoff
39230d2171
Clean up some LoggingContext stuff (#7120) 
* Pull Sentinel out of LoggingContext

... and drop a few unnecessary references to it

* Factor out LoggingContext.current_context

move `current_context` and `set_context` out to top-level functions.

Mostly this means that I can more easily trace what's actually referring to
LoggingContext, but I think it's generally neater.

* move copy-to-parent into `stop`

this really just makes `start` and `stop` more symetric. It also means that it
behaves correctly if you manually `set_log_context` rather than using the
context manager.

* Replace `LoggingContext.alive` with `finished`

Turn `alive` into `finished` and make it a bit better defined.
 2020-03-24 14:45:33 +00:00
Naugrimm
1fcf9c6f95
Fix CAS redirect url (#6634) 
Build the same service URL when requesting the CAS ticket and when calling the proxyValidate URL.
 2020-03-24 11:59:04 +00:00
Erik Johnston
c816072d47 Fix starting workers when federation sending not split out. 2020-03-24 10:35:00 +00:00
Patrick Cloke
190ab593b7
Use the proper error code when a canonical alias that does not exist is used. (#7109) 2020-03-23 15:21:54 -04:00
Richard van der Hoff
a564b92d37
Convert *StreamRow classes to inner classes (#7116) 
This just helps keep the rows closer to their streams, so that it's easier to
see what the format of each stream is.
 2020-03-23 13:59:11 +00:00
Richard van der Hoff
5126cb1253 Merge branch 'master' into develop 2020-03-23 13:54:29 +00:00
Richard van der Hoff
2fa55c0cc6 1.12.0 2020-03-23 12:13:09 +00:00
Richard van der Hoff
b3cee0ce67
Fix processing of groups stream, and use symbolic names for streams (#7117) 
`groups` != `receipts`

Introduced in #6964
 2020-03-23 11:39:36 +00:00
Dionysis Grigoropoulos
96071eea8f
Set Referrer-Policy to no-referrer for media (#7009) 2020-03-23 09:48:28 +00:00
Patrick Cloke
477c4f5b1c
Clean-up some auth/login REST code (#7115) 2020-03-20 16:22:47 -04:00
Richard van der Hoff
c165c1233b
Improve database configuration docs (#6988) 
Attempts to clarify the sample config for databases, and add some stuff about
tcp keepalives to `postgres.md`.
 2020-03-20 15:24:22 +00:00
Erik Johnston
fdb1344716
Remove concept of a non-limited stream. (#7011) 2020-03-20 14:40:47 +00:00
Patrick Cloke
caec7d4fa0
Convert some of the media REST code to async/await (#7110) 2020-03-20 07:20:02 -04:00
Patrick Cloke
c2db6599c8
Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors (#7089). 2020-03-19 08:22:56 -04:00
Erik Johnston
a319cb1dd1
Change device list streams to have one row per ID (#7010) 
* Add 'device_lists_outbound_pokes' as extra table.

This makes sure we check all the relevant tables to get the current max
stream ID.

Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.

* Change device lists stream to have one row per id.

This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.

* Change device list replication to match new semantics.

Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).

* Newsfile

* Remove handling of multiple rows per ID

* Fix worker handling

* Comments from review
 2020-03-19 11:36:53 +00:00
Richard van der Hoff
e43e78b985 1.12.0rc1 2020-03-19 11:07:16 +00:00
Richard van der Hoff
e913823a22
Fix concurrent modification errors in pusher metrics (#7106) 
add a lock to try to make this metric actually work
 2020-03-19 10:28:49 +00:00
Richard van der Hoff
8c75667ad7
Add prometheus metrics for the number of active pushers (#7103) 2020-03-19 10:00:24 +00:00
Richard van der Hoff
443162e577
Move pusherpool startup into _base.setup (#7104) 
This should be safe to do on all workers/masters because it is guarded by
a config option which will ensure it is only actually done on the worker
assigned as a pusher.
 2020-03-19 09:48:45 +00:00
Erik Johnston
4a17a647a9
Improve get auth chain difference algorithm. (#7095) 
It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.

Instead, lets try and fetch the auth chains at the same time and
calculate the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.
 2020-03-18 16:46:41 +00:00
Patrick Cloke
88b41986db
Add an option to the set password API to choose whether to logout other devices. (#7085) 2020-03-18 07:50:00 -04:00
Erik Johnston
6e6476ef07 Comments from review 2020-03-18 10:13:55 +00:00
Richard van der Hoff
c37db0211e
Share SSL contexts for non-federation requests (#7094) 
Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests.

Fixes #7092.
 2020-03-17 21:32:25 +00:00
The Stranjer
5e477c1deb
Set charset to utf-8 when adding headers for certain text content types (#7044) 
Fixes #7043
 2020-03-17 13:29:09 +00:00
Patrick Cloke
7581d30e9f
Remove unused federation endpoint (query_auth) (#7026) 2020-03-17 08:04:49 -04:00
Patrick Cloke
60724c46b7
Remove special casing of m.room.aliases events (#7034) 2020-03-17 07:37:04 -04:00
Richard van der Hoff
6a35046363 Revert "Add options to disable setting profile info for prevent changes. (#7053)" 
This reverts commit 54dd28621b, reversing
changes made to 6640460d05.
 2020-03-17 11:25:01 +00:00
Brendan Abolivier
7df04ca0e6
Populate the room version from state events (#7070) 
Fixes #7065 

This is basically the same as https://github.com/matrix-org/synapse/pull/6847 except it tries to populate events from `state_events` rather than `current_state_events`, since the latter might have been cleared from the state of some rooms too early, leaving them with a `NULL` room version.
 2020-03-16 22:31:47 +00:00
Brendan Abolivier
beb19cf61a
Fix buggy condition in account validity handler (#7074) 2020-03-16 12:16:30 +00:00
Brendan Abolivier
d8d91983bc
Merge pull request #7067 from matrix-org/babolivier/saml_error_moar 
Move the default SAML2 error HTML to a dedicated file
 2020-03-13 19:53:19 +00:00
Brendan Abolivier
ebfcbbff9c
Use innerText instead of innerHTML 2020-03-13 19:09:22 +00:00
Patrick Cloke
77d0a4507b
Add type annotations and comments to auth handler (#7063) 2020-03-12 11:36:27 -04:00
Brendan Abolivier
0de9f9486a
Lint 2020-03-11 20:39:18 +00:00
Brendan Abolivier
f9e98176bf
Put the file in the templates directory 2020-03-11 20:31:42 +00:00
Brendan Abolivier
900bca9707
Update wording and config 2020-03-11 19:40:30 +00:00
Brendan Abolivier
b8cfe79ffc
Move the default SAML2 error HTML to a dedicated file 
Also add some JS to it to process any error we might have in the URI
(see #6893).
 2020-03-11 19:33:16 +00:00
Brendan Abolivier
8120a238a4
Refactor a bit 2020-03-11 18:49:41 +00:00
Brendan Abolivier
37a9873f63
Also don't fail on aliases events in this case 2020-03-11 18:43:41 +00:00
Brendan Abolivier
e38c44b418
Lint 2020-03-11 18:06:07 +00:00
Brendan Abolivier
2dce68c651
Also don't filter out events sent by ignored users when checking state visibility 2020-03-11 17:53:22 +00:00
Brendan Abolivier
9c0775e86a
Fix condition 2020-03-11 17:53:18 +00:00
Brendan Abolivier
69ce55c510
Don't filter out dummy events when we're checking the visibility of state 2020-03-11 17:52:54 +00:00
Brendan Abolivier
54dd28621b
Add options to disable setting profile info for prevent changes. (#7053) 2020-03-10 22:23:01 +00:00
Dirk Klimpel
42ac4ca477
Update synapse/config/registration.py 
Co-Authored-By: Brendan Abolivier <github@brendanabolivier.com>
 2020-03-10 21:26:55 +01:00
Brendan Abolivier
8f826f98ac
Rephrase default message 2020-03-10 17:22:45 +00:00
Brendan Abolivier
dc6fb56c5f
Hopefully mypy is happy now 2020-03-10 14:40:28 +00:00
Brendan Abolivier
fe593ef990
Attempt at appeasing the gods of mypy 2020-03-10 14:19:06 +00:00
Brendan Abolivier
5ec2077bf9
Lint 2020-03-10 14:04:20 +00:00
Brendan Abolivier
6b0efe73e2
SAML2: render a comprehensible error page if something goes wrong 
If an error happened while processing a SAML AuthN response, or a client
ends up doing a `GET` request to `/authn_response`, then render a
customisable error page rather than a confusing error.
 2020-03-10 13:59:22 +00:00
dklimpel
39f6595b4a lint, fix tests 2020-03-09 22:13:20 +01:00
dklimpel
885134529f updates after review 2020-03-09 22:09:29 +01:00
Brendan Abolivier
14b2ebe767
Merge pull request #7055 from matrix-org/babolivier/get_time_of_last_push_action_before 
Move get_time_of_last_push_action_before to the EventPushActionsWorkerStore
 2020-03-09 14:53:50 +00:00
Brendan Abolivier
aee2bae952
Fix undefined room_id in make_summary_text 
This would break notifications about un-named rooms when processing
notifications in a batch.
 2020-03-09 14:10:19 +00:00
Brendan Abolivier
87c65576e0
Move get_time_of_last_push_action_before to the EventPushActionsWorkerStore 
Fixes #7054

I also had a look at the rest of the functions in
`EventPushActionsStore` and in the push notifications send code and it
looks to me like there shouldn't be any other method with this issue in
this part of the codebase.
 2020-03-09 13:58:38 +00:00
Patrick Cloke
06eb5cae08
Remove special auth and redaction rules for aliases events in experimental room ver. (#7037) 2020-03-09 08:58:25 -04:00
Brendan Abolivier
bbf725e7da
Merge pull request #7045 from matrix-org/babolivier/room_keys_check 
Make sure that is_verified is a boolean when processing room keys
 2020-03-09 09:54:48 +00:00
dklimpel
99bbe177b6 add disable_3pid_changes 2020-03-08 21:58:12 +01:00
dklimpel
20545a2199 lint2 2020-03-08 15:28:00 +01:00
dklimpel
ce460dc31c lint 2020-03-08 15:22:43 +01:00
dklimpel
fb078f921b changelog 2020-03-08 15:19:07 +01:00
dklimpel
1f5f3ae8b1 Add options to disable setting profile info for prevent changes. 2020-03-08 14:49:33 +01:00
Neil Pilgrim
2bff4457d9
Add type hints to logging/context.py (#6309) 
* Add type hints to logging/context.py

Signed-off-by: neiljp (Neil Pilgrim) <github@kepier.clara.net>
 2020-03-07 17:57:26 +00:00
Neil Johnson
1d66dce83e
Break down monthly active users by appservice_id (#7030) 
* Break down monthly active users by appservice_id and emit via prometheus.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2020-03-06 18:14:19 +00:00
Brendan Abolivier
54b78a0e3b
Lint 2020-03-06 15:11:13 +00:00
Brendan Abolivier
297aaf4816
Mention the session ID in the error message 2020-03-06 15:07:41 +00:00
Brendan Abolivier
45df9d35a9
Lint 2020-03-06 11:10:52 +00:00
Brendan Abolivier
80e580ae92
Make sure that is_verified is a boolean when processing room keys 2020-03-06 11:05:00 +00:00
Patrick Cloke
87972f07e5
Convert remote key resource REST layer to async/await. (#7020) 2020-03-05 11:29:56 -05:00
Richard van der Hoff
78a15b1f9d
Store room_versions in EventBase objects (#6875) 
This is a bit fiddly because it all has to be done on one fell swoop:

* Wherever we create a new event, pass in the room version (and check it matches the format version)
* When we prune an event, use the room version of the unpruned event to create the pruned version.
* When we pass an event over the replication protocol, pass the room version over alongside it, and use it when deserialising the event again.
 2020-03-05 15:46:44 +00:00
Brendan Abolivier
31a2116331
Hide extremities dummy events from clients 2020-03-04 17:28:13 +00:00
Patrick Cloke
13892776ef
Allow deleting an alias if the user has sufficient power level (#6986) 2020-03-04 11:30:46 -05:00
Richard van der Hoff
8ef8fb2c1c
Read the room version from database when fetching events (#6874) 
This is a precursor to giving EventBase objects the knowledge of which room version they belong to.
 2020-03-04 13:11:04 +00:00
Brendan Abolivier
43f874055d
Merge branch 'master' into develop 2020-03-03 15:20:49 +00:00
Brendan Abolivier
fd983fad96 v1.11.1 2020-03-03 14:58:37 +00:00
Patrick Cloke
7dcbc33a1b
Validate the alt_aliases property of canonical alias events (#6971) 2020-03-03 07:12:45 -05:00
Brendan Abolivier
a0178df104 Fix wrong handler being used in SAML handler 2020-03-03 11:29:07 +00:00
Brendan Abolivier
65c73cdfec Factor out complete_sso_login and expose it to the Module API 2020-03-03 10:54:44 +00:00
Richard van der Hoff
b68041df3d Add a whitelist for the SSO confirmation step. 2020-03-02 17:05:09 +00:00
Erik Johnston
65a941d1f8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_devices_stream 2020-03-02 16:55:55 +00:00
Erik Johnston
b29474e0aa
Always return a deferred from get_current_state_deltas. (#7019) 
This currently causes presence notify code to log exceptions when there
is no state changes to process. This doesn't actually cause any problems
as we'd simply do nothing anyway.
 2020-03-02 16:52:15 +00:00
Richard van der Hoff
27d099edd6 Merge remote-tracking branch 'origin/release-v1.11.1' into release-v1.11.1 2020-03-02 16:43:33 +00:00
Brendan Abolivier
b2bd54a2e3 Add a confirmation step to the SSO login flow 2020-03-02 16:36:32 +00:00
Dirk Klimpel
bbeee33d63 Fixed set a user as an admin with the new API (#6928) 
Fix #6910
 2020-03-02 13:28:50 +00:00
Erik Johnston
e53744c737 Fix worker handling 2020-03-02 12:52:28 +00:00
Patrick Cloke
12d4259000
Add some type annotations to the federation base & client classes (#6995) 2020-02-28 07:31:07 -05:00
Erik Johnston
f70f44abc7 Remove handling of multiple rows per ID 2020-02-28 11:45:35 +00:00
Erik Johnston
9ce4e344a8 Change device list replication to match new semantics. 
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
 2020-02-28 11:25:34 +00:00
Erik Johnston
f5caa1864e Change device lists stream to have one row per id. 
This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.
 2020-02-28 11:21:25 +00:00
Erik Johnston
c3c6c0e622 Add 'device_lists_outbound_pokes' as extra table. 
This makes sure we check all the relevant tables to get the current max
stream ID.

Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.
 2020-02-28 11:15:11 +00:00
Dirk Klimpel
9b06d8f8a6
Fixed set a user as an admin with the new API (#6928) 
Fix #6910
 2020-02-28 09:58:05 +00:00
Patrick Cloke
ab0073a6c0 Merge remote-tracking branch 'origin/release-v1.11.1' into develop 2020-02-27 13:47:44 -05:00
Erik Johnston
2201bc9795
Don't refuse to start worker if media listener configured. (#7002) 
Instead lets just warn if the worker has a media listener configured but
has the media repository disabled.

Previously non media repository workers would just ignore the media
listener.
 2020-02-27 16:33:21 +00:00
Richard van der Hoff
132b673dbe
Add some type annotations in synapse.storage (#6987) 
I cracked, and added some type definitions in synapse.storage.
 2020-02-27 11:53:40 +00:00
Richard van der Hoff
3e99528f2b
Store room version on invite (#6983) 
When we get an invite over federation, store the room version in the rooms table.

The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...
 2020-02-26 16:58:33 +00:00
Patrick Cloke
380122866f
Cast a coroutine into a Deferred in the federation base (#6996) 
Properly convert a coroutine into a Deferred in federation_base to fix an error when joining a room.
 2020-02-26 11:32:13 -05:00
Erik Johnston
1f773eec91
Port PresenceHandler to async/await (#6991) 2020-02-26 15:33:26 +00:00
Uday Bansal
7728d87fd7
Updated warning for incorrect database collation/ctype (#6985) 
Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
 2020-02-26 15:17:03 +00:00
Andrew Morgan
8c75b621bf
Ensure 'deactivated' parameter is a boolean on user admin API, Fix error handling of call to deactivate user (#6990) 2020-02-26 12:22:55 +00:00
Richard van der Hoff
e66f099ca9
Sanity-check database before running upgrades (#6982) 
Some of the database deltas rely on `config.server_name` being set correctly,
so we should check that it is before running the deltas.

Fixes #6870.
 2020-02-25 17:46:00 +00:00
Erik Johnston
bbf8886a05
Merge worker apps into one. (#6964) 2020-02-25 16:56:55 +00:00
Richard van der Hoff
691659568f
Remove redundant store_room call (#6979) 
`_process_received_pdu` is only called by `on_receive_pdu`, which ignores any
events for unknown rooms, so this is redundant.
 2020-02-24 17:20:44 +00:00
Richard van der Hoff
a301934f46
Upsert room version when we join over federation (#6968) 
This is intended as a precursor to storing room versions when we receive an
invite over federation, but has the happy side-effect of fixing #3374 at last.

In short: change the store_room with try/except to a proper upsert which
updates the right columns.
 2020-02-24 15:46:41 +00:00
Richard van der Hoff
4c2ed3f20e
Fix minor issues with email config (#6962) 
* Give `notif_template_html`, `notif_template_text` default values (fixes #6960)
 * Don't complain if `smtp_host` and `smtp_port` are unset, since they have sensible defaults (fixes #6961)
 * Set the example for `enable_notifs` to `True`, for consistency and because it's more useful
 * Raise errors as ConfigError rather than RuntimeError for nicer formatting
 2020-02-24 15:18:38 +00:00
Patrick Cloke
af6c389501
No longer use room alias events to calculate room names for push notifications. (#6966) 2020-02-21 12:50:48 -05:00
Dirk Klimpel
7b0e2d961c
Change displayname of user as admin in rooms (#6876) 2020-02-21 17:44:03 +00:00
Patrick Cloke
fcf4599488
Stop returning aliases as part of the room list. (#6970) 2020-02-21 12:40:23 -05:00
Patrick Cloke
7936d2a96e
Publishing/removing from the directory requires a power level greater than canonical aliases. 2020-02-21 07:18:33 -05:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) 
Ensure good comprehension hygiene using flake8-comprehensions.
 2020-02-21 07:15:07 -05:00
Richard van der Hoff
272eee1ae1
Merge pull request #6967 from matrix-org/rav/increase_max_events_behind 
Increase MAX_EVENTS_BEHIND for replication clients
 2020-02-21 10:17:28 +00:00
Richard van der Hoff
4f7e4fc2fb Merge branch 'master' into develop 2020-02-21 09:37:03 +00:00
Erik Johnston
0bd8cf435e Increase MAX_EVENTS_BEHIND for replication clients 2020-02-21 09:04:33 +00:00
Richard van der Hoff
9c1b83b007 1.11.0 2020-02-21 08:56:04 +00:00
Andrew Morgan
8f6d9c4cf0
Small grammar fixes to the ACME v1 deprecation notice (#6944) 
Some small fixes to the copy in #6907.
 2020-02-21 08:53:01 +00:00
Patrick Cloke
99eed85a77
Do not send alias events when creating / upgrading a room (#6941) 
Stop emitting room alias update events during room creation/upgrade.
 2020-02-20 16:24:04 -05:00
Hubert Chathi
a90d0dc5c2
don't insert into the device table for remote cross-signing keys (#6956) 2020-02-20 09:59:00 -05:00
Ruben Barkow-Kuder
4fb5f4d0ce
Add some clarifications to README.md in the database schema directory. (#6615) 
Signed-off-by: Ruben Barkow-Kuder <github@r.z11.de>
 2020-02-20 10:37:57 +00:00
Erik Johnston
7b7c3cedf2 Minor perf fixes to get_auth_chain_ids. 2020-02-19 15:47:11 +00:00
Erik Johnston
fc87d2ffb3
Freeze allocated objects on startup. (#6953) 
This may make gc go a bit faster as the gc will know things like
caches/data stores etc. are frozen without having to check.
 2020-02-19 15:09:00 +00:00
Erik Johnston
2b37eabca1
Reduce auth chains fetched during v2 state res. (#6952) 
The state res v2 algorithm only cares about the difference between auth
chains, so we can pass in the known common state to the `get_auth_chain`
storage function so that it can ignore those events.
 2020-02-19 15:04:47 +00:00
Richard van der Hoff
197b08de35 1.11.0rc1 2020-02-19 13:48:32 +00:00
Erik Johnston
099c96b89b
Revert get_auth_chain_ids changes (#6951) 2020-02-19 11:37:35 +00:00
Richard van der Hoff
2fb7794e60
Merge pull request #6949 from matrix-org/rav/list_room_aliases_peekable 
Make room alias lists peekable
 2020-02-19 11:19:11 +00:00
Richard van der Hoff
880aaac1d8
Move MSC2432 stuff onto unstable prefix (#6948) 
it's not in the spec yet, so needs to be unstable. Also add a feature flag for it. Also add a test for admin users.
 2020-02-19 10:40:27 +00:00
Richard van der Hoff
abf1e5c526
Tiny optimisation for _get_handler_for_request (#6950) 
we have hundreds of path_regexes (see #5118), so let's not convert the same
bytes to str for each of them.
 2020-02-19 10:38:20 +00:00
Erik Johnston
0d0bc35792
Increase DB/CPU perf of _is_server_still_joined check. (#6936) 
* Increase DB/CPU perf of `_is_server_still_joined` check.

For rooms with large amount of state a single user leaving could cause
us to go and load a lot of membership events and then pull out
membership state in a large number of batches.

* Newsfile

* Update synapse/storage/persist_events.py

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Fix adding if too soon

* Update docstring

* Review comments

* Woops typo

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-19 10:15:49 +00:00
Brendan Abolivier
5e4a438556
Merge pull request #6945 from matrix-org/babolivier/fix-retention-debug-log 
Fix log in message retention purge jobs
 2020-02-19 10:12:55 +00:00
Erik Johnston
5a5abd55e8
Limit size of get_auth_chain_ids query (#6947) 2020-02-19 09:39:26 +00:00
Richard van der Hoff
709e81f518 Make room alias lists peekable 
As per
https://github.com/matrix-org/matrix-doc/pull/2432#pullrequestreview-360566830,
make room alias lists accessible to users outside world_readable rooms.
 2020-02-19 08:53:32 +00:00
Richard van der Hoff
a0a1fd0bec Add allow_departed_users param to check_in_room_or_world_readable 
... and set it everywhere it's called.

while we're here, rename it for consistency with `check_user_in_room` (and to
help check that I haven't missed any instances)
 2020-02-19 08:52:51 +00:00
Richard van der Hoff
b58d17e44f Refactor the membership check methods in Auth 
these were getting a bit unwieldy, so let's combine `check_joined_room` and
`check_user_was_in_room` into a single `check_user_in_room`.
 2020-02-18 23:21:44 +00:00
Brendan Abolivier
f31a94a6dd
Fix log in message retention purge jobs 2020-02-18 17:29:57 +00:00
Richard van der Hoff
adfaea8c69
Implement GET /_matrix/client/r0/rooms/{roomId}/aliases (#6939) 
per matrix-org/matrix-doc#2432
 2020-02-18 16:23:25 +00:00
Richard van der Hoff
3f1cd14791
Merge pull request #6872 from matrix-org/rav/dictproperty 
Rewrite _EventInternalMetadata to back it with a dict
 2020-02-18 16:21:02 +00:00
Brendan Abolivier
d484126bf7
Merge pull request #6907 from matrix-org/babolivier/acme-config 
Add mention and warning about ACME v1 deprecation to the TLS config
 2020-02-18 16:11:31 +00:00
Erik Johnston
8a380d0fe2
Increase perf of get_auth_chain_ids used in state res v2. (#6937) 
We do this by moving the recursive query to be fully in the DB.
 2020-02-18 15:39:09 +00:00
Brendan Abolivier
9801a042f3
Make the log more noticeable 2020-02-18 15:15:43 +00:00
Brendan Abolivier
bfbe2f5b08
Print the error as an error log and raise the same exception we got 2020-02-18 15:10:41 +00:00
Patrick Cloke
fe3941f6e3
Stop sending events when creating or deleting aliases (#6904) 
Stop sending events when creating or deleting associations (room aliases). Send an updated canonical alias event if one of the alt_aliases is deleted.
 2020-02-18 07:29:44 -05:00
Richard van der Hoff
0a714c3abf Merge branch 'master' into develop 2020-02-17 16:33:21 +00:00
Richard van der Hoff
fd6d83ed96 1.10.1 2020-02-17 16:27:33 +00:00
Richard van der Hoff
d2455ec3aa wait for current_state_events_membership before delete_old_current_state_events (#6924) 2020-02-17 16:19:32 +00:00
Andrew Morgan
3404ad289b
Raise the default power levels for invites, tombstones and server acls (#6834) 2020-02-17 13:23:37 +00:00
Richard van der Hoff
46fa66bbfd
wait for current_state_events_membership before delete_old_current_state_events (#6924) 2020-02-17 11:30:50 +00:00
Patrick Cloke
10027c80b0
Add type hints to the spam check module (#6915) 
Add typing information to the spam checker modules.
 2020-02-14 12:49:40 -05:00
Richard van der Hoff
9551911f88 Rewrite _EventInternalMetadata to back it with a _dict 
Mostly, this gives mypy an easier time.
 2020-02-14 16:42:40 +00:00
Richard van der Hoff
43b2be9764 Replace _event_dict_property with DictProperty 
this amounts to the same thing, but replaces `_event_dict` with `_dict`, and
removes some of the function layers generated by `property`.
 2020-02-14 16:42:37 +00:00
Richard van der Hoff
97a42bbc3a
Add a warning about indentation to generated config (#6920) 
Fixes #6916.
 2020-02-14 16:22:30 +00:00
Patrick Cloke
49f877d32e
Filter the results of user directory searching via the spam checker (#6888) 
Add a method to the spam checker to filter the user directory results.
 2020-02-14 07:17:54 -05:00
Brendan Abolivier
36af094017
Linters are hard but in they end they just want what's best for us 2020-02-13 17:03:41 +00:00
Brendan Abolivier
65bdc35a1f
Lint 2020-02-13 16:14:15 +00:00
Brendan Abolivier
0cb83cde70
Lint 2020-02-13 16:06:31 +00:00
Brendan Abolivier
ef9c275d96
Add a separator for the config warning 2020-02-13 15:44:14 +00:00
Brendan Abolivier
12bbcc255a
Add a comprehensive error when failing to register for an ACME account 2020-02-13 14:58:34 +00:00
Brendan Abolivier
5820ed905f
Add mention and warning about ACME v1 deprecation to the Synapse config 2020-02-13 14:20:08 +00:00
Aaron Raimist
dc3f998706
Remove m.lazy_load_members from unstable features since it is in CS r0.5.0 (#6877) 
Fixes #5528
 2020-02-13 12:02:32 +00:00
Andrew Morgan
d8994942f2
Return a 404 for admin api user lookup if user not found (#6901) 2020-02-12 18:14:10 +00:00
Brendan Abolivier
47acbc519f Merge branch 'master' into develop 2020-02-12 13:24:09 +00:00
Brendan Abolivier
fdb816713a 1.10.0 2020-02-12 12:19:19 +00:00
Richard van der Hoff
88d41e94f5 Merge branch 'release-v1.10.0' into develop 2020-02-11 11:12:31 +00:00
Richard van der Hoff
856b2a9555 1.10.0rc5 2020-02-11 11:06:28 +00:00
Richard van der Hoff
605cd089f7 Merge branch 'release-v1.10.0' into develop 2020-02-11 10:43:47 +00:00
Richard van der Hoff
3edc65dd24 1.10.0rc4 2020-02-11 10:43:16 +00:00
Patrick Cloke
a92e703ab9
Reject device display names that are too long (#6882) 
* Reject device display names that are too long.

Too long is currently defined as 100 characters in length.

* Add a regression test for rejecting a too long device display name.
 2020-02-10 16:35:26 -05:00
Matthew Hodgson
01209382fb
filter out m.room.aliases from /sync state blocks (#6884) 
We forgot to filter out aliases from /sync state blocks as well as the timeline.
 2020-02-10 18:07:35 +00:00
Richard van der Hoff
db0fee738d Synapse 1.10.0rc3 (2020-02-10) 
==============================
 
 Features
 --------
 
 - Filter out m.room.aliases from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))
 
 Internal Changes
 ----------------
 
 - Fix continuous integration failures with old versions of `pip`, which were introduced by a release of the `zipp` library. ([\#6880](https://github.com/matrix-org/synapse/issues/6880))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl5BLBwACgkQOSor00I9
 eP/9Ogf9HZ+XD5tt1uYkGqDSvUBS4sHdNcvxpNf1UMo6VaY4K5b9kF28MjY07ur2
 nGq/nnLyLZZzwVXtV3q8upIdIK1hcMp2HCU0qDhEljepOkGmyQTqZooxrdrLTjPS
 232bbifB6VFdK8P4gPIGopCRQI9a+TII4jODBobiJR43e0gEPfSREJKAnJs7DYaD
 Quf3Svzu+jB+X8ymWLVbCFXuk4A4UqPzrRPb8TQBDN3w35G4JBDAgIpdxhQdJpdR
 RIS1HgSm95P03doS23RLUjXuIpBfYveyStuGfPx4P9Lf30M2o6bxDeTPo2+IGbST
 yhl1jB0l8Ve5/6veuTieA6vuXWU55g==
 =2rgX
 -----END PGP SIGNATURE-----

Merge tag 'v1.10.0rc3' into develop

Synapse 1.10.0rc3 (2020-02-10)
==============================

Features
--------

- Filter out m.room.aliases from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))

Internal Changes
----------------

- Fix continuous integration failures with old versions of `pip`, which were introduced by a release of the `zipp` library. ([\#6880](https://github.com/matrix-org/synapse/issues/6880))
 2020-02-10 10:15:32 +00:00
Richard van der Hoff
3de57e7062 1.10.0rc3 2020-02-10 09:56:42 +00:00
Matthew Hodgson
8e64c5a24c filter out m.room.aliases from the CS API until a better solution is specced (#6878) 
We're in the middle of properly mitigating spam caused by malicious aliases being added to a room. However, until this work fully lands, we temporarily filter out all m.room.aliases events from /sync and /messages on the CS API, to remove abusive aliases. This is considered acceptable as m.room.aliases events were never a reliable record of the given alias->id mapping and were purely informational, and in their current state do more harm than good.
 2020-02-10 09:36:23 +00:00
Erik Johnston
21db35f77e
Add support for putting fed user query API on workers (#6873) 2020-02-07 15:45:39 +00:00
Richard van der Hoff
e1d858984d
Remove unused get_room_stats_state method. (#6869) 2020-02-07 15:30:26 +00:00
Richard van der Hoff
799001f2c0
Add a make_event_from_dict method (#6858) 
... and use it in places where it's trivial to do so.

This will make it easier to pass room versions into the FrozenEvent
constructors.
 2020-02-07 15:30:04 +00:00
Erik Johnston
b08b0a22d5
Add typing to synapse.federation.sender (#6871) 2020-02-07 13:56:38 +00:00
Erik Johnston
de2d267375
Allow moving group read APIs to workers (#6866) 2020-02-07 11:14:19 +00:00
Dirk Klimpel
56ca93ef59
Admin api to add an email address (#6789) 2020-02-07 10:29:36 +00:00
Richard van der Hoff
f4884444c3
remove unused room_version_to_event_format (#6857) 2020-02-07 09:26:57 +00:00
Richard van der Hoff
e1b240329e
Merge pull request #6856 from matrix-org/rav/redact_changes/6 
Pass room_version into `event_from_pdu_json`
 2020-02-07 09:22:15 +00:00
Patrick Cloke
7765bf3989
Limit the number of events that can be requested when backfilling events (#6864) 
Limit the maximum number of events requested when backfilling events.
 2020-02-06 13:25:24 -05:00
Richard van der Hoff
928edef979 Pass room_version into event_from_pdu_json 
It's called from all over the shop, so this one's a bit messy.
 2020-02-06 16:08:27 +00:00
Richard van der Hoff
b0c8bdd49d
pass room version into FederationClient.send_join (#6854) 
... which allows us to sanity-check the create event.
 2020-02-06 15:50:39 +00:00
timfi
bce557175b
Allow empty federation_certificate_verification_whitelist (#6849) 2020-02-06 14:45:01 +00:00
PeerD
99fcc96289
Third party event rules Update (#6781) 2020-02-06 14:15:29 +00:00
Erik Johnston
ed630ea17c
Reduce amount of logging at INFO level. (#6862) 
A lot of the things we log at INFO are now a bit superfluous, so lets
make them DEBUG logs to reduce the amount we log by default.

Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
 2020-02-06 13:31:05 +00:00
Richard van der Hoff
9bcd37146e
Merge pull request #6823 from matrix-org/rav/redact_changes/5 
pass room versions around
 2020-02-06 11:32:33 +00:00
Erik Johnston
2201ef8556 Synapse 1.10.0rc2 (2020-02-06) 
==============================
 
 Bugfixes
 --------
 
 - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
 - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))
 
 Internal Changes
 ----------------
 
 - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl478l8QHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHmKfD/0ZpBV95gMk0iecfvO6IY8+0KigzDUypgXf
 zp1k+k5DBUmH5/83h7+cXWHIJyofv1At1YdIq9J/nNeyqr3V3dcQwPabOKYbI96d
 kVD0EZX+2NjuYejAuF9eJNdiwRq5yMLluCfnOXr5jCS0NNdpO1xVb4bWYm50RmcD
 WJvBvmfwXwCC3AFyNb4IAipaVUXuVXXx1YhjFs6DsbtpPUG88e4uIhpfvrS4v0t8
 GFJZNuVJStOvyKHHTfNRIMkhy4xidj3HRUMSmjNpx07WnPBnbv4LnUOD1boYxgaP
 EoODYnoAPshdiKB0AywNNjue3TmFD/Z7vVEzlFP/lNZ4GDU4kN9C/EwFYw0C2Jqq
 f9O/E2ZuP9Qqume5O9UwMQQAmhV5lMBaIsYRbixU+9bSB893zRHRffA11HypzD00
 ZXj8QDOXpiXp2jpAwN1Rk9e/aZEX3qd+zUAgRmk+kYb0KTC2/gcY956rodNSSO/U
 RFYg4DFvoCgCrRSxZ4LQMlFu3YY2E7qWH+p/OHk3WG79jW64VpEFaytvv8fR+GKq
 g3EbtWy6mUzlKYqbjZ+ZTUDe5AWdv6ZX8xJqRD/S6cpiwyh6Gp89HHNvBThXoCmK
 fxkgw8if7eIITuTlNuDrqunxyWqwd3oVlzd2mi2bg0yRfcqJ9C6OuBV1VTLFZeky
 3sjCiU0IRw==
 =kcSy
 -----END PGP SIGNATURE-----

Merge tag 'v1.10.0rc2' into develop

Synapse 1.10.0rc2 (2020-02-06)
==============================

Bugfixes
--------

- Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
- Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))

Internal Changes
----------------

- Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
 2020-02-06 11:04:03 +00:00
Erik Johnston
6a7e90ad78 1.10.0rc2 2020-02-06 10:40:08 +00:00
Richard van der Hoff
ee0525b2b2 Simplify room_version handling in FederationClient.send_invite 2020-02-05 17:43:57 +00:00
Richard van der Hoff
f84700fba8 Pass room version object into FederationClient.get_pdu 2020-02-05 17:25:46 +00:00
Richard van der Hoff
577f460369
Merge pull request #6840 from matrix-org/rav/federation_client_async 
Port much of `synapse.federation.federation_client` to async/await
 2020-02-05 16:56:39 +00:00
Richard van der Hoff
6bbd890f05 make FederationClient._do_send_invite async 2020-02-05 15:50:31 +00:00
Richard van der Hoff
146fec0820
Apply suggestions from code review 
Co-Authored-By: Erik Johnston <erik@matrix.org>
 2020-02-05 15:47:00 +00:00
Erik Johnston
a58860e480
Check sender_key matches on inbound encrypted events. (#6850) 
If they don't then the device lists are probably out of sync.
 2020-02-05 14:02:39 +00:00
Hubert Chathi
60d0672426 Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed 
add device signatures to device key query results
 2020-02-05 10:54:49 +00:00
Michael Kaye
a831d2e4e3
Reduce performance logging to DEBUG (#6833) 
* Reduce tnx performance logging to DEBUG
* Changelog.d
 2020-02-05 08:57:37 +00:00
Richard van der Hoff
d88e0ec080
Database updates to populate rooms.room_version (#6847) 
We're going to need this so that we can figure out how to handle redactions when fetching events from the database.
 2020-02-04 21:31:08 +00:00
Erik Johnston
6475382d80
Fix detecting unknown devices from remote encrypted events. (#6848) 
We were looking at the wrong event type (`m.room.encryption` vs
`m.room.encrypted`).

Also fixup the duplicate `EvenTypes` entries.

Introduced in #6776.
 2020-02-04 17:25:54 +00:00
Hubert Chathi
74bf3fdbb9
Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed 
add device signatures to device key query results
 2020-02-04 12:03:54 -05:00
Richard van der Hoff
5ef91b96f1 Merge remote-tracking branch 'origin/develop' into rav/federation_client_async 2020-02-04 12:07:05 +00:00
Richard van der Hoff
c7d6d5c69e
Merge pull request #6837 from matrix-org/rav/federation_async 
Port much of `synapse.handlers.federation` to async/await.
 2020-02-04 12:06:18 +00:00
Hubert Chathi
23d8a55c7a add device signatures to device key query results 2020-02-04 00:13:12 -05:00
Richard van der Hoff
ea23210b2d make FederationClient.send_invite async 2020-02-03 22:29:49 +00:00
Richard van der Hoff
6deeefb68c make FederationClient.get_missing_events async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
abadf44eb2 make FederationClient._do_send_leave async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
e88b90aaeb make FederationClient.send_leave.send_request async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
638001116d make FederationClient._do_send_join async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
3960527c2e make FederationClient.send_join.send_request async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
ad09ee9262 make FederationClient.make_membership_event.send_request async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
1330c311b7 make FederationClient._try_destination_list async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
a46fabf17b make FederationClient.send_leave async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
8af9f11bea make FederationClient.send_join async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
3f11cbb404 make FederationClient.make_membership_event async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
24d814ca23 make FederationClient.get_event_auth async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
d73683c363 make FederationClient.get_room_state_ids async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
0cb0c7bcd5 make FederationClient.get_pdu async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
0536d0c9be make FederationClient.backfill async 2020-02-03 22:28:45 +00:00
Richard van der Hoff
5d17c31596 make FederationHandler.send_invite async 2020-02-03 22:28:11 +00:00
Richard van der Hoff
e81c093974 make FederationHandler.on_get_missing_events async 2020-02-03 19:15:08 +00:00
Erik Johnston
b9391c9575
Add typing to SyncHandler (#6821) 
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-03 18:05:44 +00:00
Erik Johnston
ae5b3104f0
Fix stacktraces when using ObservableDeferred and async/await (#6836) 2020-02-03 17:10:54 +00:00
Richard van der Hoff
f64c96662e make FederationHandler.user_joined_room async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
52642860da make FederationHandler._clean_room_for_join async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
814cc00cb9 make FederationHandler._notify_persisted_event async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
05299599b6 make FederationHandler.persist_events_and_notify async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
3b7e0e002b make FederationHandler._make_and_verify_event async 2020-02-03 16:22:30 +00:00
Richard van der Hoff
4286e429a7 make FederationHandler.do_remotely_reject_invite async 2020-02-03 16:19:18 +00:00
Richard van der Hoff
c3f296af32 make FederationHandler._check_for_soft_fail async 2020-02-03 16:16:31 +00:00
Richard van der Hoff
dbdf843012 make FederationHandler._persist_auth_tree async 2020-02-03 16:14:58 +00:00
Richard van der Hoff
ebd6a15af3 make FederationHandler.do_invite_join async 2020-02-03 16:13:13 +00:00
Richard van der Hoff
94f7b4cd54 make FederationHandler.on_event_auth async 2020-02-03 16:06:46 +00:00
Richard van der Hoff
863087d186 make FederationHandler.on_exchange_third_party_invite_request async 2020-02-03 16:02:50 +00:00
Richard van der Hoff
957129f4a7 make FederationHandler.construct_auth_difference async 2020-02-03 16:00:46 +00:00
Richard van der Hoff
0d5f2f4bb0 make FederationHandler._update_context_for_auth_events async 2020-02-03 15:55:35 +00:00
Richard van der Hoff
a25ddf26a3 make FederationHandler._update_auth_events_and_context_for_auth async 2020-02-03 15:53:54 +00:00
Richard van der Hoff
bc9b75c6f0 make FederationHandler.do_auth async 2020-02-03 15:51:24 +00:00
Richard van der Hoff
8033b257a7 make FederationHandler._prep_event async 2020-02-03 15:49:32 +00:00
Richard van der Hoff
1cdc253e0a make FederationHandler._handle_new_event async 2020-02-03 15:48:33 +00:00
Richard van der Hoff
c556ed9e15 make FederationHandler._handle_new_events async 2020-02-03 15:43:51 +00:00
Richard van der Hoff
6e89ec5e32 make FederationHandler.on_make_leave_request async 2020-02-03 15:40:41 +00:00
Richard van der Hoff
d184cbc031 make FederationHandler.on_send_leave_request async 2020-02-03 15:39:24 +00:00
Richard van der Hoff
98681f90cb make FederationHandler.on_make_join_request async 2020-02-03 15:38:02 +00:00
Richard van der Hoff
af8ba6b525 make FederationHandler.on_invite_request async 2020-02-03 15:33:42 +00:00
Richard van der Hoff
7571bf86f0 make FederationHandler.on_send_join_request async 2020-02-03 15:32:48 +00:00
Richard van der Hoff
b3e44f0bdf make FederationHandler.on_query_auth async 2020-02-03 15:30:23 +00:00
Andrew Morgan
370080531e
Allow URL-encoded user IDs on user admin api paths (#6825) 2020-02-03 13:18:42 +00:00