Commit Graph

826 Commits

Author SHA1 Message Date
Erik Johnston
e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Erik Johnston
1531b214fc
Add ability to wait for replication streams (#7542)
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
Erik Johnston
547e4dd83e
Fix exception reporting due to HTTP request errors. (#7556)
These are business as usual errors, rather than stuff we want to log at
error.
2020-05-22 11:39:20 +01:00
Patrick Cloke
8c8858e124
Convert federation handler to async/await. (#7459) 2020-05-11 15:12:46 -04:00
Andrew Morgan
6b22921b19
async/await is_server_admin (#7363) 2020-05-01 15:15:36 +01:00
Patrick Cloke
33bceb7f70
Convert some of the federation handler methods to async/await. (#7338) 2020-04-24 14:36:38 -04:00
Richard van der Hoff
7966a1cde9
Rewrite prune_old_outbound_device_pokes for efficiency (#7159)
make sure we clear out all but one update for the user
2020-03-30 19:06:52 +01:00
Richard van der Hoff
3e99528f2b
Store room version on invite (#6983)
When we get an invite over federation, store the room version in the rooms table.

The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...
2020-02-26 16:58:33 +00:00
Richard van der Hoff
691659568f
Remove redundant store_room call (#6979)
`_process_received_pdu` is only called by `on_receive_pdu`, which ignores any
events for unknown rooms, so this is redundant.
2020-02-24 17:20:44 +00:00
Richard van der Hoff
a301934f46
Upsert room version when we join over federation (#6968)
This is intended as a precursor to storing room versions when we receive an
invite over federation, but has the happy side-effect of fixing #3374 at last.

In short: change the store_room with try/except to a proper upsert which
updates the right columns.
2020-02-24 15:46:41 +00:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Patrick Cloke
7765bf3989
Limit the number of events that can be requested when backfilling events (#6864)
Limit the maximum number of events requested when backfilling events.
2020-02-06 13:25:24 -05:00
Richard van der Hoff
b0c8bdd49d
pass room version into FederationClient.send_join (#6854)
... which allows us to sanity-check the create event.
2020-02-06 15:50:39 +00:00
Richard van der Hoff
9bcd37146e
Merge pull request #6823 from matrix-org/rav/redact_changes/5
pass room versions around
2020-02-06 11:32:33 +00:00
Erik Johnston
2201ef8556 Synapse 1.10.0rc2 (2020-02-06)
==============================
 
 Bugfixes
 --------
 
 - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
 - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))
 
 Internal Changes
 ----------------
 
 - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl478l8QHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHmKfD/0ZpBV95gMk0iecfvO6IY8+0KigzDUypgXf
 zp1k+k5DBUmH5/83h7+cXWHIJyofv1At1YdIq9J/nNeyqr3V3dcQwPabOKYbI96d
 kVD0EZX+2NjuYejAuF9eJNdiwRq5yMLluCfnOXr5jCS0NNdpO1xVb4bWYm50RmcD
 WJvBvmfwXwCC3AFyNb4IAipaVUXuVXXx1YhjFs6DsbtpPUG88e4uIhpfvrS4v0t8
 GFJZNuVJStOvyKHHTfNRIMkhy4xidj3HRUMSmjNpx07WnPBnbv4LnUOD1boYxgaP
 EoODYnoAPshdiKB0AywNNjue3TmFD/Z7vVEzlFP/lNZ4GDU4kN9C/EwFYw0C2Jqq
 f9O/E2ZuP9Qqume5O9UwMQQAmhV5lMBaIsYRbixU+9bSB893zRHRffA11HypzD00
 ZXj8QDOXpiXp2jpAwN1Rk9e/aZEX3qd+zUAgRmk+kYb0KTC2/gcY956rodNSSO/U
 RFYg4DFvoCgCrRSxZ4LQMlFu3YY2E7qWH+p/OHk3WG79jW64VpEFaytvv8fR+GKq
 g3EbtWy6mUzlKYqbjZ+ZTUDe5AWdv6ZX8xJqRD/S6cpiwyh6Gp89HHNvBThXoCmK
 fxkgw8if7eIITuTlNuDrqunxyWqwd3oVlzd2mi2bg0yRfcqJ9C6OuBV1VTLFZeky
 3sjCiU0IRw==
 =kcSy
 -----END PGP SIGNATURE-----

Merge tag 'v1.10.0rc2' into develop

Synapse 1.10.0rc2 (2020-02-06)
==============================

Bugfixes
--------

- Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
- Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))

Internal Changes
----------------

- Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
2020-02-06 11:04:03 +00:00
Richard van der Hoff
f84700fba8 Pass room version object into FederationClient.get_pdu 2020-02-05 17:25:46 +00:00
Erik Johnston
a58860e480
Check sender_key matches on inbound encrypted events. (#6850)
If they don't then the device lists are probably out of sync.
2020-02-05 14:02:39 +00:00
Erik Johnston
6475382d80
Fix detecting unknown devices from remote encrypted events. (#6848)
We were looking at the wrong event type (`m.room.encryption` vs
`m.room.encrypted`).

Also fixup the duplicate `EvenTypes` entries.

Introduced in #6776.
2020-02-04 17:25:54 +00:00
Richard van der Hoff
5d17c31596 make FederationHandler.send_invite async 2020-02-03 22:28:11 +00:00
Richard van der Hoff
e81c093974 make FederationHandler.on_get_missing_events async 2020-02-03 19:15:08 +00:00
Richard van der Hoff
f64c96662e make FederationHandler.user_joined_room async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
52642860da make FederationHandler._clean_room_for_join async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
814cc00cb9 make FederationHandler._notify_persisted_event async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
05299599b6 make FederationHandler.persist_events_and_notify async 2020-02-03 16:29:30 +00:00
Richard van der Hoff
3b7e0e002b make FederationHandler._make_and_verify_event async 2020-02-03 16:22:30 +00:00
Richard van der Hoff
4286e429a7 make FederationHandler.do_remotely_reject_invite async 2020-02-03 16:19:18 +00:00
Richard van der Hoff
c3f296af32 make FederationHandler._check_for_soft_fail async 2020-02-03 16:16:31 +00:00
Richard van der Hoff
dbdf843012 make FederationHandler._persist_auth_tree async 2020-02-03 16:14:58 +00:00
Richard van der Hoff
ebd6a15af3 make FederationHandler.do_invite_join async 2020-02-03 16:13:13 +00:00
Richard van der Hoff
94f7b4cd54 make FederationHandler.on_event_auth async 2020-02-03 16:06:46 +00:00
Richard van der Hoff
863087d186 make FederationHandler.on_exchange_third_party_invite_request async 2020-02-03 16:02:50 +00:00
Richard van der Hoff
957129f4a7 make FederationHandler.construct_auth_difference async 2020-02-03 16:00:46 +00:00
Richard van der Hoff
0d5f2f4bb0 make FederationHandler._update_context_for_auth_events async 2020-02-03 15:55:35 +00:00
Richard van der Hoff
a25ddf26a3 make FederationHandler._update_auth_events_and_context_for_auth async 2020-02-03 15:53:54 +00:00
Richard van der Hoff
bc9b75c6f0 make FederationHandler.do_auth async 2020-02-03 15:51:24 +00:00
Richard van der Hoff
8033b257a7 make FederationHandler._prep_event async 2020-02-03 15:49:32 +00:00
Richard van der Hoff
1cdc253e0a make FederationHandler._handle_new_event async 2020-02-03 15:48:33 +00:00
Richard van der Hoff
c556ed9e15 make FederationHandler._handle_new_events async 2020-02-03 15:43:51 +00:00
Richard van der Hoff
6e89ec5e32 make FederationHandler.on_make_leave_request async 2020-02-03 15:40:41 +00:00
Richard van der Hoff
d184cbc031 make FederationHandler.on_send_leave_request async 2020-02-03 15:39:24 +00:00
Richard van der Hoff
98681f90cb make FederationHandler.on_make_join_request async 2020-02-03 15:38:02 +00:00
Richard van der Hoff
af8ba6b525 make FederationHandler.on_invite_request async 2020-02-03 15:33:42 +00:00
Richard van der Hoff
7571bf86f0 make FederationHandler.on_send_join_request async 2020-02-03 15:32:48 +00:00
Richard van der Hoff
b3e44f0bdf make FederationHandler.on_query_auth async 2020-02-03 15:30:23 +00:00
Richard van der Hoff
7f93eb1903
pass room_version into compute_event_signature (#6807) 2020-01-31 13:47:43 +00:00
Richard van der Hoff
160522e32c
Merge pull request #6820 from matrix-org/rav/get_room_version_id
Make `get_room_version` return a RoomVersion object
2020-01-31 10:56:42 +00:00
Richard van der Hoff
d7bf793cc1 s/get_room_version/get_room_version_id/
... to make way for a forthcoming get_room_version which returns a RoomVersion
object.
2020-01-31 10:06:21 +00:00
Erik Johnston
7d846e8704
Fix bug with getting missing auth event during join 500'ed (#6810) 2020-01-31 09:49:13 +00:00
Richard van der Hoff
46a446828d
pass room version into FederationHandler.on_invite_request (#6805) 2020-01-30 22:13:02 +00:00
Erik Johnston
b660327056
Resync remote device list when detected as stale. (#6786) 2020-01-30 17:06:38 +00:00
Erik Johnston
e17a110661
Detect unknown remote devices and mark cache as stale (#6776)
We just mark the fact that the cache may be stale in the database for
now.
2020-01-28 14:43:21 +00:00
Richard van der Hoff
a8ce7aeb43
Pass room version object into event_auth.check and check_redaction (#6788)
These are easier to work with than the strings and we normally have one around.

This fixes `FederationHander._persist_auth_tree` which was passing a
RoomVersion object into event_auth.check instead of a string.
2020-01-28 14:18:29 +00:00
Erik Johnston
8df862e45d
Add rooms.room_version column (#6729)
This is so that we don't have to rely on pulling it out from `current_state_events` table.
2020-01-27 14:30:57 +00:00
Erik Johnston
d386f2f339
Add StateMap type alias (#6715) 2020-01-16 13:31:22 +00:00
Brendan Abolivier
cd428a93e2
Fix conditions failing if min_depth = 0
This could result in Synapse not fetching prev_events for new events in the room if it has missed some events.
2020-01-07 12:08:58 +00:00
Richard van der Hoff
56ad230efa Merge branch 'master' into develop 2019-12-20 11:11:34 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Richard van der Hoff
03d3792f3c
Fix exceptions when attempting to backfill (#6576)
Fixes #6575
2019-12-20 09:55:45 +00:00
Richard van der Hoff
6e8f8e14f2 Merge release-v1.7.1 into develop 2019-12-18 09:51:51 +00:00
Richard van der Hoff
bbb75ff6ee Exclude rejected state events when calculating state at backwards extrems (#6527)
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
2019-12-16 14:07:29 +00:00
Richard van der Hoff
ff773ff724 Persist auth/state events at backwards extremities when we fetch them (#6526)
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
2019-12-16 14:07:11 +00:00
Richard van der Hoff
83895316d4 sanity-checking for events used in state res (#6531)
When we perform state resolution, check that all of the events involved are in
the right room.
2019-12-16 13:53:53 +00:00
Richard van der Hoff
35bbe4ca79 Check the room_id of events when fetching room state/auth (#6524)
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
2019-12-16 13:47:07 +00:00
Richard van der Hoff
20d5ba16e6 Add include_event_in_state to _get_state_for_room (#6521)
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
2019-12-16 13:26:12 +00:00
Richard van der Hoff
be294d6fde Move get_state methods into FederationHandler (#6503)
This is a non-functional refactor as a precursor to some other work.
2019-12-16 13:20:21 +00:00
Richard van der Hoff
6920d88892
Exclude rejected state events when calculating state at backwards extrems (#6527)
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
2019-12-16 13:14:37 +00:00
Richard van der Hoff
bc7de87650
Persist auth/state events at backwards extremities when we fetch them (#6526)
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
2019-12-16 12:26:28 +00:00
Richard van der Hoff
1da15f05f5
sanity-checking for events used in state res (#6531)
When we perform state resolution, check that all of the events involved are in
the right room.
2019-12-13 12:55:32 +00:00
Richard van der Hoff
25f1244329
Check the room_id of events when fetching room state/auth (#6524)
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
2019-12-12 12:57:45 +00:00
Richard van der Hoff
2045356517
Add include_event_in_state to _get_state_for_room (#6521)
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
2019-12-11 16:37:51 +00:00
Richard van der Hoff
6637d90d77 convert to async: FederationHandler._process_received_pdu
also fix user_joined_room to consistently return deferreds
2019-12-11 14:39:26 +00:00
Richard van der Hoff
4db394a4b3 convert to async: FederationHandler._get_state_for_room
... and _get_events_from_store_or_dest
2019-12-11 14:39:26 +00:00
Richard van der Hoff
e77237b935 convert to async: FederationHandler.on_receive_pdu
and associated functions:
 * on_receive_pdu
 * handle_queued_pdus
 * get_missing_events_for_pdu
2019-12-11 14:39:26 +00:00
Richard van der Hoff
7712e751b8 Convert federation backfill to async
PaginationHandler.get_messages is only called by RoomMessageListRestServlet,
which is async.

Chase the code path down from there:
 - FederationHandler.maybe_backfill (and nested try_backfill)
 - FederationHandler.backfill
2019-12-11 14:39:25 +00:00
Richard van der Hoff
7c429f92d6
Clean up some logging (#6515)
This just makes some of the logging easier to follow when things start going
wrong.
2019-12-11 14:32:25 +00:00
Andrew Morgan
fc316a4894
Prevent redacted events from appearing in message search (#6377) 2019-12-11 13:39:47 +00:00
Andrew Morgan
ea0f0ad414
Prevent message search in upgraded rooms we're not in (#6385) 2019-12-11 13:07:25 +00:00
Richard van der Hoff
f8bc2ae883
Move get_state methods into FederationHandler (#6503)
This is a non-functional refactor as a precursor to some other work.
2019-12-10 17:42:46 +00:00
Richard van der Hoff
63d6ad1064
Stronger typing in the federation handler (#6480)
replace the event_info dict with an attrs thing
2019-12-05 15:02:35 +00:00
Richard van der Hoff
e1f4c83f41
Sanity-check the rooms of auth events before pulling them in. (#6472) 2019-12-05 14:14:45 +00:00
Richard van der Hoff
e203874caa
get rid of (most of) have_events from _update_auth_events_and_context_for_auth (#6468)
have_events was a map from event_id to rejection reason (or None) for events
which are in our local database. It was used as filter on the list of
event_ids being passed into get_events_as_list. However, since
get_events_as_list will ignore any event_ids that are unknown or rejected, we
can equivalently just leave it to get_events_as_list to do the filtering.

That means that we don't have to keep `have_events` up-to-date, and can use
`have_seen_events` instead of `get_seen_events_with_rejection` in the one place
we do need it.
2019-12-04 17:27:32 +00:00
Brendan Abolivier
54dd5dc12b
Add ephemeral messages support (MSC2228) (#6409)
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:

* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
2019-12-03 19:19:45 +00:00
Erik Johnston
2173785f0d Propagate reason in remotely rejected invites 2019-11-28 11:31:56 +00:00
Brendan Abolivier
d31f69afa0
Merge pull request #6358 from matrix-org/babolivier/message_retention
Implement message retention policies (MSC1763)
2019-11-27 15:04:38 +00:00
Brendan Abolivier
9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Richard van der Hoff
4d394d6415
remove confusing fixme 2019-11-26 12:32:37 +00:00
Brendan Abolivier
a6fc6754f8
Fix 3PID invite exchange 2019-11-19 14:07:39 +00:00
Richard van der Hoff
870c00e278 Merge remote-tracking branch 'origin/develop' into rav/event_auth/4 2019-11-18 12:05:36 +00:00
Andrew Morgan
bc29a19731 Replace instance variations of homeserver with correct case/spacing 2019-11-12 13:08:12 +00:00
Richard van der Hoff
f41027f746 Use get_events_as_list rather than lots of calls to get_event
It's more efficient and clearer.
2019-11-08 12:21:28 +00:00
Richard van der Hoff
f8407975e7 Update some docstrings and comments 2019-11-08 12:18:20 +00:00
Richard van der Hoff
772d414975 Simplify _update_auth_events_and_context_for_auth
move event_key calculation into _update_context_for_auth_events, since it's
only used there.
2019-11-08 11:40:11 +00:00
Richard van der Hoff
807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state (#6320)
Fixes a bug where rejected events were persisted with the wrong state group.

Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.

Fixes #6289.
2019-11-06 10:01:39 +00:00
Richard van der Hoff
0e3ab8afdc
Add some checks that we aren't using state from rejected events (#6330)
* Raise an exception if accessing state for rejected events

Add some sanity checks on accessing state_group etc for
rejected events.

* Skip calculating push actions for rejected events

It didn't actually cause any bugs, because rejected events get filtered out at
various later points, but there's not point in trying to calculate the push
actions for a rejected event.
2019-11-05 22:13:37 +00:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Richard van der Hoff
c6516adbe0
Factor out an _AsyncEventContextImpl (#6298)
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
2019-11-01 16:19:09 +00:00
Andrew Morgan
ace947e8da
Depublish a room from the public rooms list when it is upgraded (#6232) 2019-11-01 10:28:09 +00:00
Amber Brown
020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston
69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Erik Johnston
ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Andrew Morgan
a2276d4d3c
Fix log line that was printing undefined value (#6278) 2019-10-30 11:28:48 +00:00
Richard van der Hoff
172f264ed3
Improve signature checking on some federation APIs (#6262)
Make sure that we check that events sent over /send_join, /send_leave, and
/invite, are correctly signed and come from the expected servers.
2019-10-28 12:43:23 +00:00
Erik Johnston
3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Richard van der Hoff
93eaeec75a
Remove Auth.check method (#6217)
This method was somewhat redundant, and confusing.
2019-10-18 19:43:36 +02:00
Richard van der Hoff
1ba359a11f rip out some unreachable code
The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
2019-10-17 21:45:59 +01:00
Brendan Abolivier
21d51ab598
Typo 2019-10-04 11:21:24 +01:00
Brendan Abolivier
4676732ca0
Lint 2019-10-04 11:18:28 +01:00
Brendan Abolivier
81d51ce48b
Incorporate review 2019-10-04 11:16:19 +01:00
Brendan Abolivier
6527fa18c1 Add test case 2019-10-03 11:24:36 +01:00
Brendan Abolivier
972c9f65d7
Lint 2019-10-02 12:17:46 +01:00
Brendan Abolivier
5705ecaec6
Don't 500 code when trying to exchange a revoked 3PID invite
While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state.
When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events.
When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it).

This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
2019-10-02 11:19:43 +01:00
Andrew Morgan
cd17a2085e
Remove origin parameter from add_display_name_to_third_party_invite and add params to docstring (#6010)
Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument.

Since it's not used anywhere in the handler function though, we should remove it from the function arguments.
2019-09-11 10:37:17 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Amber Brown
7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Erik Johnston
0f632f3a57
Merge pull request #5790 from matrix-org/erikj/groups_request_errors
Handle RequestSendFailed exception correctly in more places.
2019-08-01 13:14:08 +01:00
Erik Johnston
b4d5ff0af7 Don't log as exception when failing durig backfill 2019-07-30 13:19:22 +01:00
Amber Brown
865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Richard van der Hoff
1a93daf353
Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch
Log when we receive a /make_* request from a different origin
2019-07-26 12:38:37 +01:00
Richard van der Hoff
d1020653fc Log when we receive a /make_* request from a different origin 2019-07-26 10:08:22 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
7dcf984075
Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_error
Handle the case of `get_missing_events` failing
2019-06-19 13:20:09 +01:00
David Baker
f2d2ae03da Add some logging to 3pid invite sig verification (#5015)
I had to add quite a lot of logging to diagnose a problem with 3pid
invites - we only logged the one failure which isn't all that
informative.

NB. I'm not convinced the logic of this loop is right: I think it
should just accept a single valid signature from a trusted source
rather than fail if *any* signature is invalid. Also it should
probably not skip the rest of middle loop if a check fails? However,
I'm deliberately not changing the logic here.
2019-06-18 22:51:24 +01:00
Erik Johnston
2b20d0fb59 Fix logline 2019-06-18 16:12:53 +01:00
Erik Johnston
19b80fe68a Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_get_missing_events_error 2019-06-18 16:11:43 +01:00
Erik Johnston
a9dab970b8
Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_state
Fix 3PID invite room state over federation.
2019-06-17 10:18:28 +01:00
Brendan Abolivier
f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Erik Johnston
304a1376c2 Fix 3PID invite room state over federation.
Fixes that when a user exchanges a 3PID invite for a proper invite over
federation it does not include the `invite_room_state` key.

This was due to synapse incorrectly sending out two invite requests.
2019-06-14 16:19:11 +01:00
Erik Johnston
bc3d6b918b Add logging when request fails and clarify we ignore errors. 2019-06-05 11:37:51 +01:00
Erik Johnston
220a733d73 Fix handling of failures when calling /event_auth.
When processing an incoming event over federation, we may try and
resolve any unexpected differences in auth events. This is a
non-essential process and so should not stop the processing of the event
if it fails (e.g. due to the remote disappearing or not implementing the
necessary endpoints).

Fixes #3330
2019-06-03 09:56:45 +01:00
Richard van der Hoff
85d1e03b9d
Simplifications and comments in do_auth (#5227)
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
Richard van der Hoff
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146)
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
2019-05-21 16:10:54 +01:00
Erik Johnston
c132c8e505 Handle the case of get_missing_events failing
Currently if a call to `/get_missing_events` fails we log an exception
and stop processing the top level event we received over federation.
Instead let's try and handle it sensibly given it is a somewhat expected
failure mode.
2019-04-10 10:39:54 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Erik Johnston
0ff8163eae Factor out soft fail checks 2019-03-08 11:26:33 +00:00
Erik Johnston
a9de04be72 Implement soft fail 2019-03-06 16:22:16 +00:00
Erik Johnston
aa06d26ae0 clarify comments 2019-03-05 09:16:35 +00:00
Erik Johnston
d1523aed6b Only check history visibility when filtering
When filtering events to send to server we check more than just history
visibility. However when deciding whether to backfill or not we only
care about the history visibility.
2019-03-04 14:43:42 +00:00
Erik Johnston
8b63fe4c26 s/get_forward_events/get_successor_events/ 2019-03-04 11:56:03 +00:00
Erik Johnston
fbc047f2a5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room 2019-03-04 11:54:58 +00:00
Erik Johnston
b183fef9ac Update comments 2019-02-27 13:06:10 +00:00
Erik Johnston
9c598dddcb Fix typo 2019-02-25 16:32:02 +00:00
Erik Johnston
d730c2c22b More comments 2019-02-25 14:45:02 +00:00
Erik Johnston
890cb048fd Assert rather than clobber the values 2019-02-25 14:42:11 +00:00
Erik Johnston
9342cc6ab1 Add comments and paranoia 2019-02-25 10:02:12 +00:00
Erik Johnston
7b288826b7 Fix backfill storing incorrect state for events 2019-02-22 11:33:51 +00:00
Erik Johnston
71b625d808 Stop backpaginating when events not visible 2019-02-20 18:14:12 +00:00
Amber Brown
3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Erik Johnston
55d9024835 Use snder and not event ID domain to check if ours
The transaction queue only sends out events that we generate. This was
done by checking domain of event ID, but that can no longer be used.
Instead, we may as well use the sender field.
2019-01-29 16:54:23 +00:00
Erik Johnston
6598992b01 Fixup calls to comput_event_signature
We currently pass FrozenEvent instead of `dict` to
`compute_event_signature`, which works by accident due to `dict(event)`
producing the correct result.

This fixes PR #4493 commit 855a151
2019-01-29 11:41:58 +00:00
Erik Johnston
073f6c2e5e
Merge pull request #4494 from matrix-org/erikj/fixup_event_validator
Split up event validation between event and builder
2019-01-29 10:55:07 +00:00
Erik Johnston
b872c7b1b4 Split up event validation between event and builder
The validator was being run on the EventBuilder objects, and so the
validator only checked a subset of fields. With the upcoming
EventBuilder refactor even fewer fields will be there to validate.

To get around this we split the validation into those that can be run
against an EventBuilder and those run against a fully fledged event.
2019-01-28 17:00:14 +00:00
Erik Johnston
ae2a957dba Pass through room version to event auth 2019-01-25 18:31:41 +00:00
Erik Johnston
a50cf929c1 Require event format version to parse or create events 2019-01-25 10:32:19 +00:00
Erik Johnston
62514bb81b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813 2019-01-25 10:07:08 +00:00
Erik Johnston
edc1e21dbe
Merge pull request #4448 from matrix-org/erikj/get_pdu_versions
Add room_version param to get_pdu
2019-01-24 18:47:15 +00:00
Erik Johnston
d148c43050 Review comments 2019-01-24 18:31:23 +00:00
Erik Johnston
9139b87be4 Remove unecessary setting of outlier bit 2019-01-24 18:04:02 +00:00
Erik Johnston
b8082a5445 Use term 'out of band membership' instead 2019-01-24 17:33:19 +00:00
Erik Johnston
7c288c2250 Clarify the invite flows 2019-01-23 20:07:47 +00:00
Erik Johnston
07f62da55a Remove unnecessary '_sign_event' 2019-01-23 20:04:57 +00:00
Erik Johnston
886e5acc76 Store rejected remote invite events as outliers
Currently they're stored as non-outliers even though the server isn't in
the room, which can be problematic in places where the code assumes it
has the state for all non outlier events.

In particular, there is an edge case where persisting the leave event
triggers a state resolution, which requires looking up the room version
from state. Since the server doesn't have the state, this causes an
exception to be thrown.
2019-01-23 20:04:57 +00:00
Erik Johnston
6a41d2a187 Add room_version param to get_pdu
When we add new event format we'll need to know the event format or room
version when parsing events.
2019-01-23 17:19:58 +00:00
Erik Johnston
67cd4dad81 Implement MSC 1813 - Add room version to make APIs
We also implement `make_membership_event` converting the returned
room version to an event format version.
2019-01-23 16:51:46 +00:00
Erik Johnston
dc59ad5334 Remove hack to support rejoining rooms 2018-11-09 14:58:09 +00:00
Erik Johnston
30dd27afff Simplify to always drop events if server isn't in the room 2018-11-09 11:36:45 +00:00
Erik Johnston
bc80b3f454 Add helpers for getting prev and auth events (#4139)
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
2018-11-06 00:35:15 +11:00
Erik Johnston
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase
Add v2 state resolution algorithm
2018-10-24 11:12:12 +01:00
Erik Johnston
dacbeb2e03 Comment 2018-10-24 09:47:49 +01:00
Erik Johnston
810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Richard van der Hoff
e7a16c6210 Remove redundant run_as_background_process() from pusherpool
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
2018-10-22 16:12:11 +01:00
Richard van der Hoff
fc0f13dd03 Fix incorrect truncation in get_missing_events
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
2018-10-16 21:10:04 +01:00
Erik Johnston
15133477ee Fix up use of resolve_events_with_factory 2018-10-16 16:28:42 +01:00
Richard van der Hoff
8174c6725b Avoid reraise, to improve stacktraces 2018-10-01 18:50:34 +01:00
Richard van der Hoff
333bee27f5 Include event when resolving state for missing prevs
If we have a forward extremity for a room as `E`, and you receive `A`, `B`,
s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need
to do state res between `X` and `E`.

When that happens, we need to make sure we include `X` in the state that goes
into the state res alg.

Fixes #3934.
2018-09-27 11:37:39 +01:00
Richard van der Hoff
bd61c82bdf Include state from remote servers in pdu handling
If we've fetched state events from remote servers in order to resolve the state
for a new event, we need to actually pass those events into
resolve_events_with_factory (so that it can do the state res) and then persist
the ones we need - otherwise other bits of the codebase get confused about why
we have state groups pointing to non-existent events.
2018-09-27 11:37:39 +01:00
Richard van der Hoff
a215b698c4 Fix "unhashable type: 'list'" exception in federation handling
get_state_groups returns a map from state_group_id to a list of FrozenEvents,
so was very much the wrong thing to be putting as one of the entries in the
list passed to resolve_events_with_factory (which expects maps from
(event_type, state_key) to event id).

We actually want get_state_groups_ids().values() rather than
get_state_groups().

This fixes the main problem in #3923, but there are other problems with this
bit of code which get discovered once you do so.
2018-09-27 11:37:39 +01:00
Richard van der Hoff
28223841e0 more comments 2018-09-27 11:31:51 +01:00
Richard van der Hoff
e3c159863d Clarifications in FederationHandler
* add some comments on things that look a bit bogus
* rename this `state` variable to avoid confusion with the `state` used
  elsewhere in this function. (There was no actual conflict, but it was
  a confusing bit of spaghetti.)
2018-09-27 11:31:51 +01:00
Richard van der Hoff
92abd3d6d6
Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2
Logging improvements
2018-09-27 11:27:46 +01:00
Richard van der Hoff
4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959)
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
2018-09-27 11:25:34 +01:00
Richard van der Hoff
e70b4ce069 Logging improvements
Some logging tweaks to help with debugging incoming federation transactions
2018-09-26 17:36:14 +01:00
Richard van der Hoff
703de4ec13 Comments and interface cleanup for on_receive_pdu
Add some informative comments about what's going on here.

Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart
from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of
`get_missing` and use `sent_to_us_directly` consistently.
2018-09-20 13:06:55 +01:00
Richard van der Hoff
642199570c
Improve the logging when handling a federation transaction (#3904)
Let's try to rationalise the logging that happens when we are processing an
incoming transaction, to make it easier to figure out what is going wrong when
they take ages. In particular:

- make everything start with a [room_id event_id] prefix
- make sure we log a warning when catching exceptions rather than just turning
  them into other, more cryptic, exceptions.
2018-09-19 17:28:18 +01:00
Richard van der Hoff
550007cb0e Bump timeout on get_missing_events request 2018-09-18 15:02:51 +01:00
Amber Brown
2608ebc04c
Port handlers/ to Python 3 (#3803) 2018-09-07 00:22:23 +10:00
Erik Johnston
5c261107c9 Remove unnecessary resolve_events_with_state_map
We only ever used the synchronous resolve_events_with_state_map in one
place, which is trivial to replace with the async version.
2018-08-22 15:41:15 +01:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston
8dee601054 Remove redundant room_version checks 2018-08-20 14:48:53 +01:00
Richard van der Hoff
66f7dc8c87 Fix logcontexts for running pushers
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.

Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
Erik Johnston
ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Erik Johnston
488ffe6fdb Use federation handler function rather than duplicate
This involves renaming _persist_events to be a public function.
2018-08-15 14:17:18 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Erik Johnston
ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston
b179537f2a Move clean_room_for_join to master 2018-08-09 10:37:38 +01:00
Erik Johnston
5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00
Erik Johnston
a3f5bf79a0 Add EDU/query handling over replication 2018-08-06 15:23:31 +01:00
Erik Johnston
e26dbd82ef Add replication APIs for persisting federation events 2018-08-06 15:02:28 +01:00
Richard van der Hoff
f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Richard van der Hoff
704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff
14fa9d4d92 Avoid extra db lookups
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
Richard van der Hoff
0a65450d04 Validation for events/rooms in fed requests
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
Erik Johnston
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
Erik Johnston
a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Erik Johnston
4b256b9271 _persist_auth_tree no longer returns anything 2018-08-01 13:39:07 +01:00
Richard van der Hoff
bdbdceeafa rename replication_layer to federation_client
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
2018-07-31 15:44:05 +01:00
Erik Johnston
143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Travis Ralston
d8e65ed7e1
Fix a minor documentation typo in on_make_leave 2018-07-25 15:44:41 -06:00
Erik Johnston
78a691d005 Split out DB writes in federation handler
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
Richard van der Hoff
c1f80effbe Handle delta_ids being None in _update_context_for_auth_events
it's easier to create the new state group as a delta from the existing one.

(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364)
2018-07-23 22:06:50 +01:00
Erik Johnston
027bc01a1b Add support for updating state 2018-07-23 13:17:25 +01:00
Erik Johnston
e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Amber Brown
3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Richard van der Hoff
dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks
Remove event re-signing hacks
2018-07-18 12:46:27 +01:00
Richard van der Hoff
ea69d35651 Move filter_events_for_server out of FederationHandler
for easier unit testing.
2018-07-16 13:06:24 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
a4ab491371
Merge branch 'develop' into rav/drop_re_signing_hacks 2018-07-04 07:13:38 +01:00
Amber Brown
99800de63d try and clean up 2018-06-27 11:40:27 +01:00
Amber Brown
f03a5d1a17 pep8 2018-06-27 11:38:14 +01:00
Amber Brown
a7ecf34b70 cleanups 2018-06-27 11:36:03 +01:00
Amber Brown
77078d6c8e handle federation not telling us about prev_events 2018-06-27 11:27:32 +01:00
Erik Johnston
df48f7ef37 Actually fix it 2018-06-25 20:03:41 +01:00
Erik Johnston
a0e8a53c6d Comment 2018-06-25 19:57:38 +01:00
Erik Johnston
7bdc5c8fa3 Fix bug with assuming wrong type 2018-06-25 19:56:02 +01:00
Erik Johnston
ea7a9c0483 Add fast path to _filter_events_for_server
Most rooms have a trivial history visibility like "shared" or
"world_readable", especially large rooms, so lets not bother getting the
full membership of those rooms in that case.
2018-06-25 19:49:13 +01:00
Erik Johnston
244484bf3c Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
This reverts commit 1d009013b3.
2018-06-25 13:42:55 +01:00
Richard van der Hoff
1d009013b3 Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
This reverts commit ce0d911156, reversing
changes made to b4a5d767a9.
2018-06-22 16:35:10 +01:00
Erik Johnston
ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility
Support hiding events from deleted users
2018-06-22 15:06:44 +01:00
Erik Johnston
b4a5d767a9
Merge pull request #3428 from matrix-org/erikj/persisted_pdu
Simplify get_persisted_pdu
2018-06-22 14:47:43 +01:00
Amber Brown
a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Richard van der Hoff
b6faef2ad7 Filter out erased messages
Redact any messges sent by erased users.
2018-06-12 09:53:18 +01:00
Richard van der Hoff
9fc5b74b24 simplify get_persisted_pdu
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
2018-06-12 09:51:31 +01:00
Richard van der Hoff
0834b49c6a Fix event filtering in get_missing_events handler 2018-06-08 11:34:46 +01:00
Richard van der Hoff
8503dd0047 Remove event re-signing hacks
These "temporary fixes" have been here three and a half years, and I can't find
any events in the matrix.org database where the calculated signature differs
from what's in the db. It's time for them to go away.
2018-06-07 16:08:29 +01:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Adrian Tschira
17a70cf6e9 Misc. py3 fixes
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
Erik Johnston
f72d5a44d5
Merge pull request #3261 from matrix-org/erikj/pagination_fixes
Fix federation backfill bugs
2018-05-24 14:52:03 +01:00
Erik Johnston
5aaa3189d5 s/values/itervalues/ 2018-05-23 10:13:05 +01:00
Erik Johnston
0a4bca4134 Use iter* methods for _filter_events_for_server 2018-05-23 10:04:23 +01:00
Erik Johnston
e85b5a0ff7 Use iter* methods 2018-05-22 19:02:48 +01:00
Erik Johnston
586b66b197 Fix that states is a dict of dicts 2018-05-22 19:02:36 +01:00
Erik Johnston
cb2a2ad791 get_domains_from_state returns list of tuples 2018-05-22 16:23:39 +01:00
Richard van der Hoff
26305788fe Make sure we reject attempts to invite the notices user 2018-05-18 11:18:39 +01:00
Richard van der Hoff
bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib
move httplib import to six
2018-04-30 01:26:42 +01:00
Adrian Tschira
94f4d7f49e move httplib import to six 2018-04-28 13:43:34 +02:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00