Commit Graph

3189 Commits

Author SHA1 Message Date
Richard van der Hoff
83895316d4 sanity-checking for events used in state res (#6531)
When we perform state resolution, check that all of the events involved are in
the right room.
2019-12-16 13:53:53 +00:00
Richard van der Hoff
35bbe4ca79 Check the room_id of events when fetching room state/auth (#6524)
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
2019-12-16 13:47:07 +00:00
Richard van der Hoff
20d5ba16e6 Add include_event_in_state to _get_state_for_room (#6521)
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
2019-12-16 13:26:12 +00:00
Richard van der Hoff
be294d6fde Move get_state methods into FederationHandler (#6503)
This is a non-functional refactor as a precursor to some other work.
2019-12-16 13:20:21 +00:00
Richard van der Hoff
6920d88892
Exclude rejected state events when calculating state at backwards extrems (#6527)
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
2019-12-16 13:14:37 +00:00
Richard van der Hoff
bc7de87650
Persist auth/state events at backwards extremities when we fetch them (#6526)
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
2019-12-16 12:26:28 +00:00
Brendan Abolivier
487f1bb49d
Use the filtered version of an event when responding to /context requests for that event
Sometimes the filtering function can return a pruned version of an event (on top of either the event itself or an empty list), if it thinks the user should be able to see that there's an event there but not the content of that event. Therefore, the previous logic of 'if filtered is empty then we can use the event we retrieved from the database' is flawed, and we should use the event returned by the filtering function.
2019-12-16 12:14:12 +00:00
Richard van der Hoff
1da15f05f5
sanity-checking for events used in state res (#6531)
When we perform state resolution, check that all of the events involved are in
the right room.
2019-12-13 12:55:32 +00:00
Erik Johnston
caa52836e4
Merge pull request #6496 from matrix-org/erikj/initial_sync_asnyc
Port synapse.handlers.initial_sync to async/await
2019-12-13 10:01:51 +00:00
Hubert Chathi
cb2db17994
look up cross-signing keys from the DB in bulk (#6486) 2019-12-12 12:03:28 -05:00
Richard van der Hoff
25f1244329
Check the room_id of events when fetching room state/auth (#6524)
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
2019-12-12 12:57:45 +00:00
Erik Johnston
6828b47c45 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/initial_sync_asnyc 2019-12-11 17:01:41 +00:00
Richard van der Hoff
2045356517
Add include_event_in_state to _get_state_for_room (#6521)
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
2019-12-11 16:37:51 +00:00
Richard van der Hoff
894d2addac
Merge pull request #6517 from matrix-org/rav/event_auth/13
Port some of FederationHandler to async/await
2019-12-11 16:36:06 +00:00
Erik Johnston
31905a518e
Merge pull request #6504 from matrix-org/erikj/account_validity_async_await
Port handlers.account_validity to async/await.
2019-12-11 14:49:26 +00:00
Richard van der Hoff
6637d90d77 convert to async: FederationHandler._process_received_pdu
also fix user_joined_room to consistently return deferreds
2019-12-11 14:39:26 +00:00
Richard van der Hoff
4db394a4b3 convert to async: FederationHandler._get_state_for_room
... and _get_events_from_store_or_dest
2019-12-11 14:39:26 +00:00
Richard van der Hoff
e77237b935 convert to async: FederationHandler.on_receive_pdu
and associated functions:
 * on_receive_pdu
 * handle_queued_pdus
 * get_missing_events_for_pdu
2019-12-11 14:39:26 +00:00
Richard van der Hoff
7712e751b8 Convert federation backfill to async
PaginationHandler.get_messages is only called by RoomMessageListRestServlet,
which is async.

Chase the code path down from there:
 - FederationHandler.maybe_backfill (and nested try_backfill)
 - FederationHandler.backfill
2019-12-11 14:39:25 +00:00
Richard van der Hoff
7c429f92d6
Clean up some logging (#6515)
This just makes some of the logging easier to follow when things start going
wrong.
2019-12-11 14:32:25 +00:00
Andrew Morgan
fc316a4894
Prevent redacted events from appearing in message search (#6377) 2019-12-11 13:39:47 +00:00
Andrew Morgan
ea0f0ad414
Prevent message search in upgraded rooms we're not in (#6385) 2019-12-11 13:07:25 +00:00
Richard van der Hoff
f8bc2ae883
Move get_state methods into FederationHandler (#6503)
This is a non-functional refactor as a precursor to some other work.
2019-12-10 17:42:46 +00:00
Andrew Morgan
4947de5a14
Allow SAML username provider plugins (#6411) 2019-12-10 17:30:16 +00:00
Erik Johnston
257ef2c727 Port handlers.account_validity to async/await. 2019-12-10 14:40:15 +00:00
Erik Johnston
e3f528c544
Merge pull request #6506 from matrix-org/erikj/remove_snapshot_cache
Remove SnapshotCache in favour of ResponseCache
2019-12-10 13:04:50 +00:00
Erik Johnston
353396e3a7 Port handlers.account_data to async/await. 2019-12-10 11:12:56 +00:00
Neil Johnson
adfdd82b21
Back out perf regression from get_cross_signing_keys_from_cache. (#6494)
Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression.
2019-12-09 13:59:27 +00:00
Erik Johnston
a1f8ea9051 Port synapse.handlers.initial_sync to async/await 2019-12-09 13:46:45 +00:00
Erik Johnston
f166a8d1f5 Remove SnapshotCache in favour of ResponseCache 2019-12-09 13:42:49 +00:00
Erik Johnston
e1544b0af8
Merge pull request #6493 from matrix-org/erikj/invite_state_config
Pull out room_invite_state_types config option once.
2019-12-09 12:23:21 +00:00
Erik Johnston
8ad8bcbed0 Pull out room_invite_state_types config option once.
Pulling things out of config is currently surprisingly expensive.
2019-12-09 11:52:19 +00:00
Erik Johnston
a9b393340f
Merge pull request #6484 from matrix-org/erikj/port_sync_handler
Port SyncHandler to async/await
2019-12-09 11:32:44 +00:00
Erik Johnston
b3a4e35ca8 Fixup functions to consistently return deferreds 2019-12-06 10:40:05 +00:00
Manuel Stahl
649b6bc088 Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925) 2019-12-05 18:12:23 +00:00
Erik Johnston
8437e2383e Port SyncHandler to async/await 2019-12-05 17:58:25 +00:00
Richard van der Hoff
63d6ad1064
Stronger typing in the federation handler (#6480)
replace the event_info dict with an attrs thing
2019-12-05 15:02:35 +00:00
Richard van der Hoff
e1f4c83f41
Sanity-check the rooms of auth events before pulling them in. (#6472) 2019-12-05 14:14:45 +00:00
Richard van der Hoff
e203874caa
get rid of (most of) have_events from _update_auth_events_and_context_for_auth (#6468)
have_events was a map from event_id to rejection reason (or None) for events
which are in our local database. It was used as filter on the list of
event_ids being passed into get_events_as_list. However, since
get_events_as_list will ignore any event_ids that are unknown or rejected, we
can equivalently just leave it to get_events_as_list to do the filtering.

That means that we don't have to keep `have_events` up-to-date, and can use
`have_seen_events` instead of `get_seen_events_with_rejection` in the one place
we do need it.
2019-12-04 17:27:32 +00:00
Erik Johnston
b9449012db
Merge pull request #6441 from syamgk/fix-parameter-mismatch
Fix issue #6406 parameter mismatch
2019-12-04 15:42:33 +00:00
Brendan Abolivier
c530f9af4d
Merge pull request #6329 from matrix-org/babolivier/context_filters
Filter state, events_before and events_after in /context requests
2019-12-04 15:24:16 +00:00
Brendan Abolivier
9dc84b7989
Merge branch 'develop' into babolivier/context_filters 2019-12-04 14:23:44 +00:00
Brendan Abolivier
08a436ecb2
Incorporate review 2019-12-04 14:18:46 +00:00
Syam G Krishnan
ce1c975ebc Issue #6406 Fix parameter mismatch
Signed-off-by: Syam G Krishnan <syamgk01@gmail.com>
2019-12-04 13:03:46 +05:30
Brendan Abolivier
54dd5dc12b
Add ephemeral messages support (MSC2228) (#6409)
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:

* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
2019-12-03 19:19:45 +00:00
Andrew Morgan
72078e4be5
Transfer power level state events on room upgrade (#6237) 2019-12-02 15:11:32 +00:00
Erik Johnston
1c3a61529f
Merge pull request #6434 from matrix-org/erikj/msc2367_membership_reasons
Implement MSC 2367 - Membership Reasons
2019-11-29 13:30:36 +00:00
Brendan Abolivier
708cef88cf
Discard retention policies when retrieving state
Purge jobs don't delete the latest event in a room in order to keep the forward extremity and not break the room. On the other hand, get_state_events, when given an at_token argument calls filter_events_for_client to know if the user can see the event that matches that (sync) token. That function uses the retention policies of the events it's given to filter out those that are too old from a client's view.

Some clients, such as Riot, when loading a room, request the list of members for the latest sync token it knows about, and get confused to the point of refusing to send any message if the server tells it that it can't get that information. This can happen very easily with the message retention feature turned on and a room with low activity so that the last event sent becomes too old according to the room's retention policy.

An easy and clean fix for that issue is to discard the room's retention policies when retrieving state.
2019-11-28 19:26:26 +00:00
Erik Johnston
2173785f0d Propagate reason in remotely rejected invites 2019-11-28 11:31:56 +00:00
Andrew Morgan
a9c44d4008
Remove local threepids on account deactivation (#6426) 2019-11-28 10:40:42 +00:00
Hubert Chathi
0d27aba900
add etag and count to key backup endpoints (#5858) 2019-11-27 16:14:44 -05:00
Brendan Abolivier
d31f69afa0
Merge pull request #6358 from matrix-org/babolivier/message_retention
Implement message retention policies (MSC1763)
2019-11-27 15:04:38 +00:00
Richard van der Hoff
651d930f16
Merge pull request #6343 from matrix-org/rav/event_auth/4
Refactor _update_auth_events_and_context_for_auth
2019-11-26 18:15:03 +00:00
Brendan Abolivier
9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Erik Johnston
f9f1c8acbb
Merge pull request #6332 from matrix-org/erikj/query_devices_fix
Fix caching devices for remote servers in worker.
2019-11-26 12:56:05 +00:00
Richard van der Hoff
4d394d6415
remove confusing fixme 2019-11-26 12:32:37 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Brendan Abolivier
83446a18fb
Merge pull request #6335 from matrix-org/erikj/rc_login_cleanups
Only do `rc_login` ratelimiting on succesful login.
2019-11-20 09:52:38 +00:00
Brendan Abolivier
a6fc6754f8
Fix 3PID invite exchange 2019-11-19 14:07:39 +00:00
Brendan Abolivier
7c24d0f443
Lint 2019-11-19 13:22:37 +00:00
Erik Johnston
c7376cdfe3
Apply suggestions from code review
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
2019-11-18 17:10:16 +00:00
Richard van der Hoff
870c00e278 Merge remote-tracking branch 'origin/develop' into rav/event_auth/4 2019-11-18 12:05:36 +00:00
Andrew Morgan
bc29a19731 Replace instance variations of homeserver with correct case/spacing 2019-11-12 13:08:12 +00:00
Richard van der Hoff
f41027f746 Use get_events_as_list rather than lots of calls to get_event
It's more efficient and clearer.
2019-11-08 12:21:28 +00:00
Richard van der Hoff
f8407975e7 Update some docstrings and comments 2019-11-08 12:18:20 +00:00
Richard van der Hoff
772d414975 Simplify _update_auth_events_and_context_for_auth
move event_key calculation into _update_context_for_auth_events, since it's
only used there.
2019-11-08 11:40:11 +00:00
Erik Johnston
f713c01e2b
Merge pull request #6295 from matrix-org/erikj/split_purge_history
Split purge API into events vs state and add PurgeEventsStorage
2019-11-08 10:19:15 +00:00
Andrew Morgan
e914cf12f6
Merge pull request #6235 from matrix-org/anoa/room_upgrade_groups 2019-11-07 11:12:22 +00:00
Brendan Abolivier
eda14737cf
Also filter state events 2019-11-06 18:14:03 +00:00
Andrew Morgan
55bc8d531e raise exception after multiple failures 2019-11-06 16:52:54 +00:00
Andrew Morgan
1fe3cc2c9c Address review comments 2019-11-06 14:54:24 +00:00
Andrew Morgan
4059d61e26 Don't forget to ratelimit calls outside of RegistrationHandler 2019-11-06 12:01:54 +00:00
Andrew Morgan
b33c4f7a82 Numeric ID checker now checks @0, don't ratelimit on checking 2019-11-06 11:55:00 +00:00
Erik Johnston
f697b4b4a2 Add failed auth ratelimiting to UIA 2019-11-06 11:08:58 +00:00
Erik Johnston
541f1b92d9 Only do rc_login ratelimiting on succesful login.
We were doing this in a number of places which meant that some login
code paths incremented the counter multiple times.

It was also applying ratelimiting to UIA endpoints, which was probably
not intentional.

In particular, some custom auth modules were calling
`check_user_exists`, which incremented the counters, meaning that people
would fail to login sometimes.
2019-11-06 11:08:58 +00:00
Richard van der Hoff
807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state (#6320)
Fixes a bug where rejected events were persisted with the wrong state group.

Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.

Fixes #6289.
2019-11-06 10:01:39 +00:00
Richard van der Hoff
0e3ab8afdc
Add some checks that we aren't using state from rejected events (#6330)
* Raise an exception if accessing state for rejected events

Add some sanity checks on accessing state_group etc for
rejected events.

* Skip calculating push actions for rejected events

It didn't actually cause any bugs, because rejected events get filtered out at
various later points, but there's not point in trying to calculate the push
actions for a rejected event.
2019-11-05 22:13:37 +00:00
Erik Johnston
c16e192e2f Fix caching devices for remote servers in worker.
When the `/keys/query` API is hit on client_reader worker Synapse may
decide that it needs to resync some remote deivces. Usually this happens
on master, and then gets cached. However, that fails on workers and so
it falls back to fetching devices from remotes directly, which may in
turn fail if the remote is down.
2019-11-05 15:49:43 +00:00
Brendan Abolivier
cb2cbe4d26
Only filter if a filter was provided 2019-11-05 15:28:36 +00:00
Brendan Abolivier
f141af4c79
Update copyright 2019-11-05 14:52:38 +00:00
Brendan Abolivier
1dffa78701
Filter events_before and events_after in /context requests
While the current version of the spec doesn't say much about how this endpoint uses filters (see https://github.com/matrix-org/matrix-doc/issues/2338), the current implementation is that some fields of an EventFilter apply (the ones that are used when running the SQL query) and others don't (the ones that are used by the filter itself) because we don't call event_filter.filter(...). This seems counter-intuitive and probably not what we want so this commit fixes it.
2019-11-05 12:21:59 +00:00
Andrew Morgan
c2203bea57 Re-add docstring, with caveats detailed 2019-11-04 18:17:11 +00:00
Andrew Morgan
0287d033ee Transfer upgraded rooms on groups 2019-11-04 18:08:50 +00:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Erik Johnston
6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Richard van der Hoff
c6516adbe0
Factor out an _AsyncEventContextImpl (#6298)
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
2019-11-01 16:19:09 +00:00
Richard van der Hoff
1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Andrew Morgan
fe1f2b4520
Remove last usages of deprecated logging.warn method (#6314) 2019-11-01 12:03:44 +00:00
Andrew Morgan
ace947e8da
Depublish a room from the public rooms list when it is upgraded (#6232) 2019-11-01 10:28:09 +00:00
Hubert Chathi
6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Amber Brown
020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston
cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Erik Johnston
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston
7c8c97e635 Split purge API into events vs state 2019-10-30 15:23:37 +00:00
Erik Johnston
69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Hubert Chathi
7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Hubert Chathi
bc32f102cd black 2019-10-30 10:07:36 -04:00
Hubert Chathi
d78b1e339d apply changes as a result of PR review 2019-10-30 10:01:53 -04:00
Erik Johnston
ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00