Commit Graph

406 Commits

Author SHA1 Message Date
Erik Johnston
c9dbee50ae
Fixup pusher pool notifications (#8287)
`pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed.

I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
2020-09-09 16:56:08 +01:00
Erik Johnston
dc9dcdbd59 Revert "Fixup pusher pool notifications"
This reverts commit e7fd336a53.
2020-09-09 16:19:22 +01:00
Erik Johnston
e7fd336a53 Fixup pusher pool notifications 2020-09-09 16:17:50 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Brendan Abolivier
9f8abdcc38
Revert "Add experimental support for sharding event persister. (#8170)" (#8242)
* Revert "Add experimental support for sharding event persister. (#8170)"

This reverts commit 82c1ee1c22.

* Changelog
2020-09-04 10:19:42 +01:00
Erik Johnston
82c1ee1c22
Add experimental support for sharding event persister. (#8170)
This is *not* ready for production yet. Caveats:

1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-02 15:48:37 +01:00
Patrick Cloke
da77520cd1
Convert additional databases to async/await part 2 (#8200) 2020-09-01 08:39:04 -04:00
Andrew Morgan
e0d6244beb
Remove unused parameter from, and add safeguard in, get_room_data (#8174)
Small cleanup PR.

* Removed the unused `is_guest` argument
* Added a safeguard to a (currently) impossible code path, fixing static checking at the same time.
2020-08-26 15:07:35 +01:00
Patrick Cloke
5099bd68da
Do not allow send_nonmember_event to be called with shadow-banned users. (#8158) 2020-08-25 10:52:15 -04:00
Patrick Cloke
cbd8d83da7
Stop shadow-banned users from sending non-member events. (#8142) 2020-08-24 13:58:56 -04:00
Patrick Cloke
eebf52be06
Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
Patrick Cloke
f40645e60b
Convert events worker database to async/await. (#8071) 2020-08-18 16:20:49 -04:00
Patrick Cloke
3c01724b33
Fix the return type of send_nonmember_events. (#8112) 2020-08-18 09:53:13 -04:00
Patrick Cloke
25e55d2598
Return the previous stream token if a non-member event is a duplicate. (#8093) 2020-08-18 07:53:23 -04:00
Richard van der Hoff
53834bb9c4
Run remove_push_actions_from_staging in foreground (#8081)
If we got an error persisting an event, we would try to remove the push actions
asynchronously, which would lead to a 'Re-starting finished log context'
warning.

I don't think there's any need for this to be asynchronous.
2020-08-13 17:05:31 +01:00
Erik Johnston
5dd73d029e
Add type hints to handlers.message and events.builder (#8067) 2020-08-12 15:05:50 +01:00
Richard van der Hoff
fcbab08cbd
Add an assertion on prev_events in create_new_client_event (#8041)
I think this would have caught all the cases in
https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes
more sense here than a 403
2020-08-10 12:29:47 +01:00
Patrick Cloke
d4a7829b12
Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke
d8a9cd8d3e
Remove hacky error handling for inlineDeferreds. (#7950) 2020-07-27 08:35:56 -04:00
Patrick Cloke
cc9bb3dc3f
Convert the message handler to async/await. (#7884) 2020-07-22 12:29:15 -04:00
Patrick Cloke
6fca1b3506
Convert _base, profile, and _receipts handlers to async/await (#7860) 2020-07-17 07:08:30 -04:00
Richard van der Hoff
2ab0b021f1
Generate real events when we reject invites (#7804)
Fixes #2181. 

The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
2020-07-09 10:40:19 +01:00
Patrick Cloke
e060bf4462
Convert directory handler to async/await (#7727) 2020-06-22 07:18:00 -04:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Andrew Morgan
f4e6495b5d
Performance improvements and refactor of Ratelimiter (#7595)
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:

* Rather undocumented, and
* causing a *lot* of config checks

This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. 

Best to be reviewed commit-by-commit.
2020-06-05 10:47:20 +01:00
Richard van der Hoff
f4269694ce
Optimise some references to hs.config (#7546)
These are surprisingly expensive, and we only really need to do them at startup.
2020-05-22 21:47:07 +01:00
Erik Johnston
e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Erik Johnston
1531b214fc
Add ability to wait for replication streams (#7542)
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
Aaron Raimist
250f3eb991
Omit displayname or avatar_url if they aren't set instead of returning null (#7497)
Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec.

Fixes https://github.com/matrix-org/synapse/issues/7333

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-05-19 10:31:25 +01:00
Erik Johnston
1124111a12
Allow censoring of events to happen on workers. (#7492)
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
2020-05-13 17:15:40 +01:00
Brendan Abolivier
d9b8d27494
Add a configuration setting for the dummy event threshold (#7422)
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
Andrew Morgan
6b22921b19
async/await is_server_admin (#7363) 2020-05-01 15:15:36 +01:00
Patrick Cloke
190ab593b7
Use the proper error code when a canonical alias that does not exist is used. (#7109) 2020-03-23 15:21:54 -04:00
Brendan Abolivier
69ce55c510
Don't filter out dummy events when we're checking the visibility of state 2020-03-11 17:52:54 +00:00
Patrick Cloke
7dcbc33a1b
Validate the alt_aliases property of canonical alias events (#6971) 2020-03-03 07:12:45 -05:00
Erik Johnston
1f773eec91
Port PresenceHandler to async/await (#6991) 2020-02-26 15:33:26 +00:00
Richard van der Hoff
a0a1fd0bec Add allow_departed_users param to check_in_room_or_world_readable
... and set it everywhere it's called.

while we're here, rename it for consistency with `check_user_in_room` (and to
help check that I haven't missed any instances)
2020-02-19 08:52:51 +00:00
Richard van der Hoff
5d17c31596 make FederationHandler.send_invite async 2020-02-03 22:28:11 +00:00
Richard van der Hoff
d7bf793cc1 s/get_room_version/get_room_version_id/
... to make way for a forthcoming get_room_version which returns a RoomVersion
object.
2020-01-31 10:06:21 +00:00
Richard van der Hoff
a8ce7aeb43
Pass room version object into event_auth.check and check_redaction (#6788)
These are easier to work with than the strings and we normally have one around.

This fixes `FederationHander._persist_auth_tree` which was passing a
RoomVersion object into event_auth.check instead of a string.
2020-01-28 14:18:29 +00:00
Richard van der Hoff
3bef62488e Remove unused hashes and depths from create_event params 2020-01-06 13:45:33 +00:00
Richard van der Hoff
66ca914dc0 Remove unused hashes and depths from create_new_client_event params 2020-01-06 13:45:33 +00:00
Richard van der Hoff
15720092ac replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in create_new_client_event 2020-01-06 13:45:33 +00:00
Richard van der Hoff
5a04781643 rename get_prev_events_for_room to get_prev_events_and_hashes_for_room
... to make way for a new method which just returns the event ids
2020-01-06 13:45:33 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Andrew Morgan
fc316a4894
Prevent redacted events from appearing in message search (#6377) 2019-12-11 13:39:47 +00:00
Erik Johnston
8ad8bcbed0 Pull out room_invite_state_types config option once.
Pulling things out of config is currently surprisingly expensive.
2019-12-09 11:52:19 +00:00
Brendan Abolivier
54dd5dc12b
Add ephemeral messages support (MSC2228) (#6409)
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:

* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
2019-12-03 19:19:45 +00:00
Brendan Abolivier
708cef88cf
Discard retention policies when retrieving state
Purge jobs don't delete the latest event in a room in order to keep the forward extremity and not break the room. On the other hand, get_state_events, when given an at_token argument calls filter_events_for_client to know if the user can see the event that matches that (sync) token. That function uses the retention policies of the events it's given to filter out those that are too old from a client's view.

Some clients, such as Riot, when loading a room, request the list of members for the latest sync token it knows about, and get confused to the point of refusing to send any message if the server tells it that it can't get that information. This can happen very easily with the message retention feature turned on and a room with low activity so that the last event sent becomes too old according to the room's retention policy.

An easy and clean fix for that issue is to discard the room's retention policies when retrieving state.
2019-11-28 19:26:26 +00:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Amber Brown
020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston
69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Erik Johnston
3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Neil Johnson
034db2ba21 Fix dummy event insertion consent bug (#6053)
Fixes #5905
2019-09-26 11:47:53 +01:00
Erik Johnston
57dd41a45b
Fix comments
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-11 13:54:50 +01:00
Erik Johnston
c64c3bb4c5 Fix how we check for self redaction 2019-09-11 11:18:04 +01:00
Erik Johnston
54ce81c86d Allow use of different ratelimits for admin redactions.
This is useful to allow room admins to quickly deal with a large number
of abusive messages.
2019-09-11 10:46:38 +01:00
Will Hunt
c998f25006
Apply suggestions from code review
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-08-23 10:28:54 +01:00
Half-Shot
9ba32f6573 Exempt bot users 2019-08-23 09:56:31 +01:00
Half-Shot
886eceba3e Return user_type in get_user_by_id 2019-08-23 09:14:52 +01:00
Erik Johnston
cf89266b98 Deny redaction of events in a different room.
We already correctly filter out such redactions, but we should also deny
them over the CS API.
2019-07-31 16:12:27 +01:00
Richard van der Hoff
f30a71a67b
Stop trying to fetch events with event_id=None. (#5753)
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.

I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
2019-07-24 13:16:18 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Richard van der Hoff
9c70a02a9c
Ignore redactions of m.room.create events (#5701) 2019-07-17 19:08:02 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
554609288b Run as background process and fix comments 2019-06-19 11:33:03 +01:00
Erik Johnston
b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Brendan Abolivier
f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Erik Johnston
d16f5574b6
Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events
Don't bundle aggregations with events in /sync or /events or state queries
2019-05-24 10:36:31 +01:00
Erik Johnston
4cb577c23f Don't bundle aggs for /state and /members etc APIs 2019-05-24 09:52:33 +01:00
Erik Johnston
44b8ba484e Fix words 2019-05-21 16:51:45 +01:00
Erik Johnston
c7ec06e8a6 Block attempts to annotate the same event twice 2019-05-20 17:39:05 +01:00
Erik Johnston
b54b03f9e1 Allow client event serialization to be async 2019-05-14 11:58:01 +01:00
Brendan Abolivier
1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128)
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
2019-05-08 17:01:30 +01:00
Andrew Morgan
4a4d5c4fd6
Fix grammar and document get_current_users_in_room (#4998) 2019-04-03 14:32:20 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Erik Johnston
cd62981a6a Revert spurious delete 2019-03-20 17:51:27 +00:00
Erik Johnston
aa959a6c07 Use flags 2019-03-20 17:40:29 +00:00
Erik Johnston
7d47cc1305 Move requester check into assert_accepted_privacy_policy 2019-03-20 17:08:36 +00:00
Erik Johnston
74c46d81fa Only require consent for events with an associated request
There are a number of instances where a server or admin may puppet a
user to join/leave rooms, which we don't want to fail if the user has
not consented to the privacy policy. We fix this by adding a check to
test if the requester has an associated access_token, which is used as a
proxy to answer the question of whether the action is being done on
behalf of a real request from the user.
2019-03-20 16:50:23 +00:00
Neil Johnson
88f0675967 fix test_auto_create_auto_join_where_no_consent (#4886) 2019-03-19 11:38:59 +00:00
Richard van der Hoff
3064952939
Fix incorrect log about not persisting duplicate state event. (#4776)
We were logging this when it was not true.
2019-03-01 16:47:12 +00:00
Erik Johnston
a696c48133 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah 2019-01-29 22:00:33 +00:00
Erik Johnston
5891a6edc8 Correctly set context.app_service 2019-01-29 12:09:10 +00:00
Erik Johnston
7709d2bd16 Implement rechecking of redactions 2019-01-29 11:56:20 +00:00
Erik Johnston
be47cfa9c9 Refactor event building into EventBuilder
This is so that everything is done in one place, making it easier to
change the event format based on room version
2019-01-29 11:13:00 +00:00
Erik Johnston
073f6c2e5e
Merge pull request #4494 from matrix-org/erikj/fixup_event_validator
Split up event validation between event and builder
2019-01-29 10:55:07 +00:00
Erik Johnston
d758d5310e Correctly use default room version if none is set 2019-01-28 17:26:39 +00:00
Erik Johnston
b872c7b1b4 Split up event validation between event and builder
The validator was being run on the EventBuilder objects, and so the
validator only checked a subset of fields. With the upcoming
EventBuilder refactor even fewer fields will be there to validate.

To get around this we split the validation into those that can be run
against an EventBuilder and those run against a fully fledged event.
2019-01-28 17:00:14 +00:00
Erik Johnston
ae2a957dba Pass through room version to event auth 2019-01-25 18:31:41 +00:00
Erik Johnston
a50cf929c1 Require event format version to parse or create events 2019-01-25 10:32:19 +00:00
Erik Johnston
be6a7e47fa
Revert "Require event format version to parse or create events" 2019-01-25 10:23:51 +00:00
Erik Johnston
f431ff3fb8 Require event format version to parse or create events 2019-01-23 20:21:33 +00:00
Richard van der Hoff
9b827c40ca
Log some bits about event creation (#4121)
I found these helpful in debugging my room upgrade tests.
2018-10-31 15:42:23 +00:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00