Commit Graph

309 Commits

Author SHA1 Message Date
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message.

`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.

To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
2021-03-17 16:51:55 +00:00
Richard van der Hoff
567f88f835
Prep work for removing outlier from internal_metadata (#9411)
* Populate `internal_metadata.outlier` based on `events` table

Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.

* Move `outlier` out of InternalMetadata._dict

Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
2021-03-17 12:33:18 +00:00
Andrew Morgan
847ecdd8fa
Pass SSO IdP information to spam checker's registration function (#9626)
Fixes https://github.com/matrix-org/synapse/issues/9572

When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.

This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16 12:41:41 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Erik Johnston
7e8083eb48 Add check_media_file_for_spam spam checker hook 2021-02-04 17:01:30 +00:00
Richard van der Hoff
9ffac2bef1
Remote dependency on distutils (#9125)
`distutils` is pretty much deprecated these days, and replaced with
`setuptools`. It's also annoying because it's you can't `pip install` it, and
it's hard to figure out which debian package we should depend on to make sure
it's there.

Since we only use it for a tiny function anyway, let's just vendor said
function into our codebase.
2021-01-15 15:59:20 +00:00
Patrick Cloke
9dde9c9f01
Implement MSC2176: Updated redaction rules (#8984)
An experimental room version ("org.matrix.msc2176") contains
the new redaction rules for testing.
2021-01-05 07:41:48 -05:00
David Teller
f14428b25c
Allow spam-checker modules to be provide async methods. (#8890)
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
2020-12-11 14:05:15 -05:00
Patrick Cloke
473dfec1e5
Use TYPE_CHECKING instead of magic MYPY variable. (#8770) 2020-11-17 09:09:40 -05:00
Erik Johnston
243d427fbc
Block clients from sending server ACLs that lock the local server out. (#8708)
Fixes #4042
2020-11-03 12:13:48 +00:00
Richard van der Hoff
0073fe914a
Use %r rather than %s for stringifying events (#8679)
otherwise non-state events get written as `<FrozenEvent ... state_key='None'>`
which is indistinguishable from state events with the actual state_key `None`.
2020-10-29 12:16:49 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Richard van der Hoff
da0090fdff
Fix modifying events in ThirdPartyRules modules (#8564)
EventBuilder.build wants auth events these days
2020-10-16 13:39:46 +01:00
Richard van der Hoff
5649669c3c
Merge pull request #8535 from matrix-org/rav/third_party_events_updates
Support modifying event content from ThirdPartyRules modules
2020-10-15 20:55:41 +01:00
Richard van der Hoff
4433d01519
Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_invite
Simplify `_locally_reject_invite`
2020-10-15 10:20:19 +01:00
Brendan Abolivier
3ee97a2748
Make sure a retention policy is a state event (#8527)
* Make sure a retention policy is a state event

* Changelog
2020-10-14 12:00:52 +01:00
Richard van der Hoff
a34b17e492 Simplify _locally_reject_invite
Update `EventCreationHandler.create_event` to accept an auth_events param, and
use it in `_locally_reject_invite` instead of reinventing the wheel.
2020-10-13 23:58:48 +01:00
Richard van der Hoff
898196f1cc guard against accidental modification 2020-10-13 23:24:50 +01:00
Richard van der Hoff
617e8a4653 Allow ThirdPartyRules modules to replace event content
Support returning a new event dict from `check_event_allowed`.
2020-10-13 23:24:50 +01:00
Richard van der Hoff
4f0637346a
Combine SpamCheckerApi with the more generic ModuleApi. (#8464)
Lots of different module apis is not easy to maintain.

Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
2020-10-07 12:03:26 +01:00
Richard van der Hoff
a024461130
Additional tests for third-party event rules (#8468)
* Optimise and test state fetching for 3p event rules

Getting all the events at once is much more efficient than getting them
individually

* Test that 3p event rules can modify events
2020-10-06 16:31:31 +01:00
Richard van der Hoff
4cd1448d0e Fix third-party event modules for check_visibility_can_be_modified check
PR #8292 tried to maintain backwards compat with modules which don't provide a
`check_visibility_can_be_modified` method, but the tests weren't being run,
and the check didn't work.
2020-10-05 20:29:52 +01:00
Andrew Morgan
0991a2da93
Allow ThirdPartyEventRules modules to manipulate public room state (#8292)
This PR allows `ThirdPartyEventRules` modules to view, manipulate and block changes to the state of whether a room is published in the public rooms directory.

While the idea of whether a room is in the public rooms list is not kept within an event in the room, `ThirdPartyEventRules` generally deal with controlling which modifications can happen to a room. Public rooms fits within that idea, even if its toggle state isn't controlled through a state event.
2020-10-05 14:57:46 +01:00
Richard van der Hoff
f31f8e6319
Remove stream ordering from Metadata dict (#8452)
There's no need for it to be in the dict as well as the events table. Instead,
we store it in a separate attribute in the EventInternalMetadata object, and
populate that on load.

This means that we can rely on it being correctly populated for any event which
has been persited to the database.
2020-10-05 14:43:14 +01:00
Erik Johnston
ea70f1c362
Various clean ups to room stream tokens. (#8423) 2020-09-29 21:48:33 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Erik Johnston
112266eafd
Add StreamStore to mypy (#8232) 2020-09-02 17:52:38 +01:00
Patrick Cloke
da77520cd1
Convert additional databases to async/await part 2 (#8200) 2020-09-01 08:39:04 -04:00
Brendan Abolivier
420484a334
Allow capping a room's retention policy (#8104) 2020-08-24 18:21:04 +01:00
Patrick Cloke
3f91638da6
Allow denying or shadow banning registrations via the spam checker (#8034) 2020-08-20 15:42:58 -04:00
Richard van der Hoff
318f4e738e
Be more tolerant of membership events in unknown rooms (#8110)
It turns out that not all out-of-band membership events are labelled as such,
so we need to be more accepting here.
2020-08-20 16:42:12 +01:00
Erik Johnston
5dd73d029e
Add type hints to handlers.message and events.builder (#8067) 2020-08-12 15:05:50 +01:00
Patrick Cloke
d4a7829b12
Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke
8553f46498
Convert a synapse.events to async/await. (#7949) 2020-07-27 13:40:22 -04:00
Patrick Cloke
b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Karthikeyan Singaravelan
a7b06a81f0
Fix deprecation warning: import ABC from collections.abc (#7892) 2020-07-20 13:33:04 -04:00
Richard van der Hoff
67593b1728
Add HomeServer.signing_key property (#7805)
... instead of duplicating `config.signing_key[0]` everywhere
2020-07-08 17:51:56 +01:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Patrick Cloke
56b66db78a
Strictly enforce canonicaljson requirements in a new room version (#7381) 2020-05-14 13:24:01 -04:00
Andrew Morgan
67feea8044
Extend spam checker to allow for multiple modules (#7435) 2020-05-08 19:25:48 +01:00
Erik Johnston
7941a70fa8
Fix bug in EventContext.deserialize. (#7393)
This caused `prev_state_ids` to be incorrect if the state event was not
replacing an existing state entry.
2020-05-05 14:17:27 +01:00
Patrick Cloke
06eb5cae08
Remove special auth and redaction rules for aliases events in experimental room ver. (#7037) 2020-03-09 08:58:25 -04:00
Richard van der Hoff
78a15b1f9d
Store room_versions in EventBase objects (#6875)
This is a bit fiddly because it all has to be done on one fell swoop:

* Wherever we create a new event, pass in the room version (and check it matches the format version)
* When we prune an event, use the room version of the unpruned event to create the pruned version.
* When we pass an event over the replication protocol, pass the room version over alongside it, and use it when deserialising the event again.
2020-03-05 15:46:44 +00:00
Richard van der Hoff
3f1cd14791
Merge pull request #6872 from matrix-org/rav/dictproperty
Rewrite _EventInternalMetadata to back it with a dict
2020-02-18 16:21:02 +00:00
Patrick Cloke
10027c80b0
Add type hints to the spam check module (#6915)
Add typing information to the spam checker modules.
2020-02-14 12:49:40 -05:00
Richard van der Hoff
9551911f88 Rewrite _EventInternalMetadata to back it with a _dict
Mostly, this gives mypy an easier time.
2020-02-14 16:42:40 +00:00
Richard van der Hoff
43b2be9764 Replace _event_dict_property with DictProperty
this amounts to the same thing, but replaces `_event_dict` with `_dict`, and
removes some of the function layers generated by `property`.
2020-02-14 16:42:37 +00:00
Patrick Cloke
49f877d32e
Filter the results of user directory searching via the spam checker (#6888)
Add a method to the spam checker to filter the user directory results.
2020-02-14 07:17:54 -05:00
Richard van der Hoff
799001f2c0
Add a make_event_from_dict method (#6858)
... and use it in places where it's trivial to do so.

This will make it easier to pass room versions into the FrozenEvent
constructors.
2020-02-07 15:30:04 +00:00
Richard van der Hoff
f4884444c3
remove unused room_version_to_event_format (#6857) 2020-02-07 09:26:57 +00:00
PeerD
99fcc96289
Third party event rules Update (#6781) 2020-02-06 14:15:29 +00:00
Erik Johnston
b9391c9575
Add typing to SyncHandler (#6821)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-02-03 18:05:44 +00:00
Richard van der Hoff
2a81393a4b Pass room_version into add_hashes_and_signatures 2020-01-30 22:15:50 +00:00
Richard van der Hoff
54f3f369bd Pass room_version into create_local_event_from_event_dict 2020-01-30 22:15:50 +00:00
Richard van der Hoff
ef6bdafb29 Store the room version in EventBuilder 2020-01-30 22:15:50 +00:00
Richard van der Hoff
5a246611e3
Type defintions for use in refactoring for redaction changes (#6803)
* Bump signedjson to 1.1

... so that we can use the type definitions

* Fix breakage caused by upgrade to signedjson 1.1

Thanks, @illicitonion...
2020-01-30 11:25:59 +00:00
Richard van der Hoff
b36095ae5c Set the PL for aliases events to 0. 2020-01-29 11:52:52 +00:00
Richard van der Hoff
ee42a5513e Factor out a copy_power_levels_contents method
I'm going to need another copy (hah!) of this.
2020-01-29 11:52:52 +00:00
Erik Johnston
fa4d609e20
Make 'event.redacts' never raise. (#6771)
There are quite a few places that we assume that a redaction event has a
corresponding `redacts` key, which is not always the case. So lets
cheekily make it so that event.redacts just returns None instead.
2020-01-23 15:19:03 +00:00
Erik Johnston
d386f2f339
Add StateMap type alias (#6715) 2020-01-16 13:31:22 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Brendan Abolivier
9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Richard van der Hoff
807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state (#6320)
Fixes a bug where rejected events were persisted with the wrong state group.

Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.

Fixes #6289.
2019-11-06 10:01:39 +00:00
Richard van der Hoff
0e3ab8afdc
Add some checks that we aren't using state from rejected events (#6330)
* Raise an exception if accessing state for rejected events

Add some sanity checks on accessing state_group etc for
rejected events.

* Skip calculating push actions for rejected events

It didn't actually cause any bugs, because rejected events get filtered out at
various later points, but there's not point in trying to calculate the push
actions for a rejected event.
2019-11-05 22:13:37 +00:00
Richard van der Hoff
4086002827
Improve documentation for EventContext fields (#6319) 2019-11-05 13:23:25 +00:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Richard van der Hoff
c6516adbe0
Factor out an _AsyncEventContextImpl (#6298)
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
2019-11-01 16:19:09 +00:00
Travis Ralston
3a74c03ffb
Expose some homeserver functionality to spam checkers (#6259)
* Offer the homeserver instance to the spam checker

* Newsfile

* Linting

* Expose a Spam Checker API instead of passing the homeserver object

* Alter changelog

* s/hs/api
2019-10-31 09:16:14 -06:00
Richard van der Hoff
9aee28927b Convert EventContext to attrs (#6218)
* make EventContext use an attr
2019-10-28 14:29:55 +02:00
Erik Johnston
55a0c98d16
Merge pull request #5805 from matrix-org/erikj/validate_state
Validate well known state events are state events.
2019-08-01 13:45:48 +01:00
Erik Johnston
76a58fdcce Fix spelling.
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-08-01 13:17:55 +01:00
Erik Johnston
dc4d74e44a Validate well-known state events are state events.
Lets disallow sending things like memberships, topics etc as non-state
events.
2019-07-31 16:36:20 +01:00
Erik Johnston
f31d4cb7a2 Don't allow clients to send tombstones that reference the same room 2019-07-31 15:52:27 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Andrew Morgan
b2a382efdb
Remove the ability to query relations when the original event was redacted. (#5629)
Fixes #5594

Forbid viewing relations on an event once it has been redacted.
2019-07-18 14:41:42 +01:00
Andrew Morgan
ad8b909ce9
Add origin_server_ts and sender fields to m.replace (#5613)
Riot team would like some extra fields as part of m.replace, so here you go.

Fixes: #5598
2019-07-05 17:20:02 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
e0be8d7016
Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events
Add experimental option to reduce extremities.
2019-06-19 13:19:18 +01:00
Erik Johnston
554609288b Run as background process and fix comments 2019-06-19 11:33:03 +01:00
Erik Johnston
b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Brendan Abolivier
33ea87be39
Make check_threepid_can_be_invited async 2019-06-17 17:39:38 +01:00
Brendan Abolivier
112cf5a73a Add third party rules hook for 3PID invites 2019-06-17 17:39:09 +01:00
Erik Johnston
187d2837a9 Add third party rules hook into create room 2019-06-17 16:41:19 +01:00
Brendan Abolivier
f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Tulir Asokan
dd64b9dbdd Fix appservice timestamp massaging (#5233)
Signed-off-by: Tulir Asokan <tulir@maunium.net>
2019-05-24 14:44:04 +01:00
Erik Johnston
de7672b78f Don't bundle events in /sync or /events
As we'll send down the annotations too anyway, so this just ends up
confusing clients.
2019-05-21 13:54:09 +01:00
Richard van der Hoff
04d53794d6
Fix error handling for rooms whose versions are unknown. (#5219)
If we remove support for a particular room version, we should behave more
gracefully. This should make client requests fail with a 400 rather than a 500,
and will ignore individiual PDUs in a federation transaction, rather than the
whole transaction.
2019-05-21 13:47:25 +01:00
Richard van der Hoff
5206648a4a
Add a test room version which updates event ID format (#5210)
Implements MSC1884
2019-05-20 15:54:42 +01:00
Erik Johnston
1dff859d6a Rename relation types to match MSC 2019-05-20 14:31:19 +01:00
Erik Johnston
5dbff34509 Fixup bsaed on review comments 2019-05-17 15:48:04 +01:00
Erik Johnston
d46aab3fa8 Add basic editing support 2019-05-16 16:54:45 +01:00
Erik Johnston
a0603523d2 Add aggregations API 2019-05-16 09:37:20 +01:00
Erik Johnston
b54b03f9e1 Allow client event serialization to be async 2019-05-14 11:58:01 +01:00
Brendan Abolivier
1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128)
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
2019-05-08 17:01:30 +01:00
Andrew Morgan
caa76e6021
Remove periods from copyright headers (#5046) 2019-04-11 17:08:13 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Erik Johnston
5536ddba75 Make prev_state field optional
The `prev_state` field on events is not specced and so synapse shouldn't
explode if an event is missing the field.

Fixes #4787
2019-03-08 15:05:32 +00:00