synapse-product

mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-20 14:04:19 -05:00

Author	SHA1	Message	Date
Richard van der Hoff	adfaea8c69	Implement GET /_matrix/client/r0/rooms/{roomId}/aliases (#6939 ) per matrix-org/matrix-doc#2432	2020-02-18 16:23:25 +00:00
Brendan Abolivier	d484126bf7	Merge pull request #6907 from matrix-org/babolivier/acme-config Add mention and warning about ACME v1 deprecation to the TLS config	2020-02-18 16:11:31 +00:00
Brendan Abolivier	9801a042f3	Make the log more noticeable	2020-02-18 15:15:43 +00:00
Brendan Abolivier	bfbe2f5b08	Print the error as an error log and raise the same exception we got	2020-02-18 15:10:41 +00:00
Patrick Cloke	fe3941f6e3	Stop sending events when creating or deleting aliases (#6904 ) Stop sending events when creating or deleting associations (room aliases). Send an updated canonical alias event if one of the alt_aliases is deleted.	2020-02-18 07:29:44 -05:00
Andrew Morgan	3404ad289b	Raise the default power levels for invites, tombstones and server acls (#6834 )	2020-02-17 13:23:37 +00:00
Patrick Cloke	49f877d32e	Filter the results of user directory searching via the spam checker (#6888 ) Add a method to the spam checker to filter the user directory results.	2020-02-14 07:17:54 -05:00
Brendan Abolivier	65bdc35a1f	Lint	2020-02-13 16:14:15 +00:00
Brendan Abolivier	0cb83cde70	Lint	2020-02-13 16:06:31 +00:00
Brendan Abolivier	12bbcc255a	Add a comprehensive error when failing to register for an ACME account	2020-02-13 14:58:34 +00:00
Richard van der Hoff	605cd089f7	Merge branch 'release-v1.10.0' into develop	2020-02-11 10:43:47 +00:00
Patrick Cloke	a92e703ab9	Reject device display names that are too long (#6882 ) * Reject device display names that are too long. Too long is currently defined as 100 characters in length. * Add a regression test for rejecting a too long device display name.	2020-02-10 16:35:26 -05:00
Matthew Hodgson	01209382fb	filter out m.room.aliases from /sync state blocks (#6884 ) We forgot to filter out aliases from /sync state blocks as well as the timeline.	2020-02-10 18:07:35 +00:00
Erik Johnston	21db35f77e	Add support for putting fed user query API on workers (#6873 )	2020-02-07 15:45:39 +00:00
Erik Johnston	de2d267375	Allow moving group read APIs to workers (#6866 )	2020-02-07 11:14:19 +00:00
Dirk Klimpel	56ca93ef59	Admin api to add an email address (#6789 )	2020-02-07 10:29:36 +00:00
Patrick Cloke	7765bf3989	Limit the number of events that can be requested when backfilling events (#6864 ) Limit the maximum number of events requested when backfilling events.	2020-02-06 13:25:24 -05:00
Richard van der Hoff	b0c8bdd49d	pass room version into FederationClient.send_join (#6854 ) ... which allows us to sanity-check the create event.	2020-02-06 15:50:39 +00:00
PeerD	99fcc96289	Third party event rules Update (#6781 )	2020-02-06 14:15:29 +00:00
Erik Johnston	ed630ea17c	Reduce amount of logging at INFO level. (#6862 ) A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>	2020-02-06 13:31:05 +00:00
Richard van der Hoff	9bcd37146e	Merge pull request #6823 from matrix-org/rav/redact_changes/5 pass room versions around	2020-02-06 11:32:33 +00:00
Erik Johnston	2201ef8556	Synapse 1.10.0rc2 (2020-02-06) ============================== Bugfixes -------- - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844)) - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848)) Internal Changes ---------------- - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850)) -----BEGIN PGP SIGNATURE----- iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl478l8QHGVyaWtAbWF0 cml4Lm9yZwAKCRA62gbtx1PRHmKfD/0ZpBV95gMk0iecfvO6IY8+0KigzDUypgXf zp1k+k5DBUmH5/83h7+cXWHIJyofv1At1YdIq9J/nNeyqr3V3dcQwPabOKYbI96d kVD0EZX+2NjuYejAuF9eJNdiwRq5yMLluCfnOXr5jCS0NNdpO1xVb4bWYm50RmcD WJvBvmfwXwCC3AFyNb4IAipaVUXuVXXx1YhjFs6DsbtpPUG88e4uIhpfvrS4v0t8 GFJZNuVJStOvyKHHTfNRIMkhy4xidj3HRUMSmjNpx07WnPBnbv4LnUOD1boYxgaP EoODYnoAPshdiKB0AywNNjue3TmFD/Z7vVEzlFP/lNZ4GDU4kN9C/EwFYw0C2Jqq f9O/E2ZuP9Qqume5O9UwMQQAmhV5lMBaIsYRbixU+9bSB893zRHRffA11HypzD00 ZXj8QDOXpiXp2jpAwN1Rk9e/aZEX3qd+zUAgRmk+kYb0KTC2/gcY956rodNSSO/U RFYg4DFvoCgCrRSxZ4LQMlFu3YY2E7qWH+p/OHk3WG79jW64VpEFaytvv8fR+GKq g3EbtWy6mUzlKYqbjZ+ZTUDe5AWdv6ZX8xJqRD/S6cpiwyh6Gp89HHNvBThXoCmK fxkgw8if7eIITuTlNuDrqunxyWqwd3oVlzd2mi2bg0yRfcqJ9C6OuBV1VTLFZeky 3sjCiU0IRw== =kcSy -----END PGP SIGNATURE----- Merge tag 'v1.10.0rc2' into develop Synapse 1.10.0rc2 (2020-02-06) ============================== Bugfixes -------- - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844)) - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848)) Internal Changes ---------------- - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))	2020-02-06 11:04:03 +00:00
Richard van der Hoff	f84700fba8	Pass room version object into `FederationClient.get_pdu`	2020-02-05 17:25:46 +00:00
Erik Johnston	a58860e480	Check sender_key matches on inbound encrypted events. (#6850 ) If they don't then the device lists are probably out of sync.	2020-02-05 14:02:39 +00:00
Erik Johnston	6475382d80	Fix detecting unknown devices from remote encrypted events. (#6848 ) We were looking at the wrong event type (`m.room.encryption` vs `m.room.encrypted`). Also fixup the duplicate `EvenTypes` entries. Introduced in #6776.	2020-02-04 17:25:54 +00:00
Richard van der Hoff	c7d6d5c69e	Merge pull request #6837 from matrix-org/rav/federation_async Port much of `synapse.handlers.federation` to async/await.	2020-02-04 12:06:18 +00:00
Richard van der Hoff	5d17c31596	make FederationHandler.send_invite async	2020-02-03 22:28:11 +00:00
Richard van der Hoff	e81c093974	make FederationHandler.on_get_missing_events async	2020-02-03 19:15:08 +00:00
Erik Johnston	b9391c9575	Add typing to SyncHandler (#6821 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2020-02-03 18:05:44 +00:00
Richard van der Hoff	f64c96662e	make FederationHandler.user_joined_room async	2020-02-03 16:29:30 +00:00
Richard van der Hoff	52642860da	make FederationHandler._clean_room_for_join async	2020-02-03 16:29:30 +00:00
Richard van der Hoff	814cc00cb9	make FederationHandler._notify_persisted_event async	2020-02-03 16:29:30 +00:00
Richard van der Hoff	05299599b6	make FederationHandler.persist_events_and_notify async	2020-02-03 16:29:30 +00:00
Richard van der Hoff	3b7e0e002b	make FederationHandler._make_and_verify_event async	2020-02-03 16:22:30 +00:00
Richard van der Hoff	4286e429a7	make FederationHandler.do_remotely_reject_invite async	2020-02-03 16:19:18 +00:00
Richard van der Hoff	c3f296af32	make FederationHandler._check_for_soft_fail async	2020-02-03 16:16:31 +00:00
Richard van der Hoff	dbdf843012	make FederationHandler._persist_auth_tree async	2020-02-03 16:14:58 +00:00
Richard van der Hoff	ebd6a15af3	make FederationHandler.do_invite_join async	2020-02-03 16:13:13 +00:00
Richard van der Hoff	94f7b4cd54	make FederationHandler.on_event_auth async	2020-02-03 16:06:46 +00:00
Richard van der Hoff	863087d186	make FederationHandler.on_exchange_third_party_invite_request async	2020-02-03 16:02:50 +00:00
Richard van der Hoff	957129f4a7	make FederationHandler.construct_auth_difference async	2020-02-03 16:00:46 +00:00
Richard van der Hoff	0d5f2f4bb0	make FederationHandler._update_context_for_auth_events async	2020-02-03 15:55:35 +00:00
Richard van der Hoff	a25ddf26a3	make FederationHandler._update_auth_events_and_context_for_auth async	2020-02-03 15:53:54 +00:00
Richard van der Hoff	bc9b75c6f0	make FederationHandler.do_auth async	2020-02-03 15:51:24 +00:00
Richard van der Hoff	8033b257a7	make FederationHandler._prep_event async	2020-02-03 15:49:32 +00:00
Richard van der Hoff	1cdc253e0a	make FederationHandler._handle_new_event async	2020-02-03 15:48:33 +00:00
Richard van der Hoff	c556ed9e15	make FederationHandler._handle_new_events async	2020-02-03 15:43:51 +00:00
Richard van der Hoff	6e89ec5e32	make FederationHandler.on_make_leave_request async	2020-02-03 15:40:41 +00:00
Richard van der Hoff	d184cbc031	make FederationHandler.on_send_leave_request async	2020-02-03 15:39:24 +00:00
Richard van der Hoff	98681f90cb	make FederationHandler.on_make_join_request async	2020-02-03 15:38:02 +00:00
Richard van der Hoff	af8ba6b525	make FederationHandler.on_invite_request async	2020-02-03 15:33:42 +00:00
Richard van der Hoff	7571bf86f0	make FederationHandler.on_send_join_request async	2020-02-03 15:32:48 +00:00
Richard van der Hoff	b3e44f0bdf	make FederationHandler.on_query_auth async	2020-02-03 15:30:23 +00:00
Richard van der Hoff	7f93eb1903	pass room_version into compute_event_signature (#6807 )	2020-01-31 13:47:43 +00:00
Richard van der Hoff	160522e32c	Merge pull request #6820 from matrix-org/rav/get_room_version_id Make `get_room_version` return a RoomVersion object	2020-01-31 10:56:42 +00:00
Richard van der Hoff	d7bf793cc1	s/get_room_version/get_room_version_id/ ... to make way for a forthcoming get_room_version which returns a RoomVersion object.	2020-01-31 10:06:21 +00:00
Erik Johnston	7d846e8704	Fix bug with getting missing auth event during join 500'ed (#6810 )	2020-01-31 09:49:13 +00:00
Richard van der Hoff	46a446828d	pass room version into FederationHandler.on_invite_request (#6805 )	2020-01-30 22:13:02 +00:00
Erik Johnston	b660327056	Resync remote device list when detected as stale. (#6786 )	2020-01-30 17:06:38 +00:00
Richard van der Hoff	dcd85b976d	Make /directory/room/<alias> handle restrictive power levels Fixes a bug where the alias would be added, but `PUT /directory/room/<alias>` would return a 403.	2020-01-29 11:52:52 +00:00
Richard van der Hoff	b36095ae5c	Set the PL for aliases events to 0.	2020-01-29 11:52:52 +00:00
Richard van der Hoff	ee42a5513e	Factor out a `copy_power_levels_contents` method I'm going to need another copy (hah!) of this.	2020-01-29 11:52:52 +00:00
Erik Johnston	2cad8baa70	Fix bug when querying remote user keys that require a resync. (#6796 ) We ended up only returning a single device, rather than all of them.	2020-01-29 09:56:41 +00:00
Erik Johnston	e17a110661	Detect unknown remote devices and mark cache as stale (#6776 ) We just mark the fact that the cache may be stale in the database for now.	2020-01-28 14:43:21 +00:00
Richard van der Hoff	a8ce7aeb43	Pass room version object into event_auth.check and check_redaction (#6788 ) These are easier to work with than the strings and we normally have one around. This fixes `FederationHander._persist_auth_tree` which was passing a RoomVersion object into event_auth.check instead of a string.	2020-01-28 14:18:29 +00:00
Erik Johnston	8df862e45d	Add `rooms.room_version` column (#6729 ) This is so that we don't have to rely on pulling it out from `current_state_events` table.	2020-01-27 14:30:57 +00:00
Andrew Morgan	9f7aaf90b5	Validate client_secret parameter (#6767 )	2020-01-24 14:28:40 +00:00
Andrew Morgan	ce84dd9e20	Remove unnecessary abstractions in admin handler (#6751 )	2020-01-22 15:09:57 +00:00
Brendan Abolivier	a17f64361c	Add more logging around message retention policies support (#6717 ) So we can debug issues like #6683 more easily	2020-01-17 20:51:44 +00:00
Brendan Abolivier	3b72bb780a	Merge pull request #6714 from matrix-org/babolivier/retention_select_event Fix instantiation of message retention purge jobs	2020-01-17 14:23:51 +00:00
Richard van der Hoff	59dc87c618	Merge pull request #6724 from matrix-org/rav/log_saml_attributes Log saml assertions rather than the whole response	2020-01-17 10:33:24 +00:00
Richard van der Hoff	2b6a77fcde	Delegate remote_user_id mapping to the saml mapping provider (#6723 ) Turns out that figuring out a remote user id for the SAML user isn't quite as obvious as it seems. Factor it out to the SamlMappingProvider so that it's easy to control.	2020-01-17 10:32:47 +00:00
Richard van der Hoff	acc7820574	Log saml assertions rather than the whole response ... since the whole response is huge. We even need to break up the assertions, since kibana otherwise truncates them.	2020-01-16 22:26:34 +00:00
Erik Johnston	d386f2f339	Add StateMap type alias (#6715 )	2020-01-16 13:31:22 +00:00
Erik Johnston	48c3a96886	Port synapse.replication.tcp to async/await (#6666 ) * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing....	2020-01-16 09:16:12 +00:00
Brendan Abolivier	855af069a4	Fix instantiation of message retention purge jobs When figuring out which topological token to start a purge job at, we need to do the following: 1. Figure out a timestamp before which events will be purged 2. Select the first stream ordering after that timestamp 3. Select info about the first event after that stream ordering 4. Build a topological token from that info In some situations (e.g. quiet rooms with a short max_lifetime), there might not be an event after the stream ordering at step 3, therefore we abort the purge with the error `No event found`. To mitigate that, this patch fetches the first event _before_ the stream ordering, instead of after.	2020-01-15 18:56:18 +00:00
Erik Johnston	28c98e51ff	Add `local_current_membership` table (#6655 ) Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.	2020-01-15 14:59:33 +00:00
Richard van der Hoff	326c893d24	Kill off RegistrationError (#6691 ) This is pretty pointless. Let's just use SynapseError.	2020-01-13 12:48:22 +00:00
Richard van der Hoff	2d07c73777	Don't assign numeric IDs for empty usernames (#6690 ) Fix a bug where we would assign a numeric userid if somebody tried registering with an empty username	2020-01-13 12:47:30 +00:00
Richard van der Hoff	dc69a1cf43	Pass client redirect URL into SAML mapping providers	2020-01-12 21:40:49 +00:00
Richard van der Hoff	47e63cc67a	Pass the module_api into the SamlMappingProvider ... for consistency with other modules, and because we'll need it sooner or later and it will be a pain to introduce later.	2020-01-12 21:40:49 +00:00
Manuel Stahl	d2906fe666	Allow admin users to create or modify users without a shared secret (#6495 ) Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>	2020-01-09 13:31:00 +00:00
Erik Johnston	be29ed7ad8	Correctly proxy remote group HTTP errors. (#6654 ) e.g. if remote returns a 404 then that shouldn't be treated as an error but should be proxied through.	2020-01-07 15:36:41 +00:00
Brendan Abolivier	0ab5853ec9	Merge pull request #6652 from matrix-org/babolivier/depth_missing_events Fix conditions failing if min_depth = 0	2020-01-07 15:22:07 +01:00
Brendan Abolivier	cd428a93e2	Fix conditions failing if min_depth = 0 This could result in Synapse not fetching prev_events for new events in the room if it has missed some events.	2020-01-07 12:08:58 +00:00
Richard van der Hoff	1807db5e73	Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashes Remove a bunch of unused code from event creation	2020-01-06 17:46:31 +00:00
Richard van der Hoff	ba897a7590	Fix some test failures when frozen_dicts are enabled (#6642 ) Fixes #4026	2020-01-06 15:22:46 +00:00
Richard van der Hoff	ab4b4ee6a7	Fix an error which was thrown by the PresenceHandler _on_shutdown handler. (#6640 )	2020-01-06 14:34:02 +00:00
Richard van der Hoff	38e0829a4c	Remove unused hashes and depths from _update_membership params	2020-01-06 13:45:33 +00:00
Richard van der Hoff	3bef62488e	Remove unused hashes and depths from create_event params	2020-01-06 13:45:33 +00:00
Richard van der Hoff	66ca914dc0	Remove unused hashes and depths from create_new_client_event params	2020-01-06 13:45:33 +00:00
Richard van der Hoff	15720092ac	replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in create_new_client_event	2020-01-06 13:45:33 +00:00
Richard van der Hoff	5a04781643	rename get_prev_events_for_room to get_prev_events_and_hashes_for_room ... to make way for a new method which just returns the event ids	2020-01-06 13:45:33 +00:00
Andrew Morgan	01c3c6c929	Fix power levels being incorrectly set in old and new rooms after a room upgrade (#6633 ) Modify a copy of an upgraded room's PL before sending to the new room	2020-01-06 09:53:07 +00:00
Richard van der Hoff	56ad230efa	Merge branch 'master' into develop	2019-12-20 11:11:34 +00:00
Erik Johnston	fa780e9721	Change EventContext to use the Storage class (#6564 )	2019-12-20 10:32:02 +00:00
Richard van der Hoff	03d3792f3c	Fix exceptions when attempting to backfill (#6576 ) Fixes #6575	2019-12-20 09:55:45 +00:00
Erik Johnston	3d46124ad0	Port some admin handlers to async/await (#6559 )	2019-12-19 15:07:28 +00:00
Erik Johnston	2284eb3a53	Add database config class (#6513 ) This encapsulates config for a given database and is the way to get new connections.	2019-12-18 10:45:12 +00:00
Richard van der Hoff	6e8f8e14f2	Merge release-v1.7.1 into develop	2019-12-18 09:51:51 +00:00
Erik Johnston	02553901ce	Remove unused `get_pagination_rows` methods. (#6557 ) Remove unused get_pagination_rows methods	2019-12-17 11:44:32 +00:00
Brendan Abolivier	6316530366	Merge pull request #6553 from matrix-org/babolivier/fix-context-filter Use the filtered version of an event when responding to /context requests for that event	2019-12-16 16:24:10 +00:00
Will Hunt	bfb95654c9	Add option to allow profile queries without sharing a room (#6523 )	2019-12-16 16:11:55 +00:00
Brendan Abolivier	a820069549	Incorporate review	2019-12-16 16:00:18 +00:00
Richard van der Hoff	bbb75ff6ee	Exclude rejected state events when calculating state at backwards extrems (#6527 ) This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.	2019-12-16 14:07:29 +00:00
Richard van der Hoff	ff773ff724	Persist auth/state events at backwards extremities when we fetch them (#6526 ) The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.	2019-12-16 14:07:11 +00:00
Richard van der Hoff	83895316d4	sanity-checking for events used in state res (#6531 ) When we perform state resolution, check that all of the events involved are in the right room.	2019-12-16 13:53:53 +00:00
Richard van der Hoff	35bbe4ca79	Check the room_id of events when fetching room state/auth (#6524 ) When we request the state/auth_events to populate a backwards extremity (on backfill or in the case of missing events in a transaction push), we should check that the returned events are in the right room rather than blindly using them in the room state or auth chain. Given that _get_events_from_store_or_dest takes a room_id, it seems clear that it should be sanity-checking the room_id of the requested events, so let's do it there.	2019-12-16 13:47:07 +00:00
Richard van der Hoff	20d5ba16e6	Add `include_event_in_state` to _get_state_for_room (#6521 ) Make it return the state after the requested event, rather than the one before it. This is a bit easier and requires fewer calls to get_events_from_store_or_dest.	2019-12-16 13:26:12 +00:00
Richard van der Hoff	be294d6fde	Move get_state methods into FederationHandler (#6503 ) This is a non-functional refactor as a precursor to some other work.	2019-12-16 13:20:21 +00:00
Richard van der Hoff	6920d88892	Exclude rejected state events when calculating state at backwards extrems (#6527 ) This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.	2019-12-16 13:14:37 +00:00
Richard van der Hoff	bc7de87650	Persist auth/state events at backwards extremities when we fetch them (#6526 ) The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.	2019-12-16 12:26:28 +00:00
Brendan Abolivier	487f1bb49d	Use the filtered version of an event when responding to /context requests for that event Sometimes the filtering function can return a pruned version of an event (on top of either the event itself or an empty list), if it thinks the user should be able to see that there's an event there but not the content of that event. Therefore, the previous logic of 'if filtered is empty then we can use the event we retrieved from the database' is flawed, and we should use the event returned by the filtering function.	2019-12-16 12:14:12 +00:00
Richard van der Hoff	1da15f05f5	sanity-checking for events used in state res (#6531 ) When we perform state resolution, check that all of the events involved are in the right room.	2019-12-13 12:55:32 +00:00
Erik Johnston	caa52836e4	Merge pull request #6496 from matrix-org/erikj/initial_sync_asnyc Port synapse.handlers.initial_sync to async/await	2019-12-13 10:01:51 +00:00
Hubert Chathi	cb2db17994	look up cross-signing keys from the DB in bulk (#6486 )	2019-12-12 12:03:28 -05:00
Richard van der Hoff	25f1244329	Check the room_id of events when fetching room state/auth (#6524 ) When we request the state/auth_events to populate a backwards extremity (on backfill or in the case of missing events in a transaction push), we should check that the returned events are in the right room rather than blindly using them in the room state or auth chain. Given that _get_events_from_store_or_dest takes a room_id, it seems clear that it should be sanity-checking the room_id of the requested events, so let's do it there.	2019-12-12 12:57:45 +00:00
Erik Johnston	6828b47c45	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/initial_sync_asnyc	2019-12-11 17:01:41 +00:00
Richard van der Hoff	2045356517	Add `include_event_in_state` to _get_state_for_room (#6521 ) Make it return the state after the requested event, rather than the one before it. This is a bit easier and requires fewer calls to get_events_from_store_or_dest.	2019-12-11 16:37:51 +00:00
Richard van der Hoff	894d2addac	Merge pull request #6517 from matrix-org/rav/event_auth/13 Port some of FederationHandler to async/await	2019-12-11 16:36:06 +00:00
Erik Johnston	31905a518e	Merge pull request #6504 from matrix-org/erikj/account_validity_async_await Port handlers.account_validity to async/await.	2019-12-11 14:49:26 +00:00
Richard van der Hoff	6637d90d77	convert to async: FederationHandler._process_received_pdu also fix user_joined_room to consistently return deferreds	2019-12-11 14:39:26 +00:00
Richard van der Hoff	4db394a4b3	convert to async: FederationHandler._get_state_for_room ... and _get_events_from_store_or_dest	2019-12-11 14:39:26 +00:00
Richard van der Hoff	e77237b935	convert to async: FederationHandler.on_receive_pdu and associated functions: * on_receive_pdu * handle_queued_pdus * get_missing_events_for_pdu	2019-12-11 14:39:26 +00:00
Richard van der Hoff	7712e751b8	Convert federation backfill to async PaginationHandler.get_messages is only called by RoomMessageListRestServlet, which is async. Chase the code path down from there: - FederationHandler.maybe_backfill (and nested try_backfill) - FederationHandler.backfill	2019-12-11 14:39:25 +00:00
Richard van der Hoff	7c429f92d6	Clean up some logging (#6515 ) This just makes some of the logging easier to follow when things start going wrong.	2019-12-11 14:32:25 +00:00
Andrew Morgan	fc316a4894	Prevent redacted events from appearing in message search (#6377 )	2019-12-11 13:39:47 +00:00
Andrew Morgan	ea0f0ad414	Prevent message search in upgraded rooms we're not in (#6385 )	2019-12-11 13:07:25 +00:00
Richard van der Hoff	f8bc2ae883	Move get_state methods into FederationHandler (#6503 ) This is a non-functional refactor as a precursor to some other work.	2019-12-10 17:42:46 +00:00
Andrew Morgan	4947de5a14	Allow SAML username provider plugins (#6411 )	2019-12-10 17:30:16 +00:00
Erik Johnston	257ef2c727	Port handlers.account_validity to async/await.	2019-12-10 14:40:15 +00:00
Erik Johnston	e3f528c544	Merge pull request #6506 from matrix-org/erikj/remove_snapshot_cache Remove SnapshotCache in favour of ResponseCache	2019-12-10 13:04:50 +00:00
Erik Johnston	353396e3a7	Port handlers.account_data to async/await.	2019-12-10 11:12:56 +00:00
Neil Johnson	adfdd82b21	Back out perf regression from get_cross_signing_keys_from_cache. (#6494 ) Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression.	2019-12-09 13:59:27 +00:00
Erik Johnston	a1f8ea9051	Port synapse.handlers.initial_sync to async/await	2019-12-09 13:46:45 +00:00
Erik Johnston	f166a8d1f5	Remove SnapshotCache in favour of ResponseCache	2019-12-09 13:42:49 +00:00
Erik Johnston	e1544b0af8	Merge pull request #6493 from matrix-org/erikj/invite_state_config Pull out room_invite_state_types config option once.	2019-12-09 12:23:21 +00:00
Erik Johnston	8ad8bcbed0	Pull out room_invite_state_types config option once. Pulling things out of config is currently surprisingly expensive.	2019-12-09 11:52:19 +00:00
Erik Johnston	a9b393340f	Merge pull request #6484 from matrix-org/erikj/port_sync_handler Port SyncHandler to async/await	2019-12-09 11:32:44 +00:00
Erik Johnston	b3a4e35ca8	Fixup functions to consistently return deferreds	2019-12-06 10:40:05 +00:00
Manuel Stahl	649b6bc088	Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925 )	2019-12-05 18:12:23 +00:00
Erik Johnston	8437e2383e	Port SyncHandler to async/await	2019-12-05 17:58:25 +00:00
Richard van der Hoff	63d6ad1064	Stronger typing in the federation handler (#6480 ) replace the event_info dict with an attrs thing	2019-12-05 15:02:35 +00:00
Richard van der Hoff	e1f4c83f41	Sanity-check the rooms of auth events before pulling them in. (#6472 )	2019-12-05 14:14:45 +00:00
Richard van der Hoff	e203874caa	get rid of (most of) have_events from _update_auth_events_and_context_for_auth (#6468 ) have_events was a map from event_id to rejection reason (or None) for events which are in our local database. It was used as filter on the list of event_ids being passed into get_events_as_list. However, since get_events_as_list will ignore any event_ids that are unknown or rejected, we can equivalently just leave it to get_events_as_list to do the filtering. That means that we don't have to keep `have_events` up-to-date, and can use `have_seen_events` instead of `get_seen_events_with_rejection` in the one place we do need it.	2019-12-04 17:27:32 +00:00
Erik Johnston	b9449012db	Merge pull request #6441 from syamgk/fix-parameter-mismatch Fix issue #6406 parameter mismatch	2019-12-04 15:42:33 +00:00
Brendan Abolivier	c530f9af4d	Merge pull request #6329 from matrix-org/babolivier/context_filters Filter state, events_before and events_after in /context requests	2019-12-04 15:24:16 +00:00
Brendan Abolivier	9dc84b7989	Merge branch 'develop' into babolivier/context_filters	2019-12-04 14:23:44 +00:00
Brendan Abolivier	08a436ecb2	Incorporate review	2019-12-04 14:18:46 +00:00
Syam G Krishnan	ce1c975ebc	Issue #6406 Fix parameter mismatch Signed-off-by: Syam G Krishnan <syamgk01@gmail.com>	2019-12-04 13:03:46 +05:30
Brendan Abolivier	54dd5dc12b	Add ephemeral messages support (MSC2228) (#6409 ) Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are: * the feature is hidden behind a configuration flag (`enable_ephemeral_messages`) * self-destruction doesn't happen for state events * only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one) * doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database	2019-12-03 19:19:45 +00:00
Andrew Morgan	72078e4be5	Transfer power level state events on room upgrade (#6237 )	2019-12-02 15:11:32 +00:00
Erik Johnston	1c3a61529f	Merge pull request #6434 from matrix-org/erikj/msc2367_membership_reasons Implement MSC 2367 - Membership Reasons	2019-11-29 13:30:36 +00:00
Brendan Abolivier	708cef88cf	Discard retention policies when retrieving state Purge jobs don't delete the latest event in a room in order to keep the forward extremity and not break the room. On the other hand, get_state_events, when given an at_token argument calls filter_events_for_client to know if the user can see the event that matches that (sync) token. That function uses the retention policies of the events it's given to filter out those that are too old from a client's view. Some clients, such as Riot, when loading a room, request the list of members for the latest sync token it knows about, and get confused to the point of refusing to send any message if the server tells it that it can't get that information. This can happen very easily with the message retention feature turned on and a room with low activity so that the last event sent becomes too old according to the room's retention policy. An easy and clean fix for that issue is to discard the room's retention policies when retrieving state.	2019-11-28 19:26:26 +00:00
Erik Johnston	2173785f0d	Propagate reason in remotely rejected invites	2019-11-28 11:31:56 +00:00
Andrew Morgan	a9c44d4008	Remove local threepids on account deactivation (#6426 )	2019-11-28 10:40:42 +00:00
Hubert Chathi	0d27aba900	add etag and count to key backup endpoints (#5858 )	2019-11-27 16:14:44 -05:00
Brendan Abolivier	d31f69afa0	Merge pull request #6358 from matrix-org/babolivier/message_retention Implement message retention policies (MSC1763)	2019-11-27 15:04:38 +00:00
Richard van der Hoff	651d930f16	Merge pull request #6343 from matrix-org/rav/event_auth/4 Refactor _update_auth_events_and_context_for_auth	2019-11-26 18:15:03 +00:00
Brendan Abolivier	9e937c28ee	Merge branch 'develop' into babolivier/message_retention	2019-11-26 17:53:57 +00:00
Erik Johnston	f9f1c8acbb	Merge pull request #6332 from matrix-org/erikj/query_devices_fix Fix caching devices for remote servers in worker.	2019-11-26 12:56:05 +00:00
Richard van der Hoff	4d394d6415	remove confusing fixme	2019-11-26 12:32:37 +00:00
Andrew Morgan	3916e1b97a	Clean up newline quote marks around the codebase (#6362 )	2019-11-21 12:00:14 +00:00
Brendan Abolivier	83446a18fb	Merge pull request #6335 from matrix-org/erikj/rc_login_cleanups Only do `rc_login` ratelimiting on succesful login.	2019-11-20 09:52:38 +00:00
Brendan Abolivier	a6fc6754f8	Fix 3PID invite exchange	2019-11-19 14:07:39 +00:00
Brendan Abolivier	7c24d0f443	Lint	2019-11-19 13:22:37 +00:00
Erik Johnston	c7376cdfe3	Apply suggestions from code review Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>	2019-11-18 17:10:16 +00:00
Richard van der Hoff	870c00e278	Merge remote-tracking branch 'origin/develop' into rav/event_auth/4	2019-11-18 12:05:36 +00:00
Andrew Morgan	bc29a19731	Replace instance variations of homeserver with correct case/spacing	2019-11-12 13:08:12 +00:00
Richard van der Hoff	f41027f746	Use get_events_as_list rather than lots of calls to get_event It's more efficient and clearer.	2019-11-08 12:21:28 +00:00
Richard van der Hoff	f8407975e7	Update some docstrings and comments	2019-11-08 12:18:20 +00:00
Richard van der Hoff	772d414975	Simplify _update_auth_events_and_context_for_auth move event_key calculation into _update_context_for_auth_events, since it's only used there.	2019-11-08 11:40:11 +00:00
Erik Johnston	f713c01e2b	Merge pull request #6295 from matrix-org/erikj/split_purge_history Split purge API into events vs state and add PurgeEventsStorage	2019-11-08 10:19:15 +00:00
Andrew Morgan	e914cf12f6	Merge pull request #6235 from matrix-org/anoa/room_upgrade_groups	2019-11-07 11:12:22 +00:00
Brendan Abolivier	eda14737cf	Also filter state events	2019-11-06 18:14:03 +00:00
Andrew Morgan	55bc8d531e	raise exception after multiple failures	2019-11-06 16:52:54 +00:00
Andrew Morgan	1fe3cc2c9c	Address review comments	2019-11-06 14:54:24 +00:00
Andrew Morgan	4059d61e26	Don't forget to ratelimit calls outside of RegistrationHandler	2019-11-06 12:01:54 +00:00
Andrew Morgan	b33c4f7a82	Numeric ID checker now checks @0, don't ratelimit on checking	2019-11-06 11:55:00 +00:00
Erik Johnston	f697b4b4a2	Add failed auth ratelimiting to UIA	2019-11-06 11:08:58 +00:00
Erik Johnston	541f1b92d9	Only do `rc_login` ratelimiting on succesful login. We were doing this in a number of places which meant that some login code paths incremented the counter multiple times. It was also applying ratelimiting to UIA endpoints, which was probably not intentional. In particular, some custom auth modules were calling `check_user_exists`, which incremented the counters, meaning that people would fail to login sometimes.	2019-11-06 11:08:58 +00:00
Richard van der Hoff	807ec3bd99	Fix bug which caused rejected events to be stored with the wrong room state (#6320 ) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.	2019-11-06 10:01:39 +00:00
Richard van der Hoff	0e3ab8afdc	Add some checks that we aren't using state from rejected events (#6330 ) * Raise an exception if accessing state for rejected events Add some sanity checks on accessing state_group etc for rejected events. * Skip calculating push actions for rejected events It didn't actually cause any bugs, because rejected events get filtered out at various later points, but there's not point in trying to calculate the push actions for a rejected event.	2019-11-05 22:13:37 +00:00
Erik Johnston	c16e192e2f	Fix caching devices for remote servers in worker. When the `/keys/query` API is hit on client_reader worker Synapse may decide that it needs to resync some remote deivces. Usually this happens on master, and then gets cached. However, that fails on workers and so it falls back to fetching devices from remotes directly, which may in turn fail if the remote is down.	2019-11-05 15:49:43 +00:00
Brendan Abolivier	cb2cbe4d26	Only filter if a filter was provided	2019-11-05 15:28:36 +00:00
Brendan Abolivier	f141af4c79	Update copyright	2019-11-05 14:52:38 +00:00
Brendan Abolivier	1dffa78701	Filter events_before and events_after in /context requests While the current version of the spec doesn't say much about how this endpoint uses filters (see https://github.com/matrix-org/matrix-doc/issues/2338), the current implementation is that some fields of an EventFilter apply (the ones that are used when running the SQL query) and others don't (the ones that are used by the filter itself) because we don't call event_filter.filter(...). This seems counter-intuitive and probably not what we want so this commit fixes it.	2019-11-05 12:21:59 +00:00
Andrew Morgan	c2203bea57	Re-add docstring, with caveats detailed	2019-11-04 18:17:11 +00:00
Andrew Morgan	0287d033ee	Transfer upgraded rooms on groups	2019-11-04 18:08:50 +00:00
Brendan Abolivier	09957ce0e4	Implement per-room message retention policies	2019-11-04 17:09:22 +00:00
Erik Johnston	6a0092d371	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-11-04 13:29:35 +00:00
Richard van der Hoff	c6516adbe0	Factor out an _AsyncEventContextImpl (#6298 ) The intention here is to make it clearer which fields we can expect to be populated when: notably, that the _event_type etc aren't used for the synchronous impl of EventContext.	2019-11-01 16:19:09 +00:00
Richard van der Hoff	1cb84c6486	Support for routing outbound HTTP requests via a proxy (#6239 ) The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will not be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.	2019-11-01 14:07:44 +00:00
Andrew Morgan	fe1f2b4520	Remove last usages of deprecated logging.warn method (#6314 )	2019-11-01 12:03:44 +00:00
Andrew Morgan	ace947e8da	Depublish a room from the public rooms list when it is upgraded (#6232 )	2019-11-01 10:28:09 +00:00
Hubert Chathi	6f4bc6d01d	Merge branch 'develop' into cross-signing_federation	2019-10-31 22:38:21 -04:00
Amber Brown	020add5099	Update black to 19.10b0 (#6304 ) * update version of black and also fix the mypy config being overridden	2019-11-01 02:43:24 +11:00
Erik Johnston	cd581338cf	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-10-31 15:19:26 +00:00
Erik Johnston	dfe0cd71b6	Merge pull request #6294 from matrix-org/erikj/add_state_storage Add StateGroupStorage interface	2019-10-31 16:17:53 +01:00
Andrew Morgan	54fef094b3	Remove usage of deprecated logger.warn method from codebase (#6271 ) Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.	2019-10-31 10:23:24 +00:00
Erik Johnston	7c8c97e635	Split purge API into events vs state	2019-10-30 15:23:37 +00:00
Erik Johnston	69f0054ce6	Port to use state storage	2019-10-30 14:46:54 +00:00
Hubert Chathi	7d7eac61be	Merge branch 'develop' into cross-signing_federation	2019-10-30 10:17:10 -04:00
Hubert Chathi	bc32f102cd	black	2019-10-30 10:07:36 -04:00
Hubert Chathi	d78b1e339d	apply changes as a result of PR review	2019-10-30 10:01:53 -04:00
Erik Johnston	ec6de1cc7d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store	2019-10-30 13:37:04 +00:00
Andrew Morgan	a2276d4d3c	Fix log line that was printing undefined value (#6278 )	2019-10-30 11:28:48 +00:00
Erik Johnston	2c35ffead2	Port receipt and read markers to async/wait	2019-10-29 15:08:22 +00:00
Richard van der Hoff	172f264ed3	Improve signature checking on some federation APIs (#6262 ) Make sure that we check that events sent over /send_join, /send_leave, and /invite, are correctly signed and come from the expected servers.	2019-10-28 12:43:23 +00:00
Erik Johnston	f666d6f5d7	Remove repeated calls to config.stats_enabled. Turns out that fetching variables from the config object is expensive, so doing it once at startup avoids unnecessary work.	2019-10-25 10:28:36 +01:00
Hubert Chathi	ff05c9b760	don't error if federation query doesn't have cross-signing keys	2019-10-24 21:46:11 -04:00
Erik Johnston	3ca4c7c516	Use new EventPersistenceStore	2019-10-23 16:15:03 +01:00
Hubert Chathi	480eac30eb	black	2019-10-22 22:37:16 -04:00
Hubert Chathi	404e8c8532	vendor-prefix the EDU name until MSC1756 is merged into the spec	2019-10-22 22:33:23 -04:00
Hubert Chathi	1fabf82d50	update to work with newer code, and fix formatting	2019-10-22 21:44:58 -04:00
Hubert Chathi	41ad35b523	add missing param	2019-10-22 19:06:29 -04:00
Hubert Chathi	cfdb84422d	make black happy	2019-10-22 19:06:06 -04:00
Hubert Chathi	8d3542a64e	implement federation parts of cross-signing	2019-10-22 19:04:35 -04:00
Hubert Chathi	2761731634	Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2 Cross-signing [3/4] -- uploading signatures edition	2019-10-22 17:05:06 +01:00
Richard van der Hoff	93eaeec75a	Remove Auth.check method (#6217 ) This method was somewhat redundant, and confusing.	2019-10-18 19:43:36 +02:00
Hubert Chathi	36adfaedab	Merge branch 'develop' into cross-signing_sig_upload	2019-10-18 18:34:42 +01:00
Hubert Chathi	125eb45e19	fix doc strings	2019-10-18 16:56:16 +01:00
Hubert Chathi	0a9d22808c	Merge branch 'uhoreg/e2e_cross-signing_merged' into develop	2019-10-18 15:57:40 +01:00
Richard van der Hoff	2b22faded7	Merge remote-tracking branch 'origin/develop' into rav/event_auth/1	2019-10-18 12:35:13 +01:00
Hubert Chathi	cfc28325a6	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-10-18 11:39:40 +01:00
Richard van der Hoff	80003dfcd5	Merge remote-tracking branch 'origin/develop' into rav/event_auth/1	2019-10-18 10:11:40 +01:00
Erik Johnston	5859a5c569	Fix presence timeouts when synchrotron restarts. (#6212 ) * Fix presence timeouts when synchrotron restarts. Handling timeouts would fail if there was an external process that had timed out, e.g. a synchrotron restarting. This was due to a couple of variable name typoes. Fixes #3715.	2019-10-18 06:42:26 +01:00
Richard van der Hoff	1ba359a11f	rip out some unreachable code The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.	2019-10-17 21:45:59 +01:00
Hubert Chathi	06fc66c81e	Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage make storage layer in charge of interpreting the device key data	2019-10-11 15:19:06 -04:00
Hubert Chathi	691dd67fcd	Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version make version optional in body of e2e backup version update	2019-10-11 10:11:59 -04:00
Hubert Chathi	7a0dce9259	make sure we actually return something	2019-10-10 20:31:30 -04:00
Hubert Chathi	4908fb3b30	make storage layer in charge of interpreting the device key data	2019-10-10 19:15:30 -04:00
werner291	b5b03b7079	Add domain validation when creating room with list of invitees (#6121 )	2019-10-10 13:05:48 +01:00
Richard van der Hoff	a139420a3c	Fix races in room stats (and other) updates. (#6187 ) Hopefully this will fix the occasional failures we were seeing in the room directory. The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted before current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8. We can solve this easily by only processing up to the stream_id where we know all events have been persisted.	2019-10-10 11:29:01 +01:00
Andrew Morgan	da815c1f69	Move tag/push rules room upgrade checking ealier (#6155 ) It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers. The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).	2019-10-10 10:06:45 +01:00
Hubert Chathi	4535a07f4a	make version optional in body of e2e backup version update to agree with latest version of the MSC	2019-10-09 17:54:03 -04:00
Andrew Morgan	ea7d938bca	Remove unused public room list timeout param (#6179 ) * Remove unused public room list timeout param * Add changelog	2019-10-08 13:51:25 +01:00
Erik Johnston	2cb7466993	Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure Don't regenerate numeric user ID if registration fails.	2019-10-07 16:56:23 +01:00
Richard van der Hoff	276ae5c63e	add some logging to the rooms stats updates, to try to track down a flaky test (#6167 )	2019-10-07 14:41:39 +01:00
Brendan Abolivier	ae0b78cb1f	Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked Don't 500 when trying to exchange a revoked 3PID invite	2019-10-04 12:09:05 +01:00
Brendan Abolivier	21d51ab598	Typo	2019-10-04 11:21:24 +01:00
Brendan Abolivier	4676732ca0	Lint	2019-10-04 11:18:28 +01:00
Brendan Abolivier	81d51ce48b	Incorporate review	2019-10-04 11:16:19 +01:00
Erik Johnston	ab8a64772b	Remove unused variable	2019-10-03 17:42:32 +01:00
Erik Johnston	693156aaf4	Don't regenerate numeric user ID if registration fails. This causes huge amounts of DB IO if registrations start to fail e.g. because the DB is struggling with IO.	2019-10-03 17:33:54 +01:00
Brendan Abolivier	6527fa18c1	Add test case	2019-10-03 11:24:36 +01:00
Erik Johnston	03cf4385e0	Fix public room list pagination. We incorrectly used `room_id` as to bound the result set, even though we order by `joined_members, room_id`, leading to incorrect results after pagination.	2019-10-02 15:11:17 +01:00
Erik Johnston	a5166e4d5f	Land improved room list based on room stats (#6019 ) Use room_stats and room_state for room directory search	2019-10-02 14:08:35 +01:00
Brendan Abolivier	972c9f65d7	Lint	2019-10-02 12:17:46 +01:00
Andrew Morgan	2a1470cd05	Fix yields and copy instead of move push rules on room upgrade (#6144 ) Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room. For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first. Also fix some missing yields that probably broke things quite a bit.	2019-10-02 12:04:22 +01:00
Brendan Abolivier	5705ecaec6	Don't 500 code when trying to exchange a revoked 3PID invite While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state. When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events. When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it). This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.	2019-10-02 11:19:43 +01:00
Brendan Abolivier	3e42d47a5a	Incorporate review	2019-09-27 16:15:01 +01:00
Brendan Abolivier	af92110c46	Update synapse/handlers/deactivate_account.py Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-27 16:12:15 +01:00
Brendan Abolivier	bbe2a0f339	Update synapse/handlers/deactivate_account.py Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-27 16:10:36 +01:00
Brendan Abolivier	873fe7883c	Lint	2019-09-27 15:21:03 +01:00
Brendan Abolivier	72a2708ac6	Fixup and add some logging	2019-09-27 15:13:39 +01:00
Brendan Abolivier	5257a2fb1c	Reject pending invites on deactivation	2019-09-27 14:49:53 +01:00
Andrew Morgan	8c27bc8b60	Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978 ) Just to have all the methods that make calls to identity services in one place.	2019-09-27 10:36:20 +01:00
Neil Johnson	034db2ba21	Fix dummy event insertion consent bug (#6053 ) Fixes #5905	2019-09-26 11:47:53 +01:00
Neil Johnson	77dc7093a7	Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. (#6104 ) Fixes #6103	2019-09-25 12:29:35 +01:00
Richard van der Hoff	990928abde	Stop advertising unsupported flows for registration (#6107 ) If email or msisdn verification aren't supported, let's stop advertising them for registration. Fixes #6100.	2019-09-25 12:10:26 +01:00
Richard van der Hoff	2cd98812ba	Refactor the user-interactive auth handling (#6105 ) Pull the checkers out to their own classes, rather than having them lost in a massive 1000-line class which does everything. This is also preparation for some more intelligent advertising of flows, as per #6100	2019-09-25 11:33:03 +01:00
Hubert Chathi	c3635c9459	make isort happy	2019-09-24 16:21:03 -04:00
Hubert Chathi	f4b6d43ec3	add some comments	2019-09-24 16:19:54 -04:00
Hubert Chathi	39864f45ec	drop some logger lines to debug	2019-09-24 15:26:45 -04:00
Hubert Chathi	26113fb7de	make changes based on PR feedback	2019-09-24 14:12:20 -04:00
Richard van der Hoff	9b7c4f4078	Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work	2019-09-24 17:03:50 +01:00
J. Ryan Stinnett	40fb00f5b7	Add sid to next_link for email validation (#6097 )	2019-09-24 14:39:50 +01:00
Richard van der Hoff	ed8b92f0d2	Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work	2019-09-24 12:57:32 +01:00
Andrew Morgan	50776261e1	Add submit_url response parameter to msisdn /requestToken (#6079 ) Second part of solving #6076 Fixes #6076 We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.	2019-09-23 21:21:03 +01:00
Andrew Morgan	e08ea43463	Use the federation blacklist for requests to untrusted Identity Servers (#6000 ) Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses. Fixes #5935	2019-09-23 20:23:20 +01:00
Andrew Morgan	2c99c63453	Add POST submit_token endpoint for MSISDN (#6078 ) First part of solving #6076	2019-09-23 17:49:00 +01:00
Andrew Morgan	30af161af2	Implement MSC2290 (#6043 ) Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away. This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints. Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.	2019-09-23 16:50:27 +01:00
Andrew Morgan	885a4726b7	Return timeout error to user for identity server calls (#6073 )	2019-09-23 14:37:23 +01:00
Andrew Morgan	df3401a71d	Allow HS to send emails when adding an email to the HS (#6042 )	2019-09-20 15:21:30 +01:00
Richard van der Hoff	366dc7d2d4	Merge commit '33757bad1' into rav/saml_mapping_work	2019-09-20 11:21:40 +01:00
Richard van der Hoff	33757bad19	More better logging	2019-09-20 11:20:02 +01:00
Richard van der Hoff	b65327ff66	Merge branch 'develop' into rav/saml_mapping_work	2019-09-19 18:13:31 +01:00
Richard van der Hoff	7423fade92	better logging	2019-09-19 17:16:50 +01:00
Andrew Morgan	6670bd4072	v2 3PID Invites (part of MSC2140) (#5979 ) 3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is. These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens). Requires: ~~#5976~~	2019-09-17 18:05:13 +01:00
Erik Johnston	70c52821ce	Fix race condition in room stats. (#6029 ) Broke in #5971 Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)	2019-09-17 12:41:23 +01:00
Richard van der Hoff	a8ac40445c	Record mappings from saml users in an external table We want to assign unique mxids to saml users based on an incrementing suffix. For that to work, we need to record the allocated mxid in a separate table.	2019-09-13 16:01:46 +01:00
Andrew Morgan	9fc71dc5ee	Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976 ) This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted. Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140). Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.	2019-09-11 16:02:42 +01:00
Erik Johnston	cbcbfe64a2	Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction Allow use of different ratelimits for admin redactions.	2019-09-11 15:39:38 +01:00
Andrew Morgan	9c555f37e3	Add note about extra arg to send_membership_event, remove arg in remote_reject_invite (#6009 ) Some small fixes to `room_member.py` found while doing other PRs. 1. Add requester to the base `_remote_reject_invite` method. 2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.	2019-09-11 14:23:24 +01:00
Erik Johnston	57dd41a45b	Fix comments Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-11 13:54:50 +01:00
Andrew Morgan	3505ffcda7	Fix existing v2 identity server calls (MSC2140) (#6013 ) Two things I missed while implementing [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R80). 1. Access tokens should be provided to the identity server as `access_token`, not `id_access_token`, even though the homeserver may accept the tokens as `id_access_token`. 2. Access tokens must be sent to the identity server in a query parameter, the JSON body is not allowed. We now send the access token as part of an `Authorization: ...` header, which fixes both things. The breaking code was added in https://github.com/matrix-org/synapse/pull/5892 Sytest PR: https://github.com/matrix-org/sytest/pull/697	2019-09-11 11:59:45 +01:00
Erik Johnston	c64c3bb4c5	Fix how we check for self redaction	2019-09-11 11:18:04 +01:00
Erik Johnston	54ce81c86d	Allow use of different ratelimits for admin redactions. This is useful to allow room admins to quickly deal with a large number of abusive messages.	2019-09-11 10:46:38 +01:00
Andrew Morgan	cd17a2085e	Remove origin parameter from add_display_name_to_third_party_invite and add params to docstring (#6010 ) Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument. Since it's not used anywhere in the handler function though, we should remove it from the function arguments.	2019-09-11 10:37:17 +01:00
Erik Johnston	5e9b05d7da	Merge pull request #6011 from matrix-org/anoa/fix_3pid_validation Use account_threepid_delegate for 3pid validation	2019-09-10 18:15:07 +01:00
Andrew Morgan	b5833a2abf	Add changelog	2019-09-10 17:56:10 +01:00
Andrew Morgan	60d3c57bd0	Use account_threepid_delegate for 3pid validation	2019-09-10 17:56:10 +01:00
Jason Robinson	be618e0551	Only count real users when checking for auto-creation of auto-join room Previously if the first registered user was a "support" or "bot" user, when the first real user registers, the auto-join rooms were not created. Fix to exclude non-real (ie users with a special user type) users when counting how many users there are to determine whether we should auto-create a room. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 14:48:08 +03:00
Hubert Chathi	d3f2fbcfe5	add function docs	2019-09-07 14:13:18 -04:00
Hubert Chathi	8e86f5b65c	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-09-07 13:20:34 -04:00
Hubert Chathi	0d61d1d735	Merge branch 'develop' into cross-signing_sig_upload	2019-09-07 13:14:45 -04:00
Hubert Chathi	415d0a00e0	run black	2019-09-06 16:46:45 -04:00
Hubert Chathi	561cbba057	split out signature processing into separate functions	2019-09-06 16:44:24 -04:00
Andrew Morgan	78801e7f9e	Ensure a sid parameter is passed to bind_threepid (#5995 ) `sid` is required to be part of `three_pid_creds`. We were 500'ing if it wasn't provided instead of returning `M_MISSING_PARAM`.	2019-09-06 15:36:50 +01:00
Erik Johnston	142c9325c2	Correctly handle non-bool m.federate flag	2019-09-06 14:21:06 +01:00
Andrew Morgan	0c0b82b6d1	Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987 ) This is a combination of a few different PRs, finally all being merged into `develop`: * #5875 * #5876 * #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](`891afb57cb (diff-e591d42d30690ffb79f63bb726200891)`) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future) * #5835 * #5969 * #5940 Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged. UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.	2019-09-06 11:35:28 +01:00
Hubert Chathi	369462da74	avoid modifying input parameter	2019-09-05 17:03:31 -04:00
Erik Johnston	1a6ae33309	Merge pull request #5984 from matrix-org/joriks/opentracing_link_send_to_edu_contexts Link the send loop with the edus contexts	2019-09-05 15:22:24 +01:00
Jorik Schellekens	1d65292e94	Link the send loop with the edus contexts The contexts were being filtered too early so the send loop wasn't being linked to them unless the destination was whitelisted.	2019-09-05 14:42:37 +01:00
Andrew Morgan	a0d294c306	Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (#5892 )	2019-09-05 14:31:22 +01:00
Andrew Morgan	90d17a3d28	Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (#5980 ) Implements `POST /_matrix/client/r0/account/3pid/unbind` from [MSC2140](https://github.com/matrix-org/matrix-doc/blob/dbkr/tos_2/proposals/2140-terms-of-service-2.md#post-_matrixclientr0account3pidunbind).	2019-09-05 14:00:30 +01:00
Hubert Chathi	c8dc740a94	update with newer coding style	2019-09-04 22:30:45 -04:00
Hubert Chathi	7d6c70fc7a	make black happy	2019-09-04 22:04:12 -04:00
Hubert Chathi	ac4746ac4b	allow uploading signatures of master key signed by devices	2019-09-04 22:03:41 -04:00
Hubert Chathi	4bb4544784	implement device signature uploading/fetching	2019-09-04 20:02:56 -04:00
Hubert Chathi	faf72a4c40	Merge branch 'develop' into cross-signing_keys	2019-09-04 19:12:29 -04:00
Andrew Morgan	b736c6cd3a	Remove bind_email and bind_msisdn (#5964 ) Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107).	2019-09-04 18:24:23 +01:00
Erik Johnston	6e834e94fc	Fix and refactor room and user stats (#5971 ) Previously the stats were not being correctly populated.	2019-09-04 13:04:27 +01:00
Andrew Morgan	ea128a3e8e	code cleanups	2019-09-03 21:05:06 +01:00
Andrew Morgan	6b6086b8bf	Fix docstring	2019-09-03 20:00:09 +01:00
Andrew Morgan	a98b8583c6	Remove unnecessary variable declaration	2019-09-03 19:58:51 +01:00
Matthew Hodgson	8401bcd206	fix typo	2019-09-03 12:44:14 +01:00
Andrew Morgan	2a44782666	Remove double return statements (#5962 ) Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.	2019-09-03 11:42:45 +01:00
Jorik Schellekens	a90d16dabc	Opentrace device lists (#5853 ) Trace device list changes.	2019-09-03 10:21:30 +01:00
Andrew Morgan	36f34e6f3d	Remove unused methods from c/s api v1 in register.py (#5963 ) These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.	2019-09-02 18:29:21 +01:00
Andrew Morgan	4548d1f87e	Remove unnecessary parentheses around return statements (#5931 ) Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)	2019-08-30 16:28:26 +01:00
Andrew Morgan	3057095a5d	Revert "Use the v2 lookup API for 3PID invites (#5897 )" (#5937 ) This reverts commit `71fc04069a`. This broke 3PID invites as #5892 was required for it to work correctly.	2019-08-30 12:00:20 +01:00
Hubert Chathi	e3d3fbf63f	Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys	2019-08-28 17:36:46 -07:00
Hubert Chathi	96bda56370	black	2019-08-28 17:18:40 -07:00
Hubert Chathi	3b0b22cb05	use stream ID generator instead of timestamp	2019-08-28 17:17:21 -07:00
Will Hunt	c8fa620d7a	Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent Exempt support users from consent	2019-08-28 16:31:40 +01:00
Andrew Morgan	71fc04069a	Use the v2 lookup API for 3PID invites (#5897 ) Fixes https://github.com/matrix-org/synapse/issues/5861 Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134). Currently this is only used for 3PID invites. Sytest PR: https://github.com/matrix-org/sytest/pull/679	2019-08-28 14:59:26 +02:00
Amber Brown	7dc398586c	Implement a structured logging output system. (#5680 )	2019-08-28 21:18:53 +10:00
reivilibre	a3f0635686	Merge pull request #5914 from matrix-org/rei/admin_getadmin Add GET method to admin API /users/@user:dom/admin	2019-08-28 09:44:22 +01:00
reivilibre	7ccc251415	Merge pull request #5859 from matrix-org/rei/msc2197 MSC2197 Search Filters over Federation	2019-08-28 09:00:21 +01:00
Olivier Wilkinson (reivilibre)	c88a119259	Add GET method to admin API /users/@user:dom/admin Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-27 13:12:27 +01:00
Richard van der Hoff	ccb15a5bbe	Merge pull request #5906 from matrix-org/neilj/increase_display_name_limit Increase profile display name limit	2019-08-27 11:52:59 +01:00
reivilibre	1a7e6eb633	Add Admin API capability to set adminship of a user (#5878 ) Admin API: Set adminship of a user	2019-08-27 10:14:00 +01:00
Neil Johnson	27d3fc421a	Increase max display name limit	2019-08-24 22:33:43 +01:00
Will Hunt	c998f25006	Apply suggestions from code review Co-Authored-By: Erik Johnston <erik@matrix.org>	2019-08-23 10:28:54 +01:00
Half-Shot	9ba32f6573	Exempt bot users	2019-08-23 09:56:31 +01:00
Half-Shot	886eceba3e	Return user_type in get_user_by_id	2019-08-23 09:14:52 +01:00
Jorik Schellekens	8767b63a82	Propagate opentracing contexts through EDUs (#5852 ) Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-08-22 18:21:10 +01:00
Brendan Abolivier	1c5b8c6222	Revert "Add "require_consent" parameter for registration" This reverts commit `3320aaab3a`.	2019-08-22 14:47:34 +01:00
Half-Shot	3320aaab3a	Add "require_consent" parameter for registration	2019-08-22 14:21:54 +01:00
Jorik Schellekens	9a6f2be572	Opentrace e2e keys (#5855 ) Add opentracing tags and logs for e2e keys	2019-08-22 11:28:12 +01:00
Richard van der Hoff	119aa31b10	Servlet to purge old rooms (#5845 )	2019-08-22 10:42:59 +01:00
Hubert Chathi	814f253f1b	make isort happy	2019-08-21 13:22:15 -07:00
Hubert Chathi	7c3abc6572	apply PR review suggestions	2019-08-21 13:19:35 -07:00
Erik Johnston	d514dac0b2	Merge pull request #5860 from matrix-org/erikj/update_5704_comments Remove logging for #5407 and update comments	2019-08-19 10:20:59 +01:00
Erik Johnston	748aa38378	Remove logging for #5407 and update comments	2019-08-15 12:02:18 +01:00
Olivier Wilkinson (reivilibre)	6fadb560fc	Support MSC2197 outbound with unstable prefix Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-15 10:59:37 +01:00
Michael Telatynski	baee288fb4	Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633 )	2019-08-15 09:45:57 +01:00
Erik Johnston	6881f21f3e	Handle TimelineBatch being limited and empty. This hopefully addresses #5407 by gracefully handling an empty but limited TimelineBatch. We also add some logging to figure out how this is happening.	2019-08-06 12:59:00 +01:00
Hubert Chathi	fac1cdc562	make changes from PR review	2019-08-01 21:51:19 -04:00
Hubert Chathi	336c546d6a	Merge branch 'cross-signing_hidden' into cross-signing_keys	2019-08-01 16:31:40 -04:00
Brendan Abolivier	8ed9e63432	Account validity: allow defining HTML templates to serve the us… (#5807 ) Account validity: allow defining HTML templates to serve the user on account renewal attempt	2019-08-01 16:09:25 +02:00
Erik Johnston	58af30a6c7	Merge pull request #5802 from matrix-org/erikj/deny_redacting_different_room Deny redaction of events in a different room.	2019-08-01 13:14:46 +01:00
Erik Johnston	0f632f3a57	Merge pull request #5790 from matrix-org/erikj/groups_request_errors Handle RequestSendFailed exception correctly in more places.	2019-08-01 13:14:08 +01:00
Brendan Abolivier	a4a9ded4d0	Allow defining HTML templates to serve the user on account renewal	2019-08-01 11:59:27 +02:00
Erik Johnston	cf89266b98	Deny redaction of events in a different room. We already correctly filter out such redactions, but we should also deny them over the CS API.	2019-07-31 16:12:27 +01:00
Andrew Morgan	58a755cdc3	Remove duplicate return statement	2019-07-31 13:24:51 +01:00
Erik Johnston	b4d5ff0af7	Don't log as exception when failing durig backfill	2019-07-30 13:19:22 +01:00
Erik Johnston	1ec7d656dd	Unwrap error	2019-07-30 13:09:02 +01:00
Erik Johnston	458e51df7a	Fix error handling when fetching remote device keys	2019-07-30 13:07:02 +01:00
Richard van der Hoff	8c97f6414c	Remove non-functional 'expire_access_token' setting (#5782 ) The `expire_access_token` didn't do what it sounded like it should do. What it actually did was make Synapse enforce the 'time' caveat on macaroons used as access tokens, but since our access token macaroons never contained such a caveat, it was always a no-op. (The code to add 'time' caveats was removed back in v0.18.5, in #1656)	2019-07-30 08:25:02 +01:00
Amber Brown	865077f1d1	Room Complexity Client Implementation (#5783 )	2019-07-30 02:47:27 +10:00
Erik Johnston	aecae8f397	Correctly handle errors doing requests to group servers	2019-07-29 17:21:57 +01:00
Amber Brown	97a8b4caf7	Move some timeout checking logs to DEBUG #5785	2019-07-30 02:02:18 +10:00
Jorik Schellekens	85b0bd8fe0	Update the device list cache when keys/query is called (#5693 )	2019-07-29 16:34:44 +01:00
Richard van der Hoff	1a93daf353	Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch Log when we receive a /make_* request from a different origin	2019-07-26 12:38:37 +01:00
Richard van der Hoff	d1020653fc	Log when we receive a /make_* request from a different origin	2019-07-26 10:08:22 +01:00
Hubert Chathi	c659b9f94f	allow uploading keys for cross-signing	2019-07-25 11:08:24 -04:00
Erik Johnston	2276936bac	Merge pull request #5743 from matrix-org/erikj/log_origin_receipts_mismatch Log when we receive receipt from a different origin	2019-07-24 13:27:57 +01:00
Richard van der Hoff	f30a71a67b	Stop trying to fetch events with event_id=None. (#5753 ) `None` is not a valid event id, so queuing up a database fetch for it seems like a silly thing to do. I considered making `get_event` return `None` if `event_id is None`, but then its interaction with `allow_none` seemed uninituitive, and strong typing ftw.	2019-07-24 13:16:18 +01:00
Amber Brown	4806651744	Replace returnValue with return (#5736 )	2019-07-23 23:00:55 +10:00
Erik Johnston	3db1377b26	Log when we receive receipt from a different origin	2019-07-23 13:31:03 +01:00
Richard van der Hoff	9c70a02a9c	Ignore redactions of m.room.create events (#5701 )	2019-07-17 19:08:02 +01:00
Andrew Morgan	18c516698e	Return a different error from Invalid Password when a user is deactivated (#5674 ) Return `This account has been deactivated` instead of `Invalid password` when a user is deactivated.	2019-07-15 11:45:29 +01:00
Erik Johnston	d86321300a	Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data Add basic function to get all data for a user out of synapse	2019-07-15 10:04:02 +01:00
Richard van der Hoff	5f158ec039	Implement access token expiry (#5660 ) Record how long an access token is valid for, and raise a soft-logout once it expires.	2019-07-12 17:26:02 +01:00
Richard van der Hoff	1890cfcf82	Inline issue_access_token (#5659 ) this is only used in one place, so it's clearer if we inline it and reduce the API surface. Also, fixes a buglet where we would create an access token even if we were about to block the user (we would never return the AT, so the user could never use it, but it was still created and added to the db.)	2019-07-11 04:10:07 +10:00
Brendan Abolivier	8ab3444fdf	Merge pull request #5658 from matrix-org/babolivier/is-json Send 3PID bind requests as JSON data	2019-07-10 17:01:26 +01:00
Richard van der Hoff	953dbb7980	Remove access-token support from RegistrationStore.register (#5642 ) The 'token' param is no longer used anywhere except the tests, so let's kill that off too.	2019-07-10 16:26:49 +01:00
Brendan Abolivier	b2a2e96ea6	Typo	2019-07-10 15:56:21 +01:00
Brendan Abolivier	f77e997619	Send 3PID bind requests as JSON data	2019-07-10 15:46:42 +01:00
Brendan Abolivier	65434da75d	Merge pull request #5638 from matrix-org/babolivier/invite-json Use JSON when querying the IS's /store-invite endpoint	2019-07-09 18:48:38 +01:00
Hubert Chathi	7b3bc755a3	remove unused and unnecessary check for FederationDeniedError (#5645 ) FederationDeniedError is a subclass of SynapseError, which is a subclass of CodeMessageException, so if e is a FederationDeniedError, then this check for FederationDeniedError will never be reached since it will be caught by the check for CodeMessageException above. The check for CodeMessageException does almost the same thing as this check (since FederationDeniedError initialises with code=403 and msg="Federation denied with %s."), so may as well just keep allowing it to handle this case.	2019-07-09 18:37:39 +01:00
Brendan Abolivier	af67c7c1de	Merge pull request #5644 from matrix-org/babolivier/profile-allow-self Allow newly-registered users to lookup their own profiles	2019-07-09 10:25:40 +01:00
Richard van der Hoff	824707383b	Remove access-token support from RegistrationHandler.register (#5641 ) Nothing uses this now, so we can remove the dead code, and clean up the API. Since we're changing the shape of the return value anyway, we take the opportunity to give the method a better name.	2019-07-08 19:01:08 +01:00
Brendan Abolivier	7556851665	Allow newly-registered users to lookup their own profiles When a user creates an account and the 'require_auth_for_profile_requests' config flag is set, and a client that performed the registration wants to lookup the newly-created profile, the request will be denied because the user doesn't share a room with themselves yet.	2019-07-08 17:31:00 +01:00
Richard van der Hoff	b70e080b59	Better logging for auto-join. (#5643 ) It was pretty unclear what was going on, so I've added a couple of log lines.	2019-07-08 17:14:51 +01:00
Brendan Abolivier	57eacee4f4	Merge branch 'develop' into babolivier/invite-json	2019-07-08 15:49:23 +01:00
Richard van der Hoff	1af2fcd492	Move get_or_create_user to test code (#5628 ) This is only used in tests, so...	2019-07-08 23:52:26 +10:00
Brendan Abolivier	f05c7d62bc	Lint	2019-07-08 14:29:27 +01:00
Brendan Abolivier	1a807dfe68	Use application/json when querying the IS's /store-invite endpoint	2019-07-08 14:19:39 +01:00
Richard van der Hoff	80cc82a445	Remove support for invite_3pid_guest. (#5625 ) This has never been documented, and I'm not sure it's ever been used outside sytest. It's quite a lot of poorly-maintained code, so I'd like to get rid of it. For now I haven't removed the database table; I suggest we leave that for a future clearout.	2019-07-05 16:47:58 +01:00
Erik Johnston	b4f5416dd9	pep8	2019-07-05 14:41:29 +01:00
Erik Johnston	eadb13d2e9	Remove FileExfiltrationWriter	2019-07-05 14:15:00 +01:00
Erik Johnston	7f0d8e4288	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data	2019-07-05 14:08:21 +01:00
Erik Johnston	9ccea16d45	Assume key existence. Update docstrings	2019-07-05 14:07:56 +01:00
Erik Johnston	c061d4f237	Fixup from review comments.	2019-07-04 11:41:06 +01:00
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606 )	2019-07-04 00:07:04 +10:00
Erik Johnston	d0b849c86d	Apply comment fixups from code review Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-07-03 15:03:38 +01:00
Richard van der Hoff	91753cae59	Fix a number of "Starting txn from sentinel context" warnings (#5605 ) Fixes #5602, #5603	2019-07-03 09:31:27 +01:00
Erik Johnston	8ee69f299c	Add basic function to get all data for a user out of synapse	2019-07-02 12:09:04 +01:00
Richard van der Hoff	6eecb6e500	Complete the SAML2 implementation (#5422 ) * SAML2 Improvements and redirect stuff Signed-off-by: Alexander Trost <galexrt@googlemail.com> * Code cleanups and simplifications. Also: share the saml client between redirect and response handlers. * changelog * Revert redundant changes to static js * Move all the saml stuff out to a centralised handler * Add support for tracking SAML2 sessions. This allows us to correctly handle `allow_unsolicited: False`. * update sample config * cleanups * update sample config * rename BaseSSORedirectServlet for consistency * Address review comments	2019-07-02 11:18:11 +01:00
Richard van der Hoff	b4fd86a9b4	Merge branch 'develop' into rav/saml2_client	2019-07-01 14:21:03 +01:00
Richard van der Hoff	3bcb13edd0	Address review comments	2019-07-01 12:13:22 +01:00
Erik Johnston	04196a4dae	Merge pull request #5507 from matrix-org/erikj/presence_sync_tighloop Fix sync tightloop bug.	2019-07-01 11:43:10 +01:00
Erik Johnston	915280f1ed	Fixup comment	2019-07-01 10:22:42 +01:00
Brendan Abolivier	c7ff297dde	Merge pull request #5576 from matrix-org/babolivier/3pid-invite-ratelimit Don't update the ratelimiter before sending a 3PID invite	2019-06-28 17:43:48 +01:00
Brendan Abolivier	15d9fc31bd	Only ratelimit when sending the email If we do the opposite, an event can arrive after or while sending the email and the 3PID invite event will get ratelimited.	2019-06-28 16:04:05 +01:00
Brendan Abolivier	01d0f8e701	Don't update the ratelimiter before sending a 3PID invite This would cause emails being sent, but Synapse responding with a 429 when creating the event. The client would then retry, and with bad timing the same scenario would happen again. Some testing I did ended up sending me 10 emails for one single invite because of this.	2019-06-28 15:22:16 +01:00
Daniel Hoffend	9646a593ac	Added possibilty to disable local password authentication (#5092 ) Signed-off-by: Daniel Hoffend <dh@dotlan.net>	2019-06-27 18:37:29 +01:00
Erik Johnston	729f5a4fb6	Review comments	2019-06-27 16:06:23 +01:00
Richard van der Hoff	28db0ae537	cleanups	2019-06-27 00:37:41 +01:00
Richard van der Hoff	36f4953dec	Add support for tracking SAML2 sessions. This allows us to correctly handle `allow_unsolicited: False`.	2019-06-26 23:50:55 +01:00
Richard van der Hoff	3705322103	Move all the saml stuff out to a centralised handler	2019-06-26 22:52:02 +01:00
Erik Johnston	8624db3194	Refactor and comment sync device list code	2019-06-26 19:39:49 +01:00
Erik Johnston	806a06daf2	Rename get_users_whose_devices_changed	2019-06-26 19:39:19 +01:00
Erik Johnston	a2f6d31a63	Refactor get_user_ids_changed to pull less from DB When a client asks for users whose devices have changed since a token we used to pull all users from the database since the token, which could easily be thousands of rows for old tokens. This PR changes this to only check for changes for users the client is actually interested in. Fixes #5553	2019-06-26 12:03:44 +01:00
Andrew Morgan	ef8c62758c	Prevent multiple upgrades on the same room at once (#5051 ) Closes #4583 Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time. Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.	2019-06-25 14:19:21 +01:00
Erik Johnston	25433f212d	Merge pull request #5531 from matrix-org/erikj/workers_pagination_token Fix /messages on workers when no from param specified.	2019-06-24 15:30:10 +01:00
Richard van der Hoff	edea4bb5be	Allow configuration of the path used for ACME account keys. Because sticking it in the same place as the config isn't necessarily the right thing to do.	2019-06-24 13:51:22 +01:00
Richard van der Hoff	21bf4318b5	Factor acme bits out to a separate file (#5521 ) This makes some of the conditional-import hoop-jumping easier.	2019-06-24 11:33:56 +01:00
Erik Johnston	dddf20e8e1	Fix /messages on workers when no from param specified. If no `from` param is specified we calculate and use the "current token" that inlcuded typing, presence, etc. These are unused during pagination and are not available on workers, so we simply don't calculate them.	2019-06-24 10:06:51 +01:00
Erik Johnston	8181e290a9	Fix sync tightloop bug. If, for some reason, presence updates take a while to persist then it can trigger clients to tightloop calling `/sync` due to the presence handler returning updates but not advancing the stream token. Fixes #5503.	2019-06-21 11:10:27 +01:00
Amber Brown	32e7c9e7f2	Run Black. (#5482 )	2019-06-20 19:32:02 +10:00
Erik Johnston	7dcf984075	Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_error Handle the case of `get_missing_events` failing	2019-06-19 13:20:09 +01:00
Erik Johnston	e0be8d7016	Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events Add experimental option to reduce extremities.	2019-06-19 13:19:18 +01:00
Erik Johnston	554609288b	Run as background process and fix comments	2019-06-19 11:33:03 +01:00
David Baker	f2d2ae03da	Add some logging to 3pid invite sig verification (#5015 ) I had to add quite a lot of logging to diagnose a problem with 3pid invites - we only logged the one failure which isn't all that informative. NB. I'm not convinced the logic of this loop is right: I think it should just accept a single valid signature from a trusted source rather than fail if any signature is invalid. Also it should probably not skip the rest of middle loop if a check fails? However, I'm deliberately not changing the logic here.	2019-06-18 22:51:24 +01:00
Erik Johnston	2b20d0fb59	Fix logline	2019-06-18 16:12:53 +01:00
Erik Johnston	19b80fe68a	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_get_missing_events_error	2019-06-18 16:11:43 +01:00
Erik Johnston	b42f90470f	Add experimental option to reduce extremities. Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.	2019-06-18 15:02:18 +01:00
Brendan Abolivier	112cf5a73a	Add third party rules hook for 3PID invites	2019-06-17 17:39:09 +01:00
Erik Johnston	187d2837a9	Add third party rules hook into create room	2019-06-17 16:41:19 +01:00
Erik Johnston	6840ebeef8	Merge pull request #5385 from matrix-org/erikj/reduce_http_exceptions Handle HttpResponseException when using federation client.	2019-06-17 13:54:47 +01:00
Erik Johnston	a9dab970b8	Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_state Fix 3PID invite room state over federation.	2019-06-17 10:18:28 +01:00
Brendan Abolivier	f12e1f029c	Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules Allow server admins to define implementations of extra rules for allowing or denying incoming events	2019-06-14 19:37:59 +01:00
Brendan Abolivier	f874b16b2e	Add plugin APIs for implementations of custom event rules.	2019-06-14 18:16:03 +01:00
Erik Johnston	304a1376c2	Fix 3PID invite room state over federation. Fixes that when a user exchanges a 3PID invite for a proper invite over federation it does not include the `invite_room_state` key. This was due to synapse incorrectly sending out two invite requests.	2019-06-14 16:19:11 +01:00
Brendan Abolivier	6d56a694f4	Don't send renewal emails to deactivated users	2019-06-14 15:05:56 +01:00
Brendan Abolivier	d0530382ee	Track deactivated accounts in the database (#5378 )	2019-06-14 13:18:24 +01:00
Erik Johnston	49e01e5710	Fix defaults on checking threepids	2019-06-10 23:09:31 +01:00
Erik Johnston	a46ef1e3a4	Handle HttpResponseException when using federation client. Otherwise we just log exceptions everywhere.	2019-06-07 10:29:35 +01:00
Andrew Morgan	3719680ee4	Add ability to perform password reset via email without trusting the identity server (#5377 ) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368	2019-06-06 17:34:07 +01:00
Richard van der Hoff	71063a69b8	Fix missing logcontext for PresenceHandler.on_shutdown. (#5369 ) Fixes some warnings, and a scary-looking stacktrace when sytest kills the process.	2019-06-06 14:45:17 +01:00
Brendan Abolivier	8f06344e11	Merge pull request #5089 from dnaf/m-heroes-empty-room-name Make /sync return heroes if room name or canonical alias are empty	2019-06-06 11:18:13 +01:00
Brendan Abolivier	64fa928792	Simplify condition	2019-06-06 10:34:12 +01:00
Brendan Abolivier	b9c43c8463	Merge pull request #5355 from matrix-org/babolivier/heroes_left_members Include left members in room summaries' heroes	2019-06-06 10:05:27 +01:00
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352 ) * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	2019-06-06 00:45:46 +10:00
Brendan Abolivier	0a2f522644	Simplify condition	2019-06-05 14:02:29 +01:00
Brendan Abolivier	40596aec0e	Merge branch 'develop' into m-heroes-empty-room-name	2019-06-05 13:38:01 +01:00
Brendan Abolivier	dbbaf25dd3	Do user_id != me checks before deciding whether we should pick heroes from the joined members or the parted ones	2019-06-05 11:50:27 +01:00
Erik Johnston	bc3d6b918b	Add logging when request fails and clarify we ignore errors.	2019-06-05 11:37:51 +01:00
Erik Johnston	220a733d73	Fix handling of failures when calling /event_auth. When processing an incoming event over federation, we may try and resolve any unexpected differences in auth events. This is a non-essential process and so should not stop the processing of the event if it fails (e.g. due to the remote disappearing or not implementing the necessary endpoints). Fixes #3330	2019-06-03 09:56:45 +01:00
Richard van der Hoff	d828d1dc57	Merge pull request #5309 from matrix-org/rav/limit_displayname_length Limit displaynames and avatar URLs	2019-06-01 11:34:50 +01:00
Richard van der Hoff	d16c6375fe	Limit displaynames and avatar URLs These end up in join events everywhere, so let's limit them. Fixes #5079	2019-06-01 10:44:36 +01:00
Richard van der Hoff	5726378ece	Fix "db txn 'update_presence' from sentinel context" log messages (#5275 ) Fixes #4414.	2019-05-28 21:20:11 +01:00
Erik Johnston	d16f5574b6	Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events Don't bundle aggregations with events in /sync or /events or state queries	2019-05-24 10:36:31 +01:00
Erik Johnston	4cb577c23f	Don't bundle aggs for /state and /members etc APIs	2019-05-24 09:52:33 +01:00
Andrew Morgan	6368150a74	Add config option for setting homeserver's default room version (#5223 ) Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present. That hardcoded value is now located in the server.py config file.	2019-05-23 15:00:20 +01:00
Richard van der Hoff	85d1e03b9d	Simplifications and comments in do_auth (#5227 ) I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.	2019-05-23 11:17:42 +01:00
Amber Brown	4a30e4acb4	Room Statistics (#4338 )	2019-05-21 11:36:50 -05:00
Erik Johnston	9526aa96a6	Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions Block attempts to annotate the same event twice	2019-05-21 17:08:14 +01:00
Erik Johnston	44b8ba484e	Fix words	2019-05-21 16:51:45 +01:00
Richard van der Hoff	c4aef549ad	Exclude soft-failed events from fwd-extremity candidates. (#5146 ) When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.	2019-05-21 16:10:54 +01:00
Erik Johnston	de7672b78f	Don't bundle events in /sync or /events As we'll send down the annotations too anyway, so this just ends up confusing clients.	2019-05-21 13:54:09 +01:00
Erik Johnston	c7ec06e8a6	Block attempts to annotate the same event twice	2019-05-20 17:39:05 +01:00
ReidAnderson	3787133c9e	Limit UserIds to a length that fits in a state key (#5198 )	2019-05-20 11:20:08 +01:00
bytepoets-blo	291e1eea5e	fix mapping of return values for get_or_register_3pid_guest (#5177 ) * fix mapping of return values for get_or_register_3pid_guest	2019-05-17 17:27:14 +01:00
Brendan Abolivier	cd32375846	Add option to disable per-room profiles	2019-05-16 14:34:28 +01:00
Matthew Hodgson	4a6d5de98c	Make /sync attempt to return device updates for both joined and invited users (#3484 )	2019-05-16 13:23:43 +01:00
Amber Brown	f1e5b41388	Make all the rate limiting options more consistent (#5181 )	2019-05-15 12:06:04 -05:00
Erik Johnston	b54b03f9e1	Allow client event serialization to be async	2019-05-14 11:58:01 +01:00
Matthew Hodgson	c0e0740bef	add options to require an access_token to GET /profile and /publicRooms on CS API (#5083 ) This commit adds two config options: * `restrict_public_rooms_to_local_users` Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API. * `require_auth_for_profile_requests` When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301. MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though. Groups have been intentionally omitted from this commit.	2019-05-08 18:26:56 +01:00
Brendan Abolivier	1473058b5e	Do checks on aliases for incoming m.room.aliases events (#5128 ) Follow-up to #5124 Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.	2019-05-08 17:01:30 +01:00
Erik Johnston	de655e669a	Merge pull request #5104 from matrix-org/erikj/ratelimit_3pid_invite Ratelimit 3pid invites	2019-05-07 10:12:49 +01:00
Erik Johnston	176f31c2e3	Rate limit early	2019-05-02 15:23:08 +01:00
Brendan Abolivier	84196cb231	Add some limitations to alias creation	2019-05-02 11:05:11 +01:00
Erik Johnston	d6118c5be6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ratelimit_3pid_invite	2019-04-26 18:14:23 +01:00
Erik Johnston	28a81ed62f	Ratelimit 3pid invites We do ratelimit sending the 3PID invite events, but that happens after spamming the identity server.	2019-04-26 18:06:25 +01:00
Richard van der Hoff	bd0d45ca69	Fix infinite loop in presence handler Fixes #5102	2019-04-26 11:14:49 +01:00
Katie Wolfe	7e07dc429f	Lint I probably should've just run autopep8 in the first place... Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:43:18 -04:00
Katie Wolfe	b3e5db402d	Clean up code Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:55 -04:00
Katie Wolfe	5d3ed79944	Show heroes if room name or canonical alias are empty Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:53 -04:00
Katie Wolfe	7f025eb425	Show heroes if room name or canonical alias are empty Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:29 -04:00
Erik Johnston	ca90336a69	Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration	2019-04-17 19:44:40 +01:00
Brendan Abolivier	eaf41a943b	Add management endpoints for account validity	2019-04-17 19:34:45 +01:00
Brendan Abolivier	91934025b9	Merge pull request #5047 from matrix-org/babolivier/account_expiration Send out emails with links to extend an account's validity period	2019-04-17 14:57:39 +01:00
Brendan Abolivier	20f0617e87	Send out emails with links to extend an account's validity period	2019-04-17 14:42:20 +01:00
Erik Johnston	c132c8e505	Handle the case of `get_missing_events` failing Currently if a call to `/get_missing_events` fails we log an exception and stop processing the top level event we received over federation. Instead let's try and handle it sensibly given it is a somewhat expected failure mode.	2019-04-10 10:39:54 +01:00
Brendan Abolivier	8e85493b0c	Add config option to block users from looking up 3PIDs (#5010 )	2019-04-04 17:25:47 +01:00
Andrew Morgan	db265f0642	Prevent kicking users who aren't in the room (#4999 ) Prevent kick events from succeeding if the user is not currently in the room.	2019-04-04 13:05:51 +01:00
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998 )	2019-04-03 14:32:20 +01:00
Neil Johnson	e8419554ff	Remove presence lists (#4989 ) Remove presence list support as per MSC 1819	2019-04-03 11:11:15 +01:00
Erik Johnston	8f549c1177	Merge pull request #4982 from matrix-org/erikj/msc1915 Implement MSC1915 - 3PID unbind APIs	2019-04-03 11:07:09 +01:00
Andrew Morgan	66e78700a2	Transfer related groups on room upgrade (#4990 ) Transfers the m.room.related_groups state event on room upgrade.	2019-04-02 17:15:24 +01:00
Erik Johnston	4ef5d17b96	Correctly handle id_server param	2019-04-02 11:20:09 +01:00
Erik Johnston	24232514bf	Remove threepid binding if id server returns 400/404/501	2019-04-02 11:20:09 +01:00
Erik Johnston	c75e2017f1	Fixup docstrings	2019-04-02 11:20:06 +01:00
Erik Johnston	057715aaa2	Allowing specifying IS to use in unbind API. By default the homeserver will use the identity server used during the binding of the 3PID to unbind the 3PID. However, we need to allow clients to explicitly ask the homeserver to unbind via a particular identity server, for the case where the 3PID was bound out of band from the homeserver. Implements MSC915.	2019-04-01 15:25:18 +01:00
Erik Johnston	9fbbc3d9e5	For unbind poke IS used during binding of 3PID This changes the behaviour from using the server specified trusted identity server to using the IS that used during the binding of the 3PID, if known. This is the behaviour specified by MSC1915.	2019-04-01 15:23:30 +01:00
Erik Johnston	1666c0696a	Track IS used to bind 3PIDs This will then be used to know which IS to default to when unbinding the threepid.	2019-04-01 15:23:01 +01:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969 ) Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Andrew Morgan	7a91b9d81c	Allow password providers to bind emails (#4947 ) This PR allows password provider modules to bind email addresses when a user is registering and is motivated by matrix-org/matrix-synapse-ldap3#58	2019-03-28 15:48:07 +00:00
Erik Johnston	4e5f0f7ca0	Use an assert	2019-03-28 14:05:05 +00:00
Erik Johnston	40e56997bc	Review comments	2019-03-28 13:48:41 +00:00
Erik Johnston	197fae1639	Use event streams to calculate presence Primarily this fixes a bug in the handling of remote users joining a room where the server sent out the presence for all local users in the room to all servers in the room. We also change to using the state delta stream, rather than the distributor, as it will make it easier to split processing out of the master process (as well as being more flexible). Finally, when sending presence states to newly joined servers we filter out old presence states to reduce the number sent. Initially we filter out states that are offline and have a last active more than a week ago, though this can be changed down the line. Fixes #3962	2019-03-27 13:41:36 +00:00
Andrew Morgan	bbd244c7b2	Support 3PID login in password providers (#4931 ) Adds a new method, check_3pid_auth, which gives password providers the chance to allow authentication with third-party identifiers such as email or msisdn.	2019-03-26 17:48:30 +00:00
Amber Brown	ac396a0d32	Refactor out state delta handling into its own class (#4917 )	2019-03-25 20:37:08 +11:00
Erik Johnston	01e6b405be	Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms Deny peeking into rooms that have been blocked	2019-03-21 14:07:17 +00:00
Erik Johnston	3959858eaa	Merge pull request #4904 from matrix-org/erikj/fix_shutdown Fixup shutdown room API	2019-03-21 11:24:42 +00:00
Erik Johnston	536a266520	Deny peeking into rooms that have been blocked	2019-03-21 11:20:13 +00:00
Erik Johnston	09f991a63d	Merge pull request #4896 from matrix-org/erikj/disable_room_directory Add option to disable search room lists	2019-03-21 10:16:54 +00:00
Erik Johnston	cd62981a6a	Revert spurious delete	2019-03-20 17:51:27 +00:00
Erik Johnston	aa959a6c07	Use flags	2019-03-20 17:40:29 +00:00
Erik Johnston	7d47cc1305	Move requester check into assert_accepted_privacy_policy	2019-03-20 17:08:36 +00:00
Erik Johnston	74c46d81fa	Only require consent for events with an associated request There are a number of instances where a server or admin may puppet a user to join/leave rooms, which we don't want to fail if the user has not consented to the privacy policy. We fix this by adding a check to test if the requester has an associated access_token, which is used as a proxy to answer the question of whether the action is being done on behalf of a real request from the user.	2019-03-20 16:50:23 +00:00
Richard van der Hoff	a902d13180	Batch up outgoing read-receipts to reduce federation traffic. (#4890 ) Rate-limit outgoing read-receipts as per #4730.	2019-03-20 16:02:25 +00:00
Erik Johnston	cc197a61a1	Disable publishing to room list when its disabled	2019-03-20 14:30:36 +00:00
Erik Johnston	2c90422146	Pull out config option	2019-03-20 14:25:58 +00:00
Erik Johnston	7529038e66	Return before we log	2019-03-20 14:25:28 +00:00
Erik Johnston	213c98c00a	Add option to disable search room lists This disables both local and remote room list searching.	2019-03-19 17:10:52 +00:00
Neil Johnson	88f0675967	fix test_auto_create_auto_join_where_no_consent (#4886 )	2019-03-19 11:38:59 +00:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864 )	2019-03-19 04:50:24 +11:00
Brendan Abolivier	651ad8bc96	Add ratelimiting on failed login attempts (#4865 )	2019-03-18 12:57:20 +00:00
Brendan Abolivier	899e523d6d	Add ratelimiting on login (#4821 ) Add two ratelimiters on login (per-IP address and per-userID).	2019-03-15 17:46:16 +00:00
Richard van der Hoff	2dee441bdb	Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker Move client receipt processing to federation sender worker.	2019-03-15 12:30:30 +00:00
Erik Johnston	9073cfc8bd	Merge pull request #4846 from matrix-org/hawkowl/userdir-search Improve searching in the userdir	2019-03-14 13:54:15 +00:00
Richard van der Hoff	fdcad8eabd	Move client receipt processing to federation sender worker. This is mostly a prerequisite for #4730, but also fits with the general theme of "move everything off the master that we possibly can".	2019-03-13 17:21:19 +00:00
Richard van der Hoff	eed7271b3b	declare a ReadReceipt class I'm going to use this in queues and things, so it'll be useful to give it more of a structure.	2019-03-13 17:20:55 +00:00
Andrew Morgan	d42c81d724	Transfer local user's push rules on room upgrade (#4838 ) Transfer push rules (notifications) on room upgrade	2019-03-12 14:42:53 +00:00
Amber Brown	c0332d095f	fixup	2019-03-13 01:30:54 +11:00
Amber Brown	10480c4348	fixup	2019-03-12 21:47:14 +11:00
Amber Brown	5ba8ceab4c	fixes	2019-03-12 00:35:31 +11:00
Amber Brown	4abc988c6a	initial	2019-03-11 21:11:36 +11:00
Erik Johnston	8c4896668f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl	2019-03-08 11:44:20 +00:00
Erik Johnston	0ff8163eae	Factor out soft fail checks	2019-03-08 11:26:33 +00:00
Erik Johnston	469b66c102	Merge pull request #4829 from matrix-org/erikj/device_list_seen_updates When re-syncing device lists reset the state	2019-03-08 09:54:54 +00:00
Erik Johnston	d42b41544a	When re-syncing device lists reset the state We keep track of what stream IDs we've seen so that we know what updates we've handled or missed. If we re-sync we don't know if the updates we've seen are included in the re-sync (there may be a race), so we should reset the seen updates.	2019-03-07 16:04:24 +00:00
Erik Johnston	c633fc02d7	Add some debug logging for device list handling	2019-03-07 15:58:03 +00:00
Amber Brown	f6135d06cf	Rewrite userdir to be faster (#4537 )	2019-03-07 01:22:53 -08:00
Erik Johnston	0a6e716600	Merge pull request #4815 from matrix-org/erikj/docstrings Add docstrings from matrix-org-hotfixes	2019-03-06 18:59:28 +00:00
Erik Johnston	8b7790e68f	Port #4422 debug logging from hotfixes	2019-03-06 17:29:15 +00:00
Erik Johnston	6d13bdec91	Add docstrings from matrix-org-hotfixes	2019-03-06 17:24:26 +00:00
Erik Johnston	a9de04be72	Implement soft fail	2019-03-06 16:22:16 +00:00
Brendan Abolivier	6f3cde8b25	Make registration ratelimiter separate from the main events one	2019-03-06 11:02:42 +00:00
Brendan Abolivier	f4195f4118	Revert "Split ratelimiters in two (one for events, one for registration)" This reverts commit `d7dbad3526`.	2019-03-06 10:55:22 +00:00
Brendan Abolivier	d7dbad3526	Split ratelimiters in two (one for events, one for registration)	2019-03-05 18:41:27 +00:00
Brendan Abolivier	a4c3a361b7	Add rate-limiting on registration (#4735 ) * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test	2019-03-05 14:25:33 +00:00
Erik Johnston	b050a10871	Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room Stop backpaginating when events not visible	2019-03-05 09:32:33 +00:00
Erik Johnston	aa06d26ae0	clarify comments	2019-03-05 09:16:35 +00:00
Erik Johnston	c3c542bb4a	Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys Allow /keys/{changes,query} API to run on worker	2019-03-05 09:06:25 +00:00
Erik Johnston	bfa7d46a10	Allow /keys/{changes,query} API to run on worker	2019-03-04 18:30:01 +00:00
Erik Johnston	157e5a8f27	Split DeviceHandler into master and worker	2019-03-04 18:29:26 +00:00
Richard van der Hoff	b29693a30b	Clean up read-receipt handling. Remove a call to run_as_background_process: there is no need to run this as a background process, because build_and_send_edu does not block. We may as well inline the whole of _push_remotes.	2019-03-04 18:16:43 +00:00
Erik Johnston	d1523aed6b	Only check history visibility when filtering When filtering events to send to server we check more than just history visibility. However when deciding whether to backfill or not we only care about the history visibility.	2019-03-04 14:43:42 +00:00
Richard van der Hoff	856c83f5f8	Avoid rebuilding Edu objects in worker mode (#4770 ) In worker mode, on the federation sender, when we receive an edu for sending over the replication socket, it is parsed into an Edu object. There is no point extracting the contents of it so that we can then immediately build another Edu.	2019-03-04 12:57:44 +00:00
Erik Johnston	8b63fe4c26	s/get_forward_events/get_successor_events/	2019-03-04 11:56:03 +00:00
Erik Johnston	fbc047f2a5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room	2019-03-04 11:54:58 +00:00
Richard van der Hoff	3064952939	Fix incorrect log about not persisting duplicate state event. (#4776 ) We were logging this when it was not true.	2019-03-01 16:47:12 +00:00
Erik Johnston	b183fef9ac	Update comments	2019-02-27 13:06:10 +00:00
Andrew Morgan	802884d4ee	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate_develop	2019-02-26 14:23:40 +00:00
Andrew Morgan	6fcb25202f	Put function def back to the way it was	2019-02-26 14:17:38 +00:00
Andrew Morgan	7a4632af9c	Prevent showing non-fed rooms in fed /publicRooms	2019-02-26 13:37:24 +00:00
Andrew Morgan	c74624a633	Revert "Prevent showing non-fed rooms in fed /publicRooms"	2019-02-26 13:20:38 +00:00
Andrew Morgan	a1a6473293	Merge pull request #4736 from matrix-org/anoa/public_rooms_federate Config option to prevent showing non-fed rooms in fed /publicRooms	2019-02-26 13:07:15 +00:00
Andrew Morgan	c4414768af	Cleaner chunk logic	2019-02-26 12:22:34 +00:00
Andrew Morgan	a712aa3a9c	Correct indent	2019-02-26 12:13:55 +00:00
Andrew Morgan	16565e67db	Correct docstring types and chunk logic	2019-02-26 12:12:48 +00:00
Andrew Morgan	40c2271680	Clean up room chunk logic	2019-02-26 12:04:34 +00:00
Andrew Morgan	6728bf3940	Make not showing non-federated rooms the default	2019-02-26 11:52:52 +00:00
Andrew Morgan	6946c20111	Result may be None	2019-02-26 11:27:19 +00:00
Andrew Morgan	71669a0fba	Address rich comments	2019-02-26 11:25:00 +00:00
Matthew Hodgson	70ea2f4e1d	switch from google.com to recaptcha.net for reCAPTCHA (#4731 ) * add trivial clarification about jemalloc * switch from google.com to recaptcha.net because https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally	2019-02-25 11:15:36 -08:00
Andrew Morgan	1330aa4a8f	elif not else if	2019-02-25 17:28:19 +00:00
Erik Johnston	65f3fbfbf7	Merge pull request #4737 from matrix-org/erikj/failure_log_tb Log tracebacks correctly	2019-02-25 17:26:30 +00:00
Erik Johnston	c7b333c545	Log tracebacks correctly	2019-02-25 16:56:41 +00:00
Erik Johnston	9c598dddcb	Fix typo	2019-02-25 16:32:02 +00:00
Andrew Morgan	8aaf7ffc44	syntax derp	2019-02-25 15:27:17 +00:00
Andrew Morgan	84c0a20dfe	Simplify call to generate_room_entry	2019-02-25 15:23:27 +00:00
Erik Johnston	4b9e5076c4	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate	2019-02-25 15:08:18 +00:00
Andrew Morgan	07493607a8	Docs and arg name clarification	2019-02-25 15:04:46 +00:00
Andrew Morgan	bd398b874e	Don't restrict non-fed rooms over client APIs	2019-02-25 15:04:46 +00:00
Andrew Morgan	e4b078a600	Config option to prevent showing non-fed rooms in fed /publicRooms	2019-02-25 15:04:46 +00:00
Erik Johnston	d730c2c22b	More comments	2019-02-25 14:45:02 +00:00
Erik Johnston	890cb048fd	Assert rather than clobber the values	2019-02-25 14:42:11 +00:00
Erik Johnston	9342cc6ab1	Add comments and paranoia	2019-02-25 10:02:12 +00:00
Erik Johnston	7b288826b7	Fix backfill storing incorrect state for events	2019-02-22 11:33:51 +00:00
Richard van der Hoff	6d65659b62	Run push_receipts_to_remotes as background job (#4707 ) I suspect the CPU usage metrics for this are going to /dev/null at the moment.	2019-02-21 17:50:30 +00:00
Erik Johnston	b9d6756b14	Merge pull request #4263 from rkfg/develop Prevent crash on pagination.	2019-02-21 17:42:15 +00:00
Erik Johnston	71b625d808	Stop backpaginating when events not visible	2019-02-20 18:14:12 +00:00
Erik Johnston	dbdc565dfd	Fix registration on workers (#4682 ) * Move RegistrationHandler init to HomeServer * Move post registration actions to RegistrationHandler * Add post regisration replication endpoint * Newsfile	2019-02-20 18:47:31 +11:00
Brendan Abolivier	a288bdf0b1	Merge pull request #4652 from matrix-org/babolivier/acme-delegated Support .well-known delegation when issuing certificates through ACME	2019-02-19 11:15:38 +00:00
Brendan Abolivier	a8626901cd	Fetch ACME domain into an instance member	2019-02-19 10:54:33 +00:00
Andrew Morgan	968a30a75c	Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade Transfer bans on room upgrade	2019-02-19 10:16:42 +00:00
Andrew Morgan	34ac75ce2c	lint	2019-02-18 18:23:37 +00:00
Andrew Morgan	f8b9ca53ce	Move member event processing and changelog fix	2019-02-18 17:28:52 +00:00
Erik Johnston	fc2c245a1f	Merge pull request #4666 from matrix-org/erikj/register_login_split Split out registration to worker	2019-02-18 17:18:06 +00:00
Erik Johnston	459d3d5046	Merge pull request #4668 from matrix-org/erikj/catch_exceptions Correctly handle HttpResponseException when handling device updates	2019-02-18 16:55:43 +00:00
Erik Johnston	af691e415c	Move register_device into handler	2019-02-18 16:49:38 +00:00
Erik Johnston	c30f73c86a	Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers Fix kicking guest users in worker mode	2019-02-18 15:55:48 +00:00
Brendan Abolivier	45bb55c6de	Use a configuration parameter to give the domain to generate a certificate for	2019-02-18 15:46:23 +00:00
Erik Johnston	94960cef03	pep8	2019-02-18 15:24:13 +00:00
Erik Johnston	e07cc31cb8	Correctly handle HttpResponseException	2019-02-18 14:55:09 +00:00
Brendan Abolivier	68a53f825f	Merge branch 'develop' into babolivier/acme-delegated	2019-02-18 14:52:23 +00:00
Erik Johnston	32e54b472a	Fix kicking guest users in worker mode When guest_access changes from allowed to forbidden all local guest users should be kicked from the room. This did not happen when revocation was received from federation on a worker. Presumably broken in #4141	2019-02-18 14:16:07 +00:00
Andrew Morgan	915421065b	Membership events are done later	2019-02-18 14:02:09 +00:00
Erik Johnston	d1b060b492	Merge pull request #4643 from matrix-org/erikj/catch_exceptions Correctly handle RequestSendFailed exceptions	2019-02-18 13:53:24 +00:00
Andrew Morgan	9caab0c364	Transfer bans on room upgrade	2019-02-18 13:52:51 +00:00
Erik Johnston	eb2b8523ae	Split out registration to worker This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker.	2019-02-18 12:12:57 +00:00
Andrew Morgan	6d02a13d81	Typo in info log Co-Authored-By: babolivier <contact@brendanabolivier.com>	2019-02-18 11:36:34 +00:00
Brendan Abolivier	f86b695cbd	Various cosmetics to make TravisCI happy	2019-02-15 12:29:34 +00:00
Brendan Abolivier	af8a2f679b	Remove unused import	2019-02-15 12:27:43 +00:00
Brendan Abolivier	1895d14e12	Support .well-known delegation when issuing certificates through ACME	2019-02-15 12:05:08 +00:00
Erik Johnston	f61b2068e6	Only fetch aliases when publishing rooms	2019-02-14 18:08:09 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Erik Johnston	7fc1196a36	Correctly handle RequestSendFailed exceptions This mainly reduces the number of exceptions we log.	2019-02-14 14:01:04 +00:00
Amber Brown	bb4fd8f927	Run `black` on user directory code (#4635 )	2019-02-13 23:05:32 +11:00
Erik Johnston	ba3f27b69a	Merge pull request #4608 from matrix-org/anoa/acls_room_upgrade Transfer Server ACLs on room upgrade	2019-02-12 13:20:06 +00:00
Andrew Morgan	a126f86eec	Transfer Server ACLs on room upgrade	2019-02-11 11:30:37 +00:00
Hubert Chathi	afae8442b5	make sure version is in body and wrap in linearizer queue also add tests	2019-02-08 01:32:45 -05:00
Hubert Chathi	d9e424bf64	re-try to make isort happy	2019-02-06 22:18:41 -05:00
Hubert Chathi	9ff620a518	fix import to make isort happy	2019-02-06 21:32:52 -05:00
Hubert Chathi	8248637173	add new endpoint to update backup versions	2019-02-06 17:57:10 -05:00
Hubert Chathi	664c81e8b7	return proper error codes for some 404s	2019-02-06 17:47:22 -05:00
Andrew Morgan	c45fd0dda0	Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable Copy over non-federatable trait on room upgrade	2019-01-31 20:28:37 +00:00
Richard van der Hoff	07dfe148de	Add some debug for membership syncing issues (#4538 ) I can't figure out what's going on with #4422 and #4436; perhaps this will help.	2019-01-31 18:30:40 +00:00
Andrew Morgan	3ed3cb4339	New function for getting room's create event	2019-01-31 18:21:39 +00:00
Andrew Morgan	bbb97a35fd	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable	2019-01-31 18:11:00 +00:00
Andrew Morgan	fb50934b8f	lint	2019-01-31 11:34:45 +00:00
Andrew Morgan	d621c5562e	Copy over non-federatable trait on room upgrade	2019-01-30 16:33:51 +00:00
Erik Johnston	ed8c5e4cda	Fix remote invite rejections not comming down sync This was broken in PR #4405, commit `886e5ac`, where we changed remote rejections to be outliers. The fix is to explicitly add the leave event in when we know its an out of band invite. We can't always add the event as if the server is/was in the room there might be more events to send down the sync than just the leave.	2019-01-30 15:46:27 +00:00
Richard van der Hoff	7615a8ced1	ACME config cleanups (#4525 ) * Handle listening for ACME requests on IPv6 addresses the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses without extra quoting. Building a string which you are about to parse again seems like a weird choice. Let's just use listenTCP, which is consistent with what we do elsewhere. * Clean up the default ACME config make it look a bit more consistent with everything else, and tweak the defaults to listen on port 80. * newsfile	2019-01-30 14:17:55 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519 )	2019-01-30 10:53:17 +00:00
Erik Johnston	67b82f1336	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:58:38 +00:00
Erik Johnston	a696c48133	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:00:33 +00:00
Erik Johnston	55d9024835	Use snder and not event ID domain to check if ours The transaction queue only sends out events that we generate. This was done by checking domain of event ID, but that can no longer be used. Instead, we may as well use the sender field.	2019-01-29 16:54:23 +00:00
Erik Johnston	b1fffca345	Remove event ID usage when checking if new room The event ID is changing, so we can no longer get the domain from it. On the other hand, the check is unnecessary.	2019-01-29 16:15:02 +00:00
Erik Johnston	b8d75ef53e	Merge pull request #4481 from matrix-org/erikj/event_builder Refactor event building into EventBuilder	2019-01-29 14:07:23 +00:00
Erik Johnston	5891a6edc8	Correctly set context.app_service	2019-01-29 12:09:10 +00:00
Erik Johnston	7709d2bd16	Implement rechecking of redactions	2019-01-29 11:56:20 +00:00
Erik Johnston	6598992b01	Fixup calls to `comput_event_signature` We currently pass FrozenEvent instead of `dict` to `compute_event_signature`, which works by accident due to `dict(event)` producing the correct result. This fixes PR #4493 commit `855a151`	2019-01-29 11:41:58 +00:00
Erik Johnston	be47cfa9c9	Refactor event building into EventBuilder This is so that everything is done in one place, making it easier to change the event format based on room version	2019-01-29 11:13:00 +00:00
Erik Johnston	073f6c2e5e	Merge pull request #4494 from matrix-org/erikj/fixup_event_validator Split up event validation between event and builder	2019-01-29 10:55:07 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Erik Johnston	d758d5310e	Correctly use default room version if none is set	2019-01-28 17:26:39 +00:00
Erik Johnston	b872c7b1b4	Split up event validation between event and builder The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.	2019-01-28 17:00:14 +00:00
Andrew Morgan	f0e96ab66a	Change return syntax in doc string	2019-01-28 14:09:45 +00:00
Andrew Morgan	4026d555fa	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-28 14:08:24 +00:00
Andrew Morgan	1ce463963d	Reuse predecessor method	2019-01-28 14:08:18 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Andrew Morgan	9244a3089e	Fixes	2019-01-25 11:48:38 +00:00
Andrew Morgan	da0d2219d2	Clean up direct_rooms access	2019-01-25 11:37:12 +00:00
Andrew Morgan	8265995498	Use python magic	2019-01-25 11:26:06 +00:00
Andrew Morgan	c4cdafa81f	Destructure account data tuple before use	2019-01-25 11:24:28 +00:00
Andrew Morgan	516456b763	Remove unnecessary null check	2019-01-25 11:22:14 +00:00
Andrew Morgan	6f3fda79ce	Move room_tag declaration to be closer to its use	2019-01-25 11:21:25 +00:00
Andrew Morgan	821b65aeb5	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-25 11:09:53 +00:00
Andrew Morgan	0862d35b8e	Move tag and direct state copying into separate function	2019-01-25 11:09:34 +00:00
Andrew Morgan	b1b6dba2d2	Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms Ability to search entire room history after upgrading room	2019-01-25 10:49:58 +00:00
Erik Johnston	a50cf929c1	Require event format version to parse or create events	2019-01-25 10:32:19 +00:00
Erik Johnston	62514bb81b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813	2019-01-25 10:07:08 +00:00
Erik Johnston	edc1e21dbe	Merge pull request #4448 from matrix-org/erikj/get_pdu_versions Add room_version param to get_pdu	2019-01-24 18:47:15 +00:00
Erik Johnston	80bcca659e	Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites Store rejected remote invite events as outliers	2019-01-24 18:33:27 +00:00
Erik Johnston	d148c43050	Review comments	2019-01-24 18:31:23 +00:00
Erik Johnston	9139b87be4	Remove unecessary setting of outlier bit	2019-01-24 18:04:02 +00:00
Erik Johnston	b8082a5445	Use term 'out of band membership' instead	2019-01-24 17:33:19 +00:00
Richard van der Hoff	03c85335d1	Apply suggestions from code review Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:22:09 +00:00
Andrew Morgan	075ff3ede9	Change default timeout value from 0 to None	2019-01-24 15:10:22 +00:00
Andrew Morgan	5b1dc94083	Use self.clock instead of datetime	2019-01-24 14:59:50 +00:00
Andrew Morgan	a2d85144e5	isort	2019-01-24 14:22:26 +00:00
Andrew Morgan	5541645e80	lint	2019-01-24 12:45:32 +00:00
Andrew Morgan	068aa1d228	Time out filtered room dir queries after 60s	2019-01-24 12:44:27 +00:00
Erik Johnston	7c288c2250	Clarify the invite flows	2019-01-23 20:07:47 +00:00
Erik Johnston	07f62da55a	Remove unnecessary '_sign_event'	2019-01-23 20:04:57 +00:00
Erik Johnston	886e5acc76	Store rejected remote invite events as outliers Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.	2019-01-23 20:04:57 +00:00
Erik Johnston	6a41d2a187	Add room_version param to get_pdu When we add new event format we'll need to know the event format or room version when parsing events.	2019-01-23 17:19:58 +00:00
Erik Johnston	67cd4dad81	Implement MSC 1813 - Add room version to make APIs We also implement `make_membership_event` converting the returned room version to an event format version.	2019-01-23 16:51:46 +00:00
Andrew Morgan	82a92ba535	Add metric for user dir current event stream position	2019-01-23 15:34:47 +00:00
Amber Brown	6129e52f43	Support ACME for certificate provisioning (#4384 )	2019-01-23 19:39:06 +11:00
Andrew Morgan	766a172b99	lint	2019-01-22 13:51:40 +00:00
Andrew Morgan	c4875d8c76	Prevent duplicate room IDs in m.direct	2019-01-22 12:13:46 +00:00
Andrew Morgan	c9bfb058d8	Fix a bug with single-room search searching all rooms * Create a new method for getting predecessor rooms * Remove formatting change	2019-01-22 12:00:41 +00:00
Andrew Morgan	8086a5c05e	Fix comments	2019-01-22 11:16:23 +00:00
Andrew Morgan	48951f437f	Join logic covers both room creator and arbitrary users	2019-01-22 11:00:04 +00:00
Andrew Morgan	8c85f0833d	tags, m.direct copying over correctly	2019-01-22 11:00:04 +00:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Andrew Morgan	ea8903fcc9	Migrating dm and room tags work for migrator	2019-01-22 11:00:04 +00:00
Andrew Morgan	887ca93a1b	Prevent crash on user who doesn't have any direct rooms	2019-01-22 11:00:04 +00:00
Andrew Morgan	4ff6d22245	Preserve DM status of a room on upgrade Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-22 11:00:04 +00:00
Andrew Morgan	75942af1db	Fix typo	2019-01-22 11:00:04 +00:00
Andrew Morgan	702c4b750c	Migrate encryption state on room upgrade (#4411 ) * Migrate encryption state on room upgrade Signed-off-by: Andrew Morgan <andrew@amorgan.xyz> * Add changelog file	2019-01-21 20:42:58 +11:00
Andrew Morgan	df3a661e4a	Search for messages across predecessor rooms Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-18 11:19:20 +00:00
Richard van der Hoff	9feb5d0b71	sign_request -> build_auth_headers (#4408 ) Just got very confused about the fact that the headers are only an output, not an input.	2019-01-17 12:40:09 +00:00
Richard van der Hoff	05e1296649	don't store more remote device lists if they have more than 1K devices (#4397 )	2019-01-16 23:14:11 +00:00
Matthew Hodgson	27128145e6	fix NPE in /messages by checking if all events were filtered out (#4330 )	2019-01-02 23:37:39 +00:00
Nathan Pennie	81b513416e	Fixed line length	2018-12-19 14:12:33 -05:00
Nathan Pennie	668e6625b0	Fix the variable names used for account_data	2018-12-19 06:53:02 -05:00
Neil Johnson	d2f7c4e6b1	create support user (#4141 ) Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272 ) When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	c588b9b9e4	Factor SSO success handling out of CAS login (#4264 ) This is mostly factoring out the post-CAS-login code to somewhere we can reuse it for other SSO flows, but it also fixes the userid mapping while we're at it.	2018-12-07 13:10:07 +01:00
rkfg	ae19a7db8c	Prevent crash on pagination.	2018-12-06 13:32:05 +03:00
Richard van der Hoff	b0c24a66ec	Rip out half-implemented m.login.saml2 support (#4265 ) * Rip out half-implemented m.login.saml2 support This was implemented in an odd way that left most of the work to the client, in a way that I really didn't understand. It's going to be a pain to maintain, so let's start by ripping it out. * drop undocumented dependency on dateutil It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 19:44:38 +11:00
Travis Ralston	158ffb92f1	Add an option to disable search for homeservers which may not be interested in it (#4230 ) This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.	2018-12-04 12:01:02 +01:00
Neil Johnson	7039ece8fb	Neilj/fix autojoin (#4223 ) * Fix auto join failures for servers that require user consent * Fix auto join failures for servers that require user consent	2018-11-28 22:24:57 +11:00
Travis Ralston	ab4526a153	Remove duplicate slashes in generated consent URLs	2018-11-15 20:41:53 -07:00
Erik Johnston	dc59ad5334	Remove hack to support rejoining rooms	2018-11-09 14:58:09 +00:00
Erik Johnston	30dd27afff	Simplify to always drop events if server isn't in the room	2018-11-09 11:36:45 +00:00
Travis Ralston	0f5e51f726	Add config variables for enabling terms auth and the policy name (#4142 ) So people can still collect consent the old way if they want to.	2018-11-06 10:32:34 +00:00
Hubert Chathi	f1087106cf	handle empty backups according to latest spec proposal (#4123 ) fixes #4056	2018-11-05 17:59:29 -05:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139 ) * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Amber Brown	cb7a6b2379	Fix typing being reset causing infinite syncs (#4127 )	2018-11-03 00:19:23 +11:00
Travis Ralston	a8d41c6aff	Include a version query string arg for the consent route	2018-10-31 13:19:28 -06:00
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00
Richard van der Hoff	1729ba1650	Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades Attempt to move room aliases on room upgrades	2018-10-31 17:52:18 +00:00
Richard van der Hoff	0f8591a5a8	Avoid else clause on exception for clarity	2018-10-31 15:43:57 +00:00
Richard van der Hoff	94c7fadc98	Attempt to move room aliases on room upgrades	2018-10-31 15:43:57 +00:00
Richard van der Hoff	9b827c40ca	Log some bits about event creation (#4121 ) I found these helpful in debugging my room upgrade tests.	2018-10-31 15:42:23 +00:00
Amber Brown	086e1a8f3e	Fix search 500ing (#4122 )	2018-10-31 06:26:46 +11:00
Amber Brown	3bade14ec0	Fix search 500ing (#4122 )	2018-10-31 04:33:41 +11:00
Amber Brown	4cd1c9f2ff	Delete the disused & unspecced identicon functionality (#4106 )	2018-10-29 23:57:24 +11:00
Richard van der Hoff	7fbfea062e	Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar Remember to copy the avatar on room upgrades	2018-10-29 12:49:21 +00:00
Richard van der Hoff	db24d7f15e	Better handling of odd PLs during room upgrades Fixes handling of rooms where we have permission to send the tombstone, but not other state. We need to (a) fail more gracefully when we can't send the PLs in the old room, and (b) not set the PLs in the new room until we are done with the other stuff.	2018-10-27 00:54:26 +01:00
Richard van der Hoff	5caf79b312	Remember to copy the avatar on room upgrades	2018-10-26 23:56:40 +01:00
Richard van der Hoff	54bbe71867	optimise state copying	2018-10-26 22:51:34 +01:00
Richard van der Hoff	193cadc988	Address review comments Improve comments, get old room state from the context we already have	2018-10-26 17:10:30 +01:00
Richard van der Hoff	e6babc27d5	restrict PLs in old room	2018-10-25 19:18:25 +01:00
Richard van der Hoff	3a263bf3ae	copy state	2018-10-25 19:18:25 +01:00
Richard van der Hoff	1b9f253e20	preserve PLs	2018-10-25 19:10:24 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Richard van der Hoff	0f7d1c9906	Basic initial support for room upgrades Currently just creates a new, empty, room, and sends a tombstone in the old room.	2018-10-25 19:10:24 +01:00
Richard van der Hoff	e1948175ee	Allow power_level_content_override=None for _send_events_for_new_room	2018-10-25 19:10:24 +01:00
Richard van der Hoff	7f7b2cd3de	Make room_member_handler a member of RoomCreationHandler ... to save passing it into `_send_events_for_new_room`	2018-10-25 19:10:18 +01:00
Richard van der Hoff	871c4abfec	Factor _generate_room_id out of create_room we're going to need this for room upgrades.	2018-10-25 18:23:09 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011 ) Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Erik Johnston	b94a43d5b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list	2018-10-25 15:25:31 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Travis Ralston	54def42c19	Merge branch 'develop' into travis/login-terms	2018-10-24 13:22:59 -06:00
Neil Johnson	9532caf6ef	remove trailing whiter space	2018-10-24 16:08:25 +01:00
Richard van der Hoff	94a49e0636	fix tuple Co-Authored-By: neilisfragile <neil@matrix.org>	2018-10-24 14:39:23 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	e0b9d5f0af	Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts Clean up the way logcontexts and threads work in the pushers	2018-10-24 09:53:57 +01:00
Erik Johnston	dacbeb2e03	Comment	2018-10-24 09:47:49 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Erik Johnston	cb23aa4c42	Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation Clean up room alias creation	2018-10-24 09:40:27 +01:00
Neil Johnson	a67d8ace9b	remove errant exception and style	2018-10-23 17:44:39 +01:00
Travis Ralston	43c3f0b02f	Merge pull request #3969 from turt2live/travis/fix-federated-group-requests Handle HttpResponseException more safely for federated groups	2018-10-23 10:41:04 -06:00
Travis Ralston	3e704822be	Comments help	2018-10-23 10:25:31 -06:00
Richard van der Hoff	b3f6dddad2	Give some more things logcontexts (#4077 )	2018-10-23 14:29:17 +01:00
Richard van der Hoff	5c445114d3	Correctly account for cpu usage by background threads (#4074 ) Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064	2018-10-23 13:12:32 +01:00
Richard van der Hoff	e7a16c6210	Remove redundant run_as_background_process() from pusherpool `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.	2018-10-22 16:12:11 +01:00
Erik Johnston	084046456e	Add config option to control alias creation	2018-10-19 10:22:45 +01:00
Erik Johnston	74e7617083	Clean up room alias creation	2018-10-19 10:11:56 +01:00
Travis Ralston	49a044aa5f	Merge branch 'develop' into travis/login-terms	2018-10-18 09:57:58 -06:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046 ) Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Neil Johnson	c6584f4b5f	clean up config error logic and imports	2018-10-17 11:36:41 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events It's quite important that get_missing_events returns the latest events in the room; however we were pulling event ids out of the database until we got at least 10, and then taking the earliest of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	15133477ee	Fix up use of resolve_events_with_factory	2018-10-16 16:28:42 +01:00
Travis Ralston	a8ed93a4b5	pep8	2018-10-15 16:10:29 -06:00
Travis Ralston	442734ff9e	Ensure the terms params are actually provided	2018-10-15 14:56:13 -06:00
Travis Ralston	762a0982aa	Python is hard	2018-10-15 14:46:09 -06:00
Travis Ralston	f293d124b6	Merge branch 'develop' into travis/login-terms	2018-10-15 14:44:32 -06:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups E2E backups	2018-10-15 10:19:43 +01:00
Neil Johnson	1ccafb0c5e	no need to join room if creator	2018-10-13 21:14:21 +01:00
Travis Ralston	dd99db846d	Update login terms structure for the proposed language support	2018-10-12 18:03:27 -06:00
Travis Ralston	7ede650956	Merge branch 'develop' into travis/login-terms	2018-10-12 16:24:07 -06:00
Travis Ralston	164f8e4843	isort	2018-10-12 15:11:59 -06:00
Travis Ralston	7bb651de6a	More sane handling of group errors and pep8	2018-10-12 14:53:30 -06:00
Travis Ralston	e3586f7c06	Merge branch 'develop' into travis/fix-federated-group-requests	2018-10-12 14:49:58 -06:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	8c0ff0287a	Linting soothes the savage PEP8 monster	2018-10-12 13:47:43 +01:00
David Baker	83e72bb2f0	PR feedback pt. 1	2018-10-12 11:26:18 +01:00
Erik Johnston	7e561b5c1a	Add description to counter metric	2018-10-10 11:41:15 +01:00
Erik Johnston	3cbe8331e6	Track number of non-empty sync responses instead	2018-10-10 11:23:17 +01:00
Erik Johnston	395276b405	Append _total to metric and fix up spelling	2018-10-10 09:24:39 +01:00
Erik Johnston	bdc27d6716	Add metric to count lazy member sync requests	2018-10-09 14:17:52 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	f4a4dbcad1	Apparently this blank line is Very Important	2018-10-09 09:47:04 +01:00
David Baker	0c905ee015	be python3 compatible	2018-10-09 09:39:13 +01:00
Erik Johnston	6982320572	Remove unnecessary extra function call layer	2018-10-08 14:06:19 +01:00
Erik Johnston	8a1817f0d2	Use errback pattern and catch async failures	2018-10-08 13:29:47 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Travis Ralston	537d0b7b36	Use a flag rather than a new route for the public policy This also means that the template now has optional parameters, which will need to be documented somehow.	2018-10-03 17:50:11 -06:00
Travis Ralston	149c4f1765	Supply params for terms auth stage As per https://github.com/matrix-org/matrix-doc/pull/1692	2018-10-03 15:57:42 -06:00
Travis Ralston	fd99787162	Incorporate Dave's work for GDPR login flows As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117	2018-10-03 15:57:42 -06:00
Erik Johnston	495a9d06bb	Fix exception handling in fetching remote profiles	2018-10-03 11:34:30 +01:00
Richard van der Hoff	8c41b0ca66	Merge pull request #3989 from matrix-org/rav/better_stacktraces Avoid reraise, to improve stacktraces	2018-10-02 10:47:38 +01:00
Erik Johnston	bc29946809	Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state Fix lazy loaded sync with rejected state events	2018-10-02 11:38:35 +02:00
Richard van der Hoff	8174c6725b	Avoid reraise, to improve stacktraces	2018-10-01 18:50:34 +01:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Erik Johnston	82f922b4af	Fix lazy loaded sync with rejected state events In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.	2018-10-01 14:19:40 +01:00
Matthew Hodgson	23b6a0537f	emit room aliases event	2018-09-29 02:19:37 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Matthew Hodgson	8f646f2d04	fix UTs	2018-09-28 15:37:28 +01:00
Richard van der Hoff	3deaad2fb4	Merge pull request #3964 from matrix-org/rav/remove_localhost_checks remove spurious federation checks on localhost	2018-09-28 13:35:47 +01:00
Richard van der Hoff	965154d60a	Fix complete fail to do the right thing	2018-09-28 12:45:54 +01:00
Matthew Hodgson	07340cdaca	untested stab at autocreating autojoin rooms	2018-09-28 01:42:53 +01:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Richard van der Hoff	36c62a67c4	Merge pull request #3794 from matrix-org/erikj/faster_typing Improve performance of getting typing updates for replication	2018-09-27 15:14:26 +01:00
Richard van der Hoff	e1e3e77bfd	Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups Clarifications in FederationHandler	2018-09-27 15:06:59 +01:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	333bee27f5	Include event when resolving state for missing prevs If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	bd61c82bdf	Include state from remote servers in pdu handling If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	a215b698c4	Fix "unhashable type: 'list'" exception in federation handling get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	28223841e0	more comments	2018-09-27 11:31:51 +01:00
Richard van der Hoff	e3c159863d	Clarifications in FederationHandler * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)	2018-09-27 11:31:51 +01:00
Richard van der Hoff	92abd3d6d6	Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 Logging improvements	2018-09-27 11:27:46 +01:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959 ) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Travis Ralston	82fa31799c	Remove debugging statement	2018-09-26 14:01:02 -06:00
Travis Ralston	b4c3bc1734	Handle HttpResponseException more safely for federated groups	2018-09-26 13:48:04 -06:00
Richard van der Hoff	e70b4ce069	Logging improvements Some logging tweaks to help with debugging incoming federation transactions	2018-09-26 17:36:14 +01:00
Amber Brown	c2185f14d7	Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu Comments and interface cleanup for on_receive_pdu	2018-09-26 21:41:26 +10:00
Matthew Hodgson	787d22ed6c	Only lazy load self-members on initial sync Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.	2018-09-25 00:49:26 +01:00
Matthew Hodgson	a2ddaa90f2	Always LL ourselves if we're in a room to simplify clients (#3916 ) Should fix https://github.com/vector-im/riot-web/issues/7209	2018-09-20 21:21:54 +01:00
Richard van der Hoff	703de4ec13	Comments and interface cleanup for on_receive_pdu Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.	2018-09-20 13:06:55 +01:00
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904 ) Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	2018-09-19 17:28:18 +01:00
Erik Johnston	ce846bb620	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing	2018-09-19 15:08:36 +01:00
Amber Brown	f773ecbd61	Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout Bump timeout on get_missing_events request	2018-09-19 17:57:48 +10:00
Richard van der Hoff	a219ce8726	Use directory server for room joins (#3899 ) When we do a join, always try the server we used for the alias lookup first. Fixes #2418	2018-09-18 18:27:37 +01:00
Richard van der Hoff	550007cb0e	Bump timeout on get_missing_events request	2018-09-18 15:02:51 +01:00
Richard van der Hoff	286d6930b7	Merge pull request #3879 from matrix-org/matthew/fix-autojoin don't ratelimit autojoins	2018-09-18 13:07:01 +01:00
Richard van der Hoff	f75b9961c6	Reinstate missing null check	2018-09-17 16:52:02 +01:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
Matthew Hodgson	d42d79e3c3	don't ratelimit autojoins	2018-09-15 22:27:41 +01:00
Matthew Hodgson	024be6cf18	don't filter membership events based on history visibility (#3874 ) don't filter membership events based on history visibility as we will already have filtered the messages in the timeline, and state events are always visible. and because @erikjohnston said so.	2018-09-14 18:12:52 +01:00
Matthew Hodgson	2ac1abbc7e	show heroes if a room has a 'deleted' name/canonical_alias (#3851 )	2018-09-12 17:11:05 +01:00
Matthew Hodgson	0403cf0783	argh pep8	2018-09-12 16:54:28 +01:00
Matthew Hodgson	0e200e366d	correctly log gappy sync metrics	2018-09-12 16:47:20 +01:00
Matthew Hodgson	11bfc2af1c	fix logline	2018-09-12 16:45:42 +01:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827 ) * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Erik Johnston	f60c9e2a01	Don't send empty tags list down sync	2018-09-06 17:01:41 +01:00
Erik Johnston	7baf66ef5d	Send existing room tags down sync on join When a user joined a room any existing tags were not sent down the sync stream. Ordinarily this isn't a problem because the user needs to be in the room to have set tags in it, however synapse will sometimes add tags for a user to a room, e.g. for server notices, which need to come down sync.	2018-09-06 16:46:51 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Amber Brown	ee7c8bd2b5	Merge pull request #3795 from matrix-org/erikj/faster_sync_state User iter* during sync state calculations	2018-09-07 00:24:43 +10:00
Amber Brown	2608ebc04c	Port handlers/ to Python 3 (#3803 )	2018-09-07 00:22:23 +10:00
Erik Johnston	7419764351	User iter* during sync state calculations	2018-09-05 16:19:50 +01:00
Erik Johnston	5f02017aea	Improve performance of getting typing updates for replication Fetching the list of all new typing notifications involved iterating over all rooms and comparing their serial. Lets move to using a stream change cache, like we do for other streams.	2018-09-05 10:20:40 +01:00
Erik Johnston	567363e497	Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func Remove unnecessary resolve_events_with_state_map	2018-09-03 16:19:41 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Matthew Hodgson	b1580f50fe	don't return non-LL-member state in incremental sync state blocks (#3760 ) don't return non-LL-member state in incremental sync state blocks	2018-08-28 23:25:58 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	5c261107c9	Remove unnecessary resolve_events_with_state_map We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.	2018-08-22 15:41:15 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	8432e2ebd7	Rename WorkerProfileHandler to BaseProfileHandler	2018-08-22 10:13:40 +01:00
Erik Johnston	a81f140880	Add assert to ensure handler is only run on master	2018-08-22 10:11:21 +01:00
Erik Johnston	47b25ba5f3	Remove redundant vars	2018-08-22 10:09:05 +01:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	8dee601054	Remove redundant room_version checks	2018-08-20 14:48:53 +01:00
Erik Johnston	e21c368b8b	Revert spurious change	2018-08-20 13:54:51 +01:00
Richard van der Hoff	c144252a8c	Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates Fix logcontexts for running pushers	2018-08-17 16:21:49 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694 )	2018-08-18 01:08:45 +10:00
Erik Johnston	ab822a2d1f	Add some fixmes	2018-08-17 15:31:50 +01:00
Erik Johnston	91cdb6de08	Call UserDirectoryHandler methods directly Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.	2018-08-17 15:26:13 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Richard van der Hoff	66f7dc8c87	Fix logcontexts for running pushers First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.	2018-08-17 00:32:39 +01:00
Neil Johnson	372bf073c1	block event creation and room creation on hitting resource limits	2018-08-16 21:25:16 +01:00
Matthew Hodgson	762a758fea	lazyload aware /messages (#3589 )	2018-08-16 14:22:47 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574 )	2018-08-16 09:46:50 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568 )	2018-08-15 16:35:22 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Erik Johnston	dc56c47dc0	Merge pull request #3653 from matrix-org/erikj/split_federation Move more federation APIs to workers	2018-08-15 14:59:02 +01:00
Neil Johnson	4601129c44	Merge pull request #3687 from matrix-org/neilj/admin_email support admin_email config and pass through into blocking errors,	2018-08-15 13:52:25 +00:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Erik Johnston	488ffe6fdb	Use federation handler function rather than duplicate This involves renaming _persist_events to be a public function.	2018-08-15 14:17:18 +01:00
Erik Johnston	fef2e65d12	Merge pull request #3667 from matrix-org/erikj/fixup_unbind Don't fail requests to unbind 3pids for non supporting ID servers	2018-08-15 10:32:12 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	c74c71128d	remove blank line	2018-08-14 15:06:24 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	bd92c8eaa7	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:56:45 +01:00
Neil Johnson	99ebaed8e6	Update register.py remove comments	2018-08-14 14:55:55 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" This reverts commit `0d43f991a1`.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423 )	2018-08-13 16:47:46 +10:00
Matthew Hodgson	72788cf9c1	support DELETE /version with no args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	5e42c45c96	switch get_current_version_info back to being get_version_info	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	14b3da63a3	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f500cb39e	more docstring for the e2e_room_keys rest	2018-08-12 19:14:31 -04:00
Matthew Hodgson	ca0b052307	fix factoring out of _should_replace_room_key	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Amber Brown	b37c472419	Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678 )	2018-08-10 23:50:21 +10:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Erik Johnston	b179537f2a	Move clean_room_for_join to master	2018-08-09 10:37:38 +01:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Erik Johnston	bf7598f582	Log when we 3pid/unbind request fails	2018-08-09 10:09:56 +01:00
Erik Johnston	2bdafaf3c1	Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet Add helper base class for generating new replication endpoints	2018-08-09 10:06:23 +01:00
Richard van der Hoff	bb89c84614	Merge pull request #3664 from matrix-org/rav/federation_metrics more metrics for the federation and appservice senders	2018-08-08 23:16:58 +01:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Erik Johnston	360ba89c50	Don't fail requests to unbind 3pids for non supporting ID servers Older identity servers may not support the unbind 3pid request, so we shouldn't fail the requests if we received one of 400/404/501. The request still fails if we receive e.g. 500 responses, allowing clients to retry requests on transient identity server errors that otherwise do support the API. Fixes #3661	2018-08-08 12:06:18 +01:00
Richard van der Hoff	53bca4690b	more metrics for the federation and appservice senders	2018-08-07 19:09:48 +01:00
Richard van der Hoff	3523f5432a	Don't expose default_room_version as config opt	2018-08-07 12:51:57 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Erik Johnston	a3f5bf79a0	Add EDU/query handling over replication	2018-08-06 15:23:31 +01:00
Erik Johnston	e26dbd82ef	Add replication APIs for persisting federation events	2018-08-06 15:02:28 +01:00
Richard van der Hoff	f900d50824	include known room versions in outgoing make_joins	2018-08-06 13:45:37 +01:00
Neil Johnson	e40a510fbf	py3 fix	2018-08-03 23:19:13 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Will Hunt	16d9701892	Return M_NOT_FOUND when a profile could not be found. (#3596 )	2018-08-03 19:08:05 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Erik Johnston	cb298ff623	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet	2018-08-03 09:25:15 +01:00
Richard van der Hoff	1fa98495d0	Merge pull request #3639 from matrix-org/rav/refactor_error_handling Clean up handling of errors from outbound requests	2018-08-02 17:38:24 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	0bf5ec0db7	Check room visibility for /event/ requests Make sure that the user has permission to view the requeseted event for /event/{eventId} and /room/{roomId}/event/{eventId} requests. Also check that the event is in the given room for /room/{roomId}/event/{eventId}, for sanity.	2018-08-02 15:03:27 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Richard van der Hoff	01e93f48ed	Kill off MatrixCodeMessageException This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.	2018-08-01 16:02:46 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569 )	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Erik Johnston	4b256b9271	_persist_auth_tree no longer returns anything	2018-08-01 13:39:07 +01:00
Neil Johnson	0aba3d361a	count_monthly_users() async	2018-08-01 11:47:58 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Richard van der Hoff	bdbdceeafa	rename replication_layer to federation_client I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.	2018-07-31 15:44:05 +01:00
Erik Johnston	443da003bc	Use new helper base class for membership requests	2018-07-31 14:32:23 +01:00
Erik Johnston	729b672823	Use new helper base class for ReplicationSendEventRestServlet	2018-07-31 14:32:23 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567 ) make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	b0b5566f36	Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid if inviter_display_name == ""\|\|None then default to inviter MXID	2018-07-27 10:08:39 +01:00
Matthew Hodgson	a75231b507	Deduplicate redundant lazy-loaded members (#3331 ) * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review	2018-07-26 22:51:30 +01:00

... 17 18 19 20 21 ...

4145 Commits