3949 Commits

Author	SHA1	Message	Date
Amber Brown	020add5099	Update black to 19.10b0 (#6304) ... * update version of black and also fix the mypy config being overridden	2019-11-01 02:43:24 +11:00
Erik Johnston	cd581338cf	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-10-31 15:19:26 +00:00
Erik Johnston	dfe0cd71b6	Merge pull request #6294 from matrix-org/erikj/add_state_storage ... Add StateGroupStorage interface	2019-10-31 16:17:53 +01:00
Andrew Morgan	54fef094b3	Remove usage of deprecated logger.warn method from codebase (#6271) ... Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.	2019-10-31 10:23:24 +00:00
Erik Johnston	7c8c97e635	Split purge API into events vs state	2019-10-30 15:23:37 +00:00
Erik Johnston	69f0054ce6	Port to use state storage	2019-10-30 14:46:54 +00:00
Hubert Chathi	7d7eac61be	Merge branch 'develop' into cross-signing_federation	2019-10-30 10:17:10 -04:00
Hubert Chathi	bc32f102cd	black	2019-10-30 10:07:36 -04:00
Hubert Chathi	d78b1e339d	apply changes as a result of PR review	2019-10-30 10:01:53 -04:00
Erik Johnston	ec6de1cc7d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store	2019-10-30 13:37:04 +00:00
Andrew Morgan	a2276d4d3c	Fix log line that was printing undefined value (#6278)	2019-10-30 11:28:48 +00:00
Erik Johnston	2c35ffead2	Port receipt and read markers to async/wait	2019-10-29 15:08:22 +00:00
Richard van der Hoff	172f264ed3	Improve signature checking on some federation APIs (#6262) ... Make sure that we check that events sent over /send_join, /send_leave, and /invite, are correctly signed and come from the expected servers.	2019-10-28 12:43:23 +00:00
Erik Johnston	f666d6f5d7	Remove repeated calls to config.stats_enabled. ... Turns out that fetching variables from the config object is expensive, so doing it once at startup avoids unnecessary work.	2019-10-25 10:28:36 +01:00
Hubert Chathi	ff05c9b760	don't error if federation query doesn't have cross-signing keys	2019-10-24 21:46:11 -04:00
Erik Johnston	3ca4c7c516	Use new EventPersistenceStore	2019-10-23 16:15:03 +01:00
Hubert Chathi	480eac30eb	black	2019-10-22 22:37:16 -04:00
Hubert Chathi	404e8c8532	vendor-prefix the EDU name until MSC1756 is merged into the spec	2019-10-22 22:33:23 -04:00
Hubert Chathi	1fabf82d50	update to work with newer code, and fix formatting	2019-10-22 21:44:58 -04:00
Hubert Chathi	41ad35b523	add missing param	2019-10-22 19:06:29 -04:00
Hubert Chathi	cfdb84422d	make black happy	2019-10-22 19:06:06 -04:00
Hubert Chathi	8d3542a64e	implement federation parts of cross-signing	2019-10-22 19:04:35 -04:00
Hubert Chathi	2761731634	Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2 ... Cross-signing [3/4] -- uploading signatures edition	2019-10-22 17:05:06 +01:00
Richard van der Hoff	93eaeec75a	Remove Auth.check method (#6217) ... This method was somewhat redundant, and confusing.	2019-10-18 19:43:36 +02:00
Hubert Chathi	36adfaedab	Merge branch 'develop' into cross-signing_sig_upload	2019-10-18 18:34:42 +01:00
Hubert Chathi	125eb45e19	fix doc strings	2019-10-18 16:56:16 +01:00
Hubert Chathi	0a9d22808c	Merge branch 'uhoreg/e2e_cross-signing_merged' into develop	2019-10-18 15:57:40 +01:00
Richard van der Hoff	2b22faded7	Merge remote-tracking branch 'origin/develop' into rav/event_auth/1	2019-10-18 12:35:13 +01:00
Hubert Chathi	cfc28325a6	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-10-18 11:39:40 +01:00
Richard van der Hoff	80003dfcd5	Merge remote-tracking branch 'origin/develop' into rav/event_auth/1	2019-10-18 10:11:40 +01:00
Erik Johnston	5859a5c569	Fix presence timeouts when synchrotron restarts. (#6212) ... * Fix presence timeouts when synchrotron restarts. Handling timeouts would fail if there was an external process that had timed out, e.g. a synchrotron restarting. This was due to a couple of variable name typoes. Fixes #3715.	2019-10-18 06:42:26 +01:00
Richard van der Hoff	1ba359a11f	rip out some unreachable code ... The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.	2019-10-17 21:45:59 +01:00
Hubert Chathi	06fc66c81e	Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage ... make storage layer in charge of interpreting the device key data	2019-10-11 15:19:06 -04:00
Hubert Chathi	691dd67fcd	Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version ... make version optional in body of e2e backup version update	2019-10-11 10:11:59 -04:00
Hubert Chathi	7a0dce9259	make sure we actually return something	2019-10-10 20:31:30 -04:00
Hubert Chathi	4908fb3b30	make storage layer in charge of interpreting the device key data	2019-10-10 19:15:30 -04:00
werner291	b5b03b7079	Add domain validation when creating room with list of invitees (#6121)	2019-10-10 13:05:48 +01:00
Richard van der Hoff	a139420a3c	Fix races in room stats (and other) updates. (#6187) ... Hopefully this will fix the occasional failures we were seeing in the room directory. The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8. We can solve this easily by only processing up to the stream_id where we know all events have been persisted.	2019-10-10 11:29:01 +01:00
Andrew Morgan	da815c1f69	Move tag/push rules room upgrade checking ealier (#6155) ... It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers. The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).	2019-10-10 10:06:45 +01:00
Hubert Chathi	4535a07f4a	make version optional in body of e2e backup version update ... to agree with latest version of the MSC	2019-10-09 17:54:03 -04:00
Andrew Morgan	ea7d938bca	Remove unused public room list timeout param (#6179) ... * Remove unused public room list timeout param * Add changelog	2019-10-08 13:51:25 +01:00
Erik Johnston	2cb7466993	Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure ... Don't regenerate numeric user ID if registration fails.	2019-10-07 16:56:23 +01:00
Richard van der Hoff	276ae5c63e	add some logging to the rooms stats updates, to try to track down a flaky test (#6167)	2019-10-07 14:41:39 +01:00
Brendan Abolivier	ae0b78cb1f	Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked ... Don't 500 when trying to exchange a revoked 3PID invite	2019-10-04 12:09:05 +01:00
Brendan Abolivier	21d51ab598	Typo	2019-10-04 11:21:24 +01:00
Brendan Abolivier	4676732ca0	Lint	2019-10-04 11:18:28 +01:00
Brendan Abolivier	81d51ce48b	Incorporate review	2019-10-04 11:16:19 +01:00
Erik Johnston	ab8a64772b	Remove unused variable	2019-10-03 17:42:32 +01:00
Erik Johnston	693156aaf4	Don't regenerate numeric user ID if registration fails. ... This causes huge amounts of DB IO if registrations start to fail e.g. because the DB is struggling with IO.	2019-10-03 17:33:54 +01:00
Brendan Abolivier	6527fa18c1	Add test case	2019-10-03 11:24:36 +01:00
Erik Johnston	03cf4385e0	Fix public room list pagination. ... We incorrectly used `room_id` as to bound the result set, even though we order by `joined_members, room_id`, leading to incorrect results after pagination.	2019-10-02 15:11:17 +01:00
Erik Johnston	a5166e4d5f	Land improved room list based on room stats (#6019) ... Use room_stats and room_state for room directory search	2019-10-02 14:08:35 +01:00
Brendan Abolivier	972c9f65d7	Lint	2019-10-02 12:17:46 +01:00
Andrew Morgan	2a1470cd05	Fix yields and copy instead of move push rules on room upgrade (#6144) ... Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room. For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first. Also fix some missing yields that probably broke things quite a bit.	2019-10-02 12:04:22 +01:00
Brendan Abolivier	5705ecaec6	Don't 500 code when trying to exchange a revoked 3PID invite ... While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state. When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events. When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it). This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.	2019-10-02 11:19:43 +01:00
Brendan Abolivier	3e42d47a5a	Incorporate review	2019-09-27 16:15:01 +01:00
Brendan Abolivier	af92110c46	Update synapse/handlers/deactivate_account.py ... Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-27 16:12:15 +01:00
Brendan Abolivier	bbe2a0f339	Update synapse/handlers/deactivate_account.py ... Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-27 16:10:36 +01:00
Brendan Abolivier	873fe7883c	Lint	2019-09-27 15:21:03 +01:00
Brendan Abolivier	72a2708ac6	Fixup and add some logging	2019-09-27 15:13:39 +01:00
Brendan Abolivier	5257a2fb1c	Reject pending invites on deactivation	2019-09-27 14:49:53 +01:00
Andrew Morgan	8c27bc8b60	Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978) ... Just to have all the methods that make calls to identity services in one place.	2019-09-27 10:36:20 +01:00
Neil Johnson	034db2ba21	Fix dummy event insertion consent bug (#6053) ... Fixes #5905	2019-09-26 11:47:53 +01:00
Neil Johnson	77dc7093a7	Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. (#6104) ... Fixes #6103	2019-09-25 12:29:35 +01:00
Richard van der Hoff	990928abde	Stop advertising unsupported flows for registration (#6107) ... If email or msisdn verification aren't supported, let's stop advertising them for registration. Fixes #6100.	2019-09-25 12:10:26 +01:00
Richard van der Hoff	2cd98812ba	Refactor the user-interactive auth handling (#6105) ... Pull the checkers out to their own classes, rather than having them lost in a massive 1000-line class which does everything. This is also preparation for some more intelligent advertising of flows, as per #6100	2019-09-25 11:33:03 +01:00
Hubert Chathi	c3635c9459	make isort happy	2019-09-24 16:21:03 -04:00
Hubert Chathi	f4b6d43ec3	add some comments	2019-09-24 16:19:54 -04:00
Hubert Chathi	39864f45ec	drop some logger lines to debug	2019-09-24 15:26:45 -04:00
Hubert Chathi	26113fb7de	make changes based on PR feedback	2019-09-24 14:12:20 -04:00
Richard van der Hoff	9b7c4f4078	Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work	2019-09-24 17:03:50 +01:00
J. Ryan Stinnett	40fb00f5b7	Add sid to next_link for email validation (#6097)	2019-09-24 14:39:50 +01:00
Richard van der Hoff	ed8b92f0d2	Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work	2019-09-24 12:57:32 +01:00
Andrew Morgan	50776261e1	Add submit_url response parameter to msisdn /requestToken (#6079) ... Second part of solving #6076 Fixes #6076 We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.	2019-09-23 21:21:03 +01:00
Andrew Morgan	e08ea43463	Use the federation blacklist for requests to untrusted Identity Servers (#6000) ... Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses. Fixes #5935	2019-09-23 20:23:20 +01:00
Andrew Morgan	2c99c63453	Add POST submit_token endpoint for MSISDN (#6078) ... First part of solving #6076	2019-09-23 17:49:00 +01:00
Andrew Morgan	30af161af2	Implement MSC2290 (#6043) ... Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away. This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints. Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.	2019-09-23 16:50:27 +01:00
Andrew Morgan	885a4726b7	Return timeout error to user for identity server calls (#6073)	2019-09-23 14:37:23 +01:00
Andrew Morgan	df3401a71d	Allow HS to send emails when adding an email to the HS (#6042)	2019-09-20 15:21:30 +01:00
Richard van der Hoff	366dc7d2d4	Merge commit '33757bad1' into rav/saml_mapping_work	2019-09-20 11:21:40 +01:00
Richard van der Hoff	33757bad19	More better logging	2019-09-20 11:20:02 +01:00
Richard van der Hoff	b65327ff66	Merge branch 'develop' into rav/saml_mapping_work	2019-09-19 18:13:31 +01:00
Richard van der Hoff	7423fade92	better logging	2019-09-19 17:16:50 +01:00
Andrew Morgan	6670bd4072	v2 3PID Invites (part of MSC2140) (#5979) ... 3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is. These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens). Requires: ~~#5976~~	2019-09-17 18:05:13 +01:00
Erik Johnston	70c52821ce	Fix race condition in room stats. (#6029) ... Broke in #5971 Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)	2019-09-17 12:41:23 +01:00
Richard van der Hoff	a8ac40445c	Record mappings from saml users in an external table ... We want to assign unique mxids to saml users based on an incrementing suffix. For that to work, we need to record the allocated mxid in a separate table.	2019-09-13 16:01:46 +01:00
Andrew Morgan	9fc71dc5ee	Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976) ... This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted. Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140). Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.	2019-09-11 16:02:42 +01:00
Erik Johnston	cbcbfe64a2	Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction ... Allow use of different ratelimits for admin redactions.	2019-09-11 15:39:38 +01:00
Andrew Morgan	9c555f37e3	Add note about extra arg to send_membership_event, remove arg in remote_reject_invite (#6009) ... Some small fixes to `room_member.py` found while doing other PRs. 1. Add requester to the base `_remote_reject_invite` method. 2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.	2019-09-11 14:23:24 +01:00
Erik Johnston	57dd41a45b	Fix comments ... Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-11 13:54:50 +01:00
Andrew Morgan	3505ffcda7	Fix existing v2 identity server calls (MSC2140) (#6013) ... Two things I missed while implementing [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R80). 1. Access tokens should be provided to the identity server as `access_token`, not `id_access_token`, even though the homeserver may accept the tokens as `id_access_token`. 2. Access tokens must be sent to the identity server in a query parameter, the JSON body is not allowed. We now send the access token as part of an `Authorization: ...` header, which fixes both things. The breaking code was added in https://github.com/matrix-org/synapse/pull/5892 Sytest PR: https://github.com/matrix-org/sytest/pull/697	2019-09-11 11:59:45 +01:00
Erik Johnston	c64c3bb4c5	Fix how we check for self redaction	2019-09-11 11:18:04 +01:00
Erik Johnston	54ce81c86d	Allow use of different ratelimits for admin redactions. ... This is useful to allow room admins to quickly deal with a large number of abusive messages.	2019-09-11 10:46:38 +01:00
Andrew Morgan	cd17a2085e	Remove origin parameter from add_display_name_to_third_party_invite and add params to docstring (#6010) ... Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument. Since it's not used anywhere in the handler function though, we should remove it from the function arguments.	2019-09-11 10:37:17 +01:00
Erik Johnston	5e9b05d7da	Merge pull request #6011 from matrix-org/anoa/fix_3pid_validation ... Use account_threepid_delegate for 3pid validation	2019-09-10 18:15:07 +01:00
Andrew Morgan	b5833a2abf	Add changelog	2019-09-10 17:56:10 +01:00
Andrew Morgan	60d3c57bd0	Use account_threepid_delegate for 3pid validation	2019-09-10 17:56:10 +01:00
Jason Robinson	be618e0551	Only count real users when checking for auto-creation of auto-join room ... Previously if the first registered user was a "support" or "bot" user, when the first real user registers, the auto-join rooms were not created. Fix to exclude non-real (ie users with a special user type) users when counting how many users there are to determine whether we should auto-create a room. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 14:48:08 +03:00
Hubert Chathi	d3f2fbcfe5	add function docs	2019-09-07 14:13:18 -04:00
Hubert Chathi	8e86f5b65c	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-09-07 13:20:34 -04:00
Hubert Chathi	0d61d1d735	Merge branch 'develop' into cross-signing_sig_upload	2019-09-07 13:14:45 -04:00
Hubert Chathi	415d0a00e0	run black	2019-09-06 16:46:45 -04:00
Hubert Chathi	561cbba057	split out signature processing into separate functions	2019-09-06 16:44:24 -04:00
Andrew Morgan	78801e7f9e	Ensure a sid parameter is passed to bind_threepid (#5995) ... `sid` is required to be part of `three_pid_creds`. We were 500'ing if it wasn't provided instead of returning `M_MISSING_PARAM`.	2019-09-06 15:36:50 +01:00
Erik Johnston	142c9325c2	Correctly handle non-bool m.federate flag	2019-09-06 14:21:06 +01:00
Andrew Morgan	0c0b82b6d1	Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987) ... This is a combination of a few different PRs, finally all being merged into `develop`: * #5875 * #5876 * #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future) * #5835 * #5969 * #5940 Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged. UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.	2019-09-06 11:35:28 +01:00
Hubert Chathi	369462da74	avoid modifying input parameter	2019-09-05 17:03:31 -04:00
Erik Johnston	1a6ae33309	Merge pull request #5984 from matrix-org/joriks/opentracing_link_send_to_edu_contexts ... Link the send loop with the edus contexts	2019-09-05 15:22:24 +01:00
Jorik Schellekens	1d65292e94	Link the send loop with the edus contexts ... The contexts were being filtered too early so the send loop wasn't being linked to them unless the destination was whitelisted.	2019-09-05 14:42:37 +01:00
Andrew Morgan	a0d294c306	Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (#5892)	2019-09-05 14:31:22 +01:00
Andrew Morgan	90d17a3d28	Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (#5980) ... Implements `POST /_matrix/client/r0/account/3pid/unbind` from [MSC2140](https://github.com/matrix-org/matrix-doc/blob/dbkr/tos_2/proposals/2140-terms-of-service-2.md#post-_matrixclientr0account3pidunbind).	2019-09-05 14:00:30 +01:00
Hubert Chathi	c8dc740a94	update with newer coding style	2019-09-04 22:30:45 -04:00
Hubert Chathi	7d6c70fc7a	make black happy	2019-09-04 22:04:12 -04:00
Hubert Chathi	ac4746ac4b	allow uploading signatures of master key signed by devices	2019-09-04 22:03:41 -04:00
Hubert Chathi	4bb4544784	implement device signature uploading/fetching	2019-09-04 20:02:56 -04:00
Hubert Chathi	faf72a4c40	Merge branch 'develop' into cross-signing_keys	2019-09-04 19:12:29 -04:00
Andrew Morgan	b736c6cd3a	Remove bind_email and bind_msisdn (#5964) ... Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107).	2019-09-04 18:24:23 +01:00
Erik Johnston	6e834e94fc	Fix and refactor room and user stats (#5971) ... Previously the stats were not being correctly populated.	2019-09-04 13:04:27 +01:00
Andrew Morgan	ea128a3e8e	code cleanups	2019-09-03 21:05:06 +01:00
Andrew Morgan	6b6086b8bf	Fix docstring	2019-09-03 20:00:09 +01:00
Andrew Morgan	a98b8583c6	Remove unnecessary variable declaration	2019-09-03 19:58:51 +01:00
Matthew Hodgson	8401bcd206	fix typo	2019-09-03 12:44:14 +01:00
Andrew Morgan	2a44782666	Remove double return statements (#5962) ... Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.	2019-09-03 11:42:45 +01:00
Jorik Schellekens	a90d16dabc	Opentrace device lists (#5853) ... Trace device list changes.	2019-09-03 10:21:30 +01:00
Andrew Morgan	36f34e6f3d	Remove unused methods from c/s api v1 in register.py (#5963) ... These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.	2019-09-02 18:29:21 +01:00
Andrew Morgan	4548d1f87e	Remove unnecessary parentheses around return statements (#5931) ... Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)	2019-08-30 16:28:26 +01:00
Andrew Morgan	3057095a5d	Revert "Use the v2 lookup API for 3PID invites (#5897)" (#5937) ... This reverts commit 71fc04069a. This broke 3PID invites as #5892 was required for it to work correctly.	2019-08-30 12:00:20 +01:00
Hubert Chathi	e3d3fbf63f	Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys	2019-08-28 17:36:46 -07:00
Hubert Chathi	96bda56370	black	2019-08-28 17:18:40 -07:00
Hubert Chathi	3b0b22cb05	use stream ID generator instead of timestamp	2019-08-28 17:17:21 -07:00
Will Hunt	c8fa620d7a	Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent ... Exempt support users from consent	2019-08-28 16:31:40 +01:00
Andrew Morgan	71fc04069a	Use the v2 lookup API for 3PID invites (#5897) ... Fixes https://github.com/matrix-org/synapse/issues/5861 Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134). Currently this is only used for 3PID invites. Sytest PR: https://github.com/matrix-org/sytest/pull/679	2019-08-28 14:59:26 +02:00
Amber Brown	7dc398586c	Implement a structured logging output system. (#5680)	2019-08-28 21:18:53 +10:00
reivilibre	a3f0635686	Merge pull request #5914 from matrix-org/rei/admin_getadmin ... Add GET method to admin API /users/@user:dom/admin	2019-08-28 09:44:22 +01:00
reivilibre	7ccc251415	Merge pull request #5859 from matrix-org/rei/msc2197 ... MSC2197 Search Filters over Federation	2019-08-28 09:00:21 +01:00
Olivier Wilkinson (reivilibre)	c88a119259	Add GET method to admin API /users/@user:dom/admin ... Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-27 13:12:27 +01:00
Richard van der Hoff	ccb15a5bbe	Merge pull request #5906 from matrix-org/neilj/increase_display_name_limit ... Increase profile display name limit	2019-08-27 11:52:59 +01:00
reivilibre	1a7e6eb633	Add Admin API capability to set adminship of a user (#5878) ... Admin API: Set adminship of a user	2019-08-27 10:14:00 +01:00
Neil Johnson	27d3fc421a	Increase max display name limit	2019-08-24 22:33:43 +01:00
Will Hunt	c998f25006	Apply suggestions from code review ... Co-Authored-By: Erik Johnston <erik@matrix.org>	2019-08-23 10:28:54 +01:00
Half-Shot	9ba32f6573	Exempt bot users	2019-08-23 09:56:31 +01:00
Half-Shot	886eceba3e	Return user_type in get_user_by_id	2019-08-23 09:14:52 +01:00
Jorik Schellekens	8767b63a82	Propagate opentracing contexts through EDUs (#5852) ... Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-08-22 18:21:10 +01:00
Brendan Abolivier	1c5b8c6222	Revert "Add "require_consent" parameter for registration" ... This reverts commit 3320aaab3a.	2019-08-22 14:47:34 +01:00
Half-Shot	3320aaab3a	Add "require_consent" parameter for registration	2019-08-22 14:21:54 +01:00
Jorik Schellekens	9a6f2be572	Opentrace e2e keys (#5855) ... Add opentracing tags and logs for e2e keys	2019-08-22 11:28:12 +01:00
Richard van der Hoff	119aa31b10	Servlet to purge old rooms (#5845)	2019-08-22 10:42:59 +01:00
Hubert Chathi	814f253f1b	make isort happy	2019-08-21 13:22:15 -07:00
Hubert Chathi	7c3abc6572	apply PR review suggestions	2019-08-21 13:19:35 -07:00
Erik Johnston	d514dac0b2	Merge pull request #5860 from matrix-org/erikj/update_5704_comments ... Remove logging for #5407 and update comments	2019-08-19 10:20:59 +01:00
Erik Johnston	748aa38378	Remove logging for #5407 and update comments	2019-08-15 12:02:18 +01:00
Olivier Wilkinson (reivilibre)	6fadb560fc	Support MSC2197 outbound with unstable prefix ... Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-15 10:59:37 +01:00
Michael Telatynski	baee288fb4	Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633)	2019-08-15 09:45:57 +01:00
Erik Johnston	6881f21f3e	Handle TimelineBatch being limited and empty. ... This hopefully addresses #5407 by gracefully handling an empty but limited TimelineBatch. We also add some logging to figure out how this is happening.	2019-08-06 12:59:00 +01:00
Hubert Chathi	fac1cdc562	make changes from PR review	2019-08-01 21:51:19 -04:00
Hubert Chathi	336c546d6a	Merge branch 'cross-signing_hidden' into cross-signing_keys	2019-08-01 16:31:40 -04:00
Brendan Abolivier	8ed9e63432	Account validity: allow defining HTML templates to serve the us… (#5807) ... Account validity: allow defining HTML templates to serve the user on account renewal attempt	2019-08-01 16:09:25 +02:00
Erik Johnston	58af30a6c7	Merge pull request #5802 from matrix-org/erikj/deny_redacting_different_room ... Deny redaction of events in a different room.	2019-08-01 13:14:46 +01:00
Erik Johnston	0f632f3a57	Merge pull request #5790 from matrix-org/erikj/groups_request_errors ... Handle RequestSendFailed exception correctly in more places.	2019-08-01 13:14:08 +01:00
Brendan Abolivier	a4a9ded4d0	Allow defining HTML templates to serve the user on account renewal	2019-08-01 11:59:27 +02:00
Erik Johnston	cf89266b98	Deny redaction of events in a different room. ... We already correctly filter out such redactions, but we should also deny them over the CS API.	2019-07-31 16:12:27 +01:00
Andrew Morgan	58a755cdc3	Remove duplicate return statement	2019-07-31 13:24:51 +01:00
Erik Johnston	b4d5ff0af7	Don't log as exception when failing durig backfill	2019-07-30 13:19:22 +01:00
Erik Johnston	1ec7d656dd	Unwrap error	2019-07-30 13:09:02 +01:00
Erik Johnston	458e51df7a	Fix error handling when fetching remote device keys	2019-07-30 13:07:02 +01:00
Richard van der Hoff	8c97f6414c	Remove non-functional 'expire_access_token' setting (#5782) ... The `expire_access_token` didn't do what it sounded like it should do. What it actually did was make Synapse enforce the 'time' caveat on macaroons used as access tokens, but since our access token macaroons never contained such a caveat, it was always a no-op. (The code to add 'time' caveats was removed back in v0.18.5, in #1656)	2019-07-30 08:25:02 +01:00
Amber Brown	865077f1d1	Room Complexity Client Implementation (#5783)	2019-07-30 02:47:27 +10:00
Erik Johnston	aecae8f397	Correctly handle errors doing requests to group servers	2019-07-29 17:21:57 +01:00
Amber Brown	97a8b4caf7	Move some timeout checking logs to DEBUG #5785	2019-07-30 02:02:18 +10:00
Jorik Schellekens	85b0bd8fe0	Update the device list cache when keys/query is called (#5693)	2019-07-29 16:34:44 +01:00
Richard van der Hoff	1a93daf353	Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch ... Log when we receive a /make_* request from a different origin	2019-07-26 12:38:37 +01:00
Richard van der Hoff	d1020653fc	Log when we receive a /make_* request from a different origin	2019-07-26 10:08:22 +01:00
Hubert Chathi	c659b9f94f	allow uploading keys for cross-signing	2019-07-25 11:08:24 -04:00
Erik Johnston	2276936bac	Merge pull request #5743 from matrix-org/erikj/log_origin_receipts_mismatch ... Log when we receive receipt from a different origin	2019-07-24 13:27:57 +01:00
Richard van der Hoff	f30a71a67b	Stop trying to fetch events with event_id=None. (#5753) ... `None` is not a valid event id, so queuing up a database fetch for it seems like a silly thing to do. I considered making `get_event` return `None` if `event_id is None`, but then its interaction with `allow_none` seemed uninituitive, and strong typing ftw.	2019-07-24 13:16:18 +01:00
Amber Brown	4806651744	Replace returnValue with return (#5736)	2019-07-23 23:00:55 +10:00
Erik Johnston	3db1377b26	Log when we receive receipt from a different origin	2019-07-23 13:31:03 +01:00
Richard van der Hoff	9c70a02a9c	Ignore redactions of m.room.create events (#5701)	2019-07-17 19:08:02 +01:00
Andrew Morgan	18c516698e	Return a different error from Invalid Password when a user is deactivated (#5674) ... Return `This account has been deactivated` instead of `Invalid password` when a user is deactivated.	2019-07-15 11:45:29 +01:00
Erik Johnston	d86321300a	Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data ... Add basic function to get all data for a user out of synapse	2019-07-15 10:04:02 +01:00
Richard van der Hoff	5f158ec039	Implement access token expiry (#5660) ... Record how long an access token is valid for, and raise a soft-logout once it expires.	2019-07-12 17:26:02 +01:00
Richard van der Hoff	1890cfcf82	Inline issue_access_token (#5659) ... this is only used in one place, so it's clearer if we inline it and reduce the API surface. Also, fixes a buglet where we would create an access token even if we were about to block the user (we would never return the AT, so the user could never use it, but it was still created and added to the db.)	2019-07-11 04:10:07 +10:00
Brendan Abolivier	8ab3444fdf	Merge pull request #5658 from matrix-org/babolivier/is-json ... Send 3PID bind requests as JSON data	2019-07-10 17:01:26 +01:00
Richard van der Hoff	953dbb7980	Remove access-token support from RegistrationStore.register (#5642) ... The 'token' param is no longer used anywhere except the tests, so let's kill that off too.	2019-07-10 16:26:49 +01:00
Brendan Abolivier	b2a2e96ea6	Typo	2019-07-10 15:56:21 +01:00
Brendan Abolivier	f77e997619	Send 3PID bind requests as JSON data	2019-07-10 15:46:42 +01:00
Brendan Abolivier	65434da75d	Merge pull request #5638 from matrix-org/babolivier/invite-json ... Use JSON when querying the IS's /store-invite endpoint	2019-07-09 18:48:38 +01:00
Hubert Chathi	7b3bc755a3	remove unused and unnecessary check for FederationDeniedError (#5645) ... FederationDeniedError is a subclass of SynapseError, which is a subclass of CodeMessageException, so if e is a FederationDeniedError, then this check for FederationDeniedError will never be reached since it will be caught by the check for CodeMessageException above. The check for CodeMessageException does almost the same thing as this check (since FederationDeniedError initialises with code=403 and msg="Federation denied with %s."), so may as well just keep allowing it to handle this case.	2019-07-09 18:37:39 +01:00
Brendan Abolivier	af67c7c1de	Merge pull request #5644 from matrix-org/babolivier/profile-allow-self ... Allow newly-registered users to lookup their own profiles	2019-07-09 10:25:40 +01:00
Richard van der Hoff	824707383b	Remove access-token support from RegistrationHandler.register (#5641) ... Nothing uses this now, so we can remove the dead code, and clean up the API. Since we're changing the shape of the return value anyway, we take the opportunity to give the method a better name.	2019-07-08 19:01:08 +01:00
Brendan Abolivier	7556851665	Allow newly-registered users to lookup their own profiles ... When a user creates an account and the 'require_auth_for_profile_requests' config flag is set, and a client that performed the registration wants to lookup the newly-created profile, the request will be denied because the user doesn't share a room with themselves yet.	2019-07-08 17:31:00 +01:00
Richard van der Hoff	b70e080b59	Better logging for auto-join. (#5643) ... It was pretty unclear what was going on, so I've added a couple of log lines.	2019-07-08 17:14:51 +01:00
Brendan Abolivier	57eacee4f4	Merge branch 'develop' into babolivier/invite-json	2019-07-08 15:49:23 +01:00
Richard van der Hoff	1af2fcd492	Move get_or_create_user to test code (#5628) ... This is only used in tests, so...	2019-07-08 23:52:26 +10:00
Brendan Abolivier	f05c7d62bc	Lint	2019-07-08 14:29:27 +01:00
Brendan Abolivier	1a807dfe68	Use application/json when querying the IS's /store-invite endpoint	2019-07-08 14:19:39 +01:00
Richard van der Hoff	80cc82a445	Remove support for invite_3pid_guest. (#5625) ... This has never been documented, and I'm not sure it's ever been used outside sytest. It's quite a lot of poorly-maintained code, so I'd like to get rid of it. For now I haven't removed the database table; I suggest we leave that for a future clearout.	2019-07-05 16:47:58 +01:00
Erik Johnston	b4f5416dd9	pep8	2019-07-05 14:41:29 +01:00
Erik Johnston	eadb13d2e9	Remove FileExfiltrationWriter	2019-07-05 14:15:00 +01:00
Erik Johnston	7f0d8e4288	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data	2019-07-05 14:08:21 +01:00
Erik Johnston	9ccea16d45	Assume key existence. Update docstrings	2019-07-05 14:07:56 +01:00
Erik Johnston	c061d4f237	Fixup from review comments.	2019-07-04 11:41:06 +01:00
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606)	2019-07-04 00:07:04 +10:00
Erik Johnston	d0b849c86d	Apply comment fixups from code review ... Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-07-03 15:03:38 +01:00
Richard van der Hoff	91753cae59	Fix a number of "Starting txn from sentinel context" warnings (#5605) ... Fixes #5602, #5603	2019-07-03 09:31:27 +01:00
Erik Johnston	8ee69f299c	Add basic function to get all data for a user out of synapse	2019-07-02 12:09:04 +01:00
Richard van der Hoff	6eecb6e500	Complete the SAML2 implementation (#5422) ... * SAML2 Improvements and redirect stuff Signed-off-by: Alexander Trost <galexrt@googlemail.com> * Code cleanups and simplifications. Also: share the saml client between redirect and response handlers. * changelog * Revert redundant changes to static js * Move all the saml stuff out to a centralised handler * Add support for tracking SAML2 sessions. This allows us to correctly handle `allow_unsolicited: False`. * update sample config * cleanups * update sample config * rename BaseSSORedirectServlet for consistency * Address review comments	2019-07-02 11:18:11 +01:00
Richard van der Hoff	b4fd86a9b4	Merge branch 'develop' into rav/saml2_client	2019-07-01 14:21:03 +01:00
Richard van der Hoff	3bcb13edd0	Address review comments	2019-07-01 12:13:22 +01:00
Erik Johnston	04196a4dae	Merge pull request #5507 from matrix-org/erikj/presence_sync_tighloop ... Fix sync tightloop bug.	2019-07-01 11:43:10 +01:00
Erik Johnston	915280f1ed	Fixup comment	2019-07-01 10:22:42 +01:00
Brendan Abolivier	c7ff297dde	Merge pull request #5576 from matrix-org/babolivier/3pid-invite-ratelimit ... Don't update the ratelimiter before sending a 3PID invite	2019-06-28 17:43:48 +01:00
Brendan Abolivier	15d9fc31bd	Only ratelimit when sending the email ... If we do the opposite, an event can arrive after or while sending the email and the 3PID invite event will get ratelimited.	2019-06-28 16:04:05 +01:00
Brendan Abolivier	01d0f8e701	Don't update the ratelimiter before sending a 3PID invite ... This would cause emails being sent, but Synapse responding with a 429 when creating the event. The client would then retry, and with bad timing the same scenario would happen again. Some testing I did ended up sending me 10 emails for one single invite because of this.	2019-06-28 15:22:16 +01:00
Daniel Hoffend	9646a593ac	Added possibilty to disable local password authentication (#5092) ... Signed-off-by: Daniel Hoffend <dh@dotlan.net>	2019-06-27 18:37:29 +01:00
Erik Johnston	729f5a4fb6	Review comments	2019-06-27 16:06:23 +01:00
Richard van der Hoff	28db0ae537	cleanups	2019-06-27 00:37:41 +01:00
Richard van der Hoff	36f4953dec	Add support for tracking SAML2 sessions. ... This allows us to correctly handle `allow_unsolicited: False`.	2019-06-26 23:50:55 +01:00
Richard van der Hoff	3705322103	Move all the saml stuff out to a centralised handler	2019-06-26 22:52:02 +01:00
Erik Johnston	8624db3194	Refactor and comment sync device list code	2019-06-26 19:39:49 +01:00
Erik Johnston	806a06daf2	Rename get_users_whose_devices_changed	2019-06-26 19:39:19 +01:00
Erik Johnston	a2f6d31a63	Refactor get_user_ids_changed to pull less from DB ... When a client asks for users whose devices have changed since a token we used to pull *all* users from the database since the token, which could easily be thousands of rows for old tokens. This PR changes this to only check for changes for users the client is actually interested in. Fixes #5553	2019-06-26 12:03:44 +01:00
Andrew Morgan	ef8c62758c	Prevent multiple upgrades on the same room at once (#5051) ... Closes #4583 Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time. Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.	2019-06-25 14:19:21 +01:00
Erik Johnston	25433f212d	Merge pull request #5531 from matrix-org/erikj/workers_pagination_token ... Fix /messages on workers when no from param specified.	2019-06-24 15:30:10 +01:00
Richard van der Hoff	edea4bb5be	Allow configuration of the path used for ACME account keys. ... Because sticking it in the same place as the config isn't necessarily the right thing to do.	2019-06-24 13:51:22 +01:00
Richard van der Hoff	21bf4318b5	Factor acme bits out to a separate file (#5521) ... This makes some of the conditional-import hoop-jumping easier.	2019-06-24 11:33:56 +01:00
Erik Johnston	dddf20e8e1	Fix /messages on workers when no from param specified. ... If no `from` param is specified we calculate and use the "current token" that inlcuded typing, presence, etc. These are unused during pagination and are not available on workers, so we simply don't calculate them.	2019-06-24 10:06:51 +01:00
Erik Johnston	8181e290a9	Fix sync tightloop bug. ... If, for some reason, presence updates take a while to persist then it can trigger clients to tightloop calling `/sync` due to the presence handler returning updates but not advancing the stream token. Fixes #5503.	2019-06-21 11:10:27 +01:00
Amber Brown	32e7c9e7f2	Run Black. (#5482)	2019-06-20 19:32:02 +10:00
Erik Johnston	7dcf984075	Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_error ... Handle the case of `get_missing_events` failing	2019-06-19 13:20:09 +01:00
Erik Johnston	e0be8d7016	Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events ... Add experimental option to reduce extremities.	2019-06-19 13:19:18 +01:00
Erik Johnston	554609288b	Run as background process and fix comments	2019-06-19 11:33:03 +01:00
David Baker	f2d2ae03da	Add some logging to 3pid invite sig verification (#5015) ... I had to add quite a lot of logging to diagnose a problem with 3pid invites - we only logged the one failure which isn't all that informative. NB. I'm not convinced the logic of this loop is right: I think it should just accept a single valid signature from a trusted source rather than fail if *any* signature is invalid. Also it should probably not skip the rest of middle loop if a check fails? However, I'm deliberately not changing the logic here.	2019-06-18 22:51:24 +01:00
Erik Johnston	2b20d0fb59	Fix logline	2019-06-18 16:12:53 +01:00
Erik Johnston	19b80fe68a	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_get_missing_events_error	2019-06-18 16:11:43 +01:00
Erik Johnston	b42f90470f	Add experimental option to reduce extremities. ... Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.	2019-06-18 15:02:18 +01:00
Brendan Abolivier	112cf5a73a	Add third party rules hook for 3PID invites	2019-06-17 17:39:09 +01:00
Erik Johnston	187d2837a9	Add third party rules hook into create room	2019-06-17 16:41:19 +01:00
Erik Johnston	6840ebeef8	Merge pull request #5385 from matrix-org/erikj/reduce_http_exceptions ... Handle HttpResponseException when using federation client.	2019-06-17 13:54:47 +01:00
Erik Johnston	a9dab970b8	Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_state ... Fix 3PID invite room state over federation.	2019-06-17 10:18:28 +01:00
Brendan Abolivier	f12e1f029c	Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules ... Allow server admins to define implementations of extra rules for allowing or denying incoming events	2019-06-14 19:37:59 +01:00
Brendan Abolivier	f874b16b2e	Add plugin APIs for implementations of custom event rules.	2019-06-14 18:16:03 +01:00
Erik Johnston	304a1376c2	Fix 3PID invite room state over federation. ... Fixes that when a user exchanges a 3PID invite for a proper invite over federation it does not include the `invite_room_state` key. This was due to synapse incorrectly sending out two invite requests.	2019-06-14 16:19:11 +01:00
Brendan Abolivier	6d56a694f4	Don't send renewal emails to deactivated users	2019-06-14 15:05:56 +01:00
Brendan Abolivier	d0530382ee	Track deactivated accounts in the database (#5378)	2019-06-14 13:18:24 +01:00
Erik Johnston	49e01e5710	Fix defaults on checking threepids	2019-06-10 23:09:31 +01:00
Erik Johnston	a46ef1e3a4	Handle HttpResponseException when using federation client. ... Otherwise we just log exceptions everywhere.	2019-06-07 10:29:35 +01:00
Andrew Morgan	3719680ee4	Add ability to perform password reset via email without trusting the identity server (#5377) ... Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368	2019-06-06 17:34:07 +01:00
Richard van der Hoff	71063a69b8	Fix missing logcontext for PresenceHandler.on_shutdown. (#5369) ... Fixes some warnings, and a scary-looking stacktrace when sytest kills the process.	2019-06-06 14:45:17 +01:00
Brendan Abolivier	8f06344e11	Merge pull request #5089 from dnaf/m-heroes-empty-room-name ... Make /sync return heroes if room name or canonical alias are empty	2019-06-06 11:18:13 +01:00
Brendan Abolivier	64fa928792	Simplify condition	2019-06-06 10:34:12 +01:00
Brendan Abolivier	b9c43c8463	Merge pull request #5355 from matrix-org/babolivier/heroes_left_members ... Include left members in room summaries' heroes	2019-06-06 10:05:27 +01:00
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352) ... * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	2019-06-06 00:45:46 +10:00
Brendan Abolivier	0a2f522644	Simplify condition	2019-06-05 14:02:29 +01:00
Brendan Abolivier	40596aec0e	Merge branch 'develop' into m-heroes-empty-room-name	2019-06-05 13:38:01 +01:00
Brendan Abolivier	dbbaf25dd3	Do user_id != me checks before deciding whether we should pick heroes from the joined members or the parted ones	2019-06-05 11:50:27 +01:00
Erik Johnston	bc3d6b918b	Add logging when request fails and clarify we ignore errors.	2019-06-05 11:37:51 +01:00
Erik Johnston	220a733d73	Fix handling of failures when calling /event_auth. ... When processing an incoming event over federation, we may try and resolve any unexpected differences in auth events. This is a non-essential process and so should not stop the processing of the event if it fails (e.g. due to the remote disappearing or not implementing the necessary endpoints). Fixes #3330	2019-06-03 09:56:45 +01:00
Richard van der Hoff	d828d1dc57	Merge pull request #5309 from matrix-org/rav/limit_displayname_length ... Limit displaynames and avatar URLs	2019-06-01 11:34:50 +01:00
Richard van der Hoff	d16c6375fe	Limit displaynames and avatar URLs ... These end up in join events everywhere, so let's limit them. Fixes #5079	2019-06-01 10:44:36 +01:00
Richard van der Hoff	5726378ece	Fix "db txn 'update_presence' from sentinel context" log messages (#5275) ... Fixes #4414.	2019-05-28 21:20:11 +01:00
Erik Johnston	d16f5574b6	Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events ... Don't bundle aggregations with events in /sync or /events or state queries	2019-05-24 10:36:31 +01:00
Erik Johnston	4cb577c23f	Don't bundle aggs for /state and /members etc APIs	2019-05-24 09:52:33 +01:00
Andrew Morgan	6368150a74	Add config option for setting homeserver's default room version (#5223) ... Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present. That hardcoded value is now located in the server.py config file.	2019-05-23 15:00:20 +01:00
Richard van der Hoff	85d1e03b9d	Simplifications and comments in do_auth (#5227) ... I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.	2019-05-23 11:17:42 +01:00
Amber Brown	4a30e4acb4	Room Statistics (#4338)	2019-05-21 11:36:50 -05:00
Erik Johnston	9526aa96a6	Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions ... Block attempts to annotate the same event twice	2019-05-21 17:08:14 +01:00
Erik Johnston	44b8ba484e	Fix words	2019-05-21 16:51:45 +01:00
Richard van der Hoff	c4aef549ad	Exclude soft-failed events from fwd-extremity candidates. (#5146) ... When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.	2019-05-21 16:10:54 +01:00
Erik Johnston	de7672b78f	Don't bundle events in /sync or /events ... As we'll send down the annotations too anyway, so this just ends up confusing clients.	2019-05-21 13:54:09 +01:00
Erik Johnston	c7ec06e8a6	Block attempts to annotate the same event twice	2019-05-20 17:39:05 +01:00
ReidAnderson	3787133c9e	Limit UserIds to a length that fits in a state key (#5198)	2019-05-20 11:20:08 +01:00
bytepoets-blo	291e1eea5e	fix mapping of return values for get_or_register_3pid_guest (#5177) ... * fix mapping of return values for get_or_register_3pid_guest	2019-05-17 17:27:14 +01:00
Brendan Abolivier	cd32375846	Add option to disable per-room profiles	2019-05-16 14:34:28 +01:00
Matthew Hodgson	4a6d5de98c	Make /sync attempt to return device updates for both joined and invited users (#3484)	2019-05-16 13:23:43 +01:00
Amber Brown	f1e5b41388	Make all the rate limiting options more consistent (#5181)	2019-05-15 12:06:04 -05:00
Erik Johnston	b54b03f9e1	Allow client event serialization to be async	2019-05-14 11:58:01 +01:00
Matthew Hodgson	c0e0740bef	add options to require an access_token to GET /profile and /publicRooms on CS API (#5083) ... This commit adds two config options: * `restrict_public_rooms_to_local_users` Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API. * `require_auth_for_profile_requests` When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301. MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though. Groups have been intentionally omitted from this commit.	2019-05-08 18:26:56 +01:00
Brendan Abolivier	1473058b5e	Do checks on aliases for incoming m.room.aliases events (#5128) ... Follow-up to #5124 Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.	2019-05-08 17:01:30 +01:00
Erik Johnston	de655e669a	Merge pull request #5104 from matrix-org/erikj/ratelimit_3pid_invite ... Ratelimit 3pid invites	2019-05-07 10:12:49 +01:00
Erik Johnston	176f31c2e3	Rate limit early	2019-05-02 15:23:08 +01:00
Brendan Abolivier	84196cb231	Add some limitations to alias creation	2019-05-02 11:05:11 +01:00
Erik Johnston	d6118c5be6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ratelimit_3pid_invite	2019-04-26 18:14:23 +01:00
Erik Johnston	28a81ed62f	Ratelimit 3pid invites ... We do ratelimit sending the 3PID invite events, but that happens after spamming the identity server.	2019-04-26 18:06:25 +01:00
Richard van der Hoff	bd0d45ca69	Fix infinite loop in presence handler ... Fixes #5102	2019-04-26 11:14:49 +01:00
Katie Wolfe	7e07dc429f	Lint ... I probably should've just run autopep8 in the first place... Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:43:18 -04:00
Katie Wolfe	b3e5db402d	Clean up code ... Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:55 -04:00
Katie Wolfe	5d3ed79944	Show heroes if room name or canonical alias are empty ... Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:53 -04:00
Katie Wolfe	7f025eb425	Show heroes if room name or canonical alias are empty ... Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:29 -04:00
Erik Johnston	ca90336a69	Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration	2019-04-17 19:44:40 +01:00
Brendan Abolivier	eaf41a943b	Add management endpoints for account validity	2019-04-17 19:34:45 +01:00
Brendan Abolivier	91934025b9	Merge pull request #5047 from matrix-org/babolivier/account_expiration ... Send out emails with links to extend an account's validity period	2019-04-17 14:57:39 +01:00
Brendan Abolivier	20f0617e87	Send out emails with links to extend an account's validity period	2019-04-17 14:42:20 +01:00
Erik Johnston	c132c8e505	Handle the case of get_missing_events failing ... Currently if a call to `/get_missing_events` fails we log an exception and stop processing the top level event we received over federation. Instead let's try and handle it sensibly given it is a somewhat expected failure mode.	2019-04-10 10:39:54 +01:00
Brendan Abolivier	8e85493b0c	Add config option to block users from looking up 3PIDs (#5010)	2019-04-04 17:25:47 +01:00
Andrew Morgan	db265f0642	Prevent kicking users who aren't in the room (#4999) ... Prevent kick events from succeeding if the user is not currently in the room.	2019-04-04 13:05:51 +01:00
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998)	2019-04-03 14:32:20 +01:00
Neil Johnson	e8419554ff	Remove presence lists (#4989) ... Remove presence list support as per MSC 1819	2019-04-03 11:11:15 +01:00
Erik Johnston	8f549c1177	Merge pull request #4982 from matrix-org/erikj/msc1915 ... Implement MSC1915 - 3PID unbind APIs	2019-04-03 11:07:09 +01:00
Andrew Morgan	66e78700a2	Transfer related groups on room upgrade (#4990) ... Transfers the m.room.related_groups state event on room upgrade.	2019-04-02 17:15:24 +01:00
Erik Johnston	4ef5d17b96	Correctly handle id_server param	2019-04-02 11:20:09 +01:00
Erik Johnston	24232514bf	Remove threepid binding if id server returns 400/404/501	2019-04-02 11:20:09 +01:00
Erik Johnston	c75e2017f1	Fixup docstrings	2019-04-02 11:20:06 +01:00
Erik Johnston	057715aaa2	Allowing specifying IS to use in unbind API. ... By default the homeserver will use the identity server used during the binding of the 3PID to unbind the 3PID. However, we need to allow clients to explicitly ask the homeserver to unbind via a particular identity server, for the case where the 3PID was bound out of band from the homeserver. Implements MSC915.	2019-04-01 15:25:18 +01:00
Erik Johnston	9fbbc3d9e5	For unbind poke IS used during binding of 3PID ... This changes the behaviour from using the server specified trusted identity server to using the IS that used during the binding of the 3PID, if known. This is the behaviour specified by MSC1915.	2019-04-01 15:23:30 +01:00
Erik Johnston	1666c0696a	Track IS used to bind 3PIDs ... This will then be used to know which IS to default to when unbinding the threepid.	2019-04-01 15:23:01 +01:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969) ... Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Andrew Morgan	7a91b9d81c	Allow password providers to bind emails (#4947) ... This PR allows password provider modules to bind email addresses when a user is registering and is motivated by matrix-org/matrix-synapse-ldap3#58	2019-03-28 15:48:07 +00:00
Erik Johnston	4e5f0f7ca0	Use an assert	2019-03-28 14:05:05 +00:00
Erik Johnston	40e56997bc	Review comments	2019-03-28 13:48:41 +00:00
Erik Johnston	197fae1639	Use event streams to calculate presence ... Primarily this fixes a bug in the handling of remote users joining a room where the server sent out the presence for all local users in the room to all servers in the room. We also change to using the state delta stream, rather than the distributor, as it will make it easier to split processing out of the master process (as well as being more flexible). Finally, when sending presence states to newly joined servers we filter out old presence states to reduce the number sent. Initially we filter out states that are offline and have a last active more than a week ago, though this can be changed down the line. Fixes #3962	2019-03-27 13:41:36 +00:00
Andrew Morgan	bbd244c7b2	Support 3PID login in password providers (#4931) ... Adds a new method, check_3pid_auth, which gives password providers the chance to allow authentication with third-party identifiers such as email or msisdn.	2019-03-26 17:48:30 +00:00
Amber Brown	ac396a0d32	Refactor out state delta handling into its own class (#4917)	2019-03-25 20:37:08 +11:00
Erik Johnston	01e6b405be	Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms ... Deny peeking into rooms that have been blocked	2019-03-21 14:07:17 +00:00
Erik Johnston	3959858eaa	Merge pull request #4904 from matrix-org/erikj/fix_shutdown ... Fixup shutdown room API	2019-03-21 11:24:42 +00:00
Erik Johnston	536a266520	Deny peeking into rooms that have been blocked	2019-03-21 11:20:13 +00:00
Erik Johnston	09f991a63d	Merge pull request #4896 from matrix-org/erikj/disable_room_directory ... Add option to disable search room lists	2019-03-21 10:16:54 +00:00
Erik Johnston	cd62981a6a	Revert spurious delete	2019-03-20 17:51:27 +00:00
Erik Johnston	aa959a6c07	Use flags	2019-03-20 17:40:29 +00:00
Erik Johnston	7d47cc1305	Move requester check into assert_accepted_privacy_policy	2019-03-20 17:08:36 +00:00
Erik Johnston	74c46d81fa	Only require consent for events with an associated request ... There are a number of instances where a server or admin may puppet a user to join/leave rooms, which we don't want to fail if the user has not consented to the privacy policy. We fix this by adding a check to test if the requester has an associated access_token, which is used as a proxy to answer the question of whether the action is being done on behalf of a real request from the user.	2019-03-20 16:50:23 +00:00
Richard van der Hoff	a902d13180	Batch up outgoing read-receipts to reduce federation traffic. (#4890) ... Rate-limit outgoing read-receipts as per #4730.	2019-03-20 16:02:25 +00:00
Erik Johnston	cc197a61a1	Disable publishing to room list when its disabled	2019-03-20 14:30:36 +00:00
Erik Johnston	2c90422146	Pull out config option	2019-03-20 14:25:58 +00:00
Erik Johnston	7529038e66	Return before we log	2019-03-20 14:25:28 +00:00
Erik Johnston	213c98c00a	Add option to disable search room lists ... This disables both local and remote room list searching.	2019-03-19 17:10:52 +00:00
Neil Johnson	88f0675967	fix test_auto_create_auto_join_where_no_consent (#4886)	2019-03-19 11:38:59 +00:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864)	2019-03-19 04:50:24 +11:00
Brendan Abolivier	651ad8bc96	Add ratelimiting on failed login attempts (#4865)	2019-03-18 12:57:20 +00:00
Brendan Abolivier	899e523d6d	Add ratelimiting on login (#4821) ... Add two ratelimiters on login (per-IP address and per-userID).	2019-03-15 17:46:16 +00:00
Richard van der Hoff	2dee441bdb	Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker ... Move client receipt processing to federation sender worker.	2019-03-15 12:30:30 +00:00
Erik Johnston	9073cfc8bd	Merge pull request #4846 from matrix-org/hawkowl/userdir-search ... Improve searching in the userdir	2019-03-14 13:54:15 +00:00
Richard van der Hoff	fdcad8eabd	Move client receipt processing to federation sender worker. ... This is mostly a prerequisite for #4730, but also fits with the general theme of "move everything off the master that we possibly can".	2019-03-13 17:21:19 +00:00
Richard van der Hoff	eed7271b3b	declare a ReadReceipt class ... I'm going to use this in queues and things, so it'll be useful to give it more of a structure.	2019-03-13 17:20:55 +00:00
Andrew Morgan	d42c81d724	Transfer local user's push rules on room upgrade (#4838) ... Transfer push rules (notifications) on room upgrade	2019-03-12 14:42:53 +00:00
Amber Brown	c0332d095f	fixup	2019-03-13 01:30:54 +11:00
Amber Brown	10480c4348	fixup	2019-03-12 21:47:14 +11:00
Amber Brown	5ba8ceab4c	fixes	2019-03-12 00:35:31 +11:00
Amber Brown	4abc988c6a	initial	2019-03-11 21:11:36 +11:00
Erik Johnston	8c4896668f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl	2019-03-08 11:44:20 +00:00
Erik Johnston	0ff8163eae	Factor out soft fail checks	2019-03-08 11:26:33 +00:00
Erik Johnston	469b66c102	Merge pull request #4829 from matrix-org/erikj/device_list_seen_updates ... When re-syncing device lists reset the state	2019-03-08 09:54:54 +00:00
Erik Johnston	d42b41544a	When re-syncing device lists reset the state ... We keep track of what stream IDs we've seen so that we know what updates we've handled or missed. If we re-sync we don't know if the updates we've seen are included in the re-sync (there may be a race), so we should reset the seen updates.	2019-03-07 16:04:24 +00:00
Erik Johnston	c633fc02d7	Add some debug logging for device list handling	2019-03-07 15:58:03 +00:00
Amber Brown	f6135d06cf	Rewrite userdir to be faster (#4537)	2019-03-07 01:22:53 -08:00
Erik Johnston	0a6e716600	Merge pull request #4815 from matrix-org/erikj/docstrings ... Add docstrings from matrix-org-hotfixes	2019-03-06 18:59:28 +00:00
Erik Johnston	8b7790e68f	Port #4422 debug logging from hotfixes	2019-03-06 17:29:15 +00:00
Erik Johnston	6d13bdec91	Add docstrings from matrix-org-hotfixes	2019-03-06 17:24:26 +00:00
Erik Johnston	a9de04be72	Implement soft fail	2019-03-06 16:22:16 +00:00
Brendan Abolivier	6f3cde8b25	Make registration ratelimiter separate from the main events one	2019-03-06 11:02:42 +00:00
Brendan Abolivier	f4195f4118	Revert "Split ratelimiters in two (one for events, one for registration)" ... This reverts commit d7dbad3526.	2019-03-06 10:55:22 +00:00
Brendan Abolivier	d7dbad3526	Split ratelimiters in two (one for events, one for registration)	2019-03-05 18:41:27 +00:00
Brendan Abolivier	a4c3a361b7	Add rate-limiting on registration (#4735) ... * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test	2019-03-05 14:25:33 +00:00
Erik Johnston	b050a10871	Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room ... Stop backpaginating when events not visible	2019-03-05 09:32:33 +00:00
Erik Johnston	aa06d26ae0	clarify comments	2019-03-05 09:16:35 +00:00
Erik Johnston	c3c542bb4a	Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys ... Allow /keys/{changes,query} API to run on worker	2019-03-05 09:06:25 +00:00
Erik Johnston	bfa7d46a10	Allow /keys/{changes,query} API to run on worker	2019-03-04 18:30:01 +00:00
Erik Johnston	157e5a8f27	Split DeviceHandler into master and worker	2019-03-04 18:29:26 +00:00
Richard van der Hoff	b29693a30b	Clean up read-receipt handling. ... Remove a call to run_as_background_process: there is no need to run this as a background process, because build_and_send_edu does not block. We may as well inline the whole of _push_remotes.	2019-03-04 18:16:43 +00:00
Erik Johnston	d1523aed6b	Only check history visibility when filtering ... When filtering events to send to server we check more than just history visibility. However when deciding whether to backfill or not we only care about the history visibility.	2019-03-04 14:43:42 +00:00
Richard van der Hoff	856c83f5f8	Avoid rebuilding Edu objects in worker mode (#4770) ... In worker mode, on the federation sender, when we receive an edu for sending over the replication socket, it is parsed into an Edu object. There is no point extracting the contents of it so that we can then immediately build another Edu.	2019-03-04 12:57:44 +00:00
Erik Johnston	8b63fe4c26	s/get_forward_events/get_successor_events/	2019-03-04 11:56:03 +00:00
Erik Johnston	fbc047f2a5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room	2019-03-04 11:54:58 +00:00
Richard van der Hoff	3064952939	Fix incorrect log about not persisting duplicate state event. (#4776) ... We were logging this when it was not true.	2019-03-01 16:47:12 +00:00
Erik Johnston	b183fef9ac	Update comments	2019-02-27 13:06:10 +00:00
Andrew Morgan	802884d4ee	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate_develop	2019-02-26 14:23:40 +00:00
Andrew Morgan	6fcb25202f	Put function def back to the way it was	2019-02-26 14:17:38 +00:00
Andrew Morgan	7a4632af9c	Prevent showing non-fed rooms in fed /publicRooms	2019-02-26 13:37:24 +00:00
Andrew Morgan	c74624a633	Revert "Prevent showing non-fed rooms in fed /publicRooms"	2019-02-26 13:20:38 +00:00
Andrew Morgan	a1a6473293	Merge pull request #4736 from matrix-org/anoa/public_rooms_federate ... Config option to prevent showing non-fed rooms in fed /publicRooms	2019-02-26 13:07:15 +00:00
Andrew Morgan	c4414768af	Cleaner chunk logic	2019-02-26 12:22:34 +00:00
Andrew Morgan	a712aa3a9c	Correct indent	2019-02-26 12:13:55 +00:00
Andrew Morgan	16565e67db	Correct docstring types and chunk logic	2019-02-26 12:12:48 +00:00
Andrew Morgan	40c2271680	Clean up room chunk logic	2019-02-26 12:04:34 +00:00
Andrew Morgan	6728bf3940	Make not showing non-federated rooms the default	2019-02-26 11:52:52 +00:00
Andrew Morgan	6946c20111	Result may be None	2019-02-26 11:27:19 +00:00
Andrew Morgan	71669a0fba	Address rich comments	2019-02-26 11:25:00 +00:00
Matthew Hodgson	70ea2f4e1d	switch from google.com to recaptcha.net for reCAPTCHA (#4731) ... * add trivial clarification about jemalloc * switch from google.com to recaptcha.net because https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally	2019-02-25 11:15:36 -08:00
Andrew Morgan	1330aa4a8f	elif not else if	2019-02-25 17:28:19 +00:00
Erik Johnston	65f3fbfbf7	Merge pull request #4737 from matrix-org/erikj/failure_log_tb ... Log tracebacks correctly	2019-02-25 17:26:30 +00:00
Erik Johnston	c7b333c545	Log tracebacks correctly	2019-02-25 16:56:41 +00:00
Erik Johnston	9c598dddcb	Fix typo	2019-02-25 16:32:02 +00:00
Andrew Morgan	8aaf7ffc44	syntax derp	2019-02-25 15:27:17 +00:00
Andrew Morgan	84c0a20dfe	Simplify call to generate_room_entry	2019-02-25 15:23:27 +00:00
Erik Johnston	4b9e5076c4	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate	2019-02-25 15:08:18 +00:00
Andrew Morgan	07493607a8	Docs and arg name clarification	2019-02-25 15:04:46 +00:00
Andrew Morgan	bd398b874e	Don't restrict non-fed rooms over client APIs	2019-02-25 15:04:46 +00:00
Andrew Morgan	e4b078a600	Config option to prevent showing non-fed rooms in fed /publicRooms	2019-02-25 15:04:46 +00:00
Erik Johnston	d730c2c22b	More comments	2019-02-25 14:45:02 +00:00
Erik Johnston	890cb048fd	Assert rather than clobber the values	2019-02-25 14:42:11 +00:00
Erik Johnston	9342cc6ab1	Add comments and paranoia	2019-02-25 10:02:12 +00:00
Erik Johnston	7b288826b7	Fix backfill storing incorrect state for events	2019-02-22 11:33:51 +00:00
Richard van der Hoff	6d65659b62	Run push_receipts_to_remotes as background job (#4707) ... I suspect the CPU usage metrics for this are going to /dev/null at the moment.	2019-02-21 17:50:30 +00:00
Erik Johnston	b9d6756b14	Merge pull request #4263 from rkfg/develop ... Prevent crash on pagination.	2019-02-21 17:42:15 +00:00
Erik Johnston	71b625d808	Stop backpaginating when events not visible	2019-02-20 18:14:12 +00:00
Erik Johnston	dbdc565dfd	Fix registration on workers (#4682) ... * Move RegistrationHandler init to HomeServer * Move post registration actions to RegistrationHandler * Add post regisration replication endpoint * Newsfile	2019-02-20 18:47:31 +11:00
Brendan Abolivier	a288bdf0b1	Merge pull request #4652 from matrix-org/babolivier/acme-delegated ... Support .well-known delegation when issuing certificates through ACME	2019-02-19 11:15:38 +00:00
Brendan Abolivier	a8626901cd	Fetch ACME domain into an instance member	2019-02-19 10:54:33 +00:00
Andrew Morgan	968a30a75c	Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade ... Transfer bans on room upgrade	2019-02-19 10:16:42 +00:00
Andrew Morgan	34ac75ce2c	lint	2019-02-18 18:23:37 +00:00
Andrew Morgan	f8b9ca53ce	Move member event processing and changelog fix	2019-02-18 17:28:52 +00:00
Erik Johnston	fc2c245a1f	Merge pull request #4666 from matrix-org/erikj/register_login_split ... Split out registration to worker	2019-02-18 17:18:06 +00:00
Erik Johnston	459d3d5046	Merge pull request #4668 from matrix-org/erikj/catch_exceptions ... Correctly handle HttpResponseException when handling device updates	2019-02-18 16:55:43 +00:00
Erik Johnston	af691e415c	Move register_device into handler	2019-02-18 16:49:38 +00:00
Erik Johnston	c30f73c86a	Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers ... Fix kicking guest users in worker mode	2019-02-18 15:55:48 +00:00
Brendan Abolivier	45bb55c6de	Use a configuration parameter to give the domain to generate a certificate for	2019-02-18 15:46:23 +00:00
Erik Johnston	94960cef03	pep8	2019-02-18 15:24:13 +00:00
Erik Johnston	e07cc31cb8	Correctly handle HttpResponseException	2019-02-18 14:55:09 +00:00
Brendan Abolivier	68a53f825f	Merge branch 'develop' into babolivier/acme-delegated	2019-02-18 14:52:23 +00:00
Erik Johnston	32e54b472a	Fix kicking guest users in worker mode ... When guest_access changes from allowed to forbidden all local guest users should be kicked from the room. This did not happen when revocation was received from federation on a worker. Presumably broken in #4141	2019-02-18 14:16:07 +00:00
Andrew Morgan	915421065b	Membership events are done later	2019-02-18 14:02:09 +00:00
Erik Johnston	d1b060b492	Merge pull request #4643 from matrix-org/erikj/catch_exceptions ... Correctly handle RequestSendFailed exceptions	2019-02-18 13:53:24 +00:00
Andrew Morgan	9caab0c364	Transfer bans on room upgrade	2019-02-18 13:52:51 +00:00
Erik Johnston	eb2b8523ae	Split out registration to worker ... This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker.	2019-02-18 12:12:57 +00:00
Andrew Morgan	6d02a13d81	Typo in info log ... Co-Authored-By: babolivier <contact@brendanabolivier.com>	2019-02-18 11:36:34 +00:00
Brendan Abolivier	f86b695cbd	Various cosmetics to make TravisCI happy	2019-02-15 12:29:34 +00:00
Brendan Abolivier	af8a2f679b	Remove unused import	2019-02-15 12:27:43 +00:00
Brendan Abolivier	1895d14e12	Support .well-known delegation when issuing certificates through ACME	2019-02-15 12:05:08 +00:00
Erik Johnston	f61b2068e6	Only fetch aliases when publishing rooms	2019-02-14 18:08:09 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules ... This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Erik Johnston	7fc1196a36	Correctly handle RequestSendFailed exceptions ... This mainly reduces the number of exceptions we log.	2019-02-14 14:01:04 +00:00
Amber Brown	bb4fd8f927	Run black on user directory code (#4635)	2019-02-13 23:05:32 +11:00
Erik Johnston	ba3f27b69a	Merge pull request #4608 from matrix-org/anoa/acls_room_upgrade ... Transfer Server ACLs on room upgrade	2019-02-12 13:20:06 +00:00
Andrew Morgan	a126f86eec	Transfer Server ACLs on room upgrade	2019-02-11 11:30:37 +00:00
Hubert Chathi	afae8442b5	make sure version is in body and wrap in linearizer queue ... also add tests	2019-02-08 01:32:45 -05:00
Hubert Chathi	d9e424bf64	re-try to make isort happy	2019-02-06 22:18:41 -05:00
Hubert Chathi	9ff620a518	fix import to make isort happy	2019-02-06 21:32:52 -05:00
Hubert Chathi	8248637173	add new endpoint to update backup versions	2019-02-06 17:57:10 -05:00
Hubert Chathi	664c81e8b7	return proper error codes for some 404s	2019-02-06 17:47:22 -05:00
Andrew Morgan	c45fd0dda0	Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable ... Copy over non-federatable trait on room upgrade	2019-01-31 20:28:37 +00:00
Richard van der Hoff	07dfe148de	Add some debug for membership syncing issues (#4538) ... I can't figure out what's going on with #4422 and #4436; perhaps this will help.	2019-01-31 18:30:40 +00:00
Andrew Morgan	3ed3cb4339	New function for getting room's create event	2019-01-31 18:21:39 +00:00
Andrew Morgan	bbb97a35fd	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable	2019-01-31 18:11:00 +00:00
Andrew Morgan	fb50934b8f	lint	2019-01-31 11:34:45 +00:00
Andrew Morgan	d621c5562e	Copy over non-federatable trait on room upgrade	2019-01-30 16:33:51 +00:00
Erik Johnston	ed8c5e4cda	Fix remote invite rejections not comming down sync ... This was broken in PR #4405, commit 886e5ac, where we changed remote rejections to be outliers. The fix is to explicitly add the leave event in when we know its an out of band invite. We can't always add the event as if the server is/was in the room there might be more events to send down the sync than just the leave.	2019-01-30 15:46:27 +00:00
Richard van der Hoff	7615a8ced1	ACME config cleanups (#4525) ... * Handle listening for ACME requests on IPv6 addresses the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses without extra quoting. Building a string which you are about to parse again seems like a weird choice. Let's just use listenTCP, which is consistent with what we do elsewhere. * Clean up the default ACME config make it look a bit more consistent with everything else, and tweak the defaults to listen on port 80. * newsfile	2019-01-30 14:17:55 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519)	2019-01-30 10:53:17 +00:00
Erik Johnston	67b82f1336	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:58:38 +00:00
Erik Johnston	a696c48133	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:00:33 +00:00
Erik Johnston	55d9024835	Use snder and not event ID domain to check if ours ... The transaction queue only sends out events that we generate. This was done by checking domain of event ID, but that can no longer be used. Instead, we may as well use the sender field.	2019-01-29 16:54:23 +00:00
Erik Johnston	b1fffca345	Remove event ID usage when checking if new room ... The event ID is changing, so we can no longer get the domain from it. On the other hand, the check is unnecessary.	2019-01-29 16:15:02 +00:00
Erik Johnston	b8d75ef53e	Merge pull request #4481 from matrix-org/erikj/event_builder ... Refactor event building into EventBuilder	2019-01-29 14:07:23 +00:00
Erik Johnston	5891a6edc8	Correctly set context.app_service	2019-01-29 12:09:10 +00:00
Erik Johnston	7709d2bd16	Implement rechecking of redactions	2019-01-29 11:56:20 +00:00
Erik Johnston	6598992b01	Fixup calls to comput_event_signature ... We currently pass FrozenEvent instead of `dict` to `compute_event_signature`, which works by accident due to `dict(event)` producing the correct result. This fixes PR #4493 commit 855a151	2019-01-29 11:41:58 +00:00
Erik Johnston	be47cfa9c9	Refactor event building into EventBuilder ... This is so that everything is done in one place, making it easier to change the event format based on room version	2019-01-29 11:13:00 +00:00
Erik Johnston	073f6c2e5e	Merge pull request #4494 from matrix-org/erikj/fixup_event_validator ... Split up event validation between event and builder	2019-01-29 10:55:07 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version ... Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Erik Johnston	d758d5310e	Correctly use default room version if none is set	2019-01-28 17:26:39 +00:00
Erik Johnston	b872c7b1b4	Split up event validation between event and builder ... The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.	2019-01-28 17:00:14 +00:00
Andrew Morgan	f0e96ab66a	Change return syntax in doc string	2019-01-28 14:09:45 +00:00
Andrew Morgan	4026d555fa	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-28 14:08:24 +00:00
Andrew Morgan	1ce463963d	Reuse predecessor method	2019-01-28 14:08:18 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version ... Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Andrew Morgan	9244a3089e	Fixes	2019-01-25 11:48:38 +00:00
Andrew Morgan	da0d2219d2	Clean up direct_rooms access	2019-01-25 11:37:12 +00:00
Andrew Morgan	8265995498	Use python magic	2019-01-25 11:26:06 +00:00
Andrew Morgan	c4cdafa81f	Destructure account data tuple before use	2019-01-25 11:24:28 +00:00
Andrew Morgan	516456b763	Remove unnecessary null check	2019-01-25 11:22:14 +00:00
Andrew Morgan	6f3fda79ce	Move room_tag declaration to be closer to its use	2019-01-25 11:21:25 +00:00
Andrew Morgan	821b65aeb5	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-25 11:09:53 +00:00
Andrew Morgan	0862d35b8e	Move tag and direct state copying into separate function	2019-01-25 11:09:34 +00:00
Andrew Morgan	b1b6dba2d2	Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms ... Ability to search entire room history after upgrading room	2019-01-25 10:49:58 +00:00
Erik Johnston	a50cf929c1	Require event format version to parse or create events	2019-01-25 10:32:19 +00:00
Erik Johnston	62514bb81b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813	2019-01-25 10:07:08 +00:00
Erik Johnston	edc1e21dbe	Merge pull request #4448 from matrix-org/erikj/get_pdu_versions ... Add room_version param to get_pdu	2019-01-24 18:47:15 +00:00
Erik Johnston	80bcca659e	Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites ... Store rejected remote invite events as outliers	2019-01-24 18:33:27 +00:00
Erik Johnston	d148c43050	Review comments	2019-01-24 18:31:23 +00:00
Erik Johnston	9139b87be4	Remove unecessary setting of outlier bit	2019-01-24 18:04:02 +00:00
Erik Johnston	b8082a5445	Use term 'out of band membership' instead	2019-01-24 17:33:19 +00:00
Richard van der Hoff	03c85335d1	Apply suggestions from code review ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:22:09 +00:00
Andrew Morgan	075ff3ede9	Change default timeout value from 0 to None	2019-01-24 15:10:22 +00:00
Andrew Morgan	5b1dc94083	Use self.clock instead of datetime	2019-01-24 14:59:50 +00:00
Andrew Morgan	a2d85144e5	isort	2019-01-24 14:22:26 +00:00
Andrew Morgan	5541645e80	lint	2019-01-24 12:45:32 +00:00
Andrew Morgan	068aa1d228	Time out filtered room dir queries after 60s	2019-01-24 12:44:27 +00:00
Erik Johnston	7c288c2250	Clarify the invite flows	2019-01-23 20:07:47 +00:00
Erik Johnston	07f62da55a	Remove unnecessary '_sign_event'	2019-01-23 20:04:57 +00:00
Erik Johnston	886e5acc76	Store rejected remote invite events as outliers ... Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.	2019-01-23 20:04:57 +00:00
Erik Johnston	6a41d2a187	Add room_version param to get_pdu ... When we add new event format we'll need to know the event format or room version when parsing events.	2019-01-23 17:19:58 +00:00
Erik Johnston	67cd4dad81	Implement MSC 1813 - Add room version to make APIs ... We also implement `make_membership_event` converting the returned room version to an event format version.	2019-01-23 16:51:46 +00:00
Andrew Morgan	82a92ba535	Add metric for user dir current event stream position	2019-01-23 15:34:47 +00:00
Amber Brown	6129e52f43	Support ACME for certificate provisioning (#4384)	2019-01-23 19:39:06 +11:00
Andrew Morgan	766a172b99	lint	2019-01-22 13:51:40 +00:00
Andrew Morgan	c4875d8c76	Prevent duplicate room IDs in m.direct	2019-01-22 12:13:46 +00:00
Andrew Morgan	c9bfb058d8	Fix a bug with single-room search searching all rooms ... * Create a new method for getting predecessor rooms * Remove formatting change	2019-01-22 12:00:41 +00:00
Andrew Morgan	8086a5c05e	Fix comments	2019-01-22 11:16:23 +00:00
Andrew Morgan	48951f437f	Join logic covers both room creator and arbitrary users	2019-01-22 11:00:04 +00:00
Andrew Morgan	8c85f0833d	tags, m.direct copying over correctly	2019-01-22 11:00:04 +00:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Andrew Morgan	ea8903fcc9	Migrating dm and room tags work for migrator	2019-01-22 11:00:04 +00:00
Andrew Morgan	887ca93a1b	Prevent crash on user who doesn't have any direct rooms	2019-01-22 11:00:04 +00:00
Andrew Morgan	4ff6d22245	Preserve DM status of a room on upgrade ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-22 11:00:04 +00:00
Andrew Morgan	75942af1db	Fix typo	2019-01-22 11:00:04 +00:00
Andrew Morgan	702c4b750c	Migrate encryption state on room upgrade (#4411) ... * Migrate encryption state on room upgrade Signed-off-by: Andrew Morgan <andrew@amorgan.xyz> * Add changelog file	2019-01-21 20:42:58 +11:00
Andrew Morgan	df3a661e4a	Search for messages across predecessor rooms ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-18 11:19:20 +00:00
Richard van der Hoff	9feb5d0b71	sign_request -> build_auth_headers (#4408) ... Just got very confused about the fact that the headers are only an output, not an input.	2019-01-17 12:40:09 +00:00
Richard van der Hoff	05e1296649	don't store more remote device lists if they have more than 1K devices (#4397)	2019-01-16 23:14:11 +00:00
Matthew Hodgson	27128145e6	fix NPE in /messages by checking if all events were filtered out (#4330)	2019-01-02 23:37:39 +00:00
Nathan Pennie	81b513416e	Fixed line length	2018-12-19 14:12:33 -05:00
Nathan Pennie	668e6625b0	Fix the variable names used for account_data	2018-12-19 06:53:02 -05:00
Neil Johnson	d2f7c4e6b1	create support user (#4141) ... Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272) ... When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	c588b9b9e4	Factor SSO success handling out of CAS login (#4264) ... This is mostly factoring out the post-CAS-login code to somewhere we can reuse it for other SSO flows, but it also fixes the userid mapping while we're at it.	2018-12-07 13:10:07 +01:00
rkfg	ae19a7db8c	Prevent crash on pagination.	2018-12-06 13:32:05 +03:00
Richard van der Hoff	b0c24a66ec	Rip out half-implemented m.login.saml2 support (#4265) ... * Rip out half-implemented m.login.saml2 support This was implemented in an odd way that left most of the work to the client, in a way that I really didn't understand. It's going to be a pain to maintain, so let's start by ripping it out. * drop undocumented dependency on dateutil It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 19:44:38 +11:00
Travis Ralston	158ffb92f1	Add an option to disable search for homeservers which may not be interested in it (#4230) ... This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.	2018-12-04 12:01:02 +01:00
Neil Johnson	7039ece8fb	Neilj/fix autojoin (#4223) ... * Fix auto join failures for servers that require user consent * Fix auto join failures for servers that require user consent	2018-11-28 22:24:57 +11:00
Travis Ralston	ab4526a153	Remove duplicate slashes in generated consent URLs	2018-11-15 20:41:53 -07:00
Erik Johnston	dc59ad5334	Remove hack to support rejoining rooms	2018-11-09 14:58:09 +00:00
Erik Johnston	30dd27afff	Simplify to always drop events if server isn't in the room	2018-11-09 11:36:45 +00:00
Travis Ralston	0f5e51f726	Add config variables for enabling terms auth and the policy name (#4142) ... So people can still collect consent the old way if they want to.	2018-11-06 10:32:34 +00:00
Hubert Chathi	f1087106cf	handle empty backups according to latest spec proposal (#4123) ... fixes #4056	2018-11-05 17:59:29 -05:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139) ... * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Amber Brown	cb7a6b2379	Fix typing being reset causing infinite syncs (#4127)	2018-11-03 00:19:23 +11:00
Travis Ralston	a8d41c6aff	Include a version query string arg for the consent route	2018-10-31 13:19:28 -06:00
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00
Richard van der Hoff	1729ba1650	Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades ... Attempt to move room aliases on room upgrades	2018-10-31 17:52:18 +00:00
Richard van der Hoff	0f8591a5a8	Avoid else clause on exception for clarity	2018-10-31 15:43:57 +00:00
Richard van der Hoff	94c7fadc98	Attempt to move room aliases on room upgrades	2018-10-31 15:43:57 +00:00
Richard van der Hoff	9b827c40ca	Log some bits about event creation (#4121) ... I found these helpful in debugging my room upgrade tests.	2018-10-31 15:42:23 +00:00
Amber Brown	086e1a8f3e	Fix search 500ing (#4122)	2018-10-31 06:26:46 +11:00
Amber Brown	3bade14ec0	Fix search 500ing (#4122)	2018-10-31 04:33:41 +11:00
Amber Brown	4cd1c9f2ff	Delete the disused & unspecced identicon functionality (#4106)	2018-10-29 23:57:24 +11:00
Richard van der Hoff	7fbfea062e	Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar ... Remember to copy the avatar on room upgrades	2018-10-29 12:49:21 +00:00
Richard van der Hoff	db24d7f15e	Better handling of odd PLs during room upgrades ... Fixes handling of rooms where we have permission to send the tombstone, but not other state. We need to (a) fail more gracefully when we can't send the PLs in the old room, and (b) not set the PLs in the new room until we are done with the other stuff.	2018-10-27 00:54:26 +01:00
Richard van der Hoff	5caf79b312	Remember to copy the avatar on room upgrades	2018-10-26 23:56:40 +01:00
Richard van der Hoff	54bbe71867	optimise state copying	2018-10-26 22:51:34 +01:00
Richard van der Hoff	193cadc988	Address review comments ... Improve comments, get old room state from the context we already have	2018-10-26 17:10:30 +01:00
Richard van der Hoff	e6babc27d5	restrict PLs in old room	2018-10-25 19:18:25 +01:00
Richard van der Hoff	3a263bf3ae	copy state	2018-10-25 19:18:25 +01:00
Richard van der Hoff	1b9f253e20	preserve PLs	2018-10-25 19:10:24 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Richard van der Hoff	0f7d1c9906	Basic initial support for room upgrades ... Currently just creates a new, empty, room, and sends a tombstone in the old room.	2018-10-25 19:10:24 +01:00
Richard van der Hoff	e1948175ee	Allow power_level_content_override=None for _send_events_for_new_room	2018-10-25 19:10:24 +01:00
Richard van der Hoff	7f7b2cd3de	Make room_member_handler a member of RoomCreationHandler ... ... to save passing it into `_send_events_for_new_room`	2018-10-25 19:10:18 +01:00
Richard van der Hoff	871c4abfec	Factor _generate_room_id out of create_room ... we're going to need this for room upgrades.	2018-10-25 18:23:09 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Erik Johnston	b94a43d5b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list	2018-10-25 15:25:31 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Travis Ralston	54def42c19	Merge branch 'develop' into travis/login-terms	2018-10-24 13:22:59 -06:00
Neil Johnson	9532caf6ef	remove trailing whiter space	2018-10-24 16:08:25 +01:00
Richard van der Hoff	94a49e0636	fix tuple ... Co-Authored-By: neilisfragile <neil@matrix.org>	2018-10-24 14:39:23 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	e0b9d5f0af	Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts ... Clean up the way logcontexts and threads work in the pushers	2018-10-24 09:53:57 +01:00
Erik Johnston	dacbeb2e03	Comment	2018-10-24 09:47:49 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Erik Johnston	cb23aa4c42	Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation ... Clean up room alias creation	2018-10-24 09:40:27 +01:00
Neil Johnson	a67d8ace9b	remove errant exception and style	2018-10-23 17:44:39 +01:00
Travis Ralston	43c3f0b02f	Merge pull request #3969 from turt2live/travis/fix-federated-group-requests ... Handle HttpResponseException more safely for federated groups	2018-10-23 10:41:04 -06:00
Travis Ralston	3e704822be	Comments help	2018-10-23 10:25:31 -06:00
Richard van der Hoff	b3f6dddad2	Give some more things logcontexts (#4077)	2018-10-23 14:29:17 +01:00
Richard van der Hoff	5c445114d3	Correctly account for cpu usage by background threads (#4074) ... Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064	2018-10-23 13:12:32 +01:00
Richard van der Hoff	e7a16c6210	Remove redundant run_as_background_process() from pusherpool ... `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.	2018-10-22 16:12:11 +01:00
Erik Johnston	084046456e	Add config option to control alias creation	2018-10-19 10:22:45 +01:00
Erik Johnston	74e7617083	Clean up room alias creation	2018-10-19 10:11:56 +01:00
Travis Ralston	49a044aa5f	Merge branch 'develop' into travis/login-terms	2018-10-18 09:57:58 -06:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046) ... Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Neil Johnson	c6584f4b5f	clean up config error logic and imports	2018-10-17 11:36:41 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	15133477ee	Fix up use of resolve_events_with_factory	2018-10-16 16:28:42 +01:00
Travis Ralston	a8ed93a4b5	pep8	2018-10-15 16:10:29 -06:00
Travis Ralston	442734ff9e	Ensure the terms params are actually provided	2018-10-15 14:56:13 -06:00
Travis Ralston	762a0982aa	Python is hard	2018-10-15 14:46:09 -06:00
Travis Ralston	f293d124b6	Merge branch 'develop' into travis/login-terms	2018-10-15 14:44:32 -06:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Neil Johnson	1ccafb0c5e	no need to join room if creator	2018-10-13 21:14:21 +01:00
Travis Ralston	dd99db846d	Update login terms structure for the proposed language support	2018-10-12 18:03:27 -06:00
Travis Ralston	7ede650956	Merge branch 'develop' into travis/login-terms	2018-10-12 16:24:07 -06:00
Travis Ralston	164f8e4843	isort	2018-10-12 15:11:59 -06:00
Travis Ralston	7bb651de6a	More sane handling of group errors and pep8	2018-10-12 14:53:30 -06:00
Travis Ralston	e3586f7c06	Merge branch 'develop' into travis/fix-federated-group-requests	2018-10-12 14:49:58 -06:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	8c0ff0287a	Linting soothes the savage PEP8 monster	2018-10-12 13:47:43 +01:00
David Baker	83e72bb2f0	PR feedback pt. 1	2018-10-12 11:26:18 +01:00
Erik Johnston	7e561b5c1a	Add description to counter metric	2018-10-10 11:41:15 +01:00
Erik Johnston	3cbe8331e6	Track number of non-empty sync responses instead	2018-10-10 11:23:17 +01:00
Erik Johnston	395276b405	Append _total to metric and fix up spelling	2018-10-10 09:24:39 +01:00
Erik Johnston	bdc27d6716	Add metric to count lazy member sync requests	2018-10-09 14:17:52 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	f4a4dbcad1	Apparently this blank line is Very Important	2018-10-09 09:47:04 +01:00
David Baker	0c905ee015	be python3 compatible	2018-10-09 09:39:13 +01:00
Erik Johnston	6982320572	Remove unnecessary extra function call layer	2018-10-08 14:06:19 +01:00
Erik Johnston	8a1817f0d2	Use errback pattern and catch async failures	2018-10-08 13:29:47 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Travis Ralston	537d0b7b36	Use a flag rather than a new route for the public policy ... This also means that the template now has optional parameters, which will need to be documented somehow.	2018-10-03 17:50:11 -06:00
Travis Ralston	149c4f1765	Supply params for terms auth stage ... As per https://github.com/matrix-org/matrix-doc/pull/1692	2018-10-03 15:57:42 -06:00
Travis Ralston	fd99787162	Incorporate Dave's work for GDPR login flows ... As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117	2018-10-03 15:57:42 -06:00
Erik Johnston	495a9d06bb	Fix exception handling in fetching remote profiles	2018-10-03 11:34:30 +01:00
Richard van der Hoff	8c41b0ca66	Merge pull request #3989 from matrix-org/rav/better_stacktraces ... Avoid reraise, to improve stacktraces	2018-10-02 10:47:38 +01:00
Erik Johnston	bc29946809	Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state ... Fix lazy loaded sync with rejected state events	2018-10-02 11:38:35 +02:00
Richard van der Hoff	8174c6725b	Avoid reraise, to improve stacktraces	2018-10-01 18:50:34 +01:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors ... Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Erik Johnston	82f922b4af	Fix lazy loaded sync with rejected state events ... In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.	2018-10-01 14:19:40 +01:00
Matthew Hodgson	23b6a0537f	emit room aliases event	2018-09-29 02:19:37 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Matthew Hodgson	8f646f2d04	fix UTs	2018-09-28 15:37:28 +01:00
Richard van der Hoff	3deaad2fb4	Merge pull request #3964 from matrix-org/rav/remove_localhost_checks ... remove spurious federation checks on localhost	2018-09-28 13:35:47 +01:00
Richard van der Hoff	965154d60a	Fix complete fail to do the right thing	2018-09-28 12:45:54 +01:00
Matthew Hodgson	07340cdaca	untested stab at autocreating autojoin rooms	2018-09-28 01:42:53 +01:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Richard van der Hoff	36c62a67c4	Merge pull request #3794 from matrix-org/erikj/faster_typing ... Improve performance of getting typing updates for replication	2018-09-27 15:14:26 +01:00
Richard van der Hoff	e1e3e77bfd	Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups ... Clarifications in FederationHandler	2018-09-27 15:06:59 +01:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 ... Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) ... Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	333bee27f5	Include event when resolving state for missing prevs ... If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	bd61c82bdf	Include state from remote servers in pdu handling ... If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	a215b698c4	Fix "unhashable type: 'list'" exception in federation handling ... get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	28223841e0	more comments	2018-09-27 11:31:51 +01:00
Richard van der Hoff	e3c159863d	Clarifications in FederationHandler ... * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)	2018-09-27 11:31:51 +01:00
Richard van der Hoff	92abd3d6d6	Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 ... Logging improvements	2018-09-27 11:27:46 +01:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Travis Ralston	82fa31799c	Remove debugging statement	2018-09-26 14:01:02 -06:00
Travis Ralston	b4c3bc1734	Handle HttpResponseException more safely for federated groups	2018-09-26 13:48:04 -06:00
Richard van der Hoff	e70b4ce069	Logging improvements ... Some logging tweaks to help with debugging incoming federation transactions	2018-09-26 17:36:14 +01:00
Amber Brown	c2185f14d7	Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu ... Comments and interface cleanup for on_receive_pdu	2018-09-26 21:41:26 +10:00
Matthew Hodgson	787d22ed6c	Only lazy load self-members on initial sync ... Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.	2018-09-25 00:49:26 +01:00
Matthew Hodgson	a2ddaa90f2	Always LL ourselves if we're in a room to simplify clients (#3916) ... Should fix https://github.com/vector-im/riot-web/issues/7209	2018-09-20 21:21:54 +01:00
Richard van der Hoff	703de4ec13	Comments and interface cleanup for on_receive_pdu ... Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.	2018-09-20 13:06:55 +01:00
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904) ... Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	2018-09-19 17:28:18 +01:00
Erik Johnston	ce846bb620	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing	2018-09-19 15:08:36 +01:00
Amber Brown	f773ecbd61	Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout ... Bump timeout on get_missing_events request	2018-09-19 17:57:48 +10:00
Richard van der Hoff	a219ce8726	Use directory server for room joins (#3899) ... When we do a join, always try the server we used for the alias lookup first. Fixes #2418	2018-09-18 18:27:37 +01:00
Richard van der Hoff	550007cb0e	Bump timeout on get_missing_events request	2018-09-18 15:02:51 +01:00
Richard van der Hoff	286d6930b7	Merge pull request #3879 from matrix-org/matthew/fix-autojoin ... don't ratelimit autojoins	2018-09-18 13:07:01 +01:00
Richard van der Hoff	f75b9961c6	Reinstate missing null check	2018-09-17 16:52:02 +01:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias ... As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
Matthew Hodgson	d42d79e3c3	don't ratelimit autojoins	2018-09-15 22:27:41 +01:00
Matthew Hodgson	024be6cf18	don't filter membership events based on history visibility (#3874) ... don't filter membership events based on history visibility as we will already have filtered the messages in the timeline, and state events are always visible. and because @erikjohnston said so.	2018-09-14 18:12:52 +01:00
Matthew Hodgson	2ac1abbc7e	show heroes if a room has a 'deleted' name/canonical_alias (#3851)	2018-09-12 17:11:05 +01:00
Matthew Hodgson	0403cf0783	argh pep8	2018-09-12 16:54:28 +01:00
Matthew Hodgson	0e200e366d	correctly log gappy sync metrics	2018-09-12 16:47:20 +01:00
Matthew Hodgson	11bfc2af1c	fix logline	2018-09-12 16:45:42 +01:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827) ... * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Erik Johnston	f60c9e2a01	Don't send empty tags list down sync	2018-09-06 17:01:41 +01:00
Erik Johnston	7baf66ef5d	Send existing room tags down sync on join ... When a user joined a room any existing tags were not sent down the sync stream. Ordinarily this isn't a problem because the user needs to be in the room to have set tags in it, however synapse will sometimes add tags for a user to a room, e.g. for server notices, which need to come down sync.	2018-09-06 16:46:51 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Amber Brown	ee7c8bd2b5	Merge pull request #3795 from matrix-org/erikj/faster_sync_state ... User iter* during sync state calculations	2018-09-07 00:24:43 +10:00
Amber Brown	2608ebc04c	Port handlers/ to Python 3 (#3803)	2018-09-07 00:22:23 +10:00
Erik Johnston	7419764351	User iter* during sync state calculations	2018-09-05 16:19:50 +01:00
Erik Johnston	5f02017aea	Improve performance of getting typing updates for replication ... Fetching the list of all new typing notifications involved iterating over all rooms and comparing their serial. Lets move to using a stream change cache, like we do for other streams.	2018-09-05 10:20:40 +01:00
Erik Johnston	567363e497	Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func ... Remove unnecessary resolve_events_with_state_map	2018-09-03 16:19:41 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Matthew Hodgson	b1580f50fe	don't return non-LL-member state in incremental sync state blocks (#3760) ... don't return non-LL-member state in incremental sync state blocks	2018-08-28 23:25:58 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	5c261107c9	Remove unnecessary resolve_events_with_state_map ... We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.	2018-08-22 15:41:15 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	8432e2ebd7	Rename WorkerProfileHandler to BaseProfileHandler	2018-08-22 10:13:40 +01:00
Erik Johnston	a81f140880	Add assert to ensure handler is only run on master	2018-08-22 10:11:21 +01:00
Erik Johnston	47b25ba5f3	Remove redundant vars	2018-08-22 10:09:05 +01:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	8dee601054	Remove redundant room_version checks	2018-08-20 14:48:53 +01:00
Erik Johnston	e21c368b8b	Revert spurious change	2018-08-20 13:54:51 +01:00
Richard van der Hoff	c144252a8c	Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates ... Fix logcontexts for running pushers	2018-08-17 16:21:49 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	2018-08-18 01:08:45 +10:00
Erik Johnston	ab822a2d1f	Add some fixmes	2018-08-17 15:31:50 +01:00
Erik Johnston	91cdb6de08	Call UserDirectoryHandler methods directly ... Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.	2018-08-17 15:26:13 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Richard van der Hoff	66f7dc8c87	Fix logcontexts for running pushers ... First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.	2018-08-17 00:32:39 +01:00
Neil Johnson	372bf073c1	block event creation and room creation on hitting resource limits	2018-08-16 21:25:16 +01:00
Matthew Hodgson	762a758fea	lazyload aware /messages (#3589)	2018-08-16 14:22:47 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus ... Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Erik Johnston	dc56c47dc0	Merge pull request #3653 from matrix-org/erikj/split_federation ... Move more federation APIs to workers	2018-08-15 14:59:02 +01:00
Neil Johnson	4601129c44	Merge pull request #3687 from matrix-org/neilj/admin_email ... support admin_email config and pass through into blocking errors,	2018-08-15 13:52:25 +00:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Erik Johnston	488ffe6fdb	Use federation handler function rather than duplicate ... This involves renaming _persist_events to be a public function.	2018-08-15 14:17:18 +01:00
Erik Johnston	fef2e65d12	Merge pull request #3667 from matrix-org/erikj/fixup_unbind ... Don't fail requests to unbind 3pids for non supporting ID servers	2018-08-15 10:32:12 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	c74c71128d	remove blank line	2018-08-14 15:06:24 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	bd92c8eaa7	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:56:45 +01:00
Neil Johnson	99ebaed8e6	Update register.py ... remove comments	2018-08-14 14:55:55 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" ... This reverts commit 0d43f991a1.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	72788cf9c1	support DELETE /version with no args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	5e42c45c96	switch get_current_version_info back to being get_version_info	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	14b3da63a3	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f500cb39e	more docstring for the e2e_room_keys rest	2018-08-12 19:14:31 -04:00
Matthew Hodgson	ca0b052307	fix factoring out of _should_replace_room_key	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Erik Johnston	b179537f2a	Move clean_room_for_join to master	2018-08-09 10:37:38 +01:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Erik Johnston	bf7598f582	Log when we 3pid/unbind request fails	2018-08-09 10:09:56 +01:00
Erik Johnston	2bdafaf3c1	Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet ... Add helper base class for generating new replication endpoints	2018-08-09 10:06:23 +01:00
Richard van der Hoff	bb89c84614	Merge pull request #3664 from matrix-org/rav/federation_metrics ... more metrics for the federation and appservice senders	2018-08-08 23:16:58 +01:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Erik Johnston	360ba89c50	Don't fail requests to unbind 3pids for non supporting ID servers ... Older identity servers may not support the unbind 3pid request, so we shouldn't fail the requests if we received one of 400/404/501. The request still fails if we receive e.g. 500 responses, allowing clients to retry requests on transient identity server errors that otherwise do support the API. Fixes #3661	2018-08-08 12:06:18 +01:00
Richard van der Hoff	53bca4690b	more metrics for the federation and appservice senders	2018-08-07 19:09:48 +01:00
Richard van der Hoff	3523f5432a	Don't expose default_room_version as config opt	2018-08-07 12:51:57 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Erik Johnston	a3f5bf79a0	Add EDU/query handling over replication	2018-08-06 15:23:31 +01:00
Erik Johnston	e26dbd82ef	Add replication APIs for persisting federation events	2018-08-06 15:02:28 +01:00
Richard van der Hoff	f900d50824	include known room versions in outgoing make_joins	2018-08-06 13:45:37 +01:00
Neil Johnson	e40a510fbf	py3 fix	2018-08-03 23:19:13 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Will Hunt	16d9701892	Return M_NOT_FOUND when a profile could not be found. (#3596)	2018-08-03 19:08:05 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Erik Johnston	cb298ff623	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet	2018-08-03 09:25:15 +01:00
Richard van der Hoff	1fa98495d0	Merge pull request #3639 from matrix-org/rav/refactor_error_handling ... Clean up handling of errors from outbound requests	2018-08-02 17:38:24 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	0bf5ec0db7	Check room visibility for /event/ requests ... Make sure that the user has permission to view the requeseted event for /event/{eventId} and /room/{roomId}/event/{eventId} requests. Also check that the event is in the given room for /room/{roomId}/event/{eventId}, for sanity.	2018-08-02 15:03:27 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Richard van der Hoff	01e93f48ed	Kill off MatrixCodeMessageException ... This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.	2018-08-01 16:02:46 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Erik Johnston	4b256b9271	_persist_auth_tree no longer returns anything	2018-08-01 13:39:07 +01:00
Neil Johnson	0aba3d361a	count_monthly_users() async	2018-08-01 11:47:58 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Richard van der Hoff	bdbdceeafa	rename replication_layer to federation_client ... I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.	2018-07-31 15:44:05 +01:00
Erik Johnston	443da003bc	Use new helper base class for membership requests	2018-07-31 14:32:23 +01:00
Erik Johnston	729b672823	Use new helper base class for ReplicationSendEventRestServlet	2018-07-31 14:32:23 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	b0b5566f36	Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid ... if inviter_display_name == ""||None then default to inviter MXID	2018-07-27 10:08:39 +01:00
Matthew Hodgson	a75231b507	Deduplicate redundant lazy-loaded members (#3331) ... * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review	2018-07-26 22:51:30 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	1bcd0490c2	Merge pull request #2970 from matrix-org/matthew/filter_members ... Implement the lazy_load_members room state filter parameter	2018-07-26 00:03:01 +01:00
Travis Ralston	d8e65ed7e1	Fix a minor documentation typo in on_make_leave	2018-07-25 15:44:41 -06:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Richard van der Hoff	55acd6856c	Fix updating of cached remote profiles ... _update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when it was called, would just return a generator object, without actually running any of the method body.	2018-07-25 10:34:48 +01:00
Michael Telatynski	87951d3891	Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid	2018-07-24 17:17:46 +01:00
Erik Johnston	8b8c4f34a3	Replace usage of get_current_toke with StreamToken.START ... This allows us to handle /context/ requests on the client_reader worker without having to pull in all the various stream handlers (e.g. precence, typing, pushers etc). The only thing the token gets used for is pagination, and that ignores everything but the room portion of the token.	2018-07-24 16:49:17 +01:00
Richard van der Hoff	a321f78991	Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups ... Fixes and optimisations for resolve_state_groups	2018-07-24 14:11:45 +01:00
Matthew Hodgson	1a01a5b964	clarify comment on p_ids	2018-07-24 14:03:15 +01:00
Matthew Hodgson	eb1d911ab7	rather than adding ll_ids, remove them from p_ids	2018-07-24 13:40:49 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Erik Johnston	536bc63a4e	Merge branch 'develop' into erikj/client_apis_move	2018-07-24 09:57:05 +01:00
Richard van der Hoff	cf2d15c6a9	another couple of logcontext leaks	2018-07-24 00:57:48 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	c1f80effbe	Handle delta_ids being None in _update_context_for_auth_events ... it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with https://github.com/matrix-org/synapse/issues/3364)	2018-07-23 22:06:50 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Erik Johnston	0faa3223cd	Fix missing attributes on workers. ... This was missed during the transition from attribute to getter for getting state from context.	2018-07-23 16:28:00 +01:00
Erik Johnston	0b0b24cb82	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move	2018-07-23 13:21:15 +01:00
Erik Johnston	027bc01a1b	Add support for updating state	2018-07-23 13:17:25 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Amber Brown	3132b89f12	Make the rest of the .iterwhatever go away (#3562)	2018-07-21 15:47:18 +10:00
Erik Johnston	5c88bb722f	Move PaginationHandler to its own file	2018-07-20 15:32:23 +01:00
Erik Johnston	0ecf68aedc	Move check_in_room_or_world_readable to Auth	2018-07-20 15:30:59 +01:00
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	2018-07-20 13:11:43 +01:00
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	2018-07-20 12:50:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Erik Johnston	bacdf0cbf9	Move RoomContextHandler out of Handlers ... This is in preparation for moving GET /context/ to a worker	2018-07-18 15:33:03 +01:00
Erik Johnston	8cb8df55e9	Split MessageHandler into read only and writers ... This will let us call the read only parts from workers, and so be able to move some APIs off of master, e.g. the `/state` API.	2018-07-18 15:33:03 +01:00
Richard van der Hoff	dab00faa83	Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks ... Remove event re-signing hacks	2018-07-18 12:46:27 +01:00
Richard van der Hoff	9c04b4abf9	Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server ... Refactor and optimze filter_events_for_server	2018-07-17 14:01:39 +01:00
Richard van der Hoff	ea69d35651	Move filter_events_for_server out of FederationHandler ... for easier unit testing.	2018-07-16 13:06:24 +01:00
Matthew Hodgson	ba22b6a456	typo	2018-07-13 12:03:39 +01:00
Matthew Hodgson	ea752bdd99	s/becuase/because/g	2018-07-10 17:58:18 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Richard van der Hoff	a4ab491371	Merge branch 'develop' into rav/drop_re_signing_hacks	2018-07-04 07:13:38 +01:00
Erik Johnston	e3b4043800	Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking ... Check the state of prev_events a bit more thoroughly when coming over federation	2018-06-29 13:55:02 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Amber Brown	99800de63d	try and clean up	2018-06-27 11:40:27 +01:00
Amber Brown	f03a5d1a17	pep8	2018-06-27 11:38:14 +01:00
Amber Brown	a7ecf34b70	cleanups	2018-06-27 11:36:03 +01:00
Amber Brown	77078d6c8e	handle federation not telling us about prev_events	2018-06-27 11:27:32 +01:00
Matthew Hodgson	c7f6b420ae	Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api ... add GDPR erase param to deactivate API	2018-06-26 10:43:14 +01:00
Matthew Hodgson	1e788db430	add GDPR erase param to deactivate API	2018-06-26 10:26:54 +01:00
Erik Johnston	d72fb9a448	Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts ... allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 20:14:18 +01:00
Erik Johnston	df48f7ef37	Actually fix it	2018-06-25 20:03:41 +01:00
Erik Johnston	a0e8a53c6d	Comment	2018-06-25 19:57:38 +01:00
Erik Johnston	7bdc5c8fa3	Fix bug with assuming wrong type	2018-06-25 19:56:02 +01:00
Erik Johnston	ea7a9c0483	Add fast path to _filter_events_for_server ... Most rooms have a trivial history visibility like "shared" or "world_readable", especially large rooms, so lets not bother getting the full membership of those rooms in that case.	2018-06-25 19:49:13 +01:00
Matthew Hodgson	0269367f18	allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 17:56:10 +01:00
Matthew Hodgson	784189b1f4	typos	2018-06-25 17:37:16 +01:00
Matthew Hodgson	6eb861b67f	typo	2018-06-25 17:37:16 +01:00
Erik Johnston	33fdcfa957	Merge pull request #3441 from matrix-org/erikj/redo_erasure ... Fix user erasure and re-enable	2018-06-25 14:37:01 +01:00
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	2018-06-25 14:08:28 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Erik Johnston	28ddc6cfbe	Also log number of events for serach context	2018-06-22 15:42:11 +01:00
Erik Johnston	4b4cec3989	Add some logging to search queries	2018-06-22 15:42:11 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	b4a5d767a9	Merge pull request #3428 from matrix-org/erikj/persisted_pdu ... Simplify get_persisted_pdu	2018-06-22 14:47:43 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	02bfc581f8	Merge pull request #3399 from costacruise/master ... Add error code to room creation error	2018-06-20 17:26:25 +01:00
Michael Wagner	19cd3120ec	Add error code to room creation error ... This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom	2018-06-14 14:08:40 +02:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Michael Telatynski	94700e55fa	if inviter_display_name == ""||None then default to inviter MXID ... to prevent email invite from "None"	2018-06-13 10:31:01 +01:00
Richard van der Hoff	b6faef2ad7	Filter out erased messages ... Redact any messges sent by erased users.	2018-06-12 09:53:18 +01:00
Richard van der Hoff	f1023ebf4b	mark accounts as erased when requested	2018-06-12 09:53:18 +01:00
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Richard van der Hoff	c6b1441c52	Fix event filtering in get_missing_events handler	2018-06-08 14:15:31 +01:00
Richard van der Hoff	0834b49c6a	Fix event filtering in get_missing_events handler	2018-06-08 11:34:46 +01:00
Richard van der Hoff	8503dd0047	Remove event re-signing hacks ... These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.	2018-06-07 16:08:29 +01:00
David Baker	d62162bbec	doc fixes	2018-06-05 18:09:13 +01:00
David Baker	6a29e815fc	Fix comment	2018-06-04 12:01:23 +01:00
David Baker	e44150a6de	Missing yield	2018-06-04 12:01:13 +01:00
David Baker	f731e42baf	docstring	2018-06-04 12:00:51 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Matthew Hodgson	5f6122fe10	more comments	2018-06-04 00:08:52 +03:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Richard van der Hoff	34b85df7f5	Update some comments and docstrings in SyncHandler	2018-05-29 22:31:18 +01:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Richard van der Hoff	41921ac01b	Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room ... Let users leave the server notice room after joining	2018-05-25 11:15:45 +01:00
Richard van der Hoff	757ed27258	Let users leave the server notice room after joining ... They still can't reject invites, but we let them leave it.	2018-05-25 11:07:21 +01:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	7a1af504d7	Remove users from user directory on deactivate	2018-05-24 15:59:58 +01:00
Erik Johnston	f72d5a44d5	Merge pull request #3261 from matrix-org/erikj/pagination_fixes ... Fix federation backfill bugs	2018-05-24 14:52:03 +01:00
Erik Johnston	68399fc4de	Merge pull request #3267 from matrix-org/erikj/iter_filter ... Use iter* methods for _filter_events_for_server	2018-05-24 14:44:57 +01:00
David Baker	9700d15611	pep8	2018-05-24 11:23:15 +01:00
David Baker	a21a41bad7	comment	2018-05-24 11:19:59 +01:00
David Baker	b3bff53178	Unbind 3pids when they're deleted too	2018-05-24 11:08:05 +01:00
Richard van der Hoff	a0b3946fe2	Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri	2018-05-23 16:06:03 +01:00
Richard van der Hoff	3ff6f50eac	Use the localpart in the consent uri ... ... because it's shorter.	2018-05-23 15:28:23 +01:00
Richard van der Hoff	82191b08f6	Support for putting %(consent_uri)s in messages ... Make it possible to put the URI in the error message and the server notice that get sent by the server	2018-05-23 15:24:31 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Erik Johnston	5aaa3189d5	s/values/itervalues/	2018-05-23 10:13:05 +01:00
Erik Johnston	0a4bca4134	Use iter* methods for _filter_events_for_server	2018-05-23 10:04:23 +01:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Erik Johnston	e85b5a0ff7	Use iter* methods	2018-05-22 19:02:48 +01:00
Erik Johnston	586b66b197	Fix that states is a dict of dicts	2018-05-22 19:02:36 +01:00
Richard van der Hoff	08bfc48abf	custom error code for not leaving server notices room	2018-05-22 17:27:27 +01:00
David Baker	0a078026ea	comment typo	2018-05-22 17:14:06 +01:00
Amber Brown	8f5a688d42	cleanups, self-registration	2018-05-22 10:56:03 -05:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Erik Johnston	cb2a2ad791	get_domains_from_state returns list of tuples	2018-05-22 16:23:39 +01:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Richard van der Hoff	6d9dc67139	Merge pull request #3232 from matrix-org/rav/server_notices_room ... Infrastructure for a server notices room	2018-05-18 11:28:04 +01:00
Richard van der Hoff	67af392712	Merge pull request #3233 from matrix-org/rav/remove_dead_code ... Remove unused `update_external_syncs`	2018-05-18 11:22:43 +01:00
Richard van der Hoff	26305788fe	Make sure we reject attempts to invite the notices user	2018-05-18 11:18:39 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	88d3405332	fix missing yield for server_notices_room	2018-05-17 18:33:45 +01:00
Richard van der Hoff	d43d480d86	Remove unused update_external_syncs ... This method isn't used anywhere. Burninate it.	2018-05-17 18:22:19 +01:00
Richard van der Hoff	fed62e21ad	Infrastructure for a server notices room ... Server Notices use a special room which the user can't dismiss. They are created on demand when some other bit of the code calls send_notice. (This doesn't actually do much yet becuse we don't call send_notice anywhere)	2018-05-17 17:58:25 +01:00
Richard van der Hoff	c46367d0d7	Move RoomCreationHandler out of synapse.handlers.Handlers ... Handlers is deprecated nowadays, so let's move this out before I add a new dependency on it. Also fix the docstrings on create_room.	2018-05-17 09:08:42 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
David Baker	6b49628e3b	Catch failure to part user from room	2018-05-10 12:23:53 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
David Baker	4d298506dd	Oops, don't call function passed to run_in_background	2018-05-10 11:57:13 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	d26bec8a43	Add comment to sync as to why code path is split	2018-05-09 15:56:07 +01:00
Erik Johnston	fcf55f2255	Fix returned token is no longer a tuple	2018-05-09 15:43:00 +01:00
David Baker	294e9a0c9b	Prefix internal functions	2018-05-09 15:10:37 +01:00
David Baker	52281e4c54	Indent fail	2018-05-09 15:06:16 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
David Baker	bf98fa0864	Part user from rooms on account deactivate ... This implements this very crudely: this probably isn't viable because parting a user from all their rooms could take a long time, and if the HS gets restarted in that time the process will be aborted.	2018-05-08 15:58:35 +01:00
Richard van der Hoff	ca7211104e	Merge branch 'release-v0.28.1' into develop	2018-05-01 18:16:57 +01:00
Richard van der Hoff	33f469ba19	Apply some limits to depth to counter abuse ... * When creating a new event, cap its depth to 2^63 - 1 * When receiving events, reject any without a sensible depth As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI	2018-05-01 17:54:19 +01:00
Richard van der Hoff	63ae5cbf34	Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn ... Remove redundant call to preserve_fn	2018-04-30 10:23:59 +01:00
Richard van der Hoff	bc2017a594	Merge pull request #3153 from NotAFile/py3-httplib ... move httplib import to six	2018-04-30 01:26:42 +01:00
Richard van der Hoff	683149c1f9	Merge pull request #3151 from NotAFile/py3-xrange-1 ... Move more xrange to six	2018-04-30 01:20:06 +01:00
Richard van der Hoff	3b0e431c82	Merge pull request #3150 from NotAFile/py3-listcomp-yield ... Don't yield in list comprehensions	2018-04-30 01:11:41 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Richard van der Hoff	af3cc50511	Remove redundant call to preserve_fn ... submit_event_for_as doesn't return a deferred anyway, so this is pointless.	2018-04-30 00:48:36 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Adrian Tschira	94f4d7f49e	move httplib import to six	2018-04-28 13:43:34 +02:00
Adrian Tschira	cdb4647a80	Don't yield in list comprehensions ... I've tried to grep for more of this with no success. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:36:30 +02:00
Richard van der Hoff	453adf00b6	pep8; remove spurious import	2018-04-27 14:32:08 +01:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	41d4b07a53	Merge pull request #3142 from matrix-org/rav/reraise ... reraise exceptions more carefully	2018-04-27 12:16:19 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Richard van der Hoff	6493b22b42	reraise exceptions more carefully ... We need to be careful (under python 2, at least) that when we reraise an exception after doing some error handling, we actually reraise the original exception rather than anything that might have been raised (and handled) during the error handling.	2018-04-27 11:40:06 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Richard van der Hoff	77ebef9d43	Merge pull request #3118 from matrix-org/rav/reject_prev_events ... Reject events which have lots of prev_events	2018-04-23 17:51:38 +01:00
Richard van der Hoff	9b9c38373c	Remove spurious param	2018-04-23 12:00:06 +01:00
Richard van der Hoff	8dc4a6144b	Merge pull request #3107 from NotAFile/py3-bool-nonzero ... add __bool__ alias to __nonzero__ methods	2018-04-20 15:43:39 +01:00
Richard van der Hoff	3de7d9fe99	accept stupid events over backfill	2018-04-20 11:41:03 +01:00
Richard van der Hoff	11a67b7c9d	Merge pull request #3093 from matrix-org/rav/response_cache_wrap ... Refactor ResponseCache usage	2018-04-20 11:31:17 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	1f4b498b73	Add some comments	2018-04-18 00:15:36 +01:00
Richard van der Hoff	e585228860	Check events on backfill too	2018-04-18 00:06:42 +01:00
Richard van der Hoff	9b7794262f	Reject events which have too many auth_events or prev_events ... ... this should protect us from being dossed by people making silly events (deliberately or otherwise)	2018-04-18 00:06:42 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	f63ff73c7f	add __bool__ alias to __nonzero__ methods ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:40:47 +02:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	d5c74b9f6c	Merge pull request #3092 from matrix-org/rav/response_cache_metrics ... Add metrics for ResponseCache	2018-04-12 12:59:36 +01:00
Erik Johnston	19ceb4851f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag	2018-04-12 11:36:07 +01:00
Erik Johnston	f67e906e18	Set all metrics at the same time	2018-04-12 11:18:19 +01:00
Erik Johnston	971059a733	Merge pull request #3088 from matrix-org/erikj/as_parallel ... Send events to ASes concurrently	2018-04-12 10:42:36 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Erik Johnston	121591568b	Send events to ASes concurrently	2018-04-11 09:56:00 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Erik Johnston	eaa2ebf20b	Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends ... Limit concurrent event sends for a room	2018-04-10 16:43:58 +01:00
Erik Johnston	f8e8ec013b	Note why we're limiting concurrent event sends	2018-04-10 14:00:46 +01:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	d2c6f4d626	Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin ... Return a 404 rather than a 500 on rejoining empty rooms	2018-04-09 17:32:36 +01:00
Richard van der Hoff	f3ef60662f	Return a 404 rather than a 500 on rejoining empty rooms ... Filter ourselves out of the server list before checking for an empty remote host list, to fix 500 error Fixes #2141	2018-04-09 12:56:22 +01:00
Erik Johnston	e5082494eb	Limit concurrent event sends for a room	2018-04-09 12:07:39 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00