10688 Commits

Author	SHA1	Message	Date
Erik Johnston	4fb44fb5b9	Expose DataStore._get_events as get_events_as_list ... This is in preparation for reaction work which requires it.	2019-05-14 13:37:44 +01:00
Erik Johnston	a80e6b53f9	Newsfile	2019-05-14 13:12:23 +01:00
Erik Johnston	b54b03f9e1	Allow client event serialization to be async	2019-05-14 11:58:01 +01:00
Amber Brown	df2ebd75d3	Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171)	2019-05-13 15:01:14 -05:00
Andrew Morgan	5a4b328f52	Add ability to blacklist ip ranges for federation traffic (#5043)	2019-05-13 19:05:06 +01:00
Brendan Abolivier	2e1129b5f7	0.99.4rc1	2019-05-13 16:11:21 +01:00
David Baker	516a5fb64b	Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only	2019-05-13 15:54:25 +01:00
David Baker	9e99143c47	Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only	2019-05-13 15:37:03 +01:00
Brendan Abolivier	bb93757b32	Fix CI after new release of isort	2019-05-13 15:19:44 +01:00
Andrew Morgan	2f48c4e1ae	URL preview blacklisting fixes (#5155) ... Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.	2019-05-10 10:32:44 -07:00
David Baker	04299132af	Re-order flows so that email auth is done last ... It's more natural for the user if the bit that takes them away from the registration flow comes last. Adding the dummy stage allows us to do the stages in this order without the ambiguity.	2019-05-10 13:58:03 +01:00
David Baker	9c61dce3c8	Comment	2019-05-10 11:14:55 +01:00
David Baker	8714ff6d51	Add a DUMMY stage to captcha-only registration flow ... This allows the client to complete the email last which is more natual for the user. Without this stage, if the client would complete the recaptcha (and terms, if enabled) stages and then the registration request would complete because you've now completed a flow, even if you were intending to complete the flow that's the same except has email auth at the end. Adding a dummy auth stage to the recaptcha-only flow means it's always unambiguous which flow the client was trying to complete. Longer term we should think about changing the protocol so the client explicitly says which flow it's trying to complete. vector-im/riot-web#9586	2019-05-10 11:09:53 +01:00
David Baker	c2bb7476c9	Revert 085ae346ac ... Accidentally went straight to develop	2019-05-10 11:08:01 +01:00
David Baker	085ae346ac	Add a DUMMY stage to captcha-only registration flow ... This allows the client to complete the email last which is more natual for the user. Without this stage, if the client would complete the recaptcha (and terms, if enabled) stages and then the registration request would complete because you've now completed a flow, even if you were intending to complete the flow that's the same except has email auth at the end. Adding a dummy auth stage to the recaptcha-only flow means it's always unambiguous which flow the client was trying to complete. Longer term we should think about changing the protocol so the client explicitly says which flow it's trying to complete. https://github.com/vector-im/riot-web/issues/9586	2019-05-10 10:52:24 +01:00
Richard van der Hoff	130f932cbc	Run black on per_destination_queue ... ... mostly to fix pep8 fails	2019-05-09 16:27:02 +01:00
Quentin Dufour	11ea16777f	Limit the number of EDUs in transactions to 100 as expected by receiver (#5138) ... Fixes #3951.	2019-05-09 11:01:41 +01:00
Matthew Hodgson	c0e0740bef	add options to require an access_token to GET /profile and /publicRooms on CS API (#5083) ... This commit adds two config options: * `restrict_public_rooms_to_local_users` Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API. * `require_auth_for_profile_requests` When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301. MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though. Groups have been intentionally omitted from this commit.	2019-05-08 18:26:56 +01:00
Erik Johnston	c8c069db92	Merge pull request #5037 from matrix-org/erikj/limit_inflight_dns ... Limit in flight DNS requests	2019-05-08 17:11:03 +01:00
Brendan Abolivier	1473058b5e	Do checks on aliases for incoming m.room.aliases events (#5128) ... Follow-up to #5124 Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.	2019-05-08 17:01:30 +01:00
Erik Johnston	de655e669a	Merge pull request #5104 from matrix-org/erikj/ratelimit_3pid_invite ... Ratelimit 3pid invites	2019-05-07 10:12:49 +01:00
Richard van der Hoff	59e2d2694d	Remove the requirement to authenticate for /admin/server_version. (#5122) ... This endpoint isn't much use for its intended purpose if you first need to get yourself an admin's auth token. I've restricted it to the `/_synapse/admin` path to make it a bit easier to lock down for those concerned about exposing this information. I don't imagine anyone is using it in anger currently.	2019-05-07 09:29:30 +01:00
Richard van der Hoff	836d3adcce	Merge branch 'master' into develop	2019-05-03 19:25:01 +01:00
Richard van der Hoff	fa21455e08	0.99.3.2	2019-05-03 18:56:24 +01:00
Richard van der Hoff	e3281d7d26	pin urllib3 to <1.25	2019-05-03 18:33:10 +01:00
Richard van der Hoff	863ec09622	0.99.3.1	2019-05-03 16:03:24 +01:00
Richard van der Hoff	a845abbf3a	Merge pull request #5134 from matrix-org/rav/url_preview_blacklist ... Blacklist 0.0.0.0 and :: by default for URL previews	2019-05-03 15:59:20 +01:00
Richard van der Hoff	1565ebec2c	more config comment updates	2019-05-03 15:50:59 +01:00
Richard van der Hoff	1acfb9e9f0	Merge pull request #5133 from matrix-org/rav/systemrandom ... Use SystemRandom for token generation.	2019-05-03 15:39:30 +01:00
Richard van der Hoff	1a7104fde3	Blacklist 0.0.0.0 and :: by default for URL previews	2019-05-03 15:35:49 +01:00
Richard van der Hoff	247dc1bd0b	Use SystemRandom for token generation	2019-05-03 13:02:55 +01:00
Erik Johnston	176f31c2e3	Rate limit early	2019-05-02 15:23:08 +01:00
Richard van der Hoff	12f9d51e82	Add admin api for sending server_notices (#5121)	2019-05-02 11:59:16 +01:00
Brendan Abolivier	c193b39134	Merge pull request #5124 from matrix-org/babolivier/aliases ... Add some limitations to alias creation	2019-05-02 11:22:40 +01:00
Brendan Abolivier	84196cb231	Add some limitations to alias creation	2019-05-02 11:05:11 +01:00
Richard van der Hoff	0836cbb9f5	Factor out an "assert_requester_is_admin" function (#5120) ... Rather than copying-and-pasting the same four lines hundreds of times	2019-05-02 10:45:52 +01:00
Richard van der Hoff	f203c98794	fix examples	2019-05-01 17:49:56 +01:00
Richard van der Hoff	40e576e29c	Move admin api impl to its own package ... It doesn't really belong under rest/client/v1 any more.	2019-05-01 15:44:30 +01:00
Richard van der Hoff	8e9ca83537	Move admin API to a new prefix	2019-05-01 15:44:30 +01:00
Richard van der Hoff	579b637b6c	Move admin API away from ClientV1RestServlet	2019-05-01 15:16:04 +01:00
Travis Ralston	8c5b1e30d4	Add a default .m.rule.tombstone push rule (#4867) ... * Add a default .m.rule.tombstone push rule In support of MSC1930: https://github.com/matrix-org/matrix-doc/pull/1930 * changelog * Appease the changelog linter	2019-04-29 15:40:31 -06:00
Richard van der Hoff	b31cc1c613	Merge pull request #5100 from matrix-org/rav/verification_hackery ... Improve logging when event-signature checking fails	2019-04-29 13:19:32 +01:00
Erik Johnston	d6118c5be6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ratelimit_3pid_invite	2019-04-26 18:14:23 +01:00
Erik Johnston	28a81ed62f	Ratelimit 3pid invites ... We do ratelimit sending the 3PID invite events, but that happens after spamming the identity server.	2019-04-26 18:06:25 +01:00
Richard van der Hoff	bd0d45ca69	Fix infinite loop in presence handler ... Fixes #5102	2019-04-26 11:14:49 +01:00
Richard van der Hoff	837d7f85a9	more logging improvements	2019-04-25 22:17:59 +01:00
Richard van der Hoff	fd8fb32bdd	remove extraneous exception logging	2019-04-25 22:02:03 +01:00
Richard van der Hoff	7ca638c761	Clarify logging when PDU signature checking fails	2019-04-25 20:55:12 +01:00
Andrew Morgan	6824ddd93d	Config option for verifying federation certificates (MSC 1711) (#4967)	2019-04-25 14:22:49 +01:00
Michael Kaye	788163e204	Remove log error for .well-known/matrix/client (#4972)	2019-04-24 17:44:06 +01:00
Katie Wolfe	7e07dc429f	Lint ... I probably should've just run autopep8 in the first place... Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:43:18 -04:00
Amber Brown	6b2b9a58c4	Prevent "producer not unregistered" message (#5009)	2019-04-24 17:37:32 +01:00
Katie Wolfe	b3e5db402d	Clean up code ... Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:55 -04:00
Katie Wolfe	5d3ed79944	Show heroes if room name or canonical alias are empty ... Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:53 -04:00
Katie Wolfe	7f025eb425	Show heroes if room name or canonical alias are empty ... Fixes #4194 Signed-off-by: Katie Wolfe <katie@dnaf.moe>	2019-04-24 12:26:29 -04:00
Brendan Abolivier	f8826d31cd	Don't crash on lack of expiry templates	2019-04-18 14:50:05 +01:00
Erik Johnston	ca90336a69	Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration	2019-04-17 19:44:40 +01:00
Brendan Abolivier	eaf41a943b	Add management endpoints for account validity	2019-04-17 19:34:45 +01:00
Brendan Abolivier	91934025b9	Merge pull request #5047 from matrix-org/babolivier/account_expiration ... Send out emails with links to extend an account's validity period	2019-04-17 14:57:39 +01:00
Brendan Abolivier	20f0617e87	Send out emails with links to extend an account's validity period	2019-04-17 14:42:20 +01:00
Brendan Abolivier	49ff74da9b	Merge pull request #5071 from matrix-org/babolivier/3pid-check ... Make sure we're not registering the same 3pid twice	2019-04-17 14:37:42 +01:00
Brendan Abolivier	600ec04739	Make sure we're not registering the same 3pid twice	2019-04-17 14:23:01 +01:00
Erik Johnston	fd2fcb817c	Merge pull request #5070 from matrix-org/erikj/postpath ... Remove usage of request.postpath	2019-04-17 09:03:26 +01:00
Erik Johnston	a1eb4c6d2f	Merge pull request #5065 from matrix-org/erikj/fix_versions ... VersionRestServlet doesn't take a param	2019-04-16 17:52:36 +01:00
Erik Johnston	ad010f6306	Remove usage of request.postpath ... This is an undocumented variable in twisted, and relies on the servlet being mounted in the right way. This also breaks getting push rules on workers.	2019-04-16 17:42:50 +01:00
Silke Hofstra	a137f4eac0	Add systemd-python to optional dependencies (#4339) ... Using systemd-python allows for logging to the systemd journal, as is documented in: `synapse/contrib/systemd/log_config.yaml`. Signed-off-by: Silke Hofstra <silke@slxh.eu>	2019-04-16 20:41:17 +10:00
Erik Johnston	38642614cf	VersionRestServlet doesn't take a param	2019-04-15 19:39:47 +01:00
Erik Johnston	6e27a8620f	Merge pull request #5063 from matrix-org/erikj/move_endpoints ... Move some rest endpoints to client reader	2019-04-15 18:55:01 +01:00
Erik Johnston	ec638a1602	Only handle GET requests for /push_rules	2019-04-15 18:51:48 +01:00
Erik Johnston	d5adf297e6	Move some rest endpoints to client reader	2019-04-15 17:21:03 +01:00
Andrew Morgan	caa76e6021	Remove periods from copyright headers (#5046)	2019-04-11 17:08:13 +01:00
Erik Johnston	c132c8e505	Handle the case of get_missing_events failing ... Currently if a call to `/get_missing_events` fails we log an exception and stop processing the top level event we received over federation. Instead let's try and handle it sensibly given it is a somewhat expected failure mode.	2019-04-10 10:39:54 +01:00
Erik Johnston	e446921def	Merge pull request #5033 from matrix-org/erikj/fix_schema_delta ... Fix schema upgrade when dropping tables	2019-04-10 10:22:35 +01:00
Richard van der Hoff	329688c161	Fix disappearing exceptions in manhole. (#5035) ... Avoid sending syntax errors from the manhole to sentry.	2019-04-10 07:23:48 +01:00
Erik Johnston	a0fc256d65	Limit in flight DNS requests ... This is to work around a bug in twisted where a large number of concurrent DNS requests cause it to tight loop forever. c.f. https://twistedmatrix.com/trac/ticket/9620#ticket	2019-04-09 17:23:42 +01:00
Brendan Abolivier	bfc8fdf1fc	Merge pull request #5027 from matrix-org/babolivier/account_expiration ... Add time-based account expiration	2019-04-09 17:02:41 +01:00
Brendan Abolivier	747aa9f8ca	Add account expiration feature	2019-04-09 16:46:04 +01:00
Erik Johnston	50d2a3059d	Fix schema upgrade when dropping tables ... We need to drop tables in the correct order due to foreign table constraints (on `application_services`), otherwise the DROP TABLE command will fail. Introduced in #4992.	2019-04-09 14:39:18 +01:00
Richard van der Hoff	644b86677f	Merge pull request #5030 from matrix-org/rav/rewrite_g_s_v_k ... Rewrite Datastore.get_server_verify_keys	2019-04-09 11:30:23 +01:00
Richard van der Hoff	4abf5aa81a	Bump psycopg requirement (#5032)	2019-04-09 11:29:50 +01:00
Richard van der Hoff	18b69be00f	Rewrite Datastore.get_server_verify_keys ... Rewrite this so that it doesn't hammer the database.	2019-04-09 00:00:10 +01:00
Richard van der Hoff	f50efcb65d	Replace SlavedKeyStore with a shim ... since we're pulling everything out of KeyStore anyway, we may as well simplify it.	2019-04-08 23:59:07 +01:00
Richard van der Hoff	f88a9e6323	Remove redundant merged_keys dict ... There's no point in collecting a merged dict of keys: it is sufficient to consider just the new keys which have been fetched by the most recent key_fetch_fns.	2019-04-08 22:36:18 +01:00
Richard van der Hoff	3352baac4b	Remove unused server_tls_certificates functions (#5028) ... These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again.	2019-04-08 21:50:18 +01:00
Neil Johnson	b25e387c0d	add context to phonehome stats (#5020) ... add context to phonehome stats	2019-04-08 15:47:39 +01:00
Richard van der Hoff	67d7b44784	Merge pull request #5024 from matrix-org/rav/record_correct_server_in_serverkeys ... Fix from_server buglet in get_keys_from_perspectives	2019-04-08 15:40:37 +01:00
Neil Johnson	2d951686a7	drop tables listed in #1830 (#4992) ... Tables dropped: * application_services, * application_services_regex, * transaction_id_to_pdu, * stats_reporting * current_state_resets * event_content_hashes * event_destinations * event_edge_hashes * event_signatures * feedback * room_hosts * state_forward_extremities	2019-04-08 15:37:26 +01:00
Richard van der Hoff	7d2a0c848e	Fix from_server buglet in get_keys_from_perspectives ... make sure we store the name of the server the keys came from, rather than the origin server, after doing a fetch-from-perspectives.	2019-04-08 12:51:16 +01:00
Richard van der Hoff	7fc1e17f4c	Merge pull request #5001 from matrix-org/rav/keyring_cleanups ... Cleanups in the Keyring	2019-04-08 12:47:09 +01:00
Richard van der Hoff	6ae9361510	Hoist server_name check out of process_v2_response ... It's easier to check it in the caller than to complicate the interface with an extra param.	2019-04-04 19:12:54 +01:00
Richard van der Hoff	ef27d434d1	Clean up Keyring.process_v2_response ... Make this just return the key dict, rather than a single-entry dict mapping the server name to the key dict. It's easy for the caller to get the server name from from the response object anyway.	2019-04-04 19:12:54 +01:00
Richard van der Hoff	b43d9a920b	Fix docstring on get_server_keys_json	2019-04-04 18:54:03 +01:00
Brendan Abolivier	8e85493b0c	Add config option to block users from looking up 3PIDs (#5010)	2019-04-04 17:25:47 +01:00
Amber Brown	a33a5abc4c	Clean up the database pagination code (#5007) ... * rewrite & simplify * changelog * cleanup potential sql injection	2019-04-05 00:21:16 +11:00
Erik Johnston	616e6a10bd	Merge pull request #5002 from matrix-org/erikj/delete_group ... Add delete group admin API	2019-04-04 14:15:41 +01:00
Andrew Morgan	db265f0642	Prevent kicking users who aren't in the room (#4999) ... Prevent kick events from succeeding if the user is not currently in the room.	2019-04-04 13:05:51 +01:00
Marcel Krüger	9f5d206c4a	Avoid redundant URL encoding (#4555) ... * Do not double encode fallback redirect URL Signed-off-by: Marcel Fabian Krüger <zauguin@gmail.com>	2019-04-04 12:05:56 +01:00
Erik Johnston	c192bf8970	Add admin API for group deletion	2019-04-03 16:29:52 +01:00
Erik Johnston	4a2e13631d	Add functions to delete a group	2019-04-03 16:29:52 +01:00
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998)	2019-04-03 14:32:20 +01:00
Neil Johnson	e8419554ff	Remove presence lists (#4989) ... Remove presence list support as per MSC 1819	2019-04-03 11:11:15 +01:00
Erik Johnston	8f549c1177	Merge pull request #4982 from matrix-org/erikj/msc1915 ... Implement MSC1915 - 3PID unbind APIs	2019-04-03 11:07:09 +01:00
Amber Brown	7efd1d87c2	Run black on the rest of the storage module (#4996)	2019-04-03 10:07:29 +01:00
Erik Johnston	3039d61baf	Merge pull request #4991 from matrix-org/erikj/stagger_push_startup ... Make starting pushers faster during start up	2019-04-02 18:23:32 +01:00
Andrew Morgan	66e78700a2	Transfer related groups on room upgrade (#4990) ... Transfers the m.room.related_groups state event on room upgrade.	2019-04-02 17:15:24 +01:00
Richard van der Hoff	8530090b16	Add config.signing_key_path. (#4974) ... As requested by @andrewshadura	2019-04-02 16:59:27 +01:00
Erik Johnston	5bec8d660d	Make starting pushers faster during start up ... We start all pushers on start up and immediately start a background process to fetch push to send. This makes start up incredibly painful when dealing with many pushers. Instead, let's do a quick fast DB check to see if there *may* be push to send and only start the background processes for those pushers. We also stagger starting up and doing those checks so that we don't try and handle all pushers at once.	2019-04-02 16:59:13 +01:00
Richard van der Hoff	297bf2547e	Fix sync bug when accepting invites (#4956) ... Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.	2019-04-02 12:42:39 +01:00
Erik Johnston	4ef5d17b96	Correctly handle id_server param	2019-04-02 11:20:09 +01:00
Erik Johnston	24232514bf	Remove threepid binding if id server returns 400/404/501	2019-04-02 11:20:09 +01:00
Erik Johnston	c75e2017f1	Fixup docstrings	2019-04-02 11:20:06 +01:00
Erik Johnston	862d6e5ba5	Add unbind API to /r0 as it is now stabalised	2019-04-01 15:25:19 +01:00
Erik Johnston	3715c124b3	Grandfather in existing user threepids ... We assume, as we did before, that users bound their threepid to one of the trusted identity servers. So we simply fill the new table with all threepids in `user_threepids` joined with the trusted identity servers.	2019-04-01 15:25:19 +01:00
Erik Johnston	057715aaa2	Allowing specifying IS to use in unbind API. ... By default the homeserver will use the identity server used during the binding of the 3PID to unbind the 3PID. However, we need to allow clients to explicitly ask the homeserver to unbind via a particular identity server, for the case where the 3PID was bound out of band from the homeserver. Implements MSC915.	2019-04-01 15:25:18 +01:00
Erik Johnston	9fbbc3d9e5	For unbind poke IS used during binding of 3PID ... This changes the behaviour from using the server specified trusted identity server to using the IS that used during the binding of the 3PID, if known. This is the behaviour specified by MSC1915.	2019-04-01 15:23:30 +01:00
Erik Johnston	1666c0696a	Track IS used to bind 3PIDs ... This will then be used to know which IS to default to when unbinding the threepid.	2019-04-01 15:23:01 +01:00
Neil Johnson	62988f73fd	Merge branch 'master' into develop	2019-04-01 14:08:53 +01:00
Neil Johnson	35442efb75	0.99.3	2019-04-01 12:49:03 +00:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969) ... Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Richard van der Hoff	902cdc63b6	Merge pull request #4955 from matrix-org/rav/merge_state_into_events ... Combine the CurrentStateDeltaStream into the EventStream	2019-03-28 18:32:13 +00:00
Richard van der Hoff	d688a51736	Merge pull request #4954 from matrix-org/rav/refactor_parse_row ... Refactors to replication stream row update/parsing	2019-03-28 18:31:17 +00:00
Neil Johnson	c7296bcb98	remove log line for password (#4965) ... Remove log line for password.	2019-03-28 17:38:01 +00:00
Andrew Morgan	7a91b9d81c	Allow password providers to bind emails (#4947) ... This PR allows password provider modules to bind email addresses when a user is registering and is motivated by matrix-org/matrix-synapse-ldap3#58	2019-03-28 15:48:07 +00:00
Erik Johnston	248014379e	Merge pull request #4942 from matrix-org/erikj/fix_presence ... Use event streams to calculate presence	2019-03-28 14:38:31 +00:00
Erik Johnston	4e5f0f7ca0	Use an assert	2019-03-28 14:05:05 +00:00
Erik Johnston	40e56997bc	Review comments	2019-03-28 13:48:41 +00:00
Richard van der Hoff	d035d62f6b	Merge remote-tracking branch 'origin/develop' into rav/refactor_parse_row	2019-03-28 13:45:14 +00:00
Richard van der Hoff	4eeb2c2f07	Merge pull request #4953 from matrix-org/rav/refactor_replication_streams ... Split up replication.tcp.streams into smaller files	2019-03-28 13:43:25 +00:00
Amber Brown	2e060774ad	Run black on some storage modules that the stats branch touches (#4959)	2019-03-29 00:37:16 +11:00
Richard van der Hoff	4b91c313a9	Combine the CurrentStateDeltaStream into the EventStream	2019-03-27 22:07:05 +00:00
Richard van der Hoff	1f6d6f918a	Make EventStream rows have a type ... ... as a precursor to combining it with the CurrentStateDelta stream.	2019-03-27 22:07:05 +00:00
Richard van der Hoff	015b3622eb	Skip building a ROW_TYPE when building updates ... We're about to turn it straight into a JSON object anyway so building a ROW_TYPE is a bit pointless, and reduces flexibility in the update_function.	2019-03-27 21:58:03 +00:00
Richard van der Hoff	f570916a3e	Add parse_row method to replication stream class ... This will allow individual stream classes to override how a row is parsed.	2019-03-27 21:32:33 +00:00
Richard van der Hoff	71dcb275f1	move FederationStream out to its own file	2019-03-27 21:13:14 +00:00
Richard van der Hoff	aa1e017864	move EventsStream out to its own file	2019-03-27 21:13:14 +00:00
Richard van der Hoff	a5798de067	Move replication.tcp.streams into a package	2019-03-27 21:13:14 +00:00
Richard van der Hoff	acaa18f7dd	Fix/improve some docstrings in the replication code. (#4949)	2019-03-27 21:12:36 +00:00
Erik Johnston	197fae1639	Use event streams to calculate presence ... Primarily this fixes a bug in the handling of remote users joining a room where the server sent out the presence for all local users in the room to all servers in the room. We also change to using the state delta stream, rather than the distributor, as it will make it easier to split processing out of the master process (as well as being more flexible). Finally, when sending presence states to newly joined servers we filter out old presence states to reduce the number sent. Initially we filter out states that are offline and have a last active more than a week ago, though this can be changed down the line. Fixes #3962	2019-03-27 13:41:36 +00:00
Neil Johnson	4aa914369b	bump version	2019-03-27 10:23:03 +00:00
Andrew Morgan	bbd244c7b2	Support 3PID login in password providers (#4931) ... Adds a new method, check_3pid_auth, which gives password providers the chance to allow authentication with third-party identifiers such as email or msisdn.	2019-03-26 17:48:30 +00:00
Amber Brown	903f04c21f	Use the state event amount for userdir import batching, not room count (#4944)	2019-03-27 02:49:28 +11:00
Andrew Morgan	4a125be138	Make federation endpoints more tolerant of trailing slashes v2 (#4935) ... Redo of https://github.com/matrix-org/synapse/pull/4840	2019-03-26 11:35:29 +00:00
Richard van der Hoff	8cbbedaa2b	Fix ClientReplicationStreamProtocol.__str__ (#4929) ... `__str__` depended on `self.addr`, which was absent from ClientReplicationStreamProtocol, so attempting to call str on such an object would raise an exception. We can calculate the peer addr from the transport, so there is no need for addr anyway.	2019-03-25 16:41:51 +00:00
Richard van der Hoff	9bde730ef8	Fix bug where read-receipts lost their timestamps (#4927) ... Make sure that they are sent correctly over the replication stream. Fixes: #4898	2019-03-25 16:38:05 +00:00
Amber Brown	ac396a0d32	Refactor out state delta handling into its own class (#4917)	2019-03-25 20:37:08 +11:00
Erik Johnston	5fee9d8067	Merge pull request #4869 from matrix-org/erikj/yaml_load ... Fix yaml warnings by using safe_load	2019-03-22 11:58:13 +00:00
Erik Johnston	3677548a82	Use yaml safe_load	2019-03-22 10:20:17 +00:00
Andrew Morgan	7bef97dfb7	Remove trailing slashes from outbound federation requests and retry on 400 (#4840) ... As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.	2019-03-21 15:07:28 +00:00
Andrew Morgan	b41c2eaadc	Clean up backoff_on_404 and metehod calls	2019-03-21 14:32:47 +00:00
Erik Johnston	01e6b405be	Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms ... Deny peeking into rooms that have been blocked	2019-03-21 14:07:17 +00:00
Erik Johnston	3959858eaa	Merge pull request #4904 from matrix-org/erikj/fix_shutdown ... Fixup shutdown room API	2019-03-21 11:24:42 +00:00
Erik Johnston	536a266520	Deny peeking into rooms that have been blocked	2019-03-21 11:20:13 +00:00
Erik Johnston	3ecec5ede2	Fix upsert	2019-03-21 10:21:15 +00:00
Erik Johnston	09f991a63d	Merge pull request #4896 from matrix-org/erikj/disable_room_directory ... Add option to disable search room lists	2019-03-21 10:16:54 +00:00
Erik Johnston	cd62981a6a	Revert spurious delete	2019-03-20 17:51:27 +00:00
Erik Johnston	8d8834d3e7	comment block_room	2019-03-20 17:49:56 +00:00
Erik Johnston	aa959a6c07	Use flags	2019-03-20 17:40:29 +00:00
Erik Johnston	7d47cc1305	Move requester check into assert_accepted_privacy_policy	2019-03-20 17:08:36 +00:00
Erik Johnston	72a14860ab	Gracefully handle failing to kick user	2019-03-20 16:54:21 +00:00
Erik Johnston	6b28890543	Log new room ID	2019-03-20 16:52:28 +00:00
Erik Johnston	74c46d81fa	Only require consent for events with an associated request ... There are a number of instances where a server or admin may puppet a user to join/leave rooms, which we don't want to fail if the user has not consented to the privacy policy. We fix this by adding a check to test if the requester has an associated access_token, which is used as a proxy to answer the question of whether the action is being done on behalf of a real request from the user.	2019-03-20 16:50:23 +00:00
Erik Johnston	67d618e111	Allow blocking a room multiple times	2019-03-20 16:50:05 +00:00
Erik Johnston	263f2c9ce1	Merge pull request #4895 from matrix-org/erikj/disable_user_search ... Add option to disable searching in the user dir	2019-03-20 16:47:15 +00:00
Amber Brown	4d53017432	Batching in the user directory import (#4900)	2019-03-21 03:06:36 +11:00
Richard van der Hoff	cdb8036161	Add a config option for torture-testing worker replication. (#4902) ... Setting this to 50 or so makes a bunch of sytests fail in worker mode.	2019-03-20 16:04:35 +00:00
Richard van der Hoff	a902d13180	Batch up outgoing read-receipts to reduce federation traffic. (#4890) ... Rate-limit outgoing read-receipts as per #4730.	2019-03-20 16:02:25 +00:00
Erik Johnston	cd8c5b91ad	Fix up sample config	2019-03-20 14:35:41 +00:00
Richard van der Hoff	ab20f85c59	Update synapse/config/user_directory.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2019-03-20 14:33:11 +00:00
Erik Johnston	cc197a61a1	Disable publishing to room list when its disabled	2019-03-20 14:30:36 +00:00
Erik Johnston	2c90422146	Pull out config option	2019-03-20 14:25:58 +00:00
Erik Johnston	7529038e66	Return before we log	2019-03-20 14:25:28 +00:00
Erik Johnston	926f29ea6d	Fix up config comments	2019-03-20 14:24:53 +00:00
Andrew Morgan	2150151abe	kwargs doesn't like commas on calling funcs either. TIL	2019-03-20 14:13:32 +00:00
Andrew Morgan	bb52a2e653	lint	2019-03-20 14:08:57 +00:00
Andrew Morgan	cd36a1283b	New test, fix issues	2019-03-20 14:00:39 +00:00
Andrew Morgan	c69df5d5d3	Fix comments. v0.99.2 -> v0.99.3	2019-03-20 11:27:18 +00:00
Andrew Morgan	551ea11559	Just return if not doing any trailing slash shennanigans	2019-03-20 11:07:36 +00:00
Erik Johnston	213c98c00a	Add option to disable search room lists ... This disables both local and remote room list searching.	2019-03-19 17:10:52 +00:00
Erik Johnston	320667a479	Add option to disable searching in the user dir ... We still populate it, as it can still be accessed via the admin API.	2019-03-19 16:40:19 +00:00
Richard van der Hoff	13bc1e0746	Use a regular HomeServerConfig object for unit tests ... Rather than using a Mock for the homeserver config, use a genuine HomeServerConfig object. This makes for a more realistic test, and means that we don't have to keep remembering to add things to the mock config every time we add a new config setting.	2019-03-19 11:44:43 +00:00
Richard van der Hoff	45c4e19c74	Merge remote-tracking branch 'origin/develop' into HEAD	2019-03-19 11:43:46 +00:00
Neil Johnson	88f0675967	fix test_auto_create_auto_join_where_no_consent (#4886)	2019-03-19 11:38:59 +00:00
Richard van der Hoff	0dbfae03f9	Enforce hs_disabled_message correctly ... Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests if there was no server_notices_mxid configured.	2019-03-19 11:30:54 +00:00
Erik Johnston	e9eeca1314	Fix user directory background update (#4887)	2019-03-19 22:13:53 +11:00
Richard van der Hoff	fd463b4f5d	Comment out most options in the generated config. (#4863) ... Make it so that most options in the config are optional, and commented out in the generated config. The reasons this is a good thing are as follows: * If we decide that we should change the default for an option, we can do so, and only those admins that have deliberately chosen to override that option will be stuck on the old setting. * It moves us towards a point where we can get rid of the super-surprising feature of synapse where the default settings for the config come from the generated yaml. * It makes setting up a test config for unit testing an order of magnitude easier (see forthcoming PR). * It makes the generated config more consistent, and hopefully easier for users to understand.	2019-03-19 10:06:40 +00:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864)	2019-03-19 04:50:24 +11:00
Andrew Morgan	a8ad39eec7	lint	2019-03-18 17:47:39 +00:00
Andrew Morgan	621e7f37f1	Better exception handling	2019-03-18 17:45:54 +00:00
Brendan Abolivier	651ad8bc96	Add ratelimiting on failed login attempts (#4865)	2019-03-18 12:57:20 +00:00
Brendan Abolivier	899e523d6d	Add ratelimiting on login (#4821) ... Add two ratelimiters on login (per-IP address and per-userID).	2019-03-15 17:46:16 +00:00
Richard van der Hoff	3b7ceb2c69	Merge pull request #4855 from matrix-org/rav/refactor_transaction_queue ... Split TransactionQueue up	2019-03-15 12:32:11 +00:00
Richard van der Hoff	2dee441bdb	Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker ... Move client receipt processing to federation sender worker.	2019-03-15 12:30:30 +00:00
Erik Johnston	b0fa3f6ff3	Merge pull request #4853 from matrix-org/erikj/worker_docker_ci ... Allow passing --daemonize to workers	2019-03-15 10:35:38 +00:00
Erik Johnston	271cb1998b	Revert "Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793)" ... This reverts commit 290552fd83.	2019-03-14 14:30:54 +00:00
Erik Johnston	9073cfc8bd	Merge pull request #4846 from matrix-org/hawkowl/userdir-search ... Improve searching in the userdir	2019-03-14 13:54:15 +00:00
Erik Johnston	9ad448c1e5	Correctly handle all command line options	2019-03-14 13:32:14 +00:00
Andrew Morgan	ecea5af491	Correct var name	2019-03-13 21:21:03 +00:00
Andrew Morgan	b2df0e8e2c	receiving a 400 caused an exception. handle it	2019-03-13 21:08:10 +00:00
Andrew Morgan	9a2e22fd41	is this what purgatory feels like	2019-03-13 20:29:38 +00:00
Andrew Morgan	45524f2f5e	i should have given up x2	2019-03-13 20:17:39 +00:00
Andrew Morgan	8d16ffaf7a	i should have given up	2019-03-13 20:03:10 +00:00
Richard van der Hoff	02e23b36bc	Rename and move the classes	2019-03-13 20:02:56 +00:00
Andrew Morgan	26f8e2d099	there comes a time when you should give up. but you dont	2019-03-13 19:49:08 +00:00
Andrew Morgan	5ca857ad84	as above	2019-03-13 19:35:23 +00:00
Andrew Morgan	7c0295f13c	no kwargs today	2019-03-13 19:27:10 +00:00
Andrew Morgan	66cdb840a6	Or perhaps I was the one who was drunk	2019-03-13 19:18:25 +00:00
Andrew Morgan	bec313818c	go home python, you're drunk	2019-03-13 19:10:56 +00:00
Andrew Morgan	c991e7aec7	Syntax checker is bork	2019-03-13 19:08:08 +00:00
Andrew Morgan	c2d848b80d	Destructure again	2019-03-13 19:04:43 +00:00
Andrew Morgan	ee8ba397e8	Are you happy now	2019-03-13 18:48:31 +00:00
Andrew Morgan	9dd0e34679	Syntax test	2019-03-13 18:45:17 +00:00
Andrew Morgan	220607a618	Remove testing code	2019-03-13 18:43:40 +00:00
Andrew Morgan	660b77f362	Add missing docstring detail	2019-03-13 18:38:16 +00:00
Andrew Morgan	5526b054aa	Fix syntax issues	2019-03-13 18:35:21 +00:00
Andrew Morgan	09626bfd39	Switch to wrapper function around _send_request	2019-03-13 18:26:06 +00:00
Richard van der Hoff	5d89a526f1	Factor per-destination stuff out of TransactionQueue ... This is easier than having to have a million fields keyed on destination.	2019-03-13 17:48:29 +00:00
Erik Johnston	72bfaf746d	Allow passing --daemonize to workers	2019-03-13 17:33:54 +00:00
Richard van der Hoff	fdcad8eabd	Move client receipt processing to federation sender worker. ... This is mostly a prerequisite for #4730, but also fits with the general theme of "move everything off the master that we possibly can".	2019-03-13 17:21:19 +00:00
Richard van der Hoff	eed7271b3b	declare a ReadReceipt class ... I'm going to use this in queues and things, so it'll be useful to give it more of a structure.	2019-03-13 17:20:55 +00:00
Andrew Morgan	7998ca3a66	Document using a certificate with a full chain (#4849)	2019-03-13 15:26:29 +00:00
Andrew Morgan	7d053cfe10	Retry on 400:M_UNRECOGNIZED	2019-03-13 12:10:33 +00:00
Andrew Morgan	7e75d9644b	Fix paranthesis indent	2019-03-13 11:15:23 +00:00
Andrew Morgan	cf301e37d8	Add workaround note	2019-03-13 11:14:43 +00:00
Andrew Morgan	d42c81d724	Transfer local user's push rules on room upgrade (#4838) ... Transfer push rules (notifications) on room upgrade	2019-03-12 14:42:53 +00:00
Amber Brown	c0332d095f	fixup	2019-03-13 01:30:54 +11:00
Amber Brown	797b6a63fc	fixup	2019-03-13 01:17:51 +11:00
Andrew Morgan	0ea8582f8b	Cleaner way of implementing trailing slashes	2019-03-12 14:11:11 +00:00
Amber Brown	81d9d1bee6	fixup	2019-03-12 22:28:48 +11:00
Amber Brown	6f5890b2fa	fixup	2019-03-12 22:27:56 +11:00
Amber Brown	10480c4348	fixup	2019-03-12 21:47:14 +11:00
Amber Brown	8b618041ef	fixup	2019-03-12 18:06:28 +11:00
Amber Brown	c980c7e31f	use the old method	2019-03-12 17:51:14 +11:00
Aaron Raimist	8ea1b41a0e	Clarify what registration_shared_secret allows for (#2885) (#4844) ... * Clarify what registration_shared_secret allows for (#2885) Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>	2019-03-11 18:21:52 +00:00
Andrew Morgan	f18dca26da	Merge branch 'develop' into anoa/trailing_slashes_client	2019-03-11 17:44:29 +00:00
Andrew Morgan	290552fd83	Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793) ... Server side of a solution towards #3622.	2019-03-11 17:44:03 +00:00
Andrew Morgan	a8a028dfce	Merge branch 'develop' into anoa/trailing_slashes_client	2019-03-11 17:07:31 +00:00
Erik Johnston	78c563b77c	Correctly log expected errors when fetching server keys	2019-03-11 14:11:10 +00:00
Amber Brown	78a6b950b3	fix	2019-03-12 00:50:28 +11:00
Amber Brown	1b77bd69fb	pep8	2019-03-12 00:39:12 +11:00
Amber Brown	5ba8ceab4c	fixes	2019-03-12 00:35:31 +11:00
Amber Brown	8da22e2b53	master startup	2019-03-11 21:13:35 +11:00
Amber Brown	30a8deeb68	Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search	2019-03-11 21:12:30 +11:00
Amber Brown	4abc988c6a	initial	2019-03-11 21:11:36 +11:00
Matthew Hodgson	2326e00bc4	fix incorrect encoding of filenames with spaces in (#2090) ... fixes https://github.com/vector-im/riot-web/issues/3155	2019-03-11 09:53:45 +00:00
Andrew Morgan	a5dd335cd8	lint	2019-03-08 18:25:59 +00:00
Andrew Morgan	64ff11019e	Retry certain federation requests on 404	2019-03-08 18:22:47 +00:00
Andrew Morgan	525dd02bbe	Remove trailing slashes from outbound federation requests	2019-03-08 16:55:52 +00:00
Erik Johnston	fe6c12e6cd	Add comment to schema	2019-03-08 16:38:23 +00:00
Erik Johnston	5536ddba75	Make prev_state field optional ... The `prev_state` field on events is not specced and so synapse shouldn't explode if an event is missing the field. Fixes #4787	2019-03-08 15:05:32 +00:00
Erik Johnston	8c4896668f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl	2019-03-08 11:44:20 +00:00
Erik Johnston	0ff8163eae	Factor out soft fail checks	2019-03-08 11:26:33 +00:00
Erik Johnston	469b66c102	Merge pull request #4829 from matrix-org/erikj/device_list_seen_updates ... When re-syncing device lists reset the state	2019-03-08 09:54:54 +00:00
Erik Johnston	d42b41544a	When re-syncing device lists reset the state ... We keep track of what stream IDs we've seen so that we know what updates we've handled or missed. If we re-sync we don't know if the updates we've seen are included in the re-sync (there may be a race), so we should reset the seen updates.	2019-03-07 16:04:24 +00:00
Erik Johnston	c633fc02d7	Add some debug logging for device list handling	2019-03-07 15:58:03 +00:00
Amber Brown	f6135d06cf	Rewrite userdir to be faster (#4537)	2019-03-07 01:22:53 -08:00
Travis Ralston	06cbf79fe3	Merge pull request #4779 from matrix-org/travis/icons ... Use static locations for Riot icons	2019-03-06 15:11:41 -07:00
Erik Johnston	1f85c2c0ce	Merge pull request #4818 from matrix-org/erikj/prefill_client_ips ... Prefill client IPs cache on workers	2019-03-06 19:17:39 +00:00
Erik Johnston	0a6e716600	Merge pull request #4815 from matrix-org/erikj/docstrings ... Add docstrings from matrix-org-hotfixes	2019-03-06 18:59:28 +00:00
Erik Johnston	c665b637de	Merge pull request #4816 from matrix-org/erikj/4422_debug ... Port #4422 debug logging from hotfixes	2019-03-06 18:59:12 +00:00
Erik Johnston	face0c5b3c	Prefill client IPs cache on workers	2019-03-06 17:39:32 +00:00
Erik Johnston	b879870b2d	Send message after room has been shutdown ... Currently the explanation message is sent to the abuse room before any users are forced joined, which means it tends to get lost in the backlog of joins. So instead we send the message *after* we've forced joined everyone.	2019-03-06 17:35:11 +00:00
Erik Johnston	8b7790e68f	Port #4422 debug logging from hotfixes	2019-03-06 17:29:15 +00:00
Erik Johnston	6d13bdec91	Add docstrings from matrix-org-hotfixes	2019-03-06 17:24:26 +00:00
Erik Johnston	a9de04be72	Implement soft fail	2019-03-06 16:22:16 +00:00
Andrew Morgan	7b8a157b79	Merge pull request #4792 from matrix-org/anoa/replication_tokens ... Support batch updates in the worker sender	2019-03-06 15:48:29 +00:00
Brendan Abolivier	6f3cde8b25	Make registration ratelimiter separate from the main events one	2019-03-06 11:02:42 +00:00
Brendan Abolivier	f4195f4118	Revert "Split ratelimiters in two (one for events, one for registration)" ... This reverts commit d7dbad3526.	2019-03-06 10:55:22 +00:00
Brendan Abolivier	d7dbad3526	Split ratelimiters in two (one for events, one for registration)	2019-03-05 18:41:27 +00:00
Brendan Abolivier	067ce795c0	Move settings from registration to ratelimiting in config file	2019-03-05 18:03:14 +00:00
Erik Johnston	16c8b4ecbd	Merge pull request #4772 from jbweston/jbweston/server-version-api ... Add 'server_version' endpoint to admin API	2019-03-05 16:31:00 +00:00
Brendan Abolivier	a4c3a361b7	Add rate-limiting on registration (#4735) ... * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test	2019-03-05 14:25:33 +00:00
Andrew Morgan	b9f6163092	Simplify token replication logic	2019-03-05 13:58:30 +00:00
Andrew Morgan	ae90531036	Merge pull request #4794 from matrix-org/anoa/erroneous_dollahs ... Remove unnecessary dollar signs	2019-03-05 09:46:59 +00:00
Erik Johnston	b050a10871	Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room ... Stop backpaginating when events not visible	2019-03-05 09:32:33 +00:00
Erik Johnston	9e8bca5667	Merge pull request #4799 from matrix-org/rav/clean_up_replication_code ... Clean ups in replication notifier	2019-03-05 09:19:48 +00:00
Erik Johnston	aa06d26ae0	clarify comments	2019-03-05 09:16:35 +00:00
Erik Johnston	c3c542bb4a	Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys ... Allow /keys/{changes,query} API to run on worker	2019-03-05 09:06:25 +00:00
Richard van der Hoff	c7325776a7	Remove redundant PreserveLoggingContext ... Both (!) things that register as replication listeners do the right thing wrt logcontexts, so this is redundant.	2019-03-04 18:31:18 +00:00
Erik Johnston	bfa7d46a10	Allow /keys/{changes,query} API to run on worker	2019-03-04 18:30:01 +00:00
Erik Johnston	157e5a8f27	Split DeviceHandler into master and worker	2019-03-04 18:29:26 +00:00
Richard van der Hoff	daa10e3e66	Remove unused wait_for_replication method ... I guess this was used once? It's not now, anyway.	2019-03-04 18:27:32 +00:00
Richard van der Hoff	2db49ea476	Add some debug about processing read receipts. ... I'm hoping to establish which rooms are having lots of RRs sent for them, and how old the events are when they are sent.	2019-03-04 18:19:40 +00:00
Richard van der Hoff	b29693a30b	Clean up read-receipt handling. ... Remove a call to run_as_background_process: there is no need to run this as a background process, because build_and_send_edu does not block. We may as well inline the whole of _push_remotes.	2019-03-04 18:16:43 +00:00
Erik Johnston	a84b8d56c2	Fixup slave stores	2019-03-04 18:04:57 +00:00
Richard van der Hoff	8e28bc5eee	Include a default configuration file in the 'docs' directory. (#4791)	2019-03-04 17:14:58 +00:00
Erik Johnston	0d2d046709	Fix missing null guard	2019-03-04 16:04:04 +00:00
Andrew Morgan	336de1d45b	Remove unnecessary dollar signs ... A dollar sign is already appended to the end of each PATH, so there's no need to add one in the PATH declaration as well.	2019-03-04 15:25:12 +00:00
Andrew Morgan	fe7bd23a85	Clean up logic and add comments	2019-03-04 15:08:15 +00:00
Erik Johnston	d1523aed6b	Only check history visibility when filtering ... When filtering events to send to server we check more than just history visibility. However when deciding whether to backfill or not we only care about the history visibility.	2019-03-04 14:43:42 +00:00
Andrew Morgan	9f7cdf3da1	Clearer branching, fix missing list clear	2019-03-04 14:36:52 +00:00
Andrew Morgan	5f0c449dd5	Prevent replication wedging	2019-03-04 14:03:18 +00:00
Richard van der Hoff	856c83f5f8	Avoid rebuilding Edu objects in worker mode (#4770) ... In worker mode, on the federation sender, when we receive an edu for sending over the replication socket, it is parsed into an Edu object. There is no point extracting the contents of it so that we can then immediately build another Edu.	2019-03-04 12:57:44 +00:00
Erik Johnston	8b63fe4c26	s/get_forward_events/get_successor_events/	2019-03-04 11:56:03 +00:00
Erik Johnston	fbc047f2a5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room	2019-03-04 11:54:58 +00:00
Travis Ralston	4dfbae18fe	Use static locations for Riot icons ... See https://github.com/vector-im/riot-web/issues/9009	2019-03-01 15:06:21 -07:00
Richard van der Hoff	3064952939	Fix incorrect log about not persisting duplicate state event. (#4776) ... We were logging this when it was not true.	2019-03-01 16:47:12 +00:00
Richard van der Hoff	1beebe916f	Merge branch 'master' into develop	2019-03-01 10:58:39 +00:00
Richard van der Hoff	9ac72d9543	0.99.2	2019-03-01 10:55:44 +00:00
Joseph Weston	1e8388b311	Add 'server_version' endpoint to admin API ... This is required because the 'Server' HTTP header is not always passed through proxies.	2019-03-01 09:56:58 +01:00
Andrew Morgan	ac61b45a75	Minor docstring fixes for MatrixFederationAgent (#4765)	2019-02-28 16:24:01 +00:00
Amber Brown	b131cc77df	Make 'event_id' a required parameter in federated state requests (#4741) ... * make 'event_id' a required parameter in federated state requests As per the spec: https://matrix.org/docs/spec/server_server/r0.1.1.html#id40 Signed-off-by: Joseph Weston <joseph@weston.cloud> * add changelog entry for bugfix Signed-off-by: Joseph Weston <joseph@weston.cloud> * Update server.py	2019-02-27 14:35:47 -08:00
Richard van der Hoff	68f47d6744	Fix parsing of Content-Disposition headers (#4763) ... * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings	2019-02-27 14:29:10 -08:00
Amber Brown	f2a753ea38	Move from TravisCI to BuildKite (#4752)	2019-02-27 13:03:14 -08:00
Erik Johnston	76550c58d2	Merge pull request #4759 from matrix-org/erikj/3pid_client_reader ... Move /account/3pid to client_reader	2019-02-27 16:11:21 +00:00
Erik Johnston	8267034a63	Merge pull request #4758 from matrix-org/erikj/use_presence_replication ... When presence is disabled don't send over replication	2019-02-27 15:46:26 +00:00
Erik Johnston	54f9ce11a7	Move /account/3pid to client_reader	2019-02-27 14:26:08 +00:00
Erik Johnston	1e315017d3	When presence is enabled don't send over replication	2019-02-27 13:53:46 +00:00
Erik Johnston	4cff9376f7	Move server key queries to federation reader	2019-02-27 13:43:53 +00:00
Erik Johnston	b183fef9ac	Update comments	2019-02-27 13:06:10 +00:00
Erik Johnston	7590e9fa28	Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff ... Fix tightloop over connecting to replication server	2019-02-27 11:00:59 +00:00
Richard van der Hoff	44a4d65586	0.99.2rc1	2019-02-27 10:48:34 +00:00
Erik Johnston	6bb1c028f1	Limit cache invalidation replication line length (#4748)	2019-02-27 10:28:37 +00:00
Erik Johnston	6870fc496f	Move connecting logic into ClientReplicationStreamProtocol	2019-02-27 10:23:51 +00:00
Richard van der Hoff	f191be822b	Add database version to phonehome stats. (#4753)	2019-02-27 10:21:49 +00:00
Erik Johnston	25814921f1	Increase the max delay between retry attempts ... Otherwise if you have many workers they can easily take out master with their connection attempts	2019-02-26 15:12:33 +00:00
Erik Johnston	313987187e	Fix tightloop over connecting to replication server ... If the client failed to process incoming commands during the initial set up of the replication connection it would immediately disconnect and reconnect, resulting in a tightloop. This can happen, for example, when subscribing to a stream that has a row that is too long in the backlog. The fix here is to not consider the connection successfully set up until the client has succesfully subscribed and caught up with the streams. This ensures that the retry logic timers aren't reset until then, meaning that if an error does happen during start up the client will continue backing off before retrying again.	2019-02-26 15:05:41 +00:00
Andrew Morgan	802884d4ee	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate_develop	2019-02-26 14:23:40 +00:00
Andrew Morgan	6fcb25202f	Put function def back to the way it was	2019-02-26 14:17:38 +00:00
Andrew Morgan	7a4632af9c	Prevent showing non-fed rooms in fed /publicRooms	2019-02-26 13:37:24 +00:00
Andrew Morgan	c74624a633	Revert "Prevent showing non-fed rooms in fed /publicRooms"	2019-02-26 13:20:38 +00:00
Andrew Morgan	a1a6473293	Merge pull request #4736 from matrix-org/anoa/public_rooms_federate ... Config option to prevent showing non-fed rooms in fed /publicRooms	2019-02-26 13:07:15 +00:00
Andrew Morgan	c4414768af	Cleaner chunk logic	2019-02-26 12:22:34 +00:00
Andrew Morgan	a712aa3a9c	Correct indent	2019-02-26 12:13:55 +00:00
Andrew Morgan	16565e67db	Correct docstring types and chunk logic	2019-02-26 12:12:48 +00:00
Andrew Morgan	40c2271680	Clean up room chunk logic	2019-02-26 12:04:34 +00:00
Andrew Morgan	6728bf3940	Make not showing non-federated rooms the default	2019-02-26 11:52:52 +00:00
Andrew Morgan	6946c20111	Result may be None	2019-02-26 11:27:19 +00:00
Andrew Morgan	71669a0fba	Address rich comments	2019-02-26 11:25:00 +00:00
Erik Johnston	899a119c2b	Don't log stack trace when client has gone away during media download (#4738) ... * Don't log stack trace when client has gone away during media download * Newsfile * Fixup newsfile	2019-02-25 11:17:22 -08:00
Richard van der Hoff	641c409e4e	Fix ACME config for python 2. (#4717) ... Fixes #4675.	2019-02-25 11:16:33 -08:00
Matthew Hodgson	70ea2f4e1d	switch from google.com to recaptcha.net for reCAPTCHA (#4731) ... * add trivial clarification about jemalloc * switch from google.com to recaptcha.net because https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally	2019-02-25 11:15:36 -08:00
Andrew Morgan	96c408273e	Fix group's call to generate_room_entry	2019-02-25 18:00:17 +00:00
Andrew Morgan	1330aa4a8f	elif not else if	2019-02-25 17:28:19 +00:00
Erik Johnston	65f3fbfbf7	Merge pull request #4737 from matrix-org/erikj/failure_log_tb ... Log tracebacks correctly	2019-02-25 17:26:30 +00:00
Erik Johnston	c7b333c545	Log tracebacks correctly	2019-02-25 16:56:41 +00:00
Erik Johnston	69efe6fb16	Merge pull request #4718 from matrix-org/erikj/fix_backfill_state_shred ... Fix backfill storing incorrect state for events	2019-02-25 16:48:52 +00:00
Erik Johnston	9c598dddcb	Fix typo	2019-02-25 16:32:02 +00:00
Erik Johnston	b1a90da82e	Merge pull request #4721 from matrix-org/erikj/msc_1866 ... MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API	2019-02-25 15:53:10 +00:00
Erik Johnston	16c7afa94c	Merge pull request #4722 from matrix-org/erikj/correctly_handle_keyring_exceptions ... Handle errors when fetching remote server keys	2019-02-25 15:53:02 +00:00
Andrew Morgan	8aaf7ffc44	syntax derp	2019-02-25 15:27:17 +00:00
Andrew Morgan	84c0a20dfe	Simplify call to generate_room_entry	2019-02-25 15:23:27 +00:00
Erik Johnston	4b9e5076c4	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate	2019-02-25 15:08:18 +00:00
Andrew Morgan	07493607a8	Docs and arg name clarification	2019-02-25 15:04:46 +00:00
Andrew Morgan	bd398b874e	Don't restrict non-fed rooms over client APIs	2019-02-25 15:04:46 +00:00
Andrew Morgan	e4b078a600	Config option to prevent showing non-fed rooms in fed /publicRooms	2019-02-25 15:04:46 +00:00
Erik Johnston	d730c2c22b	More comments	2019-02-25 14:45:02 +00:00
Erik Johnston	890cb048fd	Assert rather than clobber the values	2019-02-25 14:42:11 +00:00
Erik Johnston	65d1003d01	raise_from already raises	2019-02-25 14:34:03 +00:00
Erik Johnston	9342cc6ab1	Add comments and paranoia	2019-02-25 10:02:12 +00:00
Erik Johnston	47a7e3928d	Correctly proxy exception in frontend_proxy worker	2019-02-23 15:17:57 +00:00
Erik Johnston	41285ffe5b	Handle errors when fetching remote server keys	2019-02-23 15:09:39 +00:00
Erik Johnston	59e0112209	MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API	2019-02-23 14:50:43 +00:00
Erik Johnston	d14e94bae4	Merge pull request #4716 from matrix-org/erikj/pusher_logging ... Fix up pusher logging a bit	2019-02-22 15:29:16 +00:00
Erik Johnston	b82c9cf462	Add missing return	2019-02-22 15:27:40 +00:00
Erik Johnston	f2891d2487	Correctly handle PusherConfigException	2019-02-22 15:18:19 +00:00
Erik Johnston	a164134a53	Drop logging level of creating a pusher	2019-02-22 14:48:06 +00:00
Erik Johnston	1d9df51ff1	Correctly handle null data in HttpPusher	2019-02-22 14:47:48 +00:00
Erik Johnston	80467bbac3	Fix state cache invalidation on workers	2019-02-22 14:38:14 +00:00
Erik Johnston	7b288826b7	Fix backfill storing incorrect state for events	2019-02-22 11:33:51 +00:00
Richard van der Hoff	e07384c4e1	Add prometheus metrics for number of badge update pushes. (#4709) ... We're counting the number of push notifications, but not the number of badges; I'd like to see if they are significant.	2019-02-22 10:57:15 +00:00
Richard van der Hoff	0abb094f1a	bail out early in on_new_receipts if no pushers (#4706)	2019-02-21 17:51:21 +00:00
Richard van der Hoff	6d65659b62	Run push_receipts_to_remotes as background job (#4707) ... I suspect the CPU usage metrics for this are going to /dev/null at the moment.	2019-02-21 17:50:30 +00:00
Erik Johnston	b9d6756b14	Merge pull request #4263 from rkfg/develop ... Prevent crash on pagination.	2019-02-21 17:42:15 +00:00
Erik Johnston	71b625d808	Stop backpaginating when events not visible	2019-02-20 18:14:12 +00:00
Erik Johnston	82fca11fc1	Merge pull request #4694 from matrix-org/erikj/fix_sentry_config_format ... Fixup generated metrics config	2019-02-20 14:13:38 +00:00
Richard van der Hoff	82ca6d1f9f	Add metrics for number of outgoing EDUs, by type (#4695)	2019-02-20 14:13:14 +00:00
Erik Johnston	3d672fec51	Fixup generated metrics config	2019-02-20 13:39:37 +00:00
Amber Brown	a06614bd2a	UPSERT many functionality (#4644)	2019-02-20 23:03:30 +11:00
Richard van der Hoff	b2200a8690	Logging improvements for the pusher (#4691)	2019-02-20 11:36:50 +00:00
Richard van der Hoff	c88bc53903	Fix TaskStopped exceptions when outbound requests time out (#4690)	2019-02-20 11:35:52 +00:00
Richard van der Hoff	86920ac266	Merge pull request #4678 from matrix-org/rav/tls_install_instructions ... Attempt to clarify installation/config instructions	2019-02-20 11:10:33 +00:00
Erik Johnston	dbdc565dfd	Fix registration on workers (#4682) ... * Move RegistrationHandler init to HomeServer * Move post registration actions to RegistrationHandler * Add post regisration replication endpoint * Newsfile	2019-02-20 18:47:31 +11:00
Richard van der Hoff	5f9bdf90fe	Attempt to make default config more consistent ... The general idea here is that config examples should just have a hash and no extraneous whitespace, both to make it easier for people who don't understand yaml, and to make the examples stand out from the comments.	2019-02-19 13:54:29 +00:00
Erik Johnston	c003450057	Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation ... Batch cache invalidation over replication	2019-02-19 13:14:30 +00:00
Erik Johnston	62175a20e5	Docs	2019-02-19 11:38:40 +00:00
Erik Johnston	1bb35e3a83	Use itertools	2019-02-19 11:34:40 +00:00
Erik Johnston	bc8fa1509d	Documentation	2019-02-19 11:24:59 +00:00
Brendan Abolivier	a288bdf0b1	Merge pull request #4652 from matrix-org/babolivier/acme-delegated ... Support .well-known delegation when issuing certificates through ACME	2019-02-19 11:15:38 +00:00
Brendan Abolivier	5a707a2f9a	Improve config documentation	2019-02-19 10:59:26 +00:00
Brendan Abolivier	a8626901cd	Fetch ACME domain into an instance member	2019-02-19 10:54:33 +00:00
Erik Johnston	32590b7139	Merge pull request #4670 from matrix-org/erikj/register_login_split ... Split /login into client_reader	2019-02-19 10:46:22 +00:00
Richard van der Hoff	7c70b8f8a6	Try and make TLS federation client code faster (#4674) ... * fix to use makeContext so that we don't need to rebuild the certificateoptions each time	2019-02-19 10:19:16 +00:00
Andrew Morgan	968a30a75c	Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade ... Transfer bans on room upgrade	2019-02-19 10:16:42 +00:00
Amber Brown	561eebe170	fix to use makeContext so that we don't need to rebuild the certificateoptions each time	2019-02-19 16:18:05 +11:00
Andrew Morgan	34ac75ce2c	lint	2019-02-18 18:23:37 +00:00
Erik Johnston	a9b5ea6fc1	Batch cache invalidation over replication ... Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic.	2019-02-18 17:53:31 +00:00
Andrew Morgan	f8b9ca53ce	Move member event processing and changelog fix	2019-02-18 17:28:52 +00:00
Erik Johnston	d154f5a055	Merge pull request #4632 from matrix-org/erikj/basic_sentry ... Add basic optional sentry.io integration	2019-02-18 17:22:45 +00:00
Erik Johnston	4cc4400b4d	Split /login into client_reader	2019-02-18 17:19:01 +00:00
Erik Johnston	fc2c245a1f	Merge pull request #4666 from matrix-org/erikj/register_login_split ... Split out registration to worker	2019-02-18 17:18:06 +00:00
Erik Johnston	459d3d5046	Merge pull request #4668 from matrix-org/erikj/catch_exceptions ... Correctly handle HttpResponseException when handling device updates	2019-02-18 16:55:43 +00:00
Erik Johnston	d328a93b51	Fixup error handling and message	2019-02-18 16:53:56 +00:00
Erik Johnston	af691e415c	Move register_device into handler	2019-02-18 16:49:38 +00:00
Erik Johnston	028267acd2	Merge pull request #4669 from matrix-org/erikj/log_exception ... Cleanup top level request exception logging	2019-02-18 16:41:41 +00:00
Erik Johnston	d08bac4136	Merge pull request #4651 from matrix-org/matthew/well-known-cors ... set CORS on .well-known URI to unbreak modular	2019-02-18 16:38:11 +00:00
Erik Johnston	c30f73c86a	Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers ... Fix kicking guest users in worker mode	2019-02-18 15:55:48 +00:00
Erik Johnston	092b541401	Fixup	2019-02-18 15:52:26 +00:00
Brendan Abolivier	45bb55c6de	Use a configuration parameter to give the domain to generate a certificate for	2019-02-18 15:46:23 +00:00
Erik Johnston	8b9ae6d3a6	Update docs	2019-02-18 15:26:13 +00:00
Erik Johnston	94960cef03	pep8	2019-02-18 15:24:13 +00:00
Erik Johnston	fe725f7e45	Cleanup top level request exception logging ... Firstly, we always logged that the request was being handled via `JsonResource._async_render`, so we change that to use the servlet name we add to the request. Secondly, we pass the exception information to the logger rather than formatting it manually. This makes it consistent with other exception logging, allwoing logging hooks and formatters to access the exception information.	2019-02-18 15:11:04 +00:00
Matthew Hodgson	04dad5ac16	fix missig import	2019-02-18 15:59:23 +01:00
Erik Johnston	e07cc31cb8	Correctly handle HttpResponseException	2019-02-18 14:55:09 +00:00
Brendan Abolivier	68a53f825f	Merge branch 'develop' into babolivier/acme-delegated	2019-02-18 14:52:23 +00:00
Erik Johnston	32e54b472a	Fix kicking guest users in worker mode ... When guest_access changes from allowed to forbidden all local guest users should be kicked from the room. This did not happen when revocation was received from federation on a worker. Presumably broken in #4141	2019-02-18 14:16:07 +00:00
Andrew Morgan	915421065b	Membership events are done later	2019-02-18 14:02:09 +00:00
Erik Johnston	d1b060b492	Merge pull request #4643 from matrix-org/erikj/catch_exceptions ... Correctly handle RequestSendFailed exceptions	2019-02-18 13:53:24 +00:00
Andrew Morgan	9caab0c364	Transfer bans on room upgrade	2019-02-18 13:52:51 +00:00
Erik Johnston	dc5efc92a8	Fixup	2019-02-18 13:52:49 +00:00
Erik Johnston	eb2b8523ae	Split out registration to worker ... This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker.	2019-02-18 12:12:57 +00:00
Andrew Morgan	6d02a13d81	Typo in info log ... Co-Authored-By: babolivier <contact@brendanabolivier.com>	2019-02-18 11:36:34 +00:00
Juuso "Linda" Lapinlampi	68d2869c8d	config: Remove a repeated word from a logger warning ... The warning for missing macaroon_secret_key was "missing missing".	2019-02-15 22:24:53 -07:00
Erik Johnston	bd4505f765	Merge pull request #4647 from matrix-org/erikj/add_room_publishing_rules ... Add configurable room list publishing rules	2019-02-15 22:11:01 +00:00
Brendan Abolivier	f86b695cbd	Various cosmetics to make TravisCI happy	2019-02-15 12:29:34 +00:00
Brendan Abolivier	af8a2f679b	Remove unused import	2019-02-15 12:27:43 +00:00
Brendan Abolivier	1895d14e12	Support .well-known delegation when issuing certificates through ACME	2019-02-15 12:05:08 +00:00
Erik Johnston	b99c532c1c	Move defaults up into code	2019-02-15 10:53:39 +00:00
Erik Johnston	02c729d6b0	Hoist up checks to reduce overall work	2019-02-15 10:20:02 +00:00
Erik Johnston	02c46acc6a	Fixup comments	2019-02-15 10:17:13 +00:00
Matthew Hodgson	bfcefbb230	tabs	2019-02-14 19:53:12 +00:00
Matthew Hodgson	6f47bc3fb2	set CORS on .well-known URI to unbreak modular ... otherwise a riot/web running on foo.riot.im can't query the .well-known on foo.modular.im...	2019-02-14 19:51:41 +00:00
Erik Johnston	8e32f26cb8	Clarify comments	2019-02-14 18:21:24 +00:00
Erik Johnston	cb12a37708	Clarify and fix behaviour when there are multiple aliases	2019-02-14 18:16:32 +00:00
Erik Johnston	f61b2068e6	Only fetch aliases when publishing rooms	2019-02-14 18:08:09 +00:00
Erik Johnston	f666fe36d7	Fixup comments	2019-02-14 18:07:24 +00:00
Richard van der Hoff	bf4fd14806	Merge branch 'master' into develop	2019-02-14 17:34:09 +00:00
Richard van der Hoff	f595d6ac57	0.99.1.1	2019-02-14 17:20:02 +00:00
Richard van der Hoff	f311018823	Fix errors in acme provisioning (#4648) ... * Better logging for errors on startup * Fix "TypeError: '>' not supported" when starting without an existing certificate * Fix a bug where an existing certificate would be reprovisoned every day	2019-02-14 17:10:36 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules ... This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Richard van der Hoff	b02465b9db	Merge branch 'master' into develop	2019-02-14 14:42:03 +00:00
Richard van der Hoff	06cd757ae7	0.99.1	2019-02-14 14:24:24 +00:00
Erik Johnston	7fc1196a36	Correctly handle RequestSendFailed exceptions ... This mainly reduces the number of exceptions we log.	2019-02-14 14:01:04 +00:00
Erik Johnston	6cb415b63f	Fixup comments and add warning	2019-02-13 16:15:11 +00:00
Richard van der Hoff	c6e75c9f2d	Merge pull request #4450 from 14mRh4X0r/fix-dependency-message ... Fix error message for optional dependencies	2019-02-13 16:12:49 +00:00
Richard van der Hoff	3bc238629e	0.99.1rc2	2019-02-13 14:46:18 +00:00
Richard van der Hoff	c1dfd6a18a	Merge remote-tracking branch 'origin/release-v0.99.1' into develop	2019-02-13 14:27:45 +00:00
Erik Johnston	309f3bb322	Update synapse/app/_base.py ... Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>	2019-02-13 13:24:27 +00:00
Amber Brown	bb4fd8f927	Run black on user directory code (#4635)	2019-02-13 23:05:32 +11:00
Richard van der Hoff	767686af48	Use listen_tcp for the replication listener ... Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with what we do elsewhere.	2019-02-13 11:59:04 +00:00
Richard van der Hoff	2a5a15aff8	Improve logging around listening services ... I wanted to bring listen_tcp into line with listen_ssl in terms of returning a list of ports, and wanted to check that was a safe thing to do - hence the logging in `refresh_certificate`. Also, pull the 'Synapse now listening' message up to homeserver.py, because it was being duplicated everywhere else.	2019-02-13 11:58:54 +00:00
Richard van der Hoff	e3a0300431	Special-case the default bind_addresses for metrics listener ... turns out it doesn't really support ipv6, so let's hack around that by only listening on ipv4 by default.	2019-02-13 11:48:56 +00:00
Erik Johnston	93f7d2df3e	Comments	2019-02-12 16:03:40 +00:00
Erik Johnston	6a8f902edb	Raise an appropriate error message if sentry_sdk missing	2019-02-12 16:01:41 +00:00
Erik Johnston	ef2228c890	Basic sentry integration	2019-02-12 13:55:58 +00:00
Erik Johnston	d2fa7b7e99	Update changelog and version	2019-02-12 13:22:25 +00:00
Erik Johnston	ba3f27b69a	Merge pull request #4608 from matrix-org/anoa/acls_room_upgrade ... Transfer Server ACLs on room upgrade	2019-02-12 13:20:06 +00:00
Erik Johnston	cf82338930	Merge pull request #4627 from matrix-org/erikj/user_ips_analyze ... Analyze user_ips before running deduplication	2019-02-12 13:05:09 +00:00
Erik Johnston	3df8fcca25	Merge pull request #4626 from matrix-org/erikj/fixup_user_ips_dedupe ... Reduce user_ips bloat during dedupe background update	2019-02-12 13:02:58 +00:00
Erik Johnston	495ea92350	Fix pep8	2019-02-12 12:40:42 +00:00
Erik Johnston	483ba85c7a	Analyze user_ips before running deduplication ... Due to the table locks taken out by the naive upsert, the table statistics may be out of date. During deduplication it is important that the correct index is used as otherwise a full table scan may be incorrectly used, which can end up thrashing the database badly.	2019-02-12 11:55:27 +00:00
Erik Johnston	362d80b770	Reduce user_ips bloat during dedupe background update ... The background update to remove duplicate rows naively deleted and reinserted the duplicates. For large tables with a large number of duplicates this causes a lot of bloat (with postgres), as the inserted rows are appended to the table, since deleted rows will not be overwritten until a VACUUM has happened. This should hopefully also help ensure that the query in the last batch uses the correct index, as inserting a large number of new rows without analyzing will upset the query planner.	2019-02-12 11:39:34 +00:00
Erik Johnston	3c03c37883	Merge pull request #4625 from matrix-org/rav/fix_generate_config_warnings ... fix self-signed cert notice from generate-config	2019-02-12 11:24:45 +00:00
Richard van der Hoff	a4ce91396b	Disable TLS by default (#4614)	2019-02-12 10:52:08 +00:00
Richard van der Hoff	32b781bfe2	Fix error when loading cert if tls is disabled (#4618) ... If TLS is disabled, it should not be an error if no cert is given. Fixes #4554.	2019-02-12 10:51:31 +00:00
Richard van der Hoff	dfc846a316	fix self-signed cert notice from generate-config ... fixes #4620	2019-02-12 10:37:59 +00:00
Richard van der Hoff	0ca2908653	fix tests	2019-02-11 22:01:27 +00:00
Richard van der Hoff	4fddf8fc77	Infer no_tls from presence of TLS listeners ... Rather than have to specify `no_tls` explicitly, infer whether we need to load the TLS keys etc from whether we have any TLS-enabled listeners.	2019-02-11 21:39:14 +00:00
Richard van der Hoff	15272f837c	Merge branch 'rav/no_create_server_contexts_if_no_tls' into rav/tls_cert/work	2019-02-11 21:34:19 +00:00
Richard van der Hoff	9645728619	Don't create server contexts when TLS is disabled ... we aren't going to use them anyway.	2019-02-11 21:32:01 +00:00
Richard van der Hoff	be794c7cf7	Merge branch 'rav/tls_config_logging_fixes' into rav/tls_cert/work	2019-02-11 21:16:00 +00:00
Richard van der Hoff	2129dd1a02	Fail cleanly if listener config lacks a 'port' ... ... otherwise we would fail with a mysterious KeyError or something later.	2019-02-11 21:15:01 +00:00
Richard van der Hoff	086f6f27d4	Logging improvements around TLS certs ... Log which file we're reading keys and certs from, and refactor the code a bit in preparation for other work	2019-02-11 21:02:06 +00:00
Richard van der Hoff	5d27730a73	Move ClientTLSOptionsFactory init out of refresh_certificates (#4611) ... It's nothing to do with refreshing the certificates. No idea why it was here.	2019-02-11 18:03:30 +00:00
Erik Johnston	719e073f00	Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating ... add updating of backup versions	2019-02-11 13:45:49 +00:00
Richard van der Hoff	24b7f3916d	Clean up default listener configuration (#4586) ... Rearrange the comments to try to clarify them, and expand on what some of it means. Use a sensible default 'bind_addresses' setting. For the insecure port, only bind to localhost, and enable x_forwarded, since apparently it's for use behind a load-balancer.	2019-02-11 12:50:30 +00:00
Andrew Morgan	a126f86eec	Transfer Server ACLs on room upgrade	2019-02-11 11:30:37 +00:00
Amber Brown	6e2a5aa050	ACME Reprovisioning (#4522)	2019-02-11 10:36:26 +00:00
Amber Brown	4ffd10f46d	Be tolerant of blank TLS fingerprints config (#4589)	2019-02-11 10:04:27 +00:00
Erik Johnston	b201149c7e	Merge pull request #4420 from matrix-org/jaywink/openid-listener ... New listener resource for the federation API "openid/userinfo" endpoint	2019-02-11 09:44:00 +00:00
Valentin Anger	2dc2b6e9f1	Allow "unavailable" presence status for /sync (#4592) ... * Allow "unavailable" presence status for /sync Closes #3772, closes #3779 Signed-off-by: Valentin Anger <valentin.an.1999@gmail.com> * Add changelog for PR 4592	2019-02-08 21:09:56 +00:00
Richard van der Hoff	56710c7df5	Fix 'no unique or exclusion constraint' error (#4591) ... Add more tables to the list of tables which need a background update to complete before we can upsert into them, which fixes a race against the background updates.	2019-02-08 18:30:46 +00:00
Amber Brown	9cd33d2f4b	Deduplicate some code in synapse.app (#4567)	2019-02-08 17:25:57 +00:00
Hubert Chathi	afae8442b5	make sure version is in body and wrap in linearizer queue ... also add tests	2019-02-08 01:32:45 -05:00
Hubert Chathi	d9e424bf64	re-try to make isort happy	2019-02-06 22:18:41 -05:00
Hubert Chathi	9ff620a518	fix import to make isort happy	2019-02-06 21:32:52 -05:00
Hubert Chathi	8248637173	add new endpoint to update backup versions	2019-02-06 17:57:10 -05:00
Hubert Chathi	664c81e8b7	return proper error codes for some 404s	2019-02-06 17:47:22 -05:00
Richard van der Hoff	2475434080	Merge branch 'master' into develop	2019-02-05 18:44:49 +00:00
Richard van der Hoff	3bd9daf4b8	v0.99.0	2019-02-05 18:33:02 +00:00
Andrew Morgan	627ecd358e	Filter user directory state query to a subset of state events (#4462) ... * Filter user directory state query to a subset of state events * Add changelog	2019-02-05 12:16:28 +00:00
Richard van der Hoff	bf1e4d96ad	Fix default ACME config for py2 (#4564) ... Fixes #4559	2019-02-05 11:37:33 +00:00
Richard van der Hoff	d7e27a1f08	fix typo in config comments (#4557)	2019-02-05 11:32:45 +00:00
Richard van der Hoff	30fd2f89db	0.99.0rc4	2019-02-01 15:52:28 +00:00
Richard van der Hoff	ef43a03fc5	Merge pull request #4546 from matrix-org/rav/silence_critical_error_from_federation ... Fix noisy "twisted.internet.task.TaskStopped" errors in logs	2019-02-01 14:37:22 +00:00
Richard van der Hoff	f8db967d5a	Merge remote-tracking branch 'origin/release-v0.99.0' into develop	2019-02-01 13:20:15 +00:00
Richard van der Hoff	fa794980ec	Merge pull request #4544 from matrix-org/rav/skip_invalid_well_known ... Treat an invalid .well-known the same as an absent one	2019-02-01 13:18:36 +00:00
Richard van der Hoff	e9779a6f8f	Fix b'ab' noise in logs	2019-02-01 12:34:31 +00:00
Richard van der Hoff	9763a73af0	Merge branch 'release-v0.99.0' into develop	2019-02-01 12:30:22 +00:00
Richard van der Hoff	f0ba34f581	Fix noisy "twisted.internet.task.TaskStopped" errors in logs ... Fixes #4003	2019-02-01 12:22:57 +00:00
Richard van der Hoff	8a21b03fba	Treat an invalid .well-known the same as an absent one ... ... basically, carry on and fall back to SRV etc.	2019-02-01 11:37:31 +00:00
Richard van der Hoff	3c8a41140e	Cache failures to parse .well-known ... Also add a Measure block around the .well-known fetch	2019-02-01 00:37:52 +00:00
Richard van der Hoff	24d59c7568	better logging for federation connections	2019-01-31 23:18:20 +00:00
Richard van der Hoff	d428b46346	Update federation routing logic to check .well-known before SRV	2019-01-31 23:14:18 +00:00
Andrew Morgan	c45fd0dda0	Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable ... Copy over non-federatable trait on room upgrade	2019-01-31 20:28:37 +00:00
Richard van der Hoff	625385d684	Merge branch 'release-v0.99.0' into develop	2019-01-31 18:43:20 +00:00
Richard van der Hoff	85129d7068	v0.99.0rc3	2019-01-31 18:35:38 +00:00
Andrew Morgan	d239f67c25	Raise an exception instead of returning None	2019-01-31 18:34:15 +00:00
Richard van der Hoff	07dfe148de	Add some debug for membership syncing issues (#4538) ... I can't figure out what's going on with #4422 and #4436; perhaps this will help.	2019-01-31 18:30:40 +00:00
Andrew Morgan	3ed3cb4339	New function for getting room's create event	2019-01-31 18:21:39 +00:00
Andrew Morgan	bbb97a35fd	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable	2019-01-31 18:11:00 +00:00
Richard van der Hoff	e707e7b38d	Fix infinite loop when an event is redacted in a v3 room (#4535)	2019-01-31 15:34:17 +00:00
Andrew Morgan	563f6a832b	Reject large transactions on federation (#4513) ... * Reject large transactions on federation * Add changelog * lint * Simplify large transaction handling	2019-01-31 11:44:04 +00:00
Andrew Morgan	fb50934b8f	lint	2019-01-31 11:34:45 +00:00
Richard van der Hoff	6fba9fd20c	Merge remote-tracking branch 'origin/release-v0.99.0' into develop	2019-01-30 17:02:32 +00:00
Andrew Morgan	d621c5562e	Copy over non-federatable trait on room upgrade	2019-01-30 16:33:51 +00:00
Richard van der Hoff	b8b898666e	v0.99.0rc2	2019-01-30 16:31:07 +00:00
Matthew Hodgson	ad7ac8853c	by default include m.room.encryption on invites (#3902) ... * by default include m.room.encryption on invites * fix constant * changelog	2019-01-30 16:26:13 +00:00
Richard van der Hoff	c74b96755c	Merge remote-tracking branch 'origin/develop' into release-v0.99.0	2019-01-30 16:23:28 +00:00
Neil Johnson	81b7e7eed3	Update constants.py ... remove trailing ,	2019-01-30 16:11:36 +00:00
Erik Johnston	ed8c5e4cda	Fix remote invite rejections not comming down sync ... This was broken in PR #4405, commit 886e5ac, where we changed remote rejections to be outliers. The fix is to explicitly add the leave event in when we know its an out of band invite. We can't always add the event as if the server is/was in the room there might be more events to send down the sync than just the leave.	2019-01-30 15:46:27 +00:00
Richard van der Hoff	a5d0c771a3	0.99.0rc1	2019-01-30 15:11:18 +00:00
Erik Johnston	6587b0b89b	Merge pull request #4472 from matrix-org/neilj/room_capabilities ... Server capabilities support	2019-01-30 14:26:56 +00:00
Erik Johnston	a4f52a33fe	Fix replication for room v3 (#4523) ... * Fix replication for room v3 We were not correctly quoting the path fragments over http replication, which meant that it exploded when the event IDs had a slash in them * Newsfile	2019-01-30 14:19:52 +00:00
Richard van der Hoff	7615a8ced1	ACME config cleanups (#4525) ... * Handle listening for ACME requests on IPv6 addresses the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses without extra quoting. Building a string which you are about to parse again seems like a weird choice. Let's just use listenTCP, which is consistent with what we do elsewhere. * Clean up the default ACME config make it look a bit more consistent with everything else, and tweak the defaults to listen on port 80. * newsfile	2019-01-30 14:17:55 +00:00
Erik Johnston	e87d7a4b0f	Raise ConfigError instead	2019-01-30 12:48:09 +00:00
Erik Johnston	270f212a2a	_listener_http should return a list	2019-01-30 12:14:50 +00:00
Richard van der Hoff	a79034aedf	Merge pull request #4521 from matrix-org/rav/fed_routing/cleanups ... Tiny .well-known fixes	2019-01-30 11:47:24 +00:00
Richard van der Hoff	c7b24ac3d0	Follow redirects on .well-known (#4520)	2019-01-30 11:43:33 +00:00
Amber Brown	f6813919e8	SIGHUP for TLS cert reloading (#4495)	2019-01-30 11:00:02 +00:00
Richard van der Hoff	09a1a6b55e	fix exception text	2019-01-30 10:58:52 +00:00
Richard van der Hoff	928c50b59a	Also jitter the invalid cache period	2019-01-30 10:58:52 +00:00
Neil Johnson	b37e8c9572	Merge branch 'neilj/room_capabilities' of github.com:matrix-org/synapse into neilj/room_capabilities	2019-01-30 10:56:47 +00:00
Neil Johnson	f834d98402	isort	2019-01-30 10:55:42 +00:00
Richard van der Hoff	bc5f6e1797	Add a caching layer to .well-known responses (#4516)	2019-01-30 10:55:25 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519)	2019-01-30 10:53:17 +00:00
Neil Johnson	ee4df7fd7a	Merge branch 'develop' into neilj/room_capabilities	2019-01-30 10:28:08 +00:00
Neil Johnson	c5a0f82cca	define room dispositions for use in exposing room capabilities	2019-01-30 10:24:24 +00:00
Neil Johnson	9c850d9d5e	formatting and use constants where available	2019-01-30 10:23:26 +00:00
Neil Johnson	2f46804055	Populate default room version from Constants	2019-01-30 09:39:10 +00:00
Neil Johnson	c7837dce24	reflect that rooms v3 is a stable room version	2019-01-30 09:33:30 +00:00
Richard van der Hoff	457fbfaf22	Merge pull request #4486 from xperimental/workaround-4216 ... Implement workaround for login error.	2019-01-30 07:06:11 +00:00
Robert Jacob	2a7f0b8953	Implement workaround for login error. ... Signed-off-by: Robert Jacob <xperimental@solidproject.de>	2019-01-30 01:06:39 +01:00
Erik Johnston	0b24d58e05	No vdh tests!	2019-01-29 23:11:48 +00:00
Erik Johnston	a1b0e1879b	Enable room version v3	2019-01-29 23:09:10 +00:00
Erik Johnston	67b82f1336	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:58:38 +00:00
Erik Johnston	afeea319df	Fixup comment	2019-01-29 22:55:29 +00:00
Erik Johnston	7740eddd04	Merge pull request #4514 from matrix-org/erikj/remove_event_id ... Remove usages of event ID's domain	2019-01-29 22:54:25 +00:00
Erik Johnston	ff2f65d737	Update comment	2019-01-29 22:35:36 +00:00
Erik Johnston	655ce037fd	check event format version not room version	2019-01-29 22:33:43 +00:00
Erik Johnston	f46a818ce5	kill vdh test some more	2019-01-29 22:02:58 +00:00
Erik Johnston	a696c48133	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:00:33 +00:00
Erik Johnston	c21b7cbc09	Update synapse/storage/events_worker.py	2019-01-29 21:53:48 +00:00
Erik Johnston	4db252c073	Check redaction state when event is pulled out of the database	2019-01-29 21:48:36 +00:00
Erik Johnston	6d23ec2111	Fix typo	2019-01-29 21:45:53 +00:00
Erik Johnston	b5d510ad64	Remove unused arg	2019-01-29 21:45:28 +00:00
Erik Johnston	6f9cdc2d47	Merge pull request #4483 from matrix-org/erikj/event_v2 ... Implement event format V2	2019-01-29 21:40:00 +00:00
Erik Johnston	47e2dd1994	Drop vdh support	2019-01-29 21:24:34 +00:00
Erik Johnston	38590a4870	Add docstring	2019-01-29 21:22:47 +00:00
Richard van der Hoff	82165eeb05	Update synapse/storage/events_worker.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2019-01-29 21:14:39 +00:00
Andrew Morgan	03b086647f	Merge pull request #4512 from matrix-org/anoa/consent_dir ... Check consent dir path on startup	2019-01-29 20:08:18 +00:00
Erik Johnston	84af577356	Implement event format v2	2019-01-29 18:06:11 +00:00
Neil Johnson	e4bef9d470	rework format of change password capability	2019-01-29 18:04:56 +00:00
Erik Johnston	610f0830b0	Don't assert an event must have an event ID	2019-01-29 18:02:26 +00:00
Erik Johnston	840068bd78	Only check event ID domain for signatures for V1 events ... In future version events won't have an event ID, so we won't be able to do this check.	2019-01-29 18:02:02 +00:00
Erik Johnston	8e3d34e3c5	Use event origin for filtering incoming events ... We only process events sent to us from a server if the event ID matches the server, to help guard against federation storms. We replace this with a check against the event origin.	2019-01-29 16:57:00 +00:00
Erik Johnston	55d9024835	Use snder and not event ID domain to check if ours ... The transaction queue only sends out events that we generate. This was done by checking domain of event ID, but that can no longer be used. Instead, we may as well use the sender field.	2019-01-29 16:54:23 +00:00
Richard van der Hoff	cc2d650ef7	Relax requirement for a content-type on .well-known (#4511)	2019-01-29 16:49:17 +00:00
Erik Johnston	b1fffca345	Remove event ID usage when checking if new room ... The event ID is changing, so we can no longer get the domain from it. On the other hand, the check is unnecessary.	2019-01-29 16:15:02 +00:00
Erik Johnston	770b823445	Only check event IDs domain signed event for V1 and V2 ... Since newer versions of events don't have the same format for event ID.	2019-01-29 16:15:00 +00:00
Travis Ralston	d02c5ccb11	Merge pull request #4498 from matrix-org/travis/fix-docs-public_baseurl ... Don't recommend :8448 to people on public_baseurl	2019-01-29 09:06:16 -07:00
Neil Johnson	f03b3a7a3a	support change_password in capabilities end-point	2019-01-29 15:58:37 +00:00
Andrew Morgan	e65a17b26f	Check consent dir path on startup	2019-01-29 15:30:33 +00:00
Amber Brown	6bd4374636	Do not generate self-signed TLS certificates by default. (#4509)	2019-01-29 14:09:10 +00:00
Erik Johnston	b8d75ef53e	Merge pull request #4481 from matrix-org/erikj/event_builder ... Refactor event building into EventBuilder	2019-01-29 14:07:23 +00:00
Richard van der Hoff	99e36d5e24	Implement MSC1708 (.well-known lookups for server routing) (#4489)	2019-01-29 13:53:02 +00:00
Erik Johnston	b82a76c384	Finish comment...	2019-01-29 13:50:59 +00:00
Erik Johnston	2562319821	Merge pull request #4510 from matrix-org/erikj/fixup_compute_event_signature ... Fixup calls to `comput_event_signature`	2019-01-29 13:35:19 +00:00
Erik Johnston	5891a6edc8	Correctly set context.app_service	2019-01-29 12:09:10 +00:00
Erik Johnston	fb99dae9c8	Don't set event_id twice	2019-01-29 12:08:23 +00:00
Erik Johnston	64c1bd1d21	Remove dead function	2019-01-29 12:06:28 +00:00
Erik Johnston	7709d2bd16	Implement rechecking of redactions	2019-01-29 11:56:20 +00:00
Erik Johnston	7a3ec5b022	Add RoomVersions.V3 constant, without enabling it ... We add the constant, but don't add it to the known room versions. This lets us start adding V3 logic, but the servers will never join or create V3 rooms	2019-01-29 11:55:33 +00:00
Erik Johnston	6598992b01	Fixup calls to comput_event_signature ... We currently pass FrozenEvent instead of `dict` to `compute_event_signature`, which works by accident due to `dict(event)` producing the correct result. This fixes PR #4493 commit 855a151	2019-01-29 11:41:58 +00:00
Neil Johnson	a124025dab	enforce auth for capabilities endpoint	2019-01-29 11:37:56 +00:00
Erik Johnston	ff37acb8ce	Merge pull request #4496 from matrix-org/erikj/invite_fallback ... Implement fallback for V2 invite API	2019-01-29 11:28:23 +00:00
Erik Johnston	5180f12bae	Replace usage of builder.user_id with builder.sender ... `.user_id` is proxed to `.sender` in FrozenEvent, so this has no functional change	2019-01-29 11:18:38 +00:00
Erik Johnston	be47cfa9c9	Refactor event building into EventBuilder ... This is so that everything is done in one place, making it easier to change the event format based on room version	2019-01-29 11:13:00 +00:00
Erik Johnston	554ca58ea1	Make add_hashes_and_signatures operate on dicts	2019-01-29 11:12:38 +00:00
Amber Brown	f815bd7feb	Make linearizer more quiet (#4507)	2019-01-29 11:05:31 +00:00
Erik Johnston	073f6c2e5e	Merge pull request #4494 from matrix-org/erikj/fixup_event_validator ... Split up event validation between event and builder	2019-01-29 10:55:07 +00:00
Erik Johnston	40638ae7f5	Remove duplicate checks	2019-01-29 10:37:40 +00:00
Erik Johnston	9fa3c6ffa3	Fix up error messages	2019-01-29 10:36:46 +00:00
Erik Johnston	28efc80723	Fold validate into validate_new	2019-01-29 10:34:49 +00:00
Erik Johnston	b6b73a0bcf	Fix receiving events from federation via a worker ... This bug was introduced in PR #4470, commit 678a92cb56	2019-01-29 10:30:26 +00:00
Neil Johnson	327b992e17	register capabilities servlet	2019-01-29 10:28:35 +00:00
Amber Brown	94fb63e44f	Fix typo in upserts code (#4505) ... * fix obvious problem :| * changelog	2019-01-29 10:04:23 +00:00
Erik Johnston	17709f8f9c	Merge pull request #4493 from matrix-org/erikj/refactor_event_signing ... Refactor event signing to work on dicts	2019-01-29 09:52:54 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version ... Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Travis Ralston	6901ac7e9d	Don't recommend :8448 to people on public_baseurl	2019-01-28 12:15:22 -07:00
Erik Johnston	d414f30019	Implement fallback for V2 invite API ... If the room version is either 1 or 2 then a server should retry failed `/v2/invite` requests with the v1 API	2019-01-28 17:33:25 +00:00
Erik Johnston	d758d5310e	Correctly use default room version if none is set	2019-01-28 17:26:39 +00:00
Amber Brown	5d976c0c7c	Fix worker TLS (#4492) ... * load cert * changelog * fix	2019-01-28 17:18:33 +00:00
Erik Johnston	b872c7b1b4	Split up event validation between event and builder ... The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.	2019-01-28 17:00:14 +00:00
Erik Johnston	855a151015	Refactor event signing to work on dicts ... This is in preparation for making EventBuilder format agnostic, which means event signing should be done against the event dict rather than the EventBuilder object.	2019-01-28 16:42:10 +00:00
Amber Brown	7072fe3084	Fix UPSERTs on SQLite 3.24+ (#4477)	2019-01-28 15:43:32 +00:00
Andrew Morgan	f0e96ab66a	Change return syntax in doc string	2019-01-28 14:09:45 +00:00
Andrew Morgan	4026d555fa	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-28 14:08:24 +00:00
Andrew Morgan	1ce463963d	Reuse predecessor method	2019-01-28 14:08:18 +00:00
Richard van der Hoff	3bd0f1a4a3	docstrings for _RoutingResult	2019-01-28 12:43:09 +00:00
Richard van der Hoff	0fd5b3b53e	Handle IP literals explicitly ... We don't want to be doing .well-known lookups on these guys.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	51958df766	MatrixFederationAgent: factor out routing logic ... This is going to get too big and unmanageable.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	d840019192	Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487) ... Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.	2019-01-28 09:56:59 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Neil Johnson	893107be78	backout v3	2019-01-25 17:27:36 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version ... Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Erik Johnston	431e485914	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert	2019-01-25 14:11:17 +00:00
Richard van der Hoff	4a3f138832	Fix quoting for allowed_local_3pids example config (#4476) ... If you use double-quotes here, you have to escape your backslashes. It's much easier with single-quotes. (Note that the existing double-backslashes are already interpreted by python's """ parsing.)	2019-01-25 13:57:52 +00:00
Richard van der Hoff	8520bc3109	Fix Host header sent by MatrixFederationAgent (#4468) ... Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.	2019-01-25 12:38:16 +00:00
Andrew Morgan	9244a3089e	Fixes	2019-01-25 11:48:38 +00:00
Andrew Morgan	da0d2219d2	Clean up direct_rooms access	2019-01-25 11:37:12 +00:00
Andrew Morgan	8265995498	Use python magic	2019-01-25 11:26:06 +00:00
Andrew Morgan	0b3fd1401f	Don't require sqlite3 when using postgres (#4466)	2019-01-25 22:25:02 +11:00
Andrew Morgan	c4cdafa81f	Destructure account data tuple before use	2019-01-25 11:24:28 +00:00
Andrew Morgan	516456b763	Remove unnecessary null check	2019-01-25 11:22:14 +00:00
Andrew Morgan	6f3fda79ce	Move room_tag declaration to be closer to its use	2019-01-25 11:21:25 +00:00
Neil Johnson	95f871fc0d	Support room version capabilities in CS API (MSC1804)	2019-01-25 11:16:29 +00:00
Andrew Morgan	821b65aeb5	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-25 11:09:53 +00:00
Andrew Morgan	0862d35b8e	Move tag and direct state copying into separate function	2019-01-25 11:09:34 +00:00
Neil Johnson	53ef4da8c2	track unstable room v3	2019-01-25 11:04:11 +00:00
Andrew Morgan	b1b6dba2d2	Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms ... Ability to search entire room history after upgrading room	2019-01-25 10:49:58 +00:00
Erik Johnston	1953067136	Disable native upserts for sqlite, as they don't work	2019-01-25 10:46:49 +00:00
Erik Johnston	678a92cb56	Replace missed usages of FrozenEvent	2019-01-25 10:32:30 +00:00
Erik Johnston	a50cf929c1	Require event format version to parse or create events	2019-01-25 10:32:19 +00:00
Erik Johnston	62514bb81b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813	2019-01-25 10:07:08 +00:00
Erik Johnston	edc1e21dbe	Merge pull request #4448 from matrix-org/erikj/get_pdu_versions ... Add room_version param to get_pdu	2019-01-24 18:47:15 +00:00
Erik Johnston	80bcca659e	Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites ... Store rejected remote invite events as outliers	2019-01-24 18:33:27 +00:00
Erik Johnston	664b7a2920	Merge pull request #4437 from matrix-org/erikj/event_format_version_v2 ... Add support for persisting event format versions	2019-01-24 18:32:30 +00:00
Erik Johnston	d148c43050	Review comments	2019-01-24 18:31:23 +00:00
Erik Johnston	26f44164c8	Review comments	2019-01-24 18:28:00 +00:00
Erik Johnston	5ee1f997a8	Update make_membership_event docs	2019-01-24 18:08:08 +00:00
Erik Johnston	9139b87be4	Remove unecessary setting of outlier bit	2019-01-24 18:04:02 +00:00
Erik Johnston	b8082a5445	Use term 'out of band membership' instead	2019-01-24 17:33:19 +00:00
Andrew Morgan	a383289b0d	Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms	2019-01-24 17:23:51 +00:00
Andrew Morgan	e1781b043b	Remove redundant create event None check	2019-01-24 17:23:39 +00:00
Richard van der Hoff	03c85335d1	Apply suggestions from code review ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:22:09 +00:00
Richard van der Hoff	8ea509a935	Update synapse/api/filtering.py ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:21:35 +00:00
Andrew Morgan	50c396a7ee	Merge pull request #4461 from matrix-org/anoa/room_dir_quick_fix ... Add a 60s timeout to filtered room directory queries	2019-01-24 15:24:35 +00:00
Andrew Morgan	075ff3ede9	Change default timeout value from 0 to None	2019-01-24 15:10:22 +00:00
Andrew Morgan	5b1dc94083	Use self.clock instead of datetime	2019-01-24 14:59:50 +00:00
Andrew Morgan	a2d85144e5	isort	2019-01-24 14:22:26 +00:00
Richard van der Hoff	4a6e863843	Merge pull request #4464 from matrix-org/rav/fix_srv_lookup ... MatrixFederationAgent: Look up the right SRV record	2019-01-24 13:57:51 +00:00
Richard van der Hoff	f4697b5ec1	Fix UnboundLocalError in post_urlencoded_get_json (#4460) ... This could cause exceptions if the id server returned 4xx responses.	2019-01-24 13:38:29 +00:00
Richard van der Hoff	afd69a0920	Look up the right SRV record	2019-01-24 13:31:43 +00:00
Neil Johnson	10b89d5c2e	Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check ... Neilj/fix threepid auth check	2019-01-24 13:02:50 +00:00
Andrew Morgan	5541645e80	lint	2019-01-24 12:45:32 +00:00
Andrew Morgan	068aa1d228	Time out filtered room dir queries after 60s	2019-01-24 12:44:27 +00:00
Neil Johnson	5c41b22359	Merge pull request #4458 from matrix-org/dbkr/public_baseurl_doc ... Clarify docs for public_baseurl	2019-01-24 12:36:09 +00:00
Amber Brown	0e27501ee5	Fix UPSERT check (#4459)	2019-01-24 22:57:41 +11:00
David Baker	92d8a068ad	Clarify docs for public_baseurl ... This is leading to problems with people upgrading to clients that support MSC1730 because people have this misconfigured, so try to make the docs completely unambiguous.	2019-01-24 10:52:06 +00:00
Amber Brown	58f6c48183	Use native UPSERTs where possible (#4306)	2019-01-24 21:31:54 +11:00
Richard van der Hoff	97fd29c019	Don't send IP addresses as SNI (#4452) ... The problem here is that we have cut-and-pasted an impl from Twisted, and then failed to maintain it. It was fixed in Twisted in https://github.com/twisted/twisted/pull/1047/files; let's do the same here.	2019-01-24 09:34:44 +00:00
Erik Johnston	7c288c2250	Clarify the invite flows	2019-01-23 20:07:47 +00:00
Erik Johnston	07f62da55a	Remove unnecessary '_sign_event'	2019-01-23 20:04:57 +00:00
Erik Johnston	886e5acc76	Store rejected remote invite events as outliers ... Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.	2019-01-23 20:04:57 +00:00
Willem Mulder	d528406cb8	Fix error message for optional dependencies ... Signed-off-by: Willem Mulder <willemmaster@hotmail.com>	2019-01-23 18:44:57 +01:00
Erik Johnston	6a41d2a187	Add room_version param to get_pdu ... When we add new event format we'll need to know the event format or room version when parsing events.	2019-01-23 17:19:58 +00:00
Erik Johnston	67cd4dad81	Implement MSC 1813 - Add room version to make APIs ... We also implement `make_membership_event` converting the returned room version to an event format version.	2019-01-23 16:51:46 +00:00
Andrew Morgan	6b90ae6efc	Merge pull request #4445 from matrix-org/anoa/user_dir_develop_backport ... Add metric for user dir current event stream position	2019-01-23 15:58:40 +00:00
Richard van der Hoff	a0ae475219	Merge pull request #4428 from matrix-org/rav/matrix_federation_agent ... Move SRV magic into an Agent-like thing	2019-01-23 15:50:25 +00:00
Andrew Morgan	82a92ba535	Add metric for user dir current event stream position	2019-01-23 15:34:47 +00:00
Erik Johnston	be1065af59	isort	2019-01-23 11:48:16 +00:00
Erik Johnston	c5a296b10c	Add support for persisting event format versions ... Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.	2019-01-23 11:30:01 +00:00
Jason Robinson	6f680241bd	Fix flake8 issues ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:53:48 +02:00
Erik Johnston	90743c9d89	Fixup removal of duplicate user_ips rows (#4432) ... * Remove unnecessary ORDER BY clause * Add logging * Newsfile	2019-01-23 19:45:18 +11:00
Amber Brown	6129e52f43	Support ACME for certificate provisioning (#4384)	2019-01-23 19:39:06 +11:00
Jason Robinson	0516dc4d85	Remove openid resource from default config ... Instead document it commented out. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	d39b7b6d38	Document servlet_groups parameters ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	82e13662c0	Split federation OpenID userinfo endpoint out of the federation resource ... This allows the OpenID userinfo endpoint to be active even if the federation resource is not active. The OpenID userinfo endpoint is called by integration managers to verify user actions using the client API OpenID access token. Without this verification, the integration manager cannot know that the access token is valid. The OpenID userinfo endpoint will be loaded in the case that either "federation" or "openid" resource is defined. The new "openid" resource is defaulted to active in default configuration. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	a17bac171f	Make SynapseHomeServer _http_listener use self.get_reactor() ... For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	4f8f41c824	Make FederationReaderServer _http_listen use self.get_reactor() ... For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	899e60be80	Add parameterized Python module to test dependencies ... Allows running parameterized tests. BSD license. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Richard van der Hoff	7021784d46	put resolve_service in an object ... this makes it easier to stub things out for tests.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	53a327b4d5	Require that service_name be a byte string ... it is only ever a bytes now, so let's enforce that.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	fe212bbe4a	Kill off matrix_federation_endpoint ... this thing is now redundant.	2019-01-22 20:34:35 +00:00
Richard van der Hoff	7871146667	Make MatrixFederationClient use MatrixFederationAgent ... ... instead of the matrix_federation_endpoint	2019-01-22 20:34:35 +00:00
Richard van der Hoff	44be7513bf	MatrixFederationAgent ... Pull the magic that is currently in matrix_federation_endpoint and friends into an agent-like thing	2019-01-22 20:34:35 +00:00
Neil Johnson	c99c2d58d7	move guard out of is_threepid_reserved and into register.py	2019-01-22 17:47:00 +00:00
Neil Johnson	d619b113ed	Fix None guard in config.server.is_threepid_reserved	2019-01-22 16:52:29 +00:00
Erik Johnston	12699a701f	Merge pull request #4434 from matrix-org/erikj/fix_user_ips_dedup ... Fix bug when removing duplicate rows from user_ips	2019-01-22 16:51:57 +00:00
Erik Johnston	7f503f83b9	Refactor to rewrite the SQL instead	2019-01-22 16:31:05 +00:00
Neil Johnson	388c164aea	Merge pull request #4423 from matrix-org/neilj/disable_msisdn_on_registration ... Config option to disable requesting MSISDN on registration	2019-01-22 16:23:08 +00:00
Erik Johnston	1c9704f8ab	Don't shadow params	2019-01-22 16:20:33 +00:00
Andrew Morgan	766a172b99	lint	2019-01-22 13:51:40 +00:00
Erik Johnston	2557531f0f	Fix bug when removing duplicate rows from user_ips ... This was caused by accidentally overwritting a `last_seen` variable in a for loop, causing the wrong value to be written to the progress table. The result of which was that we didn't scan sections of the table when searching for duplicates, and so some duplicates did not get deleted.	2019-01-22 13:33:46 +00:00
Andrew Morgan	117bc94cd2	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-22 13:16:51 +00:00
Andrew Morgan	277e50462d	Do not return in a deferred function	2019-01-22 12:40:26 +00:00
Andrew Morgan	c4875d8c76	Prevent duplicate room IDs in m.direct	2019-01-22 12:13:46 +00:00
Andrew Morgan	c433f61091	Ensure new filter is actually created	2019-01-22 12:06:36 +00:00
Andrew Morgan	c9bfb058d8	Fix a bug with single-room search searching all rooms ... * Create a new method for getting predecessor rooms * Remove formatting change	2019-01-22 12:00:41 +00:00
Andrew Morgan	8086a5c05e	Fix comments	2019-01-22 11:16:23 +00:00
Richard van der Hoff	6bfa735a69	Make key fetches use regular federation client (#4426) ... All this magic is redundant.	2019-01-22 11:04:20 +00:00
Andrew Morgan	48951f437f	Join logic covers both room creator and arbitrary users	2019-01-22 11:00:04 +00:00
Andrew Morgan	8c85f0833d	tags, m.direct copying over correctly	2019-01-22 11:00:04 +00:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Andrew Morgan	ea8903fcc9	Migrating dm and room tags work for migrator	2019-01-22 11:00:04 +00:00
Andrew Morgan	887ca93a1b	Prevent crash on user who doesn't have any direct rooms	2019-01-22 11:00:04 +00:00
Andrew Morgan	4ff6d22245	Preserve DM status of a room on upgrade ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-22 11:00:04 +00:00
Andrew Morgan	75942af1db	Fix typo	2019-01-22 11:00:04 +00:00
Richard van der Hoff	33a55289cb	Refactor and bugfix for resove_service (#4427)	2019-01-22 10:59:27 +00:00
Amber Brown	23b0813599	Require ECDH key exchange & remove dh_params (#4429) ... * remove dh_params and set better cipher string	2019-01-22 21:58:50 +11:00
Neil Johnson	1b53cc3cb4	fix line length	2019-01-21 15:17:20 +00:00
Neil Johnson	5349262302	Config option to disable requesting MSISDN on registration	2019-01-21 14:59:37 +00:00
Erik Johnston	35e1d67b4e	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_v2_invite_server	2019-01-21 14:04:19 +00:00
Erik Johnston	5f54765587	Merge pull request #4390 from matrix-org/erikj/versioned_fed_apis ... Add groundwork for new versions of federation APIs	2019-01-21 11:44:05 +00:00
Andrew Morgan	702c4b750c	Migrate encryption state on room upgrade (#4411) ... * Migrate encryption state on room upgrade Signed-off-by: Andrew Morgan <andrew@amorgan.xyz> * Add changelog file	2019-01-21 20:42:58 +11:00
Erik Johnston	25dd56ace3	Fix race when persisting create event (#4404) ... * Fix race when persisting create event When persisting a chunk of DAG it is sometimes requried to do a state resolution, which requires knowledge of the room version. If this happens while we're persisting the create event then we need to use that event rather than attempting to look it up in the database.	2019-01-18 23:17:04 +11:00
Richard van der Hoff	de6888e7ce	Remove redundant WrappedConnection (#4409) ... * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.	2019-01-18 23:07:38 +11:00
Andrew Morgan	df3a661e4a	Search for messages across predecessor rooms ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-18 11:19:20 +00:00
Richard van der Hoff	676cf2ee26	Fix incorrect logcontexts after a Deferred was cancelled (#4407)	2019-01-17 14:00:23 +00:00
Richard van der Hoff	9feb5d0b71	sign_request -> build_auth_headers (#4408) ... Just got very confused about the fact that the headers are only an output, not an input.	2019-01-17 12:40:09 +00:00
Richard van der Hoff	3982a6ee07	Changing macaroon_secret_key no longer logs you out (#4387)	2019-01-16 23:14:41 +00:00
Richard van der Hoff	05e1296649	don't store more remote device lists if they have more than 1K devices (#4397)	2019-01-16 23:14:11 +00:00
Andrej Shadura	64cf6788d9	Depend on pymacaroons >= 0.13.0 instead on pymacaroons-pynacl ... Since 0.13.0, pymacaroons works correctly with pynacl, so there isn’t any more reason to depend on an outdated pynacl fork. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>	2019-01-16 10:54:41 +01:00
Andrej Shadura	fab948120f	Use msgpack instead of msgpack-python ... The package msgpack-python has been deprecated. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>	2019-01-16 10:53:49 +01:00
Neil Johnson	9ec56d6935	ALL_USER_TYPES should be a tuple	2019-01-15 14:38:15 +00:00
Erik Johnston	4a4d2e17bc	Add /v2/invite federation API	2019-01-15 13:22:44 +00:00
Erik Johnston	bb63e7ca4f	Add groundwork for new versions of federation APIs	2019-01-15 11:14:34 +00:00
Amber Brown	a35c66a00b	Remove duplicates in the user_ips table and add an index (#4370)	2019-01-12 06:21:50 +11:00
Amber Brown	522dada206	Merge remote-tracking branch 'origin/master' into develop	2019-01-11 02:22:48 +11:00
Amber Brown	6dc06c3775	version	2019-01-11 01:56:37 +11:00
Richard van der Hoff	e0910d0145	Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 ... Fixes #4371	2019-01-10 14:12:50 +00:00
Richard van der Hoff	ba41aeed6a	Revert "Fix macaroon_secret_key fallback logic" ... This is already fixed in 0.34.1, by 59f93bb This reverts commit efc522c55e.	2019-01-10 14:09:26 +00:00
Richard van der Hoff	aa70d24125	Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34	2019-01-10 12:58:33 +00:00
Richard van der Hoff	efc522c55e	Fix macaroon_secret_key fallback logic	2019-01-10 12:57:27 +00:00
Richard van der Hoff	566947ff34	Skip macaroon check for access tokens in the db	2019-01-10 12:57:21 +00:00
Richard van der Hoff	353f2407b7	Fix fallback to signing key for macaroon-secret-key	2019-01-10 12:42:56 +00:00
Richard van der Hoff	d566e6b17a	Merge branch 'master' into develop	2019-01-09 15:00:46 +00:00
Richard van der Hoff	998f5225c1	0.34.1	2019-01-09 14:53:54 +00:00
Erik Johnston	55c3e853c5	Merge pull request #4368 from matrix-org/erikj/better_errors ... Fixup docstrings for matrixfederationclient	2019-01-09 11:39:28 +00:00
Erik Johnston	484867d35d	Merge branch 'develop' into release-v0.34.1	2019-01-09 11:38:13 +00:00
Amber Brown	7960c26fda	Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369)	2019-01-09 22:26:25 +11:00
Erik Johnston	34ea14139d	Fixup docstrings for matrixfederationclient	2019-01-09 09:25:59 +00:00
Erik Johnston	d422570e9b	Use RequestSendFailed when fail to parse content type headers	2019-01-08 14:22:18 +00:00
Erik Johnston	1371d5b798	Don't log stack traces for HTTP error responses	2019-01-08 12:28:30 +00:00
Richard van der Hoff	ba85786d71	0.34.1rc1	2019-01-08 11:15:34 +00:00
Erik Johnston	b970cb0e96	Refactor request sending to have better excpetions (#4358) ... * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.	2019-01-08 11:04:28 +00:00
Amber Brown	1dcb086f33	Fix synapse.config.__main__ on python 3 (#4356)	2019-01-08 10:03:09 +11:00
Travis Ralston	bc1fa8cd01	Add GET account data routes (#4303) ... As per https://github.com/matrix-org/matrix-doc/issues/1339	2019-01-07 10:27:54 +00:00
Richard van der Hoff	b7c0218812	Check jinja version for consent resource (#4327) ... * Raise a ConfigError if an invalid resource is specified * Require Jinja 2.9 for the consent resource * changelog	2019-01-07 10:14:31 +00:00
Michael Telatynski	a27e501b09	fix the check for whether is_url to match all the other ones in codebase (#3405) ... Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2019-01-06 21:43:25 +00:00
Matthew Hodgson	27128145e6	fix NPE in /messages by checking if all events were filtered out (#4330)	2019-01-02 23:37:39 +00:00
Neil Johnson	84b6fae1f5	Ensure synchrotrons can access is_support_user in the storage layer	2019-01-02 10:19:59 +00:00
Amber Brown	d7843f47b6	Remove v1 only REST APIs now we don't ship matrix console (#4334)	2018-12-29 23:12:30 +11:00
Will Hunt	0708f437cc	Log roomid along with Unknown room (#4297)	2018-12-24 10:49:42 +01:00
Richard van der Hoff	cd018e3543	Merge pull request #4307 from matrix-org/erikj/v2_rooms ... Add v2 room version	2018-12-24 10:48:59 +01:00
Richard van der Hoff	f85676cc93	Return well_known in /login response (#4319) ... ... as per MSC1730.	2018-12-24 10:44:33 +01:00
Richard van der Hoff	9c2af7b2c5	Add a script to generate a clean config file (#4315)	2018-12-22 02:04:57 +11:00
Richard van der Hoff	f3561f8d86	Fix indentation in default config (#4313) ... These settings are not supposed to be under 'listeners'.	2018-12-22 02:01:43 +11:00
Amber Brown	ea6abf6724	Fix IP URL previews on Python 3 (#4215)	2018-12-22 01:56:13 +11:00
Amber Brown	c26f49a664	Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298)	2018-12-22 01:37:26 +11:00
Richard van der Hoff	0b26feb422	Merge branch 'master' into develop	2018-12-20 16:48:47 +00:00
Richard van der Hoff	6dd7271243	Prepare 0.34 release	2018-12-20 12:07:15 +00:00
Richard van der Hoff	d3c9c562c6	Debian packaging via dh_virtualenv	2018-12-20 11:35:23 +00:00
Richard van der Hoff	1a6d5bfa08	Debian packaging via dh_virtualenv (#4285)	2018-12-20 11:33:29 +00:00
David Baker	1c0051114a	Add 'sandbox' to CSP for media repo (#4284) ... * Add 'sandbox' to the CSP for media repo * Changelog	2018-12-20 11:09:18 +00:00
Nathan Pennie	81b513416e	Fixed line length	2018-12-19 14:12:33 -05:00
Nathan Pennie	668e6625b0	Fix the variable names used for account_data	2018-12-19 06:53:02 -05:00
Erik Johnston	df89f8afb8	Add v2 room version	2018-12-18 18:10:37 +00:00
Neil Johnson	7e22cd90f5	ensure can report mau stats when hs.config.mau_stats_only is set (#4305) ... * ensure can report mau stats when hs.config.mau_stats_only is set	2018-12-18 14:36:11 +00:00
Neil Johnson	d2f7c4e6b1	create support user (#4141) ... Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Nad Chishtie	128294d3e5	Improved welcome page (#4294) ... * Improved static/index.html look, feel and content. * Added accessibility tags to in-line SVG. * Grammar.	2018-12-14 05:13:56 +11:00
Amber Brown	c7144b105c	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:13:33 +11:00
Amber Brown	e93a0ebf50	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:10:31 +11:00
Nad Chishtie	cef8ae272a	Improved welcome page (#4294) ... * Improved static/index.html look, feel and content. * Added accessibility tags to in-line SVG. * Grammar.	2018-12-14 02:42:51 +11:00
Richard van der Hoff	f208f608cb	Merge branch 'release-v0.34.0' into develop	2018-12-11 15:43:20 +00:00
Richard van der Hoff	eb835bf65b	0.34.0rc2	2018-12-11 15:35:47 +00:00
Richard van der Hoff	188945713e	Merge pull request #4290 from matrix-org/rav/remove_webclient ... Stop installing Matrix Console by default	2018-12-11 16:24:15 +01:00
Richard van der Hoff	95c1f6500b	Merge pull request #4289 from matrix-org/rav/welcome_page ... Add a welcome page to the static resources	2018-12-11 16:24:06 +01:00
Richard van der Hoff	f537432ef9	Add a welcome page to the static resources ... This is largely a precursor for the removal of the bundled webclient. The idea is to present a page at / which reassures people that something is working, and to give them some links for next steps. The welcome page lives at `/_matrix/static/`, so is enabled alongside the other `static` resources (which, in practice, means the client API is enabled). We'll redirect to it from `/` if we have nothing better to display there. It would be nice to have a way to disable it (in the same way that you might disable the nginx welcome page), but I can't really think of a good way to do that without a load of ickiness. It's based on the work done by @krombel for #2601.	2018-12-11 13:26:22 +00:00
Richard van der Hoff	df96177ca7	Stop installing Matrix Console by default ... This is based on the work done by @krombel in #2601.	2018-12-11 13:20:33 +00:00
David Baker	89ac2a5bdb	Add 'sandbox' to CSP for media repo (#4284) ... * Add 'sandbox' to the CSP for media repo * Changelog	2018-12-11 04:05:02 +11:00
Will Hunt	fee831c040	Move imports to one line	2018-12-10 13:52:33 +00:00
Will Hunt	466c1f3e01	Use send_cors	2018-12-10 13:11:37 +00:00
Will Hunt	91206e09f2	changelog & isort	2018-12-09 17:39:44 +00:00
Will Hunt	dbf736ba66	Make /config more CORS-y	2018-12-09 13:27:22 +00:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272) ... When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	c7401a697f	Implement SAML2 authentication (#4267) ... This implements both a SAML2 metadata endpoint (at `/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at `/_matrix/saml2/authn_response`). If the SAML2 response matches what's been configured, we complete the SSO login flow by redirecting to the client url (aka `RelayState` in SAML2 jargon) with a login token. What we don't yet have is anything to build a SAML2 request and redirect the user to the identity provider. That is left as an exercise for the reader.	2018-12-07 13:11:11 +01:00
Richard van der Hoff	c588b9b9e4	Factor SSO success handling out of CAS login (#4264) ... This is mostly factoring out the post-CAS-login code to somewhere we can reuse it for other SSO flows, but it also fixes the userid mapping while we're at it.	2018-12-07 13:10:07 +01:00
rkfg	ae19a7db8c	Prevent crash on pagination.	2018-12-06 13:32:05 +03:00
Richard van der Hoff	b0c24a66ec	Rip out half-implemented m.login.saml2 support (#4265) ... * Rip out half-implemented m.login.saml2 support This was implemented in an odd way that left most of the work to the client, in a way that I really didn't understand. It's going to be a pain to maintain, so let's start by ripping it out. * drop undocumented dependency on dateutil It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 19:44:38 +11:00
Richard van der Hoff	9a3e24a13d	drop undocumented dependency on dateutil (#4266) ... It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 04:52:42 +11:00
Richard van der Hoff	e8d98466b0	Implement .well-known handling (#4262) ... Sometimes it's useful for synapse to generate its own .well-known file.	2018-12-05 14:38:58 +01:00
Richard van der Hoff	e55983defe	Prepare 0.34.0rc1	2018-12-04 13:52:16 +00:00
Travis Ralston	1737753a62	Add an option to enable recording IPs for appservice users (#3831)	2018-12-04 12:44:41 +01:00
Amber Brown	fd96dd75a3	Fix non-ASCII pushrules (#4248)	2018-12-04 12:44:02 +01:00
Travis Ralston	158ffb92f1	Add an option to disable search for homeservers which may not be interested in it (#4230) ... This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.	2018-12-04 12:01:02 +01:00
Ben Parsons	b5ac0ffa0a	add more detail to logging regarding "More than one row matched" error (#4234)	2018-12-04 11:57:39 +01:00
Richard van der Hoff	ecc23188f4	Fix UnicodeDecodeError when postgres is not configured in english (#4253) ... This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.	2018-12-04 11:55:52 +01:00
Richard van der Hoff	f144c0a210	Merge pull request #4244 from aaronraimist/drop-sent-txt ... Drop sent_transactions	2018-12-04 11:41:28 +01:00
Richard van der Hoff	a077e710a3	Merge pull request #4250 from matrix-org/hawkowl/pusher-remove-py3 ... Fix removing pushers on python 3	2018-12-04 11:22:46 +01:00
Richard van der Hoff	52e87fbfbe	Run the AS senders as background processes (#4189) ... This should fix some "Starting db connection from sentinel context" warnings, and will mean we get metrics for these processes.	2018-12-04 10:53:49 +01:00
Amber Brown	d3c61ef906	fix type error	2018-12-03 22:27:41 +11:00
Richard van der Hoff	c03324294d	Workaround for non-ascii event ids (#4241) ... It turns out that we accept events with non-ascii IDs, which would later cause an explosion during state res. Fixes #4226	2018-12-03 21:47:48 +11:00
Aaron Raimist	704c5298f0	Drop sent_transactions ... Signed-off-by: Aaron Raimist <aaron@raim.ist>	2018-12-01 23:07:35 -06:00
Neil Johnson	7039ece8fb	Neilj/fix autojoin (#4223) ... * Fix auto join failures for servers that require user consent * Fix auto join failures for servers that require user consent	2018-11-28 22:24:57 +11:00
Neil Johnson	f9b136a886	Neilj/fix mau initial reserved users (#4211) ... * fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail * towncrier * white space	2018-11-28 20:33:41 +11:00
Richard van der Hoff	944d524f18	Support m.login.sso (#4220) ... * Clean up the CSS for the fallback login form I was finding this hard to work with, so simplify a bunch of things. Each flow is now a form inside a div of class login_flow. The login_flow class now has a fixed width, as that looks much better than each flow having a differnt width. * Support m.login.sso MSC1721 renames m.login.cas to m.login.sso. This implements the change (retaining support for m.login.cas for older clients). * changelog	2018-11-27 18:51:52 +11:00
Richard van der Hoff	6c18cc4b50	Ignore __pycache__ directories in schema delta dir ... Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.	2018-11-20 22:52:34 +00:00
Neil Johnson	78ba0e7ab8	Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)	2018-11-20 12:29:25 +01:00
Richard van der Hoff	416c671474	Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes ... Fix some logcontext leaks	2018-11-20 12:19:19 +01:00
Amber Brown	31425d82a3	Merge remote-tracking branch 'origin/master' into develop	2018-11-19 12:55:25 -06:00
Amber Brown	d102e19e47	version	2018-11-19 12:42:49 -06:00
Amber Brown	80cac86b2c	Fix fallback auth on Python 3 (#4197)	2018-11-19 12:27:33 -06:00
Richard van der Hoff	f5faf6bc14	Fix logcontext leak in EmailPusher	2018-11-19 17:07:01 +00:00
Travis Ralston	ab4526a153	Remove duplicate slashes in generated consent URLs	2018-11-15 20:41:53 -07:00
Amber Brown	8b1affe7d5	Fix Content-Disposition in media repository (#4176)	2018-11-15 15:55:58 -06:00
Travis Ralston	835779f7fb	Add option to track MAU stats (but not limit people) (#3830)	2018-11-15 18:08:27 +00:00
Amber Brown	df758e155d	Use <meta> tags to discover the per-page encoding of html previews (#4183)	2018-11-15 11:05:08 -06:00
Neil Johnson	bf648c37e7	release 0.33.9rc1	2018-11-14 11:45:52 +00:00
Richard van der Hoff	c1efcd7c6a	Add a test for the public T&Cs form	2018-11-14 10:46:27 +00:00
Richard van der Hoff	83a5f459aa	Fix an internal server error when viewing the public privacy policy	2018-11-14 10:21:07 +00:00
David Baker	0869566ad3	Merge pull request #4113 from matrix-org/dbkr/e2e_backup_versions_are_numbers ... Make e2e backup versions numeric in the DB	2018-11-14 07:55:48 +00:00
David Baker	bca3b91c2d	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers	2018-11-09 18:35:02 +00:00
Erik Johnston	db5a1c059a	Merge pull request #4166 from matrix-org/erikj/drop_unknown_events ... Drop incoming events from federation for unknown rooms	2018-11-09 17:59:34 +00:00
Erik Johnston	dc59ad5334	Remove hack to support rejoining rooms	2018-11-09 14:58:09 +00:00
David Baker	d44dea0223	pep8	2018-11-09 14:38:31 +00:00
David Baker	4f93abd62d	add docs	2018-11-09 13:25:38 +00:00
Erik Johnston	30dd27afff	Simplify to always drop events if server isn't in the room	2018-11-09 11:36:45 +00:00
Richard van der Hoff	3cecf5340d	Update synapse/federation/federation_server.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2018-11-09 11:28:25 +00:00
Richard van der Hoff	9bce065a53	Update synapse/federation/federation_server.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2018-11-09 11:28:22 +00:00
David Baker	d3fa6194f7	Remove unnecessary str()	2018-11-09 11:11:31 +00:00
Erik Johnston	7b22421a7b	Merge pull request #4164 from matrix-org/erikj/fix_device_comparison ... Fix noop checks when updating device keys	2018-11-08 14:37:20 +00:00
Erik Johnston	5ebed18692	Lets convert bytes to unicode instead	2018-11-08 12:33:13 +00:00
Erik Johnston	b1a22b24ab	Fix noop checks when updating device keys ... Clients often reupload their device keys (for some reason) so its important for the server to check for no-ops before sending out device list update notifications. The check is broken in python 3 due to the fact comparing bytes and unicode always fails, and that we write bytes to the DB but get unicode when we read.	2018-11-08 12:18:38 +00:00
Erik Johnston	9417986f77	Drop PDUs of unknown rooms ... When we receive events over federation we will need to know the room version to be able to correctly handle them, e.g. once we start changing event formats. Currently, we attempt to handle events in unknown rooms.	2018-11-08 12:11:20 +00:00
Richard van der Hoff	0a1fc52971	fix parse_string docstring	2018-11-08 11:12:29 +00:00
hera	2b075fb03a	Fix encoding error for consent form on python3 ... The form was rendering this as "b'01234....'". -- richvdh	2018-11-08 11:05:39 +00:00
Amber Brown	b3708830b8	Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157)	2018-11-08 01:37:43 +11:00
Travis Ralston	0f5e51f726	Add config variables for enabling terms auth and the policy name (#4142) ... So people can still collect consent the old way if they want to.	2018-11-06 10:32:34 +00:00
Hubert Chathi	f1087106cf	handle empty backups according to latest spec proposal (#4123) ... fixes #4056	2018-11-05 17:59:29 -05:00
Amber Brown	efdcbbe46b	Tests for user consent resource (#4140)	2018-11-06 05:53:44 +11:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139) ... * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Amber Brown	0467384d2f	Set the encoding to UTF8 in the default logconfig (#4138)	2018-11-03 02:28:07 +11:00
Erik Johnston	90d713b8c6	Merge pull request #4137 from matrix-org/erikj/clean_up_events ... Clean up event accesses and tests	2018-11-02 14:12:49 +00:00
Erik Johnston	b86d05a279	Clean up event accesses and tests ... This is in preparation to refactor FrozenEvent to support different event formats for different room versions	2018-11-02 13:44:14 +00:00
Amber Brown	cb7a6b2379	Fix typing being reset causing infinite syncs (#4127)	2018-11-03 00:19:23 +11:00
Richard van der Hoff	efb9343c8c	Merge pull request #4132 from matrix-org/rav/fix_device_list_locking ... Fix locked upsert on device_lists_remote_cache	2018-11-02 10:50:53 +00:00
Richard van der Hoff	00f12e00f8	Merge pull request #4133 from matrix-org/travis/fix-terms-auth ... Fix logic error that prevented guests from seeing the privacy policy	2018-11-02 10:50:43 +00:00
Erik Johnston	b199534518	Merge pull request #4135 from matrix-org/erikj/fix_state_res_none ... Fix None exception in state res v2	2018-11-02 10:45:57 +00:00
Richard van der Hoff	350f654e7b	Add unique indexes to a couple of tables ... The indexes on device_lists_remote_extremeties can be unique, and they therefore should, to ensure that the db remains consistent.	2018-11-02 10:36:13 +00:00
Richard van der Hoff	50e328d1e7	Remove redundant database locks for device list updates ... We can rely on the application-level per-user linearizer.	2018-11-02 10:36:13 +00:00
Erik Johnston	54aec35867	Fix None exception in state res v2	2018-11-02 10:29:19 +00:00
Travis Ralston	642505abc3	Fix logic error that prevented guests from seeing the privacy policy	2018-11-01 16:48:32 -06:00
Travis Ralston	c68aab1536	Merge pull request #4004 from matrix-org/travis/login-terms ... Add m.login.terms to the registration flow	2018-11-01 11:03:38 -06:00
Erik Johnston	1b21e771d0	Merge pull request #4128 from matrix-org/erikj/state_res_v2_version ... Add STATE_V2_TEST room version	2018-11-01 13:17:57 +00:00
Erik Johnston	b3dd6fa981	Add STATE_V2_TEST room version	2018-11-01 11:43:46 +00:00
Amber Brown	073d400b84	Merge branch 'master' into develop	2018-11-01 21:32:12 +11:00
Amber Brown	aa98e38896	version	2018-11-01 21:28:35 +11:00
Travis Ralston	a8d41c6aff	Include a version query string arg for the consent route	2018-10-31 13:19:28 -06:00
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00