9812 Commits

Author	SHA1	Message	Date
Erik Johnston	be47cfa9c9	Refactor event building into EventBuilder ... This is so that everything is done in one place, making it easier to change the event format based on room version	2019-01-29 11:13:00 +00:00
Erik Johnston	554ca58ea1	Make add_hashes_and_signatures operate on dicts	2019-01-29 11:12:38 +00:00
Amber Brown	f815bd7feb	Make linearizer more quiet (#4507)	2019-01-29 11:05:31 +00:00
Erik Johnston	073f6c2e5e	Merge pull request #4494 from matrix-org/erikj/fixup_event_validator ... Split up event validation between event and builder	2019-01-29 10:55:07 +00:00
Erik Johnston	40638ae7f5	Remove duplicate checks	2019-01-29 10:37:40 +00:00
Erik Johnston	9fa3c6ffa3	Fix up error messages	2019-01-29 10:36:46 +00:00
Erik Johnston	28efc80723	Fold validate into validate_new	2019-01-29 10:34:49 +00:00
Erik Johnston	b6b73a0bcf	Fix receiving events from federation via a worker ... This bug was introduced in PR #4470, commit 678a92cb56	2019-01-29 10:30:26 +00:00
Neil Johnson	327b992e17	register capabilities servlet	2019-01-29 10:28:35 +00:00
Amber Brown	94fb63e44f	Fix typo in upserts code (#4505) ... * fix obvious problem :| * changelog	2019-01-29 10:04:23 +00:00
Erik Johnston	17709f8f9c	Merge pull request #4493 from matrix-org/erikj/refactor_event_signing ... Refactor event signing to work on dicts	2019-01-29 09:52:54 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version ... Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Travis Ralston	6901ac7e9d	Don't recommend :8448 to people on public_baseurl	2019-01-28 12:15:22 -07:00
Erik Johnston	d414f30019	Implement fallback for V2 invite API ... If the room version is either 1 or 2 then a server should retry failed `/v2/invite` requests with the v1 API	2019-01-28 17:33:25 +00:00
Erik Johnston	d758d5310e	Correctly use default room version if none is set	2019-01-28 17:26:39 +00:00
Amber Brown	5d976c0c7c	Fix worker TLS (#4492) ... * load cert * changelog * fix	2019-01-28 17:18:33 +00:00
Erik Johnston	b872c7b1b4	Split up event validation between event and builder ... The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.	2019-01-28 17:00:14 +00:00
Erik Johnston	855a151015	Refactor event signing to work on dicts ... This is in preparation for making EventBuilder format agnostic, which means event signing should be done against the event dict rather than the EventBuilder object.	2019-01-28 16:42:10 +00:00
Amber Brown	7072fe3084	Fix UPSERTs on SQLite 3.24+ (#4477)	2019-01-28 15:43:32 +00:00
Andrew Morgan	f0e96ab66a	Change return syntax in doc string	2019-01-28 14:09:45 +00:00
Andrew Morgan	4026d555fa	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-28 14:08:24 +00:00
Andrew Morgan	1ce463963d	Reuse predecessor method	2019-01-28 14:08:18 +00:00
Richard van der Hoff	3bd0f1a4a3	docstrings for _RoutingResult	2019-01-28 12:43:09 +00:00
Richard van der Hoff	0fd5b3b53e	Handle IP literals explicitly ... We don't want to be doing .well-known lookups on these guys.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	51958df766	MatrixFederationAgent: factor out routing logic ... This is going to get too big and unmanageable.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	d840019192	Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487) ... Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.	2019-01-28 09:56:59 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Neil Johnson	893107be78	backout v3	2019-01-25 17:27:36 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version ... Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Erik Johnston	431e485914	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert	2019-01-25 14:11:17 +00:00
Richard van der Hoff	4a3f138832	Fix quoting for allowed_local_3pids example config (#4476) ... If you use double-quotes here, you have to escape your backslashes. It's much easier with single-quotes. (Note that the existing double-backslashes are already interpreted by python's """ parsing.)	2019-01-25 13:57:52 +00:00
Richard van der Hoff	8520bc3109	Fix Host header sent by MatrixFederationAgent (#4468) ... Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.	2019-01-25 12:38:16 +00:00
Andrew Morgan	9244a3089e	Fixes	2019-01-25 11:48:38 +00:00
Andrew Morgan	da0d2219d2	Clean up direct_rooms access	2019-01-25 11:37:12 +00:00
Andrew Morgan	8265995498	Use python magic	2019-01-25 11:26:06 +00:00
Andrew Morgan	0b3fd1401f	Don't require sqlite3 when using postgres (#4466)	2019-01-25 22:25:02 +11:00
Andrew Morgan	c4cdafa81f	Destructure account data tuple before use	2019-01-25 11:24:28 +00:00
Andrew Morgan	516456b763	Remove unnecessary null check	2019-01-25 11:22:14 +00:00
Andrew Morgan	6f3fda79ce	Move room_tag declaration to be closer to its use	2019-01-25 11:21:25 +00:00
Neil Johnson	95f871fc0d	Support room version capabilities in CS API (MSC1804)	2019-01-25 11:16:29 +00:00
Andrew Morgan	821b65aeb5	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-25 11:09:53 +00:00
Andrew Morgan	0862d35b8e	Move tag and direct state copying into separate function	2019-01-25 11:09:34 +00:00
Neil Johnson	53ef4da8c2	track unstable room v3	2019-01-25 11:04:11 +00:00
Andrew Morgan	b1b6dba2d2	Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms ... Ability to search entire room history after upgrading room	2019-01-25 10:49:58 +00:00
Erik Johnston	1953067136	Disable native upserts for sqlite, as they don't work	2019-01-25 10:46:49 +00:00
Erik Johnston	678a92cb56	Replace missed usages of FrozenEvent	2019-01-25 10:32:30 +00:00
Erik Johnston	a50cf929c1	Require event format version to parse or create events	2019-01-25 10:32:19 +00:00
Erik Johnston	62514bb81b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813	2019-01-25 10:07:08 +00:00
Erik Johnston	edc1e21dbe	Merge pull request #4448 from matrix-org/erikj/get_pdu_versions ... Add room_version param to get_pdu	2019-01-24 18:47:15 +00:00
Erik Johnston	80bcca659e	Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites ... Store rejected remote invite events as outliers	2019-01-24 18:33:27 +00:00
Erik Johnston	664b7a2920	Merge pull request #4437 from matrix-org/erikj/event_format_version_v2 ... Add support for persisting event format versions	2019-01-24 18:32:30 +00:00
Erik Johnston	d148c43050	Review comments	2019-01-24 18:31:23 +00:00
Erik Johnston	26f44164c8	Review comments	2019-01-24 18:28:00 +00:00
Erik Johnston	5ee1f997a8	Update make_membership_event docs	2019-01-24 18:08:08 +00:00
Erik Johnston	9139b87be4	Remove unecessary setting of outlier bit	2019-01-24 18:04:02 +00:00
Erik Johnston	b8082a5445	Use term 'out of band membership' instead	2019-01-24 17:33:19 +00:00
Andrew Morgan	a383289b0d	Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms	2019-01-24 17:23:51 +00:00
Andrew Morgan	e1781b043b	Remove redundant create event None check	2019-01-24 17:23:39 +00:00
Richard van der Hoff	03c85335d1	Apply suggestions from code review ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:22:09 +00:00
Richard van der Hoff	8ea509a935	Update synapse/api/filtering.py ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:21:35 +00:00
Andrew Morgan	50c396a7ee	Merge pull request #4461 from matrix-org/anoa/room_dir_quick_fix ... Add a 60s timeout to filtered room directory queries	2019-01-24 15:24:35 +00:00
Andrew Morgan	075ff3ede9	Change default timeout value from 0 to None	2019-01-24 15:10:22 +00:00
Andrew Morgan	5b1dc94083	Use self.clock instead of datetime	2019-01-24 14:59:50 +00:00
Andrew Morgan	a2d85144e5	isort	2019-01-24 14:22:26 +00:00
Richard van der Hoff	4a6e863843	Merge pull request #4464 from matrix-org/rav/fix_srv_lookup ... MatrixFederationAgent: Look up the right SRV record	2019-01-24 13:57:51 +00:00
Richard van der Hoff	f4697b5ec1	Fix UnboundLocalError in post_urlencoded_get_json (#4460) ... This could cause exceptions if the id server returned 4xx responses.	2019-01-24 13:38:29 +00:00
Richard van der Hoff	afd69a0920	Look up the right SRV record	2019-01-24 13:31:43 +00:00
Neil Johnson	10b89d5c2e	Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check ... Neilj/fix threepid auth check	2019-01-24 13:02:50 +00:00
Andrew Morgan	5541645e80	lint	2019-01-24 12:45:32 +00:00
Andrew Morgan	068aa1d228	Time out filtered room dir queries after 60s	2019-01-24 12:44:27 +00:00
Neil Johnson	5c41b22359	Merge pull request #4458 from matrix-org/dbkr/public_baseurl_doc ... Clarify docs for public_baseurl	2019-01-24 12:36:09 +00:00
Amber Brown	0e27501ee5	Fix UPSERT check (#4459)	2019-01-24 22:57:41 +11:00
David Baker	92d8a068ad	Clarify docs for public_baseurl ... This is leading to problems with people upgrading to clients that support MSC1730 because people have this misconfigured, so try to make the docs completely unambiguous.	2019-01-24 10:52:06 +00:00
Amber Brown	58f6c48183	Use native UPSERTs where possible (#4306)	2019-01-24 21:31:54 +11:00
Richard van der Hoff	97fd29c019	Don't send IP addresses as SNI (#4452) ... The problem here is that we have cut-and-pasted an impl from Twisted, and then failed to maintain it. It was fixed in Twisted in https://github.com/twisted/twisted/pull/1047/files; let's do the same here.	2019-01-24 09:34:44 +00:00
Erik Johnston	7c288c2250	Clarify the invite flows	2019-01-23 20:07:47 +00:00
Erik Johnston	07f62da55a	Remove unnecessary '_sign_event'	2019-01-23 20:04:57 +00:00
Erik Johnston	886e5acc76	Store rejected remote invite events as outliers ... Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.	2019-01-23 20:04:57 +00:00
Willem Mulder	d528406cb8	Fix error message for optional dependencies ... Signed-off-by: Willem Mulder <willemmaster@hotmail.com>	2019-01-23 18:44:57 +01:00
Erik Johnston	6a41d2a187	Add room_version param to get_pdu ... When we add new event format we'll need to know the event format or room version when parsing events.	2019-01-23 17:19:58 +00:00
Erik Johnston	67cd4dad81	Implement MSC 1813 - Add room version to make APIs ... We also implement `make_membership_event` converting the returned room version to an event format version.	2019-01-23 16:51:46 +00:00
Andrew Morgan	6b90ae6efc	Merge pull request #4445 from matrix-org/anoa/user_dir_develop_backport ... Add metric for user dir current event stream position	2019-01-23 15:58:40 +00:00
Richard van der Hoff	a0ae475219	Merge pull request #4428 from matrix-org/rav/matrix_federation_agent ... Move SRV magic into an Agent-like thing	2019-01-23 15:50:25 +00:00
Andrew Morgan	82a92ba535	Add metric for user dir current event stream position	2019-01-23 15:34:47 +00:00
Erik Johnston	be1065af59	isort	2019-01-23 11:48:16 +00:00
Erik Johnston	c5a296b10c	Add support for persisting event format versions ... Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.	2019-01-23 11:30:01 +00:00
Jason Robinson	6f680241bd	Fix flake8 issues ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:53:48 +02:00
Erik Johnston	90743c9d89	Fixup removal of duplicate user_ips rows (#4432) ... * Remove unnecessary ORDER BY clause * Add logging * Newsfile	2019-01-23 19:45:18 +11:00
Amber Brown	6129e52f43	Support ACME for certificate provisioning (#4384)	2019-01-23 19:39:06 +11:00
Jason Robinson	0516dc4d85	Remove openid resource from default config ... Instead document it commented out. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	d39b7b6d38	Document servlet_groups parameters ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	82e13662c0	Split federation OpenID userinfo endpoint out of the federation resource ... This allows the OpenID userinfo endpoint to be active even if the federation resource is not active. The OpenID userinfo endpoint is called by integration managers to verify user actions using the client API OpenID access token. Without this verification, the integration manager cannot know that the access token is valid. The OpenID userinfo endpoint will be loaded in the case that either "federation" or "openid" resource is defined. The new "openid" resource is defaulted to active in default configuration. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	a17bac171f	Make SynapseHomeServer _http_listener use self.get_reactor() ... For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	4f8f41c824	Make FederationReaderServer _http_listen use self.get_reactor() ... For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	899e60be80	Add parameterized Python module to test dependencies ... Allows running parameterized tests. BSD license. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Richard van der Hoff	7021784d46	put resolve_service in an object ... this makes it easier to stub things out for tests.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	53a327b4d5	Require that service_name be a byte string ... it is only ever a bytes now, so let's enforce that.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	fe212bbe4a	Kill off matrix_federation_endpoint ... this thing is now redundant.	2019-01-22 20:34:35 +00:00
Richard van der Hoff	7871146667	Make MatrixFederationClient use MatrixFederationAgent ... ... instead of the matrix_federation_endpoint	2019-01-22 20:34:35 +00:00
Richard van der Hoff	44be7513bf	MatrixFederationAgent ... Pull the magic that is currently in matrix_federation_endpoint and friends into an agent-like thing	2019-01-22 20:34:35 +00:00
Neil Johnson	c99c2d58d7	move guard out of is_threepid_reserved and into register.py	2019-01-22 17:47:00 +00:00
Neil Johnson	d619b113ed	Fix None guard in config.server.is_threepid_reserved	2019-01-22 16:52:29 +00:00
Erik Johnston	12699a701f	Merge pull request #4434 from matrix-org/erikj/fix_user_ips_dedup ... Fix bug when removing duplicate rows from user_ips	2019-01-22 16:51:57 +00:00
Erik Johnston	7f503f83b9	Refactor to rewrite the SQL instead	2019-01-22 16:31:05 +00:00
Neil Johnson	388c164aea	Merge pull request #4423 from matrix-org/neilj/disable_msisdn_on_registration ... Config option to disable requesting MSISDN on registration	2019-01-22 16:23:08 +00:00
Erik Johnston	1c9704f8ab	Don't shadow params	2019-01-22 16:20:33 +00:00
Andrew Morgan	766a172b99	lint	2019-01-22 13:51:40 +00:00
Erik Johnston	2557531f0f	Fix bug when removing duplicate rows from user_ips ... This was caused by accidentally overwritting a `last_seen` variable in a for loop, causing the wrong value to be written to the progress table. The result of which was that we didn't scan sections of the table when searching for duplicates, and so some duplicates did not get deleted.	2019-01-22 13:33:46 +00:00
Andrew Morgan	117bc94cd2	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-22 13:16:51 +00:00
Andrew Morgan	277e50462d	Do not return in a deferred function	2019-01-22 12:40:26 +00:00
Andrew Morgan	c4875d8c76	Prevent duplicate room IDs in m.direct	2019-01-22 12:13:46 +00:00
Andrew Morgan	c433f61091	Ensure new filter is actually created	2019-01-22 12:06:36 +00:00
Andrew Morgan	c9bfb058d8	Fix a bug with single-room search searching all rooms ... * Create a new method for getting predecessor rooms * Remove formatting change	2019-01-22 12:00:41 +00:00
Andrew Morgan	8086a5c05e	Fix comments	2019-01-22 11:16:23 +00:00
Richard van der Hoff	6bfa735a69	Make key fetches use regular federation client (#4426) ... All this magic is redundant.	2019-01-22 11:04:20 +00:00
Andrew Morgan	48951f437f	Join logic covers both room creator and arbitrary users	2019-01-22 11:00:04 +00:00
Andrew Morgan	8c85f0833d	tags, m.direct copying over correctly	2019-01-22 11:00:04 +00:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Andrew Morgan	ea8903fcc9	Migrating dm and room tags work for migrator	2019-01-22 11:00:04 +00:00
Andrew Morgan	887ca93a1b	Prevent crash on user who doesn't have any direct rooms	2019-01-22 11:00:04 +00:00
Andrew Morgan	4ff6d22245	Preserve DM status of a room on upgrade ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-22 11:00:04 +00:00
Andrew Morgan	75942af1db	Fix typo	2019-01-22 11:00:04 +00:00
Richard van der Hoff	33a55289cb	Refactor and bugfix for resove_service (#4427)	2019-01-22 10:59:27 +00:00
Amber Brown	23b0813599	Require ECDH key exchange & remove dh_params (#4429) ... * remove dh_params and set better cipher string	2019-01-22 21:58:50 +11:00
Neil Johnson	1b53cc3cb4	fix line length	2019-01-21 15:17:20 +00:00
Neil Johnson	5349262302	Config option to disable requesting MSISDN on registration	2019-01-21 14:59:37 +00:00
Erik Johnston	35e1d67b4e	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_v2_invite_server	2019-01-21 14:04:19 +00:00
Erik Johnston	5f54765587	Merge pull request #4390 from matrix-org/erikj/versioned_fed_apis ... Add groundwork for new versions of federation APIs	2019-01-21 11:44:05 +00:00
Andrew Morgan	702c4b750c	Migrate encryption state on room upgrade (#4411) ... * Migrate encryption state on room upgrade Signed-off-by: Andrew Morgan <andrew@amorgan.xyz> * Add changelog file	2019-01-21 20:42:58 +11:00
Erik Johnston	25dd56ace3	Fix race when persisting create event (#4404) ... * Fix race when persisting create event When persisting a chunk of DAG it is sometimes requried to do a state resolution, which requires knowledge of the room version. If this happens while we're persisting the create event then we need to use that event rather than attempting to look it up in the database.	2019-01-18 23:17:04 +11:00
Richard van der Hoff	de6888e7ce	Remove redundant WrappedConnection (#4409) ... * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.	2019-01-18 23:07:38 +11:00
Andrew Morgan	df3a661e4a	Search for messages across predecessor rooms ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-18 11:19:20 +00:00
Richard van der Hoff	676cf2ee26	Fix incorrect logcontexts after a Deferred was cancelled (#4407)	2019-01-17 14:00:23 +00:00
Richard van der Hoff	9feb5d0b71	sign_request -> build_auth_headers (#4408) ... Just got very confused about the fact that the headers are only an output, not an input.	2019-01-17 12:40:09 +00:00
Richard van der Hoff	3982a6ee07	Changing macaroon_secret_key no longer logs you out (#4387)	2019-01-16 23:14:41 +00:00
Richard van der Hoff	05e1296649	don't store more remote device lists if they have more than 1K devices (#4397)	2019-01-16 23:14:11 +00:00
Andrej Shadura	64cf6788d9	Depend on pymacaroons >= 0.13.0 instead on pymacaroons-pynacl ... Since 0.13.0, pymacaroons works correctly with pynacl, so there isn’t any more reason to depend on an outdated pynacl fork. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>	2019-01-16 10:54:41 +01:00
Andrej Shadura	fab948120f	Use msgpack instead of msgpack-python ... The package msgpack-python has been deprecated. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>	2019-01-16 10:53:49 +01:00
Neil Johnson	9ec56d6935	ALL_USER_TYPES should be a tuple	2019-01-15 14:38:15 +00:00
Erik Johnston	4a4d2e17bc	Add /v2/invite federation API	2019-01-15 13:22:44 +00:00
Erik Johnston	bb63e7ca4f	Add groundwork for new versions of federation APIs	2019-01-15 11:14:34 +00:00
Amber Brown	a35c66a00b	Remove duplicates in the user_ips table and add an index (#4370)	2019-01-12 06:21:50 +11:00
Amber Brown	522dada206	Merge remote-tracking branch 'origin/master' into develop	2019-01-11 02:22:48 +11:00
Amber Brown	6dc06c3775	version	2019-01-11 01:56:37 +11:00
Richard van der Hoff	e0910d0145	Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 ... Fixes #4371	2019-01-10 14:12:50 +00:00
Richard van der Hoff	ba41aeed6a	Revert "Fix macaroon_secret_key fallback logic" ... This is already fixed in 0.34.1, by 59f93bb This reverts commit efc522c55e.	2019-01-10 14:09:26 +00:00
Richard van der Hoff	aa70d24125	Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34	2019-01-10 12:58:33 +00:00
Richard van der Hoff	efc522c55e	Fix macaroon_secret_key fallback logic	2019-01-10 12:57:27 +00:00
Richard van der Hoff	566947ff34	Skip macaroon check for access tokens in the db	2019-01-10 12:57:21 +00:00
Richard van der Hoff	353f2407b7	Fix fallback to signing key for macaroon-secret-key	2019-01-10 12:42:56 +00:00
Richard van der Hoff	d566e6b17a	Merge branch 'master' into develop	2019-01-09 15:00:46 +00:00
Richard van der Hoff	998f5225c1	0.34.1	2019-01-09 14:53:54 +00:00
Erik Johnston	55c3e853c5	Merge pull request #4368 from matrix-org/erikj/better_errors ... Fixup docstrings for matrixfederationclient	2019-01-09 11:39:28 +00:00
Erik Johnston	484867d35d	Merge branch 'develop' into release-v0.34.1	2019-01-09 11:38:13 +00:00
Amber Brown	7960c26fda	Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369)	2019-01-09 22:26:25 +11:00
Erik Johnston	34ea14139d	Fixup docstrings for matrixfederationclient	2019-01-09 09:25:59 +00:00
Erik Johnston	d422570e9b	Use RequestSendFailed when fail to parse content type headers	2019-01-08 14:22:18 +00:00
Erik Johnston	1371d5b798	Don't log stack traces for HTTP error responses	2019-01-08 12:28:30 +00:00
Richard van der Hoff	ba85786d71	0.34.1rc1	2019-01-08 11:15:34 +00:00
Erik Johnston	b970cb0e96	Refactor request sending to have better excpetions (#4358) ... * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.	2019-01-08 11:04:28 +00:00
Amber Brown	1dcb086f33	Fix synapse.config.__main__ on python 3 (#4356)	2019-01-08 10:03:09 +11:00
Travis Ralston	bc1fa8cd01	Add GET account data routes (#4303) ... As per https://github.com/matrix-org/matrix-doc/issues/1339	2019-01-07 10:27:54 +00:00
Richard van der Hoff	b7c0218812	Check jinja version for consent resource (#4327) ... * Raise a ConfigError if an invalid resource is specified * Require Jinja 2.9 for the consent resource * changelog	2019-01-07 10:14:31 +00:00
Michael Telatynski	a27e501b09	fix the check for whether is_url to match all the other ones in codebase (#3405) ... Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2019-01-06 21:43:25 +00:00
Matthew Hodgson	27128145e6	fix NPE in /messages by checking if all events were filtered out (#4330)	2019-01-02 23:37:39 +00:00
Neil Johnson	84b6fae1f5	Ensure synchrotrons can access is_support_user in the storage layer	2019-01-02 10:19:59 +00:00
Amber Brown	d7843f47b6	Remove v1 only REST APIs now we don't ship matrix console (#4334)	2018-12-29 23:12:30 +11:00
Will Hunt	0708f437cc	Log roomid along with Unknown room (#4297)	2018-12-24 10:49:42 +01:00
Richard van der Hoff	cd018e3543	Merge pull request #4307 from matrix-org/erikj/v2_rooms ... Add v2 room version	2018-12-24 10:48:59 +01:00
Richard van der Hoff	f85676cc93	Return well_known in /login response (#4319) ... ... as per MSC1730.	2018-12-24 10:44:33 +01:00
Richard van der Hoff	9c2af7b2c5	Add a script to generate a clean config file (#4315)	2018-12-22 02:04:57 +11:00
Richard van der Hoff	f3561f8d86	Fix indentation in default config (#4313) ... These settings are not supposed to be under 'listeners'.	2018-12-22 02:01:43 +11:00
Amber Brown	ea6abf6724	Fix IP URL previews on Python 3 (#4215)	2018-12-22 01:56:13 +11:00
Amber Brown	c26f49a664	Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298)	2018-12-22 01:37:26 +11:00
Richard van der Hoff	0b26feb422	Merge branch 'master' into develop	2018-12-20 16:48:47 +00:00
Richard van der Hoff	6dd7271243	Prepare 0.34 release	2018-12-20 12:07:15 +00:00
Richard van der Hoff	d3c9c562c6	Debian packaging via dh_virtualenv	2018-12-20 11:35:23 +00:00
Richard van der Hoff	1a6d5bfa08	Debian packaging via dh_virtualenv (#4285)	2018-12-20 11:33:29 +00:00
David Baker	1c0051114a	Add 'sandbox' to CSP for media repo (#4284) ... * Add 'sandbox' to the CSP for media repo * Changelog	2018-12-20 11:09:18 +00:00
Nathan Pennie	81b513416e	Fixed line length	2018-12-19 14:12:33 -05:00
Nathan Pennie	668e6625b0	Fix the variable names used for account_data	2018-12-19 06:53:02 -05:00
Erik Johnston	df89f8afb8	Add v2 room version	2018-12-18 18:10:37 +00:00
Neil Johnson	7e22cd90f5	ensure can report mau stats when hs.config.mau_stats_only is set (#4305) ... * ensure can report mau stats when hs.config.mau_stats_only is set	2018-12-18 14:36:11 +00:00
Neil Johnson	d2f7c4e6b1	create support user (#4141) ... Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Nad Chishtie	128294d3e5	Improved welcome page (#4294) ... * Improved static/index.html look, feel and content. * Added accessibility tags to in-line SVG. * Grammar.	2018-12-14 05:13:56 +11:00
Amber Brown	c7144b105c	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:13:33 +11:00
Amber Brown	e93a0ebf50	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:10:31 +11:00
Nad Chishtie	cef8ae272a	Improved welcome page (#4294) ... * Improved static/index.html look, feel and content. * Added accessibility tags to in-line SVG. * Grammar.	2018-12-14 02:42:51 +11:00
Richard van der Hoff	f208f608cb	Merge branch 'release-v0.34.0' into develop	2018-12-11 15:43:20 +00:00
Richard van der Hoff	eb835bf65b	0.34.0rc2	2018-12-11 15:35:47 +00:00
Richard van der Hoff	188945713e	Merge pull request #4290 from matrix-org/rav/remove_webclient ... Stop installing Matrix Console by default	2018-12-11 16:24:15 +01:00
Richard van der Hoff	95c1f6500b	Merge pull request #4289 from matrix-org/rav/welcome_page ... Add a welcome page to the static resources	2018-12-11 16:24:06 +01:00
Richard van der Hoff	f537432ef9	Add a welcome page to the static resources ... This is largely a precursor for the removal of the bundled webclient. The idea is to present a page at / which reassures people that something is working, and to give them some links for next steps. The welcome page lives at `/_matrix/static/`, so is enabled alongside the other `static` resources (which, in practice, means the client API is enabled). We'll redirect to it from `/` if we have nothing better to display there. It would be nice to have a way to disable it (in the same way that you might disable the nginx welcome page), but I can't really think of a good way to do that without a load of ickiness. It's based on the work done by @krombel for #2601.	2018-12-11 13:26:22 +00:00
Richard van der Hoff	df96177ca7	Stop installing Matrix Console by default ... This is based on the work done by @krombel in #2601.	2018-12-11 13:20:33 +00:00
David Baker	89ac2a5bdb	Add 'sandbox' to CSP for media repo (#4284) ... * Add 'sandbox' to the CSP for media repo * Changelog	2018-12-11 04:05:02 +11:00
Will Hunt	fee831c040	Move imports to one line	2018-12-10 13:52:33 +00:00
Will Hunt	466c1f3e01	Use send_cors	2018-12-10 13:11:37 +00:00
Will Hunt	91206e09f2	changelog & isort	2018-12-09 17:39:44 +00:00
Will Hunt	dbf736ba66	Make /config more CORS-y	2018-12-09 13:27:22 +00:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272) ... When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	c7401a697f	Implement SAML2 authentication (#4267) ... This implements both a SAML2 metadata endpoint (at `/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at `/_matrix/saml2/authn_response`). If the SAML2 response matches what's been configured, we complete the SSO login flow by redirecting to the client url (aka `RelayState` in SAML2 jargon) with a login token. What we don't yet have is anything to build a SAML2 request and redirect the user to the identity provider. That is left as an exercise for the reader.	2018-12-07 13:11:11 +01:00
Richard van der Hoff	c588b9b9e4	Factor SSO success handling out of CAS login (#4264) ... This is mostly factoring out the post-CAS-login code to somewhere we can reuse it for other SSO flows, but it also fixes the userid mapping while we're at it.	2018-12-07 13:10:07 +01:00
rkfg	ae19a7db8c	Prevent crash on pagination.	2018-12-06 13:32:05 +03:00
Richard van der Hoff	b0c24a66ec	Rip out half-implemented m.login.saml2 support (#4265) ... * Rip out half-implemented m.login.saml2 support This was implemented in an odd way that left most of the work to the client, in a way that I really didn't understand. It's going to be a pain to maintain, so let's start by ripping it out. * drop undocumented dependency on dateutil It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 19:44:38 +11:00
Richard van der Hoff	9a3e24a13d	drop undocumented dependency on dateutil (#4266) ... It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 04:52:42 +11:00
Richard van der Hoff	e8d98466b0	Implement .well-known handling (#4262) ... Sometimes it's useful for synapse to generate its own .well-known file.	2018-12-05 14:38:58 +01:00
Richard van der Hoff	e55983defe	Prepare 0.34.0rc1	2018-12-04 13:52:16 +00:00
Travis Ralston	1737753a62	Add an option to enable recording IPs for appservice users (#3831)	2018-12-04 12:44:41 +01:00
Amber Brown	fd96dd75a3	Fix non-ASCII pushrules (#4248)	2018-12-04 12:44:02 +01:00
Travis Ralston	158ffb92f1	Add an option to disable search for homeservers which may not be interested in it (#4230) ... This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.	2018-12-04 12:01:02 +01:00
Ben Parsons	b5ac0ffa0a	add more detail to logging regarding "More than one row matched" error (#4234)	2018-12-04 11:57:39 +01:00
Richard van der Hoff	ecc23188f4	Fix UnicodeDecodeError when postgres is not configured in english (#4253) ... This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.	2018-12-04 11:55:52 +01:00
Richard van der Hoff	f144c0a210	Merge pull request #4244 from aaronraimist/drop-sent-txt ... Drop sent_transactions	2018-12-04 11:41:28 +01:00
Richard van der Hoff	a077e710a3	Merge pull request #4250 from matrix-org/hawkowl/pusher-remove-py3 ... Fix removing pushers on python 3	2018-12-04 11:22:46 +01:00
Richard van der Hoff	52e87fbfbe	Run the AS senders as background processes (#4189) ... This should fix some "Starting db connection from sentinel context" warnings, and will mean we get metrics for these processes.	2018-12-04 10:53:49 +01:00
Amber Brown	d3c61ef906	fix type error	2018-12-03 22:27:41 +11:00
Richard van der Hoff	c03324294d	Workaround for non-ascii event ids (#4241) ... It turns out that we accept events with non-ascii IDs, which would later cause an explosion during state res. Fixes #4226	2018-12-03 21:47:48 +11:00
Aaron Raimist	704c5298f0	Drop sent_transactions ... Signed-off-by: Aaron Raimist <aaron@raim.ist>	2018-12-01 23:07:35 -06:00
Neil Johnson	7039ece8fb	Neilj/fix autojoin (#4223) ... * Fix auto join failures for servers that require user consent * Fix auto join failures for servers that require user consent	2018-11-28 22:24:57 +11:00
Neil Johnson	f9b136a886	Neilj/fix mau initial reserved users (#4211) ... * fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail * towncrier * white space	2018-11-28 20:33:41 +11:00
Richard van der Hoff	944d524f18	Support m.login.sso (#4220) ... * Clean up the CSS for the fallback login form I was finding this hard to work with, so simplify a bunch of things. Each flow is now a form inside a div of class login_flow. The login_flow class now has a fixed width, as that looks much better than each flow having a differnt width. * Support m.login.sso MSC1721 renames m.login.cas to m.login.sso. This implements the change (retaining support for m.login.cas for older clients). * changelog	2018-11-27 18:51:52 +11:00
Richard van der Hoff	6c18cc4b50	Ignore __pycache__ directories in schema delta dir ... Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.	2018-11-20 22:52:34 +00:00
Neil Johnson	78ba0e7ab8	Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)	2018-11-20 12:29:25 +01:00
Richard van der Hoff	416c671474	Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes ... Fix some logcontext leaks	2018-11-20 12:19:19 +01:00
Amber Brown	31425d82a3	Merge remote-tracking branch 'origin/master' into develop	2018-11-19 12:55:25 -06:00
Amber Brown	d102e19e47	version	2018-11-19 12:42:49 -06:00
Amber Brown	80cac86b2c	Fix fallback auth on Python 3 (#4197)	2018-11-19 12:27:33 -06:00
Richard van der Hoff	f5faf6bc14	Fix logcontext leak in EmailPusher	2018-11-19 17:07:01 +00:00
Travis Ralston	ab4526a153	Remove duplicate slashes in generated consent URLs	2018-11-15 20:41:53 -07:00
Amber Brown	8b1affe7d5	Fix Content-Disposition in media repository (#4176)	2018-11-15 15:55:58 -06:00
Travis Ralston	835779f7fb	Add option to track MAU stats (but not limit people) (#3830)	2018-11-15 18:08:27 +00:00
Amber Brown	df758e155d	Use <meta> tags to discover the per-page encoding of html previews (#4183)	2018-11-15 11:05:08 -06:00
Neil Johnson	bf648c37e7	release 0.33.9rc1	2018-11-14 11:45:52 +00:00
Richard van der Hoff	c1efcd7c6a	Add a test for the public T&Cs form	2018-11-14 10:46:27 +00:00
Richard van der Hoff	83a5f459aa	Fix an internal server error when viewing the public privacy policy	2018-11-14 10:21:07 +00:00
David Baker	0869566ad3	Merge pull request #4113 from matrix-org/dbkr/e2e_backup_versions_are_numbers ... Make e2e backup versions numeric in the DB	2018-11-14 07:55:48 +00:00
David Baker	bca3b91c2d	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers	2018-11-09 18:35:02 +00:00
Erik Johnston	db5a1c059a	Merge pull request #4166 from matrix-org/erikj/drop_unknown_events ... Drop incoming events from federation for unknown rooms	2018-11-09 17:59:34 +00:00
Erik Johnston	dc59ad5334	Remove hack to support rejoining rooms	2018-11-09 14:58:09 +00:00
David Baker	d44dea0223	pep8	2018-11-09 14:38:31 +00:00
David Baker	4f93abd62d	add docs	2018-11-09 13:25:38 +00:00
Erik Johnston	30dd27afff	Simplify to always drop events if server isn't in the room	2018-11-09 11:36:45 +00:00
Richard van der Hoff	3cecf5340d	Update synapse/federation/federation_server.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2018-11-09 11:28:25 +00:00
Richard van der Hoff	9bce065a53	Update synapse/federation/federation_server.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2018-11-09 11:28:22 +00:00
David Baker	d3fa6194f7	Remove unnecessary str()	2018-11-09 11:11:31 +00:00
Erik Johnston	7b22421a7b	Merge pull request #4164 from matrix-org/erikj/fix_device_comparison ... Fix noop checks when updating device keys	2018-11-08 14:37:20 +00:00
Erik Johnston	5ebed18692	Lets convert bytes to unicode instead	2018-11-08 12:33:13 +00:00
Erik Johnston	b1a22b24ab	Fix noop checks when updating device keys ... Clients often reupload their device keys (for some reason) so its important for the server to check for no-ops before sending out device list update notifications. The check is broken in python 3 due to the fact comparing bytes and unicode always fails, and that we write bytes to the DB but get unicode when we read.	2018-11-08 12:18:38 +00:00
Erik Johnston	9417986f77	Drop PDUs of unknown rooms ... When we receive events over federation we will need to know the room version to be able to correctly handle them, e.g. once we start changing event formats. Currently, we attempt to handle events in unknown rooms.	2018-11-08 12:11:20 +00:00
Richard van der Hoff	0a1fc52971	fix parse_string docstring	2018-11-08 11:12:29 +00:00
hera	2b075fb03a	Fix encoding error for consent form on python3 ... The form was rendering this as "b'01234....'". -- richvdh	2018-11-08 11:05:39 +00:00
Amber Brown	b3708830b8	Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157)	2018-11-08 01:37:43 +11:00
Travis Ralston	0f5e51f726	Add config variables for enabling terms auth and the policy name (#4142) ... So people can still collect consent the old way if they want to.	2018-11-06 10:32:34 +00:00
Hubert Chathi	f1087106cf	handle empty backups according to latest spec proposal (#4123) ... fixes #4056	2018-11-05 17:59:29 -05:00
Amber Brown	efdcbbe46b	Tests for user consent resource (#4140)	2018-11-06 05:53:44 +11:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139) ... * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Amber Brown	0467384d2f	Set the encoding to UTF8 in the default logconfig (#4138)	2018-11-03 02:28:07 +11:00
Erik Johnston	90d713b8c6	Merge pull request #4137 from matrix-org/erikj/clean_up_events ... Clean up event accesses and tests	2018-11-02 14:12:49 +00:00
Erik Johnston	b86d05a279	Clean up event accesses and tests ... This is in preparation to refactor FrozenEvent to support different event formats for different room versions	2018-11-02 13:44:14 +00:00
Amber Brown	cb7a6b2379	Fix typing being reset causing infinite syncs (#4127)	2018-11-03 00:19:23 +11:00
Richard van der Hoff	efb9343c8c	Merge pull request #4132 from matrix-org/rav/fix_device_list_locking ... Fix locked upsert on device_lists_remote_cache	2018-11-02 10:50:53 +00:00
Richard van der Hoff	00f12e00f8	Merge pull request #4133 from matrix-org/travis/fix-terms-auth ... Fix logic error that prevented guests from seeing the privacy policy	2018-11-02 10:50:43 +00:00
Erik Johnston	b199534518	Merge pull request #4135 from matrix-org/erikj/fix_state_res_none ... Fix None exception in state res v2	2018-11-02 10:45:57 +00:00
Richard van der Hoff	350f654e7b	Add unique indexes to a couple of tables ... The indexes on device_lists_remote_extremeties can be unique, and they therefore should, to ensure that the db remains consistent.	2018-11-02 10:36:13 +00:00
Richard van der Hoff	50e328d1e7	Remove redundant database locks for device list updates ... We can rely on the application-level per-user linearizer.	2018-11-02 10:36:13 +00:00
Erik Johnston	54aec35867	Fix None exception in state res v2	2018-11-02 10:29:19 +00:00
Travis Ralston	642505abc3	Fix logic error that prevented guests from seeing the privacy policy	2018-11-01 16:48:32 -06:00
Travis Ralston	c68aab1536	Merge pull request #4004 from matrix-org/travis/login-terms ... Add m.login.terms to the registration flow	2018-11-01 11:03:38 -06:00
Erik Johnston	1b21e771d0	Merge pull request #4128 from matrix-org/erikj/state_res_v2_version ... Add STATE_V2_TEST room version	2018-11-01 13:17:57 +00:00
Erik Johnston	b3dd6fa981	Add STATE_V2_TEST room version	2018-11-01 11:43:46 +00:00
Amber Brown	073d400b84	Merge branch 'master' into develop	2018-11-01 21:32:12 +11:00
Amber Brown	aa98e38896	version	2018-11-01 21:28:35 +11:00
Travis Ralston	a8d41c6aff	Include a version query string arg for the consent route	2018-10-31 13:19:28 -06:00
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00
Richard van der Hoff	1729ba1650	Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades ... Attempt to move room aliases on room upgrades	2018-10-31 17:52:18 +00:00
Richard van der Hoff	0f8591a5a8	Avoid else clause on exception for clarity	2018-10-31 15:43:57 +00:00
Richard van der Hoff	94c7fadc98	Attempt to move room aliases on room upgrades	2018-10-31 15:43:57 +00:00
Richard van der Hoff	9b827c40ca	Log some bits about event creation (#4121) ... I found these helpful in debugging my room upgrade tests.	2018-10-31 15:42:23 +00:00
Amber Brown	916efc8249	Remove fetching keys via the deprecated v1 kex method (#4120)	2018-10-31 23:14:39 +11:00
Amber Brown	f79f454485	Remove deprecated v1 key exchange endpoint (#4119)	2018-10-31 22:29:02 +11:00
Amber Brown	67c1924899	version bump	2018-10-31 06:27:05 +11:00
Amber Brown	086e1a8f3e	Fix search 500ing (#4122)	2018-10-31 06:26:46 +11:00
Amber Brown	3bade14ec0	Fix search 500ing (#4122)	2018-10-31 04:33:41 +11:00
Amber Brown	2e223a8c22	Remove the unused /pull federation API (#4118)	2018-10-31 04:24:59 +11:00
Erik Johnston	0794504bce	Merge pull request #4006 from matrix-org/erikj/purge_state_groups ... Delete unreferenced state groups during purge	2018-10-30 16:58:22 +00:00
Amber Brown	0dce9e1379	Write some tests for the email pusher (#4095)	2018-10-30 23:55:43 +11:00
David Baker	e0934acdbb	Cast to int here too	2018-10-30 11:12:23 +00:00
David Baker	12941f5f8b	Cast bacjup version to int when querying	2018-10-30 11:01:07 +00:00
David Baker	2f0f911c52	Convert version back to a string	2018-10-30 10:35:18 +00:00
David Baker	64fa557f80	Try & make it work on postgres	2018-10-30 09:51:04 +00:00
David Baker	563f9b61b1	Make e2e backup versions numeric in the DB ... We were doing max(version) which does not do what we wanted on a column of type TEXT.	2018-10-29 21:01:22 +00:00
Erik Johnston	88e5ffe6fe	Deduplicate device updates sent over replication ... We currently send several kHz of device list updates over replication occisonally, which often causes the replications streams to lag and then get dropped. A lot of those updates will actually be duplicates, since we don't send e.g. device_ids across replication, so let's deduplicate it when we pull them out of the database.	2018-10-29 17:34:34 +00:00
Erik Johnston	a163b748a5	Don't truncate command name in metrics	2018-10-29 17:34:21 +00:00
Erik Johnston	ad88460e0d	Move _find_unreferenced_groups	2018-10-29 14:24:19 +00:00
Erik Johnston	664b192a3b	Fix set operations thinko	2018-10-29 14:21:43 +00:00
Erik Johnston	f4f223aa44	Don't make temporary list	2018-10-29 14:01:49 +00:00
Erik Johnston	b2399f6281	Make SQL a bit cleaner	2018-10-29 14:01:11 +00:00
Amber Brown	4cd1c9f2ff	Delete the disused & unspecced identicon functionality (#4106)	2018-10-29 23:57:24 +11:00
Richard van der Hoff	7fbfea062e	Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar ... Remember to copy the avatar on room upgrades	2018-10-29 12:49:21 +00:00
Richard van der Hoff	56ca578f77	Merge pull request #4099 from matrix-org/rav/upgrade_odd_pls ... Better handling of odd PLs during room upgrades	2018-10-29 12:48:51 +00:00
Richard van der Hoff	bf33eed609	Merge pull request #4091 from matrix-org/rav/room_version_upgrades ... Room version upgrade support	2018-10-29 12:47:20 +00:00
Amber Brown	c4b3698a80	Make the replication logger quieter (#4108)	2018-10-29 22:59:44 +11:00
Amber Brown	2b791865c4	version bump	2018-10-29 21:52:52 +11:00
Richard van der Hoff	db24d7f15e	Better handling of odd PLs during room upgrades ... Fixes handling of rooms where we have permission to send the tombstone, but not other state. We need to (a) fail more gracefully when we can't send the PLs in the old room, and (b) not set the PLs in the new room until we are done with the other stuff.	2018-10-27 00:54:26 +01:00
Richard van der Hoff	5caf79b312	Remember to copy the avatar on room upgrades	2018-10-26 23:56:40 +01:00
Richard van der Hoff	54bbe71867	optimise state copying	2018-10-26 22:51:34 +01:00
Richard van der Hoff	193cadc988	Address review comments ... Improve comments, get old room state from the context we already have	2018-10-26 17:10:30 +01:00
Erik Johnston	03e634dad4	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-26 16:22:45 +01:00
Amber Brown	77d70a7646	Port register_new_matrix_user to Python 3 and add tests (#4085)	2018-10-26 22:05:22 +11:00
Richard van der Hoff	e6babc27d5	restrict PLs in old room	2018-10-25 19:18:25 +01:00
Richard van der Hoff	3a263bf3ae	copy state	2018-10-25 19:18:25 +01:00
Richard van der Hoff	1b9f253e20	preserve PLs	2018-10-25 19:10:24 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Richard van der Hoff	0f7d1c9906	Basic initial support for room upgrades ... Currently just creates a new, empty, room, and sends a tombstone in the old room.	2018-10-25 19:10:24 +01:00
Richard van der Hoff	e1948175ee	Allow power_level_content_override=None for _send_events_for_new_room	2018-10-25 19:10:24 +01:00
Richard van der Hoff	7f7b2cd3de	Make room_member_handler a member of RoomCreationHandler ... ... to save passing it into `_send_events_for_new_room`	2018-10-25 19:10:18 +01:00
Richard van der Hoff	871c4abfec	Factor _generate_room_id out of create_room ... we're going to need this for room upgrades.	2018-10-25 18:23:09 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Erik Johnston	c85e063302	Merge pull request #4051 from matrix-org/erikj/alias_disallow_list ... Add config option to control alias creation	2018-10-25 17:04:59 +01:00
Neil Johnson	95ad128851	Merge pull request #4081 from matrix-org/neilj/fix_mau_init ... fix race condiftion in calling initialise_reserved_users	2018-10-25 16:33:40 +01:00
Neil Johnson	fcbd488e9a	add new line	2018-10-25 16:13:43 +01:00
Erik Johnston	b94a43d5b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list	2018-10-25 15:25:31 +01:00
Erik Johnston	e5481b22aa	Use allow/deny	2018-10-25 15:25:21 +01:00
Neil Johnson	f8fe98812b	improve comments	2018-10-25 14:58:59 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Travis Ralston	54def42c19	Merge branch 'develop' into travis/login-terms	2018-10-24 13:22:59 -06:00
Neil Johnson	ea69a84bbb	fix style inconsistencies	2018-10-24 17:18:08 +01:00
Neil Johnson	663d9db8e7	commit transaction before closing	2018-10-24 17:17:30 +01:00
Neil Johnson	07126e43a4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init	2018-10-24 16:25:39 +01:00
Neil Johnson	9ec2186586	isort	2018-10-24 16:09:21 +01:00
Neil Johnson	9532caf6ef	remove trailing whiter space	2018-10-24 16:08:25 +01:00
Richard van der Hoff	94a49e0636	fix tuple ... Co-Authored-By: neilisfragile <neil@matrix.org>	2018-10-24 14:39:23 +01:00
Richard van der Hoff	3ad359e5be	Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation	2018-10-24 11:23:49 +01:00
Richard van der Hoff	7328039117	Merge pull request #4082 from matrix-org/rav/fix_pep8 ... Fix a number of flake8 errors	2018-10-24 11:23:35 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	7e07d25ed6	Allow backslashes in event field filters ... Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which meant that single backslashes were not allowed in event field filters. The intention here is to allow single-backslashes, but disallow double-backslashes.	2018-10-24 11:11:24 +01:00
Richard van der Hoff	ef771cc4c2	Fix a number of flake8 errors ... Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.	2018-10-24 10:39:03 +01:00
Erik Johnston	47a9ba435d	Use match rather than search	2018-10-24 09:54:57 +01:00
Richard van der Hoff	e0b9d5f0af	Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts ... Clean up the way logcontexts and threads work in the pushers	2018-10-24 09:53:57 +01:00
Erik Johnston	dacbeb2e03	Comment	2018-10-24 09:47:49 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Erik Johnston	cb23aa4c42	Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation ... Clean up room alias creation	2018-10-24 09:40:27 +01:00
Richard van der Hoff	c573794b22	Fix start_pushers vs _start_pushers confusion	2018-10-24 09:24:55 +01:00
Richard van der Hoff	e564306e31	sanity-check the is_processing flag ... ... and rename it, for even more sanity	2018-10-24 09:23:33 +01:00
Neil Johnson	a67d8ace9b	remove errant exception and style	2018-10-23 17:44:39 +01:00
Travis Ralston	43c3f0b02f	Merge pull request #3969 from turt2live/travis/fix-federated-group-requests ... Handle HttpResponseException more safely for federated groups	2018-10-23 10:41:04 -06:00
Travis Ralston	3e704822be	Comments help	2018-10-23 10:25:31 -06:00
Neil Johnson	329d18b39c	remove white space	2018-10-23 15:27:20 +01:00
Neil Johnson	6105c6101f	fix race condiftion in calling initialise_reserved_users	2018-10-23 15:24:58 +01:00
Richard van der Hoff	b3f6dddad2	Give some more things logcontexts (#4077)	2018-10-23 14:29:17 +01:00
Richard van der Hoff	5c445114d3	Correctly account for cpu usage by background threads (#4074) ... Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064	2018-10-23 13:12:32 +01:00
Richard van der Hoff	026cd91ac8	Run PusherPool.start as a background process ... We don't do anything with the result, so this is needed to give this code a logcontext.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	f749607c91	Make on_started synchronous too ... This brings it into line with on_new_notifications and on_new_receipts. It requires a little bit of hoop-jumping in EmailPusher to load the throttle params before the first loop.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	e7a16c6210	Remove redundant run_as_background_process() from pusherpool ... `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	c7273c11bc	Give pushers their own background logcontext ... Each pusher has its own loop which runs for as long as it has work to do. This should run in its own background thread with its own logcontext, as other similar loops elsewhere in the system do - which means that CPU usage is consistently attributed to that loop, rather than to whatever request happened to start the loop.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	5110f4e425	move get_all_pushers call down ... simplifies the interface to _start_pushers	2018-10-22 16:12:11 +01:00
Richard van der Hoff	04277d0ed8	Factor PusherPool._start_pusher out of _start_pushers ... ... and use it from start_pusher_by_id. This mostly simplifies start_pusher_by_id.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	3e8b02c939	Rename _refresh_pusher ... This is public (or at least, called from outside the class), so ought to have a better name.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	7aea00069c	Merge pull request #4076 from matrix-org/rav/fix_init_logcontexts ... Run MAU queries as background processes	2018-10-22 14:46:59 +01:00
Richard van der Hoff	911db96658	Merge pull request #4073 from matrix-org/rav/require_psutil ... Make psutil an explicit dependency	2018-10-22 12:33:21 +01:00
Richard van der Hoff	a6f421e812	Run MAU queries as background processes ... Fixes #3820	2018-10-20 02:14:35 +01:00
Amber Brown	e1728dfcbe	Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068)	2018-10-20 11:16:55 +11:00
Richard van der Hoff	e5b52d0f94	Make psutil an explicit dependency ... As of #4027, we require psutil to be installed, so it should be in our dependency list. We can also remove some of the conditional import code introduced by #992. Fixes #4062.	2018-10-19 21:51:15 +01:00
Richard van der Hoff	f62c597d14	Merge pull request #4069 from matrix-org/rav/fix_email_templates_4065 ... Calculate absolute path for email templates	2018-10-19 16:33:54 +01:00
Erik Johnston	67f7b9cb50	pep8	2018-10-19 16:06:59 +01:00
Erik Johnston	056f099126	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-19 15:48:59 +01:00
Erik Johnston	47a9da28ca	Batch process handling state groups	2018-10-19 15:48:15 +01:00
Richard van der Hoff	cc325c7069	Calculate absolute path for email templates	2018-10-19 14:01:59 +01:00
Amber Brown	e404ba9aac	Fix manhole on py3 (pt 2) (#4067)	2018-10-19 22:26:00 +11:00
Amber Brown	b69216f768	Make the metrics less racy (#4061)	2018-10-19 21:45:45 +11:00
Erik Johnston	9fafdfa97d	Anchor returned regex to start and end of string	2018-10-19 10:22:45 +01:00
Erik Johnston	084046456e	Add config option to control alias creation	2018-10-19 10:22:45 +01:00
Erik Johnston	74e7617083	Clean up room alias creation	2018-10-19 10:11:56 +01:00
Amber Brown	a36b0ec195	make a bytestring	2018-10-19 09:24:00 +11:00
Travis Ralston	49a044aa5f	Merge branch 'develop' into travis/login-terms	2018-10-18 09:57:58 -06:00
Richard van der Hoff	c632bc8654	Merge branch 'master' into develop	2018-10-18 15:07:03 +01:00
Richard van der Hoff	926da4dda8	0.33.7	2018-10-18 14:57:32 +01:00
Richard van der Hoff	c7d0f34a3c	v0.33.7rc2	2018-10-17 17:40:19 +01:00
Richard van der Hoff	0fd2321629	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-17 17:35:26 +01:00
Richard van der Hoff	c8f2c19991	Put the warning blob at the top of the file	2018-10-17 16:56:22 +01:00
Richard van der Hoff	1519572961	Ship the email templates as package_data ... move the example email templates into the synapse package so that they can be used as package data, which should mean that all of the packaging mechanisms (pip, docker, debian, arch, etc) should now come with the example templates. In order to grandfather in people who relied on the templates being in the old place, check for that situation and fall back to using the defaults if the templates directory does not exist.	2018-10-17 16:46:02 +01:00
Erik Johnston	f6a0a02a62	Fix bug where we raised StopIteration in a generator ... This made python 3.7 unhappy	2018-10-17 16:10:52 +01:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046) ... Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Neil Johnson	c6584f4b5f	clean up config error logic and imports	2018-10-17 11:36:41 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	4a28d3d36f	Update event_auth table for rejected events	2018-10-16 16:28:42 +01:00
Erik Johnston	15133477ee	Fix up use of resolve_events_with_factory	2018-10-16 16:28:42 +01:00
Erik Johnston	6bd856caa2	User event.sender rather than alias event.user_id	2018-10-16 16:16:13 +01:00
Erik Johnston	e238013c44	Add v2 state res algorithm. ... We hook this up to the vdh test room version.	2018-10-16 16:16:13 +01:00
Richard van der Hoff	b8a5b0097c	Various cleanups in the federation client code (#4031) ... - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions	2018-10-16 10:44:49 +01:00
Travis Ralston	a8ed93a4b5	pep8	2018-10-15 16:10:29 -06:00
Travis Ralston	442734ff9e	Ensure the terms params are actually provided	2018-10-15 14:56:13 -06:00
Travis Ralston	762a0982aa	Python is hard	2018-10-15 14:46:09 -06:00
Travis Ralston	f293d124b6	Merge branch 'develop' into travis/login-terms	2018-10-15 14:44:32 -06:00
Amber Brown	f726f2dc6c	version bump	2018-10-15 22:21:45 +11:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Ivan Shapovalov	06bc8d2fe5	synapse/app: frontend_proxy.py: actually make workers work on py3	2018-10-14 20:08:39 +03:00
Ivan Shapovalov	fb216a22db	synapse/visibility.py: fix SyntaxError on py3.7	2018-10-14 20:08:17 +03:00
Neil Johnson	1ccafb0c5e	no need to join room if creator	2018-10-13 21:14:21 +01:00
Travis Ralston	dd99db846d	Update login terms structure for the proposed language support	2018-10-12 18:03:27 -06:00
Travis Ralston	5119818e9d	Rely on the lack of ?u to represent public access ... also general cleanup	2018-10-12 18:03:17 -06:00
Travis Ralston	7ede650956	Merge branch 'develop' into travis/login-terms	2018-10-12 16:24:07 -06:00
Travis Ralston	164f8e4843	isort	2018-10-12 15:11:59 -06:00
Travis Ralston	7bb651de6a	More sane handling of group errors and pep8	2018-10-12 14:53:30 -06:00
Travis Ralston	e3586f7c06	Merge branch 'develop' into travis/fix-federated-group-requests	2018-10-12 14:49:58 -06:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	a45f2c3a00	missed one	2018-10-12 14:33:55 +01:00
Amber Brown	381d2cfdf0	Make workers work on Py3 (#4027)	2018-10-13 00:14:08 +11:00
Erik Johnston	67a1e315cc	Fix up comments	2018-10-12 13:49:48 +01:00
David Baker	8c0ff0287a	Linting soothes the savage PEP8 monster	2018-10-12 13:47:43 +01:00
David Baker	306361b31b	Misc PR feedback bits	2018-10-12 11:48:56 +01:00
David Baker	bddfad253a	Don't mangle exceptions	2018-10-12 11:48:02 +01:00
David Baker	86ef9760a7	Split /room_keys/version into 2 servlets	2018-10-12 11:35:08 +01:00
David Baker	83e72bb2f0	PR feedback pt. 1	2018-10-12 11:26:18 +01:00
Richard van der Hoff	8ddd0f273c	Comments on get_all_new_events_stream ... just some docstrings to clarify the behaviour here	2018-10-12 09:55:41 +01:00
Erik Johnston	7e561b5c1a	Add description to counter metric	2018-10-10 11:41:15 +01:00
Erik Johnston	3cbe8331e6	Track number of non-empty sync responses instead	2018-10-10 11:23:17 +01:00
Erik Johnston	395276b405	Append _total to metric and fix up spelling	2018-10-10 09:24:39 +01:00
David Baker	b8d9e108be	Fix mergefail	2018-10-09 18:04:21 +01:00
Erik Johnston	bdc27d6716	Add metric to count lazy member sync requests	2018-10-09 14:17:52 +01:00
David Baker	d3464ce708	isort	2018-10-09 10:33:59 +01:00
Erik Johnston	9eb1a79100	Merge pull request #4008 from matrix-org/erikj/log_looping_exceptions ... Log looping call exceptions	2018-10-09 10:12:10 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	f4a4dbcad1	Apparently this blank line is Very Important	2018-10-09 09:47:04 +01:00
David Baker	0c905ee015	be python3 compatible	2018-10-09 09:39:13 +01:00
Erik Johnston	6982320572	Remove unnecessary extra function call layer	2018-10-08 14:06:19 +01:00
Richard van der Hoff	495975e231	Optimisation for filter_events_for_server ... We're better off hashing just the event_id than the whole ((type, state_key), event_id) tuple - so use a dict instead of a set. Also, iteritems > items.	2018-10-08 13:46:52 +01:00
Erik Johnston	8a1817f0d2	Use errback pattern and catch async failures	2018-10-08 13:29:47 +01:00
David Baker	497444f1fd	Don't reuse backup versions ... Since we don't actually delete the keys, just mark the versions as deleted in the db rather than actually deleting them, then we won't reuse versions. Fixes https://github.com/vector-im/riot-web/issues/7448	2018-10-05 15:08:36 +01:00
Erik Johnston	f7199e8734	Log looping call exceptions ... If a looping call function errors, then it kills the loop entirely. Currently it throws away the exception logs, so we should make it actually log them. Fixes #3929	2018-10-05 11:24:12 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Richard van der Hoff	c6dbd216e6	Merge pull request #3995 from matrix-org/rav/no_deextrem_outliers ... Fix bug in forward_extremity update logic	2018-10-04 16:27:05 +01:00
Erik Johnston	4917ff5523	Add state_group index to event_to_state_groups ... This is needed to efficiently check for unreferenced state groups during purge.	2018-10-04 16:03:08 +01:00
Erik Johnston	17d585753f	Delete unreferened state groups during purge	2018-10-04 16:03:06 +01:00
Amber Brown	d86794325f	Merge branch 'master' into develop	2018-10-04 22:41:52 +10:00
Amber Brown	dd59dfc51f	full version	2018-10-04 22:37:55 +10:00
Travis Ralston	537d0b7b36	Use a flag rather than a new route for the public policy ... This also means that the template now has optional parameters, which will need to be documented somehow.	2018-10-03 17:50:11 -06:00
Travis Ralston	f9d34a763c	Auto-consent to the privacy policy if the user registered with terms	2018-10-03 17:39:45 -06:00
Travis Ralston	dfcad5fad5	Make the terms flow requried	2018-10-03 17:39:00 -06:00
Travis Ralston	3099d96dba	Flesh out the fallback auth for terms	2018-10-03 17:39:00 -06:00
Travis Ralston	149c4f1765	Supply params for terms auth stage ... As per https://github.com/matrix-org/matrix-doc/pull/1692	2018-10-03 15:57:42 -06:00
Travis Ralston	fd99787162	Incorporate Dave's work for GDPR login flows ... As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117	2018-10-03 15:57:42 -06:00
Richard van der Hoff	a59d899668	Pin to prometheus_client<0.4 to avoid renaming all of our metrics	2018-10-03 17:20:15 +01:00
Erik Johnston	8935ec5a93	Merge pull request #3999 from matrix-org/erikj/fix_3pid_invite_rejetion ... Fix handling of rejected threepid invites	2018-10-03 14:47:41 +01:00
Erik Johnston	81e2813948	Merge pull request #3996 from matrix-org/erikj/fix_bg_iteration ... Fix exception in background metrics collection	2018-10-03 14:14:38 +01:00
Erik Johnston	52e6e815be	Sanitise error messages when user doesn't have permission to invite	2018-10-03 14:13:07 +01:00
Erik Johnston	69e857853f	Fix handling of rejected threepid invites	2018-10-03 11:57:30 +01:00
Erik Johnston	495a9d06bb	Fix exception handling in fetching remote profiles	2018-10-03 11:34:30 +01:00
Erik Johnston	7c570bff74	Fix exception in background metrics collection ... We attempted to iterate through a list on a separate thread without doing the necessary copying.	2018-10-03 11:28:01 +01:00
Richard van der Hoff	9693625e55	actually exclude outliers	2018-10-03 10:19:41 +01:00
Richard van der Hoff	3e39783d5d	remove debugging	2018-10-02 23:44:14 +01:00
Richard van der Hoff	ae61ade891	Fix bug in forward_extremity update logic ... An event does not stop being a forward_extremity just because an outlier or rejected event refers to it.	2018-10-02 23:36:08 +01:00
Amber Brown	da8f82008d	version	2018-10-03 02:15:48 +10:00
Erik Johnston	7258d081a5	Fix bug when invalidating destination retry timings	2018-10-02 15:47:57 +01:00
Richard van der Hoff	2b8d28b095	Merge pull request #3960 from matrix-org/rav/fix_missing_create_event_error ... Fix error handling for missing auth_event	2018-10-02 13:57:54 +01:00
Amber Brown	7232917f12	Disable frozen dicts by default (#3987)	2018-10-02 22:53:47 +10:00
Erik Johnston	334e075dd8	Fix error when logging incomplete requests ... If a connection is lost before a request is read from Request, Twisted sets `method` (and `uri`) attributes to dummy values. These dummy values have incorrect types (i.e. they're not bytes), and so things like `__repr__` would raise an exception. To fix this we had a helper method to return the method with a consistent type.	2018-10-02 12:06:22 +01:00
Richard van der Hoff	8c41b0ca66	Merge pull request #3989 from matrix-org/rav/better_stacktraces ... Avoid reraise, to improve stacktraces	2018-10-02 10:47:38 +01:00
Erik Johnston	bc29946809	Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state ... Fix lazy loaded sync with rejected state events	2018-10-02 11:38:35 +02:00
Richard van der Hoff	8174c6725b	Avoid reraise, to improve stacktraces	2018-10-01 18:50:34 +01:00
Richard van der Hoff	5908a8f6f5	Merge branch 'develop' into rav/fix_missing_create_event_error	2018-10-01 17:16:20 +01:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors ... Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Amber Brown	6e05fd032c	Fix userconsent on Python 3 (#3938)	2018-10-02 00:11:58 +10:00
Erik Johnston	82f922b4af	Fix lazy loaded sync with rejected state events ... In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.	2018-10-01 14:19:40 +01:00
Erik Johnston	8f5c23d0cd	Merge pull request #3933 from matrix-org/erikj/destination_retry_cache ... Add a five minute cache to get_destination_retry_timings	2018-10-01 14:20:30 +02:00
Erik Johnston	4f3e3ac192	Correctly match 'dict.pop' api	2018-10-01 12:25:27 +01:00
Erik Johnston	8ea887856c	Don't update eviction metrics on explicit removal	2018-10-01 12:00:58 +01:00
Matthew Hodgson	23b6a0537f	emit room aliases event	2018-09-29 02:19:37 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Matthew Hodgson	8f646f2d04	fix UTs	2018-09-28 15:37:28 +01:00
Richard van der Hoff	3deaad2fb4	Merge pull request #3964 from matrix-org/rav/remove_localhost_checks ... remove spurious federation checks on localhost	2018-09-28 13:35:47 +01:00
Richard van der Hoff	965154d60a	Fix complete fail to do the right thing	2018-09-28 12:45:54 +01:00
Richard van der Hoff	19475cf337	Remove redundant call to start_get_pdu_cache ... I think this got forgotten in #3932. We were getting away with it because it was the last call in this function.	2018-09-28 12:01:23 +01:00
Richard van der Hoff	9c8cec5dab	Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache	2018-09-28 10:51:09 +01:00
Matthew Hodgson	07340cdaca	untested stab at autocreating autojoin rooms	2018-09-28 01:42:53 +01:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Richard van der Hoff	36c62a67c4	Merge pull request #3794 from matrix-org/erikj/faster_typing ... Improve performance of getting typing updates for replication	2018-09-27 15:14:26 +01:00
Richard van der Hoff	e1e3e77bfd	Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups ... Clarifications in FederationHandler	2018-09-27 15:06:59 +01:00
Amber Brown	6de3884e5e	Merge pull request #3965 from matrix-org/rav/notify_app_services_bg_process ... Run notify_app_services as a bg process	2018-09-27 23:40:30 +10:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 ... Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Amber Brown	b3064532d0	Run our oldest supported configuration in CI (#3952)	2018-09-27 23:21:54 +10:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) ... Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	51d33d5178	Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts ... fix #3854 MAU transaction errors	2018-09-27 12:31:37 +01:00
Richard van der Hoff	333bee27f5	Include event when resolving state for missing prevs ... If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	bd61c82bdf	Include state from remote servers in pdu handling ... If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	a215b698c4	Fix "unhashable type: 'list'" exception in federation handling ... get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	28223841e0	more comments	2018-09-27 11:31:51 +01:00
Richard van der Hoff	e3c159863d	Clarifications in FederationHandler ... * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)	2018-09-27 11:31:51 +01:00
Richard van der Hoff	92abd3d6d6	Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 ... Logging improvements	2018-09-27 11:27:46 +01:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Richard van der Hoff	ae6ad4cf41	docstrings and unittests for storage.state (#3958) ... I spent ages trying to figure out how I was going mad...	2018-09-27 11:22:25 +01:00
Travis Ralston	82fa31799c	Remove debugging statement	2018-09-26 14:01:02 -06:00
Travis Ralston	b4c3bc1734	Handle HttpResponseException more safely for federated groups	2018-09-26 13:48:04 -06:00
Travis Ralston	219606a6ed	Fix exception documentation in matrixfederationclient.py	2018-09-26 13:26:27 -06:00
Richard van der Hoff	e70b4ce069	Logging improvements ... Some logging tweaks to help with debugging incoming federation transactions	2018-09-26 17:36:14 +01:00
Richard van der Hoff	a87d419a85	Run notify_app_services as a bg process ... This ensures that its resource usage metrics get recorded somewhere rather than getting lost. (It also fixes an error when called from a nested logging context which completes before the bg process)	2018-09-26 17:00:40 +01:00
Richard van der Hoff	9453c65948	remove spurious federation checks on localhost ... There's really no point in checking for destinations called "localhost" because there is nothing stopping people creating other DNS entries which point to 127.0.0.1. The right fix for this is https://github.com/matrix-org/synapse/issues/3953. Blocking localhost, on the other hand, means that you get a surprise when trying to connect a test server on localhost to an existing server (with a 'normal' server_name).	2018-09-26 16:53:52 +01:00
Richard van der Hoff	607eec0456	fix docstring for FederationClient.get_state_for_room ... trivial fixes for docstring	2018-09-26 16:52:24 +01:00
Neil Johnson	28781b65e7	fix #3854	2018-09-26 16:16:41 +01:00
Richard van der Hoff	8afddf7afe	Fix error handling for missing auth_event ... When we were authorizing an event, if there was no `m.room.create` in its auth_events, we would raise a SynapseError with a cryptic message, which then meant that we would bail out of processing any incoming events, rather than storing a rejection for the faulty event and moving on. We should treat the absent event the same as any other auth failure, by raising an AuthError, so that the event is marked as rejected.	2018-09-26 14:40:16 +01:00
Richard van der Hoff	4e8276a34a	Merge pull request #3956 from matrix-org/rav/fix_expiring_cache_len ... Fix ExpiringCache.__len__ to be accurate	2018-09-26 13:10:13 +01:00
Richard van der Hoff	5b4028fa78	Merge branch 'rav/fix_expiring_cache_len' into erikj/destination_retry_cache	2018-09-26 12:55:53 +01:00
Richard van der Hoff	7ee94fc1ba	Log which cache is throwing exceptions	2018-09-26 12:43:08 +01:00
Amber Brown	66a1d57adb	Merge pull request #3948 from matrix-org/rav/no_symlink_synctl ... Move synctl into top dir to avoid a symlink	2018-09-26 21:41:58 +10:00
Amber Brown	c2185f14d7	Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu ... Comments and interface cleanup for on_receive_pdu	2018-09-26 21:41:26 +10:00
Erik Johnston	3baf6e1667	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch. (This appears to have been a longstanding bug, but one which has been made more of a problem by #3932 and #3933). (This was originally done by Erik as part of #3933. I'm cherry-picking it because really it's a fix in its own right)	2018-09-26 12:32:29 +01:00
Richard van der Hoff	a1cd37390f	Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache	2018-09-25 12:03:54 +01:00
Richard van der Hoff	4c3e7eeec5	Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches ... Fix some instances of ExpiringCache not expiring cache items	2018-09-25 12:02:57 +01:00
Jérémy Farnaud	6cf261930a	added "media-src: 'self'" to CSP for resources (#3578) ... Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem.	2018-09-25 11:55:02 +01:00
Richard van der Hoff	94f7befc31	Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered ... Fix spurious exceptions when client closes conncetion	2018-09-25 11:52:06 +01:00
Richard van der Hoff	c53336986d	Move synctl into top dir to avoid a symlink ... symlinks apparently break setuptools on python3 and alpine (https://bugs.python.org/issue31940), so let's stop using a symlink and just use the file directly.	2018-09-25 11:19:27 +01:00
Richard van der Hoff	a9d84f4e44	We require attrs 16.0.0 ... Ref: https://github.com/matrix-org/synapse/issues/3945	2018-09-25 10:43:39 +01:00
Matthew Hodgson	787d22ed6c	Only lazy load self-members on initial sync ... Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.	2018-09-25 00:49:26 +01:00
Amber Brown	fbe5ba25f6	Merge branch 'master' into develop	2018-09-25 03:10:01 +10:00
Amber Brown	6b6cb32297	bump version	2018-09-25 02:54:34 +10:00
Amber Brown	04eed80a73	Merge branch 'master' into develop	2018-09-24 23:42:25 +10:00
Amber Brown	e302f40e20	update version	2018-09-24 23:40:05 +10:00
Erik Johnston	19dc676d1a	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch.	2018-09-21 16:25:42 +01:00
Erik Johnston	fdd1a62e8d	Add a five minute cache to get_destination_retry_timings ... Hopefully helps with #3931	2018-09-21 14:56:12 +01:00
Erik Johnston	79eded1ae4	Make ExpiringCache slightly more performant	2018-09-21 14:52:21 +01:00
Erik Johnston	8601c24287	Fix some instances of ExpiringCache not expiring cache items ... ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.	2018-09-21 14:19:46 +01:00
Erik Johnston	ad53a5497d	Merge pull request #3927 from matrix-org/erikj/handle_background_errors ... Handle exceptions thrown by background tasks	2018-09-21 09:26:30 +01:00
Matthew Hodgson	a2ddaa90f2	Always LL ourselves if we're in a room to simplify clients (#3916) ... Should fix https://github.com/vector-im/riot-web/issues/7209	2018-09-20 21:21:54 +01:00
Erik Johnston	94ae1dea3c	Add missing logger	2018-09-20 17:05:34 +01:00
Erik Johnston	9ea408441f	Handle exceptions thrown by background tasks ... Fixes #3921	2018-09-20 16:15:21 +01:00
Erik Johnston	b28a7ed503	Fix spurious exceptions when client closes conncetion ... If a HTTP handler throws an exception while processing a request we automatically write a JSON error response. If the handler had already started writing a response twisted throws an exception. We should check for this case and simple abort the connection if there was an error after the response had started being written.	2018-09-20 13:44:20 +01:00
Neil Johnson	23b53b4ef8	Merge pull request #3868 from matrix-org/neilj/fix_room_invite_mail_links ... Neilj/fix room invite mail links	2018-09-20 13:32:38 +01:00
Richard van der Hoff	703de4ec13	Comments and interface cleanup for on_receive_pdu ... Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.	2018-09-20 13:06:55 +01:00
Amber Brown	1f3f5fcf52	Fix client IPs being broken on Python 3 (#3908)	2018-09-20 20:14:34 +10:00
Erik Johnston	3fd68d533b	Merge pull request #3914 from matrix-org/erikj/remove_retry_cache ... Remove get_destination_retry_timings cache	2018-09-20 10:54:49 +01:00
Amber Brown	aeca5a5ed5	Add a regression test for logging on failed connections (#3912)	2018-09-20 16:28:18 +10:00
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904) ... Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	2018-09-19 17:28:18 +01:00
Erik Johnston	ce846bb620	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing	2018-09-19 15:08:36 +01:00
Erik Johnston	bbab6ebfd9	Fix up changelog and remove spurious comment	2018-09-19 14:45:14 +01:00
Erik Johnston	392a54128c	pep8	2018-09-19 14:37:49 +01:00
Erik Johnston	b9158ac2bf	Remove get_destination_retry_timings cache ... Currently we rely on the master to invalidate this cache promptly. However, after having moved most federation endpoints off of master this no longer happens, causing outbound fedeariont to get blackholed. Fixes #3798	2018-09-19 14:22:57 +01:00
Erik Johnston	80d2d50f47	Fixup	2018-09-19 11:19:47 +01:00
Erik Johnston	9407bcf37a	Replace custom DeferredTimeoutError with defer.TimeoutError	2018-09-19 11:07:29 +01:00
Erik Johnston	6c48aa0256	Run canceller first to allow it to generate correct error	2018-09-19 11:07:27 +01:00
Erik Johnston	a334e1cace	Update to use new timeout function everywhere. ... The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.	2018-09-19 10:39:40 +01:00
Amber Brown	47c02e6332	Merge pull request #3909 from turt2live/travis/fix-logging-1 ... Fix matrixfederationclient.py logging: Destination is a string	2018-09-19 18:14:47 +10:00
Amber Brown	3d6b24fb1b	Merge pull request #3907 from matrix-org/rav/set_sni_to_server_name ... Set SNI to the server_name, not whatever was in the SRV record	2018-09-19 17:59:33 +10:00
Amber Brown	f773ecbd61	Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout ... Bump timeout on get_missing_events request	2018-09-19 17:57:48 +10:00
Travis Ralston	35aec19f0a	Destination is a string	2018-09-18 15:29:30 -06:00
Richard van der Hoff	38ead946a9	Merge remote-tracking branch 'origin/develop' into neilj/fix_room_invite_mail_links	2018-09-18 19:02:45 +01:00
Richard van der Hoff	a219ce8726	Use directory server for room joins (#3899) ... When we do a join, always try the server we used for the alias lookup first. Fixes #2418	2018-09-18 18:27:37 +01:00
Richard van der Hoff	31c15dcb80	Refactor matrixfederationclient to fix logging (#3906) ... We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.	2018-09-18 18:17:15 +01:00
Amber Brown	c600886d47	Merge pull request #3894 from matrix-org/hs/phone_home_py_version ... Add python_version phone home stat	2018-09-19 02:40:04 +10:00
Richard van der Hoff	b3097396e7	Set SNI to the server_name, not whatever was in the SRV record ... Fixes #3843	2018-09-18 17:01:12 +01:00