9139 Commits

Author	SHA1	Message	Date
Amber Brown	ee7c8bd2b5	Merge pull request #3795 from matrix-org/erikj/faster_sync_state ... User iter* during sync state calculations	2018-09-07 00:24:43 +10:00
Erik Johnston	6707a3212c	Limit the number of PDUs/EDUs per fedreation transaction	2018-09-06 15:23:55 +01:00
Amber Brown	135f3b4390	Merge pull request #3804 from matrix-org/rav/fix_openssl_dep ... bump dep on pyopenssl to 16.x	2018-09-07 00:23:39 +10:00
Amber Brown	2608ebc04c	Port handlers/ to Python 3 (#3803)	2018-09-07 00:22:23 +10:00
Erik Johnston	417e7077aa	Bump version and changelog	2018-09-06 14:08:55 +01:00
Erik Johnston	d64b24dfe6	Synapse 0.33.3.1 (2018-09-06) ... ============================= SECURITY FIXES -------------- - Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) - Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) Internal Changes ---------------- - Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAABCAAGBQJbkPLrAAoJEIofk9V1tejV7K4IAItYIX98DKN9x3FNs9Hd69Pw mZtqmuJ12YAyiVZoR7IJ5GfyctHCYUUXcqmNR1O+2/IRvezFnU6ZTcPW3OfNfnuD vlnMiK53F21T96ul3Wu47Z2wyO+WkoKeXdvlqt3Wa8HpnPbU6y5CSac2vK57ppTU DcMZKXaDae6vA7bsjryYgTRRplH6eFWZEexjbuZudbDvqkySi2zGUTs6SUesMC5B FRPgI6p6sCeQgGMJY+d1i+ZdzWkZhH5OLH2icf2MI1hnhSnRSJ/scSGm5OYDCwYy 03hOy0ZVMr8SB5/j0TjSaFXonKsmPKqxnU2g7crJmvFP4BfJrEbym0y9YhYXnVQ= =8PDU -----END PGP SIGNATURE----- Merge tag 'v0.33.3.1' into release-v0.33.4 Synapse 0.33.3.1 (2018-09-06) ============================= SECURITY FIXES -------------- - Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) - Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) Internal Changes ---------------- - Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802))	2018-09-06 14:08:33 +01:00
Richard van der Hoff	4f8baab0c4	Merge branch 'master' into develop	2018-09-06 13:05:22 +01:00
Richard van der Hoff	625542878d	bump dep on pyopenssl to 16.x	2018-09-06 12:53:15 +01:00
Richard van der Hoff	80189ed27c	prepare v0.33.3.1	2018-09-06 10:26:23 +01:00
Neil Johnson	61b05727fa	guest users should not be part of mau total	2018-09-05 22:30:36 +01:00
Richard van der Hoff	dfba1d843d	Merge pull request #3790 from matrix-org/rav/respect_event_format_in_filter ... Implement 'event_format' filter param in /sync	2018-09-05 16:24:14 +01:00
Erik Johnston	7419764351	User iter* during sync state calculations	2018-09-05 16:19:50 +01:00
Amber Brown	2d2828dcbc	Port http/ to Python 3 (#3771)	2018-09-06 00:10:47 +10:00
Richard van der Hoff	c127c8d042	Fix origin handling for pushed transactions ... Use the actual origin for push transactions, rather than whatever the remote server claimed.	2018-09-05 13:08:07 +01:00
Richard van der Hoff	804dd41e18	Check that signatures on events are valid ... We should check that both the sender's server, and the server which created the event_id (which may be different from whatever the remote server has told us the origin is), have signed the event.	2018-09-05 13:08:07 +01:00
Erik Johnston	5f02017aea	Improve performance of getting typing updates for replication ... Fetching the list of all new typing notifications involved iterating over all rooms and comparing their serial. Lets move to using a stream change cache, like we do for other streams.	2018-09-05 10:20:40 +01:00
Richard van der Hoff	87c18d12ee	Implement 'event_format' filter param in /sync ... This has been specced and part-implemented; let's implement it for /sync (but no other endpoints yet :/).	2018-09-04 15:20:09 +01:00
Neil Johnson	a6cf7d9d9a	Merge pull request #3789 from matrix-org/neilj/improve_threepid_error_strings ... improve human readable error messages	2018-09-04 13:16:00 +00:00
Amber Brown	7e9ced4178	version and towncrier	2018-09-04 21:12:04 +10:00
Neil Johnson	bae37cd811	improve human readable error message	2018-09-04 12:07:00 +01:00
Neil Johnson	c42f7fd7b9	improve human readable error messages	2018-09-04 12:03:17 +01:00
Erik Johnston	3e242dc149	Remove conn_id	2018-09-04 11:45:52 +01:00
Erik Johnston	b13836da7f	Remove conn_id from repl prometheus metrics ... `conn_id` gets set to a random string, and so we end up filling up prometheus with tonnes of data series, which is bad.	2018-09-03 17:22:49 +01:00
Erik Johnston	567363e497	Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func ... Remove unnecessary resolve_events_with_state_map	2018-09-03 16:19:41 +01:00
Neil Johnson	301cb60d0b	assert rather than warn	2018-08-31 17:29:35 +01:00
Neil Johnson	0b01281e77	move threepid checker to config, add missing yields	2018-08-31 17:11:11 +01:00
Neil Johnson	e8e540630e	fix reference to is_threepid_reserved	2018-08-31 16:09:15 +01:00
Neil Johnson	09f3cf1a7e	ensure post registration auth checks do not fail erroneously	2018-08-31 15:42:51 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Amber Brown	14e4d4f4bf	Port storage/ to Python 3 (#3725)	2018-08-31 00:19:58 +10:00
Richard van der Hoff	475253a88e	Merge pull request #3764 from matrix-org/rav/close_db_conn_after_init ... Make sure that we close db connections opened during init	2018-08-30 10:47:27 +01:00
Krombel	79a8a347a6	fix #3445 ... itervalues(d) calls d.itervalues() [PY2] and d.values() [PY3] but SortedDict only implements d.values()	2018-08-29 16:28:25 +02:00
Matthew Hodgson	b1580f50fe	don't return non-LL-member state in incremental sync state blocks (#3760) ... don't return non-LL-member state in incremental sync state blocks	2018-08-28 23:25:58 +01:00
Richard van der Hoff	32eb1dedd2	use abc.abstractproperty ... This gives clearer messages when someone gets it wrong	2018-08-28 17:10:43 +01:00
Richard van der Hoff	0b07f02e19	Make sure that we close db connections opened during init ... We should explicitly close any db connections we open, because failing to do so can block other transactions as per https://github.com/matrix-org/synapse/issues/3682. Let's also try to factor out some of the boilerplate by having server classes define their datastore class rather than duplicating the whole of `setup`.	2018-08-28 13:39:49 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Erik Johnston	9db2476991	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact	2018-08-24 17:00:37 +01:00
Erik Johnston	05077e06fa	Change admin_uri to admin_contact in config and errors	2018-08-24 16:51:27 +01:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	01a5a8b9e3	Fix checking if service notice room is already tagged ... This manifested in synapse repeatedly setting the tag for the room	2018-08-24 16:22:37 +01:00
Erik Johnston	cdd24449ee	Ensure we wake up /sync when we add tag to notice room	2018-08-24 14:50:03 +01:00
Erik Johnston	14d49c51db	Make content of tag an empty object rather than null	2018-08-24 14:44:16 +01:00
Erik Johnston	84b4e76fed	Merge pull request #3754 from matrix-org/erikj/fix_whitelist ... Allow federation_domain_whitelist to be emtpy list	2018-08-24 12:23:39 +01:00
Erik Johnston	1d67b13674	Fix bug when federation_domain_whitelist is an emtpy list ... Outbound federation were incorrectly allowed when the config option was set to an empty list	2018-08-24 12:13:12 +01:00
Erik Johnston	92d50e3c2a	Merge pull request #3753 from matrix-org/erikj/fix_no_server_noticse ... Fix bug where we broke sync when using limit_usage_by_mau	2018-08-24 11:56:08 +01:00
Richard van der Hoff	e94cdbaecf	Merge pull request #3751 from matrix-org/rav/twisted_17 ... Pin to twisted 17.1 or later	2018-08-24 11:55:52 +01:00
Erik Johnston	7f147d623b	Fix bug where we broke sync when using limit_usage_by_mau ... We assumed that we always had service notices configured, but that is not always true	2018-08-24 11:33:50 +01:00
Richard van der Hoff	f584d6108f	Pin to twisted 17.1 or later ... Fixes https://github.com/matrix-org/synapse/issues/3741.	2018-08-24 10:02:31 +01:00
Erik Johnston	cd77270a66	Implement trail users	2018-08-23 19:17:19 +01:00
Erik Johnston	7e6e588e60	Fix bug where we resent "limit exceeded" server notices ... This was due to a bug where we mutated a cached event's contents	2018-08-23 16:21:20 +01:00
Erik Johnston	7a0da69eee	Add missing yield	2018-08-23 10:28:12 +01:00
Erik Johnston	fd2dbf1836	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-22 17:06:10 +01:00
Erik Johnston	9643a6f7f2	Update notice format	2018-08-22 17:00:29 +01:00
Erik Johnston	5c261107c9	Remove unnecessary resolve_events_with_state_map ... We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.	2018-08-22 15:41:15 +01:00
Richard van der Hoff	c7181dcc6c	Merge branch 'master' into develop	2018-08-22 14:37:11 +01:00
Richard van der Hoff	74854a9719	Use recaptcha_ajax.js directly from Google ... This was originally done in commit c75b71a397, but got reverted on this branch due to the PR (#3677) being based on the wrong branch. We're ready to merge this to master now, so let's make it match release-v0.33.3.	2018-08-22 14:30:49 +01:00
Richard van der Hoff	48fec67536	Features ... -------- - Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439)) - Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184)) - speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568)) - implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574)) - Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589)) - Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633)) - Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653)) - Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654)) - Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655)) - Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662)) - Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664)) - Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670)) - set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687)) - Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694)) - For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708)) Bugfixes -------- - Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658)) - Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661)) - Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676)) - Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677)) - Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681)) - Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689)) - Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692)) - Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700)) - Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701)) - Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713)) - Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710)) - Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719)) - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723)) - Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732)) Deprecations and Removals ------------------------- - The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703)) Internal Changes ---------------- - The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423)) - Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632)) - Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647)) - Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660)) - Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668)) - Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669)) - Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678)) - Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679)) - Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684)) - Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690)) - add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707)) - Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709)) - Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbfWKDAAoJEIofk9V1tejVHOMH/RmP+axF2K+Socnnb5Fa6RTo pDJXVwsvNQxcwzZIJXh2TRilVr88jx7A1PHUde0TmkHRmis/bDzZxa59QRA4rFEy fLYMx9Tyt0RzAJrEd0p7jbIkJxxPm5PbMWfCgVfzg44q0kepRW6ZVl1+kFHDrsCo ocE03WdLe2sRHSudaVtBLOVMA1lDE22v40/7xiu9IbUOnF+WZ3cjXYX+SNr5eb6g 7YGbXOk4/E8XoUnpzH5oEnNtiDK5piRrr7o1282kecY20ZMNzKgseBHU6nYMv5ZO TUpCPwbukmzeGHITupY1RkzpmBbEIJTCnmLP8fiUw+RWznYyFC/aRFMiWTzle3A= =ngmn -----END PGP SIGNATURE----- Merge tag 'v0.33.3' Features -------- - Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439)) - Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184)) - speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568)) - implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574)) - Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589)) - Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633)) - Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653)) - Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654)) - Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655)) - Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662)) - Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664)) - Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670)) - set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687)) - Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694)) - For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708)) Bugfixes -------- - Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658)) - Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661)) - Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676)) - Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677)) - Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681)) - Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689)) - Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692)) - Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700)) - Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701)) - Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713)) - Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710)) - Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719)) - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723)) - Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732)) Deprecations and Removals ------------------------- - The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703)) Internal Changes ---------------- - The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423)) - Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632)) - Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647)) - Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660)) - Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668)) - Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669)) - Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678)) - Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679)) - Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684)) - Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690)) - add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707)) - Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709)) - Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))	2018-08-22 14:28:55 +01:00
Richard van der Hoff	4e5a4549b6	bump version to 0.33.3	2018-08-22 14:07:10 +01:00
Erik Johnston	db10f553ba	Merge pull request #3724 from Half-Shot/hs/guest-fetch-event ... Allow guests to use /rooms/:roomId/event/:eventId	2018-08-22 13:41:08 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	8432e2ebd7	Rename WorkerProfileHandler to BaseProfileHandler	2018-08-22 10:13:40 +01:00
Erik Johnston	a81f140880	Add assert to ensure handler is only run on master	2018-08-22 10:11:21 +01:00
Erik Johnston	47b25ba5f3	Remove redundant vars	2018-08-22 10:09:05 +01:00
Erik Johnston	3bf8bab8f9	Merge pull request #3673 from matrix-org/erikj/refactor_state_handler ... Refactor state module to support multiple room versions	2018-08-22 10:04:55 +01:00
Richard van der Hoff	a4cf660a32	Merge pull request #3735 from matrix-org/travis/federation-spelling ... limt -> limit	2018-08-22 09:34:21 +01:00
Richard van der Hoff	0d568ff403	Merge remote-tracking branch 'origin/release-v0.33.3' into develop	2018-08-22 09:15:44 +01:00
Matthew Hodgson	bb81e78ec6	Split the state_group_cache in two (#3726) ... Splits the state_group_cache in two. One half contains normal state events; the other contains member events. The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching. Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.	2018-08-22 00:56:37 +02:00
Richard van der Hoff	f7bf181a90	fix another consent encoding fail	2018-08-21 23:14:25 +01:00
Richard van der Hoff	f7baff6f7b	Fix 500 error from /consent form ... Fixes #3731	2018-08-21 22:47:07 +01:00
Richard van der Hoff	a52f276990	Bugfixes ... -------- - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723)) -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6 Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9 698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU Z2lL1C5ggBCkD43g3orDvehbNWCL5g== =0vEe -----END PGP SIGNATURE----- Merge tag 'v0.33.3rc2' into develop Bugfixes -------- - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))	2018-08-21 20:30:09 +01:00
Hubert Chathi	42a394caa2	allow session_data to be any JSON instead of just a string	2018-08-21 14:51:34 -04:00
Erik Johnston	46c832eaac	Merge pull request #3727 from matrix-org/erikj/dont_error_on_missing_keys ... Don't log exceptions when failing to fetch server keys	2018-08-21 17:07:20 +01:00
Erik Johnston	cd6937fb26	Fix typo	2018-08-21 16:28:10 +01:00
Hubert Chathi	8550a7e9c2	allow auth_data to be any JSON instead of a string	2018-08-21 10:38:00 -04:00
Erik Johnston	c2c153dd3b	Log more detail when we fail to authenticate request	2018-08-21 11:42:49 +01:00
Erik Johnston	808d8e06aa	Don't log exceptions when failing to fetch server keys ... Not being able to resolve or connect to remote servers is an expected error, so we shouldn't log at ERROR with stacktraces.	2018-08-21 11:19:26 +01:00
Erik Johnston	3f6762f0bb	isort	2018-08-21 09:38:38 +01:00
Amber Brown	3b5b64ac99	changelog	2018-08-21 03:48:55 +10:00
Will Hunt	f89f6b7c09	Allow guests to access /rooms/:roomId/event/:eventId	2018-08-20 18:25:54 +01:00
Richard van der Hoff	be6527325a	Fix exceptions when a connection is closed before we read the headers ... This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.	2018-08-20 18:21:10 +01:00
Richard van der Hoff	55e6bdf287	Robustness fix for logcontext filter ... Make the logcontext filter not explode if it somehow ends up with a logcontext of None, since that infinite-loops the whole logging system.	2018-08-20 18:20:07 +01:00
Erik Johnston	b01a755498	Make the in flight requests metrics thread safe	2018-08-20 17:27:52 +01:00
Erik Johnston	1058d14127	Make the in flight background process metrics thread safe	2018-08-20 17:27:24 +01:00
Amber Brown	9a2f960736	version	2018-08-21 00:00:19 +10:00
Amber Brown	324525f40c	Port over enough to get some sytests running on Python 3 (#3668)	2018-08-20 23:54:49 +10:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	8dee601054	Remove redundant room_version checks	2018-08-20 14:48:53 +01:00
Erik Johnston	e21c368b8b	Revert spurious change	2018-08-20 13:54:51 +01:00
Erik Johnston	f2a48d87df	Use get_cache_factor_for function for state_cache ... This allows the cache factor for `state_cache` to be individually specified in the enviroment	2018-08-20 13:01:46 +01:00
Erik Johnston	2aa7cc6a46	Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging ... Fix logging bug in EDU handling over replication	2018-08-20 10:51:45 +01:00
Neil Johnson	e07970165f	rename error code	2018-08-18 14:39:45 +01:00
Neil Johnson	c5171bf171	special case server_notices_mxid	2018-08-18 12:33:07 +01:00
Neil Johnson	ba1fbf7d5b	special case server_notices_mxid	2018-08-18 12:31:08 +01:00
Richard van der Hoff	3cef867cc1	Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commands ... Logcontexts for replication command handlers	2018-08-17 16:22:07 +01:00
Richard van der Hoff	c144252a8c	Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates ... Fix logcontexts for running pushers	2018-08-17 16:21:49 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	2018-08-18 01:08:45 +10:00
Amber Brown	04f5d2db62	Remove v1/register's broken shared secret functionality (#3703)	2018-08-18 00:55:01 +10:00
Erik Johnston	ab822a2d1f	Add some fixmes	2018-08-17 15:31:50 +01:00
Erik Johnston	91cdb6de08	Call UserDirectoryHandler methods directly ... Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.	2018-08-17 15:26:13 +01:00
Neil Johnson	d49b77404b	clean up, no functional changes	2018-08-17 15:21:34 +01:00
Richard van der Hoff	63260397c6	Merge pull request #3701 from matrix-org/rav/use_producer_for_responses ... Use a producer to stream back responses	2018-08-17 14:58:45 +01:00
Richard van der Hoff	3f8709ffe4	Merge pull request #3700 from matrix-org/rav/wait_for_producers ... Refactor request logging code	2018-08-17 14:57:45 +01:00
Neil Johnson	3ee57bdcbb	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-17 14:34:10 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Neil Johnson	b5f638f1f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-17 14:04:15 +01:00
Neil Johnson	9fd161c6fb	Merge branch 'neilj/limit_exceeded_error' of github.com:matrix-org/synapse into neilj/limit_exceeded_error	2018-08-17 13:58:40 +01:00
Neil Johnson	0195dfbf52	server limits config docs	2018-08-17 13:58:25 +01:00
Neil Johnson	69c49d3fa3	Merge branch 'develop' into neilj/limit_exceeded_error	2018-08-17 12:44:26 +00:00
Erik Johnston	38f708a2bb	Remote profile cache should remain in master worker	2018-08-17 11:37:42 +01:00
Erik Johnston	3b2dcfff78	Fix logging bug in EDU handling over replication	2018-08-17 11:11:06 +01:00
Neil Johnson	521d369e7a	remove errant yield	2018-08-17 10:12:11 +01:00
Richard van der Hoff	0e8d78f6aa	Logcontexts for replication command handlers ... Run the handlers for replication commands as background processes. This should improve the visibility in our metrics, and reduce the number of "running db transaction from sentinel context" warnings. Ideally it means converting the things that fire off deferreds into the night into things that actually return a Deferred when they are done. I've made a bit of a stab at this, but it will probably be leaky.	2018-08-17 00:43:43 +01:00
Richard van der Hoff	66f7dc8c87	Fix logcontexts for running pushers ... First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.	2018-08-17 00:32:39 +01:00
Neil Johnson	bcfeb44afe	call reap on start up and fix under reaping bug	2018-08-16 22:55:32 +01:00
Neil Johnson	372bf073c1	block event creation and room creation on hitting resource limits	2018-08-16 21:25:16 +01:00
Neil Johnson	13ad9930c8	add new error type ResourceLimit	2018-08-16 18:02:02 +01:00
Neil Johnson	51b17ec566	flake8	2018-08-16 17:32:22 +01:00
Neil Johnson	3c1080b6e4	refactor for readability, and reuse caching for setting tags	2018-08-16 17:02:04 +01:00
Neil Johnson	eff3ae3b9a	add room tagging	2018-08-16 15:48:34 +01:00
Erik Johnston	7d5b1a60a3	Fix inbound federation on reader worker ... Inbound federation requires calculating push, which in turn relies on having access to account data.	2018-08-16 15:32:20 +01:00
Neil Johnson	bfb6c58624	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-16 15:10:16 +01:00
Neil Johnson	a675f9c556	check for room state before deciding on action	2018-08-16 14:53:35 +01:00
Will Hunt	c151b32b1d	Add GET media/v1/config (#3184)	2018-08-16 14:23:38 +01:00
Matthew Hodgson	762a758fea	lazyload aware /messages (#3589)	2018-08-16 14:22:47 +01:00
Neil Johnson	25d2b5d55f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-16 11:37:17 +01:00
Neil Johnson	df1e4f259f	WIP impl commiting to get feedback	2018-08-16 11:10:53 +01:00
Neil Johnson	c055c91655	fix case where empty string state check is evaulated as False	2018-08-16 11:10:19 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Neil Johnson	8cfad2e686	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-15 17:20:38 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus ... Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Neil Johnson	fc5d937550	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-15 16:31:40 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Neil Johnson	eabc5f8271	wip cut at sending resource server notices	2018-08-15 15:04:52 +01:00
Neil Johnson	c24fc9797b	add new event types	2018-08-15 15:04:30 +01:00
Richard van der Hoff	afcd655ab6	Use a producer to stream back responses ... The problem with dumping all of the json response into the Request object at once is that doing so starts the timeout for the next request to be received: so if it takes longer than 60s to stream back the response to the client, the client never gets it. The correct solution is to use a Producer; then the timeout is only started once all of the content is sent over the TCP connection.	2018-08-15 15:04:16 +01:00
Erik Johnston	dc56c47dc0	Merge pull request #3653 from matrix-org/erikj/split_federation ... Move more federation APIs to workers	2018-08-15 14:59:02 +01:00
Neil Johnson	4601129c44	Merge pull request #3687 from matrix-org/neilj/admin_email ... support admin_email config and pass through into blocking errors,	2018-08-15 13:52:25 +00:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Erik Johnston	488ffe6fdb	Use federation handler function rather than duplicate ... This involves renaming _persist_events to be a public function.	2018-08-15 14:17:18 +01:00
Erik Johnston	773db62a22	Rename slave TransactionStore to SlaveTransactionStore	2018-08-15 14:17:06 +01:00
Richard van der Hoff	eaaa2248ff	Refactor request logging code ... This commit moves a bunch of the logic for deciding when to log the receipt and completion of HTTP requests into SynapseRequest, rather than in the request handling wrappers. Advantages of this are: * we get logs for *all* requests (including OPTIONS and HEADs), rather than just those that end up hitting handlers we've remembered to decorate correctly. * when a request handler wires up a Producer (as the media stuff does currently, and as other things will do soon), we log at the point that all of the traffic has been sent to the client.	2018-08-15 13:47:52 +01:00
Neil Johnson	87a824bad1	clean up AuthError	2018-08-15 11:58:03 +01:00
Neil Johnson	c4eb97518f	Merge branch 'neilj/update_limits_error_codes' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-15 11:53:01 +01:00
Neil Johnson	55afba0fc5	update admin email to uri	2018-08-15 11:41:18 +01:00
Neil Johnson	1c5e690a6b	Merge pull request #3690 from matrix-org/neilj/change_prometheus_mau_metric_name ... combine mau metrics into one group	2018-08-15 09:53:59 +00:00
Erik Johnston	fef2e65d12	Merge pull request #3667 from matrix-org/erikj/fixup_unbind ... Don't fail requests to unbind 3pids for non supporting ID servers	2018-08-15 10:32:12 +01:00
Neil Johnson	b8429c7c81	update error codes for resource limiting	2018-08-15 10:19:48 +01:00
Neil Johnson	ab035bdeac	replace admin_email with admin_uri for greater flexibility	2018-08-15 10:16:41 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	b586b8b986	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 17:43:22 +01:00
Neil Johnson	70e48cbbb1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/change_prometheus_mau_metric_name	2018-08-14 17:40:59 +01:00
Neil Johnson	9ecbaf8ba8	adding missing yield	2018-08-14 16:55:28 +01:00
Neil Johnson	e5962f845c	pep8	2018-08-14 16:36:14 +01:00
Neil Johnson	e7d091fb86	combine mau metrics into one group	2018-08-14 16:26:55 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	2545993ce4	make comments clearer	2018-08-14 15:48:12 +01:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	c74c71128d	remove blank line	2018-08-14 15:06:24 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	bd92c8eaa7	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:56:45 +01:00
Neil Johnson	99ebaed8e6	Update register.py ... remove comments	2018-08-14 14:55:55 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Neil Johnson	e2c9fe0a6a	backout ability to pass in event type to server notices	2018-08-14 13:32:56 +01:00
Amber Brown	591bf87c6a	Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres	2018-08-14 20:56:23 +10:00
Neil Johnson	9b75c78b4d	support server notice state events for resource limits	2018-08-14 11:20:41 +01:00
Neil Johnson	63417c31e9	fix typo	2018-08-13 22:36:52 +01:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Richard van der Hoff	50bcaf1a27	Merge pull request #3677 from andrewshadura/master ... Use recaptcha_ajax.js directly from Google	2018-08-13 18:52:56 +01:00
Richard van der Hoff	34f51babc5	Revert "Use recaptcha_ajax.js directly from Google"	2018-08-13 18:48:03 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" ... This reverts commit 0d43f991a1.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	4f7064f6b5	missing import	2018-08-12 19:14:31 -04:00
Matthew Hodgson	f0cede5556	missing import	2018-08-12 19:14:31 -04:00
Matthew Hodgson	54ac18e832	use parse_string	2018-08-12 19:14:31 -04:00
Matthew Hodgson	66a4ca1d28	404 nicely if you try to interact with a missing current version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	72788cf9c1	support DELETE /version with no args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	edc427a351	flake8	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	15d513f16f	fix idiocies and so make tests pass	2018-08-12 19:14:31 -04:00
Matthew Hodgson	b5eee511c7	don't needlessly return user_id	2018-08-12 19:14:31 -04:00
Matthew Hodgson	93d174bcc4	improve docstring	2018-08-12 19:14:31 -04:00
Matthew Hodgson	5e42c45c96	switch get_current_version_info back to being get_version_info	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	14b3da63a3	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f0791b7bd	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f500cb39e	more docstring for the e2e_room_keys rest	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8d14598e90	add storage docstring; remove unused set_e2e_room_keys	2018-08-12 19:14:31 -04:00
Matthew Hodgson	ca0b052307	fix factoring out of _should_replace_room_key	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cf1e2000f6	document the API	2018-08-12 19:13:09 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Neil Johnson	31fa743567	fix sqlite/postgres incompatibility in reap_monthly_active_users	2018-08-11 22:38:34 +01:00
Neil Johnson	6c6aba76e1	implementation of server notices to alert on hitting resource limits	2018-08-10 15:12:59 +01:00
Richard van der Hoff	c31793a784	Merge branch 'rav/fix_linearizer_cancellation' into develop	2018-08-10 14:57:27 +01:00
Richard van der Hoff	c08f9d95b2	log *after* reloading log config ... ... because logging *before* reloading means the log message gets lost in the old MemoryLogger	2018-08-10 14:56:48 +01:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Andrej Shadura	c75b71a397	Use recaptcha_ajax.js directly from Google ... The script recaptcha_ajax.js contains minified non-free code which we probably cannot redistribute. Since it talks to Google servers anyway, it is better to just download it from Google directly instead of shipping it. This fixes #1932.	2018-08-10 13:36:14 +02:00
Richard van der Hoff	3c0213a217	Merge pull request #3439 from vojeroen/send_sni_for_federation_requests ... send SNI for federation requests	2018-08-10 12:23:54 +01:00
Richard van der Hoff	638d35ef08	Fix linearizer cancellation on twisted < 18.7 ... Turns out that cancellation of inlineDeferreds didn't really work properly until Twisted 18.7. This commit refactors Linearizer.queue to avoid inlineCallbacks.	2018-08-10 10:59:09 +01:00
Neil Johnson	01021c812f	wip at implementing MSC 7075	2018-08-09 22:16:00 +01:00
Jeroen	2e9c73e8ca	more generic conversion of str/bytes to unicode	2018-08-09 21:31:26 +02:00
Jeroen	64899341dc	include private functions from twisted	2018-08-09 21:04:22 +02:00
Neil Johnson	885ea9c602	rename _user_last_seen_monthly_active	2018-08-09 18:02:12 +01:00
Neil Johnson	c1f9dec92a	fix errant parenthesis	2018-08-09 17:43:26 +01:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	bb99b1f550	Add fast path in state res for zero prev events	2018-08-09 14:58:49 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Erik Johnston	152c0aa58e	Add constants for room versions	2018-08-09 14:55:47 +01:00
Erik Johnston	119451dcd1	Refactor state module ... We split out the actual state resolution algorithm to prepare for having multiple versions.	2018-08-09 14:55:47 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	3dce9050cf	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:42:02 +01:00
Neil Johnson	0ad98e38d0	Merge pull request #3655 from matrix-org/neilj/disable_hs ... Flag to disable HS without disabling federation	2018-08-09 10:41:43 +00:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Erik Johnston	a6c813761a	Docstrings	2018-08-09 10:41:08 +01:00
Erik Johnston	b179537f2a	Move clean_room_for_join to master	2018-08-09 10:37:38 +01:00
Erik Johnston	72d1902bbe	Fixup doc comments	2018-08-09 10:23:49 +01:00
Amber Brown	984376745b	Merge branch 'master' into develop	2018-08-09 19:23:47 +10:00
Amber Brown	dafe90a6be	0.33.2	2018-08-09 19:20:41 +10:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Erik Johnston	bf7598f582	Log when we 3pid/unbind request fails	2018-08-09 10:09:56 +01:00
Erik Johnston	2bdafaf3c1	Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet ... Add helper base class for generating new replication endpoints	2018-08-09 10:06:23 +01:00
Erik Johnston	62564797f5	Fixup wording and remove dead code	2018-08-09 09:56:10 +01:00
Richard van der Hoff	bb89c84614	Merge pull request #3664 from matrix-org/rav/federation_metrics ... more metrics for the federation and appservice senders	2018-08-08 23:16:58 +01:00
Jeroen	d5c0ce4cad	updated docstring for ServerContextFactory	2018-08-08 19:25:01 +02:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Neil Johnson	839a317c96	fix pep8 too many lines	2018-08-08 17:39:04 +01:00
Neil Johnson	5298d79fb5	Merge branch 'develop' into neilj/disable_hs	2018-08-08 16:13:03 +00:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	d2f3ef98ac	Merge branch 'develop' into neilj/disable_hs	2018-08-08 15:55:47 +00:00
Neil Johnson	2b5baebeba	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-08 13:47:07 +01:00
Neil Johnson	be59910b93	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users	2018-08-08 13:45:21 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Neil Johnson	312ae74746	typos	2018-08-08 13:33:16 +01:00
Erik Johnston	360ba89c50	Don't fail requests to unbind 3pids for non supporting ID servers ... Older identity servers may not support the unbind 3pid request, so we shouldn't fail the requests if we received one of 400/404/501. The request still fails if we receive e.g. 500 responses, allowing clients to retry requests on transient identity server errors that otherwise do support the API. Fixes #3661	2018-08-08 12:06:18 +01:00
Erik Johnston	bebe325e6c	Rename POST param to METHOD	2018-08-08 10:36:18 +01:00
Erik Johnston	5011417632	Fixup logging and docstrings	2018-08-08 10:29:58 +01:00
Richard van der Hoff	bab94da79c	fix metric name	2018-08-07 22:11:45 +01:00
Richard van der Hoff	53bca4690b	more metrics for the federation and appservice senders	2018-08-07 19:09:48 +01:00
Neil Johnson	ef3589063a	prevent total number of reserved users being too large	2018-08-07 17:59:27 +01:00
Neil Johnson	e8eba2b4e3	implement reserved users for mau limits	2018-08-07 17:49:43 +01:00
Richard van der Hoff	3523f5432a	Don't expose default_room_version as config opt	2018-08-07 12:51:57 +01:00
Amber Brown	848431be1d	fix for rc1	2018-08-07 21:13:22 +10:00
Amber Brown	3b9662339b	version	2018-08-07 21:10:52 +10:00
Erik Johnston	cd9765805e	Allow ratelimiting on workers	2018-08-07 10:50:28 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Richard van der Hoff	ca9bc1f4fe	Fix occasional glitches in the synapse_event_persisted_position metric ... Every so often this metric glitched to a negative number. I'm assuming it was due to backfilled events.	2018-08-07 10:00:03 +01:00
Neil Johnson	a74b25faaa	WIP building out mau reserved users	2018-08-06 23:25:25 +01:00
Neil Johnson	7daa8a78c5	load mau limit threepids	2018-08-06 22:55:05 +01:00
Neil Johnson	89834d9a29	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-06 21:56:56 +01:00
Neil Johnson	e54794f5b6	Fix postgres compatibility bug	2018-08-06 21:55:54 +01:00
Neil Johnson	7bcf126b18	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-06 21:39:44 +01:00
Neil Johnson	1911c037cb	update comments to reflect new sig	2018-08-06 18:01:46 +01:00
Neil Johnson	33bd07d062	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-06 17:52:28 +01:00
Neil Johnson	16d78be315	make use of _simple_select_one_onecol, improved comments	2018-08-06 17:51:15 +01:00
Richard van der Hoff	19a17068f1	Check m.room.create for sane room_versions	2018-08-06 16:11:24 +01:00
Erik Johnston	96a9a29645	Pull in necessary stores in federation_reader	2018-08-06 15:23:57 +01:00
Erik Johnston	62ace05c45	Move necessary storage functions to worker classes	2018-08-06 15:23:38 +01:00
Erik Johnston	1e2bed9656	Import all functions from TransactionStore	2018-08-06 15:23:38 +01:00
Erik Johnston	a3f5bf79a0	Add EDU/query handling over replication	2018-08-06 15:23:31 +01:00
Erik Johnston	e26dbd82ef	Add replication APIs for persisting federation events	2018-08-06 15:02:28 +01:00
Erik Johnston	051a99c400	Fix isort	2018-08-06 14:29:31 +01:00
Richard van der Hoff	f900d50824	include known room versions in outgoing make_joins	2018-08-06 13:45:37 +01:00
Neil Johnson	42c6823827	disable HS from config	2018-08-04 22:07:04 +01:00
Neil Johnson	e40a510fbf	py3 fix	2018-08-03 23:19:13 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Will Hunt	16d9701892	Return M_NOT_FOUND when a profile could not be found. (#3596)	2018-08-03 19:08:05 +01:00
Neil Johnson	e10830e976	wip commit - tests failing	2018-08-03 17:55:50 +01:00
Richard van der Hoff	3777fa26aa	sanity check response from make_join	2018-08-03 16:08:32 +01:00
Richard van der Hoff	0d63d93ca8	Enforce compatibility when processing make_join requests ... Reject make_join requests from servers which do not support the room version. Also include the room version in the response.	2018-08-03 16:08:32 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Richard van der Hoff	15c1ae45e5	Docstrings for BaseFederationServlet ... ... to save me reverse-engineering this stuff again.	2018-08-03 16:08:32 +01:00
Neil Johnson	5593ff6773	fix (lots of) py3 test failures	2018-08-03 14:59:17 +01:00
Neil Johnson	950807d93a	fix caching and tests	2018-08-03 13:49:53 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Erik Johnston	cb298ff623	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet	2018-08-03 09:25:15 +01:00
Neil Johnson	c0affa7b4f	update generate_monthly_active_users, and reap_monthly_active_users	2018-08-02 23:03:01 +01:00
Neil Johnson	5e2f7b8084	typo	2018-08-02 22:47:48 +01:00
Neil Johnson	4ecb4bdac9	wip attempt at caching	2018-08-02 22:41:05 +01:00
Richard van der Hoff	1fa98495d0	Merge pull request #3639 from matrix-org/rav/refactor_error_handling ... Clean up handling of errors from outbound requests	2018-08-02 17:38:24 +01:00
Richard van der Hoff	bdae8f2e68	Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling ... Factor out exception handling in federation_client	2018-08-02 17:37:46 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Neil Johnson	9180061b49	remove unused count_monthly_users	2018-08-02 15:55:29 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	43ecfe0b10	Synapse 0.33.1 (2018-08-02) ... =========================== SECURITY FIXES -------------- - Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_) - Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7 SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc= =U/ir -----END PGP SIGNATURE----- Merge tag 'v0.33.1' Synapse 0.33.1 (2018-08-02) =========================== SECURITY FIXES -------------- - Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_) - Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_)	2018-08-02 15:40:44 +01:00
Richard van der Hoff	14a4e7d5a4	Prepare 0.33.1	2018-08-02 15:31:04 +01:00
Richard van der Hoff	0bf5ec0db7	Check room visibility for /event/ requests ... Make sure that the user has permission to view the requeseted event for /event/{eventId} and /room/{roomId}/event/{eventId} requests. Also check that the event is in the given room for /room/{roomId}/event/{eventId}, for sanity.	2018-08-02 15:03:27 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Neil Johnson	00f99f74b1	insertion into monthly_active_users	2018-08-02 13:47:19 +01:00
Neil Johnson	4a6725d9d1	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-02 11:04:18 +01:00
Neil Johnson	165e067033	Revert "change monthly_active_users table to be a single column" ... This reverts commit ec716a35b2.	2018-08-02 10:59:58 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	08281fe6b7	self.db_conn unused	2018-08-01 23:26:24 +01:00
Neil Johnson	c21d82bab3	normalise reaping query	2018-08-01 23:24:38 +01:00
Neil Johnson	ec716a35b2	change monthly_active_users table to be a single column	2018-08-01 17:54:37 +01:00
Neil Johnson	d766f26de9	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-01 17:49:41 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Richard van der Hoff	b8d7d3996b	Merge pull request #3620 from fuzzmz/return-404-room-not-found ... return 404 if room not found	2018-08-01 16:34:32 +01:00
Neil Johnson	b7f203a566	count_monthly_users is now async	2018-08-01 16:17:42 +01:00
Neil Johnson	7ff44d9215	improve clarity	2018-08-01 16:17:00 +01:00
Richard van der Hoff	01e93f48ed	Kill off MatrixCodeMessageException ... This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.	2018-08-01 16:02:46 +01:00
Richard van der Hoff	018d75a148	Refactor code for turning HttpResponseException into SynapseError ... This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call.	2018-08-01 16:02:46 +01:00
Richard van der Hoff	fa7dc889f1	Be more careful which errors we send back over the C-S API ... We really shouldn't be sending all CodeMessageExceptions back over the C-S API; it will include things like 401s which we shouldn't proxy. That means that we need to explicitly turn a few HttpResponseExceptions into SynapseErrors in the federation layer. The effect of the latter is that the matrix errcode will get passed through correctly to calling clients, which might help with some of the random M_UNKNOWN errors when trying to join rooms.	2018-08-01 16:02:38 +01:00
Richard van der Hoff	c82ccd3027	Factor out exception handling in federation_client ... Factor out the error handling from make_membership_event, send_join, and send_leave, so that it can be shared.	2018-08-01 16:01:04 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Erik Johnston	4b256b9271	_persist_auth_tree no longer returns anything	2018-08-01 13:39:07 +01:00
Neil Johnson	4e5ac901dd	clean up	2018-08-01 12:03:57 +01:00
Neil Johnson	f9f5559971	fix comment	2018-08-01 12:03:42 +01:00
Neil Johnson	0aba3d361a	count_monthly_users() async	2018-08-01 11:47:58 +01:00
Neil Johnson	2c54f1c225	remove need to plot limit_usage_by_mau	2018-08-01 11:46:59 +01:00
Richard van der Hoff	f49147d14f	Merge pull request #3634 from matrix-org/rav/wtf_is_a_replication_layer ... rename replication_layer to federation_client	2018-08-01 10:29:29 +01:00
Richard van der Hoff	cab782c17e	Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator ... Rewrite cache list decorator	2018-08-01 10:28:56 +01:00
Neil Johnson	6023cdd227	remove errant print	2018-08-01 10:27:17 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Neil Johnson	c507fa15ce	only need to loop if mau limiting is enabled	2018-08-01 10:20:42 +01:00
Travis Ralston	37be52ac34	limt -> limit	2018-07-31 16:29:09 -06:00
Serban Constantin	70af98e361	return NotFoundError if room not found ... Per the Client-Server API[0] we should return `M_NOT_FOUND` if the room isn't found instead of generic SynapseError. This ensures that /directory/list API returns 404 for room not found instead of 400. [0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid Signed-off-by: Serban Constantin <serban.constantin@gmail.com>	2018-07-31 21:47:23 +03:00
Travis Ralston	5e2ee64660	Merge pull request #3628 from turt2live/travis/goodby-pdu-failures ... Remove pdu_failures from transactions	2018-07-31 12:13:09 -06:00
Neil Johnson	6ef983ce5c	api into monthly_active_users table	2018-07-31 16:36:24 +01:00
Richard van der Hoff	bdbdceeafa	rename replication_layer to federation_client ... I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.	2018-07-31 15:44:05 +01:00
Erik Johnston	443da003bc	Use new helper base class for membership requests	2018-07-31 14:32:23 +01:00
Erik Johnston	729b672823	Use new helper base class for ReplicationSendEventRestServlet	2018-07-31 14:32:23 +01:00
Erik Johnston	d81602b75a	Add helper base class for generating new replication endpoints ... This will hopefully reduce the boiler plate required to implement new internal HTTP requests.	2018-07-31 14:32:20 +01:00
Richard van der Hoff	5de936caa1	Merge pull request #3612 from matrix-org/rav/store_heirarchy ... Make EventStore inherit from EventFederationStore	2018-07-31 13:44:04 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Richard van der Hoff	0bc9b9e397	reinstate explicit include of EventsWorkerStore	2018-07-31 13:11:04 +01:00
Travis Ralston	e908b86832	Remove pdu_failures from transactions ... The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.	2018-07-30 16:28:47 -06:00
Krombel	254e8267e2	Only import secrets when available ... secrets got introduced in python 3.6 so this class is not available in 3.5 and before. This now checks for the current running version and only tries using secrets if the version is 3.6 or above Signed-Off-By: Matthias Kesler <krombel@krombel.de>	2018-07-30 23:59:02 +02:00
Neil Johnson	21276ff846	remove errant logging	2018-07-30 22:42:12 +01:00
Neil Johnson	fef7e58ac6	actually close conn	2018-07-30 22:29:44 +01:00
Neil Johnson	9b13817e06	factor out metrics from __init__ to app/homeserver	2018-07-30 22:07:07 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Jeroen	2903e65aff	fix isort	2018-07-29 19:47:08 +02:00
Richard van der Hoff	a8cbce0ced	fix invalidation	2018-07-27 16:17:17 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	f102c05856	Rewrite cache list decorator ... Because it was complicated and annoyed me. I suspect this will be more efficient too.	2018-07-27 13:47:04 +01:00
Jeroen	8e3f75b39a	fix accidental removal of hs	2018-07-27 12:17:31 +02:00
Richard van der Hoff	b0b5566f36	Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid ... if inviter_display_name == ""||None then default to inviter MXID	2018-07-27 10:08:39 +01:00
Richard van der Hoff	7041cd872b	Merge branch 'develop' into send_sni_for_federation_requests	2018-07-27 09:17:11 +01:00
Richard van der Hoff	d42455cdc9	Merge pull request #3616 from matrix-org/travis/event_id_send_leave ... Update the send_leave path to be an event_id	2018-07-26 23:25:17 +01:00
Richard van der Hoff	65c8dee900	Merge pull request #3614 from matrix-org/rav/stop_populating_event_content ... Stop populating events.content	2018-07-26 22:54:08 +01:00
Matthew Hodgson	a75231b507	Deduplicate redundant lazy-loaded members (#3331) ... * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review	2018-07-26 22:51:30 +01:00
Richard van der Hoff	9e68b1bd2d	Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id ... Remove some redundant joins on event_edges.room_id	2018-07-26 22:31:01 +01:00
Travis Ralston	7d32f0d745	Update the send_leave path to be an event_id ... It's still not used, however the parameter is an event ID not a transaction ID.	2018-07-26 14:41:59 -06:00
Richard van der Hoff	ef9d51b081	Merge pull request #3610 from matrix-org/rav/fix_looping_calls ... Fix some looping_call calls which were broken in #3604	2018-07-26 14:55:57 +01:00
Richard van der Hoff	51d7df1915	Create the column nullable ... There's no real point in ever making the column non-nullable, and doing so breaks the sytests.	2018-07-26 14:54:04 +01:00
Richard van der Hoff	5c1d301fd9	Stop populating events.content ... This field is no longer read from, so we should stop populating it. Once we're happy that this doesn't break everything, and a rollback is unlikely, we can think about dropping the column.	2018-07-26 14:43:02 +01:00
Richard van der Hoff	bd4b25f4d0	Remove some redundant joins on event_edges.room_id ... We've long passed the point where it's possible to have the same event_id in different tables, so these join conditions are redundant: we can just join on event_id. event_edges is of non-trivial size, and the room_id column is wasteful, so let's stop reading from it. In future, we can stop writing to it, and then drop it.	2018-07-26 13:19:08 +01:00
Richard van der Hoff	1b4d73fa52	comment on event_edges	2018-07-26 12:53:51 +01:00
Richard van der Hoff	21e878ebb6	Make EventStore inherit from EventFederationStore ... (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py.	2018-07-26 12:48:51 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	1bcd0490c2	Merge pull request #2970 from matrix-org/matthew/filter_members ... Implement the lazy_load_members room state filter parameter	2018-07-26 00:03:01 +01:00
Matthew Hodgson	bc7944e6d2	switch missing_types to be a bool	2018-07-25 23:36:31 +01:00
Travis Ralston	d8e65ed7e1	Fix a minor documentation typo in on_make_leave	2018-07-25 15:44:41 -06:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Matthew Hodgson	7d9fb88617	incorporate more review.	2018-07-25 16:33:50 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Erik Johnston	3849f7f69f	Merge pull request #3603 from matrix-org/erikj/handle_outliers ... Correctly handle outliers during persist events	2018-07-25 13:24:04 +01:00
Richard van der Hoff	cee1ae1b72	Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more ... Fix another logcontext leak in _persist_events	2018-07-25 11:56:00 +01:00
Richard van der Hoff	32b30e15f2	Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error ... Fix occasional 'tuple index out of range' error	2018-07-25 11:55:45 +01:00
Richard van der Hoff	4081bd1560	Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache ... Fix updating of cached remote profiles	2018-07-25 11:54:46 +01:00
Erik Johnston	7780a7b47c	Actually fix it by adding continue	2018-07-25 11:13:20 +01:00
Richard van der Hoff	1be94440d3	Fix occasional 'tuple index out of range' error ... This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying.	2018-07-25 11:05:58 +01:00
Richard van der Hoff	07defd5fe6	Fix another logcontext leak in _persist_events ... We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from	2018-07-25 10:53:23 +01:00
Richard van der Hoff	55acd6856c	Fix updating of cached remote profiles ... _update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when it was called, would just return a generator object, without actually running any of the method body.	2018-07-25 10:34:48 +01:00
Richard van der Hoff	f59be4eb0e	Fix unit tests ... on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival.	2018-07-25 10:30:36 +01:00
Erik Johnston	a297ff2b16	Fix typo in conditional	2018-07-25 09:48:01 +01:00
Richard van der Hoff	371da42ae4	Wrap a number of things that run in the background ... This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.	2018-07-25 09:41:12 +01:00
Erik Johnston	ec56121b0d	Correctly handle outliers during persist events ... We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601	2018-07-25 09:35:02 +01:00
Matthew Hodgson	cb5c37a57c	handle the edge case for _get_some_state_from_cache where types is []	2018-07-24 20:34:45 +01:00
Michael Telatynski	87951d3891	Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid	2018-07-24 17:17:46 +01:00
Erik Johnston	8b8c4f34a3	Replace usage of get_current_toke with StreamToken.START ... This allows us to handle /context/ requests on the client_reader worker without having to pull in all the various stream handlers (e.g. precence, typing, pushers etc). The only thing the token gets used for is pagination, and that ignores everything but the room portion of the token.	2018-07-24 16:49:17 +01:00
Erik Johnston	3188973857	Pull out did_forget to worker store	2018-07-24 16:49:14 +01:00
Erik Johnston	60a1d147a7	Merge pull request #3595 from matrix-org/erikj/use_deltas ... Use deltas to calculate current state deltas	2018-07-24 15:41:18 +01:00
Erik Johnston	709c309b0e	Expand on docstring comment about return value	2018-07-24 15:12:50 +01:00
Erik Johnston	8f65ab98d2	Remove unnecessary iteritems	2018-07-24 15:08:01 +01:00
Erik Johnston	ed0dd68731	Fixup comment (and indent)	2018-07-24 14:31:38 +01:00
Erik Johnston	811ac73a42	Don't fetch state from the database unless needed	2018-07-24 14:18:23 +01:00
Erik Johnston	a79410e7b8	Have _get_new_state_after_events return delta ... If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.	2018-07-24 14:14:17 +01:00
Richard van der Hoff	81946db9cf	Merge pull request #3587 from matrix-org/rav/better_exception_logging ... Improve logging for exceptions when handling PDUs	2018-07-24 14:12:12 +01:00
Richard van der Hoff	a321f78991	Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups ... Fixes and optimisations for resolve_state_groups	2018-07-24 14:11:45 +01:00
Richard van der Hoff	93b0722c50	Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room ... Remove redundant checks on room forgottenness	2018-07-24 14:11:11 +01:00
Matthew Hodgson	454f59b7ad	Merge branch 'develop' into matthew/filter_members	2018-07-24 14:03:37 +01:00
Matthew Hodgson	1a01a5b964	clarify comment on p_ids	2018-07-24 14:03:15 +01:00
Erik Johnston	223341205e	Don't require to_delete to have event_ids	2018-07-24 14:02:40 +01:00
Matthew Hodgson	e22700c3dd	consider non-filter_type types as wildcards, thus missing from the state-group-cache	2018-07-24 13:59:07 +01:00
Matthew Hodgson	eb1d911ab7	rather than adding ll_ids, remove them from p_ids	2018-07-24 13:40:49 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	30bfed5aa5	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-24 11:46:09 +01:00
Erik Johnston	97acd385a3	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta	2018-07-24 11:32:13 +01:00
Erik Johnston	0fa73e4a63	Remove unnecessary if	2018-07-24 11:19:23 +01:00
Erik Johnston	ff5426f6b8	Speed up _calculate_state_delta	2018-07-24 10:55:11 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Erik Johnston	d436ad332c	Merge pull request #3555 from matrix-org/erikj/client_apis_move ... Make client_reader support some more read only APIs	2018-07-24 10:42:28 +01:00
Erik Johnston	536bc63a4e	Merge branch 'develop' into erikj/client_apis_move	2018-07-24 09:57:05 +01:00
Richard van der Hoff	cf2d15c6a9	another couple of logcontext leaks	2018-07-24 00:57:48 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	69fb5dbdab	fix idiocy	2018-07-24 00:04:44 +01:00
Richard van der Hoff	1938cffaea	Add some measure blocks to persist_events ... ... to help us figure out where 40% of CPU is going	2018-07-23 23:48:19 +01:00
Matthew Hodgson	efcdacad7d	handle case where types is [] on postgres correctly	2018-07-23 22:41:05 +01:00
Richard van der Hoff	ce0c18dec5	Improve logging for exceptions handling PDUs ... when we get an exception handling a federation PDU, log the whole stacktrace.	2018-07-23 22:13:19 +01:00
Richard van der Hoff	c1f80effbe	Handle delta_ids being None in _update_context_for_auth_events ... it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with https://github.com/matrix-org/synapse/issues/3364)	2018-07-23 22:06:50 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Richard van der Hoff	5c705f70c9	Fixes and optimisations for resolve_state_groups ... First of all, fix the logic which looks for identical input state groups so that we actually use them. This turned out to be most easily done by factoring the relevant code out to a separate function so that we could do an early return. Secondly, avoid building the whole `conflicted_state` dict (which was only ever used as a boolean flag). Thirdly, replace the construction of the `state` dict (which mapped from keys to events that set them), with an optimistic construction of the resolution result assuming there will be no conflicts. This should be no slower than building the old `state` dict, and: - in the conflicted case, we'll short-cut it, saving part of the work - in the unconflicted case, it saves rebuilding the resolution from the `state` dict. Finally, do a couple of s/values/itervalues/.	2018-07-23 19:10:11 +01:00
Erik Johnston	8b9f164fff	Comments	2018-07-23 17:43:01 +01:00
Erik Johnston	50c60e5fad	Only get cached state from context in persist_event ... We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.	2018-07-23 17:21:40 +01:00
Richard van der Hoff	4f5cc8e4e7	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-23 17:15:12 +01:00
Richard van der Hoff	dae6dc1e77	Remove redundant checks on room forgottenness ... Fixes #3550	2018-07-23 17:13:34 +01:00
Erik Johnston	0faa3223cd	Fix missing attributes on workers. ... This was missed during the transition from attribute to getter for getting state from context.	2018-07-23 16:28:00 +01:00
Erik Johnston	999bcf9d01	Fix EventContext when using workers ... We were: 1. Not correctly setting all attributes 2. Using defer.inlineCallbacks in a non-generator	2018-07-23 15:24:21 +01:00
Erik Johnston	4797ed000e	Update docstrings to make sense	2018-07-23 15:05:56 +01:00
Erik Johnston	0b0b24cb82	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move	2018-07-23 13:21:15 +01:00
Erik Johnston	027bc01a1b	Add support for updating state	2018-07-23 13:17:25 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Erik Johnston	440b8845b5	Make EventContext lazy load state	2018-07-23 12:56:56 +01:00
Erik Johnston	842cdece42	pep8	2018-07-23 12:31:35 +01:00
Erik Johnston	acbfdc3442	Refcator EventContext to accept state during init	2018-07-23 12:20:55 +01:00
Matthew Hodgson	9b34f3ea3a	Merge branch 'develop' into matthew/sync_deleted_devices	2018-07-23 10:03:28 +01:00
Matthew Hodgson	c1bf2b587e	add trailing comma	2018-07-23 09:56:23 +01:00
Amber Brown	3132b89f12	Make the rest of the .iterwhatever go away (#3562)	2018-07-21 15:47:18 +10:00
Erik Johnston	5c88bb722f	Move PaginationHandler to its own file	2018-07-20 15:32:23 +01:00
Erik Johnston	0ecf68aedc	Move check_in_room_or_world_readable to Auth	2018-07-20 15:30:59 +01:00
Richard van der Hoff	3d6df84658	Test and fix support for cancellation in Linearizer	2018-07-20 13:59:55 +01:00
Richard van der Hoff	4f67623674	Merge pull request #3571 from matrix-org/rav/limiter_fixes ... A set of improvements to the Limiter	2018-07-20 13:53:03 +01:00
Amber Brown	e1a237eaab	Admin API for creating new users (#3415)	2018-07-20 22:41:13 +10:00
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	2018-07-20 13:11:43 +01:00
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	2018-07-20 12:50:27 +01:00
Richard van der Hoff	d7275eecf3	Add a sleep to the Limiter to fix stack overflows. ... Fixes #3570	2018-07-20 12:37:12 +01:00
Matthew Hodgson	2f558300cc	fix thinkos; unbreak tests	2018-07-19 19:22:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Richard van der Hoff	5f3658baf5	Merge pull request #3377 from Valodim/note-affinity ... document that the affinity package is required for the cpu_affinity setting	2018-07-19 14:35:06 +01:00
Amber Brown	9a8acdc720	Merge branch 'master' into develop	2018-07-19 21:16:44 +10:00
Amber Brown	38f53399a2	0.33 final	2018-07-19 21:11:40 +10:00
Amber Brown	95ccb6e2ec	Don't spew errors because we can't save metrics (#3563)	2018-07-19 20:58:18 +10:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Matthew Hodgson	9e40834f74	yes, we do need to invalidate the device_id_exists_cache when deleting a remote device	2018-07-19 11:15:10 +01:00
Richard van der Hoff	f1a15ea206	revert 00bc979 ... ... we've fixed the things that caused the warnings, so we should reinstate the warning.	2018-07-19 11:14:20 +01:00
Richard van der Hoff	1ffb7bec20	Merge remote-tracking branch 'origin/release-v0.33.0' into develop	2018-07-19 11:12:33 +01:00
Richard van der Hoff	c754e006f4	Merge pull request #3556 from matrix-org/rav/background_processes ... Run things as background processes	2018-07-19 11:04:18 +01:00
Amber Brown	a97c845271	Move v1-only APIs into their own module & isolate deprecated ones (#3460)	2018-07-19 20:03:33 +10:00
Matthew Hodgson	c0685f67c0	spell out that include_deleted_devices requires include_all_devices	2018-07-19 10:59:02 +01:00
Richard van der Hoff	00bc979137	Disable logcontext warning ... Temporary workaround to #3518 while we release 0.33.0.	2018-07-19 10:51:15 +01:00
Erik Johnston	6f62a6ef21	Merge pull request #3554 from matrix-org/erikj/response_metrics_code ... Add response code to response timer metrics	2018-07-19 10:25:18 +01:00
Richard van der Hoff	8c69b735e3	Make Distributor run its processes as a background process ... This is more involved than it might otherwise be, because the current implementation just drops its logcontexts and runs everything in the sentinel context. It turns out that we aren't actually using a bunch of the functionality here (notably suppress_failures and the fact that Distributor.fire returns a deferred), so the easiest way to fix this is actually by simplifying a bunch of code.	2018-07-18 20:55:05 +01:00
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	2018-07-18 20:55:05 +01:00
Erik Johnston	9b596177ae	Add some room read only APIs to client_reader	2018-07-18 15:33:03 +01:00
Erik Johnston	bacdf0cbf9	Move RoomContextHandler out of Handlers ... This is in preparation for moving GET /context/ to a worker	2018-07-18 15:33:03 +01:00
Erik Johnston	8cb8df55e9	Split MessageHandler into read only and writers ... This will let us call the read only parts from workers, and so be able to move some APIs off of master, e.g. the `/state` API.	2018-07-18 15:33:03 +01:00
Richard van der Hoff	65d6a0e477	Merge pull request #3553 from matrix-org/rav/background_process_tracking ... Resource tracking for background processes	2018-07-18 14:27:50 +01:00
Erik Johnston	5bd0a47fcd	pep8	2018-07-18 14:19:00 +01:00
Erik Johnston	e45a46b6e4	Add response code to response timer metrics	2018-07-18 13:59:36 +01:00
Richard van der Hoff	dab00faa83	Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks ... Remove event re-signing hacks	2018-07-18 12:46:27 +01:00
David Baker	c91a44572e	Merge pull request #3514 from matrix-org/dbkr/turn_dont_add_defaults ... Comment dummy TURN parameters in default config	2018-07-18 11:56:18 +01:00
Richard van der Hoff	6e3fc657b4	Resource tracking for background processes ... This introduces a mechanism for tracking resource usage by background processes, along with an example of how it will be used. This will help address #3518, but more importantly will give us better insights into things which are happening but not being shown up by the request metrics. We *could* do this with Measure blocks, but: - I think having them pulled out as a completely separate metric class will make it easier to distinguish top-level processes from those which are nested. - I want to be able to report on in-flight background processes, and I don't think we want to do this for *all* Measure blocks.	2018-07-18 10:50:33 +01:00
Amber Brown	5f3d02f6eb	bump to 0.33.0rc1	2018-07-18 12:52:56 +10:00
Richard van der Hoff	79eb339c66	add a comment	2018-07-17 14:53:34 +01:00
Richard van der Hoff	d897be6a98	Fix visibility of events from erased users over federation	2018-07-17 14:02:07 +01:00
Richard van der Hoff	9c04b4abf9	Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server ... Refactor and optimze filter_events_for_server	2018-07-17 14:01:39 +01:00
Richard van der Hoff	94440ae994	fix imports	2018-07-17 11:51:26 +01:00
Amber Brown	bc006b3c9d	Refactor REST API tests to use explicit reactors (#3351)	2018-07-17 20:43:18 +10:00
Richard van der Hoff	2172a3d8cb	add a comment	2018-07-17 11:13:57 +01:00
Erik Johnston	b2aa05a8d6	Use efficient .intersection	2018-07-17 11:07:04 +01:00
Erik Johnston	547b1355d3	Fix perf regression in PR #3530 ... The get_entities_changed function was changed to return all changed entities since the given stream position, rather than only those changed from a given list of entities. This resulted in the function incorrectly returning large numbers of entities that, for example, caused large increases in database usage.	2018-07-17 10:27:51 +01:00
Amber Brown	3fe0938b76	Merge pull request #3530 from matrix-org/erikj/stream_cache ... Don't return unknown entities in get_entities_changed	2018-07-17 13:44:46 +10:00
Richard van der Hoff	09e29fb58b	Attempt to make _filter_events_for_server more efficient	2018-07-16 14:06:09 +01:00
Krombel	78a9ddcf9a	rerun isort with latest version	2018-07-16 14:23:25 +02:00
Richard van der Hoff	ea69d35651	Move filter_events_for_server out of FederationHandler ... for easier unit testing.	2018-07-16 13:06:24 +01:00
Krombel	4a27000548	check isort by travis	2018-07-16 13:57:33 +02:00
Jeroen	505530f36a	Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests ... # Conflicts: # synapse/crypto/context_factory.py	2018-07-14 20:24:46 +02:00
Amber Brown	8a4f05fefb	Fix develop because I broke it :( (#3535)	2018-07-14 09:51:00 +10:00
Amber Brown	8532953c04	Merge pull request #3534 from krombel/use_parse_and_asserts_from_servlet ... Use parse and asserts from http.servlet	2018-07-14 09:09:19 +10:00
Amber Brown	a2374b2c7f	fix sytests	2018-07-14 07:52:58 +10:00
Amber Brown	33b60c01b5	Make auth & transactions more testable (#3499)	2018-07-14 07:34:49 +10:00
Krombel	516f960ad8	add changelog	2018-07-13 22:19:19 +02:00
Krombel	3366b9c534	rename assert_params_in_request to assert_params_in_dict ... the method "assert_params_in_request" does handle dicts and not requests. A request body has to be parsed to json before this method can be used	2018-07-13 21:53:01 +02:00
Krombel	32fd6910d0	Use parse_{int,str} and assert from http.servlet ... parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable	2018-07-13 21:40:14 +02:00
Richard van der Hoff	2aba1f549c	Merge pull request #3533 from matrix-org/rav/fix_federation_ratelimite_queue ... Make FederationRateLimiter queue requests properly	2018-07-13 16:59:18 +01:00
Richard van der Hoff	33b40d0a25	Make FederationRateLimiter queue requests properly ... popitem removes the *most recent* item by default [1]. We want the oldest. Fixes #3524 [1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem	2018-07-13 16:19:40 +01:00
Erik Johnston	77b692e65d	Don't return unknown entities in get_entities_changed ... The stream cache keeps track of all entities that have changed since a particular stream position, so get_entities_changed does not need to return unknown entites when given a larger stream position. This makes it consistent with the behaviour of has_entity_changed.	2018-07-13 15:26:10 +01:00
Matthew Hodgson	ba22b6a456	typo	2018-07-13 12:03:39 +01:00
Richard van der Hoff	6dff49b8a9	Merge pull request #3521 from matrix-org/rav/optimise_stream_change_cache ... Reduce set building in get_entities_changed	2018-07-12 12:08:49 +01:00
Matthew Hodgson	12ec58301f	shift to using an explicit deleted flag on m.device_list_update EDUs ... and generally make it work.	2018-07-12 11:39:43 +01:00
Richard van der Hoff	fa5c2bc082	Reduce set building in get_entities_changed ... This line shows up as about 5% of cpu time on a synchrotron: not_known_entities = set(entities) - set(self._entity_to_key) Presumably the problem here is that _entity_to_key can be largeish, and building a set for its keys every time this function is called is slow. Here we rewrite the logic to avoid building so many sets.	2018-07-12 11:37:44 +01:00