2311 Commits

Author	SHA1	Message	Date
Richard van der Hoff	e9bfe719ba	Strip overlong OpenGraph data from url preview ... ... to stop people causing DoSes with malicious web pages	2019-11-05 15:51:18 +00:00
Brendan Abolivier	a6863da249	Lint	2019-11-05 14:50:19 +00:00
Brendan Abolivier	8822b33111	Update copyrights	2019-11-05 14:46:35 +00:00
Brendan Abolivier	037360e6cf	Add tests for /search	2019-11-05 14:33:18 +00:00
Brendan Abolivier	c9e4748cb7	Merge labels tests for /context and /messages	2019-11-05 13:47:47 +00:00
Brendan Abolivier	a7c818c79b	Add test case	2019-11-05 13:21:26 +00:00
Erik Johnston	ffe595381d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-11-05 10:27:41 +00:00
Amber Brown	4e1c7b79fa	Remove the psutil dependency (#6318) ... * remove psutil and replace with resource	2019-11-05 05:05:48 +11:00
Brendan Abolivier	09957ce0e4	Implement per-room message retention policies	2019-11-04 17:09:22 +00:00
Erik Johnston	6a0092d371	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-11-04 13:29:35 +00:00
Brendan Abolivier	f496d25877	Merge pull request #6301 from matrix-org/babolivier/msc2326 ... Implement MSC2326 (label based filtering)	2019-11-01 17:04:45 +00:00
Brendan Abolivier	988d8d6507	Incorporate review	2019-11-01 16:22:44 +00:00
Richard van der Hoff	c6516adbe0	Factor out an _AsyncEventContextImpl (#6298) ... The intention here is to make it clearer which fields we can expect to be populated when: notably, that the _event_type etc aren't used for the synchronous impl of EventContext.	2019-11-01 16:19:09 +00:00
Richard van der Hoff	1cb84c6486	Support for routing outbound HTTP requests via a proxy (#6239) ... The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.	2019-11-01 14:07:44 +00:00
Brendan Abolivier	c6dbca2422	Incorporate review	2019-11-01 10:30:51 +00:00
Hubert Chathi	6f4bc6d01d	Merge branch 'develop' into cross-signing_federation	2019-10-31 22:38:21 -04:00
Erik Johnston	cd581338cf	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history	2019-10-31 15:19:26 +00:00
Erik Johnston	dfe0cd71b6	Merge pull request #6294 from matrix-org/erikj/add_state_storage ... Add StateGroupStorage interface	2019-10-31 16:17:53 +01:00
Erik Johnston	97c60ccaa3	Add unit test for /purge_room API	2019-10-31 11:30:25 +00:00
Hubert Chathi	bb6cec27a5	rename get_devices_by_remote to get_device_updates_by_remote	2019-10-30 14:57:34 -04:00
Brendan Abolivier	dcc069a2e2	Lint	2019-10-30 18:01:56 +00:00
Brendan Abolivier	d8c9109aee	Add integration tests for /messages	2019-10-30 17:48:22 +00:00
Brendan Abolivier	fe51d6cacf	Add more integration testing	2019-10-30 17:28:41 +00:00
Brendan Abolivier	395683add1	Add integration tests for sync	2019-10-30 16:47:37 +00:00
Brendan Abolivier	e7943f660a	Add unit tests	2019-10-30 16:15:04 +00:00
Erik Johnston	7c8c97e635	Split purge API into events vs state	2019-10-30 15:23:37 +00:00
Erik Johnston	69f0054ce6	Port to use state storage	2019-10-30 14:46:54 +00:00
Hubert Chathi	7d7eac61be	Merge branch 'develop' into cross-signing_federation	2019-10-30 10:17:10 -04:00
Erik Johnston	ec6de1cc7d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store	2019-10-30 13:37:04 +00:00
Erik Johnston	a8d16f6c00	Review comments	2019-10-30 13:36:12 +00:00
Erik Johnston	326b3dace7	Make ObservableDeferred.observe() always return deferred. ... This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.	2019-10-30 11:35:46 +00:00
Erik Johnston	3f33879be4	Port federation_server to async/await	2019-10-29 14:13:08 +00:00
Erik Johnston	d0d8a22c13	Quick fix to ensure cache descriptors always return deferreds	2019-10-28 13:33:04 +00:00
Hubert Chathi	da78f61778	Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys ... delete keys when deleting backup versions	2019-10-25 11:28:11 -04:00
Hubert Chathi	4697c0de0b	remove unneeded imports	2019-10-25 10:47:02 -04:00
Hubert Chathi	4cf3a30a20	switch to using HomeserverTestCase	2019-10-25 10:42:07 -04:00
Erik Johnston	7e7a1461f6	Fix tests	2019-10-25 10:57:37 +01:00
Hubert Chathi	c40d7244f8	Merge branch 'develop' into cross-signing_federation	2019-10-24 22:31:25 -04:00
Hubert Chathi	29a0bc5637	remove some unnecessary lines	2019-10-24 21:43:02 -04:00
Hubert Chathi	848cd388d9	delete keys when deleting backups	2019-10-24 21:21:51 -04:00
Neil Johnson	2794b79052	Option to suppress resource exceeded alerting (#6173) ... The expected use case is to suppress MAU limiting on small instances	2019-10-24 11:48:46 +01:00
Erik Johnston	3ca4c7c516	Use new EventPersistenceStore	2019-10-23 16:15:03 +01:00
Hubert Chathi	3e3f9b684e	fix unit test	2019-10-22 22:26:30 -04:00
Erik Johnston	c17efdc01c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores	2019-10-22 18:44:29 +01:00
Erik Johnston	336eeea3ff	Fix postgres unit tests to use prepare_database	2019-10-22 11:02:01 +01:00
Erik Johnston	bb6264be0b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores	2019-10-22 10:41:18 +01:00
Erik Johnston	c66a06ac6b	Move storage classes into a main "data store". ... This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.	2019-10-21 16:05:06 +01:00
Hubert Chathi	36adfaedab	Merge branch 'develop' into cross-signing_sig_upload	2019-10-18 18:34:42 +01:00
Hubert Chathi	cfc28325a6	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-10-18 11:39:40 +01:00
Erik Johnston	5859a5c569	Fix presence timeouts when synchrotron restarts. (#6212) ... * Fix presence timeouts when synchrotron restarts. Handling timeouts would fail if there was an external process that had timed out, e.g. a synchrotron restarting. This was due to a couple of variable name typoes. Fixes #3715.	2019-10-18 06:42:26 +01:00
Hubert Chathi	06fc66c81e	Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage ... make storage layer in charge of interpreting the device key data	2019-10-11 15:19:06 -04:00
Hubert Chathi	691dd67fcd	Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version ... make version optional in body of e2e backup version update	2019-10-11 10:11:59 -04:00
Neil Johnson	a0d0ba7862	Fix MAU reaping where reserved users are specified. (#6168)	2019-10-11 09:38:26 +01:00
Hubert Chathi	4908fb3b30	make storage layer in charge of interpreting the device key data	2019-10-10 19:15:30 -04:00
Erik Johnston	bc244627ac	Fix postgres unit tests	2019-10-10 15:37:53 +01:00
Hubert Chathi	5373de6cce	change test name to be unique	2019-10-10 08:54:07 -04:00
Erik Johnston	933034e2fe	Merge pull request #6127 from matrix-org/erikj/patch_inner ... Add more log context checks when patching inlineCallbacks	2019-10-10 13:47:50 +01:00
werner291	b5b03b7079	Add domain validation when creating room with list of invitees (#6121)	2019-10-10 13:05:48 +01:00
krombel	2efd050c9d	send 404 as http-status when filter-id is unknown to the server (#2380) ... This fixed the weirdness of 400 vs 404 as http status code in the case the filter id is not known by the server. As e.g. matrix-js-sdk expects 404 to catch this situation this leads to unwanted behaviour.	2019-10-10 12:59:55 +01:00
Erik Johnston	9970f955ce	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner	2019-10-10 11:51:50 +01:00
Richard van der Hoff	a139420a3c	Fix races in room stats (and other) updates. (#6187) ... Hopefully this will fix the occasional failures we were seeing in the room directory. The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8. We can solve this easily by only processing up to the stream_id where we know all events have been persisted.	2019-10-10 11:29:01 +01:00
Erik Johnston	1d6dd1c294	Move patch_inline_callbacks into synapse/	2019-10-10 10:53:06 +01:00
Amber Brown	f743108a94	Refactor HomeserverConfig so it can be typechecked (#6137)	2019-10-10 09:39:35 +01:00
Hubert Chathi	4535a07f4a	make version optional in body of e2e backup version update ... to agree with latest version of the MSC	2019-10-09 17:54:03 -04:00
Erik Johnston	5c1f886c75	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner	2019-10-09 16:52:21 +01:00
Brendan Abolivier	ae0b78cb1f	Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked ... Don't 500 when trying to exchange a revoked 3PID invite	2019-10-04 12:09:05 +01:00
Brendan Abolivier	8a5e8e829b	Lint (again)	2019-10-03 11:30:43 +01:00
Brendan Abolivier	ebcb6a30d7	Lint	2019-10-03 11:29:07 +01:00
Brendan Abolivier	6527fa18c1	Add test case	2019-10-03 11:24:36 +01:00
Erik Johnston	a5166e4d5f	Land improved room list based on room stats (#6019) ... Use room_stats and room_state for room directory search	2019-10-02 14:08:35 +01:00
Erik Johnston	ecd254bc49	Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop	2019-10-02 11:08:07 +01:00
Erik Johnston	b4fe7e19c0	Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings ... Fix errors storing large retry intervals.	2019-10-02 11:05:22 +01:00
Erik Johnston	f44f1d2e83	Fix errors storing large retry intervals. ... We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).	2019-10-02 10:36:27 +01:00
Erik Johnston	ce7a3e7e27	Fix fetching censored redactions from DB ... Fetching a censored redactions caused an exception due to the code expecting redactions to have a `redact` key, which redacted redactions don't have.	2019-10-02 10:14:01 +01:00
Erik Johnston	6374ca40c2	Update	2019-09-27 15:58:14 +01:00
Brendan Abolivier	fbb8ff3088	ok	2019-09-27 15:23:07 +01:00
Brendan Abolivier	873fe7883c	Lint	2019-09-27 15:21:03 +01:00
Brendan Abolivier	e94ff67903	Add test to validate the change	2019-09-27 15:14:02 +01:00
Erik Johnston	132279a46f	Patch inlinecallbacks for log contexts	2019-09-27 15:11:14 +01:00
Neil Johnson	034db2ba21	Fix dummy event insertion consent bug (#6053) ... Fixes #5905	2019-09-26 11:47:53 +01:00
Erik Johnston	4fb3c129aa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2	2019-09-25 17:53:13 +01:00
Erik Johnston	d2bd0bc6b1	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips	2019-09-25 17:16:28 +01:00
Richard van der Hoff	990928abde	Stop advertising unsupported flows for registration (#6107) ... If email or msisdn verification aren't supported, let's stop advertising them for registration. Fixes #6100.	2019-09-25 12:10:26 +01:00
Richard van der Hoff	2cd98812ba	Refactor the user-interactive auth handling (#6105) ... Pull the checkers out to their own classes, rather than having them lost in a massive 1000-line class which does everything. This is also preparation for some more intelligent advertising of flows, as per #6100	2019-09-25 11:33:03 +01:00
Richard van der Hoff	8004d6ca2f	Refactor code for calculating registration flows (#6106) ... because, frankly, it looked like it was written by an axe-murderer. This should be a non-functional change, except that where `m.login.dummy` was previously advertised *before* `m.login.terms`, it will now be advertised afterwards. AFAICT that should have no effect, and will be more consistent with the flows that involve passing a 3pid.	2019-09-25 11:32:05 +01:00
Erik Johnston	faac453f08	Test that pruning of old user IPs works	2019-09-24 15:53:17 +01:00
Erik Johnston	acb62a7cc6	Test background update	2019-09-23 16:59:45 +01:00
Erik Johnston	51d28272e2	Query devices table for last seen info. ... This is a) simpler than querying user_ips directly and b) means we can purge older entries from user_ips without losing the required info. The storage functions now no longer return the access_token, since it was unused.	2019-09-23 16:59:45 +01:00
Richard van der Hoff	1e19ce00bf	Add 'failure_ts' column to 'destinations' table (#6016) ... Track the time that a server started failing at, for general analysis purposes.	2019-09-17 11:41:54 +01:00
Amber Brown	850dcfd2d3	Fix well-known lookups with the federation certificate whitelist (#5997)	2019-09-14 04:58:38 +10:00
Amber Brown	b617864cd9	Fix for structured logging tests stomping on logs (#6023)	2019-09-13 02:29:55 +10:00
Jason Robinson	6d847d8ce6	Ensure support users can be registered even if MAU limit is reached ... This allows support users to be created even on MAU limits via the admin API. Support users are excluded from MAU after creation, so it makes sense to exclude them in creation - except if the whole host is in disabled state. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-11 20:48:31 +03:00
Erik Johnston	57dd41a45b	Fix comments ... Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2019-09-11 13:54:50 +01:00
Erik Johnston	caa9d6fed7	Add test for admin redaction ratelimiting.	2019-09-11 11:18:04 +01:00
Jason Robinson	63f9317b8e	Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users ... Only count real users when checking for auto-creation of auto-join room	2019-09-09 17:37:52 +03:00
Erik Johnston	470dc621ae	Merge pull request #5934 from matrix-org/erikj/censor_redactions ... Censor redactions in DB after a month	2019-09-09 15:29:39 +01:00
Amber Brown	aeb9b2179e	Add a build info metric to Prometheus (#6005)	2019-09-10 00:14:58 +10:00
Jason Robinson	be618e0551	Only count real users when checking for auto-creation of auto-join room ... Previously if the first registered user was a "support" or "bot" user, when the first real user registers, the auto-join rooms were not created. Fix to exclude non-real (ie users with a special user type) users when counting how many users there are to determine whether we should auto-create a room. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 14:48:08 +03:00
Hubert Chathi	8e86f5b65c	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-09-07 13:20:34 -04:00
Hubert Chathi	0d61d1d735	Merge branch 'develop' into cross-signing_sig_upload	2019-09-07 13:14:45 -04:00
Hubert Chathi	ab729e31cf	use something that's the right type for user_id	2019-09-06 17:52:37 -04:00
Amber Brown	55d5b3af88	Servers-known-about statistic (#5981)	2019-09-07 01:45:51 +10:00
Hubert Chathi	e47af0f086	fix test	2019-09-05 17:03:14 -04:00
Erik Johnston	591d82f06b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions	2019-09-05 17:27:46 +01:00
Erik Johnston	ad9b64b496	Fix test	2019-09-05 17:17:47 +01:00
Erik Johnston	3ff0422d2d	Make redaction retention period configurable	2019-09-05 17:16:45 +01:00
Hubert Chathi	5914fd09c7	add test	2019-09-04 22:05:02 -04:00
Hubert Chathi	9061b4198a	make isort happy	2019-09-04 22:04:41 -04:00
Hubert Chathi	7d6c70fc7a	make black happy	2019-09-04 22:04:12 -04:00
Hubert Chathi	ac4746ac4b	allow uploading signatures of master key signed by devices	2019-09-04 22:03:41 -04:00
Hubert Chathi	faf72a4c40	Merge branch 'develop' into cross-signing_keys	2019-09-04 19:12:29 -04:00
Erik Johnston	6e834e94fc	Fix and refactor room and user stats (#5971) ... Previously the stats were not being correctly populated.	2019-09-04 13:04:27 +01:00
Andrew Morgan	4548d1f87e	Remove unnecessary parentheses around return statements (#5931) ... Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)	2019-08-30 16:28:26 +01:00
Erik Johnston	a4bf72c30c	Censor redactions in DB after a month	2019-08-30 11:29:17 +01:00
Hubert Chathi	e3d3fbf63f	Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys	2019-08-28 17:36:46 -07:00
Will Hunt	c8fa620d7a	Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent ... Exempt support users from consent	2019-08-28 16:31:40 +01:00
Jorik Schellekens	6d97843793	Config templating (#5900) ... Template config files * Imagine a system composed entirely of x, y, z etc and the basic operations.. Wait George, why XOR? Why not just neq? George: Eh, I didn't think of that.. Co-Authored-By: Erik Johnston <erik@matrix.org>	2019-08-28 13:12:22 +01:00
Amber Brown	7dc398586c	Implement a structured logging output system. (#5680)	2019-08-28 21:18:53 +10:00
Half-Shot	0fb5189072	Fix registration test	2019-08-23 09:25:35 +01:00
Erik Johnston	1e4b4d85e7	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reliable_lookups	2019-08-22 13:41:57 +01:00
Erik Johnston	1b09cf8658	Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_fail ... Retry well known on fail	2019-08-22 13:17:05 +01:00
Richard van der Hoff	5019945828	Refactor the Appservice scheduler code ... Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code (it seemed to be previously inexplicably split between `ApplicationServiceScheduler.start` and `_Recoverer.start`). Add some docstrings too.	2019-08-20 11:50:23 +01:00
Erik Johnston	7777d353bf	Remove test debugs	2019-08-20 11:46:59 +01:00
Erik Johnston	1dec31560e	Change jitter to be a factor rather than absolute value	2019-08-20 11:46:00 +01:00
Erik Johnston	f299c5414c	Refactor MatrixFederationAgent to retry SRV. ... This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool	2019-08-15 15:43:22 +01:00
Erik Johnston	e6e136decc	Retry well known on fail. ... If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.	2019-08-15 09:28:58 +01:00
Erik Johnston	17e1e80726	Retry well-known lookup before expiry. ... This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.	2019-08-13 16:20:38 +01:00
Erik Johnston	e9906b0772	Merge pull request #5836 from matrix-org/erikj/lower_bound_ttl_well_known ... Add a lower bound to well-known TTL.	2019-08-13 12:41:16 +01:00
Richard van der Hoff	fb86217553	Merge pull request #5788 from matrix-org/rav/metaredactions ... Fix handling of redactions of redactions	2019-08-12 12:25:19 +01:00
Erik Johnston	107ad133fc	Move well known lookup into a separate clas	2019-08-07 15:36:38 +01:00
Erik Johnston	af9f1c0764	Add a lower bound for TTL on well known results. ... It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".	2019-08-06 17:01:23 +01:00
Hubert Chathi	8c9adcc95d	fix formatting	2019-08-01 22:09:05 -04:00
Hubert Chathi	336c546d6a	Merge branch 'cross-signing_hidden' into cross-signing_keys	2019-08-01 16:31:40 -04:00
Brendan Abolivier	8ed9e63432	Account validity: allow defining HTML templates to serve the us… (#5807) ... Account validity: allow defining HTML templates to serve the user on account renewal attempt	2019-08-01 16:09:25 +02:00
Brendan Abolivier	bc35503528	Add tests	2019-08-01 12:00:08 +02:00
Erik Johnston	8fde611a8c	Merge pull request #5794 from matrix-org/erikj/share_ssl_options_for_well_known ... Share SSL options for well-known requests	2019-07-31 11:40:02 +01:00
Amber Brown	8f15832950	Remove DelayedCall debugging from test runs (#5787)	2019-07-31 20:39:22 +10:00
Erik Johnston	a9bcae9f50	Share SSL options for well-known requests	2019-07-31 10:39:24 +01:00
Erik Johnston	63eb4a1b62	Merge pull request #5746 from matrix-org/erikj/test_bg_update_currnet_state ... Add unit test for current state membership bg update	2019-07-30 10:00:02 +01:00
Richard van der Hoff	8c97f6414c	Remove non-functional 'expire_access_token' setting (#5782) ... The `expire_access_token` didn't do what it sounded like it should do. What it actually did was make Synapse enforce the 'time' caveat on macaroons used as access tokens, but since our access token macaroons never contained such a caveat, it was always a no-op. (The code to add 'time' caveats was removed back in v0.18.5, in #1656)	2019-07-30 08:25:02 +01:00
Richard van der Hoff	4e97eb89e5	Handle loops in redaction events	2019-07-30 00:00:34 +01:00
Amber Brown	865077f1d1	Room Complexity Client Implementation (#5783)	2019-07-30 02:47:27 +10:00
Richard van der Hoff	1cad8d7b6f	Convert RedactionTestCase to modern test style (#5768)	2019-07-26 07:38:55 +01:00
Hubert Chathi	c659b9f94f	allow uploading keys for cross-signing	2019-07-25 11:08:24 -04:00
Richard van der Hoff	618bd1ee76	Fix some error cases in the caching layer. (#5749) ... There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.	2019-07-25 15:59:45 +01:00
Andrew Morgan	baf081cd3b	Bugfixes ... -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734)) -----BEGIN PGP SIGNATURE----- iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl04Ur0THGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9F4oD/0TY6S/SEd2uAmzor64ojmbX5BOwPzf j/wzUTrfvuf40EvkNPDpnejNZSvy/ysbaGQaQusv0SQKlV3xrvdn4RuMvnOWVWck kBsO+lvzOaUTR0KHDxN4y9F5eI2NdPbub4847PPVzyqSIHAd+kolxXS8kSBBhwpL yfaICWV/AOy5L7xN+JZ9IQpnegVAvUj5DmgXzDHd6VdeiHDVJuARaBgrR5uCkwVS ZoLRqZ95XV/qiguMAUvPOwyEqht2mwO64989MswP16YYm8oMkB5QA6I5nYnACsTP qk9YcN/oNvEfQXUhttku6MxK1/4yUMPUhEoDBDH7ebc0440QDtWN+IHTdA6oPVZB IuStL9YGY16m7Ltx37ZUA4URfNMiSeLHo3zKc/mCAcwxN4HyOjJewtxbG5zKQAOZ SMs8UcDwGR4zL1hnt8ZDNYtWwfzJBQIdGjoHvjXJEY7/1csTv2lmAwewFTXiqSAr 30GW5ews94kotqBK53zZT6V0F5gHNqgGHniOz1ZpqLLxYLqO3LSAGe97CrqlWUdX GkhA9tZyweknociD9fyyBmKdcFJ4mL4a+oGI5CMnSMph8UvCY8Y5XMb1T+iYEABI tA9G3mBvgkLPj+5V+8QggNkBafSigW2Q4FX7enGsDmiiskZOtfeKrAcVkapD4ooi 3I7IW5aetZr2IQ== =+JBn -----END PGP SIGNATURE----- Merge tag 'v1.2.0rc2' into develop Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))	2019-07-24 13:47:51 +01:00
Jorik Schellekens	cf2972c818	Fix servlet metric names (#5734) ... * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths	2019-07-24 13:07:35 +01:00
Erik Johnston	73bbaf2bc6	Add unit test for current state membership bg update	2019-07-23 17:00:22 +01:00
Amber Brown	4806651744	Replace returnValue with return (#5736)	2019-07-23 23:00:55 +10:00
Richard van der Hoff	c7095be913	Refactor Keyring._start_key_lookups ... There's an awful lot of deferreds and dictionaries flying around here. The whole thing can be made much simpler and achieve the same effect.	2019-07-19 17:49:19 +01:00
Amber Brown	7ad1d76356	Support Prometheus_client 0.4.0+ (#5636)	2019-07-18 23:57:15 +10:00
Andrew Morgan	b2a382efdb	Remove the ability to query relations when the original event was redacted. (#5629) ... Fixes #5594 Forbid viewing relations on an event once it has been redacted.	2019-07-18 14:41:42 +01:00
Richard van der Hoff	9c70a02a9c	Ignore redactions of m.room.create events (#5701)	2019-07-17 19:08:02 +01:00
Richard van der Hoff	2091c91fde	More refactoring in get_events_as_list (#5707) ... We can now use `_get_events_from_cache_or_db` rather than going right back to the database, which means that (a) we can benefit from caching, and (b) it opens the way forward to more extensive checks on the original event. We now always require the original event to exist before we will serve up a redaction.	2019-07-17 17:34:13 +01:00
Erik Johnston	d86321300a	Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data ... Add basic function to get all data for a user out of synapse	2019-07-15 10:04:02 +01:00
Richard van der Hoff	5f158ec039	Implement access token expiry (#5660) ... Record how long an access token is valid for, and raise a soft-logout once it expires.	2019-07-12 17:26:02 +01:00
Richard van der Hoff	6bb0357c94	Add a mechanism for per-test configs (#5657) ... It's useful to be able to tweak the homeserver config to be used for each test. This PR adds a mechanism to do so.	2019-07-12 10:16:23 +01:00
Richard van der Hoff	0a4001eba1	Clean up exception handling for access_tokens (#5656) ... First of all, let's get rid of `TOKEN_NOT_FOUND_HTTP_STATUS`. It was a hack we did at one point when it was possible to return either a 403 or a 401 if the creds were missing. We always return a 401 in these cases now (thankfully), so it's not needed. Let's also stop abusing `AuthError` for these cases. Honestly they have nothing that relates them to the other places that `AuthError` is used, other than the fact that they are loosely under the 'Auth' banner. It makes no sense for them to share exception classes. Instead, let's add a couple of new exception classes: `InvalidClientTokenError` and `MissingClientTokenError`, for the `M_UNKNOWN_TOKEN` and `M_MISSING_TOKEN` cases respectively - and an `InvalidClientCredentialsError` base class for the two of them.	2019-07-11 11:06:23 +01:00
Richard van der Hoff	1890cfcf82	Inline issue_access_token (#5659) ... this is only used in one place, so it's clearer if we inline it and reduce the API surface. Also, fixes a buglet where we would create an access token even if we were about to block the user (we would never return the AT, so the user could never use it, but it was still created and added to the db.)	2019-07-11 04:10:07 +10:00
Richard van der Hoff	953dbb7980	Remove access-token support from RegistrationStore.register (#5642) ... The 'token' param is no longer used anywhere except the tests, so let's kill that off too.	2019-07-10 16:26:49 +01:00
Andrew Morgan	d88421ab03	Include the original event in /relations (#5626) ... When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.	2019-07-09 13:43:08 +01:00
Brendan Abolivier	af67c7c1de	Merge pull request #5644 from matrix-org/babolivier/profile-allow-self ... Allow newly-registered users to lookup their own profiles	2019-07-09 10:25:40 +01:00
Richard van der Hoff	824707383b	Remove access-token support from RegistrationHandler.register (#5641) ... Nothing uses this now, so we can remove the dead code, and clean up the API. Since we're changing the shape of the return value anyway, we take the opportunity to give the method a better name.	2019-07-08 19:01:08 +01:00
Brendan Abolivier	73cb716b3c	Lint	2019-07-08 17:44:20 +01:00
Brendan Abolivier	5e01e9ac19	Add test case	2019-07-08 17:41:16 +01:00
Richard van der Hoff	1af2fcd492	Move get_or_create_user to test code (#5628) ... This is only used in tests, so...	2019-07-08 23:52:26 +10:00
Andrew Morgan	ad8b909ce9	Add origin_server_ts and sender fields to m.replace (#5613) ... Riot team would like some extra fields as part of m.replace, so here you go. Fixes: #5598	2019-07-05 17:20:02 +01:00
Erik Johnston	7f0d8e4288	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data	2019-07-05 14:08:21 +01:00
Richard van der Hoff	9481707a52	Fixes to the federation rate limiter (#5621) ... - Put the default window_size back to 1000ms (broken by #5181) - Make the `rc_federation` config actually do something - fix an off-by-one error in the 'concurrent' limit - Avoid creating an unused `_PerHostRatelimiter` object for every single incoming request	2019-07-05 11:10:19 +01:00
Erik Johnston	c061d4f237	Fixup from review comments.	2019-07-04 11:41:06 +01:00
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606)	2019-07-04 00:07:04 +10:00
Amber Brown	0ee9076ffe	Fix media repo breaking (#5593)	2019-07-02 19:01:28 +01:00
Erik Johnston	8ee69f299c	Add basic function to get all data for a user out of synapse	2019-07-02 12:09:04 +01:00
Amber Brown	f40a7dc41f	Make the http server handle coroutine-making REST servlets (#5475)	2019-06-29 17:06:55 +10:00
Amber Brown	be3b901ccd	Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550)	2019-06-28 18:19:09 +10:00
Brendan Abolivier	bfe84e051e	Split public rooms directory auth config in two	2019-06-24 15:42:31 +01:00
Richard van der Hoff	c3c6b00d95	Pass config_dir_path and data_dir_path into Config.read_config. (#5522) ... * Pull config_dir_path and data_dir_path calculation out of read_config_files * Pass config_dir_path and data_dir_path into read_config	2019-06-24 11:34:45 +01:00
Amber Brown	32e7c9e7f2	Run Black. (#5482)	2019-06-20 19:32:02 +10:00
Erik Johnston	554609288b	Run as background process and fix comments	2019-06-19 11:33:03 +01:00
Erik Johnston	b42f90470f	Add experimental option to reduce extremities. ... Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.	2019-06-18 15:02:18 +01:00
Erik Johnston	dd927b29e1	Merge pull request #5388 from matrix-org/erikj/fix_email_push ... Fix email notifications for unnamed rooms with multiple people	2019-06-17 13:54:35 +01:00
Brendan Abolivier	f12e1f029c	Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules ... Allow server admins to define implementations of extra rules for allowing or denying incoming events	2019-06-14 19:37:59 +01:00
Erik Johnston	9ca4ae7131	Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive ... Prometheus histograms are cumalative	2019-06-14 18:21:42 +01:00
Brendan Abolivier	f874b16b2e	Add plugin APIs for implementations of custom event rules.	2019-06-14 18:16:03 +01:00
Brendan Abolivier	6d56a694f4	Don't send renewal emails to deactivated users	2019-06-14 15:05:56 +01:00
Erik Johnston	3ed595e327	Prometheus histograms are cumalative	2019-06-14 14:07:32 +01:00
Brendan Abolivier	d0530382ee	Track deactivated accounts in the database (#5378)	2019-06-14 13:18:24 +01:00
Amber H. Brown	a10c8dae85	fix prometheus rendering error	2019-06-14 21:09:33 +10:00
Amber Brown	6312d6cc7c	Expose statistics on extrems to prometheus (#5384)	2019-06-13 22:40:52 +10:00
Erik Johnston	7e68691ce9	Merge branch 'master' of github.com:matrix-org/synapse into develop	2019-06-11 17:25:16 +01:00
Erik Johnston	a766c41d25	Bump bleach version so that tests can run on old deps.	2019-06-11 12:34:18 +01:00
Erik Johnston	10383e6e6f	Change password reset links to /_matrix.	2019-06-11 11:34:33 +01:00
Erik Johnston	abce00fc6a	Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop	2019-06-10 18:33:55 +01:00
Erik Johnston	a6b1817940	Merge pull request #5417 from matrix-org/rav/shared_ssl_context ... Share an SSL context object between SSL connections	2019-06-10 18:16:12 +01:00
Richard van der Hoff	db74c4fc6c	fix ci on py2, again	2019-06-10 17:55:01 +01:00
Richard van der Hoff	19780a521e	fix CI on python 2.7	2019-06-10 17:41:10 +01:00
Richard van der Hoff	e016681221	Tests for SSL certs for federation connections ... Add some tests for bad certificates for federation and .well-known connections	2019-06-10 16:14:20 +01:00
Erik Johnston	0382b0ffee	Merge pull request #5415 from matrix-org/erikj/fix_null_valid_until_ms ... Fix key verification when key stored with null valid_until_ms	2019-06-10 15:56:36 +01:00
Erik Johnston	43badd2cd4	Fix key verification when key stored with null valid_until_ms ... Some keys are stored in the synapse database with a null valid_until_ms which caused an exception to be thrown when using that key. We fix this by treating nulls as zeroes, i.e. they keys will match verification requests with a minimum_valid_until_ms of zero (i.e. don't validate ts) but will not match requests with a non-zero minimum_valid_until_ms. Fixes #5391.	2019-06-10 15:37:10 +01:00
Brendan Abolivier	26b62796c2	Merge pull request #5363 from matrix-org/babolivier/account_validity_send_mail_auth ... Don't check whether the user's account is expired on /send_mail requests	2019-06-10 11:57:02 +01:00
Erik Johnston	2ebeda48b2	Add test	2019-06-07 12:15:33 +01:00
Erik Johnston	8182a1cfb5	Refactor email tests	2019-06-07 12:15:33 +01:00
Andrew Morgan	2d1d7b7e6f	Prevent multiple device list updates from breaking a batch send (#5156) ... fixes #5153	2019-06-06 23:54:00 +01:00
Neil Johnson	a11865016e	Set default room version to v4. (#5379) ... Set default room version to v4.	2019-06-06 20:13:47 +01:00
Andrew Morgan	3719680ee4	Add ability to perform password reset via email without trusting the identity server (#5377) ... Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368	2019-06-06 17:34:07 +01:00
Richard van der Hoff	9fbb20a531	Stop hardcoding trust of old matrix.org key (#5374) ... There are a few changes going on here: * We make checking the signature on a key server response optional: if no verify_keys are specified, we trust to TLS to validate the connection. * We change the default config so that it does not require responses to be signed by the old key. * We replace the old 'perspectives' config with 'trusted_key_servers', which is also formatted slightly differently. * We emit a warning to the logs every time we trust a key server response signed by the old key.	2019-06-06 17:33:11 +01:00
Richard van der Hoff	cb3b381fcb	Merge pull request #5359 from matrix-org/rav/enable_tls_verification ... Validate federation server TLS certificates by default.	2019-06-06 10:50:42 +01:00
Richard van der Hoff	2eb47e5ee7	Merge pull request #5353 from matrix-org/rav/verify_key_logging ... Associate a request_name with each verify request, for logging	2019-06-06 09:33:09 +01:00
Brendan Abolivier	fe13bd52ac	Don't check whether the user's account is expired on /send_mail requests	2019-06-05 16:35:05 +01:00
Richard van der Hoff	7603a706eb	Merge branch 'rav/fix_custom_ca' into rav/enable_tls_verification	2019-06-05 16:32:35 +01:00
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352) ... * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	2019-06-06 00:45:46 +10:00
Richard van der Hoff	e2dfb922e1	Validate federation server TLS certificates by default.	2019-06-05 14:17:50 +01:00
Richard van der Hoff	cea9750d11	Associate a request_name with each verify request, for logging ... Also: * rename VerifyKeyRequest->VerifyJsonRequest * calculate key_ids on VerifyJsonRequest construction * refactor things to pass around VerifyJsonRequests instead of 4-tuples	2019-06-05 10:46:26 +01:00
Richard van der Hoff	def5ea4062	Don't bomb out on direct key fetches as soon as one fails	2019-06-04 00:16:56 +01:00
Richard van der Hoff	fec2dcb1a5	Enforce validity period on server_keys for fed requests. (#5321) ... When handling incoming federation requests, make sure that we have an up-to-date copy of the signing key. We do not yet enforce the validity period for event signatures.	2019-06-03 22:59:51 +01:00
Amber Brown	2889b05554	Unify v1 and v2 REST client APIs (#5226)	2019-06-03 21:28:59 +10:00
Richard van der Hoff	d828d1dc57	Merge pull request #5309 from matrix-org/rav/limit_displayname_length ... Limit displaynames and avatar URLs	2019-06-01 11:34:50 +01:00
Richard van der Hoff	93003aa172	add some tests	2019-06-01 11:14:37 +01:00
Erik Johnston	58cce39f3a	Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta ... Allow configuring a range for the account validity startup job	2019-05-31 12:11:56 +01:00
Brendan Abolivier	4d794dae21	Move delta from +10% to -10%	2019-05-31 11:09:39 +01:00
Brendan Abolivier	847b9dcd1c	Make max_delta equal to period * 10%	2019-05-31 09:54:46 +01:00
Erik Johnston	e9e5d3392d	Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities ... Add DB bg update to cleanup extremities.	2019-05-30 14:28:26 +01:00
Amber Brown	46c8f7a517	Implement the SHHS complexity API (#5216)	2019-05-30 01:47:16 +10:00
Erik Johnston	d7add713a8	Add test	2019-05-29 15:59:06 +01:00
Richard van der Hoff	540f40f0cd	Merge pull request #5251 from matrix-org/rav/server_keys/01-check_sig ... Ensure that server_keys fetched via a notary server are correctly signed.	2019-05-28 21:32:17 +01:00
Brendan Abolivier	7e1c7cc274	Typo	2019-05-28 17:13:26 +01:00
Brendan Abolivier	52839886d6	Allow configuring a range for the account validity startup job ... When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal. In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.	2019-05-28 16:52:45 +01:00
Richard van der Hoff	fa1b293da2	Simplification to Keyring.wait_for_previous_lookups. (#5250) ... The list of server names was redundant, since it was equivalent to the keys on the server_to_deferred map. This reduces the number of large lists being passed around, and has the benefit of deduplicating the entries in `wait_on`.	2019-05-24 22:17:18 +01:00
Erik Johnston	8c41c04ee4	Merge pull request #5244 from matrix-org/rav/server_keys/00-factor-out-fetchers ... Factor out KeyFetchers from KeyRing	2019-05-23 16:28:51 +01:00
Richard van der Hoff	753b1270da	Require sig from origin server on perspectives responses	2019-05-23 15:01:09 +01:00
Andrew Morgan	6368150a74	Add config option for setting homeserver's default room version (#5223) ... Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present. That hardcoded value is now located in the server.py config file.	2019-05-23 15:00:20 +01:00
Richard van der Hoff	895b79ac2e	Factor out KeyFetchers from KeyRing ... Rather than have three methods which have to have the same interface, factor out a separate interface which is provided by three implementations. I find it easier to grok the code this way.	2019-05-23 13:46:47 +01:00
Richard van der Hoff	b75537beaf	Store key validity time in the storage layer ... This is a first step to checking that the key is valid at the required moment. The idea here is that, rather than passing VerifyKey objects in and out of the storage layer, we instead pass FetchKeyResult objects, which simply wrap the VerifyKey and add a valid_until_ts field.	2019-05-23 11:52:22 +01:00
Richard van der Hoff	2e052110ee	Rewrite store_server_verify_key to store several keys at once (#5234) ... Storing server keys hammered the database a bit. This replaces the implementation which stored a single key, with one which can do many updates at once.	2019-05-23 11:45:39 +01:00
Amber Brown	4a30e4acb4	Room Statistics (#4338)	2019-05-21 11:36:50 -05:00
Erik Johnston	9526aa96a6	Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions ... Block attempts to annotate the same event twice	2019-05-21 17:08:14 +01:00
Brendan Abolivier	6a5a70edf0	Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date ... Add startup background job for account validity	2019-05-21 14:55:15 +01:00
Erik Johnston	c7ec06e8a6	Block attempts to annotate the same event twice	2019-05-20 17:39:05 +01:00
Erik Johnston	1dff859d6a	Rename relation types to match MSC	2019-05-20 14:31:19 +01:00
Erik Johnston	57ba3451b6	Merge pull request #5209 from matrix-org/erikj/reactions_base ... Land basic reaction and edit support.	2019-05-20 14:06:40 +01:00
Erik Johnston	2ac9c965dd	Fixup comments	2019-05-20 12:32:26 +01:00
Erik Johnston	935af0da38	Correctly update aggregation counts after redaction	2019-05-20 12:09:27 +01:00
ReidAnderson	3787133c9e	Limit UserIds to a length that fits in a state key (#5198)	2019-05-20 11:20:08 +01:00
Brendan Abolivier	ad5b4074e1	Add startup background job for account validity ... If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.	2019-05-17 19:37:31 +01:00
Erik Johnston	d4ca533d70	Make tests use different user for each reaction it sends ... As users aren't allowed to react with the same emoji more than once.	2019-05-17 18:03:05 +01:00
Brendan Abolivier	fdeac1e984	Merge pull request #5196 from matrix-org/babolivier/per_room_profiles ... Add an option to disable per-room profiles	2019-05-17 12:10:49 +01:00
David Baker	07cff7b121	Merge pull request #5174 from matrix-org/dbkr/add_dummy_flow_to_recaptcha_only ... Re-order registration stages to do msisdn & email auth last	2019-05-16 17:27:39 +01:00
Erik Johnston	d46aab3fa8	Add basic editing support	2019-05-16 16:54:45 +01:00
Brendan Abolivier	8f9ce1a8a2	Lint	2019-05-16 15:25:54 +01:00
Brendan Abolivier	cc8c139a39	Lint	2019-05-16 15:20:59 +01:00
Brendan Abolivier	efdc55db75	Forgot copyright	2019-05-16 15:10:24 +01:00
Brendan Abolivier	54a582ed44	Add test case	2019-05-16 15:09:16 +01:00
Erik Johnston	95f3fcda3c	Check that event is visible in new APIs	2019-05-16 14:19:06 +01:00
Erik Johnston	a0603523d2	Add aggregations API	2019-05-16 09:37:20 +01:00
Amber Brown	f1e5b41388	Make all the rate limiting options more consistent (#5181)	2019-05-15 12:06:04 -05:00
Erik Johnston	b50641e357	Add simple pagination API	2019-05-15 13:36:51 +01:00
Erik Johnston	efe3c7977a	Add simple send_relation API and track in DB	2019-05-15 13:36:51 +01:00
Erik Johnston	d94544051b	Merge pull request #5184 from matrix-org/erikj/expose_get_events_as_array ... Expose DataStore._get_events as get_events_as_list	2019-05-15 10:17:38 +01:00
Erik Johnston	495e859e58	Merge branch 'erikj/fix_config_ratelimiting' into erikj/test	2019-05-14 14:42:47 +01:00
Erik Johnston	dc4f6d1b01	Use correct config option for ratelimiting in tests	2019-05-14 14:37:40 +01:00
Erik Johnston	4fb44fb5b9	Expose DataStore._get_events as get_events_as_list ... This is in preparation for reaction work which requires it.	2019-05-14 13:37:44 +01:00
Amber Brown	df2ebd75d3	Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171)	2019-05-13 15:01:14 -05:00
Andrew Morgan	5a4b328f52	Add ability to blacklist ip ranges for federation traffic (#5043)	2019-05-13 19:05:06 +01:00
David Baker	822072b1bb	Terms might not be the last stage	2019-05-13 16:10:26 +01:00
David Baker	9e99143c47	Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only	2019-05-13 15:37:03 +01:00
David Baker	8782bfb783	And now I realise why the test is failing...	2019-05-13 15:34:11 +01:00
Andrew Morgan	2f48c4e1ae	URL preview blacklisting fixes (#5155) ... Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.	2019-05-10 10:32:44 -07:00
Amber Brown	b36c82576e	Run Black on the tests again (#5170)	2019-05-10 00:12:11 -05:00
Brendan Abolivier	d216a36b37	Fix bogus imports in tests (#5154)	2019-05-08 21:57:03 +01:00
Matthew Hodgson	c0e0740bef	add options to require an access_token to GET /profile and /publicRooms on CS API (#5083) ... This commit adds two config options: * `restrict_public_rooms_to_local_users` Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API. * `require_auth_for_profile_requests` When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301. MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though. Groups have been intentionally omitted from this commit.	2019-05-08 18:26:56 +01:00
Brendan Abolivier	1473058b5e	Do checks on aliases for incoming m.room.aliases events (#5128) ... Follow-up to #5124 Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.	2019-05-08 17:01:30 +01:00
Richard van der Hoff	59e2d2694d	Remove the requirement to authenticate for /admin/server_version. (#5122) ... This endpoint isn't much use for its intended purpose if you first need to get yourself an admin's auth token. I've restricted it to the `/_synapse/admin` path to make it a bit easier to lock down for those concerned about exposing this information. I don't imagine anyone is using it in anger currently.	2019-05-07 09:29:30 +01:00
Richard van der Hoff	12f9d51e82	Add admin api for sending server_notices (#5121)	2019-05-02 11:59:16 +01:00
Richard van der Hoff	40e576e29c	Move admin api impl to its own package ... It doesn't really belong under rest/client/v1 any more.	2019-05-01 15:44:30 +01:00
Andrew Morgan	6824ddd93d	Config option for verifying federation certificates (MSC 1711) (#4967)	2019-04-25 14:22:49 +01:00
Erik Johnston	ca90336a69	Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration	2019-04-17 19:44:40 +01:00
Brendan Abolivier	eaf41a943b	Add management endpoints for account validity	2019-04-17 19:34:45 +01:00
Brendan Abolivier	91934025b9	Merge pull request #5047 from matrix-org/babolivier/account_expiration ... Send out emails with links to extend an account's validity period	2019-04-17 14:57:39 +01:00
Brendan Abolivier	20f0617e87	Send out emails with links to extend an account's validity period	2019-04-17 14:42:20 +01:00
Andrew Morgan	caa76e6021	Remove periods from copyright headers (#5046)	2019-04-11 17:08:13 +01:00
Brendan Abolivier	bfc8fdf1fc	Merge pull request #5027 from matrix-org/babolivier/account_expiration ... Add time-based account expiration	2019-04-09 17:02:41 +01:00
Brendan Abolivier	747aa9f8ca	Add account expiration feature	2019-04-09 16:46:04 +01:00
Richard van der Hoff	18b69be00f	Rewrite Datastore.get_server_verify_keys ... Rewrite this so that it doesn't hammer the database.	2019-04-09 00:00:10 +01:00
Richard van der Hoff	0084309cd2	Rewrite test_keys as a HomeserverTestCase	2019-04-08 23:59:57 +01:00
Richard van der Hoff	7d2a0c848e	Fix from_server buglet in get_keys_from_perspectives ... make sure we store the name of the server the keys came from, rather than the origin server, after doing a fetch-from-perspectives.	2019-04-08 12:51:16 +01:00
Richard van der Hoff	7fc1e17f4c	Merge pull request #5001 from matrix-org/rav/keyring_cleanups ... Cleanups in the Keyring	2019-04-08 12:47:09 +01:00
Richard van der Hoff	b2d574f126	test for get_keys_from_perspectives	2019-04-04 19:12:54 +01:00
Richard van der Hoff	30805237fa	add a test for get_keys_from_server	2019-04-04 19:12:54 +01:00
Brendan Abolivier	8e85493b0c	Add config option to block users from looking up 3PIDs (#5010)	2019-04-04 17:25:47 +01:00
Erik Johnston	c6a233a936	Add unit test for deleting groups	2019-04-03 16:29:52 +01:00
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998)	2019-04-03 14:32:20 +01:00
Richard van der Hoff	e4d473d855	Rewrite KeyringTestCase as a HomeServerTestCase (#4986) ... This is a bit fiddly due to the keyring doing weird things with logcontexts.	2019-04-03 14:11:27 +01:00
Neil Johnson	e8419554ff	Remove presence lists (#4989) ... Remove presence list support as per MSC 1819	2019-04-03 11:11:15 +01:00
Richard van der Hoff	297bf2547e	Fix sync bug when accepting invites (#4956) ... Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.	2019-04-02 12:42:39 +01:00
Neil Johnson	4c552ed78a	Neilj/fix threepid auth check (with tests) (#4474) ... test threepid checking	2019-04-01 17:42:18 +01:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969) ... Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Erik Johnston	248014379e	Merge pull request #4942 from matrix-org/erikj/fix_presence ... Use event streams to calculate presence	2019-03-28 14:38:31 +00:00
Erik Johnston	40e56997bc	Review comments	2019-03-28 13:48:41 +00:00
Richard van der Hoff	a5798de067	Move replication.tcp.streams into a package	2019-03-27 21:13:14 +00:00
Erik Johnston	b7fa834c40	Add unit tests	2019-03-27 13:41:36 +00:00
Andrew Morgan	4a125be138	Make federation endpoints more tolerant of trailing slashes v2 (#4935) ... Redo of https://github.com/matrix-org/synapse/pull/4840	2019-03-26 11:35:29 +00:00
Richard van der Hoff	9bde730ef8	Fix bug where read-receipts lost their timestamps (#4927) ... Make sure that they are sent correctly over the replication stream. Fixes: #4898	2019-03-25 16:38:05 +00:00
Richard van der Hoff	a54a44734f	Use an explicit dbname for postgres connections in the tests. (#4928) ... I don't have a database with the same name as my user, so leaving the database name unset fails. While we're at it, clear out some unused stuff in the test setup.	2019-03-25 16:36:56 +00:00
Erik Johnston	5fee9d8067	Merge pull request #4869 from matrix-org/erikj/yaml_load ... Fix yaml warnings by using safe_load	2019-03-22 11:58:13 +00:00
Erik Johnston	3677548a82	Use yaml safe_load	2019-03-22 10:20:17 +00:00
Amber Brown	a68e00fca8	Some more porting to HomeserverTestCase and remove old RESTHelper (#4913)	2019-03-22 02:10:21 +11:00
Andrew Morgan	7bef97dfb7	Remove trailing slashes from outbound federation requests and retry on 400 (#4840) ... As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.	2019-03-21 15:07:28 +00:00
Erik Johnston	01e6b405be	Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms ... Deny peeking into rooms that have been blocked	2019-03-21 14:07:17 +00:00
Erik Johnston	d3f640f0ac	isort	2019-03-21 11:29:48 +00:00
Erik Johnston	3959858eaa	Merge pull request #4904 from matrix-org/erikj/fix_shutdown ... Fixup shutdown room API	2019-03-21 11:24:42 +00:00
Erik Johnston	cd80cbffea	Fix typo and add description	2019-03-21 11:24:04 +00:00
Erik Johnston	536a266520	Deny peeking into rooms that have been blocked	2019-03-21 11:20:13 +00:00
Erik Johnston	4a8a1ac962	Rejig testcase to make it more extensible	2019-03-21 11:02:11 +00:00
Erik Johnston	9c9e618b93	Remove debug	2019-03-21 10:58:56 +00:00
Erik Johnston	5c6f61f81c	Add tests	2019-03-21 10:51:21 +00:00
Erik Johnston	09f991a63d	Merge pull request #4896 from matrix-org/erikj/disable_room_directory ... Add option to disable search room lists	2019-03-21 10:16:54 +00:00
Richard van der Hoff	a6f2d3053d	Log requests which are simulated by the unit tests. (#4905) ... Rather than stubbing out the access_log, make it actually log the requests, which makes it a lot more obvious what is going on during tests.	2019-03-20 18:00:02 +00:00
Erik Johnston	263f2c9ce1	Merge pull request #4895 from matrix-org/erikj/disable_user_search ... Add option to disable searching in the user dir	2019-03-20 16:47:15 +00:00
Richard van der Hoff	a902d13180	Batch up outgoing read-receipts to reduce federation traffic. (#4890) ... Rate-limit outgoing read-receipts as per #4730.	2019-03-20 16:02:25 +00:00
Erik Johnston	3660d24ebe	Add test	2019-03-20 15:16:36 +00:00
Erik Johnston	cc09685830	Add test	2019-03-20 14:53:44 +00:00
Andrew Morgan	bb52a2e653	lint	2019-03-20 14:08:57 +00:00
Andrew Morgan	cd36a1283b	New test, fix issues	2019-03-20 14:00:39 +00:00
Andrew Morgan	c69df5d5d3	Fix comments. v0.99.2 -> v0.99.3	2019-03-20 11:27:18 +00:00
Andrew Morgan	94cb7939e4	Federation test fixed!	2019-03-20 10:50:44 +00:00
Richard van der Hoff	b5d48560c7	Fix RegistrationTestCase ... turns out this relies on there being a `user_consent_version` set.	2019-03-19 12:05:05 +00:00
Richard van der Hoff	13bc1e0746	Use a regular HomeServerConfig object for unit tests ... Rather than using a Mock for the homeserver config, use a genuine HomeServerConfig object. This makes for a more realistic test, and means that we don't have to keep remembering to add things to the mock config every time we add a new config setting.	2019-03-19 11:44:43 +00:00
Richard van der Hoff	053c50bcb3	Fix resource limits tests ... Make sure that we have a `server_notices_mxid` set, given that we are relying on it.	2019-03-19 11:44:43 +00:00
Richard van der Hoff	45bb54a6c6	Fix registration test ... * Set allow_guest_access = True, since we rely on it * config doesn't have a `hostname` attribute; it is `server_name`	2019-03-19 11:44:43 +00:00
Richard van der Hoff	8c1774e821	Fix email test ... The Mailer expects the config object to have `email_smtp_pass` and `email_riot_base_url` attributes (and it won't by default, because the default config impl doesn't set any of the attributes unless email_enable_notifs is set).	2019-03-19 11:44:43 +00:00
Richard van der Hoff	45c4e19c74	Merge remote-tracking branch 'origin/develop' into HEAD	2019-03-19 11:43:46 +00:00
Neil Johnson	88f0675967	fix test_auto_create_auto_join_where_no_consent (#4886)	2019-03-19 11:38:59 +00:00
Richard van der Hoff	0dbfae03f9	Enforce hs_disabled_message correctly ... Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests if there was no server_notices_mxid configured.	2019-03-19 11:30:54 +00:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864)	2019-03-19 04:50:24 +11:00
Andrew Morgan	621e7f37f1	Better exception handling	2019-03-18 17:45:54 +00:00
Brendan Abolivier	651ad8bc96	Add ratelimiting on failed login attempts (#4865)	2019-03-18 12:57:20 +00:00
Brendan Abolivier	899e523d6d	Add ratelimiting on login (#4821) ... Add two ratelimiters on login (per-IP address and per-userID).	2019-03-15 17:46:16 +00:00
Erik Johnston	271cb1998b	Revert "Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793)" ... This reverts commit 290552fd83.	2019-03-14 14:30:54 +00:00
Erik Johnston	9073cfc8bd	Merge pull request #4846 from matrix-org/hawkowl/userdir-search ... Improve searching in the userdir	2019-03-14 13:54:15 +00:00
Andrew Morgan	ecea5af491	Correct var name	2019-03-13 21:21:03 +00:00
Andrew Morgan	86c60bda15	i should have given up x3	2019-03-13 20:19:07 +00:00
Andrew Morgan	45524f2f5e	i should have given up x2	2019-03-13 20:17:39 +00:00
Andrew Morgan	09626bfd39	Switch to wrapper function around _send_request	2019-03-13 18:26:06 +00:00
Andrew Morgan	97653ef1f4	Correct argument name	2019-03-12 14:30:26 +00:00
Amber Brown	797b6a63fc	fixup	2019-03-13 01:17:51 +11:00
Andrew Morgan	0ea8582f8b	Cleaner way of implementing trailing slashes	2019-03-12 14:11:11 +00:00
Amber Brown	d306bd1b26	fixup	2019-03-12 22:38:01 +11:00
Amber Brown	10480c4348	fixup	2019-03-12 21:47:14 +11:00
Andrew Morgan	4868b12029	and again	2019-03-11 18:22:26 +00:00
Andrew Morgan	802cb5dcf0	Fix syntax error	2019-03-11 18:08:28 +00:00
Andrew Morgan	66f205e93d	We're calling different functions now	2019-03-11 18:01:58 +00:00
Andrew Morgan	290552fd83	Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793) ... Server side of a solution towards #3622.	2019-03-11 17:44:03 +00:00
Amber Brown	5ba8ceab4c	fixes	2019-03-12 00:35:31 +11:00
Amber Brown	26eefca3b7	setup master	2019-03-11 21:16:10 +11:00
Amber Brown	30a8deeb68	Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search	2019-03-11 21:12:30 +11:00
Amber Brown	4abc988c6a	initial	2019-03-11 21:11:36 +11:00
Andrew Morgan	d6e0be92fe	Disable captcha registration by default in tests	2019-03-08 15:49:38 +00:00
Amber Brown	f6135d06cf	Rewrite userdir to be faster (#4537)	2019-03-07 01:22:53 -08:00
Erik Johnston	16c8b4ecbd	Merge pull request #4772 from jbweston/jbweston/server-version-api ... Add 'server_version' endpoint to admin API	2019-03-05 16:31:00 +00:00
Brendan Abolivier	a4c3a361b7	Add rate-limiting on registration (#4735) ... * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test	2019-03-05 14:25:33 +00:00
Richard van der Hoff	2c3548d9d8	Update test_typing to use HomeserverTestCase. (#4771)	2019-03-04 10:05:39 +00:00
Joseph Weston	1e8388b311	Add 'server_version' endpoint to admin API ... This is required because the 'Server' HTTP header is not always passed through proxies.	2019-03-01 09:56:58 +01:00
Joseph Weston	b136ee10df	Import 'admin' module rather than 'register_servlets' directly ... We will later need also to import 'register_servlets' from the 'login' module, so we un-pollute the namespace now to keep the logical changes separate.	2019-03-01 09:46:25 +01:00
Richard van der Hoff	68f47d6744	Fix parsing of Content-Disposition headers (#4763) ... * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings	2019-02-27 14:29:10 -08:00
Amber Brown	f2a753ea38	Move from TravisCI to BuildKite (#4752)	2019-02-27 13:03:14 -08:00
Erik Johnston	d4dc527a1a	Fix unit tests	2019-02-27 14:24:45 +00:00
Amber Brown	a06614bd2a	UPSERT many functionality (#4644)	2019-02-20 23:03:30 +11:00
Erik Johnston	dbdc565dfd	Fix registration on workers (#4682) ... * Move RegistrationHandler init to HomeServer * Move post registration actions to RegistrationHandler * Add post regisration replication endpoint * Newsfile	2019-02-20 18:47:31 +11:00
Erik Johnston	af691e415c	Move register_device into handler	2019-02-18 16:49:38 +00:00
Erik Johnston	41c3f21c3b	Fix unit tests	2019-02-18 13:43:16 +00:00
Erik Johnston	cb12a37708	Clarify and fix behaviour when there are multiple aliases	2019-02-14 18:16:32 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules ... This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Richard van der Hoff	32b781bfe2	Fix error when loading cert if tls is disabled (#4618) ... If TLS is disabled, it should not be an error if no cert is given. Fixes #4554.	2019-02-12 10:51:31 +00:00
Richard van der Hoff	0ca2908653	fix tests	2019-02-11 22:01:27 +00:00
Richard van der Hoff	5d27730a73	Move ClientTLSOptionsFactory init out of refresh_certificates (#4611) ... It's nothing to do with refreshing the certificates. No idea why it was here.	2019-02-11 18:03:30 +00:00
Erik Johnston	719e073f00	Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating ... add updating of backup versions	2019-02-11 13:45:49 +00:00
Erik Johnston	b201149c7e	Merge pull request #4420 from matrix-org/jaywink/openid-listener ... New listener resource for the federation API "openid/userinfo" endpoint	2019-02-11 09:44:00 +00:00
Hubert Chathi	afae8442b5	make sure version is in body and wrap in linearizer queue ... also add tests	2019-02-08 01:32:45 -05:00
Richard van der Hoff	8a21b03fba	Treat an invalid .well-known the same as an absent one ... ... basically, carry on and fall back to SRV etc.	2019-02-01 11:37:31 +00:00
Richard van der Hoff	d428b46346	Update federation routing logic to check .well-known before SRV	2019-01-31 23:14:18 +00:00
Erik Johnston	6587b0b89b	Merge pull request #4472 from matrix-org/neilj/room_capabilities ... Server capabilities support	2019-01-30 14:26:56 +00:00
Richard van der Hoff	c7b24ac3d0	Follow redirects on .well-known (#4520)	2019-01-30 11:43:33 +00:00
Richard van der Hoff	bc5f6e1797	Add a caching layer to .well-known responses (#4516)	2019-01-30 10:55:25 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519)	2019-01-30 10:53:17 +00:00
Neil Johnson	ee4df7fd7a	Merge branch 'develop' into neilj/room_capabilities	2019-01-30 10:28:08 +00:00
Neil Johnson	e4bef9d470	rework format of change password capability	2019-01-29 18:04:56 +00:00
Richard van der Hoff	cc2d650ef7	Relax requirement for a content-type on .well-known (#4511)	2019-01-29 16:49:17 +00:00
Neil Johnson	f03b3a7a3a	support change_password in capabilities end-point	2019-01-29 15:58:37 +00:00
Amber Brown	6bd4374636	Do not generate self-signed TLS certificates by default. (#4509)	2019-01-29 14:09:10 +00:00
Erik Johnston	b8d75ef53e	Merge pull request #4481 from matrix-org/erikj/event_builder ... Refactor event building into EventBuilder	2019-01-29 14:07:23 +00:00
Richard van der Hoff	99e36d5e24	Implement MSC1708 (.well-known lookups for server routing) (#4489)	2019-01-29 13:53:02 +00:00
Neil Johnson	4eeb2fb215	isort	2019-01-29 12:44:10 +00:00
Richard van der Hoff	5488cadaae	Enable configuring test log level via env var (#4506) ... I got fed up with always adding '@unittest.DEBUG' every time I needed to debug a test.	2019-01-29 12:07:00 +00:00
Neil Johnson	a124025dab	enforce auth for capabilities endpoint	2019-01-29 11:37:56 +00:00
Erik Johnston	aee39f7de8	Fix test to use valid event format	2019-01-29 11:19:50 +00:00
Erik Johnston	554ca58ea1	Make add_hashes_and_signatures operate on dicts	2019-01-29 11:12:38 +00:00
Richard van der Hoff	f2b553d656	Use SimpleResolverComplexifier in tests (#4497) ... two reasons for this. One, it saves a bunch of boilerplate. Two, it squashes unicode to IDNA-in-a-`str` (even on python 3) in a way that it turns out we rely on to give consistent behaviour between python 2 and 3.	2019-01-29 09:38:29 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version ... Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Amber Brown	7072fe3084	Fix UPSERTs on SQLite 3.24+ (#4477)	2019-01-28 15:43:32 +00:00
Richard van der Hoff	0fd5b3b53e	Handle IP literals explicitly ... We don't want to be doing .well-known lookups on these guys.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	d840019192	Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487) ... Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.	2019-01-28 09:56:59 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version ... Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Richard van der Hoff	8520bc3109	Fix Host header sent by MatrixFederationAgent (#4468) ... Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.	2019-01-25 12:38:16 +00:00
Neil Johnson	95f871fc0d	Support room version capabilities in CS API (MSC1804)	2019-01-25 11:16:29 +00:00
Erik Johnston	9770ed91c2	Fix tests	2019-01-25 10:32:26 +00:00
Erik Johnston	be6a7e47fa	Revert "Require event format version to parse or create events"	2019-01-25 10:23:51 +00:00
Erik Johnston	efb8ed1d45	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version	2019-01-24 18:52:34 +00:00
Richard van der Hoff	afd69a0920	Look up the right SRV record	2019-01-24 13:31:43 +00:00
Richard van der Hoff	e1c8440e0c	lots more tests for MatrixFederationAgent	2019-01-24 13:28:07 +00:00
Neil Johnson	10b89d5c2e	Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check ... Neilj/fix threepid auth check	2019-01-24 13:02:50 +00:00
Amber Brown	58f6c48183	Use native UPSERTs where possible (#4306)	2019-01-24 21:31:54 +11:00
Richard van der Hoff	97fd29c019	Don't send IP addresses as SNI (#4452) ... The problem here is that we have cut-and-pasted an impl from Twisted, and then failed to maintain it. It was fixed in Twisted in https://github.com/twisted/twisted/pull/1047/files; let's do the same here.	2019-01-24 09:34:44 +00:00
Erik Johnston	e79ba9eb34	Fix tests	2019-01-24 09:28:16 +00:00
Richard van der Hoff	6b574f3df7	fix python2 test failure	2019-01-23 11:25:36 +00:00
Jason Robinson	6f680241bd	Fix flake8 issues ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:53:48 +02:00
Jason Robinson	1838ef1ac3	Fix openid tests after rebase ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:38:13 +02:00
Jason Robinson	a47fac9af6	Fix sorting of imports in tests. Remove an unnecessary mock ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	5336e49b39	Add tests for the openid lister for SynapseHomeServer ... Check all possible variants of openid and federation listener on/off possibilities. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	6d25599098	Add tests for the openid lister for FederationReaderServer ... Check all possible variants of openid and federation listener on/off possibilities. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	ab97b6e33c	Fix a test docstring in frontend proxy tests ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Richard van der Hoff	d02c4532c0	Add a test for MatrixFederationAgent	2019-01-22 20:35:12 +00:00
Richard van der Hoff	7021784d46	put resolve_service in an object ... this makes it easier to stub things out for tests.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	53a327b4d5	Require that service_name be a byte string ... it is only ever a bytes now, so let's enforce that.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	7871146667	Make MatrixFederationClient use MatrixFederationAgent ... ... instead of the matrix_federation_endpoint	2019-01-22 20:34:35 +00:00
Neil Johnson	d619b113ed	Fix None guard in config.server.is_threepid_reserved	2019-01-22 16:52:29 +00:00
Richard van der Hoff	33a55289cb	Refactor and bugfix for resove_service (#4427)	2019-01-22 10:59:27 +00:00
Amber Brown	23b0813599	Require ECDH key exchange & remove dh_params (#4429) ... * remove dh_params and set better cipher string	2019-01-22 21:58:50 +11:00
Richard van der Hoff	de6888e7ce	Remove redundant WrappedConnection (#4409) ... * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.	2019-01-18 23:07:38 +11:00
Richard van der Hoff	676cf2ee26	Fix incorrect logcontexts after a Deferred was cancelled (#4407)	2019-01-17 14:00:23 +00:00
Amber Brown	522dada206	Merge remote-tracking branch 'origin/master' into develop	2019-01-11 02:22:48 +11:00
Richard van der Hoff	e0910d0145	Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 ... Fixes #4371	2019-01-10 14:12:50 +00:00
Richard van der Hoff	aa70d24125	Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34	2019-01-10 12:58:33 +00:00
Richard van der Hoff	566947ff34	Skip macaroon check for access tokens in the db	2019-01-10 12:57:21 +00:00
Amber Brown	7960c26fda	Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369)	2019-01-09 22:26:25 +11:00
Erik Johnston	b970cb0e96	Refactor request sending to have better excpetions (#4358) ... * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.	2019-01-08 11:04:28 +00:00
Amber Brown	d7843f47b6	Remove v1 only REST APIs now we don't ship matrix console (#4334)	2018-12-29 23:12:30 +11:00
Richard van der Hoff	f85676cc93	Return well_known in /login response (#4319) ... ... as per MSC1730.	2018-12-24 10:44:33 +01:00
Richard van der Hoff	3355c1a9ec	Merge pull request #4317 from de-vri-es/test-metric-prometheus-0.5 ... Fix test_metrics.py compatibility prometheus_client 0.5	2018-12-23 00:14:47 +01:00
Amber Brown	ea6abf6724	Fix IP URL previews on Python 3 (#4215)	2018-12-22 01:56:13 +11:00
Maarten de Vries	48b7ff7a35	Fix test_metrics.py compatibility prometheus_client 0.5 ... prometheus_client 0.5 has a named-tuple Sample type with more member than the old plain tuple had. This commit makes sure the unit test detects this and changes the way it reads the sample. Signed-off-by: Maarten de Vries <maarten@de-vri.es>	2018-12-21 01:53:57 +01:00
Neil Johnson	d2f7c4e6b1	create support user (#4141) ... Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272) ... When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	c7401a697f	Implement SAML2 authentication (#4267) ... This implements both a SAML2 metadata endpoint (at `/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at `/_matrix/saml2/authn_response`). If the SAML2 response matches what's been configured, we complete the SSO login flow by redirecting to the client url (aka `RelayState` in SAML2 jargon) with a login token. What we don't yet have is anything to build a SAML2 request and redirect the user to the identity provider. That is left as an exercise for the reader.	2018-12-07 13:11:11 +01:00
Richard van der Hoff	c588b9b9e4	Factor SSO success handling out of CAS login (#4264) ... This is mostly factoring out the post-CAS-login code to somewhere we can reuse it for other SSO flows, but it also fixes the userid mapping while we're at it.	2018-12-07 13:10:07 +01:00
Richard van der Hoff	b0c24a66ec	Rip out half-implemented m.login.saml2 support (#4265) ... * Rip out half-implemented m.login.saml2 support This was implemented in an odd way that left most of the work to the client, in a way that I really didn't understand. It's going to be a pain to maintain, so let's start by ripping it out. * drop undocumented dependency on dateutil It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 19:44:38 +11:00
Richard van der Hoff	e8d98466b0	Implement .well-known handling (#4262) ... Sometimes it's useful for synapse to generate its own .well-known file.	2018-12-05 14:38:58 +01:00
Richard van der Hoff	48972ce9d1	Patch defer.inlineCallbacks to check logcontexts in tests (#4205)	2018-12-04 11:30:32 +01:00
Neil Johnson	7039ece8fb	Neilj/fix autojoin (#4223) ... * Fix auto join failures for servers that require user consent * Fix auto join failures for servers that require user consent	2018-11-28 22:24:57 +11:00
Richard van der Hoff	a44c0a096f	Check logcontexts before and after each test (#4190) ... * Add better diagnostics to flakey keyring test * fix interpolation fail * Check logcontexts before and after each test * update changelog * update changelog	2018-11-27 13:47:18 +11:00
Richard van der Hoff	80527b568d	Fix more logcontext leaks in tests (#4209)	2018-11-27 13:01:04 +11:00
Richard van der Hoff	de8772a655	Do a GC after each test to fix logcontext leaks (#4227) ... * Some words about garbage collections and logcontexts * Do a GC after each test to fix logcontext leaks This feels like an awful hack, but... * changelog	2018-11-27 13:00:33 +11:00
Richard van der Hoff	416c671474	Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes ... Fix some logcontext leaks	2018-11-20 12:19:19 +01:00
Amber Brown	80cac86b2c	Fix fallback auth on Python 3 (#4197)	2018-11-19 12:27:33 -06:00
Richard van der Hoff	828f18bd8b	Fix logcontext leak in test_url_preview	2018-11-19 17:07:01 +00:00
Richard van der Hoff	a267c2e3ed	Fix logcontext leak in http pusher test	2018-11-19 17:07:01 +00:00
Richard van der Hoff	884a561447	Fix some tests which leaked logcontexts	2018-11-19 17:07:01 +00:00
Travis Ralston	3da9781c98	Fix the terms UI auth tests ... By setting the config value directly, we skip the block that adds the slash automatically for us.	2018-11-15 23:00:28 -07:00
Amber Brown	8b1affe7d5	Fix Content-Disposition in media repository (#4176)	2018-11-15 15:55:58 -06:00
Travis Ralston	835779f7fb	Add option to track MAU stats (but not limit people) (#3830)	2018-11-15 18:08:27 +00:00
Amber Brown	df758e155d	Use <meta> tags to discover the per-page encoding of html previews (#4183)	2018-11-15 11:05:08 -06:00
Richard van der Hoff	c1efcd7c6a	Add a test for the public T&Cs form	2018-11-14 10:46:27 +00:00
Erik Johnston	abaa93c158	Add test to assert set_e2e_device_keys correctly returns False on no-op	2018-11-08 14:06:44 +00:00
Amber Brown	b3708830b8	Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157)	2018-11-08 01:37:43 +11:00
Amber Brown	e62f7f17b3	Remove some boilerplate in tests (#4156)	2018-11-07 03:00:00 +11:00
Travis Ralston	0f5e51f726	Add config variables for enabling terms auth and the policy name (#4142) ... So people can still collect consent the old way if they want to.	2018-11-06 10:32:34 +00:00
Hubert Chathi	f1087106cf	handle empty backups according to latest spec proposal (#4123) ... fixes #4056	2018-11-05 17:59:29 -05:00
Amber Brown	efdcbbe46b	Tests for user consent resource (#4140)	2018-11-06 05:53:44 +11:00
Amber Brown	5a63589e80	Add some tests for the HTTP pusher (#4149)	2018-11-06 05:53:24 +11:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139) ... * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Erik Johnston	90d713b8c6	Merge pull request #4137 from matrix-org/erikj/clean_up_events ... Clean up event accesses and tests	2018-11-02 14:12:49 +00:00
Erik Johnston	b86d05a279	Clean up event accesses and tests ... This is in preparation to refactor FrozenEvent to support different event formats for different room versions	2018-11-02 13:44:14 +00:00
Amber Brown	cb7a6b2379	Fix typing being reset causing infinite syncs (#4127)	2018-11-03 00:19:23 +11:00
Erik Johnston	b199534518	Merge pull request #4135 from matrix-org/erikj/fix_state_res_none ... Fix None exception in state res v2	2018-11-02 10:45:57 +00:00
Erik Johnston	54aec35867	Fix None exception in state res v2	2018-11-02 10:29:19 +00:00
Travis Ralston	a8c9faa9a2	The tests also need a version parameter	2018-10-31 13:28:08 -06:00
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00
Amber Brown	3bade14ec0	Fix search 500ing (#4122)	2018-10-31 04:33:41 +11:00
Amber Brown	0dce9e1379	Write some tests for the email pusher (#4095)	2018-10-30 23:55:43 +11:00
Richard van der Hoff	bf33eed609	Merge pull request #4091 from matrix-org/rav/room_version_upgrades ... Room version upgrade support	2018-10-29 12:47:20 +00:00
Amber Brown	77d70a7646	Port register_new_matrix_user to Python 3 and add tests (#4085)	2018-10-26 22:05:22 +11:00
Richard van der Hoff	474810d9d5	fix broken test ... This test stubbed out some stuff in a very weird way. I have no idea why. It broke.	2018-10-25 23:15:03 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Erik Johnston	c85e063302	Merge pull request #4051 from matrix-org/erikj/alias_disallow_list ... Add config option to control alias creation	2018-10-25 17:04:59 +01:00
Neil Johnson	95ad128851	Merge pull request #4081 from matrix-org/neilj/fix_mau_init ... fix race condiftion in calling initialise_reserved_users	2018-10-25 16:33:40 +01:00
Erik Johnston	b94a43d5b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list	2018-10-25 15:25:31 +01:00
Erik Johnston	e5481b22aa	Use allow/deny	2018-10-25 15:25:21 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Travis Ralston	a5468eaadf	pep8	2018-10-24 13:54:38 -06:00
Travis Ralston	81880beff4	It helps to import things	2018-10-24 13:32:13 -06:00
Travis Ralston	4acb6fe8a3	Move test to where the other integration tests are	2018-10-24 13:24:24 -06:00
Travis Ralston	9283987f7e	Fix test ... Debug tests Try printing the channel fix Import and use six Remove debugging Disable captcha Add some mocks Define the URL Fix the clock? Less rendering? use the other render Complete the dummy auth stage Fix last stage of the test Remove mocks we don't need	2018-10-24 13:23:08 -06:00
Travis Ralston	54def42c19	Merge branch 'develop' into travis/login-terms	2018-10-24 13:22:59 -06:00
Neil Johnson	ea69a84bbb	fix style inconsistencies	2018-10-24 17:18:08 +01:00
Neil Johnson	07126e43a4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init	2018-10-24 16:25:39 +01:00
Richard van der Hoff	3ad359e5be	Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation	2018-10-24 11:23:49 +01:00
Richard van der Hoff	7328039117	Merge pull request #4082 from matrix-org/rav/fix_pep8 ... Fix a number of flake8 errors	2018-10-24 11:23:35 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	7e07d25ed6	Allow backslashes in event field filters ... Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which meant that single backslashes were not allowed in event field filters. The intention here is to allow single-backslashes, but disallow double-backslashes.	2018-10-24 11:11:24 +01:00
Richard van der Hoff	ef771cc4c2	Fix a number of flake8 errors ... Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.	2018-10-24 10:39:03 +01:00
Erik Johnston	b313b9b009	isort	2018-10-24 10:02:41 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Neil Johnson	6105c6101f	fix race condiftion in calling initialise_reserved_users	2018-10-23 15:24:58 +01:00
Erik Johnston	3c580c2b47	Add tests for alias creation rules	2018-10-19 10:22:45 +01:00
Erik Johnston	1b4bf232b9	Add tests for config generation	2018-10-19 10:22:45 +01:00
Travis Ralston	dba84fa69c	Fix terms UI auth test	2018-10-18 12:45:21 -06:00
Travis Ralston	88c5ffec33	Test for terms UI auth	2018-10-18 12:35:30 -06:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046) ... Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Erik Johnston	947c7443eb	Add some state res v2 tests	2018-10-16 16:28:39 +01:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Erik Johnston	9c834b8ee9	Add tests	2018-10-02 16:22:39 +01:00
Amber Brown	7232917f12	Disable frozen dicts by default (#3987)	2018-10-02 22:53:47 +10:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors ... Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Amber Brown	6e05fd032c	Fix userconsent on Python 3 (#3938)	2018-10-02 00:11:58 +10:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 ... Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Richard van der Hoff	484a9b8c81	Remove redundant, failing, test ... This test didn't do what it claimed to do, and what it claimed to do was the same as test_cant_hide_direct_ancestors anyway. This stuff is tested by sytest anyway.	2018-09-27 13:11:23 +01:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) ... Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Richard van der Hoff	ae6ad4cf41	docstrings and unittests for storage.state (#3958) ... I spent ages trying to figure out how I was going mad...	2018-09-27 11:22:25 +01:00
Richard van der Hoff	4c3e7eeec5	Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches ... Fix some instances of ExpiringCache not expiring cache items	2018-09-25 12:02:57 +01:00
Amber Brown	fbe5ba25f6	Merge branch 'master' into develop	2018-09-25 03:10:01 +10:00
Oleg Girko	7d3f639844	Fix compatibility issue with older Twisted in tests. ... Older Twisted (18.4.0) returns TimeoutError instead of ConnectingCancelledError when connection times out. This change allows tests to be compatible with this behaviour. Signed-off-by: Oleg Girko <ol@infoserver.lv>	2018-09-25 02:51:18 +10:00
Erik Johnston	8601c24287	Fix some instances of ExpiringCache not expiring cache items ... ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.	2018-09-21 14:19:46 +01:00
Amber Brown	1f3f5fcf52	Fix client IPs being broken on Python 3 (#3908)	2018-09-20 20:14:34 +10:00
Amber Brown	aeca5a5ed5	Add a regression test for logging on failed connections (#3912)	2018-09-20 16:28:18 +10:00
Richard van der Hoff	31c15dcb80	Refactor matrixfederationclient to fix logging (#3906) ... We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.	2018-09-18 18:17:15 +01:00
Erik Johnston	f6e82dcddb	Tests	2018-09-14 15:08:37 +01:00
Amber Brown	bfa0b759e0	Attempt to figure out what's going on with timeouts (#3857)	2018-09-14 00:15:51 +10:00
Neil Johnson	8decd6233d	improve naming	2018-09-12 16:22:15 +01:00
Neil Johnson	0ddf486724	expose number of real reserved users	2018-09-12 11:58:52 +01:00
Amber Brown	52ec6e9dfa	Port tests/ to Python 3 (#3808)	2018-09-07 02:58:18 +10:00
Neil Johnson	c5440b2ca0	Merge pull request #3800 from matrix-org/neilj/remove-guests-from-mau-count ... guest users should not be part of mau total	2018-09-06 17:45:55 +01:00
Neil Johnson	84a750e0c3	ensure guests never enter mau list	2018-09-06 17:22:53 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Richard van der Hoff	4f8baab0c4	Merge branch 'master' into develop	2018-09-06 13:05:22 +01:00
Neil Johnson	61b05727fa	guest users should not be part of mau total	2018-09-05 22:30:36 +01:00
Richard van der Hoff	c127c8d042	Fix origin handling for pushed transactions ... Use the actual origin for push transactions, rather than whatever the remote server claimed.	2018-09-05 13:08:07 +01:00
Amber Brown	77055dba92	Fix tests on postgresql (#3740)	2018-09-04 02:21:48 +10:00
Neil Johnson	0b01281e77	move threepid checker to config, add missing yields	2018-08-31 17:11:11 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Amber Brown	14e4d4f4bf	Port storage/ to Python 3 (#3725)	2018-08-31 00:19:58 +10:00
Richard van der Hoff	414fa36f3e	Fix up tests	2018-08-28 17:21:05 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Erik Johnston	9db2476991	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact	2018-08-24 17:00:37 +01:00
Erik Johnston	05077e06fa	Change admin_uri to admin_contact in config and errors	2018-08-24 16:51:27 +01:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	01a5a8b9e3	Fix checking if service notice room is already tagged ... This manifested in synapse repeatedly setting the tag for the room	2018-08-24 16:22:37 +01:00
Erik Johnston	f2cbbda956	Unit tests	2018-08-23 19:17:19 +01:00
Erik Johnston	cd77270a66	Implement trail users	2018-08-23 19:17:19 +01:00
Erik Johnston	7e6e588e60	Fix bug where we resent "limit exceeded" server notices ... This was due to a bug where we mutated a cached event's contents	2018-08-23 16:21:20 +01:00
Erik Johnston	c5842dff1a	Actually run the tests	2018-08-23 10:35:54 +01:00
Erik Johnston	fd2dbf1836	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-22 17:06:10 +01:00
Erik Johnston	9643a6f7f2	Update notice format	2018-08-22 17:00:29 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	3bf8bab8f9	Merge pull request #3673 from matrix-org/erikj/refactor_state_handler ... Refactor state module to support multiple room versions	2018-08-22 10:04:55 +01:00
Matthew Hodgson	bb81e78ec6	Split the state_group_cache in two (#3726) ... Splits the state_group_cache in two. One half contains normal state events; the other contains member events. The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching. Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.	2018-08-22 00:56:37 +02:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Neil Johnson	e07970165f	rename error code	2018-08-18 14:39:45 +01:00
Neil Johnson	ba1fbf7d5b	special case server_notices_mxid	2018-08-18 12:31:08 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	2018-08-18 01:08:45 +10:00
Neil Johnson	d49b77404b	clean up, no functional changes	2018-08-17 15:21:34 +01:00
Richard van der Hoff	63260397c6	Merge pull request #3701 from matrix-org/rav/use_producer_for_responses ... Use a producer to stream back responses	2018-08-17 14:58:45 +01:00
Neil Johnson	3ee57bdcbb	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-17 14:34:10 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Neil Johnson	b5f638f1f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-17 14:04:15 +01:00
Neil Johnson	69c49d3fa3	Merge branch 'develop' into neilj/limit_exceeded_error	2018-08-17 12:44:26 +00:00
Neil Johnson	bcfeb44afe	call reap on start up and fix under reaping bug	2018-08-16 22:55:32 +01:00
Neil Johnson	13ad9930c8	add new error type ResourceLimit	2018-08-16 18:02:02 +01:00
Neil Johnson	3c1080b6e4	refactor for readability, and reuse caching for setting tags	2018-08-16 17:02:04 +01:00
Neil Johnson	a675f9c556	check for room state before deciding on action	2018-08-16 14:53:35 +01:00
Neil Johnson	8cfad2e686	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-15 17:20:38 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus ... Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Neil Johnson	fc5d937550	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-15 16:31:40 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Amber Brown	a87af25fbb	Fix the tests	2018-08-15 15:12:23 +01:00
Neil Johnson	75c663c7b9	update error codes	2018-08-15 11:27:48 +01:00
Neil Johnson	b8429c7c81	update error codes for resource limiting	2018-08-15 10:19:48 +01:00
Neil Johnson	ab035bdeac	replace admin_email with admin_uri for greater flexibility	2018-08-15 10:16:41 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	b586b8b986	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 17:43:22 +01:00
Neil Johnson	7277216d01	fix setup_test_homeserver to be postgres compatible	2018-08-14 17:14:39 +01:00
Neil Johnson	1522ed9c07	in case max_mau is less than I think	2018-08-14 16:52:30 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	06b331ff40	fix off by 1 errors	2018-08-14 15:28:15 +01:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Amber Brown	bdfbd934d6	Implement a new test baseclass to cut down on boilerplate (#3684)	2018-08-14 20:53:43 +10:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" ... This reverts commit 0d43f991a1.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	66a4ca1d28	404 nicely if you try to interact with a missing current version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	edc427a351	flake8	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	f6a3067868	linting	2018-08-12 19:14:31 -04:00
Matthew Hodgson	15d513f16f	fix idiocies and so make tests pass	2018-08-12 19:14:31 -04:00
Matthew Hodgson	174be586e5	first cut at a UT	2018-08-12 19:14:31 -04:00
Neil Johnson	6c6aba76e1	implementation of server notices to alert on hitting resource limits	2018-08-10 15:12:59 +01:00
Amber Brown	807449d8f2	fix up a forced long line	2018-08-11 00:01:43 +10:00
black	8b3d9b6b19	Run black.	2018-08-10 23:54:09 +10:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Richard van der Hoff	3c0213a217	Merge pull request #3439 from vojeroen/send_sni_for_federation_requests ... send SNI for federation requests	2018-08-10 12:23:54 +01:00
Neil Johnson	885ea9c602	rename _user_last_seen_monthly_active	2018-08-09 18:02:12 +01:00
Neil Johnson	04df714259	fix imports	2018-08-09 17:41:52 +01:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	3e19beb941	Fix tests	2018-08-09 14:58:49 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	3dce9050cf	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:42:02 +01:00
Neil Johnson	0ad98e38d0	Merge pull request #3655 from matrix-org/neilj/disable_hs ... Flag to disable HS without disabling federation	2018-08-09 10:41:43 +00:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Amber Brown	2511f3f8a0	Test fixes for Python 3 (#3647)	2018-08-09 12:22:01 +10:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Neil Johnson	5298d79fb5	Merge branch 'develop' into neilj/disable_hs	2018-08-08 16:13:03 +00:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	d2f3ef98ac	Merge branch 'develop' into neilj/disable_hs	2018-08-08 15:55:47 +00:00
Neil Johnson	7f3d897e7a	mock config.max_mau_value	2018-08-08 11:46:23 +01:00
Neil Johnson	e8eba2b4e3	implement reserved users for mau limits	2018-08-07 17:49:43 +01:00
Neil Johnson	fbe255f9a4	add default mau_limits_reserved_threepids	2018-08-06 23:24:54 +01:00
Neil Johnson	42c6823827	disable HS from config	2018-08-04 22:07:04 +01:00
Neil Johnson	d08296f9f2	remove unused import	2018-08-03 23:00:16 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Neil Johnson	e10830e976	wip commit - tests failing	2018-08-03 17:55:50 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	5593ff6773	fix (lots of) py3 test failures	2018-08-03 14:59:17 +01:00
Neil Johnson	b2aab04d2c	fix py3 test failure	2018-08-03 14:12:56 +01:00
Neil Johnson	950807d93a	fix caching and tests	2018-08-03 13:49:53 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Neil Johnson	9180061b49	remove unused count_monthly_users	2018-08-02 15:55:29 +01:00
Neil Johnson	c4ffbecb68	fix test, update constructor call	2018-08-02 13:51:05 +01:00
Neil Johnson	00f99f74b1	insertion into monthly_active_users	2018-08-02 13:47:19 +01:00
Neil Johnson	165e067033	Revert "change monthly_active_users table to be a single column" ... This reverts commit ec716a35b2.	2018-08-02 10:59:58 +01:00
Neil Johnson	ec716a35b2	change monthly_active_users table to be a single column	2018-08-01 17:54:37 +01:00
Neil Johnson	d766f26de9	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-01 17:49:41 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	c480c4c962	fix isort	2018-08-01 14:25:58 +01:00
Neil Johnson	6eed16d8a2	fix test for py3	2018-08-01 14:02:10 +01:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Neil Johnson	4e6e00152c	fix known broken test	2018-08-01 11:48:37 +01:00
Richard van der Hoff	cab782c17e	Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator ... Rewrite cache list decorator	2018-08-01 10:28:56 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Neil Johnson	6ef983ce5c	api into monthly_active_users table	2018-07-31 16:36:24 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Neil Johnson	7d05406a07	fix user_ips counting	2018-07-31 12:03:23 +01:00
Travis Ralston	e908b86832	Remove pdu_failures from transactions ... The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.	2018-07-30 16:28:47 -06:00
Neil Johnson	cefac79c10	monthly_active_tests	2018-07-30 22:08:09 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Richard van der Hoff	a8cbce0ced	fix invalidation	2018-07-27 16:17:17 +01:00
Richard van der Hoff	f102c05856	Rewrite cache list decorator ... Because it was complicated and annoyed me. I suspect this will be more efficient too.	2018-07-27 13:47:04 +01:00
Richard van der Hoff	7041cd872b	Merge branch 'develop' into send_sni_for_federation_requests	2018-07-27 09:17:11 +01:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Matthew Hodgson	0620d27f4d	flake8	2018-07-25 17:21:17 +01:00
Matthew Hodgson	0a7ee0ab8b	add tests for _get_some_state_from_cache	2018-07-25 16:33:52 +01:00
Richard van der Hoff	f59be4eb0e	Fix unit tests ... on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival.	2018-07-25 10:30:36 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Erik Johnston	8fbe418777	Fix unit tests	2018-07-23 13:33:49 +01:00
Amber Brown	3132b89f12	Make the rest of the .iterwhatever go away (#3562)	2018-07-21 15:47:18 +10:00
Richard van der Hoff	3d6df84658	Test and fix support for cancellation in Linearizer	2018-07-20 13:59:55 +01:00
Richard van der Hoff	4f67623674	Merge pull request #3571 from matrix-org/rav/limiter_fixes ... A set of improvements to the Limiter	2018-07-20 13:53:03 +01:00
Amber Brown	e1a237eaab	Admin API for creating new users (#3415)	2018-07-20 22:41:13 +10:00
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	2018-07-20 13:11:43 +01:00
Richard van der Hoff	d7275eecf3	Add a sleep to the Limiter to fix stack overflows. ... Fixes #3570	2018-07-20 12:37:12 +01:00
Matthew Hodgson	650daf5628	make test work	2018-07-19 20:49:44 +01:00
Matthew Hodgson	1fa4f7e03e	first cut of a UT for testing state store (untested)	2018-07-19 20:19:32 +01:00
Richard van der Hoff	c754e006f4	Merge pull request #3556 from matrix-org/rav/background_processes ... Run things as background processes	2018-07-19 11:04:18 +01:00
Amber Brown	a97c845271	Move v1-only APIs into their own module & isolate deprecated ones (#3460)	2018-07-19 20:03:33 +10:00
Richard van der Hoff	8c69b735e3	Make Distributor run its processes as a background process ... This is more involved than it might otherwise be, because the current implementation just drops its logcontexts and runs everything in the sentinel context. It turns out that we aren't actually using a bunch of the functionality here (notably suppress_failures and the fact that Distributor.fire returns a deferred), so the easiest way to fix this is actually by simplifying a bunch of code.	2018-07-18 20:55:05 +01:00
Richard van der Hoff	d897be6a98	Fix visibility of events from erased users over federation	2018-07-17 14:02:07 +01:00
Richard van der Hoff	9c04b4abf9	Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server ... Refactor and optimze filter_events_for_server	2018-07-17 14:01:39 +01:00
Richard van der Hoff	94440ae994	fix imports	2018-07-17 11:51:26 +01:00
Amber Brown	bc006b3c9d	Refactor REST API tests to use explicit reactors (#3351)	2018-07-17 20:43:18 +10:00
Erik Johnston	850238b4ef	Add unit test	2018-07-17 10:59:02 +01:00
Amber Brown	3fe0938b76	Merge pull request #3530 from matrix-org/erikj/stream_cache ... Don't return unknown entities in get_entities_changed	2018-07-17 13:44:46 +10:00
Richard van der Hoff	15b13b537f	Add a test which profiles filter_events_for_server in a large room	2018-07-16 14:06:09 +01:00
Richard van der Hoff	ea69d35651	Move filter_events_for_server out of FederationHandler ... for easier unit testing.	2018-07-16 13:06:24 +01:00
Jeroen	505530f36a	Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests ... # Conflicts: # synapse/crypto/context_factory.py	2018-07-14 20:24:46 +02:00
Amber Brown	33b60c01b5	Make auth & transactions more testable (#3499)	2018-07-14 07:34:49 +10:00
Erik Johnston	bc832f822f	Fixup unit test	2018-07-13 17:03:04 +01:00
Richard van der Hoff	3b391d9c45	Fix unit tests	2018-07-13 16:28:04 +01:00
Oleg Girko	6c1ec5a1bd	Use more portable syntax using attrs package. ... Newer syntax attr.ib(factory=dict) is just a syntactic sugar for attr.ib(default=attr.Factory(dict)) It was introduced in newest version of attrs package (18.1.0) and doesn't work with older versions. We should either require minimum version of attrs to be 18.1.0, or use older (slightly more verbose) syntax. Requiring newest version is not a good solution because Linux distributions may have older version of attrs (17.4.0 in Fedora 28), and requiring to build (and package) newer version just to use newer syntactic sugar in only one test is just too much. It's much better to fix that test to use older syntax. Signed-off-by: Oleg Girko <ol@infoserver.lv>	2018-07-10 00:38:49 +01:00
Jeroen	b5e157d895	Merge branch 'develop' into send_sni_for_federation_requests ... # Conflicts: # synapse/http/endpoint.py	2018-07-09 08:51:11 +02:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Neil Johnson	feef8461d1	Merge remote-tracking branch 'hera/rav/server_acls' into develop	2018-07-05 11:04:01 +01:00
Erik Johnston	1a88640677	Merge pull request #3483 from matrix-org/rav/more_server_name_validation ... More server_name validation	2018-07-05 10:04:20 +01:00
Richard van der Hoff	3cf3e08a97	Implementation of server_acls ... ... as described at https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.	2018-07-04 19:06:20 +01:00
Richard van der Hoff	546bc9e28b	More server_name validation ... We need to do a bit more validation when we get a server name, but don't want to be re-doing it all over the shop, so factor out a separate parse_and_validate_server_name, and do the extra validation. Also, use it to verify the server name in the config file.	2018-07-04 18:59:51 +01:00
Richard van der Hoff	ea555d5633	Reinstate lost run_on_reactor in unit test ... a61738b removed a call to run_on_reactor from a unit test, but that call was doing something useful, in making the function in question asynchronous. Reinstate the call and add a check that we are testing what we wanted to be testing.	2018-07-04 09:40:01 +01:00
Richard van der Hoff	508196e08a	Reject invalid server names (#3480) ... Make sure that server_names used in auth headers are sane, and reject them with a sensible error code, before they disappear off into the depths of the system.	2018-07-03 14:36:14 +01:00
Erik Johnston	e3b4043800	Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking ... Check the state of prev_events a bit more thoroughly when coming over federation	2018-06-29 13:55:02 +01:00
Matthew Hodgson	f82cf3c7df	add test	2018-06-28 21:14:16 +01:00
Matthew Hodgson	e72234f6bd	fix tests	2018-06-28 20:56:07 +01:00
Amber Brown	f03a5d1a17	pep8	2018-06-27 11:38:14 +01:00
Amber Brown	94f09618e5	cleanups	2018-06-27 11:38:03 +01:00
Amber Brown	35cc3e8b14	stylistic cleanup	2018-06-27 11:32:09 +01:00
Amber Brown	8d62baa48c	cleanups	2018-06-27 11:31:48 +01:00
Amber Brown	77078d6c8e	handle federation not telling us about prev_events	2018-06-27 11:27:32 +01:00
Amber Brown	cd6bcdaf87	Better testing framework for homeserver-using things (#3446)	2018-06-27 10:37:24 +01:00
Jeroen	3d605853c8	send SNI for federation requests	2018-06-24 22:38:43 +02:00
Erik Johnston	f79abda87f	Merge pull request #3427 from matrix-org/erikj/remove_filters ... remove dead filter_events_for_clients	2018-06-22 14:45:42 +01:00
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	2018-06-22 11:52:07 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	9a793f861c	Merge branch 'master' into develop	2018-06-14 16:36:01 +01:00
Richard van der Hoff	1e77ac66e3	Fix broken unit test ... We need power levels for this test to do what it is supposed to do.	2018-06-14 14:21:29 +01:00
Richard van der Hoff	a502cfec00	remove spurious debug	2018-06-14 14:20:53 +01:00
Richard van der Hoff	5c9afd6f80	Make default state_default 50 ... Make it so that, before there is a power-levels event in the room, you need a power level of at least 50 to send state. Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192	2018-06-14 12:38:09 +01:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Richard van der Hoff	bd348f0af6	remove dead filter_events_for_clients ... This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.	2018-06-12 09:51:31 +01:00
Amber Brown	f7869f8f8b	Port to sortedcontainers (with tests!) (#3332)	2018-06-06 00:13:57 +10:00
Amber Brown	5dbf305444	Put python's logs into Trial when running unit tests (#3319)	2018-06-04 16:06:06 +10:00
Richard van der Hoff	a995fdae39	fix tests	2018-05-29 20:19:29 +01:00
Matthew Hodgson	adb6bac4d5	fix another dumb typo	2018-05-29 02:29:22 +01:00
Matthew Hodgson	284e36ad04	fix dumb typo	2018-05-29 02:23:11 +01:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	d5dca9a04f	Move consent config parsing into ConsentConfig ... turns out we need to reuse this, so it's better in the config class.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Amber Brown	fcc525b0b7	rest of the changes	2018-05-21 19:48:57 -05:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Richard van der Hoff	6e1cb54a05	Fix logcontext leak in HttpTransactionCache ... ONE DAY I WILL PURGE THE WORLD OF THIS EVIL	2018-05-21 16:58:20 +01:00
Richard van der Hoff	6d6e7288fe	Stop the transaction cache caching failures ... The transaction cache has some code which tries to stop it caching failures, but if the callback function failed straight away, then things would happen backwards and we'd end up with the failure stuck in the cache.	2018-05-21 16:49:59 +01:00
Erik Johnston	6406b70aeb	Use stream rather depth ordering for push actions ... This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.	2018-05-11 15:30:11 +01:00
Richard van der Hoff	902673e356	Merge pull request #3161 from NotAFile/remove-v1auth ... Make Client-Server API return 403 for invalid token	2018-05-03 10:10:57 +01:00
Erik Johnston	53a5fdf312	Merge pull request #3175 from matrix-org/erikj/escape_metric_values ... Escape label values in prometheus metrics	2018-05-03 10:01:04 +01:00
Erik Johnston	32015e1109	Escape label values in prometheus metrics	2018-05-02 16:52:42 +01:00
Richard van der Hoff	415c6b672e	Merge branch 'develop' into rav/more_logcontext_leaks	2018-05-02 16:16:01 +01:00
Richard van der Hoff	11607006d9	Remove spurious unittest.DEBUG	2018-05-02 15:48:47 +01:00
Richard van der Hoff	46beeb9a30	Fix a couple of logcontext leaks in unit tests ... ... which were making other, innocent, tests, fail. Plus remove a spurious unittest.DEBUG which was making the output noisy.	2018-05-02 15:46:22 +01:00
Richard van der Hoff	f22e7cda2c	Fix a class of logcontext leaks ... So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.	2018-05-02 11:58:00 +01:00
Richard van der Hoff	e482f8cd85	Fix incorrect reference to StringIO ... This was introduced in 4f2f5171	2018-05-02 09:12:26 +01:00
Adrian Tschira	6495dbb326	Burminate v1auth ... This closes #2602 v1auth was created to account for the differences in status code between the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid tokens). However since those protocols were merged, this makes the r0 version/endpoint internally inconsistent, and violates the specification for the r0 endpoint. This might break clients that rely on this inconsistency with the specification. This is said to affect the legacy angular reference client. However, I feel that restoring parity with the spec is more important. Either way, it is critical to inform developers about this change, in case they rely on the illegal behaviour. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-30 22:20:43 +02:00
Richard van der Hoff	9e2601f830	Merge pull request #3108 from NotAFile/py3-six-urlparse ... Use six.moves.urlparse	2018-04-30 00:33:05 +01:00
Richard van der Hoff	3d1ae61399	Merge branch 'develop' into rav/deferred_timeout	2018-04-27 12:54:43 +01:00
Richard van der Hoff	286e20f2bc	Merge pull request #3109 from NotAFile/py3-tests-fix ... Make tests py3 compatible	2018-04-23 11:59:03 +01:00
Richard van der Hoff	1ea904b9f0	Use deferred.addTimeout instead of time_bound_deferred ... This doesn't feel like a wheel we need to reinvent.	2018-04-23 00:53:18 +01:00
Richard van der Hoff	d06a9ea5f7	Merge pull request #3104 from NotAFile/py3-unittest-config ... Add some more variables to the unittest config	2018-04-20 15:35:58 +01:00
Richard van der Hoff	692a3cc806	Merge pull request #3103 from NotAFile/py3-baseexcepton-message ... Use str(e) instead of e.message	2018-04-20 15:34:49 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	a1a3c9660f	Make tests py3 compatible ... This is a mixed commit that fixes various small issues * print parentheses * 01 is invalid syntax (it was octal in py2) * [x for i in 1, 2] is invalid syntax * six moves Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-16 00:39:32 +02:00
Adrian Tschira	2a3c33ff03	Use six.moves.urlparse ... The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 21:22:43 +02:00
Adrian Tschira	cb9cdfecd0	Add some more variables to the unittest config ... These worked accidentally before (python2 doesn't complain if you compare incompatible types) but under py3 this blows up spectacularly Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:36:39 +02:00
Adrian Tschira	1515560f5c	Use str(e) instead of e.message ... Doing this I learned e.message was pretty shortlived, added in 2.6, they realized it was a bad idea and deprecated it in 2.7 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:32:42 +02:00
Erik Johnston	e939f3bca6	Fix tests	2018-04-11 14:37:11 +01:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	87478c5a60	Merge pull request #3061 from NotAFile/add-some-byte-strings ... Add b prefixes to some strings that are bytes in py3	2018-04-09 23:54:05 +01:00
Richard van der Hoff	664adb4236	Merge pull request #3016 from silkeh/improve-service-lookups ... Improve handling of SRV records for federation connections	2018-04-09 23:40:06 +01:00
Richard van der Hoff	01afc563c3	Fix overzealous cache invalidation ... Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.	2018-04-05 16:24:04 +01:00
Adrian Tschira	6168351877	Add b prefixes to some strings that are bytes in py3 ... This has no effect on python2 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-04 13:48:51 +02:00
Silke	72251d1b97	Remove address resolution of hosts in SRV records ... Signed-off-by: Silke Hofstra <silke@slxh.eu>	2018-04-04 12:26:50 +02:00
Krombel	91ea0202e6	move handling of auto_join_rooms to RegisterHandler ... Currently the handling of auto_join_rooms only works when a user registers itself via public register api. Registrations via registration_shared_secret and ModuleApi do not work This auto_joins the users in the registration handler which enables the auto join feature for all 3 registration paths. This is related to issue #2725 Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-14 16:45:37 +01:00
Erik Johnston	cb9f8e527c	s/replication_client/federation_client/	2018-03-13 13:26:52 +00:00
Erik Johnston	cea462e285	s/replication_server/federation_server	2018-03-13 13:22:21 +00:00
Erik Johnston	6ea27fafad	Fix tests	2018-03-13 10:55:47 +00:00
Erik Johnston	631a73f7ef	Fix tests	2018-03-13 10:39:19 +00:00
Richard van der Hoff	2c911d75e8	Fix comment typo	2018-03-05 12:24:49 +00:00
Richard van der Hoff	c818fcab11	Test and fix find_first_stream_ordering_after_ts ... It seemed to suffer from a bunch of off-by-one errors.	2018-03-05 12:04:02 +00:00
Erik Johnston	784f036306	Move RoomMemberHandler out of Handlers	2018-03-01 14:36:50 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	25634ed152	Fix test	2018-02-20 12:40:44 +00:00
Erik Johnston	e440e28456	Fix unit tests	2018-02-20 11:41:40 +00:00
Erik Johnston	3a061cae26	Fix unit test	2018-02-15 16:31:59 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Erik Johnston	25c0a020f4	Updates tests	2018-02-05 16:01:48 +00:00
Richard van der Hoff	db91e72ade	Merge pull request #2844 from matrix-org/rav/evicted_metrics ... montoring metrics for number of cache evictions	2018-02-05 16:34:36 +01:00
Richard van der Hoff	bc496df192	report metrics on number of cache evictions	2018-02-05 15:34:01 +00:00
Erik Johnston	a1beca0e25	Fix broken unit test for media storage	2018-02-05 12:44:03 +00:00
Erik Johnston	1f881e0746	Merge pull request #2791 from matrix-org/erikj/media_storage_refactor ... Ensure media is in local cache before thumbnailing	2018-02-05 11:28:52 +00:00
Richard van der Hoff	6da4c4d3bd	Factor out resolve_state_groups to a separate handler ... We extract the storage-independent bits of the state group resolution out to a separate functiom, and stick it in a new handler, in preparation for its use from the storage layer.	2018-02-01 12:40:04 +00:00
Richard van der Hoff	d1f3490e75	Add tests for user directory search	2018-01-27 17:21:57 +00:00
Richard van der Hoff	88b9c5cbf0	Make it possible to run tests against postgres	2018-01-27 17:15:24 +00:00
Richard van der Hoff	d7eacc4f87	Create dbpool as normal in tests ... ... instead of creating our own special SQLiteMemoryDbPool, whose purpose was a bit of a mystery. For some reason this makes one of the tests run slightly slower, so bump the sleep(). Sorry.	2018-01-27 17:15:15 +00:00
Richard van der Hoff	b178eca261	Run on_new_connection for unit tests ... Configure the connectionpool used for unit tests to run the `on_new_connection` function.	2018-01-27 17:06:04 +00:00
Richard van der Hoff	6e9bf67f18	Remove unused/bitrotted MemoryDataStore ... This isn't used, and looks thoroughly bitrotted.	2018-01-26 00:35:15 +00:00
Richard van der Hoff	2b91846497	Remove spurious unittest.DEBUG	2018-01-26 00:34:27 +00:00
Richard van der Hoff	b2da272b77	Merge pull request #2821 from matrix-org/rav/matthew_test_fixes ... Matthew's fixes to the unit tests	2018-01-22 20:43:56 +00:00
Matthew Hodgson	ab9f844aaf	Add federation_domain_whitelist option (#2820) ... Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network	2018-01-22 19:11:18 +01:00
Richard van der Hoff	5c431f421c	Matthew's fixes to the unit tests ... Extracted from https://github.com/matrix-org/synapse/pull/2820	2018-01-22 16:46:16 +00:00
Matthew Hodgson	d84f65255e	Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid ... add registrations_require_3pid and allow_local_3pids	2018-01-22 13:57:22 +00:00
Matthew Hodgson	81d037dbd8	mock registrations_require_3pid	2018-01-19 00:28:08 +00:00
Erik Johnston	ce4f66133e	Add unit tests	2018-01-18 16:32:11 +00:00
Erik Johnston	b6dc7044a9	Merge pull request #2804 from matrix-org/erikj/file_consumer ... Add decent impl of a FileConsumer	2018-01-18 16:31:33 +00:00
Erik Johnston	1432f7ccd5	Move test stuff to tests	2018-01-18 11:57:57 +00:00
Erik Johnston	bc67e7d260	Add decent impl of a FileConsumer ... Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind	2018-01-17 16:43:03 +00:00
Richard van der Hoff	44a498418c	Optimise LoggingContext creation and copying ... It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.	2018-01-16 15:49:42 +00:00
Richard van der Hoff	a767f06e3f	Merge pull request #2765 from matrix-org/rav/fix_room_uts ... Fix flaky test_rooms UTs	2018-01-09 14:21:03 +00:00
Richard van der Hoff	f8fa5ae4af	enable twisted delayedcall debugging in UTs	2018-01-09 12:06:45 +00:00
Richard van der Hoff	374c4d4ced	Remove dead code ... pointless function is pointless	2018-01-09 12:06:45 +00:00
Richard van der Hoff	142fb0a7d4	Disable user_directory updates for UTs ... Fix flakiness in the UTs caused by the user_directory being updated in the background	2018-01-09 12:06:45 +00:00
Richard van der Hoff	0211464ba2	Fix broken config UTs ... https://github.com/matrix-org/synapse/pull/2755 broke log-config generation, which in turn broke the unit tests.	2018-01-09 11:28:33 +00:00