8705 Commits

Author	SHA1	Message	Date
Matthew Hodgson	e72234f6bd	fix tests	2018-06-28 20:56:07 +01:00
Matthew Hodgson	f4f1cda928	add ip_range_whitelist parameter to limit where ASes can connect from	2018-06-28 20:32:00 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Amber Brown	72d2143ea8	Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454)	2018-06-28 11:04:18 +01:00
Amber Brown	99800de63d	try and clean up	2018-06-27 11:40:27 +01:00
Amber Brown	f03a5d1a17	pep8	2018-06-27 11:38:14 +01:00
Amber Brown	a7ecf34b70	cleanups	2018-06-27 11:36:03 +01:00
Amber Brown	77078d6c8e	handle federation not telling us about prev_events	2018-06-27 11:27:32 +01:00
Jeroen	95341a8f6f	take idna implementation from twisted	2018-06-26 21:15:14 +02:00
Jeroen	b7f34ee348	allow self-signed certificates	2018-06-26 20:41:05 +02:00
Matthew Hodgson	8057489b26	Revert "Try to not use as much CPU in the StreamChangeCache"	2018-06-26 18:09:01 +01:00
Matthew Hodgson	d91efb06cf	Merge pull request #3451 from matrix-org/hawkowl/sorteddict-api ... Try to not use as much CPU in the cache	2018-06-26 17:49:55 +01:00
Amber Brown	1202508067	fixes	2018-06-26 17:29:01 +01:00
Amber Brown	bd3d329c88	fixes	2018-06-26 17:28:12 +01:00
Amber Brown	abfe4b2957	try and make loading items from the cache faster	2018-06-26 17:25:34 +01:00
David Baker	028490afd4	Fix error on deleting users pending deactivation ... Use simple_delete instead of simple_delete_one as commented	2018-06-26 10:52:52 +01:00
Matthew Hodgson	c7f6b420ae	Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api ... add GDPR erase param to deactivate API	2018-06-26 10:43:14 +01:00
Matthew Hodgson	9570aa82eb	update doc for deactivate API	2018-06-26 10:42:50 +01:00
Matthew Hodgson	1e788db430	add GDPR erase param to deactivate API	2018-06-26 10:26:54 +01:00
Amber Brown	1d62c4a127	Merge pull request #3438 from turt2live/travis/dont-print-access-tokens-in-logs ... Stop including access tokens in warnings in the log	2018-06-26 09:55:55 +01:00
Erik Johnston	d72fb9a448	Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts ... allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 20:14:18 +01:00
Erik Johnston	df48f7ef37	Actually fix it	2018-06-25 20:03:41 +01:00
Erik Johnston	a0e8a53c6d	Comment	2018-06-25 19:57:38 +01:00
Erik Johnston	7bdc5c8fa3	Fix bug with assuming wrong type	2018-06-25 19:56:02 +01:00
Erik Johnston	ea7a9c0483	Add fast path to _filter_events_for_server ... Most rooms have a trivial history visibility like "shared" or "world_readable", especially large rooms, so lets not bother getting the full membership of those rooms in that case.	2018-06-25 19:49:13 +01:00
Matthew Hodgson	0269367f18	allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 17:56:10 +01:00
Matthew Hodgson	784189b1f4	typos	2018-06-25 17:37:16 +01:00
Matthew Hodgson	6eb861b67f	typo	2018-06-25 17:37:16 +01:00
Erik Johnston	947fea67cb	Need to pass reactor to endpoint fac	2018-06-25 15:22:57 +01:00
Erik Johnston	33fdcfa957	Merge pull request #3441 from matrix-org/erikj/redo_erasure ... Fix user erasure and re-enable	2018-06-25 14:37:01 +01:00
Erik Johnston	eb50c44eaf	Add UserErasureWorkerStore to workers	2018-06-25 14:22:24 +01:00
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	2018-06-25 14:08:28 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Jeroen	07b4f88de9	formatting changes for pep8	2018-06-25 12:31:16 +02:00
Jeroen	3d605853c8	send SNI for federation requests	2018-06-24 22:38:43 +02:00
Travis Ralston	ec1e799e17	Don't print invalid access tokens in the logs ... Tokens shouldn't be appearing the logs, valid or invalid. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-24 12:17:01 -06:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Richard van der Hoff	516f884176	Merge pull request #3435 from matrix-org/rav/fix_event_push_actions_tablescan ... Fix event_push_actions tablescan when reinserting events	2018-06-22 15:57:22 +01:00
Mark Haines	9850f66abe	Deleting from event_push_actions needs to use an index	2018-06-22 15:54:48 +01:00
Erik Johnston	28ddc6cfbe	Also log number of events for serach context	2018-06-22 15:42:11 +01:00
Erik Johnston	4b4cec3989	Add some logging to search queries	2018-06-22 15:42:11 +01:00
Richard van der Hoff	200e11c5bf	Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable ... Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:18:51 +01:00
Erik Johnston	f8272813a9	Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:12:26 +01:00
Richard van der Hoff	1d7ad11747	Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation ... Make push actions rotation configurable	2018-06-22 15:07:31 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	b4a5d767a9	Merge pull request #3428 from matrix-org/erikj/persisted_pdu ... Simplify get_persisted_pdu	2018-06-22 14:47:43 +01:00
Erik Johnston	f79abda87f	Merge pull request #3427 from matrix-org/erikj/remove_filters ... remove dead filter_events_for_clients	2018-06-22 14:45:42 +01:00
Erik Johnston	75dc3ddeab	Make push actions rotation configurable	2018-06-22 14:44:37 +01:00
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	2018-06-22 11:52:07 +01:00
Richard van der Hoff	240f192523	Merge pull request #3382 from matrix-org/rav/optimise_state_groups ... Optimise state_group_cache update	2018-06-22 11:20:20 +01:00
Richard van der Hoff	70e6501913	Merge pull request #3419 from matrix-org/rav/events_per_request ... Log number of events fetched from DB	2018-06-22 11:17:56 +01:00
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	2018-06-22 10:42:28 +01:00
Amber Brown	9a685d60ae	Merge pull request #3418 from matrix-org/rav/fix_metric_desc ... Fix description of "python_gc_time" metric	2018-06-22 09:43:26 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	cbbfaa4be8	Fix description of "python_gc_time" metric	2018-06-21 10:02:42 +01:00
Amber Brown	c2eff937ac	Populate synapse_federation_client_sent_pdu_destinations:count again (#3386)	2018-06-21 09:39:58 +01:00
Amber Brown	99b77aa829	Fix tcp protocol metrics naming (#3410)	2018-06-21 09:39:27 +01:00
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	2018-06-21 06:15:03 +01:00
Richard van der Hoff	aff3d76920	Merge pull request #3416 from matrix-org/rav/restart_indicator ... Write a clear restart indicator in logs	2018-06-20 18:14:08 +01:00
Richard van der Hoff	02bfc581f8	Merge pull request #3399 from costacruise/master ... Add error code to room creation error	2018-06-20 17:26:25 +01:00
Richard van der Hoff	245d53d32a	Write a clear restart indicator in logs ... I'm fed up with never being able to find the point a server restarted in the logs.	2018-06-20 15:33:14 +01:00
Amber Brown	f6c4d74f96	Fix inflight requests metric (incorrect name & traceback) (#3413)	2018-06-20 11:18:57 +01:00
Matthew Hodgson	ccfdaf68be	spell gauge correctly	2018-06-16 07:10:34 +01:00
Richard van der Hoff	9a793f861c	Merge branch 'master' into develop	2018-06-14 16:36:01 +01:00
Richard van der Hoff	53969e1960	SECURITY UPDATE: Prevent unauthorised users from setting state events in a room ... when there is no `m.room.power_levels` event in force in the room. (PR #3397) Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbIop9AAoJEIofk9V1tejV9lsIAJVH0l5dXROmy1KH/zt16AUA CXa6Vv4Vyo6hKad/fZ81OZVRr5ChK/TvbIJVn/SA/muCfdoIFdxhT8eo/pXzO2UW zReuLsDhAg+gSvpNus37oWj2FVsAE1HYDZ60lfaapAdZnkFit68d5DQZjO6nZHHA YUXcU3GUwj0ZYuUzFzYKMLu6uNNasNkN8h6SS2lF7Bm4JaKDW+mFMfCyJwdIVSEh BGhHoVpXdxFysD9s6Mwxqrz3KKg1Jtp7idDkk0x2S2Eh+gxyiDQQokv0oQ3+0+HG sgy5Iz2t2CkpS02/j+LOvAZljTmnD0bXu3srGR+25StsoDFP038Am3bfQwtD190= =9jsT -----END PGP SIGNATURE----- Merge tag 'v0.31.2' SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no `m.room.power_levels` event in force in the room. (PR #3397) Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.	2018-06-14 16:35:33 +01:00
Richard van der Hoff	7e1c616452	v0.31.2	2018-06-14 16:24:32 +01:00
Richard van der Hoff	61ab08a197	Merge pull request #3397 from matrix-org/rav/adjust_auth_rules ... Adjust event auth rules when there is no PL event	2018-06-14 16:09:13 +01:00
Michael Wagner	19cd3120ec	Add error code to room creation error ... This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom	2018-06-14 14:08:40 +02:00
Richard van der Hoff	5c9afd6f80	Make default state_default 50 ... Make it so that, before there is a power-levels event in the room, you need a power level of at least 50 to send state. Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192	2018-06-14 12:38:09 +01:00
Richard van der Hoff	52423607bd	Clarify interface for event_auth ... stop pretending that it returns a boolean, which just almost gave me a heart attack.	2018-06-14 12:26:17 +01:00
Amber Brown	f116f32ace	add a last seen metric (#3396)	2018-06-14 20:26:59 +10:00
Richard van der Hoff	557b686eac	Refactor get_send_level to take a power_levels event ... it makes it easier for me to reason about	2018-06-14 11:26:27 +01:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Amber Brown	0fde1896cd	Merge pull request #3389 from turt2live/travis/name_metrics ... Use the correct flag (enable_metrics) when warning about an incorrect metrics setup	2018-06-13 23:50:10 +10:00
Michael Telatynski	94700e55fa	if inviter_display_name == ""||None then default to inviter MXID ... to prevent email invite from "None"	2018-06-13 10:31:01 +01:00
Travis Ralston	45768d1640	Use the RegistryProxy for appservices too ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-12 12:55:48 -06:00
Travis Ralston	12285a1a76	The flag is named enable_metrics, not collect_metrics ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-12 12:51:31 -06:00
Richard van der Hoff	b6faef2ad7	Filter out erased messages ... Redact any messges sent by erased users.	2018-06-12 09:53:18 +01:00
Richard van der Hoff	f1023ebf4b	mark accounts as erased when requested	2018-06-12 09:53:18 +01:00
Richard van der Hoff	3ff8a619f5	UserErasureStore ... to store which users have been erased	2018-06-12 09:52:22 +01:00
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	bd348f0af6	remove dead filter_events_for_clients ... This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	eb32b2ca20	Optimise state_group_cache update ... (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.	2018-06-11 22:56:11 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	d6cc369205	fix idiotic typo in state res	2018-06-11 14:43:55 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Vincent Breitmoser	b800834351	add note that the affinity package is required for the cpu_affinity setting	2018-06-09 22:50:29 +02:00
Neil Johnson	ed5a0780a4	Merge branch 'master' into develop	2018-06-08 15:47:11 +01:00
Neil Johnson	1032393dfb	Changes in synapse v0.31.1 (2018-06-08) ... ======================================= v0.31.1 fixes a security bug in the ``get_missing_events`` federation API where event visibility rules were not applied correctly. We are not aware of it being actively exploited but please upgrade asap. Bug Fixes: * Fix event filtering in get_missing_events handler (PR #3371) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsalP8ACgkQCTxDAAxP S/RADw/+NeDu0LjVpS5Uc4ElHgRBuFSm6l2i4z8rZBBlKSYnuq0Em4WMvLloi/JF iAvTOYE7OjmF+gNvmsdH1N7hc1lKdQ2gAlpvaQR/5Qz9NtOVmM3WPZxS7n5jZHvD hVSxeO9+GQOwK7rJorqrrsnWHQt0OkLHV6WThFdrgZb1JjWCUDTvw+Hei2uMX2aq y2mkMG4TLStHwMvL2qw0h+hFtXywXI796qJR73ZxbEn24YD+kOXeEVkIFi2LT0Pj cgkg7WWT32JD43/ioumLupZuhCmpRyxn4fi5gIKpXe5kiLsxOdApzNQSwmoJ+WA8 7zlrWY+0QDN4pbA5ESLitWSWAT50Ul//uM4nwmM4xEBPHdljXvKyHPsaSCeKLvT9 RT8pc41TQAqSshlXF8zgIAtStnF3oGel3EBBl1mmM9Un1ULnBFwWDZhlIm+ZZGhJ MWoAWNG7j8AQuy0BTUAUr76x7t+/cdSqDuyVl1GO1tbDh0DUWoHZGXCUKrAXnn2T SbiFigwOLvEADbvkW7L9Je9CVOi2V5Pg/32X9O8YMEiSz+j5PQEiGefyVh/I/QvV Ha/atRpZF2OZ+XUOO5DLZMP/XCXpVgvHuskzfU6LvvVQCXgExuJhRD1PrRGeBaWr zjJW+rmY+VeredqX7QkTB3XOGLMLGJfx2FeSMb+j0w3a7/iFj+Q= =J80S -----END PGP SIGNATURE----- Merge tag 'v0.31.1' Changes in synapse v0.31.1 (2018-06-08) ======================================= v0.31.1 fixes a security bug in the ``get_missing_events`` federation API where event visibility rules were not applied correctly. We are not aware of it being actively exploited but please upgrade asap. Bug Fixes: * Fix event filtering in get_missing_events handler (PR #3371)	2018-06-08 15:46:18 +01:00
Neil Johnson	0eb4722932	changelog a bump version	2018-06-08 15:21:46 +01:00
Richard van der Hoff	c6b1441c52	Fix event filtering in get_missing_events handler	2018-06-08 14:15:31 +01:00
David Baker	0e505b1913	Merge pull request #3372 from matrix-org/rav/better_verification_logging ... Try to log more helpful info when a sig verification fails	2018-06-08 13:32:02 +01:00
Richard van der Hoff	e82db24a0e	Try to log more helpful info when a sig verification fails ... Firstly, don't swallow the reason for the failure Secondly, don't assume all exceptions are verification failures Thirdly, log a bit of info about the key being used if debug is enabled	2018-06-08 12:13:08 +01:00
Richard van der Hoff	0834b49c6a	Fix event filtering in get_missing_events handler	2018-06-08 11:34:46 +01:00
Richard van der Hoff	8503dd0047	Remove event re-signing hacks ... These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.	2018-06-07 16:08:29 +01:00
Will Hunt	13d211edc1	Merge pull request #3344 from Half-Shot/hs/as-metrics ... Add metrics to track appservice transactions	2018-06-07 11:37:12 +01:00
Richard van der Hoff	1152f495a0	Merge pull request #3363 from matrix-org/rav/fix_purge ... Fix event-purge-by-ts admin API	2018-06-07 11:34:46 +01:00
Richard van der Hoff	e2acf536d4	Merge pull request #3355 from matrix-org/rav/fix_federation_backfill ... Fix federation backfill from sqlite servers	2018-06-07 10:18:01 +01:00
Amber Brown	0160f6601f	Merge pull request #3356 from matrix-org/rav/add_missing_attr_dep ... Make sure that attr is installed	2018-06-07 16:33:30 +10:00
Richard van der Hoff	f4caf3f83d	fix log	2018-06-07 00:26:38 +01:00
Richard van der Hoff	0546715c18	Fix event-purge-by-ts admin API ... This got completely broken in 0.30. Fixes #3300.	2018-06-07 00:15:49 +01:00
Richard van der Hoff	57e3f923d2	Add missing dependency on attr ... We've rcently added a dep on `attr`. I don't know why the CI didn't pick this up, but we should make it explicit anyway.	2018-06-06 17:12:41 +01:00
Richard van der Hoff	d3a8c9c55e	Fix sql error in _get_state_groups_from_groups ... If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.	2018-06-06 14:19:01 +01:00
Neil Johnson	fef6c2cdcc	Merge branch 'master' into develop	2018-06-06 12:28:15 +01:00
Neil Johnson	752b7b32ed	Changes in synapse v0.31.0 (2018-06-06) ... ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx! -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsXxJAACgkQCTxDAAxP S/SJTg//Wtr+Qop9LJh2/leAYXpyqW6P7Ftak0w3aJ3KL3+tYg32yYNoRADCqbp3 LkrHu8MwbZagHjRUyEWNfDk4jbfq5fwh0JVGmYuUKhG9aF0HYyytKkbW79YzuhdQ dfHj9x0xSBOUvgt/husloZSDy0VHC6uyQSAFgFDyHS2y7RPAiGstqLGByv0ciZOk pO7TdjkUQcx4Ps7Wgip31NuHy3GY2int6f540pUXoZHLXs7RkfqS2cpF9Z/sTXJ4 xDLiY7uYNsTcCblwqaiijY5c90xwRB2vLs5CJdKFgyB6PNgg/2wHJqP/WHHEj+F8 BoSm3Ts7NXQf23pP9CXICe7vXX3J+ruOnC7FOSRobr6KGjn6DUrIZxo1ZepTwpp9 DIq+1eOFKKjwLQM3Jdi8WBCP63LhYXrTZxreke3jpwdcD7oIO9v6/e9J1gU5xHWa Izg+YnWn1JLfq/X8T7YTZddUXGGPkH5i6LZKKkyY8u7LkJ4WR7syuAceUzkOOIAq UWO0uEV7IiLnZzZGTtNIlEqtuklmVQTm6bvAgTPabai2JQyngFpH5M/5mPpVSiLV QRLwaM56c+5GGZJWef8vxdGeYn+8rFI/UUniJ7358kLJF2IHsxlZu8J0ZZO2HWI2 ze5Kz0AWRzXLhWzq62Qb2dsiGySrZ7hng1tDxLak2IiusY+9SjM= =Mz9U -----END PGP SIGNATURE----- Merge tag 'v0.31.0' Changes in synapse v0.31.0 (2018-06-06) ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!	2018-06-06 12:27:33 +01:00
Neil Johnson	61134debdc	bump version and changelog	2018-06-06 11:26:21 +01:00
Richard van der Hoff	ad459a106c	Merge pull request #3349 from t3chguy/redact_as_request_token ... Redact AS tokens in log (fixes to #3327)	2018-06-06 10:58:07 +01:00
Michael Telatynski	592c162516	also redact __str__ of ApplicationService used for logging	2018-06-06 10:35:29 +01:00
Michael Telatynski	330432031b	redact_uri in two missed log paths	2018-06-06 10:25:48 +01:00
David Baker	bf54c1cf6c	pep8	2018-06-06 10:15:33 +01:00
David Baker	3e4bc4488c	More doc fixes	2018-06-06 09:44:10 +01:00
Richard van der Hoff	592ee217a3	Merge commit 'b7e7fd2' into release-v0.31.0	2018-06-06 07:02:02 +01:00
Ivan Shapovalov	c88d50aa8f	federation/send_queue.py: fix usage of sortedcontainers.SortedDict	2018-06-06 02:45:18 +03:00
Richard van der Hoff	e316407b5d	Merge pull request #3327 from t3chguy/redact_as_request_token ... Strip `access_token` from outgoing requests	2018-06-05 19:08:46 +01:00
Michael Telatynski	e6cbf47773	factor out uri redaction into a method on http	2018-06-05 18:31:40 +01:00
David Baker	607bd27c83	fix pep8	2018-06-05 18:10:35 +01:00
David Baker	d62162bbec	doc fixes	2018-06-05 18:09:13 +01:00
Richard van der Hoff	522bd3c8a3	Merge remote-tracking branch 'origin/master' into develop	2018-06-05 17:42:49 +01:00
Will Hunt	d6e3c2c79b	Let's try labels instead of label, that might work	2018-06-05 17:30:45 +01:00
Amber Brown	f7869f8f8b	Port to sortedcontainers (with tests!) (#3332)	2018-06-06 00:13:57 +10:00
Will Hunt	604cff1a06	Add metrics to track appservice transactions	2018-06-05 13:21:30 +01:00
Richard van der Hoff	f29b41fde9	Merge pull request #3324 from matrix-org/rav/remove_dead_method ... Remove was_forgotten_at	2018-06-04 18:18:08 +01:00
Richard van der Hoff	28b0490dfd	Merge pull request #3334 from matrix-org/rav/cache_factor_override ... Cache factor override system for specific caches	2018-06-04 17:19:33 +01:00
Richard van der Hoff	b7e7fd2d0e	Fix replication metrics ... fix bug introduced in #3256	2018-06-04 16:23:05 +01:00
Neil Johnson	244ab974e7	bump version and changelog	2018-06-04 16:09:58 +01:00
Richard van der Hoff	694968fa81	Hopefully, fix LaterGuage error handling	2018-06-04 15:59:14 +01:00
Erik Johnston	042eedfa2b	Add hacky cache factor override system	2018-06-04 15:39:28 +01:00
David Baker	c5930d513a	Docstring	2018-06-04 12:05:58 +01:00
David Baker	6a29e815fc	Fix comment	2018-06-04 12:01:23 +01:00
David Baker	e44150a6de	Missing yield	2018-06-04 12:01:13 +01:00
David Baker	f731e42baf	docstring	2018-06-04 12:00:51 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Matthew Hodgson	5f6122fe10	more comments	2018-06-04 00:08:52 +03:00
Ivan Shapovalov	7d9d75e4e8	federation/send_queue.py: fix usage of LaterGauge ... Fixes a startup crash due to commit df9f72d9e5 "replacing portions".	2018-06-03 14:16:17 +03:00
Michael Telatynski	09503126df	Strip access_token from outgoing requests using existing regex	2018-06-02 23:25:13 +01:00
Richard van der Hoff	c1f4118bb6	Remove was_forgotten_at ... This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *	2018-06-01 18:21:49 +01:00
Richard van der Hoff	a9e97dcd65	Merge pull request #3317 from thegcat/feature/3312-add_ipv6_to_blacklist_example_config ... Add private IPv6 addresses to example config for url preview blacklist	2018-06-01 14:45:14 +01:00
Neil Johnson	71477f3317	Merge pull request #3264 from matrix-org/neil/sign-up-stats ... daily user type phone home stats	2018-06-01 13:42:01 +00:00
Richard van der Hoff	9f797a24a4	Handle RRs which arrive before their events	2018-06-01 14:01:43 +01:00
Richard van der Hoff	857e6fd8b6	Ignore depth when updating read-receipts ... Order read receipts by stream ordering instead of depth	2018-06-01 12:18:11 +01:00
Felix Schäfer	4ef76f3ac4	Add private IPv6 addresses to preview blacklist #3312 ... The added addresses are expected to be local or loopback addresses and shouldn't be spidered for previews. Signed-off-by: Felix Schäfer <felix@thegcat.net>	2018-06-01 12:18:35 +02:00
Neil Johnson	4986b084f8	remove unnecessary INSERT	2018-06-01 10:50:40 +01:00
Richard van der Hoff	7e15410f02	Enforce the specified API for report_event ... as per https://matrix.org/docs/spec/client_server/unstable.html#post-matrix-client-r0-rooms-roomid-report-eventid	2018-05-31 18:17:11 +01:00
Amber Brown	febe0ec8fd	Run Prometheus on a different port, optionally. (#3274)	2018-05-31 19:04:50 +10:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Richard van der Hoff	219c2a322b	remove trailing whitespace	2018-05-30 19:42:19 +01:00
Richard van der Hoff	2e4be8bfd9	fix english and wrap comment	2018-05-30 19:24:12 +01:00
Amber Brown	872cf43516	Merge pull request #3303 from NotAFile/py3-memoryview ... use memoryview in py3	2018-05-30 12:51:42 +10:00
Amber Brown	debff7ae09	Merge pull request #3281 from NotAFile/py3-six-isinstance ... remaining isintance fixes	2018-05-30 12:44:46 +10:00
Richard van der Hoff	34b85df7f5	Update some comments and docstrings in SyncHandler	2018-05-29 22:31:18 +01:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Neil Johnson	ab0ef31dc7	create users index on creation_ts	2018-05-29 17:51:08 +01:00
Neil Johnson	558f3d376a	create index in background	2018-05-29 17:47:55 +01:00
Neil Johnson	c379acd4fd	bump version	2018-05-29 17:47:28 +01:00
Adrian Tschira	4b9d0cde97	add remaining memoryview changes	2018-05-29 17:42:43 +02:00
Adrian Tschira	7873cde526	pep8	2018-05-29 17:35:55 +02:00
Adrian Tschira	1afafb3497	use memoryview in py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-29 17:14:34 +02:00
Matthew Hodgson	9bbb9f5556	add lazy_load_members to the filter json schema	2018-05-29 04:26:10 +01:00
Matthew Hodgson	8df7bad839	pep8	2018-05-29 02:49:36 +01:00
Matthew Hodgson	ff1bc0a279	pep8	2018-05-29 02:32:15 +01:00
Matthew Hodgson	0a240ad36e	disable CPUMetrics if no /proc/self/stat ... fixes build on macOS again	2018-05-29 02:23:30 +01:00
Matthew Hodgson	b69ff33d9e	disable CPUMetrics if no /proc/self/stat ... fixes build on macOS again	2018-05-29 02:22:27 +01:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	57ad76fa4a	fix up tests	2018-05-28 19:51:53 +10:00
Amber Brown	3ef5cd74a6	update to more consistently use seconds in any metrics or logging	2018-05-28 19:39:27 +10:00
Amber Brown	5c40ce3777	invalid syntax :(	2018-05-28 19:16:09 +10:00
Amber Brown	357c74a50f	add comment about why unreg	2018-05-28 19:14:41 +10:00
Amber Brown	a2eb5db4a0	update metrics to be in seconds	2018-05-28 19:10:27 +10:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Ruben Barkow	08ea5fe635	add link to thorough instruction how to configure consent	2018-05-25 23:19:55 +02:00
Richard van der Hoff	66bdae986f	Fix default for send_server_notice_to_guests ... bool("False") == True...	2018-05-25 11:42:05 +01:00
Richard van der Hoff	ba1b163590	Avoid sending consent notice to guest users ... we think it makes sense not to send the notices to guest users.	2018-05-25 11:36:43 +01:00
Richard van der Hoff	41921ac01b	Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room ... Let users leave the server notice room after joining	2018-05-25 11:15:45 +01:00
Richard van der Hoff	757ed27258	Let users leave the server notice room after joining ... They still can't reject invites, but we let them leave it.	2018-05-25 11:07:21 +01:00
Adrian Tschira	4ee4450d66	fix recursion error	2018-05-24 21:44:10 +02:00
Amber Brown	9c36c150e7	Merge pull request #3283 from NotAFile/py3-state ... py3-ize state.py	2018-05-24 14:23:33 -05:00
Amber Brown	cc1349c06a	Merge pull request #3279 from NotAFile/py3-more-iteritems ... more six iteritems	2018-05-24 14:23:13 -05:00
Adrian Tschira	0e61705661	py3-ize state.py	2018-05-24 20:59:00 +02:00
Adrian Tschira	dd068ca979	remaining isintance fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:55:08 +02:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Adrian Tschira	6c16a4ec1b	more iteritems	2018-05-24 20:19:06 +02:00
Amber Brown	7ea07c7305	Merge pull request #3278 from NotAFile/py3-storage-base ... Py3 storage/_base.py	2018-05-24 13:08:09 -05:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
Amber Brown	c4fb15a06c	Merge pull request #3246 from NotAFile/py3-repr-string ... use repr, not str	2018-05-24 13:00:20 -05:00
Amber Brown	36501068d8	Merge pull request #3247 from NotAFile/py3-misc ... Misc Python3 fixes	2018-05-24 12:58:37 -05:00
Amber Brown	2aff6eab6d	Merge pull request #3245 from NotAFile/batch-iter ... Add batch_iter to utils	2018-05-24 12:54:12 -05:00
Adrian Tschira	095292304f	Py3 storage/_base.py ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 18:24:12 +02:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	ecc4b88bd1	Merge pull request #3277 from matrix-org/dbkr/remove_from_user_dir ... Remove users from user directory on deactivate	2018-05-24 16:12:12 +01:00
Erik Johnston	46345187cc	Merge pull request #3243 from NotAFile/py3-six-3 ... Replace some more comparisons with six	2018-05-24 16:08:57 +01:00
Neil Johnson	037c6db85d	Merge branch 'master' into develop	2018-05-24 16:03:44 +01:00
David Baker	7a1af504d7	Remove users from user directory on deactivate	2018-05-24 15:59:58 +01:00
Erik Johnston	f72d5a44d5	Merge pull request #3261 from matrix-org/erikj/pagination_fixes ... Fix federation backfill bugs	2018-05-24 14:52:03 +01:00
Erik Johnston	68399fc4de	Merge pull request #3267 from matrix-org/erikj/iter_filter ... Use iter* methods for _filter_events_for_server	2018-05-24 14:44:57 +01:00
Neil Johnson	91d95a1d8e	bump version	2018-05-24 14:05:07 +01:00
David Baker	9700d15611	pep8	2018-05-24 11:23:15 +01:00
David Baker	a21a41bad7	comment	2018-05-24 11:19:59 +01:00
David Baker	b3bff53178	Unbind 3pids when they're deleted too	2018-05-24 11:08:05 +01:00
Richard van der Hoff	6abcb5d22d	Merge pull request #3273 from matrix-org/rav/server_notices_avatar_url ... Allow overriding the server_notices user's avatar	2018-05-24 10:31:43 +01:00
Amber Brown	389dac2c15	pepeightttt	2018-05-23 13:08:59 -05:00
Amber Brown	472a5ec4e2	add back CPU metrics	2018-05-23 13:03:56 -05:00
Amber Brown	e987079037	fixes	2018-05-23 13:03:51 -05:00
Richard van der Hoff	9bf4b2bda3	Allow overriding the server_notices user's avatar ... probably should have done this in the first place, like @turt2live suggested.	2018-05-23 17:43:30 +01:00
Richard van der Hoff	a0b3946fe2	Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri	2018-05-23 16:06:03 +01:00
Richard van der Hoff	2f7008d4eb	Merge pull request #3271 from matrix-org/rav/consent_uri_in_messages ... Support for putting %(consent_uri)s in messages	2018-05-23 16:04:30 +01:00
Richard van der Hoff	dda40fb55d	fix typo	2018-05-23 15:30:26 +01:00
Richard van der Hoff	3ff6f50eac	Use the localpart in the consent uri ... ... because it's shorter.	2018-05-23 15:28:23 +01:00
Richard van der Hoff	82191b08f6	Support for putting %(consent_uri)s in messages ... Make it possible to put the URI in the error message and the server notice that get sent by the server	2018-05-23 15:24:31 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Richard van der Hoff	321f02d263	Block attempts to send server notices to remote users	2018-05-23 14:30:47 +01:00
Neil Johnson	3601a240aa	bump version and changelog	2018-05-23 12:03:23 +01:00
Erik Johnston	5aaa3189d5	s/values/itervalues/	2018-05-23 10:13:05 +01:00
Erik Johnston	0a4bca4134	Use iter* methods for _filter_events_for_server	2018-05-23 10:04:23 +01:00
Amber Brown	b6063631c3	more cleanup	2018-05-22 17:36:20 -05:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Amber Brown	071206304d	cleanup pep8 errors	2018-05-22 16:54:22 -05:00
Amber Brown	85ba83eb51	fixes	2018-05-22 16:28:23 -05:00
Amber Brown	228f1f584e	fix the test failures	2018-05-22 15:02:38 -05:00
Erik Johnston	e85b5a0ff7	Use iter* methods	2018-05-22 19:02:48 +01:00
Erik Johnston	586b66b197	Fix that states is a dict of dicts	2018-05-22 19:02:36 +01:00
Erik Johnston	a17e901f4d	Remove unused string formatting param	2018-05-22 18:24:32 +01:00
Erik Johnston	5494c1d71e	Don't support limitless pagination ... The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.	2018-05-22 18:15:21 +01:00
Neil Johnson	d8cb7225d2	daily user type phone home stats	2018-05-22 18:09:09 +01:00
hera	ad2823ee27	fix synchrotron	2018-05-22 17:47:42 +01:00
Richard van der Hoff	08bfc48abf	custom error code for not leaving server notices room	2018-05-22 17:27:27 +01:00
David Baker	0a078026ea	comment typo	2018-05-22 17:14:06 +01:00
Amber Brown	8f5a688d42	cleanups, self-registration	2018-05-22 10:56:03 -05:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Erik Johnston	cb2a2ad791	get_domains_from_state returns list of tuples	2018-05-22 16:23:39 +01:00
Richard van der Hoff	08a14b32ae	Merge pull request #3262 from matrix-org/rav/has_already_consented ... Add a 'has_consented' template var to consent forms	2018-05-22 15:35:10 +01:00
Richard van der Hoff	7b36d06a69	Add a 'has_consented' template var to consent forms ... fixes #3260	2018-05-22 14:58:34 +01:00
Richard van der Hoff	669400e22f	Enable auto-escaping for the consent templates ... ... to reduce the risk of somebody introducing an html injection attack...	2018-05-22 14:58:34 +01:00
Richard van der Hoff	b5b2d5d64b	Fix dependency on jinja2 ... Delay the import of ConsentResource, so that we can get away without jinja2 if people don't have the consent resource enabled. Fixes #3259	2018-05-22 14:03:45 +01:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	8aeb529262	Merge pull request #3236 from matrix-org/rav/consent_notice ... Send users a server notice about consent	2018-05-22 11:56:09 +01:00
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	2018-05-22 11:54:51 +01:00
Richard van der Hoff	d5dca9a04f	Move consent config parsing into ConsentConfig ... turns out we need to reuse this, so it's better in the config class.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	d14d7b8fdc	Rename 'version' param on user consent config ... we're going to use it for the version we require too.	2018-05-22 11:54:51 +01:00
Erik Johnston	7948ecf234	Comment	2018-05-22 11:39:43 +01:00
Erik Johnston	020377a550	Fix logcontext resource usage tracking	2018-05-22 11:16:07 +01:00
Erik Johnston	13a8dfba0d	Merge pull request #3252 from matrix-org/erikj/in_flight_requests ... Add in flight request metrics	2018-05-22 09:54:30 +01:00
Erik Johnston	c435b0b441	Don't store context	2018-05-22 09:34:26 +01:00
Erik Johnston	fb2806b186	Move in_flight_requests_count to be a callback metric	2018-05-22 09:31:53 +01:00
Amber Brown	fcc525b0b7	rest of the changes	2018-05-21 19:48:57 -05:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Amber Brown	c60e0d5e02	don't need the resource portion	2018-05-21 17:03:20 -05:00
Amber Brown	02c1d29133	look at the Prometheus metrics instead	2018-05-21 17:02:20 -05:00
Amber Brown	f258deffcb	remove old metrics libs	2018-05-21 17:01:15 -05:00
Richard van der Hoff	413482f578	Merge pull request #3255 from matrix-org/rav/fix_transactions ... Stop the transaction cache caching failures	2018-05-21 17:58:08 +01:00
Richard van der Hoff	6e1cb54a05	Fix logcontext leak in HttpTransactionCache ... ONE DAY I WILL PURGE THE WORLD OF THIS EVIL	2018-05-21 16:58:20 +01:00
Richard van der Hoff	6d6e7288fe	Stop the transaction cache caching failures ... The transaction cache has some code which tries to stop it caching failures, but if the callback function failed straight away, then things would happen backwards and we'd end up with the failure stuck in the cache.	2018-05-21 16:49:59 +01:00
Erik Johnston	dfa70adc33	Add in flight request metrics ... This tracks CPU and DB usage while requests are in flight, rather than when we write the response.	2018-05-21 16:23:06 +01:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Adrian Tschira	d9fe2b2d9d	Replace some more comparisons with six ... plus a bonus b"" string I missed last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:56:31 +02:00
Adrian Tschira	45b55e23d3	Add batch_iter to utils ... There's a frequent idiom I noticed where an iterable is split up into a number of chunks/batches. Unfortunately that method does not work with iterators like dict.keys() in python3. This implementation works with iterators. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:48:30 +02:00
Adrian Tschira	dcc235b47d	use stand-in value if maxint is not available ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:44 +02:00
Adrian Tschira	73cbdef5f7	fix py3 intern and remove unnecessary py3 encode ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:31 +02:00
Adrian Tschira	aafb0f6b0d	py3-ize url preview	2018-05-19 17:35:20 +02:00
Adrian Tschira	b932b4ea25	use repr, not str ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:28:42 +02:00
Neil Johnson	644aac5f73	Tighter filtering for user_daily_visits	2018-05-18 17:10:35 +01:00
Neil Johnson	ef466b3a13	fix psql compatability bug	2018-05-18 15:51:21 +01:00
Neil Johnson	2725223f08	Merge branch 'master' into develop	2018-05-18 14:07:50 +01:00
Richard van der Hoff	6d9dc67139	Merge pull request #3232 from matrix-org/rav/server_notices_room ... Infrastructure for a server notices room	2018-05-18 11:28:04 +01:00
Richard van der Hoff	ed3125b0a1	Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred ... Fix error in handling receipts	2018-05-18 11:23:11 +01:00
Richard van der Hoff	67af392712	Merge pull request #3233 from matrix-org/rav/remove_dead_code ... Remove unused `update_external_syncs`	2018-05-18 11:22:43 +01:00
Richard van der Hoff	011e1f4010	Better docstrings	2018-05-18 11:22:12 +01:00
Richard van der Hoff	26305788fe	Make sure we reject attempts to invite the notices user	2018-05-18 11:18:39 +01:00
Richard van der Hoff	d10707c810	Replace inline docstrings with "Attributes" in class docstring	2018-05-18 11:00:55 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	8b1c856d81	Fix error in handling receipts ... Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234	2018-05-18 09:15:35 +01:00
Neil Johnson	6958459b50	bump version, change log	2018-05-17 21:35:07 +01:00
Richard van der Hoff	88d3405332	fix missing yield for server_notices_room	2018-05-17 18:33:45 +01:00
Richard van der Hoff	d43d480d86	Remove unused update_external_syncs ... This method isn't used anywhere. Burninate it.	2018-05-17 18:22:19 +01:00
Richard van der Hoff	fed62e21ad	Infrastructure for a server notices room ... Server Notices use a special room which the user can't dismiss. They are created on demand when some other bit of the code calls send_notice. (This doesn't actually do much yet becuse we don't call send_notice anywhere)	2018-05-17 17:58:25 +01:00
Richard van der Hoff	f8a1e76d64	Merge pull request #3225 from matrix-org/rav/move_creation_handler ... Move RoomCreationHandler out of synapse.handlers.Handlers	2018-05-17 17:56:42 +01:00
Erik Johnston	f7906203f6	Merge pull request #3212 from matrix-org/erikj/epa_stream ... Use stream rather depth ordering for push actions	2018-05-17 12:01:21 +01:00
Richard van der Hoff	c46367d0d7	Move RoomCreationHandler out of synapse.handlers.Handlers ... Handlers is deprecated nowadays, so let's move this out before I add a new dependency on it. Also fix the docstrings on create_room.	2018-05-17 09:08:42 +01:00
Neil Johnson	85b8acdeb4	Merge branch 'master' into develop	2018-05-16 16:48:10 +01:00
Neil Johnson	3c099219e0	bump version and changelog for 0.29.0	2018-05-16 15:44:59 +01:00
Erik Johnston	680530cc7f	Clarify comment	2018-05-16 11:47:29 +01:00
Erik Johnston	43e6e82c4d	Comments	2018-05-16 11:13:31 +01:00
Neil Johnson	dc8930ea9e	Merge pull request #3163 from matrix-org/cohort_analytics ... user visit data	2018-05-16 10:09:24 +00:00
Erik Johnston	c945af8799	Move and rename variable	2018-05-16 10:52:06 +01:00
Neil Johnson	be11a02c4f	remove empty line	2018-05-16 10:45:40 +01:00
Neil Johnson	a2204cc9cc	remove unused method recurring_user_daily_visit_stats	2018-05-16 09:47:20 +01:00
Neil Johnson	31c2502ca8	style and further contraining query	2018-05-16 09:46:43 +01:00
Richard van der Hoff	8030a825c8	Merge pull request #3213 from matrix-org/rav/consent_handler ... ConsentResource to gather policy consent from users	2018-05-16 07:19:18 +01:00
Neil Johnson	c92a8aa578	pep8	2018-05-15 17:31:11 +01:00
Neil Johnson	05ac15ae82	Limit query load of generate_user_daily_visits ... The aim is to keep track of when it was last called and only query from that point in time	2018-05-15 17:01:33 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
Erik Johnston	37dbee6490	Use events_to_purge table rather than token	2018-05-15 16:23:47 +01:00
Richard van der Hoff	47815edcfa	ConsentResource to gather policy consent from users ... Hopefully there are enough comments and docs in this that it makes sense on its own.	2018-05-15 15:11:59 +01:00
Neil Johnson	aea80a0118	v0.29.0-rc1: bump version and change log	2018-05-14 15:50:57 +01:00
Neil Johnson	f077e97914	instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day	2018-05-14 13:50:58 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
Neil Johnson	977765bde2	Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics	2018-05-14 09:31:42 +01:00
Richard van der Hoff	c25d7ba12e	Merge pull request #3208 from matrix-org/rav/more_refactor_request_handler ... Set Server header in SynapseRequest	2018-05-11 16:07:47 +01:00
Erik Johnston	6406b70aeb	Use stream rather depth ordering for push actions ... This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.	2018-05-11 15:30:11 +01:00
Richard van der Hoff	23e2dfe940	Merge pull request #3209 from damir-manapov/master ... transaction_id, destination defined twice	2018-05-11 00:35:13 +01:00
Damir Manapov	db18d854cd	transaction_id, destination twice	2018-05-10 22:13:31 +03:00
Richard van der Hoff	318711e139	Set Server header in SynapseRequest ... (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.	2018-05-10 18:50:27 +01:00
David Baker	6b49628e3b	Catch failure to part user from room	2018-05-10 12:23:53 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
Richard van der Hoff	645cb4bf06	Remove redundant request_handler decorator ... This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.	2018-05-10 12:19:53 +01:00
Richard van der Hoff	09f570b935	Factor wrap_request_handler_with_logging out of wrap_request_handler ... ... so that it can be used on non-JSON endpoints	2018-05-10 12:19:52 +01:00
Richard van der Hoff	9589a1925e	Remove include_metrics param ... The metrics are now available via the request, so this is redundant and can go away at last.	2018-05-10 12:19:52 +01:00
Richard van der Hoff	49e5a613f1	Move outgoing_responses_counter handling to RequestMetrics ... it's much neater there.	2018-05-10 12:19:52 +01:00
Richard van der Hoff	b8700dd7d0	Bump requests_counter in wrapped_request_handler ... less magic	2018-05-10 12:19:52 +01:00
Richard van der Hoff	c6f730282c	Move RequestMetrics handling into SynapseRequest.processing() ... It fits quite nicely here, and opens the path to getting rid of the "include_metrics" mess.	2018-05-10 12:19:51 +01:00
Richard van der Hoff	09b29f9c4a	Make RequestMetrics take a raw time rather than a clock ... ... which is going to make it easier to move around.	2018-05-10 12:18:52 +01:00
David Baker	4d298506dd	Oops, don't call function passed to run_in_background	2018-05-10 11:57:13 +01:00
Richard van der Hoff	8460e48d06	Move request_id management into SynapseRequest	2018-05-10 11:48:17 +01:00
Richard van der Hoff	18e144fe08	Move RequestsMetrics to its own file ... This is useful in its own right, because server.py is full of stuff; but more importantly, I want to do some refactoring that will cause a circular reference as it is.	2018-05-09 19:55:03 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	5adb75bcba	Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination ... Refactor get_recent_events_for_room return type	2018-05-09 16:07:14 +01:00
Erik Johnston	d26bec8a43	Add comment to sync as to why code path is split	2018-05-09 15:56:07 +01:00
Erik Johnston	fcf55f2255	Fix returned token is no longer a tuple	2018-05-09 15:43:00 +01:00
Erik Johnston	7ce98804ff	Fix up comment	2018-05-09 15:42:39 +01:00
Erik Johnston	cddf91c8b9	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change	2018-05-09 15:32:07 +01:00
Erik Johnston	9896dab8f6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination	2018-05-09 15:31:33 +01:00
Erik Johnston	1e5280b7d0	Merge pull request #3196 from matrix-org/erikj/pagination_return ... Refactor pagination DB API to return concrete type	2018-05-09 15:27:17 +01:00
Erik Johnston	75552d2148	Update comments	2018-05-09 15:15:38 +01:00
David Baker	294e9a0c9b	Prefix internal functions	2018-05-09 15:10:37 +01:00
David Baker	46df23f581	Add the schema file	2018-05-09 15:07:54 +01:00
David Baker	52281e4c54	Indent fail	2018-05-09 15:06:16 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	c0e08dc45b	Remove unused code path from member change DB func ... The function is never called without a from_key, so we can remove all the handling for that scenario.	2018-05-09 14:31:32 +01:00
Erik Johnston	0461ef01b7	Merge pull request #3195 from matrix-org/erikj/pagination_refactor ... Refactor recent events func to use pagination func	2018-05-09 14:12:24 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	e5ab9cd24b	Don't unnecessarily require token to be stream token ... This allows calling the `get_recent_event_ids_for_room` function in more situations.	2018-05-09 11:58:35 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
Erik Johnston	05e0a2462c	Refactor pagination DB API to return concrete type ... This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.	2018-05-09 11:34:24 +01:00
Erik Johnston	7dd13415db	Remove unused from_token param	2018-05-09 10:58:16 +01:00
Erik Johnston	27cf170558	Refactor recent events func to use pagination func ... This also removes a cache that is unlikely to ever get hit.	2018-05-09 10:55:55 +01:00
Erik Johnston	1aeb5e28a9	Merge pull request #3193 from matrix-org/erikj/pagination_refactor ... Refactor /context to reuse pagination storage functions	2018-05-09 10:22:38 +01:00
Erik Johnston	23ec51c94c	Fix up comments and make function private	2018-05-09 09:55:19 +01:00
Erik Johnston	696f532453	Reuse existing pagination code for context API	2018-05-08 16:20:19 +01:00
Erik Johnston	3e6d306e94	Parse tokens before calling DB function	2018-05-08 16:18:58 +01:00
Erik Johnston	274b8c6025	Only fetch required fields from database	2018-05-08 16:15:25 +01:00
Erik Johnston	06c0d0ed08	Split paginate_room_events storage function	2018-05-08 16:14:26 +01:00
David Baker	bf98fa0864	Part user from rooms on account deactivate ... This implements this very crudely: this probably isn't viable because parting a user from all their rooms could take a long time, and if the HS gets restarted in that time the process will be aborted.	2018-05-08 15:58:35 +01:00
Richard van der Hoff	678e649b78	Merge pull request #3190 from mujx/notif-token-fix ... notifications: Convert next_token to string according to the spec	2018-05-08 13:54:47 +01:00
Konstantinos Sideris	88868b2839	notifications: Convert next_token to string according to the spec ... Currently the parameter is serialized as an integer. Signed-off-by: Konstantinos Sideris <sideris.konstantin@gmail.com>	2018-05-05 12:55:02 +03:00
Erik Johnston	6d8ec3462d	Note that label values can be anything	2018-05-03 16:25:05 +01:00
Erik Johnston	95b6912045	Fix metrics that have integer value labels	2018-05-03 15:51:04 +01:00
Richard van der Hoff	093d8c415a	Merge remote-tracking branch 'origin/develop' into rav/warn_on_logcontext_fail	2018-05-03 14:59:29 +01:00
Richard van der Hoff	0ba609dc6f	Merge pull request #3183 from matrix-org/rav/moar_logcontext_leaks ... Fix logcontext leaks in rate limiter	2018-05-03 14:58:13 +01:00
Richard van der Hoff	2117f84323	Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdown ... Fix 'Unhandled Error' logs with Twisted 18.4	2018-05-03 12:40:11 +01:00
Richard van der Hoff	a7fe62f0cb	Fix logcontext leaks in rate limiter	2018-05-03 12:31:59 +01:00
Will Hunt	2e7a94c36b	Don't abortConnection() if the transport connection has already closed.	2018-05-03 12:31:47 +01:00
Richard van der Hoff	a2aaa9cb3c	Merge pull request #3178 from matrix-org/rav/fix_request_timeouts ... fix http request timeout code	2018-05-03 11:33:26 +01:00
Erik Johnston	0a3b51c420	Merge pull request #3141 from matrix-org/erikj/fixup_state ... Refactor event storage to prepare for changes in state calculations	2018-05-03 10:39:20 +01:00
Erik Johnston	31c7c29d43	Fix up grammar	2018-05-03 10:38:58 +01:00
Richard van der Hoff	902673e356	Merge pull request #3161 from NotAFile/remove-v1auth ... Make Client-Server API return 403 for invalid token	2018-05-03 10:10:57 +01:00
Erik Johnston	53a5fdf312	Merge pull request #3175 from matrix-org/erikj/escape_metric_values ... Escape label values in prometheus metrics	2018-05-03 10:01:04 +01:00
Richard van der Hoff	1dfd650348	add missing param to cancelled_to_request_timed_out_error ... This gets two arguments, not one.	2018-05-02 22:42:36 +01:00
Erik Johnston	a41117c63b	Make _escape_character take MatchObject	2018-05-02 17:27:27 +01:00
Erik Johnston	32015e1109	Escape label values in prometheus metrics	2018-05-02 16:52:42 +01:00
Richard van der Hoff	3a42aed9a1	Merge pull request #3170 from matrix-org/rav/more_logcontext_leaks ... Fix a class of logcontext leaks	2018-05-02 16:45:51 +01:00
Richard van der Hoff	415c6b672e	Merge branch 'develop' into rav/more_logcontext_leaks	2018-05-02 16:16:01 +01:00
Richard van der Hoff	be31adb036	Fix logcontext leak in media repo ... Make FileResponder.write_to_consumer uphold the logcontext contract	2018-05-02 16:14:50 +01:00
Richard van der Hoff	f22e7cda2c	Fix a class of logcontext leaks ... So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.	2018-05-02 11:58:00 +01:00
Richard van der Hoff	a8d8bf92e0	Merge pull request #3168 from matrix-org/rav/fix_logformatter ... Fix incorrect reference to StringIO	2018-05-02 10:03:36 +01:00
Richard van der Hoff	e482f8cd85	Fix incorrect reference to StringIO ... This was introduced in 4f2f5171	2018-05-02 09:12:26 +01:00
Neil Johnson	2414178ed6	Merge branch 'master' into develop	2018-05-01 18:53:56 +01:00
Neil Johnson	8570bb84cc	Update __init__.py ... bump version	2018-05-01 18:22:53 +01:00
Richard van der Hoff	ca7211104e	Merge branch 'release-v0.28.1' into develop	2018-05-01 18:16:57 +01:00
Richard van der Hoff	d5eee5d601	Merge commit '33f469b' into release-v0.28.1	2018-05-01 18:14:18 +01:00
Richard van der Hoff	d858f3bd4e	Miscellaneous fixes to python_dependencies ... * add some doc about wtf this thing does * pin Twisted to < 18.4 * add explicit dep on six (fixes #3089)	2018-05-01 18:13:54 +01:00
Richard van der Hoff	33f469ba19	Apply some limits to depth to counter abuse ... * When creating a new event, cap its depth to 2^63 - 1 * When receiving events, reject any without a sensible depth As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI	2018-05-01 17:54:19 +01:00
Neil Johnson	dd1a832419	remove user agent from data model, will just join on user_ips	2018-05-01 12:13:49 +01:00
Neil Johnson	d0857702e8	add inidexes based on usage	2018-05-01 12:12:57 +01:00
Neil Johnson	5917562b60	10 mins seems more reasonable that every minute	2018-05-01 12:12:22 +01:00
Adrian Tschira	6495dbb326	Burminate v1auth ... This closes #2602 v1auth was created to account for the differences in status code between the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid tokens). However since those protocols were merged, this makes the r0 version/endpoint internally inconsistent, and violates the specification for the r0 endpoint. This might break clients that rely on this inconsistency with the specification. This is said to affect the legacy angular reference client. However, I feel that restoring parity with the spec is more important. Either way, it is critical to inform developers about this change, in case they rely on the illegal behaviour. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-30 22:20:43 +02:00
Will Hunt	2ad3fc36e6	Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve (#3157) ... fixes #3135 Signed-off-by: Will Hunt will@half-shot.uk	2018-04-30 16:21:11 +01:00
Krombel	576b71dd3d	add guard for None on purge_history api	2018-04-30 14:29:48 +02:00
Matthew Hodgson	99a54bf2af	Merge pull request #3129 from matrix-org/matthew/fix_group_dups ... remove duplicates from groups tables	2018-04-30 11:47:25 +01:00
Richard van der Hoff	63ae5cbf34	Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn ... Remove redundant call to preserve_fn	2018-04-30 10:23:59 +01:00
Richard van der Hoff	fdb6849b81	Merge pull request #3144 from matrix-org/rav/run_in_background_exception_handling ... Trap exceptions thrown within run_in_background	2018-04-30 10:23:02 +01:00
Richard van der Hoff	01e8a52825	Merge pull request #3102 from NotAFile/py3-attributeerror ... Make event properties raise AttributeError instead	2018-04-30 09:22:09 +01:00
Adrian Tschira	0c9db26260	add comment explaining attributeerror	2018-04-30 09:49:10 +02:00
Richard van der Hoff	950a32eb47	Merge pull request #3152 from NotAFile/py3-local-imports ... make imports local	2018-04-30 01:28:13 +01:00
Richard van der Hoff	bc2017a594	Merge pull request #3153 from NotAFile/py3-httplib ... move httplib import to six	2018-04-30 01:26:42 +01:00
Richard van der Hoff	683149c1f9	Merge pull request #3151 from NotAFile/py3-xrange-1 ... Move more xrange to six	2018-04-30 01:20:06 +01:00
Richard van der Hoff	3b0e431c82	Merge pull request #3150 from NotAFile/py3-listcomp-yield ... Don't yield in list comprehensions	2018-04-30 01:11:41 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Richard van der Hoff	2fd96727b1	Merge pull request #3085 from NotAFile/py3-config-text-mode ... Open config file in non-bytes mode	2018-04-30 01:00:23 +01:00
Richard van der Hoff	b8ee12b978	Merge pull request #3084 from NotAFile/py3-certs-byte-mode ... Open certificate files as bytes	2018-04-30 01:00:05 +01:00
Richard van der Hoff	049b0b5af2	Merge pull request #3154 from NotAFile/py3-stringio ... Replace stringIO imports with six	2018-04-30 00:59:04 +01:00
Richard van der Hoff	af3cc50511	Remove redundant call to preserve_fn ... submit_event_for_as doesn't return a deferred anyway, so this is pointless.	2018-04-30 00:48:36 +01:00
Richard van der Hoff	dbf6f28d64	Merge pull request #3155 from NotAFile/py3-bytes-1 ... more bytes strings	2018-04-30 00:38:21 +01:00
Richard van der Hoff	aab2e4da60	Merge pull request #3140 from matrix-org/rav/use_run_in_background ... Use run_in_background in preference to preserve_fn	2018-04-30 00:34:28 +01:00
Richard van der Hoff	1315d374cc	Merge pull request #3156 from NotAFile/py3-hmac-bytes ... Construct HMAC as bytes on py3	2018-04-30 00:33:20 +01:00
Richard van der Hoff	9e2601f830	Merge pull request #3108 from NotAFile/py3-six-urlparse ... Use six.moves.urlparse	2018-04-30 00:33:05 +01:00
Adrian Tschira	122593265b	Construct HMAC as bytes on py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-29 00:19:41 +02:00
Adrian Tschira	e9143b6593	more bytes strings ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-29 00:13:57 +02:00
Matthew Hodgson	adaf3ec87f	fix missing import	2018-04-28 22:39:15 +01:00
Matthew Hodgson	006e18b6bb	pep8	2018-04-28 22:32:24 +01:00
Matthew Hodgson	42c89c8215	make it work with sqlite	2018-04-28 22:27:30 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Adrian Tschira	4f2f5171b7	replace stringIO imports	2018-04-28 13:46:23 +02:00
Adrian Tschira	94f4d7f49e	move httplib import to six	2018-04-28 13:43:34 +02:00
Adrian Tschira	57b58e2174	make imports local ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:41:41 +02:00
Adrian Tschira	cdb4647a80	Don't yield in list comprehensions ... I've tried to grep for more of this with no success. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:36:30 +02:00
Adrian Tschira	a376d8f761	open log_config in text mode too ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:34:13 +02:00
Richard van der Hoff	453adf00b6	pep8; remove spurious import	2018-04-27 14:32:08 +01:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	6146332387	Merge remote-tracking branch 'origin/develop' into rav/deferred_timeout	2018-04-27 14:18:00 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	3d1ae61399	Merge branch 'develop' into rav/deferred_timeout	2018-04-27 12:54:43 +01:00
Richard van der Hoff	9d2c1b8429	Backport deferred.addTimeout ... Twisted 16.0 doesn't have addTimeout, so let's backport it.	2018-04-27 12:52:30 +01:00
Richard van der Hoff	13843f771e	Trap exceptions thrown within run_in_background ... Turn any exceptions that get thrown synchronously within run_in_background into Failures instead.	2018-04-27 12:17:13 +01:00
Richard van der Hoff	41d4b07a53	Merge pull request #3142 from matrix-org/rav/reraise ... reraise exceptions more carefully	2018-04-27 12:16:19 +01:00
Richard van der Hoff	268e40341b	Merge pull request #3136 from matrix-org/rav/fix_dependencies ... Miscellaneous fixes to python_dependencies	2018-04-27 11:48:28 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Richard van der Hoff	53494c34df	Merge pull request #3139 from matrix-org/rav/consume_errors ... Add missing consumeErrors to improve exception handling	2018-04-27 11:47:21 +01:00
Richard van der Hoff	6493b22b42	reraise exceptions more carefully ... We need to be careful (under python 2, at least) that when we reraise an exception after doing some error handling, we actually reraise the original exception rather than anything that might have been raised (and handled) during the error handling.	2018-04-27 11:40:06 +01:00
Erik Johnston	6e10eed28e	Refactor event storage to not require state ... This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.	2018-04-27 11:38:02 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Neil Johnson	28dd536e80	update changelog and bump version to 0.28.0	2018-04-26 15:51:39 +01:00
Richard van der Hoff	d78ada3166	Miscellaneous fixes to python_dependencies ... * add some doc about wtf this thing does * pin Twisted to < 18.4 * add explicit dep on six (fixes #3089)	2018-04-26 13:11:03 +01:00
Erik Johnston	7ec8e798b4	Fix media admin APIs	2018-04-26 11:31:22 +01:00
Neil Johnson	fb6015d0a6	pep8	2018-04-25 17:56:11 +01:00
Neil Johnson	617bf40924	Generate user daily stats	2018-04-25 17:37:29 +01:00
Erik Johnston	22881b3d69	Also fix reindexing of search	2018-04-25 15:32:04 +01:00
Erik Johnston	ba3166743c	Fix quarantine media admin API	2018-04-25 15:11:18 +01:00
Matthew Hodgson	e3a373f002	remove duplicates from groups tables ... and rename inconsistently named indexes. Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!	2018-04-25 14:58:43 +01:00
Neil Johnson	2c3e995f38	Bump version and update changelog	2018-04-24 15:33:22 +01:00
Richard van der Hoff	77ebef9d43	Merge pull request #3118 from matrix-org/rav/reject_prev_events ... Reject events which have lots of prev_events	2018-04-23 17:51:38 +01:00
Richard van der Hoff	9b9c38373c	Remove spurious param	2018-04-23 12:00:06 +01:00
Richard van der Hoff	1ea904b9f0	Use deferred.addTimeout instead of time_bound_deferred ... This doesn't feel like a wheel we need to reinvent.	2018-04-23 00:53:18 +01:00
Richard van der Hoff	dc875d2712	Merge pull request #3106 from NotAFile/py3-six-itervalues-1 ... Use six.itervalues in some places	2018-04-20 15:43:52 +01:00
Richard van der Hoff	8dc4a6144b	Merge pull request #3107 from NotAFile/py3-bool-nonzero ... add __bool__ alias to __nonzero__ methods	2018-04-20 15:43:39 +01:00
Richard van der Hoff	c09a6daf09	Merge pull request #3110 from NotAFile/py3-six-queue ... Replace Queue with six.moves.queue	2018-04-20 15:35:00 +01:00
Richard van der Hoff	692a3cc806	Merge pull request #3103 from NotAFile/py3-baseexcepton-message ... Use str(e) instead of e.message	2018-04-20 15:34:49 +01:00
Richard van der Hoff	3de7d9fe99	accept stupid events over backfill	2018-04-20 11:41:03 +01:00
Richard van der Hoff	11a67b7c9d	Merge pull request #3093 from matrix-org/rav/response_cache_wrap ... Refactor ResponseCache usage	2018-04-20 11:31:17 +01:00
Richard van der Hoff	0c280d4d99	Reinstate linearizer for federation_server.on_context_state_request	2018-04-20 11:10:04 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	1f4b498b73	Add some comments	2018-04-18 00:15:36 +01:00
Richard van der Hoff	e585228860	Check events on backfill too	2018-04-18 00:06:42 +01:00
Richard van der Hoff	9b7794262f	Reject events which have too many auth_events or prev_events ... ... this should protect us from being dossed by people making silly events (deliberately or otherwise)	2018-04-18 00:06:42 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	878995e660	Replace Queue with six.moves.queue ... and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-16 00:46:21 +02:00
Adrian Tschira	2a3c33ff03	Use six.moves.urlparse ... The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 21:22:43 +02:00
Adrian Tschira	f63ff73c7f	add __bool__ alias to __nonzero__ methods ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:40:47 +02:00
Adrian Tschira	36c59ce669	Use six.itervalues in some places ... There's more where that came from Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:39:43 +02:00
Adrian Tschira	1515560f5c	Use str(e) instead of e.message ... Doing this I learned e.message was pretty shortlived, added in 2.6, they realized it was a bad idea and deprecated it in 2.7 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:32:42 +02:00
Adrian Tschira	bfc2ade9b3	Make event properties raise AttributeError instead ... They raised KeyError before. I'm changing this because the code uses hasattr() to check for the presence of a key. This worked accidentally before, because hasattr() silences all exceptions in python 2. However, in python3, this isn't the case anymore. I had a look around to see if anything depended on this raising a KeyError and I couldn't find anything. Of course, I could have simply missed it. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:16:59 +02:00
Neil Johnson	154b44c249	Merge branch 'master' of https://github.com/matrix-org/synapse into develop	2018-04-13 17:07:54 +01:00
Matthew Hodgson	78a9698650	fix federation_domain_whitelist ... we were checking the wrong server_name on inbound requests	2018-04-13 15:47:43 +01:00
Matthew Hodgson	25b0ba30b1	revert last to PR properly	2018-04-13 15:46:37 +01:00
Matthew Hodgson	f8d46cad3c	correctly auth inbound federation_domain_whitelist reqs	2018-04-13 15:41:52 +01:00
Neil Johnson	d4b2e05852	Merge branch 'master' of https://github.com/matrix-org/synapse into develop	2018-04-13 12:16:27 +01:00
Neil Johnson	51d628d28d	Bump version and Change log	2018-04-13 12:08:19 +01:00
Richard van der Hoff	d3347ad485	Revert "Use sortedcontainers instead of blist" ... This reverts commit 9fbe70a7dc. It turns out that sortedcontainers.SortedDict is not an exact match for blist.sorteddict; in particular, `popitem()` removes things from the opposite end of the dict. This is trivial to fix, but I want to add some unit tests, and potentially some more thought about it, before we do so.	2018-04-13 11:16:43 +01:00
Richard van der Hoff	fac3f9e678	Bump canonicaljson to 1.1.3 ... 1.1.2 was a bit broken too :/	2018-04-13 10:21:38 +01:00
Richard van der Hoff	60f6014bb7	ResponseCache: fix handling of completed results ... Turns out that ObservableDeferred.observe doesn't return a deferred if the result is already completed. Fix handling and improve documentation.	2018-04-13 07:32:29 +01:00
Richard van der Hoff	119596ab8f	Update canonicaljson dependency ... 1.1.0 and 1.1.1 were broken, so we're updating this to help people make sure they don't end up on a broken version. Also, 1.1.0 is speedier...	2018-04-12 17:31:44 +01:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	d5c74b9f6c	Merge pull request #3092 from matrix-org/rav/response_cache_metrics ... Add metrics for ResponseCache	2018-04-12 12:59:36 +01:00
Erik Johnston	415aeefd89	Format docstring	2018-04-12 12:07:09 +01:00
Erik Johnston	19ceb4851f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag	2018-04-12 11:36:07 +01:00
Richard van der Hoff	261124396e	Merge pull request #3059 from matrix-org/rav/doc_response_cache ... Document the behaviour of ResponseCache	2018-04-12 11:22:30 +01:00
Erik Johnston	23a7f9d7f4	Doc we raise on unknown event	2018-04-12 11:20:51 +01:00
Erik Johnston	d7bf3a68f0	s/list/tuple	2018-04-12 11:19:04 +01:00
Erik Johnston	f67e906e18	Set all metrics at the same time	2018-04-12 11:18:19 +01:00
Erik Johnston	971059a733	Merge pull request #3088 from matrix-org/erikj/as_parallel ... Send events to ASes concurrently	2018-04-12 10:42:36 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Erik Johnston	ab825aa328	Add GaugeMetric	2018-04-11 12:13:40 +01:00
Richard van der Hoff	233699c42e	Merge pull request #2760 from Valodim/pypy ... Synapse on PyPy	2018-04-11 11:20:01 +01:00
Neil Johnson	781cd8c54f	bump version/changelog	2018-04-11 10:54:43 +01:00
Neil Johnson	9ef0b179e0	Merge commit '11d2609da70af797405241cdf7d9df19db5628f2' of https://github.com/matrix-org/synapse into release-v0.27.0	2018-04-11 10:51:59 +01:00
Erik Johnston	121591568b	Send events to ASes concurrently	2018-04-11 09:56:00 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Neil Johnson	d54cfbb7a8	fix typo	2018-04-10 17:38:16 +01:00
Erik Johnston	eaa2ebf20b	Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends ... Limit concurrent event sends for a room	2018-04-10 16:43:58 +01:00
Erik Johnston	9daf82278f	Merge pull request #3078 from matrix-org/erikj/federation_sender ... Send federation events concurrently	2018-04-10 16:43:48 +01:00
Adrian Tschira	a3f9ddbede	Open certificate files as bytes ... That's what pyOpenSSL expects on python3 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-10 17:36:29 +02:00
Adrian Tschira	7f8eebc8ee	Open config file in non-bytes mode ... Nothing written into it is encoded, so it makes little sense, but it does break in python3 the way it was before. The variable names were adjusted to be less misleading. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-10 17:32:40 +02:00
Neil Johnson	dd723267b2	Merge branch 'release-v0.27.0' of https://github.com/matrix-org/synapse into develop	2018-04-10 15:03:29 +01:00
Erik Johnston	a060dfa132	Use run_in_background instead	2018-04-10 14:25:11 +01:00
Erik Johnston	f8e8ec013b	Note why we're limiting concurrent event sends	2018-04-10 14:00:46 +01:00
Erik Johnston	1246d23710	Preserve log contexts correctly	2018-04-10 12:04:32 +01:00
Erik Johnston	d49cbf712f	Log event ID on exception	2018-04-10 12:03:41 +01:00
Erik Johnston	ce72d590ed	Merge pull request #3082 from matrix-org/erikj/urlencode_paths ... URL quote path segments over federation	2018-04-10 11:31:16 +01:00
Erik Johnston	11d2609da7	Ensure slashes are escaped	2018-04-10 11:24:40 +01:00
Erik Johnston	dab87b84a3	URL quote path segments over federation	2018-04-10 11:16:08 +01:00
Vincent Breitmoser	6d7f0f8dd3	Don't disable GC when running on PyPy ... PyPy's incminimark GC can't be triggered manually. From what I observed there are no obvious issues with just letting it run normally. And unlike CPython, it actually returns unused RAM to the system. Signed-off-by: Vincent Breitmoser <look@my.amazin.horse>	2018-04-10 11:35:34 +02:00
Vincent Breitmoser	f4284d943a	In DomainSpecificString, override __repr__ in addition to __str__ ... For some reason, string interpolation on a DomainSpecificString object like "%r" % (domainSpecificStringObj) fails under PyPy, because the default __repr__ implementation wants to iterate over the object. I'm not sure why that happens, but overriding __repr__ instead of __str__ fixes this problem, and is arguably the more appropriate thing to do anyways.	2018-04-10 11:35:29 +02:00
Richard van der Hoff	d1e56cfcd1	Fix pep8 error on psycopg2cffi hack	2018-04-10 11:35:29 +02:00
Vincent Breitmoser	89de934981	Use psycopg2cffi module instead of psycopg2 if running on pypy ... The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.	2018-04-10 11:29:52 +02:00
Vincent Breitmoser	9fbe70a7dc	Use sortedcontainers instead of blist ... This commit drop-in replaces blist with SortedContainers. They are written in pure python so work with pypy, but perform as good as native implementations, at least in a couple benchmarks: http://www.grantjenks.com/docs/sortedcontainers/performance.html	2018-04-10 11:29:51 +02:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	87478c5a60	Merge pull request #3061 from NotAFile/add-some-byte-strings ... Add b prefixes to some strings that are bytes in py3	2018-04-09 23:54:05 +01:00
Richard van der Hoff	c508b2f2f0	Merge pull request #3073 from NotAFile/use-six-reraise ... Replace old-style raise with six.reraise	2018-04-09 23:53:40 +01:00
Richard van der Hoff	37354b55c9	Merge pull request #2938 from dklug/develop ... Return 401 for invalid access_token on logout	2018-04-09 23:52:56 +01:00
Richard van der Hoff	0e9aa1d091	Merge pull request #3074 from NotAFile/fix-py3-prints ... use python3-compatible prints	2018-04-09 23:44:41 +01:00
Richard van der Hoff	8eaa141d8f	Merge pull request #3075 from NotAFile/six-type-checks ... Replace some type checks with six type checks	2018-04-09 23:40:44 +01:00
Richard van der Hoff	664adb4236	Merge pull request #3016 from silkeh/improve-service-lookups ... Improve handling of SRV records for federation connections	2018-04-09 23:40:06 +01:00
Neil Johnson	41e0611895	remove errant print	2018-04-09 18:44:20 +01:00
Neil Johnson	61b439c904	Fix msec to sec, again	2018-04-09 18:43:48 +01:00
Neil Johnson	87770300d5	Fix msec to sec	2018-04-09 18:38:59 +01:00
Neil Johnson	64bc2162ef	Fix psycopg2 interpolation	2018-04-09 17:50:36 +01:00
Richard van der Hoff	d2c6f4d626	Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin ... Return a 404 rather than a 500 on rejoining empty rooms	2018-04-09 17:32:36 +01:00
Neil Johnson	5232d3bfb1	version bump v0.27.3-rc2	2018-04-09 17:25:57 +01:00
Neil Johnson	5e785d4d5b	Merge branch 'develop' of https://github.com/matrix-org/synapse into release-v0.27.0	2018-04-09 17:21:34 +01:00
Erik Johnston	6e025a97b4	Handle all events in a room correctly	2018-04-09 16:02:48 +01:00
Neil Johnson	d9ae2bc826	bump version to release candidate	2018-04-09 16:00:31 +01:00
Neil Johnson	072fb59446	bump version	2018-04-09 13:49:25 +01:00
Richard van der Hoff	13decdbf96	Revert "Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics" ... We aren't ready to release this yet, so I'm reverting it for now. This reverts commit d1679a4ed7, reversing changes made to e089100c62.	2018-04-09 12:59:12 +01:00
Richard van der Hoff	f3ef60662f	Return a 404 rather than a 500 on rejoining empty rooms ... Filter ourselves out of the server list before checking for an empty remote host list, to fix 500 error Fixes #2141	2018-04-09 12:56:22 +01:00
Erik Johnston	e5082494eb	Limit concurrent event sends for a room	2018-04-09 12:07:39 +01:00
Erik Johnston	56b0589865	Use create_and_send_nonmember_event everywhere	2018-04-09 12:04:18 +01:00
Erik Johnston	11974f3787	Send federation events concurrently	2018-04-09 11:47:10 +01:00
Erik Johnston	145d14656b	Handle exceptions in get_hosts_for_room when sending events over federation	2018-04-09 11:47:01 +01:00
Adrian Tschira	e54c202b81	Replace some type checks with six type checks ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-07 01:02:32 +02:00
Adrian Tschira	b0500d3774	use python3-compatible prints	2018-04-06 23:35:27 +02:00
Adrian Tschira	4f40d058cc	Replace old-style raise with six.reraise ... The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-06 23:06:24 +02:00
Luke Barnard	135fc5b9cd	Merge pull request #3046 from matrix-org/dbkr/join_group ... Implement group join API	2018-04-06 16:24:32 +01:00
Luke Barnard	020a501354	de-lint, quote consistency	2018-04-06 16:02:06 +01:00
Luke Barnard	db2fd801f7	Explicitly grab individual columns from group object	2018-04-06 15:57:25 +01:00
Luke Barnard	7945435587	When exposing group state, return is_openly_joinable ... as opposed to join_policy, which is really only pertinent to the synapse implementation of the group server. By doing this we keep the group server concept extensible by allowing arbitrarily complex rules for deciding whether a group is openly joinable.	2018-04-06 15:43:27 +01:00
Luke Barnard	6bd1b7053e	By default, join policy is "invite"	2018-04-06 15:43:27 +01:00
Luke Barnard	b4478e586f	add_user -> _add_user	2018-04-06 15:43:27 +01:00
Luke Barnard	112c2253e2	pep8	2018-04-06 15:43:27 +01:00
Luke Barnard	6850f8aea3	Get group_info from existing call to check_group_is_ours	2018-04-06 15:43:27 +01:00
Luke Barnard	cd087a265d	Don't use redundant inlineCallbacks	2018-04-06 15:43:27 +01:00
Luke Barnard	87c864b698	join_rule -> join_policy	2018-04-06 15:43:27 +01:00
Luke Barnard	ae85c7804e	is_joinable -> join_rule	2018-04-06 15:43:27 +01:00
Luke Barnard	f8d1917fce	Fix federation client set_group_joinable typo	2018-04-06 15:43:27 +01:00
Luke Barnard	6eb3aa94b6	Factor out add_user from accept_invite and join_group	2018-04-06 15:43:27 +01:00
David Baker	edb45aae38	pep8	2018-04-06 15:43:27 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Erik Johnston	7b824f1475	Add response size metrics	2018-04-06 13:20:11 +01:00
Krombel	1d71f484d4	use PUT instead of POST for federating groups/m.join_policy	2018-04-06 12:54:09 +02:00
Richard van der Hoff	15e8ed874f	more verbosity in synctl	2018-04-06 09:28:36 +01:00
Richard van der Hoff	c7f0969731	Merge pull request #2986 from jplatte/join_reponse_room_id ... Add room_id to the response of `rooms/{roomId}/join`	2018-04-05 17:29:06 +01:00
Richard van der Hoff	3449da3bc7	Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation ... Improve database cache performance	2018-04-05 17:21:44 +01:00
Richard van der Hoff	d1679a4ed7	Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics ... Remove redundant metrics which were deprecated in 0.27.0.	2018-04-05 17:21:18 +01:00
Richard van der Hoff	01afc563c3	Fix overzealous cache invalidation ... Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.	2018-04-05 16:24:04 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Neil Johnson	68b0ee4e8d	Merge pull request #3041 from matrix-org/r30_stats ... R30 stats	2018-04-05 15:37:37 +01:00
Richard van der Hoff	22284a6f65	Merge pull request #3060 from matrix-org/rav/kill_event_content ... Remove uses of events.content	2018-04-05 15:02:17 +01:00
Luke Barnard	917380e89d	NON NULL -> NOT NULL	2018-04-05 14:32:12 +01:00
Luke Barnard	104c0bc1d5	Use "/settings/" (plural)	2018-04-05 14:07:16 +01:00
Luke Barnard	700e5e7198	Use DEFAULT join_policy of "invite" in db	2018-04-05 14:01:17 +01:00
Luke Barnard	b214a04ffc	Document set_group_join_policy	2018-04-05 13:29:16 +01:00
Neil Johnson	0e5f479fc0	Review comments ... Use iteritems over item to loop over dict formatting	2018-04-05 12:16:46 +01:00
Richard van der Hoff	518f6de088	Remove redundant metrics which were deprecated in 0.27.0.	2018-04-04 19:46:28 +01:00
Jan Christian Grünhage	e4570c53dd	phone home cache size configurations	2018-04-04 16:46:58 +01:00
Adrian Tschira	6168351877	Add b prefixes to some strings that are bytes in py3 ... This has no effect on python2 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-04 13:48:51 +02:00
Richard van der Hoff	f92963f5db	Revert "improve mxid check performance"	2018-04-04 12:08:29 +01:00
Silke	72251d1b97	Remove address resolution of hosts in SRV records ... Signed-off-by: Silke Hofstra <silke@slxh.eu>	2018-04-04 12:26:50 +02:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Richard van der Hoff	a89f9f830c	Merge pull request #3044 from matrix-org/michaelk/performance_stats ... Add basic performance statistics to phone home	2018-04-04 10:37:25 +01:00
Richard van der Hoff	39ce38b024	Merge pull request #3053 from NotAFile/speedup-mxid-check ... improve mxid check performance	2018-04-04 10:19:52 +01:00
Richard van der Hoff	a9a74101a4	Document the behaviour of ResponseCache ... it looks like everything that uses ResponseCache expects to have to `make_deferred_yieldable` its results. It's debatable whether that is the best approach, but let's document it for now to avoid further confusion.	2018-04-04 09:06:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	3ee4ad09eb	Fix json encoding bug in replication ... json encoders have an encode method, not a dumps method.	2018-04-03 15:09:48 +01:00
Adrian Tschira	11597ddea5	improve mxid check performance ~4x ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-31 02:00:22 +02:00
Richard van der Hoff	2fe3f848b9	Remove uses of events.content	2018-03-29 23:17:12 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Neil Johnson	b4e37c6f50	pep8	2018-03-29 17:27:39 +01:00
Neil Johnson	9ee44a372d	Remove need for sqlite specific query	2018-03-29 16:45:34 +01:00
Erik Johnston	88cc9cc69e	Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation ... Measure time it takes to calculate state group ID	2018-03-28 17:40:14 +01:00
Neil Johnson	dc7c020b33	fix pep8 errors	2018-03-28 17:25:15 +01:00
David Baker	c5de6987c2	This should probably be a PUT	2018-03-28 16:44:11 +01:00
Neil Johnson	241e4e8687	remove twisted deferral cruft	2018-03-28 16:25:53 +01:00
David Baker	929b34963d	OK, smallint it is then	2018-03-28 14:53:55 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00
David Baker	a838444a70	Grr. Copy the definition from is_admin	2018-03-28 14:50:30 +01:00
Neil Johnson	4262aba17b	bump schema version	2018-03-28 14:40:03 +01:00
Neil Johnson	86932be2cb	Support multi client R30 for psql	2018-03-28 14:36:53 +01:00
David Baker	32260baa41	pep8	2018-03-28 14:29:42 +01:00
Michael Kaye	33f6195d9a	Handle review comments	2018-03-28 14:25:25 +01:00
David Baker	a164270833	Make column definition that works on both dbs	2018-03-28 14:23:00 +01:00
David Baker	352e1ff9ed	Add schema delta file	2018-03-28 14:07:57 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Krombel	6152e253d8	Merge branch 'develop' of into allow_auto_join_rooms	2018-03-28 14:45:28 +02:00
Neil Johnson	792d340572	rename stat to future proof	2018-03-28 12:25:02 +01:00
Michael Kaye	4ceaa7433a	As daemonizing will make a new process, defer call to init.	2018-03-28 12:19:01 +01:00
Neil Johnson	788e69098c	Add user_ips last seen index	2018-03-28 12:03:13 +01:00
Neil Johnson	0f890f477e	No need to cast in count_daily_users	2018-03-28 11:49:57 +01:00
Neil Johnson	545001b9e4	Fix search_user_dir multiple sqlite versions do different things	2018-03-28 11:19:45 +01:00
Erik Johnston	01ccc9e6f2	Measure time it takes to calculate state group ID	2018-03-28 11:03:52 +01:00
Neil Johnson	a32d2548d9	query and call for r30 stats	2018-03-28 10:39:13 +01:00
Neil Johnson	9187e0762f	count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home	2018-03-28 10:02:32 +01:00
Erik Johnston	f879127aaa	Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id ... Linearize calls to _generate_user_id	2018-03-28 10:00:31 +01:00
Erik Johnston	e6d87c93f3	Merge pull request #3030 from matrix-org/erikj/no_ujson ... Remove last usage of ujson	2018-03-28 10:00:06 +01:00
Erik Johnston	004cc8a328	Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics ... Add counter metrics for calculating state delta	2018-03-28 09:59:42 +01:00
Michael Kaye	ef520d8d0e	Include coarse CPU and Memory use in stats callbacks. ... This requires the psutil module, and is still opt-in based on the report_stats config option.	2018-03-27 17:56:03 +01:00
Richard van der Hoff	a134c572a6	Stringify exceptions for keys/{query,claim} ... Make sure we stringify any exceptions we return from keys/query and keys/claim, to avoid a 'not JSON serializable' error later Fixes #3010	2018-03-27 17:15:06 +01:00