8914 Commits

Author	SHA1	Message	Date
Richard van der Hoff	200e11c5bf	Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable ... Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:18:51 +01:00
Erik Johnston	f8272813a9	Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:12:26 +01:00
Richard van der Hoff	1d7ad11747	Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation ... Make push actions rotation configurable	2018-06-22 15:07:31 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	b4a5d767a9	Merge pull request #3428 from matrix-org/erikj/persisted_pdu ... Simplify get_persisted_pdu	2018-06-22 14:47:43 +01:00
Erik Johnston	f79abda87f	Merge pull request #3427 from matrix-org/erikj/remove_filters ... remove dead filter_events_for_clients	2018-06-22 14:45:42 +01:00
Erik Johnston	75dc3ddeab	Make push actions rotation configurable	2018-06-22 14:44:37 +01:00
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	2018-06-22 11:52:07 +01:00
Richard van der Hoff	240f192523	Merge pull request #3382 from matrix-org/rav/optimise_state_groups ... Optimise state_group_cache update	2018-06-22 11:20:20 +01:00
Richard van der Hoff	70e6501913	Merge pull request #3419 from matrix-org/rav/events_per_request ... Log number of events fetched from DB	2018-06-22 11:17:56 +01:00
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	2018-06-22 10:42:28 +01:00
Amber Brown	9a685d60ae	Merge pull request #3418 from matrix-org/rav/fix_metric_desc ... Fix description of "python_gc_time" metric	2018-06-22 09:43:26 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	cbbfaa4be8	Fix description of "python_gc_time" metric	2018-06-21 10:02:42 +01:00
Amber Brown	c2eff937ac	Populate synapse_federation_client_sent_pdu_destinations:count again (#3386)	2018-06-21 09:39:58 +01:00
Amber Brown	99b77aa829	Fix tcp protocol metrics naming (#3410)	2018-06-21 09:39:27 +01:00
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	2018-06-21 06:15:03 +01:00
Richard van der Hoff	aff3d76920	Merge pull request #3416 from matrix-org/rav/restart_indicator ... Write a clear restart indicator in logs	2018-06-20 18:14:08 +01:00
Richard van der Hoff	02bfc581f8	Merge pull request #3399 from costacruise/master ... Add error code to room creation error	2018-06-20 17:26:25 +01:00
Richard van der Hoff	245d53d32a	Write a clear restart indicator in logs ... I'm fed up with never being able to find the point a server restarted in the logs.	2018-06-20 15:33:14 +01:00
Amber Brown	f6c4d74f96	Fix inflight requests metric (incorrect name & traceback) (#3413)	2018-06-20 11:18:57 +01:00
Matthew Hodgson	ccfdaf68be	spell gauge correctly	2018-06-16 07:10:34 +01:00
Richard van der Hoff	9a793f861c	Merge branch 'master' into develop	2018-06-14 16:36:01 +01:00
Richard van der Hoff	53969e1960	SECURITY UPDATE: Prevent unauthorised users from setting state events in a room ... when there is no `m.room.power_levels` event in force in the room. (PR #3397) Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbIop9AAoJEIofk9V1tejV9lsIAJVH0l5dXROmy1KH/zt16AUA CXa6Vv4Vyo6hKad/fZ81OZVRr5ChK/TvbIJVn/SA/muCfdoIFdxhT8eo/pXzO2UW zReuLsDhAg+gSvpNus37oWj2FVsAE1HYDZ60lfaapAdZnkFit68d5DQZjO6nZHHA YUXcU3GUwj0ZYuUzFzYKMLu6uNNasNkN8h6SS2lF7Bm4JaKDW+mFMfCyJwdIVSEh BGhHoVpXdxFysD9s6Mwxqrz3KKg1Jtp7idDkk0x2S2Eh+gxyiDQQokv0oQ3+0+HG sgy5Iz2t2CkpS02/j+LOvAZljTmnD0bXu3srGR+25StsoDFP038Am3bfQwtD190= =9jsT -----END PGP SIGNATURE----- Merge tag 'v0.31.2' SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no `m.room.power_levels` event in force in the room. (PR #3397) Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.	2018-06-14 16:35:33 +01:00
Richard van der Hoff	7e1c616452	v0.31.2	2018-06-14 16:24:32 +01:00
Richard van der Hoff	61ab08a197	Merge pull request #3397 from matrix-org/rav/adjust_auth_rules ... Adjust event auth rules when there is no PL event	2018-06-14 16:09:13 +01:00
Michael Wagner	19cd3120ec	Add error code to room creation error ... This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom	2018-06-14 14:08:40 +02:00
Richard van der Hoff	5c9afd6f80	Make default state_default 50 ... Make it so that, before there is a power-levels event in the room, you need a power level of at least 50 to send state. Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192	2018-06-14 12:38:09 +01:00
Richard van der Hoff	52423607bd	Clarify interface for event_auth ... stop pretending that it returns a boolean, which just almost gave me a heart attack.	2018-06-14 12:26:17 +01:00
Amber Brown	f116f32ace	add a last seen metric (#3396)	2018-06-14 20:26:59 +10:00
Richard van der Hoff	557b686eac	Refactor get_send_level to take a power_levels event ... it makes it easier for me to reason about	2018-06-14 11:26:27 +01:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Amber Brown	0fde1896cd	Merge pull request #3389 from turt2live/travis/name_metrics ... Use the correct flag (enable_metrics) when warning about an incorrect metrics setup	2018-06-13 23:50:10 +10:00
Michael Telatynski	94700e55fa	if inviter_display_name == ""||None then default to inviter MXID ... to prevent email invite from "None"	2018-06-13 10:31:01 +01:00
Travis Ralston	45768d1640	Use the RegistryProxy for appservices too ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-12 12:55:48 -06:00
Travis Ralston	12285a1a76	The flag is named enable_metrics, not collect_metrics ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-12 12:51:31 -06:00
Richard van der Hoff	b6faef2ad7	Filter out erased messages ... Redact any messges sent by erased users.	2018-06-12 09:53:18 +01:00
Richard van der Hoff	f1023ebf4b	mark accounts as erased when requested	2018-06-12 09:53:18 +01:00
Richard van der Hoff	3ff8a619f5	UserErasureStore ... to store which users have been erased	2018-06-12 09:52:22 +01:00
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	bd348f0af6	remove dead filter_events_for_clients ... This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	eb32b2ca20	Optimise state_group_cache update ... (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.	2018-06-11 22:56:11 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	d6cc369205	fix idiotic typo in state res	2018-06-11 14:43:55 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Vincent Breitmoser	b800834351	add note that the affinity package is required for the cpu_affinity setting	2018-06-09 22:50:29 +02:00
Neil Johnson	ed5a0780a4	Merge branch 'master' into develop	2018-06-08 15:47:11 +01:00
Neil Johnson	1032393dfb	Changes in synapse v0.31.1 (2018-06-08) ... ======================================= v0.31.1 fixes a security bug in the ``get_missing_events`` federation API where event visibility rules were not applied correctly. We are not aware of it being actively exploited but please upgrade asap. Bug Fixes: * Fix event filtering in get_missing_events handler (PR #3371) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsalP8ACgkQCTxDAAxP S/RADw/+NeDu0LjVpS5Uc4ElHgRBuFSm6l2i4z8rZBBlKSYnuq0Em4WMvLloi/JF iAvTOYE7OjmF+gNvmsdH1N7hc1lKdQ2gAlpvaQR/5Qz9NtOVmM3WPZxS7n5jZHvD hVSxeO9+GQOwK7rJorqrrsnWHQt0OkLHV6WThFdrgZb1JjWCUDTvw+Hei2uMX2aq y2mkMG4TLStHwMvL2qw0h+hFtXywXI796qJR73ZxbEn24YD+kOXeEVkIFi2LT0Pj cgkg7WWT32JD43/ioumLupZuhCmpRyxn4fi5gIKpXe5kiLsxOdApzNQSwmoJ+WA8 7zlrWY+0QDN4pbA5ESLitWSWAT50Ul//uM4nwmM4xEBPHdljXvKyHPsaSCeKLvT9 RT8pc41TQAqSshlXF8zgIAtStnF3oGel3EBBl1mmM9Un1ULnBFwWDZhlIm+ZZGhJ MWoAWNG7j8AQuy0BTUAUr76x7t+/cdSqDuyVl1GO1tbDh0DUWoHZGXCUKrAXnn2T SbiFigwOLvEADbvkW7L9Je9CVOi2V5Pg/32X9O8YMEiSz+j5PQEiGefyVh/I/QvV Ha/atRpZF2OZ+XUOO5DLZMP/XCXpVgvHuskzfU6LvvVQCXgExuJhRD1PrRGeBaWr zjJW+rmY+VeredqX7QkTB3XOGLMLGJfx2FeSMb+j0w3a7/iFj+Q= =J80S -----END PGP SIGNATURE----- Merge tag 'v0.31.1' Changes in synapse v0.31.1 (2018-06-08) ======================================= v0.31.1 fixes a security bug in the ``get_missing_events`` federation API where event visibility rules were not applied correctly. We are not aware of it being actively exploited but please upgrade asap. Bug Fixes: * Fix event filtering in get_missing_events handler (PR #3371)	2018-06-08 15:46:18 +01:00
Neil Johnson	0eb4722932	changelog a bump version	2018-06-08 15:21:46 +01:00
Richard van der Hoff	c6b1441c52	Fix event filtering in get_missing_events handler	2018-06-08 14:15:31 +01:00
David Baker	0e505b1913	Merge pull request #3372 from matrix-org/rav/better_verification_logging ... Try to log more helpful info when a sig verification fails	2018-06-08 13:32:02 +01:00
Richard van der Hoff	e82db24a0e	Try to log more helpful info when a sig verification fails ... Firstly, don't swallow the reason for the failure Secondly, don't assume all exceptions are verification failures Thirdly, log a bit of info about the key being used if debug is enabled	2018-06-08 12:13:08 +01:00
Richard van der Hoff	0834b49c6a	Fix event filtering in get_missing_events handler	2018-06-08 11:34:46 +01:00
Richard van der Hoff	8503dd0047	Remove event re-signing hacks ... These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.	2018-06-07 16:08:29 +01:00
Will Hunt	13d211edc1	Merge pull request #3344 from Half-Shot/hs/as-metrics ... Add metrics to track appservice transactions	2018-06-07 11:37:12 +01:00
Richard van der Hoff	1152f495a0	Merge pull request #3363 from matrix-org/rav/fix_purge ... Fix event-purge-by-ts admin API	2018-06-07 11:34:46 +01:00
Richard van der Hoff	e2acf536d4	Merge pull request #3355 from matrix-org/rav/fix_federation_backfill ... Fix federation backfill from sqlite servers	2018-06-07 10:18:01 +01:00
Amber Brown	0160f6601f	Merge pull request #3356 from matrix-org/rav/add_missing_attr_dep ... Make sure that attr is installed	2018-06-07 16:33:30 +10:00
Richard van der Hoff	f4caf3f83d	fix log	2018-06-07 00:26:38 +01:00
Richard van der Hoff	0546715c18	Fix event-purge-by-ts admin API ... This got completely broken in 0.30. Fixes #3300.	2018-06-07 00:15:49 +01:00
Richard van der Hoff	57e3f923d2	Add missing dependency on attr ... We've rcently added a dep on `attr`. I don't know why the CI didn't pick this up, but we should make it explicit anyway.	2018-06-06 17:12:41 +01:00
Richard van der Hoff	d3a8c9c55e	Fix sql error in _get_state_groups_from_groups ... If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.	2018-06-06 14:19:01 +01:00
Neil Johnson	fef6c2cdcc	Merge branch 'master' into develop	2018-06-06 12:28:15 +01:00
Neil Johnson	752b7b32ed	Changes in synapse v0.31.0 (2018-06-06) ... ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx! -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsXxJAACgkQCTxDAAxP S/SJTg//Wtr+Qop9LJh2/leAYXpyqW6P7Ftak0w3aJ3KL3+tYg32yYNoRADCqbp3 LkrHu8MwbZagHjRUyEWNfDk4jbfq5fwh0JVGmYuUKhG9aF0HYyytKkbW79YzuhdQ dfHj9x0xSBOUvgt/husloZSDy0VHC6uyQSAFgFDyHS2y7RPAiGstqLGByv0ciZOk pO7TdjkUQcx4Ps7Wgip31NuHy3GY2int6f540pUXoZHLXs7RkfqS2cpF9Z/sTXJ4 xDLiY7uYNsTcCblwqaiijY5c90xwRB2vLs5CJdKFgyB6PNgg/2wHJqP/WHHEj+F8 BoSm3Ts7NXQf23pP9CXICe7vXX3J+ruOnC7FOSRobr6KGjn6DUrIZxo1ZepTwpp9 DIq+1eOFKKjwLQM3Jdi8WBCP63LhYXrTZxreke3jpwdcD7oIO9v6/e9J1gU5xHWa Izg+YnWn1JLfq/X8T7YTZddUXGGPkH5i6LZKKkyY8u7LkJ4WR7syuAceUzkOOIAq UWO0uEV7IiLnZzZGTtNIlEqtuklmVQTm6bvAgTPabai2JQyngFpH5M/5mPpVSiLV QRLwaM56c+5GGZJWef8vxdGeYn+8rFI/UUniJ7358kLJF2IHsxlZu8J0ZZO2HWI2 ze5Kz0AWRzXLhWzq62Qb2dsiGySrZ7hng1tDxLak2IiusY+9SjM= =Mz9U -----END PGP SIGNATURE----- Merge tag 'v0.31.0' Changes in synapse v0.31.0 (2018-06-06) ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!	2018-06-06 12:27:33 +01:00
Neil Johnson	61134debdc	bump version and changelog	2018-06-06 11:26:21 +01:00
Richard van der Hoff	ad459a106c	Merge pull request #3349 from t3chguy/redact_as_request_token ... Redact AS tokens in log (fixes to #3327)	2018-06-06 10:58:07 +01:00
Michael Telatynski	592c162516	also redact __str__ of ApplicationService used for logging	2018-06-06 10:35:29 +01:00
Michael Telatynski	330432031b	redact_uri in two missed log paths	2018-06-06 10:25:48 +01:00
David Baker	bf54c1cf6c	pep8	2018-06-06 10:15:33 +01:00
David Baker	3e4bc4488c	More doc fixes	2018-06-06 09:44:10 +01:00
Richard van der Hoff	592ee217a3	Merge commit 'b7e7fd2' into release-v0.31.0	2018-06-06 07:02:02 +01:00
Ivan Shapovalov	c88d50aa8f	federation/send_queue.py: fix usage of sortedcontainers.SortedDict	2018-06-06 02:45:18 +03:00
Richard van der Hoff	e316407b5d	Merge pull request #3327 from t3chguy/redact_as_request_token ... Strip `access_token` from outgoing requests	2018-06-05 19:08:46 +01:00
Michael Telatynski	e6cbf47773	factor out uri redaction into a method on http	2018-06-05 18:31:40 +01:00
David Baker	607bd27c83	fix pep8	2018-06-05 18:10:35 +01:00
David Baker	d62162bbec	doc fixes	2018-06-05 18:09:13 +01:00
Richard van der Hoff	522bd3c8a3	Merge remote-tracking branch 'origin/master' into develop	2018-06-05 17:42:49 +01:00
Will Hunt	d6e3c2c79b	Let's try labels instead of label, that might work	2018-06-05 17:30:45 +01:00
Amber Brown	f7869f8f8b	Port to sortedcontainers (with tests!) (#3332)	2018-06-06 00:13:57 +10:00
Will Hunt	604cff1a06	Add metrics to track appservice transactions	2018-06-05 13:21:30 +01:00
Richard van der Hoff	f29b41fde9	Merge pull request #3324 from matrix-org/rav/remove_dead_method ... Remove was_forgotten_at	2018-06-04 18:18:08 +01:00
Richard van der Hoff	28b0490dfd	Merge pull request #3334 from matrix-org/rav/cache_factor_override ... Cache factor override system for specific caches	2018-06-04 17:19:33 +01:00
Richard van der Hoff	b7e7fd2d0e	Fix replication metrics ... fix bug introduced in #3256	2018-06-04 16:23:05 +01:00
Neil Johnson	244ab974e7	bump version and changelog	2018-06-04 16:09:58 +01:00
Richard van der Hoff	694968fa81	Hopefully, fix LaterGuage error handling	2018-06-04 15:59:14 +01:00
Erik Johnston	042eedfa2b	Add hacky cache factor override system	2018-06-04 15:39:28 +01:00
David Baker	c5930d513a	Docstring	2018-06-04 12:05:58 +01:00
David Baker	6a29e815fc	Fix comment	2018-06-04 12:01:23 +01:00
David Baker	e44150a6de	Missing yield	2018-06-04 12:01:13 +01:00
David Baker	f731e42baf	docstring	2018-06-04 12:00:51 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Matthew Hodgson	5f6122fe10	more comments	2018-06-04 00:08:52 +03:00
Ivan Shapovalov	7d9d75e4e8	federation/send_queue.py: fix usage of LaterGauge ... Fixes a startup crash due to commit df9f72d9e5 "replacing portions".	2018-06-03 14:16:17 +03:00
Michael Telatynski	09503126df	Strip access_token from outgoing requests using existing regex	2018-06-02 23:25:13 +01:00
Richard van der Hoff	c1f4118bb6	Remove was_forgotten_at ... This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *	2018-06-01 18:21:49 +01:00
Richard van der Hoff	a9e97dcd65	Merge pull request #3317 from thegcat/feature/3312-add_ipv6_to_blacklist_example_config ... Add private IPv6 addresses to example config for url preview blacklist	2018-06-01 14:45:14 +01:00
Neil Johnson	71477f3317	Merge pull request #3264 from matrix-org/neil/sign-up-stats ... daily user type phone home stats	2018-06-01 13:42:01 +00:00
Richard van der Hoff	9f797a24a4	Handle RRs which arrive before their events	2018-06-01 14:01:43 +01:00
Richard van der Hoff	857e6fd8b6	Ignore depth when updating read-receipts ... Order read receipts by stream ordering instead of depth	2018-06-01 12:18:11 +01:00
Felix Schäfer	4ef76f3ac4	Add private IPv6 addresses to preview blacklist #3312 ... The added addresses are expected to be local or loopback addresses and shouldn't be spidered for previews. Signed-off-by: Felix Schäfer <felix@thegcat.net>	2018-06-01 12:18:35 +02:00
Neil Johnson	4986b084f8	remove unnecessary INSERT	2018-06-01 10:50:40 +01:00
Richard van der Hoff	7e15410f02	Enforce the specified API for report_event ... as per https://matrix.org/docs/spec/client_server/unstable.html#post-matrix-client-r0-rooms-roomid-report-eventid	2018-05-31 18:17:11 +01:00
Amber Brown	febe0ec8fd	Run Prometheus on a different port, optionally. (#3274)	2018-05-31 19:04:50 +10:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Richard van der Hoff	219c2a322b	remove trailing whitespace	2018-05-30 19:42:19 +01:00
Richard van der Hoff	2e4be8bfd9	fix english and wrap comment	2018-05-30 19:24:12 +01:00
Amber Brown	872cf43516	Merge pull request #3303 from NotAFile/py3-memoryview ... use memoryview in py3	2018-05-30 12:51:42 +10:00
Amber Brown	debff7ae09	Merge pull request #3281 from NotAFile/py3-six-isinstance ... remaining isintance fixes	2018-05-30 12:44:46 +10:00
Richard van der Hoff	34b85df7f5	Update some comments and docstrings in SyncHandler	2018-05-29 22:31:18 +01:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Neil Johnson	ab0ef31dc7	create users index on creation_ts	2018-05-29 17:51:08 +01:00
Neil Johnson	558f3d376a	create index in background	2018-05-29 17:47:55 +01:00
Neil Johnson	c379acd4fd	bump version	2018-05-29 17:47:28 +01:00
Adrian Tschira	4b9d0cde97	add remaining memoryview changes	2018-05-29 17:42:43 +02:00
Adrian Tschira	7873cde526	pep8	2018-05-29 17:35:55 +02:00
Adrian Tschira	1afafb3497	use memoryview in py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-29 17:14:34 +02:00
Matthew Hodgson	9bbb9f5556	add lazy_load_members to the filter json schema	2018-05-29 04:26:10 +01:00
Matthew Hodgson	8df7bad839	pep8	2018-05-29 02:49:36 +01:00
Matthew Hodgson	ff1bc0a279	pep8	2018-05-29 02:32:15 +01:00
Matthew Hodgson	0a240ad36e	disable CPUMetrics if no /proc/self/stat ... fixes build on macOS again	2018-05-29 02:23:30 +01:00
Matthew Hodgson	b69ff33d9e	disable CPUMetrics if no /proc/self/stat ... fixes build on macOS again	2018-05-29 02:22:27 +01:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	57ad76fa4a	fix up tests	2018-05-28 19:51:53 +10:00
Amber Brown	3ef5cd74a6	update to more consistently use seconds in any metrics or logging	2018-05-28 19:39:27 +10:00
Amber Brown	5c40ce3777	invalid syntax :(	2018-05-28 19:16:09 +10:00
Amber Brown	357c74a50f	add comment about why unreg	2018-05-28 19:14:41 +10:00
Amber Brown	a2eb5db4a0	update metrics to be in seconds	2018-05-28 19:10:27 +10:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Ruben Barkow	08ea5fe635	add link to thorough instruction how to configure consent	2018-05-25 23:19:55 +02:00
Richard van der Hoff	66bdae986f	Fix default for send_server_notice_to_guests ... bool("False") == True...	2018-05-25 11:42:05 +01:00
Richard van der Hoff	ba1b163590	Avoid sending consent notice to guest users ... we think it makes sense not to send the notices to guest users.	2018-05-25 11:36:43 +01:00
Richard van der Hoff	41921ac01b	Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room ... Let users leave the server notice room after joining	2018-05-25 11:15:45 +01:00
Richard van der Hoff	757ed27258	Let users leave the server notice room after joining ... They still can't reject invites, but we let them leave it.	2018-05-25 11:07:21 +01:00
Adrian Tschira	4ee4450d66	fix recursion error	2018-05-24 21:44:10 +02:00
Amber Brown	9c36c150e7	Merge pull request #3283 from NotAFile/py3-state ... py3-ize state.py	2018-05-24 14:23:33 -05:00
Amber Brown	cc1349c06a	Merge pull request #3279 from NotAFile/py3-more-iteritems ... more six iteritems	2018-05-24 14:23:13 -05:00
Adrian Tschira	0e61705661	py3-ize state.py	2018-05-24 20:59:00 +02:00
Adrian Tschira	dd068ca979	remaining isintance fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:55:08 +02:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Adrian Tschira	6c16a4ec1b	more iteritems	2018-05-24 20:19:06 +02:00
Amber Brown	7ea07c7305	Merge pull request #3278 from NotAFile/py3-storage-base ... Py3 storage/_base.py	2018-05-24 13:08:09 -05:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
Amber Brown	c4fb15a06c	Merge pull request #3246 from NotAFile/py3-repr-string ... use repr, not str	2018-05-24 13:00:20 -05:00
Amber Brown	36501068d8	Merge pull request #3247 from NotAFile/py3-misc ... Misc Python3 fixes	2018-05-24 12:58:37 -05:00
Amber Brown	2aff6eab6d	Merge pull request #3245 from NotAFile/batch-iter ... Add batch_iter to utils	2018-05-24 12:54:12 -05:00
Adrian Tschira	095292304f	Py3 storage/_base.py ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 18:24:12 +02:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	ecc4b88bd1	Merge pull request #3277 from matrix-org/dbkr/remove_from_user_dir ... Remove users from user directory on deactivate	2018-05-24 16:12:12 +01:00
Erik Johnston	46345187cc	Merge pull request #3243 from NotAFile/py3-six-3 ... Replace some more comparisons with six	2018-05-24 16:08:57 +01:00
Neil Johnson	037c6db85d	Merge branch 'master' into develop	2018-05-24 16:03:44 +01:00
David Baker	7a1af504d7	Remove users from user directory on deactivate	2018-05-24 15:59:58 +01:00
Erik Johnston	f72d5a44d5	Merge pull request #3261 from matrix-org/erikj/pagination_fixes ... Fix federation backfill bugs	2018-05-24 14:52:03 +01:00
Erik Johnston	68399fc4de	Merge pull request #3267 from matrix-org/erikj/iter_filter ... Use iter* methods for _filter_events_for_server	2018-05-24 14:44:57 +01:00
Neil Johnson	91d95a1d8e	bump version	2018-05-24 14:05:07 +01:00
David Baker	9700d15611	pep8	2018-05-24 11:23:15 +01:00
David Baker	a21a41bad7	comment	2018-05-24 11:19:59 +01:00
David Baker	b3bff53178	Unbind 3pids when they're deleted too	2018-05-24 11:08:05 +01:00
Richard van der Hoff	6abcb5d22d	Merge pull request #3273 from matrix-org/rav/server_notices_avatar_url ... Allow overriding the server_notices user's avatar	2018-05-24 10:31:43 +01:00
Amber Brown	389dac2c15	pepeightttt	2018-05-23 13:08:59 -05:00
Amber Brown	472a5ec4e2	add back CPU metrics	2018-05-23 13:03:56 -05:00
Amber Brown	e987079037	fixes	2018-05-23 13:03:51 -05:00
Richard van der Hoff	9bf4b2bda3	Allow overriding the server_notices user's avatar ... probably should have done this in the first place, like @turt2live suggested.	2018-05-23 17:43:30 +01:00
Richard van der Hoff	a0b3946fe2	Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri	2018-05-23 16:06:03 +01:00
Richard van der Hoff	2f7008d4eb	Merge pull request #3271 from matrix-org/rav/consent_uri_in_messages ... Support for putting %(consent_uri)s in messages	2018-05-23 16:04:30 +01:00
Richard van der Hoff	dda40fb55d	fix typo	2018-05-23 15:30:26 +01:00
Richard van der Hoff	3ff6f50eac	Use the localpart in the consent uri ... ... because it's shorter.	2018-05-23 15:28:23 +01:00
Richard van der Hoff	82191b08f6	Support for putting %(consent_uri)s in messages ... Make it possible to put the URI in the error message and the server notice that get sent by the server	2018-05-23 15:24:31 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Richard van der Hoff	321f02d263	Block attempts to send server notices to remote users	2018-05-23 14:30:47 +01:00
Neil Johnson	3601a240aa	bump version and changelog	2018-05-23 12:03:23 +01:00
Erik Johnston	5aaa3189d5	s/values/itervalues/	2018-05-23 10:13:05 +01:00
Erik Johnston	0a4bca4134	Use iter* methods for _filter_events_for_server	2018-05-23 10:04:23 +01:00
Amber Brown	b6063631c3	more cleanup	2018-05-22 17:36:20 -05:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Amber Brown	071206304d	cleanup pep8 errors	2018-05-22 16:54:22 -05:00
Amber Brown	85ba83eb51	fixes	2018-05-22 16:28:23 -05:00
Amber Brown	228f1f584e	fix the test failures	2018-05-22 15:02:38 -05:00
Erik Johnston	e85b5a0ff7	Use iter* methods	2018-05-22 19:02:48 +01:00
Erik Johnston	586b66b197	Fix that states is a dict of dicts	2018-05-22 19:02:36 +01:00
Erik Johnston	a17e901f4d	Remove unused string formatting param	2018-05-22 18:24:32 +01:00
Erik Johnston	5494c1d71e	Don't support limitless pagination ... The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.	2018-05-22 18:15:21 +01:00
Neil Johnson	d8cb7225d2	daily user type phone home stats	2018-05-22 18:09:09 +01:00
hera	ad2823ee27	fix synchrotron	2018-05-22 17:47:42 +01:00
Richard van der Hoff	08bfc48abf	custom error code for not leaving server notices room	2018-05-22 17:27:27 +01:00
David Baker	0a078026ea	comment typo	2018-05-22 17:14:06 +01:00
Amber Brown	8f5a688d42	cleanups, self-registration	2018-05-22 10:56:03 -05:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Erik Johnston	cb2a2ad791	get_domains_from_state returns list of tuples	2018-05-22 16:23:39 +01:00
Richard van der Hoff	08a14b32ae	Merge pull request #3262 from matrix-org/rav/has_already_consented ... Add a 'has_consented' template var to consent forms	2018-05-22 15:35:10 +01:00
Richard van der Hoff	7b36d06a69	Add a 'has_consented' template var to consent forms ... fixes #3260	2018-05-22 14:58:34 +01:00
Richard van der Hoff	669400e22f	Enable auto-escaping for the consent templates ... ... to reduce the risk of somebody introducing an html injection attack...	2018-05-22 14:58:34 +01:00
Richard van der Hoff	b5b2d5d64b	Fix dependency on jinja2 ... Delay the import of ConsentResource, so that we can get away without jinja2 if people don't have the consent resource enabled. Fixes #3259	2018-05-22 14:03:45 +01:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	8aeb529262	Merge pull request #3236 from matrix-org/rav/consent_notice ... Send users a server notice about consent	2018-05-22 11:56:09 +01:00
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	2018-05-22 11:54:51 +01:00
Richard van der Hoff	d5dca9a04f	Move consent config parsing into ConsentConfig ... turns out we need to reuse this, so it's better in the config class.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	d14d7b8fdc	Rename 'version' param on user consent config ... we're going to use it for the version we require too.	2018-05-22 11:54:51 +01:00
Erik Johnston	7948ecf234	Comment	2018-05-22 11:39:43 +01:00
Erik Johnston	020377a550	Fix logcontext resource usage tracking	2018-05-22 11:16:07 +01:00
Erik Johnston	13a8dfba0d	Merge pull request #3252 from matrix-org/erikj/in_flight_requests ... Add in flight request metrics	2018-05-22 09:54:30 +01:00
Erik Johnston	c435b0b441	Don't store context	2018-05-22 09:34:26 +01:00
Erik Johnston	fb2806b186	Move in_flight_requests_count to be a callback metric	2018-05-22 09:31:53 +01:00
Amber Brown	fcc525b0b7	rest of the changes	2018-05-21 19:48:57 -05:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Amber Brown	c60e0d5e02	don't need the resource portion	2018-05-21 17:03:20 -05:00
Amber Brown	02c1d29133	look at the Prometheus metrics instead	2018-05-21 17:02:20 -05:00
Amber Brown	f258deffcb	remove old metrics libs	2018-05-21 17:01:15 -05:00
Richard van der Hoff	413482f578	Merge pull request #3255 from matrix-org/rav/fix_transactions ... Stop the transaction cache caching failures	2018-05-21 17:58:08 +01:00
Richard van der Hoff	6e1cb54a05	Fix logcontext leak in HttpTransactionCache ... ONE DAY I WILL PURGE THE WORLD OF THIS EVIL	2018-05-21 16:58:20 +01:00
Richard van der Hoff	6d6e7288fe	Stop the transaction cache caching failures ... The transaction cache has some code which tries to stop it caching failures, but if the callback function failed straight away, then things would happen backwards and we'd end up with the failure stuck in the cache.	2018-05-21 16:49:59 +01:00
Erik Johnston	dfa70adc33	Add in flight request metrics ... This tracks CPU and DB usage while requests are in flight, rather than when we write the response.	2018-05-21 16:23:06 +01:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Adrian Tschira	d9fe2b2d9d	Replace some more comparisons with six ... plus a bonus b"" string I missed last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:56:31 +02:00
Adrian Tschira	45b55e23d3	Add batch_iter to utils ... There's a frequent idiom I noticed where an iterable is split up into a number of chunks/batches. Unfortunately that method does not work with iterators like dict.keys() in python3. This implementation works with iterators. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:48:30 +02:00
Adrian Tschira	dcc235b47d	use stand-in value if maxint is not available ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:44 +02:00
Adrian Tschira	73cbdef5f7	fix py3 intern and remove unnecessary py3 encode ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:31 +02:00
Adrian Tschira	aafb0f6b0d	py3-ize url preview	2018-05-19 17:35:20 +02:00
Adrian Tschira	b932b4ea25	use repr, not str ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:28:42 +02:00
Neil Johnson	644aac5f73	Tighter filtering for user_daily_visits	2018-05-18 17:10:35 +01:00
Neil Johnson	ef466b3a13	fix psql compatability bug	2018-05-18 15:51:21 +01:00
Neil Johnson	2725223f08	Merge branch 'master' into develop	2018-05-18 14:07:50 +01:00
Richard van der Hoff	6d9dc67139	Merge pull request #3232 from matrix-org/rav/server_notices_room ... Infrastructure for a server notices room	2018-05-18 11:28:04 +01:00
Richard van der Hoff	ed3125b0a1	Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred ... Fix error in handling receipts	2018-05-18 11:23:11 +01:00
Richard van der Hoff	67af392712	Merge pull request #3233 from matrix-org/rav/remove_dead_code ... Remove unused `update_external_syncs`	2018-05-18 11:22:43 +01:00
Richard van der Hoff	011e1f4010	Better docstrings	2018-05-18 11:22:12 +01:00
Richard van der Hoff	26305788fe	Make sure we reject attempts to invite the notices user	2018-05-18 11:18:39 +01:00
Richard van der Hoff	d10707c810	Replace inline docstrings with "Attributes" in class docstring	2018-05-18 11:00:55 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	8b1c856d81	Fix error in handling receipts ... Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234	2018-05-18 09:15:35 +01:00
Neil Johnson	6958459b50	bump version, change log	2018-05-17 21:35:07 +01:00
Richard van der Hoff	88d3405332	fix missing yield for server_notices_room	2018-05-17 18:33:45 +01:00
Richard van der Hoff	d43d480d86	Remove unused update_external_syncs ... This method isn't used anywhere. Burninate it.	2018-05-17 18:22:19 +01:00
Richard van der Hoff	fed62e21ad	Infrastructure for a server notices room ... Server Notices use a special room which the user can't dismiss. They are created on demand when some other bit of the code calls send_notice. (This doesn't actually do much yet becuse we don't call send_notice anywhere)	2018-05-17 17:58:25 +01:00
Richard van der Hoff	f8a1e76d64	Merge pull request #3225 from matrix-org/rav/move_creation_handler ... Move RoomCreationHandler out of synapse.handlers.Handlers	2018-05-17 17:56:42 +01:00
Erik Johnston	f7906203f6	Merge pull request #3212 from matrix-org/erikj/epa_stream ... Use stream rather depth ordering for push actions	2018-05-17 12:01:21 +01:00
Richard van der Hoff	c46367d0d7	Move RoomCreationHandler out of synapse.handlers.Handlers ... Handlers is deprecated nowadays, so let's move this out before I add a new dependency on it. Also fix the docstrings on create_room.	2018-05-17 09:08:42 +01:00
Neil Johnson	85b8acdeb4	Merge branch 'master' into develop	2018-05-16 16:48:10 +01:00
Neil Johnson	3c099219e0	bump version and changelog for 0.29.0	2018-05-16 15:44:59 +01:00
Erik Johnston	680530cc7f	Clarify comment	2018-05-16 11:47:29 +01:00
Erik Johnston	43e6e82c4d	Comments	2018-05-16 11:13:31 +01:00
Neil Johnson	dc8930ea9e	Merge pull request #3163 from matrix-org/cohort_analytics ... user visit data	2018-05-16 10:09:24 +00:00
Erik Johnston	c945af8799	Move and rename variable	2018-05-16 10:52:06 +01:00
Neil Johnson	be11a02c4f	remove empty line	2018-05-16 10:45:40 +01:00
Neil Johnson	a2204cc9cc	remove unused method recurring_user_daily_visit_stats	2018-05-16 09:47:20 +01:00
Neil Johnson	31c2502ca8	style and further contraining query	2018-05-16 09:46:43 +01:00
Richard van der Hoff	8030a825c8	Merge pull request #3213 from matrix-org/rav/consent_handler ... ConsentResource to gather policy consent from users	2018-05-16 07:19:18 +01:00
Neil Johnson	c92a8aa578	pep8	2018-05-15 17:31:11 +01:00
Neil Johnson	05ac15ae82	Limit query load of generate_user_daily_visits ... The aim is to keep track of when it was last called and only query from that point in time	2018-05-15 17:01:33 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
Erik Johnston	37dbee6490	Use events_to_purge table rather than token	2018-05-15 16:23:47 +01:00
Richard van der Hoff	47815edcfa	ConsentResource to gather policy consent from users ... Hopefully there are enough comments and docs in this that it makes sense on its own.	2018-05-15 15:11:59 +01:00
Neil Johnson	aea80a0118	v0.29.0-rc1: bump version and change log	2018-05-14 15:50:57 +01:00
Neil Johnson	f077e97914	instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day	2018-05-14 13:50:58 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
Neil Johnson	977765bde2	Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics	2018-05-14 09:31:42 +01:00
Richard van der Hoff	c25d7ba12e	Merge pull request #3208 from matrix-org/rav/more_refactor_request_handler ... Set Server header in SynapseRequest	2018-05-11 16:07:47 +01:00
Erik Johnston	6406b70aeb	Use stream rather depth ordering for push actions ... This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.	2018-05-11 15:30:11 +01:00
Richard van der Hoff	23e2dfe940	Merge pull request #3209 from damir-manapov/master ... transaction_id, destination defined twice	2018-05-11 00:35:13 +01:00
Damir Manapov	db18d854cd	transaction_id, destination twice	2018-05-10 22:13:31 +03:00
Richard van der Hoff	318711e139	Set Server header in SynapseRequest ... (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.	2018-05-10 18:50:27 +01:00
David Baker	6b49628e3b	Catch failure to part user from room	2018-05-10 12:23:53 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
Richard van der Hoff	645cb4bf06	Remove redundant request_handler decorator ... This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.	2018-05-10 12:19:53 +01:00
Richard van der Hoff	09f570b935	Factor wrap_request_handler_with_logging out of wrap_request_handler ... ... so that it can be used on non-JSON endpoints	2018-05-10 12:19:52 +01:00
Richard van der Hoff	9589a1925e	Remove include_metrics param ... The metrics are now available via the request, so this is redundant and can go away at last.	2018-05-10 12:19:52 +01:00
Richard van der Hoff	49e5a613f1	Move outgoing_responses_counter handling to RequestMetrics ... it's much neater there.	2018-05-10 12:19:52 +01:00
Richard van der Hoff	b8700dd7d0	Bump requests_counter in wrapped_request_handler ... less magic	2018-05-10 12:19:52 +01:00
Richard van der Hoff	c6f730282c	Move RequestMetrics handling into SynapseRequest.processing() ... It fits quite nicely here, and opens the path to getting rid of the "include_metrics" mess.	2018-05-10 12:19:51 +01:00
Richard van der Hoff	09b29f9c4a	Make RequestMetrics take a raw time rather than a clock ... ... which is going to make it easier to move around.	2018-05-10 12:18:52 +01:00
David Baker	4d298506dd	Oops, don't call function passed to run_in_background	2018-05-10 11:57:13 +01:00
Richard van der Hoff	8460e48d06	Move request_id management into SynapseRequest	2018-05-10 11:48:17 +01:00
Richard van der Hoff	18e144fe08	Move RequestsMetrics to its own file ... This is useful in its own right, because server.py is full of stuff; but more importantly, I want to do some refactoring that will cause a circular reference as it is.	2018-05-09 19:55:03 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	5adb75bcba	Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination ... Refactor get_recent_events_for_room return type	2018-05-09 16:07:14 +01:00
Erik Johnston	d26bec8a43	Add comment to sync as to why code path is split	2018-05-09 15:56:07 +01:00
Erik Johnston	fcf55f2255	Fix returned token is no longer a tuple	2018-05-09 15:43:00 +01:00
Erik Johnston	7ce98804ff	Fix up comment	2018-05-09 15:42:39 +01:00
Erik Johnston	cddf91c8b9	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change	2018-05-09 15:32:07 +01:00
Erik Johnston	9896dab8f6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination	2018-05-09 15:31:33 +01:00
Erik Johnston	1e5280b7d0	Merge pull request #3196 from matrix-org/erikj/pagination_return ... Refactor pagination DB API to return concrete type	2018-05-09 15:27:17 +01:00
Erik Johnston	75552d2148	Update comments	2018-05-09 15:15:38 +01:00
David Baker	294e9a0c9b	Prefix internal functions	2018-05-09 15:10:37 +01:00
David Baker	46df23f581	Add the schema file	2018-05-09 15:07:54 +01:00
David Baker	52281e4c54	Indent fail	2018-05-09 15:06:16 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	c0e08dc45b	Remove unused code path from member change DB func ... The function is never called without a from_key, so we can remove all the handling for that scenario.	2018-05-09 14:31:32 +01:00
Erik Johnston	0461ef01b7	Merge pull request #3195 from matrix-org/erikj/pagination_refactor ... Refactor recent events func to use pagination func	2018-05-09 14:12:24 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	e5ab9cd24b	Don't unnecessarily require token to be stream token ... This allows calling the `get_recent_event_ids_for_room` function in more situations.	2018-05-09 11:58:35 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
Erik Johnston	05e0a2462c	Refactor pagination DB API to return concrete type ... This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.	2018-05-09 11:34:24 +01:00
Erik Johnston	7dd13415db	Remove unused from_token param	2018-05-09 10:58:16 +01:00
Erik Johnston	27cf170558	Refactor recent events func to use pagination func ... This also removes a cache that is unlikely to ever get hit.	2018-05-09 10:55:55 +01:00
Erik Johnston	1aeb5e28a9	Merge pull request #3193 from matrix-org/erikj/pagination_refactor ... Refactor /context to reuse pagination storage functions	2018-05-09 10:22:38 +01:00
Erik Johnston	23ec51c94c	Fix up comments and make function private	2018-05-09 09:55:19 +01:00
Erik Johnston	696f532453	Reuse existing pagination code for context API	2018-05-08 16:20:19 +01:00
Erik Johnston	3e6d306e94	Parse tokens before calling DB function	2018-05-08 16:18:58 +01:00
Erik Johnston	274b8c6025	Only fetch required fields from database	2018-05-08 16:15:25 +01:00
Erik Johnston	06c0d0ed08	Split paginate_room_events storage function	2018-05-08 16:14:26 +01:00
David Baker	bf98fa0864	Part user from rooms on account deactivate ... This implements this very crudely: this probably isn't viable because parting a user from all their rooms could take a long time, and if the HS gets restarted in that time the process will be aborted.	2018-05-08 15:58:35 +01:00
Richard van der Hoff	678e649b78	Merge pull request #3190 from mujx/notif-token-fix ... notifications: Convert next_token to string according to the spec	2018-05-08 13:54:47 +01:00
Konstantinos Sideris	88868b2839	notifications: Convert next_token to string according to the spec ... Currently the parameter is serialized as an integer. Signed-off-by: Konstantinos Sideris <sideris.konstantin@gmail.com>	2018-05-05 12:55:02 +03:00
Erik Johnston	6d8ec3462d	Note that label values can be anything	2018-05-03 16:25:05 +01:00
Erik Johnston	95b6912045	Fix metrics that have integer value labels	2018-05-03 15:51:04 +01:00
Richard van der Hoff	093d8c415a	Merge remote-tracking branch 'origin/develop' into rav/warn_on_logcontext_fail	2018-05-03 14:59:29 +01:00
Richard van der Hoff	0ba609dc6f	Merge pull request #3183 from matrix-org/rav/moar_logcontext_leaks ... Fix logcontext leaks in rate limiter	2018-05-03 14:58:13 +01:00
Richard van der Hoff	2117f84323	Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdown ... Fix 'Unhandled Error' logs with Twisted 18.4	2018-05-03 12:40:11 +01:00
Richard van der Hoff	a7fe62f0cb	Fix logcontext leaks in rate limiter	2018-05-03 12:31:59 +01:00
Will Hunt	2e7a94c36b	Don't abortConnection() if the transport connection has already closed.	2018-05-03 12:31:47 +01:00
Richard van der Hoff	a2aaa9cb3c	Merge pull request #3178 from matrix-org/rav/fix_request_timeouts ... fix http request timeout code	2018-05-03 11:33:26 +01:00
Erik Johnston	0a3b51c420	Merge pull request #3141 from matrix-org/erikj/fixup_state ... Refactor event storage to prepare for changes in state calculations	2018-05-03 10:39:20 +01:00
Erik Johnston	31c7c29d43	Fix up grammar	2018-05-03 10:38:58 +01:00
Richard van der Hoff	902673e356	Merge pull request #3161 from NotAFile/remove-v1auth ... Make Client-Server API return 403 for invalid token	2018-05-03 10:10:57 +01:00
Erik Johnston	53a5fdf312	Merge pull request #3175 from matrix-org/erikj/escape_metric_values ... Escape label values in prometheus metrics	2018-05-03 10:01:04 +01:00
Richard van der Hoff	1dfd650348	add missing param to cancelled_to_request_timed_out_error ... This gets two arguments, not one.	2018-05-02 22:42:36 +01:00
Erik Johnston	a41117c63b	Make _escape_character take MatchObject	2018-05-02 17:27:27 +01:00
Erik Johnston	32015e1109	Escape label values in prometheus metrics	2018-05-02 16:52:42 +01:00
Richard van der Hoff	3a42aed9a1	Merge pull request #3170 from matrix-org/rav/more_logcontext_leaks ... Fix a class of logcontext leaks	2018-05-02 16:45:51 +01:00
Richard van der Hoff	415c6b672e	Merge branch 'develop' into rav/more_logcontext_leaks	2018-05-02 16:16:01 +01:00
Richard van der Hoff	be31adb036	Fix logcontext leak in media repo ... Make FileResponder.write_to_consumer uphold the logcontext contract	2018-05-02 16:14:50 +01:00
Richard van der Hoff	f22e7cda2c	Fix a class of logcontext leaks ... So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.	2018-05-02 11:58:00 +01:00
Richard van der Hoff	a8d8bf92e0	Merge pull request #3168 from matrix-org/rav/fix_logformatter ... Fix incorrect reference to StringIO	2018-05-02 10:03:36 +01:00
Richard van der Hoff	e482f8cd85	Fix incorrect reference to StringIO ... This was introduced in 4f2f5171	2018-05-02 09:12:26 +01:00
Neil Johnson	2414178ed6	Merge branch 'master' into develop	2018-05-01 18:53:56 +01:00
Neil Johnson	8570bb84cc	Update __init__.py ... bump version	2018-05-01 18:22:53 +01:00
Richard van der Hoff	ca7211104e	Merge branch 'release-v0.28.1' into develop	2018-05-01 18:16:57 +01:00
Richard van der Hoff	d5eee5d601	Merge commit '33f469b' into release-v0.28.1	2018-05-01 18:14:18 +01:00
Richard van der Hoff	d858f3bd4e	Miscellaneous fixes to python_dependencies ... * add some doc about wtf this thing does * pin Twisted to < 18.4 * add explicit dep on six (fixes #3089)	2018-05-01 18:13:54 +01:00
Richard van der Hoff	33f469ba19	Apply some limits to depth to counter abuse ... * When creating a new event, cap its depth to 2^63 - 1 * When receiving events, reject any without a sensible depth As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI	2018-05-01 17:54:19 +01:00
Neil Johnson	dd1a832419	remove user agent from data model, will just join on user_ips	2018-05-01 12:13:49 +01:00
Neil Johnson	d0857702e8	add inidexes based on usage	2018-05-01 12:12:57 +01:00
Neil Johnson	5917562b60	10 mins seems more reasonable that every minute	2018-05-01 12:12:22 +01:00
Adrian Tschira	6495dbb326	Burminate v1auth ... This closes #2602 v1auth was created to account for the differences in status code between the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid tokens). However since those protocols were merged, this makes the r0 version/endpoint internally inconsistent, and violates the specification for the r0 endpoint. This might break clients that rely on this inconsistency with the specification. This is said to affect the legacy angular reference client. However, I feel that restoring parity with the spec is more important. Either way, it is critical to inform developers about this change, in case they rely on the illegal behaviour. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-30 22:20:43 +02:00
Will Hunt	2ad3fc36e6	Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve (#3157) ... fixes #3135 Signed-off-by: Will Hunt will@half-shot.uk	2018-04-30 16:21:11 +01:00
Krombel	576b71dd3d	add guard for None on purge_history api	2018-04-30 14:29:48 +02:00
Matthew Hodgson	99a54bf2af	Merge pull request #3129 from matrix-org/matthew/fix_group_dups ... remove duplicates from groups tables	2018-04-30 11:47:25 +01:00
Richard van der Hoff	63ae5cbf34	Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn ... Remove redundant call to preserve_fn	2018-04-30 10:23:59 +01:00
Richard van der Hoff	fdb6849b81	Merge pull request #3144 from matrix-org/rav/run_in_background_exception_handling ... Trap exceptions thrown within run_in_background	2018-04-30 10:23:02 +01:00
Richard van der Hoff	01e8a52825	Merge pull request #3102 from NotAFile/py3-attributeerror ... Make event properties raise AttributeError instead	2018-04-30 09:22:09 +01:00
Adrian Tschira	0c9db26260	add comment explaining attributeerror	2018-04-30 09:49:10 +02:00
Richard van der Hoff	950a32eb47	Merge pull request #3152 from NotAFile/py3-local-imports ... make imports local	2018-04-30 01:28:13 +01:00
Richard van der Hoff	bc2017a594	Merge pull request #3153 from NotAFile/py3-httplib ... move httplib import to six	2018-04-30 01:26:42 +01:00
Richard van der Hoff	683149c1f9	Merge pull request #3151 from NotAFile/py3-xrange-1 ... Move more xrange to six	2018-04-30 01:20:06 +01:00
Richard van der Hoff	3b0e431c82	Merge pull request #3150 from NotAFile/py3-listcomp-yield ... Don't yield in list comprehensions	2018-04-30 01:11:41 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Richard van der Hoff	2fd96727b1	Merge pull request #3085 from NotAFile/py3-config-text-mode ... Open config file in non-bytes mode	2018-04-30 01:00:23 +01:00
Richard van der Hoff	b8ee12b978	Merge pull request #3084 from NotAFile/py3-certs-byte-mode ... Open certificate files as bytes	2018-04-30 01:00:05 +01:00
Richard van der Hoff	049b0b5af2	Merge pull request #3154 from NotAFile/py3-stringio ... Replace stringIO imports with six	2018-04-30 00:59:04 +01:00
Richard van der Hoff	af3cc50511	Remove redundant call to preserve_fn ... submit_event_for_as doesn't return a deferred anyway, so this is pointless.	2018-04-30 00:48:36 +01:00
Richard van der Hoff	dbf6f28d64	Merge pull request #3155 from NotAFile/py3-bytes-1 ... more bytes strings	2018-04-30 00:38:21 +01:00
Richard van der Hoff	aab2e4da60	Merge pull request #3140 from matrix-org/rav/use_run_in_background ... Use run_in_background in preference to preserve_fn	2018-04-30 00:34:28 +01:00
Richard van der Hoff	1315d374cc	Merge pull request #3156 from NotAFile/py3-hmac-bytes ... Construct HMAC as bytes on py3	2018-04-30 00:33:20 +01:00
Richard van der Hoff	9e2601f830	Merge pull request #3108 from NotAFile/py3-six-urlparse ... Use six.moves.urlparse	2018-04-30 00:33:05 +01:00
Adrian Tschira	122593265b	Construct HMAC as bytes on py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-29 00:19:41 +02:00
Adrian Tschira	e9143b6593	more bytes strings ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-29 00:13:57 +02:00
Matthew Hodgson	adaf3ec87f	fix missing import	2018-04-28 22:39:15 +01:00
Matthew Hodgson	006e18b6bb	pep8	2018-04-28 22:32:24 +01:00
Matthew Hodgson	42c89c8215	make it work with sqlite	2018-04-28 22:27:30 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Adrian Tschira	4f2f5171b7	replace stringIO imports	2018-04-28 13:46:23 +02:00
Adrian Tschira	94f4d7f49e	move httplib import to six	2018-04-28 13:43:34 +02:00
Adrian Tschira	57b58e2174	make imports local ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:41:41 +02:00
Adrian Tschira	cdb4647a80	Don't yield in list comprehensions ... I've tried to grep for more of this with no success. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:36:30 +02:00
Adrian Tschira	a376d8f761	open log_config in text mode too ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:34:13 +02:00
Richard van der Hoff	453adf00b6	pep8; remove spurious import	2018-04-27 14:32:08 +01:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	6146332387	Merge remote-tracking branch 'origin/develop' into rav/deferred_timeout	2018-04-27 14:18:00 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	3d1ae61399	Merge branch 'develop' into rav/deferred_timeout	2018-04-27 12:54:43 +01:00
Richard van der Hoff	9d2c1b8429	Backport deferred.addTimeout ... Twisted 16.0 doesn't have addTimeout, so let's backport it.	2018-04-27 12:52:30 +01:00
Richard van der Hoff	13843f771e	Trap exceptions thrown within run_in_background ... Turn any exceptions that get thrown synchronously within run_in_background into Failures instead.	2018-04-27 12:17:13 +01:00
Richard van der Hoff	41d4b07a53	Merge pull request #3142 from matrix-org/rav/reraise ... reraise exceptions more carefully	2018-04-27 12:16:19 +01:00
Richard van der Hoff	268e40341b	Merge pull request #3136 from matrix-org/rav/fix_dependencies ... Miscellaneous fixes to python_dependencies	2018-04-27 11:48:28 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Richard van der Hoff	53494c34df	Merge pull request #3139 from matrix-org/rav/consume_errors ... Add missing consumeErrors to improve exception handling	2018-04-27 11:47:21 +01:00
Richard van der Hoff	6493b22b42	reraise exceptions more carefully ... We need to be careful (under python 2, at least) that when we reraise an exception after doing some error handling, we actually reraise the original exception rather than anything that might have been raised (and handled) during the error handling.	2018-04-27 11:40:06 +01:00
Erik Johnston	6e10eed28e	Refactor event storage to not require state ... This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.	2018-04-27 11:38:02 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Neil Johnson	28dd536e80	update changelog and bump version to 0.28.0	2018-04-26 15:51:39 +01:00
Richard van der Hoff	d78ada3166	Miscellaneous fixes to python_dependencies ... * add some doc about wtf this thing does * pin Twisted to < 18.4 * add explicit dep on six (fixes #3089)	2018-04-26 13:11:03 +01:00
Erik Johnston	7ec8e798b4	Fix media admin APIs	2018-04-26 11:31:22 +01:00
Neil Johnson	fb6015d0a6	pep8	2018-04-25 17:56:11 +01:00
Neil Johnson	617bf40924	Generate user daily stats	2018-04-25 17:37:29 +01:00
Erik Johnston	22881b3d69	Also fix reindexing of search	2018-04-25 15:32:04 +01:00
Erik Johnston	ba3166743c	Fix quarantine media admin API	2018-04-25 15:11:18 +01:00
Matthew Hodgson	e3a373f002	remove duplicates from groups tables ... and rename inconsistently named indexes. Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!	2018-04-25 14:58:43 +01:00
Neil Johnson	2c3e995f38	Bump version and update changelog	2018-04-24 15:33:22 +01:00
Richard van der Hoff	77ebef9d43	Merge pull request #3118 from matrix-org/rav/reject_prev_events ... Reject events which have lots of prev_events	2018-04-23 17:51:38 +01:00
Richard van der Hoff	9b9c38373c	Remove spurious param	2018-04-23 12:00:06 +01:00
Richard van der Hoff	1ea904b9f0	Use deferred.addTimeout instead of time_bound_deferred ... This doesn't feel like a wheel we need to reinvent.	2018-04-23 00:53:18 +01:00
Richard van der Hoff	dc875d2712	Merge pull request #3106 from NotAFile/py3-six-itervalues-1 ... Use six.itervalues in some places	2018-04-20 15:43:52 +01:00
Richard van der Hoff	8dc4a6144b	Merge pull request #3107 from NotAFile/py3-bool-nonzero ... add __bool__ alias to __nonzero__ methods	2018-04-20 15:43:39 +01:00
Richard van der Hoff	c09a6daf09	Merge pull request #3110 from NotAFile/py3-six-queue ... Replace Queue with six.moves.queue	2018-04-20 15:35:00 +01:00
Richard van der Hoff	692a3cc806	Merge pull request #3103 from NotAFile/py3-baseexcepton-message ... Use str(e) instead of e.message	2018-04-20 15:34:49 +01:00
Richard van der Hoff	3de7d9fe99	accept stupid events over backfill	2018-04-20 11:41:03 +01:00
Richard van der Hoff	11a67b7c9d	Merge pull request #3093 from matrix-org/rav/response_cache_wrap ... Refactor ResponseCache usage	2018-04-20 11:31:17 +01:00
Richard van der Hoff	0c280d4d99	Reinstate linearizer for federation_server.on_context_state_request	2018-04-20 11:10:04 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	1f4b498b73	Add some comments	2018-04-18 00:15:36 +01:00
Richard van der Hoff	e585228860	Check events on backfill too	2018-04-18 00:06:42 +01:00
Richard van der Hoff	9b7794262f	Reject events which have too many auth_events or prev_events ... ... this should protect us from being dossed by people making silly events (deliberately or otherwise)	2018-04-18 00:06:42 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	878995e660	Replace Queue with six.moves.queue ... and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-16 00:46:21 +02:00
Adrian Tschira	2a3c33ff03	Use six.moves.urlparse ... The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 21:22:43 +02:00
Adrian Tschira	f63ff73c7f	add __bool__ alias to __nonzero__ methods ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:40:47 +02:00
Adrian Tschira	36c59ce669	Use six.itervalues in some places ... There's more where that came from Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:39:43 +02:00
Adrian Tschira	1515560f5c	Use str(e) instead of e.message ... Doing this I learned e.message was pretty shortlived, added in 2.6, they realized it was a bad idea and deprecated it in 2.7 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:32:42 +02:00
Adrian Tschira	bfc2ade9b3	Make event properties raise AttributeError instead ... They raised KeyError before. I'm changing this because the code uses hasattr() to check for the presence of a key. This worked accidentally before, because hasattr() silences all exceptions in python 2. However, in python3, this isn't the case anymore. I had a look around to see if anything depended on this raising a KeyError and I couldn't find anything. Of course, I could have simply missed it. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:16:59 +02:00
Neil Johnson	154b44c249	Merge branch 'master' of https://github.com/matrix-org/synapse into develop	2018-04-13 17:07:54 +01:00
Matthew Hodgson	78a9698650	fix federation_domain_whitelist ... we were checking the wrong server_name on inbound requests	2018-04-13 15:47:43 +01:00
Matthew Hodgson	25b0ba30b1	revert last to PR properly	2018-04-13 15:46:37 +01:00
Matthew Hodgson	f8d46cad3c	correctly auth inbound federation_domain_whitelist reqs	2018-04-13 15:41:52 +01:00
Neil Johnson	d4b2e05852	Merge branch 'master' of https://github.com/matrix-org/synapse into develop	2018-04-13 12:16:27 +01:00
Neil Johnson	51d628d28d	Bump version and Change log	2018-04-13 12:08:19 +01:00
Richard van der Hoff	d3347ad485	Revert "Use sortedcontainers instead of blist" ... This reverts commit 9fbe70a7dc. It turns out that sortedcontainers.SortedDict is not an exact match for blist.sorteddict; in particular, `popitem()` removes things from the opposite end of the dict. This is trivial to fix, but I want to add some unit tests, and potentially some more thought about it, before we do so.	2018-04-13 11:16:43 +01:00
Richard van der Hoff	fac3f9e678	Bump canonicaljson to 1.1.3 ... 1.1.2 was a bit broken too :/	2018-04-13 10:21:38 +01:00
Richard van der Hoff	60f6014bb7	ResponseCache: fix handling of completed results ... Turns out that ObservableDeferred.observe doesn't return a deferred if the result is already completed. Fix handling and improve documentation.	2018-04-13 07:32:29 +01:00
Richard van der Hoff	119596ab8f	Update canonicaljson dependency ... 1.1.0 and 1.1.1 were broken, so we're updating this to help people make sure they don't end up on a broken version. Also, 1.1.0 is speedier...	2018-04-12 17:31:44 +01:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	d5c74b9f6c	Merge pull request #3092 from matrix-org/rav/response_cache_metrics ... Add metrics for ResponseCache	2018-04-12 12:59:36 +01:00
Erik Johnston	415aeefd89	Format docstring	2018-04-12 12:07:09 +01:00
Erik Johnston	19ceb4851f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag	2018-04-12 11:36:07 +01:00
Richard van der Hoff	261124396e	Merge pull request #3059 from matrix-org/rav/doc_response_cache ... Document the behaviour of ResponseCache	2018-04-12 11:22:30 +01:00
Erik Johnston	23a7f9d7f4	Doc we raise on unknown event	2018-04-12 11:20:51 +01:00
Erik Johnston	d7bf3a68f0	s/list/tuple	2018-04-12 11:19:04 +01:00
Erik Johnston	f67e906e18	Set all metrics at the same time	2018-04-12 11:18:19 +01:00
Erik Johnston	971059a733	Merge pull request #3088 from matrix-org/erikj/as_parallel ... Send events to ASes concurrently	2018-04-12 10:42:36 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Erik Johnston	ab825aa328	Add GaugeMetric	2018-04-11 12:13:40 +01:00
Richard van der Hoff	233699c42e	Merge pull request #2760 from Valodim/pypy ... Synapse on PyPy	2018-04-11 11:20:01 +01:00
Neil Johnson	781cd8c54f	bump version/changelog	2018-04-11 10:54:43 +01:00
Neil Johnson	9ef0b179e0	Merge commit '11d2609da70af797405241cdf7d9df19db5628f2' of https://github.com/matrix-org/synapse into release-v0.27.0	2018-04-11 10:51:59 +01:00
Erik Johnston	121591568b	Send events to ASes concurrently	2018-04-11 09:56:00 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Neil Johnson	d54cfbb7a8	fix typo	2018-04-10 17:38:16 +01:00
Erik Johnston	eaa2ebf20b	Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends ... Limit concurrent event sends for a room	2018-04-10 16:43:58 +01:00
Erik Johnston	9daf82278f	Merge pull request #3078 from matrix-org/erikj/federation_sender ... Send federation events concurrently	2018-04-10 16:43:48 +01:00
Adrian Tschira	a3f9ddbede	Open certificate files as bytes ... That's what pyOpenSSL expects on python3 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-10 17:36:29 +02:00
Adrian Tschira	7f8eebc8ee	Open config file in non-bytes mode ... Nothing written into it is encoded, so it makes little sense, but it does break in python3 the way it was before. The variable names were adjusted to be less misleading. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-10 17:32:40 +02:00
Neil Johnson	dd723267b2	Merge branch 'release-v0.27.0' of https://github.com/matrix-org/synapse into develop	2018-04-10 15:03:29 +01:00
Erik Johnston	a060dfa132	Use run_in_background instead	2018-04-10 14:25:11 +01:00
Erik Johnston	f8e8ec013b	Note why we're limiting concurrent event sends	2018-04-10 14:00:46 +01:00
Erik Johnston	1246d23710	Preserve log contexts correctly	2018-04-10 12:04:32 +01:00
Erik Johnston	d49cbf712f	Log event ID on exception	2018-04-10 12:03:41 +01:00
Erik Johnston	ce72d590ed	Merge pull request #3082 from matrix-org/erikj/urlencode_paths ... URL quote path segments over federation	2018-04-10 11:31:16 +01:00
Erik Johnston	11d2609da7	Ensure slashes are escaped	2018-04-10 11:24:40 +01:00
Erik Johnston	dab87b84a3	URL quote path segments over federation	2018-04-10 11:16:08 +01:00
Vincent Breitmoser	6d7f0f8dd3	Don't disable GC when running on PyPy ... PyPy's incminimark GC can't be triggered manually. From what I observed there are no obvious issues with just letting it run normally. And unlike CPython, it actually returns unused RAM to the system. Signed-off-by: Vincent Breitmoser <look@my.amazin.horse>	2018-04-10 11:35:34 +02:00
Vincent Breitmoser	f4284d943a	In DomainSpecificString, override __repr__ in addition to __str__ ... For some reason, string interpolation on a DomainSpecificString object like "%r" % (domainSpecificStringObj) fails under PyPy, because the default __repr__ implementation wants to iterate over the object. I'm not sure why that happens, but overriding __repr__ instead of __str__ fixes this problem, and is arguably the more appropriate thing to do anyways.	2018-04-10 11:35:29 +02:00
Richard van der Hoff	d1e56cfcd1	Fix pep8 error on psycopg2cffi hack	2018-04-10 11:35:29 +02:00
Vincent Breitmoser	89de934981	Use psycopg2cffi module instead of psycopg2 if running on pypy ... The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.	2018-04-10 11:29:52 +02:00
Vincent Breitmoser	9fbe70a7dc	Use sortedcontainers instead of blist ... This commit drop-in replaces blist with SortedContainers. They are written in pure python so work with pypy, but perform as good as native implementations, at least in a couple benchmarks: http://www.grantjenks.com/docs/sortedcontainers/performance.html	2018-04-10 11:29:51 +02:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	87478c5a60	Merge pull request #3061 from NotAFile/add-some-byte-strings ... Add b prefixes to some strings that are bytes in py3	2018-04-09 23:54:05 +01:00
Richard van der Hoff	c508b2f2f0	Merge pull request #3073 from NotAFile/use-six-reraise ... Replace old-style raise with six.reraise	2018-04-09 23:53:40 +01:00
Richard van der Hoff	37354b55c9	Merge pull request #2938 from dklug/develop ... Return 401 for invalid access_token on logout	2018-04-09 23:52:56 +01:00
Richard van der Hoff	0e9aa1d091	Merge pull request #3074 from NotAFile/fix-py3-prints ... use python3-compatible prints	2018-04-09 23:44:41 +01:00
Richard van der Hoff	8eaa141d8f	Merge pull request #3075 from NotAFile/six-type-checks ... Replace some type checks with six type checks	2018-04-09 23:40:44 +01:00
Richard van der Hoff	664adb4236	Merge pull request #3016 from silkeh/improve-service-lookups ... Improve handling of SRV records for federation connections	2018-04-09 23:40:06 +01:00
Neil Johnson	41e0611895	remove errant print	2018-04-09 18:44:20 +01:00
Neil Johnson	61b439c904	Fix msec to sec, again	2018-04-09 18:43:48 +01:00
Neil Johnson	87770300d5	Fix msec to sec	2018-04-09 18:38:59 +01:00
Neil Johnson	64bc2162ef	Fix psycopg2 interpolation	2018-04-09 17:50:36 +01:00
Richard van der Hoff	d2c6f4d626	Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin ... Return a 404 rather than a 500 on rejoining empty rooms	2018-04-09 17:32:36 +01:00
Neil Johnson	5232d3bfb1	version bump v0.27.3-rc2	2018-04-09 17:25:57 +01:00
Neil Johnson	5e785d4d5b	Merge branch 'develop' of https://github.com/matrix-org/synapse into release-v0.27.0	2018-04-09 17:21:34 +01:00
Erik Johnston	6e025a97b4	Handle all events in a room correctly	2018-04-09 16:02:48 +01:00
Neil Johnson	d9ae2bc826	bump version to release candidate	2018-04-09 16:00:31 +01:00
Neil Johnson	072fb59446	bump version	2018-04-09 13:49:25 +01:00
Richard van der Hoff	13decdbf96	Revert "Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics" ... We aren't ready to release this yet, so I'm reverting it for now. This reverts commit d1679a4ed7, reversing changes made to e089100c62.	2018-04-09 12:59:12 +01:00
Richard van der Hoff	f3ef60662f	Return a 404 rather than a 500 on rejoining empty rooms ... Filter ourselves out of the server list before checking for an empty remote host list, to fix 500 error Fixes #2141	2018-04-09 12:56:22 +01:00
Erik Johnston	e5082494eb	Limit concurrent event sends for a room	2018-04-09 12:07:39 +01:00
Erik Johnston	56b0589865	Use create_and_send_nonmember_event everywhere	2018-04-09 12:04:18 +01:00
Erik Johnston	11974f3787	Send federation events concurrently	2018-04-09 11:47:10 +01:00
Erik Johnston	145d14656b	Handle exceptions in get_hosts_for_room when sending events over federation	2018-04-09 11:47:01 +01:00
Adrian Tschira	e54c202b81	Replace some type checks with six type checks ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-07 01:02:32 +02:00
Adrian Tschira	b0500d3774	use python3-compatible prints	2018-04-06 23:35:27 +02:00
Adrian Tschira	4f40d058cc	Replace old-style raise with six.reraise ... The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-06 23:06:24 +02:00
Luke Barnard	135fc5b9cd	Merge pull request #3046 from matrix-org/dbkr/join_group ... Implement group join API	2018-04-06 16:24:32 +01:00
Luke Barnard	020a501354	de-lint, quote consistency	2018-04-06 16:02:06 +01:00
Luke Barnard	db2fd801f7	Explicitly grab individual columns from group object	2018-04-06 15:57:25 +01:00
Luke Barnard	7945435587	When exposing group state, return is_openly_joinable ... as opposed to join_policy, which is really only pertinent to the synapse implementation of the group server. By doing this we keep the group server concept extensible by allowing arbitrarily complex rules for deciding whether a group is openly joinable.	2018-04-06 15:43:27 +01:00
Luke Barnard	6bd1b7053e	By default, join policy is "invite"	2018-04-06 15:43:27 +01:00
Luke Barnard	b4478e586f	add_user -> _add_user	2018-04-06 15:43:27 +01:00
Luke Barnard	112c2253e2	pep8	2018-04-06 15:43:27 +01:00
Luke Barnard	6850f8aea3	Get group_info from existing call to check_group_is_ours	2018-04-06 15:43:27 +01:00
Luke Barnard	cd087a265d	Don't use redundant inlineCallbacks	2018-04-06 15:43:27 +01:00
Luke Barnard	87c864b698	join_rule -> join_policy	2018-04-06 15:43:27 +01:00
Luke Barnard	ae85c7804e	is_joinable -> join_rule	2018-04-06 15:43:27 +01:00
Luke Barnard	f8d1917fce	Fix federation client set_group_joinable typo	2018-04-06 15:43:27 +01:00
Luke Barnard	6eb3aa94b6	Factor out add_user from accept_invite and join_group	2018-04-06 15:43:27 +01:00
David Baker	edb45aae38	pep8	2018-04-06 15:43:27 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Erik Johnston	7b824f1475	Add response size metrics	2018-04-06 13:20:11 +01:00
Krombel	1d71f484d4	use PUT instead of POST for federating groups/m.join_policy	2018-04-06 12:54:09 +02:00
Richard van der Hoff	15e8ed874f	more verbosity in synctl	2018-04-06 09:28:36 +01:00
Richard van der Hoff	c7f0969731	Merge pull request #2986 from jplatte/join_reponse_room_id ... Add room_id to the response of `rooms/{roomId}/join`	2018-04-05 17:29:06 +01:00
Richard van der Hoff	3449da3bc7	Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation ... Improve database cache performance	2018-04-05 17:21:44 +01:00
Richard van der Hoff	d1679a4ed7	Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics ... Remove redundant metrics which were deprecated in 0.27.0.	2018-04-05 17:21:18 +01:00
Richard van der Hoff	01afc563c3	Fix overzealous cache invalidation ... Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.	2018-04-05 16:24:04 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Neil Johnson	68b0ee4e8d	Merge pull request #3041 from matrix-org/r30_stats ... R30 stats	2018-04-05 15:37:37 +01:00
Richard van der Hoff	22284a6f65	Merge pull request #3060 from matrix-org/rav/kill_event_content ... Remove uses of events.content	2018-04-05 15:02:17 +01:00
Luke Barnard	917380e89d	NON NULL -> NOT NULL	2018-04-05 14:32:12 +01:00
Luke Barnard	104c0bc1d5	Use "/settings/" (plural)	2018-04-05 14:07:16 +01:00
Luke Barnard	700e5e7198	Use DEFAULT join_policy of "invite" in db	2018-04-05 14:01:17 +01:00
Luke Barnard	b214a04ffc	Document set_group_join_policy	2018-04-05 13:29:16 +01:00
Neil Johnson	0e5f479fc0	Review comments ... Use iteritems over item to loop over dict formatting	2018-04-05 12:16:46 +01:00
Richard van der Hoff	518f6de088	Remove redundant metrics which were deprecated in 0.27.0.	2018-04-04 19:46:28 +01:00
Jan Christian Grünhage	e4570c53dd	phone home cache size configurations	2018-04-04 16:46:58 +01:00
Adrian Tschira	6168351877	Add b prefixes to some strings that are bytes in py3 ... This has no effect on python2 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-04 13:48:51 +02:00
Richard van der Hoff	f92963f5db	Revert "improve mxid check performance"	2018-04-04 12:08:29 +01:00
Silke	72251d1b97	Remove address resolution of hosts in SRV records ... Signed-off-by: Silke Hofstra <silke@slxh.eu>	2018-04-04 12:26:50 +02:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Richard van der Hoff	a89f9f830c	Merge pull request #3044 from matrix-org/michaelk/performance_stats ... Add basic performance statistics to phone home	2018-04-04 10:37:25 +01:00
Richard van der Hoff	39ce38b024	Merge pull request #3053 from NotAFile/speedup-mxid-check ... improve mxid check performance	2018-04-04 10:19:52 +01:00
Richard van der Hoff	a9a74101a4	Document the behaviour of ResponseCache ... it looks like everything that uses ResponseCache expects to have to `make_deferred_yieldable` its results. It's debatable whether that is the best approach, but let's document it for now to avoid further confusion.	2018-04-04 09:06:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	3ee4ad09eb	Fix json encoding bug in replication ... json encoders have an encode method, not a dumps method.	2018-04-03 15:09:48 +01:00
Adrian Tschira	11597ddea5	improve mxid check performance ~4x ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-31 02:00:22 +02:00
Richard van der Hoff	2fe3f848b9	Remove uses of events.content	2018-03-29 23:17:12 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Neil Johnson	b4e37c6f50	pep8	2018-03-29 17:27:39 +01:00
Neil Johnson	9ee44a372d	Remove need for sqlite specific query	2018-03-29 16:45:34 +01:00
Erik Johnston	88cc9cc69e	Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation ... Measure time it takes to calculate state group ID	2018-03-28 17:40:14 +01:00
Neil Johnson	dc7c020b33	fix pep8 errors	2018-03-28 17:25:15 +01:00
David Baker	c5de6987c2	This should probably be a PUT	2018-03-28 16:44:11 +01:00
Neil Johnson	241e4e8687	remove twisted deferral cruft	2018-03-28 16:25:53 +01:00
David Baker	929b34963d	OK, smallint it is then	2018-03-28 14:53:55 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00
David Baker	a838444a70	Grr. Copy the definition from is_admin	2018-03-28 14:50:30 +01:00
Neil Johnson	4262aba17b	bump schema version	2018-03-28 14:40:03 +01:00
Neil Johnson	86932be2cb	Support multi client R30 for psql	2018-03-28 14:36:53 +01:00
David Baker	32260baa41	pep8	2018-03-28 14:29:42 +01:00
Michael Kaye	33f6195d9a	Handle review comments	2018-03-28 14:25:25 +01:00
David Baker	a164270833	Make column definition that works on both dbs	2018-03-28 14:23:00 +01:00
David Baker	352e1ff9ed	Add schema delta file	2018-03-28 14:07:57 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Krombel	6152e253d8	Merge branch 'develop' of into allow_auto_join_rooms	2018-03-28 14:45:28 +02:00
Neil Johnson	792d340572	rename stat to future proof	2018-03-28 12:25:02 +01:00
Michael Kaye	4ceaa7433a	As daemonizing will make a new process, defer call to init.	2018-03-28 12:19:01 +01:00
Neil Johnson	788e69098c	Add user_ips last seen index	2018-03-28 12:03:13 +01:00
Neil Johnson	0f890f477e	No need to cast in count_daily_users	2018-03-28 11:49:57 +01:00
Neil Johnson	545001b9e4	Fix search_user_dir multiple sqlite versions do different things	2018-03-28 11:19:45 +01:00
Erik Johnston	01ccc9e6f2	Measure time it takes to calculate state group ID	2018-03-28 11:03:52 +01:00
Neil Johnson	a32d2548d9	query and call for r30 stats	2018-03-28 10:39:13 +01:00
Neil Johnson	9187e0762f	count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home	2018-03-28 10:02:32 +01:00
Erik Johnston	f879127aaa	Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id ... Linearize calls to _generate_user_id	2018-03-28 10:00:31 +01:00
Erik Johnston	e6d87c93f3	Merge pull request #3030 from matrix-org/erikj/no_ujson ... Remove last usage of ujson	2018-03-28 10:00:06 +01:00
Erik Johnston	004cc8a328	Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics ... Add counter metrics for calculating state delta	2018-03-28 09:59:42 +01:00
Michael Kaye	ef520d8d0e	Include coarse CPU and Memory use in stats callbacks. ... This requires the psutil module, and is still opt-in based on the report_stats config option.	2018-03-27 17:56:03 +01:00
Richard van der Hoff	a134c572a6	Stringify exceptions for keys/{query,claim} ... Make sure we stringify any exceptions we return from keys/query and keys/claim, to avoid a 'not JSON serializable' error later Fixes #3010	2018-03-27 17:15:06 +01:00
Richard van der Hoff	c2a5cf2fe3	factor out exception handling for keys/claim and keys/query ... this stuff is badly c&p'ed	2018-03-27 17:11:23 +01:00
Erik Johnston	800cfd5774	Comment	2018-03-27 13:30:39 +01:00
Erik Johnston	152c2ac19e	Fix indent	2018-03-27 13:13:46 +01:00
Erik Johnston	e70287cff3	Comment	2018-03-27 13:13:38 +01:00
Erik Johnston	03a26e28d9	Merge pull request #3017 from matrix-org/erikj/add_cache_control_headers ... Add Cache-Control headers to all JSON APIs	2018-03-27 13:10:38 +01:00
Erik Johnston	3e0c0660b3	Also do check inside linearizer	2018-03-27 13:01:34 +01:00
Erik Johnston	3f49e131d9	Add counter metrics for calculating state delta ... This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.	2018-03-27 10:57:35 +01:00
Erik Johnston	9b8c0fb162	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-26 21:41:55 +01:00
Erik Johnston	a9d7d98d3f	Bum version and changelog	2018-03-26 16:36:53 +01:00
Erik Johnston	bdbb1eec65	Merge branch 'erikj/simplejson_replication' of github.com:matrix-org/synapse into release-v0.27.0	2018-03-26 16:35:55 +01:00
Erik Johnston	9187862002	Bump version and changelog	2018-03-26 16:20:24 +01:00
Neil Johnson	51406dab96	version bump	2018-03-26 14:48:19 +01:00
Erik Johnston	fecb45e0c3	Remove last usage of ujson	2018-03-26 13:32:29 +01:00
Erik Johnston	44cd6e1358	PEP8	2018-03-26 12:06:48 +01:00
Erik Johnston	8d6dc106d1	Don't use _cursor_to_dict in find_next_generated_user_id_localpart	2018-03-26 12:02:44 +01:00
Erik Johnston	a052aa42e7	Linearize calls to _generate_user_id	2018-03-26 12:02:20 +01:00
Matthew Hodgson	8cbbfaefc1	404 correctly on missing paths via NoResource ... fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029	2018-03-23 10:32:50 +00:00
Erik Johnston	84b5cc69f5	Merge pull request #3006 from matrix-org/erikj/state_iter ... Use .iter* to avoid copies in StateHandler	2018-03-22 11:51:13 +00:00
Erik Johnston	fde8e8f09f	Fix s/iteriterms/itervalues	2018-03-22 11:42:16 +00:00
Erik Johnston	eb9fc021e3	Merge branch 'release-v0.27.0' of github.com:matrix-org/synapse into develop	2018-03-22 10:19:53 +00:00
Erik Johnston	1c41b05c8c	Add Cache-Control headers to all JSON APIs ... It is especially important that sync requests don't get cached, as if a sync returns the same token given then the client will call sync with the same parameters again. If the previous response was cached it will get reused, resulting in the client tight looping making the same request and never making any progress. In general, clients will expect to get up to date data when requesting APIs, and so its safer to do a blanket no cache policy than only whitelisting APIs that we know will break things if they get cached.	2018-03-21 17:46:26 +00:00
Neil Johnson	e66fbcbb02	fix merge conflicts	2018-03-20 14:25:31 +00:00
Erik Johnston	9aa5a0af51	Explicitly use simplejson	2018-03-20 09:58:13 +00:00
Erik Johnston	610accbb7f	Fix replication after switch to simplejson ... Turns out that simplejson serialises namedtuple's as dictionaries rather than tuples by default.	2018-03-19 16:12:48 +00:00
Neil Johnson	c384705ee8	Update __init__.py ... bump version	2018-03-19 15:11:58 +00:00
Erik Johnston	3f961e638a	Merge pull request #3005 from matrix-org/erikj/fix_cache_size ... Fix bug where state cache used lots of memory	2018-03-19 11:44:26 +00:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Erik Johnston	9a0d783c11	Add comments	2018-03-19 11:35:53 +00:00
Matthew Hodgson	38f952b9bc	spell out not to massively increase bcrypt rounds	2018-03-19 09:27:36 +00:00
Matthew Hodgson	b2f2282947	make lazy_load_members configurable in filters	2018-03-19 01:15:13 +00:00
Matthew Hodgson	478af0f720	reshuffle todo & comments	2018-03-19 01:00:12 +00:00
Matthew Hodgson	366f730bf6	only get member state IDs for incremental syncs if we're filtering	2018-03-18 21:40:43 +00:00
Matthew Hodgson	0b56290f0b	remove stale import	2018-03-16 01:45:49 +00:00
Matthew Hodgson	fc5397fdf5	remove debug	2018-03-16 01:44:55 +00:00
Matthew Hodgson	4f0493c850	fix tsm search again	2018-03-16 01:43:37 +00:00
Matthew Hodgson	f7dcc404f2	add state_ids for timeline entries	2018-03-16 01:37:53 +00:00
Matthew Hodgson	5b3b3aada8	simplify timeline_start_members	2018-03-16 01:17:34 +00:00
Erik Johnston	bf49d2dca8	Replace some ujson with simplejson to make it work	2018-03-16 00:55:44 +00:00
Matthew Hodgson	3bc5bd2d22	make incr syncs work	2018-03-16 00:52:04 +00:00
Erik Johnston	a8ce159be4	Replace some ujson with simplejson to make it work	2018-03-16 00:27:09 +00:00
Erik Johnston	5b631ff41a	Remove wrong comment	2018-03-16 00:07:08 +00:00
Erik Johnston	ba48755d56	Bump version and changelog	2018-03-15 23:57:26 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
Richard van der Hoff	5a6e54264d	Make 'unexpected logging context' into warnings ... I think we've now fixed enough of these that the rest can be logged at warning.	2018-03-15 18:40:38 +00:00
Erik Johnston	9cf519769b	Use .iter* to avoid copies in StateHandler	2018-03-15 17:50:26 +00:00
Erik Johnston	7c7706f42b	Fix bug where state cache used lots of memory ... The state cache bases its size on the sum of the size of entries. The size of the entry is calculated once on insertion, so it is important that the size of entries does not change. The DictionaryCache modified the entries size, which caused the state cache to incorrectly think it was smaller than it actually was.	2018-03-15 15:46:54 +00:00
NotAFile	2cc9f76bc3	replace old style error catching with 'as' keyword ... This is both easier to read and compatible with python3 (not that that matters) Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-15 16:11:17 +01:00
Erik Johnston	ddb00efc1d	Bump version number	2018-03-15 14:41:30 +00:00
Krombel	91ea0202e6	move handling of auto_join_rooms to RegisterHandler ... Currently the handling of auto_join_rooms only works when a user registers itself via public register api. Registrations via registration_shared_secret and ModuleApi do not work This auto_joins the users in the registration handler which enables the auto join feature for all 3 registration paths. This is related to issue #2725 Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-14 16:45:37 +01:00
Matthew Hodgson	056a6df546	Merge branch 'develop' into matthew/filter_members	2018-03-14 15:38:05 +00:00
Erik Johnston	4f28018c83	Register membership/state servlets in event_creator	2018-03-14 14:30:06 +00:00
Erik Johnston	57db62e554	Merge pull request #2992 from matrix-org/erikj/implement_member_workre ... Implement RoomMemberWorkerHandler	2018-03-14 14:29:33 +00:00
Erik Johnston	0011ede3b0	Fix imports	2018-03-14 14:19:23 +00:00
Erik Johnston	62ad701326	s/join/joined/ in notify_user_membership_change	2018-03-14 14:17:43 +00:00
Erik Johnston	3f0f06cb31	Split RoomMemberWorkerHandler to separate file	2018-03-14 11:41:45 +00:00
Erik Johnston	3e839e0548	Merge pull request #2989 from matrix-org/erikj/profile_cache_master ... Only update remote profile cache on master	2018-03-14 09:42:27 +00:00
Erik Johnston	ebd0127999	Merge pull request #2988 from matrix-org/erikj/split_profile_store ... Split up ProfileStore	2018-03-14 09:41:06 +00:00
Erik Johnston	cfe75a9fb6	Merge pull request #2991 from matrix-org/erikj/fixup_rm ... _remote_join and co take a requester	2018-03-14 09:40:57 +00:00
Erik Johnston	f51565e023	Merge pull request #2993 from matrix-org/erikj/is_blocked ... Add is_blocked to worker store	2018-03-14 09:39:18 +00:00
Matthew Hodgson	9f77001e27	pep8	2018-03-14 00:07:47 +00:00
Matthew Hodgson	4d0cfef6ee	add copyright to nudge CI	2018-03-14 00:02:20 +00:00
Matthew Hodgson	c9d72e4571	oops	2018-03-13 23:46:45 +00:00
Matthew Hodgson	d144ed6ffb	fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990) ... Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.	2018-03-13 22:36:04 +00:00
Matthew Hodgson	ccca02846d	make it work	2018-03-13 22:31:41 +00:00
Matthew Hodgson	f0f9a0605b	remove comment now #2969 is fixed	2018-03-13 22:12:15 +00:00
Matthew Hodgson	12350e3f9a	merge proper fix to bug 2969	2018-03-13 22:11:58 +00:00
Matthew Hodgson	14a9d2f73d	ensure we always include the members for a given timeline block	2018-03-13 22:03:42 +00:00
Matthew Hodgson	afbf4d3dcc	typoe	2018-03-13 19:48:04 +00:00
Matthew Hodgson	865377a70d	disable optimisation for searching for state groups ... when type filter includes wildcards on state_key	2018-03-13 19:46:04 +00:00
Erik Johnston	a08726fc42	Add is_blocked to worker store	2018-03-13 18:28:44 +00:00
Erik Johnston	b27320b550	Implement RoomMemberWorkerHandler	2018-03-13 18:26:00 +00:00
Matthew Hodgson	b2aba9e430	build where_clause sanely	2018-03-13 18:13:44 +00:00
Matthew Hodgson	52f7e23c72	PR feedbackz	2018-03-13 18:07:55 +00:00
Matthew Hodgson	1b1c137771	fix bug #2926	2018-03-13 17:52:52 +00:00
Erik Johnston	350331d466	_remote_join and co take a requester	2018-03-13 17:50:39 +00:00
Erik Johnston	1a69c6d590	Merge pull request #2987 from matrix-org/erikj/split_room_member_handler ... Split RoomMemberHandler into base and master class	2018-03-13 17:40:00 +00:00
Erik Johnston	df8ff682a7	Only update remote profile cache on master	2018-03-13 17:38:21 +00:00
Erik Johnston	3518d0ea8f	Split up ProfileStore	2018-03-13 17:36:50 +00:00
Erik Johnston	d45a114824	Raise, don't return, exception	2018-03-13 17:24:34 +00:00
Erik Johnston	6dbebef141	Add missing param to docstrings	2018-03-13 17:15:32 +00:00
Erik Johnston	16adb11cc0	Correct import order	2018-03-13 16:57:07 +00:00
Erik Johnston	82f16faa78	Move user_*_room distributor stuff to master class ... I added yields when calling user_left_room, but they shouldn't matter on the master process as they always return None anyway.	2018-03-13 16:38:15 +00:00
Erik Johnston	b78717b87b	Split RoomMemberHandler into base and master class ... The intention here is to split the class into the bits that can be done on workers and the bits that have to be done on the master. In future there will also be a class that can be run on the worker, which will delegate work to the master when necessary.	2018-03-13 16:37:41 +00:00
Erik Johnston	95cb401ae0	Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer ... Remove ReplicationLayer and user Client/Server directly	2018-03-13 15:45:08 +00:00
Erik Johnston	5d8476d8ff	Merge pull request #2981 from matrix-org/erikj/factor_remote_leave ... Factor out _remote_reject_invite in RoomMember	2018-03-13 15:44:56 +00:00
Jonas Platte	47ce527f45	Add room_id to the response of rooms/{roomId}/join ... Fixes #2349	2018-03-13 14:48:12 +01:00
Erik Johnston	56e709857c	Merge pull request #2979 from matrix-org/erikj/no_handlers ... Don't build handlers on workers unnecessarily	2018-03-13 13:46:38 +00:00
Erik Johnston	cb9f8e527c	s/replication_client/federation_client/	2018-03-13 13:26:52 +00:00
Erik Johnston	cea462e285	s/replication_server/federation_server	2018-03-13 13:22:21 +00:00
Erik Johnston	bf8e97bd3c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave	2018-03-13 13:17:08 +00:00
Erik Johnston	ea3442c15c	Add docstring	2018-03-13 13:16:21 +00:00
Erik Johnston	16469a4f15	Merge pull request #2980 from matrix-org/erikj/rm_priv ... Make RoomMemberHandler functions private that can be	2018-03-13 13:11:11 +00:00
Erik Johnston	c82111a55f	Merge pull request #2982 from matrix-org/erikj/fix_extra_users ... extra_users is actually a list of UserIDs	2018-03-13 13:11:04 +00:00
Erik Johnston	da87791975	Merge pull request #2983 from matrix-org/erikj/rename_register_3pid ... Refactor get_or_register_3pid_guest	2018-03-13 13:10:52 +00:00
Erik Johnston	99e9b4f26c	Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix ... RoomMembershipRestServlet doesn't handle /forget	2018-03-13 13:10:44 +00:00
Erik Johnston	f5160d4a3e	RoomMembershipRestServlet doesn't handle /forget ... Due to the order we register the REST handlers `/forget` was handled by the correct handler.	2018-03-13 12:12:55 +00:00
Erik Johnston	8b3573a8b2	Refactor get_or_register_3pid_guest	2018-03-13 12:08:58 +00:00
Richard van der Hoff	299fd740c7	Merge pull request #2975 from matrix-org/rav/measure_persist_events ... Add Measure block for persist_events	2018-03-13 12:06:25 +00:00
Erik Johnston	9a2d9b4789	Merge pull request #2977 from matrix-org/erikj/replication_move_props ... Move property setting from ReplicationLayer to base classes	2018-03-13 11:45:25 +00:00
Erik Johnston	f43b6d6d9b	Fix docstring types	2018-03-13 11:29:35 +00:00
Erik Johnston	0f942f68c1	Factor out _remote_reject_invite in RoomMember	2018-03-13 11:22:45 +00:00
Erik Johnston	d0fcc48f9d	extra_users is actually a list of UserIDs	2018-03-13 11:20:06 +00:00
Erik Johnston	31becf4ac3	Make functions private that can be	2018-03-13 11:15:16 +00:00
Erik Johnston	d023ecb810	Don't build handlers on workers unnecessarily	2018-03-13 11:08:10 +00:00
Erik Johnston	ea7b3c4b1b	Remove unused ReplicationLayer	2018-03-13 11:00:04 +00:00
Erik Johnston	265b993b8a	Split replication layer into two	2018-03-13 10:55:47 +00:00
Erik Johnston	e05bf34117	Move property setting from ReplicationLayer to FederationBase	2018-03-13 10:51:30 +00:00
Erik Johnston	c3f79c9da5	Split out edu/query registration to a separate class	2018-03-13 10:24:27 +00:00
Richard van der Hoff	889a2a853a	Add Measure block for persist_events ... This seems like a useful thing to measure.	2018-03-13 10:01:42 +00:00
Richard van der Hoff	e48c7aac4d	Add transactional API to history purge ... Make the purge request return quickly, and allow scripts to poll for updates.	2018-03-12 16:22:55 +00:00
Richard van der Hoff	1708412f56	Return an error when doing two purges on a room ... Queuing up purges doesn't sound like a good thing.	2018-03-12 16:22:54 +00:00
Richard van der Hoff	b984dd0b73	Merge pull request #2961 from matrix-org/rav/run_in_background ... Factor run_in_background out from preserve_fn	2018-03-12 16:19:13 +00:00
Richard van der Hoff	ba1d08bc4b	Merge pull request #2965 from matrix-org/rav/request_logging ... Add a metric which increments when a request is received	2018-03-12 09:45:33 +00:00
Matthew Hodgson	fdedcd1f4d	correctly handle None state_keys ... and fix include_other_types thinko	2018-03-12 01:39:06 +00:00
Matthew Hodgson	97c0496cfa	fix sqlite where clause	2018-03-12 00:27:06 +00:00
Matthew Hodgson	8713365265	typos	2018-03-11 20:10:25 +00:00
Matthew Hodgson	9b334b3f97	WIP experiment in lazyloading room members	2018-03-11 20:01:41 +00:00
Richard van der Hoff	58dd148c4f	Add some docstrings to help figure this out	2018-03-09 18:05:41 +00:00
Richard van der Hoff	88541f9009	Add a metric which increments when a request is received ... It's useful to know when there are peaks in incoming requests - which isn't quite the same as there being peaks in outgoing responses, due to the time taken to handle requests.	2018-03-09 16:30:26 +00:00
Richard van der Hoff	dbe80a286b	refactor JsonResource ... rephrase the OPTIONS and unrecognised request handling so that they look similar to the common flow.	2018-03-09 16:22:16 +00:00
Richard van der Hoff	20f40348d4	Factor run_in_background out from preserve_fn ... It annoys me that we create temporary function objects when there's really no need for it. Let's factor the gubbins out of preserve_fn and start using it.	2018-03-08 11:50:11 +00:00
Erik Johnston	a56d54dcb7	Fix up log message	2018-03-07 11:55:31 +00:00
Erik Johnston	02a1296ad6	Fix typo	2018-03-07 11:55:31 +00:00
Erik Johnston	8cb44da4aa	Fix race in sync when joining room ... The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.	2018-03-07 11:55:31 +00:00
Richard van der Hoff	8ffaacbee3	Merge pull request #2949 from krombel/use_bcrypt_checkpw ... use bcrypt.checkpw	2018-03-06 11:56:06 +00:00
Richard van der Hoff	b2932107bb	Merge pull request #2946 from matrix-org/rav/timestamp_to_purge ... Implement purge_history by timestamp	2018-03-06 11:20:23 +00:00
Erik Johnston	7aed50a038	Merge pull request #2948 from matrix-org/erikj/kill_as_sync ... Remove ability for AS users to call /events and /sync	2018-03-06 11:10:09 +00:00
Erik Johnston	b6c4b851f1	Merge pull request #2947 from matrix-org/erikj/split_directory_store ... Split Directory store	2018-03-05 18:17:32 +00:00
Krombel	ed9b5eced4	use bcrypt.checkpw ... in bcrypt 3.1.0 checkpw got introduced (already 2 years ago) This makes use of that with enhancements which might get introduced by that Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-05 18:02:59 +01:00
Erik Johnston	d4ffe61d4f	Remove ability for AS users to call /events and /sync ... This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144	2018-03-05 15:44:46 +00:00
Erik Johnston	69ce365b79	Fix cache invalidation on deletion	2018-03-05 15:29:03 +00:00
Erik Johnston	2e223163ff	Split Directory store	2018-03-05 15:11:30 +00:00
Richard van der Hoff	f8bfcd7e0d	Provide a means to pass a timestamp to purge_history	2018-03-05 14:37:23 +00:00
Richard van der Hoff	d032785aa7	Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts ... Test and fix find_first_stream_ordering_after_ts	2018-03-05 12:26:14 +00:00
Richard van der Hoff	c818fcab11	Test and fix find_first_stream_ordering_after_ts ... It seemed to suffer from a bunch of off-by-one errors.	2018-03-05 12:04:02 +00:00
Richard van der Hoff	06a14876e5	Add find_first_stream_ordering_after_ts ... Expose this as a public function which can be called outside a txn	2018-03-05 11:53:39 +00:00
Erik Johnston	42174946f8	Merge pull request #2934 from matrix-org/erikj/cache_fix ... Fix bug with delayed cache invalidation stream	2018-03-05 11:33:17 +00:00
Erik Johnston	f394f5574d	Merge pull request #2929 from matrix-org/erikj/split_regististration_store ... Split registration store	2018-03-05 11:33:07 +00:00
dklug	af7ed8e1ef	Return 401 for invalid access_token on logout ... Signed-off-by: Duncan Klug <dklug@ucmerced.edu>	2018-03-02 22:01:27 -08:00
Erik Johnston	efb79820b4	Fix bug with delayed cache invalidation stream ... We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.	2018-03-02 14:45:15 +00:00
Erik Johnston	fafa3e7114	Split registration store	2018-03-02 13:48:27 +00:00
Erik Johnston	d960d23830	Add missing yield during 3pid signature checks	2018-03-02 11:03:18 +00:00
Erik Johnston	1a6c7cdf54	Merge pull request #2928 from matrix-org/erikj/read_marker_caches ... Fix typo in getting replication account data processing	2018-03-01 17:56:14 +00:00
Erik Johnston	89b7232ff8	Fix typo in getting replication account data processing	2018-03-01 17:50:30 +00:00
Erik Johnston	1773df0632	Merge pull request #2925 from matrix-org/erikj/split_sig_fed ... Split out SignatureStore and EventFederationStore	2018-03-01 17:32:58 +00:00
Erik Johnston	65cf454fd1	Remove unused DataStore	2018-03-01 17:27:53 +00:00
Erik Johnston	9e08a93a7b	Merge pull request #2927 from matrix-org/erikj/read_marker_caches ... Improve caching for read_marker API	2018-03-01 17:12:34 +00:00
Erik Johnston	4b44f05f19	Fewer lies are better	2018-03-01 17:08:17 +00:00
Erik Johnston	a83c514d1f	Improve caching for read_marker API ... We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.	2018-03-01 17:08:17 +00:00
Erik Johnston	33bebb63f3	Add some caches to help read marker API	2018-03-01 17:08:17 +00:00
Erik Johnston	483e8104db	Merge pull request #2926 from matrix-org/erikj/member_handler_move ... Move RoomMemberHandler out of Handlers	2018-03-01 17:01:25 +00:00
Erik Johnston	2ad4d5b5bb	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed	2018-03-01 16:59:39 +00:00
Erik Johnston	529c026ac1	Move back to hs.is_mine	2018-03-01 16:49:12 +00:00
Erik Johnston	7c371834cc	Stub out broken function only used for cache	2018-03-01 16:44:13 +00:00
Erik Johnston	64346be26d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store	2018-03-01 16:26:42 +00:00
Erik Johnston	22518e2833	Merge pull request #2923 from matrix-org/erikj/stream_ago_worker ... Calculate stream_ordering_month_ago correctly on workers	2018-03-01 16:23:54 +00:00
Erik Johnston	884b26ae41	Remove unused variables	2018-03-01 16:23:48 +00:00
Erik Johnston	1b2af11650	Document abstract class and method better	2018-03-01 16:20:57 +00:00
Erik Johnston	872ff95ed4	Default stream_ordering_*_ago to None	2018-03-01 16:00:05 +00:00
Erik Johnston	22004b524e	Fix comment typo	2018-03-01 15:59:40 +00:00
Erik Johnston	4bc4236faf	Merge pull request #2922 from matrix-org/erikj/split_room_store ... Split up RoomStore	2018-03-01 15:55:01 +00:00
Richard van der Hoff	2324124a72	Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable ... Rewrite make_deferred_yieldable avoiding inlineCallbacks	2018-03-01 15:39:10 +00:00
Erik Johnston	f793bc3877	Split out stream store	2018-03-01 15:13:08 +00:00
Erik Johnston	784f036306	Move RoomMemberHandler out of Handlers	2018-03-01 14:36:50 +00:00
Erik Johnston	6411f725be	Calculate stream_ordering_month_ago correctly on workers	2018-03-01 14:20:53 +00:00
Erik Johnston	a9a2d66cdd	Split out SignatureStore and EventFederationStore	2018-03-01 14:17:53 +00:00
Erik Johnston	0c8ba5dd1c	Split up RoomStore	2018-03-01 14:01:19 +00:00
Richard van der Hoff	3a75de923b	Rewrite make_deferred_yieldable avoiding inlineCallbacks ... ... because (a) it's actually simpler (b) it might be marginally more performant?	2018-03-01 12:40:05 +00:00
Erik Johnston	126b9bf96f	Log in the correct places	2018-03-01 12:05:33 +00:00
Erik Johnston	157298f986	Don't do preserve_fn for every request	2018-03-01 11:59:45 +00:00
Erik Johnston	89f90d808a	Add some logging	2018-03-01 11:59:16 +00:00
Erik Johnston	8ded8ba2c7	Make repl send_event idempotent and retry on timeouts ... If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event.	2018-03-01 11:20:34 +00:00
Erik Johnston	f381d63813	Check event auth on the worker	2018-03-01 10:18:37 +00:00
Erik Johnston	6b8604239f	Correctly send ratelimit and extra_users params	2018-03-01 10:08:39 +00:00
Erik Johnston	f756f961ea	Fixup comments	2018-03-01 10:05:27 +00:00
Erik Johnston	28e973ac11	Calculate push actions on worker	2018-02-28 18:02:30 +00:00
Erik Johnston	493e25d554	Move storage functions for push calculations ... This will allow push actions for an event to be calculated on workers.	2018-02-27 13:58:16 +00:00
Erik Johnston	3594dbc6dc	Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation ... Fix missing invalidations for receipt storage	2018-02-27 11:34:26 +00:00
Erik Johnston	2311189ee4	Merge pull request #2903 from matrix-org/erikj/split_roommember_store ... Split out RoomMemberStore	2018-02-27 11:32:10 +00:00
Erik Johnston	c57607874c	Merge pull request #2901 from matrix-org/erikj/split_as_stores ... Split AS stores	2018-02-27 10:07:07 +00:00
Erik Johnston	8956f0147a	Add comment	2018-02-27 10:06:51 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	45b5fe9122	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push	2018-02-26 13:49:24 +00:00
Erik Johnston	d62ce972f8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store	2018-02-23 11:46:24 +00:00
Erik Johnston	6ae9a3d2a6	Update copyright	2018-02-23 11:44:49 +00:00
Erik Johnston	a90c60912f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions	2018-02-23 11:26:31 +00:00
Erik Johnston	50e8657867	Merge pull request #2902 from matrix-org/erikj/split_events_store ... Split out get_events and co into a worker store	2018-02-23 11:23:52 +00:00
Erik Johnston	1cf9e071dd	Merge pull request #2899 from matrix-org/erikj/split_pushers ... Split PusherStore	2018-02-23 11:23:35 +00:00
Erik Johnston	d0957753bf	Merge pull request #2898 from matrix-org/erikj/split_push_rules_store ... Split PushRulesStore	2018-02-23 11:23:23 +00:00
Erik Johnston	199dba6c15	Merge pull request #2897 from matrix-org/erikj/split_account_data ... Split AccountDataStore and TagStore	2018-02-23 11:23:11 +00:00
Erik Johnston	70349872c2	Update copyright	2018-02-23 11:14:35 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00
Erik Johnston	bf8a36e080	Update copyright	2018-02-23 10:52:10 +00:00
Erik Johnston	5d0f665848	Remove redundant clock	2018-02-23 10:49:58 +00:00
Erik Johnston	3bd760628b	_event_persist_queue shouldn't be in worker store	2018-02-23 10:49:18 +00:00
Erik Johnston	eb9b5eec81	Update copyright	2018-02-23 10:42:39 +00:00
Erik Johnston	c2ecfcc3a4	Update copyright	2018-02-23 10:41:34 +00:00
Erik Johnston	7e6cf89dc2	Update copyright	2018-02-23 10:39:19 +00:00
Erik Johnston	26d37f7a63	Update copyright	2018-02-23 10:33:55 +00:00
Erik Johnston	bb73f55fc6	Use absolute imports	2018-02-23 10:31:16 +00:00
Erik Johnston	faeb369f15	Fix missing invalidations for receipt storage	2018-02-21 15:19:54 +00:00
Erik Johnston	3dec9c66b3	Split out RoomMemberStore	2018-02-21 12:07:26 +00:00
Erik Johnston	46244b2759	Split AS stores	2018-02-21 11:49:34 +00:00
Erik Johnston	27b094f382	Split out get_events and co into a worker store	2018-02-21 11:41:48 +00:00
Erik Johnston	573712da6b	Update comments	2018-02-21 11:29:49 +00:00
Erik Johnston	c96d547f4d	Actually use new param	2018-02-21 11:03:42 +00:00
Erik Johnston	d15d237b0d	Split out EventPushActionWorkerStore	2018-02-21 11:01:13 +00:00
Erik Johnston	27939cbb0e	Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed ... Delete from push_actions_staging in federation too	2018-02-21 11:00:06 +00:00
Erik Johnston	6f72765371	Split PusherStore	2018-02-21 10:54:21 +00:00
Erik Johnston	cbaad969f9	Split PushRulesStore	2018-02-21 10:43:31 +00:00
Erik Johnston	ca9b9d9703	Split AccountDataStore and TagStore	2018-02-21 10:15:04 +00:00
Erik Johnston	8fbb4d0d19	Raise exception in abstract method	2018-02-20 17:59:23 +00:00
Erik Johnston	95e4cffd85	Fix comment	2018-02-20 17:58:40 +00:00
Erik Johnston	e316bbb4c0	Use abstract base class to access stream IDs	2018-02-20 17:43:57 +00:00
Erik Johnston	f5ac4dc2d4	Split ReceiptsStore	2018-02-20 16:28:28 +00:00
Erik Johnston	24087bffa9	Ensure all push actions are deleted from staging	2018-02-20 12:34:31 +00:00
Erik Johnston	ad0ccf15ea	Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts	2018-02-20 12:34:28 +00:00
Erik Johnston	d874d4f2d7	Delete from push_actions_staging in federation too	2018-02-20 11:37:52 +00:00
Erik Johnston	6ff8c87484	Batch inserts into event_push_actions_staging	2018-02-20 11:33:07 +00:00
Erik Johnston	324c3e9399	Merge pull request #2868 from matrix-org/erikj/refactor_media_storage ... Make store_file use store_into_file	2018-02-20 11:31:24 +00:00
Pascal Bach	3acd616979	Update pynacl dependency to 1.2.1 or higher ... Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>	2018-02-19 10:45:22 +01:00
Richard van der Hoff	d1a3325f99	(Really) fix tablescan of event_push_actions on purge ... commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/	2018-02-16 14:02:31 +00:00
Erik Johnston	6af025d3c4	Fix typo of double is_highlight	2018-02-16 11:35:31 +00:00
Erik Johnston	012e8e142a	Comments	2018-02-16 11:35:01 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	c714c61853	Update event_push_actions table from staging table	2018-02-15 15:47:06 +00:00
Erik Johnston	acac21248c	Store push actions in staging area	2018-02-15 15:47:04 +00:00
Erik Johnston	106906a65e	Don't serialize current state over replication	2018-02-15 13:53:18 +00:00
Erik Johnston	5fb347fc41	Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log ... Don't log errors propogated from send_event	2018-02-15 12:31:49 +00:00
Erik Johnston	fd1601c596	Fix state group storage bug in workers ... We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.	2018-02-15 11:04:32 +00:00
Erik Johnston	ef344b10e5	Don't log errors propogated from send_event	2018-02-15 11:03:49 +00:00
Richard van der Hoff	b8d821aa68	Merge pull request #2867 from matrix-org/rav/rework_purge ... purge_history cleanups	2018-02-15 09:49:07 +00:00
Erik Johnston	92c52df702	Make store_file use store_into_file	2018-02-14 17:55:18 +00:00
Richard van der Hoff	d28ec43e15	Merge pull request #2769 from matrix-org/matthew/hit_the_gin ... switch back from GIST to GIN indexes	2018-02-14 16:59:03 +00:00
Richard van der Hoff	39bf47319f	purge_history: fix sqlite syntax error ... apparently sqlite insists on indexes being named	2018-02-14 16:42:19 +00:00
Richard van der Hoff	ac27f6a35e	purge_history: handle sqlite asshattery ... apparently creating a temporary table commits the transaction. because that's a useful thing.	2018-02-14 16:41:12 +00:00
Richard van der Hoff	5978dccff0	remove overzealous exception handling	2018-02-14 15:54:09 +00:00
Richard van der Hoff	278d21b5e4	purge_history: fix index use ... event_push_actions doesn't have an index on event_id, so we need to specify room_id.	2018-02-14 15:44:51 +00:00
Richard van der Hoff	5fcbf1e07c	Rework event purge to use a temporary table ... ... which should speed things up by reducing the amount of data being shuffled across the connection	2018-02-14 11:02:22 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	d627174da2	Fix log message in purge_history ... (we don't just remove remote events)	2018-02-13 16:51:21 +00:00
Richard van der Hoff	ddb6a79b68	Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin	2018-02-13 16:45:36 +00:00
Richard van der Hoff	0b27ae8dc3	move search reindex to schema 47 ... We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.	2018-02-13 16:44:46 +00:00
Richard van der Hoff	4a6d551704	GIN reindex: Fix syntax errors, improve exception handling	2018-02-13 16:44:46 +00:00
Richard van der Hoff	bfdf7b9237	Merge pull request #2864 from matrix-org/rav/persist_event_caching ... Use StateResolutionHandler to resolve state in persist_events	2018-02-13 14:45:57 +00:00
Richard van der Hoff	630caf8a70	style nit	2018-02-13 14:29:22 +00:00
Richard van der Hoff	8fd1a32456	Fix typos in purge api & doc ... * It's supposed to be purge_local_events, not ..._history * Fix the doc to have valid json	2018-02-13 13:09:39 +00:00
Richard van der Hoff	4d09366656	Merge pull request #2695 from okurz/feature/allow_recent_pysaml ... Allow use of higher versions of saml2	2018-02-13 12:24:08 +00:00
Richard van der Hoff	a9b712e9dc	Merge branch 'develop' into matthew/gin_work_mem	2018-02-13 12:16:01 +00:00
Erik Johnston	1026690cd2	Merge pull request #2857 from matrix-org/erikj/upload_store ... Tell storage providers about new file so they can upload	2018-02-12 13:52:58 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	39a6b35496	purge: move room_depth update to end ... ... to avoid locking the table for too long	2018-02-09 13:07:41 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	e571aef06d	purge: Move cache invalidation to more appropriate place ... it was a bit of a non-sequitur there	2018-02-09 13:07:41 +00:00
Richard van der Hoff	61ffaa8137	bump purge logging to info ... this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	5fa571a91b	Tell storage providers about new file so they can upload	2018-02-07 13:35:08 +00:00
Erik Johnston	50fe92cd26	Move presence handling into handle_new_client_event ... As we want to have it run on the main synapse instance	2018-02-07 10:34:09 +00:00
Erik Johnston	8ec2e638be	Add event_creator worker	2018-02-07 10:32:32 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	e3624fad5f	Remove pointless ratelimit check ... The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.	2018-02-07 10:30:25 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	3e1e69ccaf	Update copyright	2018-02-06 16:40:38 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Richard van der Hoff	b31bf0bb51	Merge pull request #2849 from matrix-org/rav/clean_up_state_delta ... Remove redundant return value from _calculate_state_delta	2018-02-05 17:42:20 +01:00
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Erik Johnston	3fa362502c	Update places where we create events	2018-02-05 16:01:48 +00:00
Erik Johnston	5ff3d23564	Split event creation into a separate handler	2018-02-05 16:01:48 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	db91e72ade	Merge pull request #2844 from matrix-org/rav/evicted_metrics ... montoring metrics for number of cache evictions	2018-02-05 16:34:36 +01:00
Richard van der Hoff	bc496df192	report metrics on number of cache evictions	2018-02-05 15:34:01 +00:00
Erik Johnston	1f881e0746	Merge pull request #2791 from matrix-org/erikj/media_storage_refactor ... Ensure media is in local cache before thumbnailing	2018-02-05 11:28:52 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	9c9356512e	Merge pull request #2845 from matrix-org/rav/urlcache_error_handling ... Handle url_previews with no content-type	2018-02-02 15:27:52 +01:00
Richard van der Hoff	18eae413af	Merge pull request #2842 from matrix-org/rav/state_resolution_handler ... Factor out resolve_state_groups to a separate handler	2018-02-02 15:27:35 +01:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Richard van der Hoff	9dcd667ac2	Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring ... Docstring fixes	2018-02-02 15:26:49 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	d5352cbba8	Handle url_previews with no content-type ... avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.	2018-02-02 00:53:46 +00:00
Richard van der Hoff	14737ba495	doc arg types for _seperate	2018-02-01 12:41:34 +00:00