11078 Commits

Author	SHA1	Message	Date
Andrew Morgan	6d02a13d81	Typo in info log ... Co-Authored-By: babolivier <contact@brendanabolivier.com>	2019-02-18 11:36:34 +00:00
Juuso "Linda" Lapinlampi	68d2869c8d	config: Remove a repeated word from a logger warning ... The warning for missing macaroon_secret_key was "missing missing".	2019-02-15 22:24:53 -07:00
Erik Johnston	bd4505f765	Merge pull request #4647 from matrix-org/erikj/add_room_publishing_rules ... Add configurable room list publishing rules	2019-02-15 22:11:01 +00:00
Brendan Abolivier	f86b695cbd	Various cosmetics to make TravisCI happy	2019-02-15 12:29:34 +00:00
Brendan Abolivier	af8a2f679b	Remove unused import	2019-02-15 12:27:43 +00:00
Brendan Abolivier	1895d14e12	Support .well-known delegation when issuing certificates through ACME	2019-02-15 12:05:08 +00:00
Erik Johnston	b99c532c1c	Move defaults up into code	2019-02-15 10:53:39 +00:00
Erik Johnston	02c729d6b0	Hoist up checks to reduce overall work	2019-02-15 10:20:02 +00:00
Erik Johnston	02c46acc6a	Fixup comments	2019-02-15 10:17:13 +00:00
Matthew Hodgson	bfcefbb230	tabs	2019-02-14 19:53:12 +00:00
Matthew Hodgson	6f47bc3fb2	set CORS on .well-known URI to unbreak modular ... otherwise a riot/web running on foo.riot.im can't query the .well-known on foo.modular.im...	2019-02-14 19:51:41 +00:00
Erik Johnston	8e32f26cb8	Clarify comments	2019-02-14 18:21:24 +00:00
Erik Johnston	cb12a37708	Clarify and fix behaviour when there are multiple aliases	2019-02-14 18:16:32 +00:00
Erik Johnston	f61b2068e6	Only fetch aliases when publishing rooms	2019-02-14 18:08:09 +00:00
Erik Johnston	f666fe36d7	Fixup comments	2019-02-14 18:07:24 +00:00
Richard van der Hoff	bf4fd14806	Merge branch 'master' into develop	2019-02-14 17:34:09 +00:00
Richard van der Hoff	f595d6ac57	0.99.1.1	2019-02-14 17:20:02 +00:00
Richard van der Hoff	f311018823	Fix errors in acme provisioning (#4648) ... * Better logging for errors on startup * Fix "TypeError: '>' not supported" when starting without an existing certificate * Fix a bug where an existing certificate would be reprovisoned every day	2019-02-14 17:10:36 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules ... This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Richard van der Hoff	b02465b9db	Merge branch 'master' into develop	2019-02-14 14:42:03 +00:00
Richard van der Hoff	06cd757ae7	0.99.1	2019-02-14 14:24:24 +00:00
Erik Johnston	7fc1196a36	Correctly handle RequestSendFailed exceptions ... This mainly reduces the number of exceptions we log.	2019-02-14 14:01:04 +00:00
Erik Johnston	6cb415b63f	Fixup comments and add warning	2019-02-13 16:15:11 +00:00
Richard van der Hoff	c6e75c9f2d	Merge pull request #4450 from 14mRh4X0r/fix-dependency-message ... Fix error message for optional dependencies	2019-02-13 16:12:49 +00:00
Richard van der Hoff	3bc238629e	0.99.1rc2	2019-02-13 14:46:18 +00:00
Richard van der Hoff	c1dfd6a18a	Merge remote-tracking branch 'origin/release-v0.99.1' into develop	2019-02-13 14:27:45 +00:00
Erik Johnston	309f3bb322	Update synapse/app/_base.py ... Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>	2019-02-13 13:24:27 +00:00
Amber Brown	bb4fd8f927	Run black on user directory code (#4635)	2019-02-13 23:05:32 +11:00
Richard van der Hoff	767686af48	Use listen_tcp for the replication listener ... Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with what we do elsewhere.	2019-02-13 11:59:04 +00:00
Richard van der Hoff	2a5a15aff8	Improve logging around listening services ... I wanted to bring listen_tcp into line with listen_ssl in terms of returning a list of ports, and wanted to check that was a safe thing to do - hence the logging in `refresh_certificate`. Also, pull the 'Synapse now listening' message up to homeserver.py, because it was being duplicated everywhere else.	2019-02-13 11:58:54 +00:00
Richard van der Hoff	e3a0300431	Special-case the default bind_addresses for metrics listener ... turns out it doesn't really support ipv6, so let's hack around that by only listening on ipv4 by default.	2019-02-13 11:48:56 +00:00
Erik Johnston	93f7d2df3e	Comments	2019-02-12 16:03:40 +00:00
Erik Johnston	6a8f902edb	Raise an appropriate error message if sentry_sdk missing	2019-02-12 16:01:41 +00:00
Erik Johnston	ef2228c890	Basic sentry integration	2019-02-12 13:55:58 +00:00
Erik Johnston	d2fa7b7e99	Update changelog and version	2019-02-12 13:22:25 +00:00
Erik Johnston	ba3f27b69a	Merge pull request #4608 from matrix-org/anoa/acls_room_upgrade ... Transfer Server ACLs on room upgrade	2019-02-12 13:20:06 +00:00
Erik Johnston	cf82338930	Merge pull request #4627 from matrix-org/erikj/user_ips_analyze ... Analyze user_ips before running deduplication	2019-02-12 13:05:09 +00:00
Erik Johnston	3df8fcca25	Merge pull request #4626 from matrix-org/erikj/fixup_user_ips_dedupe ... Reduce user_ips bloat during dedupe background update	2019-02-12 13:02:58 +00:00
Erik Johnston	495ea92350	Fix pep8	2019-02-12 12:40:42 +00:00
Erik Johnston	483ba85c7a	Analyze user_ips before running deduplication ... Due to the table locks taken out by the naive upsert, the table statistics may be out of date. During deduplication it is important that the correct index is used as otherwise a full table scan may be incorrectly used, which can end up thrashing the database badly.	2019-02-12 11:55:27 +00:00
Erik Johnston	362d80b770	Reduce user_ips bloat during dedupe background update ... The background update to remove duplicate rows naively deleted and reinserted the duplicates. For large tables with a large number of duplicates this causes a lot of bloat (with postgres), as the inserted rows are appended to the table, since deleted rows will not be overwritten until a VACUUM has happened. This should hopefully also help ensure that the query in the last batch uses the correct index, as inserting a large number of new rows without analyzing will upset the query planner.	2019-02-12 11:39:34 +00:00
Erik Johnston	3c03c37883	Merge pull request #4625 from matrix-org/rav/fix_generate_config_warnings ... fix self-signed cert notice from generate-config	2019-02-12 11:24:45 +00:00
Richard van der Hoff	a4ce91396b	Disable TLS by default (#4614)	2019-02-12 10:52:08 +00:00
Richard van der Hoff	32b781bfe2	Fix error when loading cert if tls is disabled (#4618) ... If TLS is disabled, it should not be an error if no cert is given. Fixes #4554.	2019-02-12 10:51:31 +00:00
Richard van der Hoff	dfc846a316	fix self-signed cert notice from generate-config ... fixes #4620	2019-02-12 10:37:59 +00:00
Richard van der Hoff	0ca2908653	fix tests	2019-02-11 22:01:27 +00:00
Richard van der Hoff	4fddf8fc77	Infer no_tls from presence of TLS listeners ... Rather than have to specify `no_tls` explicitly, infer whether we need to load the TLS keys etc from whether we have any TLS-enabled listeners.	2019-02-11 21:39:14 +00:00
Richard van der Hoff	15272f837c	Merge branch 'rav/no_create_server_contexts_if_no_tls' into rav/tls_cert/work	2019-02-11 21:34:19 +00:00
Richard van der Hoff	9645728619	Don't create server contexts when TLS is disabled ... we aren't going to use them anyway.	2019-02-11 21:32:01 +00:00
Richard van der Hoff	be794c7cf7	Merge branch 'rav/tls_config_logging_fixes' into rav/tls_cert/work	2019-02-11 21:16:00 +00:00
Richard van der Hoff	2129dd1a02	Fail cleanly if listener config lacks a 'port' ... ... otherwise we would fail with a mysterious KeyError or something later.	2019-02-11 21:15:01 +00:00
Richard van der Hoff	086f6f27d4	Logging improvements around TLS certs ... Log which file we're reading keys and certs from, and refactor the code a bit in preparation for other work	2019-02-11 21:02:06 +00:00
Richard van der Hoff	5d27730a73	Move ClientTLSOptionsFactory init out of refresh_certificates (#4611) ... It's nothing to do with refreshing the certificates. No idea why it was here.	2019-02-11 18:03:30 +00:00
Erik Johnston	719e073f00	Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating ... add updating of backup versions	2019-02-11 13:45:49 +00:00
Richard van der Hoff	24b7f3916d	Clean up default listener configuration (#4586) ... Rearrange the comments to try to clarify them, and expand on what some of it means. Use a sensible default 'bind_addresses' setting. For the insecure port, only bind to localhost, and enable x_forwarded, since apparently it's for use behind a load-balancer.	2019-02-11 12:50:30 +00:00
Andrew Morgan	a126f86eec	Transfer Server ACLs on room upgrade	2019-02-11 11:30:37 +00:00
Amber Brown	6e2a5aa050	ACME Reprovisioning (#4522)	2019-02-11 10:36:26 +00:00
Amber Brown	4ffd10f46d	Be tolerant of blank TLS fingerprints config (#4589)	2019-02-11 10:04:27 +00:00
Erik Johnston	b201149c7e	Merge pull request #4420 from matrix-org/jaywink/openid-listener ... New listener resource for the federation API "openid/userinfo" endpoint	2019-02-11 09:44:00 +00:00
Valentin Anger	2dc2b6e9f1	Allow "unavailable" presence status for /sync (#4592) ... * Allow "unavailable" presence status for /sync Closes #3772, closes #3779 Signed-off-by: Valentin Anger <valentin.an.1999@gmail.com> * Add changelog for PR 4592	2019-02-08 21:09:56 +00:00
Richard van der Hoff	56710c7df5	Fix 'no unique or exclusion constraint' error (#4591) ... Add more tables to the list of tables which need a background update to complete before we can upsert into them, which fixes a race against the background updates.	2019-02-08 18:30:46 +00:00
Amber Brown	9cd33d2f4b	Deduplicate some code in synapse.app (#4567)	2019-02-08 17:25:57 +00:00
Hubert Chathi	afae8442b5	make sure version is in body and wrap in linearizer queue ... also add tests	2019-02-08 01:32:45 -05:00
Hubert Chathi	d9e424bf64	re-try to make isort happy	2019-02-06 22:18:41 -05:00
Hubert Chathi	9ff620a518	fix import to make isort happy	2019-02-06 21:32:52 -05:00
Hubert Chathi	8248637173	add new endpoint to update backup versions	2019-02-06 17:57:10 -05:00
Hubert Chathi	664c81e8b7	return proper error codes for some 404s	2019-02-06 17:47:22 -05:00
Richard van der Hoff	2475434080	Merge branch 'master' into develop	2019-02-05 18:44:49 +00:00
Richard van der Hoff	3bd9daf4b8	v0.99.0	2019-02-05 18:33:02 +00:00
Andrew Morgan	627ecd358e	Filter user directory state query to a subset of state events (#4462) ... * Filter user directory state query to a subset of state events * Add changelog	2019-02-05 12:16:28 +00:00
Richard van der Hoff	bf1e4d96ad	Fix default ACME config for py2 (#4564) ... Fixes #4559	2019-02-05 11:37:33 +00:00
Richard van der Hoff	d7e27a1f08	fix typo in config comments (#4557)	2019-02-05 11:32:45 +00:00
Richard van der Hoff	30fd2f89db	0.99.0rc4	2019-02-01 15:52:28 +00:00
Richard van der Hoff	ef43a03fc5	Merge pull request #4546 from matrix-org/rav/silence_critical_error_from_federation ... Fix noisy "twisted.internet.task.TaskStopped" errors in logs	2019-02-01 14:37:22 +00:00
Richard van der Hoff	f8db967d5a	Merge remote-tracking branch 'origin/release-v0.99.0' into develop	2019-02-01 13:20:15 +00:00
Richard van der Hoff	fa794980ec	Merge pull request #4544 from matrix-org/rav/skip_invalid_well_known ... Treat an invalid .well-known the same as an absent one	2019-02-01 13:18:36 +00:00
Richard van der Hoff	e9779a6f8f	Fix b'ab' noise in logs	2019-02-01 12:34:31 +00:00
Richard van der Hoff	9763a73af0	Merge branch 'release-v0.99.0' into develop	2019-02-01 12:30:22 +00:00
Richard van der Hoff	f0ba34f581	Fix noisy "twisted.internet.task.TaskStopped" errors in logs ... Fixes #4003	2019-02-01 12:22:57 +00:00
Richard van der Hoff	8a21b03fba	Treat an invalid .well-known the same as an absent one ... ... basically, carry on and fall back to SRV etc.	2019-02-01 11:37:31 +00:00
Richard van der Hoff	3c8a41140e	Cache failures to parse .well-known ... Also add a Measure block around the .well-known fetch	2019-02-01 00:37:52 +00:00
Richard van der Hoff	24d59c7568	better logging for federation connections	2019-01-31 23:18:20 +00:00
Richard van der Hoff	d428b46346	Update federation routing logic to check .well-known before SRV	2019-01-31 23:14:18 +00:00
Andrew Morgan	c45fd0dda0	Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable ... Copy over non-federatable trait on room upgrade	2019-01-31 20:28:37 +00:00
Richard van der Hoff	625385d684	Merge branch 'release-v0.99.0' into develop	2019-01-31 18:43:20 +00:00
Richard van der Hoff	85129d7068	v0.99.0rc3	2019-01-31 18:35:38 +00:00
Andrew Morgan	d239f67c25	Raise an exception instead of returning None	2019-01-31 18:34:15 +00:00
Richard van der Hoff	07dfe148de	Add some debug for membership syncing issues (#4538) ... I can't figure out what's going on with #4422 and #4436; perhaps this will help.	2019-01-31 18:30:40 +00:00
Andrew Morgan	3ed3cb4339	New function for getting room's create event	2019-01-31 18:21:39 +00:00
Andrew Morgan	bbb97a35fd	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable	2019-01-31 18:11:00 +00:00
Richard van der Hoff	e707e7b38d	Fix infinite loop when an event is redacted in a v3 room (#4535)	2019-01-31 15:34:17 +00:00
Andrew Morgan	563f6a832b	Reject large transactions on federation (#4513) ... * Reject large transactions on federation * Add changelog * lint * Simplify large transaction handling	2019-01-31 11:44:04 +00:00
Andrew Morgan	fb50934b8f	lint	2019-01-31 11:34:45 +00:00
Richard van der Hoff	6fba9fd20c	Merge remote-tracking branch 'origin/release-v0.99.0' into develop	2019-01-30 17:02:32 +00:00
Andrew Morgan	d621c5562e	Copy over non-federatable trait on room upgrade	2019-01-30 16:33:51 +00:00
Richard van der Hoff	b8b898666e	v0.99.0rc2	2019-01-30 16:31:07 +00:00
Matthew Hodgson	ad7ac8853c	by default include m.room.encryption on invites (#3902) ... * by default include m.room.encryption on invites * fix constant * changelog	2019-01-30 16:26:13 +00:00
Richard van der Hoff	c74b96755c	Merge remote-tracking branch 'origin/develop' into release-v0.99.0	2019-01-30 16:23:28 +00:00
Neil Johnson	81b7e7eed3	Update constants.py ... remove trailing ,	2019-01-30 16:11:36 +00:00
Erik Johnston	ed8c5e4cda	Fix remote invite rejections not comming down sync ... This was broken in PR #4405, commit 886e5ac, where we changed remote rejections to be outliers. The fix is to explicitly add the leave event in when we know its an out of band invite. We can't always add the event as if the server is/was in the room there might be more events to send down the sync than just the leave.	2019-01-30 15:46:27 +00:00
Richard van der Hoff	a5d0c771a3	0.99.0rc1	2019-01-30 15:11:18 +00:00
Erik Johnston	6587b0b89b	Merge pull request #4472 from matrix-org/neilj/room_capabilities ... Server capabilities support	2019-01-30 14:26:56 +00:00
Erik Johnston	a4f52a33fe	Fix replication for room v3 (#4523) ... * Fix replication for room v3 We were not correctly quoting the path fragments over http replication, which meant that it exploded when the event IDs had a slash in them * Newsfile	2019-01-30 14:19:52 +00:00
Richard van der Hoff	7615a8ced1	ACME config cleanups (#4525) ... * Handle listening for ACME requests on IPv6 addresses the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses without extra quoting. Building a string which you are about to parse again seems like a weird choice. Let's just use listenTCP, which is consistent with what we do elsewhere. * Clean up the default ACME config make it look a bit more consistent with everything else, and tweak the defaults to listen on port 80. * newsfile	2019-01-30 14:17:55 +00:00
Erik Johnston	e87d7a4b0f	Raise ConfigError instead	2019-01-30 12:48:09 +00:00
Erik Johnston	270f212a2a	_listener_http should return a list	2019-01-30 12:14:50 +00:00
Richard van der Hoff	a79034aedf	Merge pull request #4521 from matrix-org/rav/fed_routing/cleanups ... Tiny .well-known fixes	2019-01-30 11:47:24 +00:00
Richard van der Hoff	c7b24ac3d0	Follow redirects on .well-known (#4520)	2019-01-30 11:43:33 +00:00
Amber Brown	f6813919e8	SIGHUP for TLS cert reloading (#4495)	2019-01-30 11:00:02 +00:00
Richard van der Hoff	09a1a6b55e	fix exception text	2019-01-30 10:58:52 +00:00
Richard van der Hoff	928c50b59a	Also jitter the invalid cache period	2019-01-30 10:58:52 +00:00
Neil Johnson	b37e8c9572	Merge branch 'neilj/room_capabilities' of github.com:matrix-org/synapse into neilj/room_capabilities	2019-01-30 10:56:47 +00:00
Neil Johnson	f834d98402	isort	2019-01-30 10:55:42 +00:00
Richard van der Hoff	bc5f6e1797	Add a caching layer to .well-known responses (#4516)	2019-01-30 10:55:25 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519)	2019-01-30 10:53:17 +00:00
Neil Johnson	ee4df7fd7a	Merge branch 'develop' into neilj/room_capabilities	2019-01-30 10:28:08 +00:00
Neil Johnson	c5a0f82cca	define room dispositions for use in exposing room capabilities	2019-01-30 10:24:24 +00:00
Neil Johnson	9c850d9d5e	formatting and use constants where available	2019-01-30 10:23:26 +00:00
Neil Johnson	2f46804055	Populate default room version from Constants	2019-01-30 09:39:10 +00:00
Neil Johnson	c7837dce24	reflect that rooms v3 is a stable room version	2019-01-30 09:33:30 +00:00
Richard van der Hoff	457fbfaf22	Merge pull request #4486 from xperimental/workaround-4216 ... Implement workaround for login error.	2019-01-30 07:06:11 +00:00
Robert Jacob	2a7f0b8953	Implement workaround for login error. ... Signed-off-by: Robert Jacob <xperimental@solidproject.de>	2019-01-30 01:06:39 +01:00
Erik Johnston	0b24d58e05	No vdh tests!	2019-01-29 23:11:48 +00:00
Erik Johnston	a1b0e1879b	Enable room version v3	2019-01-29 23:09:10 +00:00
Erik Johnston	67b82f1336	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:58:38 +00:00
Erik Johnston	afeea319df	Fixup comment	2019-01-29 22:55:29 +00:00
Erik Johnston	7740eddd04	Merge pull request #4514 from matrix-org/erikj/remove_event_id ... Remove usages of event ID's domain	2019-01-29 22:54:25 +00:00
Erik Johnston	ff2f65d737	Update comment	2019-01-29 22:35:36 +00:00
Erik Johnston	655ce037fd	check event format version not room version	2019-01-29 22:33:43 +00:00
Erik Johnston	f46a818ce5	kill vdh test some more	2019-01-29 22:02:58 +00:00
Erik Johnston	a696c48133	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:00:33 +00:00
Erik Johnston	c21b7cbc09	Update synapse/storage/events_worker.py	2019-01-29 21:53:48 +00:00
Erik Johnston	4db252c073	Check redaction state when event is pulled out of the database	2019-01-29 21:48:36 +00:00
Erik Johnston	6d23ec2111	Fix typo	2019-01-29 21:45:53 +00:00
Erik Johnston	b5d510ad64	Remove unused arg	2019-01-29 21:45:28 +00:00
Erik Johnston	6f9cdc2d47	Merge pull request #4483 from matrix-org/erikj/event_v2 ... Implement event format V2	2019-01-29 21:40:00 +00:00
Erik Johnston	47e2dd1994	Drop vdh support	2019-01-29 21:24:34 +00:00
Erik Johnston	38590a4870	Add docstring	2019-01-29 21:22:47 +00:00
Richard van der Hoff	82165eeb05	Update synapse/storage/events_worker.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2019-01-29 21:14:39 +00:00
Andrew Morgan	03b086647f	Merge pull request #4512 from matrix-org/anoa/consent_dir ... Check consent dir path on startup	2019-01-29 20:08:18 +00:00
Erik Johnston	84af577356	Implement event format v2	2019-01-29 18:06:11 +00:00
Neil Johnson	e4bef9d470	rework format of change password capability	2019-01-29 18:04:56 +00:00
Erik Johnston	610f0830b0	Don't assert an event must have an event ID	2019-01-29 18:02:26 +00:00
Erik Johnston	840068bd78	Only check event ID domain for signatures for V1 events ... In future version events won't have an event ID, so we won't be able to do this check.	2019-01-29 18:02:02 +00:00
Erik Johnston	8e3d34e3c5	Use event origin for filtering incoming events ... We only process events sent to us from a server if the event ID matches the server, to help guard against federation storms. We replace this with a check against the event origin.	2019-01-29 16:57:00 +00:00
Erik Johnston	55d9024835	Use snder and not event ID domain to check if ours ... The transaction queue only sends out events that we generate. This was done by checking domain of event ID, but that can no longer be used. Instead, we may as well use the sender field.	2019-01-29 16:54:23 +00:00
Richard van der Hoff	cc2d650ef7	Relax requirement for a content-type on .well-known (#4511)	2019-01-29 16:49:17 +00:00
Erik Johnston	b1fffca345	Remove event ID usage when checking if new room ... The event ID is changing, so we can no longer get the domain from it. On the other hand, the check is unnecessary.	2019-01-29 16:15:02 +00:00
Erik Johnston	770b823445	Only check event IDs domain signed event for V1 and V2 ... Since newer versions of events don't have the same format for event ID.	2019-01-29 16:15:00 +00:00
Travis Ralston	d02c5ccb11	Merge pull request #4498 from matrix-org/travis/fix-docs-public_baseurl ... Don't recommend :8448 to people on public_baseurl	2019-01-29 09:06:16 -07:00
Neil Johnson	f03b3a7a3a	support change_password in capabilities end-point	2019-01-29 15:58:37 +00:00
Andrew Morgan	e65a17b26f	Check consent dir path on startup	2019-01-29 15:30:33 +00:00
Amber Brown	6bd4374636	Do not generate self-signed TLS certificates by default. (#4509)	2019-01-29 14:09:10 +00:00
Erik Johnston	b8d75ef53e	Merge pull request #4481 from matrix-org/erikj/event_builder ... Refactor event building into EventBuilder	2019-01-29 14:07:23 +00:00
Richard van der Hoff	99e36d5e24	Implement MSC1708 (.well-known lookups for server routing) (#4489)	2019-01-29 13:53:02 +00:00
Erik Johnston	b82a76c384	Finish comment...	2019-01-29 13:50:59 +00:00
Erik Johnston	2562319821	Merge pull request #4510 from matrix-org/erikj/fixup_compute_event_signature ... Fixup calls to `comput_event_signature`	2019-01-29 13:35:19 +00:00
Erik Johnston	5891a6edc8	Correctly set context.app_service	2019-01-29 12:09:10 +00:00
Erik Johnston	fb99dae9c8	Don't set event_id twice	2019-01-29 12:08:23 +00:00
Erik Johnston	64c1bd1d21	Remove dead function	2019-01-29 12:06:28 +00:00
Erik Johnston	7709d2bd16	Implement rechecking of redactions	2019-01-29 11:56:20 +00:00
Erik Johnston	7a3ec5b022	Add RoomVersions.V3 constant, without enabling it ... We add the constant, but don't add it to the known room versions. This lets us start adding V3 logic, but the servers will never join or create V3 rooms	2019-01-29 11:55:33 +00:00
Erik Johnston	6598992b01	Fixup calls to comput_event_signature ... We currently pass FrozenEvent instead of `dict` to `compute_event_signature`, which works by accident due to `dict(event)` producing the correct result. This fixes PR #4493 commit 855a151	2019-01-29 11:41:58 +00:00
Neil Johnson	a124025dab	enforce auth for capabilities endpoint	2019-01-29 11:37:56 +00:00
Erik Johnston	ff37acb8ce	Merge pull request #4496 from matrix-org/erikj/invite_fallback ... Implement fallback for V2 invite API	2019-01-29 11:28:23 +00:00
Erik Johnston	5180f12bae	Replace usage of builder.user_id with builder.sender ... `.user_id` is proxed to `.sender` in FrozenEvent, so this has no functional change	2019-01-29 11:18:38 +00:00
Erik Johnston	be47cfa9c9	Refactor event building into EventBuilder ... This is so that everything is done in one place, making it easier to change the event format based on room version	2019-01-29 11:13:00 +00:00
Erik Johnston	554ca58ea1	Make add_hashes_and_signatures operate on dicts	2019-01-29 11:12:38 +00:00
Amber Brown	f815bd7feb	Make linearizer more quiet (#4507)	2019-01-29 11:05:31 +00:00
Erik Johnston	073f6c2e5e	Merge pull request #4494 from matrix-org/erikj/fixup_event_validator ... Split up event validation between event and builder	2019-01-29 10:55:07 +00:00
Erik Johnston	40638ae7f5	Remove duplicate checks	2019-01-29 10:37:40 +00:00
Erik Johnston	9fa3c6ffa3	Fix up error messages	2019-01-29 10:36:46 +00:00
Erik Johnston	28efc80723	Fold validate into validate_new	2019-01-29 10:34:49 +00:00
Erik Johnston	b6b73a0bcf	Fix receiving events from federation via a worker ... This bug was introduced in PR #4470, commit 678a92cb56	2019-01-29 10:30:26 +00:00
Neil Johnson	327b992e17	register capabilities servlet	2019-01-29 10:28:35 +00:00
Amber Brown	94fb63e44f	Fix typo in upserts code (#4505) ... * fix obvious problem :| * changelog	2019-01-29 10:04:23 +00:00
Erik Johnston	17709f8f9c	Merge pull request #4493 from matrix-org/erikj/refactor_event_signing ... Refactor event signing to work on dicts	2019-01-29 09:52:54 +00:00
Erik Johnston	f1a04462eb	Merge pull request #4482 from matrix-org/erikj/event_auth_room_version ... Pass through room version to event auth	2019-01-28 20:09:38 +00:00
Travis Ralston	6901ac7e9d	Don't recommend :8448 to people on public_baseurl	2019-01-28 12:15:22 -07:00
Erik Johnston	d414f30019	Implement fallback for V2 invite API ... If the room version is either 1 or 2 then a server should retry failed `/v2/invite` requests with the v1 API	2019-01-28 17:33:25 +00:00
Erik Johnston	d758d5310e	Correctly use default room version if none is set	2019-01-28 17:26:39 +00:00
Amber Brown	5d976c0c7c	Fix worker TLS (#4492) ... * load cert * changelog * fix	2019-01-28 17:18:33 +00:00
Erik Johnston	b872c7b1b4	Split up event validation between event and builder ... The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.	2019-01-28 17:00:14 +00:00
Erik Johnston	855a151015	Refactor event signing to work on dicts ... This is in preparation for making EventBuilder format agnostic, which means event signing should be done against the event dict rather than the EventBuilder object.	2019-01-28 16:42:10 +00:00
Amber Brown	7072fe3084	Fix UPSERTs on SQLite 3.24+ (#4477)	2019-01-28 15:43:32 +00:00
Andrew Morgan	f0e96ab66a	Change return syntax in doc string	2019-01-28 14:09:45 +00:00
Andrew Morgan	4026d555fa	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-28 14:08:24 +00:00
Andrew Morgan	1ce463963d	Reuse predecessor method	2019-01-28 14:08:18 +00:00
Richard van der Hoff	3bd0f1a4a3	docstrings for _RoutingResult	2019-01-28 12:43:09 +00:00
Richard van der Hoff	0fd5b3b53e	Handle IP literals explicitly ... We don't want to be doing .well-known lookups on these guys.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	51958df766	MatrixFederationAgent: factor out routing logic ... This is going to get too big and unmanageable.	2019-01-28 10:34:30 +00:00
Richard van der Hoff	d840019192	Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487) ... Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.	2019-01-28 09:56:59 +00:00
Erik Johnston	ae2a957dba	Pass through room version to event auth	2019-01-25 18:31:41 +00:00
Neil Johnson	893107be78	backout v3	2019-01-25 17:27:36 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version ... Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Erik Johnston	431e485914	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert	2019-01-25 14:11:17 +00:00
Richard van der Hoff	4a3f138832	Fix quoting for allowed_local_3pids example config (#4476) ... If you use double-quotes here, you have to escape your backslashes. It's much easier with single-quotes. (Note that the existing double-backslashes are already interpreted by python's """ parsing.)	2019-01-25 13:57:52 +00:00
Richard van der Hoff	8520bc3109	Fix Host header sent by MatrixFederationAgent (#4468) ... Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.	2019-01-25 12:38:16 +00:00
Andrew Morgan	9244a3089e	Fixes	2019-01-25 11:48:38 +00:00
Andrew Morgan	da0d2219d2	Clean up direct_rooms access	2019-01-25 11:37:12 +00:00
Andrew Morgan	8265995498	Use python magic	2019-01-25 11:26:06 +00:00
Andrew Morgan	0b3fd1401f	Don't require sqlite3 when using postgres (#4466)	2019-01-25 22:25:02 +11:00
Andrew Morgan	c4cdafa81f	Destructure account data tuple before use	2019-01-25 11:24:28 +00:00
Andrew Morgan	516456b763	Remove unnecessary null check	2019-01-25 11:22:14 +00:00
Andrew Morgan	6f3fda79ce	Move room_tag declaration to be closer to its use	2019-01-25 11:21:25 +00:00
Neil Johnson	95f871fc0d	Support room version capabilities in CS API (MSC1804)	2019-01-25 11:16:29 +00:00
Andrew Morgan	821b65aeb5	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-25 11:09:53 +00:00
Andrew Morgan	0862d35b8e	Move tag and direct state copying into separate function	2019-01-25 11:09:34 +00:00
Neil Johnson	53ef4da8c2	track unstable room v3	2019-01-25 11:04:11 +00:00
Andrew Morgan	b1b6dba2d2	Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms ... Ability to search entire room history after upgrading room	2019-01-25 10:49:58 +00:00
Erik Johnston	1953067136	Disable native upserts for sqlite, as they don't work	2019-01-25 10:46:49 +00:00
Erik Johnston	678a92cb56	Replace missed usages of FrozenEvent	2019-01-25 10:32:30 +00:00
Erik Johnston	a50cf929c1	Require event format version to parse or create events	2019-01-25 10:32:19 +00:00
Erik Johnston	62514bb81b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813	2019-01-25 10:07:08 +00:00
Erik Johnston	edc1e21dbe	Merge pull request #4448 from matrix-org/erikj/get_pdu_versions ... Add room_version param to get_pdu	2019-01-24 18:47:15 +00:00
Erik Johnston	80bcca659e	Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites ... Store rejected remote invite events as outliers	2019-01-24 18:33:27 +00:00
Erik Johnston	664b7a2920	Merge pull request #4437 from matrix-org/erikj/event_format_version_v2 ... Add support for persisting event format versions	2019-01-24 18:32:30 +00:00
Erik Johnston	d148c43050	Review comments	2019-01-24 18:31:23 +00:00
Erik Johnston	26f44164c8	Review comments	2019-01-24 18:28:00 +00:00
Erik Johnston	5ee1f997a8	Update make_membership_event docs	2019-01-24 18:08:08 +00:00
Erik Johnston	9139b87be4	Remove unecessary setting of outlier bit	2019-01-24 18:04:02 +00:00
Erik Johnston	b8082a5445	Use term 'out of band membership' instead	2019-01-24 17:33:19 +00:00
Andrew Morgan	a383289b0d	Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms	2019-01-24 17:23:51 +00:00
Andrew Morgan	e1781b043b	Remove redundant create event None check	2019-01-24 17:23:39 +00:00
Richard van der Hoff	03c85335d1	Apply suggestions from code review ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:22:09 +00:00
Richard van der Hoff	8ea509a935	Update synapse/api/filtering.py ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:21:35 +00:00
Andrew Morgan	50c396a7ee	Merge pull request #4461 from matrix-org/anoa/room_dir_quick_fix ... Add a 60s timeout to filtered room directory queries	2019-01-24 15:24:35 +00:00
Andrew Morgan	075ff3ede9	Change default timeout value from 0 to None	2019-01-24 15:10:22 +00:00
Andrew Morgan	5b1dc94083	Use self.clock instead of datetime	2019-01-24 14:59:50 +00:00
Andrew Morgan	a2d85144e5	isort	2019-01-24 14:22:26 +00:00
Richard van der Hoff	4a6e863843	Merge pull request #4464 from matrix-org/rav/fix_srv_lookup ... MatrixFederationAgent: Look up the right SRV record	2019-01-24 13:57:51 +00:00
Richard van der Hoff	f4697b5ec1	Fix UnboundLocalError in post_urlencoded_get_json (#4460) ... This could cause exceptions if the id server returned 4xx responses.	2019-01-24 13:38:29 +00:00
Richard van der Hoff	afd69a0920	Look up the right SRV record	2019-01-24 13:31:43 +00:00
Neil Johnson	10b89d5c2e	Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check ... Neilj/fix threepid auth check	2019-01-24 13:02:50 +00:00
Andrew Morgan	5541645e80	lint	2019-01-24 12:45:32 +00:00
Andrew Morgan	068aa1d228	Time out filtered room dir queries after 60s	2019-01-24 12:44:27 +00:00
Neil Johnson	5c41b22359	Merge pull request #4458 from matrix-org/dbkr/public_baseurl_doc ... Clarify docs for public_baseurl	2019-01-24 12:36:09 +00:00
Amber Brown	0e27501ee5	Fix UPSERT check (#4459)	2019-01-24 22:57:41 +11:00
David Baker	92d8a068ad	Clarify docs for public_baseurl ... This is leading to problems with people upgrading to clients that support MSC1730 because people have this misconfigured, so try to make the docs completely unambiguous.	2019-01-24 10:52:06 +00:00
Amber Brown	58f6c48183	Use native UPSERTs where possible (#4306)	2019-01-24 21:31:54 +11:00
Richard van der Hoff	97fd29c019	Don't send IP addresses as SNI (#4452) ... The problem here is that we have cut-and-pasted an impl from Twisted, and then failed to maintain it. It was fixed in Twisted in https://github.com/twisted/twisted/pull/1047/files; let's do the same here.	2019-01-24 09:34:44 +00:00
Erik Johnston	7c288c2250	Clarify the invite flows	2019-01-23 20:07:47 +00:00
Erik Johnston	07f62da55a	Remove unnecessary '_sign_event'	2019-01-23 20:04:57 +00:00
Erik Johnston	886e5acc76	Store rejected remote invite events as outliers ... Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.	2019-01-23 20:04:57 +00:00
Willem Mulder	d528406cb8	Fix error message for optional dependencies ... Signed-off-by: Willem Mulder <willemmaster@hotmail.com>	2019-01-23 18:44:57 +01:00
Erik Johnston	6a41d2a187	Add room_version param to get_pdu ... When we add new event format we'll need to know the event format or room version when parsing events.	2019-01-23 17:19:58 +00:00
Erik Johnston	67cd4dad81	Implement MSC 1813 - Add room version to make APIs ... We also implement `make_membership_event` converting the returned room version to an event format version.	2019-01-23 16:51:46 +00:00
Andrew Morgan	6b90ae6efc	Merge pull request #4445 from matrix-org/anoa/user_dir_develop_backport ... Add metric for user dir current event stream position	2019-01-23 15:58:40 +00:00
Richard van der Hoff	a0ae475219	Merge pull request #4428 from matrix-org/rav/matrix_federation_agent ... Move SRV magic into an Agent-like thing	2019-01-23 15:50:25 +00:00
Andrew Morgan	82a92ba535	Add metric for user dir current event stream position	2019-01-23 15:34:47 +00:00
Erik Johnston	be1065af59	isort	2019-01-23 11:48:16 +00:00
Erik Johnston	c5a296b10c	Add support for persisting event format versions ... Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.	2019-01-23 11:30:01 +00:00
Jason Robinson	6f680241bd	Fix flake8 issues ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:53:48 +02:00
Erik Johnston	90743c9d89	Fixup removal of duplicate user_ips rows (#4432) ... * Remove unnecessary ORDER BY clause * Add logging * Newsfile	2019-01-23 19:45:18 +11:00
Amber Brown	6129e52f43	Support ACME for certificate provisioning (#4384)	2019-01-23 19:39:06 +11:00
Jason Robinson	0516dc4d85	Remove openid resource from default config ... Instead document it commented out. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	d39b7b6d38	Document servlet_groups parameters ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	82e13662c0	Split federation OpenID userinfo endpoint out of the federation resource ... This allows the OpenID userinfo endpoint to be active even if the federation resource is not active. The OpenID userinfo endpoint is called by integration managers to verify user actions using the client API OpenID access token. Without this verification, the integration manager cannot know that the access token is valid. The OpenID userinfo endpoint will be loaded in the case that either "federation" or "openid" resource is defined. The new "openid" resource is defaulted to active in default configuration. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	a17bac171f	Make SynapseHomeServer _http_listener use self.get_reactor() ... For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	4f8f41c824	Make FederationReaderServer _http_listen use self.get_reactor() ... For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Jason Robinson	899e60be80	Add parameterized Python module to test dependencies ... Allows running parameterized tests. BSD license. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-01-23 10:32:41 +02:00
Richard van der Hoff	7021784d46	put resolve_service in an object ... this makes it easier to stub things out for tests.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	53a327b4d5	Require that service_name be a byte string ... it is only ever a bytes now, so let's enforce that.	2019-01-22 20:35:12 +00:00
Richard van der Hoff	fe212bbe4a	Kill off matrix_federation_endpoint ... this thing is now redundant.	2019-01-22 20:34:35 +00:00
Richard van der Hoff	7871146667	Make MatrixFederationClient use MatrixFederationAgent ... ... instead of the matrix_federation_endpoint	2019-01-22 20:34:35 +00:00
Richard van der Hoff	44be7513bf	MatrixFederationAgent ... Pull the magic that is currently in matrix_federation_endpoint and friends into an agent-like thing	2019-01-22 20:34:35 +00:00
Neil Johnson	c99c2d58d7	move guard out of is_threepid_reserved and into register.py	2019-01-22 17:47:00 +00:00
Neil Johnson	d619b113ed	Fix None guard in config.server.is_threepid_reserved	2019-01-22 16:52:29 +00:00
Erik Johnston	12699a701f	Merge pull request #4434 from matrix-org/erikj/fix_user_ips_dedup ... Fix bug when removing duplicate rows from user_ips	2019-01-22 16:51:57 +00:00
Erik Johnston	7f503f83b9	Refactor to rewrite the SQL instead	2019-01-22 16:31:05 +00:00
Neil Johnson	388c164aea	Merge pull request #4423 from matrix-org/neilj/disable_msisdn_on_registration ... Config option to disable requesting MSISDN on registration	2019-01-22 16:23:08 +00:00
Erik Johnston	1c9704f8ab	Don't shadow params	2019-01-22 16:20:33 +00:00
Andrew Morgan	766a172b99	lint	2019-01-22 13:51:40 +00:00
Erik Johnston	2557531f0f	Fix bug when removing duplicate rows from user_ips ... This was caused by accidentally overwritting a `last_seen` variable in a for loop, causing the wrong value to be written to the progress table. The result of which was that we didn't scan sections of the table when searching for duplicates, and so some duplicates did not get deleted.	2019-01-22 13:33:46 +00:00
Andrew Morgan	117bc94cd2	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-22 13:16:51 +00:00
Andrew Morgan	277e50462d	Do not return in a deferred function	2019-01-22 12:40:26 +00:00
Andrew Morgan	c4875d8c76	Prevent duplicate room IDs in m.direct	2019-01-22 12:13:46 +00:00
Andrew Morgan	c433f61091	Ensure new filter is actually created	2019-01-22 12:06:36 +00:00
Andrew Morgan	c9bfb058d8	Fix a bug with single-room search searching all rooms ... * Create a new method for getting predecessor rooms * Remove formatting change	2019-01-22 12:00:41 +00:00
Andrew Morgan	8086a5c05e	Fix comments	2019-01-22 11:16:23 +00:00
Richard van der Hoff	6bfa735a69	Make key fetches use regular federation client (#4426) ... All this magic is redundant.	2019-01-22 11:04:20 +00:00
Andrew Morgan	48951f437f	Join logic covers both room creator and arbitrary users	2019-01-22 11:00:04 +00:00
Andrew Morgan	8c85f0833d	tags, m.direct copying over correctly	2019-01-22 11:00:04 +00:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Andrew Morgan	ea8903fcc9	Migrating dm and room tags work for migrator	2019-01-22 11:00:04 +00:00
Andrew Morgan	887ca93a1b	Prevent crash on user who doesn't have any direct rooms	2019-01-22 11:00:04 +00:00
Andrew Morgan	4ff6d22245	Preserve DM status of a room on upgrade ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-22 11:00:04 +00:00
Andrew Morgan	75942af1db	Fix typo	2019-01-22 11:00:04 +00:00
Richard van der Hoff	33a55289cb	Refactor and bugfix for resove_service (#4427)	2019-01-22 10:59:27 +00:00
Amber Brown	23b0813599	Require ECDH key exchange & remove dh_params (#4429) ... * remove dh_params and set better cipher string	2019-01-22 21:58:50 +11:00
Neil Johnson	1b53cc3cb4	fix line length	2019-01-21 15:17:20 +00:00
Neil Johnson	5349262302	Config option to disable requesting MSISDN on registration	2019-01-21 14:59:37 +00:00
Erik Johnston	35e1d67b4e	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_v2_invite_server	2019-01-21 14:04:19 +00:00
Erik Johnston	5f54765587	Merge pull request #4390 from matrix-org/erikj/versioned_fed_apis ... Add groundwork for new versions of federation APIs	2019-01-21 11:44:05 +00:00
Andrew Morgan	702c4b750c	Migrate encryption state on room upgrade (#4411) ... * Migrate encryption state on room upgrade Signed-off-by: Andrew Morgan <andrew@amorgan.xyz> * Add changelog file	2019-01-21 20:42:58 +11:00
Erik Johnston	25dd56ace3	Fix race when persisting create event (#4404) ... * Fix race when persisting create event When persisting a chunk of DAG it is sometimes requried to do a state resolution, which requires knowledge of the room version. If this happens while we're persisting the create event then we need to use that event rather than attempting to look it up in the database.	2019-01-18 23:17:04 +11:00
Richard van der Hoff	de6888e7ce	Remove redundant WrappedConnection (#4409) ... * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.	2019-01-18 23:07:38 +11:00
Andrew Morgan	df3a661e4a	Search for messages across predecessor rooms ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-18 11:19:20 +00:00
Richard van der Hoff	676cf2ee26	Fix incorrect logcontexts after a Deferred was cancelled (#4407)	2019-01-17 14:00:23 +00:00
Richard van der Hoff	9feb5d0b71	sign_request -> build_auth_headers (#4408) ... Just got very confused about the fact that the headers are only an output, not an input.	2019-01-17 12:40:09 +00:00
Richard van der Hoff	3982a6ee07	Changing macaroon_secret_key no longer logs you out (#4387)	2019-01-16 23:14:41 +00:00
Richard van der Hoff	05e1296649	don't store more remote device lists if they have more than 1K devices (#4397)	2019-01-16 23:14:11 +00:00
Andrej Shadura	64cf6788d9	Depend on pymacaroons >= 0.13.0 instead on pymacaroons-pynacl ... Since 0.13.0, pymacaroons works correctly with pynacl, so there isn’t any more reason to depend on an outdated pynacl fork. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>	2019-01-16 10:54:41 +01:00
Andrej Shadura	fab948120f	Use msgpack instead of msgpack-python ... The package msgpack-python has been deprecated. Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>	2019-01-16 10:53:49 +01:00
Neil Johnson	9ec56d6935	ALL_USER_TYPES should be a tuple	2019-01-15 14:38:15 +00:00
Erik Johnston	4a4d2e17bc	Add /v2/invite federation API	2019-01-15 13:22:44 +00:00
Erik Johnston	bb63e7ca4f	Add groundwork for new versions of federation APIs	2019-01-15 11:14:34 +00:00
Amber Brown	a35c66a00b	Remove duplicates in the user_ips table and add an index (#4370)	2019-01-12 06:21:50 +11:00
Amber Brown	522dada206	Merge remote-tracking branch 'origin/master' into develop	2019-01-11 02:22:48 +11:00
Amber Brown	6dc06c3775	version	2019-01-11 01:56:37 +11:00
Richard van der Hoff	e0910d0145	Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 ... Fixes #4371	2019-01-10 14:12:50 +00:00
Richard van der Hoff	ba41aeed6a	Revert "Fix macaroon_secret_key fallback logic" ... This is already fixed in 0.34.1, by 59f93bb This reverts commit efc522c55e.	2019-01-10 14:09:26 +00:00
Richard van der Hoff	aa70d24125	Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34	2019-01-10 12:58:33 +00:00
Richard van der Hoff	efc522c55e	Fix macaroon_secret_key fallback logic	2019-01-10 12:57:27 +00:00
Richard van der Hoff	566947ff34	Skip macaroon check for access tokens in the db	2019-01-10 12:57:21 +00:00
Richard van der Hoff	353f2407b7	Fix fallback to signing key for macaroon-secret-key	2019-01-10 12:42:56 +00:00
Richard van der Hoff	d566e6b17a	Merge branch 'master' into develop	2019-01-09 15:00:46 +00:00
Richard van der Hoff	998f5225c1	0.34.1	2019-01-09 14:53:54 +00:00
Erik Johnston	55c3e853c5	Merge pull request #4368 from matrix-org/erikj/better_errors ... Fixup docstrings for matrixfederationclient	2019-01-09 11:39:28 +00:00
Erik Johnston	484867d35d	Merge branch 'develop' into release-v0.34.1	2019-01-09 11:38:13 +00:00
Amber Brown	7960c26fda	Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369)	2019-01-09 22:26:25 +11:00
Erik Johnston	34ea14139d	Fixup docstrings for matrixfederationclient	2019-01-09 09:25:59 +00:00
Erik Johnston	d422570e9b	Use RequestSendFailed when fail to parse content type headers	2019-01-08 14:22:18 +00:00
Erik Johnston	1371d5b798	Don't log stack traces for HTTP error responses	2019-01-08 12:28:30 +00:00
Richard van der Hoff	ba85786d71	0.34.1rc1	2019-01-08 11:15:34 +00:00
Erik Johnston	b970cb0e96	Refactor request sending to have better excpetions (#4358) ... * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.	2019-01-08 11:04:28 +00:00
Amber Brown	1dcb086f33	Fix synapse.config.__main__ on python 3 (#4356)	2019-01-08 10:03:09 +11:00
Travis Ralston	bc1fa8cd01	Add GET account data routes (#4303) ... As per https://github.com/matrix-org/matrix-doc/issues/1339	2019-01-07 10:27:54 +00:00
Richard van der Hoff	b7c0218812	Check jinja version for consent resource (#4327) ... * Raise a ConfigError if an invalid resource is specified * Require Jinja 2.9 for the consent resource * changelog	2019-01-07 10:14:31 +00:00
Michael Telatynski	a27e501b09	fix the check for whether is_url to match all the other ones in codebase (#3405) ... Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2019-01-06 21:43:25 +00:00
Matthew Hodgson	27128145e6	fix NPE in /messages by checking if all events were filtered out (#4330)	2019-01-02 23:37:39 +00:00
Neil Johnson	84b6fae1f5	Ensure synchrotrons can access is_support_user in the storage layer	2019-01-02 10:19:59 +00:00
Amber Brown	d7843f47b6	Remove v1 only REST APIs now we don't ship matrix console (#4334)	2018-12-29 23:12:30 +11:00
Will Hunt	0708f437cc	Log roomid along with Unknown room (#4297)	2018-12-24 10:49:42 +01:00
Richard van der Hoff	cd018e3543	Merge pull request #4307 from matrix-org/erikj/v2_rooms ... Add v2 room version	2018-12-24 10:48:59 +01:00
Richard van der Hoff	f85676cc93	Return well_known in /login response (#4319) ... ... as per MSC1730.	2018-12-24 10:44:33 +01:00
Richard van der Hoff	9c2af7b2c5	Add a script to generate a clean config file (#4315)	2018-12-22 02:04:57 +11:00
Richard van der Hoff	f3561f8d86	Fix indentation in default config (#4313) ... These settings are not supposed to be under 'listeners'.	2018-12-22 02:01:43 +11:00
Amber Brown	ea6abf6724	Fix IP URL previews on Python 3 (#4215)	2018-12-22 01:56:13 +11:00
Amber Brown	c26f49a664	Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298)	2018-12-22 01:37:26 +11:00
Richard van der Hoff	0b26feb422	Merge branch 'master' into develop	2018-12-20 16:48:47 +00:00
Richard van der Hoff	6dd7271243	Prepare 0.34 release	2018-12-20 12:07:15 +00:00
Richard van der Hoff	d3c9c562c6	Debian packaging via dh_virtualenv	2018-12-20 11:35:23 +00:00
Richard van der Hoff	1a6d5bfa08	Debian packaging via dh_virtualenv (#4285)	2018-12-20 11:33:29 +00:00
David Baker	1c0051114a	Add 'sandbox' to CSP for media repo (#4284) ... * Add 'sandbox' to the CSP for media repo * Changelog	2018-12-20 11:09:18 +00:00
Nathan Pennie	81b513416e	Fixed line length	2018-12-19 14:12:33 -05:00
Nathan Pennie	668e6625b0	Fix the variable names used for account_data	2018-12-19 06:53:02 -05:00
Erik Johnston	df89f8afb8	Add v2 room version	2018-12-18 18:10:37 +00:00
Neil Johnson	7e22cd90f5	ensure can report mau stats when hs.config.mau_stats_only is set (#4305) ... * ensure can report mau stats when hs.config.mau_stats_only is set	2018-12-18 14:36:11 +00:00
Neil Johnson	d2f7c4e6b1	create support user (#4141) ... Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Nad Chishtie	128294d3e5	Improved welcome page (#4294) ... * Improved static/index.html look, feel and content. * Added accessibility tags to in-line SVG. * Grammar.	2018-12-14 05:13:56 +11:00
Amber Brown	c7144b105c	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:13:33 +11:00
Amber Brown	e93a0ebf50	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:10:31 +11:00
Nad Chishtie	cef8ae272a	Improved welcome page (#4294) ... * Improved static/index.html look, feel and content. * Added accessibility tags to in-line SVG. * Grammar.	2018-12-14 02:42:51 +11:00
Richard van der Hoff	f208f608cb	Merge branch 'release-v0.34.0' into develop	2018-12-11 15:43:20 +00:00
Richard van der Hoff	eb835bf65b	0.34.0rc2	2018-12-11 15:35:47 +00:00
Richard van der Hoff	188945713e	Merge pull request #4290 from matrix-org/rav/remove_webclient ... Stop installing Matrix Console by default	2018-12-11 16:24:15 +01:00
Richard van der Hoff	95c1f6500b	Merge pull request #4289 from matrix-org/rav/welcome_page ... Add a welcome page to the static resources	2018-12-11 16:24:06 +01:00
Richard van der Hoff	f537432ef9	Add a welcome page to the static resources ... This is largely a precursor for the removal of the bundled webclient. The idea is to present a page at / which reassures people that something is working, and to give them some links for next steps. The welcome page lives at `/_matrix/static/`, so is enabled alongside the other `static` resources (which, in practice, means the client API is enabled). We'll redirect to it from `/` if we have nothing better to display there. It would be nice to have a way to disable it (in the same way that you might disable the nginx welcome page), but I can't really think of a good way to do that without a load of ickiness. It's based on the work done by @krombel for #2601.	2018-12-11 13:26:22 +00:00
Richard van der Hoff	df96177ca7	Stop installing Matrix Console by default ... This is based on the work done by @krombel in #2601.	2018-12-11 13:20:33 +00:00
David Baker	89ac2a5bdb	Add 'sandbox' to CSP for media repo (#4284) ... * Add 'sandbox' to the CSP for media repo * Changelog	2018-12-11 04:05:02 +11:00
Will Hunt	fee831c040	Move imports to one line	2018-12-10 13:52:33 +00:00
Will Hunt	466c1f3e01	Use send_cors	2018-12-10 13:11:37 +00:00
Will Hunt	91206e09f2	changelog & isort	2018-12-09 17:39:44 +00:00
Will Hunt	dbf736ba66	Make /config more CORS-y	2018-12-09 13:27:22 +00:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272) ... When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	c7401a697f	Implement SAML2 authentication (#4267) ... This implements both a SAML2 metadata endpoint (at `/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at `/_matrix/saml2/authn_response`). If the SAML2 response matches what's been configured, we complete the SSO login flow by redirecting to the client url (aka `RelayState` in SAML2 jargon) with a login token. What we don't yet have is anything to build a SAML2 request and redirect the user to the identity provider. That is left as an exercise for the reader.	2018-12-07 13:11:11 +01:00
Richard van der Hoff	c588b9b9e4	Factor SSO success handling out of CAS login (#4264) ... This is mostly factoring out the post-CAS-login code to somewhere we can reuse it for other SSO flows, but it also fixes the userid mapping while we're at it.	2018-12-07 13:10:07 +01:00
rkfg	ae19a7db8c	Prevent crash on pagination.	2018-12-06 13:32:05 +03:00
Richard van der Hoff	b0c24a66ec	Rip out half-implemented m.login.saml2 support (#4265) ... * Rip out half-implemented m.login.saml2 support This was implemented in an odd way that left most of the work to the client, in a way that I really didn't understand. It's going to be a pain to maintain, so let's start by ripping it out. * drop undocumented dependency on dateutil It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 19:44:38 +11:00
Richard van der Hoff	9a3e24a13d	drop undocumented dependency on dateutil (#4266) ... It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 04:52:42 +11:00
Richard van der Hoff	e8d98466b0	Implement .well-known handling (#4262) ... Sometimes it's useful for synapse to generate its own .well-known file.	2018-12-05 14:38:58 +01:00
Richard van der Hoff	e55983defe	Prepare 0.34.0rc1	2018-12-04 13:52:16 +00:00
Travis Ralston	1737753a62	Add an option to enable recording IPs for appservice users (#3831)	2018-12-04 12:44:41 +01:00
Amber Brown	fd96dd75a3	Fix non-ASCII pushrules (#4248)	2018-12-04 12:44:02 +01:00
Travis Ralston	158ffb92f1	Add an option to disable search for homeservers which may not be interested in it (#4230) ... This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.	2018-12-04 12:01:02 +01:00
Ben Parsons	b5ac0ffa0a	add more detail to logging regarding "More than one row matched" error (#4234)	2018-12-04 11:57:39 +01:00
Richard van der Hoff	ecc23188f4	Fix UnicodeDecodeError when postgres is not configured in english (#4253) ... This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.	2018-12-04 11:55:52 +01:00
Richard van der Hoff	f144c0a210	Merge pull request #4244 from aaronraimist/drop-sent-txt ... Drop sent_transactions	2018-12-04 11:41:28 +01:00
Richard van der Hoff	a077e710a3	Merge pull request #4250 from matrix-org/hawkowl/pusher-remove-py3 ... Fix removing pushers on python 3	2018-12-04 11:22:46 +01:00
Richard van der Hoff	52e87fbfbe	Run the AS senders as background processes (#4189) ... This should fix some "Starting db connection from sentinel context" warnings, and will mean we get metrics for these processes.	2018-12-04 10:53:49 +01:00
Amber Brown	d3c61ef906	fix type error	2018-12-03 22:27:41 +11:00
Richard van der Hoff	c03324294d	Workaround for non-ascii event ids (#4241) ... It turns out that we accept events with non-ascii IDs, which would later cause an explosion during state res. Fixes #4226	2018-12-03 21:47:48 +11:00
Aaron Raimist	704c5298f0	Drop sent_transactions ... Signed-off-by: Aaron Raimist <aaron@raim.ist>	2018-12-01 23:07:35 -06:00
Neil Johnson	7039ece8fb	Neilj/fix autojoin (#4223) ... * Fix auto join failures for servers that require user consent * Fix auto join failures for servers that require user consent	2018-11-28 22:24:57 +11:00
Neil Johnson	f9b136a886	Neilj/fix mau initial reserved users (#4211) ... * fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail * towncrier * white space	2018-11-28 20:33:41 +11:00
Richard van der Hoff	944d524f18	Support m.login.sso (#4220) ... * Clean up the CSS for the fallback login form I was finding this hard to work with, so simplify a bunch of things. Each flow is now a form inside a div of class login_flow. The login_flow class now has a fixed width, as that looks much better than each flow having a differnt width. * Support m.login.sso MSC1721 renames m.login.cas to m.login.sso. This implements the change (retaining support for m.login.cas for older clients). * changelog	2018-11-27 18:51:52 +11:00
Richard van der Hoff	6c18cc4b50	Ignore __pycache__ directories in schema delta dir ... Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.	2018-11-20 22:52:34 +00:00
Neil Johnson	78ba0e7ab8	Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207)	2018-11-20 12:29:25 +01:00
Richard van der Hoff	416c671474	Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes ... Fix some logcontext leaks	2018-11-20 12:19:19 +01:00
Amber Brown	31425d82a3	Merge remote-tracking branch 'origin/master' into develop	2018-11-19 12:55:25 -06:00
Amber Brown	d102e19e47	version	2018-11-19 12:42:49 -06:00
Amber Brown	80cac86b2c	Fix fallback auth on Python 3 (#4197)	2018-11-19 12:27:33 -06:00
Richard van der Hoff	f5faf6bc14	Fix logcontext leak in EmailPusher	2018-11-19 17:07:01 +00:00
Travis Ralston	ab4526a153	Remove duplicate slashes in generated consent URLs	2018-11-15 20:41:53 -07:00
Amber Brown	8b1affe7d5	Fix Content-Disposition in media repository (#4176)	2018-11-15 15:55:58 -06:00
Travis Ralston	835779f7fb	Add option to track MAU stats (but not limit people) (#3830)	2018-11-15 18:08:27 +00:00
Amber Brown	df758e155d	Use <meta> tags to discover the per-page encoding of html previews (#4183)	2018-11-15 11:05:08 -06:00
Neil Johnson	bf648c37e7	release 0.33.9rc1	2018-11-14 11:45:52 +00:00
Richard van der Hoff	c1efcd7c6a	Add a test for the public T&Cs form	2018-11-14 10:46:27 +00:00
Richard van der Hoff	83a5f459aa	Fix an internal server error when viewing the public privacy policy	2018-11-14 10:21:07 +00:00
David Baker	0869566ad3	Merge pull request #4113 from matrix-org/dbkr/e2e_backup_versions_are_numbers ... Make e2e backup versions numeric in the DB	2018-11-14 07:55:48 +00:00
David Baker	bca3b91c2d	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers	2018-11-09 18:35:02 +00:00
Erik Johnston	db5a1c059a	Merge pull request #4166 from matrix-org/erikj/drop_unknown_events ... Drop incoming events from federation for unknown rooms	2018-11-09 17:59:34 +00:00
Erik Johnston	dc59ad5334	Remove hack to support rejoining rooms	2018-11-09 14:58:09 +00:00
David Baker	d44dea0223	pep8	2018-11-09 14:38:31 +00:00
David Baker	4f93abd62d	add docs	2018-11-09 13:25:38 +00:00
Erik Johnston	30dd27afff	Simplify to always drop events if server isn't in the room	2018-11-09 11:36:45 +00:00
Richard van der Hoff	3cecf5340d	Update synapse/federation/federation_server.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2018-11-09 11:28:25 +00:00
Richard van der Hoff	9bce065a53	Update synapse/federation/federation_server.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2018-11-09 11:28:22 +00:00
David Baker	d3fa6194f7	Remove unnecessary str()	2018-11-09 11:11:31 +00:00
Erik Johnston	7b22421a7b	Merge pull request #4164 from matrix-org/erikj/fix_device_comparison ... Fix noop checks when updating device keys	2018-11-08 14:37:20 +00:00
Erik Johnston	5ebed18692	Lets convert bytes to unicode instead	2018-11-08 12:33:13 +00:00
Erik Johnston	b1a22b24ab	Fix noop checks when updating device keys ... Clients often reupload their device keys (for some reason) so its important for the server to check for no-ops before sending out device list update notifications. The check is broken in python 3 due to the fact comparing bytes and unicode always fails, and that we write bytes to the DB but get unicode when we read.	2018-11-08 12:18:38 +00:00
Erik Johnston	9417986f77	Drop PDUs of unknown rooms ... When we receive events over federation we will need to know the room version to be able to correctly handle them, e.g. once we start changing event formats. Currently, we attempt to handle events in unknown rooms.	2018-11-08 12:11:20 +00:00
Richard van der Hoff	0a1fc52971	fix parse_string docstring	2018-11-08 11:12:29 +00:00
hera	2b075fb03a	Fix encoding error for consent form on python3 ... The form was rendering this as "b'01234....'". -- richvdh	2018-11-08 11:05:39 +00:00
Amber Brown	b3708830b8	Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157)	2018-11-08 01:37:43 +11:00
Travis Ralston	0f5e51f726	Add config variables for enabling terms auth and the policy name (#4142) ... So people can still collect consent the old way if they want to.	2018-11-06 10:32:34 +00:00
Hubert Chathi	f1087106cf	handle empty backups according to latest spec proposal (#4123) ... fixes #4056	2018-11-05 17:59:29 -05:00
Amber Brown	efdcbbe46b	Tests for user consent resource (#4140)	2018-11-06 05:53:44 +11:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139) ... * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Amber Brown	0467384d2f	Set the encoding to UTF8 in the default logconfig (#4138)	2018-11-03 02:28:07 +11:00
Erik Johnston	90d713b8c6	Merge pull request #4137 from matrix-org/erikj/clean_up_events ... Clean up event accesses and tests	2018-11-02 14:12:49 +00:00
Erik Johnston	b86d05a279	Clean up event accesses and tests ... This is in preparation to refactor FrozenEvent to support different event formats for different room versions	2018-11-02 13:44:14 +00:00
Amber Brown	cb7a6b2379	Fix typing being reset causing infinite syncs (#4127)	2018-11-03 00:19:23 +11:00
Richard van der Hoff	efb9343c8c	Merge pull request #4132 from matrix-org/rav/fix_device_list_locking ... Fix locked upsert on device_lists_remote_cache	2018-11-02 10:50:53 +00:00
Richard van der Hoff	00f12e00f8	Merge pull request #4133 from matrix-org/travis/fix-terms-auth ... Fix logic error that prevented guests from seeing the privacy policy	2018-11-02 10:50:43 +00:00
Erik Johnston	b199534518	Merge pull request #4135 from matrix-org/erikj/fix_state_res_none ... Fix None exception in state res v2	2018-11-02 10:45:57 +00:00
Richard van der Hoff	350f654e7b	Add unique indexes to a couple of tables ... The indexes on device_lists_remote_extremeties can be unique, and they therefore should, to ensure that the db remains consistent.	2018-11-02 10:36:13 +00:00
Richard van der Hoff	50e328d1e7	Remove redundant database locks for device list updates ... We can rely on the application-level per-user linearizer.	2018-11-02 10:36:13 +00:00
Erik Johnston	54aec35867	Fix None exception in state res v2	2018-11-02 10:29:19 +00:00
Travis Ralston	642505abc3	Fix logic error that prevented guests from seeing the privacy policy	2018-11-01 16:48:32 -06:00
Travis Ralston	c68aab1536	Merge pull request #4004 from matrix-org/travis/login-terms ... Add m.login.terms to the registration flow	2018-11-01 11:03:38 -06:00
Erik Johnston	1b21e771d0	Merge pull request #4128 from matrix-org/erikj/state_res_v2_version ... Add STATE_V2_TEST room version	2018-11-01 13:17:57 +00:00
Erik Johnston	b3dd6fa981	Add STATE_V2_TEST room version	2018-11-01 11:43:46 +00:00
Amber Brown	073d400b84	Merge branch 'master' into develop	2018-11-01 21:32:12 +11:00
Amber Brown	aa98e38896	version	2018-11-01 21:28:35 +11:00
Travis Ralston	a8d41c6aff	Include a version query string arg for the consent route	2018-10-31 13:19:28 -06:00
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00
Richard van der Hoff	1729ba1650	Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades ... Attempt to move room aliases on room upgrades	2018-10-31 17:52:18 +00:00
Richard van der Hoff	0f8591a5a8	Avoid else clause on exception for clarity	2018-10-31 15:43:57 +00:00
Richard van der Hoff	94c7fadc98	Attempt to move room aliases on room upgrades	2018-10-31 15:43:57 +00:00
Richard van der Hoff	9b827c40ca	Log some bits about event creation (#4121) ... I found these helpful in debugging my room upgrade tests.	2018-10-31 15:42:23 +00:00
Amber Brown	916efc8249	Remove fetching keys via the deprecated v1 kex method (#4120)	2018-10-31 23:14:39 +11:00
Amber Brown	f79f454485	Remove deprecated v1 key exchange endpoint (#4119)	2018-10-31 22:29:02 +11:00
Amber Brown	67c1924899	version bump	2018-10-31 06:27:05 +11:00
Amber Brown	086e1a8f3e	Fix search 500ing (#4122)	2018-10-31 06:26:46 +11:00
Amber Brown	3bade14ec0	Fix search 500ing (#4122)	2018-10-31 04:33:41 +11:00
Amber Brown	2e223a8c22	Remove the unused /pull federation API (#4118)	2018-10-31 04:24:59 +11:00
Erik Johnston	0794504bce	Merge pull request #4006 from matrix-org/erikj/purge_state_groups ... Delete unreferenced state groups during purge	2018-10-30 16:58:22 +00:00
Amber Brown	0dce9e1379	Write some tests for the email pusher (#4095)	2018-10-30 23:55:43 +11:00
David Baker	e0934acdbb	Cast to int here too	2018-10-30 11:12:23 +00:00
David Baker	12941f5f8b	Cast bacjup version to int when querying	2018-10-30 11:01:07 +00:00
David Baker	2f0f911c52	Convert version back to a string	2018-10-30 10:35:18 +00:00
David Baker	64fa557f80	Try & make it work on postgres	2018-10-30 09:51:04 +00:00
David Baker	563f9b61b1	Make e2e backup versions numeric in the DB ... We were doing max(version) which does not do what we wanted on a column of type TEXT.	2018-10-29 21:01:22 +00:00
Erik Johnston	88e5ffe6fe	Deduplicate device updates sent over replication ... We currently send several kHz of device list updates over replication occisonally, which often causes the replications streams to lag and then get dropped. A lot of those updates will actually be duplicates, since we don't send e.g. device_ids across replication, so let's deduplicate it when we pull them out of the database.	2018-10-29 17:34:34 +00:00
Erik Johnston	a163b748a5	Don't truncate command name in metrics	2018-10-29 17:34:21 +00:00
Erik Johnston	ad88460e0d	Move _find_unreferenced_groups	2018-10-29 14:24:19 +00:00
Erik Johnston	664b192a3b	Fix set operations thinko	2018-10-29 14:21:43 +00:00
Erik Johnston	f4f223aa44	Don't make temporary list	2018-10-29 14:01:49 +00:00
Erik Johnston	b2399f6281	Make SQL a bit cleaner	2018-10-29 14:01:11 +00:00
Amber Brown	4cd1c9f2ff	Delete the disused & unspecced identicon functionality (#4106)	2018-10-29 23:57:24 +11:00
Richard van der Hoff	7fbfea062e	Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar ... Remember to copy the avatar on room upgrades	2018-10-29 12:49:21 +00:00
Richard van der Hoff	56ca578f77	Merge pull request #4099 from matrix-org/rav/upgrade_odd_pls ... Better handling of odd PLs during room upgrades	2018-10-29 12:48:51 +00:00
Richard van der Hoff	bf33eed609	Merge pull request #4091 from matrix-org/rav/room_version_upgrades ... Room version upgrade support	2018-10-29 12:47:20 +00:00
Amber Brown	c4b3698a80	Make the replication logger quieter (#4108)	2018-10-29 22:59:44 +11:00
Amber Brown	2b791865c4	version bump	2018-10-29 21:52:52 +11:00
Richard van der Hoff	db24d7f15e	Better handling of odd PLs during room upgrades ... Fixes handling of rooms where we have permission to send the tombstone, but not other state. We need to (a) fail more gracefully when we can't send the PLs in the old room, and (b) not set the PLs in the new room until we are done with the other stuff.	2018-10-27 00:54:26 +01:00
Richard van der Hoff	5caf79b312	Remember to copy the avatar on room upgrades	2018-10-26 23:56:40 +01:00
Richard van der Hoff	54bbe71867	optimise state copying	2018-10-26 22:51:34 +01:00
Richard van der Hoff	193cadc988	Address review comments ... Improve comments, get old room state from the context we already have	2018-10-26 17:10:30 +01:00
Erik Johnston	03e634dad4	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-26 16:22:45 +01:00
Amber Brown	77d70a7646	Port register_new_matrix_user to Python 3 and add tests (#4085)	2018-10-26 22:05:22 +11:00
Richard van der Hoff	e6babc27d5	restrict PLs in old room	2018-10-25 19:18:25 +01:00
Richard van der Hoff	3a263bf3ae	copy state	2018-10-25 19:18:25 +01:00
Richard van der Hoff	1b9f253e20	preserve PLs	2018-10-25 19:10:24 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Richard van der Hoff	0f7d1c9906	Basic initial support for room upgrades ... Currently just creates a new, empty, room, and sends a tombstone in the old room.	2018-10-25 19:10:24 +01:00
Richard van der Hoff	e1948175ee	Allow power_level_content_override=None for _send_events_for_new_room	2018-10-25 19:10:24 +01:00
Richard van der Hoff	7f7b2cd3de	Make room_member_handler a member of RoomCreationHandler ... ... to save passing it into `_send_events_for_new_room`	2018-10-25 19:10:18 +01:00
Richard van der Hoff	871c4abfec	Factor _generate_room_id out of create_room ... we're going to need this for room upgrades.	2018-10-25 18:23:09 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Erik Johnston	c85e063302	Merge pull request #4051 from matrix-org/erikj/alias_disallow_list ... Add config option to control alias creation	2018-10-25 17:04:59 +01:00
Neil Johnson	95ad128851	Merge pull request #4081 from matrix-org/neilj/fix_mau_init ... fix race condiftion in calling initialise_reserved_users	2018-10-25 16:33:40 +01:00
Neil Johnson	fcbd488e9a	add new line	2018-10-25 16:13:43 +01:00
Erik Johnston	b94a43d5b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list	2018-10-25 15:25:31 +01:00
Erik Johnston	e5481b22aa	Use allow/deny	2018-10-25 15:25:21 +01:00
Neil Johnson	f8fe98812b	improve comments	2018-10-25 14:58:59 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Travis Ralston	54def42c19	Merge branch 'develop' into travis/login-terms	2018-10-24 13:22:59 -06:00
Neil Johnson	ea69a84bbb	fix style inconsistencies	2018-10-24 17:18:08 +01:00
Neil Johnson	663d9db8e7	commit transaction before closing	2018-10-24 17:17:30 +01:00
Neil Johnson	07126e43a4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init	2018-10-24 16:25:39 +01:00
Neil Johnson	9ec2186586	isort	2018-10-24 16:09:21 +01:00
Neil Johnson	9532caf6ef	remove trailing whiter space	2018-10-24 16:08:25 +01:00
Richard van der Hoff	94a49e0636	fix tuple ... Co-Authored-By: neilisfragile <neil@matrix.org>	2018-10-24 14:39:23 +01:00
Richard van der Hoff	3ad359e5be	Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation	2018-10-24 11:23:49 +01:00
Richard van der Hoff	7328039117	Merge pull request #4082 from matrix-org/rav/fix_pep8 ... Fix a number of flake8 errors	2018-10-24 11:23:35 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	7e07d25ed6	Allow backslashes in event field filters ... Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which meant that single backslashes were not allowed in event field filters. The intention here is to allow single-backslashes, but disallow double-backslashes.	2018-10-24 11:11:24 +01:00
Richard van der Hoff	ef771cc4c2	Fix a number of flake8 errors ... Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.	2018-10-24 10:39:03 +01:00
Erik Johnston	47a9ba435d	Use match rather than search	2018-10-24 09:54:57 +01:00
Richard van der Hoff	e0b9d5f0af	Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts ... Clean up the way logcontexts and threads work in the pushers	2018-10-24 09:53:57 +01:00
Erik Johnston	dacbeb2e03	Comment	2018-10-24 09:47:49 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Erik Johnston	cb23aa4c42	Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation ... Clean up room alias creation	2018-10-24 09:40:27 +01:00
Richard van der Hoff	c573794b22	Fix start_pushers vs _start_pushers confusion	2018-10-24 09:24:55 +01:00
Richard van der Hoff	e564306e31	sanity-check the is_processing flag ... ... and rename it, for even more sanity	2018-10-24 09:23:33 +01:00
Neil Johnson	a67d8ace9b	remove errant exception and style	2018-10-23 17:44:39 +01:00
Travis Ralston	43c3f0b02f	Merge pull request #3969 from turt2live/travis/fix-federated-group-requests ... Handle HttpResponseException more safely for federated groups	2018-10-23 10:41:04 -06:00
Travis Ralston	3e704822be	Comments help	2018-10-23 10:25:31 -06:00
Neil Johnson	329d18b39c	remove white space	2018-10-23 15:27:20 +01:00
Neil Johnson	6105c6101f	fix race condiftion in calling initialise_reserved_users	2018-10-23 15:24:58 +01:00
Richard van der Hoff	b3f6dddad2	Give some more things logcontexts (#4077)	2018-10-23 14:29:17 +01:00
Richard van der Hoff	5c445114d3	Correctly account for cpu usage by background threads (#4074) ... Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064	2018-10-23 13:12:32 +01:00
Richard van der Hoff	026cd91ac8	Run PusherPool.start as a background process ... We don't do anything with the result, so this is needed to give this code a logcontext.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	f749607c91	Make on_started synchronous too ... This brings it into line with on_new_notifications and on_new_receipts. It requires a little bit of hoop-jumping in EmailPusher to load the throttle params before the first loop.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	e7a16c6210	Remove redundant run_as_background_process() from pusherpool ... `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	c7273c11bc	Give pushers their own background logcontext ... Each pusher has its own loop which runs for as long as it has work to do. This should run in its own background thread with its own logcontext, as other similar loops elsewhere in the system do - which means that CPU usage is consistently attributed to that loop, rather than to whatever request happened to start the loop.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	5110f4e425	move get_all_pushers call down ... simplifies the interface to _start_pushers	2018-10-22 16:12:11 +01:00
Richard van der Hoff	04277d0ed8	Factor PusherPool._start_pusher out of _start_pushers ... ... and use it from start_pusher_by_id. This mostly simplifies start_pusher_by_id.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	3e8b02c939	Rename _refresh_pusher ... This is public (or at least, called from outside the class), so ought to have a better name.	2018-10-22 16:12:11 +01:00
Richard van der Hoff	7aea00069c	Merge pull request #4076 from matrix-org/rav/fix_init_logcontexts ... Run MAU queries as background processes	2018-10-22 14:46:59 +01:00
Richard van der Hoff	911db96658	Merge pull request #4073 from matrix-org/rav/require_psutil ... Make psutil an explicit dependency	2018-10-22 12:33:21 +01:00
Richard van der Hoff	a6f421e812	Run MAU queries as background processes ... Fixes #3820	2018-10-20 02:14:35 +01:00
Amber Brown	e1728dfcbe	Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068)	2018-10-20 11:16:55 +11:00
Richard van der Hoff	e5b52d0f94	Make psutil an explicit dependency ... As of #4027, we require psutil to be installed, so it should be in our dependency list. We can also remove some of the conditional import code introduced by #992. Fixes #4062.	2018-10-19 21:51:15 +01:00
Richard van der Hoff	f62c597d14	Merge pull request #4069 from matrix-org/rav/fix_email_templates_4065 ... Calculate absolute path for email templates	2018-10-19 16:33:54 +01:00
Erik Johnston	67f7b9cb50	pep8	2018-10-19 16:06:59 +01:00
Erik Johnston	056f099126	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-19 15:48:59 +01:00
Erik Johnston	47a9da28ca	Batch process handling state groups	2018-10-19 15:48:15 +01:00
Richard van der Hoff	cc325c7069	Calculate absolute path for email templates	2018-10-19 14:01:59 +01:00
Amber Brown	e404ba9aac	Fix manhole on py3 (pt 2) (#4067)	2018-10-19 22:26:00 +11:00
Amber Brown	b69216f768	Make the metrics less racy (#4061)	2018-10-19 21:45:45 +11:00
Erik Johnston	9fafdfa97d	Anchor returned regex to start and end of string	2018-10-19 10:22:45 +01:00
Erik Johnston	084046456e	Add config option to control alias creation	2018-10-19 10:22:45 +01:00
Erik Johnston	74e7617083	Clean up room alias creation	2018-10-19 10:11:56 +01:00
Amber Brown	a36b0ec195	make a bytestring	2018-10-19 09:24:00 +11:00
Travis Ralston	49a044aa5f	Merge branch 'develop' into travis/login-terms	2018-10-18 09:57:58 -06:00
Richard van der Hoff	c632bc8654	Merge branch 'master' into develop	2018-10-18 15:07:03 +01:00
Richard van der Hoff	926da4dda8	0.33.7	2018-10-18 14:57:32 +01:00
Richard van der Hoff	c7d0f34a3c	v0.33.7rc2	2018-10-17 17:40:19 +01:00
Richard van der Hoff	0fd2321629	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-17 17:35:26 +01:00
Richard van der Hoff	c8f2c19991	Put the warning blob at the top of the file	2018-10-17 16:56:22 +01:00
Richard van der Hoff	1519572961	Ship the email templates as package_data ... move the example email templates into the synapse package so that they can be used as package data, which should mean that all of the packaging mechanisms (pip, docker, debian, arch, etc) should now come with the example templates. In order to grandfather in people who relied on the templates being in the old place, check for that situation and fall back to using the defaults if the templates directory does not exist.	2018-10-17 16:46:02 +01:00
Erik Johnston	f6a0a02a62	Fix bug where we raised StopIteration in a generator ... This made python 3.7 unhappy	2018-10-17 16:10:52 +01:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046) ... Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Neil Johnson	c6584f4b5f	clean up config error logic and imports	2018-10-17 11:36:41 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	4a28d3d36f	Update event_auth table for rejected events	2018-10-16 16:28:42 +01:00
Erik Johnston	15133477ee	Fix up use of resolve_events_with_factory	2018-10-16 16:28:42 +01:00
Erik Johnston	6bd856caa2	User event.sender rather than alias event.user_id	2018-10-16 16:16:13 +01:00
Erik Johnston	e238013c44	Add v2 state res algorithm. ... We hook this up to the vdh test room version.	2018-10-16 16:16:13 +01:00
Richard van der Hoff	b8a5b0097c	Various cleanups in the federation client code (#4031) ... - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions	2018-10-16 10:44:49 +01:00
Travis Ralston	a8ed93a4b5	pep8	2018-10-15 16:10:29 -06:00
Travis Ralston	442734ff9e	Ensure the terms params are actually provided	2018-10-15 14:56:13 -06:00
Travis Ralston	762a0982aa	Python is hard	2018-10-15 14:46:09 -06:00
Travis Ralston	f293d124b6	Merge branch 'develop' into travis/login-terms	2018-10-15 14:44:32 -06:00
Amber Brown	f726f2dc6c	version bump	2018-10-15 22:21:45 +11:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Ivan Shapovalov	06bc8d2fe5	synapse/app: frontend_proxy.py: actually make workers work on py3	2018-10-14 20:08:39 +03:00
Ivan Shapovalov	fb216a22db	synapse/visibility.py: fix SyntaxError on py3.7	2018-10-14 20:08:17 +03:00
Neil Johnson	1ccafb0c5e	no need to join room if creator	2018-10-13 21:14:21 +01:00
Travis Ralston	dd99db846d	Update login terms structure for the proposed language support	2018-10-12 18:03:27 -06:00
Travis Ralston	5119818e9d	Rely on the lack of ?u to represent public access ... also general cleanup	2018-10-12 18:03:17 -06:00
Travis Ralston	7ede650956	Merge branch 'develop' into travis/login-terms	2018-10-12 16:24:07 -06:00
Travis Ralston	164f8e4843	isort	2018-10-12 15:11:59 -06:00
Travis Ralston	7bb651de6a	More sane handling of group errors and pep8	2018-10-12 14:53:30 -06:00
Travis Ralston	e3586f7c06	Merge branch 'develop' into travis/fix-federated-group-requests	2018-10-12 14:49:58 -06:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	a45f2c3a00	missed one	2018-10-12 14:33:55 +01:00
Amber Brown	381d2cfdf0	Make workers work on Py3 (#4027)	2018-10-13 00:14:08 +11:00
Erik Johnston	67a1e315cc	Fix up comments	2018-10-12 13:49:48 +01:00
David Baker	8c0ff0287a	Linting soothes the savage PEP8 monster	2018-10-12 13:47:43 +01:00
David Baker	306361b31b	Misc PR feedback bits	2018-10-12 11:48:56 +01:00
David Baker	bddfad253a	Don't mangle exceptions	2018-10-12 11:48:02 +01:00
David Baker	86ef9760a7	Split /room_keys/version into 2 servlets	2018-10-12 11:35:08 +01:00
David Baker	83e72bb2f0	PR feedback pt. 1	2018-10-12 11:26:18 +01:00
Richard van der Hoff	8ddd0f273c	Comments on get_all_new_events_stream ... just some docstrings to clarify the behaviour here	2018-10-12 09:55:41 +01:00
Erik Johnston	7e561b5c1a	Add description to counter metric	2018-10-10 11:41:15 +01:00
Erik Johnston	3cbe8331e6	Track number of non-empty sync responses instead	2018-10-10 11:23:17 +01:00
Erik Johnston	395276b405	Append _total to metric and fix up spelling	2018-10-10 09:24:39 +01:00
David Baker	b8d9e108be	Fix mergefail	2018-10-09 18:04:21 +01:00
Erik Johnston	bdc27d6716	Add metric to count lazy member sync requests	2018-10-09 14:17:52 +01:00
David Baker	d3464ce708	isort	2018-10-09 10:33:59 +01:00
Erik Johnston	9eb1a79100	Merge pull request #4008 from matrix-org/erikj/log_looping_exceptions ... Log looping call exceptions	2018-10-09 10:12:10 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	f4a4dbcad1	Apparently this blank line is Very Important	2018-10-09 09:47:04 +01:00
David Baker	0c905ee015	be python3 compatible	2018-10-09 09:39:13 +01:00
Erik Johnston	6982320572	Remove unnecessary extra function call layer	2018-10-08 14:06:19 +01:00
Richard van der Hoff	495975e231	Optimisation for filter_events_for_server ... We're better off hashing just the event_id than the whole ((type, state_key), event_id) tuple - so use a dict instead of a set. Also, iteritems > items.	2018-10-08 13:46:52 +01:00
Erik Johnston	8a1817f0d2	Use errback pattern and catch async failures	2018-10-08 13:29:47 +01:00
David Baker	497444f1fd	Don't reuse backup versions ... Since we don't actually delete the keys, just mark the versions as deleted in the db rather than actually deleting them, then we won't reuse versions. Fixes https://github.com/vector-im/riot-web/issues/7448	2018-10-05 15:08:36 +01:00
Erik Johnston	f7199e8734	Log looping call exceptions ... If a looping call function errors, then it kills the loop entirely. Currently it throws away the exception logs, so we should make it actually log them. Fixes #3929	2018-10-05 11:24:12 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Richard van der Hoff	c6dbd216e6	Merge pull request #3995 from matrix-org/rav/no_deextrem_outliers ... Fix bug in forward_extremity update logic	2018-10-04 16:27:05 +01:00
Erik Johnston	4917ff5523	Add state_group index to event_to_state_groups ... This is needed to efficiently check for unreferenced state groups during purge.	2018-10-04 16:03:08 +01:00
Erik Johnston	17d585753f	Delete unreferened state groups during purge	2018-10-04 16:03:06 +01:00
Amber Brown	d86794325f	Merge branch 'master' into develop	2018-10-04 22:41:52 +10:00
Amber Brown	dd59dfc51f	full version	2018-10-04 22:37:55 +10:00
Travis Ralston	537d0b7b36	Use a flag rather than a new route for the public policy ... This also means that the template now has optional parameters, which will need to be documented somehow.	2018-10-03 17:50:11 -06:00
Travis Ralston	f9d34a763c	Auto-consent to the privacy policy if the user registered with terms	2018-10-03 17:39:45 -06:00
Travis Ralston	dfcad5fad5	Make the terms flow requried	2018-10-03 17:39:00 -06:00
Travis Ralston	3099d96dba	Flesh out the fallback auth for terms	2018-10-03 17:39:00 -06:00
Travis Ralston	149c4f1765	Supply params for terms auth stage ... As per https://github.com/matrix-org/matrix-doc/pull/1692	2018-10-03 15:57:42 -06:00
Travis Ralston	fd99787162	Incorporate Dave's work for GDPR login flows ... As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117	2018-10-03 15:57:42 -06:00
Richard van der Hoff	a59d899668	Pin to prometheus_client<0.4 to avoid renaming all of our metrics	2018-10-03 17:20:15 +01:00
Erik Johnston	8935ec5a93	Merge pull request #3999 from matrix-org/erikj/fix_3pid_invite_rejetion ... Fix handling of rejected threepid invites	2018-10-03 14:47:41 +01:00
Erik Johnston	81e2813948	Merge pull request #3996 from matrix-org/erikj/fix_bg_iteration ... Fix exception in background metrics collection	2018-10-03 14:14:38 +01:00
Erik Johnston	52e6e815be	Sanitise error messages when user doesn't have permission to invite	2018-10-03 14:13:07 +01:00
Erik Johnston	69e857853f	Fix handling of rejected threepid invites	2018-10-03 11:57:30 +01:00
Erik Johnston	495a9d06bb	Fix exception handling in fetching remote profiles	2018-10-03 11:34:30 +01:00
Erik Johnston	7c570bff74	Fix exception in background metrics collection ... We attempted to iterate through a list on a separate thread without doing the necessary copying.	2018-10-03 11:28:01 +01:00
Richard van der Hoff	9693625e55	actually exclude outliers	2018-10-03 10:19:41 +01:00
Richard van der Hoff	3e39783d5d	remove debugging	2018-10-02 23:44:14 +01:00
Richard van der Hoff	ae61ade891	Fix bug in forward_extremity update logic ... An event does not stop being a forward_extremity just because an outlier or rejected event refers to it.	2018-10-02 23:36:08 +01:00
Amber Brown	da8f82008d	version	2018-10-03 02:15:48 +10:00
Erik Johnston	7258d081a5	Fix bug when invalidating destination retry timings	2018-10-02 15:47:57 +01:00
Richard van der Hoff	2b8d28b095	Merge pull request #3960 from matrix-org/rav/fix_missing_create_event_error ... Fix error handling for missing auth_event	2018-10-02 13:57:54 +01:00
Amber Brown	7232917f12	Disable frozen dicts by default (#3987)	2018-10-02 22:53:47 +10:00
Erik Johnston	334e075dd8	Fix error when logging incomplete requests ... If a connection is lost before a request is read from Request, Twisted sets `method` (and `uri`) attributes to dummy values. These dummy values have incorrect types (i.e. they're not bytes), and so things like `__repr__` would raise an exception. To fix this we had a helper method to return the method with a consistent type.	2018-10-02 12:06:22 +01:00
Richard van der Hoff	8c41b0ca66	Merge pull request #3989 from matrix-org/rav/better_stacktraces ... Avoid reraise, to improve stacktraces	2018-10-02 10:47:38 +01:00
Erik Johnston	bc29946809	Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state ... Fix lazy loaded sync with rejected state events	2018-10-02 11:38:35 +02:00
Richard van der Hoff	8174c6725b	Avoid reraise, to improve stacktraces	2018-10-01 18:50:34 +01:00
Richard van der Hoff	5908a8f6f5	Merge branch 'develop' into rav/fix_missing_create_event_error	2018-10-01 17:16:20 +01:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors ... Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Amber Brown	6e05fd032c	Fix userconsent on Python 3 (#3938)	2018-10-02 00:11:58 +10:00
Erik Johnston	82f922b4af	Fix lazy loaded sync with rejected state events ... In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.	2018-10-01 14:19:40 +01:00
Erik Johnston	8f5c23d0cd	Merge pull request #3933 from matrix-org/erikj/destination_retry_cache ... Add a five minute cache to get_destination_retry_timings	2018-10-01 14:20:30 +02:00
Erik Johnston	4f3e3ac192	Correctly match 'dict.pop' api	2018-10-01 12:25:27 +01:00
Erik Johnston	8ea887856c	Don't update eviction metrics on explicit removal	2018-10-01 12:00:58 +01:00
Matthew Hodgson	23b6a0537f	emit room aliases event	2018-09-29 02:19:37 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Matthew Hodgson	8f646f2d04	fix UTs	2018-09-28 15:37:28 +01:00
Richard van der Hoff	3deaad2fb4	Merge pull request #3964 from matrix-org/rav/remove_localhost_checks ... remove spurious federation checks on localhost	2018-09-28 13:35:47 +01:00
Richard van der Hoff	965154d60a	Fix complete fail to do the right thing	2018-09-28 12:45:54 +01:00
Richard van der Hoff	19475cf337	Remove redundant call to start_get_pdu_cache ... I think this got forgotten in #3932. We were getting away with it because it was the last call in this function.	2018-09-28 12:01:23 +01:00
Richard van der Hoff	9c8cec5dab	Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache	2018-09-28 10:51:09 +01:00
Matthew Hodgson	07340cdaca	untested stab at autocreating autojoin rooms	2018-09-28 01:42:53 +01:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Richard van der Hoff	36c62a67c4	Merge pull request #3794 from matrix-org/erikj/faster_typing ... Improve performance of getting typing updates for replication	2018-09-27 15:14:26 +01:00
Richard van der Hoff	e1e3e77bfd	Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups ... Clarifications in FederationHandler	2018-09-27 15:06:59 +01:00
Amber Brown	6de3884e5e	Merge pull request #3965 from matrix-org/rav/notify_app_services_bg_process ... Run notify_app_services as a bg process	2018-09-27 23:40:30 +10:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 ... Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Amber Brown	b3064532d0	Run our oldest supported configuration in CI (#3952)	2018-09-27 23:21:54 +10:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) ... Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	51d33d5178	Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts ... fix #3854 MAU transaction errors	2018-09-27 12:31:37 +01:00
Richard van der Hoff	333bee27f5	Include event when resolving state for missing prevs ... If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	bd61c82bdf	Include state from remote servers in pdu handling ... If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	a215b698c4	Fix "unhashable type: 'list'" exception in federation handling ... get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	28223841e0	more comments	2018-09-27 11:31:51 +01:00
Richard van der Hoff	e3c159863d	Clarifications in FederationHandler ... * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)	2018-09-27 11:31:51 +01:00
Richard van der Hoff	92abd3d6d6	Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 ... Logging improvements	2018-09-27 11:27:46 +01:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Richard van der Hoff	ae6ad4cf41	docstrings and unittests for storage.state (#3958) ... I spent ages trying to figure out how I was going mad...	2018-09-27 11:22:25 +01:00
Travis Ralston	82fa31799c	Remove debugging statement	2018-09-26 14:01:02 -06:00
Travis Ralston	b4c3bc1734	Handle HttpResponseException more safely for federated groups	2018-09-26 13:48:04 -06:00
Travis Ralston	219606a6ed	Fix exception documentation in matrixfederationclient.py	2018-09-26 13:26:27 -06:00
Richard van der Hoff	e70b4ce069	Logging improvements ... Some logging tweaks to help with debugging incoming federation transactions	2018-09-26 17:36:14 +01:00
Richard van der Hoff	a87d419a85	Run notify_app_services as a bg process ... This ensures that its resource usage metrics get recorded somewhere rather than getting lost. (It also fixes an error when called from a nested logging context which completes before the bg process)	2018-09-26 17:00:40 +01:00
Richard van der Hoff	9453c65948	remove spurious federation checks on localhost ... There's really no point in checking for destinations called "localhost" because there is nothing stopping people creating other DNS entries which point to 127.0.0.1. The right fix for this is https://github.com/matrix-org/synapse/issues/3953. Blocking localhost, on the other hand, means that you get a surprise when trying to connect a test server on localhost to an existing server (with a 'normal' server_name).	2018-09-26 16:53:52 +01:00
Richard van der Hoff	607eec0456	fix docstring for FederationClient.get_state_for_room ... trivial fixes for docstring	2018-09-26 16:52:24 +01:00
Neil Johnson	28781b65e7	fix #3854	2018-09-26 16:16:41 +01:00
Richard van der Hoff	8afddf7afe	Fix error handling for missing auth_event ... When we were authorizing an event, if there was no `m.room.create` in its auth_events, we would raise a SynapseError with a cryptic message, which then meant that we would bail out of processing any incoming events, rather than storing a rejection for the faulty event and moving on. We should treat the absent event the same as any other auth failure, by raising an AuthError, so that the event is marked as rejected.	2018-09-26 14:40:16 +01:00
Richard van der Hoff	4e8276a34a	Merge pull request #3956 from matrix-org/rav/fix_expiring_cache_len ... Fix ExpiringCache.__len__ to be accurate	2018-09-26 13:10:13 +01:00
Richard van der Hoff	5b4028fa78	Merge branch 'rav/fix_expiring_cache_len' into erikj/destination_retry_cache	2018-09-26 12:55:53 +01:00
Richard van der Hoff	7ee94fc1ba	Log which cache is throwing exceptions	2018-09-26 12:43:08 +01:00
Amber Brown	66a1d57adb	Merge pull request #3948 from matrix-org/rav/no_symlink_synctl ... Move synctl into top dir to avoid a symlink	2018-09-26 21:41:58 +10:00
Amber Brown	c2185f14d7	Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu ... Comments and interface cleanup for on_receive_pdu	2018-09-26 21:41:26 +10:00
Erik Johnston	3baf6e1667	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch. (This appears to have been a longstanding bug, but one which has been made more of a problem by #3932 and #3933). (This was originally done by Erik as part of #3933. I'm cherry-picking it because really it's a fix in its own right)	2018-09-26 12:32:29 +01:00
Richard van der Hoff	a1cd37390f	Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache	2018-09-25 12:03:54 +01:00
Richard van der Hoff	4c3e7eeec5	Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches ... Fix some instances of ExpiringCache not expiring cache items	2018-09-25 12:02:57 +01:00
Jérémy Farnaud	6cf261930a	added "media-src: 'self'" to CSP for resources (#3578) ... Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem.	2018-09-25 11:55:02 +01:00
Richard van der Hoff	94f7befc31	Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered ... Fix spurious exceptions when client closes conncetion	2018-09-25 11:52:06 +01:00
Richard van der Hoff	c53336986d	Move synctl into top dir to avoid a symlink ... symlinks apparently break setuptools on python3 and alpine (https://bugs.python.org/issue31940), so let's stop using a symlink and just use the file directly.	2018-09-25 11:19:27 +01:00
Richard van der Hoff	a9d84f4e44	We require attrs 16.0.0 ... Ref: https://github.com/matrix-org/synapse/issues/3945	2018-09-25 10:43:39 +01:00
Matthew Hodgson	787d22ed6c	Only lazy load self-members on initial sync ... Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.	2018-09-25 00:49:26 +01:00
Amber Brown	fbe5ba25f6	Merge branch 'master' into develop	2018-09-25 03:10:01 +10:00
Amber Brown	6b6cb32297	bump version	2018-09-25 02:54:34 +10:00
Amber Brown	04eed80a73	Merge branch 'master' into develop	2018-09-24 23:42:25 +10:00
Amber Brown	e302f40e20	update version	2018-09-24 23:40:05 +10:00
Erik Johnston	19dc676d1a	Fix ExpiringCache.__len__ to be accurate ... It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch.	2018-09-21 16:25:42 +01:00
Erik Johnston	fdd1a62e8d	Add a five minute cache to get_destination_retry_timings ... Hopefully helps with #3931	2018-09-21 14:56:12 +01:00
Erik Johnston	79eded1ae4	Make ExpiringCache slightly more performant	2018-09-21 14:52:21 +01:00
Erik Johnston	8601c24287	Fix some instances of ExpiringCache not expiring cache items ... ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.	2018-09-21 14:19:46 +01:00
Erik Johnston	ad53a5497d	Merge pull request #3927 from matrix-org/erikj/handle_background_errors ... Handle exceptions thrown by background tasks	2018-09-21 09:26:30 +01:00
Matthew Hodgson	a2ddaa90f2	Always LL ourselves if we're in a room to simplify clients (#3916) ... Should fix https://github.com/vector-im/riot-web/issues/7209	2018-09-20 21:21:54 +01:00
Erik Johnston	94ae1dea3c	Add missing logger	2018-09-20 17:05:34 +01:00
Erik Johnston	9ea408441f	Handle exceptions thrown by background tasks ... Fixes #3921	2018-09-20 16:15:21 +01:00
Erik Johnston	b28a7ed503	Fix spurious exceptions when client closes conncetion ... If a HTTP handler throws an exception while processing a request we automatically write a JSON error response. If the handler had already started writing a response twisted throws an exception. We should check for this case and simple abort the connection if there was an error after the response had started being written.	2018-09-20 13:44:20 +01:00
Neil Johnson	23b53b4ef8	Merge pull request #3868 from matrix-org/neilj/fix_room_invite_mail_links ... Neilj/fix room invite mail links	2018-09-20 13:32:38 +01:00
Richard van der Hoff	703de4ec13	Comments and interface cleanup for on_receive_pdu ... Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.	2018-09-20 13:06:55 +01:00
Amber Brown	1f3f5fcf52	Fix client IPs being broken on Python 3 (#3908)	2018-09-20 20:14:34 +10:00
Erik Johnston	3fd68d533b	Merge pull request #3914 from matrix-org/erikj/remove_retry_cache ... Remove get_destination_retry_timings cache	2018-09-20 10:54:49 +01:00
Amber Brown	aeca5a5ed5	Add a regression test for logging on failed connections (#3912)	2018-09-20 16:28:18 +10:00
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904) ... Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	2018-09-19 17:28:18 +01:00
Erik Johnston	ce846bb620	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing	2018-09-19 15:08:36 +01:00
Erik Johnston	bbab6ebfd9	Fix up changelog and remove spurious comment	2018-09-19 14:45:14 +01:00
Erik Johnston	392a54128c	pep8	2018-09-19 14:37:49 +01:00
Erik Johnston	b9158ac2bf	Remove get_destination_retry_timings cache ... Currently we rely on the master to invalidate this cache promptly. However, after having moved most federation endpoints off of master this no longer happens, causing outbound fedeariont to get blackholed. Fixes #3798	2018-09-19 14:22:57 +01:00
Erik Johnston	80d2d50f47	Fixup	2018-09-19 11:19:47 +01:00
Erik Johnston	9407bcf37a	Replace custom DeferredTimeoutError with defer.TimeoutError	2018-09-19 11:07:29 +01:00
Erik Johnston	6c48aa0256	Run canceller first to allow it to generate correct error	2018-09-19 11:07:27 +01:00
Erik Johnston	a334e1cace	Update to use new timeout function everywhere. ... The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.	2018-09-19 10:39:40 +01:00
Amber Brown	47c02e6332	Merge pull request #3909 from turt2live/travis/fix-logging-1 ... Fix matrixfederationclient.py logging: Destination is a string	2018-09-19 18:14:47 +10:00
Amber Brown	3d6b24fb1b	Merge pull request #3907 from matrix-org/rav/set_sni_to_server_name ... Set SNI to the server_name, not whatever was in the SRV record	2018-09-19 17:59:33 +10:00
Amber Brown	f773ecbd61	Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout ... Bump timeout on get_missing_events request	2018-09-19 17:57:48 +10:00
Travis Ralston	35aec19f0a	Destination is a string	2018-09-18 15:29:30 -06:00
Richard van der Hoff	38ead946a9	Merge remote-tracking branch 'origin/develop' into neilj/fix_room_invite_mail_links	2018-09-18 19:02:45 +01:00
Richard van der Hoff	a219ce8726	Use directory server for room joins (#3899) ... When we do a join, always try the server we used for the alias lookup first. Fixes #2418	2018-09-18 18:27:37 +01:00
Richard van der Hoff	31c15dcb80	Refactor matrixfederationclient to fix logging (#3906) ... We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.	2018-09-18 18:17:15 +01:00
Amber Brown	c600886d47	Merge pull request #3894 from matrix-org/hs/phone_home_py_version ... Add python_version phone home stat	2018-09-19 02:40:04 +10:00
Richard van der Hoff	b3097396e7	Set SNI to the server_name, not whatever was in the SRV record ... Fixes #3843	2018-09-18 17:01:12 +01:00
Richard van der Hoff	550007cb0e	Bump timeout on get_missing_events request	2018-09-18 15:02:51 +01:00
Richard van der Hoff	286d6930b7	Merge pull request #3879 from matrix-org/matthew/fix-autojoin ... don't ratelimit autojoins	2018-09-18 13:07:01 +01:00
Richard van der Hoff	1e09a1d48a	Merge pull request #3889 from matrix-org/rav/404_on_remove_unknown_alias ... Return a 404 when deleting unknown room alias	2018-09-18 12:59:30 +01:00
Will Hunt	5baa087312	typo	2018-09-17 17:37:56 +01:00
Will Hunt	b58714789f	make pip happy?	2018-09-17 17:35:54 +01:00
Richard van der Hoff	ac80cb08fe	Fix more b'abcd' noise in metrics	2018-09-17 17:16:50 +01:00
Will Hunt	9a1cceeca9	Use a string for versions	2018-09-17 17:09:06 +01:00
Richard van der Hoff	f75b9961c6	Reinstate missing null check	2018-09-17 16:52:02 +01:00
Will Hunt	2b39494cd5	Add python_version phone home stat	2018-09-17 16:35:18 +01:00
Richard van der Hoff	f00a9d2636	Fix some b'abcd' noise in logs and metrics ... Python 3 compatibility: make sure that we decode some byte sequences before we use them to create log lines and metrics labels.	2018-09-17 16:15:42 +01:00
Amber Brown	fe88907d04	version	2018-09-17 22:33:22 +10:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias ... As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
Matthew Hodgson	d42d79e3c3	don't ratelimit autojoins	2018-09-15 22:27:41 +01:00
Erik Johnston	24efb2a70d	Fix timeout function ... Turns out deferred.cancel sometimes throws, so we do that last to ensure that we always do resolve the new deferred.	2018-09-15 11:38:39 +01:00
Erik Johnston	fcfe7a850d	Add an awful secondary timeout to fix wedged requests ... This is an attempt to mitigate #3842 by adding yet-another-timeout	2018-09-14 19:23:07 +01:00
Matthew Hodgson	024be6cf18	don't filter membership events based on history visibility (#3874) ... don't filter membership events based on history visibility as we will already have filtered the messages in the timeline, and state events are always visible. and because @erikjohnston said so.	2018-09-14 18:12:52 +01:00
Erik Johnston	3e6e94fe9f	Merge pull request #3872 from matrix-org/hawkowl/timeouts-2 ... timeouts 2: electric boogaloo	2018-09-14 16:58:44 +01:00
Amber Brown	bc9af88a2d	fix	2018-09-15 00:26:00 +10:00
Erik Johnston	d0f6c1ce21	Remove spurious comment	2018-09-14 15:12:36 +01:00
Erik Johnston	9e2f9a7b57	Measure outbound requests	2018-09-14 15:11:26 +01:00
Erik Johnston	0a81038ea0	Add in flight real time metrics for Measure blocks	2018-09-14 15:08:37 +01:00
Neil Johnson	7de1989ea2	fix link for case that config.email_riot_base_url is set	2018-09-13 22:43:50 +01:00
Amber Brown	c971aa7b9d	fix	2018-09-14 03:57:02 +10:00
Amber Brown	8f08d848f5	fix	2018-09-14 03:53:56 +10:00
Travis Ralston	f1a7264663	Fix minor typo in exception	2018-09-13 11:51:12 -06:00
Amber Brown	7c33ab76da	redact better	2018-09-14 03:45:34 +10:00
Amber Brown	63755fa4c2	we do that higher up	2018-09-14 03:21:47 +10:00
Amber Brown	73884ebac5	Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2	2018-09-14 03:11:25 +10:00
Amber Brown	7c27c4d51c	merge (#3576)	2018-09-14 03:11:11 +10:00
Amber Brown	1c3f4d9ca5	buffer?	2018-09-14 03:09:13 +10:00
David Baker	bc74925c5b	WIP e2e key backups ... Continues from uhoreg's branch This just fixed the errcode on /room_keys/version if no backup and updates the schema delta to be on the latest so it gets run	2018-09-13 17:02:59 +01:00
Erik Johnston	6c0f8d9d50	Merge pull request #3856 from matrix-org/erikj/speed_up_purge ... Make purge history slightly faster	2018-09-13 16:14:46 +01:00
Erik Johnston	ed5331a627	comment	2018-09-13 16:10:56 +01:00
Erik Johnston	89a76d1889	Fix handling of redacted events from federation ... If we receive an event that doesn't pass their content hash check (e.g. due to already being redacted) then we hit a bug which causes an exception to be raised, which then promplty stops the event (and request) from being processed. This effects all sorts of federation APIs, including joining rooms with a redacted state event.	2018-09-13 15:44:12 +01:00
Amber Brown	bfa0b759e0	Attempt to figure out what's going on with timeouts (#3857)	2018-09-14 00:15:51 +10:00
Erik Johnston	9cbd0094f0	pep8	2018-09-13 15:15:35 +01:00
Erik Johnston	9dbe38ea7d	Create indices after insertion	2018-09-13 15:05:52 +01:00
Erik Johnston	93139a1fb8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_purge	2018-09-13 12:57:09 +01:00
Erik Johnston	c857f5ef9b	Make purge history slightly faster ... Don't pull out events that are outliers and won't be deleted, as nothing should happen to them.	2018-09-13 12:48:10 +01:00
Amber Brown	b7d2fb5eb9	Remove some superfluous logging (#3855)	2018-09-13 19:59:32 +10:00
Neil Johnson	f30a303590	Merge pull request #3846 from matrix-org/neilj/expose-registered-users ... expose number of real reserved users	2018-09-12 17:14:04 +01:00
Matthew Hodgson	2ac1abbc7e	show heroes if a room has a 'deleted' name/canonical_alias (#3851)	2018-09-12 17:11:05 +01:00
Erik Johnston	fa0d464fa4	Merge pull request #3853 from matrix-org/erikj/log_outbound_each_time ... Log outbound requests when we retry	2018-09-12 16:55:40 +01:00
Matthew Hodgson	0403cf0783	argh pep8	2018-09-12 16:54:28 +01:00
Matthew Hodgson	0e200e366d	correctly log gappy sync metrics	2018-09-12 16:47:20 +01:00
Matthew Hodgson	11bfc2af1c	fix logline	2018-09-12 16:45:42 +01:00
Neil Johnson	8decd6233d	improve naming	2018-09-12 16:22:15 +01:00
Erik Johnston	8c5b84441b	Log outbound requests when we retry	2018-09-12 16:22:14 +01:00
Erik Johnston	54f8616d2c	Merge pull request #3841 from matrix-org/erikj/manhole_key_length ... Change the manhole SSH key to have more bits	2018-09-12 14:33:39 +01:00
Amber Brown	7ca097f77e	Port federation/ to py3 (#3847)	2018-09-12 23:23:32 +10:00
Neil Johnson	0ddf486724	expose number of real reserved users	2018-09-12 11:58:52 +01:00
Amber Brown	546aee7e52	Merge pull request #3835 from krombel/fix_3821 ... fix VOIP crashes under Python 3	2018-09-12 20:44:18 +10:00
Amber Brown	33716c4aea	Merge pull request #3826 from matrix-org/rav/logging_for_keyring ... add some logging for the keyring queue	2018-09-12 20:43:47 +10:00
Amber Brown	bc635026c5	Merge pull request #3824 from matrix-org/rav/fix_jwt_import ... Fix jwt import check	2018-09-12 20:41:57 +10:00
Amber Brown	02aa41809b	Port rest/ to Python 3 (#3823)	2018-09-12 20:41:31 +10:00
Amber Brown	8fd93b5eea	Port crypto/ to Python 3 (#3822)	2018-09-12 20:16:31 +10:00
Erik Johnston	4084a774a8	Timeout reading body for outbound HTTP requests	2018-09-12 10:10:20 +01:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827) ... * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Erik Johnston	9e05c8d309	Change the manhole SSH key to have more bits ... Newer versions of openssh client refuse to connect to the old key due to its length.	2018-09-11 10:42:10 +01:00
Krombel	7bc22539ff	fix VOIP crashes under Python 3 (#3821)	2018-09-10 14:30:08 +02:00
Amber Brown	9a5ea511b5	Merge pull request #3810 from matrix-org/erikj/send_tags_down_sync_on_join ... Send existing room tags down sync on join	2018-09-07 23:28:42 +10:00
Richard van der Hoff	b8ad756bd0	Fix jwt import check ... This handy code attempted to check that we could import jwt, but utterly failed to check it was the right jwt. Fixes https://github.com/matrix-org/synapse/issues/3793	2018-09-07 14:20:54 +01:00
Amber Brown	771d213ac5	Merge branch 'master' into develop	2018-09-07 21:45:38 +10:00
Amber Brown	6febd8e8f7	version	2018-09-07 21:40:57 +10:00
Richard van der Hoff	cd7ef43872	clearer logging when things fail, too	2018-09-06 23:56:47 +01:00
Richard van der Hoff	806964b5de	add some logging for the keyring queue ... why is it so damn slow?	2018-09-06 18:51:06 +01:00
Neil Johnson	c5440b2ca0	Merge pull request #3800 from matrix-org/neilj/remove-guests-from-mau-count ... guest users should not be part of mau total	2018-09-06 17:45:55 +01:00
Neil Johnson	84a750e0c3	ensure guests never enter mau list	2018-09-06 17:22:53 +01:00
Erik Johnston	f60c9e2a01	Don't send empty tags list down sync	2018-09-06 17:01:41 +01:00
Erik Johnston	7baf66ef5d	Send existing room tags down sync on join ... When a user joined a room any existing tags were not sent down the sync stream. Ordinarily this isn't a problem because the user needs to be in the room to have set tags in it, however synapse will sometimes add tags for a user to a room, e.g. for server notices, which need to come down sync.	2018-09-06 16:46:51 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Amber Brown	654324eded	Merge pull request #3805 from matrix-org/erikj/limit_transaction_pdus_edus ... Limit the number of PDUs/EDUs per fedreation transaction	2018-09-07 01:33:31 +10:00
Amber Brown	70fd75cd1d	Merge pull request #3788 from matrix-org/erikj/remove_conn_id ... Remove conn_id from repl prometheus metrics	2018-09-07 00:48:19 +10:00
Amber Brown	ee7c8bd2b5	Merge pull request #3795 from matrix-org/erikj/faster_sync_state ... User iter* during sync state calculations	2018-09-07 00:24:43 +10:00
Erik Johnston	6707a3212c	Limit the number of PDUs/EDUs per fedreation transaction	2018-09-06 15:23:55 +01:00
Amber Brown	135f3b4390	Merge pull request #3804 from matrix-org/rav/fix_openssl_dep ... bump dep on pyopenssl to 16.x	2018-09-07 00:23:39 +10:00
Amber Brown	2608ebc04c	Port handlers/ to Python 3 (#3803)	2018-09-07 00:22:23 +10:00
Erik Johnston	417e7077aa	Bump version and changelog	2018-09-06 14:08:55 +01:00
Erik Johnston	d64b24dfe6	Synapse 0.33.3.1 (2018-09-06) ... ============================= SECURITY FIXES -------------- - Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) - Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) Internal Changes ---------------- - Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAABCAAGBQJbkPLrAAoJEIofk9V1tejV7K4IAItYIX98DKN9x3FNs9Hd69Pw mZtqmuJ12YAyiVZoR7IJ5GfyctHCYUUXcqmNR1O+2/IRvezFnU6ZTcPW3OfNfnuD vlnMiK53F21T96ul3Wu47Z2wyO+WkoKeXdvlqt3Wa8HpnPbU6y5CSac2vK57ppTU DcMZKXaDae6vA7bsjryYgTRRplH6eFWZEexjbuZudbDvqkySi2zGUTs6SUesMC5B FRPgI6p6sCeQgGMJY+d1i+ZdzWkZhH5OLH2icf2MI1hnhSnRSJ/scSGm5OYDCwYy 03hOy0ZVMr8SB5/j0TjSaFXonKsmPKqxnU2g7crJmvFP4BfJrEbym0y9YhYXnVQ= =8PDU -----END PGP SIGNATURE----- Merge tag 'v0.33.3.1' into release-v0.33.4 Synapse 0.33.3.1 (2018-09-06) ============================= SECURITY FIXES -------------- - Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) - Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796)) Internal Changes ---------------- - Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802))	2018-09-06 14:08:33 +01:00
Richard van der Hoff	4f8baab0c4	Merge branch 'master' into develop	2018-09-06 13:05:22 +01:00
Richard van der Hoff	625542878d	bump dep on pyopenssl to 16.x	2018-09-06 12:53:15 +01:00
Richard van der Hoff	80189ed27c	prepare v0.33.3.1	2018-09-06 10:26:23 +01:00
Neil Johnson	61b05727fa	guest users should not be part of mau total	2018-09-05 22:30:36 +01:00
Richard van der Hoff	dfba1d843d	Merge pull request #3790 from matrix-org/rav/respect_event_format_in_filter ... Implement 'event_format' filter param in /sync	2018-09-05 16:24:14 +01:00
Erik Johnston	7419764351	User iter* during sync state calculations	2018-09-05 16:19:50 +01:00
Amber Brown	2d2828dcbc	Port http/ to Python 3 (#3771)	2018-09-06 00:10:47 +10:00
Richard van der Hoff	c127c8d042	Fix origin handling for pushed transactions ... Use the actual origin for push transactions, rather than whatever the remote server claimed.	2018-09-05 13:08:07 +01:00
Richard van der Hoff	804dd41e18	Check that signatures on events are valid ... We should check that both the sender's server, and the server which created the event_id (which may be different from whatever the remote server has told us the origin is), have signed the event.	2018-09-05 13:08:07 +01:00
Erik Johnston	5f02017aea	Improve performance of getting typing updates for replication ... Fetching the list of all new typing notifications involved iterating over all rooms and comparing their serial. Lets move to using a stream change cache, like we do for other streams.	2018-09-05 10:20:40 +01:00
Richard van der Hoff	87c18d12ee	Implement 'event_format' filter param in /sync ... This has been specced and part-implemented; let's implement it for /sync (but no other endpoints yet :/).	2018-09-04 15:20:09 +01:00
Neil Johnson	a6cf7d9d9a	Merge pull request #3789 from matrix-org/neilj/improve_threepid_error_strings ... improve human readable error messages	2018-09-04 13:16:00 +00:00
Amber Brown	7e9ced4178	version and towncrier	2018-09-04 21:12:04 +10:00
Neil Johnson	bae37cd811	improve human readable error message	2018-09-04 12:07:00 +01:00
Neil Johnson	c42f7fd7b9	improve human readable error messages	2018-09-04 12:03:17 +01:00
Erik Johnston	3e242dc149	Remove conn_id	2018-09-04 11:45:52 +01:00
Erik Johnston	b13836da7f	Remove conn_id from repl prometheus metrics ... `conn_id` gets set to a random string, and so we end up filling up prometheus with tonnes of data series, which is bad.	2018-09-03 17:22:49 +01:00
Erik Johnston	567363e497	Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func ... Remove unnecessary resolve_events_with_state_map	2018-09-03 16:19:41 +01:00
Neil Johnson	301cb60d0b	assert rather than warn	2018-08-31 17:29:35 +01:00
Neil Johnson	0b01281e77	move threepid checker to config, add missing yields	2018-08-31 17:11:11 +01:00
Neil Johnson	e8e540630e	fix reference to is_threepid_reserved	2018-08-31 16:09:15 +01:00
Neil Johnson	09f3cf1a7e	ensure post registration auth checks do not fail erroneously	2018-08-31 15:42:51 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Amber Brown	14e4d4f4bf	Port storage/ to Python 3 (#3725)	2018-08-31 00:19:58 +10:00
Richard van der Hoff	475253a88e	Merge pull request #3764 from matrix-org/rav/close_db_conn_after_init ... Make sure that we close db connections opened during init	2018-08-30 10:47:27 +01:00
Krombel	79a8a347a6	fix #3445 ... itervalues(d) calls d.itervalues() [PY2] and d.values() [PY3] but SortedDict only implements d.values()	2018-08-29 16:28:25 +02:00
Matthew Hodgson	b1580f50fe	don't return non-LL-member state in incremental sync state blocks (#3760) ... don't return non-LL-member state in incremental sync state blocks	2018-08-28 23:25:58 +01:00
Richard van der Hoff	32eb1dedd2	use abc.abstractproperty ... This gives clearer messages when someone gets it wrong	2018-08-28 17:10:43 +01:00
Richard van der Hoff	0b07f02e19	Make sure that we close db connections opened during init ... We should explicitly close any db connections we open, because failing to do so can block other transactions as per https://github.com/matrix-org/synapse/issues/3682. Let's also try to factor out some of the boilerplate by having server classes define their datastore class rather than duplicating the whole of `setup`.	2018-08-28 13:39:49 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Erik Johnston	9db2476991	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact	2018-08-24 17:00:37 +01:00
Erik Johnston	05077e06fa	Change admin_uri to admin_contact in config and errors	2018-08-24 16:51:27 +01:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	01a5a8b9e3	Fix checking if service notice room is already tagged ... This manifested in synapse repeatedly setting the tag for the room	2018-08-24 16:22:37 +01:00
Erik Johnston	cdd24449ee	Ensure we wake up /sync when we add tag to notice room	2018-08-24 14:50:03 +01:00
Erik Johnston	14d49c51db	Make content of tag an empty object rather than null	2018-08-24 14:44:16 +01:00
Erik Johnston	84b4e76fed	Merge pull request #3754 from matrix-org/erikj/fix_whitelist ... Allow federation_domain_whitelist to be emtpy list	2018-08-24 12:23:39 +01:00
Erik Johnston	1d67b13674	Fix bug when federation_domain_whitelist is an emtpy list ... Outbound federation were incorrectly allowed when the config option was set to an empty list	2018-08-24 12:13:12 +01:00
Erik Johnston	92d50e3c2a	Merge pull request #3753 from matrix-org/erikj/fix_no_server_noticse ... Fix bug where we broke sync when using limit_usage_by_mau	2018-08-24 11:56:08 +01:00
Richard van der Hoff	e94cdbaecf	Merge pull request #3751 from matrix-org/rav/twisted_17 ... Pin to twisted 17.1 or later	2018-08-24 11:55:52 +01:00
Erik Johnston	7f147d623b	Fix bug where we broke sync when using limit_usage_by_mau ... We assumed that we always had service notices configured, but that is not always true	2018-08-24 11:33:50 +01:00
Richard van der Hoff	f584d6108f	Pin to twisted 17.1 or later ... Fixes https://github.com/matrix-org/synapse/issues/3741.	2018-08-24 10:02:31 +01:00
Erik Johnston	cd77270a66	Implement trail users	2018-08-23 19:17:19 +01:00
Erik Johnston	7e6e588e60	Fix bug where we resent "limit exceeded" server notices ... This was due to a bug where we mutated a cached event's contents	2018-08-23 16:21:20 +01:00
Erik Johnston	7a0da69eee	Add missing yield	2018-08-23 10:28:12 +01:00
Erik Johnston	fd2dbf1836	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-22 17:06:10 +01:00
Erik Johnston	9643a6f7f2	Update notice format	2018-08-22 17:00:29 +01:00
Erik Johnston	5c261107c9	Remove unnecessary resolve_events_with_state_map ... We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.	2018-08-22 15:41:15 +01:00
Richard van der Hoff	c7181dcc6c	Merge branch 'master' into develop	2018-08-22 14:37:11 +01:00
Richard van der Hoff	74854a9719	Use recaptcha_ajax.js directly from Google ... This was originally done in commit c75b71a397, but got reverted on this branch due to the PR (#3677) being based on the wrong branch. We're ready to merge this to master now, so let's make it match release-v0.33.3.	2018-08-22 14:30:49 +01:00
Richard van der Hoff	48fec67536	Features ... -------- - Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439)) - Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184)) - speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568)) - implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574)) - Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589)) - Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633)) - Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653)) - Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654)) - Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655)) - Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662)) - Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664)) - Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670)) - set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687)) - Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694)) - For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708)) Bugfixes -------- - Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658)) - Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661)) - Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676)) - Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677)) - Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681)) - Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689)) - Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692)) - Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700)) - Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701)) - Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713)) - Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710)) - Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719)) - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723)) - Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732)) Deprecations and Removals ------------------------- - The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703)) Internal Changes ---------------- - The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423)) - Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632)) - Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647)) - Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660)) - Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668)) - Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669)) - Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678)) - Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679)) - Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684)) - Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690)) - add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707)) - Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709)) - Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbfWKDAAoJEIofk9V1tejVHOMH/RmP+axF2K+Socnnb5Fa6RTo pDJXVwsvNQxcwzZIJXh2TRilVr88jx7A1PHUde0TmkHRmis/bDzZxa59QRA4rFEy fLYMx9Tyt0RzAJrEd0p7jbIkJxxPm5PbMWfCgVfzg44q0kepRW6ZVl1+kFHDrsCo ocE03WdLe2sRHSudaVtBLOVMA1lDE22v40/7xiu9IbUOnF+WZ3cjXYX+SNr5eb6g 7YGbXOk4/E8XoUnpzH5oEnNtiDK5piRrr7o1282kecY20ZMNzKgseBHU6nYMv5ZO TUpCPwbukmzeGHITupY1RkzpmBbEIJTCnmLP8fiUw+RWznYyFC/aRFMiWTzle3A= =ngmn -----END PGP SIGNATURE----- Merge tag 'v0.33.3' Features -------- - Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439)) - Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184)) - speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568)) - implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574)) - Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589)) - Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633)) - Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653)) - Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654)) - Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655)) - Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662)) - Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664)) - Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670)) - set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687)) - Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694)) - For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708)) Bugfixes -------- - Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658)) - Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661)) - Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676)) - Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677)) - Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681)) - Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689)) - Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692)) - Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700)) - Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701)) - Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713)) - Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710)) - Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719)) - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723)) - Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732)) Deprecations and Removals ------------------------- - The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703)) Internal Changes ---------------- - The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423)) - Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632)) - Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647)) - Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660)) - Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668)) - Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669)) - Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678)) - Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679)) - Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684)) - Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690)) - add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707)) - Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709)) - Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))	2018-08-22 14:28:55 +01:00
Richard van der Hoff	4e5a4549b6	bump version to 0.33.3	2018-08-22 14:07:10 +01:00
Erik Johnston	db10f553ba	Merge pull request #3724 from Half-Shot/hs/guest-fetch-event ... Allow guests to use /rooms/:roomId/event/:eventId	2018-08-22 13:41:08 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	8432e2ebd7	Rename WorkerProfileHandler to BaseProfileHandler	2018-08-22 10:13:40 +01:00
Erik Johnston	a81f140880	Add assert to ensure handler is only run on master	2018-08-22 10:11:21 +01:00
Erik Johnston	47b25ba5f3	Remove redundant vars	2018-08-22 10:09:05 +01:00
Erik Johnston	3bf8bab8f9	Merge pull request #3673 from matrix-org/erikj/refactor_state_handler ... Refactor state module to support multiple room versions	2018-08-22 10:04:55 +01:00
Richard van der Hoff	a4cf660a32	Merge pull request #3735 from matrix-org/travis/federation-spelling ... limt -> limit	2018-08-22 09:34:21 +01:00
Richard van der Hoff	0d568ff403	Merge remote-tracking branch 'origin/release-v0.33.3' into develop	2018-08-22 09:15:44 +01:00
Matthew Hodgson	bb81e78ec6	Split the state_group_cache in two (#3726) ... Splits the state_group_cache in two. One half contains normal state events; the other contains member events. The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching. Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.	2018-08-22 00:56:37 +02:00
Richard van der Hoff	f7bf181a90	fix another consent encoding fail	2018-08-21 23:14:25 +01:00
Richard van der Hoff	f7baff6f7b	Fix 500 error from /consent form ... Fixes #3731	2018-08-21 22:47:07 +01:00
Richard van der Hoff	a52f276990	Bugfixes ... -------- - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723)) -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6 Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9 698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU Z2lL1C5ggBCkD43g3orDvehbNWCL5g== =0vEe -----END PGP SIGNATURE----- Merge tag 'v0.33.3rc2' into develop Bugfixes -------- - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))	2018-08-21 20:30:09 +01:00
Hubert Chathi	42a394caa2	allow session_data to be any JSON instead of just a string	2018-08-21 14:51:34 -04:00
Erik Johnston	46c832eaac	Merge pull request #3727 from matrix-org/erikj/dont_error_on_missing_keys ... Don't log exceptions when failing to fetch server keys	2018-08-21 17:07:20 +01:00
Erik Johnston	cd6937fb26	Fix typo	2018-08-21 16:28:10 +01:00
Hubert Chathi	8550a7e9c2	allow auth_data to be any JSON instead of a string	2018-08-21 10:38:00 -04:00
Erik Johnston	c2c153dd3b	Log more detail when we fail to authenticate request	2018-08-21 11:42:49 +01:00
Erik Johnston	808d8e06aa	Don't log exceptions when failing to fetch server keys ... Not being able to resolve or connect to remote servers is an expected error, so we shouldn't log at ERROR with stacktraces.	2018-08-21 11:19:26 +01:00
Erik Johnston	3f6762f0bb	isort	2018-08-21 09:38:38 +01:00
Amber Brown	3b5b64ac99	changelog	2018-08-21 03:48:55 +10:00
Will Hunt	f89f6b7c09	Allow guests to access /rooms/:roomId/event/:eventId	2018-08-20 18:25:54 +01:00
Richard van der Hoff	be6527325a	Fix exceptions when a connection is closed before we read the headers ... This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.	2018-08-20 18:21:10 +01:00
Richard van der Hoff	55e6bdf287	Robustness fix for logcontext filter ... Make the logcontext filter not explode if it somehow ends up with a logcontext of None, since that infinite-loops the whole logging system.	2018-08-20 18:20:07 +01:00
Erik Johnston	b01a755498	Make the in flight requests metrics thread safe	2018-08-20 17:27:52 +01:00
Erik Johnston	1058d14127	Make the in flight background process metrics thread safe	2018-08-20 17:27:24 +01:00
Amber Brown	9a2f960736	version	2018-08-21 00:00:19 +10:00
Amber Brown	324525f40c	Port over enough to get some sytests running on Python 3 (#3668)	2018-08-20 23:54:49 +10:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	8dee601054	Remove redundant room_version checks	2018-08-20 14:48:53 +01:00
Erik Johnston	e21c368b8b	Revert spurious change	2018-08-20 13:54:51 +01:00
Erik Johnston	f2a48d87df	Use get_cache_factor_for function for state_cache ... This allows the cache factor for `state_cache` to be individually specified in the enviroment	2018-08-20 13:01:46 +01:00
Erik Johnston	2aa7cc6a46	Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging ... Fix logging bug in EDU handling over replication	2018-08-20 10:51:45 +01:00
Neil Johnson	e07970165f	rename error code	2018-08-18 14:39:45 +01:00
Neil Johnson	c5171bf171	special case server_notices_mxid	2018-08-18 12:33:07 +01:00
Neil Johnson	ba1fbf7d5b	special case server_notices_mxid	2018-08-18 12:31:08 +01:00
Richard van der Hoff	3cef867cc1	Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commands ... Logcontexts for replication command handlers	2018-08-17 16:22:07 +01:00
Richard van der Hoff	c144252a8c	Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates ... Fix logcontexts for running pushers	2018-08-17 16:21:49 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	2018-08-18 01:08:45 +10:00
Amber Brown	04f5d2db62	Remove v1/register's broken shared secret functionality (#3703)	2018-08-18 00:55:01 +10:00
Erik Johnston	ab822a2d1f	Add some fixmes	2018-08-17 15:31:50 +01:00
Erik Johnston	91cdb6de08	Call UserDirectoryHandler methods directly ... Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.	2018-08-17 15:26:13 +01:00
Neil Johnson	d49b77404b	clean up, no functional changes	2018-08-17 15:21:34 +01:00
Richard van der Hoff	63260397c6	Merge pull request #3701 from matrix-org/rav/use_producer_for_responses ... Use a producer to stream back responses	2018-08-17 14:58:45 +01:00
Richard van der Hoff	3f8709ffe4	Merge pull request #3700 from matrix-org/rav/wait_for_producers ... Refactor request logging code	2018-08-17 14:57:45 +01:00
Neil Johnson	3ee57bdcbb	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-17 14:34:10 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Neil Johnson	b5f638f1f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-17 14:04:15 +01:00
Neil Johnson	9fd161c6fb	Merge branch 'neilj/limit_exceeded_error' of github.com:matrix-org/synapse into neilj/limit_exceeded_error	2018-08-17 13:58:40 +01:00
Neil Johnson	0195dfbf52	server limits config docs	2018-08-17 13:58:25 +01:00
Neil Johnson	69c49d3fa3	Merge branch 'develop' into neilj/limit_exceeded_error	2018-08-17 12:44:26 +00:00
Erik Johnston	38f708a2bb	Remote profile cache should remain in master worker	2018-08-17 11:37:42 +01:00
Erik Johnston	3b2dcfff78	Fix logging bug in EDU handling over replication	2018-08-17 11:11:06 +01:00
Neil Johnson	521d369e7a	remove errant yield	2018-08-17 10:12:11 +01:00
Richard van der Hoff	0e8d78f6aa	Logcontexts for replication command handlers ... Run the handlers for replication commands as background processes. This should improve the visibility in our metrics, and reduce the number of "running db transaction from sentinel context" warnings. Ideally it means converting the things that fire off deferreds into the night into things that actually return a Deferred when they are done. I've made a bit of a stab at this, but it will probably be leaky.	2018-08-17 00:43:43 +01:00
Richard van der Hoff	66f7dc8c87	Fix logcontexts for running pushers ... First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.	2018-08-17 00:32:39 +01:00
Neil Johnson	bcfeb44afe	call reap on start up and fix under reaping bug	2018-08-16 22:55:32 +01:00
Neil Johnson	372bf073c1	block event creation and room creation on hitting resource limits	2018-08-16 21:25:16 +01:00
Neil Johnson	13ad9930c8	add new error type ResourceLimit	2018-08-16 18:02:02 +01:00
Neil Johnson	51b17ec566	flake8	2018-08-16 17:32:22 +01:00
Neil Johnson	3c1080b6e4	refactor for readability, and reuse caching for setting tags	2018-08-16 17:02:04 +01:00
Neil Johnson	eff3ae3b9a	add room tagging	2018-08-16 15:48:34 +01:00
Erik Johnston	7d5b1a60a3	Fix inbound federation on reader worker ... Inbound federation requires calculating push, which in turn relies on having access to account data.	2018-08-16 15:32:20 +01:00
Neil Johnson	bfb6c58624	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-16 15:10:16 +01:00
Neil Johnson	a675f9c556	check for room state before deciding on action	2018-08-16 14:53:35 +01:00
Will Hunt	c151b32b1d	Add GET media/v1/config (#3184)	2018-08-16 14:23:38 +01:00
Matthew Hodgson	762a758fea	lazyload aware /messages (#3589)	2018-08-16 14:22:47 +01:00
Neil Johnson	25d2b5d55f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-16 11:37:17 +01:00
Neil Johnson	df1e4f259f	WIP impl commiting to get feedback	2018-08-16 11:10:53 +01:00
Neil Johnson	c055c91655	fix case where empty string state check is evaulated as False	2018-08-16 11:10:19 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Neil Johnson	8cfad2e686	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-15 17:20:38 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus ... Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Neil Johnson	fc5d937550	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking	2018-08-15 16:31:40 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Neil Johnson	eabc5f8271	wip cut at sending resource server notices	2018-08-15 15:04:52 +01:00
Neil Johnson	c24fc9797b	add new event types	2018-08-15 15:04:30 +01:00
Richard van der Hoff	afcd655ab6	Use a producer to stream back responses ... The problem with dumping all of the json response into the Request object at once is that doing so starts the timeout for the next request to be received: so if it takes longer than 60s to stream back the response to the client, the client never gets it. The correct solution is to use a Producer; then the timeout is only started once all of the content is sent over the TCP connection.	2018-08-15 15:04:16 +01:00
Erik Johnston	dc56c47dc0	Merge pull request #3653 from matrix-org/erikj/split_federation ... Move more federation APIs to workers	2018-08-15 14:59:02 +01:00
Neil Johnson	4601129c44	Merge pull request #3687 from matrix-org/neilj/admin_email ... support admin_email config and pass through into blocking errors,	2018-08-15 13:52:25 +00:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Erik Johnston	488ffe6fdb	Use federation handler function rather than duplicate ... This involves renaming _persist_events to be a public function.	2018-08-15 14:17:18 +01:00
Erik Johnston	773db62a22	Rename slave TransactionStore to SlaveTransactionStore	2018-08-15 14:17:06 +01:00
Richard van der Hoff	eaaa2248ff	Refactor request logging code ... This commit moves a bunch of the logic for deciding when to log the receipt and completion of HTTP requests into SynapseRequest, rather than in the request handling wrappers. Advantages of this are: * we get logs for *all* requests (including OPTIONS and HEADs), rather than just those that end up hitting handlers we've remembered to decorate correctly. * when a request handler wires up a Producer (as the media stuff does currently, and as other things will do soon), we log at the point that all of the traffic has been sent to the client.	2018-08-15 13:47:52 +01:00
Neil Johnson	87a824bad1	clean up AuthError	2018-08-15 11:58:03 +01:00
Neil Johnson	c4eb97518f	Merge branch 'neilj/update_limits_error_codes' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-15 11:53:01 +01:00
Neil Johnson	55afba0fc5	update admin email to uri	2018-08-15 11:41:18 +01:00
Neil Johnson	1c5e690a6b	Merge pull request #3690 from matrix-org/neilj/change_prometheus_mau_metric_name ... combine mau metrics into one group	2018-08-15 09:53:59 +00:00
Erik Johnston	fef2e65d12	Merge pull request #3667 from matrix-org/erikj/fixup_unbind ... Don't fail requests to unbind 3pids for non supporting ID servers	2018-08-15 10:32:12 +01:00
Neil Johnson	b8429c7c81	update error codes for resource limiting	2018-08-15 10:19:48 +01:00
Neil Johnson	ab035bdeac	replace admin_email with admin_uri for greater flexibility	2018-08-15 10:16:41 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	b586b8b986	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 17:43:22 +01:00
Neil Johnson	70e48cbbb1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/change_prometheus_mau_metric_name	2018-08-14 17:40:59 +01:00
Neil Johnson	9ecbaf8ba8	adding missing yield	2018-08-14 16:55:28 +01:00
Neil Johnson	e5962f845c	pep8	2018-08-14 16:36:14 +01:00
Neil Johnson	e7d091fb86	combine mau metrics into one group	2018-08-14 16:26:55 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	2545993ce4	make comments clearer	2018-08-14 15:48:12 +01:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	c74c71128d	remove blank line	2018-08-14 15:06:24 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	bd92c8eaa7	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:56:45 +01:00
Neil Johnson	99ebaed8e6	Update register.py ... remove comments	2018-08-14 14:55:55 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Neil Johnson	e2c9fe0a6a	backout ability to pass in event type to server notices	2018-08-14 13:32:56 +01:00
Amber Brown	591bf87c6a	Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres	2018-08-14 20:56:23 +10:00
Neil Johnson	9b75c78b4d	support server notice state events for resource limits	2018-08-14 11:20:41 +01:00
Neil Johnson	63417c31e9	fix typo	2018-08-13 22:36:52 +01:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Richard van der Hoff	50bcaf1a27	Merge pull request #3677 from andrewshadura/master ... Use recaptcha_ajax.js directly from Google	2018-08-13 18:52:56 +01:00
Richard van der Hoff	34f51babc5	Revert "Use recaptcha_ajax.js directly from Google"	2018-08-13 18:48:03 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" ... This reverts commit 0d43f991a1.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	4f7064f6b5	missing import	2018-08-12 19:14:31 -04:00
Matthew Hodgson	f0cede5556	missing import	2018-08-12 19:14:31 -04:00
Matthew Hodgson	54ac18e832	use parse_string	2018-08-12 19:14:31 -04:00
Matthew Hodgson	66a4ca1d28	404 nicely if you try to interact with a missing current version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	72788cf9c1	support DELETE /version with no args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	edc427a351	flake8	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	15d513f16f	fix idiocies and so make tests pass	2018-08-12 19:14:31 -04:00
Matthew Hodgson	b5eee511c7	don't needlessly return user_id	2018-08-12 19:14:31 -04:00
Matthew Hodgson	93d174bcc4	improve docstring	2018-08-12 19:14:31 -04:00
Matthew Hodgson	5e42c45c96	switch get_current_version_info back to being get_version_info	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	14b3da63a3	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f0791b7bd	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f500cb39e	more docstring for the e2e_room_keys rest	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8d14598e90	add storage docstring; remove unused set_e2e_room_keys	2018-08-12 19:14:31 -04:00
Matthew Hodgson	ca0b052307	fix factoring out of _should_replace_room_key	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cf1e2000f6	document the API	2018-08-12 19:13:09 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Neil Johnson	31fa743567	fix sqlite/postgres incompatibility in reap_monthly_active_users	2018-08-11 22:38:34 +01:00
Neil Johnson	6c6aba76e1	implementation of server notices to alert on hitting resource limits	2018-08-10 15:12:59 +01:00
Richard van der Hoff	c31793a784	Merge branch 'rav/fix_linearizer_cancellation' into develop	2018-08-10 14:57:27 +01:00
Richard van der Hoff	c08f9d95b2	log *after* reloading log config ... ... because logging *before* reloading means the log message gets lost in the old MemoryLogger	2018-08-10 14:56:48 +01:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Andrej Shadura	c75b71a397	Use recaptcha_ajax.js directly from Google ... The script recaptcha_ajax.js contains minified non-free code which we probably cannot redistribute. Since it talks to Google servers anyway, it is better to just download it from Google directly instead of shipping it. This fixes #1932.	2018-08-10 13:36:14 +02:00
Richard van der Hoff	3c0213a217	Merge pull request #3439 from vojeroen/send_sni_for_federation_requests ... send SNI for federation requests	2018-08-10 12:23:54 +01:00
Richard van der Hoff	638d35ef08	Fix linearizer cancellation on twisted < 18.7 ... Turns out that cancellation of inlineDeferreds didn't really work properly until Twisted 18.7. This commit refactors Linearizer.queue to avoid inlineCallbacks.	2018-08-10 10:59:09 +01:00
Neil Johnson	01021c812f	wip at implementing MSC 7075	2018-08-09 22:16:00 +01:00
Jeroen	2e9c73e8ca	more generic conversion of str/bytes to unicode	2018-08-09 21:31:26 +02:00
Jeroen	64899341dc	include private functions from twisted	2018-08-09 21:04:22 +02:00
Neil Johnson	885ea9c602	rename _user_last_seen_monthly_active	2018-08-09 18:02:12 +01:00
Neil Johnson	c1f9dec92a	fix errant parenthesis	2018-08-09 17:43:26 +01:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	bb99b1f550	Add fast path in state res for zero prev events	2018-08-09 14:58:49 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Erik Johnston	152c0aa58e	Add constants for room versions	2018-08-09 14:55:47 +01:00
Erik Johnston	119451dcd1	Refactor state module ... We split out the actual state resolution algorithm to prepare for having multiple versions.	2018-08-09 14:55:47 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	3dce9050cf	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:42:02 +01:00
Neil Johnson	0ad98e38d0	Merge pull request #3655 from matrix-org/neilj/disable_hs ... Flag to disable HS without disabling federation	2018-08-09 10:41:43 +00:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Erik Johnston	a6c813761a	Docstrings	2018-08-09 10:41:08 +01:00
Erik Johnston	b179537f2a	Move clean_room_for_join to master	2018-08-09 10:37:38 +01:00
Erik Johnston	72d1902bbe	Fixup doc comments	2018-08-09 10:23:49 +01:00
Amber Brown	984376745b	Merge branch 'master' into develop	2018-08-09 19:23:47 +10:00
Amber Brown	dafe90a6be	0.33.2	2018-08-09 19:20:41 +10:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Erik Johnston	bf7598f582	Log when we 3pid/unbind request fails	2018-08-09 10:09:56 +01:00
Erik Johnston	2bdafaf3c1	Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet ... Add helper base class for generating new replication endpoints	2018-08-09 10:06:23 +01:00
Erik Johnston	62564797f5	Fixup wording and remove dead code	2018-08-09 09:56:10 +01:00
Richard van der Hoff	bb89c84614	Merge pull request #3664 from matrix-org/rav/federation_metrics ... more metrics for the federation and appservice senders	2018-08-08 23:16:58 +01:00
Jeroen	d5c0ce4cad	updated docstring for ServerContextFactory	2018-08-08 19:25:01 +02:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Neil Johnson	839a317c96	fix pep8 too many lines	2018-08-08 17:39:04 +01:00
Neil Johnson	5298d79fb5	Merge branch 'develop' into neilj/disable_hs	2018-08-08 16:13:03 +00:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	d2f3ef98ac	Merge branch 'develop' into neilj/disable_hs	2018-08-08 15:55:47 +00:00
Neil Johnson	2b5baebeba	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-08 13:47:07 +01:00
Neil Johnson	be59910b93	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users	2018-08-08 13:45:21 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Neil Johnson	312ae74746	typos	2018-08-08 13:33:16 +01:00
Erik Johnston	360ba89c50	Don't fail requests to unbind 3pids for non supporting ID servers ... Older identity servers may not support the unbind 3pid request, so we shouldn't fail the requests if we received one of 400/404/501. The request still fails if we receive e.g. 500 responses, allowing clients to retry requests on transient identity server errors that otherwise do support the API. Fixes #3661	2018-08-08 12:06:18 +01:00
Erik Johnston	bebe325e6c	Rename POST param to METHOD	2018-08-08 10:36:18 +01:00
Erik Johnston	5011417632	Fixup logging and docstrings	2018-08-08 10:29:58 +01:00
Richard van der Hoff	bab94da79c	fix metric name	2018-08-07 22:11:45 +01:00
Richard van der Hoff	53bca4690b	more metrics for the federation and appservice senders	2018-08-07 19:09:48 +01:00
Neil Johnson	ef3589063a	prevent total number of reserved users being too large	2018-08-07 17:59:27 +01:00
Neil Johnson	e8eba2b4e3	implement reserved users for mau limits	2018-08-07 17:49:43 +01:00
Richard van der Hoff	3523f5432a	Don't expose default_room_version as config opt	2018-08-07 12:51:57 +01:00
Amber Brown	848431be1d	fix for rc1	2018-08-07 21:13:22 +10:00
Amber Brown	3b9662339b	version	2018-08-07 21:10:52 +10:00
Erik Johnston	cd9765805e	Allow ratelimiting on workers	2018-08-07 10:50:28 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Richard van der Hoff	ca9bc1f4fe	Fix occasional glitches in the synapse_event_persisted_position metric ... Every so often this metric glitched to a negative number. I'm assuming it was due to backfilled events.	2018-08-07 10:00:03 +01:00
Neil Johnson	a74b25faaa	WIP building out mau reserved users	2018-08-06 23:25:25 +01:00
Neil Johnson	7daa8a78c5	load mau limit threepids	2018-08-06 22:55:05 +01:00
Neil Johnson	89834d9a29	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-06 21:56:56 +01:00
Neil Johnson	e54794f5b6	Fix postgres compatibility bug	2018-08-06 21:55:54 +01:00
Neil Johnson	7bcf126b18	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-06 21:39:44 +01:00
Neil Johnson	1911c037cb	update comments to reflect new sig	2018-08-06 18:01:46 +01:00
Neil Johnson	33bd07d062	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs	2018-08-06 17:52:28 +01:00
Neil Johnson	16d78be315	make use of _simple_select_one_onecol, improved comments	2018-08-06 17:51:15 +01:00
Richard van der Hoff	19a17068f1	Check m.room.create for sane room_versions	2018-08-06 16:11:24 +01:00
Erik Johnston	96a9a29645	Pull in necessary stores in federation_reader	2018-08-06 15:23:57 +01:00
Erik Johnston	62ace05c45	Move necessary storage functions to worker classes	2018-08-06 15:23:38 +01:00
Erik Johnston	1e2bed9656	Import all functions from TransactionStore	2018-08-06 15:23:38 +01:00
Erik Johnston	a3f5bf79a0	Add EDU/query handling over replication	2018-08-06 15:23:31 +01:00
Erik Johnston	e26dbd82ef	Add replication APIs for persisting federation events	2018-08-06 15:02:28 +01:00
Erik Johnston	051a99c400	Fix isort	2018-08-06 14:29:31 +01:00
Richard van der Hoff	f900d50824	include known room versions in outgoing make_joins	2018-08-06 13:45:37 +01:00
Neil Johnson	42c6823827	disable HS from config	2018-08-04 22:07:04 +01:00
Neil Johnson	e40a510fbf	py3 fix	2018-08-03 23:19:13 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Will Hunt	16d9701892	Return M_NOT_FOUND when a profile could not be found. (#3596)	2018-08-03 19:08:05 +01:00
Neil Johnson	e10830e976	wip commit - tests failing	2018-08-03 17:55:50 +01:00
Richard van der Hoff	3777fa26aa	sanity check response from make_join	2018-08-03 16:08:32 +01:00
Richard van der Hoff	0d63d93ca8	Enforce compatibility when processing make_join requests ... Reject make_join requests from servers which do not support the room version. Also include the room version in the response.	2018-08-03 16:08:32 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Richard van der Hoff	15c1ae45e5	Docstrings for BaseFederationServlet ... ... to save me reverse-engineering this stuff again.	2018-08-03 16:08:32 +01:00
Neil Johnson	5593ff6773	fix (lots of) py3 test failures	2018-08-03 14:59:17 +01:00
Neil Johnson	950807d93a	fix caching and tests	2018-08-03 13:49:53 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Erik Johnston	cb298ff623	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet	2018-08-03 09:25:15 +01:00
Neil Johnson	c0affa7b4f	update generate_monthly_active_users, and reap_monthly_active_users	2018-08-02 23:03:01 +01:00
Neil Johnson	5e2f7b8084	typo	2018-08-02 22:47:48 +01:00
Neil Johnson	4ecb4bdac9	wip attempt at caching	2018-08-02 22:41:05 +01:00
Richard van der Hoff	1fa98495d0	Merge pull request #3639 from matrix-org/rav/refactor_error_handling ... Clean up handling of errors from outbound requests	2018-08-02 17:38:24 +01:00
Richard van der Hoff	bdae8f2e68	Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling ... Factor out exception handling in federation_client	2018-08-02 17:37:46 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Neil Johnson	9180061b49	remove unused count_monthly_users	2018-08-02 15:55:29 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	43ecfe0b10	Synapse 0.33.1 (2018-08-02) ... =========================== SECURITY FIXES -------------- - Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_) - Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7 SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc= =U/ir -----END PGP SIGNATURE----- Merge tag 'v0.33.1' Synapse 0.33.1 (2018-08-02) =========================== SECURITY FIXES -------------- - Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_) - Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_)	2018-08-02 15:40:44 +01:00
Richard van der Hoff	14a4e7d5a4	Prepare 0.33.1	2018-08-02 15:31:04 +01:00
Richard van der Hoff	0bf5ec0db7	Check room visibility for /event/ requests ... Make sure that the user has permission to view the requeseted event for /event/{eventId} and /room/{roomId}/event/{eventId} requests. Also check that the event is in the given room for /room/{roomId}/event/{eventId}, for sanity.	2018-08-02 15:03:27 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Neil Johnson	00f99f74b1	insertion into monthly_active_users	2018-08-02 13:47:19 +01:00
Neil Johnson	4a6725d9d1	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-02 11:04:18 +01:00
Neil Johnson	165e067033	Revert "change monthly_active_users table to be a single column" ... This reverts commit ec716a35b2.	2018-08-02 10:59:58 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	08281fe6b7	self.db_conn unused	2018-08-01 23:26:24 +01:00
Neil Johnson	c21d82bab3	normalise reaping query	2018-08-01 23:24:38 +01:00
Neil Johnson	ec716a35b2	change monthly_active_users table to be a single column	2018-08-01 17:54:37 +01:00
Neil Johnson	d766f26de9	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-01 17:49:41 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Richard van der Hoff	b8d7d3996b	Merge pull request #3620 from fuzzmz/return-404-room-not-found ... return 404 if room not found	2018-08-01 16:34:32 +01:00
Neil Johnson	b7f203a566	count_monthly_users is now async	2018-08-01 16:17:42 +01:00
Neil Johnson	7ff44d9215	improve clarity	2018-08-01 16:17:00 +01:00
Richard van der Hoff	01e93f48ed	Kill off MatrixCodeMessageException ... This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.	2018-08-01 16:02:46 +01:00
Richard van der Hoff	018d75a148	Refactor code for turning HttpResponseException into SynapseError ... This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call.	2018-08-01 16:02:46 +01:00
Richard van der Hoff	fa7dc889f1	Be more careful which errors we send back over the C-S API ... We really shouldn't be sending all CodeMessageExceptions back over the C-S API; it will include things like 401s which we shouldn't proxy. That means that we need to explicitly turn a few HttpResponseExceptions into SynapseErrors in the federation layer. The effect of the latter is that the matrix errcode will get passed through correctly to calling clients, which might help with some of the random M_UNKNOWN errors when trying to join rooms.	2018-08-01 16:02:38 +01:00
Richard van der Hoff	c82ccd3027	Factor out exception handling in federation_client ... Factor out the error handling from make_membership_event, send_join, and send_leave, so that it can be shared.	2018-08-01 16:01:04 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Erik Johnston	4b256b9271	_persist_auth_tree no longer returns anything	2018-08-01 13:39:07 +01:00
Neil Johnson	4e5ac901dd	clean up	2018-08-01 12:03:57 +01:00
Neil Johnson	f9f5559971	fix comment	2018-08-01 12:03:42 +01:00
Neil Johnson	0aba3d361a	count_monthly_users() async	2018-08-01 11:47:58 +01:00
Neil Johnson	2c54f1c225	remove need to plot limit_usage_by_mau	2018-08-01 11:46:59 +01:00
Richard van der Hoff	f49147d14f	Merge pull request #3634 from matrix-org/rav/wtf_is_a_replication_layer ... rename replication_layer to federation_client	2018-08-01 10:29:29 +01:00
Richard van der Hoff	cab782c17e	Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator ... Rewrite cache list decorator	2018-08-01 10:28:56 +01:00
Neil Johnson	6023cdd227	remove errant print	2018-08-01 10:27:17 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Neil Johnson	c507fa15ce	only need to loop if mau limiting is enabled	2018-08-01 10:20:42 +01:00
Travis Ralston	37be52ac34	limt -> limit	2018-07-31 16:29:09 -06:00
Serban Constantin	70af98e361	return NotFoundError if room not found ... Per the Client-Server API[0] we should return `M_NOT_FOUND` if the room isn't found instead of generic SynapseError. This ensures that /directory/list API returns 404 for room not found instead of 400. [0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid Signed-off-by: Serban Constantin <serban.constantin@gmail.com>	2018-07-31 21:47:23 +03:00
Travis Ralston	5e2ee64660	Merge pull request #3628 from turt2live/travis/goodby-pdu-failures ... Remove pdu_failures from transactions	2018-07-31 12:13:09 -06:00
Neil Johnson	6ef983ce5c	api into monthly_active_users table	2018-07-31 16:36:24 +01:00
Richard van der Hoff	bdbdceeafa	rename replication_layer to federation_client ... I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.	2018-07-31 15:44:05 +01:00
Erik Johnston	443da003bc	Use new helper base class for membership requests	2018-07-31 14:32:23 +01:00
Erik Johnston	729b672823	Use new helper base class for ReplicationSendEventRestServlet	2018-07-31 14:32:23 +01:00
Erik Johnston	d81602b75a	Add helper base class for generating new replication endpoints ... This will hopefully reduce the boiler plate required to implement new internal HTTP requests.	2018-07-31 14:32:20 +01:00
Richard van der Hoff	5de936caa1	Merge pull request #3612 from matrix-org/rav/store_heirarchy ... Make EventStore inherit from EventFederationStore	2018-07-31 13:44:04 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Richard van der Hoff	0bc9b9e397	reinstate explicit include of EventsWorkerStore	2018-07-31 13:11:04 +01:00
Travis Ralston	e908b86832	Remove pdu_failures from transactions ... The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.	2018-07-30 16:28:47 -06:00
Krombel	254e8267e2	Only import secrets when available ... secrets got introduced in python 3.6 so this class is not available in 3.5 and before. This now checks for the current running version and only tries using secrets if the version is 3.6 or above Signed-Off-By: Matthias Kesler <krombel@krombel.de>	2018-07-30 23:59:02 +02:00
Neil Johnson	21276ff846	remove errant logging	2018-07-30 22:42:12 +01:00
Neil Johnson	fef7e58ac6	actually close conn	2018-07-30 22:29:44 +01:00
Neil Johnson	9b13817e06	factor out metrics from __init__ to app/homeserver	2018-07-30 22:07:07 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Jeroen	2903e65aff	fix isort	2018-07-29 19:47:08 +02:00
Richard van der Hoff	a8cbce0ced	fix invalidation	2018-07-27 16:17:17 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	f102c05856	Rewrite cache list decorator ... Because it was complicated and annoyed me. I suspect this will be more efficient too.	2018-07-27 13:47:04 +01:00
Jeroen	8e3f75b39a	fix accidental removal of hs	2018-07-27 12:17:31 +02:00
Richard van der Hoff	b0b5566f36	Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid ... if inviter_display_name == ""||None then default to inviter MXID	2018-07-27 10:08:39 +01:00
Richard van der Hoff	7041cd872b	Merge branch 'develop' into send_sni_for_federation_requests	2018-07-27 09:17:11 +01:00
Richard van der Hoff	d42455cdc9	Merge pull request #3616 from matrix-org/travis/event_id_send_leave ... Update the send_leave path to be an event_id	2018-07-26 23:25:17 +01:00
Richard van der Hoff	65c8dee900	Merge pull request #3614 from matrix-org/rav/stop_populating_event_content ... Stop populating events.content	2018-07-26 22:54:08 +01:00
Matthew Hodgson	a75231b507	Deduplicate redundant lazy-loaded members (#3331) ... * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review	2018-07-26 22:51:30 +01:00
Richard van der Hoff	9e68b1bd2d	Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id ... Remove some redundant joins on event_edges.room_id	2018-07-26 22:31:01 +01:00
Travis Ralston	7d32f0d745	Update the send_leave path to be an event_id ... It's still not used, however the parameter is an event ID not a transaction ID.	2018-07-26 14:41:59 -06:00
Richard van der Hoff	ef9d51b081	Merge pull request #3610 from matrix-org/rav/fix_looping_calls ... Fix some looping_call calls which were broken in #3604	2018-07-26 14:55:57 +01:00
Richard van der Hoff	51d7df1915	Create the column nullable ... There's no real point in ever making the column non-nullable, and doing so breaks the sytests.	2018-07-26 14:54:04 +01:00
Richard van der Hoff	5c1d301fd9	Stop populating events.content ... This field is no longer read from, so we should stop populating it. Once we're happy that this doesn't break everything, and a rollback is unlikely, we can think about dropping the column.	2018-07-26 14:43:02 +01:00
Richard van der Hoff	bd4b25f4d0	Remove some redundant joins on event_edges.room_id ... We've long passed the point where it's possible to have the same event_id in different tables, so these join conditions are redundant: we can just join on event_id. event_edges is of non-trivial size, and the room_id column is wasteful, so let's stop reading from it. In future, we can stop writing to it, and then drop it.	2018-07-26 13:19:08 +01:00
Richard van der Hoff	1b4d73fa52	comment on event_edges	2018-07-26 12:53:51 +01:00
Richard van der Hoff	21e878ebb6	Make EventStore inherit from EventFederationStore ... (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py.	2018-07-26 12:48:51 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	1bcd0490c2	Merge pull request #2970 from matrix-org/matthew/filter_members ... Implement the lazy_load_members room state filter parameter	2018-07-26 00:03:01 +01:00
Matthew Hodgson	bc7944e6d2	switch missing_types to be a bool	2018-07-25 23:36:31 +01:00
Travis Ralston	d8e65ed7e1	Fix a minor documentation typo in on_make_leave	2018-07-25 15:44:41 -06:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Matthew Hodgson	7d9fb88617	incorporate more review.	2018-07-25 16:33:50 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Erik Johnston	3849f7f69f	Merge pull request #3603 from matrix-org/erikj/handle_outliers ... Correctly handle outliers during persist events	2018-07-25 13:24:04 +01:00
Richard van der Hoff	cee1ae1b72	Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more ... Fix another logcontext leak in _persist_events	2018-07-25 11:56:00 +01:00
Richard van der Hoff	32b30e15f2	Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error ... Fix occasional 'tuple index out of range' error	2018-07-25 11:55:45 +01:00
Richard van der Hoff	4081bd1560	Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache ... Fix updating of cached remote profiles	2018-07-25 11:54:46 +01:00
Erik Johnston	7780a7b47c	Actually fix it by adding continue	2018-07-25 11:13:20 +01:00
Richard van der Hoff	1be94440d3	Fix occasional 'tuple index out of range' error ... This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying.	2018-07-25 11:05:58 +01:00
Richard van der Hoff	07defd5fe6	Fix another logcontext leak in _persist_events ... We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from	2018-07-25 10:53:23 +01:00
Richard van der Hoff	55acd6856c	Fix updating of cached remote profiles ... _update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when it was called, would just return a generator object, without actually running any of the method body.	2018-07-25 10:34:48 +01:00
Richard van der Hoff	f59be4eb0e	Fix unit tests ... on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival.	2018-07-25 10:30:36 +01:00
Erik Johnston	a297ff2b16	Fix typo in conditional	2018-07-25 09:48:01 +01:00
Richard van der Hoff	371da42ae4	Wrap a number of things that run in the background ... This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.	2018-07-25 09:41:12 +01:00
Erik Johnston	ec56121b0d	Correctly handle outliers during persist events ... We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601	2018-07-25 09:35:02 +01:00
Matthew Hodgson	cb5c37a57c	handle the edge case for _get_some_state_from_cache where types is []	2018-07-24 20:34:45 +01:00
Michael Telatynski	87951d3891	Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid	2018-07-24 17:17:46 +01:00
Erik Johnston	8b8c4f34a3	Replace usage of get_current_toke with StreamToken.START ... This allows us to handle /context/ requests on the client_reader worker without having to pull in all the various stream handlers (e.g. precence, typing, pushers etc). The only thing the token gets used for is pagination, and that ignores everything but the room portion of the token.	2018-07-24 16:49:17 +01:00
Erik Johnston	3188973857	Pull out did_forget to worker store	2018-07-24 16:49:14 +01:00
Erik Johnston	60a1d147a7	Merge pull request #3595 from matrix-org/erikj/use_deltas ... Use deltas to calculate current state deltas	2018-07-24 15:41:18 +01:00
Erik Johnston	709c309b0e	Expand on docstring comment about return value	2018-07-24 15:12:50 +01:00
Erik Johnston	8f65ab98d2	Remove unnecessary iteritems	2018-07-24 15:08:01 +01:00
Erik Johnston	ed0dd68731	Fixup comment (and indent)	2018-07-24 14:31:38 +01:00
Erik Johnston	811ac73a42	Don't fetch state from the database unless needed	2018-07-24 14:18:23 +01:00
Erik Johnston	a79410e7b8	Have _get_new_state_after_events return delta ... If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.	2018-07-24 14:14:17 +01:00
Richard van der Hoff	81946db9cf	Merge pull request #3587 from matrix-org/rav/better_exception_logging ... Improve logging for exceptions when handling PDUs	2018-07-24 14:12:12 +01:00
Richard van der Hoff	a321f78991	Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups ... Fixes and optimisations for resolve_state_groups	2018-07-24 14:11:45 +01:00
Richard van der Hoff	93b0722c50	Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room ... Remove redundant checks on room forgottenness	2018-07-24 14:11:11 +01:00
Matthew Hodgson	454f59b7ad	Merge branch 'develop' into matthew/filter_members	2018-07-24 14:03:37 +01:00
Matthew Hodgson	1a01a5b964	clarify comment on p_ids	2018-07-24 14:03:15 +01:00
Erik Johnston	223341205e	Don't require to_delete to have event_ids	2018-07-24 14:02:40 +01:00
Matthew Hodgson	e22700c3dd	consider non-filter_type types as wildcards, thus missing from the state-group-cache	2018-07-24 13:59:07 +01:00
Matthew Hodgson	eb1d911ab7	rather than adding ll_ids, remove them from p_ids	2018-07-24 13:40:49 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	30bfed5aa5	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-24 11:46:09 +01:00
Erik Johnston	97acd385a3	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta	2018-07-24 11:32:13 +01:00
Erik Johnston	0fa73e4a63	Remove unnecessary if	2018-07-24 11:19:23 +01:00
Erik Johnston	ff5426f6b8	Speed up _calculate_state_delta	2018-07-24 10:55:11 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Erik Johnston	d436ad332c	Merge pull request #3555 from matrix-org/erikj/client_apis_move ... Make client_reader support some more read only APIs	2018-07-24 10:42:28 +01:00
Erik Johnston	536bc63a4e	Merge branch 'develop' into erikj/client_apis_move	2018-07-24 09:57:05 +01:00
Richard van der Hoff	cf2d15c6a9	another couple of logcontext leaks	2018-07-24 00:57:48 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	69fb5dbdab	fix idiocy	2018-07-24 00:04:44 +01:00
Richard van der Hoff	1938cffaea	Add some measure blocks to persist_events ... ... to help us figure out where 40% of CPU is going	2018-07-23 23:48:19 +01:00
Matthew Hodgson	efcdacad7d	handle case where types is [] on postgres correctly	2018-07-23 22:41:05 +01:00
Richard van der Hoff	ce0c18dec5	Improve logging for exceptions handling PDUs ... when we get an exception handling a federation PDU, log the whole stacktrace.	2018-07-23 22:13:19 +01:00
Richard van der Hoff	c1f80effbe	Handle delta_ids being None in _update_context_for_auth_events ... it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with https://github.com/matrix-org/synapse/issues/3364)	2018-07-23 22:06:50 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Richard van der Hoff	5c705f70c9	Fixes and optimisations for resolve_state_groups ... First of all, fix the logic which looks for identical input state groups so that we actually use them. This turned out to be most easily done by factoring the relevant code out to a separate function so that we could do an early return. Secondly, avoid building the whole `conflicted_state` dict (which was only ever used as a boolean flag). Thirdly, replace the construction of the `state` dict (which mapped from keys to events that set them), with an optimistic construction of the resolution result assuming there will be no conflicts. This should be no slower than building the old `state` dict, and: - in the conflicted case, we'll short-cut it, saving part of the work - in the unconflicted case, it saves rebuilding the resolution from the `state` dict. Finally, do a couple of s/values/itervalues/.	2018-07-23 19:10:11 +01:00
Erik Johnston	8b9f164fff	Comments	2018-07-23 17:43:01 +01:00
Erik Johnston	50c60e5fad	Only get cached state from context in persist_event ... We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.	2018-07-23 17:21:40 +01:00
Richard van der Hoff	4f5cc8e4e7	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-23 17:15:12 +01:00
Richard van der Hoff	dae6dc1e77	Remove redundant checks on room forgottenness ... Fixes #3550	2018-07-23 17:13:34 +01:00
Erik Johnston	0faa3223cd	Fix missing attributes on workers. ... This was missed during the transition from attribute to getter for getting state from context.	2018-07-23 16:28:00 +01:00
Erik Johnston	999bcf9d01	Fix EventContext when using workers ... We were: 1. Not correctly setting all attributes 2. Using defer.inlineCallbacks in a non-generator	2018-07-23 15:24:21 +01:00
Erik Johnston	4797ed000e	Update docstrings to make sense	2018-07-23 15:05:56 +01:00
Erik Johnston	0b0b24cb82	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move	2018-07-23 13:21:15 +01:00
Erik Johnston	027bc01a1b	Add support for updating state	2018-07-23 13:17:25 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Erik Johnston	440b8845b5	Make EventContext lazy load state	2018-07-23 12:56:56 +01:00
Erik Johnston	842cdece42	pep8	2018-07-23 12:31:35 +01:00
Erik Johnston	acbfdc3442	Refcator EventContext to accept state during init	2018-07-23 12:20:55 +01:00
Matthew Hodgson	9b34f3ea3a	Merge branch 'develop' into matthew/sync_deleted_devices	2018-07-23 10:03:28 +01:00
Matthew Hodgson	c1bf2b587e	add trailing comma	2018-07-23 09:56:23 +01:00
Amber Brown	3132b89f12	Make the rest of the .iterwhatever go away (#3562)	2018-07-21 15:47:18 +10:00
Erik Johnston	5c88bb722f	Move PaginationHandler to its own file	2018-07-20 15:32:23 +01:00
Erik Johnston	0ecf68aedc	Move check_in_room_or_world_readable to Auth	2018-07-20 15:30:59 +01:00
Richard van der Hoff	3d6df84658	Test and fix support for cancellation in Linearizer	2018-07-20 13:59:55 +01:00
Richard van der Hoff	4f67623674	Merge pull request #3571 from matrix-org/rav/limiter_fixes ... A set of improvements to the Limiter	2018-07-20 13:53:03 +01:00
Amber Brown	e1a237eaab	Admin API for creating new users (#3415)	2018-07-20 22:41:13 +10:00
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	2018-07-20 13:11:43 +01:00
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	2018-07-20 12:50:27 +01:00
Richard van der Hoff	d7275eecf3	Add a sleep to the Limiter to fix stack overflows. ... Fixes #3570	2018-07-20 12:37:12 +01:00
Matthew Hodgson	2f558300cc	fix thinkos; unbreak tests	2018-07-19 19:22:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Richard van der Hoff	5f3658baf5	Merge pull request #3377 from Valodim/note-affinity ... document that the affinity package is required for the cpu_affinity setting	2018-07-19 14:35:06 +01:00
Amber Brown	9a8acdc720	Merge branch 'master' into develop	2018-07-19 21:16:44 +10:00
Amber Brown	38f53399a2	0.33 final	2018-07-19 21:11:40 +10:00
Amber Brown	95ccb6e2ec	Don't spew errors because we can't save metrics (#3563)	2018-07-19 20:58:18 +10:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Matthew Hodgson	9e40834f74	yes, we do need to invalidate the device_id_exists_cache when deleting a remote device	2018-07-19 11:15:10 +01:00
Richard van der Hoff	f1a15ea206	revert 00bc979 ... ... we've fixed the things that caused the warnings, so we should reinstate the warning.	2018-07-19 11:14:20 +01:00
Richard van der Hoff	1ffb7bec20	Merge remote-tracking branch 'origin/release-v0.33.0' into develop	2018-07-19 11:12:33 +01:00
Richard van der Hoff	c754e006f4	Merge pull request #3556 from matrix-org/rav/background_processes ... Run things as background processes	2018-07-19 11:04:18 +01:00
Amber Brown	a97c845271	Move v1-only APIs into their own module & isolate deprecated ones (#3460)	2018-07-19 20:03:33 +10:00
Matthew Hodgson	c0685f67c0	spell out that include_deleted_devices requires include_all_devices	2018-07-19 10:59:02 +01:00
Richard van der Hoff	00bc979137	Disable logcontext warning ... Temporary workaround to #3518 while we release 0.33.0.	2018-07-19 10:51:15 +01:00
Erik Johnston	6f62a6ef21	Merge pull request #3554 from matrix-org/erikj/response_metrics_code ... Add response code to response timer metrics	2018-07-19 10:25:18 +01:00
Richard van der Hoff	8c69b735e3	Make Distributor run its processes as a background process ... This is more involved than it might otherwise be, because the current implementation just drops its logcontexts and runs everything in the sentinel context. It turns out that we aren't actually using a bunch of the functionality here (notably suppress_failures and the fact that Distributor.fire returns a deferred), so the easiest way to fix this is actually by simplifying a bunch of code.	2018-07-18 20:55:05 +01:00
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	2018-07-18 20:55:05 +01:00
Erik Johnston	9b596177ae	Add some room read only APIs to client_reader	2018-07-18 15:33:03 +01:00
Erik Johnston	bacdf0cbf9	Move RoomContextHandler out of Handlers ... This is in preparation for moving GET /context/ to a worker	2018-07-18 15:33:03 +01:00
Erik Johnston	8cb8df55e9	Split MessageHandler into read only and writers ... This will let us call the read only parts from workers, and so be able to move some APIs off of master, e.g. the `/state` API.	2018-07-18 15:33:03 +01:00
Richard van der Hoff	65d6a0e477	Merge pull request #3553 from matrix-org/rav/background_process_tracking ... Resource tracking for background processes	2018-07-18 14:27:50 +01:00
Erik Johnston	5bd0a47fcd	pep8	2018-07-18 14:19:00 +01:00
Erik Johnston	e45a46b6e4	Add response code to response timer metrics	2018-07-18 13:59:36 +01:00
Richard van der Hoff	dab00faa83	Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks ... Remove event re-signing hacks	2018-07-18 12:46:27 +01:00
David Baker	c91a44572e	Merge pull request #3514 from matrix-org/dbkr/turn_dont_add_defaults ... Comment dummy TURN parameters in default config	2018-07-18 11:56:18 +01:00
Richard van der Hoff	6e3fc657b4	Resource tracking for background processes ... This introduces a mechanism for tracking resource usage by background processes, along with an example of how it will be used. This will help address #3518, but more importantly will give us better insights into things which are happening but not being shown up by the request metrics. We *could* do this with Measure blocks, but: - I think having them pulled out as a completely separate metric class will make it easier to distinguish top-level processes from those which are nested. - I want to be able to report on in-flight background processes, and I don't think we want to do this for *all* Measure blocks.	2018-07-18 10:50:33 +01:00
Amber Brown	5f3d02f6eb	bump to 0.33.0rc1	2018-07-18 12:52:56 +10:00
Richard van der Hoff	79eb339c66	add a comment	2018-07-17 14:53:34 +01:00
Richard van der Hoff	d897be6a98	Fix visibility of events from erased users over federation	2018-07-17 14:02:07 +01:00
Richard van der Hoff	9c04b4abf9	Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server ... Refactor and optimze filter_events_for_server	2018-07-17 14:01:39 +01:00
Richard van der Hoff	94440ae994	fix imports	2018-07-17 11:51:26 +01:00
Amber Brown	bc006b3c9d	Refactor REST API tests to use explicit reactors (#3351)	2018-07-17 20:43:18 +10:00
Richard van der Hoff	2172a3d8cb	add a comment	2018-07-17 11:13:57 +01:00
Erik Johnston	b2aa05a8d6	Use efficient .intersection	2018-07-17 11:07:04 +01:00
Erik Johnston	547b1355d3	Fix perf regression in PR #3530 ... The get_entities_changed function was changed to return all changed entities since the given stream position, rather than only those changed from a given list of entities. This resulted in the function incorrectly returning large numbers of entities that, for example, caused large increases in database usage.	2018-07-17 10:27:51 +01:00
Amber Brown	3fe0938b76	Merge pull request #3530 from matrix-org/erikj/stream_cache ... Don't return unknown entities in get_entities_changed	2018-07-17 13:44:46 +10:00
Richard van der Hoff	09e29fb58b	Attempt to make _filter_events_for_server more efficient	2018-07-16 14:06:09 +01:00
Krombel	78a9ddcf9a	rerun isort with latest version	2018-07-16 14:23:25 +02:00
Richard van der Hoff	ea69d35651	Move filter_events_for_server out of FederationHandler ... for easier unit testing.	2018-07-16 13:06:24 +01:00
Krombel	4a27000548	check isort by travis	2018-07-16 13:57:33 +02:00
Jeroen	505530f36a	Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests ... # Conflicts: # synapse/crypto/context_factory.py	2018-07-14 20:24:46 +02:00
Amber Brown	8a4f05fefb	Fix develop because I broke it :( (#3535)	2018-07-14 09:51:00 +10:00
Amber Brown	8532953c04	Merge pull request #3534 from krombel/use_parse_and_asserts_from_servlet ... Use parse and asserts from http.servlet	2018-07-14 09:09:19 +10:00
Amber Brown	a2374b2c7f	fix sytests	2018-07-14 07:52:58 +10:00
Amber Brown	33b60c01b5	Make auth & transactions more testable (#3499)	2018-07-14 07:34:49 +10:00
Krombel	516f960ad8	add changelog	2018-07-13 22:19:19 +02:00
Krombel	3366b9c534	rename assert_params_in_request to assert_params_in_dict ... the method "assert_params_in_request" does handle dicts and not requests. A request body has to be parsed to json before this method can be used	2018-07-13 21:53:01 +02:00
Krombel	32fd6910d0	Use parse_{int,str} and assert from http.servlet ... parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable	2018-07-13 21:40:14 +02:00
Richard van der Hoff	2aba1f549c	Merge pull request #3533 from matrix-org/rav/fix_federation_ratelimite_queue ... Make FederationRateLimiter queue requests properly	2018-07-13 16:59:18 +01:00
Richard van der Hoff	33b40d0a25	Make FederationRateLimiter queue requests properly ... popitem removes the *most recent* item by default [1]. We want the oldest. Fixes #3524 [1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem	2018-07-13 16:19:40 +01:00
Erik Johnston	77b692e65d	Don't return unknown entities in get_entities_changed ... The stream cache keeps track of all entities that have changed since a particular stream position, so get_entities_changed does not need to return unknown entites when given a larger stream position. This makes it consistent with the behaviour of has_entity_changed.	2018-07-13 15:26:10 +01:00
Matthew Hodgson	ba22b6a456	typo	2018-07-13 12:03:39 +01:00
Richard van der Hoff	6dff49b8a9	Merge pull request #3521 from matrix-org/rav/optimise_stream_change_cache ... Reduce set building in get_entities_changed	2018-07-12 12:08:49 +01:00
Matthew Hodgson	12ec58301f	shift to using an explicit deleted flag on m.device_list_update EDUs ... and generally make it work.	2018-07-12 11:39:43 +01:00
Richard van der Hoff	fa5c2bc082	Reduce set building in get_entities_changed ... This line shows up as about 5% of cpu time on a synchrotron: not_known_entities = set(entities) - set(self._entity_to_key) Presumably the problem here is that _entity_to_key can be largeish, and building a set for its keys every time this function is called is slow. Here we rewrite the logic to avoid building so many sets.	2018-07-12 11:37:44 +01:00
Richard van der Hoff	482d17b58b	Merge branch 'develop' into rav/enforce_report_api	2018-07-12 09:56:28 +01:00
Erik Johnston	0456e05977	Merge pull request #3505 from matrix-org/erikj/receipts_cahce ... Use stream cache in get_linearized_receipts_for_room	2018-07-12 09:46:29 +01:00
Erik Johnston	aff1dfdf3d	Update return value docstring	2018-07-12 09:45:37 +01:00
Matthew Hodgson	5797f5542b	WIP to announce deleted devices over federation ... Previously we queued up the poke correctly when the device was deleted, but then the actual EDU wouldn't get sent, as the device was no longer known. Instead, we now send EDUs for deleted devices too if there's a poke for them.	2018-07-12 01:32:39 +01:00
David Baker	36f4fd3e1e	Comment dummy TURN parameters in default config ... This default config is parsed and used a base before the actual config is overlaid, so with these values not commented out, the code to detect when no turn params were set and refuse to generate credentials was never firing because the dummy default was always set.	2018-07-11 15:49:29 +01:00
Erik Johnston	6ccefef07a	Use 'is not None' and add comments	2018-07-10 18:12:39 +01:00
Matthew Hodgson	ea752bdd99	s/becuase/because/g	2018-07-10 17:58:18 +01:00
Erik Johnston	05f5dabc10	Use stream cache in get_linearized_receipts_for_room ... This avoids us from uncessarily hitting the database when there has been no change for the room	2018-07-10 17:22:42 +01:00
Richard van der Hoff	c3c29aa196	Attempt to include db threads in cpu usage stats (#3496) ... Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.	2018-07-10 16:12:36 +01:00
Richard van der Hoff	55370331da	Refactor logcontext resource usage tracking (#3501) ... Factor out the resource usage tracking out to a separate object, which can be passed around and copied independently of the logcontext itself.	2018-07-10 13:56:07 +01:00
Matthew Hodgson	16b10666e7	another typo	2018-07-10 12:28:42 +01:00
Matthew Hodgson	4ea391a6ae	typo (i think)	2018-07-10 12:08:09 +01:00
Richard van der Hoff	e31e5dee38	Add CPU metrics for _fetch_event_list ... add a Measure block on _fetch_event_list, in the hope that we can better measure CPU usage here.	2018-07-09 18:15:54 +01:00
Jeroen	b5e157d895	Merge branch 'develop' into send_sni_for_federation_requests ... # Conflicts: # synapse/http/endpoint.py	2018-07-09 08:51:11 +02:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Amber Brown	3060bcc8e9	version	2018-07-07 10:48:06 +10:00
Amber Brown	e845fd41c2	Correct attrs package name in requirements (#3492)	2018-07-07 10:46:59 +10:00
Richard van der Hoff	1cfc2c4790	Prepare 0.32.1 release	2018-07-06 16:50:52 +01:00
Richard van der Hoff	1464a0578a	Add explicit dependency on netaddr ... the dependencies file, causing failures on upgrade (and presumably for new installs).	2018-07-06 16:27:17 +01:00
Neil Johnson	277c561766	0.32.0 version bump, update changelog	2018-07-06 15:07:29 +01:00
Amber Brown	d196fe42a9	bump version to 0.32.0rc1	2018-07-05 20:22:35 +10:00
Richard van der Hoff	3cf3e08a97	Implementation of server_acls ... ... as described at https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.	2018-07-04 19:06:20 +01:00
Richard van der Hoff	546bc9e28b	More server_name validation ... We need to do a bit more validation when we get a server name, but don't want to be re-doing it all over the shop, so factor out a separate parse_and_validate_server_name, and do the extra validation. Also, use it to verify the server name in the config file.	2018-07-04 18:59:51 +01:00
Erik Johnston	13f7adf84b	Merge pull request #3473 from matrix-org/erikj/thread_cache ... Invalidate cache on correct thread	2018-07-04 10:11:38 +01:00
Erik Johnston	40252d13d1	Merge pull request #3474 from matrix-org/erikj/py3_auth ... Fix up auth check	2018-07-04 09:41:33 +01:00
Richard van der Hoff	a4ab491371	Merge branch 'develop' into rav/drop_re_signing_hacks	2018-07-04 07:13:38 +01:00
Richard van der Hoff	508196e08a	Reject invalid server names (#3480) ... Make sure that server_names used in auth headers are sane, and reject them with a sensible error code, before they disappear off into the depths of the system.	2018-07-03 14:36:14 +01:00
Erik Johnston	2c33b55738	Avoid relying on int vs None comparison ... Python 3 doesn't support comparing None to ints	2018-07-02 11:40:32 +01:00
Erik Johnston	cbf82dddf1	Ensure that we define sender_domain	2018-07-02 11:37:57 +01:00
Erik Johnston	3905c693c5	Invalidate cache on correct thread	2018-07-02 11:36:44 +01:00
Matthew Hodgson	fc4f8f33be	replace invalid utf8 with \ufffd	2018-07-02 11:33:02 +01:00
Matthew Hodgson	1c867f5391	a fix which doesn't NPE everywhere	2018-07-01 11:56:33 +01:00
Matthew Hodgson	f131bf8d3e	don't mix unicode strings with utf8-in-byte-strings ... otherwise we explode with: ``` Traceback (most recent call last): File /usr/lib/python2.7/logging/handlers.py, line 78, in emit logging.FileHandler.emit(self, record) File /usr/lib/python2.7/logging/__init__.py, line 950, in emit StreamHandler.emit(self, record) File /usr/lib/python2.7/logging/__init__.py, line 887, in emit self.handleError(record) File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError None, sys.stderr) File /usr/lib/python2.7/traceback.py, line 124, in print_exception _print(file, 'Traceback (most recent call last):') File /usr/lib/python2.7/traceback.py, line 13, in _print file.write(str+terminator) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write self.log.emit(self.level, format=u{log_io}, log_io=line) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit self.observer(event) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__ errorLogger = self._errorLoggerForObserver(brokenObserver) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver if obs is not observer File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__ self.log = Logger(observer=self) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__ namespace = self._namespaceFromCallingContext() File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext return currentframe(2).f_globals[__name__] File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe for x in range(n + 1): RuntimeError: maximum recursion depth exceeded while calling a Python object Logged from file site.py, line 129 File /usr/lib/python2.7/logging/__init__.py, line 859, in emit msg = self.format(record) File /usr/lib/python2.7/logging/__init__.py, line 732, in format return fmt.format(record) File /usr/lib/python2.7/logging/__init__.py, line 471, in format record.message = record.getMessage() File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage msg = msg % self.args UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128) Logged from file site.py, line 129 ``` ...where the logger apparently recurses whilst trying to log the error, hitting the maximum recursion depth and killing everything badly.	2018-07-01 05:08:58 +01:00
Erik Johnston	e3b4043800	Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking ... Check the state of prev_events a bit more thoroughly when coming over federation	2018-06-29 13:55:02 +01:00
Matthew Hodgson	e72234f6bd	fix tests	2018-06-28 20:56:07 +01:00
Matthew Hodgson	f4f1cda928	add ip_range_whitelist parameter to limit where ASes can connect from	2018-06-28 20:32:00 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Amber Brown	72d2143ea8	Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454)	2018-06-28 11:04:18 +01:00
Amber Brown	99800de63d	try and clean up	2018-06-27 11:40:27 +01:00
Amber Brown	f03a5d1a17	pep8	2018-06-27 11:38:14 +01:00
Amber Brown	a7ecf34b70	cleanups	2018-06-27 11:36:03 +01:00
Amber Brown	77078d6c8e	handle federation not telling us about prev_events	2018-06-27 11:27:32 +01:00
Jeroen	95341a8f6f	take idna implementation from twisted	2018-06-26 21:15:14 +02:00
Jeroen	b7f34ee348	allow self-signed certificates	2018-06-26 20:41:05 +02:00
Matthew Hodgson	8057489b26	Revert "Try to not use as much CPU in the StreamChangeCache"	2018-06-26 18:09:01 +01:00
Matthew Hodgson	d91efb06cf	Merge pull request #3451 from matrix-org/hawkowl/sorteddict-api ... Try to not use as much CPU in the cache	2018-06-26 17:49:55 +01:00
Amber Brown	1202508067	fixes	2018-06-26 17:29:01 +01:00
Amber Brown	bd3d329c88	fixes	2018-06-26 17:28:12 +01:00
Amber Brown	abfe4b2957	try and make loading items from the cache faster	2018-06-26 17:25:34 +01:00
David Baker	028490afd4	Fix error on deleting users pending deactivation ... Use simple_delete instead of simple_delete_one as commented	2018-06-26 10:52:52 +01:00
Matthew Hodgson	c7f6b420ae	Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api ... add GDPR erase param to deactivate API	2018-06-26 10:43:14 +01:00
Matthew Hodgson	9570aa82eb	update doc for deactivate API	2018-06-26 10:42:50 +01:00
Matthew Hodgson	1e788db430	add GDPR erase param to deactivate API	2018-06-26 10:26:54 +01:00
Amber Brown	1d62c4a127	Merge pull request #3438 from turt2live/travis/dont-print-access-tokens-in-logs ... Stop including access tokens in warnings in the log	2018-06-26 09:55:55 +01:00
Erik Johnston	d72fb9a448	Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts ... allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 20:14:18 +01:00
Erik Johnston	df48f7ef37	Actually fix it	2018-06-25 20:03:41 +01:00
Erik Johnston	a0e8a53c6d	Comment	2018-06-25 19:57:38 +01:00
Erik Johnston	7bdc5c8fa3	Fix bug with assuming wrong type	2018-06-25 19:56:02 +01:00
Erik Johnston	ea7a9c0483	Add fast path to _filter_events_for_server ... Most rooms have a trivial history visibility like "shared" or "world_readable", especially large rooms, so lets not bother getting the full membership of those rooms in that case.	2018-06-25 19:49:13 +01:00
Matthew Hodgson	0269367f18	allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 17:56:10 +01:00
Matthew Hodgson	784189b1f4	typos	2018-06-25 17:37:16 +01:00
Matthew Hodgson	6eb861b67f	typo	2018-06-25 17:37:16 +01:00
Erik Johnston	947fea67cb	Need to pass reactor to endpoint fac	2018-06-25 15:22:57 +01:00
Erik Johnston	33fdcfa957	Merge pull request #3441 from matrix-org/erikj/redo_erasure ... Fix user erasure and re-enable	2018-06-25 14:37:01 +01:00
Erik Johnston	eb50c44eaf	Add UserErasureWorkerStore to workers	2018-06-25 14:22:24 +01:00
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	2018-06-25 14:08:28 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Jeroen	07b4f88de9	formatting changes for pep8	2018-06-25 12:31:16 +02:00
Jeroen	3d605853c8	send SNI for federation requests	2018-06-24 22:38:43 +02:00
Travis Ralston	ec1e799e17	Don't print invalid access tokens in the logs ... Tokens shouldn't be appearing the logs, valid or invalid. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-24 12:17:01 -06:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Richard van der Hoff	516f884176	Merge pull request #3435 from matrix-org/rav/fix_event_push_actions_tablescan ... Fix event_push_actions tablescan when reinserting events	2018-06-22 15:57:22 +01:00
Mark Haines	9850f66abe	Deleting from event_push_actions needs to use an index	2018-06-22 15:54:48 +01:00
Erik Johnston	28ddc6cfbe	Also log number of events for serach context	2018-06-22 15:42:11 +01:00
Erik Johnston	4b4cec3989	Add some logging to search queries	2018-06-22 15:42:11 +01:00
Richard van der Hoff	200e11c5bf	Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable ... Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:18:51 +01:00
Erik Johnston	f8272813a9	Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:12:26 +01:00
Richard van der Hoff	1d7ad11747	Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation ... Make push actions rotation configurable	2018-06-22 15:07:31 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	b4a5d767a9	Merge pull request #3428 from matrix-org/erikj/persisted_pdu ... Simplify get_persisted_pdu	2018-06-22 14:47:43 +01:00
Erik Johnston	f79abda87f	Merge pull request #3427 from matrix-org/erikj/remove_filters ... remove dead filter_events_for_clients	2018-06-22 14:45:42 +01:00
Erik Johnston	75dc3ddeab	Make push actions rotation configurable	2018-06-22 14:44:37 +01:00
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	2018-06-22 11:52:07 +01:00
Richard van der Hoff	240f192523	Merge pull request #3382 from matrix-org/rav/optimise_state_groups ... Optimise state_group_cache update	2018-06-22 11:20:20 +01:00
Richard van der Hoff	70e6501913	Merge pull request #3419 from matrix-org/rav/events_per_request ... Log number of events fetched from DB	2018-06-22 11:17:56 +01:00
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	2018-06-22 10:42:28 +01:00
Amber Brown	9a685d60ae	Merge pull request #3418 from matrix-org/rav/fix_metric_desc ... Fix description of "python_gc_time" metric	2018-06-22 09:43:26 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	cbbfaa4be8	Fix description of "python_gc_time" metric	2018-06-21 10:02:42 +01:00
Amber Brown	c2eff937ac	Populate synapse_federation_client_sent_pdu_destinations:count again (#3386)	2018-06-21 09:39:58 +01:00
Amber Brown	99b77aa829	Fix tcp protocol metrics naming (#3410)	2018-06-21 09:39:27 +01:00
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	2018-06-21 06:15:03 +01:00
Richard van der Hoff	aff3d76920	Merge pull request #3416 from matrix-org/rav/restart_indicator ... Write a clear restart indicator in logs	2018-06-20 18:14:08 +01:00
Richard van der Hoff	02bfc581f8	Merge pull request #3399 from costacruise/master ... Add error code to room creation error	2018-06-20 17:26:25 +01:00
Richard van der Hoff	245d53d32a	Write a clear restart indicator in logs ... I'm fed up with never being able to find the point a server restarted in the logs.	2018-06-20 15:33:14 +01:00
Amber Brown	f6c4d74f96	Fix inflight requests metric (incorrect name & traceback) (#3413)	2018-06-20 11:18:57 +01:00
Matthew Hodgson	ccfdaf68be	spell gauge correctly	2018-06-16 07:10:34 +01:00
Richard van der Hoff	9a793f861c	Merge branch 'master' into develop	2018-06-14 16:36:01 +01:00
Richard van der Hoff	53969e1960	SECURITY UPDATE: Prevent unauthorised users from setting state events in a room ... when there is no `m.room.power_levels` event in force in the room. (PR #3397) Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJbIop9AAoJEIofk9V1tejV9lsIAJVH0l5dXROmy1KH/zt16AUA CXa6Vv4Vyo6hKad/fZ81OZVRr5ChK/TvbIJVn/SA/muCfdoIFdxhT8eo/pXzO2UW zReuLsDhAg+gSvpNus37oWj2FVsAE1HYDZ60lfaapAdZnkFit68d5DQZjO6nZHHA YUXcU3GUwj0ZYuUzFzYKMLu6uNNasNkN8h6SS2lF7Bm4JaKDW+mFMfCyJwdIVSEh BGhHoVpXdxFysD9s6Mwxqrz3KKg1Jtp7idDkk0x2S2Eh+gxyiDQQokv0oQ3+0+HG sgy5Iz2t2CkpS02/j+LOvAZljTmnD0bXu3srGR+25StsoDFP038Am3bfQwtD190= =9jsT -----END PGP SIGNATURE----- Merge tag 'v0.31.2' SECURITY UPDATE: Prevent unauthorised users from setting state events in a room when there is no `m.room.power_levels` event in force in the room. (PR #3397) Discussion around the Matrix Spec change proposal for this change can be followed at https://github.com/matrix-org/matrix-doc/issues/1304.	2018-06-14 16:35:33 +01:00
Richard van der Hoff	7e1c616452	v0.31.2	2018-06-14 16:24:32 +01:00
Richard van der Hoff	61ab08a197	Merge pull request #3397 from matrix-org/rav/adjust_auth_rules ... Adjust event auth rules when there is no PL event	2018-06-14 16:09:13 +01:00
Michael Wagner	19cd3120ec	Add error code to room creation error ... This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom	2018-06-14 14:08:40 +02:00
Richard van der Hoff	5c9afd6f80	Make default state_default 50 ... Make it so that, before there is a power-levels event in the room, you need a power level of at least 50 to send state. Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192	2018-06-14 12:38:09 +01:00
Richard van der Hoff	52423607bd	Clarify interface for event_auth ... stop pretending that it returns a boolean, which just almost gave me a heart attack.	2018-06-14 12:26:17 +01:00
Amber Brown	f116f32ace	add a last seen metric (#3396)	2018-06-14 20:26:59 +10:00
Richard van der Hoff	557b686eac	Refactor get_send_level to take a power_levels event ... it makes it easier for me to reason about	2018-06-14 11:26:27 +01:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Amber Brown	0fde1896cd	Merge pull request #3389 from turt2live/travis/name_metrics ... Use the correct flag (enable_metrics) when warning about an incorrect metrics setup	2018-06-13 23:50:10 +10:00
Michael Telatynski	94700e55fa	if inviter_display_name == ""||None then default to inviter MXID ... to prevent email invite from "None"	2018-06-13 10:31:01 +01:00
Travis Ralston	45768d1640	Use the RegistryProxy for appservices too ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-12 12:55:48 -06:00
Travis Ralston	12285a1a76	The flag is named enable_metrics, not collect_metrics ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-06-12 12:51:31 -06:00
Richard van der Hoff	b6faef2ad7	Filter out erased messages ... Redact any messges sent by erased users.	2018-06-12 09:53:18 +01:00
Richard van der Hoff	f1023ebf4b	mark accounts as erased when requested	2018-06-12 09:53:18 +01:00
Richard van der Hoff	3ff8a619f5	UserErasureStore ... to store which users have been erased	2018-06-12 09:52:22 +01:00
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	bd348f0af6	remove dead filter_events_for_clients ... This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	eb32b2ca20	Optimise state_group_cache update ... (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.	2018-06-11 22:56:11 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	d6cc369205	fix idiotic typo in state res	2018-06-11 14:43:55 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Vincent Breitmoser	b800834351	add note that the affinity package is required for the cpu_affinity setting	2018-06-09 22:50:29 +02:00
Neil Johnson	ed5a0780a4	Merge branch 'master' into develop	2018-06-08 15:47:11 +01:00
Neil Johnson	1032393dfb	Changes in synapse v0.31.1 (2018-06-08) ... ======================================= v0.31.1 fixes a security bug in the ``get_missing_events`` federation API where event visibility rules were not applied correctly. We are not aware of it being actively exploited but please upgrade asap. Bug Fixes: * Fix event filtering in get_missing_events handler (PR #3371) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsalP8ACgkQCTxDAAxP S/RADw/+NeDu0LjVpS5Uc4ElHgRBuFSm6l2i4z8rZBBlKSYnuq0Em4WMvLloi/JF iAvTOYE7OjmF+gNvmsdH1N7hc1lKdQ2gAlpvaQR/5Qz9NtOVmM3WPZxS7n5jZHvD hVSxeO9+GQOwK7rJorqrrsnWHQt0OkLHV6WThFdrgZb1JjWCUDTvw+Hei2uMX2aq y2mkMG4TLStHwMvL2qw0h+hFtXywXI796qJR73ZxbEn24YD+kOXeEVkIFi2LT0Pj cgkg7WWT32JD43/ioumLupZuhCmpRyxn4fi5gIKpXe5kiLsxOdApzNQSwmoJ+WA8 7zlrWY+0QDN4pbA5ESLitWSWAT50Ul//uM4nwmM4xEBPHdljXvKyHPsaSCeKLvT9 RT8pc41TQAqSshlXF8zgIAtStnF3oGel3EBBl1mmM9Un1ULnBFwWDZhlIm+ZZGhJ MWoAWNG7j8AQuy0BTUAUr76x7t+/cdSqDuyVl1GO1tbDh0DUWoHZGXCUKrAXnn2T SbiFigwOLvEADbvkW7L9Je9CVOi2V5Pg/32X9O8YMEiSz+j5PQEiGefyVh/I/QvV Ha/atRpZF2OZ+XUOO5DLZMP/XCXpVgvHuskzfU6LvvVQCXgExuJhRD1PrRGeBaWr zjJW+rmY+VeredqX7QkTB3XOGLMLGJfx2FeSMb+j0w3a7/iFj+Q= =J80S -----END PGP SIGNATURE----- Merge tag 'v0.31.1' Changes in synapse v0.31.1 (2018-06-08) ======================================= v0.31.1 fixes a security bug in the ``get_missing_events`` federation API where event visibility rules were not applied correctly. We are not aware of it being actively exploited but please upgrade asap. Bug Fixes: * Fix event filtering in get_missing_events handler (PR #3371)	2018-06-08 15:46:18 +01:00
Neil Johnson	0eb4722932	changelog a bump version	2018-06-08 15:21:46 +01:00
Richard van der Hoff	c6b1441c52	Fix event filtering in get_missing_events handler	2018-06-08 14:15:31 +01:00
David Baker	0e505b1913	Merge pull request #3372 from matrix-org/rav/better_verification_logging ... Try to log more helpful info when a sig verification fails	2018-06-08 13:32:02 +01:00
Richard van der Hoff	e82db24a0e	Try to log more helpful info when a sig verification fails ... Firstly, don't swallow the reason for the failure Secondly, don't assume all exceptions are verification failures Thirdly, log a bit of info about the key being used if debug is enabled	2018-06-08 12:13:08 +01:00
Richard van der Hoff	0834b49c6a	Fix event filtering in get_missing_events handler	2018-06-08 11:34:46 +01:00
Richard van der Hoff	8503dd0047	Remove event re-signing hacks ... These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.	2018-06-07 16:08:29 +01:00
Will Hunt	13d211edc1	Merge pull request #3344 from Half-Shot/hs/as-metrics ... Add metrics to track appservice transactions	2018-06-07 11:37:12 +01:00
Richard van der Hoff	1152f495a0	Merge pull request #3363 from matrix-org/rav/fix_purge ... Fix event-purge-by-ts admin API	2018-06-07 11:34:46 +01:00
Richard van der Hoff	e2acf536d4	Merge pull request #3355 from matrix-org/rav/fix_federation_backfill ... Fix federation backfill from sqlite servers	2018-06-07 10:18:01 +01:00
Amber Brown	0160f6601f	Merge pull request #3356 from matrix-org/rav/add_missing_attr_dep ... Make sure that attr is installed	2018-06-07 16:33:30 +10:00
Richard van der Hoff	f4caf3f83d	fix log	2018-06-07 00:26:38 +01:00
Richard van der Hoff	0546715c18	Fix event-purge-by-ts admin API ... This got completely broken in 0.30. Fixes #3300.	2018-06-07 00:15:49 +01:00
Richard van der Hoff	57e3f923d2	Add missing dependency on attr ... We've rcently added a dep on `attr`. I don't know why the CI didn't pick this up, but we should make it explicit anyway.	2018-06-06 17:12:41 +01:00
Richard van der Hoff	d3a8c9c55e	Fix sql error in _get_state_groups_from_groups ... If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.	2018-06-06 14:19:01 +01:00
Neil Johnson	fef6c2cdcc	Merge branch 'master' into develop	2018-06-06 12:28:15 +01:00
Neil Johnson	752b7b32ed	Changes in synapse v0.31.0 (2018-06-06) ... ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx! -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsXxJAACgkQCTxDAAxP S/SJTg//Wtr+Qop9LJh2/leAYXpyqW6P7Ftak0w3aJ3KL3+tYg32yYNoRADCqbp3 LkrHu8MwbZagHjRUyEWNfDk4jbfq5fwh0JVGmYuUKhG9aF0HYyytKkbW79YzuhdQ dfHj9x0xSBOUvgt/husloZSDy0VHC6uyQSAFgFDyHS2y7RPAiGstqLGByv0ciZOk pO7TdjkUQcx4Ps7Wgip31NuHy3GY2int6f540pUXoZHLXs7RkfqS2cpF9Z/sTXJ4 xDLiY7uYNsTcCblwqaiijY5c90xwRB2vLs5CJdKFgyB6PNgg/2wHJqP/WHHEj+F8 BoSm3Ts7NXQf23pP9CXICe7vXX3J+ruOnC7FOSRobr6KGjn6DUrIZxo1ZepTwpp9 DIq+1eOFKKjwLQM3Jdi8WBCP63LhYXrTZxreke3jpwdcD7oIO9v6/e9J1gU5xHWa Izg+YnWn1JLfq/X8T7YTZddUXGGPkH5i6LZKKkyY8u7LkJ4WR7syuAceUzkOOIAq UWO0uEV7IiLnZzZGTtNIlEqtuklmVQTm6bvAgTPabai2JQyngFpH5M/5mPpVSiLV QRLwaM56c+5GGZJWef8vxdGeYn+8rFI/UUniJ7358kLJF2IHsxlZu8J0ZZO2HWI2 ze5Kz0AWRzXLhWzq62Qb2dsiGySrZ7hng1tDxLak2IiusY+9SjM= =Mz9U -----END PGP SIGNATURE----- Merge tag 'v0.31.0' Changes in synapse v0.31.0 (2018-06-06) ====================================== Most notable change from v0.30.0 is to switch to python prometheus library to improve system stats reporting. WARNING this changes a number of prometheus metrics in a backwards-incompatible manner. For more details, see `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_. Bug Fixes: * Fix metric documentation tables (PR #3341) * Fix LaterGuage error handling (694968f) * Fix replication metrics (b7e7fd2) Changes in synapse v0.31.0-rc1 (2018-06-04) ========================================== Features: * Switch to the Python Prometheus library (PR #3256, #3274) * Let users leave the server notice room after joining (PR #3287) Changes: * daily user type phone home stats (PR #3264) * Use iter* methods for _filter_events_for_server (PR #3267) * Docs on consent bits (PR #3268) * Remove users from user directory on deactivate (PR #3277) * Avoid sending consent notice to guest users (PR #3288) * disable CPUMetrics if no /proc/self/stat (PR #3299) * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat! * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307) * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat! * Reduce stuck read-receipts: ignore depth when updating (PR #3318) * Put python's logs into Trial when running unit tests (PR #3319) Changes, python 3 migration: * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile! * replace some iteritems with six (PR #3244) Thanks to @NotAFile! * Add batch_iter to utils (PR #3245) Thanks to @NotAFile! * use repr, not str (PR #3246) Thanks to @NotAFile! * Misc Python3 fixes (PR #3247) Thanks to @NotAFile! * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile! * more six iteritems (PR #3279) Thanks to @NotAFile! * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile! * remaining isintance fixes (PR #3281) Thanks to @NotAFile! * py3-ize state.py (PR #3283) Thanks to @NotAFile! * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel! * use memoryview in py3 (PR #3303) Thanks to @NotAFile! Bugs: * Fix federation backfill bugs (PR #3261) * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!	2018-06-06 12:27:33 +01:00
Neil Johnson	61134debdc	bump version and changelog	2018-06-06 11:26:21 +01:00
Richard van der Hoff	ad459a106c	Merge pull request #3349 from t3chguy/redact_as_request_token ... Redact AS tokens in log (fixes to #3327)	2018-06-06 10:58:07 +01:00
Michael Telatynski	592c162516	also redact __str__ of ApplicationService used for logging	2018-06-06 10:35:29 +01:00
Michael Telatynski	330432031b	redact_uri in two missed log paths	2018-06-06 10:25:48 +01:00
David Baker	bf54c1cf6c	pep8	2018-06-06 10:15:33 +01:00
David Baker	3e4bc4488c	More doc fixes	2018-06-06 09:44:10 +01:00
Richard van der Hoff	592ee217a3	Merge commit 'b7e7fd2' into release-v0.31.0	2018-06-06 07:02:02 +01:00
Ivan Shapovalov	c88d50aa8f	federation/send_queue.py: fix usage of sortedcontainers.SortedDict	2018-06-06 02:45:18 +03:00
Richard van der Hoff	e316407b5d	Merge pull request #3327 from t3chguy/redact_as_request_token ... Strip `access_token` from outgoing requests	2018-06-05 19:08:46 +01:00
Michael Telatynski	e6cbf47773	factor out uri redaction into a method on http	2018-06-05 18:31:40 +01:00
David Baker	607bd27c83	fix pep8	2018-06-05 18:10:35 +01:00
David Baker	d62162bbec	doc fixes	2018-06-05 18:09:13 +01:00
Richard van der Hoff	522bd3c8a3	Merge remote-tracking branch 'origin/master' into develop	2018-06-05 17:42:49 +01:00
Will Hunt	d6e3c2c79b	Let's try labels instead of label, that might work	2018-06-05 17:30:45 +01:00
Amber Brown	f7869f8f8b	Port to sortedcontainers (with tests!) (#3332)	2018-06-06 00:13:57 +10:00
Will Hunt	604cff1a06	Add metrics to track appservice transactions	2018-06-05 13:21:30 +01:00
Richard van der Hoff	f29b41fde9	Merge pull request #3324 from matrix-org/rav/remove_dead_method ... Remove was_forgotten_at	2018-06-04 18:18:08 +01:00
Richard van der Hoff	28b0490dfd	Merge pull request #3334 from matrix-org/rav/cache_factor_override ... Cache factor override system for specific caches	2018-06-04 17:19:33 +01:00
Richard van der Hoff	b7e7fd2d0e	Fix replication metrics ... fix bug introduced in #3256	2018-06-04 16:23:05 +01:00
Neil Johnson	244ab974e7	bump version and changelog	2018-06-04 16:09:58 +01:00
Richard van der Hoff	694968fa81	Hopefully, fix LaterGuage error handling	2018-06-04 15:59:14 +01:00
Erik Johnston	042eedfa2b	Add hacky cache factor override system	2018-06-04 15:39:28 +01:00
David Baker	c5930d513a	Docstring	2018-06-04 12:05:58 +01:00
David Baker	6a29e815fc	Fix comment	2018-06-04 12:01:23 +01:00
David Baker	e44150a6de	Missing yield	2018-06-04 12:01:13 +01:00
David Baker	f731e42baf	docstring	2018-06-04 12:00:51 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Matthew Hodgson	5f6122fe10	more comments	2018-06-04 00:08:52 +03:00
Ivan Shapovalov	7d9d75e4e8	federation/send_queue.py: fix usage of LaterGauge ... Fixes a startup crash due to commit df9f72d9e5 "replacing portions".	2018-06-03 14:16:17 +03:00
Michael Telatynski	09503126df	Strip access_token from outgoing requests using existing regex	2018-06-02 23:25:13 +01:00
Richard van der Hoff	c1f4118bb6	Remove was_forgotten_at ... This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *	2018-06-01 18:21:49 +01:00
Richard van der Hoff	a9e97dcd65	Merge pull request #3317 from thegcat/feature/3312-add_ipv6_to_blacklist_example_config ... Add private IPv6 addresses to example config for url preview blacklist	2018-06-01 14:45:14 +01:00
Neil Johnson	71477f3317	Merge pull request #3264 from matrix-org/neil/sign-up-stats ... daily user type phone home stats	2018-06-01 13:42:01 +00:00
Richard van der Hoff	9f797a24a4	Handle RRs which arrive before their events	2018-06-01 14:01:43 +01:00
Richard van der Hoff	857e6fd8b6	Ignore depth when updating read-receipts ... Order read receipts by stream ordering instead of depth	2018-06-01 12:18:11 +01:00
Felix Schäfer	4ef76f3ac4	Add private IPv6 addresses to preview blacklist #3312 ... The added addresses are expected to be local or loopback addresses and shouldn't be spidered for previews. Signed-off-by: Felix Schäfer <felix@thegcat.net>	2018-06-01 12:18:35 +02:00
Neil Johnson	4986b084f8	remove unnecessary INSERT	2018-06-01 10:50:40 +01:00
Richard van der Hoff	7e15410f02	Enforce the specified API for report_event ... as per https://matrix.org/docs/spec/client_server/unstable.html#post-matrix-client-r0-rooms-roomid-report-eventid	2018-05-31 18:17:11 +01:00
Amber Brown	febe0ec8fd	Run Prometheus on a different port, optionally. (#3274)	2018-05-31 19:04:50 +10:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Richard van der Hoff	219c2a322b	remove trailing whitespace	2018-05-30 19:42:19 +01:00
Richard van der Hoff	2e4be8bfd9	fix english and wrap comment	2018-05-30 19:24:12 +01:00
Amber Brown	872cf43516	Merge pull request #3303 from NotAFile/py3-memoryview ... use memoryview in py3	2018-05-30 12:51:42 +10:00
Amber Brown	debff7ae09	Merge pull request #3281 from NotAFile/py3-six-isinstance ... remaining isintance fixes	2018-05-30 12:44:46 +10:00
Richard van der Hoff	34b85df7f5	Update some comments and docstrings in SyncHandler	2018-05-29 22:31:18 +01:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Neil Johnson	ab0ef31dc7	create users index on creation_ts	2018-05-29 17:51:08 +01:00
Neil Johnson	558f3d376a	create index in background	2018-05-29 17:47:55 +01:00
Neil Johnson	c379acd4fd	bump version	2018-05-29 17:47:28 +01:00
Adrian Tschira	4b9d0cde97	add remaining memoryview changes	2018-05-29 17:42:43 +02:00
Adrian Tschira	7873cde526	pep8	2018-05-29 17:35:55 +02:00
Adrian Tschira	1afafb3497	use memoryview in py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-29 17:14:34 +02:00
Matthew Hodgson	9bbb9f5556	add lazy_load_members to the filter json schema	2018-05-29 04:26:10 +01:00
Matthew Hodgson	8df7bad839	pep8	2018-05-29 02:49:36 +01:00
Matthew Hodgson	ff1bc0a279	pep8	2018-05-29 02:32:15 +01:00
Matthew Hodgson	0a240ad36e	disable CPUMetrics if no /proc/self/stat ... fixes build on macOS again	2018-05-29 02:23:30 +01:00
Matthew Hodgson	b69ff33d9e	disable CPUMetrics if no /proc/self/stat ... fixes build on macOS again	2018-05-29 02:22:27 +01:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	57ad76fa4a	fix up tests	2018-05-28 19:51:53 +10:00
Amber Brown	3ef5cd74a6	update to more consistently use seconds in any metrics or logging	2018-05-28 19:39:27 +10:00
Amber Brown	5c40ce3777	invalid syntax :(	2018-05-28 19:16:09 +10:00
Amber Brown	357c74a50f	add comment about why unreg	2018-05-28 19:14:41 +10:00
Amber Brown	a2eb5db4a0	update metrics to be in seconds	2018-05-28 19:10:27 +10:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Ruben Barkow	08ea5fe635	add link to thorough instruction how to configure consent	2018-05-25 23:19:55 +02:00
Richard van der Hoff	66bdae986f	Fix default for send_server_notice_to_guests ... bool("False") == True...	2018-05-25 11:42:05 +01:00
Richard van der Hoff	ba1b163590	Avoid sending consent notice to guest users ... we think it makes sense not to send the notices to guest users.	2018-05-25 11:36:43 +01:00
Richard van der Hoff	41921ac01b	Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room ... Let users leave the server notice room after joining	2018-05-25 11:15:45 +01:00
Richard van der Hoff	757ed27258	Let users leave the server notice room after joining ... They still can't reject invites, but we let them leave it.	2018-05-25 11:07:21 +01:00
Adrian Tschira	4ee4450d66	fix recursion error	2018-05-24 21:44:10 +02:00
Amber Brown	9c36c150e7	Merge pull request #3283 from NotAFile/py3-state ... py3-ize state.py	2018-05-24 14:23:33 -05:00
Amber Brown	cc1349c06a	Merge pull request #3279 from NotAFile/py3-more-iteritems ... more six iteritems	2018-05-24 14:23:13 -05:00
Adrian Tschira	0e61705661	py3-ize state.py	2018-05-24 20:59:00 +02:00
Adrian Tschira	dd068ca979	remaining isintance fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:55:08 +02:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Adrian Tschira	6c16a4ec1b	more iteritems	2018-05-24 20:19:06 +02:00
Amber Brown	7ea07c7305	Merge pull request #3278 from NotAFile/py3-storage-base ... Py3 storage/_base.py	2018-05-24 13:08:09 -05:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
Amber Brown	c4fb15a06c	Merge pull request #3246 from NotAFile/py3-repr-string ... use repr, not str	2018-05-24 13:00:20 -05:00
Amber Brown	36501068d8	Merge pull request #3247 from NotAFile/py3-misc ... Misc Python3 fixes	2018-05-24 12:58:37 -05:00
Amber Brown	2aff6eab6d	Merge pull request #3245 from NotAFile/batch-iter ... Add batch_iter to utils	2018-05-24 12:54:12 -05:00
Adrian Tschira	095292304f	Py3 storage/_base.py ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 18:24:12 +02:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	ecc4b88bd1	Merge pull request #3277 from matrix-org/dbkr/remove_from_user_dir ... Remove users from user directory on deactivate	2018-05-24 16:12:12 +01:00
Erik Johnston	46345187cc	Merge pull request #3243 from NotAFile/py3-six-3 ... Replace some more comparisons with six	2018-05-24 16:08:57 +01:00
Neil Johnson	037c6db85d	Merge branch 'master' into develop	2018-05-24 16:03:44 +01:00
David Baker	7a1af504d7	Remove users from user directory on deactivate	2018-05-24 15:59:58 +01:00
Erik Johnston	f72d5a44d5	Merge pull request #3261 from matrix-org/erikj/pagination_fixes ... Fix federation backfill bugs	2018-05-24 14:52:03 +01:00
Erik Johnston	68399fc4de	Merge pull request #3267 from matrix-org/erikj/iter_filter ... Use iter* methods for _filter_events_for_server	2018-05-24 14:44:57 +01:00
Neil Johnson	91d95a1d8e	bump version	2018-05-24 14:05:07 +01:00
David Baker	9700d15611	pep8	2018-05-24 11:23:15 +01:00
David Baker	a21a41bad7	comment	2018-05-24 11:19:59 +01:00
David Baker	b3bff53178	Unbind 3pids when they're deleted too	2018-05-24 11:08:05 +01:00
Richard van der Hoff	6abcb5d22d	Merge pull request #3273 from matrix-org/rav/server_notices_avatar_url ... Allow overriding the server_notices user's avatar	2018-05-24 10:31:43 +01:00
Amber Brown	389dac2c15	pepeightttt	2018-05-23 13:08:59 -05:00
Amber Brown	472a5ec4e2	add back CPU metrics	2018-05-23 13:03:56 -05:00
Amber Brown	e987079037	fixes	2018-05-23 13:03:51 -05:00
Richard van der Hoff	9bf4b2bda3	Allow overriding the server_notices user's avatar ... probably should have done this in the first place, like @turt2live suggested.	2018-05-23 17:43:30 +01:00
Richard van der Hoff	a0b3946fe2	Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri	2018-05-23 16:06:03 +01:00
Richard van der Hoff	2f7008d4eb	Merge pull request #3271 from matrix-org/rav/consent_uri_in_messages ... Support for putting %(consent_uri)s in messages	2018-05-23 16:04:30 +01:00
Richard van der Hoff	dda40fb55d	fix typo	2018-05-23 15:30:26 +01:00
Richard van der Hoff	3ff6f50eac	Use the localpart in the consent uri ... ... because it's shorter.	2018-05-23 15:28:23 +01:00
Richard van der Hoff	82191b08f6	Support for putting %(consent_uri)s in messages ... Make it possible to put the URI in the error message and the server notice that get sent by the server	2018-05-23 15:24:31 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Richard van der Hoff	321f02d263	Block attempts to send server notices to remote users	2018-05-23 14:30:47 +01:00
Neil Johnson	3601a240aa	bump version and changelog	2018-05-23 12:03:23 +01:00
Erik Johnston	5aaa3189d5	s/values/itervalues/	2018-05-23 10:13:05 +01:00
Erik Johnston	0a4bca4134	Use iter* methods for _filter_events_for_server	2018-05-23 10:04:23 +01:00
Amber Brown	b6063631c3	more cleanup	2018-05-22 17:36:20 -05:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Amber Brown	071206304d	cleanup pep8 errors	2018-05-22 16:54:22 -05:00
Amber Brown	85ba83eb51	fixes	2018-05-22 16:28:23 -05:00
Amber Brown	228f1f584e	fix the test failures	2018-05-22 15:02:38 -05:00
Erik Johnston	e85b5a0ff7	Use iter* methods	2018-05-22 19:02:48 +01:00
Erik Johnston	586b66b197	Fix that states is a dict of dicts	2018-05-22 19:02:36 +01:00
Erik Johnston	a17e901f4d	Remove unused string formatting param	2018-05-22 18:24:32 +01:00
Erik Johnston	5494c1d71e	Don't support limitless pagination ... The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.	2018-05-22 18:15:21 +01:00
Neil Johnson	d8cb7225d2	daily user type phone home stats	2018-05-22 18:09:09 +01:00
hera	ad2823ee27	fix synchrotron	2018-05-22 17:47:42 +01:00
Richard van der Hoff	08bfc48abf	custom error code for not leaving server notices room	2018-05-22 17:27:27 +01:00
David Baker	0a078026ea	comment typo	2018-05-22 17:14:06 +01:00
Amber Brown	8f5a688d42	cleanups, self-registration	2018-05-22 10:56:03 -05:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Erik Johnston	cb2a2ad791	get_domains_from_state returns list of tuples	2018-05-22 16:23:39 +01:00
Richard van der Hoff	08a14b32ae	Merge pull request #3262 from matrix-org/rav/has_already_consented ... Add a 'has_consented' template var to consent forms	2018-05-22 15:35:10 +01:00
Richard van der Hoff	7b36d06a69	Add a 'has_consented' template var to consent forms ... fixes #3260	2018-05-22 14:58:34 +01:00
Richard van der Hoff	669400e22f	Enable auto-escaping for the consent templates ... ... to reduce the risk of somebody introducing an html injection attack...	2018-05-22 14:58:34 +01:00
Richard van der Hoff	b5b2d5d64b	Fix dependency on jinja2 ... Delay the import of ConsentResource, so that we can get away without jinja2 if people don't have the consent resource enabled. Fixes #3259	2018-05-22 14:03:45 +01:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	8aeb529262	Merge pull request #3236 from matrix-org/rav/consent_notice ... Send users a server notice about consent	2018-05-22 11:56:09 +01:00
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	2018-05-22 11:54:51 +01:00
Richard van der Hoff	d5dca9a04f	Move consent config parsing into ConsentConfig ... turns out we need to reuse this, so it's better in the config class.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Richard van der Hoff	d14d7b8fdc	Rename 'version' param on user consent config ... we're going to use it for the version we require too.	2018-05-22 11:54:51 +01:00
Erik Johnston	7948ecf234	Comment	2018-05-22 11:39:43 +01:00
Erik Johnston	020377a550	Fix logcontext resource usage tracking	2018-05-22 11:16:07 +01:00
Erik Johnston	13a8dfba0d	Merge pull request #3252 from matrix-org/erikj/in_flight_requests ... Add in flight request metrics	2018-05-22 09:54:30 +01:00
Erik Johnston	c435b0b441	Don't store context	2018-05-22 09:34:26 +01:00
Erik Johnston	fb2806b186	Move in_flight_requests_count to be a callback metric	2018-05-22 09:31:53 +01:00
Amber Brown	fcc525b0b7	rest of the changes	2018-05-21 19:48:57 -05:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Amber Brown	c60e0d5e02	don't need the resource portion	2018-05-21 17:03:20 -05:00
Amber Brown	02c1d29133	look at the Prometheus metrics instead	2018-05-21 17:02:20 -05:00
Amber Brown	f258deffcb	remove old metrics libs	2018-05-21 17:01:15 -05:00
Richard van der Hoff	413482f578	Merge pull request #3255 from matrix-org/rav/fix_transactions ... Stop the transaction cache caching failures	2018-05-21 17:58:08 +01:00
Richard van der Hoff	6e1cb54a05	Fix logcontext leak in HttpTransactionCache ... ONE DAY I WILL PURGE THE WORLD OF THIS EVIL	2018-05-21 16:58:20 +01:00
Richard van der Hoff	6d6e7288fe	Stop the transaction cache caching failures ... The transaction cache has some code which tries to stop it caching failures, but if the callback function failed straight away, then things would happen backwards and we'd end up with the failure stuck in the cache.	2018-05-21 16:49:59 +01:00
Erik Johnston	dfa70adc33	Add in flight request metrics ... This tracks CPU and DB usage while requests are in flight, rather than when we write the response.	2018-05-21 16:23:06 +01:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Adrian Tschira	d9fe2b2d9d	Replace some more comparisons with six ... plus a bonus b"" string I missed last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:56:31 +02:00
Adrian Tschira	45b55e23d3	Add batch_iter to utils ... There's a frequent idiom I noticed where an iterable is split up into a number of chunks/batches. Unfortunately that method does not work with iterators like dict.keys() in python3. This implementation works with iterators. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:48:30 +02:00
Adrian Tschira	dcc235b47d	use stand-in value if maxint is not available ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:44 +02:00
Adrian Tschira	73cbdef5f7	fix py3 intern and remove unnecessary py3 encode ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:31 +02:00
Adrian Tschira	aafb0f6b0d	py3-ize url preview	2018-05-19 17:35:20 +02:00
Adrian Tschira	b932b4ea25	use repr, not str ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:28:42 +02:00
Neil Johnson	644aac5f73	Tighter filtering for user_daily_visits	2018-05-18 17:10:35 +01:00
Neil Johnson	ef466b3a13	fix psql compatability bug	2018-05-18 15:51:21 +01:00
Neil Johnson	2725223f08	Merge branch 'master' into develop	2018-05-18 14:07:50 +01:00
Richard van der Hoff	6d9dc67139	Merge pull request #3232 from matrix-org/rav/server_notices_room ... Infrastructure for a server notices room	2018-05-18 11:28:04 +01:00
Richard van der Hoff	ed3125b0a1	Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred ... Fix error in handling receipts	2018-05-18 11:23:11 +01:00
Richard van der Hoff	67af392712	Merge pull request #3233 from matrix-org/rav/remove_dead_code ... Remove unused `update_external_syncs`	2018-05-18 11:22:43 +01:00
Richard van der Hoff	011e1f4010	Better docstrings	2018-05-18 11:22:12 +01:00
Richard van der Hoff	26305788fe	Make sure we reject attempts to invite the notices user	2018-05-18 11:18:39 +01:00
Richard van der Hoff	d10707c810	Replace inline docstrings with "Attributes" in class docstring	2018-05-18 11:00:55 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	8b1c856d81	Fix error in handling receipts ... Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234	2018-05-18 09:15:35 +01:00
Neil Johnson	6958459b50	bump version, change log	2018-05-17 21:35:07 +01:00
Richard van der Hoff	88d3405332	fix missing yield for server_notices_room	2018-05-17 18:33:45 +01:00
Richard van der Hoff	d43d480d86	Remove unused update_external_syncs ... This method isn't used anywhere. Burninate it.	2018-05-17 18:22:19 +01:00
Richard van der Hoff	fed62e21ad	Infrastructure for a server notices room ... Server Notices use a special room which the user can't dismiss. They are created on demand when some other bit of the code calls send_notice. (This doesn't actually do much yet becuse we don't call send_notice anywhere)	2018-05-17 17:58:25 +01:00
Richard van der Hoff	f8a1e76d64	Merge pull request #3225 from matrix-org/rav/move_creation_handler ... Move RoomCreationHandler out of synapse.handlers.Handlers	2018-05-17 17:56:42 +01:00
Erik Johnston	f7906203f6	Merge pull request #3212 from matrix-org/erikj/epa_stream ... Use stream rather depth ordering for push actions	2018-05-17 12:01:21 +01:00
Richard van der Hoff	c46367d0d7	Move RoomCreationHandler out of synapse.handlers.Handlers ... Handlers is deprecated nowadays, so let's move this out before I add a new dependency on it. Also fix the docstrings on create_room.	2018-05-17 09:08:42 +01:00
Neil Johnson	85b8acdeb4	Merge branch 'master' into develop	2018-05-16 16:48:10 +01:00
Neil Johnson	3c099219e0	bump version and changelog for 0.29.0	2018-05-16 15:44:59 +01:00
Erik Johnston	680530cc7f	Clarify comment	2018-05-16 11:47:29 +01:00
Erik Johnston	43e6e82c4d	Comments	2018-05-16 11:13:31 +01:00
Neil Johnson	dc8930ea9e	Merge pull request #3163 from matrix-org/cohort_analytics ... user visit data	2018-05-16 10:09:24 +00:00
Erik Johnston	c945af8799	Move and rename variable	2018-05-16 10:52:06 +01:00
Neil Johnson	be11a02c4f	remove empty line	2018-05-16 10:45:40 +01:00
Neil Johnson	a2204cc9cc	remove unused method recurring_user_daily_visit_stats	2018-05-16 09:47:20 +01:00
Neil Johnson	31c2502ca8	style and further contraining query	2018-05-16 09:46:43 +01:00
Richard van der Hoff	8030a825c8	Merge pull request #3213 from matrix-org/rav/consent_handler ... ConsentResource to gather policy consent from users	2018-05-16 07:19:18 +01:00
Neil Johnson	c92a8aa578	pep8	2018-05-15 17:31:11 +01:00
Neil Johnson	05ac15ae82	Limit query load of generate_user_daily_visits ... The aim is to keep track of when it was last called and only query from that point in time	2018-05-15 17:01:33 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
Erik Johnston	37dbee6490	Use events_to_purge table rather than token	2018-05-15 16:23:47 +01:00
Richard van der Hoff	47815edcfa	ConsentResource to gather policy consent from users ... Hopefully there are enough comments and docs in this that it makes sense on its own.	2018-05-15 15:11:59 +01:00
Neil Johnson	aea80a0118	v0.29.0-rc1: bump version and change log	2018-05-14 15:50:57 +01:00
Neil Johnson	f077e97914	instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day	2018-05-14 13:50:58 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
Neil Johnson	977765bde2	Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics	2018-05-14 09:31:42 +01:00
Richard van der Hoff	c25d7ba12e	Merge pull request #3208 from matrix-org/rav/more_refactor_request_handler ... Set Server header in SynapseRequest	2018-05-11 16:07:47 +01:00
Erik Johnston	6406b70aeb	Use stream rather depth ordering for push actions ... This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.	2018-05-11 15:30:11 +01:00
Richard van der Hoff	23e2dfe940	Merge pull request #3209 from damir-manapov/master ... transaction_id, destination defined twice	2018-05-11 00:35:13 +01:00
Damir Manapov	db18d854cd	transaction_id, destination twice	2018-05-10 22:13:31 +03:00
Richard van der Hoff	318711e139	Set Server header in SynapseRequest ... (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.	2018-05-10 18:50:27 +01:00
David Baker	6b49628e3b	Catch failure to part user from room	2018-05-10 12:23:53 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
Richard van der Hoff	645cb4bf06	Remove redundant request_handler decorator ... This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.	2018-05-10 12:19:53 +01:00
Richard van der Hoff	09f570b935	Factor wrap_request_handler_with_logging out of wrap_request_handler ... ... so that it can be used on non-JSON endpoints	2018-05-10 12:19:52 +01:00
Richard van der Hoff	9589a1925e	Remove include_metrics param ... The metrics are now available via the request, so this is redundant and can go away at last.	2018-05-10 12:19:52 +01:00
Richard van der Hoff	49e5a613f1	Move outgoing_responses_counter handling to RequestMetrics ... it's much neater there.	2018-05-10 12:19:52 +01:00
Richard van der Hoff	b8700dd7d0	Bump requests_counter in wrapped_request_handler ... less magic	2018-05-10 12:19:52 +01:00
Richard van der Hoff	c6f730282c	Move RequestMetrics handling into SynapseRequest.processing() ... It fits quite nicely here, and opens the path to getting rid of the "include_metrics" mess.	2018-05-10 12:19:51 +01:00
Richard van der Hoff	09b29f9c4a	Make RequestMetrics take a raw time rather than a clock ... ... which is going to make it easier to move around.	2018-05-10 12:18:52 +01:00
David Baker	4d298506dd	Oops, don't call function passed to run_in_background	2018-05-10 11:57:13 +01:00
Richard van der Hoff	8460e48d06	Move request_id management into SynapseRequest	2018-05-10 11:48:17 +01:00
Richard van der Hoff	18e144fe08	Move RequestsMetrics to its own file ... This is useful in its own right, because server.py is full of stuff; but more importantly, I want to do some refactoring that will cause a circular reference as it is.	2018-05-09 19:55:03 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	5adb75bcba	Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination ... Refactor get_recent_events_for_room return type	2018-05-09 16:07:14 +01:00
Erik Johnston	d26bec8a43	Add comment to sync as to why code path is split	2018-05-09 15:56:07 +01:00
Erik Johnston	fcf55f2255	Fix returned token is no longer a tuple	2018-05-09 15:43:00 +01:00
Erik Johnston	7ce98804ff	Fix up comment	2018-05-09 15:42:39 +01:00
Erik Johnston	cddf91c8b9	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change	2018-05-09 15:32:07 +01:00
Erik Johnston	9896dab8f6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination	2018-05-09 15:31:33 +01:00
Erik Johnston	1e5280b7d0	Merge pull request #3196 from matrix-org/erikj/pagination_return ... Refactor pagination DB API to return concrete type	2018-05-09 15:27:17 +01:00
Erik Johnston	75552d2148	Update comments	2018-05-09 15:15:38 +01:00
David Baker	294e9a0c9b	Prefix internal functions	2018-05-09 15:10:37 +01:00
David Baker	46df23f581	Add the schema file	2018-05-09 15:07:54 +01:00
David Baker	52281e4c54	Indent fail	2018-05-09 15:06:16 +01:00