Synapse 1.13.0 (2020-05-19)

=========================== This release brings some potential changes necessary for certain configurations of Synapse: * If your Synapse is configured to use SSO and have a custom `sso_redirect_confirm_template_dir` configuration option set, you will need to duplicate the new `sso_auth_confirm.html`, `sso_auth_success.html` and `sso_account_deactivated.html` templates into that directory. * Synapse plugins using the `complete_sso_login` method of `synapse.module_api.ModuleApi` should instead switch to the async/await version, `complete_sso_login_async`, which includes additional checks. The former version is now deprecated. * A bug was introduced in Synapse 1.4.0 which could cause the room directory to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or earlier, to versions between v1.4.0 and v1.12.x. Please review [UPGRADE.rst](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst) for more details on these changes and for general upgrade guidance. Notice of change to the default `git` branch for Synapse -------------------------------------------------------- With the release of Synapse 1.13.0, the default `git` branch for Synapse has changed to `develop`, which is the development tip. This is more consistent with common practice and modern `git` usage. The `master` branch, which tracks the latest release, is still available. It is recommended that developers and distributors who have scripts which run builds using the default branch of Synapse should therefore consider pinning their scripts to `master`. Features -------- - Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. ([\#7006](https://github.com/matrix-org/synapse/issues/7006)) - Set `Referrer-Policy` header to `no-referrer` on media downloads. ([\#7009](https://github.com/matrix-org/synapse/issues/7009)) - Add support for running replication over Redis when using workers. ([\#7040](https://github.com/matrix-org/synapse/issues/7040), [\#7325](https://github.com/matrix-org/synapse/issues/7325), [\#7352](https://github.com/matrix-org/synapse/issues/7352), [\#7401](https://github.com/matrix-org/synapse/issues/7401), [\#7427](https://github.com/matrix-org/synapse/issues/7427), [\#7439](https://github.com/matrix-org/synapse/issues/7439), [\#7446](https://github.com/matrix-org/synapse/issues/7446), [\#7450](https://github.com/matrix-org/synapse/issues/7450), [\#7454](https://github.com/matrix-org/synapse/issues/7454)) - Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users. ([\#7051](https://github.com/matrix-org/synapse/issues/7051)) - Add options to prevent users from changing their profile or associated 3PIDs. ([\#7096](https://github.com/matrix-org/synapse/issues/7096)) - Support SSO in the user interactive authentication workflow. ([\#7102](https://github.com/matrix-org/synapse/issues/7102), [\#7186](https://github.com/matrix-org/synapse/issues/7186), [\#7279](https://github.com/matrix-org/synapse/issues/7279), [\#7343](https://github.com/matrix-org/synapse/issues/7343)) - Allow server admins to define and enforce a password policy ([MSC2000](https://github.com/matrix-org/matrix-doc/issues/2000)). ([\#7118](https://github.com/matrix-org/synapse/issues/7118)) - Improve the support for SSO authentication on the login fallback page. ([\#7152](https://github.com/matrix-org/synapse/issues/7152), [\#7235](https://github.com/matrix-org/synapse/issues/7235)) - Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. ([\#7153](https://github.com/matrix-org/synapse/issues/7153)) - Admin users are no longer required to be in a room to create an alias for it. ([\#7191](https://github.com/matrix-org/synapse/issues/7191)) - Require admin privileges to enable room encryption by default. This does not affect existing rooms. ([\#7230](https://github.com/matrix-org/synapse/issues/7230)) - Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. ([\#7265](https://github.com/matrix-org/synapse/issues/7265)) - Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. ([\#7315](https://github.com/matrix-org/synapse/issues/7315)) - Add a configuration setting to tweak the threshold for dummy events. ([\#7422](https://github.com/matrix-org/synapse/issues/7422)) Bugfixes -------- - Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. ([\#6573](https://github.com/matrix-org/synapse/issues/6573)) - Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. ([\#6634](https://github.com/matrix-org/synapse/issues/6634)) - Fix missing field `default` when fetching user-defined push rules. ([\#6639](https://github.com/matrix-org/synapse/issues/6639)) - Improve error responses when accessing remote public room lists. ([\#6899](https://github.com/matrix-org/synapse/issues/6899), [\#7368](https://github.com/matrix-org/synapse/issues/7368)) - Transfer alias mappings on room upgrade. ([\#6946](https://github.com/matrix-org/synapse/issues/6946)) - Ensure that a user interactive authentication session is tied to a single request. ([\#7068](https://github.com/matrix-org/synapse/issues/7068), [\#7455](https://github.com/matrix-org/synapse/issues/7455)) - Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. ([\#7089](https://github.com/matrix-org/synapse/issues/7089)) - Return the proper error (`M_BAD_ALIAS`) when a non-existant canonical alias is provided. ([\#7109](https://github.com/matrix-org/synapse/issues/7109)) - Fix a bug which meant that groups updates were not correctly replicated between workers. ([\#7117](https://github.com/matrix-org/synapse/issues/7117)) - Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133)) - Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. ([\#7150](https://github.com/matrix-org/synapse/issues/7150)) - Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. ([\#7151](https://github.com/matrix-org/synapse/issues/7151)) - Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155)) - Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. ([\#7159](https://github.com/matrix-org/synapse/issues/7159)) - Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177)) - Fix a bug which could cause incorrect 'cyclic dependency' error. ([\#7178](https://github.com/matrix-org/synapse/issues/7178)) - Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. ([\#7199](https://github.com/matrix-org/synapse/issues/7199)) - Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. ([\#7203](https://github.com/matrix-org/synapse/issues/7203)) - Do not allow a deactivated user to login via SSO. ([\#7240](https://github.com/matrix-org/synapse/issues/7240), [\#7259](https://github.com/matrix-org/synapse/issues/7259)) - Fix --help command-line argument. ([\#7249](https://github.com/matrix-org/synapse/issues/7249)) - Fix room publish permissions not being checked on room creation. ([\#7260](https://github.com/matrix-org/synapse/issues/7260)) - Reject unknown session IDs during user interactive authentication instead of silently creating a new session. ([\#7268](https://github.com/matrix-org/synapse/issues/7268)) - Fix a SQL query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. ([\#7274](https://github.com/matrix-org/synapse/issues/7274)) - Persist user interactive authentication sessions across workers and Synapse restarts. ([\#7302](https://github.com/matrix-org/synapse/issues/7302)) - Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. ([\#7316](https://github.com/matrix-org/synapse/issues/7316)) - Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7337](https://github.com/matrix-org/synapse/issues/7337), [\#7358](https://github.com/matrix-org/synapse/issues/7358)) - Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. ([\#7341](https://github.com/matrix-org/synapse/issues/7341)) - Fix incorrect metrics reporting for `renew_attestations` background task. ([\#7344](https://github.com/matrix-org/synapse/issues/7344)) - Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. ([\#7367](https://github.com/matrix-org/synapse/issues/7367)) - Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. ([\#7387](https://github.com/matrix-org/synapse/issues/7387)) - Fix bug in `EventContext.deserialize`. ([\#7393](https://github.com/matrix-org/synapse/issues/7393)) - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376)) - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483)) - Hash passwords as early as possible during registration. ([\#7523](https://github.com/matrix-org/synapse/issues/7523)) Improved Documentation ---------------------- - Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. ([\#6892](https://github.com/matrix-org/synapse/issues/6892)) - Improve the documentation for database configuration. ([\#6988](https://github.com/matrix-org/synapse/issues/6988)) - Improve the documentation of application service configuration files. ([\#7091](https://github.com/matrix-org/synapse/issues/7091)) - Update pre-built package name for FreeBSD. ([\#7107](https://github.com/matrix-org/synapse/issues/7107)) - Update postgres docs with login troubleshooting information. ([\#7119](https://github.com/matrix-org/synapse/issues/7119)) - Clean up INSTALL.md a bit. ([\#7141](https://github.com/matrix-org/synapse/issues/7141)) - Add documentation for running a local CAS server for testing. ([\#7147](https://github.com/matrix-org/synapse/issues/7147)) - Improve README.md by being explicit about public IP recommendation for TURN relaying. ([\#7167](https://github.com/matrix-org/synapse/issues/7167)) - Fix a small typo in the `metrics_flags` config option. ([\#7171](https://github.com/matrix-org/synapse/issues/7171)) - Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. ([\#7234](https://github.com/matrix-org/synapse/issues/7234)) - Add documentation to the `password_providers` config option. Add known password provider implementations to docs. ([\#7238](https://github.com/matrix-org/synapse/issues/7238), [\#7248](https://github.com/matrix-org/synapse/issues/7248)) - Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. ([\#7251](https://github.com/matrix-org/synapse/issues/7251)) - Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. ([\#7272](https://github.com/matrix-org/synapse/issues/7272)) - Add documentation on monitoring workers with Prometheus. ([\#7357](https://github.com/matrix-org/synapse/issues/7357)) - Clarify endpoint usage in the users admin api documentation. ([\#7361](https://github.com/matrix-org/synapse/issues/7361)) Deprecations and Removals ------------------------- - Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. ([\#7137](https://github.com/matrix-org/synapse/issues/7137)) Internal Changes ---------------- - Add benchmarks for LruCache. ([\#6446](https://github.com/matrix-org/synapse/issues/6446)) - Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6881](https://github.com/matrix-org/synapse/issues/6881)) - Change device list streams to have one row per ID. ([\#7010](https://github.com/matrix-org/synapse/issues/7010)) - Remove concept of a non-limited stream. ([\#7011](https://github.com/matrix-org/synapse/issues/7011)) - Move catchup of replication streams logic to worker. ([\#7024](https://github.com/matrix-org/synapse/issues/7024), [\#7195](https://github.com/matrix-org/synapse/issues/7195), [\#7226](https://github.com/matrix-org/synapse/issues/7226), [\#7239](https://github.com/matrix-org/synapse/issues/7239), [\#7286](https://github.com/matrix-org/synapse/issues/7286), [\#7290](https://github.com/matrix-org/synapse/issues/7290), [\#7318](https://github.com/matrix-org/synapse/issues/7318), [\#7326](https://github.com/matrix-org/synapse/issues/7326), [\#7378](https://github.com/matrix-org/synapse/issues/7378), [\#7421](https://github.com/matrix-org/synapse/issues/7421)) - Convert some of synapse.rest.media to async/await. ([\#7110](https://github.com/matrix-org/synapse/issues/7110), [\#7184](https://github.com/matrix-org/synapse/issues/7184), [\#7241](https://github.com/matrix-org/synapse/issues/7241)) - De-duplicate / remove unused REST code for login and auth. ([\#7115](https://github.com/matrix-org/synapse/issues/7115)) - Convert `*StreamRow` classes to inner classes. ([\#7116](https://github.com/matrix-org/synapse/issues/7116)) - Clean up some LoggingContext code. ([\#7120](https://github.com/matrix-org/synapse/issues/7120), [\#7181](https://github.com/matrix-org/synapse/issues/7181), [\#7183](https://github.com/matrix-org/synapse/issues/7183), [\#7408](https://github.com/matrix-org/synapse/issues/7408), [\#7426](https://github.com/matrix-org/synapse/issues/7426)) - Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. ([\#7128](https://github.com/matrix-org/synapse/issues/7128)) - Refactored the CAS authentication logic to a separate class. ([\#7136](https://github.com/matrix-org/synapse/issues/7136)) - Run replication streamers on workers. ([\#7146](https://github.com/matrix-org/synapse/issues/7146)) - Add tests for outbound device pokes. ([\#7157](https://github.com/matrix-org/synapse/issues/7157)) - Fix device list update stream ids going backward. ([\#7158](https://github.com/matrix-org/synapse/issues/7158)) - Use `stream.current_token()` and remove `stream_positions()`. ([\#7172](https://github.com/matrix-org/synapse/issues/7172)) - Move client command handling out of TCP protocol. ([\#7185](https://github.com/matrix-org/synapse/issues/7185)) - Move server command handling out of TCP protocol. ([\#7187](https://github.com/matrix-org/synapse/issues/7187)) - Fix consistency of HTTP status codes reported in log lines. ([\#7188](https://github.com/matrix-org/synapse/issues/7188)) - Only run one background database update at a time. ([\#7190](https://github.com/matrix-org/synapse/issues/7190)) - Remove sent outbound device list pokes from the database. ([\#7192](https://github.com/matrix-org/synapse/issues/7192)) - Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. ([\#7193](https://github.com/matrix-org/synapse/issues/7193)) - Remove some extraneous debugging log lines. ([\#7207](https://github.com/matrix-org/synapse/issues/7207)) - Add explicit Python build tooling as dependencies for the snapcraft build. ([\#7213](https://github.com/matrix-org/synapse/issues/7213)) - Add typing information to federation server code. ([\#7219](https://github.com/matrix-org/synapse/issues/7219)) - Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. ([\#7225](https://github.com/matrix-org/synapse/issues/7225)) - Unblacklist '/upgrade creates a new room' sytest for workers. ([\#7228](https://github.com/matrix-org/synapse/issues/7228)) - Remove redundant checks on `daemonize` from synctl. ([\#7233](https://github.com/matrix-org/synapse/issues/7233)) - Upgrade jQuery to v3.4.1 on fallback login/registration pages. ([\#7236](https://github.com/matrix-org/synapse/issues/7236)) - Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. ([\#7237](https://github.com/matrix-org/synapse/issues/7237)) - Correct the parameters of a test fixture. Contributed by Isaiah Singletary. ([\#7243](https://github.com/matrix-org/synapse/issues/7243)) - Convert auth handler to async/await. ([\#7261](https://github.com/matrix-org/synapse/issues/7261)) - Add some unit tests for replication. ([\#7278](https://github.com/matrix-org/synapse/issues/7278)) - Improve typing annotations in `synapse.replication.tcp.streams.Stream`. ([\#7291](https://github.com/matrix-org/synapse/issues/7291)) - Reduce log verbosity of url cache cleanup tasks. ([\#7295](https://github.com/matrix-org/synapse/issues/7295)) - Fix sample SAML Service Provider configuration. Contributed by @frcl. ([\#7300](https://github.com/matrix-org/synapse/issues/7300)) - Fix StreamChangeCache to work with multiple entities changing on the same stream id. ([\#7303](https://github.com/matrix-org/synapse/issues/7303)) - Fix an incorrect import in IdentityHandler. ([\#7319](https://github.com/matrix-org/synapse/issues/7319)) - Reduce logging verbosity for successful federation requests. ([\#7321](https://github.com/matrix-org/synapse/issues/7321)) - Convert some federation handler code to async/await. ([\#7338](https://github.com/matrix-org/synapse/issues/7338)) - Fix collation for postgres for unit tests. ([\#7359](https://github.com/matrix-org/synapse/issues/7359)) - Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. ([\#7363](https://github.com/matrix-org/synapse/issues/7363)) - Add an `instance_name` to `RDATA` and `POSITION` replication commands. ([\#7364](https://github.com/matrix-org/synapse/issues/7364)) - Thread through instance name to replication client. ([\#7369](https://github.com/matrix-org/synapse/issues/7369)) - Convert synapse.server_notices to async/await. ([\#7394](https://github.com/matrix-org/synapse/issues/7394)) - Convert synapse.notifier to async/await. ([\#7395](https://github.com/matrix-org/synapse/issues/7395)) - Fix issues with the Python package manifest. ([\#7404](https://github.com/matrix-org/synapse/issues/7404)) - Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. ([\#7420](https://github.com/matrix-org/synapse/issues/7420)) - Speed up fetching device lists changes when handling `/sync` requests. ([\#7423](https://github.com/matrix-org/synapse/issues/7423)) - Run group attestation renewal in series rather than parallel for performance. ([\#7442](https://github.com/matrix-org/synapse/issues/7442)) - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470)) - Update the version of dh-virtualenv we use to build debs, and add focal to the list of target distributions. ([\#7526](https://github.com/matrix-org/synapse/issues/7526)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl7D5CkACgkQM/xY9qcR MEhe9RAAtSEcFB8tYNbcdbJ87FTsvEEZKCtTmHMXNv+XuslmDWTnINTQ2lXu42Yt 2NT5gPdumRbQ6b8EzV/Xyj+BerjbFlR5qPRWdyGJCHIuLaMLtwKOhOZEy2FXvX8u XEz8isgQqo5NGCHbE+ZpQWE9HUPlWXeS+4LrE3ahEs3X0HAuQL1aKoHMi0KGP3Fe I9m9MkVfzJTa3rqZtXPAAuSOdHIcY3KLLDA7A9U4RGCpvkNnQeT3W7qVzYZUP/hL nXWnYNXfYF0kLj3EaNlnBDigaGm6CQklgPf/zQjZA8gF59d3S24y1mGU2XiBjIJO ECLmttKwSgJEF2uougOSwlGv3qexgfUNe7vaUvHYcw8V04OWewtkzRrqe2p5+pyP RzQgQhd5o8+mZt+sQBhqJGeqvhhbqHnP0j73fxhA3GlKLfXWfJwjqnFXWisxItAf m2kSCFZ+y2xpDiVLA5/ei7WcYp/1qqXzuyacMhncyKZeIaI0YoUSEVc4od++1yzO haow2D9IQNcnkjVtTfMM01+GvecC0k+E8kkh3zqwPxS7aODB50N1Do6LDLfvnCM8 RY8Enl59GxnLwhpFRrcRUfO9cOBAiP+IdkaNRvF+Z5f1kQgmsVMBiBa9y01DhF79 ElyLFhm7wTq5c+NvVi5i4sv8I+LkWSGFdByjG6/B5G3Rbjd0XnA= =ZvLt -----END PGP SIGNATURE----- Merge tag 'v1.13.0' Synapse 1.13.0 (2020-05-19) =========================== This release brings some potential changes necessary for certain configurations of Synapse: * If your Synapse is configured to use SSO and have a custom `sso_redirect_confirm_template_dir` configuration option set, you will need to duplicate the new `sso_auth_confirm.html`, `sso_auth_success.html` and `sso_account_deactivated.html` templates into that directory. * Synapse plugins using the `complete_sso_login` method of `synapse.module_api.ModuleApi` should instead switch to the async/await version, `complete_sso_login_async`, which includes additional checks. The former version is now deprecated. * A bug was introduced in Synapse 1.4.0 which could cause the room directory to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or earlier, to versions between v1.4.0 and v1.12.x. Please review [UPGRADE.rst](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst) for more details on these changes and for general upgrade guidance. Notice of change to the default `git` branch for Synapse -------------------------------------------------------- With the release of Synapse 1.13.0, the default `git` branch for Synapse has changed to `develop`, which is the development tip. This is more consistent with common practice and modern `git` usage. The `master` branch, which tracks the latest release, is still available. It is recommended that developers and distributors who have scripts which run builds using the default branch of Synapse should therefore consider pinning their scripts to `master`. Features -------- - Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. ([\#7006](https://github.com/matrix-org/synapse/issues/7006)) - Set `Referrer-Policy` header to `no-referrer` on media downloads. ([\#7009](https://github.com/matrix-org/synapse/issues/7009)) - Add support for running replication over Redis when using workers. ([\#7040](https://github.com/matrix-org/synapse/issues/7040), [\#7325](https://github.com/matrix-org/synapse/issues/7325), [\#7352](https://github.com/matrix-org/synapse/issues/7352), [\#7401](https://github.com/matrix-org/synapse/issues/7401), [\#7427](https://github.com/matrix-org/synapse/issues/7427), [\#7439](https://github.com/matrix-org/synapse/issues/7439), [\#7446](https://github.com/matrix-org/synapse/issues/7446), [\#7450](https://github.com/matrix-org/synapse/issues/7450), [\#7454](https://github.com/matrix-org/synapse/issues/7454)) - Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users. ([\#7051](https://github.com/matrix-org/synapse/issues/7051)) - Add options to prevent users from changing their profile or associated 3PIDs. ([\#7096](https://github.com/matrix-org/synapse/issues/7096)) - Support SSO in the user interactive authentication workflow. ([\#7102](https://github.com/matrix-org/synapse/issues/7102), [\#7186](https://github.com/matrix-org/synapse/issues/7186), [\#7279](https://github.com/matrix-org/synapse/issues/7279), [\#7343](https://github.com/matrix-org/synapse/issues/7343)) - Allow server admins to define and enforce a password policy ([MSC2000](https://github.com/matrix-org/matrix-doc/issues/2000)). ([\#7118](https://github.com/matrix-org/synapse/issues/7118)) - Improve the support for SSO authentication on the login fallback page. ([\#7152](https://github.com/matrix-org/synapse/issues/7152), [\#7235](https://github.com/matrix-org/synapse/issues/7235)) - Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. ([\#7153](https://github.com/matrix-org/synapse/issues/7153)) - Admin users are no longer required to be in a room to create an alias for it. ([\#7191](https://github.com/matrix-org/synapse/issues/7191)) - Require admin privileges to enable room encryption by default. This does not affect existing rooms. ([\#7230](https://github.com/matrix-org/synapse/issues/7230)) - Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. ([\#7265](https://github.com/matrix-org/synapse/issues/7265)) - Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. ([\#7315](https://github.com/matrix-org/synapse/issues/7315)) - Add a configuration setting to tweak the threshold for dummy events. ([\#7422](https://github.com/matrix-org/synapse/issues/7422)) Bugfixes -------- - Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. ([\#6573](https://github.com/matrix-org/synapse/issues/6573)) - Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. ([\#6634](https://github.com/matrix-org/synapse/issues/6634)) - Fix missing field `default` when fetching user-defined push rules. ([\#6639](https://github.com/matrix-org/synapse/issues/6639)) - Improve error responses when accessing remote public room lists. ([\#6899](https://github.com/matrix-org/synapse/issues/6899), [\#7368](https://github.com/matrix-org/synapse/issues/7368)) - Transfer alias mappings on room upgrade. ([\#6946](https://github.com/matrix-org/synapse/issues/6946)) - Ensure that a user interactive authentication session is tied to a single request. ([\#7068](https://github.com/matrix-org/synapse/issues/7068), [\#7455](https://github.com/matrix-org/synapse/issues/7455)) - Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. ([\#7089](https://github.com/matrix-org/synapse/issues/7089)) - Return the proper error (`M_BAD_ALIAS`) when a non-existant canonical alias is provided. ([\#7109](https://github.com/matrix-org/synapse/issues/7109)) - Fix a bug which meant that groups updates were not correctly replicated between workers. ([\#7117](https://github.com/matrix-org/synapse/issues/7117)) - Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133)) - Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. ([\#7150](https://github.com/matrix-org/synapse/issues/7150)) - Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. ([\#7151](https://github.com/matrix-org/synapse/issues/7151)) - Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155)) - Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. ([\#7159](https://github.com/matrix-org/synapse/issues/7159)) - Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177)) - Fix a bug which could cause incorrect 'cyclic dependency' error. ([\#7178](https://github.com/matrix-org/synapse/issues/7178)) - Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. ([\#7199](https://github.com/matrix-org/synapse/issues/7199)) - Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. ([\#7203](https://github.com/matrix-org/synapse/issues/7203)) - Do not allow a deactivated user to login via SSO. ([\#7240](https://github.com/matrix-org/synapse/issues/7240), [\#7259](https://github.com/matrix-org/synapse/issues/7259)) - Fix --help command-line argument. ([\#7249](https://github.com/matrix-org/synapse/issues/7249)) - Fix room publish permissions not being checked on room creation. ([\#7260](https://github.com/matrix-org/synapse/issues/7260)) - Reject unknown session IDs during user interactive authentication instead of silently creating a new session. ([\#7268](https://github.com/matrix-org/synapse/issues/7268)) - Fix a SQL query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. ([\#7274](https://github.com/matrix-org/synapse/issues/7274)) - Persist user interactive authentication sessions across workers and Synapse restarts. ([\#7302](https://github.com/matrix-org/synapse/issues/7302)) - Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. ([\#7316](https://github.com/matrix-org/synapse/issues/7316)) - Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7337](https://github.com/matrix-org/synapse/issues/7337), [\#7358](https://github.com/matrix-org/synapse/issues/7358)) - Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. ([\#7341](https://github.com/matrix-org/synapse/issues/7341)) - Fix incorrect metrics reporting for `renew_attestations` background task. ([\#7344](https://github.com/matrix-org/synapse/issues/7344)) - Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. ([\#7367](https://github.com/matrix-org/synapse/issues/7367)) - Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. ([\#7387](https://github.com/matrix-org/synapse/issues/7387)) - Fix bug in `EventContext.deserialize`. ([\#7393](https://github.com/matrix-org/synapse/issues/7393)) - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376)) - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483)) - Hash passwords as early as possible during registration. ([\#7523](https://github.com/matrix-org/synapse/issues/7523)) Improved Documentation ---------------------- - Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. ([\#6892](https://github.com/matrix-org/synapse/issues/6892)) - Improve the documentation for database configuration. ([\#6988](https://github.com/matrix-org/synapse/issues/6988)) - Improve the documentation of application service configuration files. ([\#7091](https://github.com/matrix-org/synapse/issues/7091)) - Update pre-built package name for FreeBSD. ([\#7107](https://github.com/matrix-org/synapse/issues/7107)) - Update postgres docs with login troubleshooting information. ([\#7119](https://github.com/matrix-org/synapse/issues/7119)) - Clean up INSTALL.md a bit. ([\#7141](https://github.com/matrix-org/synapse/issues/7141)) - Add documentation for running a local CAS server for testing. ([\#7147](https://github.com/matrix-org/synapse/issues/7147)) - Improve README.md by being explicit about public IP recommendation for TURN relaying. ([\#7167](https://github.com/matrix-org/synapse/issues/7167)) - Fix a small typo in the `metrics_flags` config option. ([\#7171](https://github.com/matrix-org/synapse/issues/7171)) - Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. ([\#7234](https://github.com/matrix-org/synapse/issues/7234)) - Add documentation to the `password_providers` config option. Add known password provider implementations to docs. ([\#7238](https://github.com/matrix-org/synapse/issues/7238), [\#7248](https://github.com/matrix-org/synapse/issues/7248)) - Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. ([\#7251](https://github.com/matrix-org/synapse/issues/7251)) - Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. ([\#7272](https://github.com/matrix-org/synapse/issues/7272)) - Add documentation on monitoring workers with Prometheus. ([\#7357](https://github.com/matrix-org/synapse/issues/7357)) - Clarify endpoint usage in the users admin api documentation. ([\#7361](https://github.com/matrix-org/synapse/issues/7361)) Deprecations and Removals ------------------------- - Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. ([\#7137](https://github.com/matrix-org/synapse/issues/7137)) Internal Changes ---------------- - Add benchmarks for LruCache. ([\#6446](https://github.com/matrix-org/synapse/issues/6446)) - Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6881](https://github.com/matrix-org/synapse/issues/6881)) - Change device list streams to have one row per ID. ([\#7010](https://github.com/matrix-org/synapse/issues/7010)) - Remove concept of a non-limited stream. ([\#7011](https://github.com/matrix-org/synapse/issues/7011)) - Move catchup of replication streams logic to worker. ([\#7024](https://github.com/matrix-org/synapse/issues/7024), [\#7195](https://github.com/matrix-org/synapse/issues/7195), [\#7226](https://github.com/matrix-org/synapse/issues/7226), [\#7239](https://github.com/matrix-org/synapse/issues/7239), [\#7286](https://github.com/matrix-org/synapse/issues/7286), [\#7290](https://github.com/matrix-org/synapse/issues/7290), [\#7318](https://github.com/matrix-org/synapse/issues/7318), [\#7326](https://github.com/matrix-org/synapse/issues/7326), [\#7378](https://github.com/matrix-org/synapse/issues/7378), [\#7421](https://github.com/matrix-org/synapse/issues/7421)) - Convert some of synapse.rest.media to async/await. ([\#7110](https://github.com/matrix-org/synapse/issues/7110), [\#7184](https://github.com/matrix-org/synapse/issues/7184), [\#7241](https://github.com/matrix-org/synapse/issues/7241)) - De-duplicate / remove unused REST code for login and auth. ([\#7115](https://github.com/matrix-org/synapse/issues/7115)) - Convert `*StreamRow` classes to inner classes. ([\#7116](https://github.com/matrix-org/synapse/issues/7116)) - Clean up some LoggingContext code. ([\#7120](https://github.com/matrix-org/synapse/issues/7120), [\#7181](https://github.com/matrix-org/synapse/issues/7181), [\#7183](https://github.com/matrix-org/synapse/issues/7183), [\#7408](https://github.com/matrix-org/synapse/issues/7408), [\#7426](https://github.com/matrix-org/synapse/issues/7426)) - Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. ([\#7128](https://github.com/matrix-org/synapse/issues/7128)) - Refactored the CAS authentication logic to a separate class. ([\#7136](https://github.com/matrix-org/synapse/issues/7136)) - Run replication streamers on workers. ([\#7146](https://github.com/matrix-org/synapse/issues/7146)) - Add tests for outbound device pokes. ([\#7157](https://github.com/matrix-org/synapse/issues/7157)) - Fix device list update stream ids going backward. ([\#7158](https://github.com/matrix-org/synapse/issues/7158)) - Use `stream.current_token()` and remove `stream_positions()`. ([\#7172](https://github.com/matrix-org/synapse/issues/7172)) - Move client command handling out of TCP protocol. ([\#7185](https://github.com/matrix-org/synapse/issues/7185)) - Move server command handling out of TCP protocol. ([\#7187](https://github.com/matrix-org/synapse/issues/7187)) - Fix consistency of HTTP status codes reported in log lines. ([\#7188](https://github.com/matrix-org/synapse/issues/7188)) - Only run one background database update at a time. ([\#7190](https://github.com/matrix-org/synapse/issues/7190)) - Remove sent outbound device list pokes from the database. ([\#7192](https://github.com/matrix-org/synapse/issues/7192)) - Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. ([\#7193](https://github.com/matrix-org/synapse/issues/7193)) - Remove some extraneous debugging log lines. ([\#7207](https://github.com/matrix-org/synapse/issues/7207)) - Add explicit Python build tooling as dependencies for the snapcraft build. ([\#7213](https://github.com/matrix-org/synapse/issues/7213)) - Add typing information to federation server code. ([\#7219](https://github.com/matrix-org/synapse/issues/7219)) - Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. ([\#7225](https://github.com/matrix-org/synapse/issues/7225)) - Unblacklist '/upgrade creates a new room' sytest for workers. ([\#7228](https://github.com/matrix-org/synapse/issues/7228)) - Remove redundant checks on `daemonize` from synctl. ([\#7233](https://github.com/matrix-org/synapse/issues/7233)) - Upgrade jQuery to v3.4.1 on fallback login/registration pages. ([\#7236](https://github.com/matrix-org/synapse/issues/7236)) - Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. ([\#7237](https://github.com/matrix-org/synapse/issues/7237)) - Correct the parameters of a test fixture. Contributed by Isaiah Singletary. ([\#7243](https://github.com/matrix-org/synapse/issues/7243)) - Convert auth handler to async/await. ([\#7261](https://github.com/matrix-org/synapse/issues/7261)) - Add some unit tests for replication. ([\#7278](https://github.com/matrix-org/synapse/issues/7278)) - Improve typing annotations in `synapse.replication.tcp.streams.Stream`. ([\#7291](https://github.com/matrix-org/synapse/issues/7291)) - Reduce log verbosity of url cache cleanup tasks. ([\#7295](https://github.com/matrix-org/synapse/issues/7295)) - Fix sample SAML Service Provider configuration. Contributed by @frcl. ([\#7300](https://github.com/matrix-org/synapse/issues/7300)) - Fix StreamChangeCache to work with multiple entities changing on the same stream id. ([\#7303](https://github.com/matrix-org/synapse/issues/7303)) - Fix an incorrect import in IdentityHandler. ([\#7319](https://github.com/matrix-org/synapse/issues/7319)) - Reduce logging verbosity for successful federation requests. ([\#7321](https://github.com/matrix-org/synapse/issues/7321)) - Convert some federation handler code to async/await. ([\#7338](https://github.com/matrix-org/synapse/issues/7338)) - Fix collation for postgres for unit tests. ([\#7359](https://github.com/matrix-org/synapse/issues/7359)) - Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. ([\#7363](https://github.com/matrix-org/synapse/issues/7363)) - Add an `instance_name` to `RDATA` and `POSITION` replication commands. ([\#7364](https://github.com/matrix-org/synapse/issues/7364)) - Thread through instance name to replication client. ([\#7369](https://github.com/matrix-org/synapse/issues/7369)) - Convert synapse.server_notices to async/await. ([\#7394](https://github.com/matrix-org/synapse/issues/7394)) - Convert synapse.notifier to async/await. ([\#7395](https://github.com/matrix-org/synapse/issues/7395)) - Fix issues with the Python package manifest. ([\#7404](https://github.com/matrix-org/synapse/issues/7404)) - Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. ([\#7420](https://github.com/matrix-org/synapse/issues/7420)) - Speed up fetching device lists changes when handling `/sync` requests. ([\#7423](https://github.com/matrix-org/synapse/issues/7423)) - Run group attestation renewal in series rather than parallel for performance. ([\#7442](https://github.com/matrix-org/synapse/issues/7442)) - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470)) - Update the version of dh-virtualenv we use to build debs, and add focal to the list of target distributions. ([\#7526](https://github.com/matrix-org/synapse/issues/7526))
2024-12-21 12:17:44 -05:00 · 2020-05-19 09:55:39 -04:00 · 2020-05-19 09:55:39 -04:00 · 13a82768ac
commit 13a82768ac
parent a564ec4d4b 45c8b1c618
257 changed files with 12274 additions and 5025 deletions
--- a/.buildkite/worker-blacklist
+++ b/.buildkite/worker-blacklist
@ -5,8 +5,6 @@ Message history can be paginated
 Can re-join room if re-invited
 /upgrade creates a new room
 The only membership state included in an initial sync is for all the senders in the timeline
 Local device key changes get to remote servers
--- a/CHANGES.md
+++ b/CHANGES.md
@ -1,3 +1,206 @@
 Synapse 1.13.0 (2020-05-19)
 ===========================
 This release brings some potential changes necessary for certain
 configurations of Synapse:
 * If your Synapse is configured to use SSO and have a custom
  `sso_redirect_confirm_template_dir` configuration option set, you will need
  to duplicate the new `sso_auth_confirm.html`, `sso_auth_success.html` and
  `sso_account_deactivated.html` templates into that directory.
 * Synapse plugins using the `complete_sso_login` method of
  `synapse.module_api.ModuleApi` should instead switch to the async/await
  version, `complete_sso_login_async`, which includes additional checks. The
  former version is now deprecated.
 * A bug was introduced in Synapse 1.4.0 which could cause the room directory
  to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or
  earlier, to versions between v1.4.0 and v1.12.x.
 Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes
 and for general upgrade guidance.
 Notice of change to the default `git` branch for Synapse
 --------------------------------------------------------
 With the release of Synapse 1.13.0, the default `git` branch for Synapse has
 changed to `develop`, which is the development tip. This is more consistent with
 common practice and modern `git` usage.
 The `master` branch, which tracks the latest release, is still available. It is
 recommended that developers and distributors who have scripts which run builds
 using the default branch of Synapse should therefore consider pinning their
 scripts to `master`.
 Internal Changes
 ----------------
 - Update the version of dh-virtualenv we use to build debs, and add focal to the list of target distributions. ([\#7526](https://github.com/matrix-org/synapse/issues/7526))
 Synapse 1.13.0rc3 (2020-05-18)
 ==============================
 Bugfixes
 --------
 - Hash passwords as early as possible during registration. ([\#7523](https://github.com/matrix-org/synapse/issues/7523))
 Synapse 1.13.0rc2 (2020-05-14)
 ==============================
 Bugfixes
 --------
 - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
 - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
 Internal Changes
 ----------------
 - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
 Synapse 1.13.0rc1 (2020-05-11)
 ==============================
 Features
 --------
 - Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. ([\#7006](https://github.com/matrix-org/synapse/issues/7006))
 - Set `Referrer-Policy` header to `no-referrer` on media downloads. ([\#7009](https://github.com/matrix-org/synapse/issues/7009))
 - Add support for running replication over Redis when using workers. ([\#7040](https://github.com/matrix-org/synapse/issues/7040), [\#7325](https://github.com/matrix-org/synapse/issues/7325), [\#7352](https://github.com/matrix-org/synapse/issues/7352), [\#7401](https://github.com/matrix-org/synapse/issues/7401), [\#7427](https://github.com/matrix-org/synapse/issues/7427), [\#7439](https://github.com/matrix-org/synapse/issues/7439), [\#7446](https://github.com/matrix-org/synapse/issues/7446), [\#7450](https://github.com/matrix-org/synapse/issues/7450), [\#7454](https://github.com/matrix-org/synapse/issues/7454))
 - Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users. ([\#7051](https://github.com/matrix-org/synapse/issues/7051))
 - Add options to prevent users from changing their profile or associated 3PIDs. ([\#7096](https://github.com/matrix-org/synapse/issues/7096))
 - Support SSO in the user interactive authentication workflow. ([\#7102](https://github.com/matrix-org/synapse/issues/7102), [\#7186](https://github.com/matrix-org/synapse/issues/7186), [\#7279](https://github.com/matrix-org/synapse/issues/7279), [\#7343](https://github.com/matrix-org/synapse/issues/7343))
 - Allow server admins to define and enforce a password policy ([MSC2000](https://github.com/matrix-org/matrix-doc/issues/2000)). ([\#7118](https://github.com/matrix-org/synapse/issues/7118))
 - Improve the support for SSO authentication on the login fallback page. ([\#7152](https://github.com/matrix-org/synapse/issues/7152), [\#7235](https://github.com/matrix-org/synapse/issues/7235))
 - Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. ([\#7153](https://github.com/matrix-org/synapse/issues/7153))
 - Admin users are no longer required to be in a room to create an alias for it. ([\#7191](https://github.com/matrix-org/synapse/issues/7191))
 - Require admin privileges to enable room encryption by default. This does not affect existing rooms. ([\#7230](https://github.com/matrix-org/synapse/issues/7230))
 - Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. ([\#7265](https://github.com/matrix-org/synapse/issues/7265))
 - Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. ([\#7315](https://github.com/matrix-org/synapse/issues/7315))
 - Add a configuration setting to tweak the threshold for dummy events. ([\#7422](https://github.com/matrix-org/synapse/issues/7422))
 Bugfixes
 --------
 - Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. ([\#6573](https://github.com/matrix-org/synapse/issues/6573))
 - Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. ([\#6634](https://github.com/matrix-org/synapse/issues/6634))
 - Fix missing field `default` when fetching user-defined push rules. ([\#6639](https://github.com/matrix-org/synapse/issues/6639))
 - Improve error responses when accessing remote public room lists. ([\#6899](https://github.com/matrix-org/synapse/issues/6899), [\#7368](https://github.com/matrix-org/synapse/issues/7368))
 - Transfer alias mappings on room upgrade. ([\#6946](https://github.com/matrix-org/synapse/issues/6946))
 - Ensure that a user interactive authentication session is tied to a single request. ([\#7068](https://github.com/matrix-org/synapse/issues/7068), [\#7455](https://github.com/matrix-org/synapse/issues/7455))
 - Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. ([\#7089](https://github.com/matrix-org/synapse/issues/7089))
 - Return the proper error (`M_BAD_ALIAS`) when a non-existant canonical alias is provided. ([\#7109](https://github.com/matrix-org/synapse/issues/7109))
 - Fix a bug which meant that groups updates were not correctly replicated between workers. ([\#7117](https://github.com/matrix-org/synapse/issues/7117))
 - Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133))
 - Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. ([\#7150](https://github.com/matrix-org/synapse/issues/7150))
 - Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. ([\#7151](https://github.com/matrix-org/synapse/issues/7151))
 - Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155))
 - Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. ([\#7159](https://github.com/matrix-org/synapse/issues/7159))
 - Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177))
 - Fix a bug which could cause incorrect 'cyclic dependency' error. ([\#7178](https://github.com/matrix-org/synapse/issues/7178))
 - Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. ([\#7199](https://github.com/matrix-org/synapse/issues/7199))
 - Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. ([\#7203](https://github.com/matrix-org/synapse/issues/7203))
 - Do not allow a deactivated user to login via SSO. ([\#7240](https://github.com/matrix-org/synapse/issues/7240), [\#7259](https://github.com/matrix-org/synapse/issues/7259))
 - Fix --help command-line argument. ([\#7249](https://github.com/matrix-org/synapse/issues/7249))
 - Fix room publish permissions not being checked on room creation. ([\#7260](https://github.com/matrix-org/synapse/issues/7260))
 - Reject unknown session IDs during user interactive authentication instead of silently creating a new session. ([\#7268](https://github.com/matrix-org/synapse/issues/7268))
 - Fix a SQL query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. ([\#7274](https://github.com/matrix-org/synapse/issues/7274))
 - Persist user interactive authentication sessions across workers and Synapse restarts. ([\#7302](https://github.com/matrix-org/synapse/issues/7302))
 - Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. ([\#7316](https://github.com/matrix-org/synapse/issues/7316))
 - Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7337](https://github.com/matrix-org/synapse/issues/7337), [\#7358](https://github.com/matrix-org/synapse/issues/7358))
 - Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. ([\#7341](https://github.com/matrix-org/synapse/issues/7341))
 - Fix incorrect metrics reporting for `renew_attestations` background task. ([\#7344](https://github.com/matrix-org/synapse/issues/7344))
 - Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. ([\#7367](https://github.com/matrix-org/synapse/issues/7367))
 - Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. ([\#7387](https://github.com/matrix-org/synapse/issues/7387))
 - Fix bug in `EventContext.deserialize`. ([\#7393](https://github.com/matrix-org/synapse/issues/7393))
 Improved Documentation
 ----------------------
 - Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. ([\#6892](https://github.com/matrix-org/synapse/issues/6892))
 - Improve the documentation for database configuration. ([\#6988](https://github.com/matrix-org/synapse/issues/6988))
 - Improve the documentation of application service configuration files. ([\#7091](https://github.com/matrix-org/synapse/issues/7091))
 - Update pre-built package name for FreeBSD. ([\#7107](https://github.com/matrix-org/synapse/issues/7107))
 - Update postgres docs with login troubleshooting information. ([\#7119](https://github.com/matrix-org/synapse/issues/7119))
 - Clean up INSTALL.md a bit. ([\#7141](https://github.com/matrix-org/synapse/issues/7141))
 - Add documentation for running a local CAS server for testing. ([\#7147](https://github.com/matrix-org/synapse/issues/7147))
 - Improve README.md by being explicit about public IP recommendation for TURN relaying. ([\#7167](https://github.com/matrix-org/synapse/issues/7167))
 - Fix a small typo in the `metrics_flags` config option. ([\#7171](https://github.com/matrix-org/synapse/issues/7171))
 - Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. ([\#7234](https://github.com/matrix-org/synapse/issues/7234))
 - Add documentation to the `password_providers` config option. Add known password provider implementations to docs. ([\#7238](https://github.com/matrix-org/synapse/issues/7238), [\#7248](https://github.com/matrix-org/synapse/issues/7248))
 - Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. ([\#7251](https://github.com/matrix-org/synapse/issues/7251))
 - Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. ([\#7272](https://github.com/matrix-org/synapse/issues/7272))
 - Add documentation on monitoring workers with Prometheus. ([\#7357](https://github.com/matrix-org/synapse/issues/7357))
 - Clarify endpoint usage in the users admin api documentation. ([\#7361](https://github.com/matrix-org/synapse/issues/7361))
 Deprecations and Removals
 -------------------------
 - Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. ([\#7137](https://github.com/matrix-org/synapse/issues/7137))
 Internal Changes
 ----------------
 - Add benchmarks for LruCache. ([\#6446](https://github.com/matrix-org/synapse/issues/6446))
 - Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6881](https://github.com/matrix-org/synapse/issues/6881))
 - Change device list streams to have one row per ID. ([\#7010](https://github.com/matrix-org/synapse/issues/7010))
 - Remove concept of a non-limited stream. ([\#7011](https://github.com/matrix-org/synapse/issues/7011))
 - Move catchup of replication streams logic to worker. ([\#7024](https://github.com/matrix-org/synapse/issues/7024), [\#7195](https://github.com/matrix-org/synapse/issues/7195), [\#7226](https://github.com/matrix-org/synapse/issues/7226), [\#7239](https://github.com/matrix-org/synapse/issues/7239), [\#7286](https://github.com/matrix-org/synapse/issues/7286), [\#7290](https://github.com/matrix-org/synapse/issues/7290), [\#7318](https://github.com/matrix-org/synapse/issues/7318), [\#7326](https://github.com/matrix-org/synapse/issues/7326), [\#7378](https://github.com/matrix-org/synapse/issues/7378), [\#7421](https://github.com/matrix-org/synapse/issues/7421))
 - Convert some of synapse.rest.media to async/await. ([\#7110](https://github.com/matrix-org/synapse/issues/7110), [\#7184](https://github.com/matrix-org/synapse/issues/7184), [\#7241](https://github.com/matrix-org/synapse/issues/7241))
 - De-duplicate / remove unused REST code for login and auth. ([\#7115](https://github.com/matrix-org/synapse/issues/7115))
 - Convert `*StreamRow` classes to inner classes. ([\#7116](https://github.com/matrix-org/synapse/issues/7116))
 - Clean up some LoggingContext code. ([\#7120](https://github.com/matrix-org/synapse/issues/7120), [\#7181](https://github.com/matrix-org/synapse/issues/7181), [\#7183](https://github.com/matrix-org/synapse/issues/7183), [\#7408](https://github.com/matrix-org/synapse/issues/7408), [\#7426](https://github.com/matrix-org/synapse/issues/7426))
 - Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. ([\#7128](https://github.com/matrix-org/synapse/issues/7128))
 - Refactored the CAS authentication logic to a separate class. ([\#7136](https://github.com/matrix-org/synapse/issues/7136))
 - Run replication streamers on workers. ([\#7146](https://github.com/matrix-org/synapse/issues/7146))
 - Add tests for outbound device pokes. ([\#7157](https://github.com/matrix-org/synapse/issues/7157))
 - Fix device list update stream ids going backward. ([\#7158](https://github.com/matrix-org/synapse/issues/7158))
 - Use `stream.current_token()` and remove `stream_positions()`. ([\#7172](https://github.com/matrix-org/synapse/issues/7172))
 - Move client command handling out of TCP protocol. ([\#7185](https://github.com/matrix-org/synapse/issues/7185))
 - Move server command handling out of TCP protocol. ([\#7187](https://github.com/matrix-org/synapse/issues/7187))
 - Fix consistency of HTTP status codes reported in log lines. ([\#7188](https://github.com/matrix-org/synapse/issues/7188))
 - Only run one background database update at a time. ([\#7190](https://github.com/matrix-org/synapse/issues/7190))
 - Remove sent outbound device list pokes from the database. ([\#7192](https://github.com/matrix-org/synapse/issues/7192))
 - Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. ([\#7193](https://github.com/matrix-org/synapse/issues/7193))
 - Remove some extraneous debugging log lines. ([\#7207](https://github.com/matrix-org/synapse/issues/7207))
 - Add explicit Python build tooling as dependencies for the snapcraft build. ([\#7213](https://github.com/matrix-org/synapse/issues/7213))
 - Add typing information to federation server code. ([\#7219](https://github.com/matrix-org/synapse/issues/7219))
 - Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. ([\#7225](https://github.com/matrix-org/synapse/issues/7225))
 - Unblacklist '/upgrade creates a new room' sytest for workers. ([\#7228](https://github.com/matrix-org/synapse/issues/7228))
 - Remove redundant checks on `daemonize` from synctl. ([\#7233](https://github.com/matrix-org/synapse/issues/7233))
 - Upgrade jQuery to v3.4.1 on fallback login/registration pages. ([\#7236](https://github.com/matrix-org/synapse/issues/7236))
 - Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. ([\#7237](https://github.com/matrix-org/synapse/issues/7237))
 - Correct the parameters of a test fixture. Contributed by Isaiah Singletary. ([\#7243](https://github.com/matrix-org/synapse/issues/7243))
 - Convert auth handler to async/await. ([\#7261](https://github.com/matrix-org/synapse/issues/7261))
 - Add some unit tests for replication. ([\#7278](https://github.com/matrix-org/synapse/issues/7278))
 - Improve typing annotations in `synapse.replication.tcp.streams.Stream`. ([\#7291](https://github.com/matrix-org/synapse/issues/7291))
 - Reduce log verbosity of url cache cleanup tasks. ([\#7295](https://github.com/matrix-org/synapse/issues/7295))
 - Fix sample SAML Service Provider configuration. Contributed by @frcl. ([\#7300](https://github.com/matrix-org/synapse/issues/7300))
 - Fix StreamChangeCache to work with multiple entities changing on the same stream id. ([\#7303](https://github.com/matrix-org/synapse/issues/7303))
 - Fix an incorrect import in IdentityHandler. ([\#7319](https://github.com/matrix-org/synapse/issues/7319))
 - Reduce logging verbosity for successful federation requests. ([\#7321](https://github.com/matrix-org/synapse/issues/7321))
 - Convert some federation handler code to async/await. ([\#7338](https://github.com/matrix-org/synapse/issues/7338))
 - Fix collation for postgres for unit tests. ([\#7359](https://github.com/matrix-org/synapse/issues/7359))
 - Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. ([\#7363](https://github.com/matrix-org/synapse/issues/7363))
 - Add an `instance_name` to `RDATA` and `POSITION` replication commands. ([\#7364](https://github.com/matrix-org/synapse/issues/7364))
 - Thread through instance name to replication client. ([\#7369](https://github.com/matrix-org/synapse/issues/7369))
 - Convert synapse.server_notices to async/await. ([\#7394](https://github.com/matrix-org/synapse/issues/7394))
 - Convert synapse.notifier to async/await. ([\#7395](https://github.com/matrix-org/synapse/issues/7395))
 - Fix issues with the Python package manifest. ([\#7404](https://github.com/matrix-org/synapse/issues/7404))
 - Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. ([\#7420](https://github.com/matrix-org/synapse/issues/7420))
 - Speed up fetching device lists changes when handling `/sync` requests. ([\#7423](https://github.com/matrix-org/synapse/issues/7423))
 - Run group attestation renewal in series rather than parallel for performance. ([\#7442](https://github.com/matrix-org/synapse/issues/7442))
 Synapse 1.12.4 (2020-04-23)
 ===========================
@ -21,7 +224,6 @@ Bugfixes
 - Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
 - Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
 Synapse 1.12.3 (2020-04-03)
 ===========================
@ -31,13 +233,10 @@ correctly fix the issue with building the Debian packages. ([\#7212](https://git
 Synapse 1.12.2 (2020-04-02)
 ===========================
-This release works around [an
+This release works around [an issue](https://github.com/matrix-org/synapse/issues/7208) with building the debian packages.
 issue](https://github.com/matrix-org/synapse/issues/7208) with building the
 debian packages.
 No other significant changes since 1.12.1.
 Synapse 1.12.1 (2020-04-02)
 ===========================
@ -54,7 +253,6 @@ Bugfixes
 - Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155)). Introduced in v1.12.0rc1.
 - Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177)). Introduced in v1.11.0.
 Synapse 1.12.0 (2020-03-23)
 ===========================
--- a/INSTALL.md
+++ b/INSTALL.md
@ -2,7 +2,6 @@
 - [Installing Synapse](#installing-synapse)
  - [Installing from source](#installing-from-source)
    - [Platform-Specific Instructions](#platform-specific-instructions)
    - [Troubleshooting Installation](#troubleshooting-installation)
  - [Prebuilt packages](#prebuilt-packages)
 - [Setting up Synapse](#setting-up-synapse)
  - [TLS certificates](#tls-certificates)
@ -10,6 +9,7 @@
  - [Registering a user](#registering-a-user)
  - [Setting up a TURN server](#setting-up-a-turn-server)
  - [URL previews](#url-previews)
 - [Troubleshooting Installation](#troubleshooting-installation)
 # Choosing your server name
@ -70,7 +70,7 @@ pip install -U matrix-synapse
 ```
 Before you can start Synapse, you will need to generate a configuration
-file. To do this, run (in your virtualenv, as before)::
+file. To do this, run (in your virtualenv, as before):
 ```
 cd ~/synapse
@ -84,22 +84,24 @@ python -m synapse.app.homeserver \
 ... substituting an appropriate value for `--server-name`.
 This command will generate you a config file that you can then customise, but it will
-also generate a set of keys for you. These keys will allow your Home Server to
+also generate a set of keys for you. These keys will allow your homeserver to
-identify itself to other Home Servers, so don't lose or delete them. It would be
+identify itself to other homeserver, so don't lose or delete them. It would be
 wise to back them up somewhere safe. (If, for whatever reason, you do need to
-change your Home Server's keys, you may find that other Home Servers have the
+change your homeserver's keys, you may find that other homeserver have the
 old key cached. If you update the signing key, you should change the name of the
 key in the `<server name>.signing.key` file (the second word) to something
 different. See the
 [spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
-for more information on key management.)
+for more information on key management).
 To actually run your new homeserver, pick a working directory for Synapse to
-run (e.g. `~/synapse`), and::
+run (e.g. `~/synapse`), and:
 ```
 cd ~/synapse
 source env/bin/activate
 synctl start
 ```
 ### Platform-Specific Instructions
@ -110,7 +112,7 @@ Installing prerequisites on Ubuntu or Debian:
 ```
 sudo apt-get install build-essential python3-dev libffi-dev \
                     python3-pip python3-setuptools sqlite3 \
-                     libssl-dev python3-virtualenv libjpeg-dev libxslt1-dev
+                     libssl-dev virtualenv libjpeg-dev libxslt1-dev
 ```
 #### ArchLinux
@ -188,7 +190,7 @@ doas pkg_add python libffi py-pip py-setuptools sqlite3 py-virtualenv \
 There is currently no port for OpenBSD. Additionally, OpenBSD's security
 settings require a slightly more difficult installation process.
-XXX: I suspect this is out of date.
+(XXX: I suspect this is out of date)
 1. Create a new directory in `/usr/local` called `_synapse`. Also, create a
   new user called `_synapse` and set that directory as the new user's home.
@ -196,7 +198,7 @@ XXX: I suspect this is out of date.
   write and execute permissions on the same memory space to be run from
   `/usr/local`.
 2. `su` to the new `_synapse` user and change to their home directory.
-3. Create a new virtualenv: `virtualenv -p python2.7 ~/.synapse`
+3. Create a new virtualenv: `virtualenv -p python3 ~/.synapse`
 4. Source the virtualenv configuration located at
   `/usr/local/_synapse/.synapse/bin/activate`. This is done in `ksh` by
   using the `.` command, rather than `bash`'s `source`.
@ -217,45 +219,6 @@ be found at https://docs.microsoft.com/en-us/windows/wsl/install-win10 for
 Windows 10 and https://docs.microsoft.com/en-us/windows/wsl/install-on-server
 for Windows Server.
 ### Troubleshooting Installation
 XXX a bunch of this is no longer relevant.
 Synapse requires pip 8 or later, so if your OS provides too old a version you
 may need to manually upgrade it::
    sudo pip install --upgrade pip
 Installing may fail with `Could not find any downloads that satisfy the requirement pymacaroons-pynacl (from matrix-synapse==0.12.0)`.
 You can fix this by manually upgrading pip and virtualenv::
    sudo pip install --upgrade virtualenv
 You can next rerun `virtualenv -p python3 synapse` to update the virtual env.
 Installing may fail during installing virtualenv with `InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.`
 You can fix this  by manually installing ndg-httpsclient::
    pip install --upgrade ndg-httpsclient
 Installing may fail with `mock requires setuptools>=17.1. Aborting installation`.
 You can fix this by upgrading setuptools::
    pip install --upgrade setuptools
 If pip crashes mid-installation for reason (e.g. lost terminal), pip may
 refuse to run until you remove the temporary installation directory it
 created. To reset the installation::
    rm -rf /tmp/pip_install_matrix
 pip seems to leak *lots* of memory during installation.  For instance, a Linux
 host with 512MB of RAM may run out of memory whilst installing Twisted.  If this
 happens, you will have to individually install the dependencies which are
 failing, e.g.::
    pip install twisted
 ## Prebuilt packages
 As an alternative to installing from source, prebuilt packages are available
@ -375,15 +338,17 @@ sudo pip install py-bcrypt
 Synapse can be found in the void repositories as 'synapse':
 ```
 xbps-install -Su
 xbps-install -S synapse
 ```
 ### FreeBSD
 Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
 - Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
- - Packages: `pkg install py27-matrix-synapse`
+ - Packages: `pkg install py37-matrix-synapse`
 ### NixOS
@ -420,6 +385,7 @@ so, you will need to edit `homeserver.yaml`, as follows:
      resources:
        - names: [client, federation]
  ```
 * You will also need to uncomment the `tls_certificate_path` and
  `tls_private_key_path` lines under the `TLS` section. You can either
  point these settings at an existing certificate and key, or you can
@ -435,7 +401,7 @@ so, you will need to edit `homeserver.yaml`, as follows:
  `cert.pem`).
 For a more detailed guide to configuring your server for federation, see
-[federate.md](docs/federate.md)
+[federate.md](docs/federate.md).
 ## Email
@ -494,7 +460,21 @@ This is critical from a security perspective to stop arbitrary Matrix users
 spidering 'internal' URLs on your network. At the very least we recommend that
 your loopback and RFC1918 IP addresses are blacklisted.
-This also requires the optional lxml and netaddr python dependencies to be
+This also requires the optional `lxml` and `netaddr` python dependencies to be
-installed.  This in turn requires the libxml2 library to be available - on
+installed. This in turn requires the `libxml2` library to be available - on
 Debian/Ubuntu this means `apt-get install libxml2-dev`, or equivalent for
 your OS.
 # Troubleshooting Installation
 `pip` seems to leak *lots* of memory during installation. For instance, a Linux
 host with 512MB of RAM may run out of memory whilst installing Twisted. If this
 happens, you will have to individually install the dependencies which are
 failing, e.g.:
 ```
 pip install twisted
 ```
 If you have any other problems, feel free to ask in
 [#synapse:matrix.org](https://matrix.to/#/#synapse:matrix.org).
--- a/MANIFEST.in
+++ b/MANIFEST.in
@ -30,23 +30,24 @@ recursive-include synapse/static *.gif
 recursive-include synapse/static *.html
 recursive-include synapse/static *.js
-exclude Dockerfile
+exclude .codecov.yml
 exclude .coveragerc
 exclude .dockerignore
 exclude test_postgresql.sh
 exclude .editorconfig
 exclude Dockerfile
 exclude mypy.ini
 exclude sytest-blacklist
 exclude test_postgresql.sh
 include pyproject.toml
 recursive-include changelog.d *
 prune .buildkite
 prune .circleci
 prune .codecov.yml
 prune .coveragerc
 prune .github
 prune contrib
 prune debian
 prune demo/etc
 prune docker
 prune mypy.ini
 prune snap
 prune stubs
--- a/UPGRADE.rst
+++ b/UPGRADE.rst
@ -75,6 +75,74 @@ for example:
     wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
     dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 Upgrading to v1.13.0
 ====================
 Incorrect database migration in old synapse versions
 ----------------------------------------------------
 A bug was introduced in Synapse 1.4.0 which could cause the room directory to
 be incomplete or empty if Synapse was upgraded directly from v1.2.1 or
 earlier, to versions between v1.4.0 and v1.12.x.
 This will *not* be a problem for Synapse installations which were:
 * created at v1.4.0 or later,
 * upgraded via v1.3.x, or
 * upgraded straight from v1.2.1 or earlier to v1.13.0 or later.
 If completeness of the room directory is a concern, installations which are
 affected can be repaired as follows:
 1. Run the following sql from a `psql` or `sqlite3` console:
   .. code:: sql
     INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
        ('populate_stats_process_rooms', '{}', 'current_state_events_membership');
     INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
        ('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
 2. Restart synapse.
 New Single Sign-on HTML Templates
 ---------------------------------
 New templates (``sso_auth_confirm.html``, ``sso_auth_success.html``, and
 ``sso_account_deactivated.html``) were added to Synapse. If your Synapse is
 configured to use SSO and a custom  ``sso_redirect_confirm_template_dir``
 configuration then these templates will need to be copied from
 `synapse/res/templates <synapse/res/templates>`_ into that directory.
 Synapse SSO Plugins Method Deprecation
 --------------------------------------
 Plugins using the ``complete_sso_login`` method of
 ``synapse.module_api.ModuleApi`` should update to using the async/await
 version ``complete_sso_login_async`` which includes additional checks. The
 non-async version is considered deprecated.
 Rolling back to v1.12.4 after a failed upgrade
 ----------------------------------------------
 v1.13.0 includes a lot of large changes. If something problematic occurs, you
 may want to roll-back to a previous version of Synapse. Because v1.13.0 also
 includes a new database schema version, reverting that version is also required
 alongside the generic rollback instructions mentioned above. In short, to roll
 back to v1.12.4 you need to:
 1. Stop the server
 2. Decrease the schema version in the database:
   .. code:: sql
      UPDATE schema_version SET version = 57;
 3. Downgrade Synapse by following the instructions for your installation method
   in the "Rolling back to older versions" section above.
 Upgrading to v1.12.0
 ====================
--- a/contrib/systemd-with-workers/README.md
+++ b/contrib/systemd-with-workers/README.md
@ -1,150 +1,2 @@
-# Setup Synapse with Workers and Systemd
+The documentation for using systemd to manage synapse workers is now part of
-
+the main synapse distribution. See [docs/systemd-with-workers](../../docs/systemd-with-workers).
 This is a setup for managing synapse with systemd including support for
 managing workers. It provides a `matrix-synapse`, as well as a
 `matrix-synapse-worker@` service for any workers you require. Additionally to
 group the required services it sets up a `matrix.target`. You can use this to
 automatically start any bot- or bridge-services. More on this in
 [Bots and Bridges](#bots-and-bridges).
 See the folder [system](system) for any service and target files.
 The folder [workers](workers) contains an example configuration for the
 `federation_reader` worker. Pay special attention to the name of the
 configuration file. In order to work with the `matrix-synapse-worker@.service`
 service, it needs to have the exact same name as the worker app.
 This setup expects neither the homeserver nor any workers to fork. Forking is
 handled by systemd.
 ## Setup
 1. Adjust your matrix configs. Make sure that the worker config files have the
 exact same name as the worker app. Compare `matrix-synapse-worker@.service` for
 why. You can find an example worker config in the [workers](workers) folder. See
 below for relevant settings in the `homeserver.yaml`.
 2. Copy the `*.service` and `*.target` files in [system](system) to
 `/etc/systemd/system`.
 3. `systemctl enable matrix-synapse.service` this adds the homeserver
 app to the `matrix.target`
 4. *Optional.* `systemctl enable
 matrix-synapse-worker@federation_reader.service` this adds the federation_reader
 app to the `matrix-synapse.service`
 5. *Optional.* Repeat step 4 for any additional workers you require.
 6. *Optional.* Add any bots or bridges by enabling them.
 7. Start all matrix related services via `systemctl start matrix.target`
 8. *Optional.* Enable autostart of all matrix related services on system boot
 via `systemctl enable matrix.target`
 ## Usage
 After you have setup you can use the following commands to manage your synapse
 installation:
 ```
 # Start matrix-synapse, all workers and any enabled bots or bridges.
 systemctl start matrix.target
 # Restart matrix-synapse and all workers (not necessarily restarting bots
 # or bridges, see "Bots and Bridges")
 systemctl restart matrix-synapse.service
 # Stop matrix-synapse and all workers (not necessarily restarting bots
 # or bridges, see "Bots and Bridges")
 systemctl stop matrix-synapse.service
 # Restart a specific worker (i. e. federation_reader), the homeserver is
 # unaffected by this.
 systemctl restart matrix-synapse-worker@federation_reader.service
 # Add a new worker (assuming all configs are setup already)
 systemctl enable matrix-synapse-worker@federation_writer.service
 systemctl restart matrix-synapse.service
 ```
 ## The Configs
 Make sure the `worker_app` is set in the `homeserver.yaml` and it does not fork.
 ```
 worker_app: synapse.app.homeserver 
 daemonize: false
 ```
 None of the workers should fork, as forking is handled by systemd. Hence make
 sure this is present in all worker config files.
 ```
 worker_daemonize: false
 ```
 The config files of all workers are expected to be located in
 `/etc/matrix-synapse/workers`. If you want to use a different location you have
 to edit the provided `*.service` files accordingly.  
 ## Bots and Bridges
 Most bots and bridges do not care if the homeserver goes down or is restarted.
 Depending on the implementation this may crash them though. So look up the docs
 or ask the community of the specific bridge or bot you want to run to make sure
 you choose the correct setup.
 Whichever configuration you choose, after the setup the following will enable
 automatically starting (and potentially restarting) your bot/bridge with the
 `matrix.target`.
 ```
 systemctl enable <yourBotOrBridgeName>.service
 ```
 **Note** that from an inactive synapse the bots/bridges will only be started with
 synapse if you start the `matrix.target`, not if you start the
 `matrix-synapse.service`. This is on purpose. Think of `matrix-synapse.service`
 as *just* synapse, but `matrix.target` being anything matrix related, including
 synapse and any and all enabled bots and bridges.
 ### Start with synapse but ignore synapse going down
 If the bridge can handle shutdowns of the homeserver you'll want to install the
 service in the `matrix.target` and optionally add a
 `After=matrix-synapse.service` dependency to have the bot/bridge start after
 synapse on starting everything.
 In this case the service file should look like this.
 ```
 [Unit]
 # ...
 # Optional, this will only ensure that if you start everything, synapse will
 # be started before the bot/bridge will be started.
 After=matrix-synapse.service
 [Service]
 # ...
 [Install]
 WantedBy=matrix.target
 ```
 ### Stop/restart when synapse stops/restarts
 If the bridge can't handle shutdowns of the homeserver you'll still want to
 install the service in the `matrix.target` but also have to specify the
 `After=matrix-synapse.service` *and* `BindsTo=matrix-synapse.service`
 dependencies to have the bot/bridge stop/restart with synapse.
 In this case the service file should look like this.
 ```
 [Unit]
 # ...
 # Mandatory
 After=matrix-synapse.service
 BindsTo=matrix-synapse.service
 [Service]
 # ...
 [Install]
 WantedBy=matrix.target
 ```
--- a/contrib/systemd-with-workers/system/matrix-synapse-worker@.service
+++ b/contrib/systemd-with-workers/system/matrix-synapse-worker@.service
@ -1,19 +0,0 @@
 [Unit]
 Description=Synapse Matrix Worker
 After=matrix-synapse.service
 BindsTo=matrix-synapse.service
 [Service]
 Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
 EnvironmentFile=/etc/default/matrix-synapse
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.%i --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=3
 SyslogIdentifier=matrix-synapse-%i
 [Install]
 WantedBy=matrix-synapse.service
--- a/contrib/systemd-with-workers/system/matrix.target
+++ b/contrib/systemd-with-workers/system/matrix.target
@ -1,7 +0,0 @@
 [Unit]
 Description=Contains matrix services like synapse, bridges and bots
 After=network.target
 AllowIsolate=no
 [Install]
 WantedBy=multi-user.target
--- a/debian/build_virtualenv
+++ b/debian/build_virtualenv
@ -36,7 +36,6 @@ esac
 dh_virtualenv \
    --install-suffix "matrix-synapse" \
    --builtin-venv \
    --setuptools \
    --python "$SNAKE" \
    --upgrade-pip \
    --preinstall="lxml" \
--- a/debian/changelog
+++ b/debian/changelog
@ -1,3 +1,13 @@
 matrix-synapse-py3 (1.13.0) stable; urgency=medium
  [ Patrick Cloke ]
  * Add information about .well-known files to Debian installation scripts.
  [ Synapse Packaging team ]
  * New synapse release 1.13.0.
 -- Synapse Packaging team <packages@matrix.org>  Tue, 19 May 2020 09:16:56 -0400
 matrix-synapse-py3 (1.12.4) stable; urgency=medium
  * New synapse release 1.12.4.
--- a/debian/po/templates.pot
+++ b/debian/po/templates.pot
@ -1,14 +1,14 @@
 # SOME DESCRIPTIVE TITLE.
 # Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
-# This file is distributed under the same license as the matrix-synapse package.
+# This file is distributed under the same license as the matrix-synapse-py3 package.
 # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
 #
 #, fuzzy
 msgid ""
 msgstr ""
-"Project-Id-Version: matrix-synapse\n"
+"Project-Id-Version: matrix-synapse-py3\n"
-"Report-Msgid-Bugs-To: matrix-synapse@packages.debian.org\n"
+"Report-Msgid-Bugs-To: matrix-synapse-py3@packages.debian.org\n"
-"POT-Creation-Date: 2017-02-21 07:51+0000\n"
+"POT-Creation-Date: 2020-04-06 16:39-0400\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@ -28,7 +28,10 @@ msgstr ""
 #: ../templates:1001
 msgid ""
 "The name that this homeserver will appear as, to clients and other servers "
-"via federation. This name should match the SRV record published in DNS."
+"via federation. This is normally the public hostname of the server running "
 "synapse, but can be different if you set up delegation. Please refer to the "
 "delegation documentation in this case: https://github.com/matrix-org/synapse/"
 "blob/master/docs/delegate.md."
 msgstr ""
 #. Type: boolean
--- a/debian/templates
+++ b/debian/templates
@ -2,8 +2,10 @@ Template: matrix-synapse/server-name
 Type: string
 _Description: Name of the server:
 The name that this homeserver will appear as, to clients and other
- servers via federation. This name should match the SRV record
+ servers via federation. This is normally the public hostname of the
- published in DNS.
+ server running synapse, but can be different if you set up delegation.
 Please refer to the delegation documentation in this case:
 https://github.com/matrix-org/synapse/blob/master/docs/delegate.md.
 Template: matrix-synapse/report-stats
 Type: boolean
--- a/docker/Dockerfile-dhvirtualenv
+++ b/docker/Dockerfile-dhvirtualenv
@ -27,15 +27,16 @@ RUN env DEBIAN_FRONTEND=noninteractive apt-get install \
        wget
 # fetch and unpack the package
-RUN wget -q -O /dh-virtuenv-1.1.tar.gz https://github.com/spotify/dh-virtualenv/archive/1.1.tar.gz
+RUN mkdir /dh-virtualenv
-RUN tar xvf /dh-virtuenv-1.1.tar.gz
+RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/matrix-org/dh-virtualenv/archive/matrixorg-20200519.tar.gz
 RUN tar -xv --strip-components=1 -C /dh-virtualenv -f /dh-virtualenv.tar.gz
 # install its build deps
-RUN cd dh-virtualenv-1.1/ \
+RUN cd /dh-virtualenv \
-    && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -yqq --no-install-recommends"
+    && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -y --no-install-recommends"
 # build it
-RUN cd dh-virtualenv-1.1 && dpkg-buildpackage -us -uc -b
+RUN cd /dh-virtualenv && dpkg-buildpackage -us -uc -b
 ###
 ### Stage 1
@ -68,12 +69,12 @@ RUN apt-get update -qq -o Acquire::Languages=none \
        sqlite3 \
        libpq-dev
-COPY --from=builder /dh-virtualenv_1.1-1_all.deb /
+COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb /
 # install dhvirtualenv. Update the apt cache again first, in case we got a
 # cached cache from docker the first time.
 RUN apt-get update -qq -o Acquire::Languages=none \
-    && apt-get install -yq /dh-virtualenv_1.1-1_all.deb
+    && apt-get install -yq /dh-virtualenv_1.2~dev-1_all.deb
 WORKDIR /synapse/source
 ENTRYPOINT ["bash","/synapse/source/docker/build_debian.sh"]
--- a/docs/admin_api/room_membership.md
+++ b/docs/admin_api/room_membership.md
@ -0,0 +1,34 @@
 # Edit Room Membership API
 This API allows an administrator to join an user account with a given `user_id`
 to a room with a given `room_id_or_alias`. You can only modify the membership of
 local users. The server administrator must be in the room and have permission to
 invite users.
 ## Parameters
 The following parameters are available:
 * `user_id` - Fully qualified user: for example, `@user:server.com`.
 * `room_id_or_alias` - The room identifier or alias to join: for example,
  `!636q39766251:server.com`.
 ## Usage
 ```
 POST /_synapse/admin/v1/join/<room_id_or_alias>
 {
  "user_id": "@user:server.com"
 }
 ```
 Including an `access_token` of a server admin.
 Response:
 ```
 {
  "room_id": "!636q39766251:server.com"
 }
 ```
--- a/docs/admin_api/rooms.md
+++ b/docs/admin_api/rooms.md
@ -11,8 +11,21 @@ The following query parameters are available:
 * `from` - Offset in the returned list. Defaults to `0`.
 * `limit` - Maximum amount of rooms to return. Defaults to `100`.
 * `order_by` - The method in which to sort the returned list of rooms. Valid values are:
-  - `alphabetical` - Rooms are ordered alphabetically by room name. This is the default.
+  - `alphabetical` - Same as `name`. This is deprecated.
-  - `size` - Rooms are ordered by the number of members. Largest to smallest.
+  - `size` - Same as `joined_members`. This is deprecated.
  - `name` - Rooms are ordered alphabetically by room name. This is the default.
  - `canonical_alias` - Rooms are ordered alphabetically by main alias address of the room.
  - `joined_members` - Rooms are ordered by the number of members. Largest to smallest.
  - `joined_local_members` - Rooms are ordered by the number of local members. Largest to smallest.
  - `version` - Rooms are ordered by room version. Largest to smallest.
  - `creator` - Rooms are ordered alphabetically by creator of the room.
  - `encryption` - Rooms are ordered alphabetically by the end-to-end encryption algorithm.
  - `federatable` - Rooms are ordered by whether the room is federatable.
  - `public` - Rooms are ordered by visibility in room list.
  - `join_rules` - Rooms are ordered alphabetically by join rules of the room.
  - `guest_access` - Rooms are ordered alphabetically by guest access option of the room.
  - `history_visibility` - Rooms are ordered alphabetically by visibility of history of the room.
  - `state_events` - Rooms are ordered by number of state events. Largest to smallest.
 * `dir` - Direction of room order. Either `f` for forwards or `b` for backwards. Setting
          this value to `b` will reverse the above sort order. Defaults to `f`.
 * `search_term` - Filter rooms by their room name. Search term can be contained in any
@ -26,6 +39,16 @@ The following fields are possible in the JSON response body:
    - `name` - The name of the room.
    - `canonical_alias` - The canonical (main) alias address of the room.
    - `joined_members` - How many users are currently in the room.
    - `joined_local_members` - How many local users are currently in the room.
    - `version` - The version of the room as a string.
    - `creator` - The `user_id` of the room creator.
    - `encryption` - Algorithm of end-to-end encryption of messages. Is `null` if encryption is not active.
    - `federatable` - Whether users on other servers can join this room.
    - `public` - Whether the room is visible in room directory.
    - `join_rules` - The type of rules used for users wishing to join this room. One of: ["public", "knock", "invite", "private"].
    - `guest_access` - Whether guests can join the room. One of: ["can_join", "forbidden"].
    - `history_visibility` - Who can see the room history. One of: ["invited", "joined", "shared", "world_readable"].
    - `state_events` - Total number of state_events of a room. Complexity of the room.
 * `offset` - The current pagination offset in rooms. This parameter should be
             used instead of `next_token` for room offset as `next_token` is
             not intended to be parsed.
@ -60,14 +83,34 @@ Response:
      "room_id": "!OGEhHVWSdvArJzumhm:matrix.org",
      "name": "Matrix HQ",
      "canonical_alias": "#matrix:matrix.org",
-      "joined_members": 8326
+      "joined_members": 8326,
      "joined_local_members": 2,
      "version": "1",
      "creator": "@foo:matrix.org",
      "encryption": null,
      "federatable": true,
      "public": true,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 93534
    },
    ... (8 hidden items) ...
    {
      "room_id": "!xYvNcQPhnkrdUmYczI:matrix.org",
      "name": "This Week In Matrix (TWIM)",
      "canonical_alias": "#twim:matrix.org",
-      "joined_members": 314
+      "joined_members": 314,
      "joined_local_members": 20,
      "version": "4",
      "creator": "@foo:matrix.org",
      "encryption": "m.megolm.v1.aes-sha2",
      "federatable": true,
      "public": false,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 8345
    }
  ],
  "offset": 0,
@ -92,7 +135,17 @@ Response:
      "room_id": "!xYvNcQPhnkrdUmYczI:matrix.org",
      "name": "This Week In Matrix (TWIM)",
      "canonical_alias": "#twim:matrix.org",
-      "joined_members": 314
+      "joined_members": 314,
      "joined_local_members": 20,
      "version": "4",
      "creator": "@foo:matrix.org",
      "encryption": "m.megolm.v1.aes-sha2",
      "federatable": true,
      "public": false,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 8
    }
  ],
  "offset": 0,
@ -117,14 +170,34 @@ Response:
      "room_id": "!OGEhHVWSdvArJzumhm:matrix.org",
      "name": "Matrix HQ",
      "canonical_alias": "#matrix:matrix.org",
-      "joined_members": 8326
+      "joined_members": 8326,
      "joined_local_members": 2,
      "version": "1",
      "creator": "@foo:matrix.org",
      "encryption": null,
      "federatable": true,
      "public": true,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 93534
    },
    ... (98 hidden items) ...
    {
      "room_id": "!xYvNcQPhnkrdUmYczI:matrix.org",
      "name": "This Week In Matrix (TWIM)",
      "canonical_alias": "#twim:matrix.org",
-      "joined_members": 314
+      "joined_members": 314,
      "joined_local_members": 20,
      "version": "4",
      "creator": "@foo:matrix.org",
      "encryption": "m.megolm.v1.aes-sha2",
      "federatable": true,
      "public": false,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 8345
    }
  ],
  "offset": 0,
@ -154,6 +227,16 @@ Response:
      "name": "Music Theory",
      "canonical_alias": "#musictheory:matrix.org",
      "joined_members": 127
      "joined_local_members": 2,
      "version": "1",
      "creator": "@foo:matrix.org",
      "encryption": null,
      "federatable": true,
      "public": true,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 93534
    },
    ... (48 hidden items) ...
    {
@ -161,6 +244,16 @@ Response:
      "name": "weechat-matrix",
      "canonical_alias": "#weechat-matrix:termina.org.uk",
      "joined_members": 137
      "joined_local_members": 20,
      "version": "4",
      "creator": "@foo:termina.org.uk",
      "encryption": null,
      "federatable": true,
      "public": true,
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
      "state_events": 8345
    }
  ],
  "offset": 100,
--- a/docs/admin_api/user_admin_api.rst
+++ b/docs/admin_api/user_admin_api.rst
@ -33,12 +33,22 @@ with a body of:
 including an ``access_token`` of a server admin.
-The parameter ``displayname`` is optional and defaults to ``user_id``.
+The parameter ``displayname`` is optional and defaults to the value of
-The parameter ``threepids`` is optional.
+``user_id``.
-The parameter ``avatar_url`` is optional.
+
-The parameter ``admin`` is optional and defaults to 'false'.
+The parameter ``threepids`` is optional and allows setting the third-party IDs
-The parameter ``deactivated`` is optional and defaults to 'false'.
+(email, msisdn) belonging to a user.
-The parameter ``password`` is optional. If provided the user's password is updated and all devices are logged out.
+
 The parameter ``avatar_url`` is optional. Must be a [MXC
 URI](https://matrix.org/docs/spec/client_server/r0.6.0#matrix-content-mxc-uris).
 The parameter ``admin`` is optional and defaults to ``false``.
 The parameter ``deactivated`` is optional and defaults to ``false``.
 The parameter ``password`` is optional. If provided, the user's password is
 updated and all devices are logged out.
 If the user already exists then optional parameters default to the current value.
 List Accounts
@ -51,16 +61,25 @@ The api is::
    GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
 including an ``access_token`` of a server admin.
-The parameters ``from`` and ``limit`` are required only for pagination.
+
-By default, a ``limit`` of 100 is used.
+The parameter ``from`` is optional but used for pagination, denoting the
-The parameter ``user_id`` can be used to select only users with user ids that
+offset in the returned results. This should be treated as an opaque value and
-contain this value.
+not explicitly set to anything other than the return value of ``next_token``
-The parameter ``guests=false`` can be used to exclude guest users,
+from a previous call.
-default is to include guest users.
+
-The parameter ``deactivated=true`` can be used to include deactivated users,
+The parameter ``limit`` is optional but is used for pagination, denoting the
-default is to exclude deactivated users.
+maximum number of items to return in this call. Defaults to ``100``.
-If the endpoint does not return a ``next_token`` then there are no more users left.
+
-It returns a JSON body like the following:
+The parameter ``user_id`` is optional and filters to only users with user IDs
 that contain this value.
 The parameter ``guests`` is optional and if ``false`` will **exclude** guest users.
 Defaults to ``true`` to include guest users.
 The parameter ``deactivated`` is optional and if ``true`` will **include** deactivated users.
 Defaults to ``false`` to exclude deactivated users.
 A JSON body is returned with the following shape:
 .. code:: json
@ -72,19 +91,29 @@ It returns a JSON body like the following:
                "is_guest": 0,
                "admin": 0,
                "user_type": null,
-                "deactivated": 0
+                "deactivated": 0,
                "displayname": "<User One>",
                "avatar_url": null
            }, {
                "name": "<user_id2>",
                "password_hash": "<password_hash2>",
                "is_guest": 0,
                "admin": 1,
                "user_type": null,
-                "deactivated": 0
+                "deactivated": 0,
                "displayname": "<User Two>",
                "avatar_url": "<avatar_url>"
            }
        ],
-        "next_token": "100"
+        "next_token": "100",
        "total": 200
    }
 To paginate, check for ``next_token`` and if present, call the endpoint again
 with ``from`` set to the value of ``next_token``. This will return a new page.
 If the endpoint does not return a ``next_token`` then there are no more users
 to paginate through.
 Query Account
 =============
--- a/docs/application_services.md
+++ b/docs/application_services.md
@ -23,9 +23,13 @@ namespaces:
  users:  # List of users we're interested in
    - exclusive: <bool>
      regex: <regex>
      group_id: <group>
    - ...
  aliases: []  # List of aliases we're interested in
  rooms: [] # List of room ids we're interested in
 ```
 `exclusive`: If enabled, only this application service is allowed to register users in its namespace(s).
 `group_id`: All users of this application service are dynamically joined to this group. This is useful for e.g user organisation or flairs.
 See the [spec](https://matrix.org/docs/spec/application_service/unstable.html) for further details on how application services work.
--- a/docs/dev/cas.md
+++ b/docs/dev/cas.md
@ -0,0 +1,64 @@
 # How to test CAS as a developer without a server
 The [django-mama-cas](https://github.com/jbittel/django-mama-cas) project is an
 easy to run CAS implementation built on top of Django.
 ## Prerequisites
 1. Create a new virtualenv: `python3 -m venv <your virtualenv>`
 2. Activate your virtualenv: `source /path/to/your/virtualenv/bin/activate`
 3. Install Django and django-mama-cas:
   ```
   python -m pip install "django<3" "django-mama-cas==2.4.0"
   ```
 4. Create a Django project in the current directory:
   ```
   django-admin startproject cas_test .
   ```
 5. Follow the [install directions](https://django-mama-cas.readthedocs.io/en/latest/installation.html#configuring) for django-mama-cas
 6. Setup the SQLite database: `python manage.py migrate`
 7. Create a user:
   ```
   python manage.py createsuperuser
   ```
   1. Use whatever you want as the username and password.
   2. Leave the other fields blank.
 8. Use the built-in Django test server to serve the CAS endpoints on port 8000:
   ```
   python manage.py runserver
   ```
 You should now have a Django project configured to serve CAS authentication with
 a single user created.
 ## Configure Synapse (and Riot) to use CAS
 1. Modify your `homeserver.yaml` to enable CAS and point it to your locally
   running Django test server:
   ```yaml
   cas_config:
     enabled: true
     server_url: "http://localhost:8000"
     service_url: "http://localhost:8081"
     #displayname_attribute: name
     #required_attributes:
     #    name: value
   ```
 2. Restart Synapse.
 Note that the above configuration assumes the homeserver is running on port 8081
 and that the CAS server is on port 8000, both on localhost.
 ## Testing the configuration
 Then in Riot:
 1. Visit the login page with a Riot pointing at your homeserver.
 2. Click the Single Sign-On button.
 3. Login using the credentials created with `createsuperuser`.
 4. You should be logged in.
 If you want to repeat this process you'll need to manually logout first:
 1. http://localhost:8000/admin/
 2. Click "logout" in the top right.
--- a/docs/dev/saml.md
+++ b/docs/dev/saml.md
@ -18,9 +18,13 @@ To make Synapse (and therefore Riot) use it:
       metadata:
         local: ["samling.xml"]   
   ```
-5. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure
+5. Ensure that your `homeserver.yaml` has a setting for `public_baseurl`:
   ```yaml
   public_baseurl: http://localhost:8080/
   ```
 6. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure
   the dependencies are installed and ready to go.
-6. Restart Synapse.
+7. Restart Synapse.
 Then in Riot:
--- a/docs/log_contexts.md
+++ b/docs/log_contexts.md
@ -29,14 +29,13 @@ from synapse.logging import context         # omitted from future snippets
 def handle_request(request_id):
    request_context = context.LoggingContext()
-    calling_context = context.LoggingContext.current_context()
+    calling_context = context.set_current_context(request_context)
    context.LoggingContext.set_current_context(request_context)
    try:
        request_context.request = request_id
        do_request_handling()
        logger.debug("finished")
    finally:
-        context.LoggingContext.set_current_context(calling_context)
+        context.set_current_context(calling_context)
 def do_request_handling():
    logger.debug("phew")  # this will be logged against request_id
--- a/docs/metrics-howto.md
+++ b/docs/metrics-howto.md
@ -60,6 +60,31 @@
 1.  Restart Prometheus.
 ## Monitoring workers
 To monitor a Synapse installation using
 [workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md),
 every worker needs to be monitored independently, in addition to
 the main homeserver process. This is because workers don't send
 their metrics to the main homeserver process, but expose them
 directly (if they are configured to do so).
 To allow collecting metrics from a worker, you need to add a
 `metrics` listener to its configuration, by adding the following
 under `worker_listeners`:
 ```yaml
 - type: metrics
   bind_address: ''
   port: 9101
 ```
 The `bind_address` and `port` parameters should be set so that
 the resulting listener can be reached by prometheus, and they
 don't clash with an existing worker.
 With this example, the worker's metrics would then be available
 on `http://127.0.0.1:9101`.
 ## Renaming of metrics & deprecation of old names in 1.2
 Synapse 1.2 updates the Prometheus metrics to match the naming
--- a/docs/password_auth_providers.md
+++ b/docs/password_auth_providers.md
@ -9,7 +9,11 @@ into Synapse, and provides a number of methods by which it can integrate
 with the authentication system.
 This document serves as a reference for those looking to implement their
-own password auth providers.
+own password auth providers. Additionally, here is a list of known
 password auth provider module implementations:
 * [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/)
 * [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth)
 ## Required methods
--- a/docs/postgres.md
+++ b/docs/postgres.md
@ -61,7 +61,33 @@ Note that the PostgreSQL database *must* have the correct encoding set
 You may need to enable password authentication so `synapse_user` can
 connect to the database. See
-<https://www.postgresql.org/docs/11/auth-pg-hba-conf.html>.
+<https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>.
 If you get an error along the lines of `FATAL:  Ident authentication failed for
 user "synapse_user"`, you may need to use an authentication method other than
 `ident`:
 * If the `synapse_user` user has a password, add the password to the `database:`
  section of `homeserver.yaml`. Then add the following to `pg_hba.conf`:
  ```
  host    synapse     synapse_user    ::1/128     md5  # or `scram-sha-256` instead of `md5` if you use that
  ```
 * If the `synapse_user` user does not have a password, then a password doesn't
  have to be added to `homeserver.yaml`. But the following does need to be added
  to `pg_hba.conf`:
  ```
  host    synapse     synapse_user    ::1/128     trust
  ```
 Note that line order matters in `pg_hba.conf`, so make sure that if you do add a
 new line, it is inserted before:
 ```
 host    all         all             ::1/128     ident
 ```
 ### Fixing incorrect `COLLATE` or `CTYPE`
@ -104,6 +130,7 @@ of free memory the database host has available.
 When you are ready to start using PostgreSQL, edit the `database`
 section in your config file to match the following lines:
 ```yaml
 database:
  name: psycopg2
  args:
@ -113,10 +140,31 @@ section in your config file to match the following lines:
    host: <host>
    cp_min: 5
    cp_max: 10
 ```
 All key, values in `args` are passed to the `psycopg2.connect(..)`
 function, except keys beginning with `cp_`, which are consumed by the
-twisted adbapi connection pool.
+twisted adbapi connection pool. See the [libpq
 documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
 for a list of options which can be passed.
 You should consider tuning the `args.keepalives_*` options if there is any danger of
 the connection between your homeserver and database dropping, otherwise Synapse
 may block for an extended period while it waits for a response from the
 database server. Example values might be:
 ```yaml
 # seconds of inactivity after which TCP should send a keepalive message to the server
 keepalives_idle: 10
 # the number of seconds after which a TCP keepalive message that is not
 # acknowledged by the server should be retransmitted
 keepalives_interval: 10
 # the number of TCP keepalives that can be lost before the client's connection
 # to the server is considered dead
 keepalives_count: 3
 ```
 ## Porting from SQLite
--- a/docs/reverse_proxy.md
+++ b/docs/reverse_proxy.md
@ -42,6 +42,9 @@ the reverse proxy and the homeserver.
            location /_matrix {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
                # Nginx by default only allows file uploads up to 1M in size
                # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
                client_max_body_size 10M;
            }
        }
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@ -33,10 +33,15 @@ server_name: "SERVERNAME"
 #
 pid_file: DATADIR/homeserver.pid
-# The path to the web client which will be served at /_matrix/client/
+# The absolute URL to the web client which /_matrix/client will redirect
-# if 'webclient' is configured under the 'listeners' configuration.
+# to if 'webclient' is configured under the 'listeners' configuration.
 #
-#web_client_location: "/path/to/web/root"
+# This option can be also set to the filesystem path to the web client
 # which will be served at /_matrix/client/ if 'webclient' is configured
 # under the 'listeners' configuration, however this is a security risk:
 # https://github.com/matrix-org/synapse#security-note
 #
 #web_client_location: https://riot.example.com/
 # The public-facing base URL that clients use to access this HS
 # (not including _matrix/...). This is the same URL a user would
@ -248,6 +253,18 @@ listeners:
  #  bind_addresses: ['::1', '127.0.0.1']
  #  type: manhole
 # Forward extremities can build up in a room due to networking delays between
 # homeservers. Once this happens in a large room, calculation of the state of
 # that room can become quite expensive. To mitigate this, once the number of
 # forward extremities reaches a given threshold, Synapse will send an
 # org.matrix.dummy_event event, which will reduce the forward extremities
 # in the room.
 #
 # This setting defines the threshold (i.e. number of forward extremities in the
 # room) at which dummy events are sent. The default value is 10.
 #
 #dummy_events_threshold: 5
 ## Homeserver blocking ##
@ -409,6 +426,16 @@ retention:
  #    longest_max_lifetime: 1y
  #    interval: 1d
 # Inhibits the /requestToken endpoints from returning an error that might leak
 # information about whether an e-mail address is in use or not on this
 # homeserver.
 # Note that for some endpoints the error situation is the e-mail already being
 # used, and for others the error is entering the e-mail being unused.
 # If this option is enabled, instead of returning an error, these endpoints will
 # act as if no error happened and return a fake session ID ('sid') to clients.
 #
 #request_token_inhibit_3pid_errors: true
 ## TLS ##
@ -578,13 +605,46 @@ acme:
 ## Database ##
 # The 'database' setting defines the database that synapse uses to store all of
 # its data.
 #
 # 'name' gives the database engine to use: either 'sqlite3' (for SQLite) or
 # 'psycopg2' (for PostgreSQL).
 #
 # 'args' gives options which are passed through to the database engine,
 # except for options starting 'cp_', which are used to configure the Twisted
 # connection pool. For a reference to valid arguments, see:
 #   * for sqlite: https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
 #   * for postgres: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
 #   * for the connection pool: https://twistedmatrix.com/documents/current/api/twisted.enterprise.adbapi.ConnectionPool.html#__init__
 #
 #
 # Example SQLite configuration:
 #
 #database:
 #  name: sqlite3
 #  args:
 #    database: /path/to/homeserver.db
 #
 #
 # Example Postgres configuration:
 #
 #database:
 #  name: psycopg2
 #  args:
 #    user: synapse
 #    password: secretpassword
 #    database: synapse
 #    host: localhost
 #    cp_min: 5
 #    cp_max: 10
 #
 # For more information on using Synapse with Postgres, see `docs/postgres.md`.
 #
 database:
-  # The database engine name
+  name: sqlite3
  name: "sqlite3"
  # Arguments to pass to the engine
  args:
-    # Path to the database
+    database: DATADIR/homeserver.db
    database: "DATADIR/homeserver.db"
 # Number of events to cache in memory.
 #
@ -697,12 +757,11 @@ media_store_path: "DATADIR/media_store"
 #
 #media_storage_providers:
 #  - module: file_system
-#    # Whether to write new local files.
+#    # Whether to store newly uploaded local files
 #    store_local: false
-#    # Whether to write new remote media
+#    # Whether to store newly downloaded remote files
 #    store_remote: false
-#    # Whether to block upload requests waiting for write to this
+#    # Whether to wait for successful storage for local uploads
 #    # provider to complete
 #    store_synchronous: false
 #    config:
 #       directory: /mnt/some/other/directory
@ -821,6 +880,31 @@ media_store_path: "DATADIR/media_store"
 #
 #max_spider_size: 10M
 # A list of values for the Accept-Language HTTP header used when
 # downloading webpages during URL preview generation. This allows
 # Synapse to specify the preferred languages that URL previews should
 # be in when communicating with remote servers.
 #
 # Each value is a IETF language tag; a 2-3 letter identifier for a
 # language, optionally followed by subtags separated by '-', specifying
 # a country or region variant.
 #
 # Multiple values can be provided, and a weight can be added to each by
 # using quality value syntax (;q=). '*' translates to any language.
 #
 # Defaults to "en".
 #
 # Example:
 #
 # url_preview_accept_language:
 #   - en-UK
 #   - en-US;q=0.9
 #   - fr;q=0.8
 #   - *;q=0.7
 #
 url_preview_accept_language:
 #   - en
 ## Captcha ##
 # See docs/CAPTCHA_SETUP for full details of configuring this.
@ -839,10 +923,6 @@ media_store_path: "DATADIR/media_store"
 #
 #enable_registration_captcha: false
 # A secret key used to bypass the captcha test entirely.
 #
 #captcha_bypass_secret: "YOUR_SECRET_HERE"
 # The API endpoint to use for verifying m.login.recaptcha responses.
 #
 #recaptcha_siteverify_api: "https://www.recaptcha.net/recaptcha/api/siteverify"
@ -1057,6 +1137,29 @@ account_threepid_delegates:
    #email: https://example.com     # Delegate email sending to example.com
    #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
 # Whether users are allowed to change their displayname after it has
 # been initially set. Useful when provisioning users based on the
 # contents of a third-party directory.
 #
 # Does not apply to server administrators. Defaults to 'true'
 #
 #enable_set_displayname: false
 # Whether users are allowed to change their avatar after it has been
 # initially set. Useful when provisioning users based on the contents
 # of a third-party directory.
 #
 # Does not apply to server administrators. Defaults to 'true'
 #
 #enable_set_avatar_url: false
 # Whether users can change the 3PIDs associated with their accounts
 # (email address and msisdn).
 #
 # Defaults to 'true'
 #
 #enable_3pid_changes: false
 # Users who register on this homeserver will automatically be joined
 # to these rooms
 #
@ -1392,6 +1495,10 @@ sso:
    # phishing attacks from evil.site. To avoid this, include a slash after the
    # hostname: "https://my.client/".
    #
    # If public_baseurl is set, then the login fallback page (used by clients
    # that don't natively support the required login flows) is whitelisted in
    # addition to any URLs in this list.
    #
    # By default, this list is empty.
    #
    #client_whitelist:
@ -1423,6 +1530,30 @@ sso:
    #
    #     * server_name: the homeserver's name.
    #
    # * HTML page which notifies the user that they are authenticating to confirm
    #   an operation on their account during the user interactive authentication
    #   process: 'sso_auth_confirm.html'.
    #
    #   When rendering, this template is given the following variables:
    #     * redirect_url: the URL the user is about to be redirected to. Needs
    #                     manual escaping (see
    #                     https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping).
    #
    #     * description: the operation which the user is being asked to confirm
    #
    # * HTML page shown after a successful user interactive authentication session:
    #   'sso_auth_success.html'.
    #
    #   Note that this page must include the JavaScript which notifies of a successful authentication
    #   (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
    #
    #   This template has no additional variables.
    #
    # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
    #   attempts to login: 'sso_account_deactivated.html'.
    #
    #   This template has no additional variables.
    #
    # You can see the default templates at:
    # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
    #
@ -1453,6 +1584,41 @@ password_config:
   #
   #pepper: "EVEN_MORE_SECRET"
   # Define and enforce a password policy. Each parameter is optional.
   # This is an implementation of MSC2000.
   #
   policy:
      # Whether to enforce the password policy.
      # Defaults to 'false'.
      #
      #enabled: true
      # Minimum accepted length for a password.
      # Defaults to 0.
      #
      #minimum_length: 15
      # Whether a password must contain at least one digit.
      # Defaults to 'false'.
      #
      #require_digit: true
      # Whether a password must contain at least one symbol.
      # A symbol is any character that's not a number or a letter.
      # Defaults to 'false'.
      #
      #require_symbol: true
      # Whether a password must contain at least one lowercase letter.
      # Defaults to 'false'.
      #
      #require_lowercase: true
      # Whether a password must contain at least one lowercase letter.
      # Defaults to 'false'.
      #
      #require_uppercase: true
 # Configuration for sending emails from Synapse.
 #
@ -1561,7 +1727,19 @@ email:
  #template_dir: "res/templates"
-#password_providers:
+# Password providers allow homeserver administrators to integrate
 # their Synapse installation with existing authentication methods
 # ex. LDAP, external tokens, etc.
 #
 # For more information and known implementations, please see
 # https://github.com/matrix-org/synapse/blob/master/docs/password_auth_providers.md
 #
 # Note: instances wishing to use SAML or CAS authentication should
 # instead use the `saml2_config` or `cas_config` options,
 # respectively.
 #
 password_providers:
 #    # Example config for an LDAP auth provider
 #    - module: "ldap_auth_provider.LdapAuthProvider"
 #      config:
 #        enabled: true
--- a/docs/systemd-with-workers/README.md
+++ b/docs/systemd-with-workers/README.md
@ -0,0 +1,67 @@
 # Setting up Synapse with Workers and Systemd
 This is a setup for managing synapse with systemd, including support for
 managing workers. It provides a `matrix-synapse` service for the master, as
 well as a `matrix-synapse-worker@` service template for any workers you
 require. Additionally, to group the required services, it sets up a
 `matrix-synapse.target`.
 See the folder [system](system) for the systemd unit files.
 The folder [workers](workers) contains an example configuration for the
 `federation_reader` worker.
 ## Synapse configuration files
 See [workers.md](../workers.md) for information on how to set up the
 configuration files and reverse-proxy correctly. You can find an example worker
 config in the [workers](workers) folder.
 Systemd manages daemonization itself, so ensure that none of the configuration
 files set either `daemonize` or `worker_daemonize`.
 The config files of all workers are expected to be located in
 `/etc/matrix-synapse/workers`. If you want to use a different location, edit
 the provided `*.service` files accordingly.
 There is no need for a separate configuration file for the master process.
 ## Set up
 1. Adjust synapse configuration files as above.
 1. Copy the `*.service` and `*.target` files in [system](system) to
 `/etc/systemd/system`.
 1. Run `systemctl deamon-reload` to tell systemd to load the new unit files.
 1. Run `systemctl enable matrix-synapse.service`. This will configure the
 synapse master process to be started as part of the `matrix-synapse.target`
 target.
 1. For each worker process to be enabled, run `systemctl enable
 matrix-synapse-worker@<worker_name>.service`. For each `<worker_name>`, there
 should be a corresponding configuration file
 `/etc/matrix-synapse/workers/<worker_name>.yaml`.
 1. Start all the synapse processes with `systemctl start matrix-synapse.target`.
 1. Tell systemd to start synapse on boot with `systemctl enable matrix-synapse.target`/
 ## Usage
 Once the services are correctly set up, you can use the following commands
 to manage your synapse installation:
 ```sh
 # Restart Synapse master and all workers
 systemctl restart matrix-synapse.target
 # Stop Synapse and all workers
 systemctl stop matrix-synapse.target
 # Restart the master alone
 systemctl start matrix-synapse.service
 # Restart a specific worker (eg. federation_reader); the master is
 # unaffected by this.
 systemctl restart matrix-synapse-worker@federation_reader.service
 # Add a new worker (assuming all configs are set up already)
 systemctl enable matrix-synapse-worker@federation_writer.service
 systemctl restart matrix-synapse.target
 ```
--- a/docs/systemd-with-workers/system/matrix-synapse-worker@.service
+++ b/docs/systemd-with-workers/system/matrix-synapse-worker@.service
@ -0,0 +1,20 @@
 [Unit]
 Description=Synapse %i
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
 [Service]
 Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
 EnvironmentFile=/etc/default/matrix-synapse
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=3
 SyslogIdentifier=matrix-synapse-%i
 [Install]
 WantedBy=matrix-synapse.target
--- a/contrib/systemd-with-workers/system/matrix-synapse.service
+++ b/contrib/systemd-with-workers/system/matrix-synapse.service
@ -1,5 +1,8 @@
 [Unit]
-Description=Synapse Matrix Homeserver
+Description=Synapse master
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
 [Service]
 Type=notify
@ -15,4 +18,4 @@ RestartSec=3
 SyslogIdentifier=matrix-synapse
 [Install]
-WantedBy=matrix.target
+WantedBy=matrix-synapse.target
--- a/docs/systemd-with-workers/system/matrix-synapse.target
+++ b/docs/systemd-with-workers/system/matrix-synapse.target
@ -0,0 +1,6 @@
 [Unit]
 Description=Synapse parent target
 After=network.target
 [Install]
 WantedBy=multi-user.target
--- a/contrib/systemd-with-workers/workers/federation_reader.yaml
+++ b/contrib/systemd-with-workers/workers/federation_reader.yaml
@ -10,5 +10,4 @@ worker_listeners:
      resources:
          - names: [federation]
 worker_daemonize: false
 worker_log_config: /etc/matrix-synapse/federation-reader-log.yaml
--- a/docs/tcp_replication.md
+++ b/docs/tcp_replication.md
@ -14,16 +14,18 @@ example flow would be (where '>' indicates master to worker and
 '<' worker to master flows):
    > SERVER example.com
-    < REPLICATE events 53
+    < REPLICATE
-    > RDATA events 54 ["$foo1:bar.com", ...]
+    > POSITION events master 53
-    > RDATA events 55 ["$foo4:bar.com", ...]
+    > RDATA events master 54 ["$foo1:bar.com", ...]
    > RDATA events master 55 ["$foo4:bar.com", ...]
-The example shows the server accepting a new connection and sending its
+The example shows the server accepting a new connection and sending its identity
-identity with the `SERVER` command, followed by the client asking to
+with the `SERVER` command, followed by the client server to respond with the
-subscribe to the `events` stream from the token `53`. The server then
+position of all streams. The server then periodically sends `RDATA` commands
-periodically sends `RDATA` commands which have the format
+which have the format `RDATA <stream_name> <instance_name> <token> <row>`, where
-`RDATA <stream_name> <token> <row>`, where the format of `<row>` is
+the format of `<row>` is defined by the individual streams. The
-defined by the individual streams.
+`<instance_name>` is the name of the Synapse process that generated the data
 (usually "master").
 Error reporting happens by either the client or server sending an ERROR
 command, and usually the connection will be closed.
@ -32,9 +34,6 @@ Since the protocol is a simple line based, its possible to manually
 connect to the server using a tool like netcat. A few things should be
 noted when manually using the protocol:
 -   When subscribing to a stream using `REPLICATE`, the special token
    `NOW` can be used to get all future updates. The special stream name
    `ALL` can be used with `NOW` to subscribe to all available streams.
 -   The federation stream is only available if federation sending has
    been disabled on the main process.
 -   The server will only time connections out that have sent a `PING`
@ -55,7 +54,7 @@ The basic structure of the protocol is line based, where the initial
 word of each line specifies the command. The rest of the line is parsed
 based on the command. For example, the RDATA command is defined as:
-    RDATA <stream_name> <token> <row_json>
+    RDATA <stream_name> <instance_name> <token> <row_json>
 (Note that <row_json> may contains spaces, but cannot contain
 newlines.)
@ -91,9 +90,7 @@ The client:
 -   Sends a `NAME` command, allowing the server to associate a human
    friendly name with the connection. This is optional.
 -   Sends a `PING` as above
-   For each stream the client wishes to subscribe to it sends a
+-   Sends a `REPLICATE` to get the current position of all streams.
    `REPLICATE` with the `stream_name` and token it wants to subscribe
    from.
 -   On receipt of a `SERVER` command, checks that the server name
    matches the expected server name.
@ -140,14 +137,12 @@ the wire:
    > PING 1490197665618
    < NAME synapse.app.appservice
    < PING 1490197665618
-    < REPLICATE events 1
+    < REPLICATE
-    < REPLICATE backfill 1
+    > POSITION events master 1
-    < REPLICATE caches 1
+    > POSITION backfill master 1
-    > POSITION events 1
+    > POSITION caches master 1
-    > POSITION backfill 1
+    > RDATA caches master 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513]
-    > POSITION caches 1
+    > RDATA events master 14 ["$149019767112vOHxz:localhost:8823",
    > RDATA caches 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513]
    > RDATA events 14 ["$149019767112vOHxz:localhost:8823",
        "!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null]
    < PING 1490197675618
    > ERROR server stopping
@ -158,10 +153,10 @@ position without needing to send data with the `RDATA` command.
 An example of a batched set of `RDATA` is:
-    > RDATA caches batch ["get_user_by_id",["@test:localhost:8823"],1490197670513]
+    > RDATA caches master batch ["get_user_by_id",["@test:localhost:8823"],1490197670513]
-    > RDATA caches batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513]
+    > RDATA caches master batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513]
-    > RDATA caches batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513]
+    > RDATA caches master batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513]
-    > RDATA caches 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513]
+    > RDATA caches master 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513]
 In this case the client shouldn't advance their caches token until it
 sees the the last `RDATA`.
@ -181,9 +176,14 @@ client (C):
 #### POSITION (S)
-   The position of the stream has been updated. Sent to the client
+   On receipt of a POSITION command clients should check if they have missed any
-    after all missing updates for a stream have been sent to the client
+   updates, and if so then fetch them out of band. Sent in response to a
-    and they're now up to date.
+   REPLICATE command (but can happen at any time).
   The POSITION command includes the source of the stream. Currently all streams
   are written by a single process (usually "master"). If fetching missing
   updates via HTTP API, rather than via the DB, then processes should make the
   request to the appropriate process.
 #### ERROR (S, C)
@ -199,24 +199,17 @@ client (C):
 #### REPLICATE (C)
-Asks the server to replicate a given stream. The syntax is:
+Asks the server for the current position of all streams.
 ```
    REPLICATE <stream_name> <token>
 ```
 Where `<token>` may be either:
 * a numeric stream_id to stream updates since (exclusive)
 * `NOW` to stream all subsequent updates.
 The `<stream_name>` is the name of a replication stream to subscribe
 to (see [here](../synapse/replication/tcp/streams/_base.py) for a list
 of streams). It can also be `ALL` to subscribe to all known streams,
 in which case the `<token>` must be set to `NOW`.
 #### USER_SYNC (C)
-   A user has started or stopped syncing
+   A user has started or stopped syncing on this process.
 #### CLEAR_USER_SYNC (C)
   The server should clear all associated user sync data from the worker.
   This is used when a worker is shutting down.
 #### FEDERATION_ACK (C)
@ -230,10 +223,6 @@ in which case the `<token>` must be set to `NOW`.
   Inform the server a cache should be invalidated
 #### SYNC (S, C)
   Used exclusively in tests
 ### REMOTE_SERVER_UP (S, C)
   Inform other processes that a remote server may have come back online.
@ -252,12 +241,12 @@ Each individual cache invalidation results in a row being sent down
 replication, which includes the cache name (the name of the function)
 and they key to invalidate. For example:
-    > RDATA caches 550953771 ["get_user_by_id", ["@bob:example.com"], 1550574873251]
+    > RDATA caches master 550953771 ["get_user_by_id", ["@bob:example.com"], 1550574873251]
 Alternatively, an entire cache can be invalidated by sending down a `null`
 instead of the key. For example:
-    > RDATA caches 550953772 ["get_user_by_id", null, 1550574873252]
+    > RDATA caches master 550953772 ["get_user_by_id", null, 1550574873252]
 However, there are times when a number of caches need to be invalidated
 at the same time with the same key. To reduce traffic we batch those
--- a/docs/turn-howto.md
+++ b/docs/turn-howto.md
@ -11,6 +11,13 @@ TURN server.
 The following sections describe how to install [coturn](<https://github.com/coturn/coturn>) (which implements the TURN REST API) and integrate it with synapse.
 ## Requirements
 For TURN relaying with `coturn` to work, it must be hosted on a server/endpoint with a public IP.
 Hosting TURN behind a NAT (even with appropriate port forwarding) is known to cause issues
 and to often not work.
 ## `coturn` Setup
 ### Initial installation
--- a/docs/workers.md
+++ b/docs/workers.md
@ -1,23 +1,31 @@
 # Scaling synapse via workers
-Synapse has experimental support for splitting out functionality into
+For small instances it recommended to run Synapse in monolith mode (the
-multiple separate python processes, helping greatly with scalability.  These
+default). For larger instances where performance is a concern it can be helpful
 to split out functionality into multiple separate python processes. These
 processes are called 'workers', and are (eventually) intended to scale
 horizontally independently.
-All of the below is highly experimental and subject to change as Synapse evolves,
+Synapse's worker support is under active development and subject to change as
-but documenting it here to help folks needing highly scalable Synapses similar
+we attempt to rapidly scale ever larger Synapse instances. However we are
-to the one running matrix.org!
+documenting it here to help admins needing a highly scalable Synapse instance
 similar to the one running `matrix.org`.
-All processes continue to share the same database instance, and as such, workers
+All processes continue to share the same database instance, and as such,
-only work with postgres based synapse deployments (sharing a single sqlite
+workers only work with PostgreSQL-based Synapse deployments. SQLite should only
-across multiple processes is a recipe for disaster, plus you should be using
+be used for demo purposes and any admin considering workers should already be
-postgres anyway if you care about scalability).
+running PostgreSQL.
-The workers communicate with the master synapse process via a synapse-specific
+## Master/worker communication
-TCP protocol called 'replication' - analogous to MySQL or Postgres style
+
-database replication; feeding a stream of relevant data to the workers so they
+The workers communicate with the master process via a Synapse-specific protocol
-can be kept in sync with the main synapse process and database state.
+called 'replication' (analogous to MySQL- or Postgres-style database
 replication) which feeds a stream of relevant data from the master to the
 workers so they can be kept in sync with the master process and database state.
 Additionally, workers may make HTTP requests to the master, to send information
 in the other direction. Typically this is used for operations which need to
 wait for a reply - such as sending an event.
 ## Configuration
@ -27,55 +35,45 @@ the correct worker, or to the main synapse instance. Note that this includes
 requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md)
 for information on setting up a reverse proxy.
-To enable workers, you need to add two replication listeners to the master
+To enable workers, you need to add *two* replication listeners to the
-synapse, e.g.:
+main Synapse configuration file (`homeserver.yaml`). For example:
 ```yaml
 listeners:
  # The TCP replication port
  - port: 9092
    bind_address: '127.0.0.1'
    type: replication
  # The HTTP replication port
  - port: 9093
    bind_address: '127.0.0.1'
    type: http
    resources:
     - names: [replication]
 ```
 Under **no circumstances** should these replication API listeners be exposed to
-the public internet; it currently implements no authentication whatsoever and is
+the public internet; they have no authentication and are unencrypted.
 unencrypted.
-(Roughly, the TCP port is used for streaming data from the master to the
+You should then create a set of configs for the various worker processes.  Each
-workers, and the HTTP port for the workers to send data to the main
+worker configuration file inherits the configuration of the main homeserver
-synapse process.)
+configuration file.  You can then override configuration specific to that
 worker, e.g. the HTTP listener that it provides (if any); logging
 configuration; etc.  You should minimise the number of overrides though to
 maintain a usable config.
-You then create a set of configs for the various worker processes.  These
+In the config file for each worker, you must specify the type of worker
-should be worker configuration files, and should be stored in a dedicated
+application (`worker_app`). The currently available worker applications are
-subdirectory, to allow synctl to manipulate them. An additional configuration
+listed below. You must also specify the replication endpoints that it should
-for the master synapse process will need to be created because the process will
+talk to on the main synapse process.  `worker_replication_host` should specify
-not be started automatically. That configuration should look like this:
+the host of the main synapse, `worker_replication_port` should point to the TCP
 replication listener port and `worker_replication_http_port` should point to
 the HTTP replication port.
-    worker_app: synapse.app.homeserver
+For example:
    daemonize: true
 Each worker configuration file inherits the configuration of the main homeserver
 configuration file.  You can then override configuration specific to that worker,
 e.g. the HTTP listener that it provides (if any); logging configuration; etc.
 You should minimise the number of overrides though to maintain a usable config.
 You must specify the type of worker application (`worker_app`). The currently
 available worker applications are listed below. You must also specify the
 replication endpoints that it's talking to on the main synapse process.
 `worker_replication_host` should specify the host of the main synapse,
 `worker_replication_port` should point to the TCP replication listener port and
 `worker_replication_http_port` should point to the HTTP replication port.
 Currently, the `event_creator` and `federation_reader` workers require specifying
 `worker_replication_http_port`.
 For instance:
 ```yaml
 worker_app: synapse.app.synchrotron
 # The replication listener on the synapse to talk to.
@ -90,9 +88,8 @@ For instance:
     - names:
       - client
    worker_daemonize: True
    worker_pid_file: /home/matrix/synapse/synchrotron.pid
 worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml
 ```
 ...is a full configuration for a synchrotron worker instance, which will expose a
 plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided
@ -101,7 +98,75 @@ by the main synapse.
 Obviously you should configure your reverse-proxy to route the relevant
 endpoints to the worker (`localhost:8083` in the above example).
-Finally, to actually run your worker-based synapse, you must pass synctl the -a
+Finally, you need to start your worker processes. This can be done with either
 `synctl` or your distribution's preferred service manager such as `systemd`. We
 recommend the use of `systemd` where available: for information on setting up
 `systemd` to start synapse workers, see
 [systemd-with-workers](systemd-with-workers). To use `synctl`, see below.
 ### **Experimental** support for replication over redis
 As of Synapse v1.13.0, it is possible to configure Synapse to send replication
 via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an
 alternative to direct TCP connections to the master: rather than all the
 workers connecting to the master, all the workers and the master connect to
 Redis, which relays replication commands between processes. This can give a
 significant cpu saving on the master and will be a prerequisite for upcoming
 performance improvements.
 Note that this support is currently experimental; you may experience lost
 messages and similar problems! It is strongly recommended that admins setting
 up workers for the first time use direct TCP replication as above.
 To configure Synapse to use Redis:
 1. Install Redis following the normal procedure for your distribution - for
   example, on Debian, `apt install redis-server`. (It is safe to use an
   existing Redis deployment if you have one: we use a pub/sub stream named
   according to the `server_name` of your synapse server.)
 2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1
   localhost 6379` and get a response of `+PONG`.
 3. Install the python prerequisites. If you installed synapse into a
   virtualenv, this can be done with:
   ```sh
   pip install matrix-synapse[redis]
   ```
   The debian packages from matrix.org already include the required
   dependencies.
 4. Add config to the shared configuration (`homeserver.yaml`):
    ```yaml
    redis:
      enabled: true
    ```
    Optional parameters which can go alongside `enabled` are `host`, `port`,
    `password`. Normally none of these are required.
 5. Restart master and all workers.
 Once redis replication is in use, `worker_replication_port` is redundant and
 can be removed from the worker configuration files. Similarly, the
 configuration for the `listener` for the TCP replication port can be removed
 from the main configuration file. Note that the HTTP replication port is
 still required.
 ### Using synctl
 If you want to use `synctl` to manage your synapse processes, you will need to
 create an an additional configuration file for the master synapse process. That
 configuration should look like this:
 ```yaml
 worker_app: synapse.app.homeserver
 ```
 Additionally, each worker app must be configured with the name of a "pid file",
 to which it will write its process ID when it starts. For example, for a
 synchrotron, you might write:
 ```yaml
 worker_pid_file: /home/matrix/synapse/synchrotron.pid
 ```
 Finally, to actually run your worker-based synapse, you must pass synctl the `-a`
 commandline option to tell it to operate on all the worker configurations found
 in the given directory, e.g.:
--- a/scripts-dev/build_debian_packages
+++ b/scripts-dev/build_debian_packages
@ -27,6 +27,7 @@ DISTS = (
    "ubuntu:cosmic",
    "ubuntu:disco",
    "ubuntu:eoan",
    "ubuntu:focal",
 )
 DESC = '''\
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@ -33,6 +33,10 @@ parts:
    python-version: python3
    python-packages:
      - '.[all]'
      - pip
      - setuptools
      - setuptools-scm
      - wheel
    build-packages:
      - libffi-dev
      - libturbojpeg0-dev
--- a/stubs/sortedcontainers/init.pyi
+++ b/stubs/sortedcontainers/init.pyi
@ -0,0 +1,13 @@
 from .sorteddict import (
    SortedDict,
    SortedKeysView,
    SortedItemsView,
    SortedValuesView,
 )
 __all__ = [
    "SortedDict",
    "SortedKeysView",
    "SortedItemsView",
    "SortedValuesView",
 ]
--- a/stubs/sortedcontainers/sorteddict.pyi
+++ b/stubs/sortedcontainers/sorteddict.pyi
@ -0,0 +1,124 @@
 # stub for SortedDict. This is a lightly edited copy of
 # https://github.com/grantjenks/python-sortedcontainers/blob/eea42df1f7bad2792e8da77335ff888f04b9e5ae/sortedcontainers/sorteddict.pyi
 # (from https://github.com/grantjenks/python-sortedcontainers/pull/107)
 from typing import (
    Any,
    Callable,
    Dict,
    Hashable,
    Iterator,
    Iterable,
    ItemsView,
    KeysView,
    List,
    Mapping,
    Optional,
    Sequence,
    Type,
    TypeVar,
    Tuple,
    Union,
    ValuesView,
    overload,
 )
 _T = TypeVar("_T")
 _S = TypeVar("_S")
 _T_h = TypeVar("_T_h", bound=Hashable)
 _KT = TypeVar("_KT", bound=Hashable)  # Key type.
 _VT = TypeVar("_VT")  # Value type.
 _KT_co = TypeVar("_KT_co", covariant=True, bound=Hashable)
 _VT_co = TypeVar("_VT_co", covariant=True)
 _SD = TypeVar("_SD", bound=SortedDict)
 _Key = Callable[[_T], Any]
 class SortedDict(Dict[_KT, _VT]):
    @overload
    def __init__(self, **kwargs: _VT) -> None: ...
    @overload
    def __init__(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
    @overload
    def __init__(
        self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT
    ) -> None: ...
    @overload
    def __init__(self, __key: _Key[_KT], **kwargs: _VT) -> None: ...
    @overload
    def __init__(
        self, __key: _Key[_KT], __map: Mapping[_KT, _VT], **kwargs: _VT
    ) -> None: ...
    @overload
    def __init__(
        self, __key: _Key[_KT], __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT
    ) -> None: ...
    @property
    def key(self) -> Optional[_Key[_KT]]: ...
    @property
    def iloc(self) -> SortedKeysView[_KT]: ...
    def clear(self) -> None: ...
    def __delitem__(self, key: _KT) -> None: ...
    def __iter__(self) -> Iterator[_KT]: ...
    def __reversed__(self) -> Iterator[_KT]: ...
    def __setitem__(self, key: _KT, value: _VT) -> None: ...
    def _setitem(self, key: _KT, value: _VT) -> None: ...
    def copy(self: _SD) -> _SD: ...
    def __copy__(self: _SD) -> _SD: ...
    @classmethod
    @overload
    def fromkeys(cls, seq: Iterable[_T_h]) -> SortedDict[_T_h, None]: ...
    @classmethod
    @overload
    def fromkeys(cls, seq: Iterable[_T_h], value: _S) -> SortedDict[_T_h, _S]: ...
    def keys(self) -> SortedKeysView[_KT]: ...
    def items(self) -> SortedItemsView[_KT, _VT]: ...
    def values(self) -> SortedValuesView[_VT]: ...
    @overload
    def pop(self, key: _KT) -> _VT: ...
    @overload
    def pop(self, key: _KT, default: _T = ...) -> Union[_VT, _T]: ...
    def popitem(self, index: int = ...) -> Tuple[_KT, _VT]: ...
    def peekitem(self, index: int = ...) -> Tuple[_KT, _VT]: ...
    def setdefault(self, key: _KT, default: Optional[_VT] = ...) -> _VT: ...
    @overload
    def update(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
    @overload
    def update(self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None: ...
    @overload
    def update(self, **kwargs: _VT) -> None: ...
    def __reduce__(
        self,
    ) -> Tuple[
        Type[SortedDict[_KT, _VT]], Tuple[Callable[[_KT], Any], List[Tuple[_KT, _VT]]],
    ]: ...
    def __repr__(self) -> str: ...
    def _check(self) -> None: ...
    def islice(
        self, start: Optional[int] = ..., stop: Optional[int] = ..., reverse=bool,
    ) -> Iterator[_KT]: ...
    def bisect_left(self, value: _KT) -> int: ...
    def bisect_right(self, value: _KT) -> int: ...
 class SortedKeysView(KeysView[_KT_co], Sequence[_KT_co]):
    @overload
    def __getitem__(self, index: int) -> _KT_co: ...
    @overload
    def __getitem__(self, index: slice) -> List[_KT_co]: ...
    def __delitem__(self, index: Union[int, slice]) -> None: ...
 class SortedItemsView(  # type: ignore
    ItemsView[_KT_co, _VT_co], Sequence[Tuple[_KT_co, _VT_co]]
 ):
    def __iter__(self) -> Iterator[Tuple[_KT_co, _VT_co]]: ...
    @overload
    def __getitem__(self, index: int) -> Tuple[_KT_co, _VT_co]: ...
    @overload
    def __getitem__(self, index: slice) -> List[Tuple[_KT_co, _VT_co]]: ...
    def __delitem__(self, index: Union[int, slice]) -> None: ...
 class SortedValuesView(ValuesView[_VT_co], Sequence[_VT_co]):
    @overload
    def __getitem__(self, index: int) -> _VT_co: ...
    @overload
    def __getitem__(self, index: slice) -> List[_VT_co]: ...
    def __delitem__(self, index: Union[int, slice]) -> None: ...
--- a/stubs/txredisapi.pyi
+++ b/stubs/txredisapi.pyi
@ -0,0 +1,43 @@
 # -*- coding: utf-8 -*-
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 """Contains *incomplete* type hints for txredisapi.
 """
 from typing import List, Optional, Union
 class RedisProtocol:
    def publish(self, channel: str, message: bytes): ...
 class SubscriberProtocol:
    password: Optional[str]
    def subscribe(self, channels: Union[str, List[str]]): ...
    def connectionMade(self): ...
    def connectionLost(self, reason): ...
 def lazyConnection(
    host: str = ...,
    port: int = ...,
    dbid: Optional[int] = ...,
    reconnect: bool = ...,
    charset: str = ...,
    password: Optional[str] = ...,
    connectTimeout: Optional[int] = ...,
    replyTimeout: Optional[int] = ...,
    convertNumbers: bool = ...,
 ) -> RedisProtocol: ...
 class SubscriberFactory:
    def buildProtocol(self, addr): ...
--- a/synapse/init.py
+++ b/synapse/init.py
@ -36,7 +36,7 @@ try:
 except ImportError:
    pass
-__version__ = "1.12.4"
+__version__ = "1.13.0"
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
--- a/synapse/api/auth.py
+++ b/synapse/api/auth.py
@ -26,16 +26,15 @@ from twisted.internet import defer
 import synapse.logging.opentracing as opentracing
 import synapse.types
 from synapse import event_auth
-from synapse.api.constants import EventTypes, LimitBlockingTypes, Membership, UserTypes
+from synapse.api.auth_blocking import AuthBlocking
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import (
    AuthError,
    Codes,
    InvalidClientTokenError,
    MissingClientTokenError,
    ResourceLimitError,
 )
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.config.server import is_threepid_reserved
 from synapse.events import EventBase
 from synapse.types import StateMap, UserID
 from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache
@ -77,7 +76,11 @@ class Auth(object):
        self.token_cache = LruCache(CACHE_SIZE_FACTOR * 10000)
        register_cache("cache", "token_cache", self.token_cache)
        self._auth_blocking = AuthBlocking(self.hs)
        self._account_validity = hs.config.account_validity
        self._track_appservice_user_ips = hs.config.track_appservice_user_ips
        self._macaroon_secret_key = hs.config.macaroon_secret_key
    @defer.inlineCallbacks
    def check_from_context(self, room_version: str, event, context, do_sig_check=True):
@ -191,7 +194,7 @@ class Auth(object):
                opentracing.set_tag("authenticated_entity", user_id)
                opentracing.set_tag("appservice_id", app_service.id)
-                if ip_addr and self.hs.config.track_appservice_user_ips:
+                if ip_addr and self._track_appservice_user_ips:
                    yield self.store.insert_client_ip(
                        user_id=user_id,
                        access_token=access_token,
@ -454,7 +457,7 @@ class Auth(object):
        # access_tokens include a nonce for uniqueness: any value is acceptable
        v.satisfy_general(lambda c: c.startswith("nonce = "))
-        v.verify(macaroon, self.hs.config.macaroon_secret_key)
+        v.verify(macaroon, self._macaroon_secret_key)
    def _verify_expiry(self, caveat):
        prefix = "time < "
@ -537,8 +540,7 @@ class Auth(object):
        return defer.succeed(auth_ids)
-    @defer.inlineCallbacks
+    async def check_can_change_room_list(self, room_id: str, user: UserID):
    def check_can_change_room_list(self, room_id: str, user: UserID):
        """Determine whether the user is allowed to edit the room's entry in the
        published room list.
@ -547,17 +549,17 @@ class Auth(object):
            user
        """
-        is_admin = yield self.is_server_admin(user)
+        is_admin = await self.is_server_admin(user)
        if is_admin:
            return True
        user_id = user.to_string()
-        yield self.check_user_in_room(room_id, user_id)
+        await self.check_user_in_room(room_id, user_id)
        # We currently require the user is a "moderator" in the room. We do this
        # by checking if they would (theoretically) be able to change the
        # m.room.canonical_alias events
-        power_level_event = yield self.state.get_current_state(
+        power_level_event = await self.state.get_current_state(
            room_id, EventTypes.PowerLevels, ""
        )
@ -664,71 +666,5 @@ class Auth(object):
                % (user_id, room_id),
            )
-    @defer.inlineCallbacks
+    def check_auth_blocking(self, *args, **kwargs):
-    def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
+        return self._auth_blocking.check_auth_blocking(*args, **kwargs)
        """Checks if the user should be rejected for some external reason,
        such as monthly active user limiting or global disable flag
        Args:
            user_id(str|None): If present, checks for presence against existing
                MAU cohort
            threepid(dict|None): If present, checks for presence against configured
                reserved threepid. Used in cases where the user is trying register
                with a MAU blocked server, normally they would be rejected but their
                threepid is on the reserved list. user_id and
                threepid should never be set at the same time.
            user_type(str|None): If present, is used to decide whether to check against
                certain blocking reasons like MAU.
        """
        # Never fail an auth check for the server notices users or support user
        # This can be a problem where event creation is prohibited due to blocking
        if user_id is not None:
            if user_id == self.hs.config.server_notices_mxid:
                return
            if (yield self.store.is_support_user(user_id)):
                return
        if self.hs.config.hs_disabled:
            raise ResourceLimitError(
                403,
                self.hs.config.hs_disabled_message,
                errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
                admin_contact=self.hs.config.admin_contact,
                limit_type=LimitBlockingTypes.HS_DISABLED,
            )
        if self.hs.config.limit_usage_by_mau is True:
            assert not (user_id and threepid)
            # If the user is already part of the MAU cohort or a trial user
            if user_id:
                timestamp = yield self.store.user_last_seen_monthly_active(user_id)
                if timestamp:
                    return
                is_trial = yield self.store.is_trial_user(user_id)
                if is_trial:
                    return
            elif threepid:
                # If the user does not exist yet, but is signing up with a
                # reserved threepid then pass auth check
                if is_threepid_reserved(
                    self.hs.config.mau_limits_reserved_threepids, threepid
                ):
                    return
            elif user_type == UserTypes.SUPPORT:
                # If the user does not exist yet and is of type "support",
                # allow registration. Support users are excluded from MAU checks.
                return
            # Else if there is no room in the MAU bucket, bail
            current_mau = yield self.store.get_monthly_active_count()
            if current_mau >= self.hs.config.max_mau_value:
                raise ResourceLimitError(
                    403,
                    "Monthly Active User Limit Exceeded",
                    admin_contact=self.hs.config.admin_contact,
                    errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
                    limit_type=LimitBlockingTypes.MONTHLY_ACTIVE_USER,
                )
--- a/synapse/api/auth_blocking.py
+++ b/synapse/api/auth_blocking.py
@ -0,0 +1,104 @@
 # -*- coding: utf-8 -*-
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
 from twisted.internet import defer
 from synapse.api.constants import LimitBlockingTypes, UserTypes
 from synapse.api.errors import Codes, ResourceLimitError
 from synapse.config.server import is_threepid_reserved
 logger = logging.getLogger(__name__)
 class AuthBlocking(object):
    def __init__(self, hs):
        self.store = hs.get_datastore()
        self._server_notices_mxid = hs.config.server_notices_mxid
        self._hs_disabled = hs.config.hs_disabled
        self._hs_disabled_message = hs.config.hs_disabled_message
        self._admin_contact = hs.config.admin_contact
        self._max_mau_value = hs.config.max_mau_value
        self._limit_usage_by_mau = hs.config.limit_usage_by_mau
        self._mau_limits_reserved_threepids = hs.config.mau_limits_reserved_threepids
    @defer.inlineCallbacks
    def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
        """Checks if the user should be rejected for some external reason,
        such as monthly active user limiting or global disable flag
        Args:
            user_id(str|None): If present, checks for presence against existing
                MAU cohort
            threepid(dict|None): If present, checks for presence against configured
                reserved threepid. Used in cases where the user is trying register
                with a MAU blocked server, normally they would be rejected but their
                threepid is on the reserved list. user_id and
                threepid should never be set at the same time.
            user_type(str|None): If present, is used to decide whether to check against
                certain blocking reasons like MAU.
        """
        # Never fail an auth check for the server notices users or support user
        # This can be a problem where event creation is prohibited due to blocking
        if user_id is not None:
            if user_id == self._server_notices_mxid:
                return
            if (yield self.store.is_support_user(user_id)):
                return
        if self._hs_disabled:
            raise ResourceLimitError(
                403,
                self._hs_disabled_message,
                errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
                admin_contact=self._admin_contact,
                limit_type=LimitBlockingTypes.HS_DISABLED,
            )
        if self._limit_usage_by_mau is True:
            assert not (user_id and threepid)
            # If the user is already part of the MAU cohort or a trial user
            if user_id:
                timestamp = yield self.store.user_last_seen_monthly_active(user_id)
                if timestamp:
                    return
                is_trial = yield self.store.is_trial_user(user_id)
                if is_trial:
                    return
            elif threepid:
                # If the user does not exist yet, but is signing up with a
                # reserved threepid then pass auth check
                if is_threepid_reserved(self._mau_limits_reserved_threepids, threepid):
                    return
            elif user_type == UserTypes.SUPPORT:
                # If the user does not exist yet and is of type "support",
                # allow registration. Support users are excluded from MAU checks.
                return
            # Else if there is no room in the MAU bucket, bail
            current_mau = yield self.store.get_monthly_active_count()
            if current_mau >= self._max_mau_value:
                raise ResourceLimitError(
                    403,
                    "Monthly Active User Limit Exceeded",
                    admin_contact=self._admin_contact,
                    errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
                    limit_type=LimitBlockingTypes.MONTHLY_ACTIVE_USER,
                )
--- a/synapse/api/constants.py
+++ b/synapse/api/constants.py
@ -61,6 +61,7 @@ class LoginType(object):
    MSISDN = "m.login.msisdn"
    RECAPTCHA = "m.login.recaptcha"
    TERMS = "m.login.terms"
    SSO = "org.matrix.login.sso"
    DUMMY = "m.login.dummy"
    # Only for C/S API v1
@ -96,6 +97,8 @@ class EventTypes(object):
    Retention = "m.room.retention"
    Presence = "m.presence"
 class RejectedReason(object):
    AUTH_ERROR = "auth_error"
--- a/synapse/api/errors.py
+++ b/synapse/api/errors.py
@ -64,6 +64,13 @@ class Codes(object):
    INCOMPATIBLE_ROOM_VERSION = "M_INCOMPATIBLE_ROOM_VERSION"
    WRONG_ROOM_KEYS_VERSION = "M_WRONG_ROOM_KEYS_VERSION"
    EXPIRED_ACCOUNT = "ORG_MATRIX_EXPIRED_ACCOUNT"
    PASSWORD_TOO_SHORT = "M_PASSWORD_TOO_SHORT"
    PASSWORD_NO_DIGIT = "M_PASSWORD_NO_DIGIT"
    PASSWORD_NO_UPPERCASE = "M_PASSWORD_NO_UPPERCASE"
    PASSWORD_NO_LOWERCASE = "M_PASSWORD_NO_LOWERCASE"
    PASSWORD_NO_SYMBOL = "M_PASSWORD_NO_SYMBOL"
    PASSWORD_IN_DICTIONARY = "M_PASSWORD_IN_DICTIONARY"
    WEAK_PASSWORD = "M_WEAK_PASSWORD"
    INVALID_SIGNATURE = "M_INVALID_SIGNATURE"
    USER_DEACTIVATED = "M_USER_DEACTIVATED"
    BAD_ALIAS = "M_BAD_ALIAS"
@ -79,7 +86,14 @@ class CodeMessageException(RuntimeError):
    def __init__(self, code, msg):
        super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
-        self.code = code
+
        # Some calls to this method pass instances of http.HTTPStatus for `code`.
        # While HTTPStatus is a subclass of int, it has magic __str__ methods
        # which emit `HTTPStatus.FORBIDDEN` when converted to a str, instead of `403`.
        # This causes inconsistency in our log lines.
        #
        # To eliminate this behaviour, we convert them to their integer equivalents here.
        self.code = int(code)
        self.msg = msg
@ -439,6 +453,20 @@ class IncompatibleRoomVersionError(SynapseError):
        return cs_error(self.msg, self.errcode, room_version=self._room_version)
 class PasswordRefusedError(SynapseError):
    """A password has been refused, either during password reset/change or registration.
    """
    def __init__(
        self,
        msg="This password doesn't comply with the server's policy",
        errcode=Codes.WEAK_PASSWORD,
    ):
        super(PasswordRefusedError, self).__init__(
            code=400, msg=msg, errcode=errcode,
        )
 class RequestSendFailed(RuntimeError):
    """Sending a HTTP request over federation failed due to not being able to
    talk to the remote server for some reason.
--- a/synapse/app/_base.py
+++ b/synapse/app/_base.py
@ -22,6 +22,7 @@ import sys
 import traceback
 from daemonize import Daemonize
 from typing_extensions import NoReturn
 from twisted.internet import defer, error, reactor
 from twisted.protocols.tls import TLSMemoryBIOFactory
@ -139,9 +140,9 @@ def start_reactor(
            run()
-def quit_with_error(error_string):
+def quit_with_error(error_string: str) -> NoReturn:
    message_lines = error_string.split("\n")
-    line_length = max(len(l) for l in message_lines if len(l) < 80) + 2
+    line_length = max(len(line) for line in message_lines if len(line) < 80) + 2
    sys.stderr.write("*" * line_length + "\n")
    for line in message_lines:
        sys.stderr.write(" %s\n" % (line.rstrip(),))
@ -270,7 +271,7 @@ def start(hs, listeners=None):
        # Start the tracer
        synapse.logging.opentracing.init_tracer(  # type: ignore[attr-defined] # noqa
-            hs.config
+            hs
        )
        # It is now safe to start your Synapse.
@ -316,7 +317,7 @@ def setup_sentry(hs):
        scope.set_tag("matrix_server_name", hs.config.server_name)
        app = hs.config.worker_app if hs.config.worker_app else "synapse.app.homeserver"
-        name = hs.config.worker_name if hs.config.worker_name else "master"
+        name = hs.get_instance_name()
        scope.set_tag("worker_app", app)
        scope.set_tag("worker_name", name)
--- a/synapse/app/admin_cmd.py
+++ b/synapse/app/admin_cmd.py
@ -43,7 +43,6 @@ from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
 from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.server import HomeServer
 from synapse.util.logcontext import LoggingContext
 from synapse.util.versionstring import get_version_string
@ -79,17 +78,6 @@ class AdminCmdServer(HomeServer):
    def start_listening(self, listeners):
        pass
    def build_tcp_replication(self):
        return AdminCmdReplicationHandler(self)
 class AdminCmdReplicationHandler(ReplicationClientHandler):
    async def on_rdata(self, stream_name, token, rows):
        pass
    def get_streams_to_replicate(self):
        return {}
@defer.inlineCallbacks
 def export_data_command(hs, args):
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@ -17,6 +17,9 @@
 import contextlib
 import logging
 import sys
 from typing import Dict, Iterable
 from typing_extensions import ContextManager
 from twisted.internet import defer, reactor
 from twisted.web.resource import NoResource
@ -38,14 +41,14 @@ from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.federation import send_queue
 from synapse.federation.transport.server import TransportLayerServer
-from synapse.handlers.presence import PresenceHandler, get_interested_parties
+from synapse.handlers.presence import BasePresenceHandler, get_interested_parties
 from synapse.http.server import JsonResource
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
 from synapse.http.site import SynapseSite
-from synapse.logging.context import LoggingContext, run_in_background
+from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.metrics.background_process_metrics import run_as_background_process
-from synapse.replication.slave.storage._base import BaseSlavedStore, __func__
+from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
 from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
@ -64,13 +67,25 @@ from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
 from synapse.replication.slave.storage.transactions import SlavedTransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
+from synapse.replication.tcp.client import ReplicationDataHandler
-from synapse.replication.tcp.streams._base import (
+from synapse.replication.tcp.commands import ClearUserSyncsCommand
 from synapse.replication.tcp.streams import (
    AccountDataStream,
    DeviceListsStream,
    GroupServerStream,
    PresenceStream,
    PushersStream,
    PushRulesStream,
    ReceiptsStream,
    TagAccountDataStream,
    ToDeviceStream,
    TypingStream,
 )
 from synapse.replication.tcp.streams.events import (
    EventsStream,
    EventsStreamEventRow,
    EventsStreamRow,
 )
 from synapse.replication.tcp.streams.events import EventsStreamEventRow, EventsStreamRow
 from synapse.rest.admin import register_servlets_for_media_repo
 from synapse.rest.client.v1 import events
 from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
@ -112,12 +127,12 @@ from synapse.storage.data_stores.main.monthly_active_users import (
    MonthlyActiveUsersWorkerStore,
 )
 from synapse.storage.data_stores.main.presence import UserPresenceState
 from synapse.storage.data_stores.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.data_stores.main.user_directory import UserDirectoryStore
 from synapse.types import ReadReceipt
 from synapse.util.async_helpers import Linearizer
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.manhole import manhole
 from synapse.util.stringutils import random_string
 from synapse.util.versionstring import get_version_string
 logger = logging.getLogger("synapse.app.generic_worker")
@ -214,21 +229,31 @@ class KeyUploadServlet(RestServlet):
            return 200, {"one_time_key_counts": result}
 class _NullContextManager(ContextManager[None]):
    """A context manager which does nothing."""
    def __exit__(self, exc_type, exc_val, exc_tb):
        pass
 UPDATE_SYNCING_USERS_MS = 10 * 1000
-class GenericWorkerPresence(object):
+class GenericWorkerPresence(BasePresenceHandler):
    def __init__(self, hs):
        super().__init__(hs)
        self.hs = hs
        self.is_mine_id = hs.is_mine_id
        self.http_client = hs.get_simple_http_client()
        self.store = hs.get_datastore()
        self.user_to_num_current_syncs = {}
        self.clock = hs.get_clock()
        self.notifier = hs.get_notifier()
-        active_presence = self.store.take_presence_startup_info()
+        self._presence_enabled = hs.config.use_presence
-        self.user_to_current_state = {state.user_id: state for state in active_presence}
+
        # The number of ongoing syncs on this process, by user id.
        # Empty if _presence_enabled is false.
        self._user_to_num_current_syncs = {}  # type: Dict[str, int]
        self.notifier = hs.get_notifier()
        self.instance_id = hs.get_instance_id()
        # user_id -> last_sync_ms. Lists the users that have stopped syncing
        # but we haven't notified the master of that yet
@ -238,13 +263,24 @@ class GenericWorkerPresence(object):
            self.send_stop_syncing, UPDATE_SYNCING_USERS_MS
        )
-        self.process_id = random_string(16)
+        hs.get_reactor().addSystemEventTrigger(
-        logger.info("Presence process_id is %r", self.process_id)
+            "before",
            "shutdown",
            run_as_background_process,
            "generic_presence.on_shutdown",
            self._on_shutdown,
        )
    def _on_shutdown(self):
        if self._presence_enabled:
            self.hs.get_tcp_replication().send_command(
                ClearUserSyncsCommand(self.instance_id)
            )
    def send_user_sync(self, user_id, is_syncing, last_sync_ms):
-        if self.hs.config.use_presence:
+        if self._presence_enabled:
            self.hs.get_tcp_replication().send_user_sync(
-                user_id, is_syncing, last_sync_ms
+                self.instance_id, user_id, is_syncing, last_sync_ms
            )
    def mark_as_coming_online(self, user_id):
@ -284,28 +320,33 @@ class GenericWorkerPresence(object):
        # TODO Hows this supposed to work?
        return defer.succeed(None)
-    get_states = __func__(PresenceHandler.get_states)
+    async def user_syncing(
-    get_state = __func__(PresenceHandler.get_state)
+        self, user_id: str, affect_presence: bool
-    current_state_for_users = __func__(PresenceHandler.current_state_for_users)
+    ) -> ContextManager[None]:
        """Record that a user is syncing.
-    def user_syncing(self, user_id, affect_presence):
+        Called by the sync and events servlets to record that a user has connected to
-        if affect_presence:
+        this worker and is waiting for some events.
-            curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
+        """
-            self.user_to_num_current_syncs[user_id] = curr_sync + 1
+        if not affect_presence or not self._presence_enabled:
            return _NullContextManager()
        curr_sync = self._user_to_num_current_syncs.get(user_id, 0)
        self._user_to_num_current_syncs[user_id] = curr_sync + 1
        # If we went from no in flight sync to some, notify replication
-            if self.user_to_num_current_syncs[user_id] == 1:
+        if self._user_to_num_current_syncs[user_id] == 1:
            self.mark_as_coming_online(user_id)
        def _end():
            # We check that the user_id is in user_to_num_current_syncs because
            # user_to_num_current_syncs may have been cleared if we are
            # shutting down.
-            if affect_presence and user_id in self.user_to_num_current_syncs:
+            if user_id in self._user_to_num_current_syncs:
-                self.user_to_num_current_syncs[user_id] -= 1
+                self._user_to_num_current_syncs[user_id] -= 1
                # If we went from one in flight sync to non, notify replication
-                if self.user_to_num_current_syncs[user_id] == 0:
+                if self._user_to_num_current_syncs[user_id] == 0:
                    self.mark_as_going_offline(user_id)
        @contextlib.contextmanager
@ -315,7 +356,7 @@ class GenericWorkerPresence(object):
            finally:
                _end()
-        return defer.succeed(_user_syncing())
+        return _user_syncing()
    @defer.inlineCallbacks
    def notify_from_replication(self, states, stream_id):
@ -350,15 +391,12 @@ class GenericWorkerPresence(object):
        stream_id = token
        yield self.notify_from_replication(states, stream_id)
-    def get_currently_syncing_users(self):
+    def get_currently_syncing_users_for_replication(self) -> Iterable[str]:
        if self.hs.config.use_presence:
        return [
            user_id
-                for user_id, count in self.user_to_num_current_syncs.items()
+            for user_id, count in self._user_to_num_current_syncs.items()
            if count > 0
        ]
        else:
            return set()
 class GenericWorkerTyping(object):
@ -375,12 +413,6 @@ class GenericWorkerTyping(object):
        # map room IDs to sets of users currently typing
        self._room_typing = {}
    def stream_positions(self):
        # We must update this typing token from the response of the previous
        # sync. In particular, the stream id may "reset" back to zero/a low
        # value which we *must* use for the next replication request.
        return {"typing": self._latest_room_serial}
    def process_replication_rows(self, token, rows):
        if self._latest_room_serial > token:
            # The master has gone backwards. To prevent inconsistent data, just
@ -394,11 +426,15 @@ class GenericWorkerTyping(object):
            self._room_serials[row.room_id] = token
            self._room_typing[row.room_id] = row.user_ids
    def get_current_token(self) -> int:
        return self._latest_room_serial
 class GenericWorkerSlavedStore(
    # FIXME(#3714): We need to add UserDirectoryStore as we write directly
    # rather than going via the correct worker.
    UserDirectoryStore,
    UIAuthWorkerStore,
    SlavedDeviceInboxStore,
    SlavedDeviceStore,
    SlavedReceiptsStore,
@ -583,7 +619,7 @@ class GenericWorkerServer(HomeServer):
    def remove_pusher(self, app_id, push_key, user_id):
        self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
-    def build_tcp_replication(self):
+    def build_replication_data_handler(self):
        return GenericWorkerReplicationHandler(self)
    def build_presence_handler(self):
@ -593,14 +629,13 @@ class GenericWorkerServer(HomeServer):
        return GenericWorkerTyping(self)
-class GenericWorkerReplicationHandler(ReplicationClientHandler):
+class GenericWorkerReplicationHandler(ReplicationDataHandler):
    def __init__(self, hs):
        super(GenericWorkerReplicationHandler, self).__init__(hs.get_datastore())
        self.store = hs.get_datastore()
        self.typing_handler = hs.get_typing_handler()
-        # NB this is a SynchrotronPresence, not a normal PresenceHandler
+        self.presence_handler = hs.get_presence_handler()  # type: GenericWorkerPresence
        self.presence_handler = hs.get_presence_handler()
        self.notifier = hs.get_notifier()
        self.notify_pushers = hs.config.start_pushers
@ -611,28 +646,18 @@ class GenericWorkerReplicationHandler(ReplicationClientHandler):
        else:
            self.send_handler = None
-    async def on_rdata(self, stream_name, token, rows):
+    async def on_rdata(self, stream_name, instance_name, token, rows):
-        await super(GenericWorkerReplicationHandler, self).on_rdata(
+        await super().on_rdata(stream_name, instance_name, token, rows)
-            stream_name, token, rows
+        await self._process_and_notify(stream_name, instance_name, token, rows)
        )
        run_in_background(self.process_and_notify, stream_name, token, rows)
-    def get_streams_to_replicate(self):
+    async def _process_and_notify(self, stream_name, instance_name, token, rows):
        args = super(GenericWorkerReplicationHandler, self).get_streams_to_replicate()
        args.update(self.typing_handler.stream_positions())
        if self.send_handler:
            args.update(self.send_handler.stream_positions())
        return args
    def get_currently_syncing_users(self):
        return self.presence_handler.get_currently_syncing_users()
    async def process_and_notify(self, stream_name, token, rows):
        try:
            if self.send_handler:
-                self.send_handler.process_replication_rows(stream_name, token, rows)
+                await self.send_handler.process_replication_rows(
                    stream_name, token, rows
                )
-            if stream_name == "events":
+            if stream_name == EventsStream.NAME:
                # We shouldn't get multiple rows per token for events stream, so
                # we don't need to optimise this for multiple rows.
                for row in rows:
@ -655,43 +680,44 @@ class GenericWorkerReplicationHandler(ReplicationClientHandler):
                    )
                await self.pusher_pool.on_new_notifications(token, token)
-            elif stream_name == "push_rules":
+            elif stream_name == PushRulesStream.NAME:
                self.notifier.on_new_event(
                    "push_rules_key", token, users=[row.user_id for row in rows]
                )
-            elif stream_name in ("account_data", "tag_account_data"):
+            elif stream_name in (AccountDataStream.NAME, TagAccountDataStream.NAME):
                self.notifier.on_new_event(
                    "account_data_key", token, users=[row.user_id for row in rows]
                )
-            elif stream_name == "receipts":
+            elif stream_name == ReceiptsStream.NAME:
                self.notifier.on_new_event(
                    "receipt_key", token, rooms=[row.room_id for row in rows]
                )
                await self.pusher_pool.on_new_receipts(
                    token, token, {row.room_id for row in rows}
                )
-            elif stream_name == "typing":
+            elif stream_name == TypingStream.NAME:
                self.typing_handler.process_replication_rows(token, rows)
                self.notifier.on_new_event(
                    "typing_key", token, rooms=[row.room_id for row in rows]
                )
-            elif stream_name == "to_device":
+            elif stream_name == ToDeviceStream.NAME:
                entities = [row.entity for row in rows if row.entity.startswith("@")]
                if entities:
                    self.notifier.on_new_event("to_device_key", token, users=entities)
-            elif stream_name == "device_lists":
+            elif stream_name == DeviceListsStream.NAME:
                all_room_ids = set()
                for row in rows:
-                    room_ids = await self.store.get_rooms_for_user(row.user_id)
+                    if row.entity.startswith("@"):
                        room_ids = await self.store.get_rooms_for_user(row.entity)
                        all_room_ids.update(room_ids)
                self.notifier.on_new_event("device_list_key", token, rooms=all_room_ids)
-            elif stream_name == "presence":
+            elif stream_name == PresenceStream.NAME:
                await self.presence_handler.process_replication_rows(token, rows)
-            elif stream_name == "receipts":
+            elif stream_name == GroupServerStream.NAME:
                self.notifier.on_new_event(
                    "groups_key", token, users=[row.user_id for row in rows]
                )
-            elif stream_name == "pushers":
+            elif stream_name == PushersStream.NAME:
                for row in rows:
                    if row.deleted:
                        self.stop_pusher(row.user_id, row.app_id, row.pushkey)
@ -758,15 +784,12 @@ class FederationSenderHandler(object):
    def wake_destination(self, server: str):
        self.federation_sender.wake_destination(server)
-    def stream_positions(self):
+    async def process_replication_rows(self, stream_name, token, rows):
        return {"federation": self.federation_position}
    def process_replication_rows(self, stream_name, token, rows):
        # The federation stream contains things that we want to send out, e.g.
        # presence, typing, etc.
        if stream_name == "federation":
            send_queue.process_rows_for_federation(self.federation_sender, rows)
-            run_in_background(self.update_token, token)
+            await self.update_token(token)
        # We also need to poke the federation sender when new events happen
        elif stream_name == "events":
@ -774,13 +797,14 @@ class FederationSenderHandler(object):
        # ... and when new receipts happen
        elif stream_name == ReceiptsStream.NAME:
-            run_as_background_process(
+            await self._on_new_receipts(rows)
                "process_receipts_for_federation", self._on_new_receipts, rows
            )
        # ... as well as device updates and messages
        elif stream_name == DeviceListsStream.NAME:
-            hosts = {row.destination for row in rows}
+            # The entities are either user IDs (starting with '@') whose devices
            # have changed, or remote servers that we need to tell about
            # changes.
            hosts = {row.entity for row in rows if not row.entity.startswith("@")}
            for host in hosts:
                self.federation_sender.send_device_messages(host)
@ -795,7 +819,7 @@ class FederationSenderHandler(object):
    async def _on_new_receipts(self, rows):
        """
        Args:
-            rows (iterable[synapse.replication.tcp.streams.ReceiptsStreamRow]):
+            rows (Iterable[synapse.replication.tcp.streams.ReceiptsStream.ReceiptsStreamRow]):
                new receipts to be processed
        """
        for receipt in rows:
@ -920,17 +944,22 @@ def start(config_options):
    synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
-    ss = GenericWorkerServer(
+    hs = GenericWorkerServer(
        config.server_name,
        config=config,
        version_string="Synapse/" + get_version_string(synapse),
    )
-    setup_logging(ss, config, use_worker_options=True)
+    setup_logging(hs, config, use_worker_options=True)
    hs.setup()
    # Ensure the replication streamer is always started in case we write to any
    # streams. Will no-op if no streams can be written to by this worker.
    hs.get_replication_streamer()
    ss.setup()
    reactor.addSystemEventTrigger(
-        "before", "startup", _base.start, ss, config.worker_listeners
+        "before", "startup", _base.start, hs, config.worker_listeners
    )
    _base.start_worker_reactor("synapse-generic-worker", config)
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@ -241,16 +241,26 @@ class SynapseHomeServer(HomeServer):
            resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)
        if name == "webclient":
-            webclient_path = self.get_config().web_client_location
+            webclient_loc = self.get_config().web_client_location
-            if webclient_path is None:
+            if webclient_loc is None:
                logger.warning(
                    "Not enabling webclient resource, as web_client_location is unset."
                )
            elif webclient_loc.startswith("http://") or webclient_loc.startswith(
                "https://"
            ):
                resources[WEB_CLIENT_PREFIX] = RootRedirect(webclient_loc)
            else:
                logger.warning(
                    "Running webclient on the same domain is not recommended: "
                    "https://github.com/matrix-org/synapse#security-note - "
                    "after you move webclient to different host you can set "
                    "web_client_location to its full URL to enable redirection."
                )
                # GZip is disabled here due to
                # https://twistedmatrix.com/trac/ticket/7678
-                resources[WEB_CLIENT_PREFIX] = File(webclient_path)
+                resources[WEB_CLIENT_PREFIX] = File(webclient_loc)
        if name == "metrics" and self.get_config().enable_metrics:
            resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
@ -263,6 +273,12 @@ class SynapseHomeServer(HomeServer):
    def start_listening(self, listeners):
        config = self.get_config()
        if config.redis_enabled:
            # If redis is enabled we connect via the replication command handler
            # in the same way as the workers (since we're effectively a client
            # rather than a server).
            self.get_tcp_replication().start_replication(self)
        for listener in listeners:
            if listener["type"] == "http":
                self._listening_services.extend(self._listener_http(config, listener))
--- a/synapse/config/_base.py
+++ b/synapse/config/_base.py
@ -294,7 +294,6 @@ class RootConfig(object):
        report_stats=None,
        open_private_ports=False,
        listeners=None,
        database_conf=None,
        tls_certificate_path=None,
        tls_private_key_path=None,
        acme_domain=None,
@ -367,7 +366,6 @@ class RootConfig(object):
                report_stats=report_stats,
                open_private_ports=open_private_ports,
                listeners=listeners,
                database_conf=database_conf,
                tls_certificate_path=tls_certificate_path,
                tls_private_key_path=tls_private_key_path,
                acme_domain=acme_domain,
@ -470,8 +468,8 @@ class RootConfig(object):
        Returns: Config object, or None if --generate-config or --generate-keys was set
        """
-        config_parser = argparse.ArgumentParser(add_help=False)
+        parser = argparse.ArgumentParser(description=description)
-        config_parser.add_argument(
+        parser.add_argument(
            "-c",
            "--config-path",
            action="append",
@ -480,7 +478,7 @@ class RootConfig(object):
            " may specify directories containing *.yaml files.",
        )
-        generate_group = config_parser.add_argument_group("Config generation")
+        generate_group = parser.add_argument_group("Config generation")
        generate_group.add_argument(
            "--generate-config",
            action="store_true",
@ -528,12 +526,13 @@ class RootConfig(object):
            ),
        )
-        config_args, remaining_args = config_parser.parse_known_args(argv)
+        cls.invoke_all_static("add_arguments", parser)
        config_args = parser.parse_args(argv)
        config_files = find_config_files(search_paths=config_args.config_path)
        if not config_files:
-            config_parser.error(
+            parser.error(
                "Must supply a config file.\nA config file can be automatically"
                ' generated using "--generate-config -H SERVER_NAME'
                ' -c CONFIG-FILE"'
@ -552,7 +551,7 @@ class RootConfig(object):
        if config_args.generate_config:
            if config_args.report_stats is None:
-                config_parser.error(
+                parser.error(
                    "Please specify either --report-stats=yes or --report-stats=no\n\n"
                    + MISSING_REPORT_STATS_SPIEL
                )
@ -611,15 +610,6 @@ class RootConfig(object):
                )
                generate_missing_configs = True
        parser = argparse.ArgumentParser(
            parents=[config_parser],
            description=description,
            formatter_class=argparse.RawDescriptionHelpFormatter,
        )
        obj.invoke_all_static("add_arguments", parser)
        args = parser.parse_args(remaining_args)
        config_dict = read_config_files(config_files)
        if generate_missing_configs:
            obj.generate_missing_files(config_dict, config_dir_path)
@ -628,7 +618,7 @@ class RootConfig(object):
        obj.parse_config_dict(
            config_dict, config_dir_path=config_dir_path, data_dir_path=data_dir_path
        )
-        obj.invoke_all("read_arguments", args)
+        obj.invoke_all("read_arguments", config_args)
        return obj
@ -667,6 +657,12 @@ def read_config_files(config_files):
    for config_file in config_files:
        with open(config_file) as file_stream:
            yaml_config = yaml.safe_load(file_stream)
        if not isinstance(yaml_config, dict):
            err = "File %r is empty or doesn't parse into a key-value map. IGNORING."
            print(err % (config_file,))
            continue
        specified_config.update(yaml_config)
    if "server_name" not in specified_config:
--- a/synapse/config/captcha.py
+++ b/synapse/config/captcha.py
@ -24,7 +24,6 @@ class CaptchaConfig(Config):
        self.enable_registration_captcha = config.get(
            "enable_registration_captcha", False
        )
        self.captcha_bypass_secret = config.get("captcha_bypass_secret")
        self.recaptcha_siteverify_api = config.get(
            "recaptcha_siteverify_api",
            "https://www.recaptcha.net/recaptcha/api/siteverify",
@ -49,10 +48,6 @@ class CaptchaConfig(Config):
        #
        #enable_registration_captcha: false
        # A secret key used to bypass the captcha test entirely.
        #
        #captcha_bypass_secret: "YOUR_SECRET_HERE"
        # The API endpoint to use for verifying m.login.recaptcha responses.
        #
        #recaptcha_siteverify_api: "https://www.recaptcha.net/recaptcha/api/siteverify"
--- a/synapse/config/database.py
+++ b/synapse/config/database.py
@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@ -14,14 +15,65 @@
 # limitations under the License.
 import logging
 import os
 from textwrap import indent
 import yaml
 from synapse.config._base import Config, ConfigError
 logger = logging.getLogger(__name__)
 NON_SQLITE_DATABASE_PATH_WARNING = """\
 Ignoring 'database_path' setting: not using a sqlite3 database.
 --------------------------------------------------------------------------------
 """
 DEFAULT_CONFIG = """\
 ## Database ##
 # The 'database' setting defines the database that synapse uses to store all of
 # its data.
 #
 # 'name' gives the database engine to use: either 'sqlite3' (for SQLite) or
 # 'psycopg2' (for PostgreSQL).
 #
 # 'args' gives options which are passed through to the database engine,
 # except for options starting 'cp_', which are used to configure the Twisted
 # connection pool. For a reference to valid arguments, see:
 #   * for sqlite: https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
 #   * for postgres: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
 #   * for the connection pool: https://twistedmatrix.com/documents/current/api/twisted.enterprise.adbapi.ConnectionPool.html#__init__
 #
 #
 # Example SQLite configuration:
 #
 #database:
 #  name: sqlite3
 #  args:
 #    database: /path/to/homeserver.db
 #
 #
 # Example Postgres configuration:
 #
 #database:
 #  name: psycopg2
 #  args:
 #    user: synapse
 #    password: secretpassword
 #    database: synapse
 #    host: localhost
 #    cp_min: 5
 #    cp_max: 10
 #
 # For more information on using Synapse with Postgres, see `docs/postgres.md`.
 #
 database:
  name: sqlite3
  args:
    database: %(database_path)s
 # Number of events to cache in memory.
 #
 #event_cache_size: 10K
 """
 class DatabaseConnectionConfig:
    """Contains the connection config for a particular database.
@ -36,10 +88,12 @@ class DatabaseConnectionConfig:
    """
    def __init__(self, name: str, db_config: dict):
-        if db_config["name"] not in ("sqlite3", "psycopg2"):
+        db_engine = db_config.get("name", "sqlite3")
            raise ConfigError("Unsupported database type %r" % (db_config["name"],))
-        if db_config["name"] == "sqlite3":
+        if db_engine not in ("sqlite3", "psycopg2"):
            raise ConfigError("Unsupported database type %r" % (db_engine,))
        if db_engine == "sqlite3":
            db_config.setdefault("args", {}).update(
                {"cp_min": 1, "cp_max": 1, "check_same_thread": False}
            )
@ -56,6 +110,11 @@ class DatabaseConnectionConfig:
 class DatabaseConfig(Config):
    section = "database"
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.databases = []
    def read_config(self, config, **kwargs):
        self.event_cache_size = self.parse_size(config.get("event_cache_size", "10K"))
@ -76,12 +135,13 @@ class DatabaseConfig(Config):
        multi_database_config = config.get("databases")
        database_config = config.get("database")
        database_path = config.get("database_path")
        if multi_database_config and database_config:
-            raise ConfigError("Can't specify both 'database' and 'datbases' in config")
+            raise ConfigError("Can't specify both 'database' and 'databases' in config")
        if multi_database_config:
-            if config.get("database_path"):
+            if database_path:
                raise ConfigError("Can't specify 'database_path' with 'databases'")
            self.databases = [
@ -89,65 +149,55 @@ class DatabaseConfig(Config):
                for name, db_conf in multi_database_config.items()
            ]
-        else:
+        if database_config:
            if database_config is None:
                database_config = {"name": "sqlite3", "args": {}}
            self.databases = [DatabaseConnectionConfig("master", database_config)]
-            self.set_databasepath(config.get("database_path"))
+        if database_path:
            if self.databases and self.databases[0].name != "sqlite3":
                logger.warning(NON_SQLITE_DATABASE_PATH_WARNING)
                return
-    def generate_config_section(self, data_dir_path, database_conf, **kwargs):
+            database_config = {"name": "sqlite3", "args": {}}
-        if not database_conf:
+            self.databases = [DatabaseConnectionConfig("master", database_config)]
-            database_path = os.path.join(data_dir_path, "homeserver.db")
+            self.set_databasepath(database_path)
            database_conf = (
                """# The database engine name
          name: "sqlite3"
          # Arguments to pass to the engine
          args:
            # Path to the database
            database: "%(database_path)s"
            """
                % locals()
            )
        else:
            database_conf = indent(yaml.dump(database_conf), " " * 10).lstrip()
-        return (
+    def generate_config_section(self, data_dir_path, **kwargs):
-            """\
+        return DEFAULT_CONFIG % {
-        ## Database ##
+            "database_path": os.path.join(data_dir_path, "homeserver.db")
-
+        }
        database:
          %(database_conf)s
        # Number of events to cache in memory.
        #
        #event_cache_size: 10K
        """
            % locals()
        )
    def read_arguments(self, args):
        """
        Cases for the cli input:
          - If no databases are configured and no database_path is set, raise.
          - No databases and only database_path available ==> sqlite3 db.
          - If there are multiple databases and a database_path raise an error.
          - If the database set in the config file is sqlite then
            overwrite with the command line argument.
        """
        if args.database_path is None:
            if not self.databases:
                raise ConfigError("No database config provided")
            return
        if len(self.databases) == 0:
            database_config = {"name": "sqlite3", "args": {}}
            self.databases = [DatabaseConnectionConfig("master", database_config)]
            self.set_databasepath(args.database_path)
            return
        if self.get_single_database().name == "sqlite3":
            self.set_databasepath(args.database_path)
        else:
            logger.warning(NON_SQLITE_DATABASE_PATH_WARNING)
    def set_databasepath(self, database_path):
        if database_path is None:
            return
        if database_path != ":memory:":
            database_path = self.abspath(database_path)
-        # We only support setting a database path if we have a single sqlite3
+        self.databases[0].config["args"]["database"] = database_path
        # database.
        if len(self.databases) != 1:
            raise ConfigError("Cannot specify 'database_path' with multiple databases")
        database = self.get_single_database()
        if database.config["name"] != "sqlite3":
            # We don't raise here as we haven't done so before for this case.
            logger.warn("Ignoring 'database_path' for non-sqlite3 database")
            return
        database.config["args"]["database"] = database_path
    @staticmethod
    def add_arguments(parser):
@ -162,7 +212,7 @@ class DatabaseConfig(Config):
    def get_single_database(self) -> DatabaseConnectionConfig:
        """Returns the database if there is only one, useful for e.g. tests
        """
-        if len(self.databases) != 1:
+        if not self.databases:
            raise Exception("More than one database exists")
        return self.databases[0]
--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@ -108,9 +108,14 @@ class EmailConfig(Config):
            if self.trusted_third_party_id_servers:
                # XXX: It's a little confusing that account_threepid_delegate_email is modified
                # both in RegistrationConfig and here. We should factor this bit out
-                self.account_threepid_delegate_email = self.trusted_third_party_id_servers[
+
-                    0
+                first_trusted_identity_server = self.trusted_third_party_id_servers[0]
-                ]  # type: Optional[str]
+
                # trusted_third_party_id_servers does not contain a scheme whereas
                # account_threepid_delegate_email is expected to. Presume https
                self.account_threepid_delegate_email = (
                    "https://" + first_trusted_identity_server
                )  # type: Optional[str]
                self.using_identity_server_from_trusted_list = True
            else:
                raise ConfigError(
--- a/synapse/config/homeserver.py
+++ b/synapse/config/homeserver.py
@ -31,6 +31,7 @@ from .password import PasswordConfig
 from .password_auth_providers import PasswordAuthProviderConfig
 from .push import PushConfig
 from .ratelimiting import RatelimitConfig
 from .redis import RedisConfig
 from .registration import RegistrationConfig
 from .repository import ContentRepositoryConfig
 from .room_directory import RoomDirectoryConfig
@ -82,4 +83,5 @@ class HomeServerConfig(RootConfig):
        RoomDirectoryConfig,
        ThirdPartyRulesConfig,
        TracerConfig,
        RedisConfig,
    ]
--- a/synapse/config/password.py
+++ b/synapse/config/password.py
@ -31,6 +31,10 @@ class PasswordConfig(Config):
        self.password_localdb_enabled = password_config.get("localdb_enabled", True)
        self.password_pepper = password_config.get("pepper", "")
        # Password policy
        self.password_policy = password_config.get("policy") or {}
        self.password_policy_enabled = self.password_policy.get("enabled", False)
    def generate_config_section(self, config_dir_path, server_name, **kwargs):
        return """\
        password_config:
@ -48,4 +52,39 @@ class PasswordConfig(Config):
           # DO NOT CHANGE THIS AFTER INITIAL SETUP!
           #
           #pepper: "EVEN_MORE_SECRET"
           # Define and enforce a password policy. Each parameter is optional.
           # This is an implementation of MSC2000.
           #
           policy:
              # Whether to enforce the password policy.
              # Defaults to 'false'.
              #
              #enabled: true
              # Minimum accepted length for a password.
              # Defaults to 0.
              #
              #minimum_length: 15
              # Whether a password must contain at least one digit.
              # Defaults to 'false'.
              #
              #require_digit: true
              # Whether a password must contain at least one symbol.
              # A symbol is any character that's not a number or a letter.
              # Defaults to 'false'.
              #
              #require_symbol: true
              # Whether a password must contain at least one lowercase letter.
              # Defaults to 'false'.
              #
              #require_lowercase: true
              # Whether a password must contain at least one lowercase letter.
              # Defaults to 'false'.
              #
              #require_uppercase: true
        """
--- a/synapse/config/password_auth_providers.py
+++ b/synapse/config/password_auth_providers.py
@ -35,7 +35,7 @@ class PasswordAuthProviderConfig(Config):
        if ldap_config.get("enabled", False):
            providers.append({"module": LDAP_PROVIDER, "config": ldap_config})
-        providers.extend(config.get("password_providers", []))
+        providers.extend(config.get("password_providers") or [])
        for provider in providers:
            mod_name = provider["module"]
@ -52,7 +52,19 @@ class PasswordAuthProviderConfig(Config):
    def generate_config_section(self, **kwargs):
        return """\
-        #password_providers:
+        # Password providers allow homeserver administrators to integrate
        # their Synapse installation with existing authentication methods
        # ex. LDAP, external tokens, etc.
        #
        # For more information and known implementations, please see
        # https://github.com/matrix-org/synapse/blob/master/docs/password_auth_providers.md
        #
        # Note: instances wishing to use SAML or CAS authentication should
        # instead use the `saml2_config` or `cas_config` options,
        # respectively.
        #
        password_providers:
        #    # Example config for an LDAP auth provider
        #    - module: "ldap_auth_provider.LdapAuthProvider"
        #      config:
        #        enabled: true
--- a/synapse/config/redis.py
+++ b/synapse/config/redis.py
@ -0,0 +1,34 @@
 # -*- coding: utf-8 -*-
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 from synapse.config._base import Config
 from synapse.python_dependencies import check_requirements
 class RedisConfig(Config):
    section = "redis"
    def read_config(self, config, **kwargs):
        redis_config = config.get("redis", {})
        self.redis_enabled = redis_config.get("enabled", False)
        if not self.redis_enabled:
            return
        check_requirements("redis")
        self.redis_host = redis_config.get("host", "localhost")
        self.redis_port = redis_config.get("port", 6379)
        self.redis_password = redis_config.get("password")
--- a/synapse/config/registration.py
+++ b/synapse/config/registration.py
@ -129,6 +129,10 @@ class RegistrationConfig(Config):
                raise ConfigError("Invalid auto_join_rooms entry %s" % (room_alias,))
        self.autocreate_auto_join_rooms = config.get("autocreate_auto_join_rooms", True)
        self.enable_set_displayname = config.get("enable_set_displayname", True)
        self.enable_set_avatar_url = config.get("enable_set_avatar_url", True)
        self.enable_3pid_changes = config.get("enable_3pid_changes", True)
        self.disable_msisdn_registration = config.get(
            "disable_msisdn_registration", False
        )
@ -330,6 +334,29 @@ class RegistrationConfig(Config):
            #email: https://example.com     # Delegate email sending to example.com
            #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
        # Whether users are allowed to change their displayname after it has
        # been initially set. Useful when provisioning users based on the
        # contents of a third-party directory.
        #
        # Does not apply to server administrators. Defaults to 'true'
        #
        #enable_set_displayname: false
        # Whether users are allowed to change their avatar after it has been
        # initially set. Useful when provisioning users based on the contents
        # of a third-party directory.
        #
        # Does not apply to server administrators. Defaults to 'true'
        #
        #enable_set_avatar_url: false
        # Whether users can change the 3PIDs associated with their accounts
        # (email address and msisdn).
        #
        # Defaults to 'true'
        #
        #enable_3pid_changes: false
        # Users who register on this homeserver will automatically be joined
        # to these rooms
        #
--- a/synapse/config/repository.py
+++ b/synapse/config/repository.py
@ -192,6 +192,10 @@ class ContentRepositoryConfig(Config):
            self.url_preview_url_blacklist = config.get("url_preview_url_blacklist", ())
            self.url_preview_accept_language = config.get(
                "url_preview_accept_language"
            ) or ["en"]
    def generate_config_section(self, data_dir_path, **kwargs):
        media_store = os.path.join(data_dir_path, "media_store")
        uploads_path = os.path.join(data_dir_path, "uploads")
@ -220,12 +224,11 @@ class ContentRepositoryConfig(Config):
        #
        #media_storage_providers:
        #  - module: file_system
-        #    # Whether to write new local files.
+        #    # Whether to store newly uploaded local files
        #    store_local: false
-        #    # Whether to write new remote media
+        #    # Whether to store newly downloaded remote files
        #    store_remote: false
-        #    # Whether to block upload requests waiting for write to this
+        #    # Whether to wait for successful storage for local uploads
        #    # provider to complete
        #    store_synchronous: false
        #    config:
        #       directory: /mnt/some/other/directory
@ -329,6 +332,31 @@ class ContentRepositoryConfig(Config):
        # The largest allowed URL preview spidering size in bytes
        #
        #max_spider_size: 10M
        # A list of values for the Accept-Language HTTP header used when
        # downloading webpages during URL preview generation. This allows
        # Synapse to specify the preferred languages that URL previews should
        # be in when communicating with remote servers.
        #
        # Each value is a IETF language tag; a 2-3 letter identifier for a
        # language, optionally followed by subtags separated by '-', specifying
        # a country or region variant.
        #
        # Multiple values can be provided, and a weight can be added to each by
        # using quality value syntax (;q=). '*' translates to any language.
        #
        # Defaults to "en".
        #
        # Example:
        #
        # url_preview_accept_language:
        #   - en-UK
        #   - en-US;q=0.9
        #   - fr;q=0.8
        #   - *;q=0.7
        #
        url_preview_accept_language:
        #   - en
        """
            % locals()
        )
--- a/synapse/config/server.py
+++ b/synapse/config/server.py
@ -505,10 +505,24 @@ class ServerConfig(Config):
            "cleanup_extremities_with_dummy_events", True
        )
        # The number of forward extremities in a room needed to send a dummy event.
        self.dummy_events_threshold = config.get("dummy_events_threshold", 10)
        self.enable_ephemeral_messages = config.get("enable_ephemeral_messages", False)
        # Inhibits the /requestToken endpoints from returning an error that might leak
        # information about whether an e-mail address is in use or not on this
        # homeserver, and instead return a 200 with a fake sid if this kind of error is
        # met, without sending anything.
        # This is a compromise between sending an email, which could be a spam vector,
        # and letting the client know which email address is bound to an account and
        # which one isn't.
        self.request_token_inhibit_3pid_errors = config.get(
            "request_token_inhibit_3pid_errors", False,
        )
    def has_tls_listener(self) -> bool:
-        return any(l["tls"] for l in self.listeners)
+        return any(listener["tls"] for listener in self.listeners)
    def generate_config_section(
        self, server_name, data_dir_path, open_private_ports, listeners, **kwargs
@ -604,10 +618,15 @@ class ServerConfig(Config):
        #
        pid_file: %(pid_file)s
-        # The path to the web client which will be served at /_matrix/client/
+        # The absolute URL to the web client which /_matrix/client will redirect
-        # if 'webclient' is configured under the 'listeners' configuration.
+        # to if 'webclient' is configured under the 'listeners' configuration.
        #
-        #web_client_location: "/path/to/web/root"
+        # This option can be also set to the filesystem path to the web client
        # which will be served at /_matrix/client/ if 'webclient' is configured
        # under the 'listeners' configuration, however this is a security risk:
        # https://github.com/matrix-org/synapse#security-note
        #
        #web_client_location: https://riot.example.com/
        # The public-facing base URL that clients use to access this HS
        # (not including _matrix/...). This is the same URL a user would
@ -807,6 +826,18 @@ class ServerConfig(Config):
          #  bind_addresses: ['::1', '127.0.0.1']
          #  type: manhole
        # Forward extremities can build up in a room due to networking delays between
        # homeservers. Once this happens in a large room, calculation of the state of
        # that room can become quite expensive. To mitigate this, once the number of
        # forward extremities reaches a given threshold, Synapse will send an
        # org.matrix.dummy_event event, which will reduce the forward extremities
        # in the room.
        #
        # This setting defines the threshold (i.e. number of forward extremities in the
        # room) at which dummy events are sent. The default value is 10.
        #
        #dummy_events_threshold: 5
        ## Homeserver blocking ##
@ -967,6 +998,16 @@ class ServerConfig(Config):
          #  - shortest_max_lifetime: 3d
          #    longest_max_lifetime: 1y
          #    interval: 1d
        # Inhibits the /requestToken endpoints from returning an error that might leak
        # information about whether an e-mail address is in use or not on this
        # homeserver.
        # Note that for some endpoints the error situation is the e-mail already being
        # used, and for others the error is entering the e-mail being unused.
        # If this option is enabled, instead of returning an error, these endpoints will
        # act as if no error happened and return a fake session ID ('sid') to clients.
        #
        #request_token_inhibit_3pid_errors: true
        """
            % locals()
        )
--- a/synapse/config/sso.py
+++ b/synapse/config/sso.py
@ -12,6 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import os
 from typing import Any, Dict
 import pkg_resources
@ -36,9 +37,32 @@ class SSOConfig(Config):
            template_dir = pkg_resources.resource_filename("synapse", "res/templates",)
        self.sso_redirect_confirm_template_dir = template_dir
        self.sso_account_deactivated_template = self.read_file(
            os.path.join(
                self.sso_redirect_confirm_template_dir, "sso_account_deactivated.html"
            ),
            "sso_account_deactivated_template",
        )
        self.sso_auth_success_template = self.read_file(
            os.path.join(
                self.sso_redirect_confirm_template_dir, "sso_auth_success.html"
            ),
            "sso_auth_success_template",
        )
        self.sso_client_whitelist = sso_config.get("client_whitelist") or []
        # Attempt to also whitelist the server's login fallback, since that fallback sets
        # the redirect URL to itself (so it can process the login token then return
        # gracefully to the client). This would make it pointless to ask the user for
        # confirmation, since the URL the confirmation page would be showing wouldn't be
        # the client's.
        # public_baseurl is an optional setting, so we only add the fallback's URL to the
        # list if it's provided (because we can't figure out what that URL is otherwise).
        if self.public_baseurl:
            login_fallback_url = self.public_baseurl + "_matrix/static/client/login"
            self.sso_client_whitelist.append(login_fallback_url)
    def generate_config_section(self, **kwargs):
        return """\
        # Additional settings to use with single-sign on systems such as SAML2 and CAS.
@ -54,6 +78,10 @@ class SSOConfig(Config):
            # phishing attacks from evil.site. To avoid this, include a slash after the
            # hostname: "https://my.client/".
            #
            # If public_baseurl is set, then the login fallback page (used by clients
            # that don't natively support the required login flows) is whitelisted in
            # addition to any URLs in this list.
            #
            # By default, this list is empty.
            #
            #client_whitelist:
@ -85,6 +113,30 @@ class SSOConfig(Config):
            #
            #     * server_name: the homeserver's name.
            #
            # * HTML page which notifies the user that they are authenticating to confirm
            #   an operation on their account during the user interactive authentication
            #   process: 'sso_auth_confirm.html'.
            #
            #   When rendering, this template is given the following variables:
            #     * redirect_url: the URL the user is about to be redirected to. Needs
            #                     manual escaping (see
            #                     https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping).
            #
            #     * description: the operation which the user is being asked to confirm
            #
            # * HTML page shown after a successful user interactive authentication session:
            #   'sso_auth_success.html'.
            #
            #   Note that this page must include the JavaScript which notifies of a successful authentication
            #   (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
            #
            #   This template has no additional variables.
            #
            # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
            #   attempts to login: 'sso_account_deactivated.html'.
            #
            #   This template has no additional variables.
            #
            # You can see the default templates at:
            # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
            #
--- a/synapse/crypto/keyring.py
+++ b/synapse/crypto/keyring.py
@ -43,8 +43,8 @@ from synapse.api.errors import (
    SynapseError,
 )
 from synapse.logging.context import (
    LoggingContext,
    PreserveLoggingContext,
    current_context,
    make_deferred_yieldable,
    preserve_fn,
    run_in_background,
@ -236,7 +236,7 @@ class Keyring(object):
        """
        try:
-            ctx = LoggingContext.current_context()
+            ctx = current_context()
            # map from server name to a set of outstanding request ids
            server_to_request_ids = {}
--- a/synapse/events/snapshot.py
+++ b/synapse/events/snapshot.py
@ -322,11 +322,14 @@ class _AsyncEventContextImpl(EventContext):
        self._current_state_ids = yield self._storage.state.get_state_ids_for_group(
            self.state_group
        )
-        if self._prev_state_id and self._event_state_key is not None:
+        if self._event_state_key is not None:
            self._prev_state_ids = dict(self._current_state_ids)
            key = (self._event_type, self._event_state_key)
            if self._prev_state_id:
                self._prev_state_ids[key] = self._prev_state_id
            else:
                self._prev_state_ids.pop(key, None)
        else:
            self._prev_state_ids = self._current_state_ids
--- a/synapse/federation/federation_base.py
+++ b/synapse/federation/federation_base.py
@ -25,19 +25,15 @@ from twisted.python.failure import Failure
 from synapse.api.constants import MAX_DEPTH, EventTypes, Membership
 from synapse.api.errors import Codes, SynapseError
-from synapse.api.room_versions import (
+from synapse.api.room_versions import EventFormatVersions, RoomVersion
    KNOWN_ROOM_VERSIONS,
    EventFormatVersions,
    RoomVersion,
 )
 from synapse.crypto.event_signing import check_event_content_hash
 from synapse.crypto.keyring import Keyring
 from synapse.events import EventBase, make_event_from_dict
 from synapse.events.utils import prune_event
 from synapse.http.servlet import assert_params_in_dict
 from synapse.logging.context import (
    LoggingContext,
    PreserveLoggingContext,
    current_context,
    make_deferred_yieldable,
 )
 from synapse.types import JsonDict, get_domain_from_id
@ -55,13 +51,15 @@ class FederationBase(object):
        self.store = hs.get_datastore()
        self._clock = hs.get_clock()
-    def _check_sigs_and_hash(self, room_version: str, pdu: EventBase) -> Deferred:
+    def _check_sigs_and_hash(
        self, room_version: RoomVersion, pdu: EventBase
    ) -> Deferred:
        return make_deferred_yieldable(
            self._check_sigs_and_hashes(room_version, [pdu])[0]
        )
    def _check_sigs_and_hashes(
-        self, room_version: str, pdus: List[EventBase]
+        self, room_version: RoomVersion, pdus: List[EventBase]
    ) -> List[Deferred]:
        """Checks that each of the received events is correctly signed by the
        sending server.
@ -80,7 +78,7 @@ class FederationBase(object):
        """
        deferreds = _check_sigs_on_pdus(self.keyring, room_version, pdus)
-        ctx = LoggingContext.current_context()
+        ctx = current_context()
        def callback(_, pdu: EventBase):
            with PreserveLoggingContext(ctx):
@ -146,7 +144,7 @@ class PduToCheckSig(
 def _check_sigs_on_pdus(
-    keyring: Keyring, room_version: str, pdus: Iterable[EventBase]
+    keyring: Keyring, room_version: RoomVersion, pdus: Iterable[EventBase]
 ) -> List[Deferred]:
    """Check that the given events are correctly signed
@ -191,10 +189,6 @@ def _check_sigs_on_pdus(
        for p in pdus
    ]
    v = KNOWN_ROOM_VERSIONS.get(room_version)
    if not v:
        raise RuntimeError("Unrecognized room version %s" % (room_version,))
    # First we check that the sender event is signed by the sender's domain
    # (except if its a 3pid invite, in which case it may be sent by any server)
    pdus_to_check_sender = [p for p in pdus_to_check if not _is_invite_via_3pid(p.pdu)]
@ -204,7 +198,7 @@ def _check_sigs_on_pdus(
            (
                p.sender_domain,
                p.redacted_pdu_json,
-                p.pdu.origin_server_ts if v.enforce_key_validity else 0,
+                p.pdu.origin_server_ts if room_version.enforce_key_validity else 0,
                p.pdu.event_id,
            )
            for p in pdus_to_check_sender
@ -227,7 +221,7 @@ def _check_sigs_on_pdus(
    # event id's domain (normally only the case for joins/leaves), and add additional
    # checks. Only do this if the room version has a concept of event ID domain
    # (ie, the room version uses old-style non-hash event IDs).
-    if v.event_format == EventFormatVersions.V1:
+    if room_version.event_format == EventFormatVersions.V1:
        pdus_to_check_event_id = [
            p
            for p in pdus_to_check
@ -239,7 +233,7 @@ def _check_sigs_on_pdus(
                (
                    get_domain_from_id(p.pdu.event_id),
                    p.redacted_pdu_json,
-                    p.pdu.origin_server_ts if v.enforce_key_validity else 0,
+                    p.pdu.origin_server_ts if room_version.enforce_key_validity else 0,
                    p.pdu.event_id,
                )
                for p in pdus_to_check_event_id
--- a/synapse/federation/federation_client.py
+++ b/synapse/federation/federation_client.py
@ -220,8 +220,7 @@ class FederationClient(FederationBase):
        # FIXME: We should handle signature failures more gracefully.
        pdus[:] = await make_deferred_yieldable(
            defer.gatherResults(
-                self._check_sigs_and_hashes(room_version.identifier, pdus),
+                self._check_sigs_and_hashes(room_version, pdus), consumeErrors=True,
                consumeErrors=True,
            ).addErrback(unwrapFirstError)
        )
@ -291,9 +290,7 @@ class FederationClient(FederationBase):
                    pdu = pdu_list[0]
                    # Check signatures are correct.
-                    signed_pdu = await self._check_sigs_and_hash(
+                    signed_pdu = await self._check_sigs_and_hash(room_version, pdu)
                        room_version.identifier, pdu
                    )
                    break
@ -350,7 +347,7 @@ class FederationClient(FederationBase):
        self,
        origin: str,
        pdus: List[EventBase],
-        room_version: str,
+        room_version: RoomVersion,
        outlier: bool = False,
        include_none: bool = False,
    ) -> List[EventBase]:
@ -396,7 +393,7 @@ class FederationClient(FederationBase):
                        self.get_pdu(
                            destinations=[pdu.origin],
                            event_id=pdu.event_id,
-                            room_version=room_version,  # type: ignore
+                            room_version=room_version,
                            outlier=outlier,
                            timeout=10000,
                        )
@ -434,7 +431,7 @@ class FederationClient(FederationBase):
        ]
        signed_auth = await self._check_sigs_and_hash_and_fetch(
-            destination, auth_chain, outlier=True, room_version=room_version.identifier
+            destination, auth_chain, outlier=True, room_version=room_version
        )
        signed_auth.sort(key=lambda e: e.depth)
@ -661,7 +658,7 @@ class FederationClient(FederationBase):
                destination,
                list(pdus.values()),
                outlier=True,
-                room_version=room_version.identifier,
+                room_version=room_version,
            )
            valid_pdus_map = {p.event_id: p for p in valid_pdus}
@ -756,7 +753,7 @@ class FederationClient(FederationBase):
        pdu = event_from_pdu_json(pdu_dict, room_version)
        # Check signatures are correct.
-        pdu = await self._check_sigs_and_hash(room_version.identifier, pdu)
+        pdu = await self._check_sigs_and_hash(room_version, pdu)
        # FIXME: We should handle signature failures more gracefully.
@ -886,18 +883,37 @@ class FederationClient(FederationBase):
    def get_public_rooms(
        self,
-        destination,
+        remote_server: str,
-        limit=None,
+        limit: Optional[int] = None,
-        since_token=None,
+        since_token: Optional[str] = None,
-        search_filter=None,
+        search_filter: Optional[Dict] = None,
-        include_all_networks=False,
+        include_all_networks: bool = False,
-        third_party_instance_id=None,
+        third_party_instance_id: Optional[str] = None,
    ):
-        if destination == self.server_name:
+        """Get the list of public rooms from a remote homeserver
            return
        Args:
            remote_server: The name of the remote server
            limit: Maximum amount of rooms to return
            since_token: Used for result pagination
            search_filter: A filter dictionary to send the remote homeserver
                and filter the result set
            include_all_networks: Whether to include results from all third party instances
            third_party_instance_id: Whether to only include results from a specific third
                party instance
        Returns:
            Deferred[Dict[str, Any]]: The response from the remote server, or None if
            `remote_server` is the same as the local server_name
        Raises:
            HttpResponseException: There was an exception returned from the remote server
            SynapseException: M_FORBIDDEN when the remote server has disallowed publicRoom
                requests over federation
        """
        return self.transport_layer.get_public_rooms(
-            destination,
+            remote_server,
            limit,
            since_token,
            search_filter,
@ -948,7 +964,7 @@ class FederationClient(FederationBase):
            ]
            signed_events = await self._check_sigs_and_hash_and_fetch(
-                destination, events, outlier=False, room_version=room_version.identifier
+                destination, events, outlier=False, room_version=room_version
            )
        except HttpResponseException as e:
            if not e.code == 400:
@ -960,14 +976,13 @@ class FederationClient(FederationBase):
        return signed_events
-    @defer.inlineCallbacks
+    async def forward_third_party_invite(self, destinations, room_id, event_dict):
    def forward_third_party_invite(self, destinations, room_id, event_dict):
        for destination in destinations:
            if destination == self.server_name:
                continue
            try:
-                yield self.transport_layer.exchange_third_party_invite(
+                await self.transport_layer.exchange_third_party_invite(
                    destination=destination, room_id=room_id, event_dict=event_dict
                )
                return None
--- a/synapse/federation/federation_server.py
+++ b/synapse/federation/federation_server.py
@ -15,7 +15,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import Dict
+from typing import Any, Callable, Dict, List, Match, Optional, Tuple, Union
 import six
 from six import iteritems
@ -38,6 +38,7 @@ from synapse.api.errors import (
    UnsupportedRoomVersionError,
 )
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.events import EventBase
 from synapse.federation.federation_base import FederationBase, event_from_pdu_json
 from synapse.federation.persistence import TransactionActions
 from synapse.federation.units import Edu, Transaction
@ -94,7 +95,9 @@ class FederationServer(FederationBase):
        # come in waves.
        self._state_resp_cache = ResponseCache(hs, "state_resp", timeout_ms=30000)
-    async def on_backfill_request(self, origin, room_id, versions, limit):
+    async def on_backfill_request(
        self, origin: str, room_id: str, versions: List[str], limit: int
    ) -> Tuple[int, Dict[str, Any]]:
        with (await self._server_linearizer.queue((origin, room_id))):
            origin_host, _ = parse_server_name(origin)
            await self.check_server_matches_acl(origin_host, room_id)
@ -107,23 +110,25 @@ class FederationServer(FederationBase):
        return 200, res
-    async def on_incoming_transaction(self, origin, transaction_data):
+    async def on_incoming_transaction(
        self, origin: str, transaction_data: JsonDict
    ) -> Tuple[int, Dict[str, Any]]:
        # keep this as early as possible to make the calculated origin ts as
        # accurate as possible.
        request_time = self._clock.time_msec()
        transaction = Transaction(**transaction_data)
-        if not transaction.transaction_id:
+        if not transaction.transaction_id:  # type: ignore
            raise Exception("Transaction missing transaction_id")
-        logger.debug("[%s] Got transaction", transaction.transaction_id)
+        logger.debug("[%s] Got transaction", transaction.transaction_id)  # type: ignore
        # use a linearizer to ensure that we don't process the same transaction
        # multiple times in parallel.
        with (
            await self._transaction_linearizer.queue(
-                (origin, transaction.transaction_id)
+                (origin, transaction.transaction_id)  # type: ignore
            )
        ):
            result = await self._handle_incoming_transaction(
@ -132,31 +137,33 @@ class FederationServer(FederationBase):
        return result
-    async def _handle_incoming_transaction(self, origin, transaction, request_time):
+    async def _handle_incoming_transaction(
        self, origin: str, transaction: Transaction, request_time: int
    ) -> Tuple[int, Dict[str, Any]]:
        """ Process an incoming transaction and return the HTTP response
        Args:
-            origin (unicode): the server making the request
+            origin: the server making the request
-            transaction (Transaction): incoming transaction
+            transaction: incoming transaction
-            request_time (int): timestamp that the HTTP request arrived at
+            request_time: timestamp that the HTTP request arrived at
        Returns:
-            Deferred[(int, object)]: http response code and body
+            HTTP response code and body
        """
        response = await self.transaction_actions.have_responded(origin, transaction)
        if response:
            logger.debug(
                "[%s] We've already responded to this request",
-                transaction.transaction_id,
+                transaction.transaction_id,  # type: ignore
            )
            return response
-        logger.debug("[%s] Transaction is new", transaction.transaction_id)
+        logger.debug("[%s] Transaction is new", transaction.transaction_id)  # type: ignore
        # Reject if PDU count > 50 or EDU count > 100
-        if len(transaction.pdus) > 50 or (
+        if len(transaction.pdus) > 50 or (  # type: ignore
-            hasattr(transaction, "edus") and len(transaction.edus) > 100
+            hasattr(transaction, "edus") and len(transaction.edus) > 100  # type: ignore
        ):
            logger.info("Transaction PDU or EDU count too large. Returning 400")
@ -204,13 +211,13 @@ class FederationServer(FederationBase):
            report back to the sending server.
        """
-        received_pdus_counter.inc(len(transaction.pdus))
+        received_pdus_counter.inc(len(transaction.pdus))  # type: ignore
        origin_host, _ = parse_server_name(origin)
-        pdus_by_room = {}
+        pdus_by_room = {}  # type: Dict[str, List[EventBase]]
-        for p in transaction.pdus:
+        for p in transaction.pdus:  # type: ignore
            if "unsigned" in p:
                unsigned = p["unsigned"]
                if "age" in unsigned:
@ -254,7 +261,7 @@ class FederationServer(FederationBase):
        # require callouts to other servers to fetch missing events), but
        # impose a limit to avoid going too crazy with ram/cpu.
-        async def process_pdus_for_room(room_id):
+        async def process_pdus_for_room(room_id: str):
            logger.debug("Processing PDUs for %s", room_id)
            try:
                await self.check_server_matches_acl(origin_host, room_id)
@ -310,7 +317,9 @@ class FederationServer(FederationBase):
            TRANSACTION_CONCURRENCY_LIMIT,
        )
-    async def on_context_state_request(self, origin, room_id, event_id):
+    async def on_context_state_request(
        self, origin: str, room_id: str, event_id: str
    ) -> Tuple[int, Dict[str, Any]]:
        origin_host, _ = parse_server_name(origin)
        await self.check_server_matches_acl(origin_host, room_id)
@ -338,7 +347,9 @@ class FederationServer(FederationBase):
        return 200, resp
-    async def on_state_ids_request(self, origin, room_id, event_id):
+    async def on_state_ids_request(
        self, origin: str, room_id: str, event_id: str
    ) -> Tuple[int, Dict[str, Any]]:
        if not event_id:
            raise NotImplementedError("Specify an event")
@ -354,7 +365,9 @@ class FederationServer(FederationBase):
        return 200, {"pdu_ids": state_ids, "auth_chain_ids": auth_chain_ids}
-    async def _on_context_state_request_compute(self, room_id, event_id):
+    async def _on_context_state_request_compute(
        self, room_id: str, event_id: str
    ) -> Dict[str, list]:
        if event_id:
            pdus = await self.handler.get_state_for_pdu(room_id, event_id)
        else:
@ -367,7 +380,9 @@ class FederationServer(FederationBase):
            "auth_chain": [pdu.get_pdu_json() for pdu in auth_chain],
        }
-    async def on_pdu_request(self, origin, event_id):
+    async def on_pdu_request(
        self, origin: str, event_id: str
    ) -> Tuple[int, Union[JsonDict, str]]:
        pdu = await self.handler.get_persisted_pdu(origin, event_id)
        if pdu:
@ -375,12 +390,16 @@ class FederationServer(FederationBase):
        else:
            return 404, ""
-    async def on_query_request(self, query_type, args):
+    async def on_query_request(
        self, query_type: str, args: Dict[str, str]
    ) -> Tuple[int, Dict[str, Any]]:
        received_queries_counter.labels(query_type).inc()
        resp = await self.registry.on_query(query_type, args)
        return 200, resp
-    async def on_make_join_request(self, origin, room_id, user_id, supported_versions):
+    async def on_make_join_request(
        self, origin: str, room_id: str, user_id: str, supported_versions: List[str]
    ) -> Dict[str, Any]:
        origin_host, _ = parse_server_name(origin)
        await self.check_server_matches_acl(origin_host, room_id)
@ -397,7 +416,7 @@ class FederationServer(FederationBase):
    async def on_invite_request(
        self, origin: str, content: JsonDict, room_version_id: str
-    ):
+    ) -> Dict[str, Any]:
        room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)
        if not room_version:
            raise SynapseError(
@ -409,12 +428,14 @@ class FederationServer(FederationBase):
        pdu = event_from_pdu_json(content, room_version)
        origin_host, _ = parse_server_name(origin)
        await self.check_server_matches_acl(origin_host, pdu.room_id)
-        pdu = await self._check_sigs_and_hash(room_version.identifier, pdu)
+        pdu = await self._check_sigs_and_hash(room_version, pdu)
        ret_pdu = await self.handler.on_invite_request(origin, pdu, room_version)
        time_now = self._clock.time_msec()
        return {"event": ret_pdu.get_pdu_json(time_now)}
-    async def on_send_join_request(self, origin, content, room_id):
+    async def on_send_join_request(
        self, origin: str, content: JsonDict, room_id: str
    ) -> Dict[str, Any]:
        logger.debug("on_send_join_request: content: %s", content)
        room_version = await self.store.get_room_version(room_id)
@ -425,7 +446,7 @@ class FederationServer(FederationBase):
        logger.debug("on_send_join_request: pdu sigs: %s", pdu.signatures)
-        pdu = await self._check_sigs_and_hash(room_version.identifier, pdu)
+        pdu = await self._check_sigs_and_hash(room_version, pdu)
        res_pdus = await self.handler.on_send_join_request(origin, pdu)
        time_now = self._clock.time_msec()
@ -434,7 +455,9 @@ class FederationServer(FederationBase):
            "auth_chain": [p.get_pdu_json(time_now) for p in res_pdus["auth_chain"]],
        }
-    async def on_make_leave_request(self, origin, room_id, user_id):
+    async def on_make_leave_request(
        self, origin: str, room_id: str, user_id: str
    ) -> Dict[str, Any]:
        origin_host, _ = parse_server_name(origin)
        await self.check_server_matches_acl(origin_host, room_id)
        pdu = await self.handler.on_make_leave_request(origin, room_id, user_id)
@ -444,7 +467,9 @@ class FederationServer(FederationBase):
        time_now = self._clock.time_msec()
        return {"event": pdu.get_pdu_json(time_now), "room_version": room_version}
-    async def on_send_leave_request(self, origin, content, room_id):
+    async def on_send_leave_request(
        self, origin: str, content: JsonDict, room_id: str
    ) -> dict:
        logger.debug("on_send_leave_request: content: %s", content)
        room_version = await self.store.get_room_version(room_id)
@ -455,12 +480,14 @@ class FederationServer(FederationBase):
        logger.debug("on_send_leave_request: pdu sigs: %s", pdu.signatures)
-        pdu = await self._check_sigs_and_hash(room_version.identifier, pdu)
+        pdu = await self._check_sigs_and_hash(room_version, pdu)
        await self.handler.on_send_leave_request(origin, pdu)
        return {}
-    async def on_event_auth(self, origin, room_id, event_id):
+    async def on_event_auth(
        self, origin: str, room_id: str, event_id: str
    ) -> Tuple[int, Dict[str, Any]]:
        with (await self._server_linearizer.queue((origin, room_id))):
            origin_host, _ = parse_server_name(origin)
            await self.check_server_matches_acl(origin_host, room_id)
@ -471,15 +498,21 @@ class FederationServer(FederationBase):
        return 200, res
    @log_function
-    def on_query_client_keys(self, origin, content):
+    async def on_query_client_keys(
-        return self.on_query_request("client_keys", content)
+        self, origin: str, content: Dict[str, str]
    ) -> Tuple[int, Dict[str, Any]]:
        return await self.on_query_request("client_keys", content)
-    async def on_query_user_devices(self, origin: str, user_id: str):
+    async def on_query_user_devices(
        self, origin: str, user_id: str
    ) -> Tuple[int, Dict[str, Any]]:
        keys = await self.device_handler.on_federation_query_user_devices(user_id)
        return 200, keys
    @trace
-    async def on_claim_client_keys(self, origin, content):
+    async def on_claim_client_keys(
        self, origin: str, content: JsonDict
    ) -> Dict[str, Any]:
        query = []
        for user_id, device_keys in content.get("one_time_keys", {}).items():
            for device_id, algorithm in device_keys.items():
@ -488,7 +521,7 @@ class FederationServer(FederationBase):
        log_kv({"message": "Claiming one time keys.", "user, device pairs": query})
        results = await self.store.claim_e2e_one_time_keys(query)
-        json_result = {}
+        json_result = {}  # type: Dict[str, Dict[str, dict]]
        for user_id, device_keys in results.items():
            for device_id, keys in device_keys.items():
                for key_id, json_bytes in keys.items():
@ -511,8 +544,13 @@ class FederationServer(FederationBase):
        return {"one_time_keys": json_result}
    async def on_get_missing_events(
-        self, origin, room_id, earliest_events, latest_events, limit
+        self,
-    ):
+        origin: str,
        room_id: str,
        earliest_events: List[str],
        latest_events: List[str],
        limit: int,
    ) -> Dict[str, list]:
        with (await self._server_linearizer.queue((origin, room_id))):
            origin_host, _ = parse_server_name(origin)
            await self.check_server_matches_acl(origin_host, room_id)
@ -541,11 +579,11 @@ class FederationServer(FederationBase):
        return {"events": [ev.get_pdu_json(time_now) for ev in missing_events]}
    @log_function
-    def on_openid_userinfo(self, token):
+    async def on_openid_userinfo(self, token: str) -> Optional[str]:
        ts_now_ms = self._clock.time_msec()
-        return self.store.get_user_id_for_open_id_token(token, ts_now_ms)
+        return await self.store.get_user_id_for_open_id_token(token, ts_now_ms)
-    def _transaction_from_pdus(self, pdu_list):
+    def _transaction_from_pdus(self, pdu_list: List[EventBase]) -> Transaction:
        """Returns a new Transaction containing the given PDUs suitable for
        transmission.
        """
@ -558,7 +596,7 @@ class FederationServer(FederationBase):
            destination=None,
        )
-    async def _handle_received_pdu(self, origin, pdu):
+    async def _handle_received_pdu(self, origin: str, pdu: EventBase) -> None:
        """ Process a PDU received in a federation /send/ transaction.
        If the event is invalid, then this method throws a FederationError.
@ -579,10 +617,8 @@ class FederationServer(FederationBase):
        until we try to backfill across the discontinuity.
        Args:
-            origin (str): server which sent the pdu
+            origin: server which sent the pdu
-            pdu (FrozenEvent): received pdu
+            pdu: received pdu
        Returns (Deferred): completes with None
        Raises: FederationError if the signatures / hash do not match, or
            if the event was unacceptable for any other reason (eg, too large,
@ -611,7 +647,7 @@ class FederationServer(FederationBase):
                logger.info("Accepting join PDU %s from %s", pdu.event_id, origin)
        # We've already checked that we know the room version by this point
-        room_version = await self.store.get_room_version_id(pdu.room_id)
+        room_version = await self.store.get_room_version(pdu.room_id)
        # Check signature.
        try:
@ -625,25 +661,27 @@ class FederationServer(FederationBase):
        return "<ReplicationLayer(%s)>" % self.server_name
    async def exchange_third_party_invite(
-        self, sender_user_id, target_user_id, room_id, signed
+        self, sender_user_id: str, target_user_id: str, room_id: str, signed: Dict
    ):
        ret = await self.handler.exchange_third_party_invite(
            sender_user_id, target_user_id, room_id, signed
        )
        return ret
-    async def on_exchange_third_party_invite_request(self, room_id, event_dict):
+    async def on_exchange_third_party_invite_request(
        self, room_id: str, event_dict: Dict
    ):
        ret = await self.handler.on_exchange_third_party_invite_request(
            room_id, event_dict
        )
        return ret
-    async def check_server_matches_acl(self, server_name, room_id):
+    async def check_server_matches_acl(self, server_name: str, room_id: str):
        """Check if the given server is allowed by the server ACLs in the room
        Args:
-            server_name (str): name of server, *without any port part*
+            server_name: name of server, *without any port part*
-            room_id (str): ID of the room to check
+            room_id: ID of the room to check
        Raises:
            AuthError if the server does not match the ACL
@ -661,15 +699,15 @@ class FederationServer(FederationBase):
        raise AuthError(code=403, msg="Server is banned from room")
-def server_matches_acl_event(server_name, acl_event):
+def server_matches_acl_event(server_name: str, acl_event: EventBase) -> bool:
    """Check if the given server is allowed by the ACL event
    Args:
-        server_name (str): name of server, without any port part
+        server_name: name of server, without any port part
-        acl_event (EventBase): m.room.server_acl event
+        acl_event: m.room.server_acl event
    Returns:
-        bool: True if this server is allowed by the ACLs
+        True if this server is allowed by the ACLs
    """
    logger.debug("Checking %s against acl %s", server_name, acl_event.content)
@ -713,7 +751,7 @@ def server_matches_acl_event(server_name, acl_event):
    return False
-def _acl_entry_matches(server_name, acl_entry):
+def _acl_entry_matches(server_name: str, acl_entry: str) -> Match:
    if not isinstance(acl_entry, six.string_types):
        logger.warning(
            "Ignoring non-str ACL entry '%s' (is %s)", acl_entry, type(acl_entry)
@ -732,13 +770,13 @@ class FederationHandlerRegistry(object):
        self.edu_handlers = {}
        self.query_handlers = {}
-    def register_edu_handler(self, edu_type, handler):
+    def register_edu_handler(self, edu_type: str, handler: Callable[[str, dict], None]):
        """Sets the handler callable that will be used to handle an incoming
        federation EDU of the given type.
        Args:
-            edu_type (str): The type of the incoming EDU to register handler for
+            edu_type: The type of the incoming EDU to register handler for
-            handler (Callable[[str, dict]]): A callable invoked on incoming EDU
+            handler: A callable invoked on incoming EDU
                of the given type. The arguments are the origin server name and
                the EDU contents.
        """
@ -749,14 +787,16 @@ class FederationHandlerRegistry(object):
        self.edu_handlers[edu_type] = handler
-    def register_query_handler(self, query_type, handler):
+    def register_query_handler(
        self, query_type: str, handler: Callable[[dict], defer.Deferred]
    ):
        """Sets the handler callable that will be used to handle an incoming
        federation query of the given type.
        Args:
-            query_type (str): Category name of the query, which should match
+            query_type: Category name of the query, which should match
                the string used by make_query.
-            handler (Callable[[dict], Deferred[dict]]): Invoked to handle
+            handler: Invoked to handle
                incoming queries of this type. The return will be yielded
                on and the result used as the response to the query request.
        """
@ -767,10 +807,11 @@ class FederationHandlerRegistry(object):
        self.query_handlers[query_type] = handler
-    async def on_edu(self, edu_type, origin, content):
+    async def on_edu(self, edu_type: str, origin: str, content: dict):
        handler = self.edu_handlers.get(edu_type)
        if not handler:
            logger.warning("No handler registered for EDU type %s", edu_type)
            return
        with start_active_span_from_edu(content, "handle_edu"):
            try:
@ -780,7 +821,7 @@ class FederationHandlerRegistry(object):
            except Exception:
                logger.exception("Failed to handle edu %r", edu_type)
-    def on_query(self, query_type, args):
+    def on_query(self, query_type: str, args: dict) -> defer.Deferred:
        handler = self.query_handlers.get(query_type)
        if not handler:
            logger.warning("No handler registered for query type %s", query_type)
@ -807,7 +848,7 @@ class ReplicationFederationHandlerRegistry(FederationHandlerRegistry):
        super(ReplicationFederationHandlerRegistry, self).__init__()
-    async def on_edu(self, edu_type, origin, content):
+    async def on_edu(self, edu_type: str, origin: str, content: dict):
        """Overrides FederationHandlerRegistry
        """
        if not self.config.use_presence and edu_type == "m.presence":
@ -821,7 +862,7 @@ class ReplicationFederationHandlerRegistry(FederationHandlerRegistry):
        return await self._send_edu(edu_type=edu_type, origin=origin, content=content)
-    async def on_query(self, query_type, args):
+    async def on_query(self, query_type: str, args: dict):
        """Overrides FederationHandlerRegistry
        """
        handler = self.query_handlers.get(query_type)
--- a/synapse/federation/send_queue.py
+++ b/synapse/federation/send_queue.py
@ -477,7 +477,7 @@ def process_rows_for_federation(transaction_queue, rows):
    Args:
        transaction_queue (FederationSender)
-        rows (list(synapse.replication.tcp.streams.FederationStreamRow))
+        rows (list(synapse.replication.tcp.streams.federation.FederationStream.FederationStreamRow))
    """
    # The federation stream contains a bunch of different types of
--- a/synapse/federation/sender/init.py
+++ b/synapse/federation/sender/init.py
@ -499,4 +499,13 @@ class FederationSender(object):
        self._get_per_destination_queue(destination).attempt_new_transaction()
    def get_current_token(self) -> int:
        # Dummy implementation for case where federation sender isn't offloaded
        # to a worker.
        return 0
    async def get_replication_rows(
        self, from_token, to_token, limit, federation_ack=None
    ):
        # Dummy implementation for case where federation sender isn't offloaded
        # to a worker.
        return []
--- a/synapse/federation/transport/client.py
+++ b/synapse/federation/transport/client.py
@ -15,13 +15,14 @@
 # limitations under the License.
 import logging
-from typing import Any, Dict
+from typing import Any, Dict, Optional
 from six.moves import urllib
 from twisted.internet import defer
 from synapse.api.constants import Membership
 from synapse.api.errors import Codes, HttpResponseException, SynapseError
 from synapse.api.urls import (
    FEDERATION_UNSTABLE_PREFIX,
    FEDERATION_V1_PREFIX,
@ -326,18 +327,25 @@ class TransportLayerClient(object):
    @log_function
    def get_public_rooms(
        self,
-        remote_server,
+        remote_server: str,
-        limit,
+        limit: Optional[int] = None,
-        since_token,
+        since_token: Optional[str] = None,
-        search_filter=None,
+        search_filter: Optional[Dict] = None,
-        include_all_networks=False,
+        include_all_networks: bool = False,
-        third_party_instance_id=None,
+        third_party_instance_id: Optional[str] = None,
    ):
        """Get the list of public rooms from a remote homeserver
        See synapse.federation.federation_client.FederationClient.get_public_rooms for
        more information.
        """
        if search_filter:
            # this uses MSC2197 (Search Filtering over Federation)
            path = _create_v1_path("/publicRooms")
-            data = {"include_all_networks": "true" if include_all_networks else "false"}
+            data = {
                "include_all_networks": "true" if include_all_networks else "false"
            }  # type: Dict[str, Any]
            if third_party_instance_id:
                data["third_party_instance_id"] = third_party_instance_id
            if limit:
@ -347,9 +355,19 @@ class TransportLayerClient(object):
            data["filter"] = search_filter
            try:
                response = yield self.client.post_json(
                    destination=remote_server, path=path, data=data, ignore_backoff=True
                )
            except HttpResponseException as e:
                if e.code == 403:
                    raise SynapseError(
                        403,
                        "You are not allowed to view the public rooms list of %s"
                        % (remote_server,),
                        errcode=Codes.FORBIDDEN,
                    )
                raise
        else:
            path = _create_v1_path("/publicRooms")
@ -363,9 +381,19 @@ class TransportLayerClient(object):
            if since_token:
                args["since"] = [since_token]
            try:
                response = yield self.client.get_json(
                    destination=remote_server, path=path, args=args, ignore_backoff=True
                )
            except HttpResponseException as e:
                if e.code == 403:
                    raise SynapseError(
                        403,
                        "You are not allowed to view the public rooms list of %s"
                        % (remote_server,),
                        errcode=Codes.FORBIDDEN,
                    )
                raise
        return response
--- a/synapse/groups/attestations.py
+++ b/synapse/groups/attestations.py
@ -37,13 +37,13 @@ An attestation is a signed blob of json that looks like:
 import logging
 import random
 from typing import Tuple
 from signedjson.sign import sign_json
 from twisted.internet import defer
 from synapse.api.errors import HttpResponseException, RequestSendFailed, SynapseError
 from synapse.logging.context import run_in_background
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.types import get_domain_from_id
@ -162,19 +162,19 @@ class GroupAttestionRenewer(object):
    def _start_renew_attestations(self):
        return run_as_background_process("renew_attestations", self._renew_attestations)
-    @defer.inlineCallbacks
+    async def _renew_attestations(self):
    def _renew_attestations(self):
        """Called periodically to check if we need to update any of our attestations
        """
        now = self.clock.time_msec()
-        rows = yield self.store.get_attestations_need_renewals(
+        rows = await self.store.get_attestations_need_renewals(
            now + UPDATE_ATTESTATION_TIME_MS
        )
        @defer.inlineCallbacks
-        def _renew_attestation(group_id, user_id):
+        def _renew_attestation(group_user: Tuple[str, str]):
            group_id, user_id = group_user
            try:
                if not self.is_mine_id(group_id):
                    destination = get_domain_from_id(group_id)
@ -208,7 +208,4 @@ class GroupAttestionRenewer(object):
                )
        for row in rows:
-            group_id = row["group_id"]
+            await _renew_attestation((row["group_id"], row["user_id"]))
            user_id = row["user_id"]
            run_in_background(_renew_attestation, group_id, user_id)
--- a/synapse/groups/groups_server.py
+++ b/synapse/groups/groups_server.py
@ -748,17 +748,18 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
        raise NotImplementedError()
-    @defer.inlineCallbacks
+    async def remove_user_from_group(
-    def remove_user_from_group(self, group_id, user_id, requester_user_id, content):
+        self, group_id, user_id, requester_user_id, content
    ):
        """Remove a user from the group; either a user is leaving or an admin
        kicked them.
        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
+        await self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
        is_kick = False
        if requester_user_id != user_id:
-            is_admin = yield self.store.is_user_admin_in_group(
+            is_admin = await self.store.is_user_admin_in_group(
                group_id, requester_user_id
            )
            if not is_admin:
@ -766,30 +767,29 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
            is_kick = True
-        yield self.store.remove_user_from_group(group_id, user_id)
+        await self.store.remove_user_from_group(group_id, user_id)
        if is_kick:
            if self.hs.is_mine_id(user_id):
                groups_local = self.hs.get_groups_local_handler()
-                yield groups_local.user_removed_from_group(group_id, user_id, {})
+                await groups_local.user_removed_from_group(group_id, user_id, {})
            else:
-                yield self.transport_client.remove_user_from_group_notification(
+                await self.transport_client.remove_user_from_group_notification(
                    get_domain_from_id(user_id), group_id, user_id, {}
                )
        if not self.hs.is_mine_id(user_id):
-            yield self.store.maybe_delete_remote_profile_cache(user_id)
+            await self.store.maybe_delete_remote_profile_cache(user_id)
        # Delete group if the last user has left
-        users = yield self.store.get_users_in_group(group_id, include_private=True)
+        users = await self.store.get_users_in_group(group_id, include_private=True)
        if not users:
-            yield self.store.delete_group(group_id)
+            await self.store.delete_group(group_id)
        return {}
-    @defer.inlineCallbacks
+    async def create_group(self, group_id, requester_user_id, content):
-    def create_group(self, group_id, requester_user_id, content):
+        group = await self.check_group_is_ours(group_id, requester_user_id)
        group = yield self.check_group_is_ours(group_id, requester_user_id)
        logger.info("Attempting to create group with ID: %r", group_id)
@ -799,7 +799,7 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
        if group:
            raise SynapseError(400, "Group already exists")
-        is_admin = yield self.auth.is_server_admin(
+        is_admin = await self.auth.is_server_admin(
            UserID.from_string(requester_user_id)
        )
        if not is_admin:
@ -822,7 +822,7 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
        long_description = profile.get("long_description")
        user_profile = content.get("user_profile", {})
-        yield self.store.create_group(
+        await self.store.create_group(
            group_id,
            requester_user_id,
            name=name,
@ -834,7 +834,7 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
        if not self.hs.is_mine_id(requester_user_id):
            remote_attestation = content["attestation"]
-            yield self.attestations.verify_attestation(
+            await self.attestations.verify_attestation(
                remote_attestation, user_id=requester_user_id, group_id=group_id
            )
@ -845,7 +845,7 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
            local_attestation = None
            remote_attestation = None
-        yield self.store.add_user_to_group(
+        await self.store.add_user_to_group(
            group_id,
            requester_user_id,
            is_admin=True,
@ -855,7 +855,7 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
        )
        if not self.hs.is_mine_id(requester_user_id):
-            yield self.store.add_remote_profile_cache(
+            await self.store.add_remote_profile_cache(
                requester_user_id,
                displayname=user_profile.get("displayname"),
                avatar_url=user_profile.get("avatar_url"),
@ -863,8 +863,7 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
        return {"group_id": group_id}
-    @defer.inlineCallbacks
+    async def delete_group(self, group_id, requester_user_id):
    def delete_group(self, group_id, requester_user_id):
        """Deletes a group, kicking out all current members.
        Only group admins or server admins can call this request
@ -877,14 +876,14 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
            Deferred
        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
+        await self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
        # Only server admins or group admins can delete groups.
-        is_admin = yield self.store.is_user_admin_in_group(group_id, requester_user_id)
+        is_admin = await self.store.is_user_admin_in_group(group_id, requester_user_id)
        if not is_admin:
-            is_admin = yield self.auth.is_server_admin(
+            is_admin = await self.auth.is_server_admin(
                UserID.from_string(requester_user_id)
            )
@ -892,18 +891,17 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
            raise SynapseError(403, "User is not an admin")
        # Before deleting the group lets kick everyone out of it
-        users = yield self.store.get_users_in_group(group_id, include_private=True)
+        users = await self.store.get_users_in_group(group_id, include_private=True)
-        @defer.inlineCallbacks
+        async def _kick_user_from_group(user_id):
        def _kick_user_from_group(user_id):
            if self.hs.is_mine_id(user_id):
                groups_local = self.hs.get_groups_local_handler()
-                yield groups_local.user_removed_from_group(group_id, user_id, {})
+                await groups_local.user_removed_from_group(group_id, user_id, {})
            else:
-                yield self.transport_client.remove_user_from_group_notification(
+                await self.transport_client.remove_user_from_group_notification(
                    get_domain_from_id(user_id), group_id, user_id, {}
                )
-                yield self.store.maybe_delete_remote_profile_cache(user_id)
+                await self.store.maybe_delete_remote_profile_cache(user_id)
        # We kick users out in the order of:
        #   1. Non-admins
@ -922,11 +920,11 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
            else:
                non_admins.append(u["user_id"])
-        yield concurrently_execute(_kick_user_from_group, non_admins, 10)
+        await concurrently_execute(_kick_user_from_group, non_admins, 10)
-        yield concurrently_execute(_kick_user_from_group, admins, 10)
+        await concurrently_execute(_kick_user_from_group, admins, 10)
-        yield _kick_user_from_group(requester_user_id)
+        await _kick_user_from_group(requester_user_id)
-        yield self.store.delete_group(group_id)
+        await self.store.delete_group(group_id)
 def _parse_join_policy_from_contents(content):
--- a/synapse/handlers/_base.py
+++ b/synapse/handlers/_base.py
@ -126,30 +126,28 @@ class BaseHandler(object):
                retry_after_ms=int(1000 * (time_allowed - time_now))
            )
-    @defer.inlineCallbacks
+    async def maybe_kick_guest_users(self, event, context=None):
    def maybe_kick_guest_users(self, event, context=None):
        # Technically this function invalidates current_state by changing it.
        # Hopefully this isn't that important to the caller.
        if event.type == EventTypes.GuestAccess:
            guest_access = event.content.get("guest_access", "forbidden")
            if guest_access != "can_join":
                if context:
-                    current_state_ids = yield context.get_current_state_ids()
+                    current_state_ids = await context.get_current_state_ids()
-                    current_state = yield self.store.get_events(
+                    current_state = await self.store.get_events(
                        list(current_state_ids.values())
                    )
                else:
-                    current_state = yield self.state_handler.get_current_state(
+                    current_state = await self.state_handler.get_current_state(
                        event.room_id
                    )
                current_state = list(current_state.values())
                logger.info("maybe_kick_guest_users %r", current_state)
-                yield self.kick_guest_users(current_state)
+                await self.kick_guest_users(current_state)
-    @defer.inlineCallbacks
+    async def kick_guest_users(self, current_state):
    def kick_guest_users(self, current_state):
        for member_event in current_state:
            try:
                if member_event.type != EventTypes.Member:
@ -180,7 +178,7 @@ class BaseHandler(object):
                # homeserver.
                requester = synapse.types.create_requester(target_user, is_guest=True)
                handler = self.hs.get_room_member_handler()
-                yield handler.update_membership(
+                await handler.update_membership(
                    requester,
                    target_user,
                    member_event.room_id,
--- a/synapse/handlers/auth.py
+++ b/synapse/handlers/auth.py
@ -18,14 +18,12 @@ import logging
 import time
 import unicodedata
 import urllib.parse
-from typing import Any, Dict, Iterable, List, Optional
+from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Union
 import attr
 import bcrypt  # type: ignore[import]
 import pymacaroons
 from twisted.internet import defer
 import synapse.util.stringutils as stringutils
 from synapse.api.constants import LoginType
 from synapse.api.errors import (
@ -43,10 +41,10 @@ from synapse.handlers.ui_auth.checkers import UserInteractiveAuthChecker
 from synapse.http.server import finish_request
 from synapse.http.site import SynapseRequest
 from synapse.logging.context import defer_to_thread
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.module_api import ModuleApi
 from synapse.push.mailer import load_jinja2_templates
 from synapse.types import Requester, UserID
 from synapse.util.caches.expiringcache import ExpiringCache
 from ._base import BaseHandler
@ -71,15 +69,6 @@ class AuthHandler(BaseHandler):
        self.bcrypt_rounds = hs.config.bcrypt_rounds
        # This is not a cache per se, but a store of all current sessions that
        # expire after N hours
        self.sessions = ExpiringCache(
            cache_name="register_sessions",
            clock=hs.get_clock(),
            expiry_ms=self.SESSION_EXPIRE_MS,
            reset_expiry_on_get=True,
        )
        account_handler = ModuleApi(hs, self)
        self.password_providers = [
            module(config=config, account_handler=account_handler)
@ -91,6 +80,7 @@ class AuthHandler(BaseHandler):
        self.hs = hs  # FIXME better possibility to access registrationHandler later?
        self.macaroon_gen = hs.get_macaroon_generator()
        self._password_enabled = hs.config.password_enabled
        self._sso_enabled = hs.config.saml2_enabled or hs.config.cas_enabled
        # we keep this as a list despite the O(N^2) implication so that we can
        # keep PASSWORD first and avoid confusing clients which pick the first
@ -106,6 +96,13 @@ class AuthHandler(BaseHandler):
                    if t not in login_types:
                        login_types.append(t)
        self._supported_login_types = login_types
        # Login types and UI Auth types have a heavy overlap, but are not
        # necessarily identical. Login types have SSO (and other login types)
        # added in the rest layer, see synapse.rest.client.v1.login.LoginRestServerlet.on_GET.
        ui_auth_types = login_types.copy()
        if self._sso_enabled:
            ui_auth_types.append(LoginType.SSO)
        self._supported_ui_auth_types = ui_auth_types
        # Ratelimiter for failed auth during UIA. Uses same ratelimit config
        # as per `rc_login.failed_attempts`.
@ -113,20 +110,52 @@ class AuthHandler(BaseHandler):
        self._clock = self.hs.get_clock()
-        # Load the SSO redirect confirmation page HTML template
+        # Expire old UI auth sessions after a period of time.
        if hs.config.worker_app is None:
            self._clock.looping_call(
                run_as_background_process,
                5 * 60 * 1000,
                "expire_old_sessions",
                self._expire_old_sessions,
            )
        # Load the SSO HTML templates.
        # The following template is shown to the user during a client login via SSO,
        # after the SSO completes and before redirecting them back to their client.
        # It notifies the user they are about to give access to their matrix account
        # to the client.
        self._sso_redirect_confirm_template = load_jinja2_templates(
            hs.config.sso_redirect_confirm_template_dir, ["sso_redirect_confirm.html"],
        )[0]
        # The following template is shown during user interactive authentication
        # in the fallback auth scenario. It notifies the user that they are
        # authenticating for an operation to occur on their account.
        self._sso_auth_confirm_template = load_jinja2_templates(
            hs.config.sso_redirect_confirm_template_dir, ["sso_auth_confirm.html"],
        )[0]
        # The following template is shown after a successful user interactive
        # authentication session. It tells the user they can close the window.
        self._sso_auth_success_template = hs.config.sso_auth_success_template
        # The following template is shown during the SSO authentication process if
        # the account is deactivated.
        self._sso_account_deactivated_template = (
            hs.config.sso_account_deactivated_template
        )
        self._server_name = hs.config.server_name
        # cast to tuple for use with str.startswith
        self._whitelisted_sso_clients = tuple(hs.config.sso_client_whitelist)
-    @defer.inlineCallbacks
+    async def validate_user_via_ui_auth(
-    def validate_user_via_ui_auth(
+        self,
-        self, requester: Requester, request_body: Dict[str, Any], clientip: str
+        requester: Requester,
-    ):
+        request: SynapseRequest,
        request_body: Dict[str, Any],
        clientip: str,
        description: str,
    ) -> dict:
        """
        Checks that the user is who they claim to be, via a UI auth.
@ -137,12 +166,17 @@ class AuthHandler(BaseHandler):
        Args:
            requester: The user, as given by the access token
            request: The request sent by the client.
            request_body: The body of the request sent by the client
            clientip: The IP address of the client.
            description: A human readable string to be displayed to the user that
                         describes the operation happening on their account.
        Returns:
-            defer.Deferred[dict]: the parameters for this request (which may
+            The parameters for this request (which may
                have been given only in a previous call).
        Raises:
@ -169,10 +203,12 @@ class AuthHandler(BaseHandler):
        )
        # build a list of supported flows
-        flows = [[login_type] for login_type in self._supported_login_types]
+        flows = [[login_type] for login_type in self._supported_ui_auth_types]
        try:
-            result, params, _ = yield self.check_auth(flows, request_body, clientip)
+            result, params, _ = await self.check_auth(
                flows, request, request_body, clientip, description
            )
        except LoginError:
            # Update the ratelimite to say we failed (`can_do_action` doesn't raise).
            self._failed_uia_attempts_ratelimiter.can_do_action(
@ -185,7 +221,7 @@ class AuthHandler(BaseHandler):
            raise
        # find the completed login type
-        for login_type in self._supported_login_types:
+        for login_type in self._supported_ui_auth_types:
            if login_type not in result:
                continue
@ -209,18 +245,18 @@ class AuthHandler(BaseHandler):
        """
        return self.checkers.keys()
-    @defer.inlineCallbacks
+    async def check_auth(
-    def check_auth(
+        self,
-        self, flows: List[List[str]], clientdict: Dict[str, Any], clientip: str
+        flows: List[List[str]],
-    ):
+        request: SynapseRequest,
        clientdict: Dict[str, Any],
        clientip: str,
        description: str,
    ) -> Tuple[dict, dict, str]:
        """
        Takes a dictionary sent by the client in the login / registration
        protocol and handles the User-Interactive Auth flow.
        As a side effect, this function fills in the 'creds' key on the user's
        session with a map, which maps each auth-type (str) to the relevant
        identity authenticated by that auth-type (mostly str, but for captcha, bool).
        If no auth flows have been completed successfully, raises an
        InteractiveAuthIncompleteError. To handle this, you can use
        synapse.rest.client.v2_alpha._base.interactive_auth_handler as a
@ -231,14 +267,18 @@ class AuthHandler(BaseHandler):
                   strings representing auth-types. At least one full
                   flow must be completed in order for auth to be successful.
            request: The request sent by the client.
            clientdict: The dictionary from the client root level, not the
                        'auth' key: this method prompts for auth if none is sent.
            clientip: The IP address of the client.
            description: A human readable string to be displayed to the user that
                         describes the operation happening on their account.
        Returns:
-            defer.Deferred[dict, dict, str]: a deferred tuple of
+            A tuple of (creds, params, session_id).
                (creds, params, session_id).
                'creds' contains the authenticated credentials of each stage.
@ -260,9 +300,26 @@ class AuthHandler(BaseHandler):
            del clientdict["auth"]
            if "session" in authdict:
                sid = authdict["session"]
        session = self._get_session_info(sid)
-        if len(clientdict) > 0:
+        # Convert the URI and method to strings.
        uri = request.uri.decode("utf-8")
        method = request.uri.decode("utf-8")
        # If there's no session ID, create a new session.
        if not sid:
            session = await self.store.create_ui_auth_session(
                clientdict, uri, method, description
            )
        else:
            try:
                session = await self.store.get_ui_auth_session(sid)
            except StoreError:
                raise SynapseError(400, "Unknown session ID: %s" % (sid,))
            # If the client provides parameters, update what is persisted,
            # otherwise use whatever was last provided.
            #
            # This was designed to allow the client to omit the parameters
            # and just supply the session in subsequent calls so it split
            # auth between devices by just sharing the session, (eg. so you
@ -270,31 +327,60 @@ class AuthHandler(BaseHandler):
            # email auth link on there). It's probably too open to abuse
            # because it lets unauthenticated clients store arbitrary objects
            # on a homeserver.
-            # Revisit: Assumimg the REST APIs do sensible validation, the data
+            #
-            # isn't arbintrary.
+            # Revisit: Assuming the REST APIs do sensible validation, the data
-            session["clientdict"] = clientdict
+            # isn't arbitrary.
-            self._save_session(session)
+            #
-        elif "clientdict" in session:
+            # Note that the registration endpoint explicitly removes the
-            clientdict = session["clientdict"]
+            # "initial_device_display_name" parameter if it is provided
            # without a "password" parameter. See the changes to
            # synapse.rest.client.v2_alpha.register.RegisterRestServlet.on_POST
            # in commit 544722bad23fc31056b9240189c3cbbbf0ffd3f9.
            if not clientdict:
                clientdict = session.clientdict
            # Ensure that the queried operation does not vary between stages of
            # the UI authentication session. This is done by generating a stable
            # comparator and storing it during the initial query. Subsequent
            # queries ensure that this comparator has not changed.
            #
            # The comparator is based on the requested URI and HTTP method. The
            # client dict (minus the auth dict) should also be checked, but some
            # clients are not spec compliant, just warn for now if the client
            # dict changes.
            if (session.uri, session.method) != (uri, method):
                raise SynapseError(
                    403,
                    "Requested operation has changed during the UI authentication session.",
                )
            if session.clientdict != clientdict:
                logger.warning(
                    "Requested operation has changed during the UI "
                    "authentication session. A future version of Synapse "
                    "will remove this capability."
                )
            # For backwards compatibility, changes to the client dict are
            # persisted as clients modify them throughout their user interactive
            # authentication flow.
            await self.store.set_ui_auth_clientdict(sid, clientdict)
        if not authdict:
            raise InteractiveAuthIncompleteError(
-                self._auth_dict_for_flows(flows, session)
+                self._auth_dict_for_flows(flows, session.session_id)
            )
        if "creds" not in session:
            session["creds"] = {}
        creds = session["creds"]
        # check auth type currently being presented
        errordict = {}  # type: Dict[str, Any]
        if "type" in authdict:
            login_type = authdict["type"]  # type: str
            try:
-                result = yield self._check_auth_dict(authdict, clientip)
+                result = await self._check_auth_dict(authdict, clientip)
                if result:
-                    creds[login_type] = result
+                    await self.store.mark_ui_auth_stage_complete(
-                    self._save_session(session)
+                        session.session_id, login_type, result
                    )
            except LoginError as e:
                if login_type == LoginType.EMAIL_IDENTITY:
                    # riot used to have a bug where it would request a new
@ -310,6 +396,7 @@ class AuthHandler(BaseHandler):
                # so that the client can have another go.
                errordict = e.error_dict()
        creds = await self.store.get_completed_ui_auth_stages(session.session_id)
        for f in flows:
            if len(set(f) - set(creds)) == 0:
                # it's very useful to know what args are stored, but this can
@ -322,15 +409,17 @@ class AuthHandler(BaseHandler):
                    creds,
                    list(clientdict),
                )
                return creds, clientdict, session["id"]
-        ret = self._auth_dict_for_flows(flows, session)
+                return creds, clientdict, session.session_id
        ret = self._auth_dict_for_flows(flows, session.session_id)
        ret["completed"] = list(creds)
        ret.update(errordict)
        raise InteractiveAuthIncompleteError(ret)
-    @defer.inlineCallbacks
+    async def add_oob_auth(
-    def add_oob_auth(self, stagetype: str, authdict: Dict[str, Any], clientip: str):
+        self, stagetype: str, authdict: Dict[str, Any], clientip: str
    ) -> bool:
        """
        Adds the result of out-of-band authentication into an existing auth
        session. Currently used for adding the result of fallback auth.
@ -340,15 +429,11 @@ class AuthHandler(BaseHandler):
        if "session" not in authdict:
            raise LoginError(400, "", Codes.MISSING_PARAM)
-        sess = self._get_session_info(authdict["session"])
+        result = await self.checkers[stagetype].check_auth(authdict, clientip)
        if "creds" not in sess:
            sess["creds"] = {}
        creds = sess["creds"]
        result = yield self.checkers[stagetype].check_auth(authdict, clientip)
        if result:
-            creds[stagetype] = result
+            await self.store.mark_ui_auth_stage_complete(
-            self._save_session(sess)
+                authdict["session"], stagetype, result
            )
            return True
        return False
@ -370,7 +455,7 @@ class AuthHandler(BaseHandler):
                sid = authdict["session"]
        return sid
-    def set_session_data(self, session_id: str, key: str, value: Any) -> None:
+    async def set_session_data(self, session_id: str, key: str, value: Any) -> None:
        """
        Store a key-value pair into the sessions data associated with this
        request. This data is stored server-side and cannot be modified by
@ -381,11 +466,12 @@ class AuthHandler(BaseHandler):
            key: The key to store the data under
            value: The data to store
        """
-        sess = self._get_session_info(session_id)
+        try:
-        sess.setdefault("serverdict", {})[key] = value
+            await self.store.set_ui_auth_session_data(session_id, key, value)
-        self._save_session(sess)
+        except StoreError:
            raise SynapseError(400, "Unknown session ID: %s" % (session_id,))
-    def get_session_data(
+    async def get_session_data(
        self, session_id: str, key: str, default: Optional[Any] = None
    ) -> Any:
        """
@ -396,11 +482,22 @@ class AuthHandler(BaseHandler):
            key: The key to store the data under
            default: Value to return if the key has not been set
        """
-        sess = self._get_session_info(session_id)
+        try:
-        return sess.setdefault("serverdict", {}).get(key, default)
+            return await self.store.get_ui_auth_session_data(session_id, key, default)
        except StoreError:
            raise SynapseError(400, "Unknown session ID: %s" % (session_id,))
-    @defer.inlineCallbacks
+    async def _expire_old_sessions(self):
-    def _check_auth_dict(self, authdict: Dict[str, Any], clientip: str):
+        """
        Invalidate any user interactive authentication sessions that have expired.
        """
        now = self._clock.time_msec()
        expiration_time = now - self.SESSION_EXPIRE_MS
        await self.store.delete_old_ui_auth_sessions(expiration_time)
    async def _check_auth_dict(
        self, authdict: Dict[str, Any], clientip: str
    ) -> Union[Dict[str, Any], str]:
        """Attempt to validate the auth dict provided by a client
        Args:
@ -408,7 +505,7 @@ class AuthHandler(BaseHandler):
            clientip: IP address of the client
        Returns:
-            Deferred: result of the stage verification.
+            Result of the stage verification.
        Raises:
            StoreError if there was a problem accessing the database
@ -418,7 +515,7 @@ class AuthHandler(BaseHandler):
        login_type = authdict["type"]
        checker = self.checkers.get(login_type)
        if checker is not None:
-            res = yield checker.check_auth(authdict, clientip=clientip)
+            res = await checker.check_auth(authdict, clientip=clientip)
            return res
        # build a v1-login-style dict out of the authdict and fall back to the
@ -428,7 +525,7 @@ class AuthHandler(BaseHandler):
        if user_id is None:
            raise SynapseError(400, "", Codes.MISSING_PARAM)
-        (canonical_id, callback) = yield self.validate_login(user_id, authdict)
+        (canonical_id, callback) = await self.validate_login(user_id, authdict)
        return canonical_id
    def _get_params_recaptcha(self) -> dict:
@ -452,7 +549,7 @@ class AuthHandler(BaseHandler):
        }
    def _auth_dict_for_flows(
-        self, flows: List[List[str]], session: Dict[str, Any]
+        self, flows: List[List[str]], session_id: str,
    ) -> Dict[str, Any]:
        public_flows = []
        for f in flows:
@ -471,31 +568,12 @@ class AuthHandler(BaseHandler):
                    params[stage] = get_params[stage]()
        return {
-            "session": session["id"],
+            "session": session_id,
            "flows": [{"stages": f} for f in public_flows],
            "params": params,
        }
-    def _get_session_info(self, session_id: Optional[str]) -> dict:
+    async def get_access_token_for_user_id(
        """
        Gets or creates a session given a session ID.
        The session can be used to track data across multiple requests, e.g. for
        interactive authentication.
        """
        if session_id not in self.sessions:
            session_id = None
        if not session_id:
            # create a new session
            while session_id is None or session_id in self.sessions:
                session_id = stringutils.random_string(24)
            self.sessions[session_id] = {"id": session_id}
        return self.sessions[session_id]
    @defer.inlineCallbacks
    def get_access_token_for_user_id(
        self, user_id: str, device_id: Optional[str], valid_until_ms: Optional[int]
    ):
        """
@ -525,10 +603,10 @@ class AuthHandler(BaseHandler):
            )
        logger.info("Logging in user %s on device %s%s", user_id, device_id, fmt_expiry)
-        yield self.auth.check_auth_blocking(user_id)
+        await self.auth.check_auth_blocking(user_id)
        access_token = self.macaroon_gen.generate_access_token(user_id)
-        yield self.store.add_access_token_to_user(
+        await self.store.add_access_token_to_user(
            user_id, access_token, device_id, valid_until_ms
        )
@ -538,15 +616,14 @@ class AuthHandler(BaseHandler):
        # device, so we double-check it here.
        if device_id is not None:
            try:
-                yield self.store.get_device(user_id, device_id)
+                await self.store.get_device(user_id, device_id)
            except StoreError:
-                yield self.store.delete_access_token(access_token)
+                await self.store.delete_access_token(access_token)
                raise StoreError(400, "Login raced against device deletion")
        return access_token
-    @defer.inlineCallbacks
+    async def check_user_exists(self, user_id: str) -> Optional[str]:
    def check_user_exists(self, user_id: str):
        """
        Checks to see if a user with the given id exists. Will check case
        insensitively, but return None if there are multiple inexact matches.
@ -555,28 +632,25 @@ class AuthHandler(BaseHandler):
            user_id: complete @user:id
        Returns:
-            defer.Deferred: (unicode) canonical_user_id, or None if zero or
+            The canonical_user_id, or None if zero or multiple matches
            multiple matches
        Raises:
            UserDeactivatedError if a user is found but is deactivated.
        """
-        res = yield self._find_user_id_and_pwd_hash(user_id)
+        res = await self._find_user_id_and_pwd_hash(user_id)
        if res is not None:
            return res[0]
        return None
-    @defer.inlineCallbacks
+    async def _find_user_id_and_pwd_hash(
-    def _find_user_id_and_pwd_hash(self, user_id: str):
+        self, user_id: str
    ) -> Optional[Tuple[str, str]]:
        """Checks to see if a user with the given id exists. Will check case
        insensitively, but will return None if there are multiple inexact
        matches.
        Returns:
-            tuple: A 2-tuple of `(canonical_user_id, password_hash)`
+            A 2-tuple of `(canonical_user_id, password_hash)` or `None`
-            None: if there is not exactly one match
+            if there is not exactly one match
        """
-        user_infos = yield self.store.get_users_by_id_case_insensitive(user_id)
+        user_infos = await self.store.get_users_by_id_case_insensitive(user_id)
        result = None
        if not user_infos:
@ -609,8 +683,9 @@ class AuthHandler(BaseHandler):
        """
        return self._supported_login_types
-    @defer.inlineCallbacks
+    async def validate_login(
-    def validate_login(self, username: str, login_submission: Dict[str, Any]):
+        self, username: str, login_submission: Dict[str, Any]
    ) -> Tuple[str, Optional[Callable[[Dict[str, str]], None]]]:
        """Authenticates the user for the /login API
        Also used by the user-interactive auth flow to validate
@ -621,7 +696,7 @@ class AuthHandler(BaseHandler):
            login_submission: the whole of the login submission
                (including 'type' and other relevant fields)
        Returns:
-            Deferred[str, func]: canonical user id, and optional callback
+            A tuple of the canonical user id, and optional callback
                to be called once the access token and device id are issued
        Raises:
            StoreError if there was a problem accessing the database
@ -650,7 +725,7 @@ class AuthHandler(BaseHandler):
        for provider in self.password_providers:
            if hasattr(provider, "check_password") and login_type == LoginType.PASSWORD:
                known_login_type = True
-                is_valid = yield provider.check_password(qualified_user_id, password)
+                is_valid = await provider.check_password(qualified_user_id, password)
                if is_valid:
                    return qualified_user_id, None
@ -682,7 +757,7 @@ class AuthHandler(BaseHandler):
                    % (login_type, missing_fields),
                )
-            result = yield provider.check_auth(username, login_type, login_dict)
+            result = await provider.check_auth(username, login_type, login_dict)
            if result:
                if isinstance(result, str):
                    result = (result, None)
@ -691,8 +766,8 @@ class AuthHandler(BaseHandler):
        if login_type == LoginType.PASSWORD and self.hs.config.password_localdb_enabled:
            known_login_type = True
-            canonical_user_id = yield self._check_local_password(
+            canonical_user_id = await self._check_local_password(
-                qualified_user_id, password
+                qualified_user_id, password  # type: ignore
            )
            if canonical_user_id:
@ -705,8 +780,9 @@ class AuthHandler(BaseHandler):
        # login, it turns all LoginErrors into a 401 anyway.
        raise LoginError(403, "Invalid password", errcode=Codes.FORBIDDEN)
-    @defer.inlineCallbacks
+    async def check_password_provider_3pid(
-    def check_password_provider_3pid(self, medium: str, address: str, password: str):
+        self, medium: str, address: str, password: str
    ) -> Tuple[Optional[str], Optional[Callable[[Dict[str, str]], None]]]:
        """Check if a password provider is able to validate a thirdparty login
        Args:
@ -715,9 +791,8 @@ class AuthHandler(BaseHandler):
            password: The password of the user.
        Returns:
-            Deferred[(str|None, func|None)]: A tuple of `(user_id,
+            A tuple of `(user_id, callback)`. If authentication is successful,
-            callback)`. If authentication is successful, `user_id` is a `str`
+            `user_id`is the authenticated, canonical user ID. `callback` is
            containing the authenticated, canonical user ID. `callback` is
            then either a function to be later run after the server has
            completed login/registration, or `None`. If authentication was
            unsuccessful, `user_id` and `callback` are both `None`.
@ -729,7 +804,7 @@ class AuthHandler(BaseHandler):
                # success, to a str (which is the user_id) or a tuple of
                # (user_id, callback_func), where callback_func should be run
                # after we've finished everything else
-                result = yield provider.check_3pid_auth(medium, address, password)
+                result = await provider.check_3pid_auth(medium, address, password)
                if result:
                    # Check if the return value is a str or a tuple
                    if isinstance(result, str):
@ -739,8 +814,7 @@ class AuthHandler(BaseHandler):
        return None, None
-    @defer.inlineCallbacks
+    async def _check_local_password(self, user_id: str, password: str) -> Optional[str]:
    def _check_local_password(self, user_id: str, password: str):
        """Authenticate a user against the local password database.
        user_id is checked case insensitively, but will return None if there are
@ -750,28 +824,26 @@ class AuthHandler(BaseHandler):
            user_id: complete @user:id
            password: the provided password
        Returns:
-            Deferred[unicode] the canonical_user_id, or Deferred[None] if
+            The canonical_user_id, or None if unknown user/bad password
                unknown user/bad password
        """
-        lookupres = yield self._find_user_id_and_pwd_hash(user_id)
+        lookupres = await self._find_user_id_and_pwd_hash(user_id)
        if not lookupres:
            return None
        (user_id, password_hash) = lookupres
        # If the password hash is None, the account has likely been deactivated
        if not password_hash:
-            deactivated = yield self.store.get_user_deactivated_status(user_id)
+            deactivated = await self.store.get_user_deactivated_status(user_id)
            if deactivated:
                raise UserDeactivatedError("This account has been deactivated")
-        result = yield self.validate_hash(password, password_hash)
+        result = await self.validate_hash(password, password_hash)
        if not result:
            logger.warning("Failed password login for user %s", user_id)
            return None
        return user_id
-    @defer.inlineCallbacks
+    async def validate_short_term_login_token_and_get_user_id(self, login_token: str):
    def validate_short_term_login_token_and_get_user_id(self, login_token: str):
        auth_api = self.hs.get_auth()
        user_id = None
        try:
@ -781,26 +853,23 @@ class AuthHandler(BaseHandler):
        except Exception:
            raise AuthError(403, "Invalid token", errcode=Codes.FORBIDDEN)
-        yield self.auth.check_auth_blocking(user_id)
+        await self.auth.check_auth_blocking(user_id)
        return user_id
-    @defer.inlineCallbacks
+    async def delete_access_token(self, access_token: str):
    def delete_access_token(self, access_token: str):
        """Invalidate a single access token
        Args:
            access_token: access token to be deleted
        Returns:
            Deferred
        """
-        user_info = yield self.auth.get_user_by_access_token(access_token)
+        user_info = await self.auth.get_user_by_access_token(access_token)
-        yield self.store.delete_access_token(access_token)
+        await self.store.delete_access_token(access_token)
        # see if any of our auth providers want to know about this
        for provider in self.password_providers:
            if hasattr(provider, "on_logged_out"):
-                yield provider.on_logged_out(
+                await provider.on_logged_out(
                    user_id=str(user_info["user"]),
                    device_id=user_info["device_id"],
                    access_token=access_token,
@ -808,12 +877,11 @@ class AuthHandler(BaseHandler):
        # delete pushers associated with this access token
        if user_info["token_id"] is not None:
-            yield self.hs.get_pusherpool().remove_pushers_by_access_token(
+            await self.hs.get_pusherpool().remove_pushers_by_access_token(
                str(user_info["user"]), (user_info["token_id"],)
            )
-    @defer.inlineCallbacks
+    async def delete_access_tokens_for_user(
    def delete_access_tokens_for_user(
        self,
        user_id: str,
        except_token_id: Optional[str] = None,
@ -827,10 +895,8 @@ class AuthHandler(BaseHandler):
            device_id:  ID of device the tokens are associated with.
                If None, tokens associated with any device (or no device) will
                be deleted
        Returns:
            Deferred
        """
-        tokens_and_devices = yield self.store.user_delete_access_tokens(
+        tokens_and_devices = await self.store.user_delete_access_tokens(
            user_id, except_token_id=except_token_id, device_id=device_id
        )
@ -838,17 +904,18 @@ class AuthHandler(BaseHandler):
        for provider in self.password_providers:
            if hasattr(provider, "on_logged_out"):
                for token, token_id, device_id in tokens_and_devices:
-                    yield provider.on_logged_out(
+                    await provider.on_logged_out(
                        user_id=user_id, device_id=device_id, access_token=token
                    )
        # delete pushers associated with the access tokens
-        yield self.hs.get_pusherpool().remove_pushers_by_access_token(
+        await self.hs.get_pusherpool().remove_pushers_by_access_token(
            user_id, (token_id for _, token_id, _ in tokens_and_devices)
        )
-    @defer.inlineCallbacks
+    async def add_threepid(
-    def add_threepid(self, user_id: str, medium: str, address: str, validated_at: int):
+        self, user_id: str, medium: str, address: str, validated_at: int
    ):
        # check if medium has a valid value
        if medium not in ["email", "msisdn"]:
            raise SynapseError(
@ -869,14 +936,13 @@ class AuthHandler(BaseHandler):
        if medium == "email":
            address = address.lower()
-        yield self.store.user_add_threepid(
+        await self.store.user_add_threepid(
            user_id, medium, address, validated_at, self.hs.get_clock().time_msec()
        )
-    @defer.inlineCallbacks
+    async def delete_threepid(
    def delete_threepid(
        self, user_id: str, medium: str, address: str, id_server: Optional[str] = None
-    ):
+    ) -> bool:
        """Attempts to unbind the 3pid on the identity servers and deletes it
        from the local database.
@ -889,7 +955,7 @@ class AuthHandler(BaseHandler):
                identity server specified when binding (if known).
        Returns:
-            Deferred[bool]: Returns True if successfully unbound the 3pid on
+            Returns True if successfully unbound the 3pid on
            the identity server, False if identity server doesn't support the
            unbind API.
        """
@ -899,28 +965,21 @@ class AuthHandler(BaseHandler):
            address = address.lower()
        identity_handler = self.hs.get_handlers().identity_handler
-        result = yield identity_handler.try_unbind_threepid(
+        result = await identity_handler.try_unbind_threepid(
            user_id, {"medium": medium, "address": address, "id_server": id_server}
        )
-        yield self.store.user_delete_threepid(user_id, medium, address)
+        await self.store.user_delete_threepid(user_id, medium, address)
        return result
-    def _save_session(self, session: Dict[str, Any]) -> None:
+    async def hash(self, password: str) -> str:
        """Update the last used time on the session to now and add it back to the session store."""
        # TODO: Persistent storage
        logger.debug("Saving session %s", session)
        session["last_used"] = self.hs.get_clock().time_msec()
        self.sessions[session["id"]] = session
    def hash(self, password: str):
        """Computes a secure hash of password.
        Args:
            password: Password to hash.
        Returns:
-            Deferred(unicode): Hashed password.
+            Hashed password.
        """
        def _do_hash():
@ -932,9 +991,11 @@ class AuthHandler(BaseHandler):
                bcrypt.gensalt(self.bcrypt_rounds),
            ).decode("ascii")
-        return defer_to_thread(self.hs.get_reactor(), _do_hash)
+        return await defer_to_thread(self.hs.get_reactor(), _do_hash)
-    def validate_hash(self, password: str, stored_hash: bytes):
+    async def validate_hash(
        self, password: str, stored_hash: Union[bytes, str]
    ) -> bool:
        """Validates that self.hash(password) == stored_hash.
        Args:
@ -942,7 +1003,7 @@ class AuthHandler(BaseHandler):
            stored_hash: Expected hash value.
        Returns:
-            Deferred(bool): Whether self.hash(password) == stored_hash.
+            Whether self.hash(password) == stored_hash.
        """
        def _do_validate_hash():
@ -958,11 +1019,57 @@ class AuthHandler(BaseHandler):
            if not isinstance(stored_hash, bytes):
                stored_hash = stored_hash.encode("ascii")
-            return defer_to_thread(self.hs.get_reactor(), _do_validate_hash)
+            return await defer_to_thread(self.hs.get_reactor(), _do_validate_hash)
        else:
-            return defer.succeed(False)
+            return False
-    def complete_sso_login(
+    async def start_sso_ui_auth(self, redirect_url: str, session_id: str) -> str:
        """
        Get the HTML for the SSO redirect confirmation page.
        Args:
            redirect_url: The URL to redirect to the SSO provider.
            session_id: The user interactive authentication session ID.
        Returns:
            The HTML to render.
        """
        try:
            session = await self.store.get_ui_auth_session(session_id)
        except StoreError:
            raise SynapseError(400, "Unknown session ID: %s" % (session_id,))
        return self._sso_auth_confirm_template.render(
            description=session.description, redirect_url=redirect_url,
        )
    async def complete_sso_ui_auth(
        self, registered_user_id: str, session_id: str, request: SynapseRequest,
    ):
        """Having figured out a mxid for this user, complete the HTTP request
        Args:
            registered_user_id: The registered user ID to complete SSO login for.
            request: The request to complete.
            client_redirect_url: The URL to which to redirect the user at the end of the
                process.
        """
        # Mark the stage of the authentication as successful.
        # Save the user who authenticated with SSO, this will be used to ensure
        # that the account be modified is also the person who logged in.
        await self.store.mark_ui_auth_stage_complete(
            session_id, LoginType.SSO, registered_user_id
        )
        # Render the HTML and return.
        html_bytes = self._sso_auth_success_template.encode("utf-8")
        request.setResponseCode(200)
        request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
        request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))
        request.write(html_bytes)
        finish_request(request)
    async def complete_sso_login(
        self,
        registered_user_id: str,
        request: SynapseRequest,
@ -976,6 +1083,32 @@ class AuthHandler(BaseHandler):
            client_redirect_url: The URL to which to redirect the user at the end of the
                process.
        """
        # If the account has been deactivated, do not proceed with the login
        # flow.
        deactivated = await self.store.get_user_deactivated_status(registered_user_id)
        if deactivated:
            html_bytes = self._sso_account_deactivated_template.encode("utf-8")
            request.setResponseCode(403)
            request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
            request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))
            request.write(html_bytes)
            finish_request(request)
            return
        self._complete_sso_login(registered_user_id, request, client_redirect_url)
    def _complete_sso_login(
        self,
        registered_user_id: str,
        request: SynapseRequest,
        client_redirect_url: str,
    ):
        """
        The synchronous portion of complete_sso_login.
        This exists purely for backwards compatibility of synapse.module_api.ModuleApi.
        """
        # Create a login token
        login_token = self.macaroon_gen.generate_short_term_login_token(
            registered_user_id
@ -1001,7 +1134,7 @@ class AuthHandler(BaseHandler):
        # URL we redirect users to.
        redirect_url_no_params = client_redirect_url.split("?")[0]
-        html = self._sso_redirect_confirm_template.render(
+        html_bytes = self._sso_redirect_confirm_template.render(
            display_url=redirect_url_no_params,
            redirect_url=redirect_url,
            server_name=self._server_name,
@ -1009,8 +1142,8 @@ class AuthHandler(BaseHandler):
        request.setResponseCode(200)
        request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
-        request.setHeader(b"Content-Length", b"%d" % (len(html),))
+        request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))
-        request.write(html)
+        request.write(html_bytes)
        finish_request(request)
    @staticmethod
--- a/synapse/handlers/cas_handler.py
+++ b/synapse/handlers/cas_handler.py
@ -0,0 +1,221 @@
 # -*- coding: utf-8 -*-
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
 import xml.etree.ElementTree as ET
 from typing import Dict, Optional, Tuple
 from six.moves import urllib
 from twisted.web.client import PartialDownloadError
 from synapse.api.errors import Codes, LoginError
 from synapse.http.site import SynapseRequest
 from synapse.types import UserID, map_username_to_mxid_localpart
 logger = logging.getLogger(__name__)
 class CasHandler:
    """
    Utility class for to handle the response from a CAS SSO service.
    Args:
        hs (synapse.server.HomeServer)
    """
    def __init__(self, hs):
        self._hostname = hs.hostname
        self._auth_handler = hs.get_auth_handler()
        self._registration_handler = hs.get_registration_handler()
        self._cas_server_url = hs.config.cas_server_url
        self._cas_service_url = hs.config.cas_service_url
        self._cas_displayname_attribute = hs.config.cas_displayname_attribute
        self._cas_required_attributes = hs.config.cas_required_attributes
        self._http_client = hs.get_proxied_http_client()
    def _build_service_param(self, args: Dict[str, str]) -> str:
        """
        Generates a value to use as the "service" parameter when redirecting or
        querying the CAS service.
        Args:
            args: Additional arguments to include in the final redirect URL.
        Returns:
            The URL to use as a "service" parameter.
        """
        return "%s%s?%s" % (
            self._cas_service_url,
            "/_matrix/client/r0/login/cas/ticket",
            urllib.parse.urlencode(args),
        )
    async def _validate_ticket(
        self, ticket: str, service_args: Dict[str, str]
    ) -> Tuple[str, Optional[str]]:
        """
        Validate a CAS ticket with the server, parse the response, and return the user and display name.
        Args:
            ticket: The CAS ticket from the client.
            service_args: Additional arguments to include in the service URL.
                Should be the same as those passed to `get_redirect_url`.
        """
        uri = self._cas_server_url + "/proxyValidate"
        args = {
            "ticket": ticket,
            "service": self._build_service_param(service_args),
        }
        try:
            body = await self._http_client.get_raw(uri, args)
        except PartialDownloadError as pde:
            # Twisted raises this error if the connection is closed,
            # even if that's being used old-http style to signal end-of-data
            body = pde.response
        user, attributes = self._parse_cas_response(body)
        displayname = attributes.pop(self._cas_displayname_attribute, None)
        for required_attribute, required_value in self._cas_required_attributes.items():
            # If required attribute was not in CAS Response - Forbidden
            if required_attribute not in attributes:
                raise LoginError(401, "Unauthorized", errcode=Codes.UNAUTHORIZED)
            # Also need to check value
            if required_value is not None:
                actual_value = attributes[required_attribute]
                # If required attribute value does not match expected - Forbidden
                if required_value != actual_value:
                    raise LoginError(401, "Unauthorized", errcode=Codes.UNAUTHORIZED)
        return user, displayname
    def _parse_cas_response(
        self, cas_response_body: str
    ) -> Tuple[str, Dict[str, Optional[str]]]:
        """
        Retrieve the user and other parameters from the CAS response.
        Args:
            cas_response_body: The response from the CAS query.
        Returns:
            A tuple of the user and a mapping of other attributes.
        """
        user = None
        attributes = {}
        try:
            root = ET.fromstring(cas_response_body)
            if not root.tag.endswith("serviceResponse"):
                raise Exception("root of CAS response is not serviceResponse")
            success = root[0].tag.endswith("authenticationSuccess")
            for child in root[0]:
                if child.tag.endswith("user"):
                    user = child.text
                if child.tag.endswith("attributes"):
                    for attribute in child:
                        # ElementTree library expands the namespace in
                        # attribute tags to the full URL of the namespace.
                        # We don't care about namespace here and it will always
                        # be encased in curly braces, so we remove them.
                        tag = attribute.tag
                        if "}" in tag:
                            tag = tag.split("}")[1]
                        attributes[tag] = attribute.text
            if user is None:
                raise Exception("CAS response does not contain user")
        except Exception:
            logger.exception("Error parsing CAS response")
            raise LoginError(401, "Invalid CAS response", errcode=Codes.UNAUTHORIZED)
        if not success:
            raise LoginError(
                401, "Unsuccessful CAS response", errcode=Codes.UNAUTHORIZED
            )
        return user, attributes
    def get_redirect_url(self, service_args: Dict[str, str]) -> str:
        """
        Generates a URL for the CAS server where the client should be redirected.
        Args:
            service_args: Additional arguments to include in the final redirect URL.
        Returns:
            The URL to redirect the client to.
        """
        args = urllib.parse.urlencode(
            {"service": self._build_service_param(service_args)}
        )
        return "%s/login?%s" % (self._cas_server_url, args)
    async def handle_ticket(
        self,
        request: SynapseRequest,
        ticket: str,
        client_redirect_url: Optional[str],
        session: Optional[str],
    ) -> None:
        """
        Called once the user has successfully authenticated with the SSO.
        Validates a CAS ticket sent by the client and completes the auth process.
        If the user interactive authentication session is provided, marks the
        UI Auth session as complete, then returns an HTML page notifying the
        user they are done.
        Otherwise, this registers the user if necessary, and then returns a
        redirect (with a login token) to the client.
        Args:
            request: the incoming request from the browser. We'll
                respond to it with a redirect or an HTML page.
            ticket: The CAS ticket provided by the client.
            client_redirect_url: the redirectUrl parameter from the `/cas/ticket` HTTP request, if given.
                This should be the same as the redirectUrl from the original `/login/sso/redirect` request.
            session: The session parameter from the `/cas/ticket` HTTP request, if given.
                This should be the UI Auth session id.
        """
        args = {}
        if client_redirect_url:
            args["redirectUrl"] = client_redirect_url
        if session:
            args["session"] = session
        username, user_display_name = await self._validate_ticket(ticket, args)
        localpart = map_username_to_mxid_localpart(username)
        user_id = UserID(localpart, self._hostname).to_string()
        registered_user_id = await self._auth_handler.check_user_exists(user_id)
        if session:
            await self._auth_handler.complete_sso_ui_auth(
                registered_user_id, session, request,
            )
        else:
            if not registered_user_id:
                registered_user_id = await self._registration_handler.register_user(
                    localpart=localpart, default_display_name=user_display_name
                )
            await self._auth_handler.complete_sso_login(
                registered_user_id, request, client_redirect_url
            )
--- a/synapse/handlers/device.py
+++ b/synapse/handlers/device.py
@ -338,9 +338,11 @@ class DeviceHandler(DeviceWorkerHandler):
            else:
                raise
-        yield self._auth_handler.delete_access_tokens_for_user(
+        yield defer.ensureDeferred(
            self._auth_handler.delete_access_tokens_for_user(
                user_id, device_id=device_id
            )
        )
        yield self.store.delete_e2e_keys_by_device(user_id=user_id, device_id=device_id)
@ -391,9 +393,11 @@ class DeviceHandler(DeviceWorkerHandler):
        # Delete access tokens and e2e keys for each device. Not optimised as it is not
        # considered as part of a critical path.
        for device_id in device_ids:
-            yield self._auth_handler.delete_access_tokens_for_user(
+            yield defer.ensureDeferred(
                self._auth_handler.delete_access_tokens_for_user(
                    user_id, device_id=device_id
                )
            )
            yield self.store.delete_e2e_keys_by_device(
                user_id=user_id, device_id=device_id
            )
--- a/synapse/handlers/directory.py
+++ b/synapse/handlers/directory.py
@ -86,8 +86,7 @@ class DirectoryHandler(BaseHandler):
            room_alias, room_id, servers, creator=creator
        )
-    @defer.inlineCallbacks
+    async def create_association(
    def create_association(
        self,
        requester: Requester,
        room_alias: RoomAlias,
@ -127,8 +126,12 @@ class DirectoryHandler(BaseHandler):
                    errcode=Codes.EXCLUSIVE,
                )
        else:
-            if self.require_membership and check_membership:
+            # Server admins are not subject to the same constraints as normal
-                rooms_for_user = yield self.store.get_rooms_for_user(user_id)
+            # users when creating an alias (e.g. being in the room).
            is_admin = await self.auth.is_server_admin(requester.user)
            if (self.require_membership and check_membership) and not is_admin:
                rooms_for_user = await self.store.get_rooms_for_user(user_id)
                if room_id not in rooms_for_user:
                    raise AuthError(
                        403, "You must be in the room to create an alias for it"
@ -145,7 +148,7 @@ class DirectoryHandler(BaseHandler):
                # per alias creation rule?
                raise SynapseError(403, "Not allowed to create alias")
-            can_create = yield self.can_modify_alias(room_alias, user_id=user_id)
+            can_create = await self.can_modify_alias(room_alias, user_id=user_id)
            if not can_create:
                raise AuthError(
                    400,
@ -153,10 +156,9 @@ class DirectoryHandler(BaseHandler):
                    errcode=Codes.EXCLUSIVE,
                )
-        yield self._create_association(room_alias, room_id, servers, creator=user_id)
+        await self._create_association(room_alias, room_id, servers, creator=user_id)
-    @defer.inlineCallbacks
+    async def delete_association(self, requester: Requester, room_alias: RoomAlias):
    def delete_association(self, requester: Requester, room_alias: RoomAlias):
        """Remove an alias from the directory
        (this is only meant for human users; AS users should call
@ -180,7 +182,7 @@ class DirectoryHandler(BaseHandler):
        user_id = requester.user.to_string()
        try:
-            can_delete = yield self._user_can_delete_alias(room_alias, user_id)
+            can_delete = await self._user_can_delete_alias(room_alias, user_id)
        except StoreError as e:
            if e.code == 404:
                raise NotFoundError("Unknown room alias")
@ -189,7 +191,7 @@ class DirectoryHandler(BaseHandler):
        if not can_delete:
            raise AuthError(403, "You don't have permission to delete the alias.")
-        can_delete = yield self.can_modify_alias(room_alias, user_id=user_id)
+        can_delete = await self.can_modify_alias(room_alias, user_id=user_id)
        if not can_delete:
            raise SynapseError(
                400,
@ -197,10 +199,10 @@ class DirectoryHandler(BaseHandler):
                errcode=Codes.EXCLUSIVE,
            )
-        room_id = yield self._delete_association(room_alias)
+        room_id = await self._delete_association(room_alias)
        try:
-            yield self._update_canonical_alias(requester, user_id, room_id, room_alias)
+            await self._update_canonical_alias(requester, user_id, room_id, room_alias)
        except AuthError as e:
            logger.info("Failed to update alias events: %s", e)
@ -292,15 +294,14 @@ class DirectoryHandler(BaseHandler):
                Codes.NOT_FOUND,
            )
-    @defer.inlineCallbacks
+    async def _update_canonical_alias(
    def _update_canonical_alias(
        self, requester: Requester, user_id: str, room_id: str, room_alias: RoomAlias
    ):
        """
        Send an updated canonical alias event if the removed alias was set as
        the canonical alias or listed in the alt_aliases field.
        """
-        alias_event = yield self.state.get_current_state(
+        alias_event = await self.state.get_current_state(
            room_id, EventTypes.CanonicalAlias, ""
        )
@ -331,7 +332,7 @@ class DirectoryHandler(BaseHandler):
                del content["alt_aliases"]
        if send_update:
-            yield self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                requester,
                {
                    "type": EventTypes.CanonicalAlias,
@ -372,8 +373,7 @@ class DirectoryHandler(BaseHandler):
        # either no interested services, or no service with an exclusive lock
        return defer.succeed(True)
-    @defer.inlineCallbacks
+    async def _user_can_delete_alias(self, alias: RoomAlias, user_id: str):
    def _user_can_delete_alias(self, alias: RoomAlias, user_id: str):
        """Determine whether a user can delete an alias.
        One of the following must be true:
@ -384,24 +384,23 @@ class DirectoryHandler(BaseHandler):
           for the current room.
        """
-        creator = yield self.store.get_room_alias_creator(alias.to_string())
+        creator = await self.store.get_room_alias_creator(alias.to_string())
        if creator is not None and creator == user_id:
            return True
        # Resolve the alias to the corresponding room.
-        room_mapping = yield self.get_association(alias)
+        room_mapping = await self.get_association(alias)
        room_id = room_mapping["room_id"]
        if not room_id:
            return False
-        res = yield self.auth.check_can_change_room_list(
+        res = await self.auth.check_can_change_room_list(
            room_id, UserID.from_string(user_id)
        )
        return res
-    @defer.inlineCallbacks
+    async def edit_published_room_list(
    def edit_published_room_list(
        self, requester: Requester, room_id: str, visibility: str
    ):
        """Edit the entry of the room in the published room list.
@ -429,11 +428,11 @@ class DirectoryHandler(BaseHandler):
                403, "This user is not permitted to publish rooms to the room list"
            )
-        room = yield self.store.get_room(room_id)
+        room = await self.store.get_room(room_id)
        if room is None:
            raise SynapseError(400, "Unknown room")
-        can_change_room_list = yield self.auth.check_can_change_room_list(
+        can_change_room_list = await self.auth.check_can_change_room_list(
            room_id, requester.user
        )
        if not can_change_room_list:
@ -445,8 +444,8 @@ class DirectoryHandler(BaseHandler):
        making_public = visibility == "public"
        if making_public:
-            room_aliases = yield self.store.get_aliases_for_room(room_id)
+            room_aliases = await self.store.get_aliases_for_room(room_id)
-            canonical_alias = yield self.store.get_canonical_alias_for_room(room_id)
+            canonical_alias = await self.store.get_canonical_alias_for_room(room_id)
            if canonical_alias:
                room_aliases.append(canonical_alias)
@ -458,7 +457,7 @@ class DirectoryHandler(BaseHandler):
                # per alias creation rule?
                raise SynapseError(403, "Not allowed to publish room")
-        yield self.store.set_room_is_public(room_id, making_public)
+        await self.store.set_room_is_public(room_id, making_public)
    @defer.inlineCallbacks
    def edit_published_appservice_room_list(
--- a/synapse/handlers/events.py
+++ b/synapse/handlers/events.py
@ -19,6 +19,7 @@ import random
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, SynapseError
 from synapse.events import EventBase
 from synapse.handlers.presence import format_user_presence_state
 from synapse.logging.utils import log_function
 from synapse.types import UserID
 from synapse.visibility import filter_events_for_client
@ -97,6 +98,8 @@ class EventStreamHandler(BaseHandler):
                explicit_room_id=room_id,
            )
            time_now = self.clock.time_msec()
            # When the user joins a new room, or another user joins a currently
            # joined room, we need to send down presence for those users.
            to_add = []
@ -112,19 +115,20 @@ class EventStreamHandler(BaseHandler):
                        users = await self.state.get_current_users_in_room(
                            event.room_id
                        )
                        states = await presence_handler.get_states(users, as_event=True)
                        to_add.extend(states)
                    else:
                        users = [event.state_key]
-                        ev = await presence_handler.get_state(
+                    states = await presence_handler.get_states(users)
-                            UserID.from_string(event.state_key), as_event=True
+                    to_add.extend(
                        {
                            "type": EventTypes.Presence,
                            "content": format_user_presence_state(state, time_now),
                        }
                        for state in states
                    )
                        to_add.append(ev)
            events.extend(to_add)
            time_now = self.clock.time_msec()
            chunks = await self._event_serializer.serialize_events(
                events,
                time_now,
--- a/synapse/handlers/federation.py
+++ b/synapse/handlers/federation.py
@ -49,6 +49,7 @@ from synapse.event_auth import auth_types_for_event
 from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.events.validator import EventValidator
 from synapse.handlers._base import BaseHandler
 from synapse.logging.context import (
    make_deferred_yieldable,
    nested_logging_context,
@ -69,10 +70,9 @@ from synapse.types import JsonDict, StateMap, UserID, get_domain_from_id
 from synapse.util.async_helpers import Linearizer, concurrently_execute
 from synapse.util.distributor import user_joined_room
 from synapse.util.retryutils import NotRetryingDestination
 from synapse.util.stringutils import shortstr
 from synapse.visibility import filter_events_for_server
 from ._base import BaseHandler
 logger = logging.getLogger(__name__)
@ -93,27 +93,6 @@ class _NewEventInfo:
    auth_events = attr.ib(type=Optional[StateMap[EventBase]], default=None)
 def shortstr(iterable, maxitems=5):
    """If iterable has maxitems or fewer, return the stringification of a list
    containing those items.
    Otherwise, return the stringification of a a list with the first maxitems items,
    followed by "...".
    Args:
        iterable (Iterable): iterable to truncate
        maxitems (int): number of items to return before truncating
    Returns:
        unicode
    """
    items = list(itertools.islice(iterable, maxitems + 1))
    if len(items) <= maxitems:
        return str(items)
    return "[" + ", ".join(repr(r) for r in items[:maxitems]) + ", ...]"
 class FederationHandler(BaseHandler):
    """Handles events that originated from federation.
        Responsible for:
@ -364,7 +343,7 @@ class FederationHandler(BaseHandler):
                    ours = await self.state_store.get_state_groups_ids(room_id, seen)
                    # state_maps is a list of mappings from (type, state_key) to event_id
-                    state_maps = list(ours.values())  # type: list[StateMap[str]]
+                    state_maps = list(ours.values())  # type: List[StateMap[str]]
                    # we don't need this any more, let's delete it.
                    del ours
@ -1715,16 +1694,15 @@ class FederationHandler(BaseHandler):
        return None
-    @defer.inlineCallbacks
+    async def get_state_for_pdu(self, room_id: str, event_id: str) -> List[EventBase]:
    def get_state_for_pdu(self, room_id, event_id):
        """Returns the state at the event. i.e. not including said event.
        """
-        event = yield self.store.get_event(
+        event = await self.store.get_event(
            event_id, allow_none=False, check_room_id=room_id
        )
-        state_groups = yield self.state_store.get_state_groups(room_id, [event_id])
+        state_groups = await self.state_store.get_state_groups(room_id, [event_id])
        if state_groups:
            _, state = list(iteritems(state_groups)).pop()
@ -1735,7 +1713,7 @@ class FederationHandler(BaseHandler):
                if "replaces_state" in event.unsigned:
                    prev_id = event.unsigned["replaces_state"]
                    if prev_id != event.event_id:
-                        prev_event = yield self.store.get_event(prev_id)
+                        prev_event = await self.store.get_event(prev_id)
                        results[(event.type, event.state_key)] = prev_event
                else:
                    del results[(event.type, event.state_key)]
@ -1745,15 +1723,14 @@ class FederationHandler(BaseHandler):
        else:
            return []
-    @defer.inlineCallbacks
+    async def get_state_ids_for_pdu(self, room_id: str, event_id: str) -> List[str]:
    def get_state_ids_for_pdu(self, room_id, event_id):
        """Returns the state at the event. i.e. not including said event.
        """
-        event = yield self.store.get_event(
+        event = await self.store.get_event(
            event_id, allow_none=False, check_room_id=room_id
        )
-        state_groups = yield self.state_store.get_state_groups_ids(room_id, [event_id])
+        state_groups = await self.state_store.get_state_groups_ids(room_id, [event_id])
        if state_groups:
            _, state = list(state_groups.items()).pop()
@ -1772,49 +1749,50 @@ class FederationHandler(BaseHandler):
        else:
            return []
    @defer.inlineCallbacks
    @log_function
-    def on_backfill_request(self, origin, room_id, pdu_list, limit):
+    async def on_backfill_request(
-        in_room = yield self.auth.check_host_in_room(room_id, origin)
+        self, origin: str, room_id: str, pdu_list: List[str], limit: int
    ) -> List[EventBase]:
        in_room = await self.auth.check_host_in_room(room_id, origin)
        if not in_room:
            raise AuthError(403, "Host not in room.")
        # Synapse asks for 100 events per backfill request. Do not allow more.
        limit = min(limit, 100)
-        events = yield self.store.get_backfill_events(room_id, pdu_list, limit)
+        events = await self.store.get_backfill_events(room_id, pdu_list, limit)
-        events = yield filter_events_for_server(self.storage, origin, events)
+        events = await filter_events_for_server(self.storage, origin, events)
        return events
    @defer.inlineCallbacks
    @log_function
-    def get_persisted_pdu(self, origin, event_id):
+    async def get_persisted_pdu(
        self, origin: str, event_id: str
    ) -> Optional[EventBase]:
        """Get an event from the database for the given server.
        Args:
-            origin [str]: hostname of server which is requesting the event; we
+            origin: hostname of server which is requesting the event; we
               will check that the server is allowed to see it.
-            event_id [str]: id of the event being requested
+            event_id: id of the event being requested
        Returns:
-            Deferred[EventBase|None]: None if we know nothing about the event;
+            None if we know nothing about the event; otherwise the (possibly-redacted) event.
                otherwise the (possibly-redacted) event.
        Raises:
            AuthError if the server is not currently in the room
        """
-        event = yield self.store.get_event(
+        event = await self.store.get_event(
            event_id, allow_none=True, allow_rejected=True
        )
        if event:
-            in_room = yield self.auth.check_host_in_room(event.room_id, origin)
+            in_room = await self.auth.check_host_in_room(event.room_id, origin)
            if not in_room:
                raise AuthError(403, "Host not in room.")
-            events = yield filter_events_for_server(self.storage, origin, [event])
+            events = await filter_events_for_server(self.storage, origin, [event])
            event = events[0]
            return event
        else:
@ -2418,7 +2396,7 @@ class FederationHandler(BaseHandler):
        """
        # exclude the state key of the new event from the current_state in the context.
        if event.is_state():
-            event_key = (event.type, event.state_key)
+            event_key = (event.type, event.state_key)  # type: Optional[Tuple[str, str]]
        else:
            event_key = None
        state_updates = {
@ -2584,9 +2562,8 @@ class FederationHandler(BaseHandler):
            "missing": [e.event_id for e in missing_locals],
        }
    @defer.inlineCallbacks
    @log_function
-    def exchange_third_party_invite(
+    async def exchange_third_party_invite(
        self, sender_user_id, target_user_id, room_id, signed
    ):
        third_party_invite = {"signed": signed}
@ -2602,16 +2579,16 @@ class FederationHandler(BaseHandler):
            "state_key": target_user_id,
        }
-        if (yield self.auth.check_host_in_room(room_id, self.hs.hostname)):
+        if await self.auth.check_host_in_room(room_id, self.hs.hostname):
-            room_version = yield self.store.get_room_version_id(room_id)
+            room_version = await self.store.get_room_version_id(room_id)
            builder = self.event_builder_factory.new(room_version, event_dict)
            EventValidator().validate_builder(builder)
-            event, context = yield self.event_creation_handler.create_new_client_event(
+            event, context = await self.event_creation_handler.create_new_client_event(
                builder=builder
            )
-            event_allowed = yield self.third_party_event_rules.check_event_allowed(
+            event_allowed = await self.third_party_event_rules.check_event_allowed(
                event, context
            )
            if not event_allowed:
@ -2623,7 +2600,7 @@ class FederationHandler(BaseHandler):
                    403, "This event is not allowed in this context", Codes.FORBIDDEN
                )
-            event, context = yield self.add_display_name_to_third_party_invite(
+            event, context = await self.add_display_name_to_third_party_invite(
                room_version, event_dict, event, context
            )
@ -2634,19 +2611,19 @@ class FederationHandler(BaseHandler):
            event.internal_metadata.send_on_behalf_of = self.hs.hostname
            try:
-                yield self.auth.check_from_context(room_version, event, context)
+                await self.auth.check_from_context(room_version, event, context)
            except AuthError as e:
                logger.warning("Denying new third party invite %r because %s", event, e)
                raise e
-            yield self._check_signature(event, context)
+            await self._check_signature(event, context)
            # We retrieve the room member handler here as to not cause a cyclic dependency
            member_handler = self.hs.get_room_member_handler()
-            yield member_handler.send_membership_event(None, event, context)
+            await member_handler.send_membership_event(None, event, context)
        else:
            destinations = {x.split(":", 1)[-1] for x in (sender_user_id, room_id)}
-            yield self.federation_client.forward_third_party_invite(
+            await self.federation_client.forward_third_party_invite(
                destinations, room_id, event_dict
            )
--- a/synapse/handlers/groups_local.py
+++ b/synapse/handlers/groups_local.py
@ -284,15 +284,14 @@ class GroupsLocalHandler(GroupsLocalWorkerHandler):
    set_group_join_policy = _create_rerouter("set_group_join_policy")
-    @defer.inlineCallbacks
+    async def create_group(self, group_id, user_id, content):
    def create_group(self, group_id, user_id, content):
        """Create a group
        """
        logger.info("Asking to create group with ID: %r", group_id)
        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.create_group(
+            res = await self.groups_server_handler.create_group(
                group_id, user_id, content
            )
            local_attestation = None
@ -301,10 +300,10 @@ class GroupsLocalHandler(GroupsLocalWorkerHandler):
            local_attestation = self.attestations.create_attestation(group_id, user_id)
            content["attestation"] = local_attestation
-            content["user_profile"] = yield self.profile_handler.get_profile(user_id)
+            content["user_profile"] = await self.profile_handler.get_profile(user_id)
            try:
-                res = yield self.transport_client.create_group(
+                res = await self.transport_client.create_group(
                    get_domain_from_id(group_id), group_id, user_id, content
                )
            except HttpResponseException as e:
@ -313,7 +312,7 @@ class GroupsLocalHandler(GroupsLocalWorkerHandler):
                raise SynapseError(502, "Failed to contact group server")
            remote_attestation = res["attestation"]
-            yield self.attestations.verify_attestation(
+            await self.attestations.verify_attestation(
                remote_attestation,
                group_id=group_id,
                user_id=user_id,
@ -321,7 +320,7 @@ class GroupsLocalHandler(GroupsLocalWorkerHandler):
            )
        is_publicised = content.get("publicise", False)
-        token = yield self.store.register_user_group_membership(
+        token = await self.store.register_user_group_membership(
            group_id,
            user_id,
            membership="join",
@ -482,12 +481,13 @@ class GroupsLocalHandler(GroupsLocalWorkerHandler):
        return {"state": "invite", "user_profile": user_profile}
-    @defer.inlineCallbacks
+    async def remove_user_from_group(
-    def remove_user_from_group(self, group_id, user_id, requester_user_id, content):
+        self, group_id, user_id, requester_user_id, content
    ):
        """Remove a user from a group
        """
        if user_id == requester_user_id:
-            token = yield self.store.register_user_group_membership(
+            token = await self.store.register_user_group_membership(
                group_id, user_id, membership="leave"
            )
            self.notifier.on_new_event("groups_key", token, users=[user_id])
@ -496,13 +496,13 @@ class GroupsLocalHandler(GroupsLocalWorkerHandler):
            # retry if the group server is currently down.
        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.remove_user_from_group(
+            res = await self.groups_server_handler.remove_user_from_group(
                group_id, user_id, requester_user_id, content
            )
        else:
            content["requester_user_id"] = requester_user_id
            try:
-                res = yield self.transport_client.remove_user_from_group(
+                res = await self.transport_client.remove_user_from_group(
                    get_domain_from_id(group_id),
                    group_id,
                    requester_user_id,
--- a/synapse/handlers/identity.py
+++ b/synapse/handlers/identity.py
@ -18,7 +18,7 @@
 """Utilities for interacting with Identity Servers"""
 import logging
-import urllib
+import urllib.parse
 from canonicaljson import json
 from signedjson.key import decode_verify_key_bytes
--- a/synapse/handlers/initial_sync.py
+++ b/synapse/handlers/initial_sync.py
@ -381,10 +381,16 @@ class InitialSyncHandler(BaseHandler):
                return []
            states = await presence_handler.get_states(
-                [m.user_id for m in room_members], as_event=True
+                [m.user_id for m in room_members]
            )
-            return states
+            return [
                {
                    "type": EventTypes.Presence,
                    "content": format_user_presence_state(s, time_now),
                }
                for s in states
            ]
        async def get_receipts():
            receipts = await self.store.get_linearized_receipts_for_room(
--- a/synapse/handlers/message.py
+++ b/synapse/handlers/message.py
@ -419,6 +419,8 @@ class EventCreationHandler(object):
        self._ephemeral_events_enabled = hs.config.enable_ephemeral_messages
        self._dummy_events_threshold = hs.config.dummy_events_threshold
    @defer.inlineCallbacks
    def create_event(
        self,
@ -626,8 +628,7 @@ class EventCreationHandler(object):
        msg = self._block_events_without_consent_error % {"consent_uri": consent_uri}
        raise ConsentNotGivenError(msg=msg, consent_uri=consent_uri)
-    @defer.inlineCallbacks
+    async def send_nonmember_event(self, requester, event, context, ratelimit=True):
    def send_nonmember_event(self, requester, event, context, ratelimit=True):
        """
        Persists and notifies local clients and federation of an event.
@ -647,7 +648,7 @@ class EventCreationHandler(object):
        assert self.hs.is_mine(user), "User must be our own: %s" % (user,)
        if event.is_state():
-            prev_state = yield self.deduplicate_state_event(event, context)
+            prev_state = await self.deduplicate_state_event(event, context)
            if prev_state is not None:
                logger.info(
                    "Not bothering to persist state event %s duplicated by %s",
@ -656,7 +657,7 @@ class EventCreationHandler(object):
                )
                return prev_state
-        yield self.handle_new_client_event(
+        await self.handle_new_client_event(
            requester=requester, event=event, context=context, ratelimit=ratelimit
        )
@ -683,8 +684,7 @@ class EventCreationHandler(object):
                return prev_event
        return
-    @defer.inlineCallbacks
+    async def create_and_send_nonmember_event(
    def create_and_send_nonmember_event(
        self, requester, event_dict, ratelimit=True, txn_id=None
    ):
        """
@ -698,8 +698,8 @@ class EventCreationHandler(object):
        # a situation where event persistence can't keep up, causing
        # extremities to pile up, which in turn leads to state resolution
        # taking longer.
-        with (yield self.limiter.queue(event_dict["room_id"])):
+        with (await self.limiter.queue(event_dict["room_id"])):
-            event, context = yield self.create_event(
+            event, context = await self.create_event(
                requester, event_dict, token_id=requester.access_token_id, txn_id=txn_id
            )
@ -709,7 +709,7 @@ class EventCreationHandler(object):
                    spam_error = "Spam is not permitted here"
                raise SynapseError(403, spam_error, Codes.FORBIDDEN)
-            yield self.send_nonmember_event(
+            await self.send_nonmember_event(
                requester, event, context, ratelimit=ratelimit
            )
        return event
@ -770,8 +770,7 @@ class EventCreationHandler(object):
        return (event, context)
    @measure_func("handle_new_client_event")
-    @defer.inlineCallbacks
+    async def handle_new_client_event(
    def handle_new_client_event(
        self, requester, event, context, ratelimit=True, extra_users=[]
    ):
        """Processes a new event. This includes checking auth, persisting it,
@ -794,9 +793,9 @@ class EventCreationHandler(object):
        ):
            room_version = event.content.get("room_version", RoomVersions.V1.identifier)
        else:
-            room_version = yield self.store.get_room_version_id(event.room_id)
+            room_version = await self.store.get_room_version_id(event.room_id)
-        event_allowed = yield self.third_party_event_rules.check_event_allowed(
+        event_allowed = await self.third_party_event_rules.check_event_allowed(
            event, context
        )
        if not event_allowed:
@ -805,7 +804,7 @@ class EventCreationHandler(object):
            )
        try:
-            yield self.auth.check_from_context(room_version, event, context)
+            await self.auth.check_from_context(room_version, event, context)
        except AuthError as err:
            logger.warning("Denying new event %r because %s", event, err)
            raise err
@ -818,7 +817,7 @@ class EventCreationHandler(object):
            logger.exception("Failed to encode content: %r", event.content)
            raise
-        yield self.action_generator.handle_push_actions_for_event(event, context)
+        await self.action_generator.handle_push_actions_for_event(event, context)
        # reraise does not allow inlineCallbacks to preserve the stacktrace, so we
        # hack around with a try/finally instead.
@ -826,7 +825,7 @@ class EventCreationHandler(object):
        try:
            # If we're a worker we need to hit out to the master.
            if self.config.worker_app:
-                yield self.send_event_to_master(
+                await self.send_event_to_master(
                    event_id=event.event_id,
                    store=self.store,
                    requester=requester,
@ -838,7 +837,7 @@ class EventCreationHandler(object):
                success = True
                return
-            yield self.persist_and_notify_client_event(
+            await self.persist_and_notify_client_event(
                requester, event, context, ratelimit=ratelimit, extra_users=extra_users
            )
@ -852,7 +851,38 @@ class EventCreationHandler(object):
                )
    @defer.inlineCallbacks
-    def persist_and_notify_client_event(
+    def _validate_canonical_alias(
        self, directory_handler, room_alias_str, expected_room_id
    ):
        """
        Ensure that the given room alias points to the expected room ID.
        Args:
            directory_handler: The directory handler object.
            room_alias_str: The room alias to check.
            expected_room_id: The room ID that the alias should point to.
        """
        room_alias = RoomAlias.from_string(room_alias_str)
        try:
            mapping = yield directory_handler.get_association(room_alias)
        except SynapseError as e:
            # Turn M_NOT_FOUND errors into M_BAD_ALIAS errors.
            if e.errcode == Codes.NOT_FOUND:
                raise SynapseError(
                    400,
                    "Room alias %s does not point to the room" % (room_alias_str,),
                    Codes.BAD_ALIAS,
                )
            raise
        if mapping["room_id"] != expected_room_id:
            raise SynapseError(
                400,
                "Room alias %s does not point to the room" % (room_alias_str,),
                Codes.BAD_ALIAS,
            )
    async def persist_and_notify_client_event(
        self, requester, event, context, ratelimit=True, extra_users=[]
    ):
        """Called when we have fully built the event, have already
@ -869,7 +899,7 @@ class EventCreationHandler(object):
            # user is actually admin or not).
            is_admin_redaction = False
            if event.type == EventTypes.Redaction:
-                original_event = yield self.store.get_event(
+                original_event = await self.store.get_event(
                    event.redacts,
                    redact_behaviour=EventRedactBehaviour.AS_IS,
                    get_prev_content=False,
@ -881,11 +911,11 @@ class EventCreationHandler(object):
                    original_event and event.sender != original_event.sender
                )
-            yield self.base_handler.ratelimit(
+            await self.base_handler.ratelimit(
                requester, is_admin_redaction=is_admin_redaction
            )
-        yield self.base_handler.maybe_kick_guest_users(event, context)
+        await self.base_handler.maybe_kick_guest_users(event, context)
        if event.type == EventTypes.CanonicalAlias:
            # Validate a newly added alias or newly added alt_aliases.
@ -895,7 +925,7 @@ class EventCreationHandler(object):
            original_event_id = event.unsigned.get("replaces_state")
            if original_event_id:
-                original_event = yield self.store.get_event(original_event_id)
+                original_event = await self.store.get_event(original_event_id)
                if original_event:
                    original_alias = original_event.content.get("alias", None)
@ -905,14 +935,8 @@ class EventCreationHandler(object):
            room_alias_str = event.content.get("alias", None)
            directory_handler = self.hs.get_handlers().directory_handler
            if room_alias_str and room_alias_str != original_alias:
-                room_alias = RoomAlias.from_string(room_alias_str)
+                await self._validate_canonical_alias(
-                mapping = yield directory_handler.get_association(room_alias)
+                    directory_handler, room_alias_str, event.room_id
                if mapping["room_id"] != event.room_id:
                    raise SynapseError(
                        400,
                        "Room alias %s does not point to the room" % (room_alias_str,),
                        Codes.BAD_ALIAS,
                )
            # Check that alt_aliases is the proper form.
@ -931,15 +955,8 @@ class EventCreationHandler(object):
            new_alt_aliases = set(alt_aliases) - set(original_alt_aliases)
            if new_alt_aliases:
                for alias_str in new_alt_aliases:
-                    room_alias = RoomAlias.from_string(alias_str)
+                    await self._validate_canonical_alias(
-                    mapping = yield directory_handler.get_association(room_alias)
+                        directory_handler, alias_str, event.room_id
                    if mapping["room_id"] != event.room_id:
                        raise SynapseError(
                            400,
                            "Room alias %s does not point to the room"
                            % (room_alias_str,),
                            Codes.BAD_ALIAS,
                    )
        federation_handler = self.hs.get_handlers().federation_handler
@ -950,7 +967,7 @@ class EventCreationHandler(object):
                def is_inviter_member_event(e):
                    return e.type == EventTypes.Member and e.sender == event.sender
-                current_state_ids = yield context.get_current_state_ids()
+                current_state_ids = await context.get_current_state_ids()
                state_to_include_ids = [
                    e_id
@ -959,7 +976,7 @@ class EventCreationHandler(object):
                    or k == (EventTypes.Member, event.sender)
                ]
-                state_to_include = yield self.store.get_events(state_to_include_ids)
+                state_to_include = await self.store.get_events(state_to_include_ids)
                event.unsigned["invite_room_state"] = [
                    {
@ -977,8 +994,8 @@ class EventCreationHandler(object):
                    # way? If we have been invited by a remote server, we need
                    # to get them to sign the event.
-                    returned_invite = yield defer.ensureDeferred(
+                    returned_invite = await federation_handler.send_invite(
-                        federation_handler.send_invite(invitee.domain, event)
+                        invitee.domain, event
                    )
                    event.unsigned.pop("room_state", None)
@ -986,7 +1003,7 @@ class EventCreationHandler(object):
                    event.signatures.update(returned_invite.signatures)
        if event.type == EventTypes.Redaction:
-            original_event = yield self.store.get_event(
+            original_event = await self.store.get_event(
                event.redacts,
                redact_behaviour=EventRedactBehaviour.AS_IS,
                get_prev_content=False,
@ -1002,14 +1019,14 @@ class EventCreationHandler(object):
                if original_event.room_id != event.room_id:
                    raise SynapseError(400, "Cannot redact event from a different room")
-            prev_state_ids = yield context.get_prev_state_ids()
+            prev_state_ids = await context.get_prev_state_ids()
-            auth_events_ids = yield self.auth.compute_auth_events(
+            auth_events_ids = await self.auth.compute_auth_events(
                event, prev_state_ids, for_verification=True
            )
-            auth_events = yield self.store.get_events(auth_events_ids)
+            auth_events = await self.store.get_events(auth_events_ids)
            auth_events = {(e.type, e.state_key): e for e in auth_events.values()}
-            room_version = yield self.store.get_room_version_id(event.room_id)
+            room_version = await self.store.get_room_version_id(event.room_id)
            room_version_obj = KNOWN_ROOM_VERSIONS[room_version]
            if event_auth.check_redaction(
@ -1028,11 +1045,11 @@ class EventCreationHandler(object):
                event.internal_metadata.recheck_redaction = False
        if event.type == EventTypes.Create:
-            prev_state_ids = yield context.get_prev_state_ids()
+            prev_state_ids = await context.get_prev_state_ids()
            if prev_state_ids:
                raise AuthError(403, "Changing the room create event is forbidden")
-        event_stream_id, max_stream_id = yield self.storage.persistence.persist_event(
+        event_stream_id, max_stream_id = await self.storage.persistence.persist_event(
            event, context=context
        )
@ -1040,7 +1057,7 @@ class EventCreationHandler(object):
            # If there's an expiry timestamp on the event, schedule its expiry.
            self._message_handler.maybe_schedule_expiry(event)
-        yield self.pusher_pool.on_new_notifications(event_stream_id, max_stream_id)
+        await self.pusher_pool.on_new_notifications(event_stream_id, max_stream_id)
        def _notify():
            try:
@ -1064,14 +1081,13 @@ class EventCreationHandler(object):
        except Exception:
            logger.exception("Error bumping presence active time")
-    @defer.inlineCallbacks
+    async def _send_dummy_events_to_fill_extremities(self):
    def _send_dummy_events_to_fill_extremities(self):
        """Background task to send dummy events into rooms that have a large
        number of extremities
        """
        self._expire_rooms_to_exclude_from_dummy_event_insertion()
-        room_ids = yield self.store.get_rooms_with_many_extremities(
+        room_ids = await self.store.get_rooms_with_many_extremities(
-            min_count=10,
+            min_count=self._dummy_events_threshold,
            limit=5,
            room_id_filter=self._rooms_to_exclude_from_dummy_event_insertion.keys(),
        )
@ -1080,9 +1096,9 @@ class EventCreationHandler(object):
            # For each room we need to find a joined member we can use to send
            # the dummy event with.
-            latest_event_ids = yield self.store.get_prev_events_for_room(room_id)
+            latest_event_ids = await self.store.get_prev_events_for_room(room_id)
-            members = yield self.state.get_current_users_in_room(
+            members = await self.state.get_current_users_in_room(
                room_id, latest_event_ids=latest_event_ids
            )
            dummy_event_sent = False
@ -1091,7 +1107,7 @@ class EventCreationHandler(object):
                    continue
                requester = create_requester(user_id)
                try:
-                    event, context = yield self.create_event(
+                    event, context = await self.create_event(
                        requester,
                        {
                            "type": "org.matrix.dummy_event",
@ -1104,7 +1120,7 @@ class EventCreationHandler(object):
                    event.internal_metadata.proactively_send = False
-                    yield self.send_nonmember_event(
+                    await self.send_nonmember_event(
                        requester, event, context, ratelimit=False
                    )
                    dummy_event_sent = True
--- a/synapse/handlers/password_policy.py
+++ b/synapse/handlers/password_policy.py
@ -0,0 +1,93 @@
 # -*- coding: utf-8 -*-
 # Copyright 2019 New Vector Ltd
 # Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
 import re
 from synapse.api.errors import Codes, PasswordRefusedError
 logger = logging.getLogger(__name__)
 class PasswordPolicyHandler(object):
    def __init__(self, hs):
        self.policy = hs.config.password_policy
        self.enabled = hs.config.password_policy_enabled
        # Regexps for the spec'd policy parameters.
        self.regexp_digit = re.compile("[0-9]")
        self.regexp_symbol = re.compile("[^a-zA-Z0-9]")
        self.regexp_uppercase = re.compile("[A-Z]")
        self.regexp_lowercase = re.compile("[a-z]")
    def validate_password(self, password):
        """Checks whether a given password complies with the server's policy.
        Args:
            password (str): The password to check against the server's policy.
        Raises:
            PasswordRefusedError: The password doesn't comply with the server's policy.
        """
        if not self.enabled:
            return
        minimum_accepted_length = self.policy.get("minimum_length", 0)
        if len(password) < minimum_accepted_length:
            raise PasswordRefusedError(
                msg=(
                    "The password must be at least %d characters long"
                    % minimum_accepted_length
                ),
                errcode=Codes.PASSWORD_TOO_SHORT,
            )
        if (
            self.policy.get("require_digit", False)
            and self.regexp_digit.search(password) is None
        ):
            raise PasswordRefusedError(
                msg="The password must include at least one digit",
                errcode=Codes.PASSWORD_NO_DIGIT,
            )
        if (
            self.policy.get("require_symbol", False)
            and self.regexp_symbol.search(password) is None
        ):
            raise PasswordRefusedError(
                msg="The password must include at least one symbol",
                errcode=Codes.PASSWORD_NO_SYMBOL,
            )
        if (
            self.policy.get("require_uppercase", False)
            and self.regexp_uppercase.search(password) is None
        ):
            raise PasswordRefusedError(
                msg="The password must include at least one uppercase letter",
                errcode=Codes.PASSWORD_NO_UPPERCASE,
            )
        if (
            self.policy.get("require_lowercase", False)
            and self.regexp_lowercase.search(password) is None
        ):
            raise PasswordRefusedError(
                msg="The password must include at least one lowercase letter",
                errcode=Codes.PASSWORD_NO_LOWERCASE,
            )
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@ -21,10 +22,10 @@ The methods that define policy are:
    - PresenceHandler._handle_timeouts
    - should_notify
 """
-
+import abc
 import logging
 from contextlib import contextmanager
-from typing import Dict, List, Set
+from typing import Dict, Iterable, List, Set
 from six import iteritems, itervalues
@ -41,7 +42,7 @@ from synapse.logging.utils import log_function
 from synapse.metrics import LaterGauge
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.storage.presence import UserPresenceState
-from synapse.types import UserID, get_domain_from_id
+from synapse.types import JsonDict, UserID, get_domain_from_id
 from synapse.util.async_helpers import Linearizer
 from synapse.util.caches.descriptors import cached
 from synapse.util.metrics import Measure
@ -99,13 +100,106 @@ EXTERNAL_PROCESS_EXPIRY = 5 * 60 * 1000
 assert LAST_ACTIVE_GRANULARITY < IDLE_TIMER
-class PresenceHandler(object):
+class BasePresenceHandler(abc.ABC):
    """Parts of the PresenceHandler that are shared between workers and master"""
    def __init__(self, hs: "synapse.server.HomeServer"):
        self.clock = hs.get_clock()
        self.store = hs.get_datastore()
        active_presence = self.store.take_presence_startup_info()
        self.user_to_current_state = {state.user_id: state for state in active_presence}
    @abc.abstractmethod
    async def user_syncing(
        self, user_id: str, affect_presence: bool
    ) -> ContextManager[None]:
        """Returns a context manager that should surround any stream requests
        from the user.
        This allows us to keep track of who is currently streaming and who isn't
        without having to have timers outside of this module to avoid flickering
        when users disconnect/reconnect.
        Args:
            user_id: the user that is starting a sync
            affect_presence: If false this function will be a no-op.
                Useful for streams that are not associated with an actual
                client that is being used by a user.
        """
    @abc.abstractmethod
    def get_currently_syncing_users_for_replication(self) -> Iterable[str]:
        """Get an iterable of syncing users on this worker, to send to the presence handler
        This is called when a replication connection is established. It should return
        a list of user ids, which are then sent as USER_SYNC commands to inform the
        process handling presence about those users.
        Returns:
            An iterable of user_id strings.
        """
    async def get_state(self, target_user: UserID) -> UserPresenceState:
        results = await self.get_states([target_user.to_string()])
        return results[0]
    async def get_states(
        self, target_user_ids: Iterable[str]
    ) -> List[UserPresenceState]:
        """Get the presence state for users."""
        updates_d = await self.current_state_for_users(target_user_ids)
        updates = list(updates_d.values())
        for user_id in set(target_user_ids) - {u.user_id for u in updates}:
            updates.append(UserPresenceState.default(user_id))
        return updates
    async def current_state_for_users(
        self, user_ids: Iterable[str]
    ) -> Dict[str, UserPresenceState]:
        """Get the current presence state for multiple users.
        Returns:
            dict: `user_id` -> `UserPresenceState`
        """
        states = {
            user_id: self.user_to_current_state.get(user_id, None)
            for user_id in user_ids
        }
        missing = [user_id for user_id, state in iteritems(states) if not state]
        if missing:
            # There are things not in our in memory cache. Lets pull them out of
            # the database.
            res = await self.store.get_presence_for_users(missing)
            states.update(res)
            missing = [user_id for user_id, state in iteritems(states) if not state]
            if missing:
                new = {
                    user_id: UserPresenceState.default(user_id) for user_id in missing
                }
                states.update(new)
                self.user_to_current_state.update(new)
        return states
    @abc.abstractmethod
    async def set_state(
        self, target_user: UserID, state: JsonDict, ignore_status_msg: bool = False
    ) -> None:
        """Set the presence state of the user. """
 class PresenceHandler(BasePresenceHandler):
    def __init__(self, hs: "synapse.server.HomeServer"):
        super().__init__(hs)
        self.hs = hs
        self.is_mine_id = hs.is_mine_id
        self.server_name = hs.hostname
        self.clock = hs.get_clock()
        self.store = hs.get_datastore()
        self.wheel_timer = WheelTimer()
        self.notifier = hs.get_notifier()
        self.federation = hs.get_federation_sender()
@ -115,13 +209,6 @@ class PresenceHandler(object):
        federation_registry.register_edu_handler("m.presence", self.incoming_presence)
        active_presence = self.store.take_presence_startup_info()
        # A dictionary of the current state of users. This is prefilled with
        # non-offline presence from the DB. We should fetch from the DB if
        # we can't find a users presence in here.
        self.user_to_current_state = {state.user_id: state for state in active_presence}
        LaterGauge(
            "synapse_handlers_presence_user_to_current_state_size",
            "",
@ -130,7 +217,7 @@ class PresenceHandler(object):
        )
        now = self.clock.time_msec()
-        for state in active_presence:
+        for state in self.user_to_current_state.values():
            self.wheel_timer.insert(
                now=now, obj=state.user_id, then=state.last_active_ts + IDLE_TIMER
            )
@ -361,10 +448,18 @@ class PresenceHandler(object):
        timers_fired_counter.inc(len(states))
        syncing_user_ids = {
            user_id
            for user_id, count in self.user_to_num_current_syncs.items()
            if count
        }
        for user_ids in self.external_process_to_current_syncs.values():
            syncing_user_ids.update(user_ids)
        changes = handle_timeouts(
            states,
            is_mine_fn=self.is_mine_id,
-            syncing_user_ids=self.get_currently_syncing_users(),
+            syncing_user_ids=syncing_user_ids,
            now=now,
        )
@ -462,22 +557,9 @@ class PresenceHandler(object):
        return _user_syncing()
-    def get_currently_syncing_users(self):
+    def get_currently_syncing_users_for_replication(self) -> Iterable[str]:
-        """Get the set of user ids that are currently syncing on this HS.
+        # since we are the process handling presence, there is nothing to do here.
-        Returns:
+        return []
            set(str): A set of user_id strings.
        """
        if self.hs.config.use_presence:
            syncing_user_ids = {
                user_id
                for user_id, count in self.user_to_num_current_syncs.items()
                if count
            }
            for user_ids in self.external_process_to_current_syncs.values():
                syncing_user_ids.update(user_ids)
            return syncing_user_ids
        else:
            return set()
    async def update_external_syncs_row(
        self, process_id, user_id, is_syncing, sync_time_msec
@ -554,34 +636,6 @@ class PresenceHandler(object):
        res = await self.current_state_for_users([user_id])
        return res[user_id]
    async def current_state_for_users(self, user_ids):
        """Get the current presence state for multiple users.
        Returns:
            dict: `user_id` -> `UserPresenceState`
        """
        states = {
            user_id: self.user_to_current_state.get(user_id, None)
            for user_id in user_ids
        }
        missing = [user_id for user_id, state in iteritems(states) if not state]
        if missing:
            # There are things not in our in memory cache. Lets pull them out of
            # the database.
            res = await self.store.get_presence_for_users(missing)
            states.update(res)
            missing = [user_id for user_id, state in iteritems(states) if not state]
            if missing:
                new = {
                    user_id: UserPresenceState.default(user_id) for user_id in missing
                }
                states.update(new)
                self.user_to_current_state.update(new)
        return states
    async def _persist_and_notify(self, states):
        """Persist states in the database, poke the notifier and send to
        interested remote servers
@ -669,40 +723,6 @@ class PresenceHandler(object):
            federation_presence_counter.inc(len(updates))
            await self._update_states(updates)
    async def get_state(self, target_user, as_event=False):
        results = await self.get_states([target_user.to_string()], as_event=as_event)
        return results[0]
    async def get_states(self, target_user_ids, as_event=False):
        """Get the presence state for users.
        Args:
            target_user_ids (list)
            as_event (bool): Whether to format it as a client event or not.
        Returns:
            list
        """
        updates = await self.current_state_for_users(target_user_ids)
        updates = list(updates.values())
        for user_id in set(target_user_ids) - {u.user_id for u in updates}:
            updates.append(UserPresenceState.default(user_id))
        now = self.clock.time_msec()
        if as_event:
            return [
                {
                    "type": "m.presence",
                    "content": format_user_presence_state(state, now),
                }
                for state in updates
            ]
        else:
            return updates
    async def set_state(self, target_user, state, ignore_status_msg=False):
        """Set the presence state of the user.
        """
@ -747,7 +767,7 @@ class PresenceHandler(object):
        return False
-    async def get_all_presence_updates(self, last_id, current_id):
+    async def get_all_presence_updates(self, last_id, current_id, limit):
        """
        Gets a list of presence update rows from between the given stream ids.
        Each row has:
@ -762,7 +782,7 @@ class PresenceHandler(object):
        """
        # TODO(markjh): replicate the unpersisted changes.
        # This could use the in-memory stores for recent changes.
-        rows = await self.store.get_all_presence_updates(last_id, current_id)
+        rows = await self.store.get_all_presence_updates(last_id, current_id, limit)
        return rows
    def notify_new_event(self):
@ -889,7 +909,7 @@ class PresenceHandler(object):
            user_ids = await self.state.get_current_users_in_room(room_id)
            user_ids = list(filter(self.is_mine_id, user_ids))
-            states = await self.current_state_for_users(user_ids)
+            states_d = await self.current_state_for_users(user_ids)
            # Filter out old presence, i.e. offline presence states where
            # the user hasn't been active for a week. We can change this
@ -899,7 +919,7 @@ class PresenceHandler(object):
            now = self.clock.time_msec()
            states = [
                state
-                for state in states.values()
+                for state in states_d.values()
                if state.state != PresenceState.OFFLINE
                or now - state.last_active_ts < 7 * 24 * 60 * 60 * 1000
                or state.status_msg is not None
--- a/synapse/handlers/profile.py
+++ b/synapse/handlers/profile.py
@ -141,8 +141,9 @@ class BaseProfileHandler(BaseHandler):
            return result["displayname"]
-    @defer.inlineCallbacks
+    async def set_displayname(
-    def set_displayname(self, target_user, requester, new_displayname, by_admin=False):
+        self, target_user, requester, new_displayname, by_admin=False
    ):
        """Set the displayname of a user
        Args:
@ -157,6 +158,15 @@ class BaseProfileHandler(BaseHandler):
        if not by_admin and target_user != requester.user:
            raise AuthError(400, "Cannot set another user's displayname")
        if not by_admin and not self.hs.config.enable_set_displayname:
            profile = await self.store.get_profileinfo(target_user.localpart)
            if profile.display_name:
                raise SynapseError(
                    400,
                    "Changing display name is disabled on this server",
                    Codes.FORBIDDEN,
                )
        if len(new_displayname) > MAX_DISPLAYNAME_LEN:
            raise SynapseError(
                400, "Displayname is too long (max %i)" % (MAX_DISPLAYNAME_LEN,)
@ -171,15 +181,15 @@ class BaseProfileHandler(BaseHandler):
        if by_admin:
            requester = create_requester(target_user)
-        yield self.store.set_profile_displayname(target_user.localpart, new_displayname)
+        await self.store.set_profile_displayname(target_user.localpart, new_displayname)
        if self.hs.config.user_directory_search_all_users:
-            profile = yield self.store.get_profileinfo(target_user.localpart)
+            profile = await self.store.get_profileinfo(target_user.localpart)
-            yield self.user_directory_handler.handle_local_profile_change(
+            await self.user_directory_handler.handle_local_profile_change(
                target_user.to_string(), profile
            )
-        yield self._update_join_states(requester, target_user)
+        await self._update_join_states(requester, target_user)
    @defer.inlineCallbacks
    def get_avatar_url(self, target_user):
@ -208,8 +218,9 @@ class BaseProfileHandler(BaseHandler):
            return result["avatar_url"]
-    @defer.inlineCallbacks
+    async def set_avatar_url(
-    def set_avatar_url(self, target_user, requester, new_avatar_url, by_admin=False):
+        self, target_user, requester, new_avatar_url, by_admin=False
    ):
        """target_user is the user whose avatar_url is to be changed;
        auth_user is the user attempting to make this change."""
        if not self.hs.is_mine(target_user):
@ -218,6 +229,13 @@ class BaseProfileHandler(BaseHandler):
        if not by_admin and target_user != requester.user:
            raise AuthError(400, "Cannot set another user's avatar_url")
        if not by_admin and not self.hs.config.enable_set_avatar_url:
            profile = await self.store.get_profileinfo(target_user.localpart)
            if profile.avatar_url:
                raise SynapseError(
                    400, "Changing avatar is disabled on this server", Codes.FORBIDDEN
                )
        if len(new_avatar_url) > MAX_AVATAR_URL_LEN:
            raise SynapseError(
                400, "Avatar URL is too long (max %i)" % (MAX_AVATAR_URL_LEN,)
@ -227,15 +245,15 @@ class BaseProfileHandler(BaseHandler):
        if by_admin:
            requester = create_requester(target_user)
-        yield self.store.set_profile_avatar_url(target_user.localpart, new_avatar_url)
+        await self.store.set_profile_avatar_url(target_user.localpart, new_avatar_url)
        if self.hs.config.user_directory_search_all_users:
-            profile = yield self.store.get_profileinfo(target_user.localpart)
+            profile = await self.store.get_profileinfo(target_user.localpart)
-            yield self.user_directory_handler.handle_local_profile_change(
+            await self.user_directory_handler.handle_local_profile_change(
                target_user.to_string(), profile
            )
-        yield self._update_join_states(requester, target_user)
+        await self._update_join_states(requester, target_user)
    @defer.inlineCallbacks
    def on_profile_query(self, args):
@ -263,21 +281,20 @@ class BaseProfileHandler(BaseHandler):
        return response
-    @defer.inlineCallbacks
+    async def _update_join_states(self, requester, target_user):
    def _update_join_states(self, requester, target_user):
        if not self.hs.is_mine(target_user):
            return
-        yield self.ratelimit(requester)
+        await self.ratelimit(requester)
-        room_ids = yield self.store.get_rooms_for_user(target_user.to_string())
+        room_ids = await self.store.get_rooms_for_user(target_user.to_string())
        for room_id in room_ids:
            handler = self.hs.get_room_member_handler()
            try:
                # Assume the target_user isn't a guest,
                # because we don't let guests set profile or avatar data.
-                yield handler.update_membership(
+                await handler.update_membership(
                    requester,
                    target_user,
                    room_id,
--- a/synapse/handlers/register.py
+++ b/synapse/handlers/register.py
@ -132,7 +132,7 @@ class RegistrationHandler(BaseHandler):
    def register_user(
        self,
        localpart=None,
-        password=None,
+        password_hash=None,
        guest_access_token=None,
        make_guest=False,
        admin=False,
@ -147,7 +147,7 @@ class RegistrationHandler(BaseHandler):
        Args:
            localpart: The local part of the user ID to register. If None,
              one will be generated.
-            password (unicode) : The password to assign to this user so they can
+            password_hash (str|None): The hashed password to assign to this user so they can
              login again. This can be None which means they cannot login again
              via a password (e.g. the user is an application service user).
            user_type (str|None): type of user. One of the values from
@ -164,9 +164,6 @@ class RegistrationHandler(BaseHandler):
        yield self.check_registration_ratelimit(address)
        yield self.auth.check_auth_blocking(threepid=threepid)
        password_hash = None
        if password:
            password_hash = yield self._auth_handler.hash(password)
        if localpart is not None:
            yield self.check_username(localpart, guest_access_token=guest_access_token)
@ -242,7 +239,7 @@ class RegistrationHandler(BaseHandler):
                    fail_count += 1
        if not self.hs.config.user_consent_at_registration:
-            yield self._auto_join_rooms(user_id)
+            yield defer.ensureDeferred(self._auto_join_rooms(user_id))
        else:
            logger.info(
                "Skipping auto-join for %s because consent is required at registration",
@ -264,8 +261,7 @@ class RegistrationHandler(BaseHandler):
        return user_id
-    @defer.inlineCallbacks
+    async def _auto_join_rooms(self, user_id):
    def _auto_join_rooms(self, user_id):
        """Automatically joins users to auto join rooms - creating the room in the first place
        if the user is the first to be created.
@ -279,9 +275,9 @@ class RegistrationHandler(BaseHandler):
        # that an auto-generated support or bot user is not a real user and will never be
        # the user to create the room
        should_auto_create_rooms = False
-        is_real_user = yield self.store.is_real_user(user_id)
+        is_real_user = await self.store.is_real_user(user_id)
        if self.hs.config.autocreate_auto_join_rooms and is_real_user:
-            count = yield self.store.count_real_users()
+            count = await self.store.count_real_users()
            should_auto_create_rooms = count == 1
        for r in self.hs.config.auto_join_rooms:
            logger.info("Auto-joining %s to %s", user_id, r)
@ -300,7 +296,7 @@ class RegistrationHandler(BaseHandler):
                        # getting the RoomCreationHandler during init gives a dependency
                        # loop
-                        yield self.hs.get_room_creation_handler().create_room(
+                        await self.hs.get_room_creation_handler().create_room(
                            fake_requester,
                            config={
                                "preset": "public_chat",
@ -309,7 +305,7 @@ class RegistrationHandler(BaseHandler):
                            ratelimit=False,
                        )
                else:
-                    yield self._join_user_to_room(fake_requester, r)
+                    await self._join_user_to_room(fake_requester, r)
            except ConsentNotGivenError as e:
                # Technically not necessary to pull out this error though
                # moving away from bare excepts is a good thing to do.
@ -317,15 +313,14 @@ class RegistrationHandler(BaseHandler):
            except Exception as e:
                logger.error("Failed to join new user to %r: %r", r, e)
-    @defer.inlineCallbacks
+    async def post_consent_actions(self, user_id):
    def post_consent_actions(self, user_id):
        """A series of registration actions that can only be carried out once consent
        has been granted
        Args:
            user_id (str): The user to join
        """
-        yield self._auto_join_rooms(user_id)
+        await self._auto_join_rooms(user_id)
    @defer.inlineCallbacks
    def appservice_register(self, user_localpart, as_token):
@ -392,14 +387,13 @@ class RegistrationHandler(BaseHandler):
        self._next_generated_user_id += 1
        return str(id)
-    @defer.inlineCallbacks
+    async def _join_user_to_room(self, requester, room_identifier):
    def _join_user_to_room(self, requester, room_identifier):
        room_member_handler = self.hs.get_room_member_handler()
        if RoomID.is_valid(room_identifier):
            room_id = room_identifier
        elif RoomAlias.is_valid(room_identifier):
            room_alias = RoomAlias.from_string(room_identifier)
-            room_id, remote_room_hosts = yield room_member_handler.lookup_room_alias(
+            room_id, remote_room_hosts = await room_member_handler.lookup_room_alias(
                room_alias
            )
            room_id = room_id.to_string()
@ -408,7 +402,7 @@ class RegistrationHandler(BaseHandler):
                400, "%s was not legal room ID or room alias" % (room_identifier,)
            )
-        yield room_member_handler.update_membership(
+        await room_member_handler.update_membership(
            requester=requester,
            target=requester.user,
            room_id=room_id,
@ -540,14 +534,15 @@ class RegistrationHandler(BaseHandler):
                user_id, ["guest = true"]
            )
        else:
-            access_token = yield self._auth_handler.get_access_token_for_user_id(
+            access_token = yield defer.ensureDeferred(
                self._auth_handler.get_access_token_for_user_id(
                    user_id, device_id=device_id, valid_until_ms=valid_until_ms
                )
            )
        return (device_id, access_token)
-    @defer.inlineCallbacks
+    async def post_registration_actions(self, user_id, auth_result, access_token):
    def post_registration_actions(self, user_id, auth_result, access_token):
        """A user has completed registration
        Args:
@ -558,7 +553,7 @@ class RegistrationHandler(BaseHandler):
                device, or None if `inhibit_login` enabled.
        """
        if self.hs.config.worker_app:
-            yield self._post_registration_client(
+            await self._post_registration_client(
                user_id=user_id, auth_result=auth_result, access_token=access_token
            )
            return
@ -570,19 +565,18 @@ class RegistrationHandler(BaseHandler):
            if is_threepid_reserved(
                self.hs.config.mau_limits_reserved_threepids, threepid
            ):
-                yield self.store.upsert_monthly_active_user(user_id)
+                await self.store.upsert_monthly_active_user(user_id)
-            yield self._register_email_threepid(user_id, threepid, access_token)
+            await self._register_email_threepid(user_id, threepid, access_token)
        if auth_result and LoginType.MSISDN in auth_result:
            threepid = auth_result[LoginType.MSISDN]
-            yield self._register_msisdn_threepid(user_id, threepid)
+            await self._register_msisdn_threepid(user_id, threepid)
        if auth_result and LoginType.TERMS in auth_result:
-            yield self._on_user_consented(user_id, self.hs.config.user_consent_version)
+            await self._on_user_consented(user_id, self.hs.config.user_consent_version)
-    @defer.inlineCallbacks
+    async def _on_user_consented(self, user_id, consent_version):
    def _on_user_consented(self, user_id, consent_version):
        """A user consented to the terms on registration
        Args:
@ -591,8 +585,8 @@ class RegistrationHandler(BaseHandler):
                consented to.
        """
        logger.info("%s has consented to the privacy policy", user_id)
-        yield self.store.user_set_consent_version(user_id, consent_version)
+        await self.store.user_set_consent_version(user_id, consent_version)
-        yield self.post_consent_actions(user_id)
+        await self.post_consent_actions(user_id)
    @defer.inlineCallbacks
    def _register_email_threepid(self, user_id, threepid, token):
@ -617,8 +611,13 @@ class RegistrationHandler(BaseHandler):
            logger.info("Can't add incomplete 3pid")
            return
-        yield self._auth_handler.add_threepid(
+        yield defer.ensureDeferred(
-            user_id, threepid["medium"], threepid["address"], threepid["validated_at"]
+            self._auth_handler.add_threepid(
                user_id,
                threepid["medium"],
                threepid["address"],
                threepid["validated_at"],
            )
        )
        # And we add an email pusher for them by default, but only
@ -670,6 +669,11 @@ class RegistrationHandler(BaseHandler):
                return None
            raise
-        yield self._auth_handler.add_threepid(
+        yield defer.ensureDeferred(
-            user_id, threepid["medium"], threepid["address"], threepid["validated_at"]
+            self._auth_handler.add_threepid(
                user_id,
                threepid["medium"],
                threepid["address"],
                threepid["validated_at"],
            )
        )
--- a/synapse/handlers/room.py
+++ b/synapse/handlers/room.py
@ -148,17 +148,16 @@ class RoomCreationHandler(BaseHandler):
        return ret
-    @defer.inlineCallbacks
+    async def _upgrade_room(
    def _upgrade_room(
        self, requester: Requester, old_room_id: str, new_version: RoomVersion
    ):
        user_id = requester.user.to_string()
        # start by allocating a new room id
-        r = yield self.store.get_room(old_room_id)
+        r = await self.store.get_room(old_room_id)
        if r is None:
            raise NotFoundError("Unknown room id %s" % (old_room_id,))
-        new_room_id = yield self._generate_room_id(
+        new_room_id = await self._generate_room_id(
            creator_id=user_id, is_public=r["is_public"], room_version=new_version,
        )
@ -169,7 +168,7 @@ class RoomCreationHandler(BaseHandler):
        (
            tombstone_event,
            tombstone_context,
-        ) = yield self.event_creation_handler.create_event(
+        ) = await self.event_creation_handler.create_event(
            requester,
            {
                "type": EventTypes.Tombstone,
@ -183,12 +182,12 @@ class RoomCreationHandler(BaseHandler):
            },
            token_id=requester.access_token_id,
        )
-        old_room_version = yield self.store.get_room_version_id(old_room_id)
+        old_room_version = await self.store.get_room_version_id(old_room_id)
-        yield self.auth.check_from_context(
+        await self.auth.check_from_context(
            old_room_version, tombstone_event, tombstone_context
        )
-        yield self.clone_existing_room(
+        await self.clone_existing_room(
            requester,
            old_room_id=old_room_id,
            new_room_id=new_room_id,
@ -197,32 +196,31 @@ class RoomCreationHandler(BaseHandler):
        )
        # now send the tombstone
-        yield self.event_creation_handler.send_nonmember_event(
+        await self.event_creation_handler.send_nonmember_event(
            requester, tombstone_event, tombstone_context
        )
-        old_room_state = yield tombstone_context.get_current_state_ids()
+        old_room_state = await tombstone_context.get_current_state_ids()
        # update any aliases
-        yield self._move_aliases_to_new_room(
+        await self._move_aliases_to_new_room(
            requester, old_room_id, new_room_id, old_room_state
        )
        # Copy over user push rules, tags and migrate room directory state
-        yield self.room_member_handler.transfer_room_state_on_room_upgrade(
+        await self.room_member_handler.transfer_room_state_on_room_upgrade(
            old_room_id, new_room_id
        )
        # finally, shut down the PLs in the old room, and update them in the new
        # room.
-        yield self._update_upgraded_room_pls(
+        await self._update_upgraded_room_pls(
            requester, old_room_id, new_room_id, old_room_state,
        )
        return new_room_id
-    @defer.inlineCallbacks
+    async def _update_upgraded_room_pls(
    def _update_upgraded_room_pls(
        self,
        requester: Requester,
        old_room_id: str,
@ -249,7 +247,7 @@ class RoomCreationHandler(BaseHandler):
            )
            return
-        old_room_pl_state = yield self.store.get_event(old_room_pl_event_id)
+        old_room_pl_state = await self.store.get_event(old_room_pl_event_id)
        # we try to stop regular users from speaking by setting the PL required
        # to send regular events and invites to 'Moderator' level. That's normally
@ -278,7 +276,7 @@ class RoomCreationHandler(BaseHandler):
        if updated:
            try:
-                yield self.event_creation_handler.create_and_send_nonmember_event(
+                await self.event_creation_handler.create_and_send_nonmember_event(
                    requester,
                    {
                        "type": EventTypes.PowerLevels,
@ -292,7 +290,7 @@ class RoomCreationHandler(BaseHandler):
            except AuthError as e:
                logger.warning("Unable to update PLs in old room: %s", e)
-        yield self.event_creation_handler.create_and_send_nonmember_event(
+        await self.event_creation_handler.create_and_send_nonmember_event(
            requester,
            {
                "type": EventTypes.PowerLevels,
@ -304,8 +302,7 @@ class RoomCreationHandler(BaseHandler):
            ratelimit=False,
        )
-    @defer.inlineCallbacks
+    async def clone_existing_room(
    def clone_existing_room(
        self,
        requester: Requester,
        old_room_id: str,
@ -338,7 +335,7 @@ class RoomCreationHandler(BaseHandler):
        # Check if old room was non-federatable
        # Get old room's create event
-        old_room_create_event = yield self.store.get_create_event_for_room(old_room_id)
+        old_room_create_event = await self.store.get_create_event_for_room(old_room_id)
        # Check if the create event specified a non-federatable room
        if not old_room_create_event.content.get("m.federate", True):
@ -361,11 +358,11 @@ class RoomCreationHandler(BaseHandler):
            (EventTypes.PowerLevels, ""),
        )
-        old_room_state_ids = yield self.store.get_filtered_current_state_ids(
+        old_room_state_ids = await self.store.get_filtered_current_state_ids(
            old_room_id, StateFilter.from_types(types_to_copy)
        )
        # map from event_id to BaseEvent
-        old_room_state_events = yield self.store.get_events(old_room_state_ids.values())
+        old_room_state_events = await self.store.get_events(old_room_state_ids.values())
        for k, old_event_id in iteritems(old_room_state_ids):
            old_event = old_room_state_events.get(old_event_id)
@ -400,7 +397,7 @@ class RoomCreationHandler(BaseHandler):
        if current_power_level < needed_power_level:
            power_levels["users"][user_id] = needed_power_level
-        yield self._send_events_for_new_room(
+        await self._send_events_for_new_room(
            requester,
            new_room_id,
            # we expect to override all the presets with initial_state, so this is
@ -412,12 +409,12 @@ class RoomCreationHandler(BaseHandler):
        )
        # Transfer membership events
-        old_room_member_state_ids = yield self.store.get_filtered_current_state_ids(
+        old_room_member_state_ids = await self.store.get_filtered_current_state_ids(
            old_room_id, StateFilter.from_types([(EventTypes.Member, None)])
        )
        # map from event_id to BaseEvent
-        old_room_member_state_events = yield self.store.get_events(
+        old_room_member_state_events = await self.store.get_events(
            old_room_member_state_ids.values()
        )
        for k, old_event in iteritems(old_room_member_state_events):
@ -426,7 +423,7 @@ class RoomCreationHandler(BaseHandler):
                "membership" in old_event.content
                and old_event.content["membership"] == "ban"
            ):
-                yield self.room_member_handler.update_membership(
+                await self.room_member_handler.update_membership(
                    requester,
                    UserID.from_string(old_event["state_key"]),
                    new_room_id,
@ -438,8 +435,7 @@ class RoomCreationHandler(BaseHandler):
        # XXX invites/joins
        # XXX 3pid invites
-    @defer.inlineCallbacks
+    async def _move_aliases_to_new_room(
    def _move_aliases_to_new_room(
        self,
        requester: Requester,
        old_room_id: str,
@ -448,13 +444,13 @@ class RoomCreationHandler(BaseHandler):
    ):
        directory_handler = self.hs.get_handlers().directory_handler
-        aliases = yield self.store.get_aliases_for_room(old_room_id)
+        aliases = await self.store.get_aliases_for_room(old_room_id)
        # check to see if we have a canonical alias.
        canonical_alias_event = None
        canonical_alias_event_id = old_room_state.get((EventTypes.CanonicalAlias, ""))
        if canonical_alias_event_id:
-            canonical_alias_event = yield self.store.get_event(canonical_alias_event_id)
+            canonical_alias_event = await self.store.get_event(canonical_alias_event_id)
        # first we try to remove the aliases from the old room (we suppress sending
        # the room_aliases event until the end).
@ -472,7 +468,7 @@ class RoomCreationHandler(BaseHandler):
        for alias_str in aliases:
            alias = RoomAlias.from_string(alias_str)
            try:
-                yield directory_handler.delete_association(requester, alias)
+                await directory_handler.delete_association(requester, alias)
                removed_aliases.append(alias_str)
            except SynapseError as e:
                logger.warning("Unable to remove alias %s from old room: %s", alias, e)
@ -485,7 +481,7 @@ class RoomCreationHandler(BaseHandler):
        # we can now add any aliases we successfully removed to the new room.
        for alias in removed_aliases:
            try:
-                yield directory_handler.create_association(
+                await directory_handler.create_association(
                    requester,
                    RoomAlias.from_string(alias),
                    new_room_id,
@ -502,7 +498,7 @@ class RoomCreationHandler(BaseHandler):
        # alias event for the new room with a copy of the information.
        try:
            if canonical_alias_event:
-                yield self.event_creation_handler.create_and_send_nonmember_event(
+                await self.event_creation_handler.create_and_send_nonmember_event(
                    requester,
                    {
                        "type": EventTypes.CanonicalAlias,
@ -518,8 +514,9 @@ class RoomCreationHandler(BaseHandler):
            # we returned the new room to the client at this point.
            logger.error("Unable to send updated alias events in new room: %s", e)
-    @defer.inlineCallbacks
+    async def create_room(
-    def create_room(self, requester, config, ratelimit=True, creator_join_profile=None):
+        self, requester, config, ratelimit=True, creator_join_profile=None
    ):
        """ Creates a new room.
        Args:
@ -547,7 +544,7 @@ class RoomCreationHandler(BaseHandler):
        """
        user_id = requester.user.to_string()
-        yield self.auth.check_auth_blocking(user_id)
+        await self.auth.check_auth_blocking(user_id)
        if (
            self._server_notices_mxid is not None
@ -556,11 +553,11 @@ class RoomCreationHandler(BaseHandler):
            # allow the server notices mxid to create rooms
            is_requester_admin = True
        else:
-            is_requester_admin = yield self.auth.is_server_admin(requester.user)
+            is_requester_admin = await self.auth.is_server_admin(requester.user)
        # Check whether the third party rules allows/changes the room create
        # request.
-        event_allowed = yield self.third_party_event_rules.on_create_room(
+        event_allowed = await self.third_party_event_rules.on_create_room(
            requester, config, is_requester_admin=is_requester_admin
        )
        if not event_allowed:
@ -574,7 +571,7 @@ class RoomCreationHandler(BaseHandler):
            raise SynapseError(403, "You are not permitted to create rooms")
        if ratelimit:
-            yield self.ratelimit(requester)
+            await self.ratelimit(requester)
        room_version_id = config.get(
            "room_version", self.config.default_room_version.identifier
@ -597,7 +594,7 @@ class RoomCreationHandler(BaseHandler):
                    raise SynapseError(400, "Invalid characters in room alias")
            room_alias = RoomAlias(config["room_alias_name"], self.hs.hostname)
-            mapping = yield self.store.get_association_from_room_alias(room_alias)
+            mapping = await self.store.get_association_from_room_alias(room_alias)
            if mapping:
                raise SynapseError(400, "Room alias already taken", Codes.ROOM_IN_USE)
@ -612,7 +609,7 @@ class RoomCreationHandler(BaseHandler):
            except Exception:
                raise SynapseError(400, "Invalid user_id: %s" % (i,))
-        yield self.event_creation_handler.assert_accepted_privacy_policy(requester)
+        await self.event_creation_handler.assert_accepted_privacy_policy(requester)
        power_level_content_override = config.get("power_level_content_override")
        if (
@ -631,13 +628,13 @@ class RoomCreationHandler(BaseHandler):
        visibility = config.get("visibility", None)
        is_public = visibility == "public"
-        room_id = yield self._generate_room_id(
+        room_id = await self._generate_room_id(
            creator_id=user_id, is_public=is_public, room_version=room_version,
        )
        directory_handler = self.hs.get_handlers().directory_handler
        if room_alias:
-            yield directory_handler.create_association(
+            await directory_handler.create_association(
                requester=requester,
                room_id=room_id,
                room_alias=room_alias,
@ -645,6 +642,13 @@ class RoomCreationHandler(BaseHandler):
                check_membership=False,
            )
        if is_public:
            if not self.config.is_publishing_room_allowed(user_id, room_id, room_alias):
                # Lets just return a generic message, as there may be all sorts of
                # reasons why we said no. TODO: Allow configurable error messages
                # per alias creation rule?
                raise SynapseError(403, "Not allowed to publish room")
        preset_config = config.get(
            "preset",
            RoomCreationPreset.PRIVATE_CHAT
@ -663,7 +667,7 @@ class RoomCreationHandler(BaseHandler):
        # override any attempt to set room versions via the creation_content
        creation_content["room_version"] = room_version.identifier
-        yield self._send_events_for_new_room(
+        await self._send_events_for_new_room(
            requester,
            room_id,
            preset_config=preset_config,
@ -677,7 +681,7 @@ class RoomCreationHandler(BaseHandler):
        if "name" in config:
            name = config["name"]
-            yield self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                requester,
                {
                    "type": EventTypes.Name,
@ -691,7 +695,7 @@ class RoomCreationHandler(BaseHandler):
        if "topic" in config:
            topic = config["topic"]
-            yield self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                requester,
                {
                    "type": EventTypes.Topic,
@ -709,7 +713,7 @@ class RoomCreationHandler(BaseHandler):
            if is_direct:
                content["is_direct"] = is_direct
-            yield self.room_member_handler.update_membership(
+            await self.room_member_handler.update_membership(
                requester,
                UserID.from_string(invitee),
                room_id,
@ -723,7 +727,7 @@ class RoomCreationHandler(BaseHandler):
            id_access_token = invite_3pid.get("id_access_token")  # optional
            address = invite_3pid["address"]
            medium = invite_3pid["medium"]
-            yield self.hs.get_room_member_handler().do_3pid_invite(
+            await self.hs.get_room_member_handler().do_3pid_invite(
                room_id,
                requester.user,
                medium,
@ -741,8 +745,7 @@ class RoomCreationHandler(BaseHandler):
        return result
-    @defer.inlineCallbacks
+    async def _send_events_for_new_room(
    def _send_events_for_new_room(
        self,
        creator,  # A Requester object.
        room_id,
@ -762,11 +765,10 @@ class RoomCreationHandler(BaseHandler):
            return e
-        @defer.inlineCallbacks
+        async def send(etype, content, **kwargs):
        def send(etype, content, **kwargs):
            event = create(etype, content, **kwargs)
            logger.debug("Sending %s in new room", etype)
-            yield self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                creator, event, ratelimit=False
            )
@ -777,10 +779,10 @@ class RoomCreationHandler(BaseHandler):
        event_keys = {"room_id": room_id, "sender": creator_id, "state_key": ""}
        creation_content.update({"creator": creator_id})
-        yield send(etype=EventTypes.Create, content=creation_content)
+        await send(etype=EventTypes.Create, content=creation_content)
        logger.debug("Sending %s in new room", EventTypes.Member)
-        yield self.room_member_handler.update_membership(
+        await self.room_member_handler.update_membership(
            creator,
            creator.user,
            room_id,
@ -793,7 +795,7 @@ class RoomCreationHandler(BaseHandler):
        # of the first events that get sent into a room.
        pl_content = initial_state.pop((EventTypes.PowerLevels, ""), None)
        if pl_content is not None:
-            yield send(etype=EventTypes.PowerLevels, content=pl_content)
+            await send(etype=EventTypes.PowerLevels, content=pl_content)
        else:
            power_level_content = {
                "users": {creator_id: 100},
@ -806,6 +808,7 @@ class RoomCreationHandler(BaseHandler):
                    EventTypes.RoomAvatar: 50,
                    EventTypes.Tombstone: 100,
                    EventTypes.ServerACL: 100,
                    EventTypes.RoomEncryption: 100,
                },
                "events_default": 0,
                "state_default": 50,
@ -825,33 +828,33 @@ class RoomCreationHandler(BaseHandler):
            if power_level_content_override:
                power_level_content.update(power_level_content_override)
-            yield send(etype=EventTypes.PowerLevels, content=power_level_content)
+            await send(etype=EventTypes.PowerLevels, content=power_level_content)
        if room_alias and (EventTypes.CanonicalAlias, "") not in initial_state:
-            yield send(
+            await send(
                etype=EventTypes.CanonicalAlias,
                content={"alias": room_alias.to_string()},
            )
        if (EventTypes.JoinRules, "") not in initial_state:
-            yield send(
+            await send(
                etype=EventTypes.JoinRules, content={"join_rule": config["join_rules"]}
            )
        if (EventTypes.RoomHistoryVisibility, "") not in initial_state:
-            yield send(
+            await send(
                etype=EventTypes.RoomHistoryVisibility,
                content={"history_visibility": config["history_visibility"]},
            )
        if config["guest_can_join"]:
            if (EventTypes.GuestAccess, "") not in initial_state:
-                yield send(
+                await send(
                    etype=EventTypes.GuestAccess, content={"guest_access": "can_join"}
                )
        for (etype, state_key), content in initial_state.items():
-            yield send(etype=etype, state_key=state_key, content=content)
+            await send(etype=etype, state_key=state_key, content=content)
    @defer.inlineCallbacks
    def _generate_room_id(
--- a/synapse/handlers/room_list.py
+++ b/synapse/handlers/room_list.py
@ -15,6 +15,7 @@
 import logging
 from collections import namedtuple
 from typing import Any, Dict, Optional
 from six import iteritems
@ -89,7 +90,11 @@ class RoomListHandler(BaseHandler):
            logger.info("Bypassing cache as search request.")
            return self._get_public_room_list(
-                limit, since_token, search_filter, network_tuple=network_tuple
+                limit,
                since_token,
                search_filter,
                network_tuple=network_tuple,
                from_federation=from_federation,
            )
        key = (limit, since_token, network_tuple)
@ -105,22 +110,22 @@ class RoomListHandler(BaseHandler):
    @defer.inlineCallbacks
    def _get_public_room_list(
        self,
-        limit=None,
+        limit: Optional[int] = None,
-        since_token=None,
+        since_token: Optional[str] = None,
-        search_filter=None,
+        search_filter: Optional[Dict] = None,
-        network_tuple=EMPTY_THIRD_PARTY_ID,
+        network_tuple: ThirdPartyInstanceID = EMPTY_THIRD_PARTY_ID,
-        from_federation=False,
+        from_federation: bool = False,
-    ):
+    ) -> Dict[str, Any]:
        """Generate a public room list.
        Args:
-            limit (int|None): Maximum amount of rooms to return.
+            limit: Maximum amount of rooms to return.
-            since_token (str|None)
+            since_token:
-            search_filter (dict|None): Dictionary to filter rooms by.
+            search_filter: Dictionary to filter rooms by.
-            network_tuple (ThirdPartyInstanceID): Which public list to use.
+            network_tuple: Which public list to use.
                This can be (None, None) to indicate the main list, or a particular
                appservice and network id to use an appservice specific one.
                Setting to None returns all public rooms across all lists.
-            from_federation (bool): Whether this request originated from a
+            from_federation: Whether this request originated from a
                federating server or a client. Used for room filtering.
        """
--- a/synapse/handlers/room_member.py
+++ b/synapse/handlers/room_member.py
@ -142,8 +142,7 @@ class RoomMemberHandler(object):
        """
        raise NotImplementedError()
-    @defer.inlineCallbacks
+    async def _local_membership_update(
    def _local_membership_update(
        self,
        requester,
        target,
@ -164,7 +163,7 @@ class RoomMemberHandler(object):
        if requester.is_guest:
            content["kind"] = "guest"
-        event, context = yield self.event_creation_handler.create_event(
+        event, context = await self.event_creation_handler.create_event(
            requester,
            {
                "type": EventTypes.Member,
@ -182,18 +181,18 @@ class RoomMemberHandler(object):
        )
        # Check if this event matches the previous membership event for the user.
-        duplicate = yield self.event_creation_handler.deduplicate_state_event(
+        duplicate = await self.event_creation_handler.deduplicate_state_event(
            event, context
        )
        if duplicate is not None:
            # Discard the new event since this membership change is a no-op.
            return duplicate
-        yield self.event_creation_handler.handle_new_client_event(
+        await self.event_creation_handler.handle_new_client_event(
            requester, event, context, extra_users=[target], ratelimit=ratelimit
        )
-        prev_state_ids = yield context.get_prev_state_ids()
+        prev_state_ids = await context.get_prev_state_ids()
        prev_member_event_id = prev_state_ids.get((EventTypes.Member, user_id), None)
@ -203,15 +202,15 @@ class RoomMemberHandler(object):
            # info.
            newly_joined = True
            if prev_member_event_id:
-                prev_member_event = yield self.store.get_event(prev_member_event_id)
+                prev_member_event = await self.store.get_event(prev_member_event_id)
                newly_joined = prev_member_event.membership != Membership.JOIN
            if newly_joined:
-                yield self._user_joined_room(target, room_id)
+                await self._user_joined_room(target, room_id)
        elif event.membership == Membership.LEAVE:
            if prev_member_event_id:
-                prev_member_event = yield self.store.get_event(prev_member_event_id)
+                prev_member_event = await self.store.get_event(prev_member_event_id)
                if prev_member_event.membership == Membership.JOIN:
-                    yield self._user_left_room(target, room_id)
+                    await self._user_left_room(target, room_id)
        return event
@ -253,8 +252,7 @@ class RoomMemberHandler(object):
        for tag, tag_content in room_tags.items():
            yield self.store.add_tag_to_room(user_id, new_room_id, tag, tag_content)
-    @defer.inlineCallbacks
+    async def update_membership(
    def update_membership(
        self,
        requester,
        target,
@ -269,8 +267,8 @@ class RoomMemberHandler(object):
    ):
        key = (room_id,)
-        with (yield self.member_linearizer.queue(key)):
+        with (await self.member_linearizer.queue(key)):
-            result = yield self._update_membership(
+            result = await self._update_membership(
                requester,
                target,
                room_id,
@ -285,8 +283,7 @@ class RoomMemberHandler(object):
        return result
-    @defer.inlineCallbacks
+    async def _update_membership(
    def _update_membership(
        self,
        requester,
        target,
@ -321,7 +318,7 @@ class RoomMemberHandler(object):
        # if this is a join with a 3pid signature, we may need to turn a 3pid
        # invite into a normal invite before we can handle the join.
        if third_party_signed is not None:
-            yield self.federation_handler.exchange_third_party_invite(
+            await self.federation_handler.exchange_third_party_invite(
                third_party_signed["sender"],
                target.to_string(),
                room_id,
@ -332,7 +329,7 @@ class RoomMemberHandler(object):
            remote_room_hosts = []
        if effective_membership_state not in ("leave", "ban"):
-            is_blocked = yield self.store.is_room_blocked(room_id)
+            is_blocked = await self.store.is_room_blocked(room_id)
            if is_blocked:
                raise SynapseError(403, "This room has been blocked on this server")
@ -351,7 +348,7 @@ class RoomMemberHandler(object):
                is_requester_admin = True
            else:
-                is_requester_admin = yield self.auth.is_server_admin(requester.user)
+                is_requester_admin = await self.auth.is_server_admin(requester.user)
            if not is_requester_admin:
                if self.config.block_non_admin_invites:
@ -370,9 +367,9 @@ class RoomMemberHandler(object):
            if block_invite:
                raise SynapseError(403, "Invites have been disabled on this server")
-        latest_event_ids = yield self.store.get_prev_events_for_room(room_id)
+        latest_event_ids = await self.store.get_prev_events_for_room(room_id)
-        current_state_ids = yield self.state_handler.get_current_state_ids(
+        current_state_ids = await self.state_handler.get_current_state_ids(
            room_id, latest_event_ids=latest_event_ids
        )
@ -381,7 +378,7 @@ class RoomMemberHandler(object):
        # transitions and generic otherwise
        old_state_id = current_state_ids.get((EventTypes.Member, target.to_string()))
        if old_state_id:
-            old_state = yield self.store.get_event(old_state_id, allow_none=True)
+            old_state = await self.store.get_event(old_state_id, allow_none=True)
            old_membership = old_state.content.get("membership") if old_state else None
            if action == "unban" and old_membership != "ban":
                raise SynapseError(
@ -413,7 +410,7 @@ class RoomMemberHandler(object):
                old_membership == Membership.INVITE
                and effective_membership_state == Membership.LEAVE
            ):
-                is_blocked = yield self._is_server_notice_room(room_id)
+                is_blocked = await self._is_server_notice_room(room_id)
                if is_blocked:
                    raise SynapseError(
                        http_client.FORBIDDEN,
@ -424,18 +421,18 @@ class RoomMemberHandler(object):
            if action == "kick":
                raise AuthError(403, "The target user is not in the room")
-        is_host_in_room = yield self._is_host_in_room(current_state_ids)
+        is_host_in_room = await self._is_host_in_room(current_state_ids)
        if effective_membership_state == Membership.JOIN:
            if requester.is_guest:
-                guest_can_join = yield self._can_guest_join(current_state_ids)
+                guest_can_join = await self._can_guest_join(current_state_ids)
                if not guest_can_join:
                    # This should be an auth check, but guests are a local concept,
                    # so don't really fit into the general auth process.
                    raise AuthError(403, "Guest access not allowed")
            if not is_host_in_room:
-                inviter = yield self._get_inviter(target.to_string(), room_id)
+                inviter = await self._get_inviter(target.to_string(), room_id)
                if inviter and not self.hs.is_mine(inviter):
                    remote_room_hosts.append(inviter.domain)
@ -443,13 +440,13 @@ class RoomMemberHandler(object):
                profile = self.profile_handler
                if not content_specified:
-                    content["displayname"] = yield profile.get_displayname(target)
+                    content["displayname"] = await profile.get_displayname(target)
-                    content["avatar_url"] = yield profile.get_avatar_url(target)
+                    content["avatar_url"] = await profile.get_avatar_url(target)
                if requester.is_guest:
                    content["kind"] = "guest"
-                remote_join_response = yield self._remote_join(
+                remote_join_response = await self._remote_join(
                    requester, remote_room_hosts, room_id, target, content
                )
@ -458,7 +455,7 @@ class RoomMemberHandler(object):
        elif effective_membership_state == Membership.LEAVE:
            if not is_host_in_room:
                # perhaps we've been invited
-                inviter = yield self._get_inviter(target.to_string(), room_id)
+                inviter = await self._get_inviter(target.to_string(), room_id)
                if not inviter:
                    raise SynapseError(404, "Not a known room")
@ -472,12 +469,12 @@ class RoomMemberHandler(object):
                else:
                    # send the rejection to the inviter's HS.
                    remote_room_hosts = remote_room_hosts + [inviter.domain]
-                    res = yield self._remote_reject_invite(
+                    res = await self._remote_reject_invite(
                        requester, remote_room_hosts, room_id, target, content,
                    )
                    return res
-        res = yield self._local_membership_update(
+        res = await self._local_membership_update(
            requester=requester,
            target=target,
            room_id=room_id,
@ -519,6 +516,9 @@ class RoomMemberHandler(object):
            yield self.store.set_room_is_public(old_room_id, False)
            yield self.store.set_room_is_public(room_id, True)
        # Transfer alias mappings in the room directory
        yield self.store.update_aliases_for_room(old_room_id, room_id)
        # Check if any groups we own contain the predecessor room
        local_group_ids = yield self.store.get_local_groups_for_room(old_room_id)
        for group_id in local_group_ids:
@ -569,8 +569,7 @@ class RoomMemberHandler(object):
                )
                continue
-    @defer.inlineCallbacks
+    async def send_membership_event(self, requester, event, context, ratelimit=True):
    def send_membership_event(self, requester, event, context, ratelimit=True):
        """
        Change the membership status of a user in a room.
@ -596,27 +595,27 @@ class RoomMemberHandler(object):
        else:
            requester = types.create_requester(target_user)
-        prev_event = yield self.event_creation_handler.deduplicate_state_event(
+        prev_event = await self.event_creation_handler.deduplicate_state_event(
            event, context
        )
        if prev_event is not None:
            return
-        prev_state_ids = yield context.get_prev_state_ids()
+        prev_state_ids = await context.get_prev_state_ids()
        if event.membership == Membership.JOIN:
            if requester.is_guest:
-                guest_can_join = yield self._can_guest_join(prev_state_ids)
+                guest_can_join = await self._can_guest_join(prev_state_ids)
                if not guest_can_join:
                    # This should be an auth check, but guests are a local concept,
                    # so don't really fit into the general auth process.
                    raise AuthError(403, "Guest access not allowed")
        if event.membership not in (Membership.LEAVE, Membership.BAN):
-            is_blocked = yield self.store.is_room_blocked(room_id)
+            is_blocked = await self.store.is_room_blocked(room_id)
            if is_blocked:
                raise SynapseError(403, "This room has been blocked on this server")
-        yield self.event_creation_handler.handle_new_client_event(
+        await self.event_creation_handler.handle_new_client_event(
            requester, event, context, extra_users=[target_user], ratelimit=ratelimit
        )
@ -630,15 +629,15 @@ class RoomMemberHandler(object):
            # info.
            newly_joined = True
            if prev_member_event_id:
-                prev_member_event = yield self.store.get_event(prev_member_event_id)
+                prev_member_event = await self.store.get_event(prev_member_event_id)
                newly_joined = prev_member_event.membership != Membership.JOIN
            if newly_joined:
-                yield self._user_joined_room(target_user, room_id)
+                await self._user_joined_room(target_user, room_id)
        elif event.membership == Membership.LEAVE:
            if prev_member_event_id:
-                prev_member_event = yield self.store.get_event(prev_member_event_id)
+                prev_member_event = await self.store.get_event(prev_member_event_id)
                if prev_member_event.membership == Membership.JOIN:
-                    yield self._user_left_room(target_user, room_id)
+                    await self._user_left_room(target_user, room_id)
    @defer.inlineCallbacks
    def _can_guest_join(self, current_state_ids):
@ -696,8 +695,7 @@ class RoomMemberHandler(object):
        if invite:
            return UserID.from_string(invite.sender)
-    @defer.inlineCallbacks
+    async def do_3pid_invite(
    def do_3pid_invite(
        self,
        room_id,
        inviter,
@ -709,7 +707,7 @@ class RoomMemberHandler(object):
        id_access_token=None,
    ):
        if self.config.block_non_admin_invites:
-            is_requester_admin = yield self.auth.is_server_admin(requester.user)
+            is_requester_admin = await self.auth.is_server_admin(requester.user)
            if not is_requester_admin:
                raise SynapseError(
                    403, "Invites have been disabled on this server", Codes.FORBIDDEN
@ -717,9 +715,9 @@ class RoomMemberHandler(object):
        # We need to rate limit *before* we send out any 3PID invites, so we
        # can't just rely on the standard ratelimiting of events.
-        yield self.base_handler.ratelimit(requester)
+        await self.base_handler.ratelimit(requester)
-        can_invite = yield self.third_party_event_rules.check_threepid_can_be_invited(
+        can_invite = await self.third_party_event_rules.check_threepid_can_be_invited(
            medium, address, room_id
        )
        if not can_invite:
@ -734,16 +732,16 @@ class RoomMemberHandler(object):
                403, "Looking up third-party identifiers is denied from this server"
            )
-        invitee = yield self.identity_handler.lookup_3pid(
+        invitee = await self.identity_handler.lookup_3pid(
            id_server, medium, address, id_access_token
        )
        if invitee:
-            yield self.update_membership(
+            await self.update_membership(
                requester, UserID.from_string(invitee), room_id, "invite", txn_id=txn_id
            )
        else:
-            yield self._make_and_store_3pid_invite(
+            await self._make_and_store_3pid_invite(
                requester,
                id_server,
                medium,
@ -754,8 +752,7 @@ class RoomMemberHandler(object):
                id_access_token=id_access_token,
            )
-    @defer.inlineCallbacks
+    async def _make_and_store_3pid_invite(
    def _make_and_store_3pid_invite(
        self,
        requester,
        id_server,
@ -766,7 +763,7 @@ class RoomMemberHandler(object):
        txn_id,
        id_access_token=None,
    ):
-        room_state = yield self.state_handler.get_current_state(room_id)
+        room_state = await self.state_handler.get_current_state(room_id)
        inviter_display_name = ""
        inviter_avatar_url = ""
@ -804,7 +801,7 @@ class RoomMemberHandler(object):
            public_keys,
            fallback_public_key,
            display_name,
-        ) = yield self.identity_handler.ask_id_server_for_third_party_invite(
+        ) = await self.identity_handler.ask_id_server_for_third_party_invite(
            requester=requester,
            id_server=id_server,
            medium=medium,
@ -820,7 +817,7 @@ class RoomMemberHandler(object):
            id_access_token=id_access_token,
        )
-        yield self.event_creation_handler.create_and_send_nonmember_event(
+        await self.event_creation_handler.create_and_send_nonmember_event(
            requester,
            {
                "type": EventTypes.ThirdPartyInvite,
@ -914,8 +911,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
        return complexity["v1"] > max_complexity
-    @defer.inlineCallbacks
+    async def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
        """Implements RoomMemberHandler._remote_join
        """
        # filter ourselves out of remote_room_hosts: do_invite_join ignores it
@ -930,7 +926,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
        if self.hs.config.limit_remote_rooms.enabled:
            # Fetch the room complexity
-            too_complex = yield self._is_remote_room_too_complex(
+            too_complex = await self._is_remote_room_too_complex(
                room_id, remote_room_hosts
            )
            if too_complex is True:
@ -944,12 +940,10 @@ class RoomMemberMasterHandler(RoomMemberHandler):
        # join dance for now, since we're kinda implicitly checking
        # that we are allowed to join when we decide whether or not we
        # need to do the invite/join dance.
-        yield defer.ensureDeferred(
+        await self.federation_handler.do_invite_join(
            self.federation_handler.do_invite_join(
            remote_room_hosts, room_id, user.to_string(), content
        )
-        )
+        await self._user_joined_room(user, room_id)
        yield self._user_joined_room(user, room_id)
        # Check the room we just joined wasn't too large, if we didn't fetch the
        # complexity of it before.
@ -959,7 +953,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
                return
            # Check again, but with the local state events
-            too_complex = yield self._is_local_room_too_complex(room_id)
+            too_complex = await self._is_local_room_too_complex(room_id)
            if too_complex is False:
                # We're under the limit.
@ -967,7 +961,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
            # The room is too large. Leave.
            requester = types.create_requester(user, None, False, None)
-            yield self.update_membership(
+            await self.update_membership(
                requester=requester, target=user, room_id=room_id, action="leave"
            )
            raise SynapseError(
@ -1005,12 +999,12 @@ class RoomMemberMasterHandler(RoomMemberHandler):
    def _user_joined_room(self, target, room_id):
        """Implements RoomMemberHandler._user_joined_room
        """
-        return user_joined_room(self.distributor, target, room_id)
+        return defer.succeed(user_joined_room(self.distributor, target, room_id))
    def _user_left_room(self, target, room_id):
        """Implements RoomMemberHandler._user_left_room
        """
-        return user_left_room(self.distributor, target, room_id)
+        return defer.succeed(user_left_room(self.distributor, target, room_id))
    @defer.inlineCallbacks
    def forget(self, user, room_id):
--- a/synapse/handlers/saml_handler.py
+++ b/synapse/handlers/saml_handler.py
@ -14,7 +14,7 @@
 # limitations under the License.
 import logging
 import re
-from typing import Tuple
+from typing import Optional, Tuple
 import attr
 import saml2
@ -26,6 +26,7 @@ from synapse.config import ConfigError
 from synapse.http.server import finish_request
 from synapse.http.servlet import parse_string
 from synapse.module_api import ModuleApi
 from synapse.module_api.errors import RedirectException
 from synapse.types import (
    UserID,
    map_username_to_mxid_localpart,
@ -43,11 +44,15 @@ class Saml2SessionData:
    # time the session was created, in milliseconds
    creation_time = attr.ib()
    # The user interactive authentication session ID associated with this SAML
    # session (or None if this SAML session is for an initial login).
    ui_auth_session_id = attr.ib(type=Optional[str], default=None)
 class SamlHandler:
    def __init__(self, hs):
        self._saml_client = Saml2Client(hs.config.saml2_sp_config)
        self._auth = hs.get_auth()
        self._auth_handler = hs.get_auth_handler()
        self._registration_handler = hs.get_registration_handler()
@ -76,12 +81,14 @@ class SamlHandler:
        self._error_html_content = hs.config.saml2_error_html_content
-    def handle_redirect_request(self, client_redirect_url):
+    def handle_redirect_request(self, client_redirect_url, ui_auth_session_id=None):
        """Handle an incoming request to /login/sso/redirect
        Args:
            client_redirect_url (bytes): the URL that we should redirect the
                client to when everything is done
            ui_auth_session_id (Optional[str]): The session ID of the ongoing UI Auth (or
                None if this is a login).
        Returns:
            bytes: URL to redirect to
@ -91,7 +98,9 @@ class SamlHandler:
        )
        now = self._clock.time_msec()
-        self._outstanding_requests_dict[reqid] = Saml2SessionData(creation_time=now)
+        self._outstanding_requests_dict[reqid] = Saml2SessionData(
            creation_time=now, ui_auth_session_id=ui_auth_session_id,
        )
        for key, value in info["headers"]:
            if key == "Location":
@ -118,7 +127,12 @@ class SamlHandler:
        self.expire_sessions()
        try:
-            user_id = await self._map_saml_response_to_user(resp_bytes, relay_state)
+            user_id, current_session = await self._map_saml_response_to_user(
                resp_bytes, relay_state
            )
        except RedirectException:
            # Raise the exception as per the wishes of the SAML module response
            raise
        except Exception as e:
            # If decoding the response or mapping it to a user failed, then log the
            # error and tell the user that something went wrong.
@ -133,9 +147,28 @@ class SamlHandler:
            finish_request(request)
            return
-        self._auth_handler.complete_sso_login(user_id, request, relay_state)
+        # Complete the interactive auth session or the login.
        if current_session and current_session.ui_auth_session_id:
            await self._auth_handler.complete_sso_ui_auth(
                user_id, current_session.ui_auth_session_id, request
            )
-    async def _map_saml_response_to_user(self, resp_bytes, client_redirect_url):
+        else:
            await self._auth_handler.complete_sso_login(user_id, request, relay_state)
    async def _map_saml_response_to_user(
        self, resp_bytes: str, client_redirect_url: str
    ) -> Tuple[str, Optional[Saml2SessionData]]:
        """
        Given a sample response, retrieve the cached session and user for it.
        Args:
            resp_bytes: The SAML response.
            client_redirect_url: The redirect URL passed in by the client.
        Returns:
             Tuple of the user ID and SAML session associated with this response.
        """
        try:
            saml2_auth = self._saml_client.parse_authn_request_response(
                resp_bytes,
@ -163,7 +196,9 @@ class SamlHandler:
        logger.info("SAML2 mapped attributes: %s", saml2_auth.ava)
-        self._outstanding_requests_dict.pop(saml2_auth.in_response_to, None)
+        current_session = self._outstanding_requests_dict.pop(
            saml2_auth.in_response_to, None
        )
        remote_user_id = self._user_mapping_provider.get_remote_user_id(
            saml2_auth, client_redirect_url
@ -184,7 +219,7 @@ class SamlHandler:
            )
            if registered_user_id is not None:
                logger.info("Found existing mapping %s", registered_user_id)
-                return registered_user_id
+                return registered_user_id, current_session
            # backwards-compatibility hack: see if there is an existing user with a
            # suitable mapping from the uid
@ -209,7 +244,7 @@ class SamlHandler:
                    await self._datastore.record_user_external_id(
                        self._auth_provider_id, remote_user_id, registered_user_id
                    )
-                    return registered_user_id
+                    return registered_user_id, current_session
            # Map saml response to user attributes using the configured mapping provider
            for i in range(1000):
@ -256,7 +291,7 @@ class SamlHandler:
            await self._datastore.record_user_external_id(
                self._auth_provider_id, remote_user_id, registered_user_id
            )
-            return registered_user_id
+            return registered_user_id, current_session
    def expire_sessions(self):
        expire_before = self._clock.time_msec() - self._saml2_session_lifetime
--- a/synapse/handlers/set_password.py
+++ b/synapse/handlers/set_password.py
@ -15,8 +15,6 @@
 import logging
 from typing import Optional
 from twisted.internet import defer
 from synapse.api.errors import Codes, StoreError, SynapseError
 from synapse.types import Requester
@ -32,9 +30,9 @@ class SetPasswordHandler(BaseHandler):
        super(SetPasswordHandler, self).__init__(hs)
        self._auth_handler = hs.get_auth_handler()
        self._device_handler = hs.get_device_handler()
        self._password_policy_handler = hs.get_password_policy_handler()
-    @defer.inlineCallbacks
+    async def set_password(
    def set_password(
        self,
        user_id: str,
        new_password: str,
@ -44,10 +42,11 @@ class SetPasswordHandler(BaseHandler):
        if not self.hs.config.password_localdb_enabled:
            raise SynapseError(403, "Password change disabled", errcode=Codes.FORBIDDEN)
-        password_hash = yield self._auth_handler.hash(new_password)
+        self._password_policy_handler.validate_password(new_password)
        password_hash = await self._auth_handler.hash(new_password)
        try:
-            yield self.store.user_set_password_hash(user_id, password_hash)
+            await self.store.user_set_password_hash(user_id, password_hash)
        except StoreError as e:
            if e.code == 404:
                raise SynapseError(404, "Unknown user", Codes.NOT_FOUND)
@ -59,12 +58,12 @@ class SetPasswordHandler(BaseHandler):
            except_access_token_id = requester.access_token_id if requester else None
            # First delete all of their other devices.
-            yield self._device_handler.delete_all_devices_for_user(
+            await self._device_handler.delete_all_devices_for_user(
                user_id, except_device_id=except_device_id
            )
            # and now delete any access tokens which weren't associated with
            # devices (or were associated with this device).
-            yield self._auth_handler.delete_access_tokens_for_user(
+            await self._auth_handler.delete_access_tokens_for_user(
                user_id, except_token_id=except_access_token_id
            )
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@ -26,7 +26,7 @@ from prometheus_client import Counter
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.filtering import FilterCollection
 from synapse.events import EventBase
-from synapse.logging.context import LoggingContext
+from synapse.logging.context import current_context
 from synapse.push.clientformat import format_push_rules_for_user
 from synapse.storage.roommember import MemberSummary
 from synapse.storage.state import StateFilter
@ -301,7 +301,7 @@ class SyncHandler(object):
        else:
            sync_type = "incremental_sync"
-        context = LoggingContext.current_context()
+        context = current_context()
        if context:
            context.tag = sync_type
@ -1143,10 +1143,14 @@ class SyncHandler(object):
                user_id
            )
-            tracked_users = set(users_who_share_room)
+            # Always tell the user about their own devices. We check as the user
            # ID is almost certainly already included (unless they're not in any
            # rooms) and taking a copy of the set is relatively expensive.
            if user_id not in users_who_share_room:
                users_who_share_room = set(users_who_share_room)
                users_who_share_room.add(user_id)
-            # Always tell the user about their own devices
+            tracked_users = users_who_share_room
            tracked_users.add(user_id)
            # Step 1a, check for changes in devices of users we share a room with
            users_that_have_changed = await self.store.get_users_whose_devices_changed(
@ -1639,7 +1643,7 @@ class SyncHandler(object):
        )
        # We loop through all room ids, even if there are no new events, in case
-        # there are non room events taht we need to notify about.
+        # there are non room events that we need to notify about.
        for room_id in sync_result_builder.joined_room_ids:
            room_entry = room_to_events.get(room_id, None)
--- a/synapse/handlers/typing.py
+++ b/synapse/handlers/typing.py
@ -15,6 +15,7 @@
 import logging
 from collections import namedtuple
 from typing import List
 from twisted.internet import defer
@ -257,7 +258,13 @@ class TypingHandler(object):
            "typing_key", self._latest_room_serial, rooms=[member.room_id]
        )
-    async def get_all_typing_updates(self, last_id, current_id):
+    async def get_all_typing_updates(
        self, last_id: int, current_id: int, limit: int
    ) -> List[dict]:
        """Get up to `limit` typing updates between the given tokens, earliest
        updates first.
        """
        if last_id == current_id:
            return []
@ -275,7 +282,7 @@ class TypingHandler(object):
                typing = self._room_typing[room_id]
                rows.append((serial, room_id, list(typing)))
        rows.sort()
-        return rows
+        return rows[:limit]
    def get_current_token(self):
        return self._latest_room_serial
--- a/synapse/http/matrixfederationclient.py
+++ b/synapse/http/matrixfederationclient.py
@ -434,6 +434,20 @@ class MatrixFederationHttpClient(object):
                        logger.info("Failed to send request: %s", e)
                        raise_from(RequestSendFailed(e, can_retry=True), e)
                    incoming_responses_counter.labels(method_bytes, response.code).inc()
                    set_tag(tags.HTTP_STATUS_CODE, response.code)
                    if 200 <= response.code < 300:
                        logger.debug(
                            "{%s} [%s] Got response headers: %d %s",
                            request.txn_id,
                            request.destination,
                            response.code,
                            response.phrase.decode("ascii", errors="replace"),
                        )
                        pass
                    else:
                        logger.info(
                            "{%s} [%s] Got response headers: %d %s",
                            request.txn_id,
@ -441,14 +455,6 @@ class MatrixFederationHttpClient(object):
                            response.code,
                            response.phrase.decode("ascii", errors="replace"),
                        )
                    incoming_responses_counter.labels(method_bytes, response.code).inc()
                    set_tag(tags.HTTP_STATUS_CODE, response.code)
                    if 200 <= response.code < 300:
                        pass
                    else:
                        # :'(
                        # Update transactions table?
                        d = treq.content(response)
--- a/synapse/http/request_metrics.py
+++ b/synapse/http/request_metrics.py
@ -19,7 +19,7 @@ import threading
 from prometheus_client.core import Counter, Histogram
-from synapse.logging.context import LoggingContext
+from synapse.logging.context import current_context
 from synapse.metrics import LaterGauge
 logger = logging.getLogger(__name__)
@ -148,7 +148,7 @@ LaterGauge(
 class RequestMetrics(object):
    def start(self, time_sec, name, method):
        self.start = time_sec
-        self.start_context = LoggingContext.current_context()
+        self.start_context = current_context()
        self.name = name
        self.method = method
@ -163,7 +163,7 @@ class RequestMetrics(object):
        with _in_flight_requests_lock:
            _in_flight_requests.discard(self)
-        context = LoggingContext.current_context()
+        context = current_context()
        tag = ""
        if context:
--- a/synapse/http/site.py
+++ b/synapse/http/site.py
@ -193,6 +193,12 @@ class SynapseRequest(Request):
        self.finish_time = time.time()
        Request.connectionLost(self, reason)
        if self.logcontext is None:
            logger.info(
                "Connection from %s lost before request headers were read", self.client
            )
            return
        # we only get here if the connection to the client drops before we send
        # the response.
        #
@ -236,13 +242,6 @@ class SynapseRequest(Request):
    def _finished_processing(self):
        """Log the completion of this request and update the metrics
        """
        if self.logcontext is None:
            # this can happen if the connection closed before we read the
            # headers (so render was never called). In that case we'll already
            # have logged a warning, so just bail out.
            return
        usage = self.logcontext.get_resource_usage()
        if self._processing_finished_time is None:
--- a/synapse/logging/_structured.py
+++ b/synapse/logging/_structured.py
@ -42,7 +42,7 @@ from synapse.logging._terse_json import (
    TerseJSONToConsoleLogObserver,
    TerseJSONToTCPLogObserver,
 )
-from synapse.logging.context import LoggingContext
+from synapse.logging.context import current_context
 def stdlib_log_level_to_twisted(level: str) -> LogLevel:
@ -86,7 +86,7 @@ class LogContextObserver(object):
            ].startswith("Timing out client"):
                return
-        context = LoggingContext.current_context()
+        context = current_context()
        # Copy the context information to the log event.
        if context is not None:
--- a/synapse/logging/context.py
+++ b/synapse/logging/context.py
@ -27,6 +27,7 @@ import inspect
 import logging
 import threading
 import types
 import warnings
 from typing import TYPE_CHECKING, Optional, Tuple, TypeVar, Union
 from typing_extensions import Literal
@ -51,7 +52,7 @@ try:
    is_thread_resource_usage_supported = True
-    def get_thread_resource_usage():
+    def get_thread_resource_usage() -> "Optional[resource._RUsage]":
        return resource.getrusage(RUSAGE_THREAD)
@ -60,7 +61,7 @@ except Exception:
    # won't track resource usage.
    is_thread_resource_usage_supported = False
-    def get_thread_resource_usage():
+    def get_thread_resource_usage() -> "Optional[resource._RUsage]":
        return None
@ -175,7 +176,54 @@ class ContextResourceUsage(object):
        return res
-LoggingContextOrSentinel = Union["LoggingContext", "LoggingContext.Sentinel"]
+LoggingContextOrSentinel = Union["LoggingContext", "_Sentinel"]
 class _Sentinel(object):
    """Sentinel to represent the root context"""
    __slots__ = ["previous_context", "finished", "request", "scope", "tag"]
    def __init__(self) -> None:
        # Minimal set for compatibility with LoggingContext
        self.previous_context = None
        self.finished = False
        self.request = None
        self.scope = None
        self.tag = None
    def __str__(self):
        return "sentinel"
    def copy_to(self, record):
        pass
    def copy_to_twisted_log_entry(self, record):
        record["request"] = None
        record["scope"] = None
    def start(self, rusage: "Optional[resource._RUsage]"):
        pass
    def stop(self, rusage: "Optional[resource._RUsage]"):
        pass
    def add_database_transaction(self, duration_sec):
        pass
    def add_database_scheduled(self, sched_sec):
        pass
    def record_event_fetch(self, event_count):
        pass
    def __nonzero__(self):
        return False
    __bool__ = __nonzero__  # python3
 SENTINEL_CONTEXT = _Sentinel()
 class LoggingContext(object):
@ -199,76 +247,33 @@ class LoggingContext(object):
        "_resource_usage",
        "usage_start",
        "main_thread",
-        "alive",
+        "finished",
        "request",
        "tag",
        "scope",
    ]
    thread_local = threading.local()
    class Sentinel(object):
        """Sentinel to represent the root context"""
        __slots__ = ["previous_context", "alive", "request", "scope", "tag"]
        def __init__(self) -> None:
            # Minimal set for compatibility with LoggingContext
            self.previous_context = None
            self.alive = None
            self.request = None
            self.scope = None
            self.tag = None
        def __str__(self):
            return "sentinel"
        def copy_to(self, record):
            pass
        def copy_to_twisted_log_entry(self, record):
            record["request"] = None
            record["scope"] = None
        def start(self):
            pass
        def stop(self):
            pass
        def add_database_transaction(self, duration_sec):
            pass
        def add_database_scheduled(self, sched_sec):
            pass
        def record_event_fetch(self, event_count):
            pass
        def __nonzero__(self):
            return False
        __bool__ = __nonzero__  # python3
    sentinel = Sentinel()
    def __init__(self, name=None, parent_context=None, request=None) -> None:
-        self.previous_context = LoggingContext.current_context()
+        self.previous_context = current_context()
        self.name = name
        # track the resources used by this context so far
        self._resource_usage = ContextResourceUsage()
-        # If alive has the thread resource usage when the logcontext last
+        # The thread resource usage when the logcontext became active. None
-        # became active.
+        # if the context is not currently active.
-        self.usage_start = None
+        self.usage_start = None  # type: Optional[resource._RUsage]
        self.main_thread = get_thread_id()
        self.request = None
        self.tag = ""
        self.alive = True
        self.scope = None  # type: Optional[_LogContextScope]
        # keep track of whether we have hit the __exit__ block for this context
        # (suggesting that the the thing that created the context thinks it should
        # be finished, and that re-activating it would suggest an error).
        self.finished = False
        self.parent_context = parent_context
        if self.parent_context is not None:
@ -287,40 +292,51 @@ class LoggingContext(object):
    def current_context(cls) -> LoggingContextOrSentinel:
        """Get the current logging context from thread local storage
        This exists for backwards compatibility. ``current_context()`` should be
        called directly.
        Returns:
            LoggingContext: the current logging context
        """
-        return getattr(cls.thread_local, "current_context", cls.sentinel)
+        warnings.warn(
            "synapse.logging.context.LoggingContext.current_context() is deprecated "
            "in favor of synapse.logging.context.current_context().",
            DeprecationWarning,
            stacklevel=2,
        )
        return current_context()
    @classmethod
    def set_current_context(
        cls, context: LoggingContextOrSentinel
    ) -> LoggingContextOrSentinel:
        """Set the current logging context in thread local storage
        This exists for backwards compatibility. ``set_current_context()`` should be
        called directly.
        Args:
            context(LoggingContext): The context to activate.
        Returns:
            The context that was previously active
        """
-        current = cls.current_context()
+        warnings.warn(
-
+            "synapse.logging.context.LoggingContext.set_current_context() is deprecated "
-        if current is not context:
+            "in favor of synapse.logging.context.set_current_context().",
-            current.stop()
+            DeprecationWarning,
-            cls.thread_local.current_context = context
+            stacklevel=2,
-            context.start()
+        )
-        return current
+        return set_current_context(context)
    def __enter__(self) -> "LoggingContext":
        """Enters this logging context into thread local storage"""
-        old_context = self.set_current_context(self)
+        old_context = set_current_context(self)
        if self.previous_context != old_context:
            logger.warning(
                "Expected previous context %r, found %r",
                self.previous_context,
                old_context,
            )
        self.alive = True
        return self
    def __exit__(self, type, value, traceback) -> None:
@ -329,24 +345,19 @@ class LoggingContext(object):
        Returns:
            None to avoid suppressing any exceptions that were thrown.
        """
-        current = self.set_current_context(self.previous_context)
+        current = set_current_context(self.previous_context)
        if current is not self:
-            if current is self.sentinel:
+            if current is SENTINEL_CONTEXT:
                logger.warning("Expected logging context %s was lost", self)
            else:
                logger.warning(
                    "Expected logging context %s but found %s", self, current
                )
        self.alive = False
-        # if we have a parent, pass our CPU usage stats on
+        # the fact that we are here suggests that the caller thinks that everything
-        if self.parent_context is not None and hasattr(
+        # is done and dusted for this logcontext, and further activity will not get
-            self.parent_context, "_resource_usage"
+        # recorded against the correct metrics.
-        ):
+        self.finished = True
            self.parent_context._resource_usage += self._resource_usage
            # reset them in case we get entered again
            self._resource_usage.reset()
    def copy_to(self, record) -> None:
        """Copy logging fields from this context to a log record or
@ -366,34 +377,62 @@ class LoggingContext(object):
        record["request"] = self.request
        record["scope"] = self.scope
-    def start(self) -> None:
+    def start(self, rusage: "Optional[resource._RUsage]") -> None:
        """
        Record that this logcontext is currently running.
        This should not be called directly: use set_current_context
        Args:
            rusage: the resources used by the current thread, at the point of
                switching to this logcontext. May be None if this platform doesn't
                support getrusuage.
        """
        if get_thread_id() != self.main_thread:
            logger.warning("Started logcontext %s on different thread", self)
            return
        if self.finished:
            logger.warning("Re-starting finished log context %s", self)
        # If we haven't already started record the thread resource usage so
        # far
-        if not self.usage_start:
+        if self.usage_start:
-            self.usage_start = get_thread_resource_usage()
+            logger.warning("Re-starting already-active log context %s", self)
        else:
            self.usage_start = rusage
-    def stop(self) -> None:
+    def stop(self, rusage: "Optional[resource._RUsage]") -> None:
        """
        Record that this logcontext is no longer running.
        This should not be called directly: use set_current_context
        Args:
            rusage: the resources used by the current thread, at the point of
                switching away from this logcontext. May be None if this platform
                doesn't support getrusuage.
        """
        try:
            if get_thread_id() != self.main_thread:
                logger.warning("Stopped logcontext %s on different thread", self)
                return
-        # When we stop, let's record the cpu used since we started
+            if not rusage:
                return
            # Record the cpu used since we started
            if not self.usage_start:
            # Log a warning on platforms that support thread usage tracking
            if is_thread_resource_usage_supported:
                logger.warning(
-                    "Called stop on logcontext %s without calling start", self
+                    "Called stop on logcontext %s without recording a start rusage",
                    self,
                )
                return
-        utime_delta, stime_delta = self._get_cputime()
+            utime_delta, stime_delta = self._get_cputime(rusage)
-        self._resource_usage.ru_utime += utime_delta
+            self.add_cputime(utime_delta, stime_delta)
-        self._resource_usage.ru_stime += stime_delta
+        finally:
            self.usage_start = None
    def get_resource_usage(self) -> ContextResourceUsage:
@ -409,25 +448,25 @@ class LoggingContext(object):
        # If we are on the correct thread and we're currently running then we
        # can include resource usage so far.
        is_main_thread = get_thread_id() == self.main_thread
-        if self.alive and self.usage_start and is_main_thread:
+        if self.usage_start and is_main_thread:
-            utime_delta, stime_delta = self._get_cputime()
+            rusage = get_thread_resource_usage()
            assert rusage is not None
            utime_delta, stime_delta = self._get_cputime(rusage)
            res.ru_utime += utime_delta
            res.ru_stime += stime_delta
        return res
-    def _get_cputime(self) -> Tuple[float, float]:
+    def _get_cputime(self, current: "resource._RUsage") -> Tuple[float, float]:
-        """Get the cpu usage time so far
+        """Get the cpu usage time between start() and the given rusage
        Args:
            rusage: the current resource usage
        Returns: Tuple[float, float]: seconds in user mode, seconds in system mode
        """
        assert self.usage_start is not None
        current = get_thread_resource_usage()
        # Indicate to mypy that we know that self.usage_start is None.
        assert self.usage_start is not None
        utime_delta = current.ru_utime - self.usage_start.ru_utime
        stime_delta = current.ru_stime - self.usage_start.ru_stime
@ -450,30 +489,52 @@ class LoggingContext(object):
        return utime_delta, stime_delta
    def add_cputime(self, utime_delta: float, stime_delta: float) -> None:
        """Update the CPU time usage of this context (and any parents, recursively).
        Args:
            utime_delta: additional user time, in seconds, spent in this context.
            stime_delta: additional system time, in seconds, spent in this context.
        """
        self._resource_usage.ru_utime += utime_delta
        self._resource_usage.ru_stime += stime_delta
        if self.parent_context:
            self.parent_context.add_cputime(utime_delta, stime_delta)
    def add_database_transaction(self, duration_sec: float) -> None:
        """Record the use of a database transaction and the length of time it took.
        Args:
            duration_sec: The number of seconds the database transaction took.
        """
        if duration_sec < 0:
            raise ValueError("DB txn time can only be non-negative")
        self._resource_usage.db_txn_count += 1
        self._resource_usage.db_txn_duration_sec += duration_sec
        if self.parent_context:
            self.parent_context.add_database_transaction(duration_sec)
    def add_database_scheduled(self, sched_sec: float) -> None:
        """Record a use of the database pool
        Args:
-            sched_sec (float): number of seconds it took us to get a
+            sched_sec: number of seconds it took us to get a connection
                connection
        """
        if sched_sec < 0:
            raise ValueError("DB scheduling time can only be non-negative")
        self._resource_usage.db_sched_duration_sec += sched_sec
        if self.parent_context:
            self.parent_context.add_database_scheduled(sched_sec)
    def record_event_fetch(self, event_count: int) -> None:
        """Record a number of events being fetched from the db
        Args:
-            event_count (int): number of events being fetched
+            event_count: number of events being fetched
        """
        self._resource_usage.evt_db_fetch_count += event_count
        if self.parent_context:
            self.parent_context.record_event_fetch(event_count)
 class LoggingContextFilter(logging.Filter):
@ -492,7 +553,7 @@ class LoggingContextFilter(logging.Filter):
        Returns:
            True to include the record in the log output.
        """
-        context = LoggingContext.current_context()
+        context = current_context()
        for key, value in self.defaults.items():
            setattr(record, key, value)
@ -512,27 +573,24 @@ class PreserveLoggingContext(object):
    __slots__ = ["current_context", "new_context", "has_parent"]
-    def __init__(self, new_context: Optional[LoggingContextOrSentinel] = None) -> None:
+    def __init__(
-        if new_context is None:
+        self, new_context: LoggingContextOrSentinel = SENTINEL_CONTEXT
-            self.new_context = LoggingContext.sentinel  # type: LoggingContextOrSentinel
+    ) -> None:
        else:
        self.new_context = new_context
    def __enter__(self) -> None:
        """Captures the current logging context"""
-        self.current_context = LoggingContext.set_current_context(self.new_context)
+        self.current_context = set_current_context(self.new_context)
        if self.current_context:
            self.has_parent = self.current_context.previous_context is not None
            if not self.current_context.alive:
                logger.debug("Entering dead context: %s", self.current_context)
    def __exit__(self, type, value, traceback) -> None:
        """Restores the current logging context"""
-        context = LoggingContext.set_current_context(self.current_context)
+        context = set_current_context(self.current_context)
        if context != self.new_context:
-            if context is LoggingContext.sentinel:
+            if not context:
                logger.warning("Expected logging context %s was lost", self.new_context)
            else:
                logger.warning(
@ -541,9 +599,37 @@ class PreserveLoggingContext(object):
                    context,
                )
-        if self.current_context is not LoggingContext.sentinel:
+
-            if not self.current_context.alive:
+_thread_local = threading.local()
-                logger.debug("Restoring dead context: %s", self.current_context)
+_thread_local.current_context = SENTINEL_CONTEXT
 def current_context() -> LoggingContextOrSentinel:
    """Get the current logging context from thread local storage"""
    return getattr(_thread_local, "current_context", SENTINEL_CONTEXT)
 def set_current_context(context: LoggingContextOrSentinel) -> LoggingContextOrSentinel:
    """Set the current logging context in thread local storage
    Args:
        context(LoggingContext): The context to activate.
    Returns:
        The context that was previously active
    """
    # everything blows up if we allow current_context to be set to None, so sanity-check
    # that now.
    if context is None:
        raise TypeError("'context' argument may not be None")
    current = current_context()
    if current is not context:
        rusage = get_thread_resource_usage()
        current.stop(rusage)
        _thread_local.current_context = context
        context.start(rusage)
    return current
 def nested_logging_context(
@ -572,7 +658,7 @@ def nested_logging_context(
    if parent_context is not None:
        context = parent_context  # type: LoggingContextOrSentinel
    else:
-        context = LoggingContext.current_context()
+        context = current_context()
    return LoggingContext(
        parent_context=context, request=str(context.request) + "-" + suffix
    )
@ -604,7 +690,7 @@ def run_in_background(f, *args, **kwargs):
    CRITICAL error about an unhandled error will be logged without much
    indication about where it came from.
    """
-    current = LoggingContext.current_context()
+    current = current_context()
    try:
        res = f(*args, **kwargs)
    except:  # noqa: E722
@ -625,7 +711,7 @@ def run_in_background(f, *args, **kwargs):
    # The function may have reset the context before returning, so
    # we need to restore it now.
-    ctx = LoggingContext.set_current_context(current)
+    ctx = set_current_context(current)
    # The original context will be restored when the deferred
    # completes, but there is nothing waiting for it, so it will
@ -674,7 +760,7 @@ def make_deferred_yieldable(deferred):
    # ok, we can't be sure that a yield won't block, so let's reset the
    # logcontext, and add a callback to the deferred to restore it.
-    prev_context = LoggingContext.set_current_context(LoggingContext.sentinel)
+    prev_context = set_current_context(SENTINEL_CONTEXT)
    deferred.addBoth(_set_context_cb, prev_context)
    return deferred
@ -684,7 +770,7 @@ ResultT = TypeVar("ResultT")
 def _set_context_cb(result: ResultT, context: LoggingContext) -> ResultT:
    """A callback function which just sets the logging context"""
-    LoggingContext.set_current_context(context)
+    set_current_context(context)
    return result
@ -752,7 +838,7 @@ def defer_to_threadpool(reactor, threadpool, f, *args, **kwargs):
        Deferred: A Deferred which fires a callback with the result of `f`, or an
            errback if `f` throws an exception.
    """
-    logcontext = LoggingContext.current_context()
+    logcontext = current_context()
    def g():
        with LoggingContext(parent_context=logcontext):
--- a/synapse/logging/opentracing.py
+++ b/synapse/logging/opentracing.py
@ -171,7 +171,7 @@ import logging
 import re
 import types
 from functools import wraps
-from typing import Dict
+from typing import TYPE_CHECKING, Dict
 from canonicaljson import json
@ -179,6 +179,9 @@ from twisted.internet import defer
 from synapse.config import ConfigError
 if TYPE_CHECKING:
    from synapse.server import HomeServer
 # Helper class
@ -297,14 +300,11 @@ def _noop_context_manager(*args, **kwargs):
 # Setup
-def init_tracer(config):
+def init_tracer(hs: "HomeServer"):
    """Set the whitelists and initialise the JaegerClient tracer
    Args:
        config (HomeserverConfig): The config used by the homeserver
    """
    global opentracing
-    if not config.opentracer_enabled:
+    if not hs.config.opentracer_enabled:
        # We don't have a tracer
        opentracing = None
        return
@ -315,18 +315,15 @@ def init_tracer(config):
            "installed."
        )
    # Include the worker name
    name = config.worker_name if config.worker_name else "master"
    # Pull out the jaeger config if it was given. Otherwise set it to something sensible.
    # See https://github.com/jaegertracing/jaeger-client-python/blob/master/jaeger_client/config.py
-    set_homeserver_whitelist(config.opentracer_whitelist)
+    set_homeserver_whitelist(hs.config.opentracer_whitelist)
    JaegerConfig(
-        config=config.jaeger_config,
+        config=hs.config.jaeger_config,
-        service_name="{} {}".format(config.server_name, name),
+        service_name="{} {}".format(hs.config.server_name, hs.get_instance_name()),
-        scope_manager=LogContextScopeManager(config),
+        scope_manager=LogContextScopeManager(hs.config),
    ).initialize_tracer()
--- a/synapse/logging/scopecontextmanager.py
+++ b/synapse/logging/scopecontextmanager.py
@ -19,7 +19,7 @@ from opentracing import Scope, ScopeManager
 import twisted
-from synapse.logging.context import LoggingContext, nested_logging_context
+from synapse.logging.context import current_context, nested_logging_context
 logger = logging.getLogger(__name__)
@ -49,10 +49,7 @@ class LogContextScopeManager(ScopeManager):
            (Scope) : the Scope that is active, or None if not
            available.
        """
-        ctx = LoggingContext.current_context()
+        ctx = current_context()
        if ctx is LoggingContext.sentinel:
            return None
        else:
        return ctx.scope
    def activate(self, span, finish_on_close):
@ -70,9 +67,9 @@ class LogContextScopeManager(ScopeManager):
        """
        enter_logcontext = False
-        ctx = LoggingContext.current_context()
+        ctx = current_context()
-        if ctx is LoggingContext.sentinel:
+        if not ctx:
            # We don't want this scope to affect.
            logger.error("Tried to activate scope outside of loggingcontext")
            return Scope(None, span)
--- a/synapse/module_api/init.py
+++ b/synapse/module_api/init.py
@ -86,7 +86,7 @@ class ModuleApi(object):
            Deferred[str|None]: Canonical (case-corrected) user_id, or None
               if the user is not registered.
        """
-        return self._auth_handler.check_user_exists(user_id)
+        return defer.ensureDeferred(self._auth_handler.check_user_exists(user_id))
    @defer.inlineCallbacks
    def register(self, localpart, displayname=None, emails=[]):
@ -196,7 +196,9 @@ class ModuleApi(object):
            yield self._hs.get_device_handler().delete_device(user_id, device_id)
        else:
            # no associated device. Just delete the access token.
-            yield self._auth_handler.delete_access_token(access_token)
+            yield defer.ensureDeferred(
                self._auth_handler.delete_access_token(access_token)
            )
    def run_db_interaction(self, desc, func, *args, **kwargs):
        """Run a function with a database connection
@ -220,6 +222,8 @@ class ModuleApi(object):
        want their access token sent to `client_redirect_url`, or redirect them to that
        URL with a token directly if the URL matches with one of the whitelisted clients.
        This is deprecated in favor of complete_sso_login_async.
        Args:
            registered_user_id: The MXID that has been registered as a previous step of
                of this SSO login.
@ -227,6 +231,24 @@ class ModuleApi(object):
            client_redirect_url: The URL to which to offer to redirect the user (or to
                redirect them directly if whitelisted).
        """
-        self._auth_handler.complete_sso_login(
+        self._auth_handler._complete_sso_login(
            registered_user_id, request, client_redirect_url,
        )
    async def complete_sso_login_async(
        self, registered_user_id: str, request: SynapseRequest, client_redirect_url: str
    ):
        """Complete a SSO login by redirecting the user to a page to confirm whether they
        want their access token sent to `client_redirect_url`, or redirect them to that
        URL with a token directly if the URL matches with one of the whitelisted clients.
        Args:
            registered_user_id: The MXID that has been registered as a previous step of
                of this SSO login.
            request: The request to respond to.
            client_redirect_url: The URL to which to offer to redirect the user (or to
                redirect them directly if whitelisted).
        """
        await self._auth_handler.complete_sso_login(
            registered_user_id, request, client_redirect_url,
        )
--- a/synapse/notifier.py
+++ b/synapse/notifier.py
@ -15,7 +15,7 @@
 import logging
 from collections import namedtuple
-from typing import Callable, List
+from typing import Callable, Iterable, List, TypeVar
 from prometheus_client import Counter
@ -42,12 +42,14 @@ users_woken_by_stream_counter = Counter(
    "synapse_notifier_users_woken_by_stream", "", ["stream"]
 )
 T = TypeVar("T")
 # TODO(paul): Should be shared somewhere
-def count(func, l):
+def count(func: Callable[[T], bool], it: Iterable[T]) -> int:
-    """Return the number of items in l for which func returns true."""
+    """Return the number of items in it for which func returns true."""
    n = 0
-    for x in l:
+    for x in it:
        if func(x):
            n += 1
    return n
@ -220,12 +222,6 @@ class Notifier(object):
        """
        self.replication_callbacks.append(cb)
    def add_remote_server_up_callback(self, cb: Callable[[str], None]):
        """Add a callback that will be called when synapse detects a server
        has been
        """
        self.remote_server_up_callbacks.append(cb)
    def on_new_room_event(
        self, event, room_stream_id, max_room_stream_id, extra_users=[]
    ):
@ -279,10 +275,9 @@ class Notifier(object):
            "room_key", room_stream_id, users=extra_users, rooms=[event.room_id]
        )
-    @defer.inlineCallbacks
+    async def _notify_app_services(self, room_stream_id):
    def _notify_app_services(self, room_stream_id):
        try:
-            yield self.appservice_handler.notify_interested_services(room_stream_id)
+            await self.appservice_handler.notify_interested_services(room_stream_id)
        except Exception:
            logger.exception("Error notifying application services of event")
@ -481,20 +476,18 @@ class Notifier(object):
        return result
-    @defer.inlineCallbacks
+    async def _get_room_ids(self, user, explicit_room_id):
-    def _get_room_ids(self, user, explicit_room_id):
+        joined_room_ids = await self.store.get_rooms_for_user(user.to_string())
        joined_room_ids = yield self.store.get_rooms_for_user(user.to_string())
        if explicit_room_id:
            if explicit_room_id in joined_room_ids:
                return [explicit_room_id], True
-            if (yield self._is_world_readable(explicit_room_id)):
+            if await self._is_world_readable(explicit_room_id):
                return [explicit_room_id], False
            raise AuthError(403, "Non-joined access not allowed")
        return joined_room_ids, True
-    @defer.inlineCallbacks
+    async def _is_world_readable(self, room_id):
-    def _is_world_readable(self, room_id):
+        state = await self.state_handler.get_current_state(
        state = yield self.state_handler.get_current_state(
            room_id, EventTypes.RoomHistoryVisibility, ""
        )
        if state and "history_visibility" in state.content:
@ -544,6 +537,3 @@ class Notifier(object):
        # circular dependencies.
        if self.federation_sender:
            self.federation_sender.wake_destination(server)
        for cb in self.remote_server_up_callbacks:
            cb(server)
--- a/Show More
+++ b/Show More