Commit Graph

12111 Commits

Author SHA1 Message Date
Erik Johnston
1e03513f9a
Fix new metric where we used ms instead of seconds (#7771)
Introduced in #7755, not yet released.
2020-07-01 15:23:58 +01:00
Richard van der Hoff
244dbb04f7
Fix incorrect error message when database CTYPE was set incorrectly. (#7760) 2020-07-01 13:56:16 +01:00
Andrew Morgan
dc80a0762d 1.16.0rc1 2020-07-01 11:26:58 +01:00
Brendan Abolivier
74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Patrick Cloke
71cccf1593
Additional configuration options for auto-join rooms (#7763) 2020-06-30 15:41:36 -04:00
Erik Johnston
a99658074d
Add some metrics for inbound and outbound federation processing times (#7755) 2020-06-30 16:58:06 +01:00
Andrew Morgan
2f6afdd8b4
Explain the purpose of the "tests" conditional dependency requirement (#7751) 2020-06-30 10:11:36 +01:00
Erik Johnston
831b31e563
Add another yield point to state res v2 (#7746) 2020-06-26 10:44:52 +01:00
Erik Johnston
0e0a2817a2
Yield during large v2 state res. (#7735)
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.

To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
2020-06-24 18:48:18 +01:00
Sorunome
6920e58136
add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675) 2020-06-24 10:23:55 +01:00
Christian Svensson
8bbe87f42d
Set Content-Length for Metrics requests (#7730)
HTTP requires the response to contain a Content-Length header unless chunked encoding is being used.
Prometheus metrics endpoint did not set this, causing software such as prometheus-proxy to not be able to scrape synapse for metrics.

Signed-off-by: Christian Svensson <blue@cmd.nu>
2020-06-23 18:06:01 +01:00
Patrick Cloke
95e41f368b
Allow local media to be marked as safe from being quarantined. (#7718) 2020-06-22 08:04:14 -04:00
Patrick Cloke
e060bf4462
Convert directory handler to async/await (#7727) 2020-06-22 07:18:00 -04:00
Erik Johnston
91e886d615
Speed up state res v2 across large state differences. (#7725) 2020-06-19 13:56:35 +01:00
Richard van der Hoff
7d2824395f add a comment 2020-06-18 10:47:06 +01:00
Brendan Abolivier
e35d44c01d
Merge pull request #7716 from matrix-org/babolivier/unread_fix
Fix unread counts in sync
2020-06-17 15:44:15 +01:00
Patrick Cloke
3630825612
Convert the typing handler to async/await. (#7679) 2020-06-17 10:37:59 -04:00
Oleg Girko
96bc110a68
Require parameterized package version to be at least 0.7.0. (#7680)
Older versions of `parameterized` package have no `parameterized_class` decorator. This decorator is used in tests.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2020-06-17 15:31:40 +01:00
Brendan Abolivier
5a5cf6460e
Fix unread counts in sync
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
2020-06-17 15:10:44 +01:00
Erik Johnston
e07a8caf58
Add support for using rust-python-jaeger-reporter (#7697) 2020-06-17 14:13:41 +01:00
Erik Johnston
b44bdd7f7b
Support running multiple media repos. (#7706)
This requires a new config option to specify which media repo should be
responsible for running background jobs to e.g. clear out expired URL
preview caches.
2020-06-17 14:13:30 +01:00
Patrick Cloke
434716e1d3
Fetch from the r0 media path instead of the unspecced v1. (#7714) 2020-06-17 08:36:46 -04:00
Brendan Abolivier
46613aaf79
Implement unread counter (MSC2625) (#7673)
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
Richard van der Hoff
e452973fd2
fix broken link in sample config (#7712) 2020-06-16 19:50:16 +01:00
Erik Johnston
f6f7511a4c
Refactor getting replication updates from database. (#7636)
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
2020-06-16 17:10:28 +01:00
Patrick Cloke
231252516c
Fix "argument of type 'ObservableDeferred' is not iterable" error (#7708) 2020-06-16 12:01:18 -04:00
Patrick Cloke
ac51bd581a
Include a user agent in federation requests. (#7677) 2020-06-16 10:43:29 -04:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke
98c4e35e3c
Convert the device message and pagination handlers to async/await. (#7678) 2020-06-16 08:06:17 -04:00
Richard van der Hoff
03619324fc
Create a ListenerConfig object (#7681)
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Brendan Abolivier
789606577a Merge branch 'master' into develop 2020-06-16 10:36:29 +01:00
Brendan Abolivier
65eb078498
1.15.1 2020-06-16 10:28:58 +01:00
Andrew Morgan
3e6b5bba71
Wrap register_device coroutine in an ensureDeferred (#7684)
Fixes https://github.com/matrix-org/synapse/issues/7683

Broke in: #7649

We had a `yield` acting on a coroutine. To be fair this one is a bit difficult to notice as there's a function in the middle that just passes the coroutine along.
2020-06-16 10:13:59 +01:00
Patrick Cloke
cc32fa7358
Ensure the body is a string before comparing push rules. (#7701) 2020-06-15 16:20:34 -04:00
Hubert Chathi
2b2344652b
Ensure etag is a string for GET room_keys/version response (#7691) 2020-06-15 13:42:44 -04:00
Andrew Morgan
b8ee03caff
Update m.id.phone to use 'phone' instead of 'number' (#7687)
The spec [states](https://matrix.org/docs/spec/client_server/r0.6.1#phone-number) that `m.id.phone` requires the field `country` and `phone`.

In Synapse, we've been enforcing `country` and `number`.

I am not currently sure whether this affects any client implementations.

This issue was introduced in #1994.
2020-06-15 17:46:23 +01:00
Erik Johnston
4241a10673
Fix "There was no active span when trying to log." error (#7698) 2020-06-15 16:42:21 +01:00
Brendan Abolivier
6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Brendan Abolivier
c2b4621630 Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:08 +01:00
Brendan Abolivier
6d5985e1f2
Enable 3PID add/bind/unbind endpoints on r0 routes 2020-06-15 16:14:13 +01:00
Patrick Cloke
7d2532be36
Discard RDATA from already seen positions. (#7648) 2020-06-15 08:44:54 -04:00
Patrick Cloke
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Brendan Abolivier
fed493c5fd
Incorporate review 2020-06-15 09:58:55 +01:00
Patrick Cloke
2d11ea385c
Fix warnings about losing log context during UI auth. (#7688) 2020-06-12 15:01:00 -04:00
Patrick Cloke
d0a43d431e
Fix a typo when comparing the URI & method during UI Auth. (#7689) 2020-06-12 14:12:04 -04:00
Brendan Abolivier
e186c660b1
Lint 2020-06-12 15:31:59 +01:00
Brendan Abolivier
e47e5a2dcd
Incorporate review bits 2020-06-12 15:13:12 +01:00
Brendan Abolivier
1e5a50302f
Pre-populate the unread_count column 2020-06-12 15:05:47 +01:00
Brendan Abolivier
9549d557ea
Don't update the schema version 2020-06-12 15:03:26 +01:00
Brendan Abolivier
cf92fbb8aa
Use attr instead of a dict 2020-06-12 15:02:15 +01:00
Brendan Abolivier
3cc7f43e8d
Fix summary rotation 2020-06-12 11:07:26 +01:00
Will Hunt
a3fbc23c39
Remove "user_id" from GET /presence. (#7606) 2020-06-11 14:13:53 -04:00
Brendan Abolivier
cb6d4d07b1
Log for invalid values of notif 2020-06-11 18:30:31 +01:00
Brendan Abolivier
803291728c
Fix SQL 2020-06-11 18:25:25 +01:00
Brendan Abolivier
34fd1f7ab5
Fix schema update 2020-06-11 18:12:12 +01:00
Brendan Abolivier
d0f095625c
Lint 2020-06-11 18:04:43 +01:00
Brendan Abolivier
ce74a6685d
Save the count of unread messages to event_push_summary 2020-06-11 17:58:26 +01:00
Brendan Abolivier
ea8f6e611b
Actually act on mark_unread 2020-06-11 15:30:42 +01:00
Brendan Abolivier
1ad06ee6eb Merge branch 'master' into develop 2020-06-11 13:31:22 +01:00
Brendan Abolivier
3b3f327a0d 1.15.0 2020-06-11 13:27:27 +01:00
Patrick Cloke
b9df7f70bb
Increase the default SAML session expirary time to 15 minutes. (#7664) 2020-06-11 07:55:45 -04:00
wondratsch
c746889bb0
fix typo in sample_config.yaml (#7652)
Just a simple typo fix.

Signed-off-by: wondratsch 28294257+wondratsch@users.noreply.github.com
2020-06-11 11:51:10 +01:00
Brendan Abolivier
9dbd006607
Appease mypy 2020-06-10 20:44:24 +01:00
Brendan Abolivier
243f0ba6ce
Lint 2020-06-10 20:35:35 +01:00
Brendan Abolivier
df3323a7cf
Use temporary prefixes as per the MSC 2020-06-10 20:32:01 +01:00
Richard van der Hoff
0df618f813
Take out a lock before modifying _CACHES (#7663)
This should fix #7610.
2020-06-10 18:27:49 +01:00
Brendan Abolivier
c7b99a1180
Use a more efficient way of calculating counters 2020-06-10 17:54:33 +01:00
Andrew Morgan
fcd6961441
Add option to enable encryption by default for new rooms (#7639)
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on #7637
2020-06-10 17:44:34 +01:00
Brendan Abolivier
ef345c5a7b
Add a new unread_counter to sync responses 2020-06-10 16:21:16 +01:00
Patrick Cloke
191dc98f80
Clean-up the fallback login code. (#7657) 2020-06-10 09:50:39 -04:00
Brendan Abolivier
6f6a4bfc07
Rename dont_push into mark_unread 2020-06-10 14:24:01 +01:00
Brendan Abolivier
ec0a7b9034 Merge branch 'develop' into babolivier/mark_unread 2020-06-10 11:42:30 +01:00
Brendan Abolivier
236d2d699d 1.15.0rc1 2020-06-09 16:37:14 +01:00
Brendan Abolivier
2dc9468c27 Revert "1.15.0rc1"
This reverts commit 8587b0426f.
2020-06-09 16:34:37 +01:00
Brendan Abolivier
8587b0426f 1.15.0rc1 2020-06-09 16:33:36 +01:00
Erik Johnston
664409b169
Fix bug in account data replication stream. (#7656)
* Ensure account data stream IDs are unique.

The account data stream is shared between three tables, and the maximum
allocated ID was tracked in a dedicated table. Updating the max ID
happened outside the transaction that allocated the ID, leading to a
race where if the server was restarted then the same ID could be
allocated but the max ID failed to be updated, leading it to be reused.

The ID generators have support for tracking across multiple tables, so
we may as well use that instead of a dedicated table.

* Fix bug in account data replication stream.

If the same stream ID was used in both global and room account data then
the getting updates for the replication stream would fail due to
`heapq.merge(..)` trying to compare a `str` with a `None`. (This is
because you'd have two rows like `(534, '!room')` and `(534, None)` from
the room and global account data tables).

Fix is just to order by stream ID, since we don't rely on the ordering
beyond that. The bug where stream IDs can be reused should be fixed now,
so this case shouldn't happen going forward.

Fixes #7617
2020-06-09 16:28:57 +01:00
Patrick Cloke
3c45a78090
Convert the registration handler to async/await. (#7649) 2020-06-08 11:15:02 -04:00
Patrick Cloke
375ca0cceb
Accept device information at the login fallback endpoint. (#7629) 2020-06-08 10:13:24 -04:00
Patrick Cloke
737b4a936e
Convert user directory handler and related classes to async/await. (#7640) 2020-06-05 14:42:55 -04:00
Travis Ralston
09099313e6
Add an option to disable autojoin for guest accounts (#6637)
Fixes https://github.com/matrix-org/synapse/issues/3177
2020-06-05 18:18:15 +01:00
Patrick Cloke
a0d2d81cf9
Update to the stable SSO prefix for UI Auth. (#7630) 2020-06-05 10:50:08 -04:00
Richard van der Hoff
eea124370b
Fix type information on assert_*_is_admin methods (#7645)
These things don't return Deferreds.
2020-06-05 14:33:49 +01:00
Richard van der Hoff
b4f8dcb4bd
Remove some unused constants. (#7644) 2020-06-05 14:33:35 +01:00
Patrick Cloke
f1e61ef85c Typo fixes. 2020-06-05 08:43:21 -04:00
Dirk Klimpel
908f9e2d24
Allow new users to be registered via the admin API even if the monthly active user limit has been reached (#7263) 2020-06-05 13:08:49 +01:00
Dirk Klimpel
2970ce8367
Add device management to admin API (#7481)
- Admin is able to
  - change displaynames
  - delete devices
  - list devices
  - get device informations

Fixes #7330
2020-06-05 13:07:22 +01:00
Patrick Cloke
02f345d053
Attempt to fix PhoneHomeStatsTestCase.test_performance_100 being flaky. (#7634) 2020-06-05 07:36:47 -04:00
Andrew Morgan
139bc86f3d
Support CS API v0.6.0 (#6585) 2020-06-05 12:27:37 +01:00
WGH
e55ee7c32f
Add support for webp thumbnailing (#7586)
Closes #4382

Signed-off-by: Maxim Plotnikov <wgh@torlan.ru>
2020-06-05 11:54:27 +01:00
Andrew Morgan
f4e6495b5d
Performance improvements and refactor of Ratelimiter (#7595)
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:

* Rather undocumented, and
* causing a *lot* of config checks

This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. 

Best to be reviewed commit-by-commit.
2020-06-05 10:47:20 +01:00
Andrew Morgan
c389bfb6ea
Fix encryption algorithm typos in tests/comments (#7637)
@uhoreg has confirmed these were both typos. They are only in comments and tests though, rather than anything critical.

Introduced in:

* https://github.com/matrix-org/synapse/pull/7157
* https://github.com/matrix-org/synapse/pull/5726
2020-06-04 20:03:40 +01:00
Patrick Cloke
f8b9ead3ee
Advertise the token login type when OpenID Connect is enabled. (#7631) 2020-06-04 06:49:51 -04:00
Richard van der Hoff
11de843626
Cleanups to the OpenID Connect integration (#7628)
docs, default configs, comments. Nothing very significant.
2020-06-03 21:13:17 +01:00
Andrew Morgan
e91abfd291
async/await get_user_id_by_threepid (#7620)
Based on #7619 

async's `get_user_id_by_threepid` and its call stack.
2020-06-03 17:15:57 +01:00
Brendan Abolivier
c9507be989
Check if the localpart is reserved for guests earlier in the registration flow (#7625)
This is so the user is warned about the username not being valid as soon as possible, rather than only once they've finished UIA.
2020-06-03 16:55:02 +02:00
Erik Johnston
11dc2b4698
Fix exceptions when fetching events from a down host. (#7622)
We already caught some exceptions, but not all.
2020-06-03 14:12:13 +01:00
Richard van der Hoff
1bbc9e2df6
Clean up exception handling in SAML2ResponseResource (#7614)
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string

* Clean up exception handling in SAML2ResponseResource

  * use the existing code in `return_html_error` instead of re-implementing it
    (giving it a jinja2 template rather than inventing a new form of template)

  * do the exception-catching in the REST layer rather than in the handler
    layer, to make sure we catch all exceptions.
2020-06-03 10:41:12 +01:00
Olof Johansson
fe434cd3c9
Fix a bug in automatic user creation with m.login.jwt. (#7585) 2020-06-01 12:55:07 -04:00
Brendan Abolivier
33c39ab93c
Process cross-signing keys when resyncing device lists (#7594)
It looks like `user_device_resync` was ignoring cross-signing keys from the results received from the remote server. This patch fixes this, by processing these keys using the same process `_handle_signing_key_updates` does (and effectively factor that part out of that function).
2020-06-01 17:47:30 +02:00
Dirk Klimpel
901b1fa561
Email notifications for new users when creating via the Admin API. (#7267) 2020-06-01 15:34:33 +01:00
Dagfinn Ilmari Mannsåker
df8a3cef6b
Improve performance of _get_state_groups_from_groups_txn (#7567)
The query keeps showing up in my slow query log.

This changes the plan under the top-level Sort node from

```
    WindowAgg  (cost=280335.88..292963.15 rows=561212 width=80) (actual time=138.651..160.562 rows=27112 loops=1)
      ->  Sort  (cost=280335.88..281738.91 rows=561212 width=84) (actual time=138.597..140.622 rows=27112 loops=1)
            Sort Key: state_groups_state.type, state_groups_state.state_key, state_groups_state.state_group
            Sort Method: quicksort  Memory: 4581kB
            ->  Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=21.548..47.657 rows=27112 loops=1)
                  ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=21.526..21.535 rows=20 loops=1)
                        Group Key: state.state_group
                        ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=21.280..21.493 rows=20 loops=1)
                  ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.005..0.991 rows=1356 loops=20)
                        Index Cond: (state_group = state.state_group)
```

to

```
    Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=24.194..52.834 rows=27112 loops=1)
      ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=24.130..24.138 rows=20 loops=1)
            Group Key: state.state_group
            ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=23.887..24.113 rows=20 loops=1)
      ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.016..1.159 rows=1356 loops=20)
            Index Cond: (state_group = state.state_group)
```

This cuts the execution time from ~190ms to ~130ms, i.e. a reduction
of ~30%.

The full plans are visualised at https://explain.depesz.com/s/WpbT and
https://explain.depesz.com/s/KlEk

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-06-01 15:23:43 +01:00
Patrick Cloke
6af9cdca24
Convert groups local and server to async/await. (#7600) 2020-06-01 07:28:43 -04:00
Brendan Abolivier
c1bdd4fac7
Don't fail all of an iteration of the device list retry loop on error (#7609)
Without this patch, if an error happens which isn't caught by `user_device_resync`, then `_maybe_retry_device_resync` would fail, without retrying the next users in the iteration. This patch fixes this so that it now only logs an error in this case.
2020-06-01 12:55:14 +02:00
Dagfinn Ilmari Mannsåker
2dc430d36e
Use upsert when inserting read receipts (#7607)
Fixes #7469

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-06-01 10:53:06 +01:00
Erik Johnston
cb495f526d
Fix 'FederationGroupsRoomsServlet' API when group has room server is not in. (#7599) 2020-05-29 17:49:47 +01:00
Erik Johnston
f5353eff21
Make inflight background metrics more efficient. (#7597)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-05-29 13:25:32 +01:00
Brendan Abolivier
5cb470b495 Merge branch 'master' into develop 2020-05-28 12:50:26 +02:00
Brendan Abolivier
61469308df
1.14.0 2020-05-28 12:36:00 +02:00
Erik Johnston
8c5f88fa4d
Merge pull request #7584 from matrix-org/erikj/save_and_send_fed_token_in_bg
Speed up processing of federation stream RDATA rows.
2020-05-27 20:06:29 +01:00
Erik Johnston
ef3934ec8f Ensure we persist and ack the same token 2020-05-27 19:45:42 +01:00
Erik Johnston
3d7f1b53d9 Remove spurious change 2020-05-27 19:41:44 +01:00
Erik Johnston
35c308731d Speed up processing of federation stream RDATA rows.
Instead of storing and sending an ACK for every single row we send
synchronously, we instead do it asynchronously while batching up
updates.
2020-05-27 19:34:07 +01:00
Christopher Cooper
c4a820b32a
allow emails to be passed through SAML (#7385)
Signed-off-by: Christopher Cooper <cooperc@ocf.berkeley.edu>
2020-05-27 17:40:08 +01:00
Brendan Abolivier
5af572ada0 Synapse 1.14.0rc2 (2020-05-27)
==============================
 
 Bugfixes
 --------
 
 - Fix cache config to not apply cache factor to event cache. Regression in v1.14.0rc1. ([\#7578](https://github.com/matrix-org/synapse/issues/7578))
 - Fix bug where `ReplicationStreamer` was not always started when replication was enabled. Bug introduced in v1.14.0rc1. ([\#7579](https://github.com/matrix-org/synapse/issues/7579))
 - Fix specifying individual cache factors for caches with special characters in their name. Regression in v1.14.0rc1. ([\#7580](https://github.com/matrix-org/synapse/issues/7580))
 
 Improved Documentation
 ----------------------
 
 - Fix the OIDC `client_auth_method` value in the sample config. ([\#7581](https://github.com/matrix-org/synapse/issues/7581))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAl7Oh9sACgkQHgFcFF8Z
 Fs2OmQ//SlksllrJ7egG0JeprfCCz9TJt7cYJJhCprqfBVNs8lIc+UT/NX0/duGe
 g2KC4kPGEyusRmGSV43Yt/m/qqhcdM4tXq4iErgHG/xOTqNN/GVvrE3RaUvo9ydS
 9IdeIkDON5ylEe8sSigbBGUnpCS20Stch1z2edoUSQHNnMMSgmTNpFaZnCEXc+Us
 EYv+HfCeShdLXbzMioYj6B5qNiRnG+hJmz+h40Bmp/HAuZRyp4kx5EawAgMXIDfy
 DGWn3H7TksqC+7zETAlQgMwFzggwQx64Dpa9w2RFAchUCG6bi8pM2U3doVDGlu1i
 4HcltfBE+fE5Sy2tT2zz8qpaGGFwAp6K/c+4h29PwVBKSHRN4nepSqNHHb3fA1Ea
 GI8DWiHGyZWzlMmKI4x85lMFyAGvGJiO1Jo8icietJHZ3+7tr6SUlnIPwlLFdkUv
 xCKmlrYzwDO5enzoF6HuddnMLbtE304Ckr+vj5gWpBwYf3FIXpUS101YBV8zXTsB
 NJBMu2fjYEkPQ/d9YjWRZwL312Cb68Kytlp2ETiTuuyfLCA5Df2wdA6yReemyg//
 ZFfN1Z/Dc6p6MVRF3sf38jcWKX3r9ErNQ0p5q//uo4JbpnMW7FLXiSv/xonj4JTv
 VLSFy6F0YIIVTol4H9Fj7f3iq8/zsJe3kBE/Kycd4uhplmfvK2w=
 =BVTL
 -----END PGP SIGNATURE-----

Merge tag 'v1.14.0rc2' into develop

Synapse 1.14.0rc2 (2020-05-27)
==============================

Bugfixes
--------

- Fix cache config to not apply cache factor to event cache. Regression in v1.14.0rc1. ([\#7578](https://github.com/matrix-org/synapse/issues/7578))
- Fix bug where `ReplicationStreamer` was not always started when replication was enabled. Bug introduced in v1.14.0rc1. ([\#7579](https://github.com/matrix-org/synapse/issues/7579))
- Fix specifying individual cache factors for caches with special characters in their name. Regression in v1.14.0rc1. ([\#7580](https://github.com/matrix-org/synapse/issues/7580))

Improved Documentation
----------------------

- Fix the OIDC `client_auth_method` value in the sample config. ([\#7581](https://github.com/matrix-org/synapse/issues/7581))
2020-05-27 17:35:29 +02:00
Andrew Morgan
0a6e837aaa
Fix incorrect placeholder syntax in database prepartion code (#7575)
We were using `logger` syntax which isn't supported by `Exception`s.
2020-05-27 16:26:59 +01:00
Brendan Abolivier
b4109499b4 1.14.0rc2 2020-05-27 17:22:28 +02:00
Jason Robinson
4be968d05d
Fix sample config docs error (#7581)
'client_auth_method' commented out value was erronously 'client_auth_basic',
when code and docstring says it should be 'client_secret_basic'.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2020-05-27 13:52:18 +01:00
Erik Johnston
d7d8a2e7ee Fix up comments 2020-05-27 13:34:46 +01:00
Erik Johnston
4ba55559ac
Fix specifying cache factors via env vars with * in name. (#7580)
This mostly applise to `*stateGroupCache*` and co.

Broke in #6391.
2020-05-27 13:17:01 +01:00
Erik Johnston
eefc6b3a0d
Don't apply cache factor to event cache. (#7578)
This is already correctly done when we instansiate the cache, but wasn't
when it got reloaded (which always happens at least once on startup).
2020-05-27 12:04:37 +01:00
Erik Johnston
9bac5d62b3
Ensure ReplicationStreamer is always started when replication enabled. (#7579)
Fixes #7566.
2020-05-27 11:44:19 +01:00
Brendan Abolivier
98483890ee Merge branch 'develop' of github.com:matrix-org/synapse into develop 2020-05-26 20:30:41 +02:00
Patrick Cloke
ef884f6d04
Convert identity handler to async/await. (#7561) 2020-05-26 13:46:22 -04:00
Brendan Abolivier
87e417c5cb
Not full release yet, this is rc1 2020-05-26 17:20:43 +02:00
Brendan Abolivier
3b19c17247 1.14.0 2020-05-26 16:45:37 +02:00
Richard van der Hoff
edd9a7214c
Replace device_27_unique_idx bg update with a fg one (#7562)
The bg update never managed to complete, because it kept being interrupted by
transactions which want to take a lock.

Just doing it in the foreground isn't that bad, and is a good deal simpler.
2020-05-26 11:43:17 +01:00
Richard van der Hoff
04729b86f8
Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563)
Introduced in #7556
2020-05-26 11:42:22 +01:00
Richard van der Hoff
00db90f409
Fix recording of federation stream token (#7564)
A couple of changes of significance:

 * remove the `_last_ack < federation_position` condition, so that
   updates will still be correctly processed after restart

 * Correctly wire up send_federation_ack to the right class.
2020-05-26 11:41:38 +01:00
Richard van der Hoff
d14c4d6b6d
Simplify reap_monthly_active_users (#7558)
we can use `make_in_list_sql_clause` rather than doing our own half-baked
equivalent, which has the benefit of working just fine with empty lists.

(This has quite a lot of tests, so I think it's pretty safe)
2020-05-23 01:20:10 +01:00
Richard van der Hoff
f4269694ce
Optimise some references to hs.config (#7546)
These are surprisingly expensive, and we only really need to do them at startup.
2020-05-22 21:47:07 +01:00
Erik Johnston
2901f54359
Fix missing CORS headers on OPTION responses (#7560)
Broke in #7534.
2020-05-22 17:42:39 +01:00
Erik Johnston
e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Patrick Cloke
4429764c9f
Return 200 OK for all OPTIONS requests (#7534) 2020-05-22 09:30:07 -04:00
Erik Johnston
1531b214fc
Add ability to wait for replication streams (#7542)
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
Erik Johnston
06a02bc1ce
Convert sending mail to async/await. (#7557)
Mainly because sometimes the email push code raises exceptions where the
stack traces have gotten lost, which is hopefully fixed by this.
2020-05-22 13:41:11 +01:00
Patrick Cloke
66f2ebc22f
Use a non-empty RelayState for user interactive auth with SAML. (#7552) 2020-05-22 07:17:30 -04:00
Erik Johnston
710d958c64
On upgrade room only send canonical alias once. (#7547)
Instead of doing a complicated dance of deleting and moving aliases one
by one, which sends a canonical alias update into the old room for each
one, lets do it all in one go.

This also changes the function to move *all* local alias events to the new
room, however that happens later on anyway.
2020-05-22 11:41:41 +01:00
Erik Johnston
547e4dd83e
Fix exception reporting due to HTTP request errors. (#7556)
These are business as usual errors, rather than stuff we want to log at
error.
2020-05-22 11:39:20 +01:00
Ivan Shapovalov
ac481a738e
synapse.metrics: implement detailed memory usage reporting on PyPy (#7536)
PyPy's gc.get_stats() returns an object containing detailed allocator statistics
which could be beneficial to collect as metrics.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2020-05-22 11:08:41 +01:00
Richard van der Hoff
a0f99f81b3
Fix stacktrace mangling in patch_inline_callbacks (#7554)
`Failure()` is more cunning than `Failure(e)`.
2020-05-22 10:17:36 +01:00
Richard van der Hoff
d84bdfe599
mypy for synapse.http.site (#7553) 2020-05-22 10:12:17 +01:00
Richard van der Hoff
66a564c859
Fix some DETECTED VIOLATIONS in the config file (#7550)
consistency ftw
2020-05-22 10:11:50 +01:00
Brendan Abolivier
d1ae1015ec
Retry to sync out of sync device lists (#7453)
When a call to `user_device_resync` fails, we don't currently mark the remote user's device list as out of sync, nor do we retry to sync it.

https://github.com/matrix-org/synapse/pull/6776 introduced some code infrastructure to mark device lists as stale/out of sync.

This commit uses that code infrastructure to mark device lists as out of sync if processing an incoming device list update makes the device handler realise that the device list is out of sync, but we can't resync right now.

It also adds a looping call to retry all failed resync every 30s. This shouldn't cause too much spam in the logs as this commit also removes the "Failed to handle device list update for..." warning logs when catching `NotRetryingDestination`.

Fixes #7418
2020-05-21 17:41:12 +02:00
Richard van der Hoff
0bbbd10513
Stub out GET presence requests in the frontend proxy (#7545)
We don't really make any promises about returning accurate presence data when
presence is disabled, so we may as well just return a static response, rather
than making the master handle a request.
2020-05-21 14:36:46 +01:00
Richard van der Hoff
075375bbc9 add a comment 2020-05-21 13:25:41 +01:00
Erik Johnston
f6f92845f8
Fix bug in persist events when dealing with non member types. (#7548)
`_is_server_still_joined` will throw if it is given state updates with non-user ID state keys with local user leaves. This is actually rarely a problem since local leaves almost always get persisted by themselves.

(I discovered this on a branch that was otherwise broken, so I haven't seen this in the wild)
2020-05-21 13:20:10 +01:00
Patrick Cloke
9dc6f3075a
Hash passwords earlier in the password reset process (#7538)
This now matches the logic of the registration process as modified in
56db0b1365 / #7523.
2020-05-20 09:48:03 -04:00
Patrick Cloke
02919bf4d8 Merge branch 'master' into develop 2020-05-19 09:56:15 -04:00
Patrick Cloke
ac3264bf1e 1.13.0 2020-05-19 09:19:09 -04:00
Aaron Raimist
250f3eb991
Omit displayname or avatar_url if they aren't set instead of returning null (#7497)
Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec.

Fixes https://github.com/matrix-org/synapse/issues/7333

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-05-19 10:31:25 +01:00