Commit Graph

12057 Commits

Author SHA1 Message Date
Patrick Cloke
71cccf1593
Additional configuration options for auto-join rooms () 2020-06-30 15:41:36 -04:00
Erik Johnston
a99658074d
Add some metrics for inbound and outbound federation processing times () 2020-06-30 16:58:06 +01:00
Andrew Morgan
2f6afdd8b4
Explain the purpose of the "tests" conditional dependency requirement () 2020-06-30 10:11:36 +01:00
Erik Johnston
831b31e563
Add another yield point to state res v2 () 2020-06-26 10:44:52 +01:00
Erik Johnston
0e0a2817a2
Yield during large v2 state res. ()
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.

To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
2020-06-24 18:48:18 +01:00
Sorunome
6920e58136
add org.matrix.login.jwt so that m.login.jwt can be deprecated () 2020-06-24 10:23:55 +01:00
Christian Svensson
8bbe87f42d
Set Content-Length for Metrics requests ()
HTTP requires the response to contain a Content-Length header unless chunked encoding is being used.
Prometheus metrics endpoint did not set this, causing software such as prometheus-proxy to not be able to scrape synapse for metrics.

Signed-off-by: Christian Svensson <blue@cmd.nu>
2020-06-23 18:06:01 +01:00
Patrick Cloke
95e41f368b
Allow local media to be marked as safe from being quarantined. () 2020-06-22 08:04:14 -04:00
Patrick Cloke
e060bf4462
Convert directory handler to async/await () 2020-06-22 07:18:00 -04:00
Erik Johnston
91e886d615
Speed up state res v2 across large state differences. () 2020-06-19 13:56:35 +01:00
Richard van der Hoff
7d2824395f add a comment 2020-06-18 10:47:06 +01:00
Brendan Abolivier
e35d44c01d
Merge pull request from matrix-org/babolivier/unread_fix
Fix unread counts in sync
2020-06-17 15:44:15 +01:00
Patrick Cloke
3630825612
Convert the typing handler to async/await. () 2020-06-17 10:37:59 -04:00
Oleg Girko
96bc110a68
Require parameterized package version to be at least 0.7.0. ()
Older versions of `parameterized` package have no `parameterized_class` decorator. This decorator is used in tests.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2020-06-17 15:31:40 +01:00
Brendan Abolivier
5a5cf6460e
Fix unread counts in sync
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
2020-06-17 15:10:44 +01:00
Erik Johnston
e07a8caf58
Add support for using rust-python-jaeger-reporter () 2020-06-17 14:13:41 +01:00
Erik Johnston
b44bdd7f7b
Support running multiple media repos. ()
This requires a new config option to specify which media repo should be
responsible for running background jobs to e.g. clear out expired URL
preview caches.
2020-06-17 14:13:30 +01:00
Patrick Cloke
434716e1d3
Fetch from the r0 media path instead of the unspecced v1. () 2020-06-17 08:36:46 -04:00
Brendan Abolivier
46613aaf79
Implement unread counter (MSC2625) ()
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
Richard van der Hoff
e452973fd2
fix broken link in sample config () 2020-06-16 19:50:16 +01:00
Erik Johnston
f6f7511a4c
Refactor getting replication updates from database. ()
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
2020-06-16 17:10:28 +01:00
Patrick Cloke
231252516c
Fix "argument of type 'ObservableDeferred' is not iterable" error () 2020-06-16 12:01:18 -04:00
Patrick Cloke
ac51bd581a
Include a user agent in federation requests. () 2020-06-16 10:43:29 -04:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents () 2020-06-16 08:51:47 -04:00
Patrick Cloke
98c4e35e3c
Convert the device message and pagination handlers to async/await. () 2020-06-16 08:06:17 -04:00
Richard van der Hoff
03619324fc
Create a ListenerConfig object ()
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Brendan Abolivier
789606577a Merge branch 'master' into develop 2020-06-16 10:36:29 +01:00
Brendan Abolivier
65eb078498
1.15.1 2020-06-16 10:28:58 +01:00
Andrew Morgan
3e6b5bba71
Wrap register_device coroutine in an ensureDeferred ()
Fixes https://github.com/matrix-org/synapse/issues/7683

Broke in: 

We had a `yield` acting on a coroutine. To be fair this one is a bit difficult to notice as there's a function in the middle that just passes the coroutine along.
2020-06-16 10:13:59 +01:00
Patrick Cloke
cc32fa7358
Ensure the body is a string before comparing push rules. () 2020-06-15 16:20:34 -04:00
Hubert Chathi
2b2344652b
Ensure etag is a string for GET room_keys/version response () 2020-06-15 13:42:44 -04:00
Andrew Morgan
b8ee03caff
Update m.id.phone to use 'phone' instead of 'number' ()
The spec [states](https://matrix.org/docs/spec/client_server/r0.6.1#phone-number) that `m.id.phone` requires the field `country` and `phone`.

In Synapse, we've been enforcing `country` and `number`.

I am not currently sure whether this affects any client implementations.

This issue was introduced in .
2020-06-15 17:46:23 +01:00
Erik Johnston
4241a10673
Fix "There was no active span when trying to log." error () 2020-06-15 16:42:21 +01:00
Brendan Abolivier
6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Brendan Abolivier
c2b4621630 Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:08 +01:00
Brendan Abolivier
6d5985e1f2
Enable 3PID add/bind/unbind endpoints on r0 routes 2020-06-15 16:14:13 +01:00
Patrick Cloke
7d2532be36
Discard RDATA from already seen positions. () 2020-06-15 08:44:54 -04:00
Patrick Cloke
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. () 2020-06-15 07:03:36 -04:00
Brendan Abolivier
fed493c5fd
Incorporate review 2020-06-15 09:58:55 +01:00
Patrick Cloke
2d11ea385c
Fix warnings about losing log context during UI auth. () 2020-06-12 15:01:00 -04:00
Patrick Cloke
d0a43d431e
Fix a typo when comparing the URI & method during UI Auth. () 2020-06-12 14:12:04 -04:00
Brendan Abolivier
e186c660b1
Lint 2020-06-12 15:31:59 +01:00
Brendan Abolivier
e47e5a2dcd
Incorporate review bits 2020-06-12 15:13:12 +01:00
Brendan Abolivier
1e5a50302f
Pre-populate the unread_count column 2020-06-12 15:05:47 +01:00
Brendan Abolivier
9549d557ea
Don't update the schema version 2020-06-12 15:03:26 +01:00
Brendan Abolivier
cf92fbb8aa
Use attr instead of a dict 2020-06-12 15:02:15 +01:00
Brendan Abolivier
3cc7f43e8d
Fix summary rotation 2020-06-12 11:07:26 +01:00
Will Hunt
a3fbc23c39
Remove "user_id" from GET /presence. () 2020-06-11 14:13:53 -04:00
Brendan Abolivier
cb6d4d07b1
Log for invalid values of notif 2020-06-11 18:30:31 +01:00
Brendan Abolivier
803291728c
Fix SQL 2020-06-11 18:25:25 +01:00
Brendan Abolivier
34fd1f7ab5
Fix schema update 2020-06-11 18:12:12 +01:00
Brendan Abolivier
d0f095625c
Lint 2020-06-11 18:04:43 +01:00
Brendan Abolivier
ce74a6685d
Save the count of unread messages to event_push_summary 2020-06-11 17:58:26 +01:00
Brendan Abolivier
ea8f6e611b
Actually act on mark_unread 2020-06-11 15:30:42 +01:00
Brendan Abolivier
1ad06ee6eb Merge branch 'master' into develop 2020-06-11 13:31:22 +01:00
Brendan Abolivier
3b3f327a0d 1.15.0 2020-06-11 13:27:27 +01:00
Patrick Cloke
b9df7f70bb
Increase the default SAML session expirary time to 15 minutes. () 2020-06-11 07:55:45 -04:00
wondratsch
c746889bb0
fix typo in sample_config.yaml ()
Just a simple typo fix.

Signed-off-by: wondratsch 28294257+wondratsch@users.noreply.github.com
2020-06-11 11:51:10 +01:00
Brendan Abolivier
9dbd006607
Appease mypy 2020-06-10 20:44:24 +01:00
Brendan Abolivier
243f0ba6ce
Lint 2020-06-10 20:35:35 +01:00
Brendan Abolivier
df3323a7cf
Use temporary prefixes as per the MSC 2020-06-10 20:32:01 +01:00
Richard van der Hoff
0df618f813
Take out a lock before modifying _CACHES ()
This should fix .
2020-06-10 18:27:49 +01:00
Brendan Abolivier
c7b99a1180
Use a more efficient way of calculating counters 2020-06-10 17:54:33 +01:00
Andrew Morgan
fcd6961441
Add option to enable encryption by default for new rooms ()
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on 
2020-06-10 17:44:34 +01:00
Brendan Abolivier
ef345c5a7b
Add a new unread_counter to sync responses 2020-06-10 16:21:16 +01:00
Patrick Cloke
191dc98f80
Clean-up the fallback login code. () 2020-06-10 09:50:39 -04:00
Brendan Abolivier
6f6a4bfc07
Rename dont_push into mark_unread 2020-06-10 14:24:01 +01:00
Brendan Abolivier
ec0a7b9034 Merge branch 'develop' into babolivier/mark_unread 2020-06-10 11:42:30 +01:00
Brendan Abolivier
236d2d699d 1.15.0rc1 2020-06-09 16:37:14 +01:00
Brendan Abolivier
2dc9468c27 Revert "1.15.0rc1"
This reverts commit 8587b0426f.
2020-06-09 16:34:37 +01:00
Brendan Abolivier
8587b0426f 1.15.0rc1 2020-06-09 16:33:36 +01:00
Erik Johnston
664409b169
Fix bug in account data replication stream. ()
* Ensure account data stream IDs are unique.

The account data stream is shared between three tables, and the maximum
allocated ID was tracked in a dedicated table. Updating the max ID
happened outside the transaction that allocated the ID, leading to a
race where if the server was restarted then the same ID could be
allocated but the max ID failed to be updated, leading it to be reused.

The ID generators have support for tracking across multiple tables, so
we may as well use that instead of a dedicated table.

* Fix bug in account data replication stream.

If the same stream ID was used in both global and room account data then
the getting updates for the replication stream would fail due to
`heapq.merge(..)` trying to compare a `str` with a `None`. (This is
because you'd have two rows like `(534, '!room')` and `(534, None)` from
the room and global account data tables).

Fix is just to order by stream ID, since we don't rely on the ordering
beyond that. The bug where stream IDs can be reused should be fixed now,
so this case shouldn't happen going forward.

Fixes 
2020-06-09 16:28:57 +01:00
Patrick Cloke
3c45a78090
Convert the registration handler to async/await. () 2020-06-08 11:15:02 -04:00
Patrick Cloke
375ca0cceb
Accept device information at the login fallback endpoint. () 2020-06-08 10:13:24 -04:00
Patrick Cloke
737b4a936e
Convert user directory handler and related classes to async/await. () 2020-06-05 14:42:55 -04:00
Travis Ralston
09099313e6
Add an option to disable autojoin for guest accounts ()
Fixes https://github.com/matrix-org/synapse/issues/3177
2020-06-05 18:18:15 +01:00
Patrick Cloke
a0d2d81cf9
Update to the stable SSO prefix for UI Auth. () 2020-06-05 10:50:08 -04:00
Richard van der Hoff
eea124370b
Fix type information on assert_*_is_admin methods ()
These things don't return Deferreds.
2020-06-05 14:33:49 +01:00
Richard van der Hoff
b4f8dcb4bd
Remove some unused constants. () 2020-06-05 14:33:35 +01:00
Patrick Cloke
f1e61ef85c Typo fixes. 2020-06-05 08:43:21 -04:00
Dirk Klimpel
908f9e2d24
Allow new users to be registered via the admin API even if the monthly active user limit has been reached () 2020-06-05 13:08:49 +01:00
Dirk Klimpel
2970ce8367
Add device management to admin API ()
- Admin is able to
  - change displaynames
  - delete devices
  - list devices
  - get device informations

Fixes 
2020-06-05 13:07:22 +01:00
Patrick Cloke
02f345d053
Attempt to fix PhoneHomeStatsTestCase.test_performance_100 being flaky. () 2020-06-05 07:36:47 -04:00
Andrew Morgan
139bc86f3d
Support CS API v0.6.0 () 2020-06-05 12:27:37 +01:00
WGH
e55ee7c32f
Add support for webp thumbnailing ()
Closes 

Signed-off-by: Maxim Plotnikov <wgh@torlan.ru>
2020-06-05 11:54:27 +01:00
Andrew Morgan
f4e6495b5d
Performance improvements and refactor of Ratelimiter ()
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:

* Rather undocumented, and
* causing a *lot* of config checks

This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. 

Best to be reviewed commit-by-commit.
2020-06-05 10:47:20 +01:00
Andrew Morgan
c389bfb6ea
Fix encryption algorithm typos in tests/comments ()
@uhoreg has confirmed these were both typos. They are only in comments and tests though, rather than anything critical.

Introduced in:

* https://github.com/matrix-org/synapse/pull/7157
* https://github.com/matrix-org/synapse/pull/5726
2020-06-04 20:03:40 +01:00
Patrick Cloke
f8b9ead3ee
Advertise the token login type when OpenID Connect is enabled. () 2020-06-04 06:49:51 -04:00
Richard van der Hoff
11de843626
Cleanups to the OpenID Connect integration ()
docs, default configs, comments. Nothing very significant.
2020-06-03 21:13:17 +01:00
Andrew Morgan
e91abfd291
async/await get_user_id_by_threepid ()
Based on  

async's `get_user_id_by_threepid` and its call stack.
2020-06-03 17:15:57 +01:00
Brendan Abolivier
c9507be989
Check if the localpart is reserved for guests earlier in the registration flow ()
This is so the user is warned about the username not being valid as soon as possible, rather than only once they've finished UIA.
2020-06-03 16:55:02 +02:00
Erik Johnston
11dc2b4698
Fix exceptions when fetching events from a down host. ()
We already caught some exceptions, but not all.
2020-06-03 14:12:13 +01:00
Richard van der Hoff
1bbc9e2df6
Clean up exception handling in SAML2ResponseResource ()
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string

* Clean up exception handling in SAML2ResponseResource

  * use the existing code in `return_html_error` instead of re-implementing it
    (giving it a jinja2 template rather than inventing a new form of template)

  * do the exception-catching in the REST layer rather than in the handler
    layer, to make sure we catch all exceptions.
2020-06-03 10:41:12 +01:00
Olof Johansson
fe434cd3c9
Fix a bug in automatic user creation with m.login.jwt. () 2020-06-01 12:55:07 -04:00
Brendan Abolivier
33c39ab93c
Process cross-signing keys when resyncing device lists ()
It looks like `user_device_resync` was ignoring cross-signing keys from the results received from the remote server. This patch fixes this, by processing these keys using the same process `_handle_signing_key_updates` does (and effectively factor that part out of that function).
2020-06-01 17:47:30 +02:00
Dirk Klimpel
901b1fa561
Email notifications for new users when creating via the Admin API. () 2020-06-01 15:34:33 +01:00
Dagfinn Ilmari Mannsåker
df8a3cef6b
Improve performance of _get_state_groups_from_groups_txn ()
The query keeps showing up in my slow query log.

This changes the plan under the top-level Sort node from

```
    WindowAgg  (cost=280335.88..292963.15 rows=561212 width=80) (actual time=138.651..160.562 rows=27112 loops=1)
      ->  Sort  (cost=280335.88..281738.91 rows=561212 width=84) (actual time=138.597..140.622 rows=27112 loops=1)
            Sort Key: state_groups_state.type, state_groups_state.state_key, state_groups_state.state_group
            Sort Method: quicksort  Memory: 4581kB
            ->  Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=21.548..47.657 rows=27112 loops=1)
                  ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=21.526..21.535 rows=20 loops=1)
                        Group Key: state.state_group
                        ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=21.280..21.493 rows=20 loops=1)
                  ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.005..0.991 rows=1356 loops=20)
                        Index Cond: (state_group = state.state_group)
```

to

```
    Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=24.194..52.834 rows=27112 loops=1)
      ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=24.130..24.138 rows=20 loops=1)
            Group Key: state.state_group
            ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=23.887..24.113 rows=20 loops=1)
      ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.016..1.159 rows=1356 loops=20)
            Index Cond: (state_group = state.state_group)
```

This cuts the execution time from ~190ms to ~130ms, i.e. a reduction
of ~30%.

The full plans are visualised at https://explain.depesz.com/s/WpbT and
https://explain.depesz.com/s/KlEk

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-06-01 15:23:43 +01:00
Patrick Cloke
6af9cdca24
Convert groups local and server to async/await. () 2020-06-01 07:28:43 -04:00
Brendan Abolivier
c1bdd4fac7
Don't fail all of an iteration of the device list retry loop on error ()
Without this patch, if an error happens which isn't caught by `user_device_resync`, then `_maybe_retry_device_resync` would fail, without retrying the next users in the iteration. This patch fixes this so that it now only logs an error in this case.
2020-06-01 12:55:14 +02:00
Dagfinn Ilmari Mannsåker
2dc430d36e
Use upsert when inserting read receipts ()
Fixes 

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-06-01 10:53:06 +01:00