Commit Graph

15079 Commits

Author SHA1 Message Date
reivilibre
9af2be192a
Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. (#14538) 2022-11-24 09:09:17 +00:00
Mathieu Velten
3b4e150868
Faster joins: use servers list approximation in assert_host_in_room (#14515)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-24 09:10:47 +01:00
Erik Johnston
f38d7d79c8
Add another index to device_lists_changes_in_room (#14534)
This helps avoid reading unnecessarily large amounts of data from the
table when querying with a set of room IDs.
2022-11-23 14:09:00 +00:00
Eric Eastwood
7f78b383ca
Optimize filter_events_for_client for faster /messages - v2 (#14527)
Fix #14108
2022-11-22 21:56:28 +00:00
realtyem
df390a8e67
Refactor federation_sender and pusher configuration loading. (#14496)
To avoid duplicating the same logic for handling legacy configuration
settings.

This should help in applying similar logic to other worker types.
2022-11-22 21:33:58 +00:00
Patrick Cloke
6d47b7e325
Add a type hint for get_device_handler() and fix incorrect types. (#14055)
This was the last untyped handler from the HomeServer object. Since
it was being treated as Any (and thus unchecked) it was being used
incorrectly in a few places.
2022-11-22 14:08:04 -05:00
Sean Quah
9cae44f49e
Track unconverted device list outbound pokes using a position instead (#14516)
When a local device list change is added to
`device_lists_changes_in_room`, the `converted_to_destinations` flag is
set to `FALSE` and the `_handle_new_device_update_async` background
process is started. This background process looks for unconverted rows
in `device_lists_changes_in_room`, copies them to
`device_lists_outbound_pokes` and updates the flag.

To update the `converted_to_destinations` flag, the database performs a
`DELETE` and `INSERT` internally, which fragments the table. To avoid
this, track unconverted rows using a `(stream ID, room ID)` position
instead of the flag.

From now on, the `converted_to_destinations` column indicates rows that
need converting to outbound pokes, but does not indicate whether the
conversion has already taken place.

Closes #14037.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-22 16:46:52 +00:00
Patrick Cloke
7eb7460042
Parallelize calls to fetch bundled aggregations. (#14510)
The bundled aggregations for annotations, references, and edits
can be parallelized.
2022-11-22 09:47:32 -05:00
Patrick Cloke
6d7523ef14
Batch fetch bundled references (#14508)
Avoid an n+1 query problem and fetch the bundled aggregations for
m.reference relations in a single query instead of a query per event.

This applies similar logic for as was previously done for edits in
8b309adb43 (#11660; threads
in b65acead42 (#11752); and
annotations in 1799a54a54 (#14491).
2022-11-22 09:41:09 -05:00
Patrick Cloke
1799a54a54
Batch fetch bundled annotations (#14491)
Avoid an n+1 query problem and fetch the bundled aggregations for
m.annotation relations in a single query instead of a query per event.

This applies similar logic for as was previously done for edits in
8b309adb43 (#11660) and threads
in b65acead42 (#11752).
2022-11-22 07:26:11 -05:00
Mathieu Velten
1526ff389f
Faster joins: filter out non local events when a room doesn't have its full state (#14404)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-21 16:46:14 +01:00
David Robertson
e1b15f25f3
Fix /key/v2/server calls with URL-unsafe key IDs (#14490)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-11-18 19:56:42 +00:00
Sean Quah
78e23eea05
Reduce default third party invite rate limit to 216 invites per day (#14487)
The previous default was the same as the `rc_message` rate limit, which
defaults to 17,280 per day.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-18 18:10:01 +00:00
David Robertson
01a0527892
Fix version that worker_main_http_uri is redundant from (#14476)
* Fix version that `worker_main_http_uri` is redundant from

* Changelog
2022-11-17 16:11:08 +00:00
Andrew Morgan
e7132c3f81
Fix check to ignore blank lines in incoming TCP replication (#14449) 2022-11-17 16:09:56 +00:00
Mathieu Velten
75888c2b1f
Faster joins: do not wait for full state when creating events to send (#14403)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-17 17:01:14 +01:00
David Robertson
115f0eb233
Reintroduce #14376, with bugfix for monoliths (#14468)
* Add tests for StreamIdGenerator

* Drive-by: annotate all defs

* Revert "Revert "Remove slaved id tracker (#14376)" (#14463)"

This reverts commit d63814fd73, which in
turn reverted 36097e88c4. This restores
the latter.

* Fix StreamIdGenerator not handling unpersisted IDs

Spotted by @erikjohnston.

Closes #14456.

* Changelog

Co-authored-by: Nick Mills-Barrett <nick@fizzadar.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-11-16 22:16:46 +00:00
realtyem
c15e9a0edb
Remove need for worker_main_http_uri setting to use /keys/upload. (#14400) 2022-11-16 22:16:25 +00:00
Andrew Morgan
618e4ab81b
Fix an invalid comparison of UserPresenceState to str (#14393) 2022-11-16 15:25:35 +00:00
Patrick Cloke
d8cc86eff4
Remove redundant types from comments. (#14412)
Remove type hints from comments which have been added
as Python type hints. This helps avoid drift between comments
and reality, as well as removing redundant information.

Also adds some missing type hints which were simple to fill in.
2022-11-16 15:25:24 +00:00
Sean Quah
882277008c
Fix background updates failing to add unique indexes on receipts (#14453)
As part of the database migration to support threaded receipts, there is
a possible window in between
`73/08thread_receipts_non_null.sql.postgres` removing the original
unique constraints on `receipts_linearized` and `receipts_graph` and the
`reeipts_linearized_unique_index` and `receipts_graph_unique_index`
background updates from `72/08thread_receipts.sql` completing where
the unique constraints on `receipts_linearized` and `receipts_graph` are
missing. Any emulated upserts on these tables must therefore be
performed with a lock held, otherwise duplicate rows can end up in the
tables when there are concurrent emulated upserts. Fix the missing lock.

Note that emulated upserts no longer happen by default on sqlite, since
the minimum supported version of sqlite supports native upserts by
default now.

Finally, clean up any duplicate receipts that may have crept in before
trying to create the `receipts_graph_unique_index` and
`receipts_linearized_unique_index` unique indexes.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-16 15:01:22 +00:00
Erik Johnston
d63814fd73
Revert "Remove slaved id tracker (#14376)" (#14463)
This reverts commit 36097e88c4.
2022-11-16 13:50:07 +00:00
Erik Johnston
945a0928c7
Don't filter state in /context response (#14461)
We don't filter state usually, so doing so here is a waste of time. This is not much of an issue for clients that enable lazy loading of members, since there will be fewer state events.
2022-11-16 12:09:33 +00:00
Erik Johnston
5cb6ad3b87
Fix HTML templates missing correct HTML tags (#14448) 2022-11-16 11:14:38 +00:00
David Robertson
1eed795fc5
Include heroes in partial join responses' state (#14442)
* Pull out hero selection logic

* Include heroes in partial join response's state

* Changelog

* Fixup trial test

* Remove TODO
2022-11-15 17:35:19 +00:00
David Robertson
258b5285b6
Fix typechecking errors introduced in #14128 (#14455)
* Fix typechecking errors introduced in #14128

* Changelog

* Correct annotations

so that context_factory works if you don't use TLS
2022-11-15 16:36:43 +00:00
DeepBlueV7.X
63cc56affa
Send content rules with pattern_type to clients (#14356) 2022-11-15 15:29:30 +00:00
Tuomas Ojamies
b5ab2c428a
Support using SSL on worker endpoints. (#14128)
* Fix missing SSL support in worker endpoints.

* Add changelog

* SSL for Replication endpoint

* Remove unit test change

* Refactor listener creation to reduce duplicated code

* Fix the logger message

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add config documentation for new TLS option

Co-authored-by: Tuomas Ojamies <tojamies@palantir.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-11-15 12:55:00 +00:00
reivilibre
634359b083
Update docstring to clarify that get_partial_state_events_batch does not just give you completely arbitrary partial-state events. (#14417) 2022-11-15 10:43:17 +00:00
Nick Mills-Barrett
36097e88c4
Remove slaved id tracker (#14376)
This matches the multi instance writer ID generator class which can
both handle advancing the current token over replication and by calling
the database.
2022-11-14 17:31:36 +00:00
David Robertson
2cc592584a
Remove unused type-ignores (#14433)
* Remove unused type-ignores

Oversights in #14427 and #14429.

* Changelog
2022-11-14 13:46:29 +00:00
Patrick Cloke
fb66fae84b
Clean-up events persistance code (#14411)
By removing unused variables and making some arguments
required which are always provided.
2022-11-14 08:13:11 -05:00
Ashish Kumar
a3623af74e
Add an Admin API endpoint for looking up users based on 3PID (#14405) 2022-11-11 15:38:17 +00:00
Nick Mills-Barrett
3a4f80f8c6
Merge/remove Slaved* stores into WorkerStores (#14375) 2022-11-11 10:51:49 +00:00
Patrick Cloke
13ca8bb2fc
Remove duplicated code to evict entries. (#14410)
This code was factored out to a method, but also left in-place.

Calling this twice in a row makes no sense: the first call will reduce
the size appropriately, but the loop will immediately exit since the
cache size was already reduced.
2022-11-10 15:33:34 -05:00
Sean Quah
b2c2b03079
Fix PostgreSQL sometimes using table scans for event_search (#14409)
PostgreSQL may underestimate the number of distinct `room_id`s in
`event_search`, which can cause it to use table scans for queries for
multiple rooms.

Fix this by setting `n_distinct` on the column.

Resolves #14402.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-10 19:02:27 +00:00
David Robertson
d10a85ec9e
Quieter logging for stateres failure at missing prev events (#14346) 2022-11-10 12:17:46 +00:00
Patrick Cloke
e9a4343cb2
Drop support for Postgres 10 in full text search code. (#14397) 2022-11-09 09:55:34 -05:00
Sean Quah
a5fcdea090
Remove support for PostgreSQL 10 (#14392)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-08 17:17:13 +00:00
Shay
7894251bce
Correctly create power level event during initial room creation (#14361) 2022-11-07 13:38:50 -08:00
Richard van der Hoff
2193513346
Fix background update table-scanning events (#14374)
When this background update did its last batch, it would try to update all the
events that had been inserted since the bgupdate started, which could cause a
table-scan. Make sure we limit the update correctly.
2022-11-07 14:28:00 +00:00
Sean Quah
e980982b59
Do not reject /sync requests with unrecognised filter fields (#14369)
For forward compatibility, Synapse needs to ignore fields it does not
recognise instead of raising an error.

Fixes #14365.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-07 13:49:31 +00:00
dependabot[bot]
8bcdd712b8
Bump flake8-bugbear from 22.9.23 to 22.10.27 (#14329)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-11-04 18:43:14 +00:00
Tulir Asokan
a4b1f64562
Fix /refresh endpoint version (#14364) 2022-11-04 16:43:51 +00:00
Sean Quah
e5d18956b9 Synapse 1.71.0rc2 (2022-11-04)
==============================
 
 Please note that, as announced in the release notes for Synapse 1.69.0, legacy Prometheus metric names are now disabled by default.
 They will be removed altogether in Synapse 1.73.0.
 If not already done, server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names.
 See the [upgrade notes](https://matrix-org.github.io/synapse/v1.71/upgrade.html#upgrading-to-v1710) for more details.
 
 Improved Documentation
 ----------------------
 
 - Document the changes to monthly active user metrics due to deprecation of legacy Prometheus metric names. ([\#14358](https://github.com/matrix-org/synapse/issues/14358), [\#14360](https://github.com/matrix-org/synapse/issues/14360))
 
 Deprecations and Removals
 -------------------------
 
 - Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. ([\#14353](https://github.com/matrix-org/synapse/issues/14353))
 
 Internal Changes
 ----------------
 
 - Run unit tests against Python 3.11. ([\#13812](https://github.com/matrix-org/synapse/issues/13812))
 -----BEGIN PGP SIGNATURE-----
 
 iQGzBAABCgAdFiEEWMTnW8Z8khaaf90R+84KzgcyGG8FAmNlD5YACgkQ+84Kzgcy
 GG8T0Qv+PkTt9+lUbDAsqWnqWtSYCit4DEZU/g0G+/xUVdZypo+pIHYzzEHEEPoB
 Siinh5sLqUagoOn0i+YU+D6WZw+20F5+qdQi6QSkpPDorReYBl+4MvyRNqjn3lAE
 a/bfyHY3+88SfVMza+GRQS7pU06CCMD5aqUZABnYwV5fpKWdF2hOESSuzqEElQDR
 7zaZuc7I/8d/VNz79FmyDfGzYnvWS+tr0BcNIyye6OIzZ2AS+q8loBzQiJ5rd1kY
 ZiGeAHfeu/r05X25kzD1E1qr04ycM7mFjDgBa/aSdf5kkimNNUISo424sawoCVOU
 oDm+QVefH4zKK9b+DVAJ7eaHIaQsRjJir+h/umXrX09I/x/eMB7Zll1wy1T+KzUS
 Zf4GCVCI1kKY3FzcPWh2Jf1j899JtbkrKasA+UYf3+PEeFxQ4LyjYlVHC+J1sCXL
 NGn5YDZkvhfpTOR300eDisUsxKIBFPxL6FjzW5whOnBddKC+VVnlSQkAbn6yHQGE
 LU1xKhta
 =m6ze
 -----END PGP SIGNATURE-----

Merge tag 'v1.71.0rc2' into develop

Synapse 1.71.0rc2 (2022-11-04)
==============================

Please note that, as announced in the release notes for Synapse 1.69.0, legacy Prometheus metric names are now disabled by default.
They will be removed altogether in Synapse 1.73.0.
If not already done, server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.71/upgrade.html#upgrading-to-v1710) for more details.

Improved Documentation
----------------------

- Document the changes to monthly active user metrics due to deprecation of legacy Prometheus metric names. ([\#14358](https://github.com/matrix-org/synapse/issues/14358), [\#14360](https://github.com/matrix-org/synapse/issues/14360))

Deprecations and Removals
-------------------------

- Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. ([\#14353](https://github.com/matrix-org/synapse/issues/14353))

Internal Changes
----------------

- Run unit tests against Python 3.11. ([\#13812](https://github.com/matrix-org/synapse/issues/13812))
2022-11-04 15:22:06 +00:00
Brendan Abolivier
86c5a710d8
Implement MSC3912: Relation-based redactions (#14260)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-03 16:21:31 +00:00
reivilibre
6546308c1e
Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. (#14353) 2022-11-02 17:33:45 +00:00
David Robertson
d4fac8a3e2
Fix typo in #13320 which could cause log spam (#14347) 2022-11-01 19:20:35 +00:00
David Robertson
2bd7f3eeab
Allow PUT/GET of aliases during faster join (#14292)
without blocking on full state.
2022-11-01 15:02:39 +00:00
David Robertson
9473ebb9e7
Revert "Fix event size checks (#13710)"
This reverts commit fab495a9e1.

As noted in
https://github.com/matrix-org/synapse/pull/13710#issuecomment-1298396007:

> We want to see this change land for the protocol's sake (and plan to
  un-revert it) but want to give this a little more time before releasing
  this.
2022-11-01 11:47:09 +00:00