Commit Graph

15059 Commits

Author SHA1 Message Date
Sean Quah
882277008c
Fix background updates failing to add unique indexes on receipts (#14453)
As part of the database migration to support threaded receipts, there is
a possible window in between
`73/08thread_receipts_non_null.sql.postgres` removing the original
unique constraints on `receipts_linearized` and `receipts_graph` and the
`reeipts_linearized_unique_index` and `receipts_graph_unique_index`
background updates from `72/08thread_receipts.sql` completing where
the unique constraints on `receipts_linearized` and `receipts_graph` are
missing. Any emulated upserts on these tables must therefore be
performed with a lock held, otherwise duplicate rows can end up in the
tables when there are concurrent emulated upserts. Fix the missing lock.

Note that emulated upserts no longer happen by default on sqlite, since
the minimum supported version of sqlite supports native upserts by
default now.

Finally, clean up any duplicate receipts that may have crept in before
trying to create the `receipts_graph_unique_index` and
`receipts_linearized_unique_index` unique indexes.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-16 15:01:22 +00:00
Erik Johnston
d63814fd73
Revert "Remove slaved id tracker (#14376)" (#14463)
This reverts commit 36097e88c4.
2022-11-16 13:50:07 +00:00
Erik Johnston
945a0928c7
Don't filter state in /context response (#14461)
We don't filter state usually, so doing so here is a waste of time. This is not much of an issue for clients that enable lazy loading of members, since there will be fewer state events.
2022-11-16 12:09:33 +00:00
Erik Johnston
5cb6ad3b87
Fix HTML templates missing correct HTML tags (#14448) 2022-11-16 11:14:38 +00:00
David Robertson
1eed795fc5
Include heroes in partial join responses' state (#14442)
* Pull out hero selection logic

* Include heroes in partial join response's state

* Changelog

* Fixup trial test

* Remove TODO
2022-11-15 17:35:19 +00:00
David Robertson
258b5285b6
Fix typechecking errors introduced in #14128 (#14455)
* Fix typechecking errors introduced in #14128

* Changelog

* Correct annotations

so that context_factory works if you don't use TLS
2022-11-15 16:36:43 +00:00
DeepBlueV7.X
63cc56affa
Send content rules with pattern_type to clients (#14356) 2022-11-15 15:29:30 +00:00
Tuomas Ojamies
b5ab2c428a
Support using SSL on worker endpoints. (#14128)
* Fix missing SSL support in worker endpoints.

* Add changelog

* SSL for Replication endpoint

* Remove unit test change

* Refactor listener creation to reduce duplicated code

* Fix the logger message

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add config documentation for new TLS option

Co-authored-by: Tuomas Ojamies <tojamies@palantir.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-11-15 12:55:00 +00:00
reivilibre
634359b083
Update docstring to clarify that get_partial_state_events_batch does not just give you completely arbitrary partial-state events. (#14417) 2022-11-15 10:43:17 +00:00
Nick Mills-Barrett
36097e88c4
Remove slaved id tracker (#14376)
This matches the multi instance writer ID generator class which can
both handle advancing the current token over replication and by calling
the database.
2022-11-14 17:31:36 +00:00
David Robertson
2cc592584a
Remove unused type-ignores (#14433)
* Remove unused type-ignores

Oversights in #14427 and #14429.

* Changelog
2022-11-14 13:46:29 +00:00
Patrick Cloke
fb66fae84b
Clean-up events persistance code (#14411)
By removing unused variables and making some arguments
required which are always provided.
2022-11-14 08:13:11 -05:00
Ashish Kumar
a3623af74e
Add an Admin API endpoint for looking up users based on 3PID (#14405) 2022-11-11 15:38:17 +00:00
Nick Mills-Barrett
3a4f80f8c6
Merge/remove Slaved* stores into WorkerStores (#14375) 2022-11-11 10:51:49 +00:00
Patrick Cloke
13ca8bb2fc
Remove duplicated code to evict entries. (#14410)
This code was factored out to a method, but also left in-place.

Calling this twice in a row makes no sense: the first call will reduce
the size appropriately, but the loop will immediately exit since the
cache size was already reduced.
2022-11-10 15:33:34 -05:00
Sean Quah
b2c2b03079
Fix PostgreSQL sometimes using table scans for event_search (#14409)
PostgreSQL may underestimate the number of distinct `room_id`s in
`event_search`, which can cause it to use table scans for queries for
multiple rooms.

Fix this by setting `n_distinct` on the column.

Resolves #14402.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-10 19:02:27 +00:00
David Robertson
d10a85ec9e
Quieter logging for stateres failure at missing prev events (#14346) 2022-11-10 12:17:46 +00:00
Patrick Cloke
e9a4343cb2
Drop support for Postgres 10 in full text search code. (#14397) 2022-11-09 09:55:34 -05:00
Sean Quah
a5fcdea090
Remove support for PostgreSQL 10 (#14392)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-08 17:17:13 +00:00
Shay
7894251bce
Correctly create power level event during initial room creation (#14361) 2022-11-07 13:38:50 -08:00
Richard van der Hoff
2193513346
Fix background update table-scanning events (#14374)
When this background update did its last batch, it would try to update all the
events that had been inserted since the bgupdate started, which could cause a
table-scan. Make sure we limit the update correctly.
2022-11-07 14:28:00 +00:00
Sean Quah
e980982b59
Do not reject /sync requests with unrecognised filter fields (#14369)
For forward compatibility, Synapse needs to ignore fields it does not
recognise instead of raising an error.

Fixes #14365.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-07 13:49:31 +00:00
dependabot[bot]
8bcdd712b8
Bump flake8-bugbear from 22.9.23 to 22.10.27 (#14329)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-11-04 18:43:14 +00:00
Tulir Asokan
a4b1f64562
Fix /refresh endpoint version (#14364) 2022-11-04 16:43:51 +00:00
Sean Quah
e5d18956b9 Synapse 1.71.0rc2 (2022-11-04)
==============================
 
 Please note that, as announced in the release notes for Synapse 1.69.0, legacy Prometheus metric names are now disabled by default.
 They will be removed altogether in Synapse 1.73.0.
 If not already done, server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names.
 See the [upgrade notes](https://matrix-org.github.io/synapse/v1.71/upgrade.html#upgrading-to-v1710) for more details.
 
 Improved Documentation
 ----------------------
 
 - Document the changes to monthly active user metrics due to deprecation of legacy Prometheus metric names. ([\#14358](https://github.com/matrix-org/synapse/issues/14358), [\#14360](https://github.com/matrix-org/synapse/issues/14360))
 
 Deprecations and Removals
 -------------------------
 
 - Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. ([\#14353](https://github.com/matrix-org/synapse/issues/14353))
 
 Internal Changes
 ----------------
 
 - Run unit tests against Python 3.11. ([\#13812](https://github.com/matrix-org/synapse/issues/13812))
 -----BEGIN PGP SIGNATURE-----
 
 iQGzBAABCgAdFiEEWMTnW8Z8khaaf90R+84KzgcyGG8FAmNlD5YACgkQ+84Kzgcy
 GG8T0Qv+PkTt9+lUbDAsqWnqWtSYCit4DEZU/g0G+/xUVdZypo+pIHYzzEHEEPoB
 Siinh5sLqUagoOn0i+YU+D6WZw+20F5+qdQi6QSkpPDorReYBl+4MvyRNqjn3lAE
 a/bfyHY3+88SfVMza+GRQS7pU06CCMD5aqUZABnYwV5fpKWdF2hOESSuzqEElQDR
 7zaZuc7I/8d/VNz79FmyDfGzYnvWS+tr0BcNIyye6OIzZ2AS+q8loBzQiJ5rd1kY
 ZiGeAHfeu/r05X25kzD1E1qr04ycM7mFjDgBa/aSdf5kkimNNUISo424sawoCVOU
 oDm+QVefH4zKK9b+DVAJ7eaHIaQsRjJir+h/umXrX09I/x/eMB7Zll1wy1T+KzUS
 Zf4GCVCI1kKY3FzcPWh2Jf1j899JtbkrKasA+UYf3+PEeFxQ4LyjYlVHC+J1sCXL
 NGn5YDZkvhfpTOR300eDisUsxKIBFPxL6FjzW5whOnBddKC+VVnlSQkAbn6yHQGE
 LU1xKhta
 =m6ze
 -----END PGP SIGNATURE-----

Merge tag 'v1.71.0rc2' into develop

Synapse 1.71.0rc2 (2022-11-04)
==============================

Please note that, as announced in the release notes for Synapse 1.69.0, legacy Prometheus metric names are now disabled by default.
They will be removed altogether in Synapse 1.73.0.
If not already done, server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.71/upgrade.html#upgrading-to-v1710) for more details.

Improved Documentation
----------------------

- Document the changes to monthly active user metrics due to deprecation of legacy Prometheus metric names. ([\#14358](https://github.com/matrix-org/synapse/issues/14358), [\#14360](https://github.com/matrix-org/synapse/issues/14360))

Deprecations and Removals
-------------------------

- Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. ([\#14353](https://github.com/matrix-org/synapse/issues/14353))

Internal Changes
----------------

- Run unit tests against Python 3.11. ([\#13812](https://github.com/matrix-org/synapse/issues/13812))
2022-11-04 15:22:06 +00:00
Brendan Abolivier
86c5a710d8
Implement MSC3912: Relation-based redactions (#14260)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-03 16:21:31 +00:00
reivilibre
6546308c1e
Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. (#14353) 2022-11-02 17:33:45 +00:00
David Robertson
d4fac8a3e2
Fix typo in #13320 which could cause log spam (#14347) 2022-11-01 19:20:35 +00:00
David Robertson
2bd7f3eeab
Allow PUT/GET of aliases during faster join (#14292)
without blocking on full state.
2022-11-01 15:02:39 +00:00
David Robertson
9473ebb9e7
Revert "Fix event size checks (#13710)"
This reverts commit fab495a9e1.

As noted in
https://github.com/matrix-org/synapse/pull/13710#issuecomment-1298396007:

> We want to see this change land for the protocol's sake (and plan to
  un-revert it) but want to give this a little more time before releasing
  this.
2022-11-01 11:47:09 +00:00
reivilibre
b922b54b61
Fix type annotation causing import time error in the Complement forking launcher. (#14084)
Co-authored-by: David Robertson <davidr@element.io>
2022-11-01 10:30:43 +00:00
David Robertson
dbfc9b803e
Fix dehydrated device REST checks (#14336) 2022-10-31 20:31:43 +00:00
Quentin Gliech
cc3a52b33d
Support OIDC backchannel logouts (#11414)
If configured an OIDC IdP can log a user's session out of
Synapse when they log out of the identity provider.

The IdP sends a request directly to Synapse (and must be
configured with an endpoint) when a user logs out.
2022-10-31 13:07:30 -04:00
David Robertson
2bb2c32e8e
Avoid incrementing bg process utime/stime counters by negative durations (#14323) 2022-10-31 13:02:07 +00:00
Andrew Morgan
7911e2835d
Prevent federation user keys query from returning device names if disallowed (#14304) 2022-10-28 18:06:02 +01:00
David Robertson
730b13dbc9
Improve RawHeaders type hints (#14303) 2022-10-28 16:04:02 +00:00
Patrick Cloke
81815e0561
Switch search SQL to triple-quote strings. (#14311)
For ease of reading we switch from concatenated strings to
triple quote strings.
2022-10-28 11:44:10 -04:00
Olivier Wilkinson (reivilibre)
1335367ca7 Merge branch 'master' into develop 2022-10-28 15:59:51 +01:00
reivilibre
6a6e1e8c07
Fix room creation being rate limited too aggressively since Synapse v1.69.0. (#14314)
* Introduce a test for the old behaviour which we want to restore

* Reintroduce the old behaviour in a simpler way

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Use 1 credit instead of 2 for creating a room: be more lenient than before

Notably, the UI in Element Web was still broken after restoring to prior behaviour.

After discussion, we agreed that it would be sensible to increase the limit.

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-10-28 11:53:34 +01:00
Eric Eastwood
aa70556699
Check appservice user interest against the local users instead of all users (get_users_in_room mis-use) (#13958) 2022-10-27 18:29:23 +00:00
Patrick Cloke
67583281e3
Fix tests for change in PostgreSQL 14 behavior change. (#14310)
PostgreSQL 14 changed the behavior of `websearch_to_tsquery` to
improve some behaviour.

The tests were hitting those edge-cases about handling of hanging double
quotes. This fixes the tests to take into account the PostgreSQL version.
2022-10-27 13:58:12 +00:00
Dirk Klimpel
1357ae869f
Add workers settings to configuration manual (#14086)
* Add workers settings to configuration manual
* Update `pusher_instances`
* update url to python logger
* update headlines
* update links after headline change
* remove link from `daemon process`

There is no docs in Synapse for this

* extend example for `federation_sender_instances` and `pusher_instances`
* more infos about stream writers
* add link to DAG
* update `pusher_instances`
* update `worker_listeners`
* update `stream_writers`
* Update `worker_name`

Co-authored-by: David Robertson <davidr@element.io>
2022-10-27 14:39:47 +01:00
Mathieu Velten
4dc05f3019
Fix presence bug introduced in 1.64 by #13313 (#14243)
* Fix presence bug introduced in 1.64 by #13313

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>

* Add changelog

* Add DISTINCT

* Apply suggestions from code review

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-10-27 13:16:00 +01:00
David Robertson
cbe01ccc3f
Reject history insertion during partial joins (#14291) 2022-10-27 10:52:23 +01:00
Eric Eastwood
40fa8294e3
Refactor MSC3030 /timestamp_to_event to move away from our snowflake pull from destination pattern (#14096)
1. `federation_client.timestamp_to_event(...)` now handles all `destination` looping and uses our generic `_try_destination_list(...)` helper.
 2. Consistently handling `NotRetryingDestination` and `FederationDeniedError` across `get_pdu` , backfill, and the generic `_try_destination_list` which is used for many places we use this pattern.
 3. `get_pdu(...)` now returns `PulledPduInfo` so we know which `destination` we ended up pulling the PDU from
2022-10-26 16:10:55 -05:00
Ashish Kumar
0cfbb35131
fix broken avatar checks when server_name contains a port (#13927)
Fixes check_avatar_size_and_mime_type() to successfully update avatars on homeservers running on non-default ports which it would mistakenly treat as remote homeserver while validating the avatar's size and mime type.

Signed-off-by: Ashish Kumar ashfame@users.noreply.github.com
2022-10-26 15:51:23 +01:00
David Robertson
04fd6221de
Fix incorrectly sending authentication tokens to application service as headers (#14301) 2022-10-26 14:00:01 +01:00
Quentin Gliech
8756d5c87e
Save login tokens in database (#13844)
* Save login tokens in database

Signed-off-by: Quentin Gliech <quenting@element.io>

* Add upgrade notes

* Track login token reuse in a Prometheus metric

Signed-off-by: Quentin Gliech <quenting@element.io>
2022-10-26 11:45:41 +01:00
James Salter
d902181de9
Unified search query syntax using the full-text search capabilities of the underlying DB. (#11635)
Support a unified search query syntax which leverages more of the full-text
search of each database supported by Synapse.

Supports, with the same syntax across Postgresql 11+ and Sqlite:

- quoted "search terms"
- `AND`, `OR`, `-` (negation) operators
- Matching words based on their stem, e.g. searches for "dog" matches
  documents containing "dogs". 

This is achieved by 

- If on postgresql 11+, pass the user input to `websearch_to_tsquery`
- If on sqlite, manually parse the query and transform it into the sqlite-specific
  query syntax.

Note that postgresql 10, which is close to end-of-life, falls back to using
`phraseto_tsquery`, which only supports a subset of the features.

Multiple terms separated by a space are implicitly ANDed.

Note that:

1. There is no escaping of full-text syntax that might be supported by the database;
  e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this
  as accidental functionality and depend on something we don't guarantee.
2. English text is assumed for stemming. To support other languages, either the target
  language needs to be known at the time of indexing the message (via room metadata,
  or otherwise), or a separate index for each language supported could be created.

Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries
Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
2022-10-25 14:05:22 -04:00
Olivier Wilkinson (reivilibre)
85fcbba595 Merge branch 'release-v1.70' into develop 2022-10-25 15:39:35 +01:00