Commit Graph

15229 Commits

Author SHA1 Message Date
Erik Johnston
cdf2707678
Fix bug in wait for stream position (#14872)
This caused some requests to fail.

This caused some requests to fail.

This really only started causing issues due to #14856
2023-01-19 22:19:56 +00:00
Andrew Morgan
a7b54ca8d8
Implement MSC3930: polls push rules (#14787) 2023-01-19 12:47:10 +00:00
Erik Johnston
9187fd940e
Wait for streams to catch up when processing HTTP replication. (#14820)
This should hopefully mitigate a class of races where data gets out of
sync due a HTTP replication request racing with the replication streams.
2023-01-18 19:35:29 +00:00
Catalan Lover
e8f2bf5c40
Change default room version to 10. Implements MSC3904 (#14111)
* Change Documentation to have v10 as default room version

* Change Default Room version to 10

* Add changelog entry for default room version swap

* Add changelog entry for v10 default room version in docs

* Clarify doc changelog entry

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Improve Documentation changes.

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Update Changelog entry to have correct format

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Update Spec Version to 1.5

* Only need 1 changelog.

* Fix test.

* Update "Changed in" line

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-01-18 18:59:48 +00:00
Patrick Cloke
4d6b1d3c47
Properly check for frozendicts in event auth code. (#14864)
Check for for an instance of a mapping instead of a dict.

This only affects room version 10 when frozen events are enabled.
2023-01-18 09:27:57 -05:00
David Robertson
5b3af1c7d0
Stabilise serving partial join responses (#14839)
Serving partial join responses is no longer experimental. They will only be served under the stable identifier if the the undocumented config flag experimental.msc3706_enabled is set to true.

Synapse continues to request a partial join only if the undocumented config flag experimental.faster_joins is set to true; this setting remains present and unaffected.
2023-01-17 12:44:15 +00:00
Erik Johnston
316590d1ea
Fix bug in wait_for_stream_position (#14856)
We were incorrectly checking if the *local* token had been advanced, rather than the token for the remote instance.

In practice, I don't think this has caused any bugs due to where we use `wait_for_stream_position`, as critically we don't use it on instances that also write to the given streams (and so the local token will lag behind all remote tokens).
2023-01-17 09:58:22 +00:00
Erik Johnston
2b084c5b71
Merge device list replication streams (#14833) 2023-01-17 09:29:58 +00:00
Sean Quah
db5145a31d
Add parameter to control whether we do a partial state join (#14843)
When the local homeserver is already joined to a room and wants to
perform another remote join, we may find it useful to do a non-partial
state join if we already have the full state for the room.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-01-16 23:15:17 +00:00
Erik Johnston
4db3331bb9
Add an early return when handling no-op presence updates. (#14855)
This stops us from incrementing the presence stream position for no-op updates.
2023-01-16 14:20:12 +00:00
Sean Quah
a302d3ecf7
Remove unnecessary reactor reference from _PerHostRatelimiter (#14842)
Fix up #14812 to avoid introducing a reference to the reactor.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-01-16 13:16:19 +00:00
David Robertson
85a7a201fa
Also use stable name in SendJoinResponse struct (#14841)
* Also use stable name in SendJoinResponse struct

follow-up to #14832

* Changelog

* Fix a rename I missed

* Run black

* Update synapse/federation/federation_client.py

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-01-16 12:40:25 +00:00
Andrew Morgan
54cd90ea60
Implement MSC3890: Remotely silence local notifications (#14775) 2023-01-13 19:32:10 +00:00
David Robertson
52ae80dd1a
Use stable identifiers for faster joins (#14832)
* Use new query param when requesting a partial join

* Read new query param when serving partial join

* Provide new field names when serving partial joins

* Read new field names from partial join response

* Changelog
2023-01-13 17:58:53 +00:00
Erik Johnston
73ff493dfb
Merge account data streams (#14826) 2023-01-13 14:57:43 +00:00
Dirk Klimpel
8d5325ec0c
Drop unused table presence (#14825) 2023-01-13 14:17:03 +00:00
Andrew Morgan
3a125625e7
Add some clarifying comments and refactor a portion of the Keyring class for readability (#14804) 2023-01-13 12:37:28 +00:00
Sean Quah
772e8c2385
Fix stack overflow in _PerHostRatelimiter due to synchronous requests (#14812)
When there are many synchronous requests waiting on a
`_PerHostRatelimiter`, each request will be started recursively just
after the previous request has completed. Under the right conditions,
this leads to stack exhaustion.

A common way for requests to become synchronous is when the remote
client disconnects early, because the homeserver is overloaded and slow
to respond.

Avoid stack exhaustion under these conditions by deferring subsequent
requests until the next reactor tick.

Fixes #14480.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-01-13 00:16:21 +00:00
Richard van der Hoff
0f061f39f0 Merge remote-tracking branch 'origin/release-v1.75' into develop 2023-01-12 16:45:23 +00:00
Erik Johnston
b50c008453
Re-enable some linting (#14821)
* Re-enable some linting

* Newsfile

* Remove comment
2023-01-12 10:52:07 +00:00
Erik Johnston
84ce93c12f
Fix race calling /members?at= (#14817)
Fixes #14814
2023-01-12 10:29:09 +00:00
Emelie Graven
dd9e71dc7f
Add set_displayname to the module API (#14629) 2023-01-11 18:41:52 +00:00
reivilibre
5172c8c403
Faster remote room joins (worker mode): do not populate external hosts-in-room cache when sending events as this requires blocking for full state. (#14749)
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Sean Quah <seanq@matrix.org>
2023-01-11 13:21:53 +00:00
reivilibre
d6bda5addd
Add index to improve performance of the /timestamp_to_event endpoint used for jumping to a specific date in the timeline of a room. (#14799) 2023-01-11 12:29:13 +00:00
Patrick Cloke
3952297f6f
Calculate rooms changed for device lists to work. (#14810)
Back-out some changes from 7e582a25f8
(#14786) which skipped necessary logic to calculate device lists properly.
2023-01-11 12:16:41 +00:00
Dirk Klimpel
73f097888e
Add listener health (#14747)
Fixes: #8780
2023-01-11 12:00:38 +00:00
Richard van der Hoff
06ab64f201
Implement MSC3925: changes to bundling of edits (#14811)
Two parts to this:

 * Bundle the whole of the replacement with any edited events. This is backwards-compatible so I haven't put it behind a flag.
 * Optionally, inhibit server-side replacement of edited events. This has scope to break things, so it is currently disabled by default.
2023-01-10 16:31:28 +00:00
reivilibre
ba4ea7d13f
Batch up replication requests to request the resyncing of remote users's devices. (#14716) 2023-01-10 11:17:59 +00:00
Jeyachandran Rathnam
58d2adc3da
Remove undocumented device from pushrules (#14727)
* Remove undocumented device from pushrules

* Add changelog

* Update changelog.d/14727.misc

* Rename 14727.misc to 14727.bugfix

Co-authored-by: David Robertson <davidr@element.io>
2023-01-09 17:17:24 +00:00
Jeyachandran Rathnam
babeeb4e7a
Unescape HTML entities in oEmbed titles. (#14781)
It doesn't seem valid that HTML entities should appear in
the title field of oEmbed responses, but a popular WordPress
plug-in seems to do it.

There should not be harm in unescaping these.
2023-01-09 14:22:02 +00:00
Patrick Cloke
7e582a25f8
Improve /sync performance of when passing filters with empty arrays. (#14786)
This has two related changes:

* It enables fast-path processing for an empty filter (`[]`) which was
  previously only used for wildcard not-filters (`["*"]`).
* It special cases a `/sync` filter with no-rooms to skip all room
  processing, previously we would partially skip processing, but would
  generally still calculate intermediate values for each room which were
  then unused.

Future changes might consider further optimizations:

* Skip calculating per-room account data when all rooms are filtered (currently
  this is thrown away).
* Make similar improvements to other endpoints which support filters.
2023-01-09 08:43:50 -05:00
Jeyachandran Rathnam
5e0888076f
Disable sending confirmation email when 3pid is disabled #14682 (#14725)
* Fixes #12277 :Disable sending confirmation email when 3pid is disabled

* Fix test_add_email_if_disabled test case to reflect changes to enable_3pid_changes flag

* Add changelog file

* Rename newsfragment.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-01-09 11:12:03 +00:00
Patrick Cloke
630d0aeaf6
Support RFC7636 PKCE in the OAuth 2.0 flow. (#14750)
PKCE can protect against certain attacks and is enabled by default. Support
can be controlled manually by setting the pkce_method of each oidc_providers
entry to 'auto' (default), 'always', or 'never'.

This is required by Twitter OAuth 2.0 support.
2023-01-04 14:58:08 -05:00
Patrick Cloke
906dfaa2cf
Support non-OpenID compliant user info endpoints (#14753)
OpenID specifies the format of the user info endpoint and some
OAuth 2.0 IdPs do not follow it, e.g. NextCloud and Twitter.

This adds subject_template and picture_template options to the
default mapping provider for more flexibility in matching those user
info responses.
2023-01-04 08:26:10 -05:00
Nick Mills-Barrett
db1cfe9c80
Update all stream IDs after processing replication rows (#14723)
This creates a new store method, `process_replication_position` that
is called after `process_replication_rows`. By moving stream ID advances
here this guarantees any relevant cache invalidations will have been
applied before the stream is advanced.

This avoids race conditions where Python switches between threads mid
way through processing the `process_replication_rows` method where stream
IDs may be advanced before caches are invalidated due to class resolution
ordering.

See this comment/issue for further discussion:
	https://github.com/matrix-org/synapse/issues/14158#issuecomment-1344048703
2023-01-04 11:49:26 +00:00
Andrew Morgan
c4456114e1
Add experimental support for MSC3391: deleting account data (#14714) 2023-01-01 03:40:46 +00:00
Patrick Cloke
044fa1a1de
Actually use the picture_claim as configured in OIDC config. (#14751)
Previously it was only using the default value ("picture") when
fetching the picture from the user info.
2022-12-29 12:18:06 -05:00
Brendan Abolivier
3854d0f949
Add a cached helper to the module API (#14663) 2022-12-28 13:48:21 +00:00
Richard van der Hoff
a52822d39c
Log to-device msgids when we return them over /sync (#14724) 2022-12-23 14:04:50 +00:00
Jeyachandran Rathnam
5c9be9c760
Check sqlite database file exists before porting. (#14692)
To avoid creating an empty SQLite file if the given path
is incorrect.
2022-12-22 13:26:37 -05:00
Patrick Cloke
7010a3d015
Switch to ruff instead of flake8. (#14633)
ruff is a flake8-compatible Python linter written in Rust.
It supports the flake8 plugins that we use and is significantly
faster in testing.
2022-12-21 13:05:21 -05:00
reivilibre
2888d7ec83
Faster remote room joins: invalidate caches and unblock requests when receiving un-partial-stated event notifications over replication. [rei:frrj/streams/unpsr] (#14546) 2022-12-19 14:57:51 +00:00
Patrick Cloke
3aeca2588b
Add missing type hints to tests.config. (#14681) 2022-12-16 08:53:28 -05:00
reivilibre
864c3f85b0
Improve type annotations for the helper methods on a CachedFunction. (#14685) 2022-12-16 13:04:54 +00:00
Patrick Cloke
652d1669c5
Add missing type hints to tests.handlers. (#14680)
And do not allow untyped defs in tests.handlers.
2022-12-16 11:53:01 +00:00
Mathieu Velten
54c012c5a8
Make handle_new_client_event throws PartialStateConflictError (#14665)
Then adapts calling code to retry when needed so it doesn't 500
to clients.

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-12-15 16:04:23 +00:00
David Robertson
4f4d690423
Allow compute_state_after_events to use partial state (#14676)
* Allow `compute_state_after_events` to use partial state

if fetching a subset of state that is trusted during a partial join.

* Changelog
2022-12-14 14:52:35 +00:00
reivilibre
fb60cb16fe
Faster remote room joins: stream the un-partial-stating of events over replication. [rei:frrj/streams/unpsr] (#14545) 2022-12-14 14:47:11 +00:00
Patrick Cloke
24a97b3e71
Delete event_push_summary_unique_index again. (#14669)
if a Synapse deployment upgraded (from < 1.62.0 to >= 1.70.0) then it
is possible for schema deltas to run before background updates causing
drift in the database schema due to:

1. A delta registered a background update to create an index.
2. A delta dropped the above index if it exists (but it yet exist won't since
  the background job hasn't run).
3. The code assumed the index was dropped.

To fix this we:

1. Cancel the background update which could create the index.
2. Drop the index again.
3. Drop a related index which is dropped by the background update.
2022-12-14 09:25:33 -05:00
Jeyachandran Rathnam
e512b25cd1
Fix #11308 : Remove dependency on jquery on reCAPTCHA page (#14672) 2022-12-14 12:02:28 +00:00
Patrick Cloke
2920e540bf
Use the room type from stats in hierarchy response. (#14263)
This avoids pulling additional state information (and events) from
the database for each item returned in the hierarchy response.

The room type might be out of date until a background update finishes
running, the worst impact of this would be spaces being treated as rooms
in the hierarchy response. This should self-heal once the background
update finishes.
2022-12-13 08:43:53 -05:00
reivilibre
62ed877433
Improve validation of field size limits in events. (#14664) 2022-12-13 13:19:19 +00:00
David Robertson
e2a1adbf5d
Allow selecting "prejoin" events by state keys (#14642)
* Declare new config

* Parse new config

* Read new config

* Don't use trial/our TestCase where it's not needed

Before:

```
$ time trial tests/events/test_utils.py > /dev/null

real	0m2.277s
user	0m2.186s
sys	0m0.083s
```

After:
```
$ time trial tests/events/test_utils.py > /dev/null

real	0m0.566s
user	0m0.508s
sys	0m0.056s
```

* Helper to upsert to event fields

without exceeding size limits.

* Use helper when adding invite/knock state

Now that we allow admins to include events in prejoin room state with
arbitrary state keys, be a good Matrix citizen and ensure they don't
accidentally create an oversized event.

* Changelog

* Move StateFilter tests

should have done this in #14668

* Add extra methods to StateFilter

* Use StateFilter

* Ensure test file enforces typed defs; alphabetise

* Workaround surprising get_current_state_ids

* Whoops, fix mypy
2022-12-13 00:54:46 +00:00
David Robertson
3d87847ecc
Enable --warn-redundant-casts option in mypy (#14671)
* Enable `--warn-redundant-casts` option in mypy

Doesn't do much but helps me sleep better at night.

* Changelog

* Fix name of the ignore

* Fix one more missed cast

Not sure why I didn't see this one locally, maybe I needed a poetry update

* Remove old comment

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-12-12 21:25:07 +00:00
Sean Quah
7982891794
Fix missing cache invalidation in application service code (#14670)
#11915 introduced the `@cached` `is_interested_in_room` method in
Synapse 1.55.0, which depends upon `get_aliases_for_room`. Add a missing
cache invalidation callback so that the `is_interested_in_room` cache is
invalidated when `get_aliases_for_room` is invalidated.

#13787 made `get_rooms_for_user` `@cached`. Add a missing cache
invalidation callback so that the `is_interested_in_presence` cache is
invalidated when `get_rooms_for_user` is invalidated.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-12-12 18:13:43 +00:00
David Robertson
b5b5f66084
Move StateFilter to synapse.types (#14668)
* Move `StateFilter` to `synapse.types`

* Changelog
2022-12-12 16:19:30 +00:00
reivilibre
74b89c2761
Revert the deletion of stale devices due to performance issues. (#14662) 2022-12-12 13:55:23 +00:00
Brendan Abolivier
2a3cd59dd0
Add optional ICU support for user search (#14464)
Fixes #13655

This change uses ICU (International Components for Unicode) to improve boundary detection in user search.

This change also adds a new dependency on libicu-dev and pkg-config for the Debian packages, which are available in all supported distros.
2022-12-12 13:21:17 +01:00
Sean Quah
373c485d8c
Handle half-created indices in receipts index background update (#14650)
When Synapse is terminated while running the background update to create
the `receipts_graph` or `receipts_linearized` indexes, the indexes may
be successfully created (or marked as invalid on postgres) while the
background update remains unfinished. When Synapse next starts up, the
background update will fail because the index already exists, or exists
but is invalid on postgres.

Use the existing code to create indices in background updates, since it
handles these edge cases.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-12-09 23:02:11 +00:00
Patrick Cloke
3ac412b4e2
Require types in tests.storage. (#14646)
Adds missing type hints to `tests.storage` package
and does not allow untyped definitions.
2022-12-09 12:36:32 -05:00
Erik Johnston
94bc21e69f
Limit the number of devices we delete at once (#14649) 2022-12-09 13:31:32 +00:00
Erik Johnston
c2de2ca630
Delete stale non-e2e devices for users, take 2 (#14595)
This should help reduce the number of devices e.g. simple bots the repeatedly login rack up.

We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
2022-12-09 09:37:07 +00:00
Ashish Kumar
a58b550eac
Fix html templates to load images only on HTTPS (#14625)
This PR changes http-based image URLs to be https in html templates.
This impacts the Synapse SSO error page, where browsers report mixed
media content warnings.

Also, https://matrix.org/img/vector-logo-email.png is currently broken
but the URL has been updated to be https anyway.

Signed-off-by: Ashish Kumar <ashfame@users.noreply.github.com>
2022-12-08 17:28:02 +00:00
Patrick Cloke
c369e95691
Rebuild the user directory and stats tables. (#14643)
Due to the various fixes to the StreamChangeCache it is not
safe to trust the information in the user directory or room/user
stats tables. Rebuild them as background jobs.

In particular see da77720752 (#14639),
and 6a8310f3df (#14435).

Maybe also be related to fac8a38525
(#14592).
2022-12-08 11:40:20 -05:00
Patrick Cloke
9d8a3234ba
Respond with proper error responses on unknown paths. (#14621)
Returns a proper 404 with an errcode of M_RECOGNIZED for
unknown endpoints per MSC3743.
2022-12-08 11:37:05 -05:00
Patrick Cloke
da77720752
Check the stream position before checking if the cache is empty. (#14639)
An empty cache does not mean the entity has no changed, if
it is earlier than the earliest known stream position return that
the entity *has* changed since the cache cannot accurately
answer that query.
2022-12-08 11:35:49 -05:00
Nick Mills-Barrett
60c3fea327
Reject receipt requests with invalid room or event IDs. (#14632)
If the room or event IDs are empty or of an invalid form they
should be rejected.
2022-12-07 17:35:41 +00:00
reivilibre
96251af50d
Fix a bug introduced in v1.67.0 where not specifying a config file or a server URL would lead to the register_new_matrix_user script failing. (#14637) 2022-12-07 13:39:27 +00:00
reivilibre
cf1059d045
Fix a long-standing bug where the user directory would return 1 more row than requested. (#14631) 2022-12-07 11:19:43 +00:00
reivilibre
9e82caac45
Faster remote room joins: unblock tasks waiting for full room state when the un-partial-stating of that room is received over the replication stream. [rei:frrj/streams/unpsr] (#14474) 2022-12-06 15:48:42 +00:00
Patrick Cloke
9b6224577e
Failover on proper error responses. (#14620)
When querying a remote server handle a 404/405 with an
errcode of M_UNRECOGNIZED as an unimplemented endpoint.
2022-12-06 07:23:03 -05:00
Richard van der Hoff
cb59e08062
Improve logging and opentracing for to-device message handling (#14598)
A batch of changes intended to make it easier to trace to-device messages through the system.

The intention here is that a client can set a property org.matrix.msgid in any to-device message it sends. That ID is then included in any tracing or logging related to the message. (Suggestions as to where this field should be documented welcome. I'm not enthusiastic about speccing it - it's very much an optional extra to help with debugging.)

I've also generally improved the data we send to opentracing for these messages.
2022-12-06 09:52:55 +00:00
Erik Johnston
cee9445884
Better return type for get_all_entities_changed (#14604)
Help callers from using the return value incorrectly by ensuring
that callers explicitly check if there was a cache hit or not.
2022-12-05 15:19:14 -05:00
Patrick Cloke
6a8310f3df
Compare to the earliest known stream pos in the stream change cache. (#14435)
The internal methods of the StreamChangeCache were inconsistently
treating the earliest known stream position as valid. It is now treated as
invalid, meaning the cache cannot determine if an entity at the earliest
known stream position has changed or not.
2022-12-05 09:00:59 -05:00
reivilibre
501f62d1a6
Faster remote room joins: stream the un-partial-stating of rooms over replication. [rei:frrj/streams/unpsr] (#14473) 2022-12-05 13:07:55 +00:00
David Robertson
93ac3c197e
Suppress empty body warnings in room servelets (#14600)
* Suppress empty body warnings in room servelets

We've already decided to allow empty bodies for backwards compat. The
change here stops us from emitting a misleading warning; see also
https://github.com/matrix-org/synapse/issues/14478#issuecomment-1319157105

* Changelog
2022-12-05 11:30:41 +00:00
Patrick Cloke
f685318c2a
Use ClientRestResource on both the main process and workers. (#14528)
Add logic to ClientRestResource to decide whether to mount servlets
or not based on whether the current process is a worker.

This is clearer to see what a worker runs than the completely separate /
copy & pasted list of servlets being mounted for workers.
2022-12-02 13:10:05 -05:00
Patrick Cloke
fac8a38525
Properly handle unknown results for the stream change cache. (#14592)
StreamChangeCache.get_all_changed_entities can return None to signify
it does not have information at the given stream position. Two callers (related
to device lists and presence) were treating this response the same as an empty
list (i.e. there being no updates).
2022-12-02 10:28:41 -05:00
Will Hunt
71f3e53ad0
Add push.enabled option to disable push notification calculation (#14551)
* Add initial option

* changelog

* Some more linting
2022-12-01 13:46:24 +00:00
David Robertson
781b14ec69
Merge branch 'release-v1.73' into develop 2022-12-01 13:43:30 +00:00
Nick Mills-Barrett
e8bce8999f
Aggregate unread notif count query for badge count calculation (#14255)
Fetch the unread notification counts used by the badge counts
in push notifications for all rooms at once (instead of fetching
them per room).
2022-11-30 08:45:06 -05:00
Mathieu Velten
4569eda944
Use servers list approx to send read receipts when in partial state (#14549)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-30 13:39:47 +01:00
Richard van der Hoff
ecb6fe9d9c
Stop using deprecated keyIds param on /key/v2/server (#14525)
Fixes #14523.
2022-11-30 11:59:57 +00:00
David Robertson
c29e2c6306
Revert "POC delete stale non-e2e devices for users (#14038)" (#14582) 2022-11-29 17:48:48 +00:00
Patrick Cloke
13aa29db1d
Advertise support for Matrix v1.5. (#14576)
All features of Matrix v1.5 were already supported: this was
mostly a maintenance release.
2022-11-29 10:49:23 -05:00
David Robertson
e860316818
Fix UndefinedColumn: column "key_json" does not exist errors when handling users with more than 50 non-E2E devices (#14580) 2022-11-29 13:05:07 +00:00
Erik Johnston
c7e29ca277
POC delete stale non-e2e devices for users (#14038)
This should help reduce the number of devices e.g. simple bots the repeatedly login rack up.

We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-29 10:36:41 +00:00
Shay
72f3e38137
Fix possible variable shadow in create_new_client_event (#14575) 2022-11-28 19:18:12 -08:00
Travis Ralston
9ccc09fe9e
Support MSC1767's content.body behaviour; Add base rules from MSC3933 (#14524)
* Support MSC1767's `content.body` behaviour in push rules

* Add the base rules from MSC3933

* Changelog entry

* Flip condition around for finding `m.markup`

* Remove forgotten import
2022-11-28 18:02:41 -07:00
Travis Ralston
dd51828120
Create MSC1767 (extensible events) room version; Implement MSC3932 (#14521)
* Add MSC1767's dedicated room version, based on v10

* Only enable MSC1767 room version if the config flag is on

Using a similar technique to knocking:
https://github.com/matrix-org/synapse/pull/6739/files#diff-3af529eedb0e00279bafb7369370c9654b37792af8eafa0925400e9281d57f0a

* Support MSC3932: Extensible events room version feature flag

* Changelog entry
2022-11-28 17:22:34 -07:00
Travis Ralston
3da6450327
Initial support for MSC3931: Room version push rule feature flags (#14520)
* Add support for MSC3931: Room Version Supports push rule condition

* Create experimental flag for future work, and use it to gate MSC3931

* Changelog entry
2022-11-28 16:29:53 -07:00
Eric Eastwood
8f10c8b054
Move MSC3030 /timestamp_to_event endpoint to stable v1 location (#14471)
Fix https://github.com/matrix-org/synapse/issues/14390

 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`
 - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`

Complement test changes: https://github.com/matrix-org/complement/pull/559
2022-11-28 15:54:18 -06:00
Andrew Ferrazzutti
1183c372fa
Use device_one_time_keys_count to match MSC3202 (#14565)
* Use `device_one_time_keys_count` to match MSC3202

Rename the `device_one_time_key_counts` key in responses to
`device_one_time_keys_count` to match the name specified by MSC3202.

Also change related variable/class names for consistency.

Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>

* Update changelog.d/14565.misc

* Revert name change for `one_time_key_counts` key

as this is a different key altogether from `device_one_time_keys_count`,
which is used for `/sync` instead of appservice transactions.

Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
2022-11-28 16:17:29 +00:00
Sean Quah
d56f48038a
Fix logging context warnings due to common usage metrics setup (#14574)
`setup()` is run under the sentinel context manager, so we wrap the
initial update in a background process. Before this change, Synapse
would log two warnings on startup:
    Starting db txn 'count_daily_users' from sentinel context
    Starting db connection from sentinel context: metrics will be lost

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-28 15:25:18 +00:00
Patrick Cloke
d748bbc8f8
Include thread information when sending receipts over federation. (#14466)
Include the thread_id field when sending read receipts over
federation. This might result in the same user having multiple
read receipts per-room, meaning multiple EDUs must be sent
to encapsulate those receipts.

This restructures the PerDestinationQueue APIs to support
multiple receipt EDUs, queue_read_receipt now becomes linear
time in the number of queued threaded receipts in the room for
the given user, it is expected this is a small number since receipt
EDUs are sent as filler in transactions.
2022-11-28 14:40:17 +00:00
Sean Quah
f792dd74e1
Remove option to skip locking of tables during emulated upserts (#14469)
To perform an emulated upsert into a table safely, we must either:
 * lock the table,
 * be the only writer upserting into the table
 * or rely on another unique index being present.

When the 2nd or 3rd cases were applicable, we previously avoided locking
the table as an optimization. However, as seen in #14406, it is easy to
slip up when adding new schema deltas and corrupt the database.

The only time we lock when performing emulated upserts is while waiting
for background updates on postgres. On sqlite, we do no locking at all.

Let's remove the option to skip locking tables, so that we don't shoot
ourselves in the foot again.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-28 13:42:06 +00:00
Ashish Kumar
09de2aecb0
Add support for handling avatar with SSO login (#13917)
This commit adds support for handling a provided avatar picture URL
when logging in via SSO.

Signed-off-by: Ashish Kumar <ashfame@users.noreply.github.com>

Fixes #9357.
2022-11-25 15:16:50 +00:00
Mathieu Velten
39cde585bf
Faster joins: use initial list of servers if we don't have the full state yet (#14408)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-24 18:09:47 +01:00
schmop
c2e06c36d4
Fix crash admin media list api when info is None (#14537)
Fixes https://github.com/matrix-org/synapse/issues/14536
2022-11-24 10:49:04 +00:00
Benjamin Kampmann
f6c74d1cb2
Implement message forward pagination from start when no from is given, fixes #12383 (#14149)
Fixes https://github.com/matrix-org/synapse/issues/12383
2022-11-24 09:10:51 +00:00