Commit Graph

24104 Commits

Author SHA1 Message Date
dependabot[bot]
ad0ee53993
Bump serde from 1.0.206 to 1.0.208 (#17581) 2024-08-23 09:52:16 +01:00
Erik Johnston
92b38c1afd
Sliding sync: Split up handler into its own module (#17595)
That file was getting long.

The changes are non functional, and simply split things up into:
- the main class
- the connection store
- the extensions
- the types
2024-08-20 18:30:23 +00:00
Quentin Gliech
a8e313836d
changelog: move SSSS some changes in the features section 2024-08-20 15:18:13 +02:00
Quentin Gliech
7c9684b5dc
1.114.0rc1 2024-08-20 14:57:22 +02:00
Erik Johnston
f1e8d2d15a
Sliding Sync: Speed up getting receipts for initial rooms (#17592)
Let's only pull out the events we care about. Note that the index isn't
necessary here, as postgres is happy to scan the set of rooms for the
events.
2024-08-20 12:57:34 +01:00
Erik Johnston
10428046e4
Add metrics for sliding sync processing time (#17593)
This should let us see how quickly we actually process things in
practice.
2024-08-20 11:36:49 +00:00
Erik Johnston
6eb98a4f1c
Sliding Sync: Handle timeline limit changes (take 2) (#17579)
This supersedes #17503, given the per-connection state is being heavily
rewritten it felt easier to recreate the PR on top of that work.

This correctly handles the case of timeline limits going up and down.

This does not handle changes in `required_state`, but that can be done
as a separate PR.

Based on #17575.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-20 10:31:25 +01:00
Erik Johnston
950ba844f7
Sliding Sync: Batch up fetching receipts (#17589)
This is to make initial sliding sync a bit faster
2024-08-20 10:13:26 +01:00
Erik Johnston
8b8d74d12f
Sliding sync: Correctly track which read receipts we have or have not sent down. (#17575)
Add connection tracking to the receipts extension.

Based on #17574

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-19 21:16:07 +01:00
Erik Johnston
261e746281
Sliding sync: Add classes for per-connection state (#17574)
This is some prep work ahead of correctly tracking receipts, where we
will also want to track the room status in terms of last receipt we had
sent down.

Essentially, we add two classes `PerConnectionState` and a mutable
version, and then operate on those.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-19 20:09:41 +01:00
Erik Johnston
993644ded0
Fix zero length media handling (#17570)
Results in:

```
AssertionError: null
  File "synapse/http/server.py", line 332, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "synapse/http/server.py", line 544, in _async_render
    callback_return = await raw_callback_return
  File "synapse/federation/transport/server/_base.py", line 369, in new_func
    response = await func(
  File "synapse/federation/transport/server/federation.py", line 826, in on_GET
    await self.media_repo.get_local_media(
  File "synapse/media/media_repository.py", line 473, in get_local_media
    await respond_with_multipart_responder(
  File "synapse/media/_base.py", line 353, in respond_with_multipart_responder
    assert content_length is not None
```
2024-08-19 15:06:44 +01:00
Erik Johnston
a5d25bb623
Test github token before running release script (#17562)
This stops people from getting half way through a step and it failing
due to the github token having expired (this happens to me every damn
time).
2024-08-19 14:15:36 +01:00
Erik Johnston
f162c92f2a
Speed up /keys/changes (#17548)
Follow on from #17537.

This is just adding a batched lookup function (you might want to hide
whitespace in the diff).
2024-08-16 16:04:02 +01:00
Erik Johnston
9ce489be5e
Add a flag to /versions about SSS support (#17571)
So that clients can check for support. Note that if the feature is only
enabled for some users, the `/versions` request must be authenticated to
pick up that SSS is enabled for the user
2024-08-16 08:54:57 +01:00
Andrew Morgan
fae75b0376
Register the media threadpool with our metrics (#17566) 2024-08-14 15:11:22 +01:00
Tulir Asokan
f77bfbfa30
Fix fetching signing keys when old_verify_keys is omitted (#17568)
`old_verify_keys` isn't marked as required in
https://spec.matrix.org/v1.11/server-server-api/#get_matrixkeyv2server
and there's no functional difference between an empty object and
omitting the object, so I don't think there's any reason synapse should
explode when the field is omitted.
2024-08-14 14:13:56 +01:00
Erik Johnston
1892ba5f67
Fix 'Producer was not unregistered' error (#17569)
Follows on from #17567
2024-08-14 13:46:22 +01:00
Erik Johnston
a51daffba5
Reduce concurrent thread usage in media (#17567)
Follow on from #17558

Basically, we want to reduce the number of threads we want to use at a
time, i.e. reduce the number of threads that are paused/blocked. We do
this by returning from the thread when the consumer pauses the producer,
rather than pausing in the thread.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-08-14 12:41:53 +01:00
Shay
b05b2e14bb
Handle lower-case http headers in _Mulitpart_Parser_Protocol (#17545) 2024-08-14 09:49:01 +01:00
Eric Eastwood
a308d99f30
Sliding Sync: Exclude partially stated rooms if we must await full state (#17538)
Previously, we just had very basic partial room exclusion based on
whether we were lazy-loading room members. Now with this PR, we added
`must_await_full_state(...)` with rules to check if we have a we're only
requesting `required_state` which is completely satisfied even with
partial state.

Partially-stated rooms should have all state events except for remote
membership events so if we require a remote membership event anywhere,
then we need to return `True`.
2024-08-13 12:27:42 -05:00
Erik Johnston
a9fc1fd112
Use a larger, dedicated threadpool for media sending (#17564) 2024-08-13 17:59:47 +01:00
Andrew Morgan
6a11bdf01d
Add a utility function for generating fake event IDs (#17557) 2024-08-13 16:55:05 +00:00
Patrick Cloke
8fea190a1f
Add missing docstrings related to profile methods. (#17559) 2024-08-13 17:04:35 +01:00
Erik Johnston
81c19c4cd2 Merge branch 'master' into develop 2024-08-13 15:58:11 +01:00
Erik Johnston
aaa3c36420
Remove logging in multipart (#17563)
This is really spurious and causes a lot of spam. I don't think there is
a use for it even at DEBUG level.
2024-08-13 15:56:18 +01:00
Erik Johnston
3e7eb45eb1
Fixup media logcontexts (#17561)
Regression from #17558
2024-08-13 15:00:57 +01:00
Erik Johnston
bab37dfc6f 1.113.0 2024-08-13 14:37:01 +01:00
Erik Johnston
9f9ec92526
Speed up responding to media requests (#17558)
We do this by reading from a threadpool, rather than blocking the main
thread.

This is broadly what we do in the [S3 storage
provider](https://github.com/matrix-org/synapse-s3-storage-provider/blob/main/s3_storage_provider.py#L234)
2024-08-13 14:06:17 +01:00
dependabot[bot]
ff7b27013e
Bump serde from 1.0.204 to 1.0.206 (#17556) 2024-08-13 08:59:35 +01:00
Andrew Morgan
e1f5f0fbb8
Bump setuptools from 67.6.0 to 72.1.0 (#17542) 2024-08-12 14:58:01 +01:00
dependabot[bot]
8c9f2743bc
Bump serde_json from 1.0.122 to 1.0.124 (#17555) 2024-08-12 14:33:12 +01:00
dependabot[bot]
b076941a36
Bump sentry-sdk from 2.10.0 to 2.12.0 (#17553) 2024-08-12 14:32:18 +01:00
dependabot[bot]
8bbe65f3c0
Bump types-pyyaml from 6.0.12.20240311 to 6.0.12.20240808 (#17552) 2024-08-12 14:32:05 +01:00
dependabot[bot]
b7faf01f26
Bump phonenumbers from 8.13.42 to 8.13.43 (#17551) 2024-08-12 14:31:56 +01:00
dependabot[bot]
4f7f6ee9a0
Bump lxml from 5.2.2 to 5.3.0 (#17550) 2024-08-12 14:31:45 +01:00
dependabot[bot]
a640b318df
Bump sigstore/cosign-installer from 3.5.0 to 3.6.0 (#17549) 2024-08-12 14:31:34 +01:00
dependabot[bot]
34b7586446
Bump types-requests from 2.31.0.20240406 to 2.32.0.20240712 (#17524)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-12 10:20:38 +01:00
Erik Johnston
70b0e38603
Fix performance of device lists in /key/changes and sliding sync (#17537)
We do this by reusing the code from sync v2.

Reviewable commit-by-commit. The function `get_user_ids_changed` has
been rewritten entirely, so I would recommend not looking at the diff.
2024-08-09 11:59:44 +01:00
devonh
f31360e34b
Start handlers for new media endpoints when media resource configured (#17483)
This is in response to issue #17473. 
Not all the necessary handlers to deal with media requests are started
now when configuring synapse to use a media worker as per the [example
config](https://element-hq.github.io/synapse/latest/workers.html#synapseappmedia_repository).
The new media endpoints introduced with authenticated media fall under
the `client` & `federation` handlers in synapse.
This PR starts up handlers for the new media endpoints if a worker has
been configured with only the `media` resource type.

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [X] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-08-08 14:35:46 +00:00
Andrew Morgan
3ad38b644d
Replace deprecated HTTPAdapter.get_connection method with get_connection_with_tls_context (#17536) 2024-08-08 14:59:37 +01:00
Erik Johnston
44ac2aa3b6
SSS: Implement PREVIOUSLY room tracking (#17535)
Implement tracking of rooms that have had updates that have not been
sent down to clients.

Simplified Sliding Sync (SSS)
2024-08-08 10:44:17 +01:00
Eric Eastwood
11db575218
Sliding Sync: Use stream_ordering based timeline pagination for incremental sync (#17510)
Use `stream_ordering` based `timeline` pagination for incremental
`/sync` in Sliding Sync. Previously, we were always using a
`topological_ordering` but we should only be using that for historical
scenarios (initial `/sync`, newly joined, or haven't sent the room down
the connection before).

This is slightly different than what the [spec
suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing)

> Events are ordered in this API according to the arrival time of the
event on the homeserver. This can conflict with other APIs which order
events based on their partial ordering in the event graph. This can
result in duplicate events being received (once per distinct API
called). Clients SHOULD de-duplicate events based on the event ID when
this happens.

But we've had a [discussion below in this
PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569)
and this matches what Sync v2 already does and seems like it makes
sense. Created a spec issue
https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this.

Related issues:

 - https://github.com/matrix-org/matrix-spec/issues/1917
 - https://github.com/matrix-org/matrix-spec/issues/852
 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033
2024-08-07 11:27:50 -05:00
dependabot[bot]
30e9f6e469
Bump bytes from 1.6.1 to 1.7.1 (#17526) 2024-08-07 10:37:54 +01:00
dependabot[bot]
eb62d12063
Bump regex from 1.10.5 to 1.10.6 (#17527) 2024-08-07 10:37:13 +01:00
Erik Johnston
ceb3686dcd
Fixup sliding sync comment (#17531)
c.f.
https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925
2024-08-07 10:32:36 +01:00
Eric Eastwood
1dfa59b238
Sliding Sync: Add more tracing (#17514)
Spawning from looking at a couple traces and wanting a little more info.

Follow-up to github.com/element-hq/synapse/pull/17501

The changes in this PR allow you to find slow Sliding Sync traces ignoring the
`wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user`
operation with `RESULT.result=true` indicating that it actually returned non-empty results.

If you want to find traces for your own user, you can use
`RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
2024-08-06 11:43:43 -05:00
Andrew Morgan
bef6568537 Merge branch 'release-v1.113' into develop 2024-08-06 14:19:12 +01:00
Andrew Morgan
244a255065
Clarify auto_accept_invites.worker_to_run_on config docs (#17515) 2024-08-06 13:26:51 +01:00
Andrew Morgan
932cb0a928 1.113.0rc1 2024-08-06 12:24:47 +01:00
dependabot[bot]
2dad718265
Bump phonenumbers from 8.13.39 to 8.13.42 (#17521) 2024-08-06 11:47:19 +01:00