Commit Graph

8040 Commits

Author SHA1 Message Date
Erik Johnston
01910b981f
Add a config to not send out device list updates for specific users (#16909)
List of users not to send out device list updates for when they register
new devices. This is useful to handle bot accounts.

This is undocumented as its mostly a hack to test on matrix.org.

Note: This will still send out device list updates if the device is
later updated, e.g. end to end keys are added.
2024-02-13 13:23:03 +00:00
Erik Johnston
ea1b30940e Merge remote-tracking branch 'origin/release-v1.101' into develop 2024-02-09 10:52:35 +00:00
Erik Johnston
bfa93d1d3b
Only do one concurrent fetch per server in keyring (#16894)
Otherwise if we've stacked a bunch of requests for the keys of a server,
we'll end up sending lots of concurrent requests for its keys,
needlessly.
2024-02-09 10:51:11 +00:00
Erik Johnston
02a147039c
Increase batching when fetching auth chains (#16893)
This basically reverts a change that was in
https://github.com/element-hq/synapse/pull/16833, where we reduced the
batching.

The smaller batching can cause performance issues on busy servers and
databases.
2024-02-09 10:51:00 +00:00
Erik Johnston
7c805f00a7 1.101.0rc1 2024-02-06 16:33:19 +00:00
David Baker
71ca199165
Accept unprefixed form of MSC3981 recurse parameter (#16842)
Now that the MSC3981 has passed FCP
2024-02-06 09:48:39 +00:00
kegsay
93edd0932e
Update docs for MacOS installs (#16854)
ICU is an optional dependency and also a pain to install. Mention that
you can just not install it and still get a working installation.
2024-02-06 09:27:38 +00:00
kegsay
505cdd044b
Fix broken links on docs (#16853)
Some links seemed to be incorrect (vector-im/sygnal and vector-im/sytest
have never been A Thing iirc) so pointed them back to matrix-org/*).
2024-02-06 09:26:55 +00:00
Will Hunt
d24d115706
Update version picker for element-hq (#16880) 2024-02-01 14:30:16 +00:00
Olivier Wilkinson (reivilibre)
458c9f6a15 1.100.0rc3 2024-01-24 14:18:35 +00:00
Erik Johnston
adf15c4f6b
Run ANALYZE after fiddling with stats (#16849)
Introduced in #16833

Fixes #16844
2024-01-24 13:57:12 +00:00
Olivier Wilkinson (reivilibre)
cf7109fc35 1.100.0rc2 2024-01-24 12:01:20 +00:00
reivilibre
d0bdd41a75
Downgrade the download-artifact and upload-artifact actions to v3 due to breaking changes. (#16847)
Partially reverts #16796

This is causing errors of the form:

```
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```

for the debs and wheels stages.


There were breaking changes that weren't included in the dependabot
changelog (:/):
https://github.com/actions/upload-artifact#breaking-changes


<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `release-v1.100` <!-- git-stack-base-branch:release-v1.100 -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Downgrade the `upload-artifact` and `download-artifact` actions to v3 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-24 11:59:15 +00:00
Olivier Wilkinson (reivilibre)
f366455682 1.100.0rc1 2024-01-23 14:26:45 +00:00
Erik Johnston
c925b45567
Speed up e2e device keys queries for bot accounts (#16841)
This helps with bot accounts with lots of non-e2e devices.

The change is basically to change the order of the join for the case of
using `INNER JOIN`
2024-01-23 11:37:16 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
14c725f73b
Preparatory work for tweaking performance of auth chain lookups (#16833) 2024-01-23 11:26:27 +00:00
reivilibre
fa2700f001
Add a --generate-only option to the Complement launcher. (#16828)
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Add a --generate-only option 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-22 16:00:04 +00:00
Arnold
8459ac9be2
listen http2 deprecated nginx (updating documentation) (#16831)
More info [here](https://www.nginx.com/blog/nginx-plus-r30-released/).

Nginx threw error's at me when I used all the options of the doc
2024-01-22 14:22:16 +00:00
Shay
a68b48a5dd
Allow room creation but not publishing to continue if room publication rules are violated when creating a new room. (#16811)
Prior to this PR, if a request to create a public (public as in
published to the rooms directory) room violated the room list
publication rules set in the
[config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules),
the request to create the room was denied and the room was not created.

This PR changes the behavior such that when a request to create a room
published to the directory violates room list publication rules, the
room is still created but the room is not published to the directory.
2024-01-22 13:59:45 +00:00
Erik Johnston
3fefb15fef
Bump minimum Rust version to 1.65.0 (#16818)
The ecosystem e.g. regex crate, have bumped up their MSRV to 1.65.0,
which was released Nov 2022. In line with our policy, let's bump to
match.
2024-01-22 11:11:52 +00:00
Mo Balaa
b99f6db039
Handle wildcard type filters properly (#14984) 2024-01-22 10:46:30 +00:00
Erik Johnston
f95cfd898e
Fix broken links in issue template (#16810)
This happened during the migration
2024-01-17 13:40:00 +00:00
Hanadi
42e1aaea68
feat: add msc4028 to versions api (#16787)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-16 14:36:08 +00:00
Erik Johnston
c43f751013
Optimize query for fetching to-device messages in /sync (#16805)
The current query supports passing in a list of users, which generates a
query using `user_id = ANY(..)`. This is generates a less efficient
query plan that is notably slower than a simple `user_id = ?` condition.

Note: The new function is mostly a copy and paste and then a
simplification of the existing function.
2024-01-11 13:37:57 +00:00
Erik Johnston
b11f7b5122
Improve DB performance of calculating badge counts for push. (#16756)
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.

Review note: this is a complete rewrite of the function, so not sure how
useful the diff is.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-11 11:52:13 +00:00
Erik Johnston
a986f86c82
Correctly handle OIDC config with no client_secret set (#16806)
In previous versions of authlib using `client_secret_basic` without a
`client_secret` would result in an invalid auth header. Since authlib
1.3 it throws an exception.

The configuration may be accepted in by very lax servers, so we don't
want to deny it outright. Instead, let's default the
`client_auth_method` to `none`, which does the right thing. If the
config specifies `client_auth_method` and no `client_secret` then that
is going to be bogus and we should reject it
2024-01-10 17:16:49 +00:00
Erik Johnston
cbe8a80d10
Faster load recents for sync (#16783)
This hopefully reduces the amount of state we need to keep in memory
2024-01-10 15:11:59 +00:00
Erik Johnston
0a96fa52a2
Pull less state out if we fail to backfill (#16788)
Sometimes we fail to fetch events during backfill due to missing state,
and we often end up querying the same bad events periodically (as people
backpaginate). In such cases its likely we will continue to fail to get
the state, and therefore we should try *before* loading the state that
we have from the DB (as otherwise it's wasted DB and memory).

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:42:13 +00:00
Erik Johnston
578c5c736e
Reduce amount of state pulled out when querying federation hierachy (#16785)
There are two changes here:

1. Only pull out the required state when handling the request.
2. Change the get filtered state return type to check that we're only
querying state that was requested

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:31:35 +00:00
Erik Johnston
4c67f0391b
Split up deleting devices into batches (#16766)
Otherwise for users with large numbers of devices this can cause a lot
of woe.
2024-01-10 13:55:16 +00:00
Erik Johnston
72e9b74bbf
Fix auto-merge CI to correctly wait for linting. (#16781)
Otherwise if you hit the `Enable auto-merge` button and the linting
fails the PR is still aut-merged.
2024-01-10 13:53:44 +00:00
Erik Johnston
8189942a1f
Remove CI check for sign off (#16776)
Since we don't require one anymore.
2024-01-10 13:53:20 +00:00
Andrew Morgan
13e3740f70
Add a link to the Request log format page from Logging Sample Config (#16778) 2024-01-10 13:34:55 +00:00
Erik Johnston
c3f2f0f063
Faster partial join to room with complex auth graph (#7)
Instead of persisting outliers in a bunch of batches, let's just do them
all at once.

This is fine because all `_auth_and_persist_outliers_inner` is doing is
checking the auth rules for each event, which requires the events to be
topologically sorted by the auth graph.
2024-01-10 12:29:42 +00:00
Erik Johnston
7109274c65 1.99.0rc1 2024-01-09 13:44:02 +00:00
reivilibre
a83a337c4d
Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. (#16759)
The idea here being that the directory server shouldn't advertise rooms
to a requesting server is the requesting server would not be allowed to
join or participate in the room.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Pass `from_federation_origin` down into room list retrieval code 

</li>
<li>

Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

fixup! Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

Cap the number of /publicRooms entries to 100 

</li>
<li>

Simplify code now that you can't request unlimited rooms 

</li>
<li>

Filter out rooms from federated requests that don't have the correct ACL

</li>
<li>

Request a handful more when filtering ACLs so that we can try to avoid
shortchanging the requester

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-08 17:24:20 +00:00
Erik Johnston
5d3850b038
Port EventInternalMetadata class to Rust (#16782)
There are a couple of things we need to be careful of here:

1. The current python code does no validation when loading from the DB,
so we need to be careful to ignore such errors (at least on jki.re there
are some old events with internal metadata fields of the wrong type).
2. We want to be memory efficient, as we often have many hundreds of
thousands of events in the cache at a time.

---------

Co-authored-by: Quentin Gliech <quenting@element.io>
2024-01-08 14:06:48 +00:00
Erik Johnston
81b1c56288
Fix linting (#16780)
Introduced in #16762
2024-01-05 13:29:00 +00:00
Erik Johnston
7469fa7585
Simplify internal metadata class. (#16762)
We remove these fields as they're just duplicating data the event
already stores, and (for reasons 🤫) I'd like to simplify
the class to only store simple types.

I'm not entirely convinced that we shouldn't instead add helper methods
to the event class to generate stream tokens, but I don't really think
that's where they belong either
2024-01-05 13:03:20 +00:00
Gaël Goinvic
9ee3db1de5
Implement cosign on docker image (#16774)
Signed-off-by: Gaël Goinvic <gaelg@element.io>
2024-01-04 11:49:33 +00:00
David Baker
25b3ba5328
Add recursion_depth to /relations if recursing (#16775)
This is an extra response parameter just added to MSC3981. In the
current impl, the recursion depth is always 3, so this just returns a
static 3 if the recurse parameter is supplied.
2024-01-04 09:57:21 +00:00
Adam Jędrzejewski
c7d0d02be7
Search non ASCII display names using Admin API (#16767)
Closes #16370

Signed-off-by: Adam Jedrzejewski <adamjedrzejewski@icloud.com>
2024-01-04 09:36:57 +00:00
FadhlanR
eabedd9520
Fix email verification redirection (#16761)
Previously, the response status of `HTMLResource` was hardcoded as
`200`. However, for proper redirection after the user verifies their
email, we require the status to be `302`. This PR addresses that issue
by using `code` as response status.
2024-01-02 16:25:26 +00:00
Andrew Morgan
1607ed5b2c
Delete the add version picker GHA workflow (#9453)
Added in https://github.com/matrix-org/synapse/pull/16533, this workflow
was intended to be run once to add the version picker to all historical
versions of the https://matrix-org.github.io/synapse documentation
website.

Note that the latest version of the docs built from this repo now exist
at https://element-hq.github.io/synapse/.

The workflow has been run successfully and the version picker was added
to the documentation. Thus we can now delete this workflow.

---

Note: Do not confuse this PR with
https://github.com/matrix-org/synapse/issues/9453. This PR was made
while we were populating this repo with "Dummy issues" after the
changeover from matrix-org/synapse to element-hq/synapse - therefore
referencing this PR may cause some confusion.
2024-01-02 16:10:19 +00:00
Christian Lölkes
35b6365317
Add another custom statistics collection server (#16769)
Signed-off-by: Christian Lölkes <christian.loelkes@gmail.com>
2024-01-02 11:54:12 +00:00
Dirk Klimpel
14ed84ac33
Enable user without password (#16770)
Closes:
- https://github.com/matrix-org/synapse/issues/10397
- #10397 

An administrator should know whether he wants to set a password or not.
There are many uses cases where a blank password is required.

- Use of only some users with SSO.
- Use of bots with password, users with SSO
2024-01-02 11:52:51 +00:00
Fredrik Lanker
c1fe945dd5
Remove config value from header (#16763)
Signed-off-by: Fredrik Lanker <fredrik@lanker.se>
2024-01-02 11:50:50 +00:00
Erik Johnston
8a50312099
Move the rust stubs inline for better IDE integration (#16757)
At least for vscode this allows click through / type checking / syntax
highlighting.
2023-12-21 13:31:36 +00:00
Erik Johnston
719014d9d5
Fix sample config doc CI (#16758)
I accidentally broke it during the move by removing a trailing new line.
2023-12-21 13:31:19 +00:00