Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
21,368 Commits 4 Branches 615 Tags 63 MiB
f7a77ad717
Commit Graph

6441 Commits

Author SHA1 Message Date
Eric Eastwood
1a209efdb2
Update get_users_in_room mis-use to get hosts with dedicated get_current_hosts_in_room (#13605) 
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
 2022-08-24 14:15:37 -05:00
Eric Eastwood
d58615c82c
Directly lookup local membership instead of getting all members in a room first (get_users_in_room mis-use) (#13608) 
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
 2022-08-24 14:13:12 -05:00
Eric Eastwood
b93bd95e8a
When loading current ids, sort by stream_id to avoid incorrect overwrite and avoid errors caused by sorting alphabetical instance name which can be null (#13585) 
When loading current ids, sort by stream ID so that we don't want to overwrite the `current_position` of an instance to a lower stream ID than we're actually at ([discussion](https://github.com/matrix-org/synapse/pull/13585#discussion_r951795379)). Previously, it sorted alphabetically by instance name which can be `null` and throw errors but more importantly, accomplishes nothing.

Fixes the following startup error which is why I started looking into this area:

```
$ poetry run synapse_homeserver --config-path homeserver.yaml
****************************************************************
 Error during initialisation:
    '<' not supported between instances of 'NoneType' and 'str'
 There may be more information in the logs.
****************************************************************
```

Somehow my database ended up looking like the following, notice the `instance_name` is `null` in the db, and we can't sort `NoneType` things. Another question is why do we see the `instance_name` as `null` sometimes instead of `master` in monolith mode?
```
$ psql synapse
synapse=# SELECT * FROM stream_positions;
   stream_name   | instance_name | stream_id
-----------------+---------------+-----------
 account_data    | master        |      1242
 events          | master        |      1787
 to_device       | master        |        58
 presence_stream | master        |    485638
 receipts        | master        |       341
 backfill        | master        |   -139106
(6 rows)
synapse=# SELECT instance_name, stream_id FROM receipts_linearized;
 instance_name | stream_id
---------------+-----------
               |       211
               |         3
               |         4
               |       212
               |       213
               |       224
               |       228
               |       164
               |       313
               |       253
               |        38
               |       321
               |       324
               |       189
               |       192
               |       193
               |       194
               |       195
               |       197
               |       198
               |       275
               |        79
               |       339
               |       340
               |        82
               |       341
               |        84
               |        85
               |        91
               |       119
```
 2022-08-24 12:53:46 -05:00
Eric Eastwood
c807b814ae
Use dedicated get_local_users_in_room to find local users when calculating join_authorised_via_users_server of a /make_join request (#13606) 
Use dedicated `get_local_users_in_room` to find local users when calculating `join_authorised_via_users_server` ("the authorising user for joining a restricted room") of a `/make_join` request.

Found while working on https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755 but it's not related.
 2022-08-24 11:14:28 -05:00
reivilibre
be4250c7a8
Add experimental configuration option to allow disabling legacy Prometheus metric names. (#13540) 
Co-authored-by: David Robertson <davidr@element.io>
 2022-08-24 11:35:54 +00:00
Nick Mills-Barrett
b687010f89
Rewrite get push actions queries (#13597) 2022-08-24 10:12:51 +01:00
reivilibre
ba882c0357
Faster Room Joins: fix /make_knock blocking indefinitely when the room in question is a partial-stated room. (#13583) 
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2022-08-24 09:09:59 +00:00
Eric Eastwood
7af07f9716
Instrument _check_sigs_and_hash_and_fetch to trace time spent in child concurrent calls (#13588) 
Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls because I've see `_check_sigs_and_hash_and_fetch` take [10.41s to process 100 events](https://github.com/matrix-org/synapse/issues/13587)

Fix https://github.com/matrix-org/synapse/issues/13587

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-23 21:53:37 -05:00
David Robertson
a25a37002c
Write about the chain cover a little. (#13602) 
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2022-08-23 17:41:55 +00:00
Erik Johnston
f7ddfe17a3
Speed up @cachedList (#13591) 
This speeds things up by ~2x.

The vast majority of the time is now spent in `LruCache` moving things around the linked lists.

We do this via two things:
1. Don't create a deferred per-key during bulk set operations in `DeferredCache`. Instead, only create them if a subsequent caller asks for the key.
2. Add a bulk lookup API to `DeferredCache` rather than use a loop.
 2022-08-23 14:53:27 +00:00
Erik Johnston
05c9c7363b
Fix regression caused by #13573 (#13600) 
Broke in #13573.
 2022-08-23 14:14:05 +00:00
David Robertson
ca3d19b05f Synapse 1.66.0rc1 (2022-08-23) 
==============================
 
 This release removes the ability for homeservers to delegate email ownership
 verification and password reset confirmation to identity servers. This removal
 was originally planned for Synapse 1.64, but was later deferred until now.
 
 See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.
 
 Features
 --------
 
 - Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](https://github.com/matrix-org/synapse/issues/13188), [\#13563](https://github.com/matrix-org/synapse/issues/13563))
 - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](https://github.com/matrix-org/synapse/issues/13503))
 - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852). ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
 - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716).  ([\#13551](https://github.com/matrix-org/synapse/issues/13551))
 - Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537))
 - Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522), [\#13547](https://github.com/matrix-org/synapse/issues/13547))
 
 Bugfixes
 --------
 
 - Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514))
 - Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525))
 - Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566))
 - Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574))
 
 Improved Documentation
 ----------------------
 
 - Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472))
 - Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491))
 - Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492))
 - Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497))
 - Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515))
 - Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536))
 - Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538))
 
 Deprecations and Removals
 -------------------------
 
 - Remove the ability for homeservers to delegate email ownership verification
   and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.
 
 Internal Changes
 ----------------
 
 - Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459))
 - Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477))
 - Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531))
 
 - Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453))
 - Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489))
 - Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499), [\#13554](https://github.com/matrix-org/synapse/issues/13554))
 - Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533))
 - Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534), [\#13541](https://github.com/matrix-org/synapse/issues/13541))
 - Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535), [\#13584](https://github.com/matrix-org/synapse/issues/13584))
 - Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544))
 - Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545))
 
 - Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024))
 - Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471))
 - Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474))
 - Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479), [\#13488](https://github.com/matrix-org/synapse/issues/13488))
 - Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485))
 - Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493))
 - Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502))
 - Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543))
 - Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
 - Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMExBwACgkQkD7OEIo5
 3t0sGA/9FLZLdTWeROLUenbInvz3POqi6DgqlE0f8IDdp3zHjjuDJIE4HZwnPhhj
 c3H5FcUnTPhQoVZ8Ybno3J3KN8jfpe4tFhnhlM1x9hOi1fujfqCnFd2qWvlzkYXX
 J4oN6Dz0BbhqdhqnS2tcNacZJOielJ47UtNNBZkzprpzjL3qqRUK3Hm/gGK8GtGN
 BLon7JK0pKG1GFMVaoIWsr+j7880QD8DGXTjoZrjo/yGdp+zJldwJTbN2ymNdMw1
 3Caq7eCk+m6jzCUJxI7tTJVioXvf7+TScIsozAXuQtBy9W1RctvVMMoR9cIlN+JJ
 dj5GKhKBv8uJpJU6dJnlYpjNC5+5hPHr1gfnkoYbcyFF6PybJrHx9jfhEIEXmI1+
 j0DlJJdFnX+eeq4deFd+LPU5hYaF8SkSVRK3G+/NmKI3YYTReD6ootZzV5O4Z9P0
 NUwLK/OSMZrsFUfuwysUttfHq0uHGGs7mytlRgeqb/1RKRC80nxVJu3qv/xF6kKj
 KF98Yr2IDgp/MFhBpGAO3ylNgK8aQ3YxED4qW+kfnDeHPkma5mTdAtJ1bpvKcec+
 3awpFGWcmYVQPflwp9J3HFJHqWkj32LMfuSgDffLs8mWuj87fP7eAdjzv6M4zeAM
 4qh+cmvNC7ArfQ+ngz3g+xVD1lHi84s+fMgo7L+ySGfgDqTLhT4=
 =1F3P
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMEz10ACgkQkD7OEIo5
 3t02GhAApioIpEedKaeSATI42N8rWhfuni3vaPFAXwmOD6BvBkZlP0gcg+vUoBjP
 I0VIkGWZ3MKIhCSJkjwPJh1vHcpE09KYqIYRKO5vSbqHZKHYUJ+ThjcCR4IkOBPD
 3Dc3IRgQaM2Nh8eMcyRY7eyxo+2y+VN2G+SXr1B+A5/4N58ZrxfV97MtLYJ1a6AH
 0iIknOxb9FPArYs4ORbk6x2rvRO2QPGh88uKwJ8uHXvHlUbJ3fmHLVvRa1m0RKAW
 ojkzj/Dysa+4gasMIvtzPePtIQnMyovaQLul+bGXzE0js4lqzSOIP12x1zNpLuuq
 LSiyD3gMgGeogumhCAU6b23xF6jBaGtgtKKpaYuK8pqkuBTRNN8kQ6LB+7g3WQ2L
 QWVos+/+wKUEHoVVD/q3u7o4CrKYqv4zE9Pj6kw5e1USOoy7803ub2ccjY1UgG3M
 mZ7c48WNQQ61o9oH2+9+Ct6bab3R/Lo1MbS3zIKN8uktB4CdrhVAz3kYgV5Bl8bo
 PsldxUccOTTeS5qmUGvm2/szVKjdqyCWRx9AY9eUH9Kc4u/GzmNq0UVOZF8yxM0q
 WEo7GFgjM3k9IIC3K/fSmSC1iBjPSHpYvqO+c8oSNnjq2iucZeIhiEB07e89+KSa
 Wiz9DDWP8Z7W6D9wRAeXyPJFB6ZdSSBgP1I90z6lQjSMzy46Kv4=
 =pJ2V
 -----END PGP SIGNATURE-----

Merge tag 'v1.66.0rc1' into develop

Synapse 1.66.0rc1 (2022-08-23)
==============================

This release removes the ability for homeservers to delegate email ownership
verification and password reset confirmation to identity servers. This removal
was originally planned for Synapse 1.64, but was later deferred until now.

See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.

Features
--------

- Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](https://github.com/matrix-org/synapse/issues/13188), [\#13563](https://github.com/matrix-org/synapse/issues/13563))
- Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](https://github.com/matrix-org/synapse/issues/13503))
- Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852). ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
- Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716).  ([\#13551](https://github.com/matrix-org/synapse/issues/13551))
- Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537))
- Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522), [\#13547](https://github.com/matrix-org/synapse/issues/13547))

Bugfixes
--------

- Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514))
- Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525))
- Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566))
- Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574))

Improved Documentation
----------------------

- Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472))
- Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491))
- Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492))
- Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497))
- Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515))
- Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536))
- Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538))

Deprecations and Removals
-------------------------

- Remove the ability for homeservers to delegate email ownership verification
  and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.

Internal Changes
----------------

- Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459))
- Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477))
- Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531))

- Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453))
- Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489))
- Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499), [\#13554](https://github.com/matrix-org/synapse/issues/13554))
- Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533))
- Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534), [\#13541](https://github.com/matrix-org/synapse/issues/13541))
- Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535), [\#13584](https://github.com/matrix-org/synapse/issues/13584))
- Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544))
- Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545))

- Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024))
- Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471))
- Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474))
- Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479), [\#13488](https://github.com/matrix-org/synapse/issues/13488))
- Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485))
- Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493))
- Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502))
- Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543))
- Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
- Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558))
 2022-08-23 14:00:09 +01:00
Erik Johnston
aec87a0f93
Speed up fetching large numbers of push rules (#13592) 2022-08-23 13:15:43 +01:00
David Robertson
ea85a2bf6c
Remove manually-added changelog 2022-08-23 12:40:35 +01:00
David Robertson
956e015413
Drop support for delegating email validation, round 2 (#13596) 2022-08-23 11:40:00 +00:00
Nick Mills-Barrett
5e7847dc92
Cache user IDs instead of profile objects (#13573) 
The profile objects are never used and increase cache size significantly.
 2022-08-23 09:49:59 +00:00
David Robertson
f0b23927fc
1.66.0rc1 2022-08-23 09:49:51 +01:00
Dirk Klimpel
37f329c9ad
Fix that sending server notices fail if avatar is None (#13566) 
Indroduced in #11846.
 2022-08-23 09:48:35 +01:00
Eric Eastwood
9385c41ba4
Fix Prometheus metrics being negative (mixed up start/end) (#13584) 
Fix:

 - https://github.com/matrix-org/synapse/pull/13535#discussion_r949582508
 - https://github.com/matrix-org/synapse/pull/13533#discussion_r949577244
 2022-08-23 08:47:30 +01:00
Quentin Gliech
3dd175b628
synapse.api.auth.Auth cleanup: make permission-related methods use Requester instead of the UserID (#13024) 
Part of #13019

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
 2022-08-22 14:17:59 +01:00
Andrew Morgan
94375f7a91
Remove redundant opentracing spans for /sendToDevice and /keys/upload (#13574) 2022-08-22 10:03:11 +01:00
Eric Eastwood
06df5d4250
MSC2716v4 room version - remove namespace from MSC2716 event content fields (#13551) 
Complement PR: https://github.com/matrix-org/complement/pull/450

As suggested in
https://github.com/matrix-org/matrix-spec-proposals/pull/2716#discussion_r941444525
 2022-08-19 15:37:01 -05:00
Andrew Morgan
f9f03426de
Implement MSC3852: Expose last_seen_user_agent to users for their own devices; also expose to Admin API (#13549) 2022-08-19 16:17:10 +00:00
Andrew Morgan
40e3e68cd7
Register homeserver modules when creating test homeserver (#13558) 2022-08-19 16:52:20 +01:00
Patrick Cloke
f3fba4914d
Reduce the number of tests using TCP replication. (#13543) 
Uses Redis replication in additional test cases (instead of
TCP replication). A small step towards dropping TCP replication.
 2022-08-19 08:25:24 -04:00
reivilibre
3a245f6cfe
Fix validation problem that occurs when a user tries to deactivate their account or change their password. (#13563) 2022-08-19 11:03:29 +00:00
Eric Eastwood
2c42673a9b
Add metrics to track /messages response time by room size (#13545) 
Follow-up to https://github.com/matrix-org/synapse/pull/13533

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-18 14:15:53 -05:00
Sean Quah
b251cff819
Fix incorrect juggling of logging contexts in _PerHostRatelimiter (#13554) 
Signed-off-by: Sean Quah <seanq@matrix.org>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2022-08-18 16:26:26 +01:00
Eric Eastwood
d64653d062
Track number of hosts affected by the rate limiter (#13541) 
Track number of hosts affected by the rate limiter so we can differentiate one really noisy homeserver from a general ratelimit tuning problem across the federation.

Follow-up to https://github.com/matrix-org/synapse/pull/13534

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-18 10:05:07 -05:00
Ayush Anand
22ea51faf9
Add support for compression to federation responses (#13537) 
Closes #13415.

Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
 2022-08-18 15:14:47 +01:00
Sean Quah
84169a82dc
Avoid blocking lazy-loading /syncs during partial joins (#13477) 
Use a state filter or accept partial state in a few places where we
request state, to avoid blocking.

To make lazy-loading `/sync`s work, we need to provide the memberships
of event senders, which are not guaranteed to be in the room state.
Instead we dig through auth events for memberships to present to
clients. The auth events of an event are guaranteed to contain a
passable membership event, otherwise the event would have been rejected.

Note that this only covers the common code paths encountered during
testing. There has been no exhaustive checking of all sync code paths.

Fixes #13146.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-08-18 11:53:02 +01:00
Eric Eastwood
49d04e43df
Add metrics to track how the rate limiter is affecting requests (sleep/reject) (#13534) 
Related to https://github.com/matrix-org/synapse/pull/13499

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-17 16:10:07 -05:00
reivilibre
8bdf2bd31e
Fix a bug in the /event_reports Admin API which meant that the total count could be larger than the number of results you can actually query for. (#13525) 
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-08-17 18:08:23 +00:00
David Robertson
82a0752f32
A first pass at pruning the Synapse README (#13491) 2022-08-17 16:46:09 +01:00
Erik Johnston
436e0eb39a
Fix breaking event sending due to bad push rule (#13547) 
Broke by #13522

It looks like we have some rules in the DB with a priority class less
than 0 that don't override the base rules. Before these were just
dropped, but #13522 made that a hard error.
 2022-08-17 11:02:38 +00:00
David Robertson
ba8938b090
Reject non-strict types in Pydantic models (#13502) 2022-08-17 11:17:04 +01:00
Antonin Loubiere
b71b41c7bd
Fix a typo in docs and in some warnings (#13538) 2022-08-17 10:59:05 +01:00
Dirk Klimpel
d75512d19e
Add forgotten status to Room Details API (#13503) 2022-08-17 09:42:01 +00:00
Eric Eastwood
c6ee9c0ee4
Add metrics to track rate limiter queue timing (#13544) 2022-08-17 10:38:05 +01:00
Eric Eastwood
088bcb7ecb
Time how long it takes us to do backfill processing (#13535) 2022-08-17 10:33:19 +01:00
Eric Eastwood
2c8cfd6d85
Add specific metric to time long-running /messages requests (#13533) 2022-08-17 10:19:21 +01:00
Eric Eastwood
0a4efbc1dd
Instrument the federation/backfill part of /messages (#13489) 
Instrument the federation/backfill part of `/messages` so it's easier to follow what's going on in Jaeger when viewing a trace.

Split out from https://github.com/matrix-org/synapse/pull/13440

Follow-up from https://github.com/matrix-org/synapse/pull/13368

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-16 12:39:40 -05:00
Sean Quah
f4ab6a4a96
Refuse to start when faster_joins is enabled on a worker deployment (#13531) 
Synapse does not currently support faster room joins on deployments with
workers.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-08-16 16:21:55 +01:00
Dirk Klimpel
14e673ef9d
Add missing links in user_consent section of configuration manual (#13536) 
Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
 2022-08-16 16:20:56 +01:00
reivilibre
c3516e9dec
Faster room joins: make /joined_members block whilst the room is partial stated. (#13514) 2022-08-16 13:16:56 +01:00
Erik Johnston
5442891cbc
Make push rules use proper structures. (#13522) 
This improves load times for push rules:

| Version              | Time per user | Time for 1k users | 
| -------------------- | ------------- | ----------------- |
| Before               |       138 µs  |             138ms |
| Now (with custom)    |       2.11 µs |            2.11ms |
| Now (without custom) |       49.7 ns |           0.05 ms |

This therefore has a large impact on send times for rooms
with large numbers of local users in the room.
 2022-08-16 12:22:17 +01:00
David Robertson
d642ce4b32
Use Pydantic to systematically validate a first batch of endpoints in synapse.rest.client.account. (#13188) 2022-08-15 19:05:57 +00:00
Shay
73c83c6411
Add a warning to retention documentation regarding the possibility of database corruption (#13497) 2022-08-15 11:54:23 -07:00
Eric Eastwood
344a2f767c
Instrument FederationStateIdsServlet - /state_ids (#13499) 
Instrument FederationStateIdsServlet - `/state_ids` so it's easier to follow what's going on in Jaeger when viewing a trace.
 2022-08-15 19:41:23 +01:00
David Robertson
19e5d44886
Revert "Update locked versions of mypy and mypy-zope (#13521)" 
This reverts commit f383b9b3ec. Other PRs
were seeing mypy failures that looked to be related to mypy-zope.
Confusingly, we didn't see this on #13521.

Revert this for now and investigate later.
 2022-08-15 14:51:05 +01:00
Patrick Cloke
46bd7f4ed9
Clarifications for event push action processing. (#13485) 
* Clarifies comments.
* Fixes an erroneous comment (about return type) added in #13455
  (ec24813220).
* Clarifies the name of a variable.
* Simplifies logic of pulling out the latest join for the requesting user.
 2022-08-15 09:33:17 -04:00
David Robertson
f383b9b3ec
Update locked versions of mypy and mypy-zope (#13521) 2022-08-15 11:32:30 +01:00
reivilibre
2c5e2ae898
Document that the DOCKER_BUILDKIT=1 flag is needed to build the docker image. (#13515) 2022-08-12 14:25:47 +01:00
James Barton
e825f7366b
Add openssl example for registration HMAC (#13472) 
Signed-off-by: James Barton <james@neodon.com>
 2022-08-11 21:10:10 +01:00
Andrew Morgan
953df2ad88
Add note to redaction_retention_period documentation mentioning that event purging runs at most every 5m (#13492) 2022-08-11 14:06:09 +01:00
Germain
4390121684
Add viewport directive to HTML templates to optimise for mobile (#13493) 2022-08-11 14:04:20 +01:00
Richard van der Hoff
507c1cb330
Update the rejected state of events during resync (#13459) 
Events can be un-rejected or newly-rejected during resync, so ensure we update
the database and caches when that happens.
 2022-08-11 10:42:24 +00:00
Dirk Klimpel
2281427175
Use literals in place of HTTPStatus constants in tests (#13488) 
* Use literals in place of `HTTPStatus` constants in tests

* newsfile

* code style

* code style
 2022-08-10 11:01:12 -07:00
Sean Quah
51c01d450a
Add some miscellaneous comments around sync (#13474) 
Add some miscellaneous comments to document sync, especially around
`compute_state_delta`.

Signed-off-by: Sean Quah <seanq@matrix.org>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2022-08-10 12:58:20 +01:00
Eric Eastwood
1b09b0832e
Allow use of both @trace and @tag_args stacked on the same function (#13453) 
```py
@trace
@tag_args
async def get_oldest_event_ids_with_depth_in_room(...)
  ...
```

Before this PR, you would see a warning in the logs and the span was not exported:
```
2022-08-03 19:11:59,383 - synapse.logging.opentracing - 835 - ERROR - GET-0 - @trace may not have wrapped EventFederationWorkerStore.get_oldest_event_ids_with_depth_in_room correctly! The function is not async but returned a coroutine.
```
 2022-08-09 14:32:33 -05:00
Dirk Klimpel
1595052b26
Use literals in place of HTTPStatus constants in tests (#13479) 
Replace
- `HTTPStatus.NOT_FOUND`
- `HTTPStatus.FORBIDDEN`
- `HTTPStatus.UNAUTHORIZED`
- `HTTPStatus.CONFLICT`
- `HTTPStatus.CREATED`

Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
 2022-08-09 14:56:43 +01:00
Olivier Wilkinson (reivilibre)
54fb517c28 Merge branch 'release-v1.65' into develop 2022-08-09 13:45:11 +01:00
Patrick Cloke
5ce2887653
Strengthen tests about deleted old push actions. (#13471) 2022-08-09 07:56:16 -04:00
Olivier Wilkinson (reivilibre)
827f0669bf 1.65.0rc1 2022-08-09 11:43:30 +01:00
Dirk Klimpel
70d3e70009
Make the configuration for the cache clearer (#13481) 2022-08-09 11:30:22 +01:00
Shay
0f954466c4
Update matrix-synapse-ldap3 version in lockfile to v0.2.2. (#13470) 2022-08-08 13:44:43 -07:00
Dirk Klimpel
c97042f7ee
Use literals in place of HTTPStatus constants in tests (#13469) 2022-08-08 13:21:27 -07:00
David Robertson
7a19995120
Correct a misnamed argument in state res v2 (#13467) 
In state res v2, we apply two passes of iterative auth checks. The first
pass replays power events and events in their auth chains, but only
those belonging to the full conflicted set. The source code as written
suggests that we want only those belonging to the auth difference (which
is a smaller set of events).

At runtime we were doing the correct thing anyway, because the only
callsite of `_reverse_topological_power_sort` passes in the
`full_conflicted_set`. So this really is just a rename.
 2022-08-08 16:59:56 +01:00
Šimon Brandner
ab18441573
Support stable identifiers for MSC2285: private read receipts. (#13273) 
This adds support for the stable identifiers of MSC2285 while
continuing to support the unstable identifiers behind the configuration
flag. These will be removed in a future version.
 2022-08-05 11:09:33 -04:00
Dirk Klimpel
e2ed1b7155
Use literals in place of HTTPStatus constants in tests (#13463) 2022-08-05 16:59:09 +02:00
Julian-Samuel Gebühr
3d2cabf966
Mark token-authenticaticated-registration API as not-experimental (#11897) 2022-08-05 11:15:35 +00:00
Matt C
026ac4486c
Update module API "update room membership" method to allow for remote joins (#13441) 
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-08-05 09:37:58 +00:00
Erik Johnston
b6a6bb4027
Add comments about how event push actions are stored. (#13445) 2022-08-04 19:38:08 +00:00
Eric Eastwood
860fdd9098
Fix @tag_args being off-by-one (ahead) (#13452) 
Fix @tag_args being off-by-one (ahead)

Example:

```
argspec.args=[
  'self',
  'room_id'
]

args=(
  <synapse.storage.databases.main.DataStore object at 0x10d0b8d00>,
  '!HBehERstyQBxyJDLfR:my.synapse.server'
)
```

---

The previous logic was also flawed and we can end up in a situation like this:

```
argspec.args=['self', 'dest', 'room_id', 'limit', 'extremities']

args=(<synapse.federation.federation_client.FederationClient object at 0x7f1651c18160>, 'hs1', '!jAEHKIubyIfuLOdfpY:hs1')
```

From this source:
```py
async def backfill(
    self, dest: str, room_id: str, limit: int, extremities: Collection[str]
) -> Optional[List[EventBase]]:
```

And this usage:
```py
events = await self._federation_client.backfill(
    dest, room_id, limit=limit, extremities=extremities
)
```

which would previously cause this error:
```
synapse_main | 2022-08-04 06:13:12,051 - synapse.handlers.federation - 424 - ERROR - GET-5 - Failed to backfill from hs1 because tuple index out of range
synapse_main | Traceback (most recent call last):
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation.py", line 392, in try_backfill
synapse_main |     await self._federation_event_handler.backfill(
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper
synapse_main |     return await func(*args, **kwargs)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 593, in backfill
synapse_main |     events = await self._federation_client.backfill(
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper
synapse_main |     return await func(*args, **kwargs)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 827, in _wrapper
synapse_main |     with wrapping_logic(func, *args, **kwargs):
synapse_main |   File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
synapse_main |     return next(self.gen)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 922, in _wrapping_logic
synapse_main |     set_attribute("ARG_" + arg, str(args[i + 1]))  # type: ignore[index]
synapse_main | IndexError: tuple index out of range
```
 2022-08-04 14:29:41 -05:00
Patrick Cloke
ec24813220
Improve comments (& avoid a duplicate query) in push actions processing. (#13455) 
* Adds docstrings and inline comments.
* Formats SQL queries using triple quoted strings.
* Minor formatting changes.
* Avoid fetching `event_push_summary_stream_ordering` multiple times
  in the same transactions.
 2022-08-04 19:24:44 +00:00
Richard van der Hoff
96d92156d0
Update type of EventContext.rejected (#13460) 2022-08-04 17:45:01 +01:00
reivilibre
e9e6aacfbe
Faster Room Joins: prevent Synapse from answering federated join requests for a room which it has not fully joined yet. (#13416) 2022-08-04 16:27:04 +01:00
Nick Mills-Barrett
41320a0554
Optimise async get event lookups (#13435) 
Still maintains local in memory lookup optimisation, but does any external
lookup as part of the deferred that prevents duplicate lookups for the same
event at once. This makes the assumption that fetching from an external
cache is a non-zero load operation.
 2022-08-04 15:49:55 +01:00
Dirk Klimpel
6dd7fa12dc
Update some outdated information on sso_mapping_providers.md (#13449) 2022-08-04 13:06:02 +01:00
Dirk Klimpel
afbdbe0634
Fix return value in example on password_auth_provider_callbacks.md (#13450) 
Fixes: #12534

Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
 2022-08-04 13:03:36 +01:00
Richard van der Hoff
166fafdf8d
synapse-workers docker: copy nginx and redis in from base images (#13447) 
Part of my continuing quest to make the docker images build quicker: copy nginx and redis in from base docker images, rather than apt installing each time.
 2022-08-04 12:59:27 +01:00
Matt C
a91078200d
Add module API method to create a room (#13429) 
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-08-04 09:34:05 +00:00
Brendan Abolivier
845732be45
Fix rooms not being properly excluded from incremental sync (#13408) 2022-08-04 11:02:29 +02:00
Shay
a648a06d52
Add some tracing spans to give insight into local joins (#13439) 2022-08-03 10:19:34 -07:00
Eric Eastwood
92d21faf12
Instrument /messages for understandable traces in Jaeger (#13368) 
In Jaeger:

 - Before: huge list of uncategorized database calls
 - After: nice and collapsible into units of work
 2022-08-03 10:57:38 -05:00
andrew do
78a3111c41
Return 404 or member list when getting joined_members after leaving (#13374) 
Signed-off-by: Andrew Doh <andrewddo@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrewm@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-08-03 14:26:31 +02:00
Jasper Spaans
503a95804e
Install cryptography build dependencies in requirements image. (#13372) 2022-08-03 11:16:32 +01:00
jejo86
668597214f
Improve documentation on becoming server admin (#13230) 
* Improved section regarding server admin

Added steps describing how to elevate an existing user to administrator by manipulating a `postgres` database.

Signed-off-by: jejo86 28619134+jejo86@users.noreply.github.com

* Improved section regarding server admin

* Reference database settings

Add instructions to check database settings to find out the database name, instead of listing all available PostgreSQL databases.

* Add suggestions from PR conversation

Replace config filename `homeserver.yaml`. with "config file".
Remove instructions to switch to `postgres` user.
Add instructions how to connect to SQLite database.

* Update changelog.d/13230.doc

Co-authored-by: reivilibre <olivier@librepush.net>
 2022-08-03 11:15:23 +01:00
Dirk Klimpel
fb7a2cc4cc
Update doc for setting macaroon_secret_key (#13443) 
* Update doc for setting `macaroon_secret_key`

* newsfile
 2022-08-03 10:41:19 +01:00
Dirk Klimpel
d6e94ad9d9
Rename RateLimitConfig to RatelimitSettings (#13442) 2022-08-03 10:40:20 +01:00
Matt C
570bf32bbb
Add module API method to resolve a room alias to a room ID (#13428) 
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-08-03 09:25:36 +00:00
Dirk Klimpel
5eccfdfafd
Remove 'Contents' section from the Configuration Manual (#13438) 
Fixes: #13053
 2022-08-03 09:19:20 +00:00
Dirk Klimpel
ec6758d472
Fix wrong headline for url_preview_accept_language in docs (#13437) 
Fixes: #13433
 2022-08-03 09:41:57 +01:00
reivilibre
1c910e2216
Add a merge-back command to the release script, which automates merging the correct branches after a release. (#13393) 2022-08-02 15:56:28 +00:00
Sean Quah
8d317f6da5
Fix error when out of servers to sync partial state with (#13432) 
so that we raise the intended error instead.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-08-02 12:12:44 +01:00
reivilibre
e17e5c97e0
Faster Room Joins: don't leave a stuck room partial state flag if the join fails. (#13403) 2022-08-01 16:45:39 +00:00
Patrick Cloke
f8e7a9418a
Fix missing import in federation_event handler. (#13431) 
#13404 removed an import of `Optional` which was still needed
due to #13413 added more usages.
 2022-08-01 14:14:29 +00:00
Sean Quah
224d792dd7
Refactor _resolve_state_at_missing_prevs to return an EventContext (#13404) 
Previously, `_resolve_state_at_missing_prevs` returned the resolved
state before an event and a partial state flag. These were unwieldy to
carry around would only ever be used to build an event context. Build
the event context directly instead.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-08-01 13:53:56 +01:00
reivilibre
05aeeb3a80
Enable Complement CI tests in the 'latest deps' test run. (#13213) 
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2022-08-01 10:55:31 +00:00
reivilibre
b817574be7
Re-enable running Complement tests against Synapse with workers. (#13420) 2022-08-01 11:51:44 +01:00
Richard van der Hoff
23768ccb4d
Faster joins: fix rejected events becoming un-rejected during resync (#13413) 
Make sure that we re-check the auth rules during state resync, otherwise
rejected events get un-rejected.
 2022-08-01 11:20:05 +01:00
Brendan Abolivier
24ef1460f6
Explicitly mention which resources support compression in the config guide (#13221) 2022-07-29 09:09:57 +00:00
Šimon Brandner
583f22780f
Use stable prefixes for MSC3827: filtering of /publicRooms by room type (#13370) 
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
 2022-07-27 19:46:57 +01:00
Patrick Cloke
922b771337
Add missing type hints for tests.unittest. (#13397) 2022-07-27 17:18:41 +00:00
Will Hunt
502f075e96
Implement MSC3848: Introduce errcodes for specific event sending failures (#13343) 
Implements MSC3848
 2022-07-27 13:44:40 +01:00
reivilibre
39be5bc550
Make minor clarifications to the error messages given when we fail to join a room via any server. (#13160) 2022-07-27 10:37:50 +00:00
Eric Eastwood
4f3082d6bf
Fix get_pdu asking every remote destination even after it finds an event (#13346) 2022-07-27 10:40:04 +01:00
Nick Mills-Barrett
bf3115584c
Copy room serials before handling in get_new_events_as (#13392) 2022-07-26 17:45:27 +00:00
reivilibre
543dc9c93e
Extend the release script to automatically push a new SyTest branch, rather than having that be a manual process. (#12978) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2022-07-26 18:08:14 +01:00
Olivier Wilkinson (reivilibre)
6236afc621 Merge branch 'release-v1.64' into develop 2022-07-26 16:26:30 +01:00
Patrick Cloke
57d334a13d
Remove the unspecced room_id field in the /hierarchy response. (#13365) 
The `room_id` field represented the parent space for each room
and was made redundant by changes in the API shape where the
`children_state` is now nested underneath each `room`.

The room ID of each child is in the `state_key` field and is still
available.
 2022-07-26 08:02:34 -04:00
Richard van der Hoff
ca3db044a3
Fix infinite loop in partial-state resync (#13353) 
Make sure that we only pull out events from the db once they have no
prev-events with partial state.
 2022-07-26 11:47:31 +00:00
Sean Quah
335ebb21cc
Faster room joins: avoid blocking when pulling events with missing prevs (#13355) 
Avoid blocking on full state in `_resolve_state_at_missing_prevs` and
return a new flag indicating whether the resolved state is partial.
Thread that flag around so that it makes it into the event context.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2022-07-26 12:39:23 +01:00
Patrick Cloke
8b603299bf
Remove unused argument for get_relations_for_event. (#13383) 2022-07-26 07:19:20 -04:00
Olivier Wilkinson (reivilibre)
641412decd 1.64.0rc1 2022-07-26 12:12:22 +01:00
Doug
549c55606a
Disable autocorrect and autocaptialisation when entering username for SSO registration. (#13350) 
When registering a new account via SSO on iOS, the text field becomes pretty annoying as it autocapitalises and autocorrects your input. This PR fixes that (although I have only tested the raw HTML file on the simulator, I'm not sure how to get the complete setup available for testing in the flow).
 2022-07-26 08:08:20 +00:00
Matt Holt
935e73efed
Update Caddy reverse proxy documentation (#13344) 
Improve/simplify Caddy examples. Remove Caddy v1 (has long been EOL'ed)

Signed-off-by: Matthew Holt <mholt@users.noreply.github.com>
 2022-07-25 16:07:26 +00:00
Jan Schär
e8519e0ed2
Support Implicit TLS for sending emails (#13317) 
Previously, TLS could only be used with STARTTLS.
Add a new option `force_tls`, where TLS is used from the start.
Implicit TLS is recommended over STARTLS,
see https://datatracker.ietf.org/doc/html/rfc8314

Fixes #8046.

Signed-off-by: Jan Schär <jan@jschaer.ch>
 2022-07-25 16:27:19 +01:00
Patrick Cloke
908aeac44a
Additional fixes for opentracing type hints. (#13362) 2022-07-25 08:34:06 -04:00
Erik Johnston
43adf2521c
Refactor presence so we can prune user in room caches (#13313) 
See #10826 and #10786 for context as to why we had to disable pruning on
those caches.

Now that `get_users_who_share_room_with_user` is called frequently only
for presence, we just need to make calls to it less frequent and then we
can remove the various levels of caching that is going on.
 2022-07-25 09:21:06 +00:00
Eric Eastwood
357561c1a2
Backfill remote event fetched by MSC3030 so we can paginate from it later (#13205) 
Depends on https://github.com/matrix-org/synapse/pull/13320

Complement tests: https://github.com/matrix-org/complement/pull/406

We could use the same method to backfill for `/context` as well in the future, see https://github.com/matrix-org/synapse/issues/3848
 2022-07-22 16:00:11 -05:00
Sean Quah
0fa41a7b17
Update locked frozendict version to 2.3.3 (#13352) 
frozendict 2.3.3 includes fixes for memory leaks that get triggered during `/sync`.
 2022-07-22 10:26:09 +01:00
Sean Quah
158782c3ce
Skip soft fail checks for rooms with partial state (#13354) 
When a room has the partial state flag, we may not have an accurate
`m.room.member` event for event senders in the room's current state, and
so cannot perform soft fail checks correctly. Skip the soft fail check
entirely in this case.

As an alternative, we could block until we have full state, but that
would prevent us from receiving incoming events over federation, which
is undesirable.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-07-22 10:13:01 +01:00
Nick Mills-Barrett
86e366a46e
Remove old empty/redundant slaved stores. (#13349) 2022-07-21 17:56:45 +00:00
Erik Johnston
0b87eb8e0c
Make DictionaryCache have better expiry properties (#13292) 2022-07-21 17:13:44 +01:00
Erik Johnston
13341dde5a
Don't hold onto full state in state cache (#13324) 2022-07-21 16:02:02 +01:00
Brendan Abolivier
10e4093839
Call out buildkit is required when building test docker images (#13338) 
Co-authored-by: David Robertson <davidr@element.io>
 2022-07-21 14:29:58 +02:00
David Robertson
34949ead1f
Track DB txn times w/ two counters, not histogram (#13342) 2022-07-21 13:23:05 +01:00
Patrick Cloke
50122754c8
Add missing types to opentracing. (#13345) 
After this change `synapse.logging` is fully typed.
 2022-07-21 12:01:52 +00:00
Nick Mills-Barrett
190f49d8ab
Use cache store remove base slaved (#13329) 
This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
 2022-07-21 11:51:30 +01:00
David Robertson
4f57ef0b18
Merge branch 'master' into develop 2022-07-21 11:27:08 +01:00
David Teller
b909d5327b
Document rc_invites.per_issuer, added in v1.63. 
Resolves #13330.
Missed in #13125.

Signed-off-by: David Teller <davidt@element.io>
 2022-07-21 11:26:34 +01:00
Eric Eastwood
0f971ca68e
Update get_pdu to return the original, pristine EventBase (#13320) 
Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier`  and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call.

Split out from https://github.com/matrix-org/synapse/pull/13205

As discussed at:

 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918365746
 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918366125

Related to https://github.com/matrix-org/synapse/issues/12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](7864f33e28/synapse/federation/federation_client.py (L581-L594)).
 2022-07-20 15:58:51 -05:00
Shay
a1b62af2af
Validate federation destinations and log an error if server name is invalid. (#13318) 2022-07-20 11:17:26 -07:00
Erik Johnston
93740cae57 1.63.1 2022-07-20 13:37:00 +01:00
Erik Johnston
b4ae3b0d44
Don't include appservice users when calculating push rules (#13332) 
This can cause a lot of extra load on servers with lots of appservice users. Introduced in #13078
 2022-07-20 12:06:13 +01:00
Sean Quah
172ce29b14
Fix spurious warning when fetching state after a missing prev event (#13258) 2022-07-19 19:15:54 +01:00
Patrick Cloke
a6895dd576
Add type annotations to trace decorator. (#13328) 
Functions that are decorated with `trace` are now properly typed
and the type hints for them are fixed.
 2022-07-19 14:14:30 -04:00
Erik Johnston
de70b25e84
Reduce memory usage of state group cache (#13323) 2022-07-19 14:40:37 +01:00
Patrick Cloke
1efe6b8c41
Stop building Ubuntu 21.10 (Impish Indri) which is end of life. (#13326) 2022-07-19 09:08:46 -04:00
Brendan Abolivier
097afd0e0b 1.63.0 2022-07-19 14:43:28 +02:00
Andrew Morgan
6faaf76a32
Remove 'anonymised' from the phone home stats documentation (#13321) 2022-07-19 12:38:29 +00:00
villepeh
84c5e6b1fd
Bash script for creating multiple stream writers (#13271) 
Add another bash script to the contrib directory. It creates multiple stream writers and also prints out the example configuration for homeserver.yaml.

Signed-off-by: Ville Petteri Huh.
 2022-07-19 12:37:20 +00:00
Jörg Behrmann
87a917e8c8
Add notes when config options were changed to config documentation (#13314) 
Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
 2022-07-19 12:36:29 +00:00
David Robertson
b977867358
Rate limit joins per-room (#13276) 2022-07-19 11:45:17 +00:00
Nick Mills-Barrett
2ee0b6ef4b
Safe async event cache (#13308) 
Fix race conditions in the async cache invalidation logic, by separating
the async & local invalidation calls and ensuring any async call i
executed first.

Signed off by Nick @ Beeper (@Fizzadar).
 2022-07-19 11:25:29 +00:00
Shay
7864f33e28
Increase batch size of bulk_get_push_rules and _get_joined_profiles_from_event_ids. (#13300) 2022-07-18 13:15:23 -07:00
Shay
15edf23626
Improve performance of query _get_subset_users_in_room_with_profiles (#13299) 2022-07-18 12:35:45 -07:00
Sean Quah
5526f9fc4f
Fix overcounting of pushers when they are replaced (#13296) 
Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-07-18 17:39:39 +01:00
Brendan Abolivier
8c60c572f0
Up the dependency on canonicaljson to ^1.5.0 (#13172) 
Co-authored-by: David Robertson <davidr@element.io>
 2022-07-18 17:30:59 +02:00