Commit Graph

21628 Commits

Author SHA1 Message Date
Hugh Nimmo-Smith
4eaf3eb840
Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-10-18 15:52:25 +00:00
David Robertson
844ce47b9b
Don't pin dev-deps in pyproject; use lower bounds (#14227)
* Don't pin dev-deps in pyproject; use lower bounds

This makes it slightly less tedious to update these things via
successive dependabot updates, by reducing the likelihood of a merge
conflict.

* Changelog

* Changelog
2022-10-18 16:44:43 +01:00
David Robertson
b951d6bd4c
Fixes to release-artifacts warnings (#14224) 2022-10-18 15:40:05 +00:00
Patrick Cloke
dbf18f514e
Update the thread_id right before use (in case the bg update hasn't finished) (#14222)
This avoids running a forced-update of a null thread_id rows.

An index is added (in the background) to hopefully make this
easier in the future.
2022-10-18 14:55:41 +00:00
Jonathan de Jong
e440f9674a
Enable URL previews in complement homeserver config. (#14198) 2022-10-18 09:52:23 -04:00
David Robertson
8e50299d8b
Fix track_memory_usage on poetry-core 1.3.x installations (#14221)
* Fix `track_memory_usage` on poetry-core 1.3.x installations

The same kind of problem as discussed in #14085:

1. we defined an extra with an underscore
2. we look it up at runtime with an underscore
3. but poetry-core 1.3.x. installs it with a dash, causing (2) to fail.

Fix by using a dash everywhere.

* Changelog
2022-10-18 13:59:04 +01:00
David Robertson
a8677bc9b8
Deal with some GHA deprecation warnings (#14216) 2022-10-18 13:45:34 +01:00
realtyem
6c5082f3e0
Flush stdout/err in Dockerfile-workers before replacing the current process (#14195)
Also update `subprocess.check_output` to the slightly newer `subprocess.run`.

Signed-off-by: Jason Little <realtyem@gmail.com>
2022-10-18 11:56:20 +00:00
David Robertson
c3a4780080
When restarting a partial join resync, prioritise the server which actioned a partial join (#14126) 2022-10-18 12:33:18 +01:00
Ivan Shapovalov
4af93bd7f6
Allow poetry-core 1.3.2 (#14217)
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2022-10-18 10:38:58 +01:00
Andrew Morgan
dc02d9f8c5
Avoid checking the event cache when backfilling events (#14164) 2022-10-18 10:33:35 +01:00
Andrew Morgan
828b5502cf
Remove _get_events_cache check optimisation from _have_seen_events_dict (#14161) 2022-10-18 10:33:21 +01:00
Eric Eastwood
2c63cdcc3f
Add debug logs to figure out why an event was filtered (#14095)
Spawned while investigating https://github.com/matrix-org/synapse/issues/13944

This way we might get some more context whenever an `403 Forbidden - body: {"errcode":"M_FORBIDDEN","error":"You don't have permission to access that event."}` error is produced.

`log_config.yaml`
```yaml
loggers:
    synapse:
        level: INFO

    synapse.visibility:
        level: DEBUG
```
2022-10-17 16:02:39 -05:00
Erik Johnston
6fee2f49f3
Cache Rust build cache when building docker images (#14130) 2022-10-17 17:21:14 +00:00
dependabot[bot]
c101fc6568
Bump click from 8.1.1 to 8.1.3 (#14201) 2022-10-17 17:54:11 +01:00
Erik Johnston
130668b66c Merge branch 'master' into develop 2022-10-17 17:04:40 +01:00
Patrick Cloke
4283bd1cf9
Support filtering the /messages API by relation type (MSC3874). (#14148)
Gated behind an experimental configuration flag.
2022-10-17 11:32:11 -04:00
dependabot[bot]
6b24235142
Bump psycopg2 from 2.9.3 to 2.9.4 (#14200) 2022-10-17 13:24:17 +00:00
dependabot[bot]
2fa1bf598d
Bump bcrypt from 3.2.0 to 4.0.1 (#14186) 2022-10-17 13:42:18 +01:00
David Robertson
ccce8cdfc5
Use Pydantic when PUTting room aliases (#14179) 2022-10-17 13:39:12 +01:00
Nick Mills-Barrett
2c2c3f8b2c
Invalidate rooms for user caches when receiving membership events (#14155)
This should fix a race where the event notification comes in over
replication before the state replication, leaving a window during
which a sync may get an incorrect list of rooms for the user.
2022-10-17 13:27:51 +01:00
David Robertson
cd01a1d3b5
Fix dead link to admin registration API (#14189)
* Fix dead link to admin registration API

* Changelog
2022-10-17 12:10:01 +01:00
Dirk Klimpel
1eb8dcf4c9
Remove not needed replication listener in docker compose example (#14107) 2022-10-17 12:00:09 +01:00
Erik Johnston
6b097a3e17 1.69.0 2022-10-17 11:31:15 +01:00
Eric Eastwood
40bb37eb27
Stop getting missing prev_events after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
dependabot[bot]
616dcc1d18
Bump types-bleach from 4.1.4 to 5.0.3 (#14188)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-14 23:57:55 +00:00
dependabot[bot]
8cd22674a1
Bump types-pillow from 9.2.2 to 9.2.2.1 (#14187)
* Bump types-pillow from 9.2.2 to 9.2.2.1

Bumps [types-pillow](https://github.com/python/typeshed) from 9.2.2 to 9.2.2.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-15 00:16:25 +01:00
realtyem
c75836fe77
Strip whitespace from worker types in Dockerfile-workers (#14165) 2022-10-14 19:38:04 +00:00
Patrick Cloke
bc2bd92b93 Merge remote-tracking branch 'origin/release-v1.69' into develop 2022-10-14 14:11:27 -04:00
Patrick Cloke
d1bdeccb50
Accept threaded receipts for events related to the root event. (#14174)
The root node of a thread (and events related to it) are considered
"part of a thread" when validating receipts. This allows clients which
show the root node in both the main timeline and the threaded timeline
to easily send receipts in either.

Note that threaded notifications are not created for these events, these
events created notifications on the main timeline.
2022-10-14 18:05:25 +00:00
Erik Johnston
b43be004b4 1.69.0rc4 2022-10-14 15:05:04 +01:00
Erik Johnston
d241a1350d
Fix background update to use an index (#14181) 2022-10-14 13:46:23 +00:00
Patrick Cloke
022f25b309
Advertise support for Matrix 1.4. (#14184)
All features / changes in Matrix 1.4 are now supported in
Synapse.
2022-10-14 13:21:55 +00:00
Patrick Cloke
5a983cccfb Fix changelog for #14171 to match #13394/#14175. 2022-10-14 08:56:37 -04:00
Andrew Morgan
97b3d037c0
Don't require optional invite_room_state field on fed v2 invite (#14083) 2022-10-14 13:48:33 +01:00
Patrick Cloke
126a15794c
Do not allow a None-limit on PaginationConfig. (#14146)
The callers either set a default limit or manually handle a None-limit
later on (by setting a default value).

Update the callers to always instantiate PaginationConfig with a default
limit and then assume the limit is non-None.
2022-10-14 12:30:05 +00:00
realtyem
c7446906bd
Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182) 2022-10-14 13:29:49 +01:00
Patrick Cloke
c3e4edb4d6
Stabilize the threads API. (#14175)
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.

This also marks the API as safe for workers since it is a read-only
API.
2022-10-14 07:16:50 -04:00
Patrick Cloke
9ff4155f6c
Properly invalidate get_thread_id cache. (#14163)
This was missed in 2b6d41ebd6 (#13824).
2022-10-14 07:10:44 -04:00
Erik Johnston
6e0dde3215 Revert "Disable push of docker images"
This reverts commit f3f303aa22.
2022-10-14 10:50:24 +01:00
Erik Johnston
424d1d28cc Fix docker workflow 2022-10-14 09:57:31 +01:00
Erik Johnston
f3f303aa22 Disable push of docker images 2022-10-14 09:54:56 +01:00
Erik Johnston
29ee4b6698
Fix docker build OOMing in CI for arm64 builds (#14173)
Co-authored-by: David Robertson <davidr@element.io>
2022-10-13 18:16:21 +00:00
David Robertson
16c5d95b59
Optimise the event_push_backfill_thread_id bg job (#14172)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-10-13 17:32:16 +00:00
Patrick Cloke
2019b60f3b
Fix sqlite syntax for upserts. (#14171) 2022-10-13 12:53:24 -04:00
Patrick Cloke
7d59a515bb Properly return the thread ID down sync. (#14159)
Fix a broken conflict in e6e876b9b1,
by not stomping over a field right after creating it.
2022-10-13 12:15:41 -04:00
Patrick Cloke
3bbe532abb
Add an API for listing threads in a room. (#13394)
Implement the /threads endpoint from MSC3856.

This is currently unstable and behind an experimental configuration
flag.

It includes a background update to backfill data, results from
the /threads endpoint will be partial until that finishes.
2022-10-13 08:02:11 -04:00
Shay
b6baa46db0
Fix a bug where the joined hosts for a given event were not being properly cached (#14125) 2022-10-12 11:01:00 -07:00
Patrick Cloke
e6e876b9b1 Return the thread ID properly down sync. (#14159)
A receipt's thread ID, if one exists, should be added to the
body of a receipt.
2022-10-12 12:18:34 -04:00
Patrick Cloke
87099b6ea5
Return the main timeline for events which are not part of a thread. (#14140)
Fixes a bug where threaded receipts could not be sent for the
main timeline.
2022-10-12 12:15:52 -04:00