Commit Graph

22237 Commits

Author SHA1 Message Date
Patrick Cloke
c9b9143655
Fix-up type hints in tests/server.py. (#15084)
This file was being ignored by mypy, we remove that
and add the missing type hints & deal with any fallout.
2023-02-17 18:19:38 +00:00
Dirk Klimpel
61bfcd669a
Add account data to export command (#14969)
* Add account data to to export command

* newsfile

* remove not needed function

* update newsfile

* adopt #14973
2023-02-17 13:54:55 +00:00
Sean Quah
4f4f27e57f
Mitigate a race where /make_join could 403 for restricted rooms (#15080)
Previously, when creating a join event in /make_join, we would decide
whether to include additional fields to satisfy restricted room checks
based on the current state of the room. Then, when building the event,
we would capture the forward extremities of the room to use as prev
events.

This is subject to race conditions. For example, when leaving and
rejoining a room, the following sequence of events leads to a misleading
403 response:
1. /make_join reads the current state of the room and sees that the user
   is still in the room. It decides to omit the field required for
   restricted room joins.
2. The leave event is persisted and the room's forward extremities are
   updated.
3. /make_join builds the event, using the post-leave forward extremities.
   The event then fails the restricted room checks.

To mitigate the race, we move the read of the forward extremities closer
to the read of the current state. Ideally, we would compute the state
based off the chosen prev events, but that can involve state resolution,
which is expensive.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-17 09:40:32 +00:00
ZAID BIN TARIQ
ad1f3fa8e1
Document how to start Synapse with Poetry (#14892)
* Add Start Synapse with Poetry

* Create 14892.doc

* Apply suggestions from code review

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Update docs/workers.md

---------

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-02-16 18:20:02 +00:00
David Robertson
ffc2ee521d
Use mypy 1.0 (#15052)
* Update mypy and mypy-zope
* Remove unused ignores

These used to suppress

```
synapse/storage/engines/__init__.py:28: error: "__new__" must return a
class instance (got "NoReturn")  [misc]
```

and

```
synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons"  [attr-defined]
```

(note that we check `hasattr(e, "reasons")` above)

* Avoid empty body warnings, sometimes by marking methods as abstract

E.g.

```
tests/handlers/test_register.py:58: error: Missing return statement  [empty-body]
tests/handlers/test_register.py:108: error: Missing return statement  [empty-body]
```

* Suppress false positive about `JaegerConfig`

Complaint was

```
synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context  [truthy-function]
```

* Fix not calling `is_state()`

Oops!

```
tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context  [truthy-function]
```

* Suppress false positives from ParamSpecs

````
synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
````

* Drive-by improvement to `wrapping_logic` annotation

* Workaround false "unreachable" positives

See https://github.com/Shoobx/mypy-zope/issues/91

```
tests/http/test_proxyagent.py:626: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:762: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:826: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:838: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:845: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:60: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:93: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:127: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:152: error: Statement is unreachable  [unreachable]
```

* Changelog

* Tweak DBAPI2 Protocol to be accepted by mypy 1.0

Some extra context in:
- https://github.com/matrix-org/python-canonicaljson/pull/57
- https://github.com/python/mypy/issues/6002
- https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected

* Pull in updated canonicaljson lib

so the protocol check just works

* Improve comments in opentracing

I tried to workaround the ignores but found it too much trouble.

I think the corresponding issue is
https://github.com/python/mypy/issues/12909. The mypy repo has a PR
claiming to fix this (https://github.com/python/mypy/pull/14677) which
might mean this gets resolved soon?

* Better annotation for INTERACTIVE_AUTH_CHECKERS

* Drive-by AUTH_TYPE annotation, to remove an ignore
2023-02-16 16:09:11 +00:00
Patrick Cloke
979f237b28
Update intentional mentions (MSC3952) to depend on exact_event_match (MSC3758). (#15037)
This replaces the specific `is_room_mention` push rule condition
used in MSC3952 with the generic `exact_event_match` push rule
condition from MSC3758.

No functionality changes due to this.
2023-02-16 09:51:22 -05:00
saddfox
d1efc47925
Fix a mistake in registration_shared_secret_path docs (#15078)
* fix a typo in registration_shared_secret_path docs
Signed-off-by: Filip Rutar <filip.rutar@gmail.com>

* changelog
2023-02-15 19:51:58 +00:00
Sean Quah
3ad817bfe5
Fix federated joins when the first server in the list is not in the room (#15074)
Previously we would give up upon receiving a 404 from the first server,
instead of trying the rest of the servers in the list.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-15 13:59:06 +00:00
David Robertson
39795b3a4e
Make it easier to use DataGrip w/ Synapse's schema (#14982)
Also tweak the schema dump script:

- add a note explaining myself how to use it
-Explicitly call `poetry run`, because not everyone uses direnv :(
2023-02-15 13:51:37 +00:00
999lakhisidhu
27a3a72a50
Support for selecting the Redis logical database. (#15034)
Note that this is only used for key-value store (cached values)
and not for the pub/sub replication used by Synapse.
2023-02-15 07:39:31 -05:00
Richard van der Hoff
5febf88b6c
Update the error code for duplicate annotation (#15075) 2023-02-15 11:47:57 +00:00
David Robertson
06ba71083e
Fix order of partial state tables when purging (#15068)
* Fix order of partial state tables when purging

`partial_state_rooms` has an FK on `events` pointing to the join event we
get from `/send_join`, so we must delete from that table before deleting
from `events`.

**NB:** It would be nice to cancel any resync processes for the room
being purged. We do not do this at present. To do so reliably we'd need
an internal HTTP "replication" endpoint, because the worker doing the
resync process may be different to that handling the purge request.

The first time the resync process tries to write data after the deletion
it will fail because we have deleted necessary data e.g. auth
events. AFAICS it will not retry the resync, so the only downside to
not cancelling the resync is a scary-looking traceback.

(This is presumably extremely race-sensitive.)

* Changelog

* admist(?) -> between

* Warn about a race

* Fix typo, thanks Sean

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

---------

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-02-14 23:42:29 +00:00
Patrick Cloke
42aea0d8af
Add final type hint to tests.unittest. (#15072)
Adds a return type to HomeServerTestCase.make_homeserver and deal
with any variables which are no longer Any.
2023-02-14 14:03:35 -05:00
Patrick Cloke
119e0795a5
Implement MSC3966: Add a push rule condition to search for a value in an array. (#15045)
The `exact_event_property_contains` condition can be used to
search for a value inside of an array.
2023-02-14 14:02:19 -05:00
Richard van der Hoff
157c571f3e
Remove spurious dont_notify action from .m.rule.reaction (#15073)
This does nothing and I want to remove it from the MSC.
2023-02-14 18:19:58 +00:00
reivilibre
e9b1ff9f31
Prevent clients from reporting nonexistent events. (#13779) 2023-02-14 15:50:59 +00:00
Mathieu Velten
69324c346c Merge branch 'master' into develop 2023-02-14 16:28:13 +01:00
Sean Quah
463c19ac36
Faster joins: Omit device list updates from partial state rooms in /sync (#15069)
...when lazy loading of members is not enabled. It's weird to notify
a client that another user's device list has changed when the client
doesn't think that they share a room.

Note that when a room is un-partial stated, device list updates are
emitted for every member in that room over /sync.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-14 12:32:19 +00:00
Mathieu Velten
3e90dfdd81 1.77.0 2023-02-14 12:59:15 +01:00
Erik Johnston
cb262713b7
Fix clashing DB txn name (#15070)
* Fix clashing DB txn name

* Newsfile
2023-02-14 11:20:25 +00:00
Erik Johnston
f09db5c991
Skip calculating unread push actions in /sync when enable_push is false. (#14980) 2023-02-14 11:10:29 +00:00
Harishankar Kumar
db2b105d69
Change collection[str] to StrCollection in event_auth code (#14929)
Signed-off-by: Harishankar Kumar <hari01584@gmail.com>
2023-02-14 09:37:08 +00:00
reivilibre
c0bf4c3cb4
Add check to ensure locked dependencies have source distributions available. (#14742) 2023-02-13 18:15:38 +00:00
reivilibre
3d7aead5d6
Tweak comment on _is_local_room_accessible as part of room visibility in /hierarchy to clarify the condition for a room being visible. (#14834) 2023-02-13 16:30:58 +00:00
V02460
bac123c9d3
Update pyo3-log to v0.8.1 (#15043)
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2023-02-13 12:25:56 +00:00
Andrew Morgan
bdccfd2477
Refactor arguments of try_unbind_threepid(_with_id_server) from dict to separate args (#15053) 2023-02-13 12:12:48 +00:00
David Robertson
c10e131250
Apply logging from hotfixes branch to develop (#15054)
* Apply logging from hotfixes branch to develop

Part of #4826.

Originally added in #11882.

* Changelog
2023-02-13 11:49:20 +00:00
dependabot[bot]
5e1b21e152
Bump sentry-sdk from 1.13.0 to 1.15.0 (#15065)
* Bump sentry-sdk from 1.13.0 to 1.15.0

Bumps [sentry-sdk](https://github.com/getsentry/sentry-python) from 1.13.0 to 1.15.0.
- [Release notes](https://github.com/getsentry/sentry-python/releases)
- [Changelog](https://github.com/getsentry/sentry-python/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-python/compare/1.13.0...1.15.0)

---
updated-dependencies:
- dependency-name: sentry-sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:19:56 +00:00
dependabot[bot]
14406d1a08
Bump types-pillow from 9.4.0.5 to 9.4.0.10 (#15064)
* Bump types-pillow from 9.4.0.5 to 9.4.0.10

Bumps [types-pillow](https://github.com/python/typeshed) from 9.4.0.5 to 9.4.0.10.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:19:39 +00:00
dependabot[bot]
fa7bbd05e2
Bump types-requests from 2.28.11.8 to 2.28.11.12 (#15063)
* Bump types-requests from 2.28.11.8 to 2.28.11.12

Bumps [types-requests](https://github.com/python/typeshed) from 2.28.11.8 to 2.28.11.12.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:19:28 +00:00
dependabot[bot]
02db6cfd28
Bump serde_json from 1.0.92 to 1.0.93 (#15062)
* Bump serde_json from 1.0.92 to 1.0.93

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.92 to 1.0.93.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.92...v1.0.93)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:19:18 +00:00
dependabot[bot]
ede0b219eb
Bump systemd-python from 234 to 235 (#15061)
* Bump systemd-python from 234 to 235

Bumps [systemd-python](https://github.com/systemd/python-systemd) from 234 to 235.
- [Release notes](https://github.com/systemd/python-systemd/releases)
- [Changelog](https://github.com/systemd/python-systemd/blob/main/NEWS)
- [Commits](https://github.com/systemd/python-systemd/compare/v234...v235)

---
updated-dependencies:
- dependency-name: systemd-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:18:50 +00:00
dependabot[bot]
81497c752b
Bump dtolnay/rust-toolchain from 9cd00a88a73addc8617065438eff914dd08d0955 to 25dc93b901a87e864900a8aec6c12e9aa794c0c3 (#15060)
* Bump dtolnay/rust-toolchain

Bumps [dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain) from 9cd00a88a73addc8617065438eff914dd08d0955 to 25dc93b901a87e864900a8aec6c12e9aa794c0c3.
- [Release notes](https://github.com/dtolnay/rust-toolchain/releases)
- [Commits](9cd00a88a7...25dc93b901)

---
updated-dependencies:
- dependency-name: dtolnay/rust-toolchain
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:18:26 +00:00
dependabot[bot]
e0bc331a94
Bump bleach from 5.0.1 to 6.0.0 (#15059)
* Bump bleach from 5.0.1 to 6.0.0

Bumps [bleach](https://github.com/mozilla/bleach) from 5.0.1 to 6.0.0.
- [Release notes](https://github.com/mozilla/bleach/releases)
- [Changelog](https://github.com/mozilla/bleach/blob/main/CHANGES)
- [Commits](https://github.com/mozilla/bleach/compare/v5.0.1...v6.0.0)

---
updated-dependencies:
- dependency-name: bleach
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-02-13 09:18:07 +00:00
Mathieu Velten
6cddf24e36
Faster joins: don't stall when a user joins during a fast join (#14606)
Fixes #12801.
Complement tests are at
https://github.com/matrix-org/complement/pull/567.

Avoid blocking on full state when handling a subsequent join into a
partial state room.

Also always perform a remote join into partial state rooms, since we do
not know whether the joining user has been banned and want to avoid
leaking history to banned users.

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:31:05 +00:00
Sean Quah
d0c713cc85
Return read-only collections from @cached methods (#13755)
It's important that collections returned from `@cached` methods are not
modified, otherwise future retrievals from the cache will return the
modified collection.

This applies to the return values from `@cached` methods and the values
inside the dictionaries returned by `@cachedList` methods. It's not
necessary for the dictionaries returned by `@cachedList` methods
themselves to be read-only.

Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:29:00 +00:00
Patrick Cloke
14be78d492
Support for MSC3758: exact_event_match push condition (#14964)
This specifies to search for an exact value match, instead of
string globbing. It only works across non-compound JSON values
(null, boolean, integer, and strings).
2023-02-10 12:37:07 -05:00
Patrick Cloke
cf5233b783
Avoid fetching unused account data in sync. (#14973)
The per-room account data is no longer unconditionally
fetched, even if all rooms will be filtered out.

Global account data will not be fetched if it will all be
filtered out.
2023-02-10 14:22:16 +00:00
David Robertson
d793fcd241
Merge branch 'release-v1.77' into develop 2023-02-10 13:43:18 +00:00
Sean Quah
b95407908d
Avoid mutating cached values in _generate_sync_entry_for_account_data (#15047) 2023-02-10 08:11:20 -05:00
Patrick Cloke
a481fb9f98
Refactor get_user_devices_from_cache to avoid mutating cached values. (#15040)
The previous version of the code could mutate a cached value,
but only if the input requested all devices of a user *and* a specific
device.

To avoid this nonsensical situation we no longer fetch a specific
device ID if all of a user's devices are returned.
2023-02-10 08:09:47 -05:00
David Robertson
73b8068ced
1.77.0rc2 2023-02-10 12:44:38 +00:00
Erik Johnston
fd296b7343
Fix exception on start up about device lists (#15041)
Fixes #15010.
2023-02-10 09:52:35 +00:00
David Robertson
a5a799722d
Tag federation request spans with the worker name (#15042)
* Systematically include worker name as process info

* Changelog

* don't bother with inner setdefault
2023-02-09 22:33:39 +00:00
Shay
03bccd542b
Add a class UnpersistedEventContext to allow for the batching up of storing state groups (#14675)
* add class UnpersistedEventContext

* modify create new client event to create unpersistedeventcontexts

* persist event contexts after creation

* fix tests to persist unpersisted event contexts

* cleanup

* misc lints + cleanup

* changelog + fix comments

* lints

* fix batch insertion?

* reduce redundant calculation

* add unpersisted event classes

* rework compute_event_context, split into function that returns unpersisted event context and then persists it

* use calculate_context_info to create unpersisted event contexts

* update typing

* $%#^&*

* black

* fix comments and consolidate classes, use attr.s for class

* requested changes

* lint

* requested changes

* requested changes

* refactor to be stupidly explicit

* clearer renaming and flow

* make partial state non-optional

* update docstrings

---------

Co-authored-by: Erik Johnston <erik@matrix.org>
2023-02-09 13:05:02 -08:00
Andrew Morgan
c1d2ce2901
Do not always start a db txn on Postgres (#14840) 2023-02-09 19:57:01 +00:00
Andrew Morgan
218a383c43
Bump ruff version from 0.0.230 to 0.0.237. (#15033) 2023-02-09 18:18:42 +00:00
Patrick Cloke
d22c1c862c
Respond correctly to unknown methods on known endpoints (#14605)
Respond with a 405 error if a request is received on a known endpoint,
but to an unknown method, per MSC3743.
2023-02-09 13:04:24 -05:00
Patrick Cloke
8a6e043488
Avoid mutating cached room aliases. (#15038)
This might cause incorrect data in other callers which
are not expecting the canonical alias to be added into
the response.
2023-02-09 15:56:02 +00:00
David Robertson
cd2484dc2e
Bump schema version (#15036)
* Bump schema version

This should have been included in
f10caa73ee (and #14979).

* Changelog
2023-02-09 15:28:26 +00:00