Commit Graph

21388 Commits

Author SHA1 Message Date
Erik Johnston
5442891cbc
Make push rules use proper structures. (#13522)
This improves load times for push rules:

| Version              | Time per user | Time for 1k users | 
| -------------------- | ------------- | ----------------- |
| Before               |       138 µs  |             138ms |
| Now (with custom)    |       2.11 µs |            2.11ms |
| Now (without custom) |       49.7 ns |           0.05 ms |

This therefore has a large impact on send times for rooms
with large numbers of local users in the room.
2022-08-16 12:22:17 +01:00
David Robertson
d642ce4b32
Use Pydantic to systematically validate a first batch of endpoints in synapse.rest.client.account. (#13188) 2022-08-15 19:05:57 +00:00
Shay
73c83c6411
Add a warning to retention documentation regarding the possibility of database corruption (#13497) 2022-08-15 11:54:23 -07:00
Eric Eastwood
344a2f767c
Instrument FederationStateIdsServlet - /state_ids (#13499)
Instrument FederationStateIdsServlet - `/state_ids` so it's easier to follow what's going on in Jaeger when viewing a trace.
2022-08-15 19:41:23 +01:00
David Robertson
19e5d44886
Revert "Update locked versions of mypy and mypy-zope (#13521)"
This reverts commit f383b9b3ec. Other PRs
were seeing mypy failures that looked to be related to mypy-zope.
Confusingly, we didn't see this on #13521.

Revert this for now and investigate later.
2022-08-15 14:51:05 +01:00
Patrick Cloke
46bd7f4ed9
Clarifications for event push action processing. (#13485)
* Clarifies comments.
* Fixes an erroneous comment (about return type) added in #13455
  (ec24813220).
* Clarifies the name of a variable.
* Simplifies logic of pulling out the latest join for the requesting user.
2022-08-15 09:33:17 -04:00
David Robertson
f383b9b3ec
Update locked versions of mypy and mypy-zope (#13521) 2022-08-15 11:32:30 +01:00
Richard van der Hoff
434fd82d5f Update grafana dashboard 2022-08-13 21:50:20 +01:00
reivilibre
2c5e2ae898
Document that the DOCKER_BUILDKIT=1 flag is needed to build the docker image. (#13515) 2022-08-12 14:25:47 +01:00
James Barton
e825f7366b
Add openssl example for registration HMAC (#13472)
Signed-off-by: James Barton <james@neodon.com>
2022-08-11 21:10:10 +01:00
Andrew Morgan
953df2ad88
Add note to redaction_retention_period documentation mentioning that event purging runs at most every 5m (#13492) 2022-08-11 14:06:09 +01:00
Germain
4390121684
Add viewport directive to HTML templates to optimise for mobile (#13493) 2022-08-11 14:04:20 +01:00
Olivier Wilkinson (reivilibre)
05bc059e0d Merge branch 'release-v1.65' into develop 2022-08-11 13:19:12 +01:00
Richard van der Hoff
507c1cb330
Update the rejected state of events during resync (#13459)
Events can be un-rejected or newly-rejected during resync, so ensure we update
the database and caches when that happens.
2022-08-11 10:42:24 +00:00
Olivier Wilkinson (reivilibre)
7a25b4302c 1.65.0rc2 2022-08-11 11:40:40 +01:00
reivilibre
12abd72497
Revert 'Remove the unspecced field in the response. (#13365)' to give more time for clients to update. (#13501) 2022-08-11 10:27:48 +00:00
Dirk Klimpel
2281427175
Use literals in place of HTTPStatus constants in tests (#13488)
* Use literals in place of `HTTPStatus` constants in tests

* newsfile

* code style

* code style
2022-08-10 11:01:12 -07:00
Sean Quah
51c01d450a
Add some miscellaneous comments around sync (#13474)
Add some miscellaneous comments to document sync, especially around
`compute_state_delta`.

Signed-off-by: Sean Quah <seanq@matrix.org>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-08-10 12:58:20 +01:00
Eric Eastwood
1b09b0832e
Allow use of both @trace and @tag_args stacked on the same function (#13453)
```py
@trace
@tag_args
async def get_oldest_event_ids_with_depth_in_room(...)
  ...
```

Before this PR, you would see a warning in the logs and the span was not exported:
```
2022-08-03 19:11:59,383 - synapse.logging.opentracing - 835 - ERROR - GET-0 - @trace may not have wrapped EventFederationWorkerStore.get_oldest_event_ids_with_depth_in_room correctly! The function is not async but returned a coroutine.
```
2022-08-09 14:32:33 -05:00
Dirk Klimpel
1595052b26
Use literals in place of HTTPStatus constants in tests (#13479)
Replace
- `HTTPStatus.NOT_FOUND`
- `HTTPStatus.FORBIDDEN`
- `HTTPStatus.UNAUTHORIZED`
- `HTTPStatus.CONFLICT`
- `HTTPStatus.CREATED`

Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
2022-08-09 14:56:43 +01:00
Olivier Wilkinson (reivilibre)
54fb517c28 Merge branch 'release-v1.65' into develop 2022-08-09 13:45:11 +01:00
Olivier Wilkinson (reivilibre)
3d1b860f90 Fix changelog mistake 2022-08-09 13:13:22 +01:00
Patrick Cloke
5ce2887653
Strengthen tests about deleted old push actions. (#13471) 2022-08-09 07:56:16 -04:00
Olivier Wilkinson (reivilibre)
c962f87d6f Tweak the changelog 2022-08-09 12:54:59 +01:00
Olivier Wilkinson (reivilibre)
827f0669bf 1.65.0rc1 2022-08-09 11:43:30 +01:00
Dirk Klimpel
70d3e70009
Make the configuration for the cache clearer (#13481) 2022-08-09 11:30:22 +01:00
Shay
0f954466c4
Update matrix-synapse-ldap3 version in lockfile to v0.2.2. (#13470) 2022-08-08 13:44:43 -07:00
Dirk Klimpel
c97042f7ee
Use literals in place of HTTPStatus constants in tests (#13469) 2022-08-08 13:21:27 -07:00
David Robertson
7a19995120
Correct a misnamed argument in state res v2 (#13467)
In state res v2, we apply two passes of iterative auth checks. The first
pass replays power events and events in their auth chains, but only
those belonging to the full conflicted set. The source code as written
suggests that we want only those belonging to the auth difference (which
is a smaller set of events).

At runtime we were doing the correct thing anyway, because the only
callsite of `_reverse_topological_power_sort` passes in the
`full_conflicted_set`. So this really is just a rename.
2022-08-08 16:59:56 +01:00
Šimon Brandner
ab18441573
Support stable identifiers for MSC2285: private read receipts. (#13273)
This adds support for the stable identifiers of MSC2285 while
continuing to support the unstable identifiers behind the configuration
flag. These will be removed in a future version.
2022-08-05 11:09:33 -04:00
Dirk Klimpel
e2ed1b7155
Use literals in place of HTTPStatus constants in tests (#13463) 2022-08-05 16:59:09 +02:00
Julian-Samuel Gebühr
3d2cabf966
Mark token-authenticaticated-registration API as not-experimental (#11897) 2022-08-05 11:15:35 +00:00
Matt C
026ac4486c
Update module API "update room membership" method to allow for remote joins (#13441)
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-05 09:37:58 +00:00
Erik Johnston
b6a6bb4027
Add comments about how event push actions are stored. (#13445) 2022-08-04 19:38:08 +00:00
Eric Eastwood
860fdd9098
Fix @tag_args being off-by-one (ahead) (#13452)
Fix @tag_args being off-by-one (ahead)

Example:

```
argspec.args=[
  'self',
  'room_id'
]

args=(
  <synapse.storage.databases.main.DataStore object at 0x10d0b8d00>,
  '!HBehERstyQBxyJDLfR:my.synapse.server'
)
```

---

The previous logic was also flawed and we can end up in a situation like this:

```
argspec.args=['self', 'dest', 'room_id', 'limit', 'extremities']

args=(<synapse.federation.federation_client.FederationClient object at 0x7f1651c18160>, 'hs1', '!jAEHKIubyIfuLOdfpY:hs1')
```

From this source:
```py
async def backfill(
    self, dest: str, room_id: str, limit: int, extremities: Collection[str]
) -> Optional[List[EventBase]]:
```

And this usage:
```py
events = await self._federation_client.backfill(
    dest, room_id, limit=limit, extremities=extremities
)
```

which would previously cause this error:
```
synapse_main | 2022-08-04 06:13:12,051 - synapse.handlers.federation - 424 - ERROR - GET-5 - Failed to backfill from hs1 because tuple index out of range
synapse_main | Traceback (most recent call last):
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation.py", line 392, in try_backfill
synapse_main |     await self._federation_event_handler.backfill(
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper
synapse_main |     return await func(*args, **kwargs)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 593, in backfill
synapse_main |     events = await self._federation_client.backfill(
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper
synapse_main |     return await func(*args, **kwargs)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 827, in _wrapper
synapse_main |     with wrapping_logic(func, *args, **kwargs):
synapse_main |   File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
synapse_main |     return next(self.gen)
synapse_main |   File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 922, in _wrapping_logic
synapse_main |     set_attribute("ARG_" + arg, str(args[i + 1]))  # type: ignore[index]
synapse_main | IndexError: tuple index out of range
```
2022-08-04 14:29:41 -05:00
Patrick Cloke
ec24813220
Improve comments (& avoid a duplicate query) in push actions processing. (#13455)
* Adds docstrings and inline comments.
* Formats SQL queries using triple quoted strings.
* Minor formatting changes.
* Avoid fetching `event_push_summary_stream_ordering` multiple times
  in the same transactions.
2022-08-04 19:24:44 +00:00
Richard van der Hoff
96d92156d0
Update type of EventContext.rejected (#13460) 2022-08-04 17:45:01 +01:00
reivilibre
e9e6aacfbe
Faster Room Joins: prevent Synapse from answering federated join requests for a room which it has not fully joined yet. (#13416) 2022-08-04 16:27:04 +01:00
Nick Mills-Barrett
41320a0554
Optimise async get event lookups (#13435)
Still maintains local in memory lookup optimisation, but does any external
lookup as part of the deferred that prevents duplicate lookups for the same
event at once. This makes the assumption that fetching from an external
cache is a non-zero load operation.
2022-08-04 15:49:55 +01:00
Dirk Klimpel
6dd7fa12dc
Update some outdated information on sso_mapping_providers.md (#13449) 2022-08-04 13:06:02 +01:00
Dirk Klimpel
afbdbe0634
Fix return value in example on password_auth_provider_callbacks.md (#13450)
Fixes: #12534

Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
2022-08-04 13:03:36 +01:00
Richard van der Hoff
166fafdf8d
synapse-workers docker: copy nginx and redis in from base images (#13447)
Part of my continuing quest to make the docker images build quicker: copy nginx and redis in from base docker images, rather than apt installing each time.
2022-08-04 12:59:27 +01:00
Matt C
a91078200d
Add module API method to create a room (#13429)
Co-authored-by: MattC <buffless-matt@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-04 09:34:05 +00:00
Brendan Abolivier
845732be45
Fix rooms not being properly excluded from incremental sync (#13408) 2022-08-04 11:02:29 +02:00
Shay
a648a06d52
Add some tracing spans to give insight into local joins (#13439) 2022-08-03 10:19:34 -07:00
Eric Eastwood
92d21faf12
Instrument /messages for understandable traces in Jaeger (#13368)
In Jaeger:

 - Before: huge list of uncategorized database calls
 - After: nice and collapsible into units of work
2022-08-03 10:57:38 -05:00
andrew do
78a3111c41
Return 404 or member list when getting joined_members after leaving (#13374)
Signed-off-by: Andrew Doh <andrewddo@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrewm@element.io>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-03 14:26:31 +02:00
Jasper Spaans
503a95804e
Install cryptography build dependencies in requirements image. (#13372) 2022-08-03 11:16:32 +01:00
jejo86
668597214f
Improve documentation on becoming server admin (#13230)
* Improved section regarding server admin

Added steps describing how to elevate an existing user to administrator by manipulating a `postgres` database.

Signed-off-by: jejo86 28619134+jejo86@users.noreply.github.com

* Improved section regarding server admin

* Reference database settings

Add instructions to check database settings to find out the database name, instead of listing all available PostgreSQL databases.

* Add suggestions from PR conversation

Replace config filename `homeserver.yaml`. with "config file".
Remove instructions to switch to `postgres` user.
Add instructions how to connect to SQLite database.

* Update changelog.d/13230.doc

Co-authored-by: reivilibre <olivier@librepush.net>
2022-08-03 11:15:23 +01:00
Dirk Klimpel
fb7a2cc4cc
Update doc for setting macaroon_secret_key (#13443)
* Update doc for setting `macaroon_secret_key`

* newsfile
2022-08-03 10:41:19 +01:00