Commit Graph

13777 Commits

Author SHA1 Message Date
Samuel Philipp
4e393af52f
Fixed config parse bug in review_recent_signups (#11191) 2021-10-27 17:25:18 +01:00
Patrick Cloke
19d5dc6931
Refactor Filter to handle fields according to data being filtered. (#11194)
This avoids filtering against fields which cannot exist on an
event source. E.g. presence updates don't have a room.
2021-10-27 11:26:30 -04:00
Dirk Klimpel
8d46fac98e
Delete messages from device_inbox table when deleting device (#10969)
Fixes: #9346
2021-10-27 16:01:18 +01:00
Patrick Cloke
a930da3291
Include the stable identifier for MSC3288. (#11187)
Includes both the stable and unstable identifier to store-invite
calls to the identity server. In the future we should remove the
unstable identifier.
2021-10-27 14:19:19 +00:00
Erik Johnston
179dc8ae9e Merge remote-tracking branch 'origin/release-v1.46' into develop 2021-10-27 14:45:40 +01:00
Sean Quah
f3a4be8700
Shut down the DNS threadpool (#11190)
The DNS threadpool must be explicitly stopped, otherwise Synapse will
hang indefinitely when asked to shut down.
2021-10-27 13:04:56 +01:00
Erik Johnston
72626b78ef
Fix thread BG update to not seq scan event_json (#11192)
For some reason the query optimiser decided to seq scan both tables,
rather than index scanning `event_json`.
2021-10-27 11:33:21 +00:00
Brendan Abolivier
c7a5e49664
Implement an on_new_event callback (#11126)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
Sean Quah
cc75a6b1b2 1.46.0rc1 2021-10-26 14:04:51 +01:00
Erik Johnston
7004f43da1
Move DNS lookups into separate thread pool (#11177)
This is to stop large bursts of lookups starving out other users of the
thread pools.

Fixes #11049.
2021-10-26 13:45:38 +01:00
Patrick Cloke
d52c58dfa3
Add a background update for updating MSC3440 relation threads. (#11181) 2021-10-26 07:38:45 -04:00
Brendan Abolivier
8c8e36af0d
Document the version each module API method was added to Synapse (#11183) 2021-10-26 11:09:10 +02:00
Jason Robinson
63cbdd8af0
Enable changing user type via users admin API (#11174)
Users admin API can now also modify user
type in addition to allowing it to be
set on user creation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-26 09:01:06 +00:00
Brendan Abolivier
c1510c97b5
Fix cyclic import in the module API (#11180)
Introduced in #10548

See https://github.com/matrix-org/synapse-email-account-validity/runs/3979337154?check_suite_focus=true for an example of a module's CI choking over this issue.
2021-10-25 16:45:19 +00:00
AndrewFerr
4387b791e0
Don't set new room alias before potential 403 (#10930)
Fixes: #10929 

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-10-25 15:24:49 +01:00
Richard van der Hoff
da957a60e8
Ensure that we correctly auth events returned by send_join (#11012)
This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected.

To do this I've reused the existing code we use when persisting outliers elsewhere.

I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.
2021-10-25 15:21:09 +01:00
Sean Quah
85a09f8b8b
Fix module API's get_user_ip_and_agents function when run on workers (#11112) 2021-10-25 13:01:04 +01:00
Sean Quah
2b82ec425f
Add type hints for most HomeServer parameters (#11095) 2021-10-22 18:15:41 +01:00
Jason Robinson
b9ce53e878
Fix synapse.config module "read" command (#11145)
`synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse.

 The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example.

 Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-22 12:00:52 +02:00
Patrick Cloke
ba00e20234
Add a thread relation type per MSC3440. (#11088)
Adds experimental support for MSC3440's `io.element.thread` relation
type (and the aggregation for it).
2021-10-21 14:39:16 -04:00
David Robertson
2d91b6256e
Fix adding excluded users to the private room sharing tables when joining a room (#11143)
* We only need to fetch users in private rooms

* Filter out `user_id` at the top

* Discard excluded users in the top loop

We weren't doing this in the "First, if they're our user" branch so this
is a bugfix.

* The caller must check that `user_id` is included

This is in the docstring. There are two call sites:
- one in `_handle_room_publicity_change`, which explicitly checks before calling;
- and another in `_handle_room_membership_event`, which returns early if
  the user is excluded.

So this change is safe.

* Test joining a private room with an excluded user

* Tweak an existing test

* Changelog

* test docstring

* lint
2021-10-21 17:48:59 +01:00
Andrew Morgan
6408372234
Improve docstrings for methods related to sending EDUs to application services (#11138) 2021-10-21 17:42:25 +01:00
Patrick Cloke
0f9adc99ad
Add missing type hints to synapse.crypto. (#11146)
And require type hints for this module.
2021-10-21 13:07:07 +00:00
Dirk Klimpel
ef7fe09778
Fix setting a user's external_id via the admin API returns 500 and deletes users existing external mappings if that external ID is already mapped (#11051)
Fixes #10846
2021-10-21 09:52:32 +01:00
Robert Edström
62db603fa0
Consider IP whitelist for identity server resolution (#11120)
Signed-off-by: Robert Edström <github@legogris.se>
2021-10-20 18:43:49 +01:00
Richard van der Hoff
0930e9ae12
Clean up _update_auth_events_and_context_for_auth (#11122)
Remove some redundant code, and generally simplify.
2021-10-20 18:22:40 +01:00
Aaron R
2c61a318cc
Show error when timestamp in seconds is provided to the /purge_media_cache API (#11101) 2021-10-20 14:41:48 +00:00
Sean Quah
ee2cee5f52 Merge branch 'master' into develop 2021-10-20 13:43:02 +01:00
Sean Quah
522489fbcd 1.45.1 2021-10-20 12:00:03 +01:00
Sean Quah
df95d3aec2
Revert change to counting of deactivated users towards the monthly active users limit (#11127)
Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)".

This reverts commit eda8c88b84.
2021-10-20 11:05:29 +01:00
Patrick Cloke
0dd0c40329
Add missing type hints to event fetching. (#11121)
Updates the event rows returned from the database to be
attrs classes instead of dictionaries.
2021-10-19 14:29:03 +00:00
David Robertson
a6c318735d
Merge branch 'master' into develop 2021-10-19 12:44:33 +01:00
David Robertson
8b1185347a
1.45.0 2021-10-19 11:19:55 +01:00
Richard van der Hoff
f3efa0036b
Move _persist_auth_tree into FederationEventHandler (#11115)
This is just a lift-and-shift, because it fits more naturally here. We do
rename it to `process_remote_join` at the same time though.
2021-10-19 10:24:09 +01:00
Richard van der Hoff
0170774b19
Rename _auth_and_persist_fetched_events (#11116)
... to `_auth_and_persist_outliers`, since that reflects its purpose better.
2021-10-19 10:23:55 +01:00
Richard van der Hoff
d85bc9a4a7
Include rejected status when we log events. (#11008)
If we find ourselves dealing with rejected events, we proably want to know
about it. Let's include it in the stringification of the event so that it gets
logged.
2021-10-19 10:21:50 +01:00
Patrick Cloke
3ab55d43bd
Add missing type hints to synapse.api. (#11109)
* Convert UserPresenceState to attrs.
* Remove args/kwargs from error classes and explicitly pass msg/errorcode.
2021-10-18 15:01:10 -04:00
Richard van der Hoff
cc33d9eee2
Check auth on received events' auth_events (#11001)
Currently, when we receive an event whose auth_events differ from those we expect, we state-resolve between the two state sets, and check that the event passes auth based on the resolved state.

This means that it's possible for us to accept events which don't pass auth at their declared auth_events (or where the auth events themselves were rejected), leading to problems down the line like #10083.

This change means we will:

 * ignore any events where we cannot find the auth events
 * reject any events whose auth events were rejected
 * reject any events which do not pass auth at their declared auth_events.

Together with a whole raft of previous work, this is a partial fix to #9595.

Fixes #6643.

Based on #11009.
2021-10-18 18:29:37 +01:00
Richard van der Hoff
a5d2ea3d08
Check *all* auth events for room id and rejection (#11009)
This fixes a bug where we would accept an event whose `auth_events` include
rejected events, if the rejected event was shadowed by another `auth_event`
with same `(type, state_key)`.

The approach is to pass a list of auth events into
`check_auth_rules_for_event` instead of a dict, which of course means updating
the call sites.

This is an extension of #10956.
2021-10-18 18:28:30 +01:00
Richard van der Hoff
e8f24b6c35
_run_push_actions_and_persist_event: handle no min_depth (#11014)
Make sure that we correctly handle rooms where we do not yet have a
`min_depth`, and also add some comments and logging.
2021-10-18 17:17:15 +01:00
Hillery Shay
7d70582eb0
Fix broken export-data admin command and add a test for it to CI (#11078)
Fix broken export-data admin command and add a test for it to CI
2021-10-18 08:14:12 -07:00
David Robertson
37b845dabc
Don't remove local users from dir when the leave their last room (#11103) 2021-10-18 13:20:04 +00:00
David Robertson
e09be0c87a
Correctly exclude users when making a room public or private (#11075)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-15 15:53:05 +01:00
Patrick Cloke
5573133348
Move experimental & retention config out of the server module. (#11070) 2021-10-15 14:30:48 +00:00
Sean Quah
6a67f3786a
Fix logging context warnings when losing replication connection (#10984)
Instead of triggering `__exit__` manually on the replication handler's
logging context, use it as a context manager so that there is an
`__enter__` call to balance the `__exit__`.
2021-10-15 13:10:58 +01:00
Eric Eastwood
daf498e099
Fix 500 error on /messages when we accumulate more than 5 backward extremities (#11027)
Found while working on the Gitter backfill script and noticed
it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390

When there are more than 5 backward extremities for a given depth,
backfill will throw an error because we sliced the extremity list
to 5 but then try to iterate over the full list. This causes
us to look for state that we never fetched and we get a `KeyError`.

Before when calling `/messages` when there are more than 5 backward extremities:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render
    callback_return = await raw_callback_return
  File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET
    msgs = await self.pagination_handler.get_messages(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages
    await self.hs.get_federation_handler().maybe_backfill(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill
    return await self._maybe_backfill_inner(room_id, current_depth, limit)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner
    likely_extremeties_domains = get_domains_from_state(states[e_id])
KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl'
```
2021-10-14 18:53:45 -05:00
Patrick Cloke
efd0074ab7
Ensure each charset is attempted only once during media preview. (#11089)
There's no point in trying more than once since it is guaranteed to
continually fail.
2021-10-14 18:51:44 +00:00
Patrick Cloke
e2f0b49b3f
Attempt different character encodings when previewing a URL. (#11077)
This follows similar logic to BeautifulSoup where we attempt different
character encodings until we find one which works.
2021-10-14 10:17:20 -04:00
David Robertson
b3698f945c Synapse 1.45.0rc2 (2021-10-14)
==============================
 
 **Note:** This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) the user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1. However, the [performance issue](https://github.com/matrix-org/synapse/issues/11049) which appeared in v1.44.0 is yet to be resolved.
 
 Bugfixes
 --------
 
 - Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045))
 - Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a
   user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053))
 - Fix a bug introduced in Synapse v1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061))
 
 Internal Changes
 ----------------
 
 - Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825))
 - Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970))
 - Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](https://github.com/matrix-org/synapse/issues/10996))
 - Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmFoBHYACgkQkD7OEIo5
 3t2+wRAAlk2GR9Ezi8E/AG6f5Dsq0+guswHK9b34PtI0wcLQFgy76y200g8D/V0F
 XFdby4IZsSLVmP2EVF+vIaU0/zviClHR81b8pvNcIRxORBAhV6fH5JkCRPNAlurS
 aGnoKA1TSSg4HwNHn+dWyg7VCHl76oLEBhixRdz7aWPhLnCNkCM62E+jlwVq19BH
 lPIDcwEvtq2ovHIGyE/sTg6Shrf9ELy0mYMQpPJNfDmxqjWqRhKaSpx5JSEJnzih
 amONFY7Nc8kIUdP6JSR6105IhUDAnItScmyyUowyCvwtAQsXsgMd8Cs0FdiJ0JLy
 BtE9qsBmBpmVWjfk7ESXE1eTHqH9Eexsy5dcB43hiVZ3ihhCCLmtVANej3QaRnRL
 rYAiH+W9WK7n3189hzkmEG/BKk+zu65dLaHSMjBvEDMyiYLoPRlYnmunLeo6mESq
 HHa25o5WiY8+UQ4RgcYqScIqvlgFqODmZT21OlknZfupUd2kDCN7Ga7TY599sBp2
 XSGNB6l/eGWRWtOPcdrandsDnppcoYsfJkXIfAUIxwjnSN/RU4kDPDiYDXhU8d+D
 hW/f59XwfMy3D3zZI4a+pEY8ZhcSHTnLJni95UFyb/7zOwnT+pZXUfVX46727nxg
 imUfXsXIxUYQnDkAGBXdt7QMZPy4gvTGenllqaA987xjfd5/fKs=
 =TTLK
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmFoCZQACgkQkD7OEIo5
 3t2Ygw/9E48TTeW7F5j16aCVVmRYnlAoh+rBlrKiaIE4MSJtwI+TEdIIrwGm+JdB
 lh59N3YmkXrn2No4ew0UY9fjD8LgofX3GAmlP2vN/fHst9kfe/UtoIDHWdQG1Xhh
 xeHnt9vXirQMOLxapkDM4ymBW1WAQdvgdn3Ozy3hQ+ZUQiMqJg4CQVMb7mUpjdJz
 D6Jb0Ub5FGnCGcSrofBk9KvHSZQb9Qrp8Rz3kokib98/+eJGdJDXhf0Yg4junJDy
 mOvTAhbJYG9TS3taFajuw8IfcngJYyBMUOf7HlcqMtyEPdHFW2wClUHzeQIPOurU
 YQmpqpDO7LSrHhFOt3cdmIlCGpGxKgR9AIUn8Volb2pxwH2i/BA7ryP9ZKz9dht3
 VnPFLEHYumBYOAB05TsUkDlpr/0x3YS1JCgzHGABM8spsi8XO/ZPRkwpjdc/jSRB
 3A4I7WGpKddk7pY3mCOnDx8hm2asgTPgTHkvJr2v2wYndIHxW91BDKhhSZ4tzjIY
 aOr2CtJmoFb3Buek/s1S2AGVLRXMYa/IIRHl8FjmL7WhsqebJkcRfcaMfHHtCq5Y
 tLhK2CYavEOu8TrkFWyIPPCCn8Y6Y8qnfg8c8f1ncfHsVHE4sOs9hjsVVz+n4QBd
 LD+ICWBeLvpGQK+0cZu3QFtIyo6XKBJlYuAyLD5saNkuLnupA3Y=
 =A9jl
 -----END PGP SIGNATURE-----

Merge tag 'v1.45.0rc2' into develop

Synapse 1.45.0rc2 (2021-10-14)
==============================

**Note:** This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) the user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1. However, the [performance issue](https://github.com/matrix-org/synapse/issues/11049) which appeared in v1.44.0 is yet to be resolved.

Bugfixes
--------

- Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045))
- Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a
  user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053))
- Fix a bug introduced in Synapse v1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061))

Internal Changes
----------------

- Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825))
- Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970))
- Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](https://github.com/matrix-org/synapse/issues/10996))
- Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))
2021-10-14 11:42:16 +01:00
David Robertson
99a4e5222d
1.45.0rc2 2021-10-14 10:59:27 +01:00