Commit Graph

206 Commits

Author SHA1 Message Date
Erik Johnston
eee6fcf5fa
Use execute_batch instead of executemany in places (#9181)
`execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
2021-01-21 10:22:53 +00:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster (#9124)
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
2021-01-15 17:18:37 +00:00
Richard van der Hoff
2de7e263ed
Ensure we store pusher data as text (#9117)
I don't think there's any need to use canonicaljson here.

Fixes: #4475.
2021-01-15 16:57:23 +00:00
Patrick Cloke
d34c6e1279
Add type hints to media rest resources. (#9093) 2021-01-15 10:57:37 -05:00
Erik Johnston
d2479c6870
Fix perf of get_cross_signing_keys (#9116) 2021-01-14 17:57:09 +00:00
Erik Johnston
659c415ed4
Fix chain cover background update to work with split out event persisters (#9115) 2021-01-14 17:19:35 +00:00
Erik Johnston
631dd06f2c
Fix get destinations to catch up query. (#9114)
t was doing a sequential scan on `destination_rooms`, which took
minutes.
2021-01-14 16:47:21 +00:00
Erik Johnston
7036e24e98
Add background update for add chain cover index (#9029) 2021-01-14 15:18:27 +00:00
Patrick Cloke
d1eb1b96e8
Register the /devices endpoint on workers. (#9092) 2021-01-13 12:35:40 -05:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932)
This only applies if the user's data is to be erased.
2021-01-12 16:30:15 -05:00
Erik Johnston
c9195744a4
Move more encryption endpoints off master (#9068) 2021-01-11 18:01:27 +00:00
Erik Johnston
1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Erik Johnston
4e04435bda
Remove old tables after schema version bump (#9055)
These tables are unused, and can be dropped now the schema version has been bumped.
2021-01-11 13:58:19 +00:00
Erik Johnston
63f4990298
Ensure rejected events get added to some metadata tables (#9016)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-01-11 13:57:33 +00:00
Jerin J Titus
c21d8f1c1d
Drop last_used column from access_tokens (#9025)
* Dropped last_used column from access_tokens

Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
2021-01-11 10:23:49 +00:00
Erik Johnston
fa5f5cbc74
Fix error handling during insertion of client IPs (#9051)
You can't continue using a transaction once an exception has been
raised, so catching and dropping the error here is pointless and just
causes more errors.
2021-01-08 14:15:20 +00:00
Erik Johnston
b530eaa262
Allow running sendToDevice on workers (#9044) 2021-01-07 20:19:26 +00:00
Erik Johnston
63593134a1
Some cleanups to device inbox store. (#9041) 2021-01-07 17:20:44 +00:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms (#9024)
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-01-07 13:03:38 +00:00
Patrick Cloke
1c9a850562
Add type hints to the crypto module. (#8999) 2021-01-04 10:04:50 -05:00
Patrick Cloke
9999eb2d02
Add type hints to admin and room list handlers. (#8973) 2020-12-29 17:42:10 -05:00
Patrick Cloke
d0c3c24eb2
Drop the unused local_invites table. (#8979)
This table has been unused since Synapse v1.17.0.
2020-12-29 07:26:29 -05:00
Patrick Cloke
5d4c330ed9
Allow re-using a UI auth validation for a period of time (#8970) 2020-12-18 07:33:57 -05:00
Brendan Abolivier
f2783fc201
Use the simple dictionary in full text search for the user directory (#8959)
* Use the simple dictionary in fts for the user directory

* Clarify naming
2020-12-17 14:42:30 +01:00
Dirk Klimpel
06006058d7
Make search statement in List Room and User Admin API case-insensitive (#8931) 2020-12-17 10:43:37 +00:00
Patrick Cloke
bd30cfe86a
Convert internal pusher dicts to attrs classes. (#8940)
This improves type hinting and should use less memory.
2020-12-16 11:25:30 -05:00
Patrick Cloke
be2db93b3c
Do not assume that the contents dictionary includes history_visibility. (#8945) 2020-12-16 08:46:37 -05:00
Patrick Cloke
b3a4b53587
Fix handling of stream tokens for push. (#8943)
Removes faulty assertions and fixes the logic to ensure the max
stream token is always set.
2020-12-15 10:41:34 -05:00
Dirk Klimpel
0a34cdfc66
Add number of local devices to Room Details Admin API (#8886) 2020-12-11 10:42:47 +00:00
Erik Johnston
df4b1e9c74
Pass room_id to get_auth_chain_difference (#8879)
This is so that we can choose which algorithm to use based on the room ID.
2020-12-04 15:52:49 +00:00
Richard van der Hoff
0bac276890 UIA: offer only available auth flows
During user-interactive auth, do not offer password auth to users with no
password, nor SSO auth to users with no SSO.

Fixes #7559.
2020-12-02 18:54:15 +00:00
Richard van der Hoff
9edff901d1
Add missing ordering to background updates (#8850)
It's important that we make sure our background updates happen in a defined
order, to avoid disasters like #6923.

Add an ordering to all of the background updates that have landed since #7190.
2020-12-01 15:52:49 +00:00
Richard van der Hoff
f8d13ca13d
Drop (almost) unused index on event_json (#8845) 2020-11-30 18:44:09 +00:00
Andrew Morgan
d963c69ba5
Speed up remote invite rejection database call (#8815)
This is another PR that grew out of #6739.

The existing code for checking whether a user is currently invited to a room when they want to leave the room looks like the following:

f737368a26/synapse/handlers/room_member.py (L518-L540)

It calls `get_invite_for_local_user_in_room`, which will actually query *all* rooms the user has been invited to, before iterating over them and matching via the room ID. It will then return a tuple of a lot of information which we pull the event ID out of.

I need to do a similar check for knocking, but this code wasn't very efficient. I then tried to write a different implementation using `StateHandler.get_current_state` but this actually didn't work as we haven't *joined* the room yet - we've only been invited to it. That means that only certain tables in Synapse have our desired `invite` membership state. One of those tables is `local_current_membership`.

So I wrote a store method that just queries that table instead
2020-11-25 20:06:13 +00:00
Will Hunt
51338491c9
Improve appservice handler to send only the most recent read receipts when no stream_id is stored. (#8744)
* Make this line debug (it's noisy)

* Don't include from_key for presence if we are at 0

* Limit read receipts for all rooms to 100

* changelog.d/8744.bugfix

* Allow from_key to be None

* Update 8744.bugfix

* The from_key is superflous

* Update comment
2020-11-18 18:54:09 +00:00
Erik Johnston
f737368a26
Add admin API for logging in as a user (#8617) 2020-11-17 10:51:25 +00:00
Andrew Morgan
e8d0853739
Generalise _maybe_store_room_on_invite (#8754)
There's a handy function called maybe_store_room_on_invite which allows us to create an entry in the rooms table for a room and its version for which we aren't joined to yet, but we can reference when ingesting events about.

This is currently used for invites where we receive some stripped state about the room and pass it down via /sync to the client, without us being in the room yet.

There is a similar requirement for knocking, where we will eventually do the same thing, and need an entry in the rooms table as well. Thus, reusing this function works, however its name needs to be generalised a bit.

Separated out from #6739.
2020-11-13 16:24:04 +00:00
Erik Johnston
4cb00d297f
Cache event ID to auth event IDs lookups (#8752)
This should hopefully speed up `get_auth_chain_difference` a bit in the case of repeated state res on the same rooms.

`get_auth_chain_difference` does a breadth first walk of the auth graphs by repeatedly looking up events' auth events. Different state resolutions on the same room will end up doing a lot of the same event to auth events lookups, so by caching them we should speed things up in cases of repeated state resolutions on the same room.
2020-11-13 11:29:18 +00:00
Dirk Klimpel
c3119d1536
Add an admin API for users' media statistics (#8700)
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users.
Related to #6094
It is the first API for statistics.
Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats)

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-11-05 18:59:12 +00:00
Erik Johnston
1eb9de90c0
Improve start time by adding index to e2e_cross_signing_keys (#8694)
We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
2020-11-02 13:55:56 +00:00
Brendan Abolivier
7a0fd6f98d
Fix error handling around when completing an AS transaction (#8693) 2020-10-30 16:50:48 +00:00
Erik Johnston
46f4be94b4
Fix race for concurrent downloads of remote media. (#8682)
Fixes #6755
2020-10-30 10:55:24 +00:00
Andrew Morgan
4504151546
Fix optional parameter in stripped state storage method (#8688)
Missed in #8671.
2020-10-30 00:22:31 +00:00
Erik Johnston
f21e24ffc2
Add ability for access tokens to belong to one user but grant access to another user. (#8616)
We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).

A future PR will add an API for creating such a token.

When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29 15:58:44 +00:00
Richard van der Hoff
c97da1e45d
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
Fix serialisation errors when using third-party event rules.
2020-10-28 20:41:42 +00:00
Richard van der Hoff
b6ca69e4f1 Remove frozendict_json_encoder and support frozendicts everywhere
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
2020-10-28 15:56:57 +00:00
Patrick Cloke
31d721fbf6
Add type hints to application services. (#8655) 2020-10-28 11:12:21 -04:00
Erik Johnston
a6ea1a957e
Don't pull event from DB when handling replication traffic. (#8669)
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-28 12:11:45 +00:00