Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-10-01 08:25:44 -04:00
Code Issues Packages Projects Releases Wiki Activity
17,575 Commits 4 Branches 609 Tags 60 MiB
916bb9d0d1
Commit Graph

3844 Commits

Author SHA1 Message Date
Patrick Cloke
37db6252b7
Convert additional databases to async/await part 3 (#8201) 2020-09-01 11:04:17 -04:00
Richard van der Hoff
5615eb5cb4
Rename _get_e2e_device_keys_txn (#8222) 
... to `_get_e2e_device_keys_and_signatures_txn`, to better reflect what it
does.
 2020-09-01 16:02:17 +01:00
Patrick Cloke
54f8d73c00
Convert additional databases to async/await (#8199) 2020-09-01 09:21:48 -04:00
Patrick Cloke
da77520cd1
Convert additional databases to async/await part 2 (#8200) 2020-09-01 08:39:04 -04:00
Erik Johnston
bbb3c8641c
Make MultiWriterIDGenerator work for streams that use negative stream IDs (#8203) 
This is so that we can use it for the backfill events stream.
 2020-09-01 13:36:25 +01:00
Richard van der Hoff
aa07c37cf0
Move and rename get_devices_with_keys_by_user (#8204) 
* Move `get_devices_with_keys_by_user` to `EndToEndKeyWorkerStore`

this seems a better fit for it.

This commit simply moves the existing code: no other changes at all.

* Rename `get_devices_with_keys_by_user`

to better reflect what it does.

* get_device_stream_token abstract method

To avoid referencing fields which are declared in the derived classes, make
`get_device_stream_token` abstract, and define that in the classes which define
`_device_list_id_gen`.
 2020-09-01 12:41:21 +01:00
Richard van der Hoff
45e8f7726f
Rename get_e2e_device_keys to better reflect its purpose (#8205) 
... and to show that it does something slightly different to
`_get_e2e_device_keys_txn`.

`include_all_devices` and `include_deleted_devices` were never used (and
`include_deleted_devices` was broken, since that would cause `None`s in the
result which were not handled in the loop below.

Add some typing too.
 2020-08-29 00:14:17 +01:00
Andrew Morgan
b4826d6eb1 Fix incorrect return signature 2020-08-28 17:39:48 +01:00
Patrick Cloke
d58fda99ff
Convert event_push_actions, registration, and roommember datastores to async (#8197) 2020-08-28 11:34:50 -04:00
Richard van der Hoff
22b926c284
Only return devices with keys from /federation/v1/user/devices/ (#8198) 
There's not much point in returning all the others, and some people have a
silly number of devices.
 2020-08-28 15:59:28 +01:00
Patrick Cloke
aec7085179
Convert state and stream stores and related code to async (#8194) 2020-08-28 09:37:55 -04:00
Patrick Cloke
5c03134d0f
Convert additional database code to async/await. (#8195) 2020-08-28 07:54:27 -04:00
Patrick Cloke
b49a5b9307
Convert stats and related calls to async/await (#8192) 2020-08-27 17:24:37 -04:00
Patrick Cloke
b71d4a094c
Convert simple_delete to async/await. (#8191) 2020-08-27 14:16:41 -04:00
Patrick Cloke
9b7ac03af3
Convert calls of async database methods to async (#8166) 2020-08-27 13:38:41 -04:00
Patrick Cloke
c9fa696ea2
simple_search_list_txn should return None, not 0. (#8187) 2020-08-27 12:07:13 -04:00
Erik Johnston
5649b7f3d0
Fix missing _add_persisted_position (#8179) 
This was forgotten in #8164.
 2020-08-27 13:20:34 +01:00
Patrick Cloke
30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) (#8168) 2020-08-27 07:41:01 -04:00
Patrick Cloke
4a739c73b4
Convert simple_update* and simple_select* to async (#8173) 2020-08-27 07:08:38 -04:00
Erik Johnston
e3c91a3c55
Make SlavedIdTracker.advance have same interface as MultiWriterIDGenerator (#8171) 2020-08-26 13:15:20 +01:00
Patrick Cloke
4c6c56dc58
Convert simple_select_one and simple_select_one_onecol to async (#8162) 2020-08-26 07:19:32 -04:00
Erik Johnston
eba98fb024
Add functions to MultiWriterIdGen used by events stream (#8164) 2020-08-25 17:32:30 +01:00
Erik Johnston
2231dffee6
Make StreamIdGen get_next and get_next_mult async (#8161) 
This is mainly so that `StreamIdGenerator` and `MultiWriterIdGenerator`
will have the same interface, allowing them to be used interchangeably.
 2020-08-25 15:10:08 +01:00
Andrew Morgan
74bf8d4d06
Wording fixes to 'name' user admin api filter (#8163) 
Some fixes to wording I noticed after merging #7377.
 2020-08-25 15:03:24 +01:00
Manuel Stahl
97962ad17b
Search in columns 'name' and 'displayname' in the admin users endpoint (#7377) 
* Search in columns 'name' and 'displayname' in the admin users endpoint

Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
 2020-08-25 14:18:14 +01:00
Brendan Abolivier
3f49f74610
Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on (#7991) 
* Don't raise session_id errors on submit_token if request_token_inhibit_3pid_errors is set

* Changelog

* Also wait some time before responding to /requestToken

* Incorporate review

* Update synapse/storage/databases/main/registration.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Incorporate review

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2020-08-24 11:33:55 +01:00
Richard van der Hoff
09fd0eda81
Micro-optimisations to get_auth_chain_ids (#8132) 2020-08-21 10:06:45 +01:00
Patrick Cloke
3f91638da6
Allow denying or shadow banning registrations via the spam checker (#8034) 2020-08-20 15:42:58 -04:00
Richard van der Hoff
318f4e738e
Be more tolerant of membership events in unknown rooms (#8110) 
It turns out that not all out-of-band membership events are labelled as such,
so we need to be more accepting here.
 2020-08-20 16:42:12 +01:00
Patrick Cloke
dbc630a628
Use the JSON encoder without whitespace in more places. (#8124) 2020-08-20 10:32:33 -04:00
Patrick Cloke
5eac0b7e76
Add more types to synapse.storage.database. (#8127) 2020-08-20 09:00:59 -04:00
Patrick Cloke
76c43f086a
Do not assume calls to runInteraction return Deferreds. (#8133) 2020-08-20 06:39:55 -04:00
Richard van der Hoff
12aebdfa5a
Close the database connection we create during startup (#8131) 
... otherwise it gets leaked.
 2020-08-19 20:41:53 +01:00
Erik Johnston
c9c544cda5
Remove ChainedIdGenerator. (#8123) 
It's just a thin wrapper around two ID gens to make `get_current_token`
and `get_next` return tuples. This can easily be replaced by calling the
appropriate methods on the underlying ID gens directly.
 2020-08-19 13:41:51 +01:00
Patrick Cloke
eebf52be06
Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
Patrick Cloke
d89692ea84
Convert runWithConnection to async. (#8121) 2020-08-19 07:09:24 -04:00
Erik Johnston
76d21d14a0
Separate get_current_token into two. (#8113) 
The function is used for two purposes: 1) for subscribers of streams to
get a token they can use to get further updates with, and 2) for
replication to track position of the writers of the stream.

For streams with a single writer the two scenarios produce the same
result, however the situation becomes complicated for streams with
multiple writers. The current `MultiWriterIdGenerator` does not
correctly handle the first case (which is not an issue as its only used
for the `caches` stream which nothing subscribes to outside of
replication).
 2020-08-19 10:39:31 +01:00
Patrick Cloke
f40645e60b
Convert events worker database to async/await. (#8071) 2020-08-18 16:20:49 -04:00
Patrick Cloke
3c01724b33
Fix the return type of send_nonmember_events. (#8112) 2020-08-18 09:53:13 -04:00
Patrick Cloke
050e20e7ca
Convert some of the general database methods to async (#8100) 2020-08-17 12:18:01 -04:00
Patrick Cloke
ad6190c925
Convert stream database to async/await. (#8074) 2020-08-17 07:24:46 -04:00
Patrick Cloke
ac77cdb64e
Add a shadow-banned flag to users. (#8092) 2020-08-14 12:37:59 -04:00
Patrick Cloke
b069b78bb4
Convert pusher databases to async/await. (#8075) 2020-08-14 10:30:16 -04:00
Patrick Cloke
e8861957d9
Convert receipts and events databases to async/await. (#8076) 2020-08-14 10:05:19 -04:00
Patrick Cloke
6b7ce1d332
Remove some unused database functions. (#8085) 2020-08-14 09:25:40 -04:00
Patrick Cloke
894dae74fe
Convert misc database code to async (#8087) 2020-08-14 07:24:26 -04:00
Patrick Cloke
fbe930dad2
Convert the roommember database to async/await. (#8070) 2020-08-12 12:14:34 -04:00
Patrick Cloke
5ecc8b5825
Convert devices database to async/await. (#8069) 2020-08-12 10:51:42 -04:00
Patrick Cloke
d68e10f308
Convert account data, device inbox, and censor events databases to async/await (#8063) 2020-08-12 09:29:06 -04:00
Patrick Cloke
a3a59bab7b
Convert appservice, group server, profile and more databases to async (#8066) 2020-08-12 09:28:48 -04:00
Erik Johnston
6ba621d786
Merge pull request #8060 from matrix-org/erikj/type_server 
Change HomeServer definition to work with typing.
 2020-08-11 22:32:14 +01:00
Patrick Cloke
04faa0bfa9
Convert tags and metrics databases to async/await (#8062) 2020-08-11 17:21:20 -04:00
Patrick Cloke
a0acdfa9e9
Converts event_federation and registration databases to async/await (#8061) 2020-08-11 17:21:13 -04:00
Erik Johnston
0f1afbe8dc Change HomeServer definition to work with typing. 
Duplicating function signatures between server.py and server.pyi is
silly. This commit changes that by changing all `build_*` methods to
`get_*` methods and changing the `_make_dependency_method` to work work
as a descriptor that caches the produced value.

There are some changes in other files that were made to fix the typing
in server.py.
 2020-08-11 18:00:17 +01:00
Brendan Abolivier
cdbb8e6d6e
Implement new experimental push rules (#7997) 
With an undocumented configuration setting to enable them for specific users.
 2020-08-10 11:48:01 +01:00
Patrick Cloke
7f837959ea
Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users database to async (#8042) 2020-08-07 13:36:29 -04:00
Patrick Cloke
f3fe6961b2
Convert additional database stores to async/await (#8045) 2020-08-07 12:17:17 -04:00
David Vo
4dd27e6d11
Reduce unnecessary whitespace in JSON. (#7372) 2020-08-07 08:02:55 -04:00
Brendan Abolivier
2ffd6783c7
Revert #7736 (#8039) 2020-08-06 17:15:35 +01:00
Patrick Cloke
d4a7829b12
Convert synapse.api to async/await (#8031) 2020-08-06 08:30:06 -04:00
Brendan Abolivier
118a9eafb3 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules 2020-08-06 10:52:50 +01:00
Brendan Abolivier
dd11f575a2
Incorporate review 2020-08-06 10:52:26 +01:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke
e19de43eb5
Convert streams to async. (#8014) 2020-08-04 07:21:47 -04:00
Andrew Morgan
5d92a1428c
Prevent join->join membership transitions changing member count (#7977) 
`StatsHandler` handles updates to the `current_state_delta_stream`, and updates room stats such as the amount of state events, joined users, etc.

However, it counts every new join membership as a new user entering a room (and that user being in another room), whereas it's possible for a user's membership status to go from join -> join, for instance when they change their per-room profile information.

This PR adds a check for join->join membership transitions, and bails out early, as none of the further checks are necessary at that point.

Due to this bug, membership stats in many rooms have ended up being wildly larger than their true values. I am not sure if we also want to include a migration step which recalculates these statistics (possibly using the `_populate_stats_process_rooms` bg update).

Bug introduced in the initial implementation https://github.com/matrix-org/synapse/pull/4338.
 2020-08-03 21:54:24 +01:00
Brendan Abolivier
1678057b56
Back out the database hack and replace it with a temporary config setting 2020-08-03 11:22:22 +01:00
Brendan Abolivier
cf42d0a60c
Fix cache name 2020-07-31 15:06:41 +01:00
Brendan Abolivier
79d991eff0
Fix cache invalidation calls 2020-07-31 13:58:42 +01:00
Brendan Abolivier
69158e554f Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules 2020-07-30 19:00:29 +01:00
Olivier Wilkinson (reivilibre)
3aa36b782c Merge branch 'master' into develop 2020-07-30 15:18:36 +01:00
Patrick Cloke
b3a97d6dac
Convert some of the data store to async. (#7976) 2020-07-30 07:20:41 -04:00
Erik Johnston
2c1b9d6763
Update worker docs with recent enhancements (#7969) 2020-07-29 23:22:13 +01:00
Patrick Cloke
d90087cffa
Remove from the event_relations table when purging historical events. (#7978) 2020-07-29 13:55:01 -04:00
Brendan Abolivier
8dff4a1242
Re-implement unread counts (#7736) 2020-07-29 18:26:55 +01:00
Patrick Cloke
3345c166a4
Convert storage layer to async/await. (#7963) 2020-07-28 16:09:53 -04:00
Brendan Abolivier
9725c59247
Implement new experimental push rules with a database hack to enable them 2020-07-28 19:20:55 +01:00
Andrew Morgan
8a25332d94
Move some log lines from default logger to sql/transaction loggers (#7952) 
Idea from matrix-org/synapse-dinsic#49
 2020-07-28 18:52:13 +01:00
Patrick Cloke
8144bc26a7
Convert push to async/await. (#7948) 2020-07-27 12:21:34 -04:00
Patrick Cloke
b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Patrick Cloke
13d77464c9
Follow-up to admin API to re-activate accounts (#7908) 2020-07-22 12:33:19 -04:00
Erik Johnston
2d2acc1cf2
Stop using 'device_max_stream_id' (#7882) 
It serves no purpose and updating everytime we write to the device inbox
stream means all such transactions will conflict, causing lots of
transaction failures and retries.
 2020-07-17 17:03:27 +01:00
Erik Johnston
4642fd66df
Change "unknown room ver" logging to warning. (#7881) 
It's somewhat expected for us to have unknown room versions in the
database due to room version experiments.
 2020-07-17 12:10:43 +01:00
Patrick Cloke
f460da6031
Consistently use db_to_json to convert from database values to JSON objects. (#7849) 2020-07-16 11:32:19 -04:00
Richard van der Hoff
a827838706
Merge pull request #7866 from matrix-org/rav/fix_guest_user_id 
Fix guest user registration with lots of client readers
 2020-07-16 13:54:45 +01:00
Richard van der Hoff
c445bc0cad Use a postgres sequence to generate guest user IDs 2020-07-16 13:00:25 +01:00
Richard van der Hoff
3c36ae17a5 Use SequenceGenerator for state group ID allocation 2020-07-16 11:25:08 +01:00
Richard van der Hoff
42509b8fb6 Use PostgresSequenceGenerator from MultiWriterIdGenerator 
partly just to show it works, but alwo to remove a bit of code duplication.
 2020-07-16 11:25:08 +01:00
Richard van der Hoff
90b0cdda42 Add some helper classes for generating ID sequences 2020-07-16 11:25:08 +01:00
Richard van der Hoff
a57df9b827
Avoid brand new rooms in delete_old_current_state_events (#7854) 
When considering rooms to clean up in `delete_old_current_state_events`, skip
rooms which we are creating, which otherwise look a bit like rooms we have
left.

Fixes #7834.
 2020-07-15 18:33:03 +01:00
Richard van der Hoff
1d9dca02f9
remove retry_on_integrity_error wrapper for persist_events (#7848) 
As far as I can tell from the sentry logs, the only time this has actually done
anything in the last two years is when we had two master workers running at
once, and even then, it made a bit of a mess of it (see
https://github.com/matrix-org/synapse/issues/7845#issuecomment-658238739).

Generally I feel like this code is doing more harm than good.
 2020-07-15 10:34:53 +01:00
Patrick Cloke
8d0097bef1
Fix bug in per-room message retention policies. (#7850) 2020-07-14 15:51:13 -04:00
Dirk Klimpel
491f0dab1b
Add delete room admin endpoint (#7613) 
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2020-07-14 12:36:23 +01:00
Richard van der Hoff
6cef918a4b Merge branch 'release-v1.17.0' into develop 2020-07-10 18:38:50 +01:00
Richard van der Hoff
8ccb7f08d9 Merge branch 'master' into release-v1.17.0 2020-07-10 18:38:18 +01:00
Erik Johnston
f299441cc6
Add ability to shard the federation sender (#7798) 2020-07-10 18:26:36 +01:00
Richard van der Hoff
b1beb3ff59 fix migration, again 2020-07-10 12:18:35 +01:00
Richard van der Hoff
cbabcec05c
Drop incorrectly-added table local_rejections_stream. (#7816) 2020-07-10 12:07:59 +01:00
Richard van der Hoff
2ab0b021f1
Generate real events when we reject invites (#7804) 
Fixes #2181. 

The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
 2020-07-09 10:40:19 +01:00
Patrick Cloke
e7efd8f827
Do not use simplejson in Synapse. (#7800) 2020-07-08 07:15:08 -04:00
Richard van der Hoff
76dbd7b8d6
Stop populating unused table local_invites. (#7793) 
This table is no longer used, so we may as well stop populating it. Removing it
would prevent people rolling back to older releases of Synapse, so that can
happen in a future release.
 2020-07-07 14:20:40 +01:00