Commit Graph

4030 Commits

Author SHA1 Message Date
Manuel Stahl
a4a5ec4096
Add room details admin endpoint (#7317) 2020-05-07 15:33:07 -04:00
Erik Johnston
d7983b63a6
Support any process writing to cache invalidation stream. (#7436) 2020-05-07 13:51:08 +01:00
Richard van der Hoff
62ee862119 Merge branch 'release-v1.13.0' into develop 2020-05-06 15:56:03 +01:00
Richard van der Hoff
fa0b2bd28d
Merge pull request #7428 from matrix-org/rav/cross_signing_keys_cache
Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
2020-05-06 12:00:01 +01:00
Richard van der Hoff
16b67c404d Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
... mostly because the latter has a cache.
2020-05-06 11:59:19 +01:00
Richard van der Hoff
db5f9031b7 Fix batching for fetching cross-signing keys
There's no point carefully dividing a list into batches, and then completely
ignoring the batches.
2020-05-06 11:59:19 +01:00
Richard van der Hoff
30a19daa02 Merge branch 'develop' into rav/upsert_for_device_list 2020-05-06 11:43:11 +01:00
Richard van der Hoff
e48361545d use an upsert to update device_lists_outbound_last_success 2020-05-06 11:41:23 +01:00
Richard van der Hoff
0f6ebf393d Better type annotations for simple_upsert_txn
most of these params don't really need to be lists.
2020-05-06 11:41:23 +01:00
Richard van der Hoff
13dd458b8d Merge branch 'release-v1.13.0' into erikj/faster_device_lists_fetch 2020-05-05 18:14:00 +01:00
Erik Johnston
f9073893af Speed up fetching device lists changes in sync.
Currently we copy `users_who_share_room` needlessly about three times,
which is expensive when the set is large (which it can easily be).
2020-05-05 17:40:29 +01:00
Erik Johnston
8123b2f909
Add MultiWriterIdGenerator. (#7281)
This will be used to coordinate stream IDs across multiple writers.

Functions as the equivalent of both `StreamIdGenerator` and
`SlavedIdTracker`.
2020-05-04 17:17:45 +01:00
Richard van der Hoff
97ef1471c6
Wait for current_state_events_membership before populate_stats_process_rooms (#7387)
populate_stats_process_rooms was added in #5971 / v1.4.0; current_state_events_membership was added in #5706 / v1.3.0.

Fixes #7380.
2020-05-01 18:05:58 +01:00
Andrew Morgan
6b22921b19
async/await is_server_admin (#7363) 2020-05-01 15:15:36 +01:00
Patrick Cloke
627b0f5f27
Persist user interactive authentication sessions (#7302)
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
2020-04-30 13:47:49 -04:00
Richard van der Hoff
c2e1a2110f
Fix limit logic for EventsStream (#7358)
* Factor out functions for injecting events into database

I want to add some more flexibility to the tools for injecting events into the
database, and I don't want to clutter up HomeserverTestCase with them, so let's
factor them out to a new file.

* Rework TestReplicationDataHandler

This wasn't very easy to work with: the mock wrapping was largely superfluous,
and it's useful to be able to inspect the received rows, and clear out the
received list.

* Fix AssertionErrors being thrown by EventsStream

Part of the problem was that there was an off-by-one error in the assertion,
but also the limit logic was too simple. Fix it all up and add some tests.
2020-04-29 12:30:36 +01:00
Manuel Stahl
04dd7d182d
Return total number of users and profile attributes in admin users endpoint (#6881)
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-04-28 18:19:36 +01:00
Richard van der Hoff
ce428a1abe Fix EventsStream raising assertions when it falls behind
Figuring out how to correctly limit updates from this stream without dropping
entries is far more complicated than just counting the number of rows being
returned. We need to consider each query separately and, if any one query hits
the limit, truncate the results from the others.

I think this also fixes some potentially long-standing bugs where events or
state changes could get missed if we hit the limit on either query.
2020-04-24 13:59:21 +01:00
Dirk Klimpel
6b6685db9f
Extend room admin api with additional attributes (#7225) 2020-04-22 13:38:41 +01:00
Richard van der Hoff
f2049a8d21
Fix a potentially-huge sql query (#7274)
We could end up looking up tens of thousands of events, which could cause large
amounts of data to be logged to the postgres log.
2020-04-15 10:16:35 +01:00
Matthew Hodgson
118b58f0c9 typos 2020-04-11 20:55:18 +01:00
Richard van der Hoff
f31e65a749
bg update to clear out duplicate outbound_device_list_pokes (#7193)
We seem to have some duplicates, which could do with being cleared out.
2020-04-07 23:06:39 +01:00
Patrick Cloke
1722b8a527
Convert delete_url_cache_media to async/await. (#7241) 2020-04-07 21:56:34 +01:00
Richard van der Hoff
2e105c156b
Remove sent outbound device list pokes from the database (#7192)
They just get in the way.
2020-04-07 15:19:19 +01:00
Erik Johnston
ce72355d7f
Fix race in replication (#7226)
Fixes a race between handling `POSITION` and `RDATA` commands. We do this by simply linearizing handling of them.
2020-04-07 11:01:04 +01:00
Andrew Morgan
4b0f00ad0c
Remove stream before/after debug log lines (#7207) 2020-04-06 12:40:34 +01:00
Richard van der Hoff
fd4c975b5b
Merge pull request #7190 from matrix-org/rav/one_bg_update_at_a_time
Only run one background update at a time
2020-04-03 13:17:30 +01:00
Richard van der Hoff
fcc2de7a0c Update docstring per review comments 2020-04-03 10:51:32 +01:00
Richard van der Hoff
daa1ac89a0
Fix device list update stream ids going backward (#7158)
Occasionally we could get a federation device list update transaction which
looked like:

```
[
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D2', 'prev_id': [], 'stream_id': 12, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D1', 'prev_id': [12], 'stream_id': 11, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D3', 'prev_id': [11], 'stream_id': 13, 'deleted': True}}
]
```

Having `stream_ids` which are lower than `prev_ids` looks odd. It might work
(I'm not actually sure), but in any case it doesn't seem like a reasonable
thing to expect other implementations to support.
2020-04-03 10:40:22 +01:00
Richard van der Hoff
af47264b78
review comment 2020-04-02 12:04:55 +01:00
Richard van der Hoff
7b608cf468 Only run one background update at a time 2020-03-31 17:43:58 +01:00
Richard van der Hoff
b4c2234232 Make do_next_background_update return a bool
returning a None or an int that we don't use is confusing.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
26d17b9bdc Make has_completed_background_updates async
(Almost) everywhere that uses it is happy with an awaitable.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
cfe8c8ab8e Remove unused start_background_update
This was only used in a unit test, so let's just inline it in the test.
2020-03-31 17:24:06 +01:00
Karlinde
fe1580bfd9
Fill in the 'default' field for user-defined push rules (#6639)
Signed-off-by: Karl Linderhed <git@karlinde.se>
2020-03-31 15:08:56 +01:00
Andrew Morgan
b994e86e35
Only setdefault for signatures if device has key_json (#7177) 2020-03-31 14:51:22 +01:00
Patrick Cloke
0a7b0882c1
Fix use of async/await in media code (#7184) 2020-03-31 09:33:02 -04:00
Richard van der Hoff
7966a1cde9
Rewrite prune_old_outbound_device_pokes for efficiency (#7159)
make sure we clear out all but one update for the user
2020-03-30 19:06:52 +01:00
Andrew Morgan
7042840b32
Transfer alias mappings when joining an upgraded room (#6946) 2020-03-30 17:53:25 +01:00
Andrew Morgan
12aa5a7fa7
Ensure is_verified on /_matrix/client/r0/room_keys/keys is a boolean (#7150) 2020-03-27 13:30:22 +00:00
David Vo
fbf0782c63
Only import sqlite3 when type checking (#7155)
Fixes: #7127
Signed-off-by: David Vo <david@vovo.id.au>
2020-03-27 13:20:00 +00:00
Erik Johnston
4cff617df1
Move catchup of replication streams to worker. (#7024)
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
2020-03-25 14:54:01 +00:00
Richard van der Hoff
39230d2171
Clean up some LoggingContext stuff (#7120)
* Pull Sentinel out of LoggingContext

... and drop a few unnecessary references to it

* Factor out LoggingContext.current_context

move `current_context` and `set_context` out to top-level functions.

Mostly this means that I can more easily trace what's actually referring to
LoggingContext, but I think it's generally neater.

* move copy-to-parent into `stop`

this really just makes `start` and `stop` more symetric. It also means that it
behaves correctly if you manually `set_log_context` rather than using the
context manager.

* Replace `LoggingContext.alive` with `finished`

Turn `alive` into `finished` and make it a bit better defined.
2020-03-24 14:45:33 +00:00
Erik Johnston
fdb1344716
Remove concept of a non-limited stream. (#7011) 2020-03-20 14:40:47 +00:00
Erik Johnston
a319cb1dd1
Change device list streams to have one row per ID (#7010)
* Add 'device_lists_outbound_pokes' as extra table.

This makes sure we check all the relevant tables to get the current max
stream ID.

Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.

* Change device lists stream to have one row per id.

This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.

* Change device list replication to match new semantics.

Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).

* Newsfile

* Remove handling of multiple rows per ID

* Fix worker handling

* Comments from review
2020-03-19 11:36:53 +00:00
Erik Johnston
4a17a647a9
Improve get auth chain difference algorithm. (#7095)
It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.

Instead, lets try and fetch the auth chains at the same time and
calculate the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.
2020-03-18 16:46:41 +00:00
Erik Johnston
6e6476ef07 Comments from review 2020-03-18 10:13:55 +00:00
Brendan Abolivier
7df04ca0e6
Populate the room version from state events (#7070)
Fixes #7065 

This is basically the same as https://github.com/matrix-org/synapse/pull/6847 except it tries to populate events from `state_events` rather than `current_state_events`, since the latter might have been cleared from the state of some rooms too early, leaving them with a `NULL` room version.
2020-03-16 22:31:47 +00:00
Brendan Abolivier
dc6fb56c5f
Hopefully mypy is happy now 2020-03-10 14:40:28 +00:00
Brendan Abolivier
14b2ebe767
Merge pull request #7055 from matrix-org/babolivier/get_time_of_last_push_action_before
Move get_time_of_last_push_action_before to the EventPushActionsWorkerStore
2020-03-09 14:53:50 +00:00
Brendan Abolivier
87c65576e0
Move get_time_of_last_push_action_before to the EventPushActionsWorkerStore
Fixes #7054

I also had a look at the rest of the functions in
`EventPushActionsStore` and in the push notifications send code and it
looks to me like there shouldn't be any other method with this issue in
this part of the codebase.
2020-03-09 13:58:38 +00:00
Patrick Cloke
06eb5cae08
Remove special auth and redaction rules for aliases events in experimental room ver. (#7037) 2020-03-09 08:58:25 -04:00
Neil Johnson
1d66dce83e
Break down monthly active users by appservice_id (#7030)
* Break down monthly active users by appservice_id and emit via prometheus.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2020-03-06 18:14:19 +00:00
Richard van der Hoff
8ef8fb2c1c
Read the room version from database when fetching events (#6874)
This is a precursor to giving EventBase objects the knowledge of which room version they belong to.
2020-03-04 13:11:04 +00:00
Erik Johnston
65a941d1f8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_devices_stream 2020-03-02 16:55:55 +00:00
Erik Johnston
b29474e0aa
Always return a deferred from get_current_state_deltas. (#7019)
This currently causes presence notify code to log exceptions when there
is no state changes to process. This doesn't actually cause any problems
as we'd simply do nothing anyway.
2020-03-02 16:52:15 +00:00
Erik Johnston
f70f44abc7 Remove handling of multiple rows per ID 2020-02-28 11:45:35 +00:00
Erik Johnston
9ce4e344a8 Change device list replication to match new semantics.
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
2020-02-28 11:25:34 +00:00
Erik Johnston
f5caa1864e Change device lists stream to have one row per id.
This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.
2020-02-28 11:21:25 +00:00
Erik Johnston
c3c6c0e622 Add 'device_lists_outbound_pokes' as extra table.
This makes sure we check all the relevant tables to get the current max
stream ID.

Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.
2020-02-28 11:15:11 +00:00
Dirk Klimpel
9b06d8f8a6
Fixed set a user as an admin with the new API (#6928)
Fix #6910
2020-02-28 09:58:05 +00:00
Richard van der Hoff
132b673dbe
Add some type annotations in synapse.storage (#6987)
I cracked, and added some type definitions in synapse.storage.
2020-02-27 11:53:40 +00:00
Richard van der Hoff
3e99528f2b
Store room version on invite (#6983)
When we get an invite over federation, store the room version in the rooms table.

The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...
2020-02-26 16:58:33 +00:00
Uday Bansal
7728d87fd7
Updated warning for incorrect database collation/ctype (#6985)
Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
2020-02-26 15:17:03 +00:00
Richard van der Hoff
e66f099ca9
Sanity-check database before running upgrades (#6982)
Some of the database deltas rely on `config.server_name` being set correctly,
so we should check that it is before running the deltas.

Fixes #6870.
2020-02-25 17:46:00 +00:00
Erik Johnston
bbf8886a05
Merge worker apps into one. (#6964) 2020-02-25 16:56:55 +00:00
Richard van der Hoff
a301934f46
Upsert room version when we join over federation (#6968)
This is intended as a precursor to storing room versions when we receive an
invite over federation, but has the happy side-effect of fixing #3374 at last.

In short: change the store_room with try/except to a proper upsert which
updates the right columns.
2020-02-24 15:46:41 +00:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Hubert Chathi
a90d0dc5c2
don't insert into the device table for remote cross-signing keys (#6956) 2020-02-20 09:59:00 -05:00
Ruben Barkow-Kuder
4fb5f4d0ce
Add some clarifications to README.md in the database schema directory. (#6615)
Signed-off-by: Ruben Barkow-Kuder <github@r.z11.de>
2020-02-20 10:37:57 +00:00
Erik Johnston
7b7c3cedf2 Minor perf fixes to get_auth_chain_ids. 2020-02-19 15:47:11 +00:00
Erik Johnston
2b37eabca1
Reduce auth chains fetched during v2 state res. (#6952)
The state res v2 algorithm only cares about the difference between auth
chains, so we can pass in the known common state to the `get_auth_chain`
storage function so that it can ignore those events.
2020-02-19 15:04:47 +00:00
Erik Johnston
099c96b89b
Revert get_auth_chain_ids changes (#6951) 2020-02-19 11:37:35 +00:00
Erik Johnston
0d0bc35792
Increase DB/CPU perf of _is_server_still_joined check. (#6936)
* Increase DB/CPU perf of `_is_server_still_joined` check.

For rooms with large amount of state a single user leaving could cause
us to go and load a lot of membership events and then pull out
membership state in a large number of batches.

* Newsfile

* Update synapse/storage/persist_events.py

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Fix adding if too soon

* Update docstring

* Review comments

* Woops typo

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-02-19 10:15:49 +00:00
Erik Johnston
5a5abd55e8
Limit size of get_auth_chain_ids query (#6947) 2020-02-19 09:39:26 +00:00
Erik Johnston
8a380d0fe2
Increase perf of get_auth_chain_ids used in state res v2. (#6937)
We do this by moving the recursive query to be fully in the DB.
2020-02-18 15:39:09 +00:00
Richard van der Hoff
46fa66bbfd
wait for current_state_events_membership before delete_old_current_state_events (#6924) 2020-02-17 11:30:50 +00:00
Richard van der Hoff
e1d858984d
Remove unused get_room_stats_state method. (#6869) 2020-02-07 15:30:26 +00:00
Erik Johnston
de2d267375
Allow moving group read APIs to workers (#6866) 2020-02-07 11:14:19 +00:00
Erik Johnston
ed630ea17c
Reduce amount of logging at INFO level. (#6862)
A lot of the things we log at INFO are now a bit superfluous, so lets
make them DEBUG logs to reduce the amount we log by default.

Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2020-02-06 13:31:05 +00:00
Erik Johnston
2201ef8556 Synapse 1.10.0rc2 (2020-02-06)
==============================
 
 Bugfixes
 --------
 
 - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
 - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))
 
 Internal Changes
 ----------------
 
 - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl478l8QHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHmKfD/0ZpBV95gMk0iecfvO6IY8+0KigzDUypgXf
 zp1k+k5DBUmH5/83h7+cXWHIJyofv1At1YdIq9J/nNeyqr3V3dcQwPabOKYbI96d
 kVD0EZX+2NjuYejAuF9eJNdiwRq5yMLluCfnOXr5jCS0NNdpO1xVb4bWYm50RmcD
 WJvBvmfwXwCC3AFyNb4IAipaVUXuVXXx1YhjFs6DsbtpPUG88e4uIhpfvrS4v0t8
 GFJZNuVJStOvyKHHTfNRIMkhy4xidj3HRUMSmjNpx07WnPBnbv4LnUOD1boYxgaP
 EoODYnoAPshdiKB0AywNNjue3TmFD/Z7vVEzlFP/lNZ4GDU4kN9C/EwFYw0C2Jqq
 f9O/E2ZuP9Qqume5O9UwMQQAmhV5lMBaIsYRbixU+9bSB893zRHRffA11HypzD00
 ZXj8QDOXpiXp2jpAwN1Rk9e/aZEX3qd+zUAgRmk+kYb0KTC2/gcY956rodNSSO/U
 RFYg4DFvoCgCrRSxZ4LQMlFu3YY2E7qWH+p/OHk3WG79jW64VpEFaytvv8fR+GKq
 g3EbtWy6mUzlKYqbjZ+ZTUDe5AWdv6ZX8xJqRD/S6cpiwyh6Gp89HHNvBThXoCmK
 fxkgw8if7eIITuTlNuDrqunxyWqwd3oVlzd2mi2bg0yRfcqJ9C6OuBV1VTLFZeky
 3sjCiU0IRw==
 =kcSy
 -----END PGP SIGNATURE-----

Merge tag 'v1.10.0rc2' into develop

Synapse 1.10.0rc2 (2020-02-06)
==============================

Bugfixes
--------

- Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
- Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))

Internal Changes
----------------

- Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
2020-02-06 11:04:03 +00:00
Hubert Chathi
60d0672426 Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed
add device signatures to device key query results
2020-02-05 10:54:49 +00:00
Michael Kaye
a831d2e4e3
Reduce performance logging to DEBUG (#6833)
* Reduce tnx performance logging to DEBUG
* Changelog.d
2020-02-05 08:57:37 +00:00
Richard van der Hoff
d88e0ec080
Database updates to populate rooms.room_version (#6847)
We're going to need this so that we can figure out how to handle redactions when fetching events from the database.
2020-02-04 21:31:08 +00:00
Erik Johnston
6475382d80
Fix detecting unknown devices from remote encrypted events. (#6848)
We were looking at the wrong event type (`m.room.encryption` vs
`m.room.encrypted`).

Also fixup the duplicate `EvenTypes` entries.

Introduced in #6776.
2020-02-04 17:25:54 +00:00
Hubert Chathi
23d8a55c7a add device signatures to device key query results 2020-02-04 00:13:12 -05:00
Erik Johnston
83b0ea047b
Fix deleting of stale marker for device lists (#6819)
We were in fact only deleting stale marker when we got an incremental
update, rather than when we did a full resync.
2020-01-31 14:04:15 +00:00
Richard van der Hoff
08f41a6f05 Add get_room_version method
So that we can start factoring out some of this boilerplatey boilerplate.
2020-01-31 10:28:15 +00:00
Richard van der Hoff
d7bf793cc1 s/get_room_version/get_room_version_id/
... to make way for a forthcoming get_room_version which returns a RoomVersion
object.
2020-01-31 10:06:21 +00:00
Erik Johnston
e0992fcc5b
Log when we delete room in bg update (#6816) 2020-01-30 17:55:34 +00:00
Erik Johnston
57ad702af0
Backgroud update to clean out rooms from current state (#6802) 2020-01-30 17:17:44 +00:00
Erik Johnston
a5bab2d058
When server leaves room check for stale device lists. (#6801)
When a server leaves a room it may stop sharing a room with remote
users, and thus not get any updates to their device lists. So we need to
check for this case and delete those device lists from the cache.

We don't need to do this if we stop sharing a room because the remote
user leaves the room, because we track that case via looking at
membership changes.
2020-01-30 16:10:30 +00:00
Erik Johnston
c80a9fe13d
When a client asks for remote keys check if should resync. (#6797)
If we detect that the remote users' keys may have changed then we should
attempt to resync against the remote server rather than using the
(potentially) stale local cache.
2020-01-30 15:06:58 +00:00
Erik Johnston
611215a49c
Delete current state when server leaves a room (#6792)
Otherwise its just stale data, which may get deleted later anyway so
can't be relied on. It's also a bit of a shotgun if we're trying to get
the current state of a room we're not in.
2020-01-29 11:01:32 +00:00
Erik Johnston
e17a110661
Detect unknown remote devices and mark cache as stale (#6776)
We just mark the fact that the cache may be stale in the database for
now.
2020-01-28 14:43:21 +00:00
Erik Johnston
02b44db922
Warn if postgres database has non-C locale. (#6734)
As using non-C locale can cause issues on upgrading OS.
2020-01-28 13:44:21 +00:00
Erik Johnston
33f904835a Merge branch 'master' into develop 2020-01-28 13:39:39 +00:00
Erik Johnston
bdbeeb94ec
Fix setting mau_limit_reserved_threepids config (#6793)
Calling the invalidation function during initialisation of the data
stores introduces a circular dependency, causing Synapse to fail to
start.
2020-01-28 13:05:24 +00:00
Erik Johnston
8df862e45d
Add rooms.room_version column (#6729)
This is so that we don't have to rely on pulling it out from `current_state_events` table.
2020-01-27 14:30:57 +00:00
Erik Johnston
d5275fc55f
Propagate cache invalidates from workers to other workers. (#6748)
Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers.
2020-01-27 13:47:50 +00:00
Erik Johnston
fa4d609e20
Make 'event.redacts' never raise. (#6771)
There are quite a few places that we assume that a redaction event has a
corresponding `redacts` key, which is not always the case. So lets
cheekily make it so that event.redacts just returns None instead.
2020-01-23 15:19:03 +00:00
Andrew Morgan
ce84dd9e20
Remove unnecessary abstractions in admin handler (#6751) 2020-01-22 15:09:57 +00:00
Andrew Morgan
90a28fb475
Admin API to list, filter and sort rooms (#6720) 2020-01-22 13:36:43 +00:00
Neil Johnson
5e52d8563b Allow monthly active user limiting support for worker mode, fixes #4639. (#6742) 2020-01-22 11:05:14 +00:00
Erik Johnston
5d7a6ad223
Allow streaming cache invalidate all to workers. (#6749) 2020-01-22 10:37:00 +00:00
Erik Johnston
0e68760078
Add a DeltaState to track changes to be made to current state (#6716) 2020-01-20 18:07:20 +00:00
Satsuki Yanagi
722b4f302d Fix syntax error in run_upgrade for schema 57 (#6728)
Fix #6727
Related #6655

Co-authored-by: Erik Johnston <erikj@jki.re>
2020-01-17 14:30:35 +00:00
Brendan Abolivier
3b72bb780a
Merge pull request #6714 from matrix-org/babolivier/retention_select_event
Fix instantiation of message retention purge jobs
2020-01-17 14:23:51 +00:00
Richard van der Hoff
14d8f342d5 move batch_iter to a separate module 2020-01-16 22:25:32 +00:00
Brendan Abolivier
dac148341b
Fixup diff 2020-01-16 20:25:09 +00:00
Brendan Abolivier
842c2cfbf1
Remove get_room_event_after_stream_ordering entirely 2020-01-16 20:24:17 +00:00
Erik Johnston
d386f2f339
Add StateMap type alias (#6715) 2020-01-16 13:31:22 +00:00
Brendan Abolivier
e601f35d3b
Lint 2020-01-16 09:55:11 +00:00
Brendan Abolivier
066b9f52b8
Correctly order when selecting before stream ordering 2020-01-15 19:32:47 +00:00
Brendan Abolivier
8363588237
Fix typo 2020-01-15 19:13:22 +00:00
Brendan Abolivier
855af069a4
Fix instantiation of message retention purge jobs
When figuring out which topological token to start a purge job at, we
need to do the following:

1. Figure out a timestamp before which events will be purged
2. Select the first stream ordering after that timestamp
3. Select info about the first event after that stream ordering
4. Build a topological token from that info

In some situations (e.g. quiet rooms with a short max_lifetime), there
might not be an event after the stream ordering at step 3, therefore we
abort the purge with the error `No event found`. To mitigate that, this
patch fetches the first event _before_ the stream ordering, instead of
after.
2020-01-15 18:56:18 +00:00
Erik Johnston
19a1aac48c
Fix purge_room admin API (#6711) 2020-01-15 18:13:47 +00:00
Erik Johnston
28c98e51ff
Add local_current_membership table (#6655)
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
2020-01-15 14:59:33 +00:00
Andrew Morgan
1177d3f3a3
Quarantine media by ID or user ID (#6681) 2020-01-13 18:10:43 +00:00
Richard van der Hoff
bf46821180 Refuse to start if sqlite is older than 3.11.0 2020-01-09 18:11:04 +00:00
Richard van der Hoff
e48ba84e0b Check postgres version in check_database
this saves doing it on each connection, and will allow us to pass extra options
in.
2020-01-09 18:05:59 +00:00
Richard van der Hoff
e97d1cf001 Modify check_database to take a connection rather than a cursor
We might not need the cursor at all.
2020-01-09 18:05:50 +00:00
Manuel Stahl
d2906fe666 Allow admin users to create or modify users without a shared secret (#6495)
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-01-09 13:31:00 +00:00
Erik Johnston
7c232bd98b
Merge pull request #6664 from matrix-org/erikj/media_admin_apis
Fix media repo admin APIs when using a media worker.
2020-01-08 15:50:06 +00:00
Erik Johnston
d74054afda Shuffle the code 2020-01-08 14:57:45 +00:00
Erik Johnston
bca3455b38 Comments 2020-01-08 14:27:35 +00:00
Erik Johnston
187dc6ad02 Do not rely on streaming events, as media repo doesn't 2020-01-08 14:24:28 +00:00
Manuel Stahl
7caaa29daa Fix GET request on /_synapse/admin/v2/users endpoint (#6563)
Fixes #6552
2020-01-08 13:26:40 +00:00
Erik Johnston
3cf7d6d5b6 Move media admin store functions to worker store 2020-01-08 13:26:20 +00:00
Brendan Abolivier
0ab5853ec9
Merge pull request #6652 from matrix-org/babolivier/depth_missing_events
Fix conditions failing if min_depth = 0
2020-01-07 15:22:07 +01:00
Richard van der Hoff
85db7f73be
Add a background update to clear tombstoned rooms from the directory (#6648)
* Add a background update to clear tombstoned rooms from the directory

* use the ABC metaclass
2020-01-07 14:18:43 +00:00
Richard van der Hoff
9824a39d80
Async/await for background updates (#6647)
so that bg update routines can be async
2020-01-07 14:12:42 +00:00
Richard van der Hoff
1ff5491117
Merge pull request #6645 from matrix-org/rav/fix_synchrotron_error
Fix exceptions in the synchrotron worker log when events are rejected.
2020-01-07 14:02:14 +00:00
Brendan Abolivier
cd428a93e2
Fix conditions failing if min_depth = 0
This could result in Synapse not fetching prev_events for new events in the room if it has missed some events.
2020-01-07 12:08:58 +00:00
Richard van der Hoff
1807db5e73
Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashes
Remove a bunch of unused code from event creation
2020-01-06 17:46:31 +00:00
Richard van der Hoff
bc42da4ab8 Clarify documentation on get_event* methods
Make it clearer how they behave in the face of rejected and/or missing events.
2020-01-06 17:12:06 +00:00
Richard van der Hoff
ba897a7590
Fix some test failures when frozen_dicts are enabled (#6642)
Fixes #4026
2020-01-06 15:22:46 +00:00
Erik Johnston
9f6c1befbb
Add experimental 'databases' config (#6580) 2020-01-06 14:44:01 +00:00
Richard van der Hoff
a7d2e5b37f Remove unused get_latest_event_ids_and_hashes_in_room 2020-01-06 13:45:33 +00:00
Richard van der Hoff
dc41fbf0dd Remove unused get_prev_events_and_hashes_for_room 2020-01-06 13:45:33 +00:00
Richard van der Hoff
15720092ac replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in create_new_client_event 2020-01-06 13:45:33 +00:00
Richard van der Hoff
5a04781643 rename get_prev_events_for_room to get_prev_events_and_hashes_for_room
... to make way for a new method which just returns the event ids
2020-01-06 13:45:33 +00:00
Erik Johnston
75d8f26ac8
Split state groups into a separate data store (#6296) 2019-12-20 10:48:24 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Erik Johnston
0b5dbadd96
Explode on duplicate delta file names. (#6565) 2019-12-19 15:07:37 +00:00
Richard van der Hoff
bca30cefee
Improve diagnostics on database upgrade failure (#6570)
`Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst
has anything useful.
2019-12-19 14:53:15 +00:00
Richard van der Hoff
d6752ce5da
Clean up startup for the pusher (#6558)
* Remove redundant python2 support code

`str.decode()` doesn't exist on python3, so presumably this code was doing
nothing

* Filter out pushers with corrupt data

When we get a row with unparsable json, drop the row, rather than returning a
row with null `data`, which will then cause an explosion later on.

* Improve logging when we can't start a pusher

Log the ID to help us understand the problem

* Make email pusher setup more robust

We know we'll have a `data` member, since that comes from the database. What we
*don't* know is if that is a dict, and if that has a `brand` member, and if
that member is a string.
2019-12-18 14:26:58 +00:00
Andrew Morgan
7963ca83cb
Add delta file to fix missing default table data (#6555) 2019-12-18 11:13:33 +00:00
Erik Johnston
2284eb3a53
Add database config class (#6513)
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Werner Sembach
9d173b312c Automatically delete empty groups/communities (#6453)
Signed-off-by: Werner Sembach <werner.sembach@fau.de>
2019-12-16 12:12:40 +00:00
Hubert Chathi
cb2db17994
look up cross-signing keys from the DB in bulk (#6486) 2019-12-12 12:03:28 -05:00
Andrew Morgan
5bfd8855d6
Fix redacted events being returned in search results ordered by "recent" (#6522) 2019-12-12 15:53:49 +00:00
Erik Johnston
adb3a873fd Synapse 1.7.0rc2 (2019-12-11)
=============================
 
 Bugfixes
 --------
 
 - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
 - Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
 - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl3w+RwQHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHijfD/43EQ58jqKvD+qKZwVFOE6JJ3SCS7UJbi3f
 zq9KWDuCB6EAFjAgmJikbBBqHPO5qq2WtNXaCpexx44s8Mk8SZKHg56dP6Fk651C
 assAb/Nsh6CAlPUcRkx8I0L/kYXMPDyATlLVBHVOi3pFDJ093mdOQ4q8yP9iUTM+
 OPsbT8k/pMhrhCH951bGmB6/SEcju+ubObW+bRFe8o3v1KE9jVYQjUGMhuoXp3pM
 z/OB8idZcqOvCc6HMo83tg9FuI613Jy80PMIc1ofyJgvnu+aDBepWuldvFEMnmSR
 D862jMor7+WdnDOTeWZrC+DXjl0qCP8F6ahs5rEllRglt/Ep2wEDPA/8YrRoEI3V
 RWe9W7XDdFCXdzlvXheOfETqTu9kdsurTwBEeJrWQ0vOLY86hxt9KKKcHVhy5eKq
 kNfRtvSVLmRhIssp7hVBcywRwnaxN7R2OoRq/TWnTZz+xEOPzYFU6r0l9kk5dbg6
 fqYYxIXbgZlhjihLWNIMNwwZH9ll/eoPiinkRTZNz40THP/VDTR9DM4tQ6jrhrr6
 sP0qM7LljvrdiimXtV00tUDyUspNgJl6xDJyGDHWM9uoCB7uorEpMQZSzlZhZe8s
 6q+fQPHlfW4JYOejfihPkjrV1ViawvEucqWPaIsD+v26C4RP7qCTtYj3NPiA65of
 zhOjomWWcg==
 =ABoZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.7.0rc2' into develop

Synapse 1.7.0rc2 (2019-12-11)
=============================

Bugfixes
--------

- Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
- Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
- Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
2019-12-11 14:14:30 +00:00
Andrew Morgan
fc316a4894
Prevent redacted events from appearing in message search (#6377) 2019-12-11 13:39:47 +00:00
Andrew Morgan
6676ee9c4a
Add dev script to generate full SQL schema files (#6394) 2019-12-11 13:16:01 +00:00
Andrew Morgan
ea0f0ad414
Prevent message search in upgraded rooms we're not in (#6385) 2019-12-11 13:07:25 +00:00
Erik Johnston
d21577bdcb Merge branch 'erikj/fix_sqlite_7' of github.com:matrix-org/synapse into release-v1.7.0 2019-12-11 11:34:50 +00:00
Richard van der Hoff
40eda84933
Fix race which caused deleted devices to reappear (#6514)
Stop the `update_client_ips` background job from recreating deleted devices.
2019-12-10 16:22:29 +00:00
Brendan Abolivier
3bd049bbb7
Give the server config to the RoomWorkerStore 2019-12-10 13:05:35 +00:00
Erik Johnston
52346990c8 Drop unused index 2019-12-10 12:46:14 +00:00
Erik Johnston
31da85e467 Convert _censor_redactions to async since it awaits on coroutines 2019-12-10 12:46:00 +00:00
Erik Johnston
cc5f6eb608 Only start censor background job after indices are created 2019-12-10 11:39:31 +00:00
Erik Johnston
4cade96616 Fix support for SQLite 3.7.
Partial indices support was added in 3.8.0, so we need to use the
background updates that handles this correctly.
2019-12-09 15:09:16 +00:00
Erik Johnston
30e9adf32f
Merge pull request #6487 from matrix-org/erikj/pass_in_db
Pass in Database object to data stores.
2019-12-09 13:53:21 +00:00
Erik Johnston
65b37f6729 Fix comment
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-12-09 11:56:43 +00:00
Erik Johnston
a9b393340f
Merge pull request #6484 from matrix-org/erikj/port_sync_handler
Port SyncHandler to async/await
2019-12-09 11:32:44 +00:00
Erik Johnston
71ee22c0ba Fix port db script 2019-12-06 16:41:48 +00:00
Erik Johnston
75f87450d8 Move start up DB checks to main data store. 2019-12-06 16:02:21 +00:00
Erik Johnston
d537be1ebd Pass Database into the data store 2019-12-06 15:49:44 +00:00
Erik Johnston
d64bb32a73 Move are_all_users_on_domain checks to main data store. 2019-12-06 13:43:40 +00:00
Erik Johnston
9a4fb457cf Change DataStores to accept 'database' param. 2019-12-06 13:30:06 +00:00
Erik Johnston
2ace775d88 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_database_class 2019-12-06 11:33:34 +00:00
Erik Johnston
e216ec381a Remove unused var 2019-12-06 11:16:37 +00:00
Erik Johnston
b3a4e35ca8 Fixup functions to consistently return deferreds 2019-12-06 10:40:05 +00:00
Manuel Stahl
649b6bc088 Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925) 2019-12-05 18:12:23 +00:00
Erik Johnston
4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston
8863624f78 Comments 2019-12-05 10:46:37 +00:00
Erik Johnston
756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston
ddbbfc9512
Merge pull request #6464 from matrix-org/erikj/make_public_sql_base
Clean up SQLBaseStore private function usage
2019-12-05 10:43:49 +00:00
Brendan Abolivier
fe799f353d
Merge pull request #6470 from matrix-org/babolivier/port_db_ci_failure
Make synapse_port_db exit with a non-0 code if something failed
2019-12-04 18:20:36 +00:00
Brendan Abolivier
f8421a1404
Fix background updates for synapse_port_db 2019-12-04 17:57:35 +00:00
Richard van der Hoff
e203874caa
get rid of (most of) have_events from _update_auth_events_and_context_for_auth (#6468)
have_events was a map from event_id to rejection reason (or None) for events
which are in our local database. It was used as filter on the list of
event_ids being passed into get_events_as_list. However, since
get_events_as_list will ignore any event_ids that are unknown or rejected, we
can equivalently just leave it to get_events_as_list to do the filtering.

That means that we don't have to keep `have_events` up-to-date, and can use
`have_seen_events` instead of `get_seen_events_with_rejection` in the one place
we do need it.
2019-12-04 17:27:32 +00:00
Erik Johnston
ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Erik Johnston
c2f525a525 Don't call SQLBaseStore methods from outside stores 2019-12-04 16:23:43 +00:00
Erik Johnston
3eb15c01d9
Merge pull request #6454 from matrix-org/erikj/clean_base_Store
Move things out of SQLBaseStore
2019-12-04 16:23:19 +00:00
Erik Johnston
9186c105a0 Revert "Move get_user_count_txn out of base store"
This reverts commit 00f0d67566.

Its going to get removed soon, so lets not make merge conflicts.
2019-12-04 15:46:19 +00:00
Erik Johnston
a7f20500ff _CURRENT_STATE_CACHE_NAME is public 2019-12-04 15:45:42 +00:00
Brendan Abolivier
c530f9af4d
Merge pull request #6329 from matrix-org/babolivier/context_filters
Filter state, events_before and events_after in /context requests
2019-12-04 15:24:16 +00:00
Erik Johnston
00f0d67566 Move get_user_count_txn out of base store 2019-12-04 15:21:14 +00:00
Erik Johnston
1056d6885a Move cache invalidation to main data store 2019-12-04 15:21:14 +00:00
Erik Johnston
6b2867096b Move event fetch vars to EventWorkStore 2019-12-04 15:21:14 +00:00
Erik Johnston
ddd48b6851 Move account validity bg updates to registration store 2019-12-04 15:21:14 +00:00
Brendan Abolivier
9dc84b7989
Merge branch 'develop' into babolivier/context_filters 2019-12-04 14:23:44 +00:00
Erik Johnston
2aa8943809
Merge pull request #6451 from matrix-org/uhoreg/cross_signing_signatures_index
make cross signing signature index non-unique
2019-12-04 09:57:09 +00:00
Richard van der Hoff
0120875462
Fix exception when a cross-signed device is deleted (#6462)
(hopefully)

... and deobfuscate the relevant bit of code.
2019-12-04 07:38:35 +00:00
Hubert Chathi
418813b205 apply changes from review 2019-12-03 15:27:00 -05:00
Brendan Abolivier
54dd5dc12b
Add ephemeral messages support (MSC2228) (#6409)
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:

* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
2019-12-03 19:19:45 +00:00
Hubert Chathi
2252680a98 make cross signing signature index non-unique 2019-12-02 17:01:59 -05:00
Andrew Morgan
a9c44d4008
Remove local threepids on account deactivation (#6426) 2019-11-28 10:40:42 +00:00
Hubert Chathi
0d27aba900
add etag and count to key backup endpoints (#5858) 2019-11-27 16:14:44 -05:00
Brendan Abolivier
9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Erik Johnston
f8f14ba466 Don't construct a set 2019-11-26 16:06:41 +00:00
Erik Johnston
8bb7b15894 Fix find_next_generated_user_id_localpart 2019-11-26 15:54:48 +00:00
Andrew Morgan
a8175d0f96
Prevent account_data content from being sent over TCP replication (#6333) 2019-11-26 13:58:39 +00:00
Richard van der Hoff
c01d543584
Make sure that we close cursors before returning from a query (#6408)
There are lots of words in the comment as to why this is a good idea.

Fixes #6403.
2019-11-25 21:03:17 +00:00
Richard van der Hoff
07929bd62f Synapse 1.6.0rc2 (2019-11-25)
=============================
 
 Bugfixes
 --------
 
 - Fix a bug which could cause the background database update hander for event labels to get stuck in a loop raising exceptions. ([\#6407](https://github.com/matrix-org/synapse/issues/6407))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl3b1yUACgkQOSor00I9
 eP/dWQf+ORS/B853qyH5KPZ66o6d7WudSewPmEkFD3747CBevxBsTPETijkqTBlo
 WPOmQy9i5OUWpsFYrsrCH+ATpr0JYaIuuoHsIFq/BPFFUx64qrgDwL+X4QEShwAm
 kjGNtCMP6VNGjM6MqFepRHSTbIEamCCS665CgVJtqgRYRaAJYI3SQDQ64+ALcbx3
 clFZowKV2EtfqhYR7HuBUuxuRjRPGcciNVyjMQFkKq91gKsO4rjPttvE4Bok29ia
 /uqFB6T0qty/81T708teZGgB/3/bYK4RtUA4lZCHBNeUejj26bESTI691RfBAEde
 to+D7xjA5zaMP3atYNlrvRrqK7Mm3w==
 =ucJN
 -----END PGP SIGNATURE-----

Merge tag 'v1.6.0rc2' into develop

Synapse 1.6.0rc2 (2019-11-25)
=============================

Bugfixes
--------

- Fix a bug which could cause the background database update hander for event labels to get stuck in a loop raising exceptions. ([\#6407](https://github.com/matrix-org/synapse/issues/6407))
2019-11-25 17:51:39 +00:00
Richard van der Hoff
b7367c339d
Fix exceptions from background database update for event labels. (#6407)
Add some exception handling here so that events whose json cannot be parsed are
ignored rather than getting us stuck in a loop.

Fixes #6404.
2019-11-25 13:26:59 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Brendan Abolivier
cdd3cb870d
Fix worker mode 2019-11-19 14:40:21 +00:00
Brendan Abolivier
7c24d0f443
Lint 2019-11-19 13:22:37 +00:00
Andrew Morgan
657d614f6a
Replace UPDATE with UPSERT on device_max_stream_id table (#6363) 2019-11-15 14:02:34 +00:00
Andrew Morgan
745a48625d
Fix guest -> real account upgrade with account validity enabled (#6359) 2019-11-14 12:02:05 +00:00
Andrew Morgan
c4bdf2d785 Remove content from being sent for account data rdata stream 2019-11-08 15:44:02 +00:00
Brendan Abolivier
963ffb60b9
Merge pull request #6340 from matrix-org/babolivier/pagination_query
Fix the SQL SELECT query in _paginate_room_events_txn
2019-11-08 11:12:24 +00:00
Brendan Abolivier
b16fa43386
Incorporate review 2019-11-08 10:34:09 +00:00
Erik Johnston
f713c01e2b
Merge pull request #6295 from matrix-org/erikj/split_purge_history
Split purge API into events vs state and add PurgeEventsStorage
2019-11-08 10:19:15 +00:00
Erik Johnston
e4ec82ce0f Move type annotation into docstring 2019-11-08 09:50:48 +00:00
Brendan Abolivier
46e5db9eb2
Merge pull request #6310 from matrix-org/babolivier/msc2326_bg_update
MSC2326: Add background update to take previous events into account
2019-11-07 22:54:56 +00:00
Brendan Abolivier
dad8d68c99
Update synapse/storage/data_stores/main/events_bg_updates.py
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-11-07 17:01:53 +00:00
Brendan Abolivier
6d360f099f
Update synapse/storage/data_stores/main/events_bg_updates.py
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-11-07 17:01:43 +00:00
Brendan Abolivier
c9b27d0044
Copy results 2019-11-07 16:47:45 +00:00
Brendan Abolivier
cd31201267 Revert "Back to using cursor_to_dict"
This reverts commit 1186612d6c.
2019-11-07 16:47:15 +00:00
Brendan Abolivier
1186612d6c
Back to using cursor_to_dict 2019-11-07 16:46:41 +00:00
Brendan Abolivier
ec2cb9f298
Initialise value before looping 2019-11-07 16:18:40 +00:00
Brendan Abolivier
bb78276bdc
Incorporate review 2019-11-07 15:25:27 +00:00
Brendan Abolivier
b9cba07962
Lint 2019-11-07 14:57:15 +00:00
Brendan Abolivier
70804392ae
Only join on event_labels if we're filtering on labels 2019-11-07 14:55:10 +00:00
Brendan Abolivier
15a1a02e70
Handle lack of filter 2019-11-07 12:04:37 +00:00
Brendan Abolivier
3f9b61ff95
Fix the SQL SELECT query in _paginate_room_events_txn
Doing a SELECT DISTINCT when paginating is quite expensive, because it requires the engine to do sorting on the entire events table. However, we only need to run it if we're filtering on 2+ labels, so this PR is changing the request so that DISTINCT is only used then.
2019-11-07 11:51:11 +00:00
Andrew Morgan
e914cf12f6
Merge pull request #6235 from matrix-org/anoa/room_upgrade_groups 2019-11-07 11:12:22 +00:00
Erik Johnston
5c3363233c Fix deleting state groups during room purge.
And fix the tests to actually test that things got deleted.
2019-11-06 17:02:08 +00:00
Erik Johnston
71f3bd734f Use correct type annotation 2019-11-06 17:00:18 +00:00
Andrew Morgan
b33c4f7a82 Numeric ID checker now checks @0, don't ratelimit on checking 2019-11-06 11:55:00 +00:00
Brendan Abolivier
24a214bd1b
Fix field name 2019-11-06 11:04:19 +00:00
Brendan Abolivier
70d93cafdb
Update insert 2019-11-06 10:59:03 +00:00
Richard van der Hoff
807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state (#6320)
Fixes a bug where rejected events were persisted with the wrong state group.

Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.

Fixes #6289.
2019-11-06 10:01:39 +00:00
Brendan Abolivier
8822b33111
Update copyrights 2019-11-05 14:46:35 +00:00
Andrew Morgan
c2203bea57 Re-add docstring, with caveats detailed 2019-11-04 18:17:11 +00:00
Andrew Morgan
0287d033ee Transfer upgraded rooms on groups 2019-11-04 18:08:50 +00:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Erik Johnston
7134ca7daa Change to not require a state_groups.room_id index.
This does mean that we won't clean up orphaned state groups (i.e. state
groups that were persisted but the associated event wasn't).
2019-11-04 13:36:57 +00:00
Erik Johnston
6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Brendan Abolivier
3b29a73f9f
Print out the actual number of affected rows 2019-11-04 09:56:11 +00:00
Brendan Abolivier
824bba2f78
Correctly order results 2019-11-04 09:56:11 +00:00
Brendan Abolivier
49008e674f
TODO 2019-11-04 09:56:11 +00:00
Brendan Abolivier
1586f2c7e7
Fix exit condition 2019-11-04 09:56:11 +00:00
Brendan Abolivier
1c1268245d
Lint 2019-11-04 09:56:11 +00:00
Brendan Abolivier
911b03ca31
Don't try to process events we already have a label for 2019-11-04 09:56:10 +00:00
Brendan Abolivier
07cb38e965
Use a sensible default value for labels 2019-11-04 09:56:10 +00:00
Brendan Abolivier
a46574281d
Use the right format for rows 2019-11-04 09:56:10 +00:00
Brendan Abolivier
c9a1b80a74
MSC2326: Add background update to take previous events into account 2019-11-04 09:56:04 +00:00
Brendan Abolivier
f496d25877
Merge pull request #6301 from matrix-org/babolivier/msc2326
Implement MSC2326 (label based filtering)
2019-11-01 17:04:45 +00:00
Brendan Abolivier
988d8d6507
Incorporate review 2019-11-01 16:22:44 +00:00
Brendan Abolivier
5598445655
Update synapse/storage/data_stores/main/schema/delta/56/event_labels.sql
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-11-01 16:18:34 +00:00
Hubert Chathi
fa7e52caf1
Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schema
fix hidden field in devices table for older sqlite
2019-11-01 10:52:46 -04:00
Brendan Abolivier
a2c63c619a
Add more data to the event_labels table and fix the indexes 2019-11-01 11:47:28 +00:00
Erik Johnston
669b6cbda3 Fix up comment 2019-11-01 11:32:20 +00:00
Brendan Abolivier
57cdb046e4
Lint 2019-11-01 10:39:14 +00:00
Brendan Abolivier
c6dbca2422
Incorporate review 2019-11-01 10:30:51 +00:00
Hubert Chathi
c61db13183 fix hidden field in devices table for older sqlite 2019-10-31 22:52:55 -04:00
Hubert Chathi
c3fc176c60
Update synapse/storage/data_stores/main/devices.py
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 22:49:48 -04:00
Hubert Chathi
6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Hubert Chathi
9c94b48bf1 Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify 2019-10-31 12:32:07 -04:00
Erik Johnston
fb1a6914cf Update log line to lie a little less 2019-10-31 15:45:48 +00:00
Amber Brown
020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston
61be1a2926 Add state_groups.room_id index 2019-10-31 15:39:26 +00:00
Erik Johnston
f91f2a1f92 Docstrings 2019-10-31 15:26:00 +00:00
Erik Johnston
8f5bbdb987 Fix purge room API 2019-10-31 15:22:08 +00:00
Erik Johnston
cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Erik Johnston
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Erik Johnston
64f2b8c3d8
Apply suggestions from code review
Fix docstring

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 15:44:31 +01:00
Erik Johnston
c6bcd38841 Fix /purge_room API.
It fails trying to clean the `topic` table which was recently removed.
2019-10-31 11:17:23 +00:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Hubert Chathi
998f7fe7d4 make user signatures a separate stream 2019-10-30 17:22:52 -04:00
Hubert Chathi
670972c0e1 Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify 2019-10-30 16:46:31 -04:00
Hubert Chathi
bb6cec27a5 rename get_devices_by_remote to get_device_updates_by_remote 2019-10-30 14:57:34 -04:00
Richard van der Hoff
0467f33584
fix delete_existing for _persist_events (#6300)
this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
2019-10-30 18:05:00 +00:00
Brendan Abolivier
dcc069a2e2
Lint 2019-10-30 18:01:56 +00:00
Brendan Abolivier
fe51d6cacf
Add more integration testing 2019-10-30 17:28:41 +00:00
Brendan Abolivier
233b14ebe1
Add index on label 2019-10-30 15:58:05 +00:00
Brendan Abolivier
acd16ad86a
Implement filtering 2019-10-30 15:56:33 +00:00
Erik Johnston
7c8c97e635 Split purge API into events vs state 2019-10-30 15:23:37 +00:00
Erik Johnston
5db03535d5 Add StateGroupStorage interface 2019-10-30 14:46:49 +00:00
Brendan Abolivier
fa0dcbc8fa
Store labels for new events 2019-10-30 14:27:15 +00:00
Hubert Chathi
7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Hubert Chathi
d78b1e339d apply changes as a result of PR review 2019-10-30 10:01:53 -04:00
Erik Johnston
ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Erik Johnston
a8d16f6c00 Review comments 2019-10-30 13:36:12 +00:00
Brendan Abolivier
47f767269c
Add database table for keeping track of labels on events 2019-10-29 16:56:22 +00:00
Erik Johnston
2c35ffead2 Port receipt and read markers to async/wait 2019-10-29 15:08:22 +00:00
Erik Johnston
561133c3c5
Merge pull request #6263 from matrix-org/erikj/caches_return_deferreds
Quick fix to ensure cache descriptors always return deferreds
2019-10-29 12:53:21 +01:00
Erik Johnston
e419c44ba4 Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-29 11:41:27 +00:00
Erik Johnston
342d871d76 Make room directory search case insensitive 2019-10-29 10:24:02 +00:00
Brendan Abolivier
14504ad573
Add CI for synapse_port_db (#6140)
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Erik Johnston
d0d8a22c13 Quick fix to ensure cache descriptors always return deferreds 2019-10-28 13:33:04 +00:00
Richard van der Hoff
9aee28927b Convert EventContext to attrs (#6218)
* make EventContext use an attr
2019-10-28 14:29:55 +02:00
Hubert Chathi
da78f61778
Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys
delete keys when deleting backup versions
2019-10-25 11:28:11 -04:00
Erik Johnston
a71b8c87ec Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:32:24 +01:00
Erik Johnston
a411f2b177 Fix /keys/query API on workers.
The necessary getters were added only to the master store and not the
worker stores.
2019-10-25 11:08:03 +01:00
Erik Johnston
44ab048cfe
Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_logging
Reduce debug logging overhead
2019-10-25 10:05:44 +01:00
Hubert Chathi
c40d7244f8 Merge branch 'develop' into cross-signing_federation 2019-10-24 22:31:25 -04:00
Hubert Chathi
8ac766c44a make notification of signatures work with workers 2019-10-24 22:14:58 -04:00
Hubert Chathi
848cd388d9 delete keys when deleting backups 2019-10-24 21:21:51 -04:00
Michael Kaye
e4d98188da Address codestyle concerns 2019-10-24 18:43:13 +01:00
Michael Kaye
9eebc1e73b use %r to __repr__ objects
This avoids calculating __repr__ unless we are going to log.
2019-10-24 18:18:56 +01:00
Michael Kaye
39266a9c9f Make user/room stats log line less verbose. 2019-10-24 17:55:53 +01:00
Erik Johnston
3aa2a90556 Move schema delta files to the correct data store.
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
Erik Johnston
73cf63784b Add DataStores and Storage classes. 2019-10-23 16:15:03 +01:00
Hubert Chathi
dc2cd6f79d move get_e2e_cross_signing_key to EndToEndKeyWorkerStore so it works with workers 2019-10-23 09:13:47 -04:00
Erik Johnston
22a9847670 Move persist_events out from main data store.
This is in preparation for splitting out of state_groups_state from the
main store into it own one, as persisting events depends on calculating
state.
2019-10-23 13:29:44 +01:00
Hubert Chathi
404e8c8532 vendor-prefix the EDU name until MSC1756 is merged into the spec 2019-10-22 22:33:23 -04:00
Hubert Chathi
1fabf82d50 update to work with newer code, and fix formatting 2019-10-22 21:44:58 -04:00
Hubert Chathi
cfdb84422d make black happy 2019-10-22 19:06:06 -04:00
Hubert Chathi
a1aaf3eea6 don't crash if the user doesn't have cross-signing keys 2019-10-22 19:04:37 -04:00
Hubert Chathi
8d3542a64e implement federation parts of cross-signing 2019-10-22 19:04:35 -04:00
Erik Johnston
c17efdc01c Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 18:44:29 +01:00
Erik Johnston
23d62eded2 Clean up prepare_database.py a bit and add comments 2019-10-22 18:43:31 +01:00
Erik Johnston
6cc497f99b Delete background_update table creation in main data_store 2019-10-22 18:02:50 +01:00
Erik Johnston
1bbc5444a8 Move README into synapse/storage/__init__.py 2019-10-22 17:59:31 +01:00
Erik Johnston
acf47c7698 Add a basic README to synapse.storage 2019-10-22 11:55:46 +01:00
Erik Johnston
bb6264be0b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 10:41:18 +01:00
Erik Johnston
ffd24545bb Fix schema management to work with multiple data stores. 2019-10-21 16:08:40 +01:00
Erik Johnston
c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Hubert Chathi
36adfaedab Merge branch 'develop' into cross-signing_sig_upload 2019-10-18 18:34:42 +01:00
Hubert Chathi
cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Hubert Chathi
f0f6a2b360 use the right function for when we're already in runInteraction 2019-10-18 10:56:54 +01:00
Hubert Chathi
06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
Hubert Chathi
132b251e29 expand on comment 2019-10-11 14:24:52 -04:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Hubert Chathi
7a0dce9259 make sure we actually return something 2019-10-10 20:31:30 -04:00
Hubert Chathi
4908fb3b30 make storage layer in charge of interpreting the device key data 2019-10-10 19:15:30 -04:00
Erik Johnston
83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
Erik Johnston
b54b1e759a Fix SQLite take 2 2019-10-10 16:19:40 +01:00
Erik Johnston
afb6d9d53b Fix SQLite 2019-10-10 15:55:41 +01:00
Erik Johnston
3bc687508f Remove add_in_list_sql_clause 2019-10-10 15:35:46 +01:00
Erik Johnston
9d06fb9cb1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any 2019-10-10 14:38:21 +01:00
Erik Johnston
8bc529c04d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 14:30:51 +01:00
Erik Johnston
b161786c14 Replace IN usage with helper funcs 2019-10-10 13:15:49 +01:00
Erik Johnston
b4fbf71187 Add helper funcs to use postgres ANY
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
Erik Johnston
91f43dca39 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 13:10:57 +01:00
Erik Johnston
9970f955ce Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner 2019-10-10 11:51:50 +01:00
Richard van der Hoff
a139420a3c
Fix races in room stats (and other) updates. (#6187)
Hopefully this will fix the occasional failures we were seeing in the room directory.

The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.

We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
Richard van der Hoff
562b4e51dd
Rewrite the user_filter migration again (#6184)
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
Erik Johnston
5c1f886c75 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner 2019-10-09 16:52:21 +01:00
Erik Johnston
7f18b3d526 Do the update as a background index 2019-10-09 16:03:24 +01:00
Erik Johnston
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
Erik Johnston
1d3858371e Disable bytes usage with postgres
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
Erik Johnston
e7631d84e6 Fix existing hex encoded json values in DB 2019-10-08 16:18:43 +01:00
Erik Johnston
ced4784592 Fix inserting bytes as text 2019-10-08 16:18:43 +01:00
Brendan Abolivier
c69324ffb5
Fix RegistrationStore 2019-10-08 14:48:33 +01:00
Brendan Abolivier
b1c0a4ceb3
Cleanup client_ips 2019-10-08 14:38:14 +01:00
Brendan Abolivier
8f1b385acc
Don't end up with 4 classes in registration 2019-10-08 14:36:33 +01:00
Brendan Abolivier
66ebea1723 Lint 2019-10-07 17:44:41 +01:00
Brendan Abolivier
0496eafbf4 Move roommember's bg updates to a dedicated store 2019-10-07 17:35:01 +01:00
Brendan Abolivier
e106a0e4db Move user_directory's bg updates to a dedicated store 2019-10-07 17:34:45 +01:00
Brendan Abolivier
cfccd2d78a Move state's bg updates to a dedicated store 2019-10-07 17:34:39 +01:00
Brendan Abolivier
841054ad96 Move search's bg updates to a dedicated store 2019-10-07 17:34:35 +01:00
Brendan Abolivier
81e6ffb536 Move registration's bg updates to a dedicated store 2019-10-07 17:34:29 +01:00
Brendan Abolivier
54f87e0734 Move media_repository's bg updates to a dedicated store 2019-10-07 17:34:26 +01:00
Brendan Abolivier
cef9f6753e Move devices's bg updates to a dedicated store 2019-10-07 17:34:20 +01:00
Brendan Abolivier
2d3b4f42f0 Move deviceinbox's bg updates to a dedicated store 2019-10-07 17:34:16 +01:00
Brendan Abolivier
88957199e7 Move client_ips's bg updates to a dedicated store 2019-10-07 17:34:12 +01:00
Richard van der Hoff
276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test (#6167) 2019-10-07 14:41:39 +01:00
Brendan Abolivier
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
Brendan Abolivier
c8e6c308c6
Fix unique_user_filter_index schema update 2019-10-07 13:15:35 +01:00
Erik Johnston
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
Erik Johnston
5119a4cac7 Fix bug where we didn't pull out event ID 2019-10-07 12:21:17 +01:00
Alexander Maznev
13c4345c84 Update user_filters table to have a unique index, and non-null columns (#1172) 2019-10-04 10:34:16 +01:00
Erik Johnston
6511071837 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships 2019-10-03 17:51:06 +01:00
Richard van der Hoff
66537e10ce
add some metrics on the federation sender (#6160) 2019-10-03 17:47:20 +01:00
Erik Johnston
91f61fc6d7 Use the right error.... 2019-10-03 17:28:31 +01:00
Erik Johnston
84691da6c3 pep8 2019-10-03 17:27:18 +01:00
Erik Johnston
a9610cdf02 Fixup names and comments 2019-10-03 17:26:56 +01:00
Erik Johnston
d89ebf7c25 cachedList descriptor doesn't like typing 2019-10-03 17:23:11 +01:00
Erik Johnston
c8145af8a9 Cache room membership lookups in _get_joined_users_from_context 2019-10-03 17:11:04 +01:00
Erik Johnston
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
Erik Johnston
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
Erik Johnston
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
Erik Johnston
7a5f080f91 Fix appservice room list pagination 2019-10-02 15:47:22 +01:00
Erik Johnston
4c4f44930d Fix not showing non-federatable rooms to remote room list queries 2019-10-02 15:20:36 +01:00
Erik Johnston
03cf4385e0 Fix public room list pagination.
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
Erik Johnston
a5166e4d5f
Land improved room list based on room stats (#6019)
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Andrew Morgan
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade (#6144)
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.

For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.

Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
Erik Johnston
d69fd53f74 Bound find_next_generated_user_id DB query.
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
Erik Johnston
ecd254bc49 Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop 2019-10-02 11:08:07 +01:00
Erik Johnston
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
Erik Johnston
35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions
Fix fetching censored redactions from DB
2019-10-02 10:57:30 +01:00
Erik Johnston
f44f1d2e83 Fix errors storing large retry intervals.
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.

To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
Erik Johnston
ce7a3e7e27 Fix fetching censored redactions from DB
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
Erik Johnston
5e8387af9e Use received_ts to find uncensored redacted events
Joining against `events` and ordering by `stream_ordering` is
inefficient as it forced scanning the entirety of the redactions table.

This isn't the case if we use `redactions.received_ts` column as we can
then use an index.
2019-10-01 13:43:48 +01:00
Erik Johnston
898dde981b Add received_ts column to redactions.
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
Erik Johnston
a27fb7d5ca Don't repeatedly attempt to censor events we don't have.
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.

When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
Erik Johnston
1d349fb159 Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop 2019-10-01 10:17:24 +01:00
Erik Johnston
9267741a5f Fix devices_last_seen background update.
Fixes #6134.
2019-09-30 11:58:36 +01:00
Richard van der Hoff
16cb9a71b8
Drop unused tables (#6115)
These tables are unused since #5893 (as amended by #6047), so we can now drop
them.

Fixes #6048.
2019-09-30 09:38:41 +01:00
Erik Johnston
132279a46f Patch inlinecallbacks for log contexts 2019-09-27 15:11:14 +01:00
Neil Johnson
034db2ba21 Fix dummy event insertion consent bug (#6053)
Fixes #5905
2019-09-26 11:47:53 +01:00
Erik Johnston
4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston
39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Erik Johnston
d2bd0bc6b1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips 2019-09-25 17:16:28 +01:00
Erik Johnston
50572db837
Use if is not None
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
Hubert Chathi
26113fb7de make changes based on PR feedback 2019-09-24 14:12:20 -04:00
Brendan Abolivier
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
Richard van der Hoff
9b7c4f4078 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 17:03:50 +01:00
Richard van der Hoff
566ac40939
remove unused parameter to get_user_id_by_threepid (#6099)
Added in #5377, apparently in error
2019-09-24 17:01:09 +01:00
Erik Johnston
242017db8b Prune rows in user_ips older than configured period
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
Erik Johnston
2135c198d1 Add has_completed_background_update
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Brendan Abolivier
12fe2a29bc
Incorporate review 2019-09-24 14:43:38 +01:00
Brendan Abolivier
f02f14e09a
Fix logging 2019-09-24 14:39:07 +01:00
Richard van der Hoff
ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Brendan Abolivier
323d685bf7
Typo 2019-09-23 17:23:49 +01:00
Brendan Abolivier
2858d10671
Fix the return value in the users_set_deactivated_flag background job 2019-09-23 17:22:01 +01:00
Erik Johnston
51d28272e2 Query devices table for last seen info.
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.

The storage functions now no longer return the access_token, since it
was unused.
2019-09-23 16:59:45 +01:00
Erik Johnston
ed80231ade Add BG update to populate devices last seen info 2019-09-23 16:59:45 +01:00
Erik Johnston
2ade05dca3 Add last seen info to devices table.
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
Andrew Morgan
30af161af2 Implement MSC2290 (#6043)
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.

This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints.

Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
Andrew Morgan
df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Richard van der Hoff
b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff
bcd9132869
Undo the deletion of some tables (#6047)
This is a partial revert of #5893. The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.

So let's leave the tables in place for now, and remember to drop them in a
subsequent release.

(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
2019-09-19 15:06:27 +01:00
Matthew Hodgson
2292dc35fc Add experimental "dont_push" push action to suppress push for notifications
This is a potential solution to https://github.com/vector-im/riot-web/issues/3374
and https://github.com/vector-im/riot-web/issues/5953
as raised by Mozilla at https://github.com/vector-im/riot-web/issues/10868.

This lets you define a push rule action which increases the badge count (unread notification)
count on a given room, but doesn't actually send a push for that notification via email or HTTP.
We might want to define this as the default behaviour for group chats in future
to solve https://github.com/vector-im/riot-web/issues/3268 at last.

This is implemented as a string action rather than a tweak because:
 * Other pushers don't care about the tweak, given they won't ever get pushed
 * The DB can store the tweak more efficiently using the existing `notify` table.
 * It avoids breaking the default_notif/highlight_action optimisations.

Clients which generate their own notifs (e.g. desktop notifs from Riot/Web
would need to be aware of the new push action) to uphold it.

An alternative way to do this would be to maintain a `msg_count` alongside
`highlight_count` and `notification_count` in `unread_notifications` in sync responses.
However, doing this by counting the rows in `events` since the `stream_position`
of the user's last read receipt turns out to be painfully slow (~200ms), perhaps
due to the size of the events table.  So instead, we use the highly optimised
existing event_push_actions (and event_push_actions_staging) table to maintain
the counts - using the code paths which already exist for tracking unread
notification counts efficiently.  These queries are typically ~3ms or so.

The biggest issues I see here are:
 * We're slightly repurposing the `notif` field on `event_push_actions` to
   track whether a given action actually sent a `push` or not.  This doesn't
   seem unreasonable, but it's slightly naughty given that previously the
   field explicitly tracked whether `notify` was true for the action (and
   as a result, it was uselessly always set to 1 in the DB).
 * We're going to put more load on the `event_push_actions` table for all the
   random group chats which people had previously muted. In practice i don't
   think there are many of these though.
 * There isn't an MSC for this yet (although this comment could become one).
2019-09-19 00:54:05 +01:00
Richard van der Hoff
1e19ce00bf
Add 'failure_ts' column to 'destinations' table (#6016)
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
Richard van der Hoff
a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Richard van der Hoff
7902bf1e1d
Clean up some code in the retry logic (#6017)
* remove some unused code
* make things which were constants into constants for efficiency and clarity
2019-09-11 15:14:56 +01:00
Jason Robinson
63f9317b8e
Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users
Only count real users when checking for auto-creation of auto-join room
2019-09-09 17:37:52 +03:00
Erik Johnston
470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions
Censor redactions in DB after a month
2019-09-09 15:29:39 +01:00
Jason Robinson
aaed6b39e1 Fix code style, again
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 17:10:02 +03:00
Erik Johnston
580f3df9b2 Fix comments 2019-09-09 15:08:24 +01:00
Jason Robinson
e89fea4f04 Simplify count_real_users SQL to only count user_type is null rows
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 16:43:32 +03:00
Jason Robinson
8c03cd0e5f Simplify is_real_user_txn check to trust user_type is null if real user
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 16:40:40 +03:00
Erik Johnston
e7184a4370 Use better names in SQL 2019-09-09 13:33:38 +01:00
Erik Johnston
916c697228 Fixup comment 2019-09-09 13:31:00 +01:00
Erik Johnston
fffe17b77d Don't start looping call unless enabled 2019-09-09 13:24:24 +01:00
Erik Johnston
80e14a8546 Handle setting retention period to 0 2019-09-09 13:23:41 +01:00
Jason Robinson
62fac9d969 Auto-fix a few code style issues
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:59:35 +03:00
Jason Robinson
be618e0551 Only count real users when checking for auto-creation of auto-join room
Previously if the first registered user was a "support" or "bot" user,
when the first real user registers, the auto-join rooms were not
created.

Fix to exclude non-real (ie users with a special user type) users
when counting how many users there are to determine whether we should
auto-create a room.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:48:08 +03:00
Hubert Chathi
8e86f5b65c Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-09-07 13:20:34 -04:00
Hubert Chathi
0d61d1d735 Merge branch 'develop' into cross-signing_sig_upload 2019-09-07 13:14:45 -04:00
Amber Brown
55d5b3af88
Servers-known-about statistic (#5981) 2019-09-07 01:45:51 +10:00
Erik Johnston
a2a695b7ec
Merge pull request #5998 from matrix-org/erikj/fixup_federate_flag
Correctly handle non-bool m.federate flag
2019-09-06 15:32:43 +01:00
Erik Johnston
142c9325c2 Correctly handle non-bool m.federate flag 2019-09-06 14:21:06 +01:00
Andrew Morgan
6ddda8152e Move delete_threepid_session into RegistrationWorkerStore 2019-09-06 13:23:10 +01:00
Andrew Morgan
e059c5e648 Move get_threepid_validation_session into RegistrationWorkerStore 2019-09-06 13:10:11 +01:00
Erik Johnston
591d82f06b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions 2019-09-05 17:27:46 +01:00
Erik Johnston
3ff0422d2d Make redaction retention period configurable 2019-09-05 17:16:45 +01:00
Jorik Schellekens
5ade977d08 Opentracing context cannot be none 2019-09-05 15:06:13 +01:00
Hubert Chathi
7d6c70fc7a make black happy 2019-09-04 22:04:12 -04:00
Hubert Chathi
ac4746ac4b allow uploading signatures of master key signed by devices 2019-09-04 22:03:41 -04:00
Hubert Chathi
4bb4544784 implement device signature uploading/fetching 2019-09-04 20:02:56 -04:00
Hubert Chathi
faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Erik Johnston
6e834e94fc
Fix and refactor room and user stats (#5971)
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00