Commit Graph

3058 Commits

Author SHA1 Message Date
Richard van der Hoff
15bf32e062 Use monotonic clock where possible for metrics
Fixes intermittent errors observed on Apple hardware which were caused by
time.clock() appearing to go backwards when called from different threads.

Also fixes a bug where database activity times were logged as 1/1000 of their
correct ratio due to confusion between milliseconds and seconds.
2019-06-19 21:09:43 +01:00
Erik Johnston
49e01e5710 Fix defaults on checking threepids 2019-06-10 23:09:31 +01:00
Erik Johnston
c413540fb9 Fix bug sending federation transactions with lots of EDUs
If we try and send a transaction with lots of EDUs and we run out of
space, we call get_new_device_msgs_for_remote with a limit of 0, which
then failed.
2019-06-10 16:22:45 +01:00
Erik Johnston
43badd2cd4 Fix key verification when key stored with null valid_until_ms
Some keys are stored in the synapse database with a null valid_until_ms
which caused an exception to be thrown when using that key. We fix this
by treating nulls as zeroes, i.e. they keys will match verification
requests with a minimum_valid_until_ms of zero (i.e. don't validate ts)
but will not match requests with a non-zero minimum_valid_until_ms.

Fixes #5391.
2019-06-10 15:37:10 +01:00
Andrew Morgan
2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156)
fixes #5153
2019-06-06 23:54:00 +01:00
Erik Johnston
1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
Make a full SQL schema
2019-06-06 18:35:38 +01:00
Amber H. Brown
ed872db8df fix maybe 2019-06-07 02:53:47 +10:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Amber H. Brown
837e32ef55 just user dir? 2019-06-07 01:49:25 +10:00
Amber H. Brown
3b6645d3bf remove background updates that arent needed 2019-06-07 01:20:58 +10:00
Erik Johnston
89d3d7b2c0
Merge pull request #5221 from matrix-org/erikj/fix_worker_sytest
Fix get_max_topological_token to never return None
2019-06-06 13:54:46 +01:00
Erik Johnston
7f08a3523a Better words 2019-06-06 11:09:38 +01:00
Amber Brown
6362e3af14 add more comments 2019-06-06 04:20:35 +10:00
Erik Johnston
75538813fc Fix background updates to handle redactions/rejections (#5352)
* Fix background updates to handle redactions/rejections

In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
Erik Johnston
0a56966f7d Fix 2019-06-03 17:42:52 +01:00
Erik Johnston
fa4b54aca5 Ignore room state with null bytes in for room stats 2019-06-03 17:06:54 +01:00
Amber Brown
2198b7ce2a add stuff in bg updates 2019-06-04 01:06:00 +10:00
Amber Brown
4e75c5e02a WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING 2019-06-03 22:42:12 +10:00
Amber Brown
be452fc9ac more fix 2019-06-03 22:24:23 +10:00
Amber Brown
7f81b967ca fix schemas 2019-06-03 22:23:40 +10:00
Amber Brown
dc72b90cd6 full schema 2019-06-03 22:03:28 +10:00
Amber Brown
37057d5d60 prepare 2019-06-03 22:02:47 +10:00
Erik Johnston
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
Brendan Abolivier
4d794dae21 Move delta from +10% to -10% 2019-05-31 11:09:39 +01:00
Erik Johnston
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
Speed up room stats background update
2019-05-31 10:48:51 +01:00
Erik Johnston
5037326d66 Add indices. Remove room_ids accidentally added
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.

We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
Brendan Abolivier
847b9dcd1c Make max_delta equal to period * 10% 2019-05-31 09:54:46 +01:00
Erik Johnston
5ac75fc9a2 Join against events to use its room_id index 2019-05-30 15:26:55 +01:00
Erik Johnston
e2c46ed851 Move deletion from table inside txn 2019-05-30 15:26:38 +01:00
Erik Johnston
04710cc2d7 Fetch membership counts all at once 2019-05-30 15:25:41 +01:00
Erik Johnston
54d50fbfdf Get events all at once 2019-05-30 15:15:13 +01:00
Erik Johnston
6cdfb0207e Add index to temp table 2019-05-30 15:02:31 +01:00
Erik Johnston
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities
Add DB bg update to cleanup extremities.
2019-05-30 14:28:26 +01:00
Erik Johnston
cb967e2346
Update synapse/storage/events_bg_updates.py
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-30 14:06:42 +01:00
Erik Johnston
468bd090ff Rename constant 2019-05-30 11:24:42 +01:00
Erik Johnston
5c1ece0ffc Move event background updates to a separate file 2019-05-30 11:22:59 +01:00
Erik Johnston
640fcbb07f Fixup comments and logging 2019-05-30 10:55:55 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Erik Johnston
7e8e683754 Log actual number of entries deleted 2019-05-29 15:11:28 +01:00
Erik Johnston
d79c9994f4 Add DB bg update to cleanup extremities.
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
Erik Johnston
58c8ed5b0d
Correctly filter out extremities with soft failed prevs (#5274)
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
2019-05-29 11:56:24 +01:00
Brendan Abolivier
52839886d6
Allow configuring a range for the account validity startup job
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
2019-05-28 16:52:45 +01:00
Brendan Abolivier
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
Brendan Abolivier
ba17de7fbc
Fix schema update for account validity 2019-05-28 10:11:38 +01:00
Travis Ralston
bc4b2ecf70 Fix logging for room stats background update 2019-05-25 12:02:48 -06:00
Richard van der Hoff
b75537beaf Store key validity time in the storage layer
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234)
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Richard van der Hoff
85d1e03b9d
Simplifications and comments in do_auth (#5227)
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
Amber Brown
4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Erik Johnston
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender
Only count aggregations from distinct senders
2019-05-21 17:10:48 +01:00