Hubert Chathi
72d296a7f3
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-08-28 17:33:44 -07:00
Hubert Chathi
3b0b22cb05
use stream ID generator instead of timestamp
2019-08-28 17:17:21 -07:00
Will Hunt
c8fa620d7a
Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent
...
Exempt support users from consent
2019-08-28 16:31:40 +01:00
Richard van der Hoff
49ef8ec399
Fix a cache-invalidation bug for worker-based deployments ( #5920 )
...
Some of the caches on worker processes were not being correctly invalidated
when a room's state was changed in a way that did not affect the membership
list of the room.
We need to make sure we send out cache invalidations even when no memberships
are changing.
2019-08-28 10:18:16 +01:00
reivilibre
322ccac33f
Allow schema deltas to be engine-specific ( #5911 )
...
* Allow schema deltas to be engine-specific
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Code style (Black)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 11:53:21 +01:00
reivilibre
1a7e6eb633
Add Admin API capability to set adminship of a user ( #5878 )
...
Admin API: Set adminship of a user
2019-08-27 10:14:00 +01:00
Olivier Wilkinson (reivilibre)
d1e0b91083
Code style (Black)
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:39:11 +01:00
Olivier Wilkinson (reivilibre)
aefa76f5cd
Allow schema deltas to be engine-specific
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:14:00 +01:00
Half-Shot
ae38e0569f
Ignore consent for support users
2019-08-23 09:15:10 +01:00
Jorik Schellekens
8767b63a82
Propagate opentracing contexts through EDUs ( #5852 )
...
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Jorik Schellekens
9a6f2be572
Opentrace e2e keys ( #5855 )
...
Add opentracing tags and logs for e2e keys
2019-08-22 11:28:12 +01:00
Richard van der Hoff
c9f11d09fc
Add missing index on users_in_public_rooms. ( #5894 )
2019-08-22 10:43:13 +01:00
Richard van der Hoff
119aa31b10
Servlet to purge old rooms ( #5845 )
2019-08-22 10:42:59 +01:00
Hubert Chathi
7c3abc6572
apply PR review suggestions
2019-08-21 13:19:35 -07:00
Richard van der Hoff
4dab867288
Drop some unused tables. ( #5893 )
...
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
Hubert Chathi
8a5f6ed130
Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index
...
add the version field to the index for e2e_room_keys
2019-08-16 17:45:50 -07:00
Hubert Chathi
c058aeb88d
update set_e2e_room_key to agree with fixed index
2019-08-14 18:02:58 -07:00
Hubert Chathi
b7f7cc7ace
add the version field to the index for e2e_room_keys
2019-08-14 17:14:40 -07:00
Erik Johnston
af187805b3
Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop
...
Handle pusher being deleted during processing.
2019-08-13 14:08:29 +01:00
Erik Johnston
fb3469f53a
Clarify docstring
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-13 10:17:23 +01:00
Erik Johnston
2546f32b90
Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses
...
Don't unnecessarily block notifying of new events.
2019-08-13 09:36:25 +01:00
Hubert Chathi
f63ba7a795
Cross-signing [1/4] -- hidden devices ( #5759 )
...
* allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
Richard van der Hoff
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions
...
Fix handling of redactions of redactions
2019-08-12 12:25:19 +01:00
Erik Johnston
bf4db42920
Don't unnecessarily block notifying of new events.
...
When persisting events we calculate new stream orderings up front.
Before we notify about an event all events with lower stream orderings
must have finished being persisted.
This PR moves the assignment of stream ordering till *after* calculated
the new current state and split the batch of events into separate chunks
for persistence. This means that if it takes a long time to calculate
new current state then it will not block events in other rooms being
notified about.
This should help reduce some global pauses in the events stream which
can last for tens of seconds (if not longer), caused by some
particularly expensive state resolutions.
2019-08-06 13:32:02 +01:00
Hubert Chathi
fac1cdc562
make changes from PR review
2019-08-01 21:51:19 -04:00
Hubert Chathi
336c546d6a
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-08-01 16:31:40 -04:00
Andrew Morgan
5d018d23f0
Have ClientReaderSlavedStore inherit RegistrationStore ( #5806 )
...
Fixes #5803
2019-08-01 13:54:56 +01:00
Erik Johnston
d02e41dcb2
Handle pusher being deleted during processing.
...
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
2019-08-01 13:44:12 +01:00
Hubert Chathi
d78a4fe156
don't need to return the hidden column any more
2019-08-01 02:16:09 -04:00
Hubert Chathi
73b26f827c
really fix queries to work with Postgres
...
(by going back to not using SQL directly)
2019-07-31 18:37:05 -04:00
Hubert Chathi
430ea08186
PostgreSQL, Y U no like?
2019-07-31 15:38:11 -04:00
Hubert Chathi
185188be03
remove extra SQL query param
2019-07-31 15:18:15 -04:00
Hubert Chathi
c1f0a5636c
Merge branch 'develop' into cross-signing_hidden
2019-07-30 23:33:06 -04:00
Hubert Chathi
bc95890822
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-07-30 23:21:26 -04:00
Hubert Chathi
781ade836b
apply changes from PR review
2019-07-30 23:09:50 -04:00
Erik Johnston
123c04daa7
Don't recreate column
2019-07-30 16:42:48 +01:00
Erik Johnston
15056ca208
Fix current_state_events membership background update.
...
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
2019-07-30 14:51:41 +01:00
Richard van der Hoff
4e97eb89e5
Handle loops in redaction events
2019-07-30 00:00:34 +01:00
Richard van der Hoff
448bcfd0f9
recursively fetch redactions
2019-07-30 00:00:34 +01:00
Richard van der Hoff
e6a6c4fbab
split _get_events_from_db out of _enqueue_events
2019-07-29 23:15:15 +01:00
Richard van der Hoff
c9964ba600
Return dicts from _fetch_event_list
2019-07-29 23:15:15 +01:00
Erik Johnston
7c8c3b8437
Merge pull request #5774 from matrix-org/erikj/fix_rejected_membership
...
Fix room summary when rejected events are in state
2019-07-29 17:15:15 +01:00
Erik Johnston
3e013b7c8e
Merge pull request #5752 from matrix-org/erikj/forgotten_user
...
Remove some more joins on room_memberships
2019-07-29 17:15:01 +01:00
Erik Johnston
df3a5db629
Expand comment
2019-07-29 16:40:25 +01:00
Erik Johnston
105e7f6ed3
Remove lost comment
2019-07-29 16:09:48 +01:00
Erik Johnston
45df38e61b
Fix current_state bg update to work on old SQLite
2019-07-29 13:04:10 +01:00
Richard van der Hoff
d74595e2ca
Merge branch 'master' into develop
2019-07-26 12:39:33 +01:00
Erik Johnston
14c24c9037
Fix room summary when rejected events are in state
...
Annoyingly, `current_state_events` table can include rejected events,
in which case the membership column will be null. To work around this
lets just always filter out null membership for now.
2019-07-26 10:11:36 +01:00
Richard van der Hoff
0f2ecb961e
Fix DoS when there is a cycle in redaction events
...
Make sure that synapse doesn't explode when a redaction redacts itself, or
there is a larger cycle.
2019-07-26 06:36:48 +00:00
Richard van der Hoff
26d742fed6
Merge pull request #5767 from matrix-org/rav/redactions/cross_room_id
...
log when a redaction attempts to redact an event in a different room
2019-07-25 18:49:56 +01:00
Richard van der Hoff
b1605cdd23
log when a redaction attempts to redact an event in a different room
2019-07-25 18:26:20 +01:00
Hubert Chathi
c659b9f94f
allow uploading keys for cross-signing
2019-07-25 11:08:24 -04:00
Hubert Chathi
d1c7c2a98a
allow devices to be marked as "hidden"
...
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-07-24 23:21:52 -04:00
Richard van der Hoff
f30a71a67b
Stop trying to fetch events with event_id=None. ( #5753 )
...
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.
I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
2019-07-24 13:16:18 +01:00
Erik Johnston
0c4a99607e
Remove join when calculating room summaries.
2019-07-24 11:49:15 +01:00
Erik Johnston
62921fb53e
Remove join on room_memberships when fetching rooms for user.
2019-07-24 11:45:58 +01:00
Erik Johnston
32768e96d4
Add function to get all forgotten rooms for user
...
This will allow us to efficiently filter out rooms that have been
forgotten in other queries without having to join against the
`room_memberships` table.
2019-07-24 11:44:23 +01:00
Amber Brown
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
Erik Johnston
b2a629ef49
Speed up current state background update.
...
Turns out that storing huge JSON arrays in the progress JSON isn't
something that postgres particularly likes.
2019-07-23 10:01:30 +01:00
Jason Robinson
66f5ff72fd
Add user_type
to returned fields in admin API user list endpoints
...
Mostly user type will be empty (normal user) but there is also the
"support" user type.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-07-22 15:29:18 +03:00
Erik Johnston
7704873cb8
Merge pull request #5720 from matrix-org/erikj/transactions_upsert
...
Use upsert when updating destination retry interval
2019-07-19 16:51:16 +01:00
Erik Johnston
d7bd9651bc
Merge pull request #5713 from matrix-org/erikj/use_cache_for_filtered_state
...
Delegate to cached version when using get_filtered_current_state_ids
2019-07-19 16:30:49 +01:00
Erik Johnston
5c07c97c09
Merge pull request #5706 from matrix-org/erikj/add_memberships_to_current_state
...
Add membership column to current_state_events table
2019-07-19 16:30:33 +01:00
Erik Johnston
2410335507
Use upsert when updating destination retry interval
2019-07-19 13:40:24 +01:00
Erik Johnston
bd2e1a2aa8
LoggingTransaction accepts None for callback lists.
...
Its a bit disingenuousto give LoggingTransaction lists to append
callbacks to if we're not going to run the callbacks.
2019-07-19 13:36:04 +01:00
Erik Johnston
ebc5ed1296
Update comment for new column
2019-07-19 13:29:02 +01:00
Erik Johnston
10523241d8
Delegate to cached version when using get_filtered_current_state_ids
...
In the case where it gets called with `StateFilter.all()`
2019-07-18 15:17:39 +01:00
Erik Johnston
8e1ada9e6f
Use the current_state_events.membership column
2019-07-18 14:16:01 +01:00
Erik Johnston
059d8c1a4e
Track if current_state_events.membership is up to date
2019-07-18 14:16:01 +01:00
Erik Johnston
c618a5d348
Add background update for current_state_events.membership column
2019-07-18 14:16:01 +01:00
Erik Johnston
6de09e07a6
Add membership column to current_state_events table.
...
It turns out that doing a join is surprisingly expensive for the DB to
do when room_membership table is larger than the disk cache.
2019-07-18 14:15:57 +01:00
Richard van der Hoff
9c70a02a9c
Ignore redactions of m.room.create events ( #5701 )
2019-07-17 19:08:02 +01:00
Richard van der Hoff
2091c91fde
More refactoring in get_events_as_list
( #5707 )
...
We can now use `_get_events_from_cache_or_db` rather than going right back to
the database, which means that (a) we can benefit from caching, and (b) it
opens the way forward to more extensive checks on the original event.
We now always require the original event to exist before we will serve up a
redaction.
2019-07-17 17:34:13 +01:00
Richard van der Hoff
375162b3c3
Fix redaction authentication ( #5700 )
...
Ensures that redactions are correctly authenticated for recent room versions.
There are a few things going on here:
* `_fetch_event_rows` is updated to return a dict rather than a list of rows.
* Rather than returning multiple copies of an event which was redacted
multiple times, it returns the redactions as a list within the dict.
* It also returns the actual rejection reason, rather than merely the fact
that it was rejected, so that we don't have to query the table again in
`_get_event_from_row`.
* The redaction handling is factored out of `_get_event_from_row`, and now
checks if any of the redactions are valid.
2019-07-17 16:52:02 +01:00
Richard van der Hoff
65c5592b8e
Refactor get_events_as_list
( #5699 )
...
A couple of changes here:
* get rid of a redundant `allow_rejected` condition - we should already have filtered out any rejected
events before we get to that point in the code, and the redundancy is confusing. Instead, let's stick in
an assertion just to make double-sure we aren't leaking rejected events by mistake.
* factor out a `_get_events_from_cache_or_db` method, which is going to be important for a
forthcoming fix to redactions.
2019-07-17 16:49:19 +01:00
Erik Johnston
d86321300a
Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data
...
Add basic function to get all data for a user out of synapse
2019-07-15 10:04:02 +01:00
Richard van der Hoff
5f158ec039
Implement access token expiry ( #5660 )
...
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Erik Johnston
db0a50bc40
Fixup docstrings
2019-07-12 16:59:59 +01:00
Andrew Morgan
24aa0e0a5b
fix typo: backgroud -> background
2019-07-12 15:29:40 +01:00
Andrew Morgan
78a1cd36b5
small typo fix ( #5655 )
2019-07-11 13:33:23 +01:00
Richard van der Hoff
953dbb7980
Remove access-token support from RegistrationStore.register ( #5642 )
...
The 'token' param is no longer used anywhere except the tests, so let's kill
that off too.
2019-07-10 16:26:49 +01:00
Andrew Morgan
d88421ab03
Include the original event in /relations ( #5626 )
...
When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.
2019-07-09 13:43:08 +01:00
Richard van der Hoff
80cc82a445
Remove support for invite_3pid_guest. ( #5625 )
...
This has never been documented, and I'm not sure it's ever been used outside
sytest.
It's quite a lot of poorly-maintained code, so I'd like to get rid of it.
For now I haven't removed the database table; I suggest we leave that for a
future clearout.
2019-07-05 16:47:58 +01:00
Erik Johnston
7f0d8e4288
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data
2019-07-05 14:08:21 +01:00
Erik Johnston
9ccea16d45
Assume key existence. Update docstrings
2019-07-05 14:07:56 +01:00
Richard van der Hoff
a6a776f3d8
remove dead transaction persist code ( #5622 )
...
this hasn't done anything for years
2019-07-05 12:59:42 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
Richard van der Hoff
91753cae59
Fix a number of "Starting txn from sentinel context" warnings ( #5605 )
...
Fixes #5602 , #5603
2019-07-03 09:31:27 +01:00
Erik Johnston
8ee69f299c
Add basic function to get all data for a user out of synapse
2019-07-02 12:09:04 +01:00
Erik Johnston
729f5a4fb6
Review comments
2019-06-27 16:06:23 +01:00
Erik Johnston
f335e77d53
Use batch_iter and correct docstring
2019-06-26 19:39:46 +01:00
Erik Johnston
806a06daf2
Rename get_users_whose_devices_changed
2019-06-26 19:39:19 +01:00
Erik Johnston
a2f6d31a63
Refactor get_user_ids_changed to pull less from DB
...
When a client asks for users whose devices have changed since a token we
used to pull *all* users from the database since the token, which could
easily be thousands of rows for old tokens.
This PR changes this to only check for changes for users the client is
actually interested in.
Fixes #5553
2019-06-26 12:03:44 +01:00
Richard van der Hoff
e59a8cd2e5
Merge pull request #5499 from matrix-org/rav/cleanup_metrics
...
Cleanups and sanity-checking in cpu and db metrics
2019-06-24 17:12:54 +01:00
Richard van der Hoff
1793de6c6d
black
2019-06-24 11:16:13 +01:00
Richard van der Hoff
5097aee740
Merge branch 'develop' into rav/cleanup_metrics
2019-06-24 10:00:13 +01:00
Richard van der Hoff
c753c098dd
Merge pull request #5498 from matrix-org/rav/fix_clock_reversal
...
Use monotonic clock where possible for metrics
2019-06-24 09:55:12 +01:00
Richard van der Hoff
0e8b35f7b0
Fix "Unexpected entry in 'full_schemas'" log warning ( #5509 )
...
There is a README.txt which always sets off this warning, which is a bit
alarming when you first start synapse. I don't think we need to warn about
this.
2019-06-21 15:11:57 +01:00
Erik Johnston
f8bd30af2a
Black
2019-06-20 13:10:06 +01:00
Erik Johnston
45f28a9d2f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/histogram_extremities
2019-06-20 11:59:14 +01:00
Amber Brown
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
Richard van der Hoff
f682af052d
Simplify PerformanceCounters.update interface
...
we already have the duration for the update, so may as well use it rather than
passing extra params around and recalculating it.
2019-06-19 21:18:38 +01:00
Richard van der Hoff
68128d5626
Remove unused _get_event_counters
...
This has been redundant since cdb3757942
.
2019-06-19 21:14:25 +01:00
Richard van der Hoff
15bf32e062
Use monotonic clock where possible for metrics
...
Fixes intermittent errors observed on Apple hardware which were caused by
time.clock() appearing to go backwards when called from different threads.
Also fixes a bug where database activity times were logged as 1/1000 of their
correct ratio due to confusion between milliseconds and seconds.
2019-06-19 21:09:43 +01:00
Erik Johnston
e0be8d7016
Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events
...
Add experimental option to reduce extremities.
2019-06-19 13:19:18 +01:00
Erik Johnston
65787b0f7c
Add descriptions and remove redundant set(..)
2019-06-19 11:49:39 +01:00
Brendan Abolivier
092e36457a
Fix typo in deactivation background job
2019-06-19 11:09:41 +01:00
Erik Johnston
b42f90470f
Add experimental option to reduce extremities.
...
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.
THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Erik Johnston
499d4a32cd
Add metrics for len of new extremities persisted.
...
Of new events being persisted add metrics for total size of forward
extremities and number of unchanged, "stale" extremities.
2019-06-17 17:02:48 +01:00
Amber Brown
eba7caf09f
Remove Postgres 9.4 support ( #5448 )
2019-06-18 00:59:00 +10:00
Erik Johnston
9ca4ae7131
Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive
...
Prometheus histograms are cumalative
2019-06-14 18:21:42 +01:00
Brendan Abolivier
e0b77b004d
Fix background job for deactivated flag
2019-06-14 16:00:45 +01:00
Brendan Abolivier
6d56a694f4
Don't send renewal emails to deactivated users
2019-06-14 15:05:56 +01:00
Erik Johnston
3ed595e327
Prometheus histograms are cumalative
2019-06-14 14:07:32 +01:00
Brendan Abolivier
d0530382ee
Track deactivated accounts in the database ( #5378 )
2019-06-14 13:18:24 +01:00
Amber Brown
6312d6cc7c
Expose statistics on extrems to prometheus ( #5384 )
2019-06-13 22:40:52 +10:00
Erik Johnston
5bc9484537
Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop
2019-06-11 10:37:43 +01:00
Neil Johnson
94dac0f3e5
add monthly active users to phonehome stats ( #5252 )
...
* add monthly active users to phonehome stats
2019-06-10 23:33:59 +01:00
Erik Johnston
49e01e5710
Fix defaults on checking threepids
2019-06-10 23:09:31 +01:00
Erik Johnston
c413540fb9
Fix bug sending federation transactions with lots of EDUs
...
If we try and send a transaction with lots of EDUs and we run out of
space, we call get_new_device_msgs_for_remote with a limit of 0, which
then failed.
2019-06-10 16:22:45 +01:00
Erik Johnston
43badd2cd4
Fix key verification when key stored with null valid_until_ms
...
Some keys are stored in the synapse database with a null valid_until_ms
which caused an exception to be thrown when using that key. We fix this
by treating nulls as zeroes, i.e. they keys will match verification
requests with a minimum_valid_until_ms of zero (i.e. don't validate ts)
but will not match requests with a non-zero minimum_valid_until_ms.
Fixes #5391 .
2019-06-10 15:37:10 +01:00
Andrew Morgan
2d1d7b7e6f
Prevent multiple device list updates from breaking a batch send ( #5156 )
...
fixes #5153
2019-06-06 23:54:00 +01:00
Erik Johnston
1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
...
Make a full SQL schema
2019-06-06 18:35:38 +01:00
Amber H. Brown
ed872db8df
fix maybe
2019-06-07 02:53:47 +10:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server ( #5377 )
...
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.
This PR is a culmination of 3 smaller PRs which have each been separately reviewed:
* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Amber H. Brown
837e32ef55
just user dir?
2019-06-07 01:49:25 +10:00
Amber H. Brown
3b6645d3bf
remove background updates that arent needed
2019-06-07 01:20:58 +10:00
Erik Johnston
89d3d7b2c0
Merge pull request #5221 from matrix-org/erikj/fix_worker_sytest
...
Fix get_max_topological_token to never return None
2019-06-06 13:54:46 +01:00
Erik Johnston
7f08a3523a
Better words
2019-06-06 11:09:38 +01:00
Amber Brown
6362e3af14
add more comments
2019-06-06 04:20:35 +10:00
Erik Johnston
75538813fc
Fix background updates to handle redactions/rejections ( #5352 )
...
* Fix background updates to handle redactions/rejections
In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
Erik Johnston
0a56966f7d
Fix
2019-06-03 17:42:52 +01:00
Erik Johnston
fa4b54aca5
Ignore room state with null bytes in for room stats
2019-06-03 17:06:54 +01:00
Amber Brown
2198b7ce2a
add stuff in bg updates
2019-06-04 01:06:00 +10:00
Amber Brown
4e75c5e02a
WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING
2019-06-03 22:42:12 +10:00
Amber Brown
be452fc9ac
more fix
2019-06-03 22:24:23 +10:00
Amber Brown
7f81b967ca
fix schemas
2019-06-03 22:23:40 +10:00
Amber Brown
dc72b90cd6
full schema
2019-06-03 22:03:28 +10:00
Amber Brown
37057d5d60
prepare
2019-06-03 22:02:47 +10:00
Erik Johnston
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
...
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
Brendan Abolivier
4d794dae21
Move delta from +10% to -10%
2019-05-31 11:09:39 +01:00
Erik Johnston
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
...
Speed up room stats background update
2019-05-31 10:48:51 +01:00
Erik Johnston
5037326d66
Add indices. Remove room_ids accidentally added
...
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.
We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
Brendan Abolivier
847b9dcd1c
Make max_delta equal to period * 10%
2019-05-31 09:54:46 +01:00
Erik Johnston
5ac75fc9a2
Join against events to use its room_id index
2019-05-30 15:26:55 +01:00
Erik Johnston
e2c46ed851
Move deletion from table inside txn
2019-05-30 15:26:38 +01:00
Erik Johnston
04710cc2d7
Fetch membership counts all at once
2019-05-30 15:25:41 +01:00
Erik Johnston
54d50fbfdf
Get events all at once
2019-05-30 15:15:13 +01:00
Erik Johnston
6cdfb0207e
Add index to temp table
2019-05-30 15:02:31 +01:00
Erik Johnston
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities
...
Add DB bg update to cleanup extremities.
2019-05-30 14:28:26 +01:00
Erik Johnston
cb967e2346
Update synapse/storage/events_bg_updates.py
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-30 14:06:42 +01:00
Erik Johnston
468bd090ff
Rename constant
2019-05-30 11:24:42 +01:00
Erik Johnston
5c1ece0ffc
Move event background updates to a separate file
2019-05-30 11:22:59 +01:00
Erik Johnston
640fcbb07f
Fixup comments and logging
2019-05-30 10:55:55 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API ( #5216 )
2019-05-30 01:47:16 +10:00
Erik Johnston
7e8e683754
Log actual number of entries deleted
2019-05-29 15:11:28 +01:00
Erik Johnston
d79c9994f4
Add DB bg update to cleanup extremities.
...
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
Erik Johnston
58c8ed5b0d
Correctly filter out extremities with soft failed prevs ( #5274 )
...
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.
Fixes #5269
2019-05-29 11:56:24 +01:00
Brendan Abolivier
52839886d6
Allow configuring a range for the account validity startup job
...
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
2019-05-28 16:52:45 +01:00
Brendan Abolivier
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
...
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
Brendan Abolivier
ba17de7fbc
Fix schema update for account validity
2019-05-28 10:11:38 +01:00
Travis Ralston
bc4b2ecf70
Fix logging for room stats background update
2019-05-25 12:02:48 -06:00
Richard van der Hoff
b75537beaf
Store key validity time in the storage layer
...
This is a first step to checking that the key is valid at the required moment.
The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once ( #5234 )
...
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Richard van der Hoff
85d1e03b9d
Simplifications and comments in do_auth ( #5227 )
...
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
Amber Brown
4a30e4acb4
Room Statistics ( #4338 )
2019-05-21 11:36:50 -05:00
Erik Johnston
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender
...
Only count aggregations from distinct senders
2019-05-21 17:10:48 +01:00
Erik Johnston
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
...
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
Richard van der Hoff
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. ( #5146 )
...
When considering the candidates to be forward-extremities, we must exclude soft
failures.
Hopefully fixes #5090 .
2019-05-21 16:10:54 +01:00
Erik Johnston
7b0e804a4a
Fix get_max_topological_token to never return None
2019-05-21 15:21:43 +01:00
Brendan Abolivier
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
...
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
Brendan Abolivier
5ceee46c6b
Do the select and insert in a single transaction
2019-05-21 13:38:51 +01:00
Erik Johnston
c7ec06e8a6
Block attempts to annotate the same event twice
2019-05-20 17:39:05 +01:00
Erik Johnston
1dff859d6a
Rename relation types to match MSC
2019-05-20 14:31:19 +01:00
Erik Johnston
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
...
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
Erik Johnston
935af0da38
Correctly update aggregation counts after redaction
2019-05-20 12:09:27 +01:00
Brendan Abolivier
ad5b4074e1
Add startup background job for account validity
...
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
2019-05-17 19:37:31 +01:00
Erik Johnston
b63cc325a9
Only count aggregations from distinct senders
...
As a user isn't allowed to send a single emoji more than once.
2019-05-17 18:03:10 +01:00
Erik Johnston
8dd9cca8ea
Spelling and clarifications
2019-05-17 16:40:51 +01:00
Erik Johnston
5dbff34509
Fixup bsaed on review comments
2019-05-17 15:48:04 +01:00
Erik Johnston
d46aab3fa8
Add basic editing support
2019-05-16 16:54:45 +01:00
Erik Johnston
895179a4dc
Update docstring
2019-05-16 16:41:05 +01:00
Erik Johnston
7a7eba8302
Move parsing of tokens out of storage layer
2019-05-16 14:26:23 +01:00
Erik Johnston
2c662ddde4
Indirect tuple conversion
2019-05-16 14:21:39 +01:00
Erik Johnston
b5c62c6b26
Fix relations in worker mode
2019-05-16 10:38:13 +01:00
Erik Johnston
33453419b0
Add cache to relations
2019-05-16 10:02:14 +01:00
Erik Johnston
a0603523d2
Add aggregations API
2019-05-16 09:37:20 +01:00
Erik Johnston
5be34fc3e3
Actually check for None rather falsey
2019-05-15 17:30:23 +01:00
Erik Johnston
b50641e357
Add simple pagination API
2019-05-15 13:36:51 +01:00
Erik Johnston
efe3c7977a
Add simple send_relation API and track in DB
2019-05-15 13:36:51 +01:00
Erik Johnston
54d77107c1
Make generating SQL bounds for pagination generic
...
This will allow us to reuse the same structure when we paginate e.g.
relations
2019-05-15 11:30:05 +01:00
Erik Johnston
52ddc6c0ed
Update docstring with correct type
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-15 09:52:15 +01:00
Erik Johnston
4fb44fb5b9
Expose DataStore._get_events as get_events_as_list
...
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
Quentin Dufour
11ea16777f
Limit the number of EDUs in transactions to 100 as expected by receiver ( #5138 )
...
Fixes #3951 .
2019-05-09 11:01:41 +01:00
Richard van der Hoff
bd0d45ca69
Fix infinite loop in presence handler
...
Fixes #5102
2019-04-26 11:14:49 +01:00
Erik Johnston
ca90336a69
Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration
2019-04-17 19:44:40 +01:00
Brendan Abolivier
eaf41a943b
Add management endpoints for account validity
2019-04-17 19:34:45 +01:00
Brendan Abolivier
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
...
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier
20f0617e87
Send out emails with links to extend an account's validity period
2019-04-17 14:42:20 +01:00
Erik Johnston
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta
...
Fix schema upgrade when dropping tables
2019-04-10 10:22:35 +01:00