Erik Johnston
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
Richard van der Hoff
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
Richard van der Hoff
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
Erik Johnston
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
Richard van der Hoff
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Erik Johnston
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
Matthew Hodgson
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
Erik Johnston
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
Erik Johnston
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
Erik Johnston
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
Erik Johnston
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
Erik Johnston
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
Erik Johnston
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
Erik Johnston
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Richard van der Hoff
93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
...
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
Matthew Hodgson
454f59b7ad
Merge branch 'develop' into matthew/filter_members
2018-07-24 14:03:37 +01:00
Erik Johnston
223341205e
Don't require to_delete to have event_ids
2018-07-24 14:02:40 +01:00
Matthew Hodgson
e22700c3dd
consider non-filter_type types as wildcards, thus missing from the state-group-cache
2018-07-24 13:59:07 +01:00
Matthew Hodgson
d19fba3655
Merge branch 'develop' into matthew/filter_members
2018-07-24 12:39:54 +01:00
Matthew Hodgson
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
Richard van der Hoff
30bfed5aa5
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-24 11:46:09 +01:00
Erik Johnston
97acd385a3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta
2018-07-24 11:32:13 +01:00
Erik Johnston
0fa73e4a63
Remove unnecessary if
2018-07-24 11:19:23 +01:00
Erik Johnston
ff5426f6b8
Speed up _calculate_state_delta
2018-07-24 10:55:11 +01:00
Richard van der Hoff
a678145010
Merge branch 'develop' into rav/logcontext_fixes
2018-07-24 10:43:30 +01:00
Richard van der Hoff
8dff6e0322
Logcontext fixes
...
Fix some random logcontext leaks.
2018-07-24 00:37:17 +01:00
Richard van der Hoff
69fb5dbdab
fix idiocy
2018-07-24 00:04:44 +01:00
Richard van der Hoff
1938cffaea
Add some measure blocks to persist_events
...
... to help us figure out where 40% of CPU is going
2018-07-23 23:48:19 +01:00
Matthew Hodgson
efcdacad7d
handle case where types is [] on postgres correctly
2018-07-23 22:41:05 +01:00
Matthew Hodgson
adfe29ec0b
Merge branch 'develop' into matthew/filter_members
2018-07-23 19:21:37 +01:00
Matthew Hodgson
254fb430d1
incorporate review
2018-07-23 19:21:20 +01:00
Erik Johnston
8b9f164fff
Comments
2018-07-23 17:43:01 +01:00
Erik Johnston
50c60e5fad
Only get cached state from context in persist_event
...
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
2018-07-23 17:21:40 +01:00
Richard van der Hoff
4f5cc8e4e7
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-23 17:15:12 +01:00
Richard van der Hoff
dae6dc1e77
Remove redundant checks on room forgottenness
...
Fixes #3550
2018-07-23 17:13:34 +01:00
Erik Johnston
e42510ba63
Use new getters
2018-07-23 13:17:22 +01:00
Matthew Hodgson
9b34f3ea3a
Merge branch 'develop' into matthew/sync_deleted_devices
2018-07-23 10:03:28 +01:00
Matthew Hodgson
c1bf2b587e
add trailing comma
2018-07-23 09:56:23 +01:00
Matthew Hodgson
2f558300cc
fix thinkos; unbreak tests
2018-07-19 19:22:27 +01:00
Matthew Hodgson
bcaec2915a
incorporate review
2018-07-19 19:03:50 +01:00
Matthew Hodgson
924eb34d94
add a filtered_types param to limit filtering to specific types
2018-07-19 18:32:02 +01:00
Matthew Hodgson
be3adfc331
merge develop pydoc for _get_state_for_groups
2018-07-19 11:26:04 +01:00
Matthew Hodgson
9e40834f74
yes, we do need to invalidate the device_id_exists_cache when deleting a remote device
2018-07-19 11:15:10 +01:00
Richard van der Hoff
f1a15ea206
revert 00bc979
...
... we've fixed the things that caused the warnings, so we should reinstate the
warning.
2018-07-19 11:14:20 +01:00
Richard van der Hoff
1ffb7bec20
Merge remote-tracking branch 'origin/release-v0.33.0' into develop
2018-07-19 11:12:33 +01:00
Matthew Hodgson
c0685f67c0
spell out that include_deleted_devices requires include_all_devices
2018-07-19 10:59:02 +01:00
Richard van der Hoff
00bc979137
Disable logcontext warning
...
Temporary workaround to #3518 while we release 0.33.0.
2018-07-19 10:51:15 +01:00
Richard van der Hoff
667fba68f3
Run things as background processes
...
This fixes #3518 , and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
Matthew Hodgson
12ec58301f
shift to using an explicit deleted flag on m.device_list_update EDUs
...
and generally make it work.
2018-07-12 11:39:43 +01:00
Erik Johnston
0456e05977
Merge pull request #3505 from matrix-org/erikj/receipts_cahce
...
Use stream cache in get_linearized_receipts_for_room
2018-07-12 09:46:29 +01:00
Erik Johnston
aff1dfdf3d
Update return value docstring
2018-07-12 09:45:37 +01:00
Matthew Hodgson
5797f5542b
WIP to announce deleted devices over federation
...
Previously we queued up the poke correctly when the device was deleted,
but then the actual EDU wouldn't get sent, as the device was no longer known.
Instead, we now send EDUs for deleted devices too if there's a poke for them.
2018-07-12 01:32:39 +01:00
Erik Johnston
6ccefef07a
Use 'is not None' and add comments
2018-07-10 18:12:39 +01:00
Matthew Hodgson
ea752bdd99
s/becuase/because/g
2018-07-10 17:58:18 +01:00
Erik Johnston
05f5dabc10
Use stream cache in get_linearized_receipts_for_room
...
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
Richard van der Hoff
c3c29aa196
Attempt to include db threads in cpu usage stats ( #3496 )
...
Let's try to include time spent in the DB threads in the per-request/block cpu
usage metrics.
2018-07-10 16:12:36 +01:00
Richard van der Hoff
e31e5dee38
Add CPU metrics for _fetch_event_list
...
add a Measure block on _fetch_event_list, in the hope that we can better
measure CPU usage here.
2018-07-09 18:15:54 +01:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Erik Johnston
3905c693c5
Invalidate cache on correct thread
2018-07-02 11:36:44 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
David Baker
028490afd4
Fix error on deleting users pending deactivation
...
Use simple_delete instead of simple_delete_one as commented
2018-06-26 10:52:52 +01:00
Erik Johnston
244484bf3c
Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
...
This reverts commit 1d009013b3
.
2018-06-25 13:42:55 +01:00
Richard van der Hoff
1d009013b3
Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
...
This reverts commit ce0d911156
, reversing
changes made to b4a5d767a9
.
2018-06-22 16:35:10 +01:00
Mark Haines
9850f66abe
Deleting from event_push_actions needs to use an index
2018-06-22 15:54:48 +01:00
Richard van der Hoff
200e11c5bf
Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable
...
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:18:51 +01:00
Erik Johnston
f8272813a9
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:12:26 +01:00
Richard van der Hoff
1d7ad11747
Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation
...
Make push actions rotation configurable
2018-06-22 15:07:31 +01:00
Erik Johnston
ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility
...
Support hiding events from deleted users
2018-06-22 15:06:44 +01:00
Erik Johnston
f79abda87f
Merge pull request #3427 from matrix-org/erikj/remove_filters
...
remove dead filter_events_for_clients
2018-06-22 14:45:42 +01:00
Erik Johnston
75dc3ddeab
Make push actions rotation configurable
2018-06-22 14:44:37 +01:00
Richard van der Hoff
43e02c409d
Disable partial state group caching for wildcard lookups
...
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
2018-06-22 11:52:07 +01:00
Richard van der Hoff
240f192523
Merge pull request #3382 from matrix-org/rav/optimise_state_groups
...
Optimise state_group_cache update
2018-06-22 11:20:20 +01:00
Richard van der Hoff
70e6501913
Merge pull request #3419 from matrix-org/rav/events_per_request
...
Log number of events fetched from DB
2018-06-22 11:17:56 +01:00
Richard van der Hoff
0495fe0035
Indirect evt_count updates via method call
...
so that we can stub it for the sentinel and not have a billion failing UTs
2018-06-22 10:42:28 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
Richard van der Hoff
b088aafcae
Log number of events fetched from DB
...
When we finish processing a request, log the number of events we fetched from
the database to handle it.
[I'm trying to figure out which requests are responsible for large amounts of
event cache churn. It may turn out to be more helpful to add counts to the
prometheus per-request/block metrics, but that is an extension to this code
anyway.]
2018-06-21 06:15:03 +01:00
Richard van der Hoff
3ff8a619f5
UserErasureStore
...
to store which users have been erased
2018-06-12 09:52:22 +01:00
Richard van der Hoff
bd348f0af6
remove dead filter_events_for_clients
...
This is only used by filter_events_for_client, so we can simplify the whole
thing by just doing one user at a time, and removing a dead storage function to
boot.
2018-06-12 09:51:31 +01:00
Richard van der Hoff
eb32b2ca20
Optimise state_group_cache update
...
(1) matrix-org-hotfixes has removed the intern calls; let's do the same here.
(2) remove redundant iteritems() so we can used an optimised db update.
2018-06-11 22:56:11 +01:00
David Baker
187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind
...
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
Matthew Hodgson
c96d882a02
Merge branch 'develop' into matthew/filter_members
2018-06-10 12:26:14 +03:00
Richard van der Hoff
d3a8c9c55e
Fix sql error in _get_state_groups_from_groups
...
If this was called with a `(type, None)` entry in types (which is supposed to
return all state of type `type`), it would explode with a sql error.
2018-06-06 14:19:01 +01:00
Richard van der Hoff
f29b41fde9
Merge pull request #3324 from matrix-org/rav/remove_dead_method
...
Remove was_forgotten_at
2018-06-04 18:18:08 +01:00
Erik Johnston
042eedfa2b
Add hacky cache factor override system
2018-06-04 15:39:28 +01:00
Matthew Hodgson
28f09fcdd5
Merge branch 'develop' into matthew/filter_members
2018-06-04 00:09:17 +03:00
Richard van der Hoff
c1f4118bb6
Remove was_forgotten_at
...
This is unused. IT MUST DIE!!!1
̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢
̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *
2018-06-01 18:21:49 +01:00
Neil Johnson
71477f3317
Merge pull request #3264 from matrix-org/neil/sign-up-stats
...
daily user type phone home stats
2018-06-01 13:42:01 +00:00
Richard van der Hoff
9f797a24a4
Handle RRs which arrive before their events
2018-06-01 14:01:43 +01:00
Richard van der Hoff
857e6fd8b6
Ignore depth when updating read-receipts
...
Order read receipts by stream ordering instead of depth
2018-06-01 12:18:11 +01:00
Neil Johnson
4986b084f8
remove unnecessary INSERT
2018-06-01 10:50:40 +01:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy ( #3307 )
2018-05-31 19:03:47 +10:00
Amber Brown
872cf43516
Merge pull request #3303 from NotAFile/py3-memoryview
...
use memoryview in py3
2018-05-30 12:51:42 +10:00
Amber Brown
debff7ae09
Merge pull request #3281 from NotAFile/py3-six-isinstance
...
remaining isintance fixes
2018-05-30 12:44:46 +10:00
Richard van der Hoff
4a9cbdbc15
Exempt AS-registered users from doing gdpr
2018-05-29 19:54:32 +01:00
Neil Johnson
ab0ef31dc7
create users index on creation_ts
2018-05-29 17:51:08 +01:00
Neil Johnson
558f3d376a
create index in background
2018-05-29 17:47:55 +01:00
Neil Johnson
c379acd4fd
bump version
2018-05-29 17:47:28 +01:00
Adrian Tschira
4b9d0cde97
add remaining memoryview changes
2018-05-29 17:42:43 +02:00
Adrian Tschira
1afafb3497
use memoryview in py3
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-29 17:14:34 +02:00
Matthew Hodgson
a6c8f7c875
add pydoc
2018-05-29 01:09:55 +01:00
Matthew Hodgson
7a6df013cc
merge develop
2018-05-29 00:25:22 +01:00
Amber Brown
a2eb5db4a0
update metrics to be in seconds
2018-05-28 19:10:27 +10:00
Amber Brown
754826a830
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-28 18:57:23 +10:00
Amber Brown
cc1349c06a
Merge pull request #3279 from NotAFile/py3-more-iteritems
...
more six iteritems
2018-05-24 14:23:13 -05:00
Adrian Tschira
dd068ca979
remaining isintance fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:55:08 +02:00
Adrian Tschira
17a70cf6e9
Misc. py3 fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
Adrian Tschira
6c16a4ec1b
more iteritems
2018-05-24 20:19:06 +02:00
Amber Brown
7ea07c7305
Merge pull request #3278 from NotAFile/py3-storage-base
...
Py3 storage/_base.py
2018-05-24 13:08:09 -05:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
...
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown
36501068d8
Merge pull request #3247 from NotAFile/py3-misc
...
Misc Python3 fixes
2018-05-24 12:58:37 -05:00
Adrian Tschira
095292304f
Py3 storage/_base.py
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 18:24:12 +02:00
David Baker
77a23e2e05
Merge remote-tracking branch 'origin/develop' into dbkr/unbind
2018-05-24 16:20:53 +01:00
David Baker
2c7866d664
Hit the 3pid unbind endpoint on deactivation
2018-05-23 14:38:56 +01:00
Amber Brown
53cc2cde1f
cleanup
2018-05-22 17:32:57 -05:00
Erik Johnston
a17e901f4d
Remove unused string formatting param
2018-05-22 18:24:32 +01:00
Erik Johnston
5494c1d71e
Don't support limitless pagination
...
The pagination storage function supported not specifiying a limit on the
number of events returned. This was triggered when using the search or
context API with a limit of zero, which the storage function took to
mean not being limited.
2018-05-22 18:15:21 +01:00
Neil Johnson
d8cb7225d2
daily user type phone home stats
2018-05-22 18:09:09 +01:00
Amber Brown
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Amber Brown
fcc525b0b7
rest of the changes
2018-05-21 19:48:57 -05:00
Adrian Tschira
933bf2dd35
replace some iteritems with six
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira
dcc235b47d
use stand-in value if maxint is not available
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:44 +02:00
Neil Johnson
644aac5f73
Tighter filtering for user_daily_visits
2018-05-18 17:10:35 +01:00
Neil Johnson
ef466b3a13
fix psql compatability bug
2018-05-18 15:51:21 +01:00
Richard van der Hoff
ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred
...
Fix error in handling receipts
2018-05-18 11:23:11 +01:00
Erik Johnston
fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token
...
Make purge_history operate on tokens
2018-05-18 10:35:23 +01:00
Richard van der Hoff
8b1c856d81
Fix error in handling receipts
...
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):
> TypeError: argument of type 'ObservableDeferred' is not iterable
fixes #3234
2018-05-18 09:15:35 +01:00
Erik Johnston
f7906203f6
Merge pull request #3212 from matrix-org/erikj/epa_stream
...
Use stream rather depth ordering for push actions
2018-05-17 12:01:21 +01:00
Erik Johnston
680530cc7f
Clarify comment
2018-05-16 11:47:29 +01:00
Erik Johnston
43e6e82c4d
Comments
2018-05-16 11:13:31 +01:00
Neil Johnson
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
Erik Johnston
c945af8799
Move and rename variable
2018-05-16 10:52:06 +01:00
Neil Johnson
be11a02c4f
remove empty line
2018-05-16 10:45:40 +01:00
Neil Johnson
31c2502ca8
style and further contraining query
2018-05-16 09:46:43 +01:00
Richard van der Hoff
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Neil Johnson
05ac15ae82
Limit query load of generate_user_daily_visits
...
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
Erik Johnston
5f27ed75ad
Make purge_history operate on tokens
...
As we're soon going to change how topological_ordering works
2018-05-15 16:23:50 +01:00
Erik Johnston
37dbee6490
Use events_to_purge table rather than token
2018-05-15 16:23:47 +01:00
Richard van der Hoff
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson
f077e97914
instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day
2018-05-14 13:50:58 +01:00
David Baker
8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate
...
Part user from rooms on account deactivate
2018-05-14 11:31:48 +01:00
Neil Johnson
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
Erik Johnston
6406b70aeb
Use stream rather depth ordering for push actions
...
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
David Baker
217bc53c98
Many docstrings
2018-05-10 12:20:40 +01:00
Erik Johnston
bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync
...
Refactor sync APIs to reuse pagination API
2018-05-09 16:16:56 +01:00
Erik Johnston
5adb75bcba
Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination
...
Refactor get_recent_events_for_room return type
2018-05-09 16:07:14 +01:00
Erik Johnston
7ce98804ff
Fix up comment
2018-05-09 15:42:39 +01:00
Erik Johnston
cddf91c8b9
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change
2018-05-09 15:32:07 +01:00
Erik Johnston
9896dab8f6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination
2018-05-09 15:31:33 +01:00
Erik Johnston
75552d2148
Update comments
2018-05-09 15:15:38 +01:00
David Baker
46df23f581
Add the schema file
2018-05-09 15:07:54 +01:00
David Baker
7e8726b8fb
Part deactivated users in the background
...
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
2018-05-09 14:54:28 +01:00
Erik Johnston
c0e08dc45b
Remove unused code path from member change DB func
...
The function is never called without a from_key, so we can remove all
the handling for that scenario.
2018-05-09 14:31:32 +01:00
Erik Johnston
e2accd7f1d
Refactor sync APIs to reuse pagination API
...
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
2018-05-09 13:43:39 +01:00
Erik Johnston
e5ab9cd24b
Don't unnecessarily require token to be stream token
...
This allows calling the `get_recent_event_ids_for_room` function in more
situations.
2018-05-09 11:58:35 +01:00
Erik Johnston
c4af4c24ca
Refactor get_recent_events_for_room return type
...
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
2018-05-09 11:55:34 +01:00
Erik Johnston
05e0a2462c
Refactor pagination DB API to return concrete type
...
This makes it easier to document what is being returned by the storage
functions and what some functions expect as arguments.
2018-05-09 11:34:24 +01:00
Erik Johnston
7dd13415db
Remove unused from_token param
2018-05-09 10:58:16 +01:00
Erik Johnston
27cf170558
Refactor recent events func to use pagination func
...
This also removes a cache that is unlikely to ever get hit.
2018-05-09 10:55:55 +01:00
Erik Johnston
23ec51c94c
Fix up comments and make function private
2018-05-09 09:55:19 +01:00
Erik Johnston
696f532453
Reuse existing pagination code for context API
2018-05-08 16:20:19 +01:00
Erik Johnston
3e6d306e94
Parse tokens before calling DB function
2018-05-08 16:18:58 +01:00
Erik Johnston
274b8c6025
Only fetch required fields from database
2018-05-08 16:15:25 +01:00
Erik Johnston
06c0d0ed08
Split paginate_room_events storage function
2018-05-08 16:14:26 +01:00
Erik Johnston
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
...
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
Erik Johnston
31c7c29d43
Fix up grammar
2018-05-03 10:38:58 +01:00
Neil Johnson
dd1a832419
remove user agent from data model, will just join on user_ips
2018-05-01 12:13:49 +01:00
Neil Johnson
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
Matthew Hodgson
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
Matthew Hodgson
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
Matthew Hodgson
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
Matthew Hodgson
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
Adrian Tschira
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Richard van der Hoff
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
Erik Johnston
6e10eed28e
Refactor event storage to not require state
...
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.
We also add a check to see if the state groups of the old extremities
are the same as the new ones.
2018-04-27 11:38:02 +01:00
Richard van der Hoff
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston
7ec8e798b4
Fix media admin APIs
2018-04-26 11:31:22 +01:00
Neil Johnson
fb6015d0a6
pep8
2018-04-25 17:56:11 +01:00
Neil Johnson
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
Erik Johnston
22881b3d69
Also fix reindexing of search
2018-04-25 15:32:04 +01:00
Erik Johnston
ba3166743c
Fix quarantine media admin API
2018-04-25 15:11:18 +01:00
Matthew Hodgson
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
Richard van der Hoff
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
...
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
Richard van der Hoff
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
Richard van der Hoff
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
Richard van der Hoff
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira
878995e660
Replace Queue with six.moves.queue
...
and a six.range change which I missed the last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
Erik Johnston
415aeefd89
Format docstring
2018-04-12 12:07:09 +01:00
Erik Johnston
23a7f9d7f4
Doc we raise on unknown event
2018-04-12 11:20:51 +01:00
Erik Johnston
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
Richard van der Hoff
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
Neil Johnson
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
Richard van der Hoff
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
Vincent Breitmoser
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
Neil Johnson
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
Neil Johnson
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
Neil Johnson
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
Neil Johnson
64bc2162ef
Fix psycopg2 interpolation
2018-04-09 17:50:36 +01:00
Luke Barnard
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
Luke Barnard
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
Luke Barnard
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
David Baker
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
David Baker
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
Richard van der Hoff
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
Luke Barnard
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
Luke Barnard
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
Luke Barnard
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
Neil Johnson
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
Luke Barnard
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
Richard van der Hoff
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
Richard van der Hoff
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
Richard van der Hoff
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
Neil Johnson
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
Neil Johnson
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
Neil Johnson
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
Neil Johnson
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
David Baker
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
David Baker
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
Neil Johnson
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
Neil Johnson
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
David Baker
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
David Baker
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
David Baker
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
Neil Johnson
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
Neil Johnson
0f890f477e
No need to cast in count_daily_users
2018-03-28 11:49:57 +01:00
Neil Johnson
545001b9e4
Fix search_user_dir multiple sqlite versions do different things
2018-03-28 11:19:45 +01:00
Neil Johnson
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
Neil Johnson
9187e0762f
count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home
2018-03-28 10:02:32 +01:00
Erik Johnston
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
...
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
Erik Johnston
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
...
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
Erik Johnston
800cfd5774
Comment
2018-03-27 13:30:39 +01:00
Erik Johnston
152c2ac19e
Fix indent
2018-03-27 13:13:46 +01:00
Erik Johnston
e70287cff3
Comment
2018-03-27 13:13:38 +01:00
Erik Johnston
3f49e131d9
Add counter metrics for calculating state delta
...
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
Erik Johnston
fecb45e0c3
Remove last usage of ujson
2018-03-26 13:32:29 +01:00
Erik Johnston
8d6dc106d1
Don't use _cursor_to_dict in find_next_generated_user_id_localpart
2018-03-26 12:02:44 +01:00
Erik Johnston
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
Erik Johnston
bf49d2dca8
Replace some ujson with simplejson to make it work
2018-03-16 00:55:44 +00:00
Erik Johnston
a8ce159be4
Replace some ujson with simplejson to make it work
2018-03-16 00:27:09 +00:00
Erik Johnston
5b631ff41a
Remove wrong comment
2018-03-16 00:07:08 +00:00
Erik Johnston
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
Matthew Hodgson
056a6df546
Merge branch 'develop' into matthew/filter_members
2018-03-14 15:38:05 +00:00
Erik Johnston
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
...
Split up ProfileStore
2018-03-14 09:41:06 +00:00
Erik Johnston
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
...
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
Matthew Hodgson
9f77001e27
pep8
2018-03-14 00:07:47 +00:00
Matthew Hodgson
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) ( #2990 )
...
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
Matthew Hodgson
ccca02846d
make it work
2018-03-13 22:31:41 +00:00
Matthew Hodgson
f0f9a0605b
remove comment now #2969 is fixed
2018-03-13 22:12:15 +00:00
Matthew Hodgson
12350e3f9a
merge proper fix to bug 2969
2018-03-13 22:11:58 +00:00
Matthew Hodgson
afbf4d3dcc
typoe
2018-03-13 19:48:04 +00:00
Matthew Hodgson
865377a70d
disable optimisation for searching for state groups
...
when type filter includes wildcards on state_key
2018-03-13 19:46:04 +00:00
Erik Johnston
a08726fc42
Add is_blocked to worker store
2018-03-13 18:28:44 +00:00
Matthew Hodgson
b2aba9e430
build where_clause sanely
2018-03-13 18:13:44 +00:00
Matthew Hodgson
52f7e23c72
PR feedbackz
2018-03-13 18:07:55 +00:00
Matthew Hodgson
1b1c137771
fix bug #2926
2018-03-13 17:52:52 +00:00
Erik Johnston
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
Richard van der Hoff
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
Matthew Hodgson
fdedcd1f4d
correctly handle None state_keys
...
and fix include_other_types thinko
2018-03-12 01:39:06 +00:00
Matthew Hodgson
97c0496cfa
fix sqlite where clause
2018-03-12 00:27:06 +00:00
Matthew Hodgson
8713365265
typos
2018-03-11 20:10:25 +00:00
Matthew Hodgson
9b334b3f97
WIP experiment in lazyloading room members
2018-03-11 20:01:41 +00:00
Erik Johnston
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
Richard van der Hoff
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
Erik Johnston
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
Erik Johnston
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
Erik Johnston
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
Erik Johnston
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
Erik Johnston
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
Richard van der Hoff
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
Richard van der Hoff
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
Richard van der Hoff
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Richard van der Hoff
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
Erik Johnston
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
Erik Johnston
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
Erik Johnston
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
Erik Johnston
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
Erik Johnston
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
Erik Johnston
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
Erik Johnston
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
Erik Johnston
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
Erik Johnston
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
Erik Johnston
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
Erik Johnston
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
Erik Johnston
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
Erik Johnston
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
Erik Johnston
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
Erik Johnston
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
Erik Johnston
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
Erik Johnston
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
Erik Johnston
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
Erik Johnston
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
Erik Johnston
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
Erik Johnston
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
Erik Johnston
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
Erik Johnston
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
Erik Johnston
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
Erik Johnston
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
Erik Johnston
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
Erik Johnston
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
Erik Johnston
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
Erik Johnston
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
Erik Johnston
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
Erik Johnston
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
Erik Johnston
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
Erik Johnston
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
Erik Johnston
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
Erik Johnston
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
Erik Johnston
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
Erik Johnston
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
Erik Johnston
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
Erik Johnston
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
Erik Johnston
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
Erik Johnston
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
Erik Johnston
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
Richard van der Hoff
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
added new code to avoid the tablescan, but didn't remove the
old :/
2018-02-16 14:02:31 +00:00
Erik Johnston
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
Erik Johnston
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
Erik Johnston
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
Erik Johnston
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
Erik Johnston
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
Erik Johnston
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
Erik Johnston
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
Erik Johnston
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
Richard van der Hoff
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
Richard van der Hoff
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
Richard van der Hoff
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
Richard van der Hoff
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
Richard van der Hoff
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
Richard van der Hoff
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
Richard van der Hoff
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
Richard van der Hoff
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
Richard van der Hoff
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
Richard van der Hoff
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
Travis Ralston
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
Travis Ralston
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
Richard van der Hoff
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
Travis Ralston
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
Richard van der Hoff
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
Richard van der Hoff
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
Richard van der Hoff
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
Travis Ralston
a94d9b6b82
Appease the linter
...
These are ids anyways, not mxc uris.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
Travis Ralston
5552ed9a7f
Add an admin route to get all the media in a room
...
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00
Richard van der Hoff
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
...
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
Richard van der Hoff
1224612a79
Log room when doing state resolution
...
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
Erik Johnston
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed
...
Keep track of last access time for local media
2018-01-17 16:39:02 +00:00
Matthew Hodgson
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql
...
fix SQL when searching all users
2018-01-17 16:27:54 +00:00
Matthew Hodgson
2d9ab533f9
fix SQL when searching all users
2018-01-17 15:58:52 +00:00
Richard van der Hoff
390093d45e
Split resolve_events into two functions
...
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
Erik Johnston
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
Erik Johnston
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
Richard van der Hoff
3d12d97415
Track DB scheduling delay per-request
...
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
Richard van der Hoff
8615f19d20
rework runInteraction in terms of runConnection
...
... so that we can share the code
2018-01-16 17:08:29 +00:00
Richard van der Hoff
64ddec1bc0
Fix a logcontext leak in persist_events
...
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.
It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.
(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
hera
174eacc8ba
oops
2018-01-09 18:14:32 +00:00
Matthew Hodgson
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
Matthew Hodgson
e79db0a673
switch back from GIST to GIN indexes
2018-01-09 16:37:48 +00:00
Matthew Hodgson
e365ad329f
oops, tweak work_mem when actually storing
2018-01-09 16:30:30 +00:00
Matthew Hodgson
19f9227643
avoid 80s GIN inserts by tweaking work_mem
...
see https://github.com/matrix-org/synapse/issues/2753 for details
2018-01-09 16:25:04 +00:00
Erik Johnston
b30cd5b107
Remove dead code related to default thumbnails
2018-01-09 14:38:33 +00:00
Matthew Hodgson
cdc2cb5d11
fix StoreError syntax
2017-12-05 11:09:31 +00:00
Matthew Hodgson
74e0cc74ce
fix pep8 and tests
2017-12-04 15:11:38 +00:00
Matthew Hodgson
1bd40ca73e
switch to a simpler 'search_all_users' button as per review feedback
2017-12-04 14:58:39 +00:00
Matthew Hodgson
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
Matthew Hodgson
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
Matthew Hodgson
4b1fceb913
fix alternation operator for FTS4 - how did this ever work!?
2017-11-30 01:34:03 +00:00
Matthew Hodgson
a4bb133b68
fix thinkos galore
2017-11-30 01:17:15 +00:00
Matthew Hodgson
3241c7aac3
untested WIP but might actually work
2017-11-29 18:27:05 +00:00
Matthew Hodgson
47d99a20d5
Add user_directory_include_pattern config param to expand search results to additional users
...
Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive.
It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users...
...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too.
Committing it here as a WIP
2017-11-29 16:46:45 +00:00
Richard van der Hoff
2c6d63922a
Remove pushers when deleting access tokens
...
Whenever an access token is invalidated, we should remove the associated
pushers.
2017-11-29 16:44:35 +00:00
Richard van der Hoff
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error
...
Fix error on sqlite 3.7
2017-11-27 12:25:48 +00:00
Richard van der Hoff
63ccaa5873
Avoid retrying forever on IntegrityError
2017-11-27 12:00:07 +00:00
Richard van der Hoff
6b48b3e277
fix sql fails
2017-11-22 18:06:24 +00:00
Richard van der Hoff
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
Richard van der Hoff
0b4aa2dc21
Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert
...
Avoid locking account_data tables for upserts
2017-11-21 13:39:14 +00:00
Richard van der Hoff
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
Richard van der Hoff
c46139a17e
Avoid locking account_data tables for upserts
2017-11-16 18:08:01 +00:00
Richard van der Hoff
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
Richard van der Hoff
77a1227870
Fix broken ref to IntegrityError
2017-11-16 16:03:38 +00:00
Richard van der Hoff
7ab2b69e18
Avoid locking pushers
table on upsert
...
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the
table.
2017-11-16 15:32:01 +00:00
Richard van der Hoff
10aaa1bc15
_simple_upsert: retry on IntegrityError
...
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
2017-11-16 15:30:15 +00:00
Richard van der Hoff
cdc9e50a5d
Cleanup in _simple_upsert_txn
...
Bail out early to reduce indentation
2017-11-16 15:29:10 +00:00
Richard van der Hoff
f959c01600
Merge pull request #2661 from matrix-org/rav/statereadstore
...
Pull out bits of StateStore to a mixin
2017-11-15 17:23:01 +00:00
Richard van der Hoff
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs
...
Remove preserve_context_over_{fn, deferred}
2017-11-15 11:13:53 +00:00
Richard van der Hoff
35a4b63240
Pull out bits of StateStore to a mixin
...
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.
fixes https://github.com/matrix-org/synapse/issues/2655 .
2017-11-14 11:43:58 +00:00
Richard van der Hoff
4dd1bfa8c1
Revert "Revert "move _state_group_cache to statestore""
...
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.
This reverts commit ab335edb02
.
2017-11-14 11:43:58 +00:00
Richard van der Hoff
7e6fa29cb5
Remove preserve_context_over_{fn, deferred}
...
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
Richard van der Hoff
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston
ab335edb02
Revert "move _state_group_cache to statestore"
...
This reverts commit f5cf3638e9
.
2017-11-13 10:05:33 +00:00
Erik Johnston
bfbf1e1f1a
Up cache size of get_global_account_data_by_type_for_user
2017-11-13 09:52:11 +00:00
Erik Johnston
2dce6b15c3
Fix typo
2017-11-09 15:56:16 +00:00
Richard van der Hoff
618c7b816a
Merge pull request #2656 from matrix-org/rav/fix_deactivate
...
Fix 'NoneType' not iterable in /deactivate
2017-11-09 15:20:35 +00:00
Richard van der Hoff
889102315e
Fix 'NoneType' not iterable in /deactivate
...
make sure we actually return a value from user_delete_access_tokens
2017-11-09 15:15:33 +00:00
Erik Johnston
e8814410ef
Have an explicit API to update room config
2017-11-08 16:13:27 +00:00
Erik Johnston
94ff2cda73
Revert "Modify group room association API to allow modification of is_public"
2017-11-08 15:43:34 +00:00
Erik Johnston
167eb01d83
Merge pull request #2637 from spantaleev/avoid-noop-media-deletes
...
Avoid no-op media deletes
2017-11-08 11:53:27 +00:00
Richard van der Hoff
02a9a93bde
Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res
...
Fix bug in state group storage
2017-11-08 09:22:13 +00:00
Matthew Hodgson
228ccf1fe3
Merge pull request #2643 from matrix-org/matthew/user_dir_typos
...
Fix various embarrassing typos around user_directory and add some doc.
2017-11-07 17:31:11 +00:00
Richard van der Hoff
f5cf3638e9
move _state_group_cache to statestore
...
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
Luke Barnard
38b265cb51
Remember to pick is_admin out of the db
2017-11-07 11:24:04 +00:00
Matthew Hodgson
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
Matthew Hodgson
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
Matthew Hodgson
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
Matthew Hodgson
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
Matthew Hodgson
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
Slavi Pantaleev
805196fbeb
Avoid no-op media deletes
...
If there are no media entries to delete,
avoid creating transactions, prepared statements
and unnecessary log entries.
Signed-off-by: Slavi Pantaleev <slavi@devture.com>
2017-11-04 09:50:15 +02:00
Richard van der Hoff
bc8a5c0330
Notify auth providers on logout
...
Provide a hook by which auth providers can be notified of logouts.
2017-11-01 16:51:51 +00:00
Matthew Hodgson
275826f234
Merge pull request #2617 from matrix-org/matthew/auto-displayname
...
automatically set default displayname on register
2017-11-01 16:21:16 +00:00
David Baker
fbbacb284e
Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens
...
Remove the last vestiges of refresh_tokens
2017-11-01 15:57:35 +00:00
Matthew Hodgson
9f7a555b4e
switch to setting default displayname in the storage layer
...
to avoid clobbering guest user displaynames on registration
2017-11-01 15:51:30 +00:00
Luke Barnard
99354b430e
Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public
...
Modify group room association API to allow modification of is_public
2017-11-01 11:08:36 +00:00
Luke Barnard
318a249c8b
Leave is_public
as required argument of update_room_group_association
2017-11-01 09:36:01 +00:00
Richard van der Hoff
356bcafc44
Remove the last vestiges of refresh_tokens
2017-10-31 20:35:58 +00:00
Luke Barnard
20fe347906
Modify group room association API to allow modification of is_public
...
also includes renamings to make things more consistent.
2017-10-31 17:04:28 +00:00
Richard van der Hoff
1650eb5847
DB schema interface for password auth providers
...
Provide an interface by which password auth providers can register db schema
files to be run at startup
2017-10-31 14:01:53 +00:00
Erik Johnston
4ab8abbc2b
Merge branch 'erikj/attestation_local_fix' of github.com:matrix-org/synapse into develop
2017-10-27 15:07:08 +01:00
Luke Barnard
5451cc7792
Request is_public from database
2017-10-27 11:27:43 +01:00
Luke Barnard
6362298fa5
Create groups with is_public = True
2017-10-27 11:04:20 +01:00
Erik Johnston
82d8c1bacb
Fixup
2017-10-27 10:30:21 +01:00
Erik Johnston
195abfe7a5
Remove incorrect attestations
2017-10-27 09:58:13 +01:00
Luke Barnard
007cd48af6
Recreate groups table instead of adding column
...
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
Luke Barnard
713e60b9b6
Awful hack to get default true
2017-10-26 17:38:14 +01:00
Luke Barnard
cfa4e658e0
Bump schema version to 46
2017-10-26 17:23:49 +01:00
Luke Barnard
9b2feef9eb
Add is_public to groups table to allow for private groups
...
Prevent group API access to non-members for private groups
Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff
2e9f5ea31a
Fix logcontext handling for persist_events
...
* don't use preserve_context_over_deferred, which is known broken.
* remove a redundant preserve_fn.
* add/improve some comments
2017-10-17 10:59:30 +01:00
Luke Barnard
2c5972f87f
Implement GET /groups/$groupId/invited_users
2017-10-16 15:31:11 +01:00
Erik Johnston
818b08d0e4
peeeeeeeeep8888888888888888888888888888
2017-10-11 15:54:00 +01:00
Erik Johnston
ea18996f54
Fix group stream replication
...
The stream update functions expect the storage function to return a list
of tuples.
2017-10-11 15:44:39 +01:00
Erik Johnston
c2c47550f9
Fix schema delta versions
2017-10-11 13:23:15 +01:00
Erik Johnston
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
Erik Johnston
e8496efe84
Fix up comment
2017-10-09 15:17:34 +01:00
Erik Johnston
11d62f43c9
Invalidate cache
2017-10-03 14:12:28 +01:00
Erik Johnston
e4ab96021e
Update comments
2017-10-03 14:10:41 +01:00
Erik Johnston
30848c0fcd
Ignore incoming events for rooms that we have left
...
When synapse receives an event for a room its not in over federation, it
double checks with the remote server to see if it is in fact in the
room. This is done so that if the server has forgotten about the room
(usually as a result of the database being dropped) it can recover from
it.
However, in the presence of state resets in large rooms, this can cause
a lot of work for servers that have legitimately left. As a hacky
solution that supports both cases we drop incoming events for rooms that
we have explicitly left.
This means that we no longer support the case of servers having
forgotten that they've rejoined a room, but that is sufficiently rare
that we're not going to support it for now.
2017-10-03 11:18:21 +01:00
David Baker
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
Erik Johnston
768f00dedb
Up the limits on number of url cache entries to delete at one time
2017-09-28 14:27:27 +01:00
Erik Johnston
4dc07e93a8
Add old indices
2017-09-28 14:10:33 +01:00
Erik Johnston
93247a424a
Only pull out local media that were for url cache
2017-09-28 13:48:14 +01:00
Erik Johnston
7a44c01d89
Fix typo
2017-09-28 12:46:04 +01:00
Erik Johnston
ae79764fe5
Change expires column to expires_ts
2017-09-28 12:37:53 +01:00
Erik Johnston
77f1d24de3
More brackets
2017-09-28 12:23:15 +01:00
Erik Johnston
9ccb4226ba
Delete expired url cache data
2017-09-28 12:18:06 +01:00
Erik Johnston
17b8e2bd02
Add remove room API
2017-09-26 15:52:41 +01:00
Erik Johnston
a8e2a3df32
Add unique index to group_rooms table
2017-09-26 15:39:21 +01:00
Erik Johnston
e1dec2f1a7
Remove user from group summary when the leave the group
2017-09-21 16:09:57 +01:00
Erik Johnston
197d82dc07
Correctly return next token
2017-09-20 11:12:11 +01:00
Erik Johnston
069ae2df12
Fix initial sync
2017-09-20 10:52:12 +01:00
Richard van der Hoff
e76d1135dd
Invalidate signing key cache when we gat an update
...
This might make the cache slightly more efficient.
2017-09-20 01:32:42 +01:00
Erik Johnston
4a9b1cf253
Add user profiles to summary from group server
2017-08-25 16:23:58 +01:00
Erik Johnston
27ebc5c8f2
Add remote profile cache
2017-08-25 11:25:47 +01:00
Erik Johnston
97c544f91f
Add _simple_update
2017-08-25 11:11:37 +01:00
Erik Johnston
2800983f3e
Merge pull request #2410 from matrix-org/erikj/groups_publicise
...
Add ability to publicise group membership
2017-08-21 16:51:56 +01:00
Erik Johnston
8b50fe5330
Use BOOLEAN rather than TEXT type
2017-08-21 16:37:29 +01:00
Erik Johnston
175a01f56c
Groups: Fix mising json.load in initial sync
2017-08-21 14:45:56 +01:00
Erik Johnston
ef8e578677
Add bulk group publicised lookup API
2017-08-09 13:36:22 +01:00
Erik Johnston
b880ff190a
Allow update group publicity
2017-08-08 14:19:41 +01:00
Erik Johnston
05e21285aa
Store whether the user wants to publicise their membership of a group
2017-08-08 13:01:46 +01:00
Erik Johnston
966a70f1fa
Update comment
2017-07-24 17:49:39 +01:00
Erik Johnston
629cdfb124
Use join rather than joined, etc.
2017-07-24 14:54:05 +01:00
Erik Johnston
ed666d3969
Fix all the typos
2017-07-24 14:05:09 +01:00
Erik Johnston
b76ef6ccb8
Include users membership in group in summary API
2017-07-24 13:55:39 +01:00
Erik Johnston
851aeae7c7
Check users/rooms are in group before adding to summary
2017-07-24 13:40:56 +01:00
Erik Johnston
96917d5552
Merge pull request #2378 from matrix-org/erikj/group_sync_support
...
Add groups to sync stream
2017-07-21 11:05:39 +01:00
Erik Johnston
0401604222
Merge pull request #2377 from matrix-org/erikj/group_profile_update
...
Add update group profile API
2017-07-20 17:53:39 +01:00
Erik Johnston
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
Erik Johnston
c544188ee3
Add groups to sync stream
2017-07-20 16:36:42 +01:00
Erik Johnston
8209b5f033
Fix a storage desc
2017-07-20 16:22:22 +01:00
Erik Johnston
57826d645b
Fix typo
2017-07-20 13:15:22 +01:00
Erik Johnston
6f443a74cf
Add update group profile API
2017-07-20 09:46:33 +01:00
Erik Johnston
94ecd871a0
Fix typos
2017-07-18 16:38:54 +01:00
Erik Johnston
3e703eb04e
Comment
2017-07-18 10:17:25 +01:00
Erik Johnston
508460f240
Remove sync stuff
2017-07-18 09:55:46 +01:00
Erik Johnston
4540730111
Remove unused tables
2017-07-18 09:38:15 +01:00
Erik Johnston
e96ee95a7e
Remove sync stuff
2017-07-18 09:38:08 +01:00
Erik Johnston
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
Erik Johnston
85fda57208
Add DEFAULT_ROLE_ID
2017-07-14 14:03:54 +01:00
Erik Johnston
4b203bdba5
Correctly increment orders
2017-07-14 14:02:31 +01:00
Erik Johnston
3b0470dba5
Remove unused functions
2017-07-13 13:54:01 +01:00
Erik Johnston
8575e3160f
Comments
2017-07-13 13:52:41 +01:00
Erik Johnston
26451a09eb
Comments
2017-07-12 14:47:18 +01:00
Erik Johnston
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
Erik Johnston
6d586dc05c
Comment
2017-07-12 09:58:37 +01:00
Erik Johnston
410b4e14a1
Move comment
2017-07-11 15:44:18 +01:00
Erik Johnston
e52c391cd4
Rename column to attestation_json
2017-07-11 14:25:46 +01:00
Erik Johnston
0aac30d53b
Comments
2017-07-11 14:23:50 +01:00
Erik Johnston
83936293eb
Comments
2017-07-11 11:42:25 +01:00
Erik Johnston
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
Erik Johnston
f502b0dea1
Perf: Don't filter events for push
...
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
Erik Johnston
ed9a7f5436
Merge pull request #2309 from matrix-org/erikj/user_ip_repl
...
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston
b5e8d529e6
Define CACHE_SIZE_FACTOR once
2017-07-04 09:56:44 +01:00
Erik Johnston
6ff14ddd2e
Make into list
2017-06-29 15:47:37 +01:00
Erik Johnston
5946aa0877
Prefill forward extrems and event to state groups
2017-06-29 15:38:48 +01:00
Erik Johnston
8c23221666
Fix up
2017-06-27 15:53:45 +01:00
Erik Johnston
a0a561ae85
Fix up client ips to read from pending data
2017-06-27 14:46:12 +01:00
Erik Johnston
ed3d0170d9
Batch upsert user ips
2017-06-27 13:37:04 +01:00
Erik Johnston
8abdd7b553
Fix up indices for users_who_share_rooms
2017-06-26 14:01:30 +01:00
Erik Johnston
7fe8ed1787
Store URL cache preview downloads seperately
...
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
Erik Johnston
e0004aa28a
Add desc
2017-06-22 10:03:48 +01:00
Erik Johnston
b668112320
Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar
...
Don't work out users who share room with appservice users
2017-06-21 14:50:24 +01:00
Erik Johnston
dae9a00a28
Initialise exclusive_user_regex
2017-06-21 14:19:33 +01:00
Krombel
4202fba82a
Merge branch 'develop' into avoid_duplicate_filters
2017-06-21 14:48:21 +02:00
Krombel
812c030e87
replaced json.dumps with encode_canonical_json
2017-06-21 14:48:12 +02:00
Erik Johnston
385dcb7c60
Handle thumbnail urls
2017-06-19 17:48:28 +01:00
Erik Johnston
b8b936a6ea
Add API to quarantine media
2017-06-19 17:39:21 +01:00
Erik Johnston
d7fe6b356c
Add shutdown room API
2017-06-19 12:37:27 +01:00
Erik Johnston
dfeca6cf40
Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
...
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
Erik Johnston
d8f47d2efa
Merge pull request #2280 from matrix-org/erikj/share_room_user_dir
...
Include users who you share a room with in user directory
2017-06-16 11:06:00 +01:00
Erik Johnston
052c5d19d5
Merge pull request #2281 from matrix-org/erikj/phone_home_stats
...
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston
a9d6fa8b2b
Include users who share room with requester in user directory
2017-06-15 10:17:21 +01:00
Erik Johnston
4564b05483
Implement updating users who share rooms on the fly
2017-06-15 10:17:17 +01:00
Erik Johnston
72613bc379
Implement initial population of users who share rooms table
2017-06-15 09:59:04 +01:00
Erik Johnston
ebcd55d641
Add DB schema for tracking users who share rooms
2017-06-15 09:45:48 +01:00
Erik Johnston
4b461a6931
Add some more stats
2017-06-15 09:39:39 +01:00
Erik Johnston
617304b2cf
Fix phone home stats
2017-06-14 19:47:15 +01:00
Erik Johnston
d9fd937e39
Fix user directory insertion due to missing room_id
2017-06-13 11:50:24 +01:00
Erik Johnston
505e7e8b9d
Fix up sql
2017-06-13 11:19:18 +01:00
Erik Johnston
6fd7e6db3d
Fix user dir to not assume existence of user
2017-06-13 11:11:26 +01:00
Erik Johnston
fdca6e36ee
Merge pull request #2274 from matrix-org/erikj/cache_is_host_joined
...
Add cache for is_host_joined
2017-06-13 10:58:53 +01:00
Erik Johnston
b58e24cc3c
Tweak the ranking of PG user dir search
2017-06-13 10:16:31 +01:00
Erik Johnston
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
Erik Johnston
a837765e8c
Merge pull request #2266 from matrix-org/erikj/host_in_room
...
Change is_host_joined to use current_state table
2017-06-12 09:49:51 +01:00
Erik Johnston
8060974344
Fix replication
2017-06-09 16:40:52 +01:00
Erik Johnston
b0d975e216
Comments
2017-06-09 16:25:42 +01:00
Erik Johnston
e54d7d536e
Cache state deltas
2017-06-09 16:24:00 +01:00
Erik Johnston
935e588799
Tweak SQL
2017-06-09 13:01:23 +01:00
Erik Johnston
0185b75381
Change is_host_joined to use current_state table
...
This bypasses a bug where using the state groups to figure out if a host
is in a room sometimes errors if the servers isn't in the room. (For
example when the server rejected an invite to a remote room)
2017-06-09 10:52:26 +01:00
Erik Johnston
ea11ee09f3
Ensure we don't use unpersisted state group as prev group
2017-06-08 11:59:57 +01:00
Erik Johnston
c62c480dc6
Merge pull request #2259 from matrix-org/erikj/fix_state_woes
...
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0
Fix bug where state_group tables got corrupted
...
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.
We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston
a053ff3979
Merge pull request #2248 from matrix-org/erikj/state_fixup
...
Faster cache for get_joined_hosts
2017-06-07 14:01:06 +01:00
Erik Johnston
64ed74c01e
When pruning, delete from device_lists_outbound_last_success
2017-06-07 11:20:47 +01:00
Erik Johnston
1a81a1898e
Keep pruning background task
2017-06-07 11:16:56 +01:00
Erik Johnston
6ba21bf2b8
Comments
2017-06-07 11:08:36 +01:00
Erik Johnston
09e4bc0501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup
2017-06-07 11:05:23 +01:00
Erik Johnston
6e2a7ee1bc
Remove spurious log lines
2017-06-07 11:05:17 +01:00
Erik Johnston
65f0513a33
Split up device_lists_outbound_pokes table for faster updates.
2017-06-07 11:02:38 +01:00
Erik Johnston
6f83c4537c
Increase size of IP cache
2017-06-07 10:18:44 +01:00
Erik Johnston
1a01af079e
Handle profile updates in user directory
2017-06-01 15:39:51 +01:00
Erik Johnston
4d039aa2ca
Fix sqlite
2017-06-01 14:58:48 +01:00
Erik Johnston
21e255a8f1
Split the table in two
2017-06-01 14:50:46 +01:00
Erik Johnston
d5477c7afd
Tweak search query
2017-06-01 13:28:01 +01:00
Erik Johnston
02a6108235
Tweak search query
2017-06-01 13:16:40 +01:00
Erik Johnston
59dbb47065
Remove spurious inlineCallbacks
2017-06-01 11:41:29 +01:00
Erik Johnston
0fe6f3c521
Bug fixes and logging
...
- Check if room is public when a user joins before adding to user dir
- Fix typo of field name "content.join_rules" -> "content.join_rule"
2017-06-01 11:09:49 +01:00
Erik Johnston
036362ede6
Order by if they have profile info
2017-06-01 09:41:08 +01:00
Erik Johnston
a757dd4863
Use prefix matching
2017-06-01 09:40:37 +01:00