Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-24 21:49:22 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,307 Commits 4 Branches 609 Tags 60 MiB
be3adfc331
Commit Graph

2593 Commits

Author SHA1 Message Date
Neil Johnson
558f3d376a create index in background 2018-05-29 17:47:55 +01:00
Neil Johnson
c379acd4fd bump version 2018-05-29 17:47:28 +01:00
Adrian Tschira
4b9d0cde97 add remaining memoryview changes 2018-05-29 17:42:43 +02:00
Adrian Tschira
1afafb3497 use memoryview in py3 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-29 17:14:34 +02:00
Matthew Hodgson
a6c8f7c875 add pydoc 2018-05-29 01:09:55 +01:00
Matthew Hodgson
7a6df013cc merge develop 2018-05-29 00:25:22 +01:00
Amber Brown
a2eb5db4a0 update metrics to be in seconds 2018-05-28 19:10:27 +10:00
Amber Brown
754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Amber Brown
cc1349c06a
Merge pull request #3279 from NotAFile/py3-more-iteritems 
more six iteritems
 2018-05-24 14:23:13 -05:00
Adrian Tschira
dd068ca979 remaining isintance fixes 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-24 20:55:08 +02:00
Adrian Tschira
17a70cf6e9 Misc. py3 fixes 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-24 20:20:33 +02:00
Adrian Tschira
6c16a4ec1b more iteritems 2018-05-24 20:19:06 +02:00
Amber Brown
7ea07c7305
Merge pull request #3278 from NotAFile/py3-storage-base 
Py3 storage/_base.py
 2018-05-24 13:08:09 -05:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4 
replace some iteritems with six
 2018-05-24 13:04:07 -05:00
Amber Brown
36501068d8
Merge pull request #3247 from NotAFile/py3-misc 
Misc Python3 fixes
 2018-05-24 12:58:37 -05:00
Adrian Tschira
095292304f Py3 storage/_base.py 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-24 18:24:12 +02:00
David Baker
77a23e2e05 Merge remote-tracking branch 'origin/develop' into dbkr/unbind 2018-05-24 16:20:53 +01:00
David Baker
2c7866d664 Hit the 3pid unbind endpoint on deactivation 2018-05-23 14:38:56 +01:00
Amber Brown
53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Erik Johnston
a17e901f4d Remove unused string formatting param 2018-05-22 18:24:32 +01:00
Erik Johnston
5494c1d71e Don't support limitless pagination 
The pagination storage function supported not specifiying a limit on the
number of events returned. This was triggered when using the search or
context API with a limit of zero, which the storage function took to
mean not being limited.
 2018-05-22 18:15:21 +01:00
Neil Johnson
d8cb7225d2 daily user type phone home stats 2018-05-22 18:09:09 +01:00
Amber Brown
a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Richard van der Hoff
9ea219c514 Send users a server notice about consent 
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
 2018-05-22 11:54:51 +01:00
Amber Brown
fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Adrian Tschira
933bf2dd35 replace some iteritems with six 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-19 17:59:26 +02:00
Adrian Tschira
dcc235b47d use stand-in value if maxint is not available 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-19 17:35:44 +02:00
Neil Johnson
644aac5f73 Tighter filtering for user_daily_visits 2018-05-18 17:10:35 +01:00
Neil Johnson
ef466b3a13 fix psql compatability bug 2018-05-18 15:51:21 +01:00
Richard van der Hoff
ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred 
Fix error in handling receipts
 2018-05-18 11:23:11 +01:00
Erik Johnston
fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token 
Make purge_history operate on tokens
 2018-05-18 10:35:23 +01:00
Richard van der Hoff
8b1c856d81 Fix error in handling receipts 
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):

> TypeError: argument of type 'ObservableDeferred' is not iterable

fixes #3234
 2018-05-18 09:15:35 +01:00
Erik Johnston
f7906203f6
Merge pull request #3212 from matrix-org/erikj/epa_stream 
Use stream rather depth ordering for push actions
 2018-05-17 12:01:21 +01:00
Erik Johnston
680530cc7f Clarify comment 2018-05-16 11:47:29 +01:00
Erik Johnston
43e6e82c4d Comments 2018-05-16 11:13:31 +01:00
Neil Johnson
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics 
user visit data
 2018-05-16 10:09:24 +00:00
Erik Johnston
c945af8799 Move and rename variable 2018-05-16 10:52:06 +01:00
Neil Johnson
be11a02c4f remove empty line 2018-05-16 10:45:40 +01:00
Neil Johnson
31c2502ca8 style and further contraining query 2018-05-16 09:46:43 +01:00
Richard van der Hoff
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler 
ConsentResource to gather policy consent from users
 2018-05-16 07:19:18 +01:00
Neil Johnson
05ac15ae82 Limit query load of generate_user_daily_visits 
The aim is to keep track of when it was last called and only query from that point in time
 2018-05-15 17:01:33 +01:00
Erik Johnston
5f27ed75ad Make purge_history operate on tokens 
As we're soon going to change how topological_ordering works
 2018-05-15 16:23:50 +01:00
Erik Johnston
37dbee6490 Use events_to_purge table rather than token 2018-05-15 16:23:47 +01:00
Richard van der Hoff
47815edcfa ConsentResource to gather policy consent from users 
Hopefully there are enough comments and docs in this that it makes sense on its
own.
 2018-05-15 15:11:59 +01:00
Neil Johnson
f077e97914 instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day 2018-05-14 13:50:58 +01:00
David Baker
8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate 
Part user from rooms on account deactivate
 2018-05-14 11:31:48 +01:00
Neil Johnson
977765bde2 Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics 2018-05-14 09:31:42 +01:00
Erik Johnston
6406b70aeb Use stream rather depth ordering for push actions 
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
 2018-05-11 15:30:11 +01:00
David Baker
217bc53c98 Many docstrings 2018-05-10 12:20:40 +01:00
Erik Johnston
bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync 
Refactor sync APIs to reuse pagination API
 2018-05-09 16:16:56 +01:00
Erik Johnston
5adb75bcba
Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination 
Refactor get_recent_events_for_room return type
 2018-05-09 16:07:14 +01:00
Erik Johnston
7ce98804ff Fix up comment 2018-05-09 15:42:39 +01:00
Erik Johnston
cddf91c8b9 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change 2018-05-09 15:32:07 +01:00
Erik Johnston
9896dab8f6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination 2018-05-09 15:31:33 +01:00
Erik Johnston
75552d2148 Update comments 2018-05-09 15:15:38 +01:00
David Baker
46df23f581 Add the schema file 2018-05-09 15:07:54 +01:00
David Baker
7e8726b8fb Part deactivated users in the background 
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
 2018-05-09 14:54:28 +01:00
Erik Johnston
c0e08dc45b Remove unused code path from member change DB func 
The function is never called without a from_key, so we can remove all
the handling for that scenario.
 2018-05-09 14:31:32 +01:00
Erik Johnston
e2accd7f1d Refactor sync APIs to reuse pagination API 
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
 2018-05-09 13:43:39 +01:00
Erik Johnston
e5ab9cd24b Don't unnecessarily require token to be stream token 
This allows calling the `get_recent_event_ids_for_room` function in more
situations.
 2018-05-09 11:58:35 +01:00
Erik Johnston
c4af4c24ca Refactor get_recent_events_for_room return type 
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
 2018-05-09 11:55:34 +01:00
Erik Johnston
05e0a2462c Refactor pagination DB API to return concrete type 
This makes it easier to document what is being returned by the storage
functions and what some functions expect as arguments.
 2018-05-09 11:34:24 +01:00
Erik Johnston
7dd13415db Remove unused from_token param 2018-05-09 10:58:16 +01:00
Erik Johnston
27cf170558 Refactor recent events func to use pagination func 
This also removes a cache that is unlikely to ever get hit.
 2018-05-09 10:55:55 +01:00
Erik Johnston
23ec51c94c Fix up comments and make function private 2018-05-09 09:55:19 +01:00
Erik Johnston
696f532453 Reuse existing pagination code for context API 2018-05-08 16:20:19 +01:00
Erik Johnston
3e6d306e94 Parse tokens before calling DB function 2018-05-08 16:18:58 +01:00
Erik Johnston
274b8c6025 Only fetch required fields from database 2018-05-08 16:15:25 +01:00
Erik Johnston
06c0d0ed08 Split paginate_room_events storage function 2018-05-08 16:14:26 +01:00
Erik Johnston
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state 
Refactor event storage to prepare for changes in state calculations
 2018-05-03 10:39:20 +01:00
Erik Johnston
31c7c29d43 Fix up grammar 2018-05-03 10:38:58 +01:00
Neil Johnson
dd1a832419 remove user agent from data model, will just join on user_ips 2018-05-01 12:13:49 +01:00
Neil Johnson
d0857702e8 add inidexes based on usage 2018-05-01 12:12:57 +01:00
Matthew Hodgson
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups 
remove duplicates from groups tables
 2018-04-30 11:47:25 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1 2018-04-30 01:02:25 +01:00
Matthew Hodgson
adaf3ec87f fix missing import 2018-04-28 22:39:15 +01:00
Matthew Hodgson
006e18b6bb pep8 2018-04-28 22:32:24 +01:00
Matthew Hodgson
42c89c8215 make it work with sqlite 2018-04-28 22:27:30 +01:00
Adrian Tschira
d82b6ea9e6 Move more xrange to six 
plus a bonus next()

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-28 13:57:00 +02:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn 
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
 2018-04-27 12:55:51 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions 
Improve exception handling for background processes
 2018-04-27 11:47:49 +01:00
Erik Johnston
6e10eed28e Refactor event storage to not require state 
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.

We also add a check to see if the state groups of the old extremities
are the same as the new ones.
 2018-04-27 11:38:02 +01:00
Richard van der Hoff
605defb9e4 Add missing consumeErrors 
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
 2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17 Improve exception handling for background processes 
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
 2018-04-27 11:07:40 +01:00
Erik Johnston
7ec8e798b4 Fix media admin APIs 2018-04-26 11:31:22 +01:00
Neil Johnson
fb6015d0a6 pep8 2018-04-25 17:56:11 +01:00
Neil Johnson
617bf40924 Generate user daily stats 2018-04-25 17:37:29 +01:00
Erik Johnston
22881b3d69 Also fix reindexing of search 2018-04-25 15:32:04 +01:00
Erik Johnston
ba3166743c Fix quarantine media admin API 2018-04-25 15:11:18 +01:00
Matthew Hodgson
e3a373f002 remove duplicates from groups tables 
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
 2018-04-25 14:58:43 +01:00
Richard van der Hoff
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue 
Replace Queue with six.moves.queue
 2018-04-20 15:35:00 +01:00
Richard van der Hoff
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events 
Refactor store.have_events
 2018-04-20 10:26:12 +01:00
Richard van der Hoff
b1dfbc3c40 Refactor store.have_events 
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
 2018-04-20 10:25:56 +01:00
Richard van der Hoff
639480e14a Avoid creating events with huge numbers of prev_events 
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
 2018-04-16 18:41:37 +01:00
Adrian Tschira
878995e660 Replace Queue with six.moves.queue 
and a six.range change which I missed the last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-16 00:46:21 +02:00
Erik Johnston
415aeefd89 Format docstring 2018-04-12 12:07:09 +01:00
Erik Johnston
23a7f9d7f4 Doc we raise on unknown event 2018-04-12 11:20:51 +01:00
Erik Johnston
4dae4a97ed Track last processed event received_ts 2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5 Track where event stream processing have gotten up to 2018-04-11 12:13:40 +01:00