Amber Brown
872cf43516
Merge pull request #3303 from NotAFile/py3-memoryview
...
use memoryview in py3
2018-05-30 12:51:42 +10:00
Amber Brown
debff7ae09
Merge pull request #3281 from NotAFile/py3-six-isinstance
...
remaining isintance fixes
2018-05-30 12:44:46 +10:00
Richard van der Hoff
4a9cbdbc15
Exempt AS-registered users from doing gdpr
2018-05-29 19:54:32 +01:00
Neil Johnson
ab0ef31dc7
create users index on creation_ts
2018-05-29 17:51:08 +01:00
Neil Johnson
558f3d376a
create index in background
2018-05-29 17:47:55 +01:00
Neil Johnson
c379acd4fd
bump version
2018-05-29 17:47:28 +01:00
Adrian Tschira
4b9d0cde97
add remaining memoryview changes
2018-05-29 17:42:43 +02:00
Adrian Tschira
1afafb3497
use memoryview in py3
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-29 17:14:34 +02:00
Matthew Hodgson
a6c8f7c875
add pydoc
2018-05-29 01:09:55 +01:00
Matthew Hodgson
7a6df013cc
merge develop
2018-05-29 00:25:22 +01:00
Amber Brown
a2eb5db4a0
update metrics to be in seconds
2018-05-28 19:10:27 +10:00
Amber Brown
754826a830
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-28 18:57:23 +10:00
Amber Brown
cc1349c06a
Merge pull request #3279 from NotAFile/py3-more-iteritems
...
more six iteritems
2018-05-24 14:23:13 -05:00
Adrian Tschira
dd068ca979
remaining isintance fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:55:08 +02:00
Adrian Tschira
17a70cf6e9
Misc. py3 fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
Adrian Tschira
6c16a4ec1b
more iteritems
2018-05-24 20:19:06 +02:00
Amber Brown
7ea07c7305
Merge pull request #3278 from NotAFile/py3-storage-base
...
Py3 storage/_base.py
2018-05-24 13:08:09 -05:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
...
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown
36501068d8
Merge pull request #3247 from NotAFile/py3-misc
...
Misc Python3 fixes
2018-05-24 12:58:37 -05:00
Adrian Tschira
095292304f
Py3 storage/_base.py
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 18:24:12 +02:00
David Baker
77a23e2e05
Merge remote-tracking branch 'origin/develop' into dbkr/unbind
2018-05-24 16:20:53 +01:00
David Baker
2c7866d664
Hit the 3pid unbind endpoint on deactivation
2018-05-23 14:38:56 +01:00
Amber Brown
53cc2cde1f
cleanup
2018-05-22 17:32:57 -05:00
Erik Johnston
a17e901f4d
Remove unused string formatting param
2018-05-22 18:24:32 +01:00
Erik Johnston
5494c1d71e
Don't support limitless pagination
...
The pagination storage function supported not specifiying a limit on the
number of events returned. This was triggered when using the search or
context API with a limit of zero, which the storage function took to
mean not being limited.
2018-05-22 18:15:21 +01:00
Neil Johnson
d8cb7225d2
daily user type phone home stats
2018-05-22 18:09:09 +01:00
Amber Brown
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Amber Brown
fcc525b0b7
rest of the changes
2018-05-21 19:48:57 -05:00
Adrian Tschira
933bf2dd35
replace some iteritems with six
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira
dcc235b47d
use stand-in value if maxint is not available
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:44 +02:00
Neil Johnson
644aac5f73
Tighter filtering for user_daily_visits
2018-05-18 17:10:35 +01:00
Neil Johnson
ef466b3a13
fix psql compatability bug
2018-05-18 15:51:21 +01:00
Richard van der Hoff
ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred
...
Fix error in handling receipts
2018-05-18 11:23:11 +01:00
Erik Johnston
fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token
...
Make purge_history operate on tokens
2018-05-18 10:35:23 +01:00
Richard van der Hoff
8b1c856d81
Fix error in handling receipts
...
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):
> TypeError: argument of type 'ObservableDeferred' is not iterable
fixes #3234
2018-05-18 09:15:35 +01:00
Erik Johnston
f7906203f6
Merge pull request #3212 from matrix-org/erikj/epa_stream
...
Use stream rather depth ordering for push actions
2018-05-17 12:01:21 +01:00
Erik Johnston
680530cc7f
Clarify comment
2018-05-16 11:47:29 +01:00
Erik Johnston
43e6e82c4d
Comments
2018-05-16 11:13:31 +01:00
Neil Johnson
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
Erik Johnston
c945af8799
Move and rename variable
2018-05-16 10:52:06 +01:00
Neil Johnson
be11a02c4f
remove empty line
2018-05-16 10:45:40 +01:00
Neil Johnson
31c2502ca8
style and further contraining query
2018-05-16 09:46:43 +01:00
Richard van der Hoff
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Neil Johnson
05ac15ae82
Limit query load of generate_user_daily_visits
...
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
Erik Johnston
5f27ed75ad
Make purge_history operate on tokens
...
As we're soon going to change how topological_ordering works
2018-05-15 16:23:50 +01:00
Erik Johnston
37dbee6490
Use events_to_purge table rather than token
2018-05-15 16:23:47 +01:00
Richard van der Hoff
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson
f077e97914
instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day
2018-05-14 13:50:58 +01:00
David Baker
8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate
...
Part user from rooms on account deactivate
2018-05-14 11:31:48 +01:00
Neil Johnson
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
Erik Johnston
6406b70aeb
Use stream rather depth ordering for push actions
...
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
David Baker
217bc53c98
Many docstrings
2018-05-10 12:20:40 +01:00
Erik Johnston
bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync
...
Refactor sync APIs to reuse pagination API
2018-05-09 16:16:56 +01:00
Erik Johnston
5adb75bcba
Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination
...
Refactor get_recent_events_for_room return type
2018-05-09 16:07:14 +01:00
Erik Johnston
7ce98804ff
Fix up comment
2018-05-09 15:42:39 +01:00
Erik Johnston
cddf91c8b9
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change
2018-05-09 15:32:07 +01:00
Erik Johnston
9896dab8f6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination
2018-05-09 15:31:33 +01:00
Erik Johnston
75552d2148
Update comments
2018-05-09 15:15:38 +01:00
David Baker
46df23f581
Add the schema file
2018-05-09 15:07:54 +01:00
David Baker
7e8726b8fb
Part deactivated users in the background
...
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
2018-05-09 14:54:28 +01:00
Erik Johnston
c0e08dc45b
Remove unused code path from member change DB func
...
The function is never called without a from_key, so we can remove all
the handling for that scenario.
2018-05-09 14:31:32 +01:00
Erik Johnston
e2accd7f1d
Refactor sync APIs to reuse pagination API
...
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
2018-05-09 13:43:39 +01:00
Erik Johnston
e5ab9cd24b
Don't unnecessarily require token to be stream token
...
This allows calling the `get_recent_event_ids_for_room` function in more
situations.
2018-05-09 11:58:35 +01:00
Erik Johnston
c4af4c24ca
Refactor get_recent_events_for_room return type
...
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
2018-05-09 11:55:34 +01:00
Erik Johnston
05e0a2462c
Refactor pagination DB API to return concrete type
...
This makes it easier to document what is being returned by the storage
functions and what some functions expect as arguments.
2018-05-09 11:34:24 +01:00
Erik Johnston
7dd13415db
Remove unused from_token param
2018-05-09 10:58:16 +01:00
Erik Johnston
27cf170558
Refactor recent events func to use pagination func
...
This also removes a cache that is unlikely to ever get hit.
2018-05-09 10:55:55 +01:00
Erik Johnston
23ec51c94c
Fix up comments and make function private
2018-05-09 09:55:19 +01:00
Erik Johnston
696f532453
Reuse existing pagination code for context API
2018-05-08 16:20:19 +01:00
Erik Johnston
3e6d306e94
Parse tokens before calling DB function
2018-05-08 16:18:58 +01:00
Erik Johnston
274b8c6025
Only fetch required fields from database
2018-05-08 16:15:25 +01:00
Erik Johnston
06c0d0ed08
Split paginate_room_events storage function
2018-05-08 16:14:26 +01:00
Erik Johnston
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
...
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
Erik Johnston
31c7c29d43
Fix up grammar
2018-05-03 10:38:58 +01:00
Neil Johnson
dd1a832419
remove user agent from data model, will just join on user_ips
2018-05-01 12:13:49 +01:00
Neil Johnson
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
Matthew Hodgson
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
Matthew Hodgson
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
Matthew Hodgson
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
Matthew Hodgson
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
Adrian Tschira
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Richard van der Hoff
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
Erik Johnston
6e10eed28e
Refactor event storage to not require state
...
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.
We also add a check to see if the state groups of the old extremities
are the same as the new ones.
2018-04-27 11:38:02 +01:00
Richard van der Hoff
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston
7ec8e798b4
Fix media admin APIs
2018-04-26 11:31:22 +01:00
Neil Johnson
fb6015d0a6
pep8
2018-04-25 17:56:11 +01:00
Neil Johnson
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
Erik Johnston
22881b3d69
Also fix reindexing of search
2018-04-25 15:32:04 +01:00
Erik Johnston
ba3166743c
Fix quarantine media admin API
2018-04-25 15:11:18 +01:00
Matthew Hodgson
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
Richard van der Hoff
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
...
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
Richard van der Hoff
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
Richard van der Hoff
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
Richard van der Hoff
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira
878995e660
Replace Queue with six.moves.queue
...
and a six.range change which I missed the last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
Erik Johnston
415aeefd89
Format docstring
2018-04-12 12:07:09 +01:00
Erik Johnston
23a7f9d7f4
Doc we raise on unknown event
2018-04-12 11:20:51 +01:00
Erik Johnston
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
Richard van der Hoff
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
Neil Johnson
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
Richard van der Hoff
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
Vincent Breitmoser
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
Neil Johnson
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
Neil Johnson
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
Neil Johnson
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
Neil Johnson
64bc2162ef
Fix psycopg2 interpolation
2018-04-09 17:50:36 +01:00
Luke Barnard
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
Luke Barnard
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
Luke Barnard
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
David Baker
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
David Baker
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
Richard van der Hoff
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
Luke Barnard
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
Luke Barnard
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
Luke Barnard
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
Neil Johnson
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
Luke Barnard
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
Richard van der Hoff
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
Richard van der Hoff
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
Richard van der Hoff
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
Neil Johnson
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
Neil Johnson
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
Neil Johnson
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
Neil Johnson
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
David Baker
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
David Baker
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
Neil Johnson
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
Neil Johnson
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
David Baker
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
David Baker
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
David Baker
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
Neil Johnson
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
Neil Johnson
0f890f477e
No need to cast in count_daily_users
2018-03-28 11:49:57 +01:00
Neil Johnson
545001b9e4
Fix search_user_dir multiple sqlite versions do different things
2018-03-28 11:19:45 +01:00
Neil Johnson
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
Neil Johnson
9187e0762f
count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home
2018-03-28 10:02:32 +01:00
Erik Johnston
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
...
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
Erik Johnston
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
...
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
Erik Johnston
800cfd5774
Comment
2018-03-27 13:30:39 +01:00
Erik Johnston
152c2ac19e
Fix indent
2018-03-27 13:13:46 +01:00
Erik Johnston
e70287cff3
Comment
2018-03-27 13:13:38 +01:00
Erik Johnston
3f49e131d9
Add counter metrics for calculating state delta
...
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
Erik Johnston
fecb45e0c3
Remove last usage of ujson
2018-03-26 13:32:29 +01:00
Erik Johnston
8d6dc106d1
Don't use _cursor_to_dict in find_next_generated_user_id_localpart
2018-03-26 12:02:44 +01:00
Erik Johnston
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
Erik Johnston
bf49d2dca8
Replace some ujson with simplejson to make it work
2018-03-16 00:55:44 +00:00
Erik Johnston
a8ce159be4
Replace some ujson with simplejson to make it work
2018-03-16 00:27:09 +00:00
Erik Johnston
5b631ff41a
Remove wrong comment
2018-03-16 00:07:08 +00:00
Erik Johnston
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
Matthew Hodgson
056a6df546
Merge branch 'develop' into matthew/filter_members
2018-03-14 15:38:05 +00:00
Erik Johnston
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
...
Split up ProfileStore
2018-03-14 09:41:06 +00:00
Erik Johnston
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
...
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
Matthew Hodgson
9f77001e27
pep8
2018-03-14 00:07:47 +00:00
Matthew Hodgson
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) ( #2990 )
...
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
Matthew Hodgson
ccca02846d
make it work
2018-03-13 22:31:41 +00:00
Matthew Hodgson
f0f9a0605b
remove comment now #2969 is fixed
2018-03-13 22:12:15 +00:00
Matthew Hodgson
12350e3f9a
merge proper fix to bug 2969
2018-03-13 22:11:58 +00:00
Matthew Hodgson
afbf4d3dcc
typoe
2018-03-13 19:48:04 +00:00
Matthew Hodgson
865377a70d
disable optimisation for searching for state groups
...
when type filter includes wildcards on state_key
2018-03-13 19:46:04 +00:00
Erik Johnston
a08726fc42
Add is_blocked to worker store
2018-03-13 18:28:44 +00:00
Matthew Hodgson
b2aba9e430
build where_clause sanely
2018-03-13 18:13:44 +00:00
Matthew Hodgson
52f7e23c72
PR feedbackz
2018-03-13 18:07:55 +00:00
Matthew Hodgson
1b1c137771
fix bug #2926
2018-03-13 17:52:52 +00:00
Erik Johnston
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
Richard van der Hoff
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
Matthew Hodgson
fdedcd1f4d
correctly handle None state_keys
...
and fix include_other_types thinko
2018-03-12 01:39:06 +00:00
Matthew Hodgson
97c0496cfa
fix sqlite where clause
2018-03-12 00:27:06 +00:00
Matthew Hodgson
8713365265
typos
2018-03-11 20:10:25 +00:00
Matthew Hodgson
9b334b3f97
WIP experiment in lazyloading room members
2018-03-11 20:01:41 +00:00
Erik Johnston
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
Richard van der Hoff
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
Erik Johnston
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
Erik Johnston
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
Erik Johnston
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
Erik Johnston
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
Erik Johnston
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
Richard van der Hoff
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
Richard van der Hoff
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
Richard van der Hoff
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Richard van der Hoff
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
Erik Johnston
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
Erik Johnston
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
Erik Johnston
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
Erik Johnston
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
Erik Johnston
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
Erik Johnston
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
Erik Johnston
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
Erik Johnston
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
Erik Johnston
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
Erik Johnston
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
Erik Johnston
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
Erik Johnston
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
Erik Johnston
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
Erik Johnston
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
Erik Johnston
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
Erik Johnston
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
Erik Johnston
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
Erik Johnston
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
Erik Johnston
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
Erik Johnston
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
Erik Johnston
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
Erik Johnston
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
Erik Johnston
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
Erik Johnston
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
Erik Johnston
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
Erik Johnston
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
Erik Johnston
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
Erik Johnston
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
Erik Johnston
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
Erik Johnston
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
Erik Johnston
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
Erik Johnston
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
Erik Johnston
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
Erik Johnston
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
Erik Johnston
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
Erik Johnston
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
Erik Johnston
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
Erik Johnston
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
Erik Johnston
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
Erik Johnston
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
Erik Johnston
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
Erik Johnston
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
Richard van der Hoff
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
Erik Johnston
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
Erik Johnston
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
Erik Johnston
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
Erik Johnston
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
Erik Johnston
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
Erik Johnston
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
Erik Johnston
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
Erik Johnston
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
Richard van der Hoff
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
Richard van der Hoff
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
Richard van der Hoff
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
Richard van der Hoff
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
Richard van der Hoff
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
Richard van der Hoff
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
Richard van der Hoff
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
Richard van der Hoff
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
Richard van der Hoff
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
Richard van der Hoff
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
Travis Ralston
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
Travis Ralston
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
Richard van der Hoff
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
Travis Ralston
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
Richard van der Hoff
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
Richard van der Hoff
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
Richard van der Hoff
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
Travis Ralston
a94d9b6b82
Appease the linter
...
These are ids anyways, not mxc uris.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
Travis Ralston
5552ed9a7f
Add an admin route to get all the media in a room
...
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00
Richard van der Hoff
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
...
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
Richard van der Hoff
1224612a79
Log room when doing state resolution
...
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
Erik Johnston
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed
...
Keep track of last access time for local media
2018-01-17 16:39:02 +00:00
Matthew Hodgson
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql
...
fix SQL when searching all users
2018-01-17 16:27:54 +00:00
Matthew Hodgson
2d9ab533f9
fix SQL when searching all users
2018-01-17 15:58:52 +00:00
Richard van der Hoff
390093d45e
Split resolve_events into two functions
...
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
Erik Johnston
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
Erik Johnston
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
Richard van der Hoff
3d12d97415
Track DB scheduling delay per-request
...
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
Richard van der Hoff
8615f19d20
rework runInteraction in terms of runConnection
...
... so that we can share the code
2018-01-16 17:08:29 +00:00
Richard van der Hoff
64ddec1bc0
Fix a logcontext leak in persist_events
...
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.
It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.
(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
hera
174eacc8ba
oops
2018-01-09 18:14:32 +00:00
Matthew Hodgson
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
Matthew Hodgson
e79db0a673
switch back from GIST to GIN indexes
2018-01-09 16:37:48 +00:00
Matthew Hodgson
e365ad329f
oops, tweak work_mem when actually storing
2018-01-09 16:30:30 +00:00
Matthew Hodgson
19f9227643
avoid 80s GIN inserts by tweaking work_mem
...
see https://github.com/matrix-org/synapse/issues/2753 for details
2018-01-09 16:25:04 +00:00
Erik Johnston
b30cd5b107
Remove dead code related to default thumbnails
2018-01-09 14:38:33 +00:00
Matthew Hodgson
cdc2cb5d11
fix StoreError syntax
2017-12-05 11:09:31 +00:00
Matthew Hodgson
74e0cc74ce
fix pep8 and tests
2017-12-04 15:11:38 +00:00
Matthew Hodgson
1bd40ca73e
switch to a simpler 'search_all_users' button as per review feedback
2017-12-04 14:58:39 +00:00
Matthew Hodgson
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
Matthew Hodgson
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
Matthew Hodgson
4b1fceb913
fix alternation operator for FTS4 - how did this ever work!?
2017-11-30 01:34:03 +00:00
Matthew Hodgson
a4bb133b68
fix thinkos galore
2017-11-30 01:17:15 +00:00
Matthew Hodgson
3241c7aac3
untested WIP but might actually work
2017-11-29 18:27:05 +00:00
Matthew Hodgson
47d99a20d5
Add user_directory_include_pattern config param to expand search results to additional users
...
Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive.
It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users...
...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too.
Committing it here as a WIP
2017-11-29 16:46:45 +00:00
Richard van der Hoff
2c6d63922a
Remove pushers when deleting access tokens
...
Whenever an access token is invalidated, we should remove the associated
pushers.
2017-11-29 16:44:35 +00:00
Richard van der Hoff
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error
...
Fix error on sqlite 3.7
2017-11-27 12:25:48 +00:00
Richard van der Hoff
63ccaa5873
Avoid retrying forever on IntegrityError
2017-11-27 12:00:07 +00:00
Richard van der Hoff
6b48b3e277
fix sql fails
2017-11-22 18:06:24 +00:00
Richard van der Hoff
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
Richard van der Hoff
0b4aa2dc21
Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert
...
Avoid locking account_data tables for upserts
2017-11-21 13:39:14 +00:00
Richard van der Hoff
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
Richard van der Hoff
c46139a17e
Avoid locking account_data tables for upserts
2017-11-16 18:08:01 +00:00
Richard van der Hoff
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
Richard van der Hoff
77a1227870
Fix broken ref to IntegrityError
2017-11-16 16:03:38 +00:00
Richard van der Hoff
7ab2b69e18
Avoid locking pushers table on upsert
...
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the
table.
2017-11-16 15:32:01 +00:00
Richard van der Hoff
10aaa1bc15
_simple_upsert: retry on IntegrityError
...
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
2017-11-16 15:30:15 +00:00
Richard van der Hoff
cdc9e50a5d
Cleanup in _simple_upsert_txn
...
Bail out early to reduce indentation
2017-11-16 15:29:10 +00:00
Richard van der Hoff
f959c01600
Merge pull request #2661 from matrix-org/rav/statereadstore
...
Pull out bits of StateStore to a mixin
2017-11-15 17:23:01 +00:00
Richard van der Hoff
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs
...
Remove preserve_context_over_{fn, deferred}
2017-11-15 11:13:53 +00:00
Richard van der Hoff
35a4b63240
Pull out bits of StateStore to a mixin
...
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.
fixes https://github.com/matrix-org/synapse/issues/2655 .
2017-11-14 11:43:58 +00:00
Richard van der Hoff
4dd1bfa8c1
Revert "Revert "move _state_group_cache to statestore""
...
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.
This reverts commit ab335edb02
2017-11-14 11:43:58 +00:00
Richard van der Hoff
7e6fa29cb5
Remove preserve_context_over_{fn, deferred}
...
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
Richard van der Hoff
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston
ab335edb02
Revert "move _state_group_cache to statestore"
...
This reverts commit f5cf3638e9
2017-11-13 10:05:33 +00:00
Erik Johnston
bfbf1e1f1a
Up cache size of get_global_account_data_by_type_for_user
2017-11-13 09:52:11 +00:00
Erik Johnston
2dce6b15c3
Fix typo
2017-11-09 15:56:16 +00:00
Richard van der Hoff
618c7b816a
Merge pull request #2656 from matrix-org/rav/fix_deactivate
...
Fix 'NoneType' not iterable in /deactivate
2017-11-09 15:20:35 +00:00
Richard van der Hoff
889102315e
Fix 'NoneType' not iterable in /deactivate
...
make sure we actually return a value from user_delete_access_tokens
2017-11-09 15:15:33 +00:00
Erik Johnston
e8814410ef
Have an explicit API to update room config
2017-11-08 16:13:27 +00:00
Erik Johnston
94ff2cda73
Revert "Modify group room association API to allow modification of is_public"
2017-11-08 15:43:34 +00:00
Erik Johnston
167eb01d83
Merge pull request #2637 from spantaleev/avoid-noop-media-deletes
...
Avoid no-op media deletes
2017-11-08 11:53:27 +00:00
Richard van der Hoff
02a9a93bde
Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res
...
Fix bug in state group storage
2017-11-08 09:22:13 +00:00
Matthew Hodgson
228ccf1fe3
Merge pull request #2643 from matrix-org/matthew/user_dir_typos
...
Fix various embarrassing typos around user_directory and add some doc.
2017-11-07 17:31:11 +00:00
Richard van der Hoff
f5cf3638e9
move _state_group_cache to statestore
...
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
Luke Barnard
38b265cb51
Remember to pick is_admin out of the db
2017-11-07 11:24:04 +00:00
Matthew Hodgson
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
Matthew Hodgson
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
Matthew Hodgson
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
Matthew Hodgson
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
Matthew Hodgson
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
Slavi Pantaleev
805196fbeb
Avoid no-op media deletes
...
If there are no media entries to delete,
avoid creating transactions, prepared statements
and unnecessary log entries.
Signed-off-by: Slavi Pantaleev <slavi@devture.com>
2017-11-04 09:50:15 +02:00
Richard van der Hoff
bc8a5c0330
Notify auth providers on logout
...
Provide a hook by which auth providers can be notified of logouts.
2017-11-01 16:51:51 +00:00
Matthew Hodgson
275826f234
Merge pull request #2617 from matrix-org/matthew/auto-displayname
...
automatically set default displayname on register
2017-11-01 16:21:16 +00:00
David Baker
fbbacb284e
Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens
...
Remove the last vestiges of refresh_tokens
2017-11-01 15:57:35 +00:00
Matthew Hodgson
9f7a555b4e
switch to setting default displayname in the storage layer
...
to avoid clobbering guest user displaynames on registration
2017-11-01 15:51:30 +00:00
Luke Barnard
99354b430e
Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public
...
Modify group room association API to allow modification of is_public
2017-11-01 11:08:36 +00:00
Luke Barnard
318a249c8b
Leave is_public as required argument of update_room_group_association
2017-11-01 09:36:01 +00:00
Richard van der Hoff
356bcafc44
Remove the last vestiges of refresh_tokens
2017-10-31 20:35:58 +00:00
Luke Barnard
20fe347906
Modify group room association API to allow modification of is_public
...
also includes renamings to make things more consistent.
2017-10-31 17:04:28 +00:00
Richard van der Hoff
1650eb5847
DB schema interface for password auth providers
...
Provide an interface by which password auth providers can register db schema
files to be run at startup
2017-10-31 14:01:53 +00:00
Erik Johnston
4ab8abbc2b
Merge branch 'erikj/attestation_local_fix' of github.com:matrix-org/synapse into develop
2017-10-27 15:07:08 +01:00
Luke Barnard
5451cc7792
Request is_public from database
2017-10-27 11:27:43 +01:00
Luke Barnard
6362298fa5
Create groups with is_public = True
2017-10-27 11:04:20 +01:00
Erik Johnston
82d8c1bacb
Fixup
2017-10-27 10:30:21 +01:00
Erik Johnston
195abfe7a5
Remove incorrect attestations
2017-10-27 09:58:13 +01:00
Luke Barnard
007cd48af6
Recreate groups table instead of adding column
...
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
Luke Barnard
713e60b9b6
Awful hack to get default true
2017-10-26 17:38:14 +01:00
Luke Barnard
cfa4e658e0
Bump schema version to 46
2017-10-26 17:23:49 +01:00
Luke Barnard
9b2feef9eb
Add is_public to groups table to allow for private groups
...
Prevent group API access to non-members for private groups
Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff
2e9f5ea31a
Fix logcontext handling for persist_events
...
* don't use preserve_context_over_deferred, which is known broken.
* remove a redundant preserve_fn.
* add/improve some comments
2017-10-17 10:59:30 +01:00
Luke Barnard
2c5972f87f
Implement GET /groups/$groupId/invited_users
2017-10-16 15:31:11 +01:00
Erik Johnston
818b08d0e4
peeeeeeeeep8888888888888888888888888888
2017-10-11 15:54:00 +01:00
Erik Johnston
ea18996f54
Fix group stream replication
...
The stream update functions expect the storage function to return a list
of tuples.
2017-10-11 15:44:39 +01:00
Erik Johnston
c2c47550f9
Fix schema delta versions
2017-10-11 13:23:15 +01:00
Erik Johnston
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
Erik Johnston
e8496efe84
Fix up comment
2017-10-09 15:17:34 +01:00
Erik Johnston
11d62f43c9
Invalidate cache
2017-10-03 14:12:28 +01:00
Erik Johnston
e4ab96021e
Update comments
2017-10-03 14:10:41 +01:00
Erik Johnston
30848c0fcd
Ignore incoming events for rooms that we have left
...
When synapse receives an event for a room its not in over federation, it
double checks with the remote server to see if it is in fact in the
room. This is done so that if the server has forgotten about the room
(usually as a result of the database being dropped) it can recover from
it.
However, in the presence of state resets in large rooms, this can cause
a lot of work for servers that have legitimately left. As a hacky
solution that supports both cases we drop incoming events for rooms that
we have explicitly left.
This means that we no longer support the case of servers having
forgotten that they've rejoined a room, but that is sufficiently rare
that we're not going to support it for now.
2017-10-03 11:18:21 +01:00
David Baker
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
Erik Johnston
768f00dedb
Up the limits on number of url cache entries to delete at one time
2017-09-28 14:27:27 +01:00
Erik Johnston
4dc07e93a8
Add old indices
2017-09-28 14:10:33 +01:00
Erik Johnston
93247a424a
Only pull out local media that were for url cache
2017-09-28 13:48:14 +01:00
Erik Johnston
7a44c01d89
Fix typo
2017-09-28 12:46:04 +01:00
Erik Johnston
ae79764fe5
Change expires column to expires_ts
2017-09-28 12:37:53 +01:00
Erik Johnston
77f1d24de3
More brackets
2017-09-28 12:23:15 +01:00
Erik Johnston
9ccb4226ba
Delete expired url cache data
2017-09-28 12:18:06 +01:00
Erik Johnston
17b8e2bd02
Add remove room API
2017-09-26 15:52:41 +01:00
Erik Johnston
a8e2a3df32
Add unique index to group_rooms table
2017-09-26 15:39:21 +01:00
Erik Johnston
e1dec2f1a7
Remove user from group summary when the leave the group
2017-09-21 16:09:57 +01:00
Erik Johnston
197d82dc07
Correctly return next token
2017-09-20 11:12:11 +01:00
Erik Johnston
069ae2df12
Fix initial sync
2017-09-20 10:52:12 +01:00
Richard van der Hoff
e76d1135dd
Invalidate signing key cache when we gat an update
...
This might make the cache slightly more efficient.
2017-09-20 01:32:42 +01:00
Erik Johnston
4a9b1cf253
Add user profiles to summary from group server
2017-08-25 16:23:58 +01:00
Erik Johnston
27ebc5c8f2
Add remote profile cache
2017-08-25 11:25:47 +01:00
Erik Johnston
97c544f91f
Add _simple_update
2017-08-25 11:11:37 +01:00
Erik Johnston
2800983f3e
Merge pull request #2410 from matrix-org/erikj/groups_publicise
...
Add ability to publicise group membership
2017-08-21 16:51:56 +01:00
Erik Johnston
8b50fe5330
Use BOOLEAN rather than TEXT type
2017-08-21 16:37:29 +01:00
Erik Johnston
175a01f56c
Groups: Fix mising json.load in initial sync
2017-08-21 14:45:56 +01:00
Erik Johnston
ef8e578677
Add bulk group publicised lookup API
2017-08-09 13:36:22 +01:00
Erik Johnston
b880ff190a
Allow update group publicity
2017-08-08 14:19:41 +01:00
Erik Johnston
05e21285aa
Store whether the user wants to publicise their membership of a group
2017-08-08 13:01:46 +01:00
Erik Johnston
966a70f1fa
Update comment
2017-07-24 17:49:39 +01:00
Erik Johnston
629cdfb124
Use join rather than joined, etc.
2017-07-24 14:54:05 +01:00
Erik Johnston
ed666d3969
Fix all the typos
2017-07-24 14:05:09 +01:00
Erik Johnston
b76ef6ccb8
Include users membership in group in summary API
2017-07-24 13:55:39 +01:00
Erik Johnston
851aeae7c7
Check users/rooms are in group before adding to summary
2017-07-24 13:40:56 +01:00
Erik Johnston
96917d5552
Merge pull request #2378 from matrix-org/erikj/group_sync_support
...
Add groups to sync stream
2017-07-21 11:05:39 +01:00
Erik Johnston
0401604222
Merge pull request #2377 from matrix-org/erikj/group_profile_update
...
Add update group profile API
2017-07-20 17:53:39 +01:00
Erik Johnston
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
Erik Johnston
c544188ee3
Add groups to sync stream
2017-07-20 16:36:42 +01:00
Erik Johnston
8209b5f033
Fix a storage desc
2017-07-20 16:22:22 +01:00
Erik Johnston
57826d645b
Fix typo
2017-07-20 13:15:22 +01:00
Erik Johnston
6f443a74cf
Add update group profile API
2017-07-20 09:46:33 +01:00
Erik Johnston
94ecd871a0
Fix typos
2017-07-18 16:38:54 +01:00
Erik Johnston
3e703eb04e
Comment
2017-07-18 10:17:25 +01:00
Erik Johnston
508460f240
Remove sync stuff
2017-07-18 09:55:46 +01:00
Erik Johnston
4540730111
Remove unused tables
2017-07-18 09:38:15 +01:00
Erik Johnston
e96ee95a7e
Remove sync stuff
2017-07-18 09:38:08 +01:00
Erik Johnston
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
Erik Johnston
85fda57208
Add DEFAULT_ROLE_ID
2017-07-14 14:03:54 +01:00
Erik Johnston
4b203bdba5
Correctly increment orders
2017-07-14 14:02:31 +01:00
Erik Johnston
3b0470dba5
Remove unused functions
2017-07-13 13:54:01 +01:00
Erik Johnston
8575e3160f
Comments
2017-07-13 13:52:41 +01:00
Erik Johnston
26451a09eb
Comments
2017-07-12 14:47:18 +01:00
Erik Johnston
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
Erik Johnston
6d586dc05c
Comment
2017-07-12 09:58:37 +01:00
Erik Johnston
410b4e14a1
Move comment
2017-07-11 15:44:18 +01:00
Erik Johnston
e52c391cd4
Rename column to attestation_json
2017-07-11 14:25:46 +01:00
Erik Johnston
0aac30d53b
Comments
2017-07-11 14:23:50 +01:00
Erik Johnston
83936293eb
Comments
2017-07-11 11:42:25 +01:00
Erik Johnston
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
Erik Johnston
f502b0dea1
Perf: Don't filter events for push
...
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
Erik Johnston
ed9a7f5436
Merge pull request #2309 from matrix-org/erikj/user_ip_repl
...
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston
b5e8d529e6
Define CACHE_SIZE_FACTOR once
2017-07-04 09:56:44 +01:00
Erik Johnston
6ff14ddd2e
Make into list
2017-06-29 15:47:37 +01:00
Erik Johnston
5946aa0877
Prefill forward extrems and event to state groups
2017-06-29 15:38:48 +01:00
Erik Johnston
8c23221666
Fix up
2017-06-27 15:53:45 +01:00
Erik Johnston
a0a561ae85
Fix up client ips to read from pending data
2017-06-27 14:46:12 +01:00
Erik Johnston
ed3d0170d9
Batch upsert user ips
2017-06-27 13:37:04 +01:00
Erik Johnston
8abdd7b553
Fix up indices for users_who_share_rooms
2017-06-26 14:01:30 +01:00
Erik Johnston
7fe8ed1787
Store URL cache preview downloads seperately
...
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
Erik Johnston
e0004aa28a
Add desc
2017-06-22 10:03:48 +01:00
Erik Johnston
b668112320
Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar
...
Don't work out users who share room with appservice users
2017-06-21 14:50:24 +01:00
Erik Johnston
dae9a00a28
Initialise exclusive_user_regex
2017-06-21 14:19:33 +01:00
Krombel
4202fba82a
Merge branch 'develop' into avoid_duplicate_filters
2017-06-21 14:48:21 +02:00
Krombel
812c030e87
replaced json.dumps with encode_canonical_json
2017-06-21 14:48:12 +02:00
Erik Johnston
385dcb7c60
Handle thumbnail urls
2017-06-19 17:48:28 +01:00
Erik Johnston
b8b936a6ea
Add API to quarantine media
2017-06-19 17:39:21 +01:00
Erik Johnston
d7fe6b356c
Add shutdown room API
2017-06-19 12:37:27 +01:00
Erik Johnston
dfeca6cf40
Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
...
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
Erik Johnston
d8f47d2efa
Merge pull request #2280 from matrix-org/erikj/share_room_user_dir
...
Include users who you share a room with in user directory
2017-06-16 11:06:00 +01:00
Erik Johnston
052c5d19d5
Merge pull request #2281 from matrix-org/erikj/phone_home_stats
...
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston
a9d6fa8b2b
Include users who share room with requester in user directory
2017-06-15 10:17:21 +01:00
Erik Johnston
4564b05483
Implement updating users who share rooms on the fly
2017-06-15 10:17:17 +01:00
Erik Johnston
72613bc379
Implement initial population of users who share rooms table
2017-06-15 09:59:04 +01:00
Erik Johnston
ebcd55d641
Add DB schema for tracking users who share rooms
2017-06-15 09:45:48 +01:00
Erik Johnston
4b461a6931
Add some more stats
2017-06-15 09:39:39 +01:00
Erik Johnston
617304b2cf
Fix phone home stats
2017-06-14 19:47:15 +01:00
Erik Johnston
d9fd937e39
Fix user directory insertion due to missing room_id
2017-06-13 11:50:24 +01:00
Erik Johnston
505e7e8b9d
Fix up sql
2017-06-13 11:19:18 +01:00
Erik Johnston
6fd7e6db3d
Fix user dir to not assume existence of user
2017-06-13 11:11:26 +01:00
Erik Johnston
fdca6e36ee
Merge pull request #2274 from matrix-org/erikj/cache_is_host_joined
...
Add cache for is_host_joined
2017-06-13 10:58:53 +01:00
Erik Johnston
b58e24cc3c
Tweak the ranking of PG user dir search
2017-06-13 10:16:31 +01:00
Erik Johnston
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
Erik Johnston
a837765e8c
Merge pull request #2266 from matrix-org/erikj/host_in_room
...
Change is_host_joined to use current_state table
2017-06-12 09:49:51 +01:00
Erik Johnston
8060974344
Fix replication
2017-06-09 16:40:52 +01:00
Erik Johnston
b0d975e216
Comments
2017-06-09 16:25:42 +01:00
Erik Johnston
e54d7d536e
Cache state deltas
2017-06-09 16:24:00 +01:00
Erik Johnston
935e588799
Tweak SQL
2017-06-09 13:01:23 +01:00
Erik Johnston
0185b75381
Change is_host_joined to use current_state table
...
This bypasses a bug where using the state groups to figure out if a host
is in a room sometimes errors if the servers isn't in the room. (For
example when the server rejected an invite to a remote room)
2017-06-09 10:52:26 +01:00
Erik Johnston
ea11ee09f3
Ensure we don't use unpersisted state group as prev group
2017-06-08 11:59:57 +01:00
Erik Johnston
c62c480dc6
Merge pull request #2259 from matrix-org/erikj/fix_state_woes
...
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0
Fix bug where state_group tables got corrupted
...
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.
We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston
a053ff3979
Merge pull request #2248 from matrix-org/erikj/state_fixup
...
Faster cache for get_joined_hosts
2017-06-07 14:01:06 +01:00
Erik Johnston
64ed74c01e
When pruning, delete from device_lists_outbound_last_success
2017-06-07 11:20:47 +01:00
Erik Johnston
1a81a1898e
Keep pruning background task
2017-06-07 11:16:56 +01:00
Erik Johnston
6ba21bf2b8
Comments
2017-06-07 11:08:36 +01:00
Erik Johnston
09e4bc0501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup
2017-06-07 11:05:23 +01:00
Erik Johnston
6e2a7ee1bc
Remove spurious log lines
2017-06-07 11:05:17 +01:00
Erik Johnston
65f0513a33
Split up device_lists_outbound_pokes table for faster updates.
2017-06-07 11:02:38 +01:00
Erik Johnston
6f83c4537c
Increase size of IP cache
2017-06-07 10:18:44 +01:00
Erik Johnston
1a01af079e
Handle profile updates in user directory
2017-06-01 15:39:51 +01:00
Erik Johnston
4d039aa2ca
Fix sqlite
2017-06-01 14:58:48 +01:00
Erik Johnston
21e255a8f1
Split the table in two
2017-06-01 14:50:46 +01:00
Erik Johnston
d5477c7afd
Tweak search query
2017-06-01 13:28:01 +01:00
Erik Johnston
02a6108235
Tweak search query
2017-06-01 13:16:40 +01:00
Erik Johnston
59dbb47065
Remove spurious inlineCallbacks
2017-06-01 11:41:29 +01:00
Erik Johnston
0fe6f3c521
Bug fixes and logging
...
- Check if room is public when a user joins before adding to user dir
- Fix typo of field name "content.join_rules" -> "content.join_rule"
2017-06-01 11:09:49 +01:00
Erik Johnston
036362ede6
Order by if they have profile info
2017-06-01 09:41:08 +01:00
Erik Johnston
a757dd4863
Use prefix matching
2017-06-01 09:40:37 +01:00
Erik Johnston
f5cc22bdc6
Comment on why arbitrary comments
2017-05-31 17:30:26 +01:00
Erik Johnston
5dd1b2c525
Use unique indices
2017-05-31 17:29:12 +01:00
Erik Johnston
f1378aef91
Convert to int
2017-05-31 17:03:08 +01:00
Erik Johnston
f091061711
Fix tests
2017-05-31 16:34:40 +01:00
Erik Johnston
4abcff0177
Fix typo
2017-05-31 16:22:36 +01:00
Erik Johnston
63c58c2a3f
Limit number of things we fetch out of the db
2017-05-31 16:17:58 +01:00
Erik Johnston
304880d185
Add stream change cache
2017-05-31 15:46:36 +01:00
Erik Johnston
5d79d728f5
Split out directory and search tables
2017-05-31 15:23:49 +01:00
Erik Johnston
dc51af3d03
Pull max id from correct table
2017-05-31 15:13:49 +01:00
Erik Johnston
350622a107
Handle the server leaving a public room
2017-05-31 15:11:36 +01:00
Erik Johnston
63fda37e20
Add comments
2017-05-31 15:00:29 +01:00
Erik Johnston
293ef29655
Weight differently
2017-05-31 14:29:32 +01:00
Erik Johnston
3b5f22ca40
Add search
2017-05-31 14:00:01 +01:00
Erik Johnston
b5db4ed5f6
Update room column when room becomes unpublic
2017-05-31 13:40:28 +01:00
Erik Johnston
eeb2f9e546
Add user_directory to database
2017-05-31 11:51:01 +01:00
Erik Johnston
dd48f7204c
Add comment
2017-05-30 15:01:22 +01:00
Erik Johnston
04095f7581
Add clobbered event_id
2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e
Add current_state_delta_stream table
2017-05-30 14:44:09 +01:00
Erik Johnston
619e8ecd0c
Handle None state group correctly
2017-05-26 10:46:03 +01:00
Erik Johnston
dfbda5e025
Faster cache for get_joined_hosts
2017-05-25 17:24:44 +01:00
Erik Johnston
11f139a647
Merge pull request #2247 from matrix-org/erikj/auth_event
...
Only store event_auth for state events
2017-05-24 16:46:34 +01:00
Erik Johnston
6e614e9e10
Add background task to clear out old event_auth
2017-05-24 15:23:34 +01:00
Erik Johnston
c049472b8a
Only store event_auth for state events
2017-05-24 15:23:31 +01:00
Erik Johnston
8cf9f0a3e7
Remove redundant invalidation
2017-05-23 09:46:59 +01:00
Erik Johnston
e6618ece2d
Missed an invalidation
2017-05-23 09:36:52 +01:00
Erik Johnston
74bf4ee7bf
Stream count_e2e_one_time_keys cache invalidation
2017-05-22 16:19:22 +01:00
Erik Johnston
58ebb96cce
Fix invalidation of get_users_with_read_receipts_in_room
2017-05-19 14:38:50 +01:00
Erik Johnston
107ac7ac96
Increase size of push rule caches
2017-05-18 17:17:53 +01:00
Erik Johnston
760625acba
Make get_if_app_services_interested_in_user faster
2017-05-18 16:34:44 +01:00
Erik Johnston
d9e3a4b5db
Merge pull request #2230 from matrix-org/erikj/speed_up_get_state
...
Make get_state_groups_from_groups faster.
2017-05-17 17:23:04 +01:00
Erik Johnston
ace23463c5
Merge pull request #2216 from slipeer/app_services_interested_in_user
...
Fix users claimed non-exclusively by an app service don't get notific…
2017-05-17 16:28:50 +01:00
Erik Johnston
bbfe4e996c
Make get_state_groups_from_groups faster.
...
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.
By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
2017-05-17 15:12:15 +01:00
Erik Johnston
85e8092cca
Comment
2017-05-17 10:03:09 +01:00
Erik Johnston
ad53fc3cf4
Short circuit when we have delta ids
2017-05-17 09:57:34 +01:00
Erik Johnston
13f540ef1b
Speed up get_joined_hosts
2017-05-16 16:05:22 +01:00
Erik Johnston
b8492b6c2f
Merge pull request #2224 from matrix-org/erikj/prefill_state
...
Prefill state caches
2017-05-16 15:50:11 +01:00
Erik Johnston
331570ea6f
Remove spurious merge artifacts
2017-05-16 15:33:07 +01:00
Krombel
55af207321
Merge branch 'develop' into avoid_duplicate_filters
2017-05-16 15:29:59 +02:00
Erik Johnston
608b5a6317
Take a copy before prefilling, as it may be a frozendict
2017-05-16 12:55:29 +01:00
Krombel
64953c8ed2
avoid access-error if no filter_id matches
2017-05-15 18:36:37 +02:00
Erik Johnston
e4435b014e
Update comment
2017-05-15 15:11:30 +01:00
Erik Johnston
871605f4e2
Comments
2017-05-15 15:11:30 +01:00
Erik Johnston
e0d2f6d5b0
Add more granular event send metrics
2017-05-15 15:11:30 +01:00
Erik Johnston
bfbc907cec
Prefill state caches
2017-05-15 15:11:13 +01:00
Krombel
eb7cbf27bc
insert whitespace to fix travis build
2017-05-12 12:09:42 +02:00
Krombel
6b95e35e96
add check to only add a new filter if the same filter does not exist previously
...
Signed-off-by: Matthias Kesler <krombel@krombel.de>
2017-05-11 16:05:30 +02:00
Richard van der Hoff
ff3d810ea8
Add a comment to old delta
2017-05-11 12:48:50 +01:00
Richard van der Hoff
34194aaff7
Don't create event_search index on sqlite
...
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff
114f290947
Add more logging for purging
...
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
Richard van der Hoff
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff
dc026bb16f
Tidy purge code and add some comments
...
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
Slipeer
328378f9cb
Fix users claimed non-exclusively by an app service don't get notifications #2211
2017-05-11 11:42:08 +03:00
Richard van der Hoff
8e345ce465
Don't de-delta state groups we're about to delete
2017-05-10 18:44:22 +01:00
Richard van der Hoff
b64d312421
add some logging to purge_history
2017-05-10 18:44:22 +01:00
Erik Johnston
a3648f84b2
Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid
...
Add per user ratelimiting overrides
2017-05-10 15:54:48 +01:00
Erik Johnston
b990b2fce5
Add per user ratelimiting overrides
2017-05-10 11:05:43 +01:00
Richard van der Hoff
a7e9d8762d
Allow clients to upload one-time-keys with new sigs
...
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).
Fixes https://github.com/vector-im/riot-android/issues/1208 , hopefully.
2017-05-09 19:04:56 +01:00
Erik Johnston
40dcf0d856
Merge pull request #2203 from matrix-org/erikj/event_cache_hit_ratio
...
Don't update event cache hit ratio from get_joined_users
2017-05-08 16:52:39 +01:00
Erik Johnston
d3c3026496
Merge pull request #2201 from matrix-org/erikj/store_device_cache
...
Cache check to see if device exists
2017-05-08 16:23:04 +01:00
Erik Johnston
093f7e47cc
Expand docstring a bit
2017-05-08 16:14:46 +01:00
Erik Johnston
6a12998a83
Add missing yields
2017-05-08 16:10:51 +01:00
Erik Johnston
ffad4fe35b
Don't update event cache hit ratio from get_joined_users
...
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston
94e6ad71f5
Invalidate cache on device deletion
2017-05-08 15:55:59 +01:00
Erik Johnston
8571f864d2
Cache one time key counts
2017-05-08 15:34:27 +01:00
Erik Johnston
fc6d4974a6
Comment
2017-05-08 15:33:57 +01:00
Erik Johnston
738ccf61c0
Cache check to see if device exists
2017-05-08 15:32:18 +01:00
Erik Johnston
dcabef952c
Increase client_ip cache size
2017-05-08 15:09:19 +01:00
Erik Johnston
e0f20e9425
Revert "Remove unused import"
...
This reverts commit ab37bef83b
2017-05-08 13:07:43 +01:00
Erik Johnston
fe7c1b969c
Revert "We don't care about forgotten rooms"
...
This reverts commit ad8b316939
2017-05-08 13:07:43 +01:00
Erik Johnston
78f306a6f7
Revert "Speed up filtering of a single event in push"
...
This reverts commit 421fdf7460
2017-05-08 13:07:41 +01:00
Erik Johnston
66f371e8b8
Merge pull request #2176 from matrix-org/erikj/faster_get_joined
...
Make get_joined_users faster
2017-05-05 10:59:55 +01:00
Erik Johnston
e3aa8a7aa8
Merge pull request #2185 from matrix-org/erikj/smaller_caches
...
Optimise caches for single key
2017-05-05 10:19:05 +01:00
Erik Johnston
cf589f2c1e
Fixes
2017-05-05 10:17:56 +01:00
Erik Johnston
8af4569583
Merge pull request #2174 from matrix-org/erikj/current_cache_hosts
...
Add cache for get_current_hosts_in_room
2017-05-05 10:15:24 +01:00
Erik Johnston
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
Erik Johnston
aa93cb9f44
Add comment
2017-05-04 14:59:28 +01:00
Erik Johnston
537dbadea0
Intern host strings
2017-05-04 14:55:28 +01:00
Erik Johnston
07a07588a0
Make caches bigger
2017-05-04 14:52:28 +01:00
Erik Johnston
dfaa58f72d
Fix comment and num args
2017-05-04 14:50:24 +01:00
Erik Johnston
5d8290429c
Reduce size of get_users_in_room
2017-05-04 13:43:19 +01:00
Erik Johnston
7ebf518c02
Make get_joined_users faster
2017-05-03 15:55:54 +01:00
Erik Johnston
2c2dcf81d0
Update comment
2017-05-03 10:00:29 +01:00
Erik Johnston
1827057acc
Comments
2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-03 09:46:40 +01:00
Erik Johnston
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
Erik Johnston
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
Erik Johnston
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
Erik Johnston
3033261891
Merge pull request #2080 from matrix-org/erikj/filter_speed
...
Speed up filtering of a single event in push
2017-04-28 14:17:13 +01:00
Erik Johnston
ab37bef83b
Remove unused import
2017-04-28 09:57:23 +01:00
Erik Johnston
ad8b316939
We don't care about forgotten rooms
2017-04-28 09:52:36 +01:00
Erik Johnston
421fdf7460
Speed up filtering of a single event in push
2017-04-28 09:52:36 +01:00
Richard van der Hoff
c84770b877
Fix bgupdate error if index already exists ( #2167 )
...
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135 .
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
Erik Johnston
f053a1409e
Make state caches cache in ascii
2017-04-25 17:22:55 +01:00
Erik Johnston
f144365281
Comment
2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86
Reduce size of joined_user cache
...
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.
We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76
Reduce _get_state_group_for_event cache size
2017-04-25 11:43:03 +01:00
Erik Johnston
119cb9bbcf
Reduce cache size by not storing deferreds
...
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.
However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.
This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.
As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Erik Johnston
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
Luke Barnard
78f0ddbfad
Merge pull request #2120 from matrix-org/luke/read-markers
...
Implement Read Marker API
2017-04-13 14:21:31 +01:00
Luke Barnard
6a70647d45
Correct logic in is_event_after
2017-04-13 13:46:17 +01:00
Luke Barnard
b9557064bf
Simplify is_event_after logic
2017-04-12 14:36:20 +01:00
Luke Barnard
7f94709066
travis flake8..
2017-04-11 18:35:45 +01:00
Luke Barnard
73880268ef
Refactor event ordering check to events store
2017-04-11 17:34:09 +01:00
Erik Johnston
85657eedf8
Bail on where clause instead
2017-04-11 16:24:31 +01:00
Erik Johnston
b48045a8f5
Don't bother with outer check for now
2017-04-11 16:23:24 +01:00
Erik Johnston
34840cdcef
Fix getting latest device IP for user with no devices
2017-04-11 09:56:54 +01:00
Erik Johnston
2a3e822f44
Comment
2017-04-07 13:47:04 +01:00
Erik Johnston
d72667fcce
Speed up get_current_state_ids
...
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).
get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
2017-04-07 10:10:49 +01:00
Erik Johnston
9f26d3b75b
Deduplicate new deviceinbox rows for replication
2017-04-04 16:21:21 +01:00
Erik Johnston
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
Erik Johnston
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c
Use txn.fetchall() so we can reuse txn
2017-03-30 10:30:05 +01:00
Erik Johnston
3ce8d59176
Increase cache size for _get_state_group_for_event
2017-03-29 14:31:46 +01:00
Erik Johnston
e4df0e189d
Decrank last commit
2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert
2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512
Correctly look up key
2017-03-29 10:56:26 +01:00
Erik Johnston
d28db583da
Merge pull request #2063 from matrix-org/erikj/device_list_batch
...
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be
The algorithm is part of the key id
2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
Erik Johnston
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6
Merge pull request #2062 from matrix-org/erikj/presence_replication
...
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4
Actually call invalidate
2017-03-24 13:28:20 +00:00
Erik Johnston
09f79aaad0
Use presence replication stream to invalidate cache
...
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
Erik Johnston
7fc1f1e2b6
Cache hosts in room
2017-03-24 11:46:24 +00:00
Erik Johnston
d58b1ffe94
Replace some calls to cursor_to_dict
...
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Erik Johnston
6ebe2d23b1
Raise a more helpful exception
2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d
Don't user upsert to persist new one time keys
...
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
2017-03-23 13:17:00 +00:00
Richard van der Hoff
64778693be
fix up some key verif docstrings
2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
Richard van der Hoff
eddce9d74a
Merge pull request #2027 from matrix-org/rav/logcontext_leaks
...
A few fixes to logcontext things
2017-03-20 11:53:36 +00:00
Richard van der Hoff
994d7ae7c5
Remove broken use of clock.call_later
...
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.
We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
Erik Johnston
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae
Avoid resetting state on rejected events
...
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.
This should fix https://github.com/matrix-org/synapse/issues/1935 .
2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5
Refactoring and cleanups
...
A few non-functional changes:
* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
mind-bending conditional.
2017-03-17 15:06:07 +00:00
Erik Johnston
da146657c9
Comments
2017-03-16 13:04:07 +00:00
Erik Johnston
6957bfdca6
Don't recreate so many sets
2017-03-16 11:54:26 +00:00
Richard van der Hoff
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Erik Johnston
bb256ac96f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed
2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e
Fix current_state_events table to not lie
...
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
2017-03-14 10:57:43 +00:00
Luke Barnard
bbeeb97f75
Implement _simple_delete_many_txn, use it to delete devices
...
(But this doesn't implement the same for deleting access tokens or e2e keys.
Also respond to code review.
2017-03-13 17:53:23 +00:00
Erik Johnston
79926e016e
Assume rooms likely haven't changed
2017-03-13 09:50:10 +00:00
Erik Johnston
8ffbe43ba1
Get current state by using current_state_events table
2017-03-10 17:39:35 +00:00
Erik Johnston
64d62e41b8
Noop repated delete device inbox calls from /sync
2017-03-10 10:36:43 +00:00
Richard van der Hoff
6ad71cc29d
Remove spurious SQL logging ( #1972 )
...
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
2017-03-08 18:00:44 +00:00
Erik Johnston
ac5491f563
Select distinct devices from DB
...
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
2017-03-06 11:10:14 +00:00
Erik Johnston
da52d3af31
Fix up
2017-03-03 15:29:13 +00:00
Erik Johnston
d766343668
Add index to device_lists_stream
2017-03-01 15:56:30 +00:00
Erik Johnston
3365117151
Clobber old device list stream entries
2017-03-01 10:21:30 +00:00
Erik Johnston
d638a7484b
Merge pull request #1959 from matrix-org/erikj/intern_once
...
Intern table column names once
2017-02-28 15:15:16 +00:00
Erik Johnston
b84907bdbb
Intern table column names once
2017-02-28 14:38:16 +00:00
Erik Johnston
e4919b9329
Add stream_id index to device_lists_outbound_pokes
...
As this is used for replication streaming
2017-02-28 11:19:06 +00:00
Erik Johnston
8a12b6f1eb
Fix up txn name
2017-02-28 10:15:50 +00:00
Erik Johnston
a41dce8f8a
Remove needless check
2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776
It should be all
2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709
Don't fetch current state in common case
...
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
2017-02-27 18:33:41 +00:00
Erik Johnston
fc5d876dba
Merge pull request #1954 from matrix-org/erikj/cache_device2
...
Cache get_user_devices_from_cache
2017-02-27 16:36:40 +00:00
Erik Johnston
f58dbb02a6
Cache get_user_devices_from_cache
2017-02-27 16:22:12 +00:00
Erik Johnston
1a4f8022e6
Strip newlines from SQL queries
2017-02-23 11:15:31 +00:00
Erik Johnston
b2d20e94fa
Remove lock from rotate notifs
2017-02-22 14:24:02 +00:00
Erik Johnston
b7442c3e2b
Store looping call
2017-02-21 13:59:25 +00:00
Erik Johnston
699be7d1be
Fix up notif rotation
2017-02-18 14:42:39 +00:00
David Baker
4aa29508af
Use TEXT rather than VARCHAR
...
While we're changing anyway
2017-02-17 10:51:49 +00:00
David Baker
b4017539d4
Make the pushers lang field column longer
...
To accommodate things like zh-Hans-CN
Fixes https://github.com/vector-im/riot-ios/issues/1031
2017-02-17 10:42:57 +00:00
Erik Johnston
138e030cfe
Comment
2017-02-16 15:03:36 +00:00
Erik Johnston
502ae6c663
Comment
2017-02-16 14:47:11 +00:00
Erik Johnston
e6acf0c399
Store the default push actions in a more efficient manner
2017-02-16 14:40:24 +00:00
Erik Johnston
04eca2589d
Merge pull request #1916 from matrix-org/erikj/push_actions_delete
...
Aggregate event push actions
2017-02-16 15:28:58 +01:00
Richard van der Hoff
7dcbcca68c
Merge pull request #1921 from matrix-org/rav/fix_key_changes
...
Fix bugs in the /keys/changes api
2017-02-15 11:25:16 +00:00
Richard van der Hoff
fc2f29c1d0
Fix bugs in the /keys/changes api
...
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
passing a `RoomStreamToken`, which meant that it returned the *current*
extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
that a `TypeError` was thrown and we ended up acting as though there was *no*
prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
dictionary - just looking up the state keys in it again meant that we acted
as though there was no prev state. We now check if each member's state has
changed since *any* of the extremities.
Also add/fix some comments.
2017-02-14 13:59:50 +00:00
Erik Johnston
ce3c8df6df
Less aggressive timers
2017-02-14 13:41:24 +00:00
Erik Johnston
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
Erik Johnston
795f8e3fe7
Merge pull request #1873 from matrix-org/erikj/delete_push_actions
...
Be more agressive about purging old room event_push_actions
2017-02-14 14:29:04 +01:00
Erik Johnston
9e617cd4c2
Cache get_presence storage
2017-02-13 13:50:03 +00:00
Erik Johnston
52cd019a54
Make None check explicit
2017-02-08 16:04:29 +00:00
Erik Johnston
f20cd34858
Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem
...
Ignore new rejected events when working out forward extremities.
2017-02-08 16:59:06 +01:00
Erik Johnston
7723b4caa4
Ignore new rejected events when working out forward extremeties.
2017-02-08 14:48:06 +00:00
Erik Johnston
af6da6db2d
Merge pull request #1784 from morteza-araby/user-admin
...
Administration functionalities
2017-02-06 16:21:10 +01:00
Erik Johnston
38258a0976
Bump cache sizes for common membership queries
2017-02-02 18:45:55 +00:00
Erik Johnston
832e9c52ca
Comment
2017-02-02 13:09:56 +00:00
Erik Johnston
54a79c1d37
Make presence.get_new_events a bit faster
...
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
Morteza Araby
2849d3f29d
admin,storage: added more administrator functionalities
...
administrators can now:
- Set displayname of users
- Update user avatars
- Search for users by user_id
- Browse all users in a paginated API
- Reset user passwords
- Deactivate users
Helpers for doing paginated queries has also been added to storage
Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
Erik Johnston
fac3c03087
Be more agressive about purging old room event_push_actions
2017-02-01 18:27:24 +00:00
Erik Johnston
fbfe44bb4d
Doc args
2017-02-01 17:52:57 +00:00
Erik Johnston
d61a04583e
Comment
2017-02-01 17:35:23 +00:00
Erik Johnston
7e919bdbd0
Include newly joined users in /keys/changes API
2017-02-01 17:33:16 +00:00
Erik Johnston
ebfaff84c9
Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
...
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
Erik Johnston
62f6b86ba7
Merge pull request #1868 from matrix-org/erikj/replication_cache
...
Only invalidate membership caches based on the cache stream
2017-02-01 13:12:30 +00:00
Erik Johnston
f6124311fd
Add m.room.member type to query
2017-02-01 11:59:17 +00:00
Erik Johnston
88a4d54883
Merge pull request #1867 from matrix-org/erikj/member_index
...
Add an index to make membership queries faster
2017-02-01 11:44:27 +00:00
Erik Johnston
368c88c487
Add a small cache get_all_new_events
2017-02-01 10:50:44 +00:00
Erik Johnston
5deaf9e30b
Up get_latest_event_ids_in_room cache
2017-02-01 10:39:41 +00:00
Erik Johnston
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
Erik Johnston
ac001dabdc
Merge pull request #1864 from matrix-org/erikj/device_list_fixes
...
Fix clearing out old device list outbound pokes
2017-01-31 13:35:35 +00:00
Erik Johnston
bfb3d255b1
Merge pull request #1862 from matrix-org/erikj/presence_update
...
Use DB cache of joined users for presence
2017-01-31 13:23:24 +00:00
Erik Johnston
ab55794b6f
Fix deletion of old sent devices correctly
2017-01-31 13:22:41 +00:00
Erik Johnston
d3169e8d28
Only fetch with row ts and count > 1
2017-01-31 11:20:03 +00:00
Erik Johnston
05b9f48ee5
Fix clearing out old device list outbound pokes
2017-01-31 10:08:55 +00:00
Erik Johnston
4b3403ca9b
Stream cache invalidations for room membership storage functions
2017-01-30 17:28:22 +00:00
Erik Johnston
1c13c9f6b6
Don't have such a large cache
2017-01-30 17:12:14 +00:00
Erik Johnston
c7a26b7c32
Fix unit tests
2017-01-30 17:11:24 +00:00
Erik Johnston
c2c9a78db9
Noop device key changes if they're the same
2017-01-30 16:55:04 +00:00
Erik Johnston
e75a779d9e
Fix query
2017-01-30 16:38:20 +00:00
Erik Johnston
828db669ec
Use get_users_in_room and declare it iterable
2017-01-30 16:37:22 +00:00
Erik Johnston
3670025e64
Rename func
2017-01-30 14:11:31 +00:00
Erik Johnston
d360c97ae1
Clear out old destination pokes.
2017-01-30 10:14:37 +00:00
Erik Johnston
76100203ab
Always use the latest stream_id, sent or unsent
2017-01-30 10:14:25 +00:00
Erik Johnston
d1e1fd6210
Add ts column to device_lists_outbound_pokes
2017-01-27 15:23:48 +00:00
Erik Johnston
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
Erik Johnston
84a35f32c7
Comment
2017-01-27 10:35:12 +00:00
Erik Johnston
c517a19c2d
Comment
2017-01-27 10:33:26 +00:00
Erik Johnston
738a2867c8
SQL param ordering
2017-01-27 10:31:29 +00:00
Erik Johnston
755adff0e4
User if rather than for
2017-01-27 10:31:06 +00:00
Erik Johnston
888c59c955
Better name
2017-01-27 10:29:47 +00:00
Erik Johnston
f25a4a4692
Remove unused param
2017-01-27 10:27:39 +00:00
Erik Johnston
31aca5589c
Fix on sqlite: use left rather than outer join
2017-01-26 16:55:50 +00:00
Erik Johnston
76d40f4904
Handle users leaving rooms
2017-01-26 16:39:33 +00:00
Erik Johnston
fbfad76c03
Add comments
2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4
Fix up sending of m.device_list_update edus
2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
Erik Johnston
fdf2a31a51
Typo
2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092
Refactor to calculate state delta outside transaction
2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485
Comments
2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5
Spelling
2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67
Update all call sites after rename
2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9
Comments
2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e
Calculate the forward extremeties once
2017-01-20 14:28:53 +00:00
Erik Johnston
09eb08f910
Derive current_state_events from state groups
2017-01-20 11:52:51 +00:00
Erik Johnston
4ec1cf49e2
Lower loading events log to DEBUG
2017-01-17 17:28:32 +00:00
Erik Johnston
d906206049
Increase state_group_cache_size
2017-01-17 11:31:08 +00:00
Erik Johnston
897f8752da
Up cache max entries for state
2017-01-16 15:08:17 +00:00
Erik Johnston
01521299c7
Increase cache size limit
2017-01-16 11:56:51 +00:00
Erik Johnston
2fae34bd2c
Optionally measure size of cache by sum of length of values
2017-01-13 17:46:17 +00:00
Erik Johnston
f0325a9ccc
Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index
...
Change device_inbox stream index to include user
2017-01-13 15:14:51 +00:00
Erik Johnston
c050f493dd
Add comment
2017-01-13 15:14:41 +00:00
Erik Johnston
ab655dca33
Explicitly close the cursor
2017-01-10 15:15:25 +00:00
Erik Johnston
5a32e9273e
Don't disable autocommit
2017-01-10 15:11:27 +00:00
Erik Johnston
caddadfc5a
Change device_inbox stream index to include user
...
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
Erik Johnston
dd52d4de4c
Limit number of entries to prefill from cache
...
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.
In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
Matthew Hodgson
bd85434cb3
Merge branch 'release-v0.18.6' into develop
2017-01-05 13:58:19 +00:00
Matthew Hodgson
8cfc0165e9
fix annoying typos
2017-01-05 13:39:43 +00:00
Matthew Hodgson
b31ed22738
Merge branch 'release-v0.18.6' into develop
2017-01-05 13:03:02 +00:00
Mark Haines
dd3df11c55
More logging for the linearizer and for get_events
2017-01-05 12:32:47 +00:00
David Baker
0c88ab1844
Add /account/3pid/delete endpoint
...
Also fix a typo in a comment
2016-12-20 18:27:30 +00:00
Richard van der Hoff
edc6a1e4f9
Add some logging for syncing to_device events
...
Attempt to track down the loss of to_device events
(https://github.com/vector-im/riot-web/issues/2711 etc).
2016-12-15 18:16:10 +00:00
Erik Johnston
c3208e45c9
Fixup membership query
2016-12-14 10:46:58 +00:00
Erik Johnston
8588d0eb3d
Merge pull request #1697 from matrix-org/erikj/fix_bg_member
...
Fix background update that prematurely stopped
2016-12-12 17:20:17 +00:00
Erik Johnston
1574b839e0
Merge pull request #1676 from matrix-org/erikj/room_list
...
Add new API appservice specific public room list
2016-12-12 17:00:10 +00:00
Erik Johnston
7ec2bf9b77
Fix background update that prematurely stopped
2016-12-12 16:54:58 +00:00
Erik Johnston
09cbcb78d3
Add cache to get_public_room_ids_at_stream_id
2016-12-12 14:41:51 +00:00
Erik Johnston
631376e2ac
Notify replication. Use correct network_id
2016-12-12 14:28:15 +00:00
Erik Johnston
d45c984653
Docstring
2016-12-12 11:00:27 +00:00
Erik Johnston
1697f6a323
Merge pull request #1680 from matrix-org/erikj/joined_rooms
...
Add new room membership APIs
2016-12-09 11:35:48 +00:00
Erik Johnston
52d12ca782
Add /room/<room_id>/joined_members API
...
This returns the currently joined members in the room with their display
names and avatar urls. This is more efficient than /members for large
rooms where you don't need the full events.
2016-12-08 13:32:07 +00:00
Erik Johnston
c45d8e9ba2
Add profile data to the room_membership table for joins
2016-12-08 13:08:41 +00:00
Richard van der Hoff
bd07a35c29
Fix result of insert_receipt
...
This should fix the absence of notifications when new receipts arrive.
2016-12-08 12:11:34 +00:00
Erik Johnston
2687af82d4
Comments
2016-12-07 09:58:33 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Richard van der Hoff
aa09d6b8f0
Rip out more refresh_token code
...
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.
Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston
46ca345b06
Don't send old events as federation
2016-11-24 12:29:02 +00:00
Erik Johnston
1985860c6e
Comment
2016-11-23 15:59:59 +00:00
Erik Johnston
2ac516850b
More efficient notif count queries
2016-11-23 15:57:04 +00:00
Erik Johnston
302fbd218d
Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn
...
Split out federation transaction sending to a worker
2016-11-23 15:39:12 +00:00
Erik Johnston
b69f76c106
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn
2016-11-23 11:31:53 +00:00
Erik Johnston
0163466d72
Ignore AS users when fetching push rules
...
By ignoring AS users early on when fetching push rules for a room we can
avoid needlessly hitting the DB and filling up the caches.
2016-11-23 11:01:01 +00:00
Erik Johnston
54fed21c04
Fix tests and flake8
2016-11-22 18:18:31 +00:00
Erik Johnston
90565d015e
Invalidate retry cache in both directions
2016-11-22 17:45:44 +00:00
Kegan Dougal
a2a6c1c22f
Fail with a coherent error message if /sync?filter= is invalid
2016-11-21 13:15:25 +00:00
Erik Johnston
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
c7546b3cdb
Merge pull request #1617 from matrix-org/erikj/intern_state_dict
...
Correctly intern keys in state cache
2016-11-15 16:45:55 +00:00
Erik Johnston
f9d156d270
New Flake8 fixes
2016-11-15 11:22:29 +00:00
Erik Johnston
587d8ac60f
Correctly intern keys in state cache
2016-11-08 11:53:25 +00:00
Erik Johnston
63772443e6
Comment
2016-11-04 10:53:42 +00:00
Erik Johnston
a3f6576084
Remove unused but buggy function
2016-11-04 10:48:20 +00:00
Erik Johnston
64c6566980
Remove spurious comment
2016-11-03 15:04:32 +00:00
Erik Johnston
8fd4d9129f
Replace postgres GIN with GIST
...
This is because GIN can be slow to write too, especially when the table
gets large.
2016-11-03 15:00:03 +00:00
Erik Johnston
760469c812
Continue to clean up received_transactions
2016-11-01 11:42:08 +00:00
Erik Johnston
f09d2b692f
Removed unused stuff
2016-10-31 17:10:56 +00:00
Erik Johnston
4c3eb14d68
Increase batching of sent transaction inserts
...
This should further reduce the number of individual inserts,
transactions and updates that are required for keeping sent_transactions
up to date.
2016-10-31 16:07:45 +00:00
Erik Johnston
a9111786f9
Use most recently added binding, not most recently seen user.
2016-10-27 14:32:45 +01:00
Erik Johnston
1fc1bc2a51
Fix user_threepids schema delta
...
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.
This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
2016-10-27 14:14:44 +01:00
Erik Johnston
d04e2ff3a4
Fix incredubly slow back pagination query
...
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
2016-10-24 13:35:51 +01:00
Erik Johnston
78c083f159
Merge pull request #1164 from pik/error-codes
...
Clarify Error codes for GET /filter/
2016-10-19 14:26:17 +01:00
David Baker
0108ed8ae6
Latest delta is now 37
2016-10-19 11:40:35 +01:00
David Baker
a7f48320b1
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-19 11:28:56 +01:00
David Baker
df2a616c7b
Convert emails to lowercase when storing
...
And db migration sql to convert existing addresses.
2016-10-19 11:13:55 +01:00
pik
e8b1d2a452
Refactor test_filter to use real DataStore
...
* add tests for filter api errors
2016-10-18 12:17:38 -05:00
Erik Johnston
6942d68247
Bump schema version
2016-10-17 11:17:45 +01:00
Erik Johnston
b59994b454
Remove TODO
2016-10-17 11:17:02 +01:00
Erik Johnston
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
David Baker
29c5922021
Revert part of 6207399
...
older sqlite doesn't support indexes on expressions, lets just
store things lowercase in the db
2016-10-14 16:20:24 +01:00
David Baker
bcb1245a2d
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-14 15:10:38 +01:00
David Baker
62073992c5
Make password reset email field case insensitive
2016-10-14 13:56:53 +01:00
Erik Johnston
3061dac53e
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop
2016-10-11 14:08:29 +01:00
Patrik Oldsberg
9bfc617791
storage/appservice: make appservice methods only relying on the cache synchronous
2016-10-06 15:24:59 +02:00
Erik Johnston
5875a65253
Merge pull request #1145 from matrix-org/erikj/fix_reindex
...
Fix background reindex of origin_server_ts
2016-09-29 13:53:48 +01:00
Erik Johnston
9040c9ffa1
Fix background reindex of origin_server_ts
...
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
2016-09-27 11:23:49 +01:00
Erik Johnston
4974147aa3
Remove duplication
2016-09-27 09:27:54 +01:00
Erik Johnston
13122e5e24
Remove unused variable
2016-09-27 09:21:51 +01:00
Erik Johnston
cf3e1cc200
Fix perf of fetching state in SQLite
2016-09-26 17:16:24 +01:00
Erik Johnston
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston
8009d84364
Match against event_id, rather than room_id
2016-09-21 16:46:59 +01:00
Erik Johnston
dc692556d6
Remove spurious AS clause
2016-09-21 16:28:47 +01:00
Erik Johnston
dc78db8c56
Update correct table
2016-09-21 15:52:44 +01:00
Erik Johnston
4f78108d8c
Readd entries to public_room_list_stream that were deleted
2016-09-21 15:24:22 +01:00
Erik Johnston
0b78d8adf2
Fix _delete_old_forward_extrem_cache query
2016-09-21 15:20:56 +01:00
Erik Johnston
a68807d426
Comment
2016-09-16 11:36:20 +01:00
Erik Johnston
e58a9d781c
Filter remote rooms lists locally
2016-09-16 10:19:32 +01:00
Erik Johnston
de4f798f01
Handling expiring stream extrems correctly.
2016-09-15 17:34:59 +01:00
Erik Johnston
955f34d23e
Change get_pos_of_last_change to return upper bound
2016-09-15 15:12:07 +01:00
Erik Johnston
cb3edec6af
Use stream_change cache to make get_forward_extremeties_for_room cache more effective
2016-09-15 14:28:13 +01:00
Erik Johnston
55e6fc917c
Add cache to get_forward_extremeties_for_room
2016-09-15 14:04:28 +01:00
Erik Johnston
e457034e99
Merge pull request #1121 from matrix-org/erikj/public_room_paginate
...
Add pagination support to publicRooms
2016-09-15 13:27:09 +01:00
Erik Johnston
211786ecd6
Stream public room changes down replication
2016-09-15 11:47:23 +01:00
Erik Johnston
4fb65a1091
Base public room list off of public_rooms stream
2016-09-15 11:27:04 +01:00
Erik Johnston
c566f0ee17
Calculate the public room list from a stream_ordering
2016-09-14 17:42:47 +01:00
Erik Johnston
baffe96d95
Add a room visibility stream
2016-09-14 17:29:19 +01:00
Erik Johnston
ed992ae6ba
Add a DB index to figure out past state at a stream ordering in a room
2016-09-14 16:20:27 +01:00
Erik Johnston
00f51493f5
Fix reindex
2016-09-14 10:18:30 +01:00
Erik Johnston
d5ae1f1291
Ensure we don't mutate state cache entries
2016-09-14 10:03:48 +01:00
Erik Johnston
43253c10b8
Remove redundant event_auth index
2016-09-13 11:47:48 +01:00
Erik Johnston
b17af156c7
Remove where clause
2016-09-12 17:05:54 +01:00
Erik Johnston
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
Erik Johnston
c94de0ab60
Add WHERE clause support to index creation
2016-09-12 16:55:01 +01:00
Erik Johnston
7c679b1118
Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent
...
Create new index concurrently
2016-09-12 14:38:40 +01:00
Erik Johnston
d080b3425c
Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast
...
Optimise /notifications query
2016-09-12 14:34:12 +01:00
Erik Johnston
03a98aff3c
Create new index concurrently
2016-09-12 14:27:01 +01:00
Erik Johnston
fa20c9ce94
Change the index to be stream_ordering, highlight
2016-09-12 14:04:08 +01:00
Erik Johnston
5ef5435529
Remove unused import
2016-09-12 13:32:58 +01:00
Mark Haines
aa7b890cfe
Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync
...
Fix direct to device messages recieved over federation to notify sync
2016-09-12 13:25:23 +01:00
Erik Johnston
7cd6edb947
Use register_background_index_update
2016-09-12 12:54:48 +01:00
Erik Johnston
0294c14ec4
Add back in query change
2016-09-12 12:43:56 +01:00
Erik Johnston
7fe42cf949
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast
2016-09-12 12:37:09 +01:00
Erik Johnston
15ca0c6a4d
Make reindex happen in bg
2016-09-12 12:36:36 +01:00
Mark Haines
a232e06100
Fix direct to device messages recieved over federation to notify sync
2016-09-12 12:30:46 +01:00
Mark Haines
31f85f9db9
Add comments to existing schema deltas that used "CREATE INDEX" directly
2016-09-12 11:00:26 +01:00
Erik Johnston
54417999b6
Revert "Add index to event_push_actions"
2016-09-12 10:39:55 +01:00
Erik Johnston
897d57bc58
Change state fetch query for postgres to be faster
...
It turns out that postgres doesn't like doing a list of OR's and is
about 1000x slower, so we just issue a query for each specific type
seperately.
2016-09-12 10:05:07 +01:00
David Baker
b91e2833b3
Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast
2016-09-09 19:11:34 +01:00
David Baker
f2acc3dcf9
Add index to event_push_actions
...
and remove room_id caluse so it uses it
Mostly from @negativemjark
2016-09-09 18:54:54 +01:00
Mark Haines
647c724573
Use the previous MAX value if any to set the stream_id
2016-09-09 11:52:44 +01:00
Mark Haines
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
Mark Haines
303b6f29f0
Merge pull request #1087 from matrix-org/markjh/reapply_delta
...
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
2016-09-09 11:24:27 +01:00
Erik Johnston
1fe7ca1362
Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop
2016-09-09 11:15:40 +01:00
Mark Haines
0877157353
Just move the schema and add some DROPs
2016-09-09 11:04:47 +01:00
Erik Johnston
2ffec928e2
Reduce batch size to be under SQL limit
2016-09-09 11:03:31 +01:00
Matthew Hodgson
b8f84f99ff
Merge pull request #1081 from matrix-org/dbkr/notifications_only_highlight
...
Implement `only=highlight` on `/notifications`
2016-09-09 00:09:51 +01:00
Mark Haines
43b77c5d97
Only catch databas errors
2016-09-08 17:44:21 +01:00
Mark Haines
7d5b142547
Add a stub run_upgrade
2016-09-08 17:39:11 +01:00
David Baker
c3276aef25
Merge pull request #1080 from matrix-org/dbkr/fix_notifications_api_with_from
...
Fix /notifications API when used with `from` param
2016-09-08 17:35:35 +01:00
Mark Haines
fa722a699c
Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop
2016-09-08 17:35:16 +01:00
Erik Johnston
5c688739d6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reindex_state_groups
2016-09-08 16:52:09 +01:00
Erik Johnston
ebb46497ba
Add delta file
2016-09-08 16:38:54 +01:00
Erik Johnston
5beda10bbd
Reindex state_groups_state after pruning
2016-09-08 16:18:01 +01:00
Mark Haines
a1c8f268e5
Support wildcard device_ids for direct to device messages
2016-09-08 15:13:05 +01:00
Mark Haines
fa9d36e050
Merge branch 'develop' into markjh/direct_to_device_federation
2016-09-08 13:43:43 +01:00
David Baker
4ef222ab61
Implement only=highlight on /notifications
2016-09-08 13:43:35 +01:00
Erik Johnston
61cd9af09b
Log delta files we're applying
2016-09-08 13:40:46 +01:00
David Baker
c5b49eb7ca
Fix /notifications API when used with from param
2016-09-08 09:40:10 +01:00
Erik Johnston
b568ca309c
Temporarily disable sequential scans for state fetching
2016-09-08 09:38:54 +01:00
Mark Haines
3c320c006c
Merge pull request #1077 from matrix-org/markjh/device_logging
...
Log the types and values when failing to store devices
2016-09-07 18:24:24 +01:00
Mark Haines
85b51fdd6b
Log the types and values when failing to store devices
2016-09-07 17:19:18 +01:00
Mark Haines
2a0159b8ae
Fix the stream change cache to work over replication
2016-09-07 15:58:00 +01:00
Mark Haines
31a07d2335
Add stream change caches for device messages
2016-09-07 15:27:07 +01:00
Erik Johnston
513188aa56
Comment
2016-09-07 14:53:23 +01:00
Erik Johnston
fadb01551a
Add appopriate framing clause
2016-09-07 14:39:01 +01:00
Erik Johnston
d25c20ccbe
Use windowing function to make use of index
2016-09-07 14:22:22 +01:00
Mark Haines
7d893beebe
Comment the add_messages storage functions
2016-09-07 12:03:37 +01:00
Mark Haines
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
Mark Haines
e020834e4f
Add storage methods for federated device messages
2016-09-06 15:12:13 +01:00
Mark Haines
2ad72da931
Add tables for federated device messages
...
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
2016-09-06 15:10:40 +01:00
Mark Haines
f4164edb70
Move _add_messages_to_device_inbox_txn into a separate method
2016-09-06 11:26:37 +01:00
Erik Johnston
0595413c0f
Scale the batch size so that we're not bitten by the minimum
2016-09-05 15:49:57 +01:00
Erik Johnston
a7032abb2e
Correctly handle reindexing state groups that already have an edge
2016-09-05 15:07:23 +01:00
Erik Johnston
9e6d88f4e2
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-05 15:01:33 +01:00
Erik Johnston
70332a12dd
Take value in a better way
2016-09-05 14:57:14 +01:00
Erik Johnston
373654c635
Comment about sqlite and WITH RECURSIVE
2016-09-05 14:50:36 +01:00
Erik Johnston
485d999c8a
Correctly delete old state groups in purge history API
2016-09-05 14:49:08 +01:00
Erik Johnston
bb36b93f71
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-05 11:53:11 +01:00
Erik Johnston
caed150363
Remove unused imports
2016-09-05 10:52:01 +01:00
Erik Johnston
80a6a445fa
Only fetch local pushers
2016-09-05 10:43:32 +01:00
Erik Johnston
628e65721b
Add comments
2016-09-05 10:41:27 +01:00
Erik Johnston
a99e933550
Add upgrade script that will slowly prune state_groups_state entries
2016-09-05 10:05:36 +01:00
Erik Johnston
3847fa38c4
Make bulk_get_push_rules_for_room use get_joined_users_from_context cache
2016-09-05 10:02:38 +01:00
Mark Haines
81b94c5750
Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound
...
Only return new device messages in /sync
2016-09-02 16:18:34 +01:00
Mark Haines
65fa37ac5e
Only return new device messages in /sync
2016-09-02 15:50:37 +01:00
Erik Johnston
3baf641a48
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-02 14:54:07 +01:00
Erik Johnston
cce957e254
Bump max_entries on get_destination_retry_timings
2016-09-02 14:08:33 +01:00
Erik Johnston
657847e4c6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-02 11:04:48 +01:00
Erik Johnston
598317927c
Limit the length of state chains
2016-09-02 10:41:38 +01:00
Erik Johnston
9e25443db8
Move to storing state_groups_state as deltas
2016-09-01 14:31:26 +01:00
Erik Johnston
44982606ee
Merge pull request #1060 from matrix-org/erikj/state_ids
...
Assign state groups in state handler.
2016-09-01 14:20:42 +01:00
Erik Johnston
0cfd6c3161
Use state_groups table to test existence
2016-08-31 16:25:57 +01:00
Erik Johnston
5405351b14
Lower get_linearized_receipts_for_room cache size
2016-08-31 16:19:44 +01:00
Erik Johnston
1671913287
Merge pull request #1059 from matrix-org/erikj/sent_transaction_delete
...
Clean up old sent transactions
2016-08-31 15:07:33 +01:00
Erik Johnston
c10cb581c6
Correctly handle the difference between prev and current state
2016-08-31 14:26:22 +01:00
Erik Johnston
ef0cc648cf
Clean up old sent transactions
2016-08-31 11:12:02 +01:00
Mark Haines
a662252758
Return the current stream position from add_messages_to_device_inbox
2016-08-31 10:42:52 +01:00
Mark Haines
1aa3e1d287
Add a replication stream for direct to device messages
2016-08-31 10:38:58 +01:00
Erik Johnston
1bb8ec296d
Generate state group ids in state layer
2016-08-31 10:09:46 +01:00
Erik Johnston
5dc2a702cf
Make _state_groups_id_gen a normal IdGenerator
2016-08-30 16:55:11 +01:00
Erik Johnston
55fc17cf4b
Merge pull request #1049 from matrix-org/erikj/presence_users_in_room
...
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-30 10:50:37 +01:00
Erik Johnston
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
Mark Haines
4bbef62124
Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device
2016-08-26 14:35:31 +01:00
Erik Johnston
4daa397a00
Add is_host_joined to slave storage
2016-08-26 13:10:56 +01:00
Erik Johnston
1ccdc1e93a
Cache check_host_in_room
2016-08-26 10:59:40 +01:00
Erik Johnston
778fa85f47
Make sync not pull out full state
2016-08-25 18:59:44 +01:00
Mark Haines
ab34fdecb7
Merge branch 'develop' into markjh/direct_to_device
2016-08-25 18:34:46 +01:00
Mark Haines
641efb6a39
Fix the deduplication of incoming direct-to-device messages
2016-08-25 18:14:02 +01:00
Erik Johnston
721414d98a
Add desc
2016-08-25 17:49:05 +01:00
Mark Haines
e993925279
Add store-and-forward direct-to-device messaging
2016-08-25 17:35:37 +01:00
Erik Johnston
a3dc1e9cbe
Replace context.current_state with context.current_state_ids
2016-08-25 17:32:22 +01:00
Erik Johnston
17f4f14df7
Pull out event ids rather than full events for state
2016-08-25 13:42:44 +01:00
Erik Johnston
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
Erik Johnston
122c7a43c9
Merge pull request #1038 from matrix-org/erikj/receved_txn_purge
...
Delete old received_transactions rows
2016-08-23 11:02:40 +01:00
Erik Johnston
d143f211c8
Merge pull request #1028 from matrix-org/dbkr/notifications_api
...
Add the Notifications API
2016-08-22 18:23:24 +01:00
Erik Johnston
aad8a1a825
Delete old received_transactions
2016-08-22 16:29:46 +01:00
Erik Johnston
e65bc7d315
Merge pull request #1031 from matrix-org/erikj/measure_notifier
...
Add more Measure blocks
2016-08-22 12:13:07 +01:00
Matthew Hodgson
6e80c03d45
Merge branch 'develop' into dbkr/notifications_api
2016-08-20 00:16:18 +01:00
Erik Johnston
37adde32dc
Move defer.returnValue out of Measure
2016-08-19 18:23:44 +01:00
Erik Johnston
47dd8f02a1
Measure _get_event_from_row
2016-08-19 18:23:44 +01:00
Erik Johnston
e6784daf07
Merge pull request #1030 from matrix-org/erikj/cache_contexts
...
Add concept of cache contexts
2016-08-19 16:29:58 +01:00
Erik Johnston
c0d7d9d642
Rename to on_invalidate
2016-08-19 15:13:58 +01:00
Erik Johnston
dc76a3e909
Make cache_context an explicit option
2016-08-19 15:02:38 +01:00
Erik Johnston
f164fd9220
Move _bulk_get_push_rules_for_room to storage layer
2016-08-19 14:29:20 +01:00
Erik Johnston
ba214a5e32
Remove lru option
2016-08-19 14:17:11 +01:00
Erik Johnston
b770435389
Make get_new_events_for_appservice use indices
2016-08-19 10:28:42 +01:00
David Baker
1e4217c90c
Explicit join
2016-08-18 17:53:44 +01:00
David Baker
602c84cd9c
Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api
2016-08-18 17:15:26 +01:00
Erik Johnston
07229bbdae
Add appservice worker
2016-08-18 14:59:55 +01:00
Erik Johnston
9da84a9a1e
Make AppserviceHandler stream events from database
...
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
2016-08-18 11:54:41 +01:00
Erik Johnston
732cf72b86
Fix push_display_name_rename schema update
2016-08-17 18:12:21 +01:00
David Baker
0d7cef0943
Merge pull request #1011 from matrix-org/dbkr/contains_display_name_override
...
Move display name rule
2016-08-17 16:15:45 +01:00
Erik Johnston
949629291c
Do it in storage function
2016-08-16 17:05:34 +01:00
David Baker
ad42322257
Add migration script
...
To port existing rule actions & enable entries to the new name
2016-08-16 16:56:30 +01:00
Erik Johnston
25c2332071
Merge pull request #1010 from matrix-org/erikj/refactor_deletions
...
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
2016-08-16 11:37:53 +01:00
Erik Johnston
a2427981b7
Use cached get_user_by_access_token in slaves
2016-08-16 11:24:32 +01:00
Erik Johnston
6cbd1b495e
Merge branch 'fix_integrity_retry' of https://github.com/Ralith/synapse into Ralith-fix_integrity_retry
2016-08-16 10:50:24 +01:00
Erik Johnston
dc3a00f24f
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
2016-08-15 17:04:39 +01:00
Erik Johnston
89e786bd85
Doc get_next() context manager usage
2016-08-15 13:45:26 +01:00
Erik Johnston
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
Erik Johnston
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
Erik Johnston
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
Erik Johnston
99bbd90b0d
Always run txn.after_callbacks
2016-08-15 09:45:44 +01:00
Benjamin Saunders
8a57cc3123
Add missing database corruption recovery case
...
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
2016-08-14 11:50:22 -07:00
David Baker
b4ecf0b886
Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api
2016-08-11 14:09:13 +01:00
Erik Johnston
a8b946decb
Raise 404 when couldn't find event
2016-08-05 15:31:02 +01:00
Erik Johnston
f0fa66f495
Delete more tables
2016-08-05 10:40:08 +01:00
Benjamin Saunders
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
Erik Johnston
8ad8490cff
Fix typo
2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88
Retry event persistence on IntegrityError
...
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.
This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
2016-08-04 15:02:15 +01:00
Richard van der Hoff
98385888b8
PEP8
2016-08-03 15:42:08 +01:00
Richard van der Hoff
68264d7404
Include device name in /keys/query response
...
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
Richard van der Hoff
530bc862dc
Merge branch 'rav/null_default_device_displayname' into develop
2016-08-03 14:30:32 +01:00
Erik Johnston
80ad710217
Remove other bit of deduplication
2016-08-03 13:25:59 +01:00
Richard van der Hoff
4fec5e57be
Default device_display_name to null
...
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
2016-08-03 11:53:00 +01:00
Erik Johnston
a8a32d2714
Ensure we only persist an event once at a time
2016-08-03 11:23:39 +01:00
Erik Johnston
5aa024e501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader
2016-07-29 11:24:56 +01:00
Mark Haines
8dad08a950
Fix SQL to supply arguments in the same order
2016-07-29 09:57:13 +01:00
Mark Haines
0a7d3cd00f
Create separate methods for getting messages to push
...
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.
The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.
The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Kegan Dougal
370135ad0b
Comment get_unread_push_actions_for_user_in_range function
2016-07-28 16:47:37 +01:00
Erik Johnston
6ede23ff1b
Add more key storage funcs into slave store
2016-07-28 15:41:26 +01:00
Richard van der Hoff
40e539683c
Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload
...
Make the device id on e2e key upload optional
2016-07-27 18:16:57 +01:00
Richard van der Hoff
26cb0efa88
SQL syntax fix
2016-07-27 12:30:22 +01:00
Richard van der Hoff
d47115ff8b
Delete e2e keys on device delete
2016-07-27 12:24:52 +01:00
Richard van der Hoff
2e3d90d67c
Make the device id on e2e key upload optional
...
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.
For active access_tokens without an associated device_id, we ought to register
the device in the devices table.
Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
2016-07-26 23:38:12 +01:00
Mark Haines
87ffd21b29
Fix a couple of bugs in the transaction and keyring code
2016-07-26 19:19:08 +01:00
Richard van der Hoff
33d7776473
Fix typo
2016-07-26 13:32:15 +01:00
Richard van der Hoff
9ad8d9b17c
Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device
2016-07-26 13:29:46 +01:00
Mark Haines
9c4cf83259
Merge pull request #948 from matrix-org/markjh/auth_fixes
...
Don't add rejections to the state_group, persist all rejections
2016-07-26 13:22:57 +01:00
Richard van der Hoff
8e02494166
Delete refresh tokens when deleting devices
2016-07-26 11:10:37 +01:00
Mark Haines
a6f06ce3e2
Fix how push_actions are redacted.
2016-07-26 11:05:39 +01:00
Mark Haines
efeb6176c1
Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically
2016-07-26 10:49:52 +01:00
Richard van der Hoff
012b4c1913
Implement updating devices
...
You can update the displayname of devices now.
2016-07-26 07:35:48 +01:00
Richard van der Hoff
436bffd15f
Implement deleting devices
2016-07-26 07:35:48 +01:00
Mark Haines
1b3c3e6d68
Only update the events and event_json tables for rejected events
2016-07-25 18:44:30 +01:00
Mark Haines
8f7f4cb92b
Don't add the events to forward extremities if the event is rejected
2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874
Don't add rejections to the state_group, persist all rejections
2016-07-25 16:12:16 +01:00
Mark Haines
955ef1f06c
fix: defer.returnValue takes one argument
2016-07-25 16:04:45 +01:00
Richard van der Hoff
2ee4c9ee02
background updates: fix assert again
2016-07-25 16:01:46 +01:00
Richard van der Hoff
9dbd903f41
background updates: Fix assertion to do something
2016-07-25 14:05:23 +01:00
Richard van der Hoff
465117d7ca
Fix background_update tests
...
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00
Richard van der Hoff
363786845b
PEP8
2016-07-22 13:21:07 +01:00
Richard van der Hoff
ec5717caf5
Create index on user_ips in the background
...
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.
I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
2016-07-22 13:16:39 +01:00
Erik Johnston
aede7248ab
Split out a FederationReader process
2016-07-21 17:37:44 +01:00
Richard van der Hoff
c445f5fec7
storage/client_ips: remove some dead code
2016-07-21 11:58:47 +01:00
Richard van der Hoff
7314bf4682
Merge branch 'develop' into rav/get_devices_api
...
(pick up PR #938 in the hope of fixing the UTs)
2016-07-20 17:40:00 +01:00
Richard van der Hoff
bc8f265f0a
GET /devices endpoint
...
implement a GET /devices endpoint which lists all of the user's devices.
It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
Richard van der Hoff
ec041b335e
Record device_id in client_ips
...
Record the device_id when we add a client ip; it's somewhat redundant as we
could get it via the access_token, but it will make querying rather easier.
2016-07-20 16:41:03 +01:00
Richard van der Hoff
053e83dafb
More doc-comments
...
Fix some more comments on some things
2016-07-20 16:40:28 +01:00
David Baker
e967bc86e7
Merge pull request #932 from matrix-org/rav/register_refactor
...
Further registration refactoring
2016-07-20 11:03:33 +01:00
Erik Johnston
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Erik Johnston
f91faf09b3
Comment
2016-07-20 10:18:09 +01:00
Richard van der Hoff
40cbffb2d2
Further registration refactoring
...
* `RegistrationHandler.appservice_register` no longer issues an access token:
instead it is left for the caller to do it. (There are two of these, one in
`synapse/rest/client/v1/register.py`, which now simply calls
`AuthHandler.issue_access_token`, and the other in
`synapse/rest/client/v2_alpha/register.py`, which is covered below).
* In `synapse/rest/client/v2_alpha/register.py`, move the generation of
access_tokens into `_create_registration_details`. This means that the normal
flow no longer needs to call `AuthHandler.issue_access_token`; the
shared-secret flow can tell `RegistrationHandler.register` not to generate a
token; and the appservice flow continues to work despite the above change.
2016-07-19 18:46:19 +01:00
Richard van der Hoff
022b9176fe
schema fix
...
device_id should be text, not bigint.
2016-07-19 11:44:05 +01:00
Richard van der Hoff
f863a52cea
Add device_id support to /login
...
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.
Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.
Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
Erik Johnston
f52565de50
Fix /purge_history bug
...
This was caused by trying to insert duplicate backward extremeties
2016-07-15 14:23:15 +01:00
Erik Johnston
978fa53cc2
Pull out min stream_ordering from ex_outlier_stream
2016-07-15 10:22:30 +01:00
Erik Johnston
eec9609e96
event_backwards_extremeties may not be empty
2016-07-15 10:22:09 +01:00
Erik Johnston
d554ca5e1d
Add support for filters in paginate_room_events
2016-07-14 15:59:04 +01:00
Erik Johnston
b64aa6d687
Add sender and contains_url field to events table
2016-07-14 15:35:43 +01:00
Mark Haines
dfde67a6fe
Add a comment explaining allow_none
2016-07-08 15:57:06 +01:00
Mark Haines
10c843fcfb
Ensure that the guest user is in the database when upgrading accounts
2016-07-08 15:15:55 +01:00
Erik Johnston
58930da52b
Merge branch 'master' of github.com:matrix-org/synapse into develop
2016-07-08 14:11:37 +01:00
Erik Johnston
b92e7955be
Comment
2016-07-07 11:42:15 +01:00
Erik Johnston
c98e1479bd
Return 400 rather than 500
2016-07-07 11:41:07 +01:00
Erik Johnston
67f2c901ea
Add rest servlet. Fix SQL.
2016-07-06 15:56:59 +01:00
Erik Johnston
a17e7caeb7
Merge branch 'erikj/shared_secret' into erikj/test2
2016-07-06 14:46:31 +01:00
Erik Johnston
be3548f7e1
Remove spurious txn
2016-07-05 17:46:51 +01:00
Erik Johnston
4adf93e0f7
Fix for postgres
2016-07-05 17:34:25 +01:00
Erik Johnston
651faee698
Add an admin option to shared secret registration
2016-07-05 17:30:22 +01:00
Erik Johnston
8f8798bc0d
Add ReadWriteLock for pagination and history prune
2016-07-05 15:30:25 +01:00
Mark Haines
dd2ccee27d
Fix typo
2016-07-05 14:06:07 +01:00
Mark Haines
b6b0132ac7
Make get_events_around more efficient on sqlite3
2016-07-05 13:55:18 +01:00
Mark Haines
d44d11d864
Use true/false for boolean parameter inclusive to avoid potential for sqli, and possibly make the code clearer
2016-07-05 10:39:13 +01:00
Erik Johnston
2d21d43c34
Add purge_history API
2016-07-05 10:28:51 +01:00
Mark Haines
0fb76c71ac
Use different SQL for postgres and sqlite3 for when using multicolumn indexes
2016-07-04 19:44:55 +01:00
Erik Johnston
a67bf0b074
Add storage function to purge history for a room
2016-07-04 16:02:50 +01:00
Mark Haines
f18d7546c6
Use a query that postgresql optimises better for get_events_around
2016-07-04 15:48:25 +01:00
Erik Johnston
f328d95cef
Feature: Add deactivate account admin API
...
Allows server admins to "deactivate" accounts, which:
- Revokes all access tokens
- Removes all threepids
- Removes password
The API is a POST to `/admin/deactivate/<user_id>`
2016-06-30 15:40:58 +01:00
Mark Haines
6783534a0f
Merge pull request #886 from matrix-org/markjh/async_commit
...
Optionally make committing to postgres asynchronous.
2016-06-29 15:21:58 +01:00
Erik Johnston
a70688445d
Implement purge_media_cache admin API
2016-06-29 14:57:59 +01:00
Erik Johnston
314b146b2e
Track approximate last access time for remote media
2016-06-29 11:41:20 +01:00
David Baker
aa3a4944d5
more pep8
2016-06-24 11:45:23 +01:00
David Baker
870c45913e
Use similar naming we use in email notifs for push
...
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
David Baker
b5fb7458d5
Actually we need to order these properly
...
otherwise we'll end up returning the wrong 20
2016-06-22 18:07:14 +01:00
David Baker
f73fdb04a6
Style
2016-06-22 17:51:40 +01:00
David Baker
3a4120e49a
Put most recent 20 messages in notif
...
Fixes https://github.com/vector-im/vector-web/issues/1648
2016-06-22 17:47:18 +01:00
Mark Haines
d5fb561709
Optionally make committing to postgres asynchronous.
...
Useful when running tests when you don't care whether the server
will lose data that it claims that it has committed.
2016-06-20 17:53:38 +01:00
Mark Haines
41e4b2efea
Add the create_profile method back since the tests use it
2016-06-17 19:20:47 +01:00
Mark Haines
9f1800fba8
Remove registered_users from the distributor.
...
The only place that was observed was to set the profile. I've made it
so that the profile is set within store.register in the same transaction
that creates the user.
This required some slight changes to the registration code for upgrading
guest users, since it previously relied on the distributor swallowing errors
if the profile already existed.
2016-06-17 19:14:16 +01:00
Erik Johnston
1a815fb04f
Don't hit DB for noop replications queries
2016-06-08 11:33:30 +01:00
Mark Haines
0b2158719c
Remove dead code.
...
Loading push rules now happens in the datastore, so we can remove
the methods that loaded them outside the datastore.
The ``waiting_for_join_list`` in federation handler is populated by
anything, so can be removed.
The ``_get_members_events_txn`` method isn't called from anywhere
so can be removed.
2016-06-07 15:07:11 +01:00
Erik Johnston
84379062f9
Fix AS retries, but with correct ordering
2016-06-07 10:24:50 +01:00
Erik Johnston
310197bab5
Fix AS retries
2016-06-07 09:34:50 +01:00
Erik Johnston
e4134c5e13
Merge pull request #841 from matrix-org/erikj/event_counter
...
Add metric counter for number of persisted events
2016-06-06 14:17:40 +01:00
Erik Johnston
7aa778fba9
Add metric counter for number of persisted events
2016-06-06 11:58:09 +01:00
Erik Johnston
70aee0717c
Add events to cache when we persist them
2016-06-06 11:34:53 +01:00
Erik Johnston
cffe46408f
Don't rely on options when inserting event into cache
2016-06-03 18:25:21 +01:00
Erik Johnston
10ea3f46ba
Change the way we cache events
2016-06-03 17:57:50 +01:00
Erik Johnston
05e01f21d7
Remove event fetching from DB threads
2016-06-03 17:22:13 +01:00
Mark Haines
eef541a291
Move insert_client_ip to a separate class
2016-06-03 14:42:35 +01:00
Erik Johnston
d06f2a229e
Merge pull request #828 from matrix-org/erikj/joined_hosts_for_room
...
Make get_joined_hosts_for_room use get_users_in_room
2016-06-03 11:50:30 +01:00
Erik Johnston
722472b48c
Merge pull request #825 from matrix-org/erikj/cache_push_rules
...
Load push rules in storage layer so that they get cached
2016-06-03 11:44:32 +01:00
Mark Haines
b09f348530
Merge pull request #824 from matrix-org/markjh/slaved_presence_store
...
Add a slaved store for presence
2016-06-03 11:26:33 +01:00
Erik Johnston
ccb56fc24b
Make get_joined_hosts_for_room use get_users_in_room
2016-06-03 11:20:23 +01:00
Erik Johnston
6a0afa582a
Load push rules in storage layer, so that they get cached
2016-06-03 11:10:00 +01:00
Mark Haines
3ae915b27e
Add a slaved store for presence
2016-06-03 11:05:53 +01:00
Erik Johnston
9c26b390a2
Only get local users
2016-06-03 11:04:31 +01:00
Erik Johnston
065e739d6e
Merge pull request #811 from matrix-org/erikj/state_users_in_room
...
Use state to calculate get_users_in_room
2016-06-03 10:58:27 +01:00
Erik Johnston
696d7c5937
Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room
...
Add get_users_with_read_receipts_in_room cache
2016-06-03 10:58:24 +01:00
Mark Haines
80f34d7b57
Fix setting the _clock in SQLBaseStore
2016-06-02 15:23:56 +01:00
Erik Johnston
a04a2d043c
Merge pull request #807 from matrix-org/erikj/push_rules_cache
...
Ensure we always return boolean in push rules
2016-06-01 18:07:48 +01:00
Erik Johnston
0f06b496d1
Merge pull request #806 from matrix-org/erikj/hash_cache
...
Cache get_event_reference_hashes
2016-06-01 18:07:42 +01:00
Erik Johnston
c8285564a3
Use state to calculate get_users_in_room
2016-06-01 15:25:25 +01:00
David Baker
d60eed0710
Limit number of notifications in an email notification
2016-06-01 11:45:43 +01:00
Erik Johnston
43db0d9f6a
Add get_users_with_read_receipts_in_room cache
2016-06-01 10:54:32 +01:00
Erik Johnston
dea9f20f8c
Force boolean
2016-05-31 17:24:30 +01:00
Mark Haines
c8c5bf950a
Fix synapse/storage/schema/delta/30/as_users.py
2016-05-31 17:10:40 +01:00
Erik Johnston
c9ca285d33
Merge pull request #805 from matrix-org/erikj/push_rules_cache
...
Fix GET /push_rules
2016-05-31 16:42:21 +01:00
Erik Johnston
1d4ee854e2
Fix typo
2016-05-31 15:45:53 +01:00
Erik Johnston
cca0093fa9
Change fix
2016-05-31 15:44:08 +01:00
Erik Johnston
aefd2d1cbc
Cache get_event_reference_hashes
2016-05-31 15:32:32 +01:00
Erik Johnston
10de8c2631
Merge pull request #804 from matrix-org/erikj/push_rules_cache
...
Add caches to bulk_get_push_rules*
2016-05-31 15:04:40 +01:00
Erik Johnston
e5b0bbcd33
Add caches to bulk_get_push_rules*
2016-05-31 13:46:58 +01:00
Erik Johnston
85b992f621
Fix to allow start with postgres
2016-05-27 10:44:44 +01:00
David Baker
37b7e84620
Include the ts the notif was received at
2016-05-24 11:33:32 +01:00
David Baker
a24bc5b2dc
Add GET /notifications API
2016-05-23 18:33:51 +01:00
David Baker
31b5395ab6
Remove debug logging
2016-05-23 16:32:01 +01:00
David Baker
c2da3406fc
Oops, missing comma
2016-05-20 18:03:31 +01:00
David Baker
ccffb0965d
Remove stale line
2016-05-20 17:59:10 +01:00
David Baker
18d68bfee4
Handle empty events table
2016-05-20 17:58:09 +01:00
David Baker
d4503e25ed
Make deleting push actions more efficient
...
There's no index on received_ts, so manually binary search using the stream_ordering index, and only update it once an hour.
2016-05-20 17:56:10 +01:00
David Baker
149fa411e2
Only delete push actions after 30 days
2016-05-20 15:25:12 +01:00
Erik Johnston
42368ea8db
Add desc to get_presence_for_users
2016-05-18 11:38:10 +01:00
Mark Haines
6a30a0bfd3
Move the functions for parsing app service config
2016-05-17 11:28:58 +01:00
Mark Haines
816df9f267
get_room_members is unused now
2016-05-16 19:51:43 +01:00
Mark Haines
1a3a2002ff
Spell "domain" correctly
...
s/domian/domain/g
2016-05-16 19:17:23 +01:00
Mark Haines
b7381d5338
Allow receipts for events we haven't seen in the db
2016-05-13 15:46:41 +01:00
Erik Johnston
ba5c616ff4
Merge pull request #778 from matrix-org/erikj/add_pusher
...
Fixup add_pusher
2016-05-13 14:43:23 +01:00
Erik Johnston
0c11c1be88
Spelling
2016-05-13 14:42:25 +01:00
Erik Johnston
e00e8f2166
Merge pull request #769 from matrix-org/erikj/push_actions_delete
...
Delete old pushers
2016-05-13 14:41:36 +01:00
Erik Johnston
fd8e921b6e
Merge pull request #779 from matrix-org/erikj/receipts
...
Use tree cache for get_linearized_receipts_for_room
2016-05-13 14:41:21 +01:00
Erik Johnston
c9aff0736c
Remove topics table
2016-05-13 14:40:38 +01:00
Mark Haines
9295fa30a8
Annotate the removed indicies with why they were removed.
2016-05-13 14:16:57 +01:00
Erik Johnston
5e50058473
Remove unused indices
...
This includes removing both unused indices and indices that are subsets
of other indices.
2016-05-13 13:28:07 +01:00
Erik Johnston
6da7f39d95
Use tree cache for get_linearized_receipts_for_room
2016-05-13 11:41:23 +01:00
Erik Johnston
13d37c3c56
Fixup add_pusher
2016-05-13 11:25:02 +01:00
Matrix
3b1930e8ec
unbreak schema
2016-05-10 16:42:37 +01:00
Matthew Hodgson
fe97b81c09
Merge pull request #759 from matrix-org/dbkr/email_notifs
...
Send email notifications for missed messages
2016-05-10 16:30:05 +02:00
David Baker
997db04648
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-05-10 14:40:19 +02:00
Erik Johnston
c9eb6dfc1b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ignore_user
2016-05-09 13:21:06 +01:00
Erik Johnston
3f84da139c
Merge pull request #773 from matrix-org/erikj/get_domian_from_id
...
Add and use get_domain_from_id
2016-05-09 13:21:00 +01:00
Erik Johnston
def64d6ef3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ignore_user
2016-05-09 13:05:09 +01:00
Erik Johnston
8715731559
Merge pull request #772 from matrix-org/erikj/get_user_cache
...
Add cache to get_user_by_id
2016-05-09 11:12:11 +01:00
Erik Johnston
08dfa8eee2
Add and use get_domian_from_id
2016-05-09 10:36:03 +01:00
Erik Johnston
4ea762c1a2
Add cache to get_user_by_id
2016-05-09 10:08:21 +01:00
Erik Johnston
012cb5416c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/push_actions_delete
2016-05-06 15:59:20 +01:00
Erik Johnston
fcb2c3f0db
Remove unused import
2016-05-06 15:47:40 +01:00
Erik Johnston
fd85b167ec
Pull loop one level up
2016-05-06 15:38:42 +01:00
Erik Johnston
b6e0be701e
Queue events for persistence
2016-05-06 14:31:44 +01:00
Erik Johnston
d13459636f
Pull prev txn from in memory
2016-05-06 11:30:55 +01:00
Erik Johnston
1d275dba69
Don't needlessly enter transaction
2016-05-06 11:25:58 +01:00
Erik Johnston
56b5e83e36
Reduce database inserts when sending transactions
2016-05-06 11:20:18 +01:00
Mark Haines
9c272da05f
Add an openidish mechanism for proving to third parties that you own a given user_id
2016-05-05 13:42:44 +01:00
Erik Johnston
1f0f5ffa1e
Add bulk fetch storage API
2016-05-05 10:03:15 +01:00
Erik Johnston
fee1118a20
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ignore_user
2016-05-04 19:08:27 +01:00
Erik Johnston
fcd1eb642d
Add primary key
2016-05-04 16:51:51 +01:00
Erik Johnston
8e6a163f27
Add timestamp and auto incrementing ID
2016-05-04 15:19:12 +01:00
David Baker
80be396464
Correct SQL statement for postgres
...
In standard sql, join binds tighter than comma, so we were joining on the wrong table. Postgres follows the standard (apparently).
2016-05-04 13:19:59 +01:00
Erik Johnston
984d4a2c0f
Add /report endpoint
2016-05-04 11:28:10 +01:00
Erik Johnston
a438a6d2bc
Implement basic ignore user
2016-05-04 10:16:46 +01:00
Erik Johnston
183f23f10d
Delete old pushers
2016-05-03 14:22:33 +01:00
David Baker
35b7b8e4bc
Remove unused function
2016-04-29 20:10:34 +01:00
David Baker
b0a1036d93
Use explicit join
2016-04-29 19:28:56 +01:00
David Baker
8f99cd5996
Oops, actually specify the user id
2016-04-29 19:27:03 +01:00
David Baker
60f44c098d
Remove unnecessary if
2016-04-29 19:17:10 +01:00
David Baker
50ad8005e4
Put spaces at start of line
2016-04-29 19:16:15 +01:00
David Baker
acded821c4
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-04-29 10:05:20 +01:00
Mark Haines
871357d539
Check that somethign has happend before running the selects
2016-04-27 11:54:13 +01:00
Erik Johnston
183cacac90
Simplify query and handle finishing correctly
2016-04-22 10:01:57 +01:00
Erik Johnston
ae571810f2
Order NULLs first
2016-04-21 18:14:18 +01:00
Erik Johnston
3ddbb1687c
Fix query
2016-04-21 18:02:36 +01:00
Erik Johnston
8fae3d7b1e
Use special UPDATE syntax
2016-04-21 18:01:49 +01:00
Erik Johnston
b57dcb4b51
Typo
2016-04-21 17:49:00 +01:00
Erik Johnston
26db18bc90
Need to do _background_update_progress_txn in actual transaction
2016-04-21 17:45:56 +01:00
Erik Johnston
e395eb1108
Update progress when creating index
2016-04-21 17:39:24 +01:00
Erik Johnston
3b0fa77f50
Fix SQL statement
2016-04-21 17:37:42 +01:00
Erik Johnston
129e403487
Create index must be on a conn
2016-04-21 17:35:51 +01:00
Erik Johnston
51bb339ab2
Create index concurrently
2016-04-21 17:16:11 +01:00
Erik Johnston
b743c1237e
Add missing run_upgrade
2016-04-21 17:12:04 +01:00
Erik Johnston
c877f0f034
Optimise event_search in postgres
2016-04-21 16:56:14 +01:00
David Baker
f63bd4ff47
Send a rather basic email notif
...
Also pep8 fixes
2016-04-20 13:02:01 +01:00
Erik Johnston
61c7edfd34
Add cache to _get_state_groups_from_groups
2016-04-19 17:22:03 +01:00
David Baker
07d765209d
First bits of emailpusher
...
Mostly logic of when to send an email
2016-04-19 14:24:36 +01:00
Erik Johnston
d213d69fe3
Add desc arg
2016-04-14 11:36:23 +01:00
Erik Johnston
96bcfb29c7
Add index
2016-04-14 11:26:33 +01:00
Erik Johnston
a2546b9082
Fix query for get_unread_push_actions_for_user_in_range
2016-04-14 11:08:31 +01:00
David Baker
767fc0b739
pep8
2016-04-13 14:23:27 +01:00
David Baker
54d08c8868
Only return unread notifications
...
Make get_unread_push_actions_for_user_in_range only return unread event actions, being more true to its name. Done in two separate sql queries to get actions after a read receipt and those in a room wiht no receipt at all. SQL queries by Erik.
2016-04-13 14:16:45 +01:00
David Baker
d33d623f0d
Merge pull request #716 from matrix-org/dbkr/get_pushers
...
Add get endpoint for pushers
2016-04-12 14:40:37 +01:00
David Baker
7984ffdc6a
Unneccessarywhitespaceisunnecessary
2016-04-12 13:55:57 +01:00
David Baker
c1267d04c5
Oops, forgot the desc.
2016-04-12 13:55:32 +01:00
David Baker
44891b4a0a
Tidy up get_pusher functions
...
Decodes pushers rows on the main thread rather than the db thread and uses _simple_select_list. Also do the same to the function I copied and factor out the duplication into a helper function.
2016-04-12 13:47:17 +01:00
David Baker
7b39bcdaae
Mis-named function
2016-04-12 13:35:08 +01:00
Erik Johnston
762ada1e07
Add back backfilled parameter that was removed
2016-04-12 11:58:04 +01:00
Erik Johnston
cccf86dd05
Check if we've already backfilled events
2016-04-12 11:19:32 +01:00
David Baker
8a76094965
Add get endpoint for pushers
...
As per https://github.com/matrix-org/matrix-doc/pull/308
2016-04-11 18:00:03 +01:00
David Baker
2547dffccc
Merge pull request #705 from matrix-org/dbkr/pushers_use_event_actions
...
Change pushers to use the event_actions table
2016-04-11 12:58:55 +01:00
Matthew Hodgson
4bd3d25218
Merge pull request #688 from matrix-org/matthew/preview_urls
...
URL previewing support
2016-04-11 10:40:29 +01:00
Matthew Hodgson
6ff7a79308
move local_media_repository_url_cache.sql to schema v31
2016-04-08 19:09:02 +01:00
Matthew Hodgson
2460d904bd
fix error checking for new SQL
2016-04-08 19:04:29 +01:00
Matthew Hodgson
fb83f6a1fc
fix SQL based on PR feedback
2016-04-08 18:55:38 +01:00
David Baker
ed3979df5f
Fix invite pushes
...
* If the event is an invite event, add the invitee to list of user we run push rules for (if they have a pusher etc)
* Move invite_for_me to be higher prio than member events otherwise member events matches them
* Spell override right
2016-04-08 15:29:59 +01:00
Mark Haines
7e2f971c08
Remove some unused functions ( #711 )
...
* Remove some unused functions
* get_room_events_stream is only used in tests
* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
David Baker
d9f38561c8
Literally a dictionary
2016-04-07 17:45:01 +01:00
David Baker
4836864f56
generate id in the main thread
2016-04-07 17:38:48 +01:00
David Baker
3fb35cbd6f
Oops, inequality fail
2016-04-07 17:33:37 +01:00
Matthew Hodgson
d6e7333ae4
Merge branch 'develop' into matthew/preview_urls
2016-04-07 17:26:44 +01:00
Mark Haines
ceb599e789
Add tests for redactions
2016-04-07 16:52:07 +01:00
David Baker
05d044aac3
pep8
2016-04-07 16:45:38 +01:00
David Baker
2d5c693fd3
Fix port script for changes merged from develop
2016-04-07 16:43:54 +01:00
David Baker
9c99ab4572
Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions
2016-04-07 16:35:22 +01:00
David Baker
92e3071623
Send badge count pushes.
...
Also fix bugs with retrying.
2016-04-07 15:39:53 +01:00
David Baker
0fd1cd2400
pep8
2016-04-06 16:50:47 +01:00
Mark Haines
e815763b7f
Merge pull request #697 from matrix-org/markjh/slaveI
...
Add a slaved events store class
2016-04-06 16:19:25 +01:00
David Baker
7e2c89a37f
Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do:
...
* Make badges work again
* Remove old, unused code
2016-04-06 15:42:15 +01:00
Erik Johnston
b713934b2e
Merge pull request #698 from matrix-org/erikj/port_script_fix
...
Don't require config to create database
2016-04-06 14:32:45 +01:00
Mark Haines
75fb9ac1be
Add a slaved events store class
...
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
2016-04-06 14:18:35 +01:00
Erik Johnston
8aab9d87fa
Don't require config to create database
2016-04-06 14:15:45 +01:00
Mark Haines
7d11f825aa
Merge pull request #694 from matrix-org/markjh/caches
...
Move _get_cache_dict into the SQLBaseStore
2016-04-06 13:21:25 +01:00
Mark Haines
87f2dec8d4
Make the cache objects be per instance rather than being global
2016-04-06 13:08:05 +01:00
Mark Haines
a1e0d316ea
Move _get_cache_dict into the SQLBaseStore
2016-04-06 13:05:19 +01:00
Erik Johnston
b29f98377d
Merge pull request #691 from matrix-org/erikj/member
...
Fix stuck invites
2016-04-05 12:44:39 +01:00
Erik Johnston
df727f2126
Fix stuck invites
...
If rejecting a remote invite fails with an error response don't fail
the entire request; instead mark the invite as locally rejected.
This fixes the bug where users can get stuck invites which they can
neither accept nor reject.
2016-04-05 11:13:24 +01:00
Erik Johnston
7a77f8b6d5
Merge pull request #690 from matrix-org/erikj/member
...
Store invites in a separate table.
2016-04-05 09:12:27 +01:00
Erik Johnston
0c53d750e7
Docs and indents
2016-04-04 18:02:48 +01:00
Erik Johnston
92ab45a330
Add upgrade path, rename table
2016-04-04 17:07:43 +01:00
Erik Johnston
3d76b7cb2b
Store invites in a separate table.
2016-04-04 16:30:15 +01:00
Erik Johnston
bf14883a04
Merge pull request #689 from matrix-org/erikj/member
...
Do checks for memberships before creating events
2016-04-04 11:56:40 +01:00
Matthew Hodgson
9f7dc2bef7
Merge branch 'develop' into matthew/preview_urls
2016-04-04 00:38:21 +01:00
Matthew Hodgson
8b98a7e8c3
pep8
2016-04-03 12:56:29 +01:00
Matthew Hodgson
7426c86eb8
add a persistent cache of URL lookups, and fix up the in-memory one to work
2016-04-03 00:31:57 +01:00
Erik Johnston
d76d89323c
Use computed prev event ids
2016-04-01 17:39:32 +01:00
Mark Haines
89e6839a48
Merge pull request #686 from matrix-org/markjh/doc_strings
...
Use google style doc strings.
2016-04-01 16:20:09 +01:00
Mark Haines
2a37467fa1
Use google style doc strings.
...
pycharm supports them so there is no need to use the other format.
Might as well convert the existing strings to reduce the risk of
people accidentally cargo culting the wrong doc string format.
2016-04-01 16:12:07 +01:00
Mark Haines
f2b916534b
Merge pull request #684 from matrix-org/markjh/backfill_id_gen
...
Use a stream id generator for backfilled ids
2016-04-01 15:13:14 +01:00
Mark Haines
9bc5b4c663
Assert that the step != 0
2016-04-01 15:08:20 +01:00
Mark Haines
35b5c4ba1b
use google style doc strings
2016-04-01 15:07:01 +01:00
Mark Haines
a2866e2e6a
Rename direction to step, apply checks consistently
2016-04-01 13:50:54 +01:00
Mark Haines
e36bfbab38
Use a stream id generator for backfilled ids
2016-04-01 13:29:05 +01:00
Mark Haines
7753fc6570
Fix the invalidation of the names and aliases cache
2016-04-01 10:34:51 +01:00
Mark Haines
5d06929169
Move the check for backfilled outside the for loop
2016-03-31 15:09:09 +01:00
Mark Haines
76503f95ed
Remove the is_new_state argument to persist event.
...
Move the checks for whether an event is new state inside persist
event itself.
This was harder than expected because there wasn't enough information
passed to persist event to correctly handle invites from remote servers
for new rooms.
2016-03-31 15:00:42 +01:00
Mark Haines
62e395f0e3
Merge pull request #676 from matrix-org/markjh/replicate_stateIII
...
Add replication streams for ex outliers and current state resets
2016-03-31 11:20:57 +01:00
Mark Haines
2ec5426035
Use a namedtuple rather than tuple unpacking
2016-03-31 10:33:02 +01:00
Mark Haines
1fbb094c6f
Add replication streams for ex outliers and current state resets
2016-03-30 17:19:56 +01:00
Mark Haines
8b8052909f
return the state_group for backfill
2016-03-30 16:20:07 +01:00
Mark Haines
61407986b4
Add a entry to current_state_resets table when the current state is reset
2016-03-30 16:18:46 +01:00
Mark Haines
31a9eceda5
Add a replication stream for state groups
2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6
Use a stream id generator to assign state group ids
2016-03-30 12:55:02 +01:00
Erik Johnston
2f0180b09e
Don't bother interning keys that are already interned
2016-03-23 16:29:46 +00:00
Erik Johnston
acdfef7b14
Intern all the things
2016-03-23 16:25:54 +00:00
Erik Johnston
75daede92f
String intern
2016-03-23 14:53:53 +00:00
Erik Johnston
fbdeb1778d
Merge pull request #664 from matrix-org/erikj/public_room_list
...
Don't require alias in public room list.
2016-03-23 14:42:01 +00:00
Erik Johnston
b275765545
Comment about weird SQL
2016-03-23 14:15:32 +00:00
Erik Johnston
0c1a27b787
SQLite and postgres doesn't share a true literal
2016-03-23 14:10:49 +00:00
Erik Johnston
84afeb41f3
Ensure all old public rooms have aliases
2016-03-23 13:59:34 +00:00
Erik Johnston
2749da542c
Merge pull request #663 from matrix-org/erikj/invite_for_user
...
Make get_invites return RoomsForUser
2016-03-23 13:19:26 +00:00
Erik Johnston
e14baa7a3b
Merge pull request #665 from matrix-org/erikj/dont_cache_events
...
Only cache events in the event cache
2016-03-23 13:19:12 +00:00
Erik Johnston
d87a846ebc
Don't cache events in get_recent_events_for_room
2016-03-23 11:42:50 +00:00
Erik Johnston
8b0dfc9fc4
Don't cache events in get_current_state_for_key
2016-03-23 11:42:17 +00:00
Erik Johnston
b6507869cd
Make get_invites return RoomsForUser
2016-03-23 10:32:10 +00:00
Erik Johnston
99f929f36b
Make StateHandler._state_cache only store event_ids.
2016-03-22 16:06:04 +00:00
Erik Johnston
6cf0ba1466
Bump get_unread_event_push_actions_by_room_for_user cache
2016-03-22 14:18:21 +00:00
Erik Johnston
76d18a5776
Bump get_aliases_for_room cache
2016-03-22 14:08:13 +00:00
Erik Johnston
fa2f96c2e3
Merge pull request #660 from matrix-org/erikj/state_cache
...
Don't cache events in _state_group_cache
2016-03-22 13:12:06 +00:00
Erik Johnston
f93304e77f
Merge pull request #659 from matrix-org/erikj/state_cache_factor
...
Make stateGroupCache honour CACHE_SIZE_FACTOR
2016-03-22 13:12:01 +00:00
Erik Johnston
2c86187a1b
Don't cache events in _state_group_cache
...
Instead, simply cache the event ids, relying on the event cache to cache
the actual events.
The problem was that while the state groups cache was limited in the
number of groups it could hold, each individual group could consist of
thousands of events.
2016-03-22 12:00:09 +00:00
Erik Johnston
d6ac752538
Merge pull request #657 from matrix-org/erikj/roomlist
...
Add published room list edit API
2016-03-22 11:57:39 +00:00
Erik Johnston
b591277620
Make stateGroupCache honour CACHE_SIZE_FACTOR
2016-03-22 10:32:50 +00:00
Mark Haines
5244c0b48e
Remove unused backfilled parameter from persist_event
2016-03-21 18:06:08 +00:00
Erik Johnston
3e7fac0d56
Add published room list edit API
2016-03-21 15:06:07 +00:00
Erik Johnston
67ed8065db
Dedupe requested event list in _get_events
2016-03-18 14:31:31 +00:00
Mark Haines
673c96ce97
Remove dead code left over from presence changes
2016-03-17 11:03:47 +00:00
Mark Haines
ba660ecde2
Add a comment to offer a hint to an explanation for why we have a unique constraint on (app_id, pushkey, user_id)
2016-03-16 10:35:00 +00:00
Mark Haines
ee32d622ce
Fix a couple of errors when deleting pushers
2016-03-15 17:47:36 +00:00
Mark Haines
b6e8420aee
Add replication stream for pushers
2016-03-15 17:33:10 +00:00
Erik Johnston
15122da0e2
Thats not how transactions work.
2016-03-11 16:45:27 +00:00
Erik Johnston
b13035cc91
Implement logout
2016-03-11 16:27:50 +00:00
Erik Johnston
c081228439
Fix SQL statement
2016-03-11 15:09:17 +00:00
Erik Johnston
b5afe6bc38
Merge pull request #641 from matrix-org/dbkr/fix_change_password
...
Fix logout on password change
2016-03-11 14:48:20 +00:00
David Baker
2dee03aee5
more pep8
2016-03-11 14:38:23 +00:00
David Baker
af59826a2f
Make select more sensible when dseleting access tokens, rename pusher deletion to match access token deletion and make exception arg optional.
2016-03-11 14:34:09 +00:00
David Baker
f523177850
Delete old, unused methods and rename new one to just be user_delete_access_tokens with an except_token_ids argument doing what it says on the tin.
2016-03-11 14:29:01 +00:00
David Baker
57c444b3ad
Dear PyCharm, please indent sensibly for me. Thx.
2016-03-11 14:25:05 +00:00
Erik Johnston
58443a022d
Merge pull request #635 from matrix-org/erikj/sync_order
...
Use topological orders for initial sync timeline
2016-03-11 13:17:49 +00:00
David Baker
aa11db5f11
Fix cache invalidation so deleting access tokens (which we did when changing password) actually takes effect without HS restart. Reinstate the code to avoid logging out the session that changed the password, removed in 415c2f0549
2016-03-11 13:14:18 +00:00
Daniel Wagner-Hall
465605d616
Store appservice ID on register
2016-03-10 15:58:22 +00:00
Erik Johnston
9669a99d1a
Update users table in a batched manner
2016-03-10 15:12:19 +00:00
Erik Johnston
8a88684736
Add comment
2016-03-09 16:51:22 +00:00
Erik Johnston
af2fe6110c
Return the correct token form
2016-03-09 16:11:53 +00:00
Erik Johnston
3ecaabc7fd
Use topological orders for initial sync timeline
2016-03-09 15:45:34 +00:00
Erik Johnston
158a322e82
Ensure integer is an integer
2016-03-09 10:20:48 +00:00
Mark Haines
5f5817ab05
Merge pull request #632 from matrix-org/markjh/py3v2
...
Fix relative imports so they work in both py3 and py27
2016-03-08 18:11:10 +00:00
Daniel Wagner-Hall
3b97797c8d
Merge branch 'develop' into daniel/ick
2016-03-08 17:35:09 +00:00
Daniel Wagner-Hall
edca2d9891
Idempotent-ise schema update script
...
If any ASes don't have an ID, the schema will fail, and then it will
error when trying to add the column again.
2016-03-08 17:32:29 +00:00
Mark Haines
7076082ae6
Fix relative imports so they work in both py3 and py27
2016-03-08 11:45:50 +00:00
Mark Haines
9848b54cac
Prefill from the correct stream
2016-03-04 16:20:22 +00:00
Mark Haines
deda48068c
prefill the push rules stream change cache
2016-03-04 16:19:42 +00:00
Mark Haines
ebcbb23226
s/stream_ordering/event_stream_ordering/ in push
2016-03-04 16:15:23 +00:00
Mark Haines
7e9fc9b6af
/FNARG/d
2016-03-04 15:54:09 +00:00
Mark Haines
ec7460b4f2
Merge branch 'develop' into markjh/pushrule_stream
2016-03-04 14:44:34 +00:00
Mark Haines
1b4f4a936f
Hook up the push rules stream to account_data in /sync
2016-03-04 14:44:01 +00:00
Erik Johnston
dd463e246d
Merge pull request #614 from matrix-org/erikj/alias_delete
...
Allow alias creators to delete aliases
2016-03-04 09:02:33 +00:00
Daniel Wagner-Hall
95481e7ba7
Merge pull request #571 from matrix-org/daniel/asids
...
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
Mark Haines
2223204eba
Hook push rules up to the replication API
2016-03-02 17:26:20 +00:00
Mark Haines
a1cf9e3bf3
Add a stream for push rule updates
2016-03-01 18:16:37 +00:00
Mark Haines
a612ce6659
Merge pull request #489 from matrix-org/markjh/replication
...
Add a /replication API for extracting the updates that happened on synapse.
2016-03-01 15:08:24 +00:00
Mark Haines
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Erik Johnston
f9af8962f8
Allow alias creators to delete aliases
2016-03-01 14:46:31 +00:00
Mark Haines
54172924c8
Load the current id in the IdGenerator constructor
...
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.
It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
2016-03-01 14:32:56 +00:00
Mark Haines
de27f7fc79
Add support for changing the actions for default rules
...
See matrix-org/matrix-doc#283
Works by adding dummy rules to the push rules table with a negative priority class and then using those rules to clobber the default rule actions when adding the default rules in ``list_with_base_rules``
2016-02-26 14:28:19 +00:00
Daniel Wagner-Hall
33300673b7
Generate guest access token on 3pid invites
...
This means that following the same link across multiple sessions or
devices can re-use the same guest account.
Note that this is somewhat of an abuse vector; we can't throw up
captchas on this flow, so this is a way of registering ephemeral
accounts for spam, whose sign-up we don't rate limit.
2016-02-24 14:41:25 +00:00
Erik Johnston
6451fcd085
Create a new stream_id per presence update
2016-02-23 15:51:39 +00:00
Erik Johnston
e5ad2e5267
Merge pull request #582 from matrix-org/erikj/presence
...
Rewrite presence for performance.
2016-02-19 09:37:50 +00:00
Erik Johnston
e6c5e3f28a
Close cursor
2016-02-18 16:39:28 +00:00
Erik Johnston
42109a62a4
Remove unused param from get_max_token
2016-02-18 16:37:28 +00:00
Mark Haines
b9977ea667
Remove dead code for setting device specific rules.
...
It wasn't possible to hit the code from the API because of a typo
in parsing the request path. Since no-one was using the feature
we might as well remove the dead code.
2016-02-18 16:05:13 +00:00
Erik Johnston
112283e230
Prefix TS fields with _ts
2016-02-18 10:11:43 +00:00
Daniel Wagner-Hall
7e90fb6a57
Merge branch 'develop' into daniel/roomcleanupincremental
...
Conflicts:
synapse/rest/client/v1/room.py
2016-02-17 15:53:59 +00:00
Erik Johnston
e5999bfb1a
Initial cut
2016-02-17 15:40:50 +00:00
Daniel Wagner-Hall
a4e278bfe7
Respond to federated invite with non-empty context
...
Currently, we magically perform an extra database hit to find the
inviter, and use this to guess where we should send the event. Instead,
fill in a valid context, so that other callers relying on the context
actually have one.
2016-02-17 15:25:12 +00:00
Mark Haines
a9c9868957
Make adding push rules idempotent
...
Also remove the **kwargs from the add_push_rule method.
Fixes https://matrix.org/jira/browse/SYN-391
2016-02-16 15:53:38 +00:00
Daniel Wagner-Hall
763360594d
Mark AS users with their AS's ID
2016-02-11 17:26:42 +00:00
Erik Johnston
ce14c7a995
Fix SYN-627, events are in incorrect room in /sync
2016-02-11 15:02:56 +00:00
Erik Johnston
0eff740523
Return events in correct order for /events
2016-02-11 10:07:27 +00:00
Erik Johnston
5189bfdef4
Batch fetch _get_state_groups_from_groups
2016-02-10 13:24:42 +00:00
Erik Johnston
24f00a6c33
Use _simple_select_many for _get_state_group_for_events
2016-02-10 12:57:50 +00:00
Erik Johnston
f7ef5c1d57
Merge pull request #568 from matrix-org/erikj/unread_notif
...
Atomically persit push actions when we persist the event
2016-02-10 11:25:32 +00:00
Erik Johnston
0214745239
Rename functions
2016-02-10 11:09:56 +00:00
Erik Johnston
7b0d846407
Atomically persit push actions when we persist the event
2016-02-09 16:19:15 +00:00
Erik Johnston
78d6c1b5be
Change a log from debug to info
2016-02-09 14:44:12 +00:00
Erik Johnston
70a8608749
Invalidate get_last_receipt_event_id_for_user cache
2016-02-09 14:27:29 +00:00
Erik Johnston
eff12e838c
Don't load all ephemeral state for a room on every sync
2016-02-09 13:55:59 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
79a1c0574b
Allocate guest user IDs numericcally
...
The current random IDs are ugly and confusing when presented in UIs.
This makes them prettier and easier to read.
Also, disable non-automated registration of numeric IDs so that we don't
need to worry so much about people carving out our automated address
space and us needing to keep retrying ID registration.
2016-02-05 11:22:30 +00:00
Erik Johnston
4d36e73230
Actually return something sensible
2016-02-03 16:35:00 +00:00
Erik Johnston
aa4af94c69
We return dicts now.
2016-02-03 16:29:32 +00:00
Erik Johnston
b84d59c5f0
Add descriptions
2016-02-03 16:22:35 +00:00
Erik Johnston
8450114098
Merge pull request #554 from matrix-org/erikj/event_push
...
Change event_push_actions_rm_tokens schema
2016-02-03 15:02:47 +00:00
Erik Johnston
f8aae79a72
Simplify get_rooms
2016-02-03 13:24:35 +00:00
Erik Johnston
6f52e90065
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_room_fix
2016-02-03 11:06:29 +00:00
Erik Johnston
771528ab13
Change event_push_actions_rm_tokens schema
2016-02-03 10:50:49 +00:00
Erik Johnston
b32121a5d1
Unused import
2016-02-03 10:30:56 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Erik Johnston
8a391e33ae
s/get_room_changes_for_user/get_membership_changes_for_user/
2016-02-02 16:12:10 +00:00
Erik Johnston
477b1ed6cf
Fetch events in a separate transaction.
...
This has a couple of benefits:
- It reduces the time of transactions, allowing other database requests
to run.
- Fetching events is given a dedicated database thread, and so can't
starve other database requests.
2016-02-02 15:58:14 +00:00
Erik Johnston
65e92eca49
Change the way we do public room list fetching
2016-02-02 15:21:10 +00:00
Erik Johnston
89b40b225c
Order things correctly
2016-02-01 16:32:46 +00:00
Erik Johnston
4bf448be25
Switch over /events to use per room caches
2016-02-01 16:26:51 +00:00
Erik Johnston
2d3837bec7
Merge pull request #543 from matrix-org/erikj/sync
...
Cache get_room_changes_for_user
2016-02-01 15:05:06 +00:00
Erik Johnston
ceb6b8680a
Only use room_ids if in get_room_events_stream if is_guest
2016-02-01 10:33:52 +00:00
Erik Johnston
25c311eaf6
Cache get_room_changes_for_user
2016-01-29 16:52:48 +00:00
Erik Johnston
cc9c97e0dc
Invalidate _account_data_stream_cache correctly
2016-01-29 16:41:51 +00:00
Erik Johnston
8da95b6f1b
Comment. Remove superfluous order by
2016-01-29 15:39:17 +00:00
Erik Johnston
b5dbced938
Don't prefill account data
2016-01-29 14:53:59 +00:00
Erik Johnston
3d60686c0c
Actually use cache
2016-01-29 14:49:11 +00:00
Erik Johnston
45488e0ffa
Max is not a function
2016-01-29 14:42:01 +00:00
Erik Johnston
f67d60496a
Convert param style
2016-01-29 14:41:16 +00:00
Erik Johnston
18579534ea
Prefill stream change caches
2016-01-29 14:37:59 +00:00
Erik Johnston
ebc5f00efe
Bump AccountDataAndTagsChangeCache size
2016-01-29 13:37:40 +00:00
Erik Johnston
b18114e19e
Merge pull request #536 from matrix-org/erikj/sync
...
Make /sync "better".
2016-01-29 13:04:51 +00:00
Erik Johnston
467c27a1f9
Amalgamate tags and account data stream caches
2016-01-28 18:20:00 +00:00
Erik Johnston
03b2c2577c
Don't use defer.returnValue
2016-01-28 17:29:24 +00:00
Erik Johnston
8fe8951a8d
Cache filters
2016-01-28 17:09:09 +00:00
Erik Johnston
fdca8ec418
Add events index
2016-01-28 16:41:59 +00:00
Erik Johnston
45cf827c8f
Change name and doc has_entity_changed
2016-01-28 16:39:18 +00:00
Erik Johnston
00cb3eb24b
Cache tags and account data
2016-01-28 16:37:41 +00:00
Erik Johnston
c23a8c7833
Ensure keys to RoomStreamChangeCache are ints
2016-01-28 15:55:26 +00:00
Erik Johnston
e1941442d4
Invalidate caches properly. Remove unused arg
2016-01-28 15:02:41 +00:00
Erik Johnston
19fd425928
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sync
2016-01-28 14:46:08 +00:00
Erik Johnston
167d1df699
Merge pull request #534 from matrix-org/erikj/setup
...
Add a Homeserver.setup method
2016-01-28 14:45:36 +00:00
Erik Johnston
7ed2bbeb11
Clean up a bit. Add comment
2016-01-28 14:32:05 +00:00
Erik Johnston
4e7948b47a
Allow paginating backwards from stream token
2016-01-28 11:52:34 +00:00
Erik Johnston
ba8931829b
Return correct type of token
2016-01-28 11:34:17 +00:00
Erik Johnston
61eaa6ec64
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sync
2016-01-28 10:50:57 +00:00
Erik Johnston
c5e7c0e436
Up get_rooms_for_user cache size
2016-01-28 09:58:45 +00:00
Erik Johnston
e7febf4fbb
PEP8
2016-01-27 17:33:27 +00:00
Erik Johnston
aca3193efb
Use the same path for incremental with gap or without gap
2016-01-27 17:33:27 +00:00
Erik Johnston
b97f6626b6
Add cache to room stream
2016-01-27 17:33:26 +00:00
Erik Johnston
a955cbfa49
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/setup
2016-01-27 17:22:35 +00:00
Mark Haines
8c94833b72
Fix adding push rules relative to other rules
2016-01-27 10:24:20 +00:00
Erik Johnston
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
Erik Johnston
8431f62ebb
Merge pull request #525 from matrix-org/erikj/select_many
...
Implement a `_simple_select_many_batch`
2016-01-25 16:30:36 +00:00
Erik Johnston
f091b73e69
Merge pull request #527 from matrix-org/erikj/push_cache
...
Push: Use storage apis that are cached
2016-01-25 16:16:34 +00:00
Erik Johnston
aea5da0ef6
Guard against empty iterables
2016-01-25 15:59:29 +00:00
Erik Johnston
1ebf5e3d03
Correct docstring
2016-01-25 15:53:36 +00:00
Erik Johnston
86896408b0
Add index to event_push_actions
2016-01-25 15:30:32 +00:00
Erik Johnston
53cb173663
Push: Use storage apis that are cached
2016-01-25 13:55:18 +00:00
Erik Johnston
ddd25def01
Implement a _simple_select_many_batch
2016-01-25 13:36:02 +00:00
David Baker
52bdd1b834
Overlay the push_rules_enabled map for users, otherwise they won't be able to disable server default rules.
2016-01-22 14:58:19 +00:00
David Baker
10f76dc5da
Make LRU cache not default to treecache & add options to use it
2016-01-22 12:10:33 +00:00
David Baker
42eae4634f
Use new invalidate_many cache invalidation to invalidate the event_push_actions cache appropriately.
2016-01-22 11:22:48 +00:00
Erik Johnston
8f66fe6392
Cache get_unread_event_push_actions_by_room_for_user
2016-01-21 15:02:07 +00:00
David Baker
367cfab4e6
peppate
2016-01-20 16:05:09 +00:00
David Baker
d4315bbf6b
Add index by user id on receipts_linearized
2016-01-20 15:33:27 +00:00
David Baker
3fa344c037
Add storage function to get all receipts for a user. Also add some cache invalidation to the receipts storage because there wasn't any, and remove a method that was unused.
2016-01-20 15:30:31 +00:00
Erik Johnston
5a7d1ecffc
Add regex cache. Only caculate push actions for users that have sent read receipts, and are on that server
2016-01-19 16:01:05 +00:00
Erik Johnston
3adcc4c86a
Return highlight_count in /sync
2016-01-19 11:35:50 +00:00
Erik Johnston
5cd2126a6a
Remove dead code
2016-01-18 16:49:46 +00:00
Erik Johnston
2c176e02ae
Make unit tests work
2016-01-18 14:48:50 +00:00
Erik Johnston
f59b564507
Make notifications go quicker
2016-01-18 14:48:29 +00:00
Erik Johnston
cc66a9a5e3
Allow filtering events for multiple users at once
2016-01-18 14:43:50 +00:00
Daniel Wagner-Hall
2680043bc6
Require ID and as_token be unique for ASs
...
Defaults ID to as_token if not specified. This will change
when IDs are fully supported.
2016-01-14 14:34:01 +00:00
Daniel Wagner-Hall
27927463a1
Merge pull request #494 from matrix-org/daniel/2
...
Don't start server if ASes are invalidly configured
2016-01-14 11:06:19 +00:00
Daniel Wagner-Hall
fcb6df45e5
Merge pull request #493 from matrix-org/daniel/1
...
Delete unused code
2016-01-14 11:06:14 +00:00
Daniel Wagner-Hall
f6fcff3602
Don't start server if ASes are invalidly configured
2016-01-13 17:09:24 +00:00
Daniel Wagner-Hall
244b356a37
Delete unused code
2016-01-13 17:03:58 +00:00
Mark Haines
9c1f853d58
Rename 'user_name' to 'user_id' in push to make it consistent with the rest of the code
2016-01-13 13:32:59 +00:00
Mark Haines
f4dad9f639
Merge remote-tracking branch 'origin/erikj/bulk_get_push_rules' into markjh/table_name
...
Conflicts:
synapse/storage/push_rule.py
2016-01-13 11:46:07 +00:00
Erik Johnston
8740e4e94a
bulk_get_push_rules should handle empty lists
2016-01-13 11:37:17 +00:00
Mark Haines
c0a279e808
Delete the table objects from TransactionStore
2016-01-13 11:15:20 +00:00
Mark Haines
96e400fee5
Remove the RoomsTable object
2016-01-13 11:07:32 +00:00
Mark Haines
a8e9e0b916
Remove the PushersTable and EventPushActionsTable objects
2016-01-12 14:41:26 +00:00
Mark Haines
31de2953a3
Remove the PushRuleTable and PushRuleEnableTable objects
2016-01-12 14:36:16 +00:00
David Baker
b5d33a656f
Postgres doesn't like booleans
2016-01-11 17:13:52 +00:00
David Baker
fe56138142
Remove rogue 'admin'
2016-01-11 17:09:03 +00:00
David Baker
c232780081
Merge pull request #456 from matrix-org/store_event_actions
...
Send unread notification counts
2016-01-08 14:47:15 +00:00
David Baker
daadcf36c0
This comma is actually important
2016-01-07 10:15:35 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
David Baker
442fcc02f7
Merge remote-tracking branch 'origin/develop' into store_event_actions
2016-01-06 17:28:55 +00:00
David Baker
b6a585348a
Adding is_guest here won't work because it just constructs a dict of uid -> password hash
2016-01-06 17:16:02 +00:00
Daniel Wagner-Hall
5880de186b
Log when we skip daily messages
2016-01-06 13:52:16 +00:00
David Baker
992928304f
Delete notifications for redacted events
2016-01-06 11:58:46 +00:00
David Baker
ae1262a241
Add schema change file for is_guest flag
2016-01-06 11:58:20 +00:00
David Baker
c79f221192
Add is_guest flag to users db to track whether a user is a guest user or not. Use this so we can run _filter_events_for_client when calculating event_push_actions.
2016-01-06 11:38:09 +00:00
David Baker
eb03625626
Merge remote-tracking branch 'origin/develop' into store_event_actions
2016-01-05 18:39:50 +00:00
Daniel Wagner-Hall
cfd07aafff
Allow guests to upgrade their accounts
2016-01-05 18:01:18 +00:00
David Baker
4eb7b950c8
= not == in sql
2016-01-04 18:11:17 +00:00
David Baker
c914d67cda
Rename event-actions to event_push_actions as per PR request
2016-01-04 14:05:37 +00:00
David Baker
3051c9d002
Address minor PR issues
2016-01-04 13:39:29 +00:00
David Baker
4c8f6a7e42
Insert push actions in a single db query rather than one per user/profile_tag
2015-12-22 17:04:31 +00:00
David Baker
65c451cb38
Add bulk push rule evaluator which actually still evaluates rules one by one, but does far fewer db queries to fetch the rules
2015-12-22 15:19:34 +00:00
David Baker
c061b47c57
Merge remote-tracking branch 'origin/develop' into store_event_actions
2015-12-21 15:30:26 +00:00
David Baker
091c545c4f
pep8
2015-12-21 10:14:57 +00:00
David Baker
413d0d6a24
Make unread notification count sending work: put the correct count in incremental syncs too, where necessary, and fix silly bugs like only select the event actions for that user...
2015-12-18 17:47:00 +00:00
Richard van der Hoff
a64f9bbfe0
Fix 500 error when back-paginating search results
...
We were mistakenly adding pagination clauses to the count query, which then
failed because the count query doesn't join to the events table.
2015-12-17 12:50:46 +00:00
David Baker
42ad49f5b7
still very WIP, but now sends unread_notifications_count in the room object on sync (only actually corrrect in a full sync: hardcoded to 0 in incremental syncs).
2015-12-16 18:42:09 +00:00
Mark Haines
4fab578b43
Merge branch 'release-v0.12.0' into develop
2015-12-16 11:18:45 +00:00
Mark Haines
dcfc70e8ed
Allow users to change which account a 3pid is bound to
2015-12-15 17:02:21 +00:00
Mark Haines
15e9885197
Merge branch 'release-v0.12.0' into develop
2015-12-14 14:46:55 +00:00
Mark Haines
98dfa7d24f
Skip events that where the body, name or topic isn't a string when back populating the FTS index
2015-12-14 13:55:46 +00:00
Daniel Wagner-Hall
76e69cc8de
Fix typo
2015-12-14 12:38:55 +00:00
Mark Haines
bfc52a2342
Fix typo in sql for full text search on sqlite3
2015-12-14 11:38:11 +00:00
Mark Haines
deeebbfcb7
Merge branch 'release-v0.12.0' into develop
2015-12-12 14:21:49 +00:00
Erik Johnston
d9a5c56930
Include approximate count of search results
2015-12-11 11:40:23 +00:00
Erik Johnston
51fb590c0e
Use more efficient query form
2015-12-11 11:12:57 +00:00
David Baker
aa667ee396
Save event actions to the db
2015-12-10 17:51:15 +00:00
Mark Haines
7d6b313312
Add caches for whether a room has been forgotten by a user
2015-12-10 17:49:34 +00:00
David Baker
21f135ba76
Very first cut of calculating actions for events as they come in. Doesn't store them yet. Not very efficient.
2015-12-10 16:26:08 +00:00
Mark Haines
6a5ff5f223
Track the time spent in the database per request.
...
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
Mark Haines
d57c5cda71
Bump schema version.
...
As we released version 26 in v0.11.1
2015-12-04 15:28:39 +00:00
Erik Johnston
b8680b82c3
Merge pull request #414 from matrix-org/erikj/if_not_exists
...
Older versions of SQLite don't like IF NOT EXISTS in virtual tables
2015-12-03 13:21:09 +00:00
David Baker
e515b48929
Just replace the table definition with the one from full_schema 16
2015-12-02 17:23:52 +00:00
David Baker
8810eb8c39
Fix schema delta 15 on postgres in the very unlikley event that anyone upgrades to 15...
2015-12-02 17:19:11 +00:00
Erik Johnston
f721fdbf87
Merge pull request #412 from matrix-org/erikj/search
...
Search: Add prefix matching support
2015-12-02 13:56:41 +00:00
Erik Johnston
976cb5aaa8
Throw if unrecognized DB type
2015-12-02 13:50:43 +00:00
Erik Johnston
b2def42bfd
Older versions of SQLite don't like IF NOT EXISTS in virtual tables
2015-12-02 13:29:14 +00:00
Erik Johnston
b9acef5301
Fix so highlight matching works again
2015-12-02 13:28:13 +00:00
Erik Johnston
7dd6e5efca
Remove deuplication. Add comment about regex.
2015-12-02 13:09:37 +00:00
Erik Johnston
477da77b46
Search: Add prefix matching support
2015-12-02 11:40:52 +00:00
Mark Haines
95f30ecd1f
Add API for setting account_data globaly or on a per room basis
2015-12-01 18:41:32 +00:00
Erik Johnston
4dcaa42b6d
Allow paginating search ordered by recents
2015-11-30 17:45:31 +00:00
Erik Johnston
76936f43ae
Return words to highlight in search results
2015-11-27 16:40:42 +00:00
Daniel Wagner-Hall
df7cf6c0eb
Fix SQL for postgres again
2015-11-23 18:54:41 +00:00
Daniel Wagner-Hall
3e573a5c6b
Fix SQL for postgres
2015-11-23 18:48:53 +00:00
Daniel Wagner-Hall
7dfa455508
Remove size specifier for database column
...
Postgres doesn't support them like this.
We don't have a bool type in common between postgres and sqlite.
2015-11-23 18:35:25 +00:00
Daniel Wagner-Hall
95c3306798
Merge branch 'daniel/forgetrooms' of github.com:matrix-org/synapse into daniel/forgetrooms
2015-11-19 15:00:14 -05:00
Daniel Wagner-Hall
df6824a008
Ignore forgotten rooms in v2 sync
2015-11-19 14:54:47 -05:00
Paul "LeoNerd" Evans
1cfda3d2d8
Merge branch 'develop' into daniel/forgetrooms
2015-11-19 16:53:13 +00:00
Daniel Wagner-Hall
9da4c5340d
Simplify code
2015-11-19 10:07:21 -05:00
Daniel Wagner-Hall
bed7889703
Apply forgetting properly to historical events
2015-11-18 18:11:12 -05:00
Mark Haines
d153f482dd
Rename the database table
2015-11-18 15:33:02 +00:00
Mark Haines
1c960fbb80
s/private_user_data/account_data/
2015-11-18 15:31:04 +00:00
Daniel Wagner-Hall
ba26eb3d5d
Allow users to forget rooms
2015-11-17 17:17:30 -05:00
Richard van der Hoff
e4d622aaaf
Implementation of state rollback in /sync
...
Implementation of SPEC-254: roll back the state dictionary to how it looked at
the start of the timeline.
Merged PR https://github.com/matrix-org/synapse/pull/373
2015-11-13 10:58:56 +00:00
Richard van der Hoff
fddedd51d9
Fix a few race conditions in the state calculation
...
Be a bit more careful about how we calculate the state to be returned by
/sync. In a few places, it was possible for /sync to return slightly later
state than that represented by the next_batch token and the timeline. In
particular, the following cases were susceptible:
* On a full state sync, for an active room
* During a per-room incremental sync with a timeline gap
* When the user has just joined a room. (Refactor check_joined_room to make it
less magical)
Also, use store.get_state_for_events() (and thus the existing stategroups) to
calculate the state corresponding to a particular sync position, rather than
state_handler.compute_event_context(), which recalculates from first principles
(and tends to miss some state).
Merged from PR https://github.com/matrix-org/synapse/pull/372
2015-11-13 10:39:09 +00:00
Erik Johnston
3de46c7755
Trailing whitespace
2015-11-12 15:36:43 +00:00
Erik Johnston
8fd8e72cec
Expand comment
2015-11-12 15:33:47 +00:00
Erik Johnston
764e79d051
Comment
2015-11-12 15:19:56 +00:00
Erik Johnston
320408ef47
Fix SQL syntax
2015-11-12 15:09:45 +00:00
Erik Johnston
14a9d805b9
Use a (hopefully) more efficient SQL query for doing recency based room search
2015-11-12 14:48:39 +00:00
Erik Johnston
39de87869c
Fix bug where assumed dict was namedtuple
2015-11-12 14:47:48 +00:00
Erik Johnston
e21cef9bb5
Merge pull request #359 from matrix-org/markjh/incremental_indexing
...
Incremental background updates for db indexes
2015-11-11 17:19:51 +00:00
Mark Haines
e1627388d1
Fix param style to work on both sqlite and postgres
2015-11-11 17:14:56 +00:00
Mark Haines
940a161192
Fix the background update
2015-11-11 13:59:40 +00:00
Daniel Wagner-Hall
cf437900e0
Return world_readable and guest_can_join in /publicRooms
2015-11-10 17:10:27 +00:00
Mark Haines
90b503216c
Use a background task to update databases to use the full text search
2015-11-10 16:20:13 +00:00
Mark Haines
a412b9a465
Run the background updates when starting synapse.
2015-11-10 15:50:58 +00:00
Mark Haines
2ede7aa8a1
Add background update task for reindexing event search
2015-11-09 19:29:32 +00:00
Mark Haines
c6a01f2ed0
Add storage module for tracking background updates.
...
The progress for each background update is stored as a JSON blob in the
database. Each background update is broken up into separate batches.
The batch size is automatically tuned to try avoid blocking single
threaded databases for too long.
2015-11-09 17:26:27 +00:00
Mark Haines
bbe10e8be7
Merge branch 'develop' into markjh/SYN-513
...
Conflicts:
synapse/storage/tags.py
2015-11-09 15:01:59 +00:00
Mark Haines
c4135d85e1
SYN-513: Include updates for rooms that have had all their tags deleted
2015-11-09 14:53:08 +00:00
Matthew Hodgson
dd40fb68e4
fix comedy important missing comma breaking recent-ordered FTS on sqlite
2015-11-08 16:04:37 +00:00
Matthew Hodgson
767c20a869
add a key existence check to tags_by_room to avoid /events 500'ing when testing against vector
2015-11-06 20:49:57 +01:00
Daniel Wagner-Hall
f2c4ee41b9
Remove accidentally added ID column
2015-11-06 14:27:49 +00:00
Erik Johnston
3640ddfbf6
Error handling
2015-11-05 16:10:54 +00:00
Erik Johnston
729ea933ea
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/search
2015-11-05 15:43:52 +00:00
Erik Johnston
7301e05122
Implement basic pagination for search results
2015-11-05 15:04:08 +00:00
Daniel Wagner-Hall
ca2f90742d
Open up /events to anonymous users for room events only
...
Squash-merge of PR #345 from daniel/anonymousevents
2015-11-05 14:32:26 +00:00
Erik Johnston
05c326d445
Implement order and group by
2015-11-04 17:57:44 +00:00
Daniel Wagner-Hall
4e62ffdb21
Merge branch 'develop' of github.com:matrix-org/synapse into develop
2015-11-04 17:31:01 +00:00
Daniel Wagner-Hall
f522f50a08
Allow guests to register and call /events?room_id=
...
This follows the same flows-based flow as regular registration, but as
the only implemented flow has no requirements, it auto-succeeds. In the
future, other flows (e.g. captcha) may be required, so clients should
treat this like the regular registration flow choices.
2015-11-04 17:29:07 +00:00
Mark Haines
33b3e04049
Merge branch 'develop' into daniel/removesomelies
...
Conflicts:
synapse/notifier.py
2015-11-04 16:01:00 +00:00
Erik Johnston
6a3a840b19
Merge pull request #343 from matrix-org/erikj/fix_retries
...
Fix broken cache for getting retry times.
2015-11-03 17:51:49 +00:00
Mark Haines
6797fcd9ab
Merge pull request #335 from matrix-org/markjh/room_tags
...
Add APIs for adding and removing tags from rooms
2015-11-03 16:45:53 +00:00
Erik Johnston
7ce264ce5f
Fix broken cache for getting retry times. This meant we retried remote destinations way more frequently than we should
2015-11-03 16:24:03 +00:00
Mark Haines
5897e773fd
Spell "deferred" more correctly
2015-11-03 14:27:35 +00:00
Daniel Wagner-Hall
771ca56c88
Remove more unused parameters
2015-11-02 15:31:57 +00:00
Mark Haines
ddd8566f41
Store room tag content and return the content in the m.tag event
2015-11-02 15:11:31 +00:00
Mark Haines
0e36756383
Merge branch 'develop' into markjh/room_tags
2015-11-02 10:57:00 +00:00
Daniel Wagner-Hall
621e84d9a0
Add missing column
2015-10-30 16:25:53 +00:00
Mark Haines
f40b0ed5e1
Inform the client of new room tags using v1 /events
2015-10-29 15:21:09 +00:00
Mark Haines
a89b86dc47
Fix pyflakes errors
2015-10-28 16:45:57 +00:00
Mark Haines
892e70ec84
Add APIs for adding and removing tags from rooms
2015-10-28 16:06:57 +00:00
Erik Johnston
56dbcd1524
Docs
2015-10-28 14:05:50 +00:00
Erik Johnston
5cb298c934
Add room context api
2015-10-28 13:45:56 +00:00
Erik Johnston
4cf633d5e9
Pull out sender when computing search results
2015-10-23 15:41:36 +01:00
Erik Johnston
0c36098c1f
Implement rank function for SQLite FTS
2015-10-23 13:23:48 +01:00
Erik Johnston
671ac699f1
Actually filter results
2015-10-22 16:54:56 +01:00
Erik Johnston
fb0fecd0b9
LESS THAN
2015-10-22 16:18:35 +01:00
Erik Johnston
232beb3a3c
Use namedtuple as return value
2015-10-22 15:02:35 +01:00
Erik Johnston
ba02bba88c
Limit max number of SQL vars
2015-10-22 13:25:27 +01:00
Erik Johnston
8a98f0dc5b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/search
2015-10-22 13:16:35 +01:00
Erik Johnston
f142898f52
PEP8
2015-10-22 11:18:01 +01:00
Erik Johnston
4d25bc6c92
Move FTS to delta 25
2015-10-22 11:12:28 +01:00
Mark Haines
e94ffd89d6
Merge pull request #316 from matrix-org/markjh/v2_sync_archived
...
Add rooms that the user has left under archived in v2 sync.
2015-10-21 15:46:41 +01:00
Mark Haines
e3d75f564a
Include banned rooms in the archived section of v2 sync
2015-10-21 11:15:48 +01:00
Mark Haines
68b7fc3e2b
Add rooms that the user has left under archived in v2 sync.
2015-10-19 17:26:18 +01:00
Erik Johnston
46d39343d9
Explicitly check for Sqlite3Engine
2015-10-16 16:58:00 +01:00
Erik Johnston
f2d698cb52
Typing
2015-10-16 16:46:48 +01:00
Erik Johnston
33646eb000
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/search
2015-10-16 15:35:35 +01:00
Mark Haines
fc012aa8dc
Fix FilteringStore.get_user_filter to work with postgres
2015-10-16 15:28:43 +01:00
Erik Johnston
12122bfc36
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/search
2015-10-16 14:46:32 +01:00
Erik Johnston
edb998ba23
Explicitly check for Sqlite3Engine
2015-10-16 14:37:14 +01:00
Erik Johnston
3cf9948b8d
Add docstring
2015-10-16 11:28:12 +01:00
Erik Johnston
73260ad01f
Comment on the LIMIT 500
2015-10-16 11:24:02 +01:00
Erik Johnston
22a8c91448
Split up run_upgrade
2015-10-16 11:19:44 +01:00
Mark Haines
6296590bf7
Encode the filter JSON as UTF-8 before storing in the database.
...
Because we are using a binary column type to store the filter JSON.
2015-10-16 10:50:32 +01:00
Erik Johnston
bcfb653816
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/search
2015-10-15 16:37:32 +01:00
Mark Haines
a059760954
Merge pull request #305 from matrix-org/markjh/v2_sync_api
...
Update the v2 sync API to work as specified in the current spec.
2015-10-14 13:56:23 +01:00
Mark Haines
f50c43464c
Merge branch 'develop' into markjh/v2_sync_api
2015-10-14 11:40:45 +01:00
Erik Johnston
99c7fbfef7
Fix to work with SQLite
2015-10-14 09:52:40 +01:00
Erik Johnston
858634e1d0
Remove unused room_id arg
2015-10-14 09:31:20 +01:00
Erik Johnston
7ecd11accb
Add paranoia limit
2015-10-13 15:50:56 +01:00
Erik Johnston
3e2a1297b5
Remove constraints in preperation of using filters
2015-10-13 15:22:14 +01:00
Mark Haines
cacf0688c6
Add a get_invites_for_user method to the storage to find out the rooms a user is invited to
2015-10-13 14:08:38 +01:00
Erik Johnston
88971fd034
Merge branch 'erikj/store_engine' into erikj/search
2015-10-13 14:03:30 +01:00
Erik Johnston
17c80c8a3d
rename schema_prepare to prepare_database
2015-10-13 13:56:22 +01:00
Erik Johnston
cfd39d6b55
Add SQLite support
2015-10-13 13:47:50 +01:00
Erik Johnston
f9340ea0d5
Merge branch 'erikj/store_engine' into erikj/search
2015-10-13 13:29:02 +01:00
Erik Johnston
ec398af41c
Expose error more nicely
2015-10-13 11:43:43 +01:00
Erik Johnston
40b6a5aad1
Split out the schema preparation and update logic into its own module
2015-10-13 11:38:48 +01:00
Erik Johnston
1a40afa756
Add sqlite schema
2015-10-13 10:36:25 +01:00
Erik Johnston
ca53ad7425
Filter events to only thsoe that the user is allowed to see
2015-10-12 15:52:55 +01:00
Erik Johnston
927004e349
Remove unused room_id parameter
2015-10-12 15:06:14 +01:00
Erik Johnston
61561b9df7
Keep FTS indexes up to date. Only search through rooms currently joined
2015-10-12 10:49:53 +01:00
Erik Johnston
c85c912562
Add basic full text search impl.
2015-10-09 15:48:31 +01:00
Mark Haines
301141515a
Merge pull request #288 from matrix-org/markjh/unused_definitions
...
Remove some of the unused definitions from synapse
2015-09-28 14:22:44 +01:00
Mark Haines
cf1100887b
Fix order of ON constraints in _get_rooms_for_user_where_membership_is_txn
2015-09-24 17:35:10 +01:00
Mark Haines
1cd65a8d1e
synapse/storage/state.py: _make_group_id was unused
2015-09-23 10:37:58 +01:00
Mark Haines
973ebb66ba
Remove unused functions from synapse/storage/signatures.py
2015-09-23 10:36:33 +01:00
Mark Haines
e51aa4be96
synapse/storage/roommember.py:_get_members_query was unused
2015-09-23 10:35:10 +01:00
Mark Haines
92d8d724c5
Remove unused functions from synapse/storage/events.py
2015-09-23 10:33:06 +01:00
Mark Haines
c292dba70c
Remove unused functions from synapse/storage/event_federation.py
2015-09-23 10:31:25 +01:00
Mark Haines
396834f1c0
synapse/storage/_base.py:_simple_max_id was unused
2015-09-23 10:30:38 +01:00
Mark Haines
1d9036aff2
synapse/storage/_base.py:_simple_delete was unused
2015-09-23 10:30:25 +01:00
Mark Haines
1ee3d26432
synapse/storage/_base.py:_simple_selectupdate_one was unused
2015-09-23 10:30:03 +01:00
Mark Haines
3559a835a2
synapse/storage/event_federation.py:_get_auth_events is unused
2015-09-22 18:39:46 +01:00
Mark Haines
7dd4f79c49
synapse/storage/_base.py:_execute_and_decode was unused
2015-09-22 18:37:07 +01:00
Mark Haines
527d95dea0
synapse/storage/_base.py:Table was unused
2015-09-22 18:14:15 +01:00
Daniel Wagner-Hall
6d59ffe1ce
Add some docstrings
2015-09-22 13:47:40 +01:00
Daniel Wagner-Hall
eb011cd99b
Add docstring
2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Mark Haines
ee2d722f0f
Merge pull request #276 from matrix-org/markjh/history_for_rooms_that_have_been_left
...
SPEC-216: Allow users to view the history of rooms that they have left.
2015-09-21 14:38:13 +01:00
Erik Johnston
2048388cfd
Merge pull request #281 from matrix-org/erikj/update_extremeties
...
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:57:25 +01:00
Erik Johnston
8148c48f11
"Comments"
2015-09-15 16:54:48 +01:00
Erik Johnston
3b05b67c89
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:34:42 +01:00
Erik Johnston
dffc9c4ae0
Drop unused index
2015-09-14 14:41:37 +01:00
Mark Haines
09cb5c7d33
Allow users that have left a room to get the messages that happend in the room before they left
2015-09-09 17:31:09 +01:00
Mark Haines
3c166a24c5
Remove undocumented and unimplemented 'feedback' parameter from the Client-Server API
2015-09-09 16:05:09 +01:00
Mark Haines
89ae0166de
Allow room initialSync for users that have left the room, returning a snapshot of how the room was when they left it
2015-09-09 13:25:22 +01:00
Mark Haines
417485eefa
Include the event_id and stream_ordering of membership events when looking up which rooms a user is in
2015-09-08 18:14:54 +01:00
Erik Johnston
b62c1395d6
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-09-01 13:11:55 +01:00
Erik Johnston
cd800ad99a
Lower size of 'stateGroupCache' now that we have data from matrix.org to support doing so
2015-09-01 10:09:03 +01:00
Mark Haines
a7122692d9
Merge branch 'release-v0.10.0' into develop
...
Conflicts:
synapse/handlers/auth.py
synapse/python_dependencies.py
synapse/rest/client/v1/login.py
2015-08-28 11:15:27 +01:00
Daniel Wagner-Hall
d3c0e48859
Merge erikj/user_dedup to develop
2015-08-26 13:42:45 +01:00
Daniel Wagner-Hall
a2355fae7e
Merge pull request #251 from matrix-org/removeadmin
...
Stop looking up "admin", which we never read
2015-08-25 17:23:05 +01:00
Daniel Wagner-Hall
ee3fa1a99c
Merge pull request #248 from matrix-org/deviceid
...
Remove completely unused concepts from codebase
2015-08-25 17:19:06 +01:00
Daniel Wagner-Hall
a9d8bd95e7
Stop looking up "admin", which we never read
2015-08-25 16:29:39 +01:00
Daniel Wagner-Hall
a0b181bd17
Remove completely unused concepts from codebase
...
Removes device_id and ClientInfo
device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.
This doesn't remove the columns from the database, because that's
fiddly.
2015-08-25 16:23:06 +01:00
Mark Haines
1c847af28a
Merge pull request #243 from matrix-org/markjh/remove_syutil
...
Replace syutil dependency with smaller, single-purpose libraries
2015-08-25 10:52:16 +01:00
Mark Haines
037481a033
Remove autoincrement since we incrementing the ID in the storage layer
2015-08-24 17:48:57 +01:00
Mark Haines
01fc3943f1
Fix indent
2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
Erik Johnston
457970c724
Don't insert events into 'event_*_extremeties' tables if they're outliers
2015-08-23 13:44:23 +01:00
Erik Johnston
1bd1a43073
Actually check if event_id isn't returned by _get_state_groups
2015-08-21 14:30:34 +01:00
Erik Johnston
745b72660a
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-08-21 11:39:38 +01:00
Erik Johnston
42f12ad92f
When logging in fetch user by user_id case insensitively, *unless* there are multiple case insensitive matches, in which case require the exact user_id
2015-08-21 11:38:44 +01:00
Erik Johnston
aa3c9c7bd0
Don't allow people to register user ids which only differ by case to an existing one
2015-08-21 10:57:47 +01:00
Erik Johnston
1f7642efa9
Fix bug where we didn't correctly serialize the redacted_because key over federation
2015-08-21 09:36:07 +01:00
Daniel Wagner-Hall
e8cf77fa49
Merge branch 'develop' into refresh
...
Conflicts:
synapse/rest/client/v1/login.py
2015-08-20 16:25:40 +01:00
Daniel Wagner-Hall
cecbd636e9
/tokenrefresh POST endpoint
...
This allows refresh tokens to be exchanged for (access_token,
refresh_token).
It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
Daniel Wagner-Hall
13a6517d89
s/by_token/by_access_token/g
...
We're about to have two kinds of token, access and refresh
2015-08-20 16:01:29 +01:00
Mark Haines
a0b8e5f2fe
Merge pull request #211 from matrix-org/email_in_use
...
Changes for unique emails
2015-08-20 10:04:04 +01:00
Erik Johnston
e624cdec64
Merge pull request #228 from matrix-org/erikj/_get_state_for_groups
...
Ensure we never return a None event from _get_state_for_groups
2015-08-18 16:30:17 +01:00
Erik Johnston
38a965b816
Merge pull request #227 from matrix-org/erikj/receipts_take2
...
Re-enable receipts API.
2015-08-18 16:30:04 +01:00
Erik Johnston
a82938416d
Remove newline because vertical whitespace makes mjark sad
2015-08-18 16:28:13 +01:00
Erik Johnston
0bfdaf1f4f
Rejig the code to make it nicer
2015-08-18 16:26:07 +01:00
Erik Johnston
128ed32e6b
Bump size of get_presence_state cache
2015-08-18 15:51:23 +01:00
Erik Johnston
f704c10f29
Rename unhelpful variable name
2015-08-18 11:54:03 +01:00
Erik Johnston
8199475ce0
Ensure we never return a None event from _get_state_for_groups
2015-08-18 11:44:10 +01:00
Erik Johnston
85d0bc3bdc
Reduce cache size from obscenely large to quite large
2015-08-18 11:00:38 +01:00
Erik Johnston
cfc503681f
Comments
2015-08-18 10:49:23 +01:00
Erik Johnston
8f4165628b
Add index receipts_linearized_room_stream
2015-08-17 14:43:54 +01:00
Erik Johnston
d3d582bc1c
Remove unused import
2015-08-17 13:38:09 +01:00
Erik Johnston
afef6f5d16
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2
2015-08-17 13:23:44 +01:00
Erik Johnston
2d97e65558
Remember to invalidate caches
2015-08-17 10:46:55 +01:00
Erik Johnston
1a9510bb84
Implement a batched presence_handler.get_state and use it
2015-08-17 10:40:23 +01:00
Erik Johnston
47abebfd6d
Add batched version of store.get_presence_state
2015-08-17 09:50:50 +01:00
Erik Johnston
f9d4da7f45
Fix bug where we were leaking None into state event lists
2015-08-17 09:39:45 +01:00
Daniel Wagner-Hall
30883d8409
Merge pull request #221 from matrix-org/auth
...
Simplify LoginHander and AuthHandler
2015-08-14 17:02:22 +01:00
Erik Johnston
68b255c5a1
Batch _get_linearized_receipts_for_rooms
2015-08-14 15:06:22 +01:00
Erik Johnston
129ee4e149
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2
2015-08-13 17:28:43 +01:00
Erik Johnston
2bb2c02571
Remove some vertical space
2015-08-13 17:11:30 +01:00
Erik Johnston
57877b01d7
Replace list comprehension
2015-08-13 17:00:17 +01:00
Erik Johnston
5db5677969
Add metrics to the receipts cache
2015-08-13 16:58:23 +01:00
Erik Johnston
0fbed2a8fa
Comment
2015-08-12 17:22:54 +01:00
Erik Johnston
c10ac7806e
Explain why we're prefilling dict with Nones
2015-08-12 17:16:30 +01:00
Erik Johnston
101ee3fd00
Better variable name
2015-08-12 17:08:05 +01:00
Erik Johnston
df361d08f7
Split _get_state_for_group_from_cache into two
2015-08-12 17:06:21 +01:00
Daniel Wagner-Hall
415c2f0549
Simplify LoginHander and AuthHandler
...
* Merge LoginHandler -> AuthHandler
* Add a bunch of documentation
* Improve some naming
* Remove unused branches
I will start merging the actual logic of the two handlers shortly
2015-08-12 15:49:37 +01:00
Erik Johnston
2df8dd9b37
Move all the caches into their own package, synapse.util.caches
2015-08-11 18:00:59 +01:00
Erik Johnston
53a817518b
Comments
2015-08-11 11:40:40 +01:00
Erik Johnston
6eaa116867
Comment
2015-08-11 11:35:24 +01:00
Erik Johnston
4762c276cb
Docs
2015-08-11 11:33:41 +01:00
Erik Johnston
1b994a97dd
Fix application of ACLs
2015-08-11 10:41:40 +01:00
Erik Johnston
10b874067b
Fix state cache
2015-08-11 09:12:41 +01:00
Erik Johnston
017b798e4f
Clean up StateStore
2015-08-10 15:01:06 +01:00
Erik Johnston
2c019eea11
Remove unused function
2015-08-10 14:44:41 +01:00
Erik Johnston
bb0a475c30
Comments
2015-08-10 14:27:38 +01:00
Mark Haines
559c51debc
Use TypeError instead of ValueError and give a nicer error mesasge
...
when someone calls Cache.invalidate with the wrong type.
2015-08-10 14:07:17 +01:00
Erik Johnston
7ce71f2ffc
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 13:47:51 +01:00
Erik Johnston
8c3a62b5c7
Merge pull request #215 from matrix-org/erikj/cache_varargs_interface
...
Change Cache to not use *args in its interface
2015-08-10 13:47:45 +01:00
Erik Johnston
86eaaa885b
Rename keyargs to args in CacheDescriptor
2015-08-10 13:44:44 +01:00
Erik Johnston
e0b6e49466
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 10:39:22 +01:00
Erik Johnston
2cd6cb9f65
Rename keyargs to args in Cache
2015-08-10 10:38:47 +01:00
Erik Johnston
aa88582e00
Do bounds check
2015-08-10 10:08:15 +01:00
Erik Johnston
5119e416e8
Line length
2015-08-10 10:05:30 +01:00
Erik Johnston
8f04b6fa7a
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:30:25 +01:00
Erik Johnston
7dec0b2bee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:28:39 +01:00
Erik Johnston
06218ab125
Merge pull request #212 from matrix-org/erikj/cache_deferreds
...
Make CacheDescriptor cache deferreds rather than the deferreds' values
2015-08-07 19:28:05 +01:00
Erik Johnston
2352974aab
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:26:54 +01:00
Erik Johnston
9c5385b53a
s/observed/observer/
2015-08-07 19:26:38 +01:00
Erik Johnston
ffab798a38
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:18:47 +01:00
Erik Johnston
62126c996c
Propogate stale cache errors to calling functions
2015-08-07 19:17:58 +01:00
Erik Johnston
3213ff630c
Remove unnecessary cache
2015-08-07 19:14:05 +01:00
Erik Johnston
20addfa358
Change Cache to not use *args in its interface
2015-08-07 18:32:47 +01:00
Erik Johnston
9eb5b23d3a
Batch up various DB requests for event -> state
2015-08-07 18:16:02 +01:00
Erik Johnston
0211890134
Implement a CacheListDescriptor
2015-08-07 18:14:49 +01:00
Paul "LeoNerd" Evans
0db40d3e93
Don't complain about extra .pyc files we find while hunting for database schemas
2015-08-07 17:22:11 +01:00
Erik Johnston
b2c7bd4b09
Cache get_recent_events_for_room
2015-08-07 14:42:34 +01:00
Erik Johnston
b3768ec10a
Remove unncessary cache
2015-08-07 13:41:05 +01:00
Erik Johnston
b8e386db59
Change Cache to not use *args in its interface
2015-08-07 11:52:21 +01:00
Erik Johnston
fe994e728f
Store absence of state in cache
2015-08-07 10:17:38 +01:00
Erik Johnston
1d08bf7c17
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 14:03:15 +01:00
Erik Johnston
63b1eaf32c
Docs
2015-08-06 14:02:50 +01:00
Erik Johnston
b811c98574
Remove failed deferreds from cache
2015-08-06 14:01:27 +01:00
Erik Johnston
433314cc34
Re-implement DEBUG_CACHES flag
2015-08-06 14:01:05 +01:00
Erik Johnston
f596ff402e
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 13:37:56 +01:00
Erik Johnston
2efb93af52
Merge branch 'erikj/cached_keyword_args' into erikj/cache_deferreds
2015-08-06 13:35:28 +01:00
Erik Johnston
953dbd28a7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cached_keyword_args
2015-08-06 13:35:03 +01:00
Erik Johnston
7eea3e356f
Make @cached cache deferreds rather than the deferreds' values
2015-08-06 13:33:34 +01:00
Erik Johnston
3e1b77efc2
Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 16:45:56 +01:00
Erik Johnston
b52b4a84ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 15:41:20 +01:00
Erik Johnston
1e62a3d3a9
Up the cache size for 'get_joined_hosts_for_room' and 'get_users_in_room'
2015-08-05 15:40:40 +01:00
Erik Johnston
a89559d797
Use LRU cache by default
2015-08-05 15:39:47 +01:00
Erik Johnston
07507643cb
Use dictionary cache to do group -> state fetching
2015-08-05 15:11:42 +01:00
Erik Johnston
c67ba143fa
Move DictionaryCache
2015-08-04 15:58:28 +01:00
Erik Johnston
e7768e77f5
Add basic dictionary cache
2015-08-04 15:56:56 +01:00
David Baker
e124128542
Bump schema version
2015-08-04 14:50:31 +01:00
David Baker
c77048e12f
Add endpoint that proxies ID server request token and errors if the given email is in use on this Home Server.
2015-08-04 14:37:09 +01:00
Erik Johnston
413a4c289b
Add comment
2015-08-04 11:08:07 +01:00
Erik Johnston
4d6cb8814e
Speed up event filtering (for ACL) logic
2015-08-04 09:32:23 +01:00
Erik Johnston
39e21ea51c
Add support for using keyword arguments with cached functions
2015-07-27 13:57:29 +01:00
David Baker
20c0324e9c
Dodesn't seem to make any difference: guess it does work with the object reference
2015-07-21 16:21:37 -07:00
David Baker
cf7a40b08a
I think this was what was intended...
2015-07-21 16:08:00 -07:00
Mark Haines
3b5823c74d
s/take/claim/ for end to end key APIs
2015-07-20 18:23:54 +01:00
Matrix
4f973eb657
Up default cache size for _RoomStreamChangeCache
2015-07-18 19:07:33 +01:00
Erik Johnston
b6d4a4c6d8
Merge pull request #199 from matrix-org/erikj/receipts
...
Implement read receipts.
2015-07-16 18:18:36 +01:00
Erik Johnston
baa55fb69e
Merge pull request #193 from matrix-org/erikj/bulk_persist_event
...
Add bulk insert events API
2015-07-14 10:49:24 +01:00
Erik Johnston
4624d6035e
Docs
2015-07-14 10:19:07 +01:00
Erik Johnston
5989637f37
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-13 13:50:57 +01:00
Erik Johnston
e5991af629
Comments
2015-07-13 13:30:43 +01:00
Erik Johnston
17bb9a7eb9
Remove commented out code
2015-07-10 14:07:57 +01:00
Erik Johnston
532fcc997a
Merge pull request #196 from matrix-org/erikj/room_history
...
Add ability to restrict room history.
2015-07-10 13:47:04 +01:00
Erik Johnston
0b3389bcd2
Merge pull request #194 from matrix-org/erikj/bulk_verify_sigs
...
Implement bulk verify_signed_json API
2015-07-10 13:46:53 +01:00
Erik Johnston
0d7f0febf4
Uniquely name unique constraint
2015-07-10 13:43:03 +01:00
Erik Johnston
ed88720952
Handle error slightly better
2015-07-09 16:14:46 +01:00
Mark Haines
bf0d59ed30
Don't bother with a timeout for one time keys on the server.
2015-07-09 14:04:03 +01:00
Erik Johnston
1af188209a
Change format of receipts to allow inclusion of data
2015-07-09 11:39:30 +01:00
Mark Haines
8fb79eeea4
Only remove one time keys when new one time keys are added
2015-07-08 17:04:29 +01:00
Erik Johnston
304343f4d7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-08 15:37:33 +01:00
Erik Johnston
af812b68dd
Add a cache to fetching of receipt streams
2015-07-08 15:35:00 +01:00
Erik Johnston
87311d1b8c
Hook up receipts to v1 initialSync
2015-07-08 11:02:04 +01:00
Erik Johnston
ca041d5526
Wire together receipts and the notifer/federation
2015-07-07 15:25:30 +01:00
Erik Johnston
716e426933
Fix various typos
2015-07-07 10:55:31 +01:00
Mark Haines
b5770f8947
Add store for client end to end keys
2015-07-06 18:46:47 +01:00
Erik Johnston
65e69dec8b
Don't explode if we don't recognize one of the event_ids in the backfill request
2015-07-06 09:33:03 +01:00
Erik Johnston
1a60545626
Add basic impl for room history ACL on GET /messages client API
2015-07-02 16:20:10 +01:00
Erik Johnston
ac78e60de6
Add stream_id index
2015-07-02 13:18:41 +01:00
Erik Johnston
80a61330ee
Add basic storage functions for handling of receipts
2015-07-01 17:19:12 +01:00
Erik Johnston
901f56fa63
Add tables for receipts
2015-06-30 15:29:47 +01:00
Erik Johnston
b5f55a1d85
Implement bulk verify_signed_json API
2015-06-26 10:39:34 +01:00
Erik Johnston
5130d80d79
Add bulk insert events API
2015-06-25 17:29:34 +01:00
Erik Johnston
6924852592
Batch SELECTs in _get_auth_chain_ids_txn
2015-06-23 11:01:04 +01:00
Erik Johnston
2eda996a63
Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py
2015-06-13 12:21:58 +01:00
Erik Johnston
1e365e88bd
Bump schema version
2015-06-11 15:50:39 +01:00
David Baker
da84946de4
pep8
2015-06-04 16:43:45 +01:00
David Baker
63a7b3ad1e
Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them.
2015-06-04 16:16:01 +01:00
Erik Johnston
0d7d9c37b6
Add cache to get_state_groups
2015-06-03 14:45:55 +01:00
Erik Johnston
d8866d7277
Caches should be bound to instances.
...
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
2015-06-03 14:45:17 +01:00
Erik Johnston
6e17463228
Don't explode if we don't have the event
2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf
Add config option to turn off freezing events. Use new encode_json api and ujson.loads
2015-05-29 12:17:33 +01:00
Mark Haines
d9f60e8dc8
Merge pull request #163 from matrix-org/markjh/presence_list_cache
...
Add a cache for the presence list
2015-05-22 17:02:23 +01:00
Mark Haines
0e42dfbe22
Merge pull request #164 from matrix-org/markjh/pusher_performance_2
...
Add a cache for get_push rules for user, fix cache invalidation
2015-05-22 17:01:56 +01:00
Mark Haines
1a9a9abcc7
Add a cache for getting the presence list for a user
2015-05-22 16:11:17 +01:00
Erik Johnston
74b7de83ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-22 16:10:42 +01:00
Mark Haines
36317f3dad
Merge pull request #156 from matrix-org/erikj/join_perf
...
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
Mark Haines
b6adfc59f5
Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities
2015-05-22 13:01:03 +01:00
Mark Haines
a04cde613e
Add a cache for get_push rules for user, fix cache invalidation
2015-05-22 10:39:45 +01:00
Erik Johnston
4429e720ae
Merge branch 'master' of github.com:matrix-org/synapse into develop
2015-05-22 10:33:00 +01:00
Mark Haines
2043527b9b
Don't try to use a txn when not in one, remove spurious debug logging
2015-05-21 16:53:03 +01:00
Mark Haines
53447e9cd3
Add caches for things requested by the pushers
2015-05-21 16:41:39 +01:00
Mark Haines
d61ce3f670
Add a cache for get_current_state with state_key
2015-05-21 16:41:39 +01:00
Erik Johnston
a910984b58
Actually return something from lambda
2015-05-21 15:58:41 +01:00
Erik Johnston
e309b1045d
Sort backfill events
2015-05-21 15:57:35 +01:00
Erik Johnston
0180bfe4aa
Remove dead code
2015-05-21 15:53:41 +01:00
Erik Johnston
1f3d1d85a9
Only get non-state
2015-05-21 15:52:29 +01:00
Erik Johnston
39a3340f73
Skip events we've already seen
2015-05-21 15:48:56 +01:00
Erik Johnston
ae3bff3491
Correctly prepopulate queue
2015-05-21 15:46:07 +01:00
Erik Johnston
dc085ddf8c
Don't prepopulate event_results
2015-05-21 15:44:05 +01:00
Erik Johnston
73d23c6ae8
Don't readd things that are already in event_results
2015-05-21 15:40:22 +01:00
Erik Johnston
6189d8e54d
PriorityQueue gives lowest first
2015-05-21 15:38:08 +01:00
Erik Johnston
115ef3ddac
Correctly capture Queue.Empty exception
2015-05-21 15:37:43 +01:00
Erik Johnston
c2633907c5
Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-21 14:58:47 +01:00
Erik Johnston
ebfdd2eb5b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf
2015-05-21 14:54:52 +01:00
Erik Johnston
a551c5dad7
Merge pull request #155 from matrix-org/erikj/perf
...
Bulk and batch retrieval of events.
2015-05-21 14:54:40 +01:00
Erik Johnston
27e4b45c06
s/for events/for requests for events/
2015-05-21 14:52:23 +01:00
Erik Johnston
ac5f2bf9db
s/for events/for requests for events/
2015-05-21 14:50:57 +01:00
Erik Johnston
80a167b1f0
Add comments
2015-05-21 11:19:04 +01:00
Erik Johnston
9118a92862
Split up _get_events into defer and txn versions
2015-05-20 13:27:16 +01:00
Erik Johnston
ab45e12d31
Make not return a deferred _get_event_from_row_txn
2015-05-20 13:07:19 +01:00
Erik Johnston
f407cbd2f1
PEP8
2015-05-20 13:02:01 +01:00
Erik Johnston
227f8ef031
Split out _get_event_from_row back into defer and _txn version
2015-05-20 13:00:57 +01:00
Erik Johnston
2bc60c55af
Fix _get_backfill_events to return events in the correct order
2015-05-20 12:57:00 +01:00
Erik Johnston
9084cdd70f
Ensure event_results is a set
2015-05-19 16:34:31 +01:00
Erik Johnston
aa729349dd
Fix event_backwards_extrem insertion to ignore outliers
2015-05-19 15:27:00 +01:00
Mark Haines
99914ec9f8
Merge pull request #152 from matrix-org/notifier_performance
...
Notifier performance
2015-05-18 17:49:59 +01:00
Erik Johnston
165eb2dbe6
Comments and shuffle of functions
2015-05-18 15:18:41 +01:00
Erik Johnston
65878a2319
Remove unused metric
2015-05-18 14:06:30 +01:00
Erik Johnston
4d1b6f4ad1
Remove rejected events if we don't want rejected events
2015-05-18 14:03:46 +01:00
Mark Haines
0b0033c40b
Merge branch 'develop' into notifier_performance
2015-05-18 13:50:01 +01:00
Erik Johnston
10f1bdb9a2
Move get_events functions to storage.events
2015-05-18 10:21:40 +01:00
Erik Johnston
c71176858b
Newline, remove debug logging
2015-05-18 10:11:14 +01:00
Erik Johnston
c3b37abdfd
PEP8
2015-05-15 16:59:58 +01:00
Erik Johnston
6c74fd62a0
Revert limiting of fetching, it didn't help perf.
2015-05-15 16:45:35 +01:00
Erik Johnston
9ff7f66a2b
init j
2015-05-15 16:36:03 +01:00
Erik Johnston
70f272f71c
Don't completely drain the list
2015-05-15 16:34:17 +01:00
Erik Johnston
8763dd80ef
Don't fetch prev_content for current_state
2015-05-15 15:33:01 +01:00
Erik Johnston
acb12cc811
Make store.get_current_state fetch events asyncly
2015-05-15 15:20:05 +01:00
Erik Johnston
d62dee7eae
Remove more debug logging
2015-05-15 15:06:37 +01:00
Erik Johnston
0f29cfabc3
Remove debug logging
2015-05-15 14:06:42 +01:00
Erik Johnston
e275a9c0d9
preserve log context
2015-05-15 11:54:51 +01:00
Erik Johnston
aa32bd38e4
Add a wait
2015-05-15 11:35:04 +01:00
Erik Johnston
372d4c6d7b
Srsly. Don't use closures. Baaaaaad
2015-05-15 11:26:00 +01:00
Erik Johnston
575ec91d82
Correctly pass through params
2015-05-15 11:15:10 +01:00
Erik Johnston
de01438a57
Sort out error handling
2015-05-15 11:00:50 +01:00
Erik Johnston
a2c4f3f150
Fix daedlock
2015-05-15 10:54:04 +01:00
Erik Johnston
1d566edb81
Remove race condition
2015-05-14 16:54:35 +01:00
Erik Johnston
ef3d8754f5
Call from right thread
2015-05-14 15:41:55 +01:00
Erik Johnston
142934084a
Count and loop
2015-05-14 15:40:21 +01:00
Erik Johnston
96c5b9f87c
Don't start up more fetch_events
2015-05-14 15:36:04 +01:00
Erik Johnston
7cd6a6f6cf
Awful idea for speeding up fetching of events
2015-05-14 15:34:02 +01:00
Erik Johnston
7f4105a5c9
Turn off preemptive transactions
2015-05-14 14:51:06 +01:00
Erik Johnston
f4d58deba1
PEP8
2015-05-14 14:45:42 +01:00
Erik Johnston
386b7330d2
Move from _base to events
2015-05-14 14:45:22 +01:00
Erik Johnston
7d6a1dae31
Jump out early
2015-05-14 14:27:58 +01:00
Erik Johnston
656223fbd3
Actually, we probably want to run this in a transaction
2015-05-14 14:26:35 +01:00
Erik Johnston
2f7f8e1c2b
Preemptively jump into a transaction if we ask for get_prev_content
2015-05-14 14:17:36 +01:00
Erik Johnston
e1e9f0c5b2
loop -> gatherResults
2015-05-14 13:58:49 +01:00
Erik Johnston
ab78a8926e
Err, we probably want a bigger limit
2015-05-14 13:47:16 +01:00
Erik Johnston
f6f902d459
Move fetching of events into their own transactions
2015-05-14 13:45:48 +01:00
Erik Johnston
cdb3757942
Refactor _get_events
2015-05-14 13:31:55 +01:00
Erik Johnston
36ea26c5c0
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf
2015-05-14 12:01:38 +01:00
David Baker
7c549dd557
Add ID generator for push_rules_enable to #resolve SYN-378
2015-05-14 11:44:03 +01:00
Mark Haines
899d4675dd
Merge branch 'notifier_unify' into notifier_performance
2015-05-14 11:36:44 +01:00
Erik Johnston
4f1d984e56
Add index on events
2015-05-13 17:22:26 +01:00
Erik Johnston
968b01a91a
Actually use async method
2015-05-13 17:02:46 +01:00
Erik Johnston
4071f29653
Fetch events from events_id in their own transactions
2015-05-13 16:59:41 +01:00
Erik Johnston
a988361aea
Typo
2015-05-13 15:44:15 +01:00
Erik Johnston
8888982db3
Don't insert None
2015-05-13 15:43:32 +01:00
Erik Johnston
cf706cc6ef
Don't return None
2015-05-13 15:31:25 +01:00
Erik Johnston
5971d240d4
Limit batch size
2015-05-13 15:26:49 +01:00
Erik Johnston
ca4f458787
Fetch events in bulk
2015-05-13 15:13:42 +01:00
Erik Johnston
6edff11a88
Don't fetch redaction and rejection stuff for each event, so we can use index only scan
2015-05-13 14:39:05 +01:00
Mark Haines
63878c0379
Don't bother checking for updates if the stream token hasn't advanced for a user
2015-05-13 13:42:21 +01:00
Erik Johnston
02590c3e1d
Temp turn off checking for rejections and redactions
2015-05-13 11:31:28 +01:00
Erik Johnston
619a21812b
defer.gatherResults loop
2015-05-13 11:29:03 +01:00
Erik Johnston
fec4485e28
Batch fetching of events for state groups
2015-05-13 11:22:42 +01:00
Erik Johnston
409bcc76bd
Load events for state group seperately
2015-05-13 11:13:31 +01:00
Erik Johnston
e122685978
You need to call contextmanager
2015-05-12 16:12:37 +01:00
Mark Haines
a6fb2aa2a5
Merge pull request #144 from matrix-org/erikj/logging_context
...
Preserving logging contexts
2015-05-12 15:23:50 +01:00
Mark Haines
1fce36b111
Merge pull request #149 from matrix-org/erikj/backfill
...
Backfill support
2015-05-12 15:20:32 +01:00
Erik Johnston
8b28209c60
Err, delete the right stuff
2015-05-12 15:02:53 +01:00
Erik Johnston
30c72d377e
Newlines
2015-05-12 14:47:40 +01:00
Erik Johnston
e4eddf9b36
We do actually want to delete rows out of event_backward_extremities
2015-05-12 14:47:23 +01:00
Erik Johnston
c1779a79bc
Fix up _handle_prev_events to not try to insert duplicate rows
2015-05-12 14:41:50 +01:00
Erik Johnston
74850d7f75
Do state groups persistence /after/ checking if we have already persisted the event
2015-05-12 14:14:58 +01:00
Erik Johnston
da6a7bbdde
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context
2015-05-12 13:10:42 +01:00
Mark Haines
5002056b16
SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes.
2015-05-12 11:20:40 +01:00
Erik Johnston
4df11b5039
Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards'
2015-05-12 10:28:10 +01:00
Erik Johnston
84e6b4001f
Initial hack at wiring together pagination and backfill
2015-05-11 18:01:31 +01:00
Erik Johnston
17653a5dfe
Move storage.stream._StreamToken to types.RoomStreamToken
2015-05-11 18:01:01 +01:00
Erik Johnston
b036596b75
Prefer to use _simple_*.
2015-05-11 11:24:01 +01:00
Erik Johnston
cd525c0f5a
push_rules table expects an 'id' field
2015-05-11 11:24:01 +01:00
Erik Johnston
22d2f498fa
Fix push rule bug: can't insert bool into small int column
2015-05-10 10:50:51 +01:00
Erik Johnston
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
Mark Haines
1c7912751e
Drop the old table not the new table
2015-05-08 16:04:32 +01:00
Mark Haines
9d36eb4eab
Rename unique constraint
2015-05-08 16:01:55 +01:00
Mark Haines
b0f71db3ff
Remove unsigned
2015-05-08 15:59:51 +01:00
Mark Haines
84e1cacea4
Bump schema version
2015-05-08 15:58:14 +01:00
Mark Haines
6538d445e8
Make the timestamps in server_keys_json bigints
2015-05-08 15:55:17 +01:00
Erik Johnston
0c4ac271ca
Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371
2015-05-05 18:21:19 +01:00
Erik Johnston
0cf7e480b4
And use buffer(...) there as well
2015-05-05 18:20:01 +01:00
Erik Johnston
ed2584050f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany
2015-05-05 18:15:20 +01:00
Erik Johnston
977338a7af
Use buffer(...) when inserting into bytea column
2015-05-05 18:12:53 +01:00
Mark Haines
31049c4d72
Merge pull request #139 from matrix-org/bugs/SYN-369
...
Fix race with cache invalidation. SYN-369
2015-05-05 17:46:13 +01:00
Mark Haines
deb0237166
Add some doc-string
2015-05-05 17:45:11 +01:00
Mark Haines
d18f37e026
Collect the invalidate callbacks on the transaction object rather than passing around a separate list
2015-05-05 17:32:21 +01:00
Erik Johnston
9951542393
Add a comment about the zip(*[zip(sorted(...),...)])
2015-05-05 17:06:55 +01:00
Mark Haines
041b6cba61
SYN-369: Add comments to the sequence number logic in the cache
2015-05-05 16:32:44 +01:00
Mark Haines
63075118a5
Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access
2015-05-05 16:24:04 +01:00
Erik Johnston
531d7955fd
Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely
2015-05-05 16:12:28 +01:00
Mark Haines
bfa4a7f8b0
Invalidate the room_member cache if the current state events updates
2015-05-05 15:43:49 +01:00
Mark Haines
d0fece8d3c
Missing return for when the event was already persisted
2015-05-05 15:39:09 +01:00
Erik Johnston
bdcd7693c8
Fix indentation
2015-05-05 15:14:48 +01:00
Erik Johnston
43c2e8deae
Add support for using executemany
2015-05-05 15:13:25 +01:00
Erik Johnston
1692dc019d
Don't call 'encode_parameter' no-op
2015-05-05 15:00:30 +01:00
Mark Haines
a9aea68fd5
Invalidate the caches from the correct thread
2015-05-05 14:57:08 +01:00
Mark Haines
261d809a47
Sequence the modifications to the cache so that selects don't race with inserts
2015-05-05 14:13:50 +01:00
Erik Johnston
d9cc5de9e5
Correctly name transaction
2015-05-05 10:24:10 +01:00
Erik Johnston
b8940cd902
Remove some unused indexes
2015-05-01 16:14:25 +01:00
David Baker
eb9bd2d949
user_id now in user_threepids
2015-05-01 15:04:37 +01:00
David Baker
9eeb03c0dd
Don't use self.execute: it's designed for fetching stuff
2015-05-01 14:21:25 +01:00
David Baker
7693f24792
No id field on user
2015-05-01 13:55:42 +01:00
David Baker
92b20713d7
More missed get_user_by_id API changes
2015-05-01 13:45:54 +01:00
Erik Johnston
4011cf1c42
Cache latest_event_ids_in_room
2015-05-01 13:06:26 +01:00
Erik Johnston
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
Erik Johnston
fabb7acd45
Fix bug where we reconnected to the database on every query.
2015-05-01 10:24:24 +01:00
Erik Johnston
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
Erik Johnston
8be5284e91
Remove pointless join
2015-04-30 18:49:26 +01:00
Erik Johnston
503e4d3d52
Fix broken SQL
2015-04-30 18:44:47 +01:00
Erik Johnston
00718ae7a9
Need more yield
2015-04-30 18:43:39 +01:00
Erik Johnston
0465560c1a
Add missing param
2015-04-30 18:42:44 +01:00
Erik Johnston
61d05daab1
More join conditions
2015-04-30 18:41:05 +01:00
Erik Johnston
6ead27ddda
Add more conditions on JOINs to make postgres go a little faster.
2015-04-30 18:32:03 +01:00
Erik Johnston
f383d5a801
Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction
2015-04-30 17:12:52 +01:00
Erik Johnston
69d4063651
Add get_rooms_for_user cache
2015-04-30 16:47:51 +01:00
Erik Johnston
3c4c229788
Don't use sub queries, it makes postgres sad
2015-04-30 16:16:53 +01:00
Mark Haines
29400b45b9
SYN-367: Use upsert rather than insert_or_replace
2015-04-30 15:21:31 +01:00
Erik Johnston
2366d28780
Don't needlessly join on state_events
2015-04-30 14:02:06 +01:00
Erik Johnston
1d7702833d
Make simple query rather than long one and then throw away half the results
2015-04-30 10:16:12 +01:00
Erik Johnston
b1ca784aca
Correctly decode, for sqlite and postgres, rows from pushers table
2015-04-29 19:41:14 +01:00
Erik Johnston
4a9dc5b2f5
pushkey' are also bytes.
2015-04-29 19:27:02 +01:00
Erik Johnston
0ade2712d1
Typo
2015-04-29 19:17:25 +01:00
Erik Johnston
50f96f256f
Also remove yield from within lock in the other generator
2015-04-29 19:17:00 +01:00
Erik Johnston
d2d61a8288
Fix deadlock in id_generators. No idea why this was an actual deadlock.
2015-04-29 19:15:23 +01:00
Erik Johnston
3864b3a8e6
Actually return rows
2015-04-29 18:07:36 +01:00
Erik Johnston
0618978238
Typo, args wrong way round
2015-04-29 18:04:35 +01:00
Erik Johnston
09177f4f2e
Decode buffers in same thread
2015-04-29 18:03:42 +01:00
Erik Johnston
472be88674
We store pusher data as bytes
2015-04-29 17:43:46 +01:00
David Baker
12d381bd5d
Decode the data json in the storage layer (was moved but this part was missed)
2015-04-29 17:13:51 +01:00
David Baker
f8c30faf25
Oops, update the contraint too
2015-04-29 16:58:42 +01:00
David Baker
61cd5d9045
Be more postgressive
2015-04-29 16:57:14 +01:00
David Baker
fb95035a65
Be postgressive
2015-04-29 16:53:41 +01:00
David Baker
4669def000
Oops, forgot the schema delta file
2015-04-29 16:50:16 +01:00
Erik Johnston
0337eaf321
txn.execute doesn't return cursors
2015-04-29 16:43:39 +01:00
Erik Johnston
884fb88e28
txn.execute doesn't return cursors
2015-04-29 16:35:20 +01:00
Erik Johnston
d76c058eea
Fix invalid SQL to work in postgres land
2015-04-29 16:30:25 +01:00
Erik Johnston
c8ed9bd278
pushers table requires an access_token
2015-04-29 15:33:27 +01:00
Erik Johnston
970a9b9d2b
We can't use REPLACE when upgrading databases now we have postgres
2015-04-29 13:55:44 +01:00
Erik Johnston
64991b0c8b
Merge pull request #129 from matrix-org/key_distribution
...
Key distribution v2
2015-04-29 13:34:38 +01:00
Mark Haines
e26a3d8d9e
bump database schema version
2015-04-29 13:32:32 +01:00
Mark Haines
a9549fdce3
Use bytea rather than BLOB
2015-04-29 13:16:09 +01:00
Mark Haines
4ad8b45155
Merge branch 'develop' into key_distribution
...
Conflicts:
synapse/config/homeserver.py
2015-04-29 13:15:14 +01:00
Erik Johnston
cd0864121b
Make postgres database error slightly more helpful
2015-04-29 12:12:25 +01:00
Erik Johnston
4932a7e2d9
Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__
2015-04-29 12:12:25 +01:00
Erik Johnston
a08bf11138
Appease PEP8
2015-04-29 11:44:48 +01:00
Erik Johnston
204132a998
Check that postgres database has correct charset set
2015-04-29 11:42:28 +01:00
Erik Johnston
327ca883ec
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-28 13:39:42 +01:00
Erik Johnston
18f8247701
Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit
2015-04-28 12:41:33 +01:00
Mark Haines
9182f87664
Merge pull request #126 from matrix-org/csauth
...
Client / Server Auth Refactor
2015-04-28 11:00:27 +01:00
Mark Haines
55e1bc8920
And don't bump the schema version unnecessarily
2015-04-28 10:54:15 +01:00
Mark Haines
b96c133034
Add server_keys.sql to the current delta rather than creating a new delta
2015-04-28 10:50:00 +01:00
Erik Johnston
8558e1ec73
Make get_max_token into inlineCallbacks so that the lock works.
2015-04-27 15:19:44 +01:00
Erik Johnston
6f8e2d517e
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-27 14:41:40 +01:00
Erik Johnston
2732be83d9
Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.
2015-04-27 13:22:30 +01:00
Erik Johnston
e4c4664d73
Handle the fact that postgres databases can be restarted from under us
2015-04-27 12:40:49 +01:00
David Baker
8a5be236e0
pep8
2015-04-27 11:49:18 +01:00
David Baker
b02e1006b9
Run database check before daemonizing, at the cost of database hygiene.
2015-04-27 11:46:00 +01:00
David Baker
f8152f2708
rename db method to be more informative
2015-04-27 10:16:26 +01:00
David Baker
2f475bd5d5
pep8
2015-04-24 18:15:07 +01:00
David Baker
a7b51f4539
Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266
2015-04-24 18:11:21 +01:00
Mark Haines
eede182df7
Merge branch 'develop' into key_distribution
2015-04-24 10:35:49 +01:00
Mark Haines
4e2f8b8722
Copyright notices
2015-04-24 10:35:29 +01:00
Mark Haines
f30d47c876
Implement remote key lookup api
2015-04-22 14:21:08 +01:00
David Baker
91c8f828e1
pep8
2015-04-17 19:56:04 +01:00
David Baker
8db6832db8
Password reset, finally.
2015-04-17 19:53:47 +01:00
David Baker
117f35ac4a
Add endpoint to get threepids from server
2015-04-17 17:20:18 +01:00
David Baker
4eea5cf6c2
pep8
2015-04-17 16:46:45 +01:00
David Baker
f96ab9d18d
make add3pid servlet work
2015-04-17 16:44:49 +01:00
Erik Johnston
e3417bbbe0
Revert needless change to storage.profile
2015-04-17 16:13:14 +01:00
David Baker
cb03fafdf1
Merge branch 'develop' into csauth
2015-04-17 13:51:10 +01:00
Erik Johnston
0ec346d942
Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea
2015-04-17 11:10:20 +01:00
Erik Johnston
e1c0970c11
PEP8
2015-04-16 11:18:45 +01:00
Erik Johnston
b8092fbc82
Go back to storing JSON in TEXT
2015-04-16 11:17:52 +01:00
Erik Johnston
bc9e69e160
Move encoding and decoding of JSON into storage layer
2015-04-16 11:01:09 +01:00
Erik Johnston
ffad75bd62
Remove mysql/maria support
2015-04-15 17:00:50 +01:00
Mark Haines
a429515bdd
Add methods for storing and retrieving the raw key json
2015-04-15 16:58:35 +01:00
Erik Johnston
cf04cedf21
Change full_schemas/11 to work with postgres
2015-04-15 16:53:47 +01:00
Erik Johnston
5b31afcbd1
Remove debug logging
2015-04-15 16:27:04 +01:00
Erik Johnston
6e91f14d09
Add missing yield
2015-04-15 16:25:07 +01:00
Erik Johnston
ed26e4012b
pushers table requires a unique id.
2015-04-15 16:24:14 +01:00
Erik Johnston
a5c72780e6
Don't pass in removed flag
2015-04-15 15:13:22 +01:00
Erik Johnston
4af32a2817
Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction
2015-04-15 14:51:21 +01:00
Erik Johnston
25d2914fba
Turn off persistance of sent_transactions.response_json
2015-04-15 14:09:35 +01:00
Erik Johnston
cce5d057d3
Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses
2015-04-15 11:43:25 +01:00
Erik Johnston
a971fa9d58
Use try..finally in contextlib.contextmanager
2015-04-15 10:25:43 +01:00
Erik Johnston
ded4128965
Use True for True rather than 1
2015-04-15 10:24:24 +01:00
Erik Johnston
f9e12f79ca
Add missing yield in storage func
2015-04-15 10:24:07 +01:00
Erik Johnston
c756dfeb14
Correctly identify deadlocks
2015-04-15 10:23:42 +01:00
Mark Haines
32e14d8181
Return a sha256 fingerprint rather than the entire tls certificate
2015-04-14 19:10:09 +01:00
Erik Johnston
127fad17dd
Add postgres database engine
2015-04-14 14:50:29 +01:00
Erik Johnston
5a95cd4442
Rename user_ips.user -> user_id
2015-04-14 13:54:09 +01:00
Erik Johnston
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
Erik Johnston
3c741682e5
Correctly increment the _next_id initially
2015-04-14 09:54:44 +01:00
Erik Johnston
86fc9b617c
For backwards compat, make state_groups.id have a type of int, not varchar
2015-04-13 17:03:49 +01:00
Erik Johnston
93937b2b31
Remove duplicate rows
2015-04-10 18:47:03 +01:00
Erik Johnston
c5365dee56
Use case sensitive collations
2015-04-10 18:46:33 +01:00
Erik Johnston
4103b1c470
DROP indexes before recreating them
2015-04-10 18:46:09 +01:00
Erik Johnston
4d5b098626
Use LONGBLOB and TEXT for arbitary length rows
2015-04-10 18:45:09 +01:00
Erik Johnston
7ed2ec3061
Handle the fact that in sqlite binary data might be stored as unicode or bytes
2015-04-10 13:41:54 +01:00
Erik Johnston
ce797ad373
Bump schema version
2015-04-10 11:22:30 +01:00
Erik Johnston
7e863c51e6
Use unsigned bigint
2015-04-10 11:22:04 +01:00
Erik Johnston
0f12772e32
SQLite wants INTEGER and not BIGINT for primary keys
2015-04-10 11:16:09 +01:00
Erik Johnston
d5d4281647
Update full_schemas/16 to match delta files. Add delta/16 scripts
2015-04-10 10:59:46 +01:00
Erik Johnston
cda4a6f93f
Revert non-trivial changes to upgrade scripts
2015-04-10 10:19:50 +01:00
Erik Johnston
e2722f58ee
Fix schema again
2015-04-10 10:16:29 +01:00
Erik Johnston
a1665c5094
Revert non-trivial schema changes and move them to a new schema version.
2015-04-10 10:05:44 +01:00
Erik Johnston
2ded344620
Remove unused import
2015-04-09 13:46:06 +01:00
Erik Johnston
9707acfc40
Remove spurious spaces
2015-04-09 13:45:20 +01:00
Erik Johnston
8bf285e082
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-09 13:13:26 +01:00
Erik Johnston
8ad0f4912e
Stream ordering and out of order insertions.
...
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
2015-04-09 11:41:36 +01:00
Erik Johnston
279a547a8b
Use generic db exceptions rather than sqlite3 specific ones
2015-04-08 16:53:48 +01:00
Mark Haines
83f5125d52
Merge pull request #114 from matrix-org/improve_get_event_cache
...
Improve get event cache
2015-04-08 16:50:27 +01:00
Erik Johnston
3887350e47
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-08 14:11:07 +01:00
Erik Johnston
c1b34af441
Move database timer logging to seperate logger
2015-04-08 13:31:19 +01:00
Erik Johnston
9a05795619
Retry transaction, not SQL query
2015-04-08 13:11:28 +01:00
Erik Johnston
24d8134ac1
Fix maria engine to correctly recognize deadlocks
2015-04-08 13:10:54 +01:00
Erik Johnston
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
Erik Johnston
49d6aa1394
Retry on deadlock
2015-04-07 15:28:37 +01:00
Erik Johnston
0bfa78b39b
PEP8
2015-04-07 12:16:05 +01:00
Erik Johnston
6bc9edd8b2
Fix prepare_sqlite3_database's convert_param_style
2015-04-07 12:13:58 +01:00
Erik Johnston
05a35d62b6
Bump database version
2015-04-07 12:10:15 +01:00
Erik Johnston
8574bf62dc
Add index to presence table
2015-04-07 12:09:36 +01:00
Erik Johnston
0af5f5efaf
Don't use multiple UNIQUE constraints; it will cause deadlocks
2015-04-07 12:08:35 +01:00
Erik Johnston
c8d3f6486d
Implement or_ignore flag on inserts
2015-04-07 12:06:22 +01:00
Erik Johnston
304111afd0
Don't use AUTOINCREMENT, use an in memory version
2015-04-07 12:05:36 +01:00
Erik Johnston
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
Erik Johnston
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
Kegan Dougal
5e88a09a42
Add same user_id char checks as registration.
2015-03-31 14:00:25 +01:00
Kegan Dougal
cf1fa59f4b
Use a sender localpart instead of a user ID.
...
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
2015-03-31 13:48:03 +01:00
Kegan Dougal
c217504949
Edit SQL schema to use string IDs not ints. Use token as ID. Update tests.
2015-03-31 12:07:56 +01:00
Kegan Dougal
b59aa74556
Fix tests and missing returns on deferreds.
2015-03-31 11:35:45 +01:00
Kegan Dougal
d33ae65efc
Remove more reg/unreg methods. Read config not database for cache.
2015-03-31 11:00:00 +01:00
Paul "LeoNerd" Evans
3e420aebd8
Revert "Add another @cached wrapper, this time on get_presence_state()"
...
This reverts commit ff1fa0fbf8
2015-03-27 16:16:58 +00:00
Paul "LeoNerd" Evans
ff1fa0fbf8
Add another @cached wrapper, this time on get_presence_state()
2015-03-27 15:57:16 +00:00
David Baker
a32e876ef4
Delete pushers when changing password
2015-03-26 13:40:16 +00:00
Paul "LeoNerd" Evans
a198894bf7
Appease pep8
2015-03-26 11:53:58 +00:00
Kegan Dougal
32206dde3f
Fixes from PR comments
2015-03-26 10:11:52 +00:00
Kegan Dougal
4edcbcee3b
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
Paul "LeoNerd" Evans
953e40f9dc
Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one
2015-03-25 19:12:16 +00:00
David Baker
c1a256cc4c
Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API.
2015-03-25 19:06:22 +00:00
Paul "LeoNerd" Evans
f173d40a32
Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key
2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
1b988b051b
Store the rejected reason in (Frozen)Event structs
2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
033a517feb
Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file
2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
9ba6487b3f
Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()
2015-03-25 19:05:34 +00:00
Erik Johnston
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
David Baker
c7023f2155
1) Pushers are now associated with an access token
...
2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
2015-03-24 17:24:15 +00:00
Erik Johnston
0ba393924a
Escape non printing ascii character
2015-03-24 16:31:52 +00:00
Erik Johnston
f488293d96
Don't reinsert into event_edges
2015-03-24 16:20:26 +00:00
Erik Johnston
1aa44939fc
Fix bugs in transactions storage
2015-03-24 16:20:05 +00:00
Erik Johnston
5a447098dd
Don't use room hosts table
2015-03-24 16:19:24 +00:00
Erik Johnston
9e98f1022a
Don't order by rowid
2015-03-24 16:19:01 +00:00
Erik Johnston
9115421ace
Use _simple_upsert
2015-03-24 16:17:39 +00:00
David Baker
d19e79ecc9
Make deleting other access tokens when you change your password actually work
2015-03-24 15:33:48 +00:00
Erik Johnston
6e7131f02f
Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.
2015-03-23 15:38:56 +00:00
Erik Johnston
9a7f496298
Sanitize RoomMemberStore
2015-03-23 15:29:04 +00:00
David Baker
d98660a60d
Implement password changing (finally) along with a start on making client/server auth more general.
2015-03-23 14:20:28 +00:00
Erik Johnston
d5272b1d2c
Use 'update or insert' rather than on 'conflict replace'
2015-03-23 14:02:34 +00:00
Erik Johnston
278149f533
Sanitize TransactionStore
2015-03-23 13:43:21 +00:00
Paul "LeoNerd" Evans
72d8406409
Put a cache on get_aliases_for_room
2015-03-20 19:21:13 +00:00
Paul "LeoNerd" Evans
a63b4f7101
Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale
2015-03-20 18:25:49 +00:00
Paul "LeoNerd" Evans
0f86312c4c
Pull out the cache logic from the @cached wrapper into its own class we can reuse
2015-03-20 18:25:42 +00:00
Paul "LeoNerd" Evans
b1022ed8b5
func(*EXPR) is valid Python syntax, really...
2015-03-20 17:47:45 +00:00
Erik Johnston
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
Erik Johnston
80cd08c190
PEP8
2015-03-20 16:03:25 +00:00
Erik Johnston
9517f4da4d
Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement
2015-03-20 16:02:47 +00:00
Erik Johnston
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
Paul "LeoNerd" Evans
ceb61daa70
Add the tiniest of tiny one-element caches to get_room_events_max_id() as it's read every time someone hits eventstream
2015-03-20 15:44:06 +00:00
Erik Johnston
fce0114005
Start removing Tables
2015-03-20 15:05:44 +00:00
Erik Johnston
7e282a53a5
Tidy up _simple_... methods
2015-03-20 15:05:10 +00:00
Paul "LeoNerd" Evans
91cb46191d
Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though
2015-03-20 14:59:45 +00:00
Erik Johnston
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
Erik Johnston
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
Erik Johnston
d288d273e1
Generate transaction id in code
2015-03-20 10:57:44 +00:00
Erik Johnston
d4f50f3ae5
decode_result takes an iterable
2015-03-20 10:57:26 +00:00
Erik Johnston
0d0610870d
Fix up schemas some more
2015-03-20 10:55:31 +00:00
Erik Johnston
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
Erik Johnston
58ed393235
Remove redundant key
2015-03-19 15:12:05 +00:00
Erik Johnston
fae059cc18
Fix up schemas to work with mariadb
2015-03-19 13:42:39 +00:00
Erik Johnston
bb24609158
Clean out event_forward_extremities table when the server rejoins the room
2015-03-18 11:19:47 +00:00
Erik Johnston
89036579ed
Update schema to work with mariadb
2015-03-18 11:18:49 +00:00
Paul "LeoNerd" Evans
93978c5e2b
@cached() annotate get_user_by_token() - achieves a minor DB performance improvement
2015-03-17 17:24:51 +00:00
Paul "LeoNerd" Evans
1489521ee5
Be polite and ensure we use @functools.wraps() when creating a function decorator
2015-03-17 17:19:22 +00:00
Paul "LeoNerd" Evans
be170b1426
Add a metric for the scheduling latency of SQL queries
2015-03-16 17:21:59 +00:00
Erik Johnston
cd2539ab2a
Merge pull request #110 from matrix-org/fix_ban
...
Fix ban
2015-03-16 15:36:52 +00:00
Kegan Dougal
f0d6f724a2
Set the service ID as soon as it is known.
2015-03-16 15:24:32 +00:00
Erik Johnston
f1d2b94e0b
Copy dict of context.current_state before changing it.
2015-03-16 15:13:05 +00:00
Kegan Dougal
835e01fc70
Minor PR comment tweaks.
2015-03-16 10:16:59 +00:00
Kegan Dougal
f9232c7917
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/appservice.py
2015-03-16 10:09:15 +00:00
Paul "LeoNerd" Evans
e75fa8bbbf
Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
63cb7ece62
Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f1fbe3e09f
Rename TimerMetric to DistributionMetric; as it could count more than just time
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
642f725fd7
Pretend the 'getEvent' cache is just another cache in the set of all the others for metric
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
fa319a5786
Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
25187ab674
Collect per-SQL-verb timer stats on query execution time
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
e1a7e3564f
Delete a couple of TODO markers of monitoring stats now done
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
8664599af7
Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
b0cdf097f4
Sprinkle some CacheCounterMetrics around the synapse.storage layer
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
a7a28f85ae
Appease pep8
2015-03-11 17:32:43 +00:00
Paul "LeoNerd" Evans
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
Paul "LeoNerd" Evans
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
David Baker
04f8478aaa
Add the master push rule for the break-my-push button. Allow server default rules to be disabled by default.
2015-03-10 17:26:25 +00:00
David Baker
8916acbc13
These aren't defined for redacted events so don't crash
2015-03-10 11:21:37 +00:00
Kegan Dougal
db1fbc6c6f
Fix remaining scheduler bugs. Add more informative logging.
2015-03-10 10:04:20 +00:00
Kegan Dougal
7e0bba555c
Remove unused import
2015-03-09 17:48:37 +00:00
Kegan Dougal
04c9751f24
Bug fixes whilst putting it all together
2015-03-09 17:45:41 +00:00
Kegan Dougal
b98cd03193
Use event IDs instead of dumping event content in the txns table.
2015-03-09 17:25:20 +00:00
Kegan Dougal
21fd84dcb8
Use seconds; start gluing in the AS scheduler into the AS handler.
2015-03-09 17:01:19 +00:00
Kegan Dougal
0a60bbf4fa
Finish appservice txn storage impl and tests.
2015-03-09 15:53:03 +00:00
Kegan Dougal
1ead1caa18
Implement create_appservice_txn with tests.
2015-03-09 13:54:20 +00:00
Kegan Dougal
1c2dcf762a
Partially implement txn store methods with tests.
2015-03-09 13:10:31 +00:00
Kegan Dougal
406d32f8b5
Start implementing ApplicationServiceTransactionStore
2015-03-06 17:35:14 +00:00
Kegan Dougal
34ce2ca62f
Merge branch 'develop' into application-services-txn-reliability
2015-03-06 17:28:49 +00:00
Kegan Dougal
4a6afa6abf
Assign the AS ID from the database; replace old placeholder txn id.
2015-03-06 17:27:55 +00:00
Kegan Dougal
01c099d9ef
Add appservice txns sql schema
2015-03-06 17:16:47 +00:00
Kegan Dougal
2602ddc379
Apply clarity and docstrings
2015-03-06 16:16:14 +00:00
Kegan Dougal
0354659f9d
Finish synapse.appservice.scheduler implementation.
...
With tests to assert behaviour. Not hooked up yet. Stub datastore methods
not implemented yet.
2015-03-06 16:09:05 +00:00
Kegan Dougal
f260cb72cd
Flesh out more stub functions.
2015-03-06 15:12:24 +00:00
Kegan Dougal
141ec04d19
Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions.
2015-03-06 14:53:35 +00:00
David Baker
cf66ddc1b4
Schema change as delta in v14
2015-03-06 14:11:49 +00:00
David Baker
ed72fc3a50
Merge branch 'develop' into pushrules2
...
Conflicts:
synapse/storage/schema/pusher.sql
2015-03-04 15:24:21 +00:00
Erik Johnston
d56c01fff4
Note that we don't specify execution order
2015-03-04 15:10:05 +00:00
Erik Johnston
17d319a20d
s/schema_deltas/applied_schema_deltas/
2015-03-04 15:06:22 +00:00
David Baker
92b3dc3219
Merge branch 'develop' into pushrules2
2015-03-04 14:56:41 +00:00
Erik Johnston
5681264faa
s/%r/%s/
2015-03-04 14:21:53 +00:00
Erik Johnston
f701197227
Add example directory structures in doc
2015-03-04 14:20:14 +00:00
