Erik Johnston
7723b4caa4
Ignore new rejected events when working out forward extremeties.
2017-02-08 14:48:06 +00:00
Erik Johnston
af6da6db2d
Merge pull request #1784 from morteza-araby/user-admin
...
Administration functionalities
2017-02-06 16:21:10 +01:00
Erik Johnston
38258a0976
Bump cache sizes for common membership queries
2017-02-02 18:45:55 +00:00
Erik Johnston
832e9c52ca
Comment
2017-02-02 13:09:56 +00:00
Erik Johnston
54a79c1d37
Make presence.get_new_events a bit faster
...
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
Morteza Araby
2849d3f29d
admin,storage: added more administrator functionalities
...
administrators can now:
- Set displayname of users
- Update user avatars
- Search for users by user_id
- Browse all users in a paginated API
- Reset user passwords
- Deactivate users
Helpers for doing paginated queries has also been added to storage
Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
Erik Johnston
fac3c03087
Be more agressive about purging old room event_push_actions
2017-02-01 18:27:24 +00:00
Erik Johnston
fbfe44bb4d
Doc args
2017-02-01 17:52:57 +00:00
Erik Johnston
d61a04583e
Comment
2017-02-01 17:35:23 +00:00
Erik Johnston
7e919bdbd0
Include newly joined users in /keys/changes API
2017-02-01 17:33:16 +00:00
Erik Johnston
ebfaff84c9
Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
...
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
Erik Johnston
62f6b86ba7
Merge pull request #1868 from matrix-org/erikj/replication_cache
...
Only invalidate membership caches based on the cache stream
2017-02-01 13:12:30 +00:00
Erik Johnston
f6124311fd
Add m.room.member type to query
2017-02-01 11:59:17 +00:00
Erik Johnston
88a4d54883
Merge pull request #1867 from matrix-org/erikj/member_index
...
Add an index to make membership queries faster
2017-02-01 11:44:27 +00:00
Erik Johnston
368c88c487
Add a small cache get_all_new_events
2017-02-01 10:50:44 +00:00
Erik Johnston
5deaf9e30b
Up get_latest_event_ids_in_room cache
2017-02-01 10:39:41 +00:00
Erik Johnston
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
Erik Johnston
ac001dabdc
Merge pull request #1864 from matrix-org/erikj/device_list_fixes
...
Fix clearing out old device list outbound pokes
2017-01-31 13:35:35 +00:00
Erik Johnston
bfb3d255b1
Merge pull request #1862 from matrix-org/erikj/presence_update
...
Use DB cache of joined users for presence
2017-01-31 13:23:24 +00:00
Erik Johnston
ab55794b6f
Fix deletion of old sent devices correctly
2017-01-31 13:22:41 +00:00
Erik Johnston
d3169e8d28
Only fetch with row ts and count > 1
2017-01-31 11:20:03 +00:00
Erik Johnston
05b9f48ee5
Fix clearing out old device list outbound pokes
2017-01-31 10:08:55 +00:00
Erik Johnston
4b3403ca9b
Stream cache invalidations for room membership storage functions
2017-01-30 17:28:22 +00:00
Erik Johnston
1c13c9f6b6
Don't have such a large cache
2017-01-30 17:12:14 +00:00
Erik Johnston
c7a26b7c32
Fix unit tests
2017-01-30 17:11:24 +00:00
Erik Johnston
c2c9a78db9
Noop device key changes if they're the same
2017-01-30 16:55:04 +00:00
Erik Johnston
e75a779d9e
Fix query
2017-01-30 16:38:20 +00:00
Erik Johnston
828db669ec
Use get_users_in_room and declare it iterable
2017-01-30 16:37:22 +00:00
Erik Johnston
3670025e64
Rename func
2017-01-30 14:11:31 +00:00
Erik Johnston
d360c97ae1
Clear out old destination pokes.
2017-01-30 10:14:37 +00:00
Erik Johnston
76100203ab
Always use the latest stream_id, sent or unsent
2017-01-30 10:14:25 +00:00
Erik Johnston
d1e1fd6210
Add ts column to device_lists_outbound_pokes
2017-01-27 15:23:48 +00:00
Erik Johnston
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
Erik Johnston
84a35f32c7
Comment
2017-01-27 10:35:12 +00:00
Erik Johnston
c517a19c2d
Comment
2017-01-27 10:33:26 +00:00
Erik Johnston
738a2867c8
SQL param ordering
2017-01-27 10:31:29 +00:00
Erik Johnston
755adff0e4
User if rather than for
2017-01-27 10:31:06 +00:00
Erik Johnston
888c59c955
Better name
2017-01-27 10:29:47 +00:00
Erik Johnston
f25a4a4692
Remove unused param
2017-01-27 10:27:39 +00:00
Erik Johnston
31aca5589c
Fix on sqlite: use left rather than outer join
2017-01-26 16:55:50 +00:00
Erik Johnston
76d40f4904
Handle users leaving rooms
2017-01-26 16:39:33 +00:00
Erik Johnston
fbfad76c03
Add comments
2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4
Fix up sending of m.device_list_update edus
2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
Erik Johnston
fdf2a31a51
Typo
2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092
Refactor to calculate state delta outside transaction
2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485
Comments
2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5
Spelling
2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67
Update all call sites after rename
2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9
Comments
2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e
Calculate the forward extremeties once
2017-01-20 14:28:53 +00:00
Erik Johnston
09eb08f910
Derive current_state_events from state groups
2017-01-20 11:52:51 +00:00
Erik Johnston
4ec1cf49e2
Lower loading events log to DEBUG
2017-01-17 17:28:32 +00:00
Erik Johnston
d906206049
Increase state_group_cache_size
2017-01-17 11:31:08 +00:00
Erik Johnston
897f8752da
Up cache max entries for state
2017-01-16 15:08:17 +00:00
Erik Johnston
01521299c7
Increase cache size limit
2017-01-16 11:56:51 +00:00
Erik Johnston
2fae34bd2c
Optionally measure size of cache by sum of length of values
2017-01-13 17:46:17 +00:00
Erik Johnston
f0325a9ccc
Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index
...
Change device_inbox stream index to include user
2017-01-13 15:14:51 +00:00
Erik Johnston
c050f493dd
Add comment
2017-01-13 15:14:41 +00:00
Erik Johnston
ab655dca33
Explicitly close the cursor
2017-01-10 15:15:25 +00:00
Erik Johnston
5a32e9273e
Don't disable autocommit
2017-01-10 15:11:27 +00:00
Erik Johnston
caddadfc5a
Change device_inbox stream index to include user
...
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
Erik Johnston
dd52d4de4c
Limit number of entries to prefill from cache
...
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.
In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
Matthew Hodgson
bd85434cb3
Merge branch 'release-v0.18.6' into develop
2017-01-05 13:58:19 +00:00
Matthew Hodgson
8cfc0165e9
fix annoying typos
2017-01-05 13:39:43 +00:00
Matthew Hodgson
b31ed22738
Merge branch 'release-v0.18.6' into develop
2017-01-05 13:03:02 +00:00
Mark Haines
dd3df11c55
More logging for the linearizer and for get_events
2017-01-05 12:32:47 +00:00
David Baker
0c88ab1844
Add /account/3pid/delete endpoint
...
Also fix a typo in a comment
2016-12-20 18:27:30 +00:00
Richard van der Hoff
edc6a1e4f9
Add some logging for syncing to_device events
...
Attempt to track down the loss of to_device events
(https://github.com/vector-im/riot-web/issues/2711 etc).
2016-12-15 18:16:10 +00:00
Erik Johnston
c3208e45c9
Fixup membership query
2016-12-14 10:46:58 +00:00
Erik Johnston
8588d0eb3d
Merge pull request #1697 from matrix-org/erikj/fix_bg_member
...
Fix background update that prematurely stopped
2016-12-12 17:20:17 +00:00
Erik Johnston
1574b839e0
Merge pull request #1676 from matrix-org/erikj/room_list
...
Add new API appservice specific public room list
2016-12-12 17:00:10 +00:00
Erik Johnston
7ec2bf9b77
Fix background update that prematurely stopped
2016-12-12 16:54:58 +00:00
Erik Johnston
09cbcb78d3
Add cache to get_public_room_ids_at_stream_id
2016-12-12 14:41:51 +00:00
Erik Johnston
631376e2ac
Notify replication. Use correct network_id
2016-12-12 14:28:15 +00:00
Erik Johnston
d45c984653
Docstring
2016-12-12 11:00:27 +00:00
Erik Johnston
1697f6a323
Merge pull request #1680 from matrix-org/erikj/joined_rooms
...
Add new room membership APIs
2016-12-09 11:35:48 +00:00
Erik Johnston
52d12ca782
Add /room/<room_id>/joined_members API
...
This returns the currently joined members in the room with their display
names and avatar urls. This is more efficient than /members for large
rooms where you don't need the full events.
2016-12-08 13:32:07 +00:00
Erik Johnston
c45d8e9ba2
Add profile data to the room_membership table for joins
2016-12-08 13:08:41 +00:00
Richard van der Hoff
bd07a35c29
Fix result of insert_receipt
...
This should fix the absence of notifications when new receipts arrive.
2016-12-08 12:11:34 +00:00
Erik Johnston
2687af82d4
Comments
2016-12-07 09:58:33 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Richard van der Hoff
aa09d6b8f0
Rip out more refresh_token code
...
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.
Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston
46ca345b06
Don't send old events as federation
2016-11-24 12:29:02 +00:00
Erik Johnston
1985860c6e
Comment
2016-11-23 15:59:59 +00:00
Erik Johnston
2ac516850b
More efficient notif count queries
2016-11-23 15:57:04 +00:00
Erik Johnston
302fbd218d
Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn
...
Split out federation transaction sending to a worker
2016-11-23 15:39:12 +00:00
Erik Johnston
b69f76c106
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn
2016-11-23 11:31:53 +00:00
Erik Johnston
0163466d72
Ignore AS users when fetching push rules
...
By ignoring AS users early on when fetching push rules for a room we can
avoid needlessly hitting the DB and filling up the caches.
2016-11-23 11:01:01 +00:00
Erik Johnston
54fed21c04
Fix tests and flake8
2016-11-22 18:18:31 +00:00
Erik Johnston
90565d015e
Invalidate retry cache in both directions
2016-11-22 17:45:44 +00:00
Kegan Dougal
a2a6c1c22f
Fail with a coherent error message if /sync?filter=
is invalid
2016-11-21 13:15:25 +00:00
Erik Johnston
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
c7546b3cdb
Merge pull request #1617 from matrix-org/erikj/intern_state_dict
...
Correctly intern keys in state cache
2016-11-15 16:45:55 +00:00
Erik Johnston
f9d156d270
New Flake8 fixes
2016-11-15 11:22:29 +00:00
Erik Johnston
587d8ac60f
Correctly intern keys in state cache
2016-11-08 11:53:25 +00:00
Erik Johnston
63772443e6
Comment
2016-11-04 10:53:42 +00:00
Erik Johnston
a3f6576084
Remove unused but buggy function
2016-11-04 10:48:20 +00:00
Erik Johnston
64c6566980
Remove spurious comment
2016-11-03 15:04:32 +00:00
Erik Johnston
8fd4d9129f
Replace postgres GIN with GIST
...
This is because GIN can be slow to write too, especially when the table
gets large.
2016-11-03 15:00:03 +00:00
Erik Johnston
760469c812
Continue to clean up received_transactions
2016-11-01 11:42:08 +00:00
Erik Johnston
f09d2b692f
Removed unused stuff
2016-10-31 17:10:56 +00:00
Erik Johnston
4c3eb14d68
Increase batching of sent transaction inserts
...
This should further reduce the number of individual inserts,
transactions and updates that are required for keeping sent_transactions
up to date.
2016-10-31 16:07:45 +00:00
Erik Johnston
a9111786f9
Use most recently added binding, not most recently seen user.
2016-10-27 14:32:45 +01:00
Erik Johnston
1fc1bc2a51
Fix user_threepids schema delta
...
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.
This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
2016-10-27 14:14:44 +01:00
Erik Johnston
d04e2ff3a4
Fix incredubly slow back pagination query
...
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
2016-10-24 13:35:51 +01:00
Erik Johnston
78c083f159
Merge pull request #1164 from pik/error-codes
...
Clarify Error codes for GET /filter/
2016-10-19 14:26:17 +01:00
David Baker
0108ed8ae6
Latest delta is now 37
2016-10-19 11:40:35 +01:00
David Baker
a7f48320b1
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-19 11:28:56 +01:00
David Baker
df2a616c7b
Convert emails to lowercase when storing
...
And db migration sql to convert existing addresses.
2016-10-19 11:13:55 +01:00
pik
e8b1d2a452
Refactor test_filter to use real DataStore
...
* add tests for filter api errors
2016-10-18 12:17:38 -05:00
Erik Johnston
6942d68247
Bump schema version
2016-10-17 11:17:45 +01:00
Erik Johnston
b59994b454
Remove TODO
2016-10-17 11:17:02 +01:00
Erik Johnston
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
David Baker
29c5922021
Revert part of 6207399
...
older sqlite doesn't support indexes on expressions, lets just
store things lowercase in the db
2016-10-14 16:20:24 +01:00
David Baker
bcb1245a2d
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-14 15:10:38 +01:00
David Baker
62073992c5
Make password reset email field case insensitive
2016-10-14 13:56:53 +01:00
Erik Johnston
3061dac53e
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop
2016-10-11 14:08:29 +01:00
Patrik Oldsberg
9bfc617791
storage/appservice: make appservice methods only relying on the cache synchronous
2016-10-06 15:24:59 +02:00
Erik Johnston
5875a65253
Merge pull request #1145 from matrix-org/erikj/fix_reindex
...
Fix background reindex of origin_server_ts
2016-09-29 13:53:48 +01:00
Erik Johnston
9040c9ffa1
Fix background reindex of origin_server_ts
...
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
2016-09-27 11:23:49 +01:00
Erik Johnston
4974147aa3
Remove duplication
2016-09-27 09:27:54 +01:00
Erik Johnston
13122e5e24
Remove unused variable
2016-09-27 09:21:51 +01:00
Erik Johnston
cf3e1cc200
Fix perf of fetching state in SQLite
2016-09-26 17:16:24 +01:00
Erik Johnston
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston
8009d84364
Match against event_id, rather than room_id
2016-09-21 16:46:59 +01:00
Erik Johnston
dc692556d6
Remove spurious AS clause
2016-09-21 16:28:47 +01:00
Erik Johnston
dc78db8c56
Update correct table
2016-09-21 15:52:44 +01:00
Erik Johnston
4f78108d8c
Readd entries to public_room_list_stream that were deleted
2016-09-21 15:24:22 +01:00
Erik Johnston
0b78d8adf2
Fix _delete_old_forward_extrem_cache query
2016-09-21 15:20:56 +01:00
Erik Johnston
a68807d426
Comment
2016-09-16 11:36:20 +01:00
Erik Johnston
e58a9d781c
Filter remote rooms lists locally
2016-09-16 10:19:32 +01:00
Erik Johnston
de4f798f01
Handling expiring stream extrems correctly.
2016-09-15 17:34:59 +01:00
Erik Johnston
955f34d23e
Change get_pos_of_last_change to return upper bound
2016-09-15 15:12:07 +01:00
Erik Johnston
cb3edec6af
Use stream_change cache to make get_forward_extremeties_for_room cache more effective
2016-09-15 14:28:13 +01:00
Erik Johnston
55e6fc917c
Add cache to get_forward_extremeties_for_room
2016-09-15 14:04:28 +01:00
Erik Johnston
e457034e99
Merge pull request #1121 from matrix-org/erikj/public_room_paginate
...
Add pagination support to publicRooms
2016-09-15 13:27:09 +01:00
Erik Johnston
211786ecd6
Stream public room changes down replication
2016-09-15 11:47:23 +01:00
Erik Johnston
4fb65a1091
Base public room list off of public_rooms stream
2016-09-15 11:27:04 +01:00
Erik Johnston
c566f0ee17
Calculate the public room list from a stream_ordering
2016-09-14 17:42:47 +01:00
Erik Johnston
baffe96d95
Add a room visibility stream
2016-09-14 17:29:19 +01:00
Erik Johnston
ed992ae6ba
Add a DB index to figure out past state at a stream ordering in a room
2016-09-14 16:20:27 +01:00
Erik Johnston
00f51493f5
Fix reindex
2016-09-14 10:18:30 +01:00
Erik Johnston
d5ae1f1291
Ensure we don't mutate state cache entries
2016-09-14 10:03:48 +01:00
Erik Johnston
43253c10b8
Remove redundant event_auth index
2016-09-13 11:47:48 +01:00
Erik Johnston
b17af156c7
Remove where clause
2016-09-12 17:05:54 +01:00
Erik Johnston
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
Erik Johnston
c94de0ab60
Add WHERE clause support to index creation
2016-09-12 16:55:01 +01:00
Erik Johnston
7c679b1118
Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent
...
Create new index concurrently
2016-09-12 14:38:40 +01:00
Erik Johnston
d080b3425c
Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast
...
Optimise /notifications query
2016-09-12 14:34:12 +01:00
Erik Johnston
03a98aff3c
Create new index concurrently
2016-09-12 14:27:01 +01:00
Erik Johnston
fa20c9ce94
Change the index to be stream_ordering, highlight
2016-09-12 14:04:08 +01:00
Erik Johnston
5ef5435529
Remove unused import
2016-09-12 13:32:58 +01:00
Mark Haines
aa7b890cfe
Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync
...
Fix direct to device messages recieved over federation to notify sync
2016-09-12 13:25:23 +01:00
Erik Johnston
7cd6edb947
Use register_background_index_update
2016-09-12 12:54:48 +01:00
Erik Johnston
0294c14ec4
Add back in query change
2016-09-12 12:43:56 +01:00
Erik Johnston
7fe42cf949
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast
2016-09-12 12:37:09 +01:00
Erik Johnston
15ca0c6a4d
Make reindex happen in bg
2016-09-12 12:36:36 +01:00
Mark Haines
a232e06100
Fix direct to device messages recieved over federation to notify sync
2016-09-12 12:30:46 +01:00
Mark Haines
31f85f9db9
Add comments to existing schema deltas that used "CREATE INDEX" directly
2016-09-12 11:00:26 +01:00
Erik Johnston
54417999b6
Revert "Add index to event_push_actions"
2016-09-12 10:39:55 +01:00
Erik Johnston
897d57bc58
Change state fetch query for postgres to be faster
...
It turns out that postgres doesn't like doing a list of OR's and is
about 1000x slower, so we just issue a query for each specific type
seperately.
2016-09-12 10:05:07 +01:00
David Baker
b91e2833b3
Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast
2016-09-09 19:11:34 +01:00
David Baker
f2acc3dcf9
Add index to event_push_actions
...
and remove room_id caluse so it uses it
Mostly from @negativemjark
2016-09-09 18:54:54 +01:00
Mark Haines
647c724573
Use the previous MAX value if any to set the stream_id
2016-09-09 11:52:44 +01:00
Mark Haines
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
Mark Haines
303b6f29f0
Merge pull request #1087 from matrix-org/markjh/reapply_delta
...
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
2016-09-09 11:24:27 +01:00
Erik Johnston
1fe7ca1362
Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop
2016-09-09 11:15:40 +01:00
Mark Haines
0877157353
Just move the schema and add some DROPs
2016-09-09 11:04:47 +01:00
Erik Johnston
2ffec928e2
Reduce batch size to be under SQL limit
2016-09-09 11:03:31 +01:00
Matthew Hodgson
b8f84f99ff
Merge pull request #1081 from matrix-org/dbkr/notifications_only_highlight
...
Implement `only=highlight` on `/notifications`
2016-09-09 00:09:51 +01:00
Mark Haines
43b77c5d97
Only catch databas errors
2016-09-08 17:44:21 +01:00
Mark Haines
7d5b142547
Add a stub run_upgrade
2016-09-08 17:39:11 +01:00
David Baker
c3276aef25
Merge pull request #1080 from matrix-org/dbkr/fix_notifications_api_with_from
...
Fix /notifications API when used with `from` param
2016-09-08 17:35:35 +01:00
Mark Haines
fa722a699c
Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop
2016-09-08 17:35:16 +01:00
Erik Johnston
5c688739d6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reindex_state_groups
2016-09-08 16:52:09 +01:00
Erik Johnston
ebb46497ba
Add delta file
2016-09-08 16:38:54 +01:00
Erik Johnston
5beda10bbd
Reindex state_groups_state after pruning
2016-09-08 16:18:01 +01:00
Mark Haines
a1c8f268e5
Support wildcard device_ids for direct to device messages
2016-09-08 15:13:05 +01:00
Mark Haines
fa9d36e050
Merge branch 'develop' into markjh/direct_to_device_federation
2016-09-08 13:43:43 +01:00
David Baker
4ef222ab61
Implement only=highlight
on /notifications
2016-09-08 13:43:35 +01:00
Erik Johnston
61cd9af09b
Log delta files we're applying
2016-09-08 13:40:46 +01:00
David Baker
c5b49eb7ca
Fix /notifications API when used with from
param
2016-09-08 09:40:10 +01:00
Erik Johnston
b568ca309c
Temporarily disable sequential scans for state fetching
2016-09-08 09:38:54 +01:00
Mark Haines
3c320c006c
Merge pull request #1077 from matrix-org/markjh/device_logging
...
Log the types and values when failing to store devices
2016-09-07 18:24:24 +01:00
Mark Haines
85b51fdd6b
Log the types and values when failing to store devices
2016-09-07 17:19:18 +01:00
Mark Haines
2a0159b8ae
Fix the stream change cache to work over replication
2016-09-07 15:58:00 +01:00
Mark Haines
31a07d2335
Add stream change caches for device messages
2016-09-07 15:27:07 +01:00
Erik Johnston
513188aa56
Comment
2016-09-07 14:53:23 +01:00
Erik Johnston
fadb01551a
Add appopriate framing clause
2016-09-07 14:39:01 +01:00
Erik Johnston
d25c20ccbe
Use windowing function to make use of index
2016-09-07 14:22:22 +01:00
Mark Haines
7d893beebe
Comment the add_messages storage functions
2016-09-07 12:03:37 +01:00
Mark Haines
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
Mark Haines
e020834e4f
Add storage methods for federated device messages
2016-09-06 15:12:13 +01:00
Mark Haines
2ad72da931
Add tables for federated device messages
...
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
2016-09-06 15:10:40 +01:00
Mark Haines
f4164edb70
Move _add_messages_to_device_inbox_txn into a separate method
2016-09-06 11:26:37 +01:00
Erik Johnston
0595413c0f
Scale the batch size so that we're not bitten by the minimum
2016-09-05 15:49:57 +01:00
Erik Johnston
a7032abb2e
Correctly handle reindexing state groups that already have an edge
2016-09-05 15:07:23 +01:00
Erik Johnston
9e6d88f4e2
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-05 15:01:33 +01:00
Erik Johnston
70332a12dd
Take value in a better way
2016-09-05 14:57:14 +01:00
Erik Johnston
373654c635
Comment about sqlite and WITH RECURSIVE
2016-09-05 14:50:36 +01:00
Erik Johnston
485d999c8a
Correctly delete old state groups in purge history API
2016-09-05 14:49:08 +01:00
Erik Johnston
bb36b93f71
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-05 11:53:11 +01:00
Erik Johnston
caed150363
Remove unused imports
2016-09-05 10:52:01 +01:00
Erik Johnston
80a6a445fa
Only fetch local pushers
2016-09-05 10:43:32 +01:00
Erik Johnston
628e65721b
Add comments
2016-09-05 10:41:27 +01:00
Erik Johnston
a99e933550
Add upgrade script that will slowly prune state_groups_state entries
2016-09-05 10:05:36 +01:00
Erik Johnston
3847fa38c4
Make bulk_get_push_rules_for_room use get_joined_users_from_context cache
2016-09-05 10:02:38 +01:00
Mark Haines
81b94c5750
Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound
...
Only return new device messages in /sync
2016-09-02 16:18:34 +01:00
Mark Haines
65fa37ac5e
Only return new device messages in /sync
2016-09-02 15:50:37 +01:00
Erik Johnston
3baf641a48
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-02 14:54:07 +01:00
Erik Johnston
cce957e254
Bump max_entries on get_destination_retry_timings
2016-09-02 14:08:33 +01:00
Erik Johnston
657847e4c6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-02 11:04:48 +01:00
Erik Johnston
598317927c
Limit the length of state chains
2016-09-02 10:41:38 +01:00
Erik Johnston
9e25443db8
Move to storing state_groups_state as deltas
2016-09-01 14:31:26 +01:00
Erik Johnston
44982606ee
Merge pull request #1060 from matrix-org/erikj/state_ids
...
Assign state groups in state handler.
2016-09-01 14:20:42 +01:00
Erik Johnston
0cfd6c3161
Use state_groups table to test existence
2016-08-31 16:25:57 +01:00
Erik Johnston
5405351b14
Lower get_linearized_receipts_for_room cache size
2016-08-31 16:19:44 +01:00
Erik Johnston
1671913287
Merge pull request #1059 from matrix-org/erikj/sent_transaction_delete
...
Clean up old sent transactions
2016-08-31 15:07:33 +01:00
Erik Johnston
c10cb581c6
Correctly handle the difference between prev and current state
2016-08-31 14:26:22 +01:00
Erik Johnston
ef0cc648cf
Clean up old sent transactions
2016-08-31 11:12:02 +01:00
Mark Haines
a662252758
Return the current stream position from add_messages_to_device_inbox
2016-08-31 10:42:52 +01:00
Mark Haines
1aa3e1d287
Add a replication stream for direct to device messages
2016-08-31 10:38:58 +01:00
Erik Johnston
1bb8ec296d
Generate state group ids in state layer
2016-08-31 10:09:46 +01:00
Erik Johnston
5dc2a702cf
Make _state_groups_id_gen a normal IdGenerator
2016-08-30 16:55:11 +01:00
Erik Johnston
55fc17cf4b
Merge pull request #1049 from matrix-org/erikj/presence_users_in_room
...
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-30 10:50:37 +01:00
Erik Johnston
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
Mark Haines
4bbef62124
Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device
2016-08-26 14:35:31 +01:00
Erik Johnston
4daa397a00
Add is_host_joined to slave storage
2016-08-26 13:10:56 +01:00
Erik Johnston
1ccdc1e93a
Cache check_host_in_room
2016-08-26 10:59:40 +01:00
Erik Johnston
778fa85f47
Make sync not pull out full state
2016-08-25 18:59:44 +01:00
Mark Haines
ab34fdecb7
Merge branch 'develop' into markjh/direct_to_device
2016-08-25 18:34:46 +01:00
Mark Haines
641efb6a39
Fix the deduplication of incoming direct-to-device messages
2016-08-25 18:14:02 +01:00
Erik Johnston
721414d98a
Add desc
2016-08-25 17:49:05 +01:00
Mark Haines
e993925279
Add store-and-forward direct-to-device messaging
2016-08-25 17:35:37 +01:00
Erik Johnston
a3dc1e9cbe
Replace context.current_state with context.current_state_ids
2016-08-25 17:32:22 +01:00
Erik Johnston
17f4f14df7
Pull out event ids rather than full events for state
2016-08-25 13:42:44 +01:00
Erik Johnston
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
Erik Johnston
122c7a43c9
Merge pull request #1038 from matrix-org/erikj/receved_txn_purge
...
Delete old received_transactions rows
2016-08-23 11:02:40 +01:00
Erik Johnston
d143f211c8
Merge pull request #1028 from matrix-org/dbkr/notifications_api
...
Add the Notifications API
2016-08-22 18:23:24 +01:00
Erik Johnston
aad8a1a825
Delete old received_transactions
2016-08-22 16:29:46 +01:00
Erik Johnston
e65bc7d315
Merge pull request #1031 from matrix-org/erikj/measure_notifier
...
Add more Measure blocks
2016-08-22 12:13:07 +01:00
Matthew Hodgson
6e80c03d45
Merge branch 'develop' into dbkr/notifications_api
2016-08-20 00:16:18 +01:00
Erik Johnston
37adde32dc
Move defer.returnValue out of Measure
2016-08-19 18:23:44 +01:00
Erik Johnston
47dd8f02a1
Measure _get_event_from_row
2016-08-19 18:23:44 +01:00
Erik Johnston
e6784daf07
Merge pull request #1030 from matrix-org/erikj/cache_contexts
...
Add concept of cache contexts
2016-08-19 16:29:58 +01:00
Erik Johnston
c0d7d9d642
Rename to on_invalidate
2016-08-19 15:13:58 +01:00
Erik Johnston
dc76a3e909
Make cache_context an explicit option
2016-08-19 15:02:38 +01:00
Erik Johnston
f164fd9220
Move _bulk_get_push_rules_for_room to storage layer
2016-08-19 14:29:20 +01:00
Erik Johnston
ba214a5e32
Remove lru option
2016-08-19 14:17:11 +01:00
Erik Johnston
b770435389
Make get_new_events_for_appservice use indices
2016-08-19 10:28:42 +01:00
David Baker
1e4217c90c
Explicit join
2016-08-18 17:53:44 +01:00
David Baker
602c84cd9c
Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api
2016-08-18 17:15:26 +01:00
Erik Johnston
07229bbdae
Add appservice worker
2016-08-18 14:59:55 +01:00
Erik Johnston
9da84a9a1e
Make AppserviceHandler stream events from database
...
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
2016-08-18 11:54:41 +01:00
Erik Johnston
732cf72b86
Fix push_display_name_rename schema update
2016-08-17 18:12:21 +01:00
David Baker
0d7cef0943
Merge pull request #1011 from matrix-org/dbkr/contains_display_name_override
...
Move display name rule
2016-08-17 16:15:45 +01:00
Erik Johnston
949629291c
Do it in storage function
2016-08-16 17:05:34 +01:00
David Baker
ad42322257
Add migration script
...
To port existing rule actions & enable entries to the new name
2016-08-16 16:56:30 +01:00
Erik Johnston
25c2332071
Merge pull request #1010 from matrix-org/erikj/refactor_deletions
...
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
2016-08-16 11:37:53 +01:00
Erik Johnston
a2427981b7
Use cached get_user_by_access_token in slaves
2016-08-16 11:24:32 +01:00
Erik Johnston
6cbd1b495e
Merge branch 'fix_integrity_retry' of https://github.com/Ralith/synapse into Ralith-fix_integrity_retry
2016-08-16 10:50:24 +01:00
Erik Johnston
dc3a00f24f
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
2016-08-15 17:04:39 +01:00
Erik Johnston
89e786bd85
Doc get_next() context manager usage
2016-08-15 13:45:26 +01:00
Erik Johnston
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
Erik Johnston
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
Erik Johnston
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
Erik Johnston
99bbd90b0d
Always run txn.after_callbacks
2016-08-15 09:45:44 +01:00
Benjamin Saunders
8a57cc3123
Add missing database corruption recovery case
...
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
2016-08-14 11:50:22 -07:00
David Baker
b4ecf0b886
Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api
2016-08-11 14:09:13 +01:00
Erik Johnston
a8b946decb
Raise 404 when couldn't find event
2016-08-05 15:31:02 +01:00
Erik Johnston
f0fa66f495
Delete more tables
2016-08-05 10:40:08 +01:00
Benjamin Saunders
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
Erik Johnston
8ad8490cff
Fix typo
2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88
Retry event persistence on IntegrityError
...
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.
This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
2016-08-04 15:02:15 +01:00
Richard van der Hoff
98385888b8
PEP8
2016-08-03 15:42:08 +01:00
Richard van der Hoff
68264d7404
Include device name in /keys/query response
...
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
Richard van der Hoff
530bc862dc
Merge branch 'rav/null_default_device_displayname' into develop
2016-08-03 14:30:32 +01:00
Erik Johnston
80ad710217
Remove other bit of deduplication
2016-08-03 13:25:59 +01:00
Richard van der Hoff
4fec5e57be
Default device_display_name to null
...
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
2016-08-03 11:53:00 +01:00
Erik Johnston
a8a32d2714
Ensure we only persist an event once at a time
2016-08-03 11:23:39 +01:00
Erik Johnston
5aa024e501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader
2016-07-29 11:24:56 +01:00
Mark Haines
8dad08a950
Fix SQL to supply arguments in the same order
2016-07-29 09:57:13 +01:00
Mark Haines
0a7d3cd00f
Create separate methods for getting messages to push
...
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.
The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.
The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Kegan Dougal
370135ad0b
Comment get_unread_push_actions_for_user_in_range function
2016-07-28 16:47:37 +01:00
Erik Johnston
6ede23ff1b
Add more key storage funcs into slave store
2016-07-28 15:41:26 +01:00
Richard van der Hoff
40e539683c
Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload
...
Make the device id on e2e key upload optional
2016-07-27 18:16:57 +01:00
Richard van der Hoff
26cb0efa88
SQL syntax fix
2016-07-27 12:30:22 +01:00
Richard van der Hoff
d47115ff8b
Delete e2e keys on device delete
2016-07-27 12:24:52 +01:00
Richard van der Hoff
2e3d90d67c
Make the device id on e2e key upload optional
...
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.
For active access_tokens without an associated device_id, we ought to register
the device in the devices table.
Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
2016-07-26 23:38:12 +01:00
Mark Haines
87ffd21b29
Fix a couple of bugs in the transaction and keyring code
2016-07-26 19:19:08 +01:00
Richard van der Hoff
33d7776473
Fix typo
2016-07-26 13:32:15 +01:00
Richard van der Hoff
9ad8d9b17c
Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device
2016-07-26 13:29:46 +01:00