Erik Johnston
8b50fe5330
Use BOOLEAN rather than TEXT type
2017-08-21 16:37:29 +01:00
Erik Johnston
175a01f56c
Groups: Fix mising json.load in initial sync
2017-08-21 14:45:56 +01:00
Erik Johnston
ef8e578677
Add bulk group publicised lookup API
2017-08-09 13:36:22 +01:00
Erik Johnston
b880ff190a
Allow update group publicity
2017-08-08 14:19:41 +01:00
Erik Johnston
05e21285aa
Store whether the user wants to publicise their membership of a group
2017-08-08 13:01:46 +01:00
Erik Johnston
966a70f1fa
Update comment
2017-07-24 17:49:39 +01:00
Erik Johnston
629cdfb124
Use join rather than joined, etc.
2017-07-24 14:54:05 +01:00
Erik Johnston
ed666d3969
Fix all the typos
2017-07-24 14:05:09 +01:00
Erik Johnston
b76ef6ccb8
Include users membership in group in summary API
2017-07-24 13:55:39 +01:00
Erik Johnston
851aeae7c7
Check users/rooms are in group before adding to summary
2017-07-24 13:40:56 +01:00
Erik Johnston
96917d5552
Merge pull request #2378 from matrix-org/erikj/group_sync_support
...
Add groups to sync stream
2017-07-21 11:05:39 +01:00
Erik Johnston
0401604222
Merge pull request #2377 from matrix-org/erikj/group_profile_update
...
Add update group profile API
2017-07-20 17:53:39 +01:00
Erik Johnston
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
Erik Johnston
c544188ee3
Add groups to sync stream
2017-07-20 16:36:42 +01:00
Erik Johnston
8209b5f033
Fix a storage desc
2017-07-20 16:22:22 +01:00
Erik Johnston
57826d645b
Fix typo
2017-07-20 13:15:22 +01:00
Erik Johnston
6f443a74cf
Add update group profile API
2017-07-20 09:46:33 +01:00
Erik Johnston
94ecd871a0
Fix typos
2017-07-18 16:38:54 +01:00
Erik Johnston
3e703eb04e
Comment
2017-07-18 10:17:25 +01:00
Erik Johnston
508460f240
Remove sync stuff
2017-07-18 09:55:46 +01:00
Erik Johnston
4540730111
Remove unused tables
2017-07-18 09:38:15 +01:00
Erik Johnston
e96ee95a7e
Remove sync stuff
2017-07-18 09:38:08 +01:00
Erik Johnston
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
Erik Johnston
85fda57208
Add DEFAULT_ROLE_ID
2017-07-14 14:03:54 +01:00
Erik Johnston
4b203bdba5
Correctly increment orders
2017-07-14 14:02:31 +01:00
Erik Johnston
3b0470dba5
Remove unused functions
2017-07-13 13:54:01 +01:00
Erik Johnston
8575e3160f
Comments
2017-07-13 13:52:41 +01:00
Erik Johnston
26451a09eb
Comments
2017-07-12 14:47:18 +01:00
Erik Johnston
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
Erik Johnston
6d586dc05c
Comment
2017-07-12 09:58:37 +01:00
Erik Johnston
410b4e14a1
Move comment
2017-07-11 15:44:18 +01:00
Erik Johnston
e52c391cd4
Rename column to attestation_json
2017-07-11 14:25:46 +01:00
Erik Johnston
0aac30d53b
Comments
2017-07-11 14:23:50 +01:00
Erik Johnston
83936293eb
Comments
2017-07-11 11:42:25 +01:00
Erik Johnston
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
Erik Johnston
f502b0dea1
Perf: Don't filter events for push
...
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
Erik Johnston
ed9a7f5436
Merge pull request #2309 from matrix-org/erikj/user_ip_repl
...
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston
b5e8d529e6
Define CACHE_SIZE_FACTOR once
2017-07-04 09:56:44 +01:00
Erik Johnston
6ff14ddd2e
Make into list
2017-06-29 15:47:37 +01:00
Erik Johnston
5946aa0877
Prefill forward extrems and event to state groups
2017-06-29 15:38:48 +01:00
Erik Johnston
8c23221666
Fix up
2017-06-27 15:53:45 +01:00
Erik Johnston
a0a561ae85
Fix up client ips to read from pending data
2017-06-27 14:46:12 +01:00
Erik Johnston
ed3d0170d9
Batch upsert user ips
2017-06-27 13:37:04 +01:00
Erik Johnston
8abdd7b553
Fix up indices for users_who_share_rooms
2017-06-26 14:01:30 +01:00
Erik Johnston
7fe8ed1787
Store URL cache preview downloads seperately
...
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
Erik Johnston
e0004aa28a
Add desc
2017-06-22 10:03:48 +01:00
Erik Johnston
b668112320
Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar
...
Don't work out users who share room with appservice users
2017-06-21 14:50:24 +01:00
Erik Johnston
dae9a00a28
Initialise exclusive_user_regex
2017-06-21 14:19:33 +01:00
Krombel
4202fba82a
Merge branch 'develop' into avoid_duplicate_filters
2017-06-21 14:48:21 +02:00
Krombel
812c030e87
replaced json.dumps with encode_canonical_json
2017-06-21 14:48:12 +02:00
Erik Johnston
385dcb7c60
Handle thumbnail urls
2017-06-19 17:48:28 +01:00
Erik Johnston
b8b936a6ea
Add API to quarantine media
2017-06-19 17:39:21 +01:00
Erik Johnston
d7fe6b356c
Add shutdown room API
2017-06-19 12:37:27 +01:00
Erik Johnston
dfeca6cf40
Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
...
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
Erik Johnston
d8f47d2efa
Merge pull request #2280 from matrix-org/erikj/share_room_user_dir
...
Include users who you share a room with in user directory
2017-06-16 11:06:00 +01:00
Erik Johnston
052c5d19d5
Merge pull request #2281 from matrix-org/erikj/phone_home_stats
...
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston
a9d6fa8b2b
Include users who share room with requester in user directory
2017-06-15 10:17:21 +01:00
Erik Johnston
4564b05483
Implement updating users who share rooms on the fly
2017-06-15 10:17:17 +01:00
Erik Johnston
72613bc379
Implement initial population of users who share rooms table
2017-06-15 09:59:04 +01:00
Erik Johnston
ebcd55d641
Add DB schema for tracking users who share rooms
2017-06-15 09:45:48 +01:00
Erik Johnston
4b461a6931
Add some more stats
2017-06-15 09:39:39 +01:00
Erik Johnston
617304b2cf
Fix phone home stats
2017-06-14 19:47:15 +01:00
Erik Johnston
d9fd937e39
Fix user directory insertion due to missing room_id
2017-06-13 11:50:24 +01:00
Erik Johnston
505e7e8b9d
Fix up sql
2017-06-13 11:19:18 +01:00
Erik Johnston
6fd7e6db3d
Fix user dir to not assume existence of user
2017-06-13 11:11:26 +01:00
Erik Johnston
fdca6e36ee
Merge pull request #2274 from matrix-org/erikj/cache_is_host_joined
...
Add cache for is_host_joined
2017-06-13 10:58:53 +01:00
Erik Johnston
b58e24cc3c
Tweak the ranking of PG user dir search
2017-06-13 10:16:31 +01:00
Erik Johnston
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
Erik Johnston
a837765e8c
Merge pull request #2266 from matrix-org/erikj/host_in_room
...
Change is_host_joined to use current_state table
2017-06-12 09:49:51 +01:00
Erik Johnston
8060974344
Fix replication
2017-06-09 16:40:52 +01:00
Erik Johnston
b0d975e216
Comments
2017-06-09 16:25:42 +01:00
Erik Johnston
e54d7d536e
Cache state deltas
2017-06-09 16:24:00 +01:00
Erik Johnston
935e588799
Tweak SQL
2017-06-09 13:01:23 +01:00
Erik Johnston
0185b75381
Change is_host_joined to use current_state table
...
This bypasses a bug where using the state groups to figure out if a host
is in a room sometimes errors if the servers isn't in the room. (For
example when the server rejected an invite to a remote room)
2017-06-09 10:52:26 +01:00
Erik Johnston
ea11ee09f3
Ensure we don't use unpersisted state group as prev group
2017-06-08 11:59:57 +01:00
Erik Johnston
c62c480dc6
Merge pull request #2259 from matrix-org/erikj/fix_state_woes
...
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0
Fix bug where state_group tables got corrupted
...
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.
We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston
a053ff3979
Merge pull request #2248 from matrix-org/erikj/state_fixup
...
Faster cache for get_joined_hosts
2017-06-07 14:01:06 +01:00
Erik Johnston
64ed74c01e
When pruning, delete from device_lists_outbound_last_success
2017-06-07 11:20:47 +01:00
Erik Johnston
1a81a1898e
Keep pruning background task
2017-06-07 11:16:56 +01:00
Erik Johnston
6ba21bf2b8
Comments
2017-06-07 11:08:36 +01:00
Erik Johnston
09e4bc0501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup
2017-06-07 11:05:23 +01:00
Erik Johnston
6e2a7ee1bc
Remove spurious log lines
2017-06-07 11:05:17 +01:00
Erik Johnston
65f0513a33
Split up device_lists_outbound_pokes table for faster updates.
2017-06-07 11:02:38 +01:00
Erik Johnston
6f83c4537c
Increase size of IP cache
2017-06-07 10:18:44 +01:00
Erik Johnston
1a01af079e
Handle profile updates in user directory
2017-06-01 15:39:51 +01:00
Erik Johnston
4d039aa2ca
Fix sqlite
2017-06-01 14:58:48 +01:00
Erik Johnston
21e255a8f1
Split the table in two
2017-06-01 14:50:46 +01:00
Erik Johnston
d5477c7afd
Tweak search query
2017-06-01 13:28:01 +01:00
Erik Johnston
02a6108235
Tweak search query
2017-06-01 13:16:40 +01:00
Erik Johnston
59dbb47065
Remove spurious inlineCallbacks
2017-06-01 11:41:29 +01:00
Erik Johnston
0fe6f3c521
Bug fixes and logging
...
- Check if room is public when a user joins before adding to user dir
- Fix typo of field name "content.join_rules" -> "content.join_rule"
2017-06-01 11:09:49 +01:00
Erik Johnston
036362ede6
Order by if they have profile info
2017-06-01 09:41:08 +01:00
Erik Johnston
a757dd4863
Use prefix matching
2017-06-01 09:40:37 +01:00
Erik Johnston
f5cc22bdc6
Comment on why arbitrary comments
2017-05-31 17:30:26 +01:00
Erik Johnston
5dd1b2c525
Use unique indices
2017-05-31 17:29:12 +01:00
Erik Johnston
f1378aef91
Convert to int
2017-05-31 17:03:08 +01:00
Erik Johnston
f091061711
Fix tests
2017-05-31 16:34:40 +01:00
Erik Johnston
4abcff0177
Fix typo
2017-05-31 16:22:36 +01:00
Erik Johnston
63c58c2a3f
Limit number of things we fetch out of the db
2017-05-31 16:17:58 +01:00
Erik Johnston
304880d185
Add stream change cache
2017-05-31 15:46:36 +01:00
Erik Johnston
5d79d728f5
Split out directory and search tables
2017-05-31 15:23:49 +01:00
Erik Johnston
dc51af3d03
Pull max id from correct table
2017-05-31 15:13:49 +01:00
Erik Johnston
350622a107
Handle the server leaving a public room
2017-05-31 15:11:36 +01:00
Erik Johnston
63fda37e20
Add comments
2017-05-31 15:00:29 +01:00
Erik Johnston
293ef29655
Weight differently
2017-05-31 14:29:32 +01:00
Erik Johnston
3b5f22ca40
Add search
2017-05-31 14:00:01 +01:00
Erik Johnston
b5db4ed5f6
Update room column when room becomes unpublic
2017-05-31 13:40:28 +01:00
Erik Johnston
eeb2f9e546
Add user_directory to database
2017-05-31 11:51:01 +01:00
Erik Johnston
dd48f7204c
Add comment
2017-05-30 15:01:22 +01:00
Erik Johnston
04095f7581
Add clobbered event_id
2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e
Add current_state_delta_stream table
2017-05-30 14:44:09 +01:00
Erik Johnston
619e8ecd0c
Handle None state group correctly
2017-05-26 10:46:03 +01:00
Erik Johnston
dfbda5e025
Faster cache for get_joined_hosts
2017-05-25 17:24:44 +01:00
Erik Johnston
11f139a647
Merge pull request #2247 from matrix-org/erikj/auth_event
...
Only store event_auth for state events
2017-05-24 16:46:34 +01:00
Erik Johnston
6e614e9e10
Add background task to clear out old event_auth
2017-05-24 15:23:34 +01:00
Erik Johnston
c049472b8a
Only store event_auth for state events
2017-05-24 15:23:31 +01:00
Erik Johnston
8cf9f0a3e7
Remove redundant invalidation
2017-05-23 09:46:59 +01:00
Erik Johnston
e6618ece2d
Missed an invalidation
2017-05-23 09:36:52 +01:00
Erik Johnston
74bf4ee7bf
Stream count_e2e_one_time_keys cache invalidation
2017-05-22 16:19:22 +01:00
Erik Johnston
58ebb96cce
Fix invalidation of get_users_with_read_receipts_in_room
2017-05-19 14:38:50 +01:00
Erik Johnston
107ac7ac96
Increase size of push rule caches
2017-05-18 17:17:53 +01:00
Erik Johnston
760625acba
Make get_if_app_services_interested_in_user faster
2017-05-18 16:34:44 +01:00
Erik Johnston
d9e3a4b5db
Merge pull request #2230 from matrix-org/erikj/speed_up_get_state
...
Make get_state_groups_from_groups faster.
2017-05-17 17:23:04 +01:00
Erik Johnston
ace23463c5
Merge pull request #2216 from slipeer/app_services_interested_in_user
...
Fix users claimed non-exclusively by an app service don't get notific…
2017-05-17 16:28:50 +01:00
Erik Johnston
bbfe4e996c
Make get_state_groups_from_groups faster.
...
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.
By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
2017-05-17 15:12:15 +01:00
Erik Johnston
85e8092cca
Comment
2017-05-17 10:03:09 +01:00
Erik Johnston
ad53fc3cf4
Short circuit when we have delta ids
2017-05-17 09:57:34 +01:00
Erik Johnston
13f540ef1b
Speed up get_joined_hosts
2017-05-16 16:05:22 +01:00
Erik Johnston
b8492b6c2f
Merge pull request #2224 from matrix-org/erikj/prefill_state
...
Prefill state caches
2017-05-16 15:50:11 +01:00
Erik Johnston
331570ea6f
Remove spurious merge artifacts
2017-05-16 15:33:07 +01:00
Krombel
55af207321
Merge branch 'develop' into avoid_duplicate_filters
2017-05-16 15:29:59 +02:00
Erik Johnston
608b5a6317
Take a copy before prefilling, as it may be a frozendict
2017-05-16 12:55:29 +01:00
Krombel
64953c8ed2
avoid access-error if no filter_id matches
2017-05-15 18:36:37 +02:00
Erik Johnston
e4435b014e
Update comment
2017-05-15 15:11:30 +01:00
Erik Johnston
871605f4e2
Comments
2017-05-15 15:11:30 +01:00
Erik Johnston
e0d2f6d5b0
Add more granular event send metrics
2017-05-15 15:11:30 +01:00
Erik Johnston
bfbc907cec
Prefill state caches
2017-05-15 15:11:13 +01:00
Krombel
eb7cbf27bc
insert whitespace to fix travis build
2017-05-12 12:09:42 +02:00
Krombel
6b95e35e96
add check to only add a new filter if the same filter does not exist previously
...
Signed-off-by: Matthias Kesler <krombel@krombel.de>
2017-05-11 16:05:30 +02:00
Richard van der Hoff
ff3d810ea8
Add a comment to old delta
2017-05-11 12:48:50 +01:00
Richard van der Hoff
34194aaff7
Don't create event_search index on sqlite
...
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff
114f290947
Add more logging for purging
...
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
Richard van der Hoff
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff
dc026bb16f
Tidy purge code and add some comments
...
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
Slipeer
328378f9cb
Fix users claimed non-exclusively by an app service don't get notifications #2211
2017-05-11 11:42:08 +03:00
Richard van der Hoff
8e345ce465
Don't de-delta state groups we're about to delete
2017-05-10 18:44:22 +01:00
Richard van der Hoff
b64d312421
add some logging to purge_history
2017-05-10 18:44:22 +01:00
Erik Johnston
a3648f84b2
Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid
...
Add per user ratelimiting overrides
2017-05-10 15:54:48 +01:00
Erik Johnston
b990b2fce5
Add per user ratelimiting overrides
2017-05-10 11:05:43 +01:00
Richard van der Hoff
a7e9d8762d
Allow clients to upload one-time-keys with new sigs
...
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).
Fixes https://github.com/vector-im/riot-android/issues/1208 , hopefully.
2017-05-09 19:04:56 +01:00
Erik Johnston
40dcf0d856
Merge pull request #2203 from matrix-org/erikj/event_cache_hit_ratio
...
Don't update event cache hit ratio from get_joined_users
2017-05-08 16:52:39 +01:00
Erik Johnston
d3c3026496
Merge pull request #2201 from matrix-org/erikj/store_device_cache
...
Cache check to see if device exists
2017-05-08 16:23:04 +01:00
Erik Johnston
093f7e47cc
Expand docstring a bit
2017-05-08 16:14:46 +01:00
Erik Johnston
6a12998a83
Add missing yields
2017-05-08 16:10:51 +01:00
Erik Johnston
ffad4fe35b
Don't update event cache hit ratio from get_joined_users
...
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston
94e6ad71f5
Invalidate cache on device deletion
2017-05-08 15:55:59 +01:00
Erik Johnston
8571f864d2
Cache one time key counts
2017-05-08 15:34:27 +01:00
Erik Johnston
fc6d4974a6
Comment
2017-05-08 15:33:57 +01:00
Erik Johnston
738ccf61c0
Cache check to see if device exists
2017-05-08 15:32:18 +01:00
Erik Johnston
dcabef952c
Increase client_ip cache size
2017-05-08 15:09:19 +01:00
Erik Johnston
e0f20e9425
Revert "Remove unused import"
...
This reverts commit ab37bef83b
2017-05-08 13:07:43 +01:00
Erik Johnston
fe7c1b969c
Revert "We don't care about forgotten rooms"
...
This reverts commit ad8b316939
2017-05-08 13:07:43 +01:00
Erik Johnston
78f306a6f7
Revert "Speed up filtering of a single event in push"
...
This reverts commit 421fdf7460
2017-05-08 13:07:41 +01:00
Erik Johnston
66f371e8b8
Merge pull request #2176 from matrix-org/erikj/faster_get_joined
...
Make get_joined_users faster
2017-05-05 10:59:55 +01:00
Erik Johnston
e3aa8a7aa8
Merge pull request #2185 from matrix-org/erikj/smaller_caches
...
Optimise caches for single key
2017-05-05 10:19:05 +01:00
Erik Johnston
cf589f2c1e
Fixes
2017-05-05 10:17:56 +01:00
Erik Johnston
8af4569583
Merge pull request #2174 from matrix-org/erikj/current_cache_hosts
...
Add cache for get_current_hosts_in_room
2017-05-05 10:15:24 +01:00
Erik Johnston
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
Erik Johnston
aa93cb9f44
Add comment
2017-05-04 14:59:28 +01:00
Erik Johnston
537dbadea0
Intern host strings
2017-05-04 14:55:28 +01:00
Erik Johnston
07a07588a0
Make caches bigger
2017-05-04 14:52:28 +01:00
Erik Johnston
dfaa58f72d
Fix comment and num args
2017-05-04 14:50:24 +01:00
Erik Johnston
5d8290429c
Reduce size of get_users_in_room
2017-05-04 13:43:19 +01:00
Erik Johnston
7ebf518c02
Make get_joined_users faster
2017-05-03 15:55:54 +01:00
Erik Johnston
2c2dcf81d0
Update comment
2017-05-03 10:00:29 +01:00
Erik Johnston
1827057acc
Comments
2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-03 09:46:40 +01:00
Erik Johnston
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
Erik Johnston
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
Erik Johnston
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
Erik Johnston
3033261891
Merge pull request #2080 from matrix-org/erikj/filter_speed
...
Speed up filtering of a single event in push
2017-04-28 14:17:13 +01:00
Erik Johnston
ab37bef83b
Remove unused import
2017-04-28 09:57:23 +01:00
Erik Johnston
ad8b316939
We don't care about forgotten rooms
2017-04-28 09:52:36 +01:00
Erik Johnston
421fdf7460
Speed up filtering of a single event in push
2017-04-28 09:52:36 +01:00
Richard van der Hoff
c84770b877
Fix bgupdate error if index already exists ( #2167 )
...
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135 .
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
Erik Johnston
f053a1409e
Make state caches cache in ascii
2017-04-25 17:22:55 +01:00
Erik Johnston
f144365281
Comment
2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86
Reduce size of joined_user cache
...
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.
We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76
Reduce _get_state_group_for_event cache size
2017-04-25 11:43:03 +01:00
Erik Johnston
119cb9bbcf
Reduce cache size by not storing deferreds
...
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.
However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.
This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.
As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Erik Johnston
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
Luke Barnard
78f0ddbfad
Merge pull request #2120 from matrix-org/luke/read-markers
...
Implement Read Marker API
2017-04-13 14:21:31 +01:00
Luke Barnard
6a70647d45
Correct logic in is_event_after
2017-04-13 13:46:17 +01:00
Luke Barnard
b9557064bf
Simplify is_event_after logic
2017-04-12 14:36:20 +01:00
Luke Barnard
7f94709066
travis flake8..
2017-04-11 18:35:45 +01:00
Luke Barnard
73880268ef
Refactor event ordering check to events store
2017-04-11 17:34:09 +01:00
Erik Johnston
85657eedf8
Bail on where clause instead
2017-04-11 16:24:31 +01:00
Erik Johnston
b48045a8f5
Don't bother with outer check for now
2017-04-11 16:23:24 +01:00
Erik Johnston
34840cdcef
Fix getting latest device IP for user with no devices
2017-04-11 09:56:54 +01:00
Erik Johnston
2a3e822f44
Comment
2017-04-07 13:47:04 +01:00
Erik Johnston
d72667fcce
Speed up get_current_state_ids
...
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).
get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
2017-04-07 10:10:49 +01:00
Erik Johnston
9f26d3b75b
Deduplicate new deviceinbox rows for replication
2017-04-04 16:21:21 +01:00
Erik Johnston
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
Erik Johnston
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c
Use txn.fetchall() so we can reuse txn
2017-03-30 10:30:05 +01:00
Erik Johnston
3ce8d59176
Increase cache size for _get_state_group_for_event
2017-03-29 14:31:46 +01:00
Erik Johnston
e4df0e189d
Decrank last commit
2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert
2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512
Correctly look up key
2017-03-29 10:56:26 +01:00
Erik Johnston
d28db583da
Merge pull request #2063 from matrix-org/erikj/device_list_batch
...
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be
The algorithm is part of the key id
2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
Erik Johnston
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6
Merge pull request #2062 from matrix-org/erikj/presence_replication
...
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4
Actually call invalidate
2017-03-24 13:28:20 +00:00
Erik Johnston
09f79aaad0
Use presence replication stream to invalidate cache
...
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
Erik Johnston
7fc1f1e2b6
Cache hosts in room
2017-03-24 11:46:24 +00:00
Erik Johnston
d58b1ffe94
Replace some calls to cursor_to_dict
...
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Erik Johnston
6ebe2d23b1
Raise a more helpful exception
2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d
Don't user upsert to persist new one time keys
...
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
2017-03-23 13:17:00 +00:00
Richard van der Hoff
64778693be
fix up some key verif docstrings
2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
Richard van der Hoff
eddce9d74a
Merge pull request #2027 from matrix-org/rav/logcontext_leaks
...
A few fixes to logcontext things
2017-03-20 11:53:36 +00:00
Richard van der Hoff
994d7ae7c5
Remove broken use of clock.call_later
...
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.
We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
Erik Johnston
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae
Avoid resetting state on rejected events
...
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.
This should fix https://github.com/matrix-org/synapse/issues/1935 .
2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5
Refactoring and cleanups
...
A few non-functional changes:
* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
mind-bending conditional.
2017-03-17 15:06:07 +00:00
Erik Johnston
da146657c9
Comments
2017-03-16 13:04:07 +00:00
Erik Johnston
6957bfdca6
Don't recreate so many sets
2017-03-16 11:54:26 +00:00
Richard van der Hoff
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Erik Johnston
bb256ac96f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed
2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e
Fix current_state_events table to not lie
...
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
2017-03-14 10:57:43 +00:00
Luke Barnard
bbeeb97f75
Implement _simple_delete_many_txn, use it to delete devices
...
(But this doesn't implement the same for deleting access tokens or e2e keys.
Also respond to code review.
2017-03-13 17:53:23 +00:00
Erik Johnston
79926e016e
Assume rooms likely haven't changed
2017-03-13 09:50:10 +00:00
Erik Johnston
8ffbe43ba1
Get current state by using current_state_events table
2017-03-10 17:39:35 +00:00
Erik Johnston
64d62e41b8
Noop repated delete device inbox calls from /sync
2017-03-10 10:36:43 +00:00
Richard van der Hoff
6ad71cc29d
Remove spurious SQL logging ( #1972 )
...
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
2017-03-08 18:00:44 +00:00
Erik Johnston
ac5491f563
Select distinct devices from DB
...
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
2017-03-06 11:10:14 +00:00
Erik Johnston
da52d3af31
Fix up
2017-03-03 15:29:13 +00:00
Erik Johnston
d766343668
Add index to device_lists_stream
2017-03-01 15:56:30 +00:00
Erik Johnston
3365117151
Clobber old device list stream entries
2017-03-01 10:21:30 +00:00
Erik Johnston
d638a7484b
Merge pull request #1959 from matrix-org/erikj/intern_once
...
Intern table column names once
2017-02-28 15:15:16 +00:00
Erik Johnston
b84907bdbb
Intern table column names once
2017-02-28 14:38:16 +00:00
Erik Johnston
e4919b9329
Add stream_id index to device_lists_outbound_pokes
...
As this is used for replication streaming
2017-02-28 11:19:06 +00:00
Erik Johnston
8a12b6f1eb
Fix up txn name
2017-02-28 10:15:50 +00:00
Erik Johnston
a41dce8f8a
Remove needless check
2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776
It should be all
2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709
Don't fetch current state in common case
...
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
2017-02-27 18:33:41 +00:00
Erik Johnston
fc5d876dba
Merge pull request #1954 from matrix-org/erikj/cache_device2
...
Cache get_user_devices_from_cache
2017-02-27 16:36:40 +00:00
Erik Johnston
f58dbb02a6
Cache get_user_devices_from_cache
2017-02-27 16:22:12 +00:00
Erik Johnston
1a4f8022e6
Strip newlines from SQL queries
2017-02-23 11:15:31 +00:00
Erik Johnston
b2d20e94fa
Remove lock from rotate notifs
2017-02-22 14:24:02 +00:00
Erik Johnston
b7442c3e2b
Store looping call
2017-02-21 13:59:25 +00:00
Erik Johnston
699be7d1be
Fix up notif rotation
2017-02-18 14:42:39 +00:00
David Baker
4aa29508af
Use TEXT rather than VARCHAR
...
While we're changing anyway
2017-02-17 10:51:49 +00:00
David Baker
b4017539d4
Make the pushers lang field column longer
...
To accommodate things like zh-Hans-CN
Fixes https://github.com/vector-im/riot-ios/issues/1031
2017-02-17 10:42:57 +00:00
Erik Johnston
138e030cfe
Comment
2017-02-16 15:03:36 +00:00
Erik Johnston
502ae6c663
Comment
2017-02-16 14:47:11 +00:00
Erik Johnston
e6acf0c399
Store the default push actions in a more efficient manner
2017-02-16 14:40:24 +00:00
Erik Johnston
04eca2589d
Merge pull request #1916 from matrix-org/erikj/push_actions_delete
...
Aggregate event push actions
2017-02-16 15:28:58 +01:00
Richard van der Hoff
7dcbcca68c
Merge pull request #1921 from matrix-org/rav/fix_key_changes
...
Fix bugs in the /keys/changes api
2017-02-15 11:25:16 +00:00
Richard van der Hoff
fc2f29c1d0
Fix bugs in the /keys/changes api
...
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
passing a `RoomStreamToken`, which meant that it returned the *current*
extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
that a `TypeError` was thrown and we ended up acting as though there was *no*
prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
dictionary - just looking up the state keys in it again meant that we acted
as though there was no prev state. We now check if each member's state has
changed since *any* of the extremities.
Also add/fix some comments.
2017-02-14 13:59:50 +00:00
Erik Johnston
ce3c8df6df
Less aggressive timers
2017-02-14 13:41:24 +00:00
Erik Johnston
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
Erik Johnston
795f8e3fe7
Merge pull request #1873 from matrix-org/erikj/delete_push_actions
...
Be more agressive about purging old room event_push_actions
2017-02-14 14:29:04 +01:00
Erik Johnston
9e617cd4c2
Cache get_presence storage
2017-02-13 13:50:03 +00:00
Erik Johnston
52cd019a54
Make None check explicit
2017-02-08 16:04:29 +00:00
Erik Johnston
f20cd34858
Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem
...
Ignore new rejected events when working out forward extremities.
2017-02-08 16:59:06 +01:00
Erik Johnston
7723b4caa4
Ignore new rejected events when working out forward extremeties.
2017-02-08 14:48:06 +00:00
Erik Johnston
af6da6db2d
Merge pull request #1784 from morteza-araby/user-admin
...
Administration functionalities
2017-02-06 16:21:10 +01:00
Erik Johnston
38258a0976
Bump cache sizes for common membership queries
2017-02-02 18:45:55 +00:00
Erik Johnston
832e9c52ca
Comment
2017-02-02 13:09:56 +00:00
Erik Johnston
54a79c1d37
Make presence.get_new_events a bit faster
...
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
Morteza Araby
2849d3f29d
admin,storage: added more administrator functionalities
...
administrators can now:
- Set displayname of users
- Update user avatars
- Search for users by user_id
- Browse all users in a paginated API
- Reset user passwords
- Deactivate users
Helpers for doing paginated queries has also been added to storage
Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
Erik Johnston
fac3c03087
Be more agressive about purging old room event_push_actions
2017-02-01 18:27:24 +00:00
Erik Johnston
fbfe44bb4d
Doc args
2017-02-01 17:52:57 +00:00
Erik Johnston
d61a04583e
Comment
2017-02-01 17:35:23 +00:00
Erik Johnston
7e919bdbd0
Include newly joined users in /keys/changes API
2017-02-01 17:33:16 +00:00
Erik Johnston
ebfaff84c9
Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
...
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
Erik Johnston
62f6b86ba7
Merge pull request #1868 from matrix-org/erikj/replication_cache
...
Only invalidate membership caches based on the cache stream
2017-02-01 13:12:30 +00:00
Erik Johnston
f6124311fd
Add m.room.member type to query
2017-02-01 11:59:17 +00:00
Erik Johnston
88a4d54883
Merge pull request #1867 from matrix-org/erikj/member_index
...
Add an index to make membership queries faster
2017-02-01 11:44:27 +00:00
Erik Johnston
368c88c487
Add a small cache get_all_new_events
2017-02-01 10:50:44 +00:00
Erik Johnston
5deaf9e30b
Up get_latest_event_ids_in_room cache
2017-02-01 10:39:41 +00:00
Erik Johnston
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
Erik Johnston
ac001dabdc
Merge pull request #1864 from matrix-org/erikj/device_list_fixes
...
Fix clearing out old device list outbound pokes
2017-01-31 13:35:35 +00:00
Erik Johnston
bfb3d255b1
Merge pull request #1862 from matrix-org/erikj/presence_update
...
Use DB cache of joined users for presence
2017-01-31 13:23:24 +00:00
Erik Johnston
ab55794b6f
Fix deletion of old sent devices correctly
2017-01-31 13:22:41 +00:00
Erik Johnston
d3169e8d28
Only fetch with row ts and count > 1
2017-01-31 11:20:03 +00:00
Erik Johnston
05b9f48ee5
Fix clearing out old device list outbound pokes
2017-01-31 10:08:55 +00:00
Erik Johnston
4b3403ca9b
Stream cache invalidations for room membership storage functions
2017-01-30 17:28:22 +00:00
Erik Johnston
1c13c9f6b6
Don't have such a large cache
2017-01-30 17:12:14 +00:00
Erik Johnston
c7a26b7c32
Fix unit tests
2017-01-30 17:11:24 +00:00
Erik Johnston
c2c9a78db9
Noop device key changes if they're the same
2017-01-30 16:55:04 +00:00
Erik Johnston
e75a779d9e
Fix query
2017-01-30 16:38:20 +00:00
Erik Johnston
828db669ec
Use get_users_in_room and declare it iterable
2017-01-30 16:37:22 +00:00
Erik Johnston
3670025e64
Rename func
2017-01-30 14:11:31 +00:00
Erik Johnston
d360c97ae1
Clear out old destination pokes.
2017-01-30 10:14:37 +00:00
Erik Johnston
76100203ab
Always use the latest stream_id, sent or unsent
2017-01-30 10:14:25 +00:00
Erik Johnston
d1e1fd6210
Add ts column to device_lists_outbound_pokes
2017-01-27 15:23:48 +00:00
Erik Johnston
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
Erik Johnston
84a35f32c7
Comment
2017-01-27 10:35:12 +00:00
Erik Johnston
c517a19c2d
Comment
2017-01-27 10:33:26 +00:00
Erik Johnston
738a2867c8
SQL param ordering
2017-01-27 10:31:29 +00:00
Erik Johnston
755adff0e4
User if rather than for
2017-01-27 10:31:06 +00:00
Erik Johnston
888c59c955
Better name
2017-01-27 10:29:47 +00:00
Erik Johnston
f25a4a4692
Remove unused param
2017-01-27 10:27:39 +00:00
Erik Johnston
31aca5589c
Fix on sqlite: use left rather than outer join
2017-01-26 16:55:50 +00:00
Erik Johnston
76d40f4904
Handle users leaving rooms
2017-01-26 16:39:33 +00:00
Erik Johnston
fbfad76c03
Add comments
2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4
Fix up sending of m.device_list_update edus
2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
Erik Johnston
fdf2a31a51
Typo
2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092
Refactor to calculate state delta outside transaction
2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485
Comments
2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5
Spelling
2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67
Update all call sites after rename
2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9
Comments
2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e
Calculate the forward extremeties once
2017-01-20 14:28:53 +00:00
Erik Johnston
09eb08f910
Derive current_state_events from state groups
2017-01-20 11:52:51 +00:00
Erik Johnston
4ec1cf49e2
Lower loading events log to DEBUG
2017-01-17 17:28:32 +00:00
Erik Johnston
d906206049
Increase state_group_cache_size
2017-01-17 11:31:08 +00:00
Erik Johnston
897f8752da
Up cache max entries for state
2017-01-16 15:08:17 +00:00
Erik Johnston
01521299c7
Increase cache size limit
2017-01-16 11:56:51 +00:00
Erik Johnston
2fae34bd2c
Optionally measure size of cache by sum of length of values
2017-01-13 17:46:17 +00:00
Erik Johnston
f0325a9ccc
Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index
...
Change device_inbox stream index to include user
2017-01-13 15:14:51 +00:00
Erik Johnston
c050f493dd
Add comment
2017-01-13 15:14:41 +00:00
Erik Johnston
ab655dca33
Explicitly close the cursor
2017-01-10 15:15:25 +00:00
Erik Johnston
5a32e9273e
Don't disable autocommit
2017-01-10 15:11:27 +00:00
Erik Johnston
caddadfc5a
Change device_inbox stream index to include user
...
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
Erik Johnston
dd52d4de4c
Limit number of entries to prefill from cache
...
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.
In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
Matthew Hodgson
bd85434cb3
Merge branch 'release-v0.18.6' into develop
2017-01-05 13:58:19 +00:00
Matthew Hodgson
8cfc0165e9
fix annoying typos
2017-01-05 13:39:43 +00:00
Matthew Hodgson
b31ed22738
Merge branch 'release-v0.18.6' into develop
2017-01-05 13:03:02 +00:00
Mark Haines
dd3df11c55
More logging for the linearizer and for get_events
2017-01-05 12:32:47 +00:00
David Baker
0c88ab1844
Add /account/3pid/delete endpoint
...
Also fix a typo in a comment
2016-12-20 18:27:30 +00:00
Richard van der Hoff
edc6a1e4f9
Add some logging for syncing to_device events
...
Attempt to track down the loss of to_device events
(https://github.com/vector-im/riot-web/issues/2711 etc).
2016-12-15 18:16:10 +00:00
Erik Johnston
c3208e45c9
Fixup membership query
2016-12-14 10:46:58 +00:00
Erik Johnston
8588d0eb3d
Merge pull request #1697 from matrix-org/erikj/fix_bg_member
...
Fix background update that prematurely stopped
2016-12-12 17:20:17 +00:00
Erik Johnston
1574b839e0
Merge pull request #1676 from matrix-org/erikj/room_list
...
Add new API appservice specific public room list
2016-12-12 17:00:10 +00:00
Erik Johnston
7ec2bf9b77
Fix background update that prematurely stopped
2016-12-12 16:54:58 +00:00
Erik Johnston
09cbcb78d3
Add cache to get_public_room_ids_at_stream_id
2016-12-12 14:41:51 +00:00
Erik Johnston
631376e2ac
Notify replication. Use correct network_id
2016-12-12 14:28:15 +00:00
Erik Johnston
d45c984653
Docstring
2016-12-12 11:00:27 +00:00
Erik Johnston
1697f6a323
Merge pull request #1680 from matrix-org/erikj/joined_rooms
...
Add new room membership APIs
2016-12-09 11:35:48 +00:00
Erik Johnston
52d12ca782
Add /room/<room_id>/joined_members API
...
This returns the currently joined members in the room with their display
names and avatar urls. This is more efficient than /members for large
rooms where you don't need the full events.
2016-12-08 13:32:07 +00:00
Erik Johnston
c45d8e9ba2
Add profile data to the room_membership table for joins
2016-12-08 13:08:41 +00:00
Richard van der Hoff
bd07a35c29
Fix result of insert_receipt
...
This should fix the absence of notifications when new receipts arrive.
2016-12-08 12:11:34 +00:00
Erik Johnston
2687af82d4
Comments
2016-12-07 09:58:33 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Richard van der Hoff
aa09d6b8f0
Rip out more refresh_token code
...
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.
Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston
46ca345b06
Don't send old events as federation
2016-11-24 12:29:02 +00:00
Erik Johnston
1985860c6e
Comment
2016-11-23 15:59:59 +00:00
Erik Johnston
2ac516850b
More efficient notif count queries
2016-11-23 15:57:04 +00:00
Erik Johnston
302fbd218d
Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn
...
Split out federation transaction sending to a worker
2016-11-23 15:39:12 +00:00
Erik Johnston
b69f76c106
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn
2016-11-23 11:31:53 +00:00
Erik Johnston
0163466d72
Ignore AS users when fetching push rules
...
By ignoring AS users early on when fetching push rules for a room we can
avoid needlessly hitting the DB and filling up the caches.
2016-11-23 11:01:01 +00:00
Erik Johnston
54fed21c04
Fix tests and flake8
2016-11-22 18:18:31 +00:00
Erik Johnston
90565d015e
Invalidate retry cache in both directions
2016-11-22 17:45:44 +00:00
Kegan Dougal
a2a6c1c22f
Fail with a coherent error message if /sync?filter= is invalid
2016-11-21 13:15:25 +00:00
Erik Johnston
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
c7546b3cdb
Merge pull request #1617 from matrix-org/erikj/intern_state_dict
...
Correctly intern keys in state cache
2016-11-15 16:45:55 +00:00
Erik Johnston
f9d156d270
New Flake8 fixes
2016-11-15 11:22:29 +00:00
Erik Johnston
587d8ac60f
Correctly intern keys in state cache
2016-11-08 11:53:25 +00:00
Erik Johnston
63772443e6
Comment
2016-11-04 10:53:42 +00:00
Erik Johnston
a3f6576084
Remove unused but buggy function
2016-11-04 10:48:20 +00:00
Erik Johnston
64c6566980
Remove spurious comment
2016-11-03 15:04:32 +00:00
Erik Johnston
8fd4d9129f
Replace postgres GIN with GIST
...
This is because GIN can be slow to write too, especially when the table
gets large.
2016-11-03 15:00:03 +00:00
Erik Johnston
760469c812
Continue to clean up received_transactions
2016-11-01 11:42:08 +00:00
Erik Johnston
f09d2b692f
Removed unused stuff
2016-10-31 17:10:56 +00:00
Erik Johnston
4c3eb14d68
Increase batching of sent transaction inserts
...
This should further reduce the number of individual inserts,
transactions and updates that are required for keeping sent_transactions
up to date.
2016-10-31 16:07:45 +00:00
Erik Johnston
a9111786f9
Use most recently added binding, not most recently seen user.
2016-10-27 14:32:45 +01:00
Erik Johnston
1fc1bc2a51
Fix user_threepids schema delta
...
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.
This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
2016-10-27 14:14:44 +01:00
Erik Johnston
d04e2ff3a4
Fix incredubly slow back pagination query
...
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
2016-10-24 13:35:51 +01:00
Erik Johnston
78c083f159
Merge pull request #1164 from pik/error-codes
...
Clarify Error codes for GET /filter/
2016-10-19 14:26:17 +01:00
David Baker
0108ed8ae6
Latest delta is now 37
2016-10-19 11:40:35 +01:00
David Baker
a7f48320b1
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-19 11:28:56 +01:00
David Baker
df2a616c7b
Convert emails to lowercase when storing
...
And db migration sql to convert existing addresses.
2016-10-19 11:13:55 +01:00
pik
e8b1d2a452
Refactor test_filter to use real DataStore
...
* add tests for filter api errors
2016-10-18 12:17:38 -05:00
Erik Johnston
6942d68247
Bump schema version
2016-10-17 11:17:45 +01:00
Erik Johnston
b59994b454
Remove TODO
2016-10-17 11:17:02 +01:00
Erik Johnston
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
David Baker
29c5922021
Revert part of 6207399
...
older sqlite doesn't support indexes on expressions, lets just
store things lowercase in the db
2016-10-14 16:20:24 +01:00
David Baker
bcb1245a2d
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-14 15:10:38 +01:00
David Baker
62073992c5
Make password reset email field case insensitive
2016-10-14 13:56:53 +01:00
Erik Johnston
3061dac53e
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop
2016-10-11 14:08:29 +01:00
Patrik Oldsberg
9bfc617791
storage/appservice: make appservice methods only relying on the cache synchronous
2016-10-06 15:24:59 +02:00
Erik Johnston
5875a65253
Merge pull request #1145 from matrix-org/erikj/fix_reindex
...
Fix background reindex of origin_server_ts
2016-09-29 13:53:48 +01:00
Erik Johnston
9040c9ffa1
Fix background reindex of origin_server_ts
...
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
2016-09-27 11:23:49 +01:00
Erik Johnston
4974147aa3
Remove duplication
2016-09-27 09:27:54 +01:00
Erik Johnston
13122e5e24
Remove unused variable
2016-09-27 09:21:51 +01:00
Erik Johnston
cf3e1cc200
Fix perf of fetching state in SQLite
2016-09-26 17:16:24 +01:00
Erik Johnston
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston
8009d84364
Match against event_id, rather than room_id
2016-09-21 16:46:59 +01:00
Erik Johnston
dc692556d6
Remove spurious AS clause
2016-09-21 16:28:47 +01:00
Erik Johnston
dc78db8c56
Update correct table
2016-09-21 15:52:44 +01:00
Erik Johnston
4f78108d8c
Readd entries to public_room_list_stream that were deleted
2016-09-21 15:24:22 +01:00
Erik Johnston
0b78d8adf2
Fix _delete_old_forward_extrem_cache query
2016-09-21 15:20:56 +01:00
Erik Johnston
a68807d426
Comment
2016-09-16 11:36:20 +01:00
Erik Johnston
e58a9d781c
Filter remote rooms lists locally
2016-09-16 10:19:32 +01:00
Erik Johnston
de4f798f01
Handling expiring stream extrems correctly.
2016-09-15 17:34:59 +01:00
Erik Johnston
955f34d23e
Change get_pos_of_last_change to return upper bound
2016-09-15 15:12:07 +01:00
Erik Johnston
cb3edec6af
Use stream_change cache to make get_forward_extremeties_for_room cache more effective
2016-09-15 14:28:13 +01:00
Erik Johnston
55e6fc917c
Add cache to get_forward_extremeties_for_room
2016-09-15 14:04:28 +01:00
Erik Johnston
e457034e99
Merge pull request #1121 from matrix-org/erikj/public_room_paginate
...
Add pagination support to publicRooms
2016-09-15 13:27:09 +01:00
Erik Johnston
211786ecd6
Stream public room changes down replication
2016-09-15 11:47:23 +01:00
Erik Johnston
4fb65a1091
Base public room list off of public_rooms stream
2016-09-15 11:27:04 +01:00
Erik Johnston
c566f0ee17
Calculate the public room list from a stream_ordering
2016-09-14 17:42:47 +01:00
Erik Johnston
baffe96d95
Add a room visibility stream
2016-09-14 17:29:19 +01:00
Erik Johnston
ed992ae6ba
Add a DB index to figure out past state at a stream ordering in a room
2016-09-14 16:20:27 +01:00
Erik Johnston
00f51493f5
Fix reindex
2016-09-14 10:18:30 +01:00
Erik Johnston
d5ae1f1291
Ensure we don't mutate state cache entries
2016-09-14 10:03:48 +01:00
Erik Johnston
43253c10b8
Remove redundant event_auth index
2016-09-13 11:47:48 +01:00
Erik Johnston
b17af156c7
Remove where clause
2016-09-12 17:05:54 +01:00
Erik Johnston
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
Erik Johnston
c94de0ab60
Add WHERE clause support to index creation
2016-09-12 16:55:01 +01:00
Erik Johnston
7c679b1118
Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent
...
Create new index concurrently
2016-09-12 14:38:40 +01:00
Erik Johnston
d080b3425c
Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast
...
Optimise /notifications query
2016-09-12 14:34:12 +01:00
Erik Johnston
03a98aff3c
Create new index concurrently
2016-09-12 14:27:01 +01:00
Erik Johnston
fa20c9ce94
Change the index to be stream_ordering, highlight
2016-09-12 14:04:08 +01:00
Erik Johnston
5ef5435529
Remove unused import
2016-09-12 13:32:58 +01:00
Mark Haines
aa7b890cfe
Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync
...
Fix direct to device messages recieved over federation to notify sync
2016-09-12 13:25:23 +01:00
Erik Johnston
7cd6edb947
Use register_background_index_update
2016-09-12 12:54:48 +01:00
Erik Johnston
0294c14ec4
Add back in query change
2016-09-12 12:43:56 +01:00
Erik Johnston
7fe42cf949
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast
2016-09-12 12:37:09 +01:00
Erik Johnston
15ca0c6a4d
Make reindex happen in bg
2016-09-12 12:36:36 +01:00
Mark Haines
a232e06100
Fix direct to device messages recieved over federation to notify sync
2016-09-12 12:30:46 +01:00
Mark Haines
31f85f9db9
Add comments to existing schema deltas that used "CREATE INDEX" directly
2016-09-12 11:00:26 +01:00
Erik Johnston
54417999b6
Revert "Add index to event_push_actions"
2016-09-12 10:39:55 +01:00
Erik Johnston
897d57bc58
Change state fetch query for postgres to be faster
...
It turns out that postgres doesn't like doing a list of OR's and is
about 1000x slower, so we just issue a query for each specific type
seperately.
2016-09-12 10:05:07 +01:00
David Baker
b91e2833b3
Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast
2016-09-09 19:11:34 +01:00
David Baker
f2acc3dcf9
Add index to event_push_actions
...
and remove room_id caluse so it uses it
Mostly from @negativemjark
2016-09-09 18:54:54 +01:00
Mark Haines
647c724573
Use the previous MAX value if any to set the stream_id
2016-09-09 11:52:44 +01:00
Mark Haines
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
Mark Haines
303b6f29f0
Merge pull request #1087 from matrix-org/markjh/reapply_delta
...
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
2016-09-09 11:24:27 +01:00
Erik Johnston
1fe7ca1362
Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop
2016-09-09 11:15:40 +01:00
Mark Haines
0877157353
Just move the schema and add some DROPs
2016-09-09 11:04:47 +01:00
Erik Johnston
2ffec928e2
Reduce batch size to be under SQL limit
2016-09-09 11:03:31 +01:00
