Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
8,937 Commits 4 Branches 615 Tags 63 MiB
b0d975e216
Commit Graph

2056 Commits

Author SHA1 Message Date
Erik Johnston
1827057acc Comments 2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state 2017-05-03 09:46:40 +01:00
Erik Johnston
3e5a62ecd8 Add more granular event send metrics 2017-05-02 14:23:26 +01:00
Erik Johnston
a2c89a225c Prefill state caches 2017-05-02 10:40:31 +01:00
Erik Johnston
7166854f41 Add cache for get_current_hosts_in_room 2017-05-02 10:36:35 +01:00
Erik Johnston
3033261891 Merge pull request #2080 from matrix-org/erikj/filter_speed 
Speed up filtering of a single event in push
 2017-04-28 14:17:13 +01:00
Erik Johnston
ab37bef83b Remove unused import 2017-04-28 09:57:23 +01:00
Erik Johnston
ad8b316939 We don't care about forgotten rooms 2017-04-28 09:52:36 +01:00
Erik Johnston
421fdf7460 Speed up filtering of a single event in push 2017-04-28 09:52:36 +01:00
Richard van der Hoff
c84770b877 Fix bgupdate error if index already exists (#2167) 
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135.

Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
 2017-04-27 15:27:48 +01:00
Erik Johnston
f053a1409e Make state caches cache in ascii 2017-04-25 17:22:55 +01:00
Erik Johnston
f144365281 Comment 2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86 Reduce size of joined_user cache 
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
 2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76 Reduce _get_state_group_for_event cache size 2017-04-25 11:43:03 +01:00
Erik Johnston
119cb9bbcf Reduce cache size by not storing deferreds 
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
 2017-04-25 10:23:11 +01:00
Erik Johnston
e4f3431116 Remove unused cache 2017-04-24 13:27:38 +01:00
Luke Barnard
78f0ddbfad Merge pull request #2120 from matrix-org/luke/read-markers 
Implement Read Marker API
 2017-04-13 14:21:31 +01:00
Luke Barnard
6a70647d45 Correct logic in is_event_after 2017-04-13 13:46:17 +01:00
Luke Barnard
b9557064bf Simplify is_event_after logic 2017-04-12 14:36:20 +01:00
Luke Barnard
7f94709066 travis flake8.. 2017-04-11 18:35:45 +01:00
Luke Barnard
73880268ef Refactor event ordering check to events store 2017-04-11 17:34:09 +01:00
Erik Johnston
85657eedf8 Bail on where clause instead 2017-04-11 16:24:31 +01:00
Erik Johnston
b48045a8f5 Don't bother with outer check for now 2017-04-11 16:23:24 +01:00
Erik Johnston
34840cdcef Fix getting latest device IP for user with no devices 2017-04-11 09:56:54 +01:00
Erik Johnston
2a3e822f44 Comment 2017-04-07 13:47:04 +01:00
Erik Johnston
d72667fcce Speed up get_current_state_ids 
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).

get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
 2017-04-07 10:10:49 +01:00
Erik Johnston
9f26d3b75b Deduplicate new deviceinbox rows for replication 2017-04-04 16:21:21 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Erik Johnston
3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston
e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Erik Johnston
d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch 
Batch sending of device list pokes
 2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication 
Use presence replication stream to invalidate cache
 2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict 
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
 2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall 
This prevents unnecessary construction of lists
 2017-03-23 17:53:49 +00:00
Erik Johnston
6ebe2d23b1 Raise a more helpful exception 2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d Don't user upsert to persist new one time keys 
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
 2017-03-23 13:17:00 +00:00
Richard van der Hoff
64778693be fix up some key verif docstrings 2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed 
Don't send the full event json over replication
 2017-03-21 13:11:15 +00:00
Richard van der Hoff
eddce9d74a Merge pull request #2027 from matrix-org/rav/logcontext_leaks 
A few fixes to logcontext things
 2017-03-20 11:53:36 +00:00
Richard van der Hoff
994d7ae7c5 Remove broken use of clock.call_later 
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.

We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
 2017-03-18 00:01:37 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae Avoid resetting state on rejected events 
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.

This should fix https://github.com/matrix-org/synapse/issues/1935.
 2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5 Refactoring and cleanups 
A few non-functional changes:

* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
  readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
  mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
  mind-bending conditional.
 2017-03-17 15:06:07 +00:00
Erik Johnston
da146657c9 Comments 2017-03-16 13:04:07 +00:00
Erik Johnston
6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Richard van der Hoff
29ed09e80a Fix assertion to stop transaction queue getting wedged 
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
 2017-03-15 12:16:55 +00:00
Erik Johnston
bb256ac96f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed 2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e Fix current_state_events table to not lie 
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
 2017-03-14 10:57:43 +00:00
Luke Barnard
bbeeb97f75 Implement _simple_delete_many_txn, use it to delete devices 
(But this doesn't implement the same for deleting access tokens or e2e keys.

Also respond to code review.
 2017-03-13 17:53:23 +00:00
Erik Johnston
79926e016e Assume rooms likely haven't changed 2017-03-13 09:50:10 +00:00
Erik Johnston
8ffbe43ba1 Get current state by using current_state_events table 2017-03-10 17:39:35 +00:00
Erik Johnston
64d62e41b8 Noop repated delete device inbox calls from /sync 2017-03-10 10:36:43 +00:00
Richard van der Hoff
6ad71cc29d Remove spurious SQL logging (#1972) 
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
 2017-03-08 18:00:44 +00:00
Erik Johnston
ac5491f563 Select distinct devices from DB 
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
 2017-03-06 11:10:14 +00:00
Erik Johnston
da52d3af31 Fix up 2017-03-03 15:29:13 +00:00
Erik Johnston
d766343668 Add index to device_lists_stream 2017-03-01 15:56:30 +00:00
Erik Johnston
3365117151 Clobber old device list stream entries 2017-03-01 10:21:30 +00:00
Erik Johnston
d638a7484b Merge pull request #1959 from matrix-org/erikj/intern_once 
Intern table column names once
 2017-02-28 15:15:16 +00:00
Erik Johnston
b84907bdbb Intern table column names once 2017-02-28 14:38:16 +00:00
Erik Johnston
e4919b9329 Add stream_id index to device_lists_outbound_pokes 
As this is used for replication streaming
 2017-02-28 11:19:06 +00:00
Erik Johnston
8a12b6f1eb Fix up txn name 2017-02-28 10:15:50 +00:00
Erik Johnston
a41dce8f8a Remove needless check 2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776 It should be all 2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709 Don't fetch current state in common case 
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
 2017-02-27 18:33:41 +00:00
Erik Johnston
fc5d876dba Merge pull request #1954 from matrix-org/erikj/cache_device2 
Cache get_user_devices_from_cache
 2017-02-27 16:36:40 +00:00
Erik Johnston
f58dbb02a6 Cache get_user_devices_from_cache 2017-02-27 16:22:12 +00:00
Erik Johnston
1a4f8022e6 Strip newlines from SQL queries 2017-02-23 11:15:31 +00:00
Erik Johnston
b2d20e94fa Remove lock from rotate notifs 2017-02-22 14:24:02 +00:00
Erik Johnston
b7442c3e2b Store looping call 2017-02-21 13:59:25 +00:00
Erik Johnston
699be7d1be Fix up notif rotation 2017-02-18 14:42:39 +00:00
David Baker
4aa29508af Use TEXT rather than VARCHAR 
While we're changing anyway
 2017-02-17 10:51:49 +00:00
David Baker
b4017539d4 Make the pushers lang field column longer 
To accommodate things like zh-Hans-CN

Fixes https://github.com/vector-im/riot-ios/issues/1031
 2017-02-17 10:42:57 +00:00
Erik Johnston
138e030cfe Comment 2017-02-16 15:03:36 +00:00
Erik Johnston
502ae6c663 Comment 2017-02-16 14:47:11 +00:00
Erik Johnston
e6acf0c399 Store the default push actions in a more efficient manner 2017-02-16 14:40:24 +00:00
Erik Johnston
04eca2589d Merge pull request #1916 from matrix-org/erikj/push_actions_delete 
Aggregate event push actions
 2017-02-16 15:28:58 +01:00
Richard van der Hoff
7dcbcca68c Merge pull request #1921 from matrix-org/rav/fix_key_changes 
Fix bugs in the /keys/changes api
 2017-02-15 11:25:16 +00:00
Richard van der Hoff
fc2f29c1d0 Fix bugs in the /keys/changes api 
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
  passing a `RoomStreamToken`, which meant that it returned the *current*
  extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
  that a `TypeError` was thrown and we ended up acting as though there was *no*
  prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
  dictionary - just looking up the state keys in it again meant that we acted
  as though there was no prev state. We now check if each member's state has
  changed since *any* of the extremities.

Also add/fix some comments.
 2017-02-14 13:59:50 +00:00
Erik Johnston
ce3c8df6df Less aggressive timers 2017-02-14 13:41:24 +00:00
Erik Johnston
095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston
795f8e3fe7 Merge pull request #1873 from matrix-org/erikj/delete_push_actions 
Be more agressive about purging old room event_push_actions
 2017-02-14 14:29:04 +01:00
Erik Johnston
9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Erik Johnston
52cd019a54 Make None check explicit 2017-02-08 16:04:29 +00:00
Erik Johnston
f20cd34858 Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem 
Ignore new rejected events when working out forward extremities.
 2017-02-08 16:59:06 +01:00
Erik Johnston
7723b4caa4 Ignore new rejected events when working out forward extremeties. 2017-02-08 14:48:06 +00:00
Erik Johnston
af6da6db2d Merge pull request #1784 from morteza-araby/user-admin 
Administration functionalities
 2017-02-06 16:21:10 +01:00
Erik Johnston
38258a0976 Bump cache sizes for common membership queries 2017-02-02 18:45:55 +00:00
Erik Johnston
832e9c52ca Comment 2017-02-02 13:09:56 +00:00
Erik Johnston
54a79c1d37 Make presence.get_new_events a bit faster 
We do this by caching the set of users a user shares rooms with.
 2017-02-02 13:07:18 +00:00
Morteza Araby
2849d3f29d admin,storage: added more administrator functionalities 
administrators can now:
 - Set displayname of users
 - Update user avatars
 - Search for users by user_id
 - Browse all users in a paginated API
 - Reset user passwords
 - Deactivate users

Helpers for doing paginated queries has also been added to storage

Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
 2017-02-02 14:02:26 +01:00
Erik Johnston
fac3c03087 Be more agressive about purging old room event_push_actions 2017-02-01 18:27:24 +00:00
Erik Johnston
fbfe44bb4d Doc args 2017-02-01 17:52:57 +00:00
Erik Johnston
d61a04583e Comment 2017-02-01 17:35:23 +00:00
Erik Johnston
7e919bdbd0 Include newly joined users in /keys/changes API 2017-02-01 17:33:16 +00:00
Erik Johnston
ebfaff84c9 Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events 
Add a small cache get_all_new_events
 2017-02-01 13:22:02 +00:00
Erik Johnston
62f6b86ba7 Merge pull request #1868 from matrix-org/erikj/replication_cache 
Only invalidate membership caches based on the cache stream
 2017-02-01 13:12:30 +00:00
Erik Johnston
f6124311fd Add m.room.member type to query 2017-02-01 11:59:17 +00:00
Erik Johnston
88a4d54883 Merge pull request #1867 from matrix-org/erikj/member_index 
Add an index to make membership queries faster
 2017-02-01 11:44:27 +00:00
Erik Johnston
368c88c487 Add a small cache get_all_new_events 2017-02-01 10:50:44 +00:00
Erik Johnston
5deaf9e30b Up get_latest_event_ids_in_room cache 2017-02-01 10:39:41 +00:00
Erik Johnston
458b6f4733 Only invalidate membership caches based on the cache stream 
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
 2017-01-31 16:09:03 +00:00
Erik Johnston
21b7375778 Add an index to make membership queries faster 2017-01-31 15:15:57 +00:00
Erik Johnston
ac001dabdc Merge pull request #1864 from matrix-org/erikj/device_list_fixes 
Fix clearing out old device list outbound pokes
 2017-01-31 13:35:35 +00:00
Erik Johnston
bfb3d255b1 Merge pull request #1862 from matrix-org/erikj/presence_update 
Use DB cache of joined users for presence
 2017-01-31 13:23:24 +00:00
Erik Johnston
ab55794b6f Fix deletion of old sent devices correctly 2017-01-31 13:22:41 +00:00
Erik Johnston
d3169e8d28 Only fetch with row ts and count > 1 2017-01-31 11:20:03 +00:00
Erik Johnston
05b9f48ee5 Fix clearing out old device list outbound pokes 2017-01-31 10:08:55 +00:00
Erik Johnston
4b3403ca9b Stream cache invalidations for room membership storage functions 2017-01-30 17:28:22 +00:00
Erik Johnston
1c13c9f6b6 Don't have such a large cache 2017-01-30 17:12:14 +00:00
Erik Johnston
c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston
c2c9a78db9 Noop device key changes if they're the same 2017-01-30 16:55:04 +00:00
Erik Johnston
e75a779d9e Fix query 2017-01-30 16:38:20 +00:00
Erik Johnston
828db669ec Use get_users_in_room and declare it iterable 2017-01-30 16:37:22 +00:00
Erik Johnston
3670025e64 Rename func 2017-01-30 14:11:31 +00:00
Erik Johnston
d360c97ae1 Clear out old destination pokes. 2017-01-30 10:14:37 +00:00
Erik Johnston
76100203ab Always use the latest stream_id, sent or unsent 2017-01-30 10:14:25 +00:00
Erik Johnston
d1e1fd6210 Add ts column to device_lists_outbound_pokes 2017-01-27 15:23:48 +00:00
Erik Johnston
252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston
84a35f32c7 Comment 2017-01-27 10:35:12 +00:00
Erik Johnston
c517a19c2d Comment 2017-01-27 10:33:26 +00:00
Erik Johnston
738a2867c8 SQL param ordering 2017-01-27 10:31:29 +00:00
Erik Johnston
755adff0e4 User if rather than for 2017-01-27 10:31:06 +00:00
Erik Johnston
888c59c955 Better name 2017-01-27 10:29:47 +00:00
Erik Johnston
f25a4a4692 Remove unused param 2017-01-27 10:27:39 +00:00
Erik Johnston
31aca5589c Fix on sqlite: use left rather than outer join 2017-01-26 16:55:50 +00:00
Erik Johnston
76d40f4904 Handle users leaving rooms 2017-01-26 16:39:33 +00:00
Erik Johnston
fbfad76c03 Add comments 2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4 Fix up sending of m.device_list_update edus 2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Erik Johnston
fdf2a31a51 Typo 2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092 Refactor to calculate state delta outside transaction 2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485 Comments 2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5 Spelling 2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67 Update all call sites after rename 2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9 Comments 2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e Calculate the forward extremeties once 2017-01-20 14:28:53 +00:00