Commit Graph

1952 Commits

Author SHA1 Message Date
Erik Johnston
a2c89a225c Prefill state caches 2017-05-02 10:40:31 +01:00
Erik Johnston
3033261891 Merge pull request #2080 from matrix-org/erikj/filter_speed
Speed up filtering of a single event in push
2017-04-28 14:17:13 +01:00
Erik Johnston
ab37bef83b Remove unused import 2017-04-28 09:57:23 +01:00
Erik Johnston
ad8b316939 We don't care about forgotten rooms 2017-04-28 09:52:36 +01:00
Erik Johnston
421fdf7460 Speed up filtering of a single event in push 2017-04-28 09:52:36 +01:00
Richard van der Hoff
c84770b877 Fix bgupdate error if index already exists (#2167)
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135.

Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
Erik Johnston
f053a1409e Make state caches cache in ascii 2017-04-25 17:22:55 +01:00
Erik Johnston
f144365281 Comment 2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86 Reduce size of joined_user cache
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76 Reduce _get_state_group_for_event cache size 2017-04-25 11:43:03 +01:00
Erik Johnston
119cb9bbcf Reduce cache size by not storing deferreds
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Erik Johnston
e4f3431116 Remove unused cache 2017-04-24 13:27:38 +01:00
Luke Barnard
78f0ddbfad Merge pull request #2120 from matrix-org/luke/read-markers
Implement Read Marker API
2017-04-13 14:21:31 +01:00
Luke Barnard
6a70647d45 Correct logic in is_event_after 2017-04-13 13:46:17 +01:00
Luke Barnard
b9557064bf Simplify is_event_after logic 2017-04-12 14:36:20 +01:00
Luke Barnard
7f94709066 travis flake8.. 2017-04-11 18:35:45 +01:00
Luke Barnard
73880268ef Refactor event ordering check to events store 2017-04-11 17:34:09 +01:00
Erik Johnston
85657eedf8 Bail on where clause instead 2017-04-11 16:24:31 +01:00
Erik Johnston
b48045a8f5 Don't bother with outer check for now 2017-04-11 16:23:24 +01:00
Erik Johnston
34840cdcef Fix getting latest device IP for user with no devices 2017-04-11 09:56:54 +01:00
Erik Johnston
2a3e822f44 Comment 2017-04-07 13:47:04 +01:00
Erik Johnston
d72667fcce Speed up get_current_state_ids
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).

get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
2017-04-07 10:10:49 +01:00
Erik Johnston
9f26d3b75b Deduplicate new deviceinbox rows for replication 2017-04-04 16:21:21 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Erik Johnston
3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston
e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Erik Johnston
d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
Erik Johnston
7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Erik Johnston
6ebe2d23b1 Raise a more helpful exception 2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d Don't user upsert to persist new one time keys
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
2017-03-23 13:17:00 +00:00
Richard van der Hoff
64778693be fix up some key verif docstrings 2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
Richard van der Hoff
eddce9d74a Merge pull request #2027 from matrix-org/rav/logcontext_leaks
A few fixes to logcontext things
2017-03-20 11:53:36 +00:00
Richard van der Hoff
994d7ae7c5 Remove broken use of clock.call_later
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.

We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae Avoid resetting state on rejected events
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.

This should fix https://github.com/matrix-org/synapse/issues/1935.
2017-03-17 15:06:08 +00:00