Commit Graph

358 Commits

Author SHA1 Message Date
Matthew Hodgson
b041115415
Speed up lazy loading (#3827)
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
2018-09-12 00:50:39 +01:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Erik Johnston
ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Erik Johnston
ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston
5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00
Richard van der Hoff
ca9bc1f4fe Fix occasional glitches in the synapse_event_persisted_position metric
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
Erik Johnston
62ace05c45 Move necessary storage functions to worker classes 2018-08-06 15:23:38 +01:00
Erik Johnston
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Erik Johnston
a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Richard van der Hoff
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
Richard van der Hoff
0bc9b9e397 reinstate explicit include of EventsWorkerStore 2018-07-31 13:11:04 +01:00
Erik Johnston
143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Richard van der Hoff
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
Stop populating events.content
2018-07-26 22:54:08 +01:00
Richard van der Hoff
5c1d301fd9 Stop populating events.content
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff
bd4b25f4d0 Remove some redundant joins on event_edges.room_id
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.

event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
Richard van der Hoff
21e878ebb6 Make EventStore inherit from EventFederationStore
(since it uses methods therein)

Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
Erik Johnston
78a691d005 Split out DB writes in federation handler
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
Erik Johnston
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
Richard van der Hoff
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
Erik Johnston
7780a7b47c Actually fix it by adding continue 2018-07-25 11:13:20 +01:00
Richard van der Hoff
1be94440d3 Fix occasional 'tuple index out of range' error
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
Richard van der Hoff
07defd5fe6 Fix another logcontext leak in _persist_events
We need to run the errback in the sentinel context to avoid losing our own
context.

Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
Erik Johnston
a297ff2b16 Fix typo in conditional 2018-07-25 09:48:01 +01:00
Erik Johnston
ec56121b0d Correctly handle outliers during persist events
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.

However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.

Fixes #3601
2018-07-25 09:35:02 +01:00
Erik Johnston
709c309b0e Expand on docstring comment about return value 2018-07-24 15:12:50 +01:00
Erik Johnston
8f65ab98d2 Remove unnecessary iteritems 2018-07-24 15:08:01 +01:00
Erik Johnston
ed0dd68731 Fixup comment (and indent) 2018-07-24 14:31:38 +01:00
Erik Johnston
811ac73a42 Don't fetch state from the database unless needed 2018-07-24 14:18:23 +01:00
Erik Johnston
a79410e7b8 Have _get_new_state_after_events return delta
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Erik Johnston
223341205e Don't require to_delete to have event_ids 2018-07-24 14:02:40 +01:00
Erik Johnston
97acd385a3 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta 2018-07-24 11:32:13 +01:00
Erik Johnston
0fa73e4a63 Remove unnecessary if 2018-07-24 11:19:23 +01:00
Erik Johnston
ff5426f6b8 Speed up _calculate_state_delta 2018-07-24 10:55:11 +01:00
Richard van der Hoff
a678145010
Merge branch 'develop' into rav/logcontext_fixes 2018-07-24 10:43:30 +01:00
Richard van der Hoff
8dff6e0322 Logcontext fixes
Fix some random logcontext leaks.
2018-07-24 00:37:17 +01:00
Richard van der Hoff
69fb5dbdab fix idiocy 2018-07-24 00:04:44 +01:00
Richard van der Hoff
1938cffaea Add some measure blocks to persist_events
... to help us figure out where 40% of CPU is going
2018-07-23 23:48:19 +01:00
Erik Johnston
8b9f164fff Comments 2018-07-23 17:43:01 +01:00
Erik Johnston
50c60e5fad Only get cached state from context in persist_event
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
2018-07-23 17:21:40 +01:00
Erik Johnston
e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Richard van der Hoff
667fba68f3 Run things as background processes
This fixes #3518, and ensures that we get useful logs and metrics for lots of
things that happen in the background.

(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Erik Johnston
3905c693c5 Invalidate cache on correct thread 2018-07-02 11:36:44 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Mark Haines
9850f66abe Deleting from event_push_actions needs to use an index 2018-06-22 15:54:48 +01:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00