Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-12-27 03:59:25 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,868 Commits 4 Branches 615 Tags 63 MiB
619e8ecd0c
Commit Graph

2016 Commits

Author SHA1 Message Date
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict 
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
 2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall 
This prevents unnecessary construction of lists
 2017-03-23 17:53:49 +00:00
Erik Johnston
6ebe2d23b1 Raise a more helpful exception 2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d Don't user upsert to persist new one time keys 
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
 2017-03-23 13:17:00 +00:00
Richard van der Hoff
64778693be fix up some key verif docstrings 2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed 
Don't send the full event json over replication
 2017-03-21 13:11:15 +00:00
Richard van der Hoff
eddce9d74a Merge pull request #2027 from matrix-org/rav/logcontext_leaks 
A few fixes to logcontext things
 2017-03-20 11:53:36 +00:00
Richard van der Hoff
994d7ae7c5 Remove broken use of clock.call_later 
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.

We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
 2017-03-18 00:01:37 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae Avoid resetting state on rejected events 
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.

This should fix https://github.com/matrix-org/synapse/issues/1935.
 2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5 Refactoring and cleanups 
A few non-functional changes:

* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
  readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
  mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
  mind-bending conditional.
 2017-03-17 15:06:07 +00:00
Erik Johnston
da146657c9 Comments 2017-03-16 13:04:07 +00:00
Erik Johnston
6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Richard van der Hoff
29ed09e80a Fix assertion to stop transaction queue getting wedged 
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
 2017-03-15 12:16:55 +00:00
Erik Johnston
bb256ac96f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed 2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e Fix current_state_events table to not lie 
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
 2017-03-14 10:57:43 +00:00
Luke Barnard
bbeeb97f75 Implement _simple_delete_many_txn, use it to delete devices 
(But this doesn't implement the same for deleting access tokens or e2e keys.

Also respond to code review.
 2017-03-13 17:53:23 +00:00
Erik Johnston
79926e016e Assume rooms likely haven't changed 2017-03-13 09:50:10 +00:00
Erik Johnston
8ffbe43ba1 Get current state by using current_state_events table 2017-03-10 17:39:35 +00:00
Erik Johnston
64d62e41b8 Noop repated delete device inbox calls from /sync 2017-03-10 10:36:43 +00:00
Richard van der Hoff
6ad71cc29d Remove spurious SQL logging (#1972) 
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
 2017-03-08 18:00:44 +00:00
Erik Johnston
ac5491f563 Select distinct devices from DB 
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
 2017-03-06 11:10:14 +00:00
Erik Johnston
da52d3af31 Fix up 2017-03-03 15:29:13 +00:00
Erik Johnston
d766343668 Add index to device_lists_stream 2017-03-01 15:56:30 +00:00
Erik Johnston
3365117151 Clobber old device list stream entries 2017-03-01 10:21:30 +00:00
Erik Johnston
d638a7484b Merge pull request #1959 from matrix-org/erikj/intern_once 
Intern table column names once
 2017-02-28 15:15:16 +00:00
Erik Johnston
b84907bdbb Intern table column names once 2017-02-28 14:38:16 +00:00
Erik Johnston
e4919b9329 Add stream_id index to device_lists_outbound_pokes 
As this is used for replication streaming
 2017-02-28 11:19:06 +00:00
Erik Johnston
8a12b6f1eb Fix up txn name 2017-02-28 10:15:50 +00:00
Erik Johnston
a41dce8f8a Remove needless check 2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776 It should be all 2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709 Don't fetch current state in common case 
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
 2017-02-27 18:33:41 +00:00
Erik Johnston
fc5d876dba Merge pull request #1954 from matrix-org/erikj/cache_device2 
Cache get_user_devices_from_cache
 2017-02-27 16:36:40 +00:00
Erik Johnston
f58dbb02a6 Cache get_user_devices_from_cache 2017-02-27 16:22:12 +00:00
Erik Johnston
1a4f8022e6 Strip newlines from SQL queries 2017-02-23 11:15:31 +00:00
Erik Johnston
b2d20e94fa Remove lock from rotate notifs 2017-02-22 14:24:02 +00:00
Erik Johnston
b7442c3e2b Store looping call 2017-02-21 13:59:25 +00:00
Erik Johnston
699be7d1be Fix up notif rotation 2017-02-18 14:42:39 +00:00
David Baker
4aa29508af Use TEXT rather than VARCHAR 
While we're changing anyway
 2017-02-17 10:51:49 +00:00
David Baker
b4017539d4 Make the pushers lang field column longer 
To accommodate things like zh-Hans-CN

Fixes https://github.com/vector-im/riot-ios/issues/1031
 2017-02-17 10:42:57 +00:00
Erik Johnston
138e030cfe Comment 2017-02-16 15:03:36 +00:00
Erik Johnston
502ae6c663 Comment 2017-02-16 14:47:11 +00:00
Erik Johnston
e6acf0c399 Store the default push actions in a more efficient manner 2017-02-16 14:40:24 +00:00
Erik Johnston
04eca2589d Merge pull request #1916 from matrix-org/erikj/push_actions_delete 
Aggregate event push actions
 2017-02-16 15:28:58 +01:00
Richard van der Hoff
7dcbcca68c Merge pull request #1921 from matrix-org/rav/fix_key_changes 
Fix bugs in the /keys/changes api
 2017-02-15 11:25:16 +00:00
Richard van der Hoff
fc2f29c1d0 Fix bugs in the /keys/changes api 
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
  passing a `RoomStreamToken`, which meant that it returned the *current*
  extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
  that a `TypeError` was thrown and we ended up acting as though there was *no*
  prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
  dictionary - just looking up the state keys in it again meant that we acted
  as though there was no prev state. We now check if each member's state has
  changed since *any* of the extremities.

Also add/fix some comments.
 2017-02-14 13:59:50 +00:00
Erik Johnston
ce3c8df6df Less aggressive timers 2017-02-14 13:41:24 +00:00
Erik Johnston
095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston
795f8e3fe7 Merge pull request #1873 from matrix-org/erikj/delete_push_actions 
Be more agressive about purging old room event_push_actions
 2017-02-14 14:29:04 +01:00
Erik Johnston
9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Erik Johnston
52cd019a54 Make None check explicit 2017-02-08 16:04:29 +00:00
Erik Johnston
f20cd34858 Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem 
Ignore new rejected events when working out forward extremities.
 2017-02-08 16:59:06 +01:00
Erik Johnston
7723b4caa4 Ignore new rejected events when working out forward extremeties. 2017-02-08 14:48:06 +00:00
Erik Johnston
af6da6db2d Merge pull request #1784 from morteza-araby/user-admin 
Administration functionalities
 2017-02-06 16:21:10 +01:00
Erik Johnston
38258a0976 Bump cache sizes for common membership queries 2017-02-02 18:45:55 +00:00
Erik Johnston
832e9c52ca Comment 2017-02-02 13:09:56 +00:00
Erik Johnston
54a79c1d37 Make presence.get_new_events a bit faster 
We do this by caching the set of users a user shares rooms with.
 2017-02-02 13:07:18 +00:00
Morteza Araby
2849d3f29d admin,storage: added more administrator functionalities 
administrators can now:
 - Set displayname of users
 - Update user avatars
 - Search for users by user_id
 - Browse all users in a paginated API
 - Reset user passwords
 - Deactivate users

Helpers for doing paginated queries has also been added to storage

Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
 2017-02-02 14:02:26 +01:00
Erik Johnston
fac3c03087 Be more agressive about purging old room event_push_actions 2017-02-01 18:27:24 +00:00
Erik Johnston
fbfe44bb4d Doc args 2017-02-01 17:52:57 +00:00
Erik Johnston
d61a04583e Comment 2017-02-01 17:35:23 +00:00
Erik Johnston
7e919bdbd0 Include newly joined users in /keys/changes API 2017-02-01 17:33:16 +00:00
Erik Johnston
ebfaff84c9 Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events 
Add a small cache get_all_new_events
 2017-02-01 13:22:02 +00:00
Erik Johnston
62f6b86ba7 Merge pull request #1868 from matrix-org/erikj/replication_cache 
Only invalidate membership caches based on the cache stream
 2017-02-01 13:12:30 +00:00
Erik Johnston
f6124311fd Add m.room.member type to query 2017-02-01 11:59:17 +00:00
Erik Johnston
88a4d54883 Merge pull request #1867 from matrix-org/erikj/member_index 
Add an index to make membership queries faster
 2017-02-01 11:44:27 +00:00
Erik Johnston
368c88c487 Add a small cache get_all_new_events 2017-02-01 10:50:44 +00:00
Erik Johnston
5deaf9e30b Up get_latest_event_ids_in_room cache 2017-02-01 10:39:41 +00:00
Erik Johnston
458b6f4733 Only invalidate membership caches based on the cache stream 
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
 2017-01-31 16:09:03 +00:00
Erik Johnston
21b7375778 Add an index to make membership queries faster 2017-01-31 15:15:57 +00:00
Erik Johnston
ac001dabdc Merge pull request #1864 from matrix-org/erikj/device_list_fixes 
Fix clearing out old device list outbound pokes
 2017-01-31 13:35:35 +00:00
Erik Johnston
bfb3d255b1 Merge pull request #1862 from matrix-org/erikj/presence_update 
Use DB cache of joined users for presence
 2017-01-31 13:23:24 +00:00
Erik Johnston
ab55794b6f Fix deletion of old sent devices correctly 2017-01-31 13:22:41 +00:00
Erik Johnston
d3169e8d28 Only fetch with row ts and count > 1 2017-01-31 11:20:03 +00:00
Erik Johnston
05b9f48ee5 Fix clearing out old device list outbound pokes 2017-01-31 10:08:55 +00:00
Erik Johnston
4b3403ca9b Stream cache invalidations for room membership storage functions 2017-01-30 17:28:22 +00:00
Erik Johnston
1c13c9f6b6 Don't have such a large cache 2017-01-30 17:12:14 +00:00
Erik Johnston
c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston
c2c9a78db9 Noop device key changes if they're the same 2017-01-30 16:55:04 +00:00
Erik Johnston
e75a779d9e Fix query 2017-01-30 16:38:20 +00:00
Erik Johnston
828db669ec Use get_users_in_room and declare it iterable 2017-01-30 16:37:22 +00:00
Erik Johnston
3670025e64 Rename func 2017-01-30 14:11:31 +00:00
Erik Johnston
d360c97ae1 Clear out old destination pokes. 2017-01-30 10:14:37 +00:00
Erik Johnston
76100203ab Always use the latest stream_id, sent or unsent 2017-01-30 10:14:25 +00:00
Erik Johnston
d1e1fd6210 Add ts column to device_lists_outbound_pokes 2017-01-27 15:23:48 +00:00
Erik Johnston
252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston
84a35f32c7 Comment 2017-01-27 10:35:12 +00:00
Erik Johnston
c517a19c2d Comment 2017-01-27 10:33:26 +00:00
Erik Johnston
738a2867c8 SQL param ordering 2017-01-27 10:31:29 +00:00
Erik Johnston
755adff0e4 User if rather than for 2017-01-27 10:31:06 +00:00
Erik Johnston
888c59c955 Better name 2017-01-27 10:29:47 +00:00
Erik Johnston
f25a4a4692 Remove unused param 2017-01-27 10:27:39 +00:00
Erik Johnston
31aca5589c Fix on sqlite: use left rather than outer join 2017-01-26 16:55:50 +00:00
Erik Johnston
76d40f4904 Handle users leaving rooms 2017-01-26 16:39:33 +00:00
Erik Johnston
fbfad76c03 Add comments 2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4 Fix up sending of m.device_list_update edus 2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Erik Johnston
fdf2a31a51 Typo 2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092 Refactor to calculate state delta outside transaction 2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485 Comments 2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5 Spelling 2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67 Update all call sites after rename 2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9 Comments 2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e Calculate the forward extremeties once 2017-01-20 14:28:53 +00:00
Erik Johnston
09eb08f910 Derive current_state_events from state groups 2017-01-20 11:52:51 +00:00
Erik Johnston
4ec1cf49e2 Lower loading events log to DEBUG 2017-01-17 17:28:32 +00:00
Erik Johnston
d906206049 Increase state_group_cache_size 2017-01-17 11:31:08 +00:00
Erik Johnston
897f8752da Up cache max entries for state 2017-01-16 15:08:17 +00:00
Erik Johnston
01521299c7 Increase cache size limit 2017-01-16 11:56:51 +00:00
Erik Johnston
2fae34bd2c Optionally measure size of cache by sum of length of values 2017-01-13 17:46:17 +00:00
Erik Johnston
f0325a9ccc Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index 
Change device_inbox stream index to include user
 2017-01-13 15:14:51 +00:00
Erik Johnston
c050f493dd Add comment 2017-01-13 15:14:41 +00:00
Erik Johnston
ab655dca33 Explicitly close the cursor 2017-01-10 15:15:25 +00:00
Erik Johnston
5a32e9273e Don't disable autocommit 2017-01-10 15:11:27 +00:00
Erik Johnston
caddadfc5a Change device_inbox stream index to include user 
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
 2017-01-10 15:04:57 +00:00
Erik Johnston
dd52d4de4c Limit number of entries to prefill from cache 
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.

In future, we should add a better index to make it even faster.
 2017-01-10 14:34:50 +00:00
Matthew Hodgson
bd85434cb3 Merge branch 'release-v0.18.6' into develop 2017-01-05 13:58:19 +00:00
Matthew Hodgson
8cfc0165e9 fix annoying typos 2017-01-05 13:39:43 +00:00
Matthew Hodgson
b31ed22738 Merge branch 'release-v0.18.6' into develop 2017-01-05 13:03:02 +00:00
Mark Haines
dd3df11c55 More logging for the linearizer and for get_events 2017-01-05 12:32:47 +00:00
David Baker
0c88ab1844 Add /account/3pid/delete endpoint 
Also fix a typo in a comment
 2016-12-20 18:27:30 +00:00
Richard van der Hoff
edc6a1e4f9 Add some logging for syncing to_device events 
Attempt to track down the loss of to_device events
(https://github.com/vector-im/riot-web/issues/2711 etc).
 2016-12-15 18:16:10 +00:00
Erik Johnston
c3208e45c9 Fixup membership query 2016-12-14 10:46:58 +00:00
Erik Johnston
8588d0eb3d Merge pull request #1697 from matrix-org/erikj/fix_bg_member 
Fix background update that prematurely stopped
 2016-12-12 17:20:17 +00:00
Erik Johnston
1574b839e0 Merge pull request #1676 from matrix-org/erikj/room_list 
Add new API appservice specific public room list
 2016-12-12 17:00:10 +00:00
Erik Johnston
7ec2bf9b77 Fix background update that prematurely stopped 2016-12-12 16:54:58 +00:00
Erik Johnston
09cbcb78d3 Add cache to get_public_room_ids_at_stream_id 2016-12-12 14:41:51 +00:00
Erik Johnston
631376e2ac Notify replication. Use correct network_id 2016-12-12 14:28:15 +00:00
Erik Johnston
d45c984653 Docstring 2016-12-12 11:00:27 +00:00
Erik Johnston
1697f6a323 Merge pull request #1680 from matrix-org/erikj/joined_rooms 
Add new room membership APIs
 2016-12-09 11:35:48 +00:00
Erik Johnston
52d12ca782 Add /room/<room_id>/joined_members API 
This returns the currently joined members in the room with their display
names and avatar urls. This is more efficient than /members for large
rooms where you don't need the full events.
 2016-12-08 13:32:07 +00:00
Erik Johnston
c45d8e9ba2 Add profile data to the room_membership table for joins 2016-12-08 13:08:41 +00:00
Richard van der Hoff
bd07a35c29 Fix result of insert_receipt 
This should fix the absence of notifications when new receipts arrive.
 2016-12-08 12:11:34 +00:00
Erik Johnston
2687af82d4 Comments 2016-12-07 09:58:33 +00:00
Erik Johnston
f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Richard van der Hoff
aa09d6b8f0 Rip out more refresh_token code 
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.

Still TODO: removing the table from the schema.
 2016-11-30 17:40:18 +00:00
Erik Johnston
46ca345b06 Don't send old events as federation 2016-11-24 12:29:02 +00:00
Erik Johnston
1985860c6e Comment 2016-11-23 15:59:59 +00:00
Erik Johnston
2ac516850b More efficient notif count queries 2016-11-23 15:57:04 +00:00
Erik Johnston
302fbd218d Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn 
Split out federation transaction sending to a worker
 2016-11-23 15:39:12 +00:00
Erik Johnston
b69f76c106 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn 2016-11-23 11:31:53 +00:00
Erik Johnston
0163466d72 Ignore AS users when fetching push rules 
By ignoring AS users early on when fetching push rules for a room we can
avoid needlessly hitting the DB and filling up the caches.
 2016-11-23 11:01:01 +00:00
Erik Johnston
54fed21c04 Fix tests and flake8 2016-11-22 18:18:31 +00:00
Erik Johnston
90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00