Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
10,004 Commits 5 Branches 723 Tags 124 MiB
c46e75d3d8
Commit Graph

7269 Commits

Author SHA1 Message Date
Erik Johnston
95298783bb Add is_publicised to group summary 2017-09-26 11:04:37 +01:00
Erik Johnston
1a398b19fd Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.23.0 2017-09-26 10:08:59 +01:00
Erik Johnston
f4c8cd5e85 Bump changelog and version 2017-09-26 10:02:48 +01:00
Erik Johnston
e3edca3b5d Refactor to speed up incremental syncs 2017-09-25 17:35:39 +01:00
Erik Johnston
44edac0497 Merge branch 'release-v0.23.0' of github.com:matrix-org/synapse into develop 2017-09-25 14:52:46 +01:00
Richard van der Hoff
94133d7ce8 Merge branch 'develop' into develop 2017-09-25 11:50:11 +01:00
Erik Johnston
ba8fdc925c Bump version and changes 2017-09-25 11:01:31 +01:00
Richard van der Hoff
79b3cf3e02 Fix logcontxt leak in keyclient (#2465) 
preserve_context_over_function doesn't do what you want it to do.
 2017-09-25 09:51:39 +01:00
Richard van der Hoff
f65e31d22f Do an AAAA lookup on SRV record targets (#2462) 
Support SRV records which point at AAAA records, as well as A records.

Fixes https://github.com/matrix-org/synapse/issues/2405
 2017-09-22 20:26:47 +01:00
Matthew Hodgson
f496399ac4 fix thinko'd docstring 2017-09-22 15:34:14 +01:00
Erik Johnston
3166ed55b2 Fix device list when rejoining room (#2461) 2017-09-22 14:44:17 +01:00
Erik Johnston
e1dec2f1a7 Remove user from group summary when the leave the group 2017-09-21 16:09:57 +01:00
Erik Johnston
bb746a9de1 Revert: Keep room_id's in group summary 2017-09-21 15:57:22 +01:00
Erik Johnston
ae8d4bb0f0 Keep room_id's in group summary 2017-09-21 15:55:18 +01:00
Erik Johnston
197d82dc07 Correctly return next token 2017-09-20 11:12:11 +01:00
Erik Johnston
069ae2df12 Fix initial sync 2017-09-20 10:52:12 +01:00
Richard van der Hoff
6de74ea6d7 Fix logcontexts in _check_sigs_and_hashes 2017-09-20 01:32:42 +01:00
Richard van der Hoff
c5c24c239b Fix logcontext handling in verify_json_objects_for_server 
preserve_context_over_fn is essentially broken, because (a) it pointlessly
drops the current logcontext before calling its wrapped function, which means
we don't get any useful logcontexts for _handle_key_deferred; (b) it wraps the
resulting deferred in a _PreservingContextDeferred, which is very dangerous
because you then can't yield on it without leaking context back into the
reactor.

Instead, let's specify that the resultant deferreds call their callbacks with
no logcontext.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
c5b0e9f485 Turn _start_key_lookups into an inlineCallbacks function 
... which means that logcontexts can be correctly preserved for the stuff it
does.

get_server_verify_keys is now called with the logcontext, so needs to
preserve_fn when it fires off its nested inlineCallbacks function.

Also renames get_server_verify_keys to reflect the fact it's meant to be
private.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
abdefb8a01 Fix potential race in _start_key_lookups 
If the verify_request.deferred has already completed, then `remove_deferreds`
will be called immediately. It therefore might resolve the server_to_deferred
deferred while there are still other requests for that server in flight.

To avoid that, we should build the complete list of requests, and *then* add the
callbacks.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
afbd773dc6 Add some comments to _start_key_lookups 2017-09-20 01:32:42 +01:00
Richard van der Hoff
2a4b9ea233 Consistency for how verify_request.deferred is called 
Define that it is run with no log context, and make sure that happens.

If we aren't careful to reset the logcontext, we can't bung the deferreds into
defer.gatherResults etc. We don't actually do that directly, but we *do*
resolve other deferreds from affected callbacks (notably the server_to_deferred
map in _start_key_lookups), and those *do* get passed into
defer.gatherResults. It turns out that this way ends up being least confusing.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
3b98439eca Factor out _start_key_lookups 
... to make it easier to see what's going on.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
fde63b880d Replace server_and_json with verify_requests 
This is a precursor to factoring some of this code out.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
2d511defd9 pull out handle_key_deferred to top level 
There's no need for this to be a nested definition; pulling it out not only
makes it more efficient, but makes it easier to check that it's not accessing
any local variables it shouldn't be.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
dd1ea9763a Fix incorrect key_ids in error message 2017-09-20 01:32:42 +01:00
Richard van der Hoff
e76d1135dd Invalidate signing key cache when we gat an update 
This might make the cache slightly more efficient.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
fcf2c0fd1a Remove redundant preserve_fn 
preserve_fn is a no-op unless the wrapped function returns a
Deferred. verify_json_objects_for_server returns a list, so this is doing
nothing.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
9864efa532 Fix concurrent server_key requests (#2458) 
Fix a bug where we could end up firing off multiple requests for server_keys
for the same server at the same time.
 2017-09-19 23:25:44 +01:00
Richard van der Hoff
aa620d09a0 Add a config option to block all room invites (#2457) 
- allows sysadmins the ability to lock down their servers so that people can't
send their users room invites.
 2017-09-19 16:08:14 +01:00
Richard van der Hoff
2eabdf3f98 add some comments to on_exchange_third_party_invite_request 2017-09-19 12:20:36 +01:00
Richard van der Hoff
5ed109d59f PoC for filtering spammy events (#2456) 
Demonstration of how you might add some hooks to filter out spammy events.
 2017-09-19 12:20:11 +01:00
Erik Johnston
93e504d04e Ensure that creator of group sees group down /sync 2017-09-19 11:08:16 +01:00
Erik Johnston
b5feaa5a49 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_merged 2017-09-19 11:07:45 +01:00
Richard van der Hoff
3f405b34e9 Fix overzealous kicking of guest users (#2453) 
We should only kick guest users if the guest access event is authorised.
 2017-09-19 08:52:52 +01:00
Richard van der Hoff
290777b3d9 Clean up and document handling of logcontexts in Keyring (#2452) 
I'm still unclear on what the intended behaviour for
`verify_json_objects_for_server` is, but at least I now understand the
behaviour of most of the things it calls...
 2017-09-18 18:31:01 +01:00
Erik Johnston
77c81ca6ea Merge pull request #2451 from matrix-org/erikj/add_state_to_timeline 
Don't filter out current state events from timeline
 2017-09-18 17:22:33 +01:00
Erik Johnston
2d1b7955ae Don't filter out current state events from timeline 2017-09-18 17:13:03 +01:00
David Baker
862c8da560 Merge pull request #2450 from matrix-org/dbkr/push_event_id_only 
Add support for event_id_only push format
 2017-09-18 16:41:29 +01:00
David Baker
436ee0a2ea Also include the room_id 
as really it's part of the event ID
 2017-09-18 15:58:38 +01:00
David Baker
b393f5db51 Use .get - it's much shorter 2017-09-18 15:50:26 +01:00
David Baker
a2562f9d74 Add support for event_id_only push format 
Param in the data dict of a pusher that tells an HTTP pusher to
send just the event_id of the event it's notifying about and the
notification counts. For clients that want to go & fetch the body
of the event themselves anyway.
 2017-09-18 15:39:39 +01:00
Erik Johnston
d6dadd95ac Correctly handle leaving room in /key/changes 2017-09-18 15:38:22 +01:00
Erik Johnston
4a94eb3ea4 Fix typo 2017-09-15 09:56:54 +01:00
Erik Johnston
3a0cee28d6 Actually hook leave notifs up 2017-09-14 11:49:37 +01:00
Erik Johnston
4f845a0713 Handle joining/leaving rooms in /keys/changes 2017-09-13 16:28:08 +01:00
Erik Johnston
473700f016 Get left rooms 2017-09-13 15:13:41 +01:00
Erik Johnston
9ce866ed4f In sync handle device lists for newly joined/left rooms 2017-09-12 16:44:26 +01:00
Erik Johnston
69ef4987a6 Add left section to /keys/changes 2017-09-08 14:44:36 +01:00
Erik Johnston
53cc8ad35a Send down device list change notif when member leaves/rejoins room 2017-09-07 15:08:39 +01:00
Erik Johnston
4a9b1cf253 Add user profiles to summary from group server 2017-08-25 16:23:58 +01:00
Erik Johnston
258409ef61 Fix typos and reinherit 2017-08-25 14:45:20 +01:00
Erik Johnston
bf81f3cf2c Split out profile handler to fix tests 2017-08-25 14:34:56 +01:00
Erik Johnston
27ebc5c8f2 Add remote profile cache 2017-08-25 11:25:47 +01:00
Erik Johnston
97c544f91f Add _simple_update 2017-08-25 11:11:37 +01:00
Richard van der Hoff
a7effa8400 Merge pull request #2288 from kyrias/bcrypt 
python_dependencies: Use bcrypt module instead of py-bcrypt
 2017-08-23 14:14:56 +01:00
Erik Johnston
2800983f3e Merge pull request #2410 from matrix-org/erikj/groups_publicise 
Add ability to publicise group membership
 2017-08-21 16:51:56 +01:00
Erik Johnston
8b50fe5330 Use BOOLEAN rather than TEXT type 2017-08-21 16:37:29 +01:00
Erik Johnston
175a01f56c Groups: Fix mising json.load in initial sync 2017-08-21 14:45:56 +01:00
Richard van der Hoff
692250c6be Fix user_dir startup 
Add missing parameter to _base.start_worker_reactor
 2017-08-16 15:11:29 +01:00
Richard van der Hoff
d2352347cf Fix process startup 
escape the % that got added in 92168cb so that the process starts up ok.
 2017-08-16 14:57:35 +01:00
Matthew Hodgson
92168cbbc5 explain why CPU affinity is a good idea 2017-08-15 18:27:42 +01:00
Richard van der Hoff
10d8b701a1 Allow configuration of CPU affinity 
Make it possible to set the CPU affinity in the config file, so that we don't
need to remember to do it manually every time.
 2017-08-15 17:08:28 +01:00
Richard van der Hoff
543c794a76 Factor out common application start 
We have 10 copies of this code, and I don't really want to update each one
separately.
 2017-08-15 17:04:40 +01:00
Erik Johnston
ba3ff7918b Fixup 2017-08-11 13:42:42 +01:00
Erik Johnston
ef8e578677 Add bulk group publicised lookup API 2017-08-09 13:36:22 +01:00
Erik Johnston
b880ff190a Allow update group publicity 2017-08-08 14:19:41 +01:00
Erik Johnston
05e21285aa Store whether the user wants to publicise their membership of a group 2017-08-08 13:01:46 +01:00
hera
eae04f1952 fix english 2017-08-04 23:56:42 +01:00
hera
5699b05072 typo 2017-08-04 23:44:37 +01:00
Erik Johnston
a1e67bcb97 Remove stale TODO comments 2017-08-04 10:07:10 +01:00
Erik Johnston
09552f9d9c Reduce spammy log line in synchrotrons 2017-08-02 17:29:51 +01:00
Kenny Keslar
f18373dc5d Fix iteration of requests_missing_keys; list doesn't have .values() 
Signed-off-by: Kenny Keslar <r3dey3@r3dey3.com>
 2017-07-26 22:44:19 -05:00
Erik Johnston
966a70f1fa Update comment 2017-07-24 17:49:39 +01:00
Erik Johnston
629cdfb124 Use join rather than joined, etc. 2017-07-24 14:54:05 +01:00
Erik Johnston
ed666d3969 Fix all the typos 2017-07-24 14:05:09 +01:00
Erik Johnston
b76ef6ccb8 Include users membership in group in summary API 2017-07-24 13:55:39 +01:00
Erik Johnston
851aeae7c7 Check users/rooms are in group before adding to summary 2017-07-24 13:40:56 +01:00
Erik Johnston
d5e32c843f Correctly add joins to correct segment 2017-07-24 13:31:26 +01:00
Erik Johnston
96917d5552 Merge pull request #2378 from matrix-org/erikj/group_sync_support 
Add groups to sync stream
 2017-07-21 11:05:39 +01:00
Erik Johnston
0401604222 Merge pull request #2377 from matrix-org/erikj/group_profile_update 
Add update group profile API
 2017-07-20 17:53:39 +01:00
Erik Johnston
b238cf7f6b Remove spurious content param 2017-07-20 17:49:55 +01:00
Erik Johnston
960dae3340 Add notifier 2017-07-20 17:14:44 +01:00
Erik Johnston
2cc998fed8 Fix replication. And notify 2017-07-20 17:13:18 +01:00
Erik Johnston
139fe30f47 Remember to cast to bool 2017-07-20 16:47:35 +01:00
Erik Johnston
4d793626ff Fix bug in generating current token 2017-07-20 16:42:44 +01:00
Erik Johnston
c544188ee3 Add groups to sync stream 2017-07-20 16:36:42 +01:00
Erik Johnston
0ab153d201 Check values are strings 2017-07-20 16:24:18 +01:00
Erik Johnston
8209b5f033 Fix a storage desc 2017-07-20 16:22:22 +01:00
Erik Johnston
57826d645b Fix typo 2017-07-20 13:15:22 +01:00
Erik Johnston
6f443a74cf Add update group profile API 2017-07-20 09:46:33 +01:00
Erik Johnston
14a34f12d7 Comments 2017-07-18 17:28:42 +01:00
Erik Johnston
3431ec55dc Comments 2017-07-18 17:23:50 +01:00
Erik Johnston
6027b1992f Fix permissions 2017-07-18 16:51:25 +01:00
Erik Johnston
e884ff31d8 Add DELETE 2017-07-18 16:41:44 +01:00
Erik Johnston
05c13f6c22 Add 'args' param to post_json 2017-07-18 16:40:21 +01:00
Erik Johnston
94ecd871a0 Fix typos 2017-07-18 16:38:54 +01:00
Erik Johnston
12ed4ee48e Correctly parse query params 2017-07-18 15:33:09 +01:00
Erik Johnston
332839f6ea Update federation client pokes 2017-07-18 14:45:37 +01:00
Erik Johnston
e5ea6dd021 Add client apis 2017-07-18 14:37:06 +01:00
Erik Johnston
cccfcfa7b9 Comments 2017-07-18 10:35:18 +01:00
Erik Johnston
68f34e85ce Use transport client directly 2017-07-18 10:29:57 +01:00
Erik Johnston
3e703eb04e Comment 2017-07-18 10:17:25 +01:00
Erik Johnston
508460f240 Remove sync stuff 2017-07-18 09:55:46 +01:00
Erik Johnston
6e9f147faa Add GroupID type 2017-07-18 09:47:25 +01:00
Erik Johnston
4540730111 Remove unused tables 2017-07-18 09:38:15 +01:00
Erik Johnston
e96ee95a7e Remove sync stuff 2017-07-18 09:38:08 +01:00
Erik Johnston
2f9eafdd36 Add local group server support 2017-07-17 12:03:49 +01:00
Erik Johnston
bfde076022 Increase cache hit ratio for push 
We don't update the cache in all code paths, which causes subsequent
calls to miss the cache
 2017-07-14 16:11:26 +01:00
Erik Johnston
cb3aee8219 Ensure category and role ids are non-null 2017-07-14 14:06:55 +01:00
Erik Johnston
85fda57208 Add DEFAULT_ROLE_ID 2017-07-14 14:03:54 +01:00
Erik Johnston
4b203bdba5 Correctly increment orders 2017-07-14 14:02:31 +01:00
Erik Johnston
8d26385d76 Add more metrics to push rule evaluation 2017-07-13 14:37:30 +01:00
Erik Johnston
3b0470dba5 Remove unused functions 2017-07-13 13:54:01 +01:00
Erik Johnston
8575e3160f Comments 2017-07-13 13:52:41 +01:00
Erik Johnston
f60218ec41 Push: Don't acquire lock unless necessary 2017-07-13 11:23:53 +01:00
Erik Johnston
a78cda4baf Remove TODO 2017-07-13 11:17:07 +01:00
Erik Johnston
7a39da8cc6 Add summary APIs to federation 2017-07-13 11:13:19 +01:00
Erik Johnston
5bbb53580a raise NotImplementedError 2017-07-13 10:25:29 +01:00
Erik Johnston
26451a09eb Comments 2017-07-12 14:47:18 +01:00
Erik Johnston
8d55877c9e Simplify checking if admin 2017-07-12 11:43:39 +01:00
Erik Johnston
a62406aaa5 Add group summary APIs 2017-07-12 11:36:15 +01:00
Erik Johnston
e9aec001f4 Use less DB for device list handling in sync 2017-07-12 10:30:10 +01:00
Erik Johnston
28e8c46f29 Merge pull request #2352 from matrix-org/erikj/group_server_split 
Initial Group Server
 2017-07-12 10:26:29 +01:00
Erik Johnston
6d586dc05c Comment 2017-07-12 09:58:37 +01:00
Erik Johnston
410b4e14a1 Move comment 2017-07-11 15:44:18 +01:00
Erik Johnston
fe4e885f54 Add federation API for adding room to group 2017-07-11 14:35:07 +01:00
Erik Johnston
bbb739d24a Comment 2017-07-11 14:31:36 +01:00
Erik Johnston
26752df503 Typo 2017-07-11 14:29:03 +01:00
Erik Johnston
e52c391cd4 Rename column to attestation_json 2017-07-11 14:25:46 +01:00
Erik Johnston
0aac30d53b Comments 2017-07-11 14:23:50 +01:00
Erik Johnston
0184a97dbd Merge pull request #2354 from krombel/reduce_static_sync_reply 
encode sync-response statically
 2017-07-11 14:19:56 +01:00
Krombel
85b9f76f1d split out reducing stuff; just make encode_* static 2017-07-11 13:14:35 +02:00
Erik Johnston
6322fbbd41 Comment 2017-07-11 11:52:03 +01:00
Erik Johnston
8ba89f1050 Remove u/ requirement 2017-07-11 11:45:32 +01:00
Erik Johnston
429925a5e9 Lift out visibility parsing 2017-07-11 11:44:08 +01:00
Erik Johnston
83936293eb Comments 2017-07-11 11:42:25 +01:00
Erik Johnston
e2cb760dcc Merge pull request #2357 from matrix-org/erikj/push 
Don't compute push actions for backfilled events
 2017-07-11 10:53:22 +01:00
Erik Johnston
925b3638ff Reduce log levels in tcp replication 2017-07-11 10:04:21 +01:00
Erik Johnston
9a6fd3ef29 Don't compute push actions for backfilled events 2017-07-11 10:02:21 +01:00
Krombel
2f82de18ee fix test 2017-07-10 17:34:58 +02:00
Erik Johnston
b8ca494ee9 Initial group server implementation 2017-07-10 15:44:15 +01:00
Krombel
6e16aca8b0 encode sync-response statically; omit empty objects from sync-response 2017-07-10 16:42:17 +02:00
Erik Johnston
d4d12daed9 Include registration and as stores in frontend proxy 2017-07-07 18:36:45 +01:00
Erik Johnston
f467a8f66d Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-07-07 18:26:28 +01:00
Erik Johnston
1fc4a962e4 Add a frontend proxy 2017-07-07 18:19:46 +01:00
Erik Johnston
f502b0dea1 Perf: Don't filter events for push 
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
 2017-07-07 14:04:40 +01:00
Erik Johnston
76ed3476d3 Bump version and changelog 2017-07-06 18:11:22 +01:00
Erik Johnston
58dc1f2c78 Merge pull request #2342 from matrix-org/erikj/pusher_pool_instantiate 
Fix bug where pusherpool didn't start and broke some rooms
 2017-07-06 18:08:43 +01:00
Erik Johnston
5a7f561a9b Fix bug where pusherpool didn't start and broke some rooms 
Since we didn't instansiate the PusherPool at start time it could fail
at run time, which it did for some users.

This may or may not fix things for those users, but it should happen at
start time and stop the server from starting.
 2017-07-06 17:55:51 +01:00
Erik Johnston
ed9a7f5436 Merge pull request #2309 from matrix-org/erikj/user_ip_repl 
Fix up user_ip replication commands
 2017-07-06 14:33:14 +01:00
Erik Johnston
1f64207f26 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-07-06 13:57:45 +01:00
Erik Johnston
6264cf9666 Bump version and changelog 2017-07-06 10:35:56 +01:00
Erik Johnston
f386632800 Merge pull request #2334 from matrix-org/erikj/refactor_transport_server 
Separate federation servlet into different lists
 2017-07-05 17:09:07 +01:00
Erik Johnston
5e49a57ecc Separate federation servlet into different lists 2017-07-05 14:32:24 +01:00
Richard van der Hoff
3d31b39297 Merge pull request #2332 from matrix-org/rav/fix_pushes 
Fix caching error in the push evaluator
 2017-07-05 11:10:53 +01:00
Richard van der Hoff
73cfe48031 Fix caching error in the push evaluator 
Initialising `result` to `{}` in the parameters meant that every call to
_flatten_dict used the *same* target dictionary.

I'm hopeful this will fix https://github.com/matrix-org/synapse/issues/2270,
but I suspect it won't. (This code seems to have been here since forever,
unlike the bug, and I don't really think it explains the observed
behaviour). Still, it makes it hard to investigate the problem.
 2017-07-05 00:28:43 +01:00
Erik Johnston
05538587ef Bump version and changelog 2017-07-04 14:02:21 +01:00
Erik Johnston
f92d7416d7 Merge pull request #2330 from matrix-org/erikj/cache_size_factor 
Increase default cache size
 2017-07-04 10:51:21 +01:00
Erik Johnston
495f075b41 Increase default cache factor size. 2017-07-04 09:58:32 +01:00
Erik Johnston
b5e8d529e6 Define CACHE_SIZE_FACTOR once 2017-07-04 09:56:44 +01:00
Mark Haines
3e279411fe Improve the error handling for bad invites received over federation 2017-06-30 16:20:30 +01:00
Erik Johnston
6ff14ddd2e Make into list 2017-06-29 15:47:37 +01:00
Erik Johnston
5946aa0877 Prefill forward extrems and event to state groups 2017-06-29 15:38:48 +01:00
Erik Johnston
2c365f4723 Cache macaroon parse and validation 
Turns out this can be quite expensive for requests, and is easily
cachable. We don't cache the lookup to the DB so invalidation still
works.
 2017-06-29 14:50:18 +01:00
Erik Johnston
c72058bcc6 Use an ExpiringCache for storing registration sessions 
This is because pruning them was a significant performance drain on
matrix.org
 2017-06-29 14:08:37 +01:00
Erik Johnston
27f26e48b7 Serialize user ip command as json 2017-06-27 16:25:38 +01:00
Erik Johnston
8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston
731f3c37a0 Merge branch 'release-v0.22.0' of github.com:matrix-org/synapse into develop 2017-06-27 15:41:34 +01:00
Erik Johnston
4b444723f0 Merge pull request #2308 from matrix-org/erikj/user_ip_repl 
Make workers report to master for user ip updates
 2017-06-27 15:36:47 +01:00
Erik Johnston
816605a137 Merge pull request #2307 from matrix-org/erikj/user_ip_batch 
Batch upsert user ips
 2017-06-27 15:08:32 +01:00
Erik Johnston
78cefd78d6 Make workers report to master for user ip updates 2017-06-27 14:58:10 +01:00
Erik Johnston
a0a561ae85 Fix up client ips to read from pending data 2017-06-27 14:46:12 +01:00
Erik Johnston
ed3d0170d9 Batch upsert user ips 2017-06-27 13:37:04 +01:00
Erik Johnston
976128f368 Update version and changelog 2017-06-26 16:14:56 +01:00
Erik Johnston
d04d672a80 Merge pull request #2290 from matrix-org/erikj/ensure_round_trip 
Reject local events that don't round trip the DB
 2017-06-26 15:12:02 +01:00
Erik Johnston
1bce3e6b35 Remove unused variables 2017-06-26 14:03:27 +01:00
Erik Johnston
e3cbec10c1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ensure_round_trip 2017-06-26 14:02:44 +01:00
Erik Johnston
8abdd7b553 Fix up indices for users_who_share_rooms 2017-06-26 14:01:30 +01:00
Erik Johnston
ff13c5e7af Merge pull request #2301 from xwiki-labs/push-redact-content 
Add configuration parameter to allow redaction of content from push m…
 2017-06-24 13:13:51 +01:00
Caleb James DeLisle
27bd0b9a91 Change the config file generator to more descriptive explanation of push.redact_content 2017-06-24 10:32:12 +02:00
Caleb James DeLisle
bce144595c Fix TravisCI tests for PR #2301 - Fat finger mistake 2017-06-23 15:26:09 +02:00
Caleb James DeLisle
75eba3b07d Fix TravisCI tests for PR #2301 2017-06-23 15:15:18 +02:00
Caleb James DeLisle
1591eddaea Add configuration parameter to allow redaction of content from push messages for google/apple devices 2017-06-23 13:01:04 +02:00
Erik Johnston
7fe8ed1787 Store URL cache preview downloads seperately 
This makes it easier to clear old media out at a later date
 2017-06-23 11:14:11 +01:00
Erik Johnston
44c722931b Make some more params configurable 2017-06-22 14:59:52 +01:00
Erik Johnston
2d520a9826 Typo. ARGH. 2017-06-22 14:42:39 +01:00
Erik Johnston
24d894e2e2 Fix thinko in unhandled user spam 2017-06-22 14:39:05 +01:00
Matthew Hodgson
ccfcef6b59 Merge branch 'master' into develop 2017-06-22 13:03:44 +01:00
Erik Johnston
e0004aa28a Add desc 2017-06-22 10:03:48 +01:00
Erik Johnston
b668112320 Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar 
Don't work out users who share room with appservice users
 2017-06-21 14:50:24 +01:00
Erik Johnston
dae9a00a28 Initialise exclusive_user_regex 2017-06-21 14:19:33 +01:00
Erik Johnston
8177563ebe Fix for workers 2017-06-21 13:57:49 +01:00
Krombel
4202fba82a Merge branch 'develop' into avoid_duplicate_filters 2017-06-21 14:48:21 +02:00
Krombel
812c030e87 replaced json.dumps with encode_canonical_json 2017-06-21 14:48:12 +02:00
Erik Johnston
1217c7da91 Don't work out users who share room with appservice users 2017-06-21 12:00:41 +01:00
Erik Johnston
385dcb7c60 Handle thumbnail urls 2017-06-19 17:48:28 +01:00
Erik Johnston
b8b936a6ea Add API to quarantine media 2017-06-19 17:39:21 +01:00
Erik Johnston
e5ae386ea4 Handle all cases of sending membership events 2017-06-19 16:07:54 +01:00
Erik Johnston
36e51aad3c Remove unused import 2017-06-19 14:42:21 +01:00
Erik Johnston
b490299a3b Change to create new room and join other users 2017-06-19 14:10:13 +01:00
Erik Johnston
5db7070dd1 Forget room 2017-06-19 12:40:29 +01:00
Erik Johnston
d7fe6b356c Add shutdown room API 2017-06-19 12:37:27 +01:00
Erik Johnston
fcf01dd88e Reject local events that don't round trip the DB 2017-06-19 11:33:40 +01:00
Johannes Löthberg
4f66312df8 python_dependencies: Use bcrypt module instead of py-bcrypt 
py-bcrypt has been unmaintained for a long while, while bcrypt is
actively maintained. And since ff8b87118d
we're compatible with the bcrypt anyway.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
 2017-06-17 17:39:35 +02:00
Erik Johnston
dfeca6cf40 Merge pull request #2286 from matrix-org/erikj/split_out_user_dir 
Split out user directory to a separate process
 2017-06-16 13:01:19 +01:00
Erik Johnston
6aa5bc8635 Initial worker impl 2017-06-16 11:47:11 +01:00
Erik Johnston
d8f47d2efa Merge pull request #2280 from matrix-org/erikj/share_room_user_dir 
Include users who you share a room with in user directory
 2017-06-16 11:06:00 +01:00
Krombel
1ff419d343 allow Authorization header which handling got implemented in #1098 
Signed-off-by: Matthias Kesler <krombel@krombel.de>
 2017-06-16 11:21:14 +02:00
Erik Johnston
fdf1ca30f0 Bump version and changelog 2017-06-15 12:59:06 +01:00
Erik Johnston
052c5d19d5 Merge pull request #2281 from matrix-org/erikj/phone_home_stats 
Fix phone home stats
 2017-06-15 12:46:23 +01:00
Erik Johnston
5ddd199870 Typo 2017-06-15 10:49:10 +01:00
Erik Johnston
a9d6fa8b2b Include users who share room with requester in user directory 2017-06-15 10:17:21 +01:00
Erik Johnston
4564b05483 Implement updating users who share rooms on the fly 2017-06-15 10:17:17 +01:00
Erik Johnston
72613bc379 Implement initial population of users who share rooms table 2017-06-15 09:59:04 +01:00
Erik Johnston
ebcd55d641 Add DB schema for tracking users who share rooms 2017-06-15 09:45:48 +01:00
Erik Johnston
4b461a6931 Add some more stats 2017-06-15 09:39:39 +01:00
Erik Johnston
617304b2cf Fix phone home stats 2017-06-14 19:47:15 +01:00
Erik Johnston
d9fd937e39 Fix user directory insertion due to missing room_id 2017-06-13 11:50:24 +01:00
Erik Johnston
505e7e8b9d Fix up sql 2017-06-13 11:19:18 +01:00
Erik Johnston
6fd7e6db3d Fix user dir to not assume existence of user 2017-06-13 11:11:26 +01:00
Erik Johnston
fdca6e36ee Merge pull request #2274 from matrix-org/erikj/cache_is_host_joined 
Add cache for is_host_joined
 2017-06-13 10:58:53 +01:00
Erik Johnston
90ae0cffec Merge pull request #2275 from matrix-org/erikj/tweark_user_directory_search 
Tweak the ranking of PG user dir search
 2017-06-13 10:58:43 +01:00
Erik Johnston
de4cb50ca6 Merge pull request #2276 from matrix-org/erikj/fix_user_di 
Don't assume existence of events when updating user directory
 2017-06-13 10:55:41 +01:00
Erik Johnston
48d2949416 Throw exception when not retrying when downloading media 2017-06-13 10:23:14 +01:00
Erik Johnston
6ae8373d40 Don't assume existance of events when updating user directory 2017-06-13 10:19:26 +01:00
Erik Johnston
b58e24cc3c Tweak the ranking of PG user dir search 2017-06-13 10:16:31 +01:00
Erik Johnston
d53fe399eb Add cache for is_host_joined 2017-06-13 09:56:18 +01:00
Erik Johnston
a837765e8c Merge pull request #2266 from matrix-org/erikj/host_in_room 
Change is_host_joined to use current_state table
 2017-06-12 09:49:51 +01:00
Erik Johnston
8060974344 Fix replication 2017-06-09 16:40:52 +01:00
Erik Johnston
b0d975e216 Comments 2017-06-09 16:25:42 +01:00
Erik Johnston
e54d7d536e Cache state deltas 2017-06-09 16:24:00 +01:00
Erik Johnston
1e9b4d5a95 Merge pull request #2268 from matrix-org/erikj/entity_has_changed 
Fix has_any_entity_changed
 2017-06-09 15:30:55 +01:00
Erik Johnston
efc2b7db95 Rewrite conditional 2017-06-09 13:35:15 +01:00
Erik Johnston
bfd68019c2 Merge pull request #2267 from matrix-org/erikj/missing_notifier 
Fix removing of pushers when using workers
 2017-06-09 13:07:29 +01:00
Erik Johnston
1664948e41 Comment 2017-06-09 13:05:05 +01:00
Erik Johnston
935e588799 Tweak SQL 2017-06-09 13:01:23 +01:00
Erik Johnston
eed59dcc1e Fix has_any_entity_changed 
Occaisonally has_any_entity_changed would throw the error: "Set changed
size during iteration" when taking the max of the `sorteddict`. While
its uncertain how that happens, its quite inefficient to iterate over
the entire dict anyway so we change to using the more traditional
`bisect_*` functions.
 2017-06-09 11:44:01 +01:00
Erik Johnston
2cac7623a5 Add missing notifier 2017-06-09 11:24:41 +01:00
Erik Johnston
298d83b340 Fix replication 2017-06-09 11:01:28 +01:00
Erik Johnston
0185b75381 Change is_host_joined to use current_state table 
This bypasses a bug where using the state groups to figure out if a host
is in a room sometimes errors if the servers isn't in the room. (For
example when the server rejected an invite to a remote room)
 2017-06-09 10:52:26 +01:00
Erik Johnston
7132e5cdff Mark remote invite rejections as outliers 2017-06-09 10:08:18 +01:00
Erik Johnston
ea11ee09f3 Ensure we don't use unpersisted state group as prev group 2017-06-08 11:59:57 +01:00
Erik Johnston
c62c480dc6 Merge pull request #2259 from matrix-org/erikj/fix_state_woes 
Fix bug where state_group tables got corrupted
 2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0 Fix bug where state_group tables got corrupted 
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.

We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
 2017-06-07 17:39:36 +01:00
Erik Johnston
f45f07ab86 Merge pull request #2258 from matrix-org/erikj/user_dir 
Don't start user_directory handling on workers
 2017-06-07 14:04:50 +01:00
Erik Johnston
a053ff3979 Merge pull request #2248 from matrix-org/erikj/state_fixup 
Faster cache for get_joined_hosts
 2017-06-07 14:01:06 +01:00
Erik Johnston
ecdd2a3658 Don't start user_directory handling on workers 2017-06-07 12:02:53 +01:00
Erik Johnston
2f34ad31ac Add some logging to user directory 2017-06-07 11:50:44 +01:00
Erik Johnston
64ed74c01e When pruning, delete from device_lists_outbound_last_success 2017-06-07 11:20:47 +01:00
Erik Johnston
1a81a1898e Keep pruning background task 2017-06-07 11:16:56 +01:00
Erik Johnston
6ba21bf2b8 Comments 2017-06-07 11:08:36 +01:00
Erik Johnston
09e4bc0501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup 2017-06-07 11:05:23 +01:00
Erik Johnston
6e2a7ee1bc Remove spurious log lines 2017-06-07 11:05:17 +01:00
Erik Johnston
65f0513a33 Split up device_lists_outbound_pokes table for faster updates. 2017-06-07 11:02:38 +01:00
Erik Johnston
6f83c4537c Increase size of IP cache 2017-06-07 10:18:44 +01:00
Erik Johnston
cca94272fa Fix typo when getting app name 2017-06-06 11:50:07 +01:00
Erik Johnston
66b121b2fc Fix wrong number of arguments 2017-06-06 11:46:38 +01:00
Erik Johnston
1a01af079e Handle profile updates in user directory 2017-06-01 15:39:51 +01:00
Erik Johnston
4d039aa2ca Fix sqlite 2017-06-01 14:58:48 +01:00
Erik Johnston
21e255a8f1 Split the table in two 2017-06-01 14:50:46 +01:00
Erik Johnston
d5477c7afd Tweak search query 2017-06-01 13:28:01 +01:00
Erik Johnston
02a6108235 Tweak search query 2017-06-01 13:16:40 +01:00
Erik Johnston
7233341eac Comments 2017-06-01 13:11:38 +01:00
Erik Johnston
8be6fd95a3 Check if host is still in room 2017-06-01 13:05:39 +01:00
Erik Johnston
59dbb47065 Remove spurious inlineCallbacks 2017-06-01 11:41:29 +01:00
Erik Johnston
9c7db2491b Fix removing users 2017-06-01 11:36:50 +01:00
Erik Johnston
0fe6f3c521 Bug fixes and logging 
- Check if room is public when a user joins before adding to user dir
- Fix typo of field name "content.join_rules" -> "content.join_rule"
 2017-06-01 11:09:49 +01:00
Erik Johnston
036362ede6 Order by if they have profile info 2017-06-01 09:41:08 +01:00
Erik Johnston
a757dd4863 Use prefix matching 2017-06-01 09:40:37 +01:00
Erik Johnston
f5cc22bdc6 Comment on why arbitrary comments 2017-05-31 17:30:26 +01:00
Erik Johnston
5dd1b2c525 Use unique indices 2017-05-31 17:29:12 +01:00
Erik Johnston
cc7609aa9f Comment briefly on how we keep user_directory up to date 2017-05-31 17:11:18 +01:00
Erik Johnston
f1378aef91 Convert to int 2017-05-31 17:03:08 +01:00
Erik Johnston
b2d8d07109 Lifts things into separate function 2017-05-31 17:00:24 +01:00
Erik Johnston
f9791498ae Typos 2017-05-31 16:50:57 +01:00
Erik Johnston
f091061711 Fix tests 2017-05-31 16:34:40 +01:00
Erik Johnston
4abcff0177 Fix typo 2017-05-31 16:22:36 +01:00
Erik Johnston
63c58c2a3f Limit number of things we fetch out of the db 2017-05-31 16:17:58 +01:00
Erik Johnston
304880d185 Add stream change cache 2017-05-31 15:46:36 +01:00
Erik Johnston
5d79d728f5 Split out directory and search tables 2017-05-31 15:23:49 +01:00
Erik Johnston
dc51af3d03 Pull max id from correct table 2017-05-31 15:13:49 +01:00
Erik Johnston
350622a107 Handle the server leaving a public room 2017-05-31 15:11:36 +01:00
Erik Johnston
63fda37e20 Add comments 2017-05-31 15:00:29 +01:00
Erik Johnston
293ef29655 Weight differently 2017-05-31 14:29:32 +01:00
Erik Johnston
535c99f157 Use POST 2017-05-31 14:15:45 +01:00
Erik Johnston
45a5df5914 Add REST API 2017-05-31 14:11:55 +01:00
Erik Johnston
3b5f22ca40 Add search 2017-05-31 14:00:01 +01:00
Erik Johnston
b5db4ed5f6 Update room column when room becomes unpublic 2017-05-31 13:40:28 +01:00
Erik Johnston
168524543f Add call later 2017-05-31 11:59:36 +01:00
Erik Johnston
3e123b8497 Start later 2017-05-31 11:56:27 +01:00
Erik Johnston
42137efde7 Don't go round in circles 2017-05-31 11:55:13 +01:00
Erik Johnston
eeb2f9e546 Add user_directory to database 2017-05-31 11:51:01 +01:00
Erik Johnston
dd48f7204c Add comment 2017-05-30 15:01:22 +01:00
Erik Johnston
04095f7581 Add clobbered event_id 2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e Add current_state_delta_stream table 2017-05-30 14:44:09 +01:00
Erik Johnston
619e8ecd0c Handle None state group correctly 2017-05-26 10:46:03 +01:00
Erik Johnston
23da638360 Fix typing tests 2017-05-26 10:02:04 +01:00
Erik Johnston
dfbda5e025 Faster cache for get_joined_hosts 2017-05-25 17:24:44 +01:00
Erik Johnston
2b03751c3c Don't return weird prev_group 2017-05-25 14:47:39 +01:00
Erik Johnston
dbc0dfd2d5 Remove unused options 2017-05-25 14:28:34 +01:00
Erik Johnston
11f139a647 Merge pull request #2247 from matrix-org/erikj/auth_event 
Only store event_auth for state events
 2017-05-24 16:46:34 +01:00
Erik Johnston
6e614e9e10 Add background task to clear out old event_auth 2017-05-24 15:23:34 +01:00
Erik Johnston
c049472b8a Only store event_auth for state events 2017-05-24 15:23:31 +01:00
Erik Johnston
9a804b2812 Merge pull request #2243 from matrix-org/matthew/fix-url-preview-length-again 
actually trim oversize og:description meta
 2017-05-23 13:26:28 +01:00
Erik Johnston
fbbc40f385 Merge pull request #2237 from matrix-org/erikj/sync_key_count 
Add count of one time keys to sync stream
 2017-05-23 11:18:13 +01:00
Erik Johnston
8cf9f0a3e7 Remove redundant invalidation 2017-05-23 09:46:59 +01:00
Erik Johnston
e6618ece2d Missed an invalidation 2017-05-23 09:36:52 +01:00
Matthew Hodgson
836d5c44b6 actually trim oversize og:description meta 2017-05-22 21:14:20 +01:00
Erik Johnston
11c2a3655f Only load jinja2 templates once 
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
 2017-05-22 17:48:58 +01:00
Erik Johnston
539aa4d333 Merge pull request #2241 from matrix-org/erikj/fix_notifs 
Correctly calculate push rules for member events
 2017-05-22 16:46:58 +01:00
Erik Johnston
f85a415279 Add missing storage function to slave store 2017-05-22 16:31:24 +01:00
Erik Johnston
6489455bed Comment 2017-05-22 16:22:04 +01:00
Erik Johnston
d668caa79c Remove spurious log level guards 2017-05-22 16:21:06 +01:00
Erik Johnston
74bf4ee7bf Stream count_e2e_one_time_keys cache invalidation 2017-05-22 16:19:22 +01:00
Erik Johnston
33ba90c6e9 Merge pull request #2240 from matrix-org/erikj/cache_list_fix 
Update list cache to handle one arg case
 2017-05-22 16:10:46 +01:00
Erik Johnston
bd7bb5df71 Pull out if statement from for loop 2017-05-22 15:12:19 +01:00
Erik Johnston
e3417a06e2 Update list cache to handle one arg case 
We update the normal cache descriptors to handle caches with a single
argument specially so that the key wasn't a 1-tuple. We need to update
the cache list to be aware of this.
 2017-05-22 15:04:42 +01:00
Erik Johnston
7fb80b5eae Check if current event is a membership event 2017-05-22 15:02:12 +01:00
Erik Johnston
2d17b09a6d Add debug logging 2017-05-22 15:01:36 +01:00
Erik Johnston
24c8f38784 Comment 2017-05-22 14:59:27 +01:00
Erik Johnston
25f03cf8e9 Use tuple unpacking 2017-05-22 14:58:22 +01:00
Erik Johnston
270e1c904a Speed up calculating push rules 2017-05-19 16:51:05 +01:00
Erik Johnston
b4f59c7e27 Add count of one time keys to sync stream 2017-05-19 15:47:55 +01:00
Erik Johnston
58ebb96cce Fix invalidation of get_users_with_read_receipts_in_room 2017-05-19 14:38:50 +01:00
Erik Johnston
1c1c0257f4 Move invalidation cb to its own structure 2017-05-19 11:44:11 +01:00
Erik Johnston
cafe659f72 Store ActionGenerator in HomeServer 2017-05-19 10:09:56 +01:00
Erik Johnston
72ed8196b3 Don't push users who have left 2017-05-18 17:48:36 +01:00
Erik Johnston
107ac7ac96 Increase size of push rule caches 2017-05-18 17:17:53 +01:00
Erik Johnston
760625acba Make get_if_app_services_interested_in_user faster 2017-05-18 16:34:44 +01:00
Erik Johnston
c57789d138 Remove size of push get_rules cache 2017-05-18 16:17:23 +01:00
Erik Johnston
f33df30732 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-05-18 13:56:37 +01:00
Erik Johnston
a5425b2e5b Bump changelog and version 2017-05-18 13:53:48 +01:00
Erik Johnston
6e381180ae Merge pull request #2177 from matrix-org/erikj/faster_push_rules 
Make calculating push actions faster
 2017-05-18 11:46:18 +01:00
Erik Johnston
056ba9b795 Add comment 2017-05-18 11:45:56 +01:00
Erik Johnston
d9e3a4b5db Merge pull request #2230 from matrix-org/erikj/speed_up_get_state 
Make get_state_groups_from_groups faster.
 2017-05-17 17:23:04 +01:00
Erik Johnston
66d8ffabbd Faster push rule calculation via push specific cache 
We add a push rule specific cache that ensures that we can reuse
calculated push rules appropriately when a user join/leaves.
 2017-05-17 16:55:40 +01:00
Erik Johnston
ace23463c5 Merge pull request #2216 from slipeer/app_services_interested_in_user 
Fix users claimed non-exclusively by an app service don't get notific…
 2017-05-17 16:28:50 +01:00
Erik Johnston
bbfe4e996c Make get_state_groups_from_groups faster. 
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.

By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
 2017-05-17 15:12:15 +01:00
Erik Johnston
a8bc7cae56 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.21.0 2017-05-17 13:11:43 +01:00
Erik Johnston
3a431a126d Bump changelog and version 2017-05-17 11:26:57 +01:00
Erik Johnston
85e8092cca Comment 2017-05-17 10:03:09 +01:00
Erik Johnston
ad53fc3cf4 Short circuit when we have delta ids 2017-05-17 09:57:34 +01:00
Erik Johnston
6fa8148ccb Merge pull request #2228 from matrix-org/erikj/speed_up_get_hosts 
Speed up get_joined_hosts
 2017-05-16 17:40:55 +01:00
Erik Johnston
7c69849a0d Merge pull request #2227 from matrix-org/erikj/presence_caches 
Make presence use cached users/hosts in room
 2017-05-16 17:40:47 +01:00
Erik Johnston
13f540ef1b Speed up get_joined_hosts 2017-05-16 16:05:22 +01:00
Erik Johnston
ec5c4499f4 Make presence use cached users/hosts in room 2017-05-16 16:01:43 +01:00
Erik Johnston
f2a5b6dbfd Speed up get_domain_from_id 2017-05-16 15:59:37 +01:00
Erik Johnston
b8492b6c2f Merge pull request #2224 from matrix-org/erikj/prefill_state 
Prefill state caches
 2017-05-16 15:50:11 +01:00
Erik Johnston
331570ea6f Remove spurious merge artifacts 2017-05-16 15:33:07 +01:00
Krombel
55af207321 Merge branch 'develop' into avoid_duplicate_filters 2017-05-16 15:29:59 +02:00
Richard van der Hoff
d648f65aaf Merge pull request #2218 from matrix-org/rav/event_search_index 
Add an index to event_search
 2017-05-16 13:26:07 +01:00
Erik Johnston
608b5a6317 Take a copy before prefilling, as it may be a frozendict 2017-05-16 12:55:29 +01:00
Krombel
64953c8ed2 avoid access-error if no filter_id matches 2017-05-15 18:36:37 +02:00
Erik Johnston
f451b64c8f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state 2017-05-15 16:09:32 +01:00
Erik Johnston
2c9475b58e Merge pull request #2221 from psaavedra/sync_timeline_limit_filter_by_name 
Configurable maximum number of events requested by /sync and /messages
 2017-05-15 16:08:46 +01:00
Erik Johnston
d12ae7fd1c Don't log exceptions for NotRetryingDestination 2017-05-15 15:42:18 +01:00
Pablo Saavedra
224137fcf9 Fixed syntax nits 2017-05-15 16:21:02 +02:00
Erik Johnston
e4435b014e Update comment 2017-05-15 15:11:30 +01:00
Erik Johnston
871605f4e2 Comments 2017-05-15 15:11:30 +01:00
Erik Johnston
e0d2f6d5b0 Add more granular event send metrics 2017-05-15 15:11:30 +01:00
Erik Johnston
bfbc907cec Prefill state caches 2017-05-15 15:11:13 +01:00
Pablo Saavedra
627e6ea2b0 Fixed implementation errors 
* Added HS as property in SyncRestServlet
* Fixed set_timeline_upper_limit function implementat¡ion
 2017-05-15 14:51:43 +02:00
Pablo Saavedra
9da4316ca5 Configurable maximum number of events requested by /sync and /messages (#2220) 
Set the limit on the returned events in the timeline in the get and sync
operations. The default value is -1, means no upper limit.

For example, using `filter_timeline_limit: 5000`:

POST /_matrix/client/r0/user/user:id/filter
{
room: {
    timeline: {
      limit: 1000000000000000000
    }
}
}

GET /_matrix/client/r0/user/user:id/filter/filter:id

{
room: {
    timeline: {
      limit: 5000
    }
}
}

The server cuts down the room.timeline.limit.
 2017-05-13 18:17:54 +02:00
Krombel
eb7cbf27bc insert whitespace to fix travis build 2017-05-12 12:09:42 +02:00
Krombel
6b95e35e96 add check to only add a new filter if the same filter does not exist previously 
Signed-off-by: Matthias Kesler <krombel@krombel.de>
 2017-05-11 16:05:30 +02:00
Richard van der Hoff
ff3d810ea8 Add a comment to old delta 2017-05-11 12:48:50 +01:00
Richard van der Hoff
34194aaff7 Don't create event_search index on sqlite 
... because the table is virtual
 2017-05-11 12:46:55 +01:00
Richard van der Hoff
114f290947 Add more logging for purging 
Log the number of events we will be deleting at info.
 2017-05-11 12:08:47 +01:00
Richard van der Hoff
baafb85ba4 Add an index to event_search 
- to make the purge API quicker
 2017-05-11 12:05:22 +01:00
Richard van der Hoff
29ded770b1 Merge pull request #2214 from matrix-org/rav/hurry_up_purge 
When purging, don't de-delta state groups we're about to delete
 2017-05-11 12:04:25 +01:00
Richard van der Hoff
dc026bb16f Tidy purge code and add some comments 
Try to make this clearer with more comments and some variable renames
 2017-05-11 10:56:12 +01:00
Slipeer
328378f9cb Fix users claimed non-exclusively by an app service don't get notifications #2211 2017-05-11 11:42:08 +03:00
Richard van der Hoff
8e345ce465 Don't de-delta state groups we're about to delete 2017-05-10 18:44:22 +01:00
Richard van der Hoff
b64d312421 add some logging to purge_history 2017-05-10 18:44:22 +01:00
Luke Barnard
ccad2ed824 Modify condition on empty localpart 2017-05-10 17:34:30 +01:00
Luke Barnard
369195caa5 Modify register/available to be GET with query param 
- GET is now the method for register/available
- a query parameter "username" is now used

Also, empty usernames are now handled with an error message on registration or via register/available: `User ID cannot be empty`
 2017-05-10 17:23:55 +01:00
Erik Johnston
a3648f84b2 Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid 
Add per user ratelimiting overrides
 2017-05-10 15:54:48 +01:00
Richard van der Hoff
5331cd150a Merge pull request #2206 from matrix-org/rav/one_time_key_upload_change_sig 
Allow clients to upload one-time-keys with new sigs
 2017-05-10 14:18:40 +01:00
Luke Barnard
f7278e612e Change register/available to POST (from GET) 2017-05-10 11:40:18 +01:00
Erik Johnston
b990b2fce5 Add per user ratelimiting overrides 2017-05-10 11:05:43 +01:00
Richard van der Hoff
aedaba018f Replace some instances of preserve_context_over_deferred 2017-05-09 19:04:56 +01:00
Richard van der Hoff
de042b3b88 Do some logging when one-time-keys get claimed 
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
 2017-05-09 19:04:56 +01:00
Richard van der Hoff
a7e9d8762d Allow clients to upload one-time-keys with new sigs 
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).

Fixes https://github.com/vector-im/riot-android/issues/1208, hopefully.
 2017-05-09 19:04:56 +01:00
Erik Johnston
ca238bc023 Merge branch 'release-v0.21.0' of github.com:matrix-org/synapse into develop 2017-05-08 17:35:11 +01:00
Erik Johnston
40dcf0d856 Merge pull request #2203 from matrix-org/erikj/event_cache_hit_ratio 
Don't update event cache hit ratio from get_joined_users
 2017-05-08 16:52:39 +01:00
Erik Johnston
d3c3026496 Merge pull request #2201 from matrix-org/erikj/store_device_cache 
Cache check to see if device exists
 2017-05-08 16:23:04 +01:00
Erik Johnston
093f7e47cc Expand docstring a bit 2017-05-08 16:14:46 +01:00
Erik Johnston
6a12998a83 Add missing yields 2017-05-08 16:10:51 +01:00
Erik Johnston
ffad4fe35b Don't update event cache hit ratio from get_joined_users 
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
 2017-05-08 16:06:17 +01:00
Erik Johnston
94e6ad71f5 Invalidate cache on device deletion 2017-05-08 15:55:59 +01:00
Erik Johnston
8571f864d2 Cache one time key counts 2017-05-08 15:34:27 +01:00
Erik Johnston
fc6d4974a6 Comment 2017-05-08 15:33:57 +01:00
Erik Johnston
738ccf61c0 Cache check to see if device exists 2017-05-08 15:32:18 +01:00
Erik Johnston
dcabef952c Increase client_ip cache size 2017-05-08 15:09:19 +01:00
Erik Johnston
771c8a83c7 Bump version and changelog 2017-05-08 13:23:46 +01:00
Erik Johnston
6631985990 Merge pull request #2200 from matrix-org/erikj/revert_push 
Revert speed up push
 2017-05-08 13:20:52 +01:00
Erik Johnston
e0f20e9425 Revert "Remove unused import" 
This reverts commit ab37bef83b.
 2017-05-08 13:07:43 +01:00
Erik Johnston
fe7c1b969c Revert "We don't care about forgotten rooms" 
This reverts commit ad8b316939.
 2017-05-08 13:07:43 +01:00
Erik Johnston
78f306a6f7 Revert "Speed up filtering of a single event in push" 
This reverts commit 421fdf7460.
 2017-05-08 13:07:41 +01:00
Erik Johnston
9ac98197bb Bump version and changelog 2017-05-08 11:07:54 +01:00
Erik Johnston
27c28eaa27 Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more 
Always mark remotes as up if we receive a signed request from them
 2017-05-05 14:08:12 +01:00
Erik Johnston
be2672716d Merge pull request #2189 from matrix-org/erikj/handle_remote_device_list 
Handle exceptions thrown in handling remote device list updates
 2017-05-05 14:01:27 +01:00
Erik Johnston
653d90c1a5 Comment 2017-05-05 14:01:17 +01:00
Erik Johnston
310b1ccdc1 Use preserve_fn and add logs 2017-05-05 13:41:19 +01:00
Kegsay
a59b0ad1a1 Merge pull request #2192 from matrix-org/kegan/simple-http-client-timeouts 
Rewrite SimpleHttpClient.request to include timeouts
 2017-05-05 11:52:43 +01:00
Erik Johnston
7b222fc56e Remove redundant reset of destination timers 2017-05-05 11:14:09 +01:00
Kegan Dougal
d0debb2116 Remember how twisted works 2017-05-05 11:00:21 +01:00
Erik Johnston
66f371e8b8 Merge pull request #2176 from matrix-org/erikj/faster_get_joined 
Make get_joined_users faster
 2017-05-05 10:59:55 +01:00
Erik Johnston
b843631d71 Add comment and TODO 2017-05-05 10:59:32 +01:00
Kegan Dougal
c2ddd773bc Include the clock 2017-05-05 10:52:46 +01:00
Kegan Dougal
7dd3bf5e24 Rewrite SimpleHttpClient.request to include timeouts 
Fixes #2191
 2017-05-05 10:49:19 +01:00
Erik Johnston
db7d0c3127 Always mark remotes as up if we receive a signed request from them 2017-05-05 10:34:53 +01:00
Erik Johnston
f346048a6e Handle exceptions thrown in handling remote device list updates 2017-05-05 10:34:10 +01:00
Erik Johnston
e3aa8a7aa8 Merge pull request #2185 from matrix-org/erikj/smaller_caches 
Optimise caches for single key
 2017-05-05 10:19:05 +01:00
Erik Johnston
cf589f2c1e Fixes 2017-05-05 10:17:56 +01:00
Erik Johnston
8af4569583 Merge pull request #2174 from matrix-org/erikj/current_cache_hosts 
Add cache for get_current_hosts_in_room
 2017-05-05 10:15:24 +01:00
Erik Johnston
587f07543f Revert "Prefill state caches" 2017-05-04 15:07:27 +01:00
Erik Johnston
aa93cb9f44 Add comment 2017-05-04 14:59:28 +01:00
Erik Johnston
537dbadea0 Intern host strings 2017-05-04 14:55:28 +01:00
Erik Johnston
07a07588a0 Make caches bigger 2017-05-04 14:52:28 +01:00
Erik Johnston
dfaa58f72d Fix comment and num args 2017-05-04 14:50:24 +01:00
Erik Johnston
9ac263ed1b Add new storage functions to slave store 2017-05-04 14:29:03 +01:00
Erik Johnston
d2d8ed4884 Optimise caches with single key 2017-05-04 14:18:46 +01:00
Erik Johnston
5d8290429c Reduce size of get_users_in_room 2017-05-04 13:43:19 +01:00
Luke Barnard
6aa423a1a8 Merge pull request #2183 from matrix-org/luke/username-availability 
Implement username availability checker
 2017-05-04 09:58:40 +01:00
Luke Barnard
3669065466 Appease the flake8 gods 2017-05-03 18:05:49 +01:00
Erik Johnston
7ebf518c02 Make get_joined_users faster 2017-05-03 15:55:54 +01:00
Luke Barnard
34ed4f4206 Implement username availability checker 
Outlined here: https://github.com/vector-im/riot-web/issues/3605#issuecomment-298679388

```HTTP
GET /_matrix/.../register/available
{
    "username": "desiredlocalpart123"
}
```

If available, the response looks like
```HTTP
HTTP/1.1 200 OK
{
    "available": true
}
```

Otherwise,
```HTTP
HTTP/1.1 429
{
    "errcode": "M_LIMIT_EXCEEDED",
    "error": "Too Many Requests",
    "retry_after_ms": 2000
}
```
or
```HTTP
HTTP/1.1 400
{
    "errcode": "M_USER_IN_USE",
    "error": "User ID already taken."
}

```
or
```HTTP
HTTP/1.1 400
{
    "errcode": "M_INVALID_USERNAME",
    "error": "Some reason for username being invalid"
}
```
 2017-05-03 12:04:12 +01:00
David Baker
60833c8978 Merge pull request #2147 from matrix-org/dbkr/http_request_propagate_error 
Propagate errors sensibly from proxied IS requests
 2017-05-03 11:23:25 +01:00
David Baker
482a2ad122 No need for the exception variable 2017-05-03 11:02:59 +01:00
David Baker
c0380402bc List caught expection types 2017-05-03 10:56:22 +01:00
Erik Johnston
cdbf38728d Merge pull request #2175 from matrix-org/erikj/prefill_state 
Prefill state caches
 2017-05-03 10:54:11 +01:00
Erik Johnston
0c27383dd7 Merge pull request #2170 from matrix-org/erikj/fed_hole_state 
Don't fetch state for missing events that we fetched
 2017-05-03 10:49:37 +01:00
Erik Johnston
ef862186dd Merge together redundant calculations/logging 2017-05-03 10:06:43 +01:00
Erik Johnston
2c2dcf81d0 Update comment 2017-05-03 10:00:29 +01:00
Erik Johnston
1827057acc Comments 2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state 2017-05-03 09:46:40 +01:00
Erik Johnston
e4c15fcb5c Merge pull request #2178 from matrix-org/erikj/message_metrics 
Add more granular event send metrics
 2017-05-02 17:57:34 +01:00
Erik Johnston
3e5a62ecd8 Add more granular event send metrics 2017-05-02 14:23:26 +01:00
Richard van der Hoff
82475a18d9 Merge pull request #2180 from matrix-org/rav/fix_timeout_on_timeout 
Instantiate DeferredTimedOutError correctly
 2017-05-02 13:32:58 +01:00
Richard van der Hoff
2e996271fe Instantiate DeferredTimedOutError correctly 
Call `super` correctly, so that we correctly initialise the `errcode` field.

Fixes https://github.com/matrix-org/synapse/issues/2179.
 2017-05-02 13:26:17 +01:00
Erik Johnston
a2c89a225c Prefill state caches 2017-05-02 10:40:31 +01:00
Erik Johnston
7166854f41 Add cache for get_current_hosts_in_room 2017-05-02 10:36:35 +01:00
Erik Johnston
3033261891 Merge pull request #2080 from matrix-org/erikj/filter_speed 
Speed up filtering of a single event in push
 2017-04-28 14:17:13 +01:00
Erik Johnston
2347efc065 Fixup 2017-04-28 12:46:53 +01:00
Erik Johnston
9b147cd730 Remove unncessary call in _get_missing_events_for_pdu 2017-04-28 11:55:25 +01:00
Erik Johnston
3a9f5bf6dd Don't fetch state for missing events that we fetched 2017-04-28 11:26:46 +01:00
Erik Johnston
ab37bef83b Remove unused import 2017-04-28 09:57:23 +01:00
Erik Johnston
ad8b316939 We don't care about forgotten rooms 2017-04-28 09:52:36 +01:00
Erik Johnston
421fdf7460 Speed up filtering of a single event in push 2017-04-28 09:52:36 +01:00
Erik Johnston
25a96e0c63 Merge pull request #2163 from matrix-org/erikj/fix_invite_state 
Fix invite state to always include all events
 2017-04-27 17:36:30 +01:00
Erik Johnston
46826bb078 Comment and remove spurious logging 2017-04-27 17:25:44 +01:00
Erik Johnston
f87b287291 Merge pull request #2127 from APwhitehat/alreadystarted 
print something legible if synapse already running
 2017-04-27 15:46:53 +01:00
Erik Johnston
bb9246e525 Merge pull request #2131 from matthewjwolff/develop 
web_client_location documentation fix
 2017-04-27 15:46:40 +01:00
Richard van der Hoff
c84770b877 Fix bgupdate error if index already exists (#2167) 
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135.

Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
 2017-04-27 15:27:48 +01:00
Erik Johnston
380fb87ecc Merge pull request #2168 from matrix-org/erikj/federation_logging 
Add some extra logging for edge cases of federation
 2017-04-27 15:19:12 +01:00
Erik Johnston
87ae59f5e9 Typo 2017-04-27 15:16:21 +01:00
Erik Johnston
e42b4ebf0f Add some extra logging for edge cases of federation 2017-04-27 14:38:21 +01:00
Erik Johnston
d3c150411c Merge pull request #2130 from APwhitehat/roomexists 
Check that requested room_id exists
 2017-04-27 09:20:26 +01:00
Erik Johnston
1e166470ab Fix tests 2017-04-26 16:23:30 +01:00
Erik Johnston
34e682d385 Fix invite state to always include all events 2017-04-26 16:18:08 +01:00
David Baker
5fd12dce01 Remove debugging 2017-04-26 12:36:26 +01:00
David Baker
82ae0238f9 Revert accidental commit 2017-04-26 11:43:16 +01:00
David Baker
81804909d3 Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error 2017-04-26 11:31:55 +01:00
David Baker
c366276056 Fix get_json 2017-04-26 10:07:01 +01:00
David Baker
1a9255c12e Use CodeMessageException subclass instead 
Parse json errors from get_json client methods and throw special
errors.
 2017-04-25 19:30:55 +01:00
Erik Johnston
f053a1409e Make state caches cache in ascii 2017-04-25 17:22:55 +01:00
Erik Johnston
acb58bfb6a fix up 2017-04-25 15:39:19 +01:00
Erik Johnston
f7181615f2 Don't specify default as dict 2017-04-25 15:22:59 +01:00
Erik Johnston
f144365281 Comment 2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86 Reduce size of joined_user cache 
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
 2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76 Reduce _get_state_group_for_event cache size 2017-04-25 11:43:03 +01:00
Erik Johnston
b4da08cad8 Merge pull request #2158 from matrix-org/erikj/reduce_cache_size 
Reduce cache size by not storing deferreds
 2017-04-25 11:18:07 +01:00
Erik Johnston
efab1dadde Remove DEBUG_CACHES 2017-04-25 10:54:09 +01:00
Erik Johnston
119cb9bbcf Reduce cache size by not storing deferreds 
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
 2017-04-25 10:23:11 +01:00
Mark Haines
e6e2627636 Fix code for reporting old verify keys in synapse 2017-04-24 18:51:25 +01:00
Richard van der Hoff
30f7bfa121 Merge pull request #2145 from matrix-org/rav/reject_invite_to_unreachable_server 
Fix rejection of invites to unreachable servers
 2017-04-24 15:20:52 +01:00
Erik Johnston
7af825bae4 Merge pull request #2155 from matrix-org/erikj/string_intern 
Only intern ascii strings
 2017-04-24 14:28:41 +01:00
Erik Johnston
d134d0935e Only intern ascii strings 2017-04-24 14:07:48 +01:00
Erik Johnston
e4f3431116 Remove unused cache 2017-04-24 13:27:38 +01:00
David Baker
a46982cee9 Need the HTTP status code 2017-04-21 16:20:12 +01:00
David Baker
70caf49914 Do the same for get_json 2017-04-21 16:09:03 +01:00
David Baker
a1595cec78 Don't error for 3xx responses 2017-04-21 11:51:17 +01:00
David Baker
2e165295b7 Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error 2017-04-21 11:35:52 +01:00
David Baker
a90a0f5c8a Propagate errors sensibly from proxied IS requests 
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
 2017-04-21 11:32:48 +01:00
Richard van der Hoff
91b3981800 Try harder when sending leave events 
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
 2017-04-21 01:50:36 +01:00
Richard van der Hoff
0cdb32fc43 Remove redundant try/except clauses 
The `except SynapseError` clauses were pointless because the wrapped functions
would never throw a `SynapseError` (they either throw a `CodeMessageException`
or a `RuntimeError`).

The `except CodeMessageException` is now also pointless because the caller
treats all exceptions equally, so we may as well just throw the
`CodeMessageException`.
 2017-04-21 01:32:01 +01:00
Richard van der Hoff
838810b76a Broaden the conditions for locally_rejecting invites 
The logic for marking invites as locally rejected was all well and good, but
didn't happen when the remote server returned a 500, or wasn't reachable, or
had no DNS, or whatever.

Just expand the except clause to catch everything.

Fixes https://github.com/matrix-org/synapse/issues/761.
 2017-04-21 01:31:37 +01:00
Richard van der Hoff
736b9a4784 Remove redundant function 
inline `reject_remote_invite`, which only existed to make tracing the callflow
more difficult.
 2017-04-21 01:31:09 +01:00
Richard van der Hoff
4903ccf159 Fix some lies, and other clarifications, in docstrings 
The documentation on get_json has been wrong ever since the very first commit
to synapse...
 2017-04-21 01:31:09 +01:00
Matthew Wolff
d4040e9e28
Queried CONDITIONAL_REQUIREMENTS 2017-04-18 16:19:48 -05:00
Luke Barnard
3fb8784c92 m.read_marker -> m.fully_read (#2128) 
Also:
 - change the REST endpoint to have a "S" on the end (so it's now /read_markers)
 - change the content of the m.read_up_to event to have the key "event_id" instead of "marker".
 2017-04-18 17:46:15 +01:00
Matthew Wolff
54f9a4cb59
Fixed travis build failure 
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
 2017-04-17 01:38:27 -05:00
Matthew Wolff
8e780b113d
web_server_root documentation fix 
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
 2017-04-17 00:49:11 -05:00
Anant Prakash
574d573ac2 Check that requested room_id exists 2017-04-14 23:50:59 +05:30
Luke Barnard
78f0ddbfad Merge pull request #2120 from matrix-org/luke/read-markers 
Implement Read Marker API
 2017-04-13 14:21:31 +01:00
Luke Barnard
6a70647d45 Correct logic in is_event_after 2017-04-13 13:46:17 +01:00
Anant Prakash
c1f52a321d synctl.py: Check if synapse is already running 2017-04-13 18:00:02 +05:30
Luke Barnard
b9557064bf Simplify is_event_after logic 2017-04-12 14:36:20 +01:00
Luke Barnard
cf6121e3da More null-guard changes 2017-04-12 14:02:03 +01:00
Erik Johnston
247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl 
Reduce federation replication traffic
 2017-04-12 11:07:13 +01:00
Paul Evans
8fbc0d29ee Merge pull request #2121 from matrix-org/paul/sent-transactions-metric 
Add a counter metric for successfully-sent transactions
 2017-04-12 11:04:31 +01:00
Erik Johnston
c06c00190f Merge pull request #2116 from matrix-org/erikj/dedupe_federation_repl2 
Dedupe KeyedEdu and Devices federation repl traffic
 2017-04-12 10:57:24 +01:00
Luke Barnard
c0aba0a23e Remove Unused ref to hs 2017-04-12 10:52:11 +01:00
Luke Barnard
b9676a75f6 Move a space 2017-04-12 10:51:17 +01:00
Luke Barnard
69a18514e9 Only notify user, not entire room 2017-04-12 10:50:37 +01:00
Luke Barnard
122cd52ce4 Remove comment, simplify null-guard 2017-04-12 10:48:32 +01:00
Erik Johnston
26ae5178a4 Add some comments 2017-04-12 10:36:29 +01:00
Erik Johnston
bf9060156a Merge pull request #2117 from matrix-org/erikj/remove_http_replication 
Remove HTTP replication APIs
 2017-04-12 10:21:42 +01:00
Erik Johnston
1745069543 Comment 2017-04-12 10:17:10 +01:00
Erik Johnston
c7ddb5ef7a Reuse get_interested_parties 2017-04-12 10:16:26 +01:00
Erik Johnston
7b41013102 Merge pull request #2118 from matrix-org/erikj/no_devices 
Fix getting latest device IP for user with no devices
 2017-04-12 10:14:32 +01:00
Luke Barnard
77fb2b72ae Handle no previous RM 2017-04-12 09:47:29 +01:00
Luke Barnard
7f94709066 travis flake8.. 2017-04-11 18:35:45 +01:00
Luke Barnard
867822fa1e flake8 2017-04-11 17:36:04 +01:00
Luke Barnard
73880268ef Refactor event ordering check to events store 2017-04-11 17:34:09 +01:00
Luke Barnard
131485ef66 Copyright 2017-04-11 17:33:51 +01:00
Paul "LeoNerd" Evans
11dbceb761 Add a counter metric for successfully-sent transactions 2017-04-11 17:16:12 +01:00
Luke Barnard
0127423027 flake8 2017-04-11 17:07:07 +01:00
Erik Johnston
85657eedf8 Bail on where clause instead 2017-04-11 16:24:31 +01:00
Erik Johnston
b48045a8f5 Don't bother with outer check for now 2017-04-11 16:23:24 +01:00
Erik Johnston
9c712a366f Move get_presence_list_* to SlaveStore 2017-04-11 16:07:33 +01:00
Erik Johnston
a8c8e4efd4 Comment 2017-04-11 15:35:49 +01:00
Erik Johnston
414522aed5 Move get_interested_parties 2017-04-11 15:33:26 +01:00
Erik Johnston
2be8a281d2 Comments 2017-04-11 15:28:24 +01:00
Erik Johnston
6308ac45b0 Move get_interested_remotes back to presence handler 2017-04-11 15:19:26 +01:00
Erik Johnston
b9b72bc6e2 Comments 2017-04-11 15:15:34 +01:00
Luke Barnard
d892079844 Finish implementing RM endpoint 
- This change causes a 405 to be sent if "m.read_marker" is set via /account_data
 - This also fixes-up the RM endpoint so that it actually Works.
 2017-04-11 15:01:39 +01:00
lukebarnard
e263c26690 Initial commit of RM server-side impl 
(See https://docs.google.com/document/d/1UWqdS-e1sdwkLDUY0wA4gZyIkRp-ekjsLZ8k6g_Zvso/edit#heading=h.lndohpg8at5u)
 2017-04-11 11:55:30 +01:00
Erik Johnston
f3cf3ff8b6 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-04-11 11:13:32 +01:00
Erik Johnston
34840cdcef Fix getting latest device IP for user with no devices 2017-04-11 09:56:54 +01:00
Erik Johnston
28a4649785 Remove HTTP replication APIs 2017-04-11 09:52:11 +01:00
Matthew Hodgson
7c551ec445 trust a hypothetical future riot.im IS 2017-04-10 17:58:36 +01:00
Erik Johnston
84fbb80c8f Use generators 2017-04-10 16:55:56 +01:00
Erik Johnston
40453b3f84 Dedupe KeyedEdu and Devices federation repl traffic 2017-04-10 16:49:51 +01:00
Erik Johnston
29574fd5b3 Reduce federation presence replication traffic 
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
 2017-04-10 16:48:30 +01:00
Erik Johnston
2e6f5a4910 Typo 2017-04-10 16:17:40 +01:00
David Baker
405ba4178a Merge pull request #2102 from DanielDent/add-auth-email 
Support authenticated SMTP
 2017-04-10 15:42:16 +01:00
Erik Johnston
efcb6db688 Merge pull request #2109 from matrix-org/erikj/send_queue_fix 
Fix up federation SendQueue and document types
 2017-04-10 13:09:25 +01:00
Erik Johnston
0018491af2 Rename variable 2017-04-10 12:44:43 +01:00
Erik Johnston
0364d23210 Up replication ping timeout 2017-04-10 11:32:05 +01:00
Erik Johnston
8c5f03cec7 Revert to sending the same data type as before 2017-04-10 10:07:18 +01:00
Erik Johnston
f8434db549 Change name 2017-04-10 10:03:07 +01:00
Erik Johnston
ab904caf33 Comments 2017-04-10 10:02:17 +01:00
Richard van der Hoff
64765e5199 When we do an invite rejection, save the signed leave event to the db 
During a rejection of an invite received over federation, we ask a remote
server to make us a `leave` event, then sign it, then send that with
`send_leave`.

We were saving the *unsigned* version of the event (which has a different event
id to the signed version) to our db (and sending it to the clients), whereas
other servers in the room will have seen the *signed* version. We're not aware
of any actual problems that caused, except that it makes the database confusing
to look at and generally leaves the room in a weird state.
 2017-04-07 14:39:32 +01:00
Erik Johnston
2a3e822f44 Comment 2017-04-07 13:47:04 +01:00
Erik Johnston
a828a64b75 Comment 2017-04-07 11:54:03 +01:00
Erik Johnston
d4d176e5d0 Add logging 2017-04-07 11:51:28 +01:00
Erik Johnston
449d1297ca Fix up federation SendQueue and document types 2017-04-07 11:48:33 +01:00
Erik Johnston
d72667fcce Speed up get_current_state_ids 
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).

get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
 2017-04-07 10:10:49 +01:00
Erik Johnston
a41fe500d6 Bump version and changelog 2017-04-07 10:03:48 +01:00
Erik Johnston
54f59bd7d4 Merge pull request #2107 from HarHarLinks/patch-1 
fix typo in synctl help
 2017-04-07 09:54:37 +01:00
Erik Johnston
98ce212093 Merge pull request #2103 from matrix-org/erikj/no-double-encode 
Don't double encode replication data
 2017-04-07 09:39:52 +01:00
Kim Brose
8a1137ceab fix typo in synctl help 2017-04-06 17:10:20 +02:00
Erik Johnston
877c029c16 Use iteritems 2017-04-06 15:51:22 +01:00
Erik Johnston
944692ef69 Merge pull request #2106 from matrix-org/erikj/reduce_user_sync 
Reduce rate of USER_SYNC repl commands
 2017-04-06 13:35:31 +01:00
Erik Johnston
391712a4f9 Comment 2017-04-06 13:35:00 +01:00
Erik Johnston
ad544c803a Document types of the replication streams 2017-04-06 13:28:52 +01:00
Erik Johnston
dbf87282d3 Docs 2017-04-06 13:11:21 +01:00
Erik Johnston
69b3fd485d Fix incorrect type when using InvalidateCacheCommand 2017-04-06 09:36:38 +01:00
Daniel Dent
5058292537 Support authenticated SMTP 
Closes (SYN-714) #1385

Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
 2017-04-05 21:01:08 -07:00
Erik Johnston
fcc803b2bf Add log lines 2017-04-05 17:13:44 +01:00
Erik Johnston
3f213d908d Rearrange metrics 2017-04-05 14:15:09 +01:00
Erik Johnston
1ca0e78ca1 Fix typo 2017-04-05 13:43:39 +01:00
Erik Johnston
b43d3267e2 Fixup some metrics for tcp repl 2017-04-05 13:34:54 +01:00
Erik Johnston
b5cb6347a4 Don't immediately notify the master about users whose syncs have gone away 2017-04-05 13:25:40 +01:00
Erik Johnston
96b9b6c127 Don't double json encode typing replication data 2017-04-05 11:34:20 +01:00
Erik Johnston
f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston
a5c401bd12 Merge pull request #2097 from matrix-org/erikj/repl_tcp_client 
Move to using TCP replication
 2017-04-05 09:36:21 +01:00
Erik Johnston
d1d5362267 Add comment 2017-04-04 16:41:03 +01:00
Erik Johnston
9f26d3b75b Deduplicate new deviceinbox rows for replication 2017-04-04 16:21:21 +01:00
Erik Johnston
a76886726b Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix 
Advance replication streams even if nothing is listening
 2017-04-04 15:40:51 +01:00
Erik Johnston
ac66e11f2b Add the appropriate amount of preserve_fn 2017-04-04 15:22:54 +01:00
Erik Johnston
4264ceb31c Fiddle tcp replication logging 2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be Advance replication streams even if nothing is listening 
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
 2017-04-04 13:19:26 +01:00
Erik Johnston
d1605794ad Remove unused worker config option 2017-04-04 11:17:00 +01:00
Erik Johnston
3376f16012 Shuffle and comment synchrotron presence 2017-04-04 11:14:16 +01:00
Erik Johnston
6ce6bbedcb Move where we ack federation 2017-04-04 11:02:44 +01:00
Erik Johnston
62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Richard van der Hoff
773e64cc1a Merge pull request #2095 from matrix-org/rav/cull_log_preserves 
Cull spurious PreserveLoggingContexts
 2017-04-03 17:02:25 +01:00
Richard van der Hoff
2d05eb3cf5 Merge remote-tracking branch 'origin/release-v0.20.0' into develop 2017-04-03 16:23:23 +01:00
Richard van der Hoff
30bcbf775a Accept join events from all servers 
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.

(Fixes #1893).
 2017-04-03 15:58:07 +01:00
Richard van der Hoff
7eb9f34cc3 Remove spurious yield 
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
 2017-04-03 15:44:19 +01:00
Richard van der Hoff
0b08c48fc5 Remove more spurious PreserveLoggingContexts 
Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`;
there is no problem if the logcontext is set when calling it.
 2017-04-03 15:43:37 +01:00
Richard van der Hoff
65e1683680 Remove spurious PreserveLoggingContext 
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
 2017-04-03 15:42:38 +01:00
Richard van der Hoff
feb496056e preserve_fn some deferred-returning things 
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
 2017-04-03 15:41:17 +01:00
Richard van der Hoff
e2eebf1696 Fix fixme in preserve_fn 
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
 2017-04-03 15:38:02 +01:00
Erik Johnston
36c28bc467 Update all the workers and master to use TCP replication 2017-04-03 15:35:52 +01:00
Erik Johnston
3a1f3f8388 Change slave storage to use new replication interface 
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
 2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1 Add basic replication client handler and factory 2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b Always advance stream tokens 2017-04-03 15:22:56 +01:00
Richard van der Hoff
773e1c6d68 Remove spurious @preserve_fn decorators 
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
 2017-04-03 15:14:11 +01:00
Erik Johnston
0d1c85e643 Merge branch 'release-v0.20.0' of github.com:matrix-org/synapse into develop 2017-04-03 14:58:14 +01:00
Erik Johnston
1df7c28661 Use callbacks to notify tcp replication rather than deferreds 2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90 Add a timestamp to USER_SYNC command 
This timestamp is used to indicate when the user last sync'd
 2017-03-31 15:42:22 +01:00
Erik Johnston
8a240e4f9c Merge pull request #2078 from APwhitehat/assertuserfriendly 
add user friendly report of assertion error in synctl.py
 2017-03-31 14:41:49 +01:00
Erik Johnston
142b6b4abf Merge pull request #2011 from matrix-org/matthew/turn_allow_guests 
add setting (on by default) to support TURN for guests
 2017-03-31 14:37:09 +01:00
Erik Johnston
2a06b44be2 Merge pull request #1986 from matrix-org/matthew/enable_guest_3p 
enable guest access for the 3pl/3pid APIs
 2017-03-31 14:36:03 +01:00
Matthew Hodgson
9ee397b440 switch to allow_guest=True for authing 3Ps as per PR feedback 2017-03-31 13:54:26 +01:00
Erik Johnston
9d0170ac6c Fix up presence 2017-03-31 11:36:32 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed 
Speed up cached function access
 2017-03-31 10:10:56 +01:00
Erik Johnston
350333a09a Merge pull request #2076 from matrix-org/erikj/as_perf 
Make AS's faster
 2017-03-31 09:43:10 +01:00
Erik Johnston
4d17add8de Remove unused instance variable 2017-03-31 09:38:27 +01:00
Erik Johnston
27b1b4a2c9 Speed up copy_and_replace 2017-03-30 17:50:31 +01:00
Erik Johnston
5b5b171f3e Docs 2017-03-30 17:05:53 +01:00
Erik Johnston
b282fe7170 Revert log context change 2017-03-30 17:03:59 +01:00
Erik Johnston
9ff4e0e91b Bump version and changelog 2017-03-30 16:37:40 +01:00
Erik Johnston
0834d1a70c Merge pull request #2079 from matrix-org/erikj/push_regex_cache 
Cache glob to regex at a higher level for push
 2017-03-30 16:28:18 +01:00
Erik Johnston
63fcc42990 Remove user from process_presence when stops syncing 2017-03-30 14:26:08 +01:00
Erik Johnston
6194a64ae9 Doc new instance variables 2017-03-30 14:19:10 +01:00
Erik Johnston
014fee93b3 Manually calculate cache key as getcallargs is expensive 
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
 2017-03-30 14:14:46 +01:00
Erik Johnston
86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Erik Johnston
3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Erik Johnston
e9dd8370b0 Add functions to presence to support remote syncs 
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
 2017-03-30 13:25:14 +01:00
Erik Johnston
4d7fc7f977 Add server side resource for tcp replication 2017-03-30 13:24:45 +01:00
Richard van der Hoff
f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077) 
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
 2017-03-30 13:22:24 +01:00
Erik Johnston
7450693435 Initial TCP protocol implementation 
This defines the low level TCP replication protocol
 2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48 Define the various streams we will replicate 2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
7984708a55 Add a simple hook to wait for replication traffic 2017-03-30 11:57:52 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Anant Prakash
6cdca71079 synctl.py: wait for synapse to stop before restarting (#2020) 2017-03-29 19:20:13 +01:00
Anant Prakash
305d16d612 add user friendly report of assertion error in synctl.py 
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
 2017-03-29 20:41:39 +05:30
Erik Johnston
a3810136fe Cache glob to regex at a higher level for push 2017-03-29 15:53:14 +01:00
Erik Johnston
3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston
b9c2ae6788 Merge pull request #1849 from matrix-org/erikj/state_typo 
Fix bug where current_state_events renamed to current_state_ids
 2017-03-29 11:53:42 +01:00
Erik Johnston
85be3dde81 Bail early if remote wouldn't be retried (#2064) 
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
 2017-03-29 11:48:27 +01:00
Erik Johnston
2f8b580b64 Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert 
Don't user upsert to persist new one time keys
 2017-03-29 11:44:23 +01:00
Erik Johnston
c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed 
Notify on new federation traffic
 2017-03-29 11:41:37 +01:00
Erik Johnston
e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Erik Johnston
69efd77749 Add comment 2017-03-29 09:50:05 +01:00
Erik Johnston
51b156d48a Cache whether an AS is interested based on members 2017-03-28 13:27:21 +01:00
Erik Johnston
30f5ffdca2 Remove param and cast at call site 2017-03-28 13:27:21 +01:00
Erik Johnston
650f0e69f2 Compile the regex's used in ASes 2017-03-28 13:27:21 +01:00
Erik Johnston
d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch 
Batch sending of device list pokes
 2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication 
Use presence replication stream to invalidate cache
 2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
31d56c3fb5 Merge pull request #2061 from matrix-org/erikj/add_transaction_store 
Add slave transaction store to workers who send federation requests
 2017-03-24 13:24:27 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
23e0ff840a Merge pull request #2060 from matrix-org/erikj/cache_hosts_in_room 
Cache hosts in room
 2017-03-24 13:07:22 +00:00