Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
14,182 Commits 5 Branches 723 Tags 124 MiB
c5d60eadd5
Commit Graph

10033 Commits

Author SHA1 Message Date
Richard van der Hoff
8530090b16
Add config.signing_key_path. (#4974) 
As requested by @andrewshadura
 2019-04-02 16:59:27 +01:00
Erik Johnston
5bec8d660d Make starting pushers faster during start up 
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.

Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
 2019-04-02 16:59:13 +01:00
Richard van der Hoff
297bf2547e
Fix sync bug when accepting invites (#4956) 
Hopefully this time we really will fix #4422.

We need to make sure that the cache on
`get_rooms_for_user_with_stream_ordering` is invalidated *before* the
SyncHandler is notified for the new events, and we can now do so reliably via
the `events` stream.
 2019-04-02 12:42:39 +01:00
Erik Johnston
4ef5d17b96 Correctly handle id_server param 2019-04-02 11:20:09 +01:00
Erik Johnston
24232514bf Remove threepid binding if id server returns 400/404/501 2019-04-02 11:20:09 +01:00
Erik Johnston
c75e2017f1 Fixup docstrings 2019-04-02 11:20:06 +01:00
Erik Johnston
862d6e5ba5 Add unbind API to /r0 as it is now stabalised 2019-04-01 15:25:19 +01:00
Erik Johnston
3715c124b3 Grandfather in existing user threepids 
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
 2019-04-01 15:25:19 +01:00
Erik Johnston
057715aaa2 Allowing specifying IS to use in unbind API. 
By default the homeserver will use the identity server used during the
binding of the 3PID to unbind the 3PID. However, we need to allow
clients to explicitly ask the homeserver to unbind via a particular
identity server, for the case where the 3PID was bound out of band from
the homeserver.

Implements MSC915.
 2019-04-01 15:25:18 +01:00
Erik Johnston
9fbbc3d9e5 For unbind poke IS used during binding of 3PID 
This changes the behaviour from using the server specified trusted
identity server to using the IS that used during the binding of the
3PID, if known.

This is the behaviour specified by MSC1915.
 2019-04-01 15:23:30 +01:00
Erik Johnston
1666c0696a Track IS used to bind 3PIDs 
This will then be used to know which IS to default to when unbinding the
threepid.
 2019-04-01 15:23:01 +01:00
Neil Johnson
62988f73fd Merge branch 'master' into develop 2019-04-01 14:08:53 +01:00
Neil Johnson
35442efb75 0.99.3 2019-04-01 12:49:03 +00:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969) 
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
 2019-04-01 10:24:38 +01:00
Richard van der Hoff
902cdc63b6
Merge pull request #4955 from matrix-org/rav/merge_state_into_events 
Combine the CurrentStateDeltaStream into the EventStream
 2019-03-28 18:32:13 +00:00
Richard van der Hoff
d688a51736
Merge pull request #4954 from matrix-org/rav/refactor_parse_row 
Refactors to replication stream row update/parsing
 2019-03-28 18:31:17 +00:00
Neil Johnson
c7296bcb98
remove log line for password (#4965) 
Remove log line for password.
 2019-03-28 17:38:01 +00:00
Andrew Morgan
7a91b9d81c
Allow password providers to bind emails (#4947) 
This PR allows password provider modules to bind email addresses when a user is registering and is motivated by matrix-org/matrix-synapse-ldap3#58
 2019-03-28 15:48:07 +00:00
Erik Johnston
248014379e
Merge pull request #4942 from matrix-org/erikj/fix_presence 
Use event streams to calculate presence
 2019-03-28 14:38:31 +00:00
Erik Johnston
4e5f0f7ca0 Use an assert 2019-03-28 14:05:05 +00:00
Erik Johnston
40e56997bc Review comments 2019-03-28 13:48:41 +00:00
Richard van der Hoff
d035d62f6b Merge remote-tracking branch 'origin/develop' into rav/refactor_parse_row 2019-03-28 13:45:14 +00:00
Richard van der Hoff
4eeb2c2f07
Merge pull request #4953 from matrix-org/rav/refactor_replication_streams 
Split up replication.tcp.streams into smaller files
 2019-03-28 13:43:25 +00:00
Amber Brown
2e060774ad
Run black on some storage modules that the stats branch touches (#4959) 2019-03-29 00:37:16 +11:00
Richard van der Hoff
4b91c313a9 Combine the CurrentStateDeltaStream into the EventStream 2019-03-27 22:07:05 +00:00
Richard van der Hoff
1f6d6f918a Make EventStream rows have a type 
... as a precursor to combining it with the CurrentStateDelta stream.
 2019-03-27 22:07:05 +00:00
Richard van der Hoff
015b3622eb Skip building a ROW_TYPE when building updates 
We're about to turn it straight into a JSON object anyway so building a
ROW_TYPE is a bit pointless, and reduces flexibility in the update_function.
 2019-03-27 21:58:03 +00:00
Richard van der Hoff
f570916a3e Add parse_row method to replication stream class 
This will allow individual stream classes to override how a row is parsed.
 2019-03-27 21:32:33 +00:00
Richard van der Hoff
71dcb275f1 move FederationStream out to its own file 2019-03-27 21:13:14 +00:00
Richard van der Hoff
aa1e017864 move EventsStream out to its own file 2019-03-27 21:13:14 +00:00
Richard van der Hoff
a5798de067 Move replication.tcp.streams into a package 2019-03-27 21:13:14 +00:00
Richard van der Hoff
acaa18f7dd
Fix/improve some docstrings in the replication code. (#4949) 2019-03-27 21:12:36 +00:00
Erik Johnston
197fae1639 Use event streams to calculate presence 
Primarily this fixes a bug in the handling of remote users joining a
room where the server sent out the presence for all local users in the
room to all servers in the room.

We also change to using the state delta stream, rather than the
distributor, as it will make it easier to split processing out of the
master process (as well as being more flexible).

Finally, when sending presence states to newly joined servers we filter
out old presence states to reduce the number sent. Initially we filter
out states that are offline and have a last active more than a week ago,
though this can be changed down the line.

Fixes #3962
 2019-03-27 13:41:36 +00:00
Neil Johnson
4aa914369b bump version 2019-03-27 10:23:03 +00:00
Andrew Morgan
bbd244c7b2
Support 3PID login in password providers (#4931) 
Adds a new method, check_3pid_auth, which gives password providers
the chance to allow authentication with third-party identifiers such
as email or msisdn.
 2019-03-26 17:48:30 +00:00
Amber Brown
903f04c21f
Use the state event amount for userdir import batching, not room count (#4944) 2019-03-27 02:49:28 +11:00
Andrew Morgan
4a125be138
Make federation endpoints more tolerant of trailing slashes v2 (#4935) 
Redo of https://github.com/matrix-org/synapse/pull/4840
 2019-03-26 11:35:29 +00:00
Richard van der Hoff
8cbbedaa2b
Fix ClientReplicationStreamProtocol.__str__ (#4929) 
`__str__` depended on `self.addr`, which was absent from
ClientReplicationStreamProtocol, so attempting to call str on such an object
would raise an exception.

We can calculate the peer addr from the transport, so there is no need for addr
anyway.
 2019-03-25 16:41:51 +00:00
Richard van der Hoff
9bde730ef8
Fix bug where read-receipts lost their timestamps (#4927) 
Make sure that they are sent correctly over the replication stream.

Fixes: #4898
 2019-03-25 16:38:05 +00:00
Amber Brown
ac396a0d32
Refactor out state delta handling into its own class (#4917) 2019-03-25 20:37:08 +11:00
Erik Johnston
5fee9d8067
Merge pull request #4869 from matrix-org/erikj/yaml_load 
Fix yaml warnings by using safe_load
 2019-03-22 11:58:13 +00:00
Erik Johnston
3677548a82 Use yaml safe_load 2019-03-22 10:20:17 +00:00
Andrew Morgan
7bef97dfb7
Remove trailing slashes from outbound federation requests and retry on 400 (#4840) 
As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.
 2019-03-21 15:07:28 +00:00
Andrew Morgan
b41c2eaadc Clean up backoff_on_404 and metehod calls 2019-03-21 14:32:47 +00:00
Erik Johnston
01e6b405be
Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms 
Deny peeking into rooms that have been blocked
 2019-03-21 14:07:17 +00:00
Erik Johnston
3959858eaa
Merge pull request #4904 from matrix-org/erikj/fix_shutdown 
Fixup shutdown room API
 2019-03-21 11:24:42 +00:00
Erik Johnston
536a266520 Deny peeking into rooms that have been blocked 2019-03-21 11:20:13 +00:00
Erik Johnston
3ecec5ede2 Fix upsert 2019-03-21 10:21:15 +00:00
Erik Johnston
09f991a63d
Merge pull request #4896 from matrix-org/erikj/disable_room_directory 
Add option to disable search room lists
 2019-03-21 10:16:54 +00:00
Erik Johnston
cd62981a6a Revert spurious delete 2019-03-20 17:51:27 +00:00
Erik Johnston
8d8834d3e7 comment block_room 2019-03-20 17:49:56 +00:00
Erik Johnston
aa959a6c07 Use flags 2019-03-20 17:40:29 +00:00
Erik Johnston
7d47cc1305 Move requester check into assert_accepted_privacy_policy 2019-03-20 17:08:36 +00:00
Erik Johnston
72a14860ab Gracefully handle failing to kick user 2019-03-20 16:54:21 +00:00
Erik Johnston
6b28890543 Log new room ID 2019-03-20 16:52:28 +00:00
Erik Johnston
74c46d81fa Only require consent for events with an associated request 
There are a number of instances where a server or admin may puppet a
user to join/leave rooms, which we don't want to fail if the user has
not consented to the privacy policy. We fix this by adding a check to
test if the requester has an associated access_token, which is used as a
proxy to answer the question of whether the action is being done on
behalf of a real request from the user.
 2019-03-20 16:50:23 +00:00
Erik Johnston
67d618e111 Allow blocking a room multiple times 2019-03-20 16:50:05 +00:00
Erik Johnston
263f2c9ce1
Merge pull request #4895 from matrix-org/erikj/disable_user_search 
Add option to disable searching in the user dir
 2019-03-20 16:47:15 +00:00
Amber Brown
4d53017432
Batching in the user directory import (#4900) 2019-03-21 03:06:36 +11:00
Richard van der Hoff
cdb8036161
Add a config option for torture-testing worker replication. (#4902) 
Setting this to 50 or so makes a bunch of sytests fail in worker mode.
 2019-03-20 16:04:35 +00:00
Richard van der Hoff
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890) 
Rate-limit outgoing read-receipts as per #4730.
 2019-03-20 16:02:25 +00:00
Erik Johnston
cd8c5b91ad Fix up sample config 2019-03-20 14:35:41 +00:00
Richard van der Hoff
ab20f85c59
Update synapse/config/user_directory.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-03-20 14:33:11 +00:00
Erik Johnston
cc197a61a1 Disable publishing to room list when its disabled 2019-03-20 14:30:36 +00:00
Erik Johnston
2c90422146 Pull out config option 2019-03-20 14:25:58 +00:00
Erik Johnston
7529038e66 Return before we log 2019-03-20 14:25:28 +00:00
Erik Johnston
926f29ea6d Fix up config comments 2019-03-20 14:24:53 +00:00
Andrew Morgan
2150151abe kwargs doesn't like commas on calling funcs either. TIL 2019-03-20 14:13:32 +00:00
Andrew Morgan
bb52a2e653 lint 2019-03-20 14:08:57 +00:00
Andrew Morgan
cd36a1283b New test, fix issues 2019-03-20 14:00:39 +00:00
Andrew Morgan
c69df5d5d3 Fix comments. v0.99.2 -> v0.99.3 2019-03-20 11:27:18 +00:00
Andrew Morgan
551ea11559 Just return if not doing any trailing slash shennanigans 2019-03-20 11:07:36 +00:00
Erik Johnston
213c98c00a Add option to disable search room lists 
This disables both local and remote room list searching.
 2019-03-19 17:10:52 +00:00
Erik Johnston
320667a479 Add option to disable searching in the user dir 
We still populate it, as it can still be accessed via the admin API.
 2019-03-19 16:40:19 +00:00
Richard van der Hoff
13bc1e0746 Use a regular HomeServerConfig object for unit tests 
Rather than using a Mock for the homeserver config, use a genuine
HomeServerConfig object. This makes for a more realistic test, and means that
we don't have to keep remembering to add things to the mock config every time
we add a new config setting.
 2019-03-19 11:44:43 +00:00
Richard van der Hoff
45c4e19c74 Merge remote-tracking branch 'origin/develop' into HEAD 2019-03-19 11:43:46 +00:00
Neil Johnson
88f0675967 fix test_auto_create_auto_join_where_no_consent (#4886) 2019-03-19 11:38:59 +00:00
Richard van der Hoff
0dbfae03f9 Enforce hs_disabled_message correctly 
Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests
if there was no server_notices_mxid configured.
 2019-03-19 11:30:54 +00:00
Erik Johnston
e9eeca1314 Fix user directory background update (#4887) 2019-03-19 22:13:53 +11:00
Richard van der Hoff
fd463b4f5d
Comment out most options in the generated config. (#4863) 
Make it so that most options in the config are optional, and commented out in
the generated config.

The reasons this is a good thing are as follows:

* If we decide that we should change the default for an option, we can do so,
  and only those admins that have deliberately chosen to override that option
  will be stuck on the old setting.

* It moves us towards a point where we can get rid of the super-surprising
  feature of synapse where the default settings for the config come from the
  generated yaml.

* It makes setting up a test config for unit testing an order of magnitude
  easier (see forthcoming PR).

* It makes the generated config more consistent, and hopefully easier for users
  to understand.
 2019-03-19 10:06:40 +00:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Andrew Morgan
a8ad39eec7 lint 2019-03-18 17:47:39 +00:00
Andrew Morgan
621e7f37f1 Better exception handling 2019-03-18 17:45:54 +00:00
Brendan Abolivier
651ad8bc96 Add ratelimiting on failed login attempts (#4865) 2019-03-18 12:57:20 +00:00
Brendan Abolivier
899e523d6d
Add ratelimiting on login (#4821) 
Add two ratelimiters on login (per-IP address and per-userID).
 2019-03-15 17:46:16 +00:00
Richard van der Hoff
3b7ceb2c69
Merge pull request #4855 from matrix-org/rav/refactor_transaction_queue 
Split TransactionQueue up
 2019-03-15 12:32:11 +00:00
Richard van der Hoff
2dee441bdb
Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker 
Move client receipt processing to federation sender worker.
 2019-03-15 12:30:30 +00:00
Erik Johnston
b0fa3f6ff3
Merge pull request #4853 from matrix-org/erikj/worker_docker_ci 
Allow passing --daemonize to workers
 2019-03-15 10:35:38 +00:00
Erik Johnston
271cb1998b Revert "Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793)" 
This reverts commit 290552fd83.
 2019-03-14 14:30:54 +00:00
Erik Johnston
9073cfc8bd
Merge pull request #4846 from matrix-org/hawkowl/userdir-search 
Improve searching in the userdir
 2019-03-14 13:54:15 +00:00
Erik Johnston
9ad448c1e5 Correctly handle all command line options 2019-03-14 13:32:14 +00:00
Andrew Morgan
ecea5af491 Correct var name 2019-03-13 21:21:03 +00:00
Andrew Morgan
b2df0e8e2c receiving a 400 caused an exception. handle it 2019-03-13 21:08:10 +00:00
Andrew Morgan
9a2e22fd41 is this what purgatory feels like 2019-03-13 20:29:38 +00:00
Andrew Morgan
45524f2f5e i should have given up x2 2019-03-13 20:17:39 +00:00
Andrew Morgan
8d16ffaf7a i should have given up 2019-03-13 20:03:10 +00:00
Richard van der Hoff
02e23b36bc Rename and move the classes 2019-03-13 20:02:56 +00:00
Andrew Morgan
26f8e2d099 there comes a time when you should give up. but you dont 2019-03-13 19:49:08 +00:00
Andrew Morgan
5ca857ad84 as above 2019-03-13 19:35:23 +00:00
Andrew Morgan
7c0295f13c no kwargs today 2019-03-13 19:27:10 +00:00
Andrew Morgan
66cdb840a6 Or perhaps I was the one who was drunk 2019-03-13 19:18:25 +00:00
Andrew Morgan
bec313818c go home python, you're drunk 2019-03-13 19:10:56 +00:00
Andrew Morgan
c991e7aec7 Syntax checker is bork 2019-03-13 19:08:08 +00:00
Andrew Morgan
c2d848b80d Destructure again 2019-03-13 19:04:43 +00:00
Andrew Morgan
ee8ba397e8 Are you happy now 2019-03-13 18:48:31 +00:00
Andrew Morgan
9dd0e34679 Syntax test 2019-03-13 18:45:17 +00:00
Andrew Morgan
220607a618 Remove testing code 2019-03-13 18:43:40 +00:00
Andrew Morgan
660b77f362 Add missing docstring detail 2019-03-13 18:38:16 +00:00
Andrew Morgan
5526b054aa Fix syntax issues 2019-03-13 18:35:21 +00:00
Andrew Morgan
09626bfd39 Switch to wrapper function around _send_request 2019-03-13 18:26:06 +00:00
Richard van der Hoff
5d89a526f1 Factor per-destination stuff out of TransactionQueue 
This is easier than having to have a million fields keyed on destination.
 2019-03-13 17:48:29 +00:00
Erik Johnston
72bfaf746d Allow passing --daemonize to workers 2019-03-13 17:33:54 +00:00
Richard van der Hoff
fdcad8eabd Move client receipt processing to federation sender worker. 
This is mostly a prerequisite for #4730, but also fits with the general theme
of "move everything off the master that we possibly can".
 2019-03-13 17:21:19 +00:00
Richard van der Hoff
eed7271b3b declare a ReadReceipt class 
I'm going to use this in queues and things, so it'll be useful to give it more
of a structure.
 2019-03-13 17:20:55 +00:00
Andrew Morgan
7998ca3a66
Document using a certificate with a full chain (#4849) 2019-03-13 15:26:29 +00:00
Andrew Morgan
7d053cfe10 Retry on 400:M_UNRECOGNIZED 2019-03-13 12:10:33 +00:00
Andrew Morgan
7e75d9644b Fix paranthesis indent 2019-03-13 11:15:23 +00:00
Andrew Morgan
cf301e37d8 Add workaround note 2019-03-13 11:14:43 +00:00
Andrew Morgan
d42c81d724
Transfer local user's push rules on room upgrade (#4838) 
Transfer push rules (notifications) on room upgrade
 2019-03-12 14:42:53 +00:00
Amber Brown
c0332d095f fixup 2019-03-13 01:30:54 +11:00
Amber Brown
797b6a63fc fixup 2019-03-13 01:17:51 +11:00
Andrew Morgan
0ea8582f8b Cleaner way of implementing trailing slashes 2019-03-12 14:11:11 +00:00
Amber Brown
81d9d1bee6 fixup 2019-03-12 22:28:48 +11:00
Amber Brown
6f5890b2fa fixup 2019-03-12 22:27:56 +11:00
Amber Brown
10480c4348 fixup 2019-03-12 21:47:14 +11:00
Amber Brown
8b618041ef fixup 2019-03-12 18:06:28 +11:00
Amber Brown
c980c7e31f use the old method 2019-03-12 17:51:14 +11:00
Aaron Raimist
8ea1b41a0e Clarify what registration_shared_secret allows for (#2885) (#4844) 
* Clarify what registration_shared_secret allows for (#2885)

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-03-11 18:21:52 +00:00
Andrew Morgan
f18dca26da Merge branch 'develop' into anoa/trailing_slashes_client 2019-03-11 17:44:29 +00:00
Andrew Morgan
290552fd83
Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793) 
Server side of a solution towards #3622.
 2019-03-11 17:44:03 +00:00
Andrew Morgan
a8a028dfce Merge branch 'develop' into anoa/trailing_slashes_client 2019-03-11 17:07:31 +00:00
Erik Johnston
78c563b77c Correctly log expected errors when fetching server keys 2019-03-11 14:11:10 +00:00
Amber Brown
78a6b950b3 fix 2019-03-12 00:50:28 +11:00
Amber Brown
1b77bd69fb pep8 2019-03-12 00:39:12 +11:00
Amber Brown
5ba8ceab4c fixes 2019-03-12 00:35:31 +11:00
Amber Brown
8da22e2b53 master startup 2019-03-11 21:13:35 +11:00
Amber Brown
30a8deeb68 Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search 2019-03-11 21:12:30 +11:00
Amber Brown
4abc988c6a
initial 2019-03-11 21:11:36 +11:00
Matthew Hodgson
2326e00bc4 fix incorrect encoding of filenames with spaces in (#2090) 
fixes https://github.com/vector-im/riot-web/issues/3155
 2019-03-11 09:53:45 +00:00
Andrew Morgan
a5dd335cd8 lint 2019-03-08 18:25:59 +00:00
Andrew Morgan
64ff11019e Retry certain federation requests on 404 2019-03-08 18:22:47 +00:00
Andrew Morgan
525dd02bbe Remove trailing slashes from outbound federation requests 2019-03-08 16:55:52 +00:00
Erik Johnston
fe6c12e6cd Add comment to schema 2019-03-08 16:38:23 +00:00
Erik Johnston
5536ddba75 Make prev_state field optional 
The `prev_state` field on events is not specced and so synapse shouldn't
explode if an event is missing the field.

Fixes #4787
 2019-03-08 15:05:32 +00:00
Erik Johnston
8c4896668f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl 2019-03-08 11:44:20 +00:00
Erik Johnston
0ff8163eae Factor out soft fail checks 2019-03-08 11:26:33 +00:00
Erik Johnston
469b66c102
Merge pull request #4829 from matrix-org/erikj/device_list_seen_updates 
When re-syncing device lists reset the state
 2019-03-08 09:54:54 +00:00
Erik Johnston
d42b41544a When re-syncing device lists reset the state 
We keep track of what stream IDs we've seen so that we know what updates
we've handled or missed. If we re-sync we don't know if the updates
we've seen are included in the re-sync (there may be a race), so we
should reset the seen updates.
 2019-03-07 16:04:24 +00:00
Erik Johnston
c633fc02d7 Add some debug logging for device list handling 2019-03-07 15:58:03 +00:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Travis Ralston
06cbf79fe3
Merge pull request #4779 from matrix-org/travis/icons 
Use static locations for Riot icons
 2019-03-06 15:11:41 -07:00
Erik Johnston
1f85c2c0ce
Merge pull request #4818 from matrix-org/erikj/prefill_client_ips 
Prefill client IPs cache on workers
 2019-03-06 19:17:39 +00:00
Erik Johnston
0a6e716600
Merge pull request #4815 from matrix-org/erikj/docstrings 
Add docstrings from matrix-org-hotfixes
 2019-03-06 18:59:28 +00:00
Erik Johnston
c665b637de
Merge pull request #4816 from matrix-org/erikj/4422_debug 
Port #4422 debug logging from hotfixes
 2019-03-06 18:59:12 +00:00
Erik Johnston
face0c5b3c Prefill client IPs cache on workers 2019-03-06 17:39:32 +00:00
Erik Johnston
b879870b2d Send message after room has been shutdown 
Currently the explanation message is sent to the abuse room before any
users are forced joined, which means it tends to get lost in the backlog
of joins.

So instead we send the message *after* we've forced joined everyone.
 2019-03-06 17:35:11 +00:00
Erik Johnston
8b7790e68f Port #4422 debug logging from hotfixes 2019-03-06 17:29:15 +00:00
Erik Johnston
6d13bdec91 Add docstrings from matrix-org-hotfixes 2019-03-06 17:24:26 +00:00
Erik Johnston
a9de04be72 Implement soft fail 2019-03-06 16:22:16 +00:00
Andrew Morgan
7b8a157b79
Merge pull request #4792 from matrix-org/anoa/replication_tokens 
Support batch updates in the worker sender
 2019-03-06 15:48:29 +00:00
Brendan Abolivier
6f3cde8b25 Make registration ratelimiter separate from the main events one 2019-03-06 11:02:42 +00:00
Brendan Abolivier
f4195f4118 Revert "Split ratelimiters in two (one for events, one for registration)" 
This reverts commit d7dbad3526.
 2019-03-06 10:55:22 +00:00
Brendan Abolivier
d7dbad3526 Split ratelimiters in two (one for events, one for registration) 2019-03-05 18:41:27 +00:00
Brendan Abolivier
067ce795c0 Move settings from registration to ratelimiting in config file 2019-03-05 18:03:14 +00:00
Erik Johnston
16c8b4ecbd
Merge pull request #4772 from jbweston/jbweston/server-version-api 
Add 'server_version' endpoint to admin API
 2019-03-05 16:31:00 +00:00
Brendan Abolivier
a4c3a361b7
Add rate-limiting on registration (#4735) 
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
 2019-03-05 14:25:33 +00:00
Andrew Morgan
b9f6163092 Simplify token replication logic 2019-03-05 13:58:30 +00:00
Andrew Morgan
ae90531036
Merge pull request #4794 from matrix-org/anoa/erroneous_dollahs 
Remove unnecessary dollar signs
 2019-03-05 09:46:59 +00:00
Erik Johnston
b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room 
Stop backpaginating when events not visible
 2019-03-05 09:32:33 +00:00
Erik Johnston
9e8bca5667
Merge pull request #4799 from matrix-org/rav/clean_up_replication_code 
Clean ups in replication notifier
 2019-03-05 09:19:48 +00:00
Erik Johnston
aa06d26ae0 clarify comments 2019-03-05 09:16:35 +00:00
Erik Johnston
c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys 
Allow /keys/{changes,query} API to run on worker
 2019-03-05 09:06:25 +00:00
Richard van der Hoff
c7325776a7 Remove redundant PreserveLoggingContext 
Both (!) things that register as replication listeners do the right thing wrt
logcontexts, so this is redundant.
 2019-03-04 18:31:18 +00:00
Erik Johnston
bfa7d46a10 Allow /keys/{changes,query} API to run on worker 2019-03-04 18:30:01 +00:00
Erik Johnston
157e5a8f27 Split DeviceHandler into master and worker 2019-03-04 18:29:26 +00:00
Richard van der Hoff
daa10e3e66 Remove unused wait_for_replication method 
I guess this was used once? It's not now, anyway.
 2019-03-04 18:27:32 +00:00
Richard van der Hoff
2db49ea476 Add some debug about processing read receipts. 
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
 2019-03-04 18:19:40 +00:00
Richard van der Hoff
b29693a30b Clean up read-receipt handling. 
Remove a call to run_as_background_process: there is no need to run this as a
background process, because build_and_send_edu does not block.

We may as well inline the whole of _push_remotes.
 2019-03-04 18:16:43 +00:00
Erik Johnston
a84b8d56c2 Fixup slave stores 2019-03-04 18:04:57 +00:00
Richard van der Hoff
8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Erik Johnston
0d2d046709 Fix missing null guard 2019-03-04 16:04:04 +00:00
Andrew Morgan
336de1d45b Remove unnecessary dollar signs 
A dollar sign is already appended to the end of each PATH, so there's
no need to add one in the PATH declaration as well.
 2019-03-04 15:25:12 +00:00
Andrew Morgan
fe7bd23a85 Clean up logic and add comments 2019-03-04 15:08:15 +00:00
Erik Johnston
d1523aed6b Only check history visibility when filtering 
When filtering events to send to server we check more than just history
visibility. However when deciding whether to backfill or not we only
care about the history visibility.
 2019-03-04 14:43:42 +00:00
Andrew Morgan
9f7cdf3da1 Clearer branching, fix missing list clear 2019-03-04 14:36:52 +00:00
Andrew Morgan
5f0c449dd5 Prevent replication wedging 2019-03-04 14:03:18 +00:00
Richard van der Hoff
856c83f5f8
Avoid rebuilding Edu objects in worker mode (#4770) 
In worker mode, on the federation sender, when we receive an edu for sending
over the replication socket, it is parsed into an Edu object. There is no point
extracting the contents of it so that we can then immediately build another Edu.
 2019-03-04 12:57:44 +00:00
Erik Johnston
8b63fe4c26 s/get_forward_events/get_successor_events/ 2019-03-04 11:56:03 +00:00
Erik Johnston
fbc047f2a5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room 2019-03-04 11:54:58 +00:00
Travis Ralston
4dfbae18fe Use static locations for Riot icons 
See https://github.com/vector-im/riot-web/issues/9009
 2019-03-01 15:06:21 -07:00
Richard van der Hoff
3064952939
Fix incorrect log about not persisting duplicate state event. (#4776) 
We were logging this when it was not true.
 2019-03-01 16:47:12 +00:00
Richard van der Hoff
1beebe916f Merge branch 'master' into develop 2019-03-01 10:58:39 +00:00
Richard van der Hoff
9ac72d9543 0.99.2 2019-03-01 10:55:44 +00:00
Joseph Weston
1e8388b311 Add 'server_version' endpoint to admin API 
This is required because the 'Server' HTTP header is not always
passed through proxies.
 2019-03-01 09:56:58 +01:00
Andrew Morgan
ac61b45a75 Minor docstring fixes for MatrixFederationAgent (#4765) 2019-02-28 16:24:01 +00:00
Amber Brown
b131cc77df
Make 'event_id' a required parameter in federated state requests (#4741) 
* make 'event_id' a required parameter in federated state requests

As per the spec: https://matrix.org/docs/spec/server_server/r0.1.1.html#id40

Signed-off-by: Joseph Weston <joseph@weston.cloud>

* add changelog entry for bugfix

Signed-off-by: Joseph Weston <joseph@weston.cloud>

* Update server.py
 2019-02-27 14:35:47 -08:00
Richard van der Hoff
68f47d6744 Fix parsing of Content-Disposition headers (#4763) 
* Fix parsing of Content-Disposition headers

TIL: filenames in content-dispostion headers can contain semicolons, and aren't
%-encoded.

* fix python2 incompatibility

* Fix docstrings
 2019-02-27 14:29:10 -08:00
Amber Brown
f2a753ea38
Move from TravisCI to BuildKite (#4752) 2019-02-27 13:03:14 -08:00
Erik Johnston
76550c58d2
Merge pull request #4759 from matrix-org/erikj/3pid_client_reader 
Move /account/3pid to client_reader
 2019-02-27 16:11:21 +00:00
Erik Johnston
8267034a63
Merge pull request #4758 from matrix-org/erikj/use_presence_replication 
When presence is disabled don't send over replication
 2019-02-27 15:46:26 +00:00
Erik Johnston
54f9ce11a7 Move /account/3pid to client_reader 2019-02-27 14:26:08 +00:00
Erik Johnston
1e315017d3 When presence is enabled don't send over replication 2019-02-27 13:53:46 +00:00
Erik Johnston
4cff9376f7 Move server key queries to federation reader 2019-02-27 13:43:53 +00:00
Erik Johnston
b183fef9ac Update comments 2019-02-27 13:06:10 +00:00
Erik Johnston
7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff 
Fix tightloop over connecting to replication server
 2019-02-27 11:00:59 +00:00
Richard van der Hoff
44a4d65586 0.99.2rc1 2019-02-27 10:48:34 +00:00
Erik Johnston
6bb1c028f1 Limit cache invalidation replication line length (#4748) 2019-02-27 10:28:37 +00:00
Erik Johnston
6870fc496f Move connecting logic into ClientReplicationStreamProtocol 2019-02-27 10:23:51 +00:00
Richard van der Hoff
f191be822b
Add database version to phonehome stats. (#4753) 2019-02-27 10:21:49 +00:00
Erik Johnston
25814921f1 Increase the max delay between retry attempts 
Otherwise if you have many workers they can easily take out master with
their connection attempts
 2019-02-26 15:12:33 +00:00
Erik Johnston
313987187e Fix tightloop over connecting to replication server 
If the client failed to process incoming commands during the initial set
up of the replication connection it would immediately disconnect and
reconnect, resulting in a tightloop.

This can happen, for example, when subscribing to a stream that has a
row that is too long in the backlog.

The fix here is to not consider the connection successfully set up until
the client has succesfully subscribed and caught up with the streams.
This ensures that the retry logic timers aren't reset until then,
meaning that if an error does happen during start up the client will
continue backing off before retrying again.
 2019-02-26 15:05:41 +00:00
Andrew Morgan
802884d4ee Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate_develop 2019-02-26 14:23:40 +00:00
Andrew Morgan
6fcb25202f Put function def back to the way it was 2019-02-26 14:17:38 +00:00
Andrew Morgan
7a4632af9c Prevent showing non-fed rooms in fed /publicRooms 2019-02-26 13:37:24 +00:00
Andrew Morgan
c74624a633
Revert "Prevent showing non-fed rooms in fed /publicRooms" 2019-02-26 13:20:38 +00:00
Andrew Morgan
a1a6473293
Merge pull request #4736 from matrix-org/anoa/public_rooms_federate 
Config option to prevent showing non-fed rooms in fed /publicRooms
 2019-02-26 13:07:15 +00:00
Andrew Morgan
c4414768af Cleaner chunk logic 2019-02-26 12:22:34 +00:00
Andrew Morgan
a712aa3a9c Correct indent 2019-02-26 12:13:55 +00:00
Andrew Morgan
16565e67db Correct docstring types and chunk logic 2019-02-26 12:12:48 +00:00
Andrew Morgan
40c2271680 Clean up room chunk logic 2019-02-26 12:04:34 +00:00
Andrew Morgan
6728bf3940 Make not showing non-federated rooms the default 2019-02-26 11:52:52 +00:00
Andrew Morgan
6946c20111 Result may be None 2019-02-26 11:27:19 +00:00
Andrew Morgan
71669a0fba Address rich comments 2019-02-26 11:25:00 +00:00
Erik Johnston
899a119c2b Don't log stack trace when client has gone away during media download (#4738) 
* Don't log stack trace when client has gone away during media download

* Newsfile

* Fixup newsfile
 2019-02-25 11:17:22 -08:00
Richard van der Hoff
641c409e4e Fix ACME config for python 2. (#4717) 
Fixes #4675.
 2019-02-25 11:16:33 -08:00
Matthew Hodgson
70ea2f4e1d switch from google.com to recaptcha.net for reCAPTCHA (#4731) 
* add trivial clarification about jemalloc

* switch from google.com to recaptcha.net

because https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally
 2019-02-25 11:15:36 -08:00
Andrew Morgan
96c408273e Fix group's call to generate_room_entry 2019-02-25 18:00:17 +00:00
Andrew Morgan
1330aa4a8f elif not else if 2019-02-25 17:28:19 +00:00
Erik Johnston
65f3fbfbf7
Merge pull request #4737 from matrix-org/erikj/failure_log_tb 
Log tracebacks correctly
 2019-02-25 17:26:30 +00:00
Erik Johnston
c7b333c545 Log tracebacks correctly 2019-02-25 16:56:41 +00:00
Erik Johnston
69efe6fb16
Merge pull request #4718 from matrix-org/erikj/fix_backfill_state_shred 
Fix backfill storing incorrect state for events
 2019-02-25 16:48:52 +00:00
Erik Johnston
9c598dddcb Fix typo 2019-02-25 16:32:02 +00:00
Erik Johnston
b1a90da82e
Merge pull request #4721 from matrix-org/erikj/msc_1866 
MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API
 2019-02-25 15:53:10 +00:00
Erik Johnston
16c7afa94c
Merge pull request #4722 from matrix-org/erikj/correctly_handle_keyring_exceptions 
Handle errors when fetching remote server keys
 2019-02-25 15:53:02 +00:00
Andrew Morgan
8aaf7ffc44 syntax derp 2019-02-25 15:27:17 +00:00
Andrew Morgan
84c0a20dfe Simplify call to generate_room_entry 2019-02-25 15:23:27 +00:00
Erik Johnston
4b9e5076c4 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate 2019-02-25 15:08:18 +00:00
Andrew Morgan
07493607a8 Docs and arg name clarification 2019-02-25 15:04:46 +00:00
Andrew Morgan
bd398b874e Don't restrict non-fed rooms over client APIs 2019-02-25 15:04:46 +00:00
Andrew Morgan
e4b078a600 Config option to prevent showing non-fed rooms in fed /publicRooms 2019-02-25 15:04:46 +00:00
Erik Johnston
d730c2c22b More comments 2019-02-25 14:45:02 +00:00
Erik Johnston
890cb048fd Assert rather than clobber the values 2019-02-25 14:42:11 +00:00
Erik Johnston
65d1003d01 raise_from already raises 2019-02-25 14:34:03 +00:00
Erik Johnston
9342cc6ab1 Add comments and paranoia 2019-02-25 10:02:12 +00:00
Erik Johnston
47a7e3928d Correctly proxy exception in frontend_proxy worker 2019-02-23 15:17:57 +00:00
Erik Johnston
41285ffe5b Handle errors when fetching remote server keys 2019-02-23 15:09:39 +00:00
Erik Johnston
59e0112209 MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API 2019-02-23 14:50:43 +00:00
Erik Johnston
d14e94bae4
Merge pull request #4716 from matrix-org/erikj/pusher_logging 
Fix up pusher logging a bit
 2019-02-22 15:29:16 +00:00
Erik Johnston
b82c9cf462 Add missing return 2019-02-22 15:27:40 +00:00
Erik Johnston
f2891d2487 Correctly handle PusherConfigException 2019-02-22 15:18:19 +00:00
Erik Johnston
a164134a53 Drop logging level of creating a pusher 2019-02-22 14:48:06 +00:00
Erik Johnston
1d9df51ff1 Correctly handle null data in HttpPusher 2019-02-22 14:47:48 +00:00
Erik Johnston
80467bbac3 Fix state cache invalidation on workers 2019-02-22 14:38:14 +00:00
Erik Johnston
7b288826b7 Fix backfill storing incorrect state for events 2019-02-22 11:33:51 +00:00
Richard van der Hoff
e07384c4e1
Add prometheus metrics for number of badge update pushes. (#4709) 
We're counting the number of push notifications, but not the number of badges;
I'd like to see if they are significant.
 2019-02-22 10:57:15 +00:00
Richard van der Hoff
0abb094f1a
bail out early in on_new_receipts if no pushers (#4706) 2019-02-21 17:51:21 +00:00
Richard van der Hoff
6d65659b62
Run push_receipts_to_remotes as background job (#4707) 
I suspect the CPU usage metrics for this are going to /dev/null at the moment.
 2019-02-21 17:50:30 +00:00
Erik Johnston
b9d6756b14
Merge pull request #4263 from rkfg/develop 
Prevent crash on pagination.
 2019-02-21 17:42:15 +00:00
Erik Johnston
71b625d808 Stop backpaginating when events not visible 2019-02-20 18:14:12 +00:00
Erik Johnston
82fca11fc1
Merge pull request #4694 from matrix-org/erikj/fix_sentry_config_format 
Fixup generated metrics config
 2019-02-20 14:13:38 +00:00
Richard van der Hoff
82ca6d1f9f
Add metrics for number of outgoing EDUs, by type (#4695) 2019-02-20 14:13:14 +00:00
Erik Johnston
3d672fec51 Fixup generated metrics config 2019-02-20 13:39:37 +00:00
Amber Brown
a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Richard van der Hoff
b2200a8690
Logging improvements for the pusher (#4691) 2019-02-20 11:36:50 +00:00
Richard van der Hoff
c88bc53903
Fix TaskStopped exceptions when outbound requests time out (#4690) 2019-02-20 11:35:52 +00:00
Richard van der Hoff
86920ac266
Merge pull request #4678 from matrix-org/rav/tls_install_instructions 
Attempt to clarify installation/config instructions
 2019-02-20 11:10:33 +00:00
Erik Johnston
dbdc565dfd Fix registration on workers (#4682) 
* Move RegistrationHandler init to HomeServer

* Move post registration actions to RegistrationHandler

* Add post regisration replication endpoint

* Newsfile
 2019-02-20 18:47:31 +11:00
Richard van der Hoff
5f9bdf90fe Attempt to make default config more consistent 
The general idea here is that config examples should just have a hash and no
extraneous whitespace, both to make it easier for people who don't understand
yaml, and to make the examples stand out from the comments.
 2019-02-19 13:54:29 +00:00
Erik Johnston
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation 
Batch cache invalidation over replication
 2019-02-19 13:14:30 +00:00
Erik Johnston
62175a20e5 Docs 2019-02-19 11:38:40 +00:00
Erik Johnston
1bb35e3a83 Use itertools 2019-02-19 11:34:40 +00:00
Erik Johnston
bc8fa1509d Documentation 2019-02-19 11:24:59 +00:00
Brendan Abolivier
a288bdf0b1
Merge pull request #4652 from matrix-org/babolivier/acme-delegated 
Support .well-known delegation when issuing certificates through ACME
 2019-02-19 11:15:38 +00:00
Brendan Abolivier
5a707a2f9a Improve config documentation 2019-02-19 10:59:26 +00:00
Brendan Abolivier
a8626901cd Fetch ACME domain into an instance member 2019-02-19 10:54:33 +00:00
Erik Johnston
32590b7139
Merge pull request #4670 from matrix-org/erikj/register_login_split 
Split /login into client_reader
 2019-02-19 10:46:22 +00:00
Richard van der Hoff
7c70b8f8a6
Try and make TLS federation client code faster (#4674) 
* fix to use makeContext so that we don't need to rebuild the certificateoptions each time
 2019-02-19 10:19:16 +00:00
Andrew Morgan
968a30a75c
Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade 
Transfer bans on room upgrade
 2019-02-19 10:16:42 +00:00
Amber Brown
561eebe170 fix to use makeContext so that we don't need to rebuild the certificateoptions each time 2019-02-19 16:18:05 +11:00
Andrew Morgan
34ac75ce2c lint 2019-02-18 18:23:37 +00:00
Erik Johnston
a9b5ea6fc1 Batch cache invalidation over replication 
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.

Hopefully this will reduce load on the master process by substantially
reducing traffic.
 2019-02-18 17:53:31 +00:00
Andrew Morgan
f8b9ca53ce Move member event processing and changelog fix 2019-02-18 17:28:52 +00:00
Erik Johnston
d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry 
Add basic optional sentry.io integration
 2019-02-18 17:22:45 +00:00
Erik Johnston
4cc4400b4d Split /login into client_reader 2019-02-18 17:19:01 +00:00
Erik Johnston
fc2c245a1f
Merge pull request #4666 from matrix-org/erikj/register_login_split 
Split out registration to worker
 2019-02-18 17:18:06 +00:00
Erik Johnston
459d3d5046
Merge pull request #4668 from matrix-org/erikj/catch_exceptions 
Correctly handle HttpResponseException when handling device updates
 2019-02-18 16:55:43 +00:00
Erik Johnston
d328a93b51 Fixup error handling and message 2019-02-18 16:53:56 +00:00
Erik Johnston
af691e415c Move register_device into handler 2019-02-18 16:49:38 +00:00
Erik Johnston
028267acd2
Merge pull request #4669 from matrix-org/erikj/log_exception 
Cleanup top level request exception logging
 2019-02-18 16:41:41 +00:00
Erik Johnston
d08bac4136
Merge pull request #4651 from matrix-org/matthew/well-known-cors 
set CORS on .well-known URI to unbreak modular
 2019-02-18 16:38:11 +00:00
Erik Johnston
c30f73c86a
Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers 
Fix kicking guest users in worker mode
 2019-02-18 15:55:48 +00:00
Erik Johnston
092b541401 Fixup 2019-02-18 15:52:26 +00:00
Brendan Abolivier
45bb55c6de Use a configuration parameter to give the domain to generate a certificate for 2019-02-18 15:46:23 +00:00
Erik Johnston
8b9ae6d3a6 Update docs 2019-02-18 15:26:13 +00:00
Erik Johnston
94960cef03 pep8 2019-02-18 15:24:13 +00:00
Erik Johnston
fe725f7e45 Cleanup top level request exception logging 
Firstly, we always logged that the request was being handled via
`JsonResource._async_render`, so we change that to use the servlet name
we add to the request.

Secondly, we pass the exception information to the logger rather than
formatting it manually. This makes it consistent with other exception
logging, allwoing logging hooks and formatters to access the exception
information.
 2019-02-18 15:11:04 +00:00
Matthew Hodgson
04dad5ac16 fix missig import 2019-02-18 15:59:23 +01:00
Erik Johnston
e07cc31cb8 Correctly handle HttpResponseException 2019-02-18 14:55:09 +00:00
Brendan Abolivier
68a53f825f Merge branch 'develop' into babolivier/acme-delegated 2019-02-18 14:52:23 +00:00
Erik Johnston
32e54b472a Fix kicking guest users in worker mode 
When guest_access changes from allowed to forbidden all local guest
users should be kicked from the room. This did not happen when
revocation was received from federation on a worker.

Presumably broken in #4141
 2019-02-18 14:16:07 +00:00