Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-12-13 23:54:19 -05:00
Code Issues Packages Projects Releases Wiki Activity
15,841 Commits 4 Branches 615 Tags 63 MiB
332f3b36e5
Commit Graph

11136 Commits

Author SHA1 Message Date
Amber Brown
2889b05554
Unify v1 and v2 REST client APIs (#5226) 2019-06-03 21:28:59 +10:00
Erik Johnston
220a733d73 Fix handling of failures when calling /event_auth. 
When processing an incoming event over federation, we may try and
resolve any unexpected differences in auth events. This is a
non-essential process and so should not stop the processing of the event
if it fails (e.g. due to the remote disappearing or not implementing the
necessary endpoints).

Fixes #3330
 2019-06-03 09:56:45 +01:00
Alexander Trost
dc3e586938
SAML2 Improvements and redirect stuff 
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
 2019-06-02 18:14:40 +02:00
Richard van der Hoff
d828d1dc57
Merge pull request #5309 from matrix-org/rav/limit_displayname_length 
Limit displaynames and avatar URLs
 2019-06-01 11:34:50 +01:00
Richard van der Hoff
d16c6375fe Limit displaynames and avatar URLs 
These end up in join events everywhere, so let's limit them.

Fixes #5079
 2019-06-01 10:44:36 +01:00
Richard van der Hoff
37b165620d
Merge pull request #5299 from matrix-org/rav/server_keys/05-rewrite-gsvk-again 
Rewrite get_server_verify_keys, again.
 2019-05-31 17:07:31 +01:00
Richard van der Hoff
3600f5568b Stop overwriting server keys with other keys 
Fix a bug where we would discard a key result which the origin server is no
longer returning. Fixes #5305.
 2019-05-31 15:58:35 +01:00
Erik Johnston
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta 
Allow configuring a range for the account validity startup job
 2019-05-31 12:11:56 +01:00
Richard van der Hoff
c605da97bf Merge remote-tracking branch 'origin/develop' into rav/server_keys/05-rewrite-gsvk-again 2019-05-31 11:38:13 +01:00
Richard van der Hoff
fe79b5e521
Merge pull request #5300 from matrix-org/rav/server_keys/06-fix-serverkeys-handling 
Remove some pointless exception handling
 2019-05-31 11:35:29 +01:00
Richard van der Hoff
2ae3cc287e
Merge pull request #5296 from matrix-org/rav/server_keys/04-use-attrs-for_verify-request 
use attr.s for VerifyKeyRequest
 2019-05-31 11:34:09 +01:00
Brendan Abolivier
4d794dae21 Move delta from +10% to -10% 2019-05-31 11:09:39 +01:00
Erik Johnston
e9981d58ca
Merge pull request #5293 from Kagamihime/messages-federation-format 
Fix ignored filter field in `/messages` endpoint
 2019-05-31 10:52:59 +01:00
Erik Johnston
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats 
Speed up room stats background update
 2019-05-31 10:48:51 +01:00
Erik Johnston
5037326d66 Add indices. Remove room_ids accidentally added 
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.

We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
 2019-05-31 10:26:56 +01:00
Brendan Abolivier
0c2362861e Gah python 2019-05-31 09:56:52 +01:00
Brendan Abolivier
847b9dcd1c Make max_delta equal to period * 10% 2019-05-31 09:54:46 +01:00
Richard van der Hoff
8ea2f756a9 Remove some pointless exception handling 
The verify_request deferred already returns a suitable SynapseError, so I don't
really know what we expect to achieve by doing more wrapping, other than log
spam.

Fixes #4278.
 2019-05-30 18:29:56 +01:00
Richard van der Hoff
a82c96b87f Rewrite get_server_verify_keys, again. 
Attempt to simplify the logic in get_server_verify_keys by splitting it into
two methods.
 2019-05-30 18:20:40 +01:00
Richard van der Hoff
099829d5a9 use attr.s for VerifyKeyRequest 
because namedtuple is awful
 2019-05-30 17:39:28 +01:00
Erik Johnston
99113e40ba Merge branch 'master' into develop 2019-05-30 16:39:49 +01:00
Erik Johnston
c831748f4d 0.99.5.2 2019-05-30 16:29:47 +01:00
Erik Johnston
e2c3660a0f Add index to temp table 2019-05-30 16:18:40 +01:00
Erik Johnston
06eb408da5 Update synapse/storage/events_bg_updates.py 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-05-30 16:15:37 +01:00
Erik Johnston
7386c35f58 Rename constant 2019-05-30 16:15:37 +01:00
Erik Johnston
98f438b52a Move event background updates to a separate file 2019-05-30 16:15:37 +01:00
Erik Johnston
9b8cd66524 Fixup comments and logging 2019-05-30 16:15:37 +01:00
Erik Johnston
1d818fde14 Log actual number of entries deleted 2019-05-30 16:15:37 +01:00
Erik Johnston
6ebc08c09d Add DB bg update to cleanup extremities. 
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
 2019-05-30 16:15:37 +01:00
Erik Johnston
df9d900544 Correctly filter out extremities with soft failed prevs (#5274) 
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
 2019-05-30 16:12:50 +01:00
Eisha Chen-yen-su
8824325b82 Fix ignored filter field in /messages endpoint 
This fixes a bug which were causing the "event_format" field to be
ignored in the filter of requests to the `/messages` endpoint of the
CS API.

Signed-off-by: Eisha Chen-yen-su <chenyensu0@gmail.com>
 2019-05-30 16:58:53 +02:00
Erik Johnston
5ac75fc9a2 Join against events to use its room_id index 2019-05-30 15:26:55 +01:00
Erik Johnston
e2c46ed851 Move deletion from table inside txn 2019-05-30 15:26:38 +01:00
Erik Johnston
04710cc2d7 Fetch membership counts all at once 2019-05-30 15:25:41 +01:00
Erik Johnston
54d50fbfdf Get events all at once 2019-05-30 15:15:13 +01:00
Erik Johnston
6cdfb0207e Add index to temp table 2019-05-30 15:02:31 +01:00
Erik Johnston
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities 
Add DB bg update to cleanup extremities.
 2019-05-30 14:28:26 +01:00
Erik Johnston
cb967e2346
Update synapse/storage/events_bg_updates.py 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-05-30 14:06:42 +01:00
Erik Johnston
45f5d8f3fd
Merge pull request #5256 from aaronraimist/logout-correct-error 
Show correct error when logging out and access token is missing
 2019-05-30 13:33:44 +01:00
Erik Johnston
468bd090ff Rename constant 2019-05-30 11:24:42 +01:00
Erik Johnston
5c1ece0ffc Move event background updates to a separate file 2019-05-30 11:22:59 +01:00
Erik Johnston
640fcbb07f Fixup comments and logging 2019-05-30 10:55:55 +01:00
Aaron Raimist
123918b739
Lint 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-29 14:44:28 -05:00
Richard van der Hoff
8d92329214
Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) 
This is just unhelpful spam
 2019-05-29 19:31:52 +01:00
Richard van der Hoff
3dcf2feba8
Improve logging for logcontext leaks. (#5288) 2019-05-29 19:27:50 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Amber Brown
532b825ed9
Serve CAS login over r0 (#5286) 2019-05-30 00:55:18 +10:00
Erik Johnston
7e8e683754 Log actual number of entries deleted 2019-05-29 15:11:28 +01:00
Erik Johnston
d79c9994f4 Add DB bg update to cleanup extremities. 
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
 2019-05-29 15:11:26 +01:00
Aaron Raimist
30858ff461 Fix error when downloading thumbnail with width/height param missing (#5258) 
Fix error when downloading thumbnail with width/height param missing

Fixes #2748

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-29 14:27:41 +01:00
Erik Johnston
58c8ed5b0d
Correctly filter out extremities with soft failed prevs (#5274) 
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
 2019-05-29 11:56:24 +01:00
Richard van der Hoff
f76d407ef3
Fix dropped logcontexts during high outbound traffic. (#5277) 
Fixes #5271.
 2019-05-29 09:17:33 +01:00
Aaron Raimist
9b6f72663e
Fix docs on resetting the user directory (#5036) 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-28 20:54:01 -05:00
Richard van der Hoff
540f40f0cd
Merge pull request #5251 from matrix-org/rav/server_keys/01-check_sig 
Ensure that server_keys fetched via a notary server are correctly signed.
 2019-05-28 21:32:17 +01:00
Richard van der Hoff
5726378ece
Fix "db txn 'update_presence' from sentinel context" log messages (#5275) 
Fixes #4414.
 2019-05-28 21:20:11 +01:00
Brendan Abolivier
52839886d6
Allow configuring a range for the account validity startup job 
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
 2019-05-28 16:52:45 +01:00
Brendan Abolivier
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema 
Fix schema update for account validity
 2019-05-28 10:30:07 +01:00
Brendan Abolivier
ba17de7fbc
Fix schema update for account validity 2019-05-28 10:11:38 +01:00
Aaron Raimist
119c9c10b0
Get rid of try except 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-27 00:13:56 -05:00
Erik Johnston
d0bba35197
Merge pull request #5260 from matrix-org/travis/fix-room-bg-task 
Fix logging for room stats background update
 2019-05-25 19:59:42 +01:00
Travis Ralston
bc4b2ecf70 Fix logging for room stats background update 2019-05-25 12:02:48 -06:00
Erik Johnston
338dca58c0
Merge pull request #5257 from aaronraimist/fix-error-code-publicrooms 
Fix error code for invalid parameter
 2019-05-25 14:09:27 +01:00
Aaron Raimist
2d4853039f
Fix error code for invalid parameter 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-24 17:13:16 -05:00
Aaron Raimist
56f07d980a
Show correct error when logging out and access token is missing 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-24 16:32:27 -05:00
Richard van der Hoff
fa1b293da2
Simplification to Keyring.wait_for_previous_lookups. (#5250) 
The list of server names was redundant, since it was equivalent to the keys on
the server_to_deferred map. This reduces the number of large lists being passed
around, and has the benefit of deduplicating the entries in `wait_on`.
 2019-05-24 22:17:18 +01:00
Richard van der Hoff
b825d1c800 Improve error handling/logging for perspectives-key fetching. 
In particular, don't give up on the first failure.
 2019-05-24 15:46:25 +01:00
Tulir Asokan
dd64b9dbdd Fix appservice timestamp massaging (#5233) 
Signed-off-by: Tulir Asokan <tulir@maunium.net>
 2019-05-24 14:44:04 +01:00
Richard van der Hoff
dba9152d15
Add missing blank line in config (#5249) 2019-05-24 14:12:38 +01:00
Erik Johnston
d16f5574b6
Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events 
Don't bundle aggregations with events in /sync or /events or state queries
 2019-05-24 10:36:31 +01:00
Erik Johnston
4cb577c23f Don't bundle aggs for /state and /members etc APIs 2019-05-24 09:52:33 +01:00
Erik Johnston
8c41c04ee4
Merge pull request #5244 from matrix-org/rav/server_keys/00-factor-out-fetchers 
Factor out KeyFetchers from KeyRing
 2019-05-23 16:28:51 +01:00
Richard van der Hoff
753b1270da Require sig from origin server on perspectives responses 2019-05-23 15:01:09 +01:00
Andrew Morgan
6368150a74
Add config option for setting homeserver's default room version (#5223) 
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
 2019-05-23 15:00:20 +01:00
Richard van der Hoff
ec24108cc2 Fix remote_key_resource 2019-05-23 14:52:13 +01:00
Richard van der Hoff
895b79ac2e Factor out KeyFetchers from KeyRing 
Rather than have three methods which have to have the same interface,
factor out a separate interface which is provided by three implementations.

I find it easier to grok the code this way.
 2019-05-23 13:46:47 +01:00
Richard van der Hoff
b75537beaf Store key validity time in the storage layer 
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
 2019-05-23 11:52:22 +01:00
Richard van der Hoff
84660d91b2
Simplify process_v2_response (#5236) 
* Pass time_added_ms into process_v2_response

* Simplify process_v2_response

We can merge old_verify_keys into verify_keys, and reduce the number of dicts
flying around.
 2019-05-23 11:51:39 +01:00
Richard van der Hoff
cc187f9337
Remove unused VerifyKey.expired and .time_added fields (#5235) 
These were never used, and poking arbitary data into objects from other
packages seems confusing at best.
 2019-05-23 11:46:05 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234) 
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
 2019-05-23 11:45:39 +01:00
Richard van der Hoff
85d1e03b9d
Simplifications and comments in do_auth (#5227) 
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
 2019-05-23 11:17:42 +01:00
Richard van der Hoff
1a94de60e8
Run black on synapse.crypto.keyring (#5232) 2019-05-22 18:39:33 +01:00
Neil Johnson
73f1de31d1 Merge branch 'master' into develop 2019-05-22 17:59:43 +01:00
Neil Johnson
3d5bba581b 0.99.5.1 2019-05-22 17:52:44 +01:00
Neil Johnson
006bd8f4f6 Revert "0.99.5" 
This reverts commit c31e375ade.
 2019-05-22 17:49:53 +01:00
Neil Johnson
c31e375ade 0.99.5 2019-05-22 17:45:44 +01:00
Marcus Hoffmann
62388a1e44 remove urllib3 pin (#5230) 
requests 2.22.0 as been released supporting urllib3 1.25.2

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
 2019-05-22 16:48:12 +01:00
Neil Johnson
ae5521be9c Merge branch 'master' into develop 2019-05-22 15:56:55 +01:00
Neil Johnson
8031a6f3d5 0.99.5 2019-05-22 15:40:28 +01:00
Neil Johnson
66b75e2d81
Neilj/ensure get profileinfo available in client reader slaved store (#5213) 
* expose SlavedProfileStore to ClientReaderSlavedStore
 2019-05-22 13:55:32 +01:00
Richard van der Hoff
b898a5600a Merge branch 'master' into develop 2019-05-22 11:38:27 +01:00
Amber Brown
4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Richard van der Hoff
f3ff64e000 Merge commit 'f4c80d70f' into release-v0.99.5 2019-05-21 17:35:31 +01:00
Erik Johnston
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender 
Only count aggregations from distinct senders
 2019-05-21 17:10:48 +01:00
Erik Johnston
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions 
Block attempts to annotate the same event twice
 2019-05-21 17:08:14 +01:00
Richard van der Hoff
959550b645 0.99.5rc1 2019-05-21 16:51:49 +01:00
Erik Johnston
44b8ba484e Fix words 2019-05-21 16:51:45 +01:00
Richard van der Hoff
17f6804837
Introduce room v4 which updates event ID format. (#5217) 
Implements https://github.com/matrix-org/matrix-doc/pull/2002.
 2019-05-21 16:22:54 +01:00
Richard van der Hoff
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146) 
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
 2019-05-21 16:10:54 +01:00
Richard van der Hoff
bab3eddac4
Pin eliot to <1.8 on python 3.5.2 (#5218) 
* Pin eliot to <1.8 on python 3.5.2

Fixes https://github.com/matrix-org/synapse/issues/5199

* Add support for 'markers' to python_dependencies

* tell xargs not to strip quotes
 2019-05-21 15:58:01 +01:00
Erik Johnston
7b0e804a4a Fix get_max_topological_token to never return None 2019-05-21 15:21:43 +01:00
Brendan Abolivier
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date 
Add startup background job for account validity
 2019-05-21 14:55:15 +01:00
Brendan Abolivier
384122efa8
Doc 2019-05-21 14:39:36 +01:00
Erik Johnston
de7672b78f Don't bundle events in /sync or /events 
As we'll send down the annotations too anyway, so this just ends up
confusing clients.
 2019-05-21 13:54:09 +01:00
Richard van der Hoff
04d53794d6
Fix error handling for rooms whose versions are unknown. (#5219) 
If we remove support for a particular room version, we should behave more
gracefully. This should make client requests fail with a 400 rather than a 500,
and will ignore individiual PDUs in a federation transaction, rather than the
whole transaction.
 2019-05-21 13:47:25 +01:00
Brendan Abolivier
5ceee46c6b
Do the select and insert in a single transaction 2019-05-21 13:38:51 +01:00
Erik Johnston
c7ec06e8a6 Block attempts to annotate the same event twice 2019-05-20 17:39:05 +01:00
Richard van der Hoff
24b93b9c76 Revert "expose SlavedProfileStore to ClientReaderSlavedStore (#5200)" 
This reverts commit ce5bcefc60.

This caused:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/client_reader.py", line 32, in <module>
    from synapse.replication.slave.storage import SlavedProfileStore
ImportError: cannot import name 'SlavedProfileStore' from 'synapse.replication.slave.storage' (/home/synapse/src/synapse/replication/slave/storage/__init__.py)
error starting synapse.app.client_reader('/home/synapse/config/workers/client_reader.yaml') (exit code: 1); see above for logs
```
 2019-05-20 16:21:34 +01:00
Richard van der Hoff
5206648a4a
Add a test room version which updates event ID format (#5210) 
Implements MSC1884
 2019-05-20 15:54:42 +01:00
Erik Johnston
1dff859d6a Rename relation types to match MSC 2019-05-20 14:31:19 +01:00
Erik Johnston
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base 
Land basic reaction and edit support.
 2019-05-20 14:06:40 +01:00
Erik Johnston
935af0da38 Correctly update aggregation counts after redaction 2019-05-20 12:09:27 +01:00
ReidAnderson
3787133c9e Limit UserIds to a length that fits in a state key (#5198) 2019-05-20 11:20:08 +01:00
Brendan Abolivier
ad5b4074e1
Add startup background job for account validity 
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
 2019-05-17 19:37:31 +01:00
Erik Johnston
b63cc325a9 Only count aggregations from distinct senders 
As a user isn't allowed to send a single emoji more than once.
 2019-05-17 18:03:10 +01:00
bytepoets-blo
291e1eea5e fix mapping of return values for get_or_register_3pid_guest (#5177) 
* fix mapping of return values for get_or_register_3pid_guest
 2019-05-17 17:27:14 +01:00
Erik Johnston
85ece3df46
Merge pull request #5191 from matrix-org/erikj/refactor_pagination_bounds 
Make generating SQL bounds for pagination generic
 2019-05-17 17:24:36 +01:00
Erik Johnston
8dd9cca8ea Spelling and clarifications 2019-05-17 16:40:51 +01:00
Erik Johnston
5dbff34509 Fixup bsaed on review comments 2019-05-17 15:48:04 +01:00
Neil Johnson
ce5bcefc60
expose SlavedProfileStore to ClientReaderSlavedStore (#5200) 
* expose SlavedProfileStore to ClientReaderSlavedStore
 2019-05-17 13:27:19 +01:00
Brendan Abolivier
fdeac1e984
Merge pull request #5196 from matrix-org/babolivier/per_room_profiles 
Add an option to disable per-room profiles
 2019-05-17 12:10:49 +01:00
PauRE
f89f688a55 Fix image orientation when generating thumbnail (#5039) 2019-05-16 19:04:26 +01:00
David Baker
07cff7b121
Merge pull request #5174 from matrix-org/dbkr/add_dummy_flow_to_recaptcha_only 
Re-order registration stages to do msisdn & email auth last
 2019-05-16 17:27:39 +01:00
Erik Johnston
d46aab3fa8 Add basic editing support 2019-05-16 16:54:45 +01:00
Erik Johnston
895179a4dc Update docstring 2019-05-16 16:41:05 +01:00
Brendan Abolivier
cd32375846 Add option to disable per-room profiles 2019-05-16 14:34:28 +01:00
Erik Johnston
7a7eba8302 Move parsing of tokens out of storage layer 2019-05-16 14:26:23 +01:00
Erik Johnston
2c662ddde4 Indirect tuple conversion 2019-05-16 14:21:39 +01:00
Erik Johnston
95f3fcda3c Check that event is visible in new APIs 2019-05-16 14:19:06 +01:00
Matthew Hodgson
4a6d5de98c Make /sync attempt to return device updates for both joined and invited users (#3484) 2019-05-16 13:23:43 +01:00
David Baker
fafb936de5
Merge pull request #5187 from matrix-org/dbkr/only_check_threepid_not_in_use_if_actually_registering 
Only check 3pids not in use when registering
 2019-05-16 10:58:09 +01:00
Erik Johnston
b5c62c6b26 Fix relations in worker mode 2019-05-16 10:38:13 +01:00
Erik Johnston
33453419b0 Add cache to relations 2019-05-16 10:02:14 +01:00
Erik Johnston
a0603523d2 Add aggregations API 2019-05-16 09:37:20 +01:00
Amber Brown
f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Richard van der Hoff
5f027a315f
Drop support for v2_alpha API prefix (#5190) 2019-05-15 17:37:46 +01:00
Erik Johnston
5be34fc3e3 Actually check for None rather falsey 2019-05-15 17:30:23 +01:00
Erik Johnston
e6459c26b4 Actually implement idempotency 2019-05-15 17:28:33 +01:00
Richard van der Hoff
1757e2d7c3 Merge branch 'master' into develop 2019-05-15 14:09:30 +01:00
Richard van der Hoff
4a926f528e 0.99.4 2019-05-15 13:58:45 +01:00
Erik Johnston
b50641e357 Add simple pagination API 2019-05-15 13:36:51 +01:00
Erik Johnston
efe3c7977a Add simple send_relation API and track in DB 2019-05-15 13:36:51 +01:00
Erik Johnston
54d77107c1 Make generating SQL bounds for pagination generic 
This will allow us to reuse the same structure when we paginate e.g.
relations
 2019-05-15 11:30:05 +01:00
Erik Johnston
0aba6c8251
Merge pull request #5183 from matrix-org/erikj/async_serialize_event 
Allow client event serialization to be async
 2019-05-15 10:36:30 +01:00
Erik Johnston
d94544051b
Merge pull request #5184 from matrix-org/erikj/expose_get_events_as_array 
Expose DataStore._get_events as get_events_as_list
 2019-05-15 10:17:38 +01:00
Erik Johnston
8ed2f182f7
Update docstring with correct return type 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-05-15 09:52:52 +01:00
Erik Johnston
52ddc6c0ed
Update docstring with correct type 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-05-15 09:52:15 +01:00
David Baker
6ca88c4693 Only check 3pids not in use when registering 
We checked that 3pids were not already in use before we checked if
we were going to return the account previously registered in the
same UI auth session, in which case the 3pids will definitely
be in use.

https://github.com/vector-im/riot-web/issues/9586
 2019-05-14 19:04:59 +01:00
Richard van der Hoff
daa2fb6317 comment about user_joined_room 2019-05-14 18:53:09 +01:00
Erik Johnston
4fb44fb5b9 Expose DataStore._get_events as get_events_as_list 
This is in preparation for reaction work which requires it.
 2019-05-14 13:37:44 +01:00
Erik Johnston
a80e6b53f9 Newsfile 2019-05-14 13:12:23 +01:00
Erik Johnston
b54b03f9e1 Allow client event serialization to be async 2019-05-14 11:58:01 +01:00
Amber Brown
df2ebd75d3
Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171) 2019-05-13 15:01:14 -05:00
Andrew Morgan
5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
Brendan Abolivier
2e1129b5f7 0.99.4rc1 2019-05-13 16:11:21 +01:00
David Baker
516a5fb64b Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:54:25 +01:00
David Baker
9e99143c47 Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:37:03 +01:00
Brendan Abolivier
bb93757b32 Fix CI after new release of isort 2019-05-13 15:19:44 +01:00
Andrew Morgan
2f48c4e1ae
URL preview blacklisting fixes (#5155) 
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
 2019-05-10 10:32:44 -07:00
David Baker
04299132af Re-order flows so that email auth is done last 
It's more natural for the user if the bit that takes them away
from the registration flow comes last. Adding the dummy stage allows
us to do the stages in this order without the ambiguity.
 2019-05-10 13:58:03 +01:00
David Baker
9c61dce3c8 Comment 2019-05-10 11:14:55 +01:00
David Baker
8714ff6d51 Add a DUMMY stage to captcha-only registration flow 
This allows the client to complete the email last which is more
natual for the user. Without this stage, if the client would
complete the recaptcha (and terms, if enabled) stages and then the
registration request would complete because you've now completed a
flow, even if you were intending to complete the flow that's the
same except has email auth at the end.

Adding a dummy auth stage to the recaptcha-only flow means it's
always unambiguous which flow the client was trying to complete.
Longer term we should think about changing the protocol so the
client explicitly says which flow it's trying to complete.

vector-im/riot-web#9586
 2019-05-10 11:09:53 +01:00
David Baker
c2bb7476c9 Revert 085ae346ac 
Accidentally went straight to develop
 2019-05-10 11:08:01 +01:00
David Baker
085ae346ac Add a DUMMY stage to captcha-only registration flow 
This allows the client to complete the email last which is more
natual for the user. Without this stage, if the client would
complete the recaptcha (and terms, if enabled) stages and then the
registration request would complete because you've now completed a
flow, even if you were intending to complete the flow that's the
same except has email auth at the end.

Adding a dummy auth stage to the recaptcha-only flow means it's
always unambiguous which flow the client was trying to complete.
Longer term we should think about changing the protocol so the
client explicitly says which flow it's trying to complete.

https://github.com/vector-im/riot-web/issues/9586
 2019-05-10 10:52:24 +01:00
Richard van der Hoff
130f932cbc Run black on per_destination_queue 
... mostly to fix pep8 fails
 2019-05-09 16:27:02 +01:00
Quentin Dufour
11ea16777f Limit the number of EDUs in transactions to 100 as expected by receiver (#5138) 
Fixes #3951.
 2019-05-09 11:01:41 +01:00
Matthew Hodgson
c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083) 
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
 2019-05-08 18:26:56 +01:00
Erik Johnston
c8c069db92
Merge pull request #5037 from matrix-org/erikj/limit_inflight_dns 
Limit in flight DNS requests
 2019-05-08 17:11:03 +01:00
Brendan Abolivier
1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128) 
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
 2019-05-08 17:01:30 +01:00
Erik Johnston
de655e669a
Merge pull request #5104 from matrix-org/erikj/ratelimit_3pid_invite 
Ratelimit 3pid invites
 2019-05-07 10:12:49 +01:00
Richard van der Hoff
59e2d2694d
Remove the requirement to authenticate for /admin/server_version. (#5122) 
This endpoint isn't much use for its intended purpose if you first need to get
yourself an admin's auth token.

I've restricted it to the `/_synapse/admin` path to make it a bit easier to
lock down for those concerned about exposing this information. I don't imagine
anyone is using it in anger currently.
 2019-05-07 09:29:30 +01:00
Richard van der Hoff
836d3adcce Merge branch 'master' into develop 2019-05-03 19:25:01 +01:00
Richard van der Hoff
fa21455e08 0.99.3.2 2019-05-03 18:56:24 +01:00
Richard van der Hoff
e3281d7d26 pin urllib3 to <1.25 2019-05-03 18:33:10 +01:00
Richard van der Hoff
863ec09622 0.99.3.1 2019-05-03 16:03:24 +01:00
Richard van der Hoff
a845abbf3a
Merge pull request #5134 from matrix-org/rav/url_preview_blacklist 
Blacklist 0.0.0.0 and :: by default for URL previews
 2019-05-03 15:59:20 +01:00
Richard van der Hoff
1565ebec2c more config comment updates 2019-05-03 15:50:59 +01:00
Richard van der Hoff
1acfb9e9f0
Merge pull request #5133 from matrix-org/rav/systemrandom 
Use SystemRandom for token generation.
 2019-05-03 15:39:30 +01:00
Richard van der Hoff
1a7104fde3 Blacklist 0.0.0.0 and :: by default for URL previews 2019-05-03 15:35:49 +01:00
Richard van der Hoff
247dc1bd0b Use SystemRandom for token generation 2019-05-03 13:02:55 +01:00
Erik Johnston
176f31c2e3 Rate limit early 2019-05-02 15:23:08 +01:00
Richard van der Hoff
12f9d51e82
Add admin api for sending server_notices (#5121) 2019-05-02 11:59:16 +01:00
Brendan Abolivier
c193b39134
Merge pull request #5124 from matrix-org/babolivier/aliases 
Add some limitations to alias creation
 2019-05-02 11:22:40 +01:00
Brendan Abolivier
84196cb231 Add some limitations to alias creation 2019-05-02 11:05:11 +01:00
Richard van der Hoff
0836cbb9f5
Factor out an "assert_requester_is_admin" function (#5120) 
Rather than copying-and-pasting the same four lines hundreds of times
 2019-05-02 10:45:52 +01:00
Richard van der Hoff
f203c98794 fix examples 2019-05-01 17:49:56 +01:00
Richard van der Hoff
40e576e29c Move admin api impl to its own package 
It doesn't really belong under rest/client/v1 any more.
 2019-05-01 15:44:30 +01:00
Richard van der Hoff
8e9ca83537 Move admin API to a new prefix 2019-05-01 15:44:30 +01:00
Richard van der Hoff
579b637b6c Move admin API away from ClientV1RestServlet 2019-05-01 15:16:04 +01:00
Travis Ralston
8c5b1e30d4
Add a default .m.rule.tombstone push rule (#4867) 
* Add a default .m.rule.tombstone push rule

In support of MSC1930: https://github.com/matrix-org/matrix-doc/pull/1930

* changelog

* Appease the changelog linter
 2019-04-29 15:40:31 -06:00
Richard van der Hoff
b31cc1c613
Merge pull request #5100 from matrix-org/rav/verification_hackery 
Improve logging when event-signature checking fails
 2019-04-29 13:19:32 +01:00
Erik Johnston
d6118c5be6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ratelimit_3pid_invite 2019-04-26 18:14:23 +01:00
Erik Johnston
28a81ed62f Ratelimit 3pid invites 
We do ratelimit sending the 3PID invite events, but that happens after
spamming the identity server.
 2019-04-26 18:06:25 +01:00
Richard van der Hoff
bd0d45ca69 Fix infinite loop in presence handler 
Fixes #5102
 2019-04-26 11:14:49 +01:00
Richard van der Hoff
837d7f85a9 more logging improvements 2019-04-25 22:17:59 +01:00
Richard van der Hoff
fd8fb32bdd remove extraneous exception logging 2019-04-25 22:02:03 +01:00
Richard van der Hoff
7ca638c761 Clarify logging when PDU signature checking fails 2019-04-25 20:55:12 +01:00
Andrew Morgan
6824ddd93d Config option for verifying federation certificates (MSC 1711) (#4967) 2019-04-25 14:22:49 +01:00
Michael Kaye
788163e204 Remove log error for .well-known/matrix/client (#4972) 2019-04-24 17:44:06 +01:00
Katie Wolfe
7e07dc429f
Lint 
I probably should've just run autopep8 in the first place...

Signed-off-by: Katie Wolfe <katie@dnaf.moe>
 2019-04-24 12:43:18 -04:00
Amber Brown
6b2b9a58c4 Prevent "producer not unregistered" message (#5009) 2019-04-24 17:37:32 +01:00
Katie Wolfe
b3e5db402d
Clean up code 
Signed-off-by: Katie Wolfe <katie@dnaf.moe>
 2019-04-24 12:26:55 -04:00
Katie Wolfe
5d3ed79944
Show heroes if room name or canonical alias are empty 
Fixes #4194

Signed-off-by: Katie Wolfe <katie@dnaf.moe>
 2019-04-24 12:26:53 -04:00
Katie Wolfe
7f025eb425
Show heroes if room name or canonical alias are empty 
Fixes #4194

Signed-off-by: Katie Wolfe <katie@dnaf.moe>
 2019-04-24 12:26:29 -04:00
Brendan Abolivier
f8826d31cd Don't crash on lack of expiry templates 2019-04-18 14:50:05 +01:00
Erik Johnston
ca90336a69 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration 2019-04-17 19:44:40 +01:00
Brendan Abolivier
eaf41a943b Add management endpoints for account validity 2019-04-17 19:34:45 +01:00
Brendan Abolivier
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration 
Send out emails with links to extend an account's validity period
 2019-04-17 14:57:39 +01:00
Brendan Abolivier
20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Brendan Abolivier
49ff74da9b
Merge pull request #5071 from matrix-org/babolivier/3pid-check 
Make sure we're not registering the same 3pid twice
 2019-04-17 14:37:42 +01:00
Brendan Abolivier
600ec04739 Make sure we're not registering the same 3pid twice 2019-04-17 14:23:01 +01:00
Erik Johnston
fd2fcb817c
Merge pull request #5070 from matrix-org/erikj/postpath 
Remove usage of request.postpath
 2019-04-17 09:03:26 +01:00
Erik Johnston
a1eb4c6d2f
Merge pull request #5065 from matrix-org/erikj/fix_versions 
VersionRestServlet doesn't take a param
 2019-04-16 17:52:36 +01:00
Erik Johnston
ad010f6306 Remove usage of request.postpath 
This is an undocumented variable in twisted, and relies on the servlet
being mounted in the right way.

This also breaks getting push rules on workers.
 2019-04-16 17:42:50 +01:00
Silke Hofstra
a137f4eac0 Add systemd-python to optional dependencies (#4339) 
Using systemd-python allows for logging to the systemd journal,
as is documented in: `synapse/contrib/systemd/log_config.yaml`.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
 2019-04-16 20:41:17 +10:00
Erik Johnston
38642614cf VersionRestServlet doesn't take a param 2019-04-15 19:39:47 +01:00
Erik Johnston
6e27a8620f
Merge pull request #5063 from matrix-org/erikj/move_endpoints 
Move some rest endpoints to client reader
 2019-04-15 18:55:01 +01:00
Erik Johnston
ec638a1602 Only handle GET requests for /push_rules 2019-04-15 18:51:48 +01:00
Erik Johnston
d5adf297e6 Move some rest endpoints to client reader 2019-04-15 17:21:03 +01:00
Andrew Morgan
caa76e6021
Remove periods from copyright headers (#5046) 2019-04-11 17:08:13 +01:00
Erik Johnston
c132c8e505 Handle the case of get_missing_events failing 
Currently if a call to `/get_missing_events` fails we log an exception
and stop processing the top level event we received over federation.
Instead let's try and handle it sensibly given it is a somewhat expected
failure mode.
 2019-04-10 10:39:54 +01:00
Erik Johnston
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta 
Fix schema upgrade when dropping tables
 2019-04-10 10:22:35 +01:00
Richard van der Hoff
329688c161
Fix disappearing exceptions in manhole. (#5035) 
Avoid sending syntax errors from the manhole to sentry.
 2019-04-10 07:23:48 +01:00
Erik Johnston
a0fc256d65 Limit in flight DNS requests 
This is to work around a bug in twisted where a large number of
concurrent DNS requests cause it to tight loop forever.

c.f. https://twistedmatrix.com/trac/ticket/9620#ticket
 2019-04-09 17:23:42 +01:00
Brendan Abolivier
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration 
Add time-based account expiration
 2019-04-09 17:02:41 +01:00
Brendan Abolivier
747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Erik Johnston
50d2a3059d Fix schema upgrade when dropping tables 
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.

Introduced in #4992.
 2019-04-09 14:39:18 +01:00
Richard van der Hoff
644b86677f
Merge pull request #5030 from matrix-org/rav/rewrite_g_s_v_k 
Rewrite Datastore.get_server_verify_keys
 2019-04-09 11:30:23 +01:00
Richard van der Hoff
4abf5aa81a
Bump psycopg requirement (#5032) 2019-04-09 11:29:50 +01:00
Richard van der Hoff
18b69be00f Rewrite Datastore.get_server_verify_keys 
Rewrite this so that it doesn't hammer the database.
 2019-04-09 00:00:10 +01:00
Richard van der Hoff
f50efcb65d Replace SlavedKeyStore with a shim 
since we're pulling everything out of KeyStore anyway, we may as well simplify
it.
 2019-04-08 23:59:07 +01:00
Richard van der Hoff
f88a9e6323 Remove redundant merged_keys dict 
There's no point in collecting a merged dict of keys: it is sufficient to
consider just the new keys which have been fetched by the most recent
key_fetch_fns.
 2019-04-08 22:36:18 +01:00
Richard van der Hoff
3352baac4b
Remove unused server_tls_certificates functions (#5028) 
These have been unused since #4120, and with the demise of perspectives, it is
unlikely that they will ever be used again.
 2019-04-08 21:50:18 +01:00
Neil Johnson
b25e387c0d
add context to phonehome stats (#5020) 
add context to phonehome stats
 2019-04-08 15:47:39 +01:00
Richard van der Hoff
67d7b44784
Merge pull request #5024 from matrix-org/rav/record_correct_server_in_serverkeys 
Fix from_server buglet in get_keys_from_perspectives
 2019-04-08 15:40:37 +01:00
Neil Johnson
2d951686a7
drop tables listed in #1830 (#4992) 
Tables dropped: 
 * application_services, 
 * application_services_regex, 
 * transaction_id_to_pdu, 
 * stats_reporting
 * current_state_resets
 * event_content_hashes
 * event_destinations
 * event_edge_hashes
 * event_signatures
 * feedback
 * room_hosts
 * state_forward_extremities
 2019-04-08 15:37:26 +01:00
Richard van der Hoff
7d2a0c848e Fix from_server buglet in get_keys_from_perspectives 
make sure we store the name of the server the keys came from, rather than the
origin server, after doing a fetch-from-perspectives.
 2019-04-08 12:51:16 +01:00
Richard van der Hoff
7fc1e17f4c
Merge pull request #5001 from matrix-org/rav/keyring_cleanups 
Cleanups in the Keyring
 2019-04-08 12:47:09 +01:00
Richard van der Hoff
6ae9361510 Hoist server_name check out of process_v2_response 
It's easier to check it in the caller than to complicate the interface with an
extra param.
 2019-04-04 19:12:54 +01:00
Richard van der Hoff
ef27d434d1 Clean up Keyring.process_v2_response 
Make this just return the key dict, rather than a single-entry dict mapping the
server name to the key dict. It's easy for the caller to get the server name
from from the response object anyway.
 2019-04-04 19:12:54 +01:00
Richard van der Hoff
b43d9a920b Fix docstring on get_server_keys_json 2019-04-04 18:54:03 +01:00
Brendan Abolivier
8e85493b0c
Add config option to block users from looking up 3PIDs (#5010) 2019-04-04 17:25:47 +01:00
Amber Brown
a33a5abc4c
Clean up the database pagination code (#5007) 
* rewrite & simplify

* changelog

* cleanup potential sql injection
 2019-04-05 00:21:16 +11:00
Erik Johnston
616e6a10bd
Merge pull request #5002 from matrix-org/erikj/delete_group 
Add delete group admin API
 2019-04-04 14:15:41 +01:00
Andrew Morgan
db265f0642
Prevent kicking users who aren't in the room (#4999) 
Prevent kick events from succeeding if the user is not currently in the room.
 2019-04-04 13:05:51 +01:00
Marcel Krüger
9f5d206c4a Avoid redundant URL encoding (#4555) 
* Do not double encode fallback redirect URL

Signed-off-by: Marcel Fabian Krüger <zauguin@gmail.com>
 2019-04-04 12:05:56 +01:00
Erik Johnston
c192bf8970 Add admin API for group deletion 2019-04-03 16:29:52 +01:00
Erik Johnston
4a2e13631d Add functions to delete a group 2019-04-03 16:29:52 +01:00
Andrew Morgan
4a4d5c4fd6
Fix grammar and document get_current_users_in_room (#4998) 2019-04-03 14:32:20 +01:00
Neil Johnson
e8419554ff
Remove presence lists (#4989) 
Remove presence list support as per MSC 1819
 2019-04-03 11:11:15 +01:00
Erik Johnston
8f549c1177
Merge pull request #4982 from matrix-org/erikj/msc1915 
Implement MSC1915 - 3PID unbind APIs
 2019-04-03 11:07:09 +01:00
Amber Brown
7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Erik Johnston
3039d61baf
Merge pull request #4991 from matrix-org/erikj/stagger_push_startup 
Make starting pushers faster during start up
 2019-04-02 18:23:32 +01:00
Andrew Morgan
66e78700a2
Transfer related groups on room upgrade (#4990) 
Transfers the m.room.related_groups state event on room upgrade.
 2019-04-02 17:15:24 +01:00
Richard van der Hoff
8530090b16
Add config.signing_key_path. (#4974) 
As requested by @andrewshadura
 2019-04-02 16:59:27 +01:00
Erik Johnston
5bec8d660d Make starting pushers faster during start up 
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.

Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
 2019-04-02 16:59:13 +01:00
Richard van der Hoff
297bf2547e
Fix sync bug when accepting invites (#4956) 
Hopefully this time we really will fix #4422.

We need to make sure that the cache on
`get_rooms_for_user_with_stream_ordering` is invalidated *before* the
SyncHandler is notified for the new events, and we can now do so reliably via
the `events` stream.
 2019-04-02 12:42:39 +01:00
Erik Johnston
4ef5d17b96 Correctly handle id_server param 2019-04-02 11:20:09 +01:00
Erik Johnston
24232514bf Remove threepid binding if id server returns 400/404/501 2019-04-02 11:20:09 +01:00
Erik Johnston
c75e2017f1 Fixup docstrings 2019-04-02 11:20:06 +01:00
Erik Johnston
862d6e5ba5 Add unbind API to /r0 as it is now stabalised 2019-04-01 15:25:19 +01:00
Erik Johnston
3715c124b3 Grandfather in existing user threepids 
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
 2019-04-01 15:25:19 +01:00
Erik Johnston
057715aaa2 Allowing specifying IS to use in unbind API. 
By default the homeserver will use the identity server used during the
binding of the 3PID to unbind the 3PID. However, we need to allow
clients to explicitly ask the homeserver to unbind via a particular
identity server, for the case where the 3PID was bound out of band from
the homeserver.

Implements MSC915.
 2019-04-01 15:25:18 +01:00
Erik Johnston
9fbbc3d9e5 For unbind poke IS used during binding of 3PID 
This changes the behaviour from using the server specified trusted
identity server to using the IS that used during the binding of the
3PID, if known.

This is the behaviour specified by MSC1915.
 2019-04-01 15:23:30 +01:00
Erik Johnston
1666c0696a Track IS used to bind 3PIDs 
This will then be used to know which IS to default to when unbinding the
threepid.
 2019-04-01 15:23:01 +01:00
Neil Johnson
62988f73fd Merge branch 'master' into develop 2019-04-01 14:08:53 +01:00
Neil Johnson
35442efb75 0.99.3 2019-04-01 12:49:03 +00:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969) 
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
 2019-04-01 10:24:38 +01:00
Richard van der Hoff
902cdc63b6
Merge pull request #4955 from matrix-org/rav/merge_state_into_events 
Combine the CurrentStateDeltaStream into the EventStream
 2019-03-28 18:32:13 +00:00
Richard van der Hoff
d688a51736
Merge pull request #4954 from matrix-org/rav/refactor_parse_row 
Refactors to replication stream row update/parsing
 2019-03-28 18:31:17 +00:00
Neil Johnson
c7296bcb98
remove log line for password (#4965) 
Remove log line for password.
 2019-03-28 17:38:01 +00:00
Andrew Morgan
7a91b9d81c
Allow password providers to bind emails (#4947) 
This PR allows password provider modules to bind email addresses when a user is registering and is motivated by matrix-org/matrix-synapse-ldap3#58
 2019-03-28 15:48:07 +00:00
Erik Johnston
248014379e
Merge pull request #4942 from matrix-org/erikj/fix_presence 
Use event streams to calculate presence
 2019-03-28 14:38:31 +00:00
Erik Johnston
4e5f0f7ca0 Use an assert 2019-03-28 14:05:05 +00:00
Erik Johnston
40e56997bc Review comments 2019-03-28 13:48:41 +00:00
Richard van der Hoff
d035d62f6b Merge remote-tracking branch 'origin/develop' into rav/refactor_parse_row 2019-03-28 13:45:14 +00:00
Richard van der Hoff
4eeb2c2f07
Merge pull request #4953 from matrix-org/rav/refactor_replication_streams 
Split up replication.tcp.streams into smaller files
 2019-03-28 13:43:25 +00:00
Amber Brown
2e060774ad
Run black on some storage modules that the stats branch touches (#4959) 2019-03-29 00:37:16 +11:00
Richard van der Hoff
4b91c313a9 Combine the CurrentStateDeltaStream into the EventStream 2019-03-27 22:07:05 +00:00
Richard van der Hoff
1f6d6f918a Make EventStream rows have a type 
... as a precursor to combining it with the CurrentStateDelta stream.
 2019-03-27 22:07:05 +00:00
Richard van der Hoff
015b3622eb Skip building a ROW_TYPE when building updates 
We're about to turn it straight into a JSON object anyway so building a
ROW_TYPE is a bit pointless, and reduces flexibility in the update_function.
 2019-03-27 21:58:03 +00:00
Richard van der Hoff
f570916a3e Add parse_row method to replication stream class 
This will allow individual stream classes to override how a row is parsed.
 2019-03-27 21:32:33 +00:00
Richard van der Hoff
71dcb275f1 move FederationStream out to its own file 2019-03-27 21:13:14 +00:00
Richard van der Hoff
aa1e017864 move EventsStream out to its own file 2019-03-27 21:13:14 +00:00
Richard van der Hoff
a5798de067 Move replication.tcp.streams into a package 2019-03-27 21:13:14 +00:00
Richard van der Hoff
acaa18f7dd
Fix/improve some docstrings in the replication code. (#4949) 2019-03-27 21:12:36 +00:00
Erik Johnston
197fae1639 Use event streams to calculate presence 
Primarily this fixes a bug in the handling of remote users joining a
room where the server sent out the presence for all local users in the
room to all servers in the room.

We also change to using the state delta stream, rather than the
distributor, as it will make it easier to split processing out of the
master process (as well as being more flexible).

Finally, when sending presence states to newly joined servers we filter
out old presence states to reduce the number sent. Initially we filter
out states that are offline and have a last active more than a week ago,
though this can be changed down the line.

Fixes #3962
 2019-03-27 13:41:36 +00:00
Neil Johnson
4aa914369b bump version 2019-03-27 10:23:03 +00:00
Andrew Morgan
bbd244c7b2
Support 3PID login in password providers (#4931) 
Adds a new method, check_3pid_auth, which gives password providers
the chance to allow authentication with third-party identifiers such
as email or msisdn.
 2019-03-26 17:48:30 +00:00
Amber Brown
903f04c21f
Use the state event amount for userdir import batching, not room count (#4944) 2019-03-27 02:49:28 +11:00
Andrew Morgan
4a125be138
Make federation endpoints more tolerant of trailing slashes v2 (#4935) 
Redo of https://github.com/matrix-org/synapse/pull/4840
 2019-03-26 11:35:29 +00:00
Richard van der Hoff
8cbbedaa2b
Fix ClientReplicationStreamProtocol.__str__ (#4929) 
`__str__` depended on `self.addr`, which was absent from
ClientReplicationStreamProtocol, so attempting to call str on such an object
would raise an exception.

We can calculate the peer addr from the transport, so there is no need for addr
anyway.
 2019-03-25 16:41:51 +00:00
Richard van der Hoff
9bde730ef8
Fix bug where read-receipts lost their timestamps (#4927) 
Make sure that they are sent correctly over the replication stream.

Fixes: #4898
 2019-03-25 16:38:05 +00:00
Amber Brown
ac396a0d32
Refactor out state delta handling into its own class (#4917) 2019-03-25 20:37:08 +11:00
Erik Johnston
5fee9d8067
Merge pull request #4869 from matrix-org/erikj/yaml_load 
Fix yaml warnings by using safe_load
 2019-03-22 11:58:13 +00:00
Erik Johnston
3677548a82 Use yaml safe_load 2019-03-22 10:20:17 +00:00
Andrew Morgan
7bef97dfb7
Remove trailing slashes from outbound federation requests and retry on 400 (#4840) 
As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.
 2019-03-21 15:07:28 +00:00
Andrew Morgan
b41c2eaadc Clean up backoff_on_404 and metehod calls 2019-03-21 14:32:47 +00:00
Erik Johnston
01e6b405be
Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms 
Deny peeking into rooms that have been blocked
 2019-03-21 14:07:17 +00:00
Erik Johnston
3959858eaa
Merge pull request #4904 from matrix-org/erikj/fix_shutdown 
Fixup shutdown room API
 2019-03-21 11:24:42 +00:00
Erik Johnston
536a266520 Deny peeking into rooms that have been blocked 2019-03-21 11:20:13 +00:00
Erik Johnston
3ecec5ede2 Fix upsert 2019-03-21 10:21:15 +00:00
Erik Johnston
09f991a63d
Merge pull request #4896 from matrix-org/erikj/disable_room_directory 
Add option to disable search room lists
 2019-03-21 10:16:54 +00:00
Erik Johnston
cd62981a6a Revert spurious delete 2019-03-20 17:51:27 +00:00
Erik Johnston
8d8834d3e7 comment block_room 2019-03-20 17:49:56 +00:00
Erik Johnston
aa959a6c07 Use flags 2019-03-20 17:40:29 +00:00
Erik Johnston
7d47cc1305 Move requester check into assert_accepted_privacy_policy 2019-03-20 17:08:36 +00:00
Erik Johnston
72a14860ab Gracefully handle failing to kick user 2019-03-20 16:54:21 +00:00
Erik Johnston
6b28890543 Log new room ID 2019-03-20 16:52:28 +00:00
Erik Johnston
74c46d81fa Only require consent for events with an associated request 
There are a number of instances where a server or admin may puppet a
user to join/leave rooms, which we don't want to fail if the user has
not consented to the privacy policy. We fix this by adding a check to
test if the requester has an associated access_token, which is used as a
proxy to answer the question of whether the action is being done on
behalf of a real request from the user.
 2019-03-20 16:50:23 +00:00
Erik Johnston
67d618e111 Allow blocking a room multiple times 2019-03-20 16:50:05 +00:00
Erik Johnston
263f2c9ce1
Merge pull request #4895 from matrix-org/erikj/disable_user_search 
Add option to disable searching in the user dir
 2019-03-20 16:47:15 +00:00
Amber Brown
4d53017432
Batching in the user directory import (#4900) 2019-03-21 03:06:36 +11:00
Richard van der Hoff
cdb8036161
Add a config option for torture-testing worker replication. (#4902) 
Setting this to 50 or so makes a bunch of sytests fail in worker mode.
 2019-03-20 16:04:35 +00:00
Richard van der Hoff
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890) 
Rate-limit outgoing read-receipts as per #4730.
 2019-03-20 16:02:25 +00:00
Erik Johnston
cd8c5b91ad Fix up sample config 2019-03-20 14:35:41 +00:00
Richard van der Hoff
ab20f85c59
Update synapse/config/user_directory.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-03-20 14:33:11 +00:00
Erik Johnston
cc197a61a1 Disable publishing to room list when its disabled 2019-03-20 14:30:36 +00:00
Erik Johnston
2c90422146 Pull out config option 2019-03-20 14:25:58 +00:00
Erik Johnston
7529038e66 Return before we log 2019-03-20 14:25:28 +00:00
Erik Johnston
926f29ea6d Fix up config comments 2019-03-20 14:24:53 +00:00
Andrew Morgan
2150151abe kwargs doesn't like commas on calling funcs either. TIL 2019-03-20 14:13:32 +00:00
Andrew Morgan
bb52a2e653 lint 2019-03-20 14:08:57 +00:00
Andrew Morgan
cd36a1283b New test, fix issues 2019-03-20 14:00:39 +00:00
Andrew Morgan
c69df5d5d3 Fix comments. v0.99.2 -> v0.99.3 2019-03-20 11:27:18 +00:00
Andrew Morgan
551ea11559 Just return if not doing any trailing slash shennanigans 2019-03-20 11:07:36 +00:00
Erik Johnston
213c98c00a Add option to disable search room lists 
This disables both local and remote room list searching.
 2019-03-19 17:10:52 +00:00
Erik Johnston
320667a479 Add option to disable searching in the user dir 
We still populate it, as it can still be accessed via the admin API.
 2019-03-19 16:40:19 +00:00
Richard van der Hoff
13bc1e0746 Use a regular HomeServerConfig object for unit tests 
Rather than using a Mock for the homeserver config, use a genuine
HomeServerConfig object. This makes for a more realistic test, and means that
we don't have to keep remembering to add things to the mock config every time
we add a new config setting.
 2019-03-19 11:44:43 +00:00
Richard van der Hoff
45c4e19c74 Merge remote-tracking branch 'origin/develop' into HEAD 2019-03-19 11:43:46 +00:00
Neil Johnson
88f0675967 fix test_auto_create_auto_join_where_no_consent (#4886) 2019-03-19 11:38:59 +00:00
Richard van der Hoff
0dbfae03f9 Enforce hs_disabled_message correctly 
Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests
if there was no server_notices_mxid configured.
 2019-03-19 11:30:54 +00:00
Erik Johnston
e9eeca1314 Fix user directory background update (#4887) 2019-03-19 22:13:53 +11:00
Richard van der Hoff
fd463b4f5d
Comment out most options in the generated config. (#4863) 
Make it so that most options in the config are optional, and commented out in
the generated config.

The reasons this is a good thing are as follows:

* If we decide that we should change the default for an option, we can do so,
  and only those admins that have deliberately chosen to override that option
  will be stuck on the old setting.

* It moves us towards a point where we can get rid of the super-surprising
  feature of synapse where the default settings for the config come from the
  generated yaml.

* It makes setting up a test config for unit testing an order of magnitude
  easier (see forthcoming PR).

* It makes the generated config more consistent, and hopefully easier for users
  to understand.
 2019-03-19 10:06:40 +00:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Andrew Morgan
a8ad39eec7 lint 2019-03-18 17:47:39 +00:00
Andrew Morgan
621e7f37f1 Better exception handling 2019-03-18 17:45:54 +00:00
Brendan Abolivier
651ad8bc96 Add ratelimiting on failed login attempts (#4865) 2019-03-18 12:57:20 +00:00
Brendan Abolivier
899e523d6d
Add ratelimiting on login (#4821) 
Add two ratelimiters on login (per-IP address and per-userID).
 2019-03-15 17:46:16 +00:00
Richard van der Hoff
3b7ceb2c69
Merge pull request #4855 from matrix-org/rav/refactor_transaction_queue 
Split TransactionQueue up
 2019-03-15 12:32:11 +00:00
Richard van der Hoff
2dee441bdb
Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker 
Move client receipt processing to federation sender worker.
 2019-03-15 12:30:30 +00:00
Erik Johnston
b0fa3f6ff3
Merge pull request #4853 from matrix-org/erikj/worker_docker_ci 
Allow passing --daemonize to workers
 2019-03-15 10:35:38 +00:00
Erik Johnston
271cb1998b Revert "Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793)" 
This reverts commit 290552fd83.
 2019-03-14 14:30:54 +00:00
Erik Johnston
9073cfc8bd
Merge pull request #4846 from matrix-org/hawkowl/userdir-search 
Improve searching in the userdir
 2019-03-14 13:54:15 +00:00
Erik Johnston
9ad448c1e5 Correctly handle all command line options 2019-03-14 13:32:14 +00:00
Andrew Morgan
ecea5af491 Correct var name 2019-03-13 21:21:03 +00:00
Andrew Morgan
b2df0e8e2c receiving a 400 caused an exception. handle it 2019-03-13 21:08:10 +00:00
Andrew Morgan
9a2e22fd41 is this what purgatory feels like 2019-03-13 20:29:38 +00:00
Andrew Morgan
45524f2f5e i should have given up x2 2019-03-13 20:17:39 +00:00
Andrew Morgan
8d16ffaf7a i should have given up 2019-03-13 20:03:10 +00:00
Richard van der Hoff
02e23b36bc Rename and move the classes 2019-03-13 20:02:56 +00:00
Andrew Morgan
26f8e2d099 there comes a time when you should give up. but you dont 2019-03-13 19:49:08 +00:00
Andrew Morgan
5ca857ad84 as above 2019-03-13 19:35:23 +00:00
Andrew Morgan
7c0295f13c no kwargs today 2019-03-13 19:27:10 +00:00
Andrew Morgan
66cdb840a6 Or perhaps I was the one who was drunk 2019-03-13 19:18:25 +00:00
Andrew Morgan
bec313818c go home python, you're drunk 2019-03-13 19:10:56 +00:00
Andrew Morgan
c991e7aec7 Syntax checker is bork 2019-03-13 19:08:08 +00:00
Andrew Morgan
c2d848b80d Destructure again 2019-03-13 19:04:43 +00:00
Andrew Morgan
ee8ba397e8 Are you happy now 2019-03-13 18:48:31 +00:00
Andrew Morgan
9dd0e34679 Syntax test 2019-03-13 18:45:17 +00:00
Andrew Morgan
220607a618 Remove testing code 2019-03-13 18:43:40 +00:00
Andrew Morgan
660b77f362 Add missing docstring detail 2019-03-13 18:38:16 +00:00
Andrew Morgan
5526b054aa Fix syntax issues 2019-03-13 18:35:21 +00:00
Andrew Morgan
09626bfd39 Switch to wrapper function around _send_request 2019-03-13 18:26:06 +00:00
Richard van der Hoff
5d89a526f1 Factor per-destination stuff out of TransactionQueue 
This is easier than having to have a million fields keyed on destination.
 2019-03-13 17:48:29 +00:00
Erik Johnston
72bfaf746d Allow passing --daemonize to workers 2019-03-13 17:33:54 +00:00
Richard van der Hoff
fdcad8eabd Move client receipt processing to federation sender worker. 
This is mostly a prerequisite for #4730, but also fits with the general theme
of "move everything off the master that we possibly can".
 2019-03-13 17:21:19 +00:00
Richard van der Hoff
eed7271b3b declare a ReadReceipt class 
I'm going to use this in queues and things, so it'll be useful to give it more
of a structure.
 2019-03-13 17:20:55 +00:00
Andrew Morgan
7998ca3a66
Document using a certificate with a full chain (#4849) 2019-03-13 15:26:29 +00:00
Andrew Morgan
7d053cfe10 Retry on 400:M_UNRECOGNIZED 2019-03-13 12:10:33 +00:00
Andrew Morgan
7e75d9644b Fix paranthesis indent 2019-03-13 11:15:23 +00:00
Andrew Morgan
cf301e37d8 Add workaround note 2019-03-13 11:14:43 +00:00
Andrew Morgan
d42c81d724
Transfer local user's push rules on room upgrade (#4838) 
Transfer push rules (notifications) on room upgrade
 2019-03-12 14:42:53 +00:00
Amber Brown
c0332d095f fixup 2019-03-13 01:30:54 +11:00
Amber Brown
797b6a63fc fixup 2019-03-13 01:17:51 +11:00
Andrew Morgan
0ea8582f8b Cleaner way of implementing trailing slashes 2019-03-12 14:11:11 +00:00
Amber Brown
81d9d1bee6 fixup 2019-03-12 22:28:48 +11:00
Amber Brown
6f5890b2fa fixup 2019-03-12 22:27:56 +11:00
Amber Brown
10480c4348 fixup 2019-03-12 21:47:14 +11:00
Amber Brown
8b618041ef fixup 2019-03-12 18:06:28 +11:00
Amber Brown
c980c7e31f use the old method 2019-03-12 17:51:14 +11:00
Aaron Raimist
8ea1b41a0e Clarify what registration_shared_secret allows for (#2885) (#4844) 
* Clarify what registration_shared_secret allows for (#2885)

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-03-11 18:21:52 +00:00
Andrew Morgan
f18dca26da Merge branch 'develop' into anoa/trailing_slashes_client 2019-03-11 17:44:29 +00:00
Andrew Morgan
290552fd83
Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793) 
Server side of a solution towards #3622.
 2019-03-11 17:44:03 +00:00
Andrew Morgan
a8a028dfce Merge branch 'develop' into anoa/trailing_slashes_client 2019-03-11 17:07:31 +00:00
Erik Johnston
78c563b77c Correctly log expected errors when fetching server keys 2019-03-11 14:11:10 +00:00
Amber Brown
78a6b950b3 fix 2019-03-12 00:50:28 +11:00
Amber Brown
1b77bd69fb pep8 2019-03-12 00:39:12 +11:00
Amber Brown
5ba8ceab4c fixes 2019-03-12 00:35:31 +11:00
Amber Brown
8da22e2b53 master startup 2019-03-11 21:13:35 +11:00
Amber Brown
30a8deeb68 Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search 2019-03-11 21:12:30 +11:00
Amber Brown
4abc988c6a
initial 2019-03-11 21:11:36 +11:00
Matthew Hodgson
2326e00bc4 fix incorrect encoding of filenames with spaces in (#2090) 
fixes https://github.com/vector-im/riot-web/issues/3155
 2019-03-11 09:53:45 +00:00
Andrew Morgan
a5dd335cd8 lint 2019-03-08 18:25:59 +00:00
Andrew Morgan
64ff11019e Retry certain federation requests on 404 2019-03-08 18:22:47 +00:00
Andrew Morgan
525dd02bbe Remove trailing slashes from outbound federation requests 2019-03-08 16:55:52 +00:00
Erik Johnston
fe6c12e6cd Add comment to schema 2019-03-08 16:38:23 +00:00
Erik Johnston
5536ddba75 Make prev_state field optional 
The `prev_state` field on events is not specced and so synapse shouldn't
explode if an event is missing the field.

Fixes #4787
 2019-03-08 15:05:32 +00:00
Erik Johnston
8c4896668f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl 2019-03-08 11:44:20 +00:00
Erik Johnston
0ff8163eae Factor out soft fail checks 2019-03-08 11:26:33 +00:00
Erik Johnston
469b66c102
Merge pull request #4829 from matrix-org/erikj/device_list_seen_updates 
When re-syncing device lists reset the state
 2019-03-08 09:54:54 +00:00
Erik Johnston
d42b41544a When re-syncing device lists reset the state 
We keep track of what stream IDs we've seen so that we know what updates
we've handled or missed. If we re-sync we don't know if the updates
we've seen are included in the re-sync (there may be a race), so we
should reset the seen updates.
 2019-03-07 16:04:24 +00:00
Erik Johnston
c633fc02d7 Add some debug logging for device list handling 2019-03-07 15:58:03 +00:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Travis Ralston
06cbf79fe3
Merge pull request #4779 from matrix-org/travis/icons 
Use static locations for Riot icons
 2019-03-06 15:11:41 -07:00
Erik Johnston
1f85c2c0ce
Merge pull request #4818 from matrix-org/erikj/prefill_client_ips 
Prefill client IPs cache on workers
 2019-03-06 19:17:39 +00:00
Erik Johnston
0a6e716600
Merge pull request #4815 from matrix-org/erikj/docstrings 
Add docstrings from matrix-org-hotfixes
 2019-03-06 18:59:28 +00:00
Erik Johnston
c665b637de
Merge pull request #4816 from matrix-org/erikj/4422_debug 
Port #4422 debug logging from hotfixes
 2019-03-06 18:59:12 +00:00
Erik Johnston
face0c5b3c Prefill client IPs cache on workers 2019-03-06 17:39:32 +00:00
Erik Johnston
b879870b2d Send message after room has been shutdown 
Currently the explanation message is sent to the abuse room before any
users are forced joined, which means it tends to get lost in the backlog
of joins.

So instead we send the message *after* we've forced joined everyone.
 2019-03-06 17:35:11 +00:00
Erik Johnston
8b7790e68f Port #4422 debug logging from hotfixes 2019-03-06 17:29:15 +00:00
Erik Johnston
6d13bdec91 Add docstrings from matrix-org-hotfixes 2019-03-06 17:24:26 +00:00
Erik Johnston
a9de04be72 Implement soft fail 2019-03-06 16:22:16 +00:00
Andrew Morgan
7b8a157b79
Merge pull request #4792 from matrix-org/anoa/replication_tokens 
Support batch updates in the worker sender
 2019-03-06 15:48:29 +00:00
Brendan Abolivier
6f3cde8b25 Make registration ratelimiter separate from the main events one 2019-03-06 11:02:42 +00:00
Brendan Abolivier
f4195f4118 Revert "Split ratelimiters in two (one for events, one for registration)" 
This reverts commit d7dbad3526.
 2019-03-06 10:55:22 +00:00
Brendan Abolivier
d7dbad3526 Split ratelimiters in two (one for events, one for registration) 2019-03-05 18:41:27 +00:00
Brendan Abolivier
067ce795c0 Move settings from registration to ratelimiting in config file 2019-03-05 18:03:14 +00:00
Erik Johnston
16c8b4ecbd
Merge pull request #4772 from jbweston/jbweston/server-version-api 
Add 'server_version' endpoint to admin API
 2019-03-05 16:31:00 +00:00
Brendan Abolivier
a4c3a361b7
Add rate-limiting on registration (#4735) 
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
 2019-03-05 14:25:33 +00:00
Andrew Morgan
b9f6163092 Simplify token replication logic 2019-03-05 13:58:30 +00:00
Andrew Morgan
ae90531036
Merge pull request #4794 from matrix-org/anoa/erroneous_dollahs 
Remove unnecessary dollar signs
 2019-03-05 09:46:59 +00:00
Erik Johnston
b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room 
Stop backpaginating when events not visible
 2019-03-05 09:32:33 +00:00
Erik Johnston
9e8bca5667
Merge pull request #4799 from matrix-org/rav/clean_up_replication_code 
Clean ups in replication notifier
 2019-03-05 09:19:48 +00:00
Erik Johnston
aa06d26ae0 clarify comments 2019-03-05 09:16:35 +00:00
Erik Johnston
c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys 
Allow /keys/{changes,query} API to run on worker
 2019-03-05 09:06:25 +00:00
Richard van der Hoff
c7325776a7 Remove redundant PreserveLoggingContext 
Both (!) things that register as replication listeners do the right thing wrt
logcontexts, so this is redundant.
 2019-03-04 18:31:18 +00:00
Erik Johnston
bfa7d46a10 Allow /keys/{changes,query} API to run on worker 2019-03-04 18:30:01 +00:00
Erik Johnston
157e5a8f27 Split DeviceHandler into master and worker 2019-03-04 18:29:26 +00:00
Richard van der Hoff
daa10e3e66 Remove unused wait_for_replication method 
I guess this was used once? It's not now, anyway.
 2019-03-04 18:27:32 +00:00
Richard van der Hoff
2db49ea476 Add some debug about processing read receipts. 
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
 2019-03-04 18:19:40 +00:00
Richard van der Hoff
b29693a30b Clean up read-receipt handling. 
Remove a call to run_as_background_process: there is no need to run this as a
background process, because build_and_send_edu does not block.

We may as well inline the whole of _push_remotes.
 2019-03-04 18:16:43 +00:00
Erik Johnston
a84b8d56c2 Fixup slave stores 2019-03-04 18:04:57 +00:00
Richard van der Hoff
8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Erik Johnston
0d2d046709 Fix missing null guard 2019-03-04 16:04:04 +00:00
Andrew Morgan
336de1d45b Remove unnecessary dollar signs 
A dollar sign is already appended to the end of each PATH, so there's
no need to add one in the PATH declaration as well.
 2019-03-04 15:25:12 +00:00
Andrew Morgan
fe7bd23a85 Clean up logic and add comments 2019-03-04 15:08:15 +00:00
Erik Johnston
d1523aed6b Only check history visibility when filtering 
When filtering events to send to server we check more than just history
visibility. However when deciding whether to backfill or not we only
care about the history visibility.
 2019-03-04 14:43:42 +00:00
Andrew Morgan
9f7cdf3da1 Clearer branching, fix missing list clear 2019-03-04 14:36:52 +00:00
Andrew Morgan
5f0c449dd5 Prevent replication wedging 2019-03-04 14:03:18 +00:00
Richard van der Hoff
856c83f5f8
Avoid rebuilding Edu objects in worker mode (#4770) 
In worker mode, on the federation sender, when we receive an edu for sending
over the replication socket, it is parsed into an Edu object. There is no point
extracting the contents of it so that we can then immediately build another Edu.
 2019-03-04 12:57:44 +00:00
Erik Johnston
8b63fe4c26 s/get_forward_events/get_successor_events/ 2019-03-04 11:56:03 +00:00
Erik Johnston
fbc047f2a5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room 2019-03-04 11:54:58 +00:00
Travis Ralston
4dfbae18fe Use static locations for Riot icons 
See https://github.com/vector-im/riot-web/issues/9009
 2019-03-01 15:06:21 -07:00
Richard van der Hoff
3064952939
Fix incorrect log about not persisting duplicate state event. (#4776) 
We were logging this when it was not true.
 2019-03-01 16:47:12 +00:00
Richard van der Hoff
1beebe916f Merge branch 'master' into develop 2019-03-01 10:58:39 +00:00
Richard van der Hoff
9ac72d9543 0.99.2 2019-03-01 10:55:44 +00:00
Joseph Weston
1e8388b311 Add 'server_version' endpoint to admin API 
This is required because the 'Server' HTTP header is not always
passed through proxies.
 2019-03-01 09:56:58 +01:00
Andrew Morgan
ac61b45a75 Minor docstring fixes for MatrixFederationAgent (#4765) 2019-02-28 16:24:01 +00:00
Amber Brown
b131cc77df
Make 'event_id' a required parameter in federated state requests (#4741) 
* make 'event_id' a required parameter in federated state requests

As per the spec: https://matrix.org/docs/spec/server_server/r0.1.1.html#id40

Signed-off-by: Joseph Weston <joseph@weston.cloud>

* add changelog entry for bugfix

Signed-off-by: Joseph Weston <joseph@weston.cloud>

* Update server.py
 2019-02-27 14:35:47 -08:00
Richard van der Hoff
68f47d6744 Fix parsing of Content-Disposition headers (#4763) 
* Fix parsing of Content-Disposition headers

TIL: filenames in content-dispostion headers can contain semicolons, and aren't
%-encoded.

* fix python2 incompatibility

* Fix docstrings
 2019-02-27 14:29:10 -08:00
Amber Brown
f2a753ea38
Move from TravisCI to BuildKite (#4752) 2019-02-27 13:03:14 -08:00
Erik Johnston
76550c58d2
Merge pull request #4759 from matrix-org/erikj/3pid_client_reader 
Move /account/3pid to client_reader
 2019-02-27 16:11:21 +00:00
Erik Johnston
8267034a63
Merge pull request #4758 from matrix-org/erikj/use_presence_replication 
When presence is disabled don't send over replication
 2019-02-27 15:46:26 +00:00
Erik Johnston
54f9ce11a7 Move /account/3pid to client_reader 2019-02-27 14:26:08 +00:00
Erik Johnston
1e315017d3 When presence is enabled don't send over replication 2019-02-27 13:53:46 +00:00
Erik Johnston
4cff9376f7 Move server key queries to federation reader 2019-02-27 13:43:53 +00:00
Erik Johnston
b183fef9ac Update comments 2019-02-27 13:06:10 +00:00
Erik Johnston
7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff 
Fix tightloop over connecting to replication server
 2019-02-27 11:00:59 +00:00
Richard van der Hoff
44a4d65586 0.99.2rc1 2019-02-27 10:48:34 +00:00
Erik Johnston
6bb1c028f1 Limit cache invalidation replication line length (#4748) 2019-02-27 10:28:37 +00:00
Erik Johnston
6870fc496f Move connecting logic into ClientReplicationStreamProtocol 2019-02-27 10:23:51 +00:00
Richard van der Hoff
f191be822b
Add database version to phonehome stats. (#4753) 2019-02-27 10:21:49 +00:00
Erik Johnston
25814921f1 Increase the max delay between retry attempts 
Otherwise if you have many workers they can easily take out master with
their connection attempts
 2019-02-26 15:12:33 +00:00
Erik Johnston
313987187e Fix tightloop over connecting to replication server 
If the client failed to process incoming commands during the initial set
up of the replication connection it would immediately disconnect and
reconnect, resulting in a tightloop.

This can happen, for example, when subscribing to a stream that has a
row that is too long in the backlog.

The fix here is to not consider the connection successfully set up until
the client has succesfully subscribed and caught up with the streams.
This ensures that the retry logic timers aren't reset until then,
meaning that if an error does happen during start up the client will
continue backing off before retrying again.
 2019-02-26 15:05:41 +00:00
Andrew Morgan
802884d4ee Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate_develop 2019-02-26 14:23:40 +00:00
Andrew Morgan
6fcb25202f Put function def back to the way it was 2019-02-26 14:17:38 +00:00
Andrew Morgan
7a4632af9c Prevent showing non-fed rooms in fed /publicRooms 2019-02-26 13:37:24 +00:00
Andrew Morgan
c74624a633
Revert "Prevent showing non-fed rooms in fed /publicRooms" 2019-02-26 13:20:38 +00:00
Andrew Morgan
a1a6473293
Merge pull request #4736 from matrix-org/anoa/public_rooms_federate 
Config option to prevent showing non-fed rooms in fed /publicRooms
 2019-02-26 13:07:15 +00:00
Andrew Morgan
c4414768af Cleaner chunk logic 2019-02-26 12:22:34 +00:00
Andrew Morgan
a712aa3a9c Correct indent 2019-02-26 12:13:55 +00:00
Andrew Morgan
16565e67db Correct docstring types and chunk logic 2019-02-26 12:12:48 +00:00
Andrew Morgan
40c2271680 Clean up room chunk logic 2019-02-26 12:04:34 +00:00
Andrew Morgan
6728bf3940 Make not showing non-federated rooms the default 2019-02-26 11:52:52 +00:00
Andrew Morgan
6946c20111 Result may be None 2019-02-26 11:27:19 +00:00
Andrew Morgan
71669a0fba Address rich comments 2019-02-26 11:25:00 +00:00
Erik Johnston
899a119c2b Don't log stack trace when client has gone away during media download (#4738) 
* Don't log stack trace when client has gone away during media download

* Newsfile

* Fixup newsfile
 2019-02-25 11:17:22 -08:00
Richard van der Hoff
641c409e4e Fix ACME config for python 2. (#4717) 
Fixes #4675.
 2019-02-25 11:16:33 -08:00
Matthew Hodgson
70ea2f4e1d switch from google.com to recaptcha.net for reCAPTCHA (#4731) 
* add trivial clarification about jemalloc

* switch from google.com to recaptcha.net

because https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally
 2019-02-25 11:15:36 -08:00
Andrew Morgan
96c408273e Fix group's call to generate_room_entry 2019-02-25 18:00:17 +00:00
Andrew Morgan
1330aa4a8f elif not else if 2019-02-25 17:28:19 +00:00
Erik Johnston
65f3fbfbf7
Merge pull request #4737 from matrix-org/erikj/failure_log_tb 
Log tracebacks correctly
 2019-02-25 17:26:30 +00:00
Erik Johnston
c7b333c545 Log tracebacks correctly 2019-02-25 16:56:41 +00:00
Erik Johnston
69efe6fb16
Merge pull request #4718 from matrix-org/erikj/fix_backfill_state_shred 
Fix backfill storing incorrect state for events
 2019-02-25 16:48:52 +00:00
Erik Johnston
9c598dddcb Fix typo 2019-02-25 16:32:02 +00:00
Erik Johnston
b1a90da82e
Merge pull request #4721 from matrix-org/erikj/msc_1866 
MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API
 2019-02-25 15:53:10 +00:00
Erik Johnston
16c7afa94c
Merge pull request #4722 from matrix-org/erikj/correctly_handle_keyring_exceptions 
Handle errors when fetching remote server keys
 2019-02-25 15:53:02 +00:00
Andrew Morgan
8aaf7ffc44 syntax derp 2019-02-25 15:27:17 +00:00
Andrew Morgan
84c0a20dfe Simplify call to generate_room_entry 2019-02-25 15:23:27 +00:00
Erik Johnston
4b9e5076c4 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate 2019-02-25 15:08:18 +00:00
Andrew Morgan
07493607a8 Docs and arg name clarification 2019-02-25 15:04:46 +00:00
Andrew Morgan
bd398b874e Don't restrict non-fed rooms over client APIs 2019-02-25 15:04:46 +00:00
Andrew Morgan
e4b078a600 Config option to prevent showing non-fed rooms in fed /publicRooms 2019-02-25 15:04:46 +00:00
Erik Johnston
d730c2c22b More comments 2019-02-25 14:45:02 +00:00
Erik Johnston
890cb048fd Assert rather than clobber the values 2019-02-25 14:42:11 +00:00
Erik Johnston
65d1003d01 raise_from already raises 2019-02-25 14:34:03 +00:00
Erik Johnston
9342cc6ab1 Add comments and paranoia 2019-02-25 10:02:12 +00:00
Erik Johnston
47a7e3928d Correctly proxy exception in frontend_proxy worker 2019-02-23 15:17:57 +00:00
Erik Johnston
41285ffe5b Handle errors when fetching remote server keys 2019-02-23 15:09:39 +00:00
Erik Johnston
59e0112209 MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API 2019-02-23 14:50:43 +00:00
Erik Johnston
d14e94bae4
Merge pull request #4716 from matrix-org/erikj/pusher_logging 
Fix up pusher logging a bit
 2019-02-22 15:29:16 +00:00
Erik Johnston
b82c9cf462 Add missing return 2019-02-22 15:27:40 +00:00
Erik Johnston
f2891d2487 Correctly handle PusherConfigException 2019-02-22 15:18:19 +00:00
Erik Johnston
a164134a53 Drop logging level of creating a pusher 2019-02-22 14:48:06 +00:00
Erik Johnston
1d9df51ff1 Correctly handle null data in HttpPusher 2019-02-22 14:47:48 +00:00
Erik Johnston
80467bbac3 Fix state cache invalidation on workers 2019-02-22 14:38:14 +00:00
Erik Johnston
7b288826b7 Fix backfill storing incorrect state for events 2019-02-22 11:33:51 +00:00
Richard van der Hoff
e07384c4e1
Add prometheus metrics for number of badge update pushes. (#4709) 
We're counting the number of push notifications, but not the number of badges;
I'd like to see if they are significant.
 2019-02-22 10:57:15 +00:00
Richard van der Hoff
0abb094f1a
bail out early in on_new_receipts if no pushers (#4706) 2019-02-21 17:51:21 +00:00
Richard van der Hoff
6d65659b62
Run push_receipts_to_remotes as background job (#4707) 
I suspect the CPU usage metrics for this are going to /dev/null at the moment.
 2019-02-21 17:50:30 +00:00
Erik Johnston
b9d6756b14
Merge pull request #4263 from rkfg/develop 
Prevent crash on pagination.
 2019-02-21 17:42:15 +00:00
Erik Johnston
71b625d808 Stop backpaginating when events not visible 2019-02-20 18:14:12 +00:00
Erik Johnston
82fca11fc1
Merge pull request #4694 from matrix-org/erikj/fix_sentry_config_format 
Fixup generated metrics config
 2019-02-20 14:13:38 +00:00
Richard van der Hoff
82ca6d1f9f
Add metrics for number of outgoing EDUs, by type (#4695) 2019-02-20 14:13:14 +00:00
Erik Johnston
3d672fec51 Fixup generated metrics config 2019-02-20 13:39:37 +00:00
Amber Brown
a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Richard van der Hoff
b2200a8690
Logging improvements for the pusher (#4691) 2019-02-20 11:36:50 +00:00
Richard van der Hoff
c88bc53903
Fix TaskStopped exceptions when outbound requests time out (#4690) 2019-02-20 11:35:52 +00:00
Richard van der Hoff
86920ac266
Merge pull request #4678 from matrix-org/rav/tls_install_instructions 
Attempt to clarify installation/config instructions
 2019-02-20 11:10:33 +00:00
Erik Johnston
dbdc565dfd Fix registration on workers (#4682) 
* Move RegistrationHandler init to HomeServer

* Move post registration actions to RegistrationHandler

* Add post regisration replication endpoint

* Newsfile
 2019-02-20 18:47:31 +11:00
Richard van der Hoff
5f9bdf90fe Attempt to make default config more consistent 
The general idea here is that config examples should just have a hash and no
extraneous whitespace, both to make it easier for people who don't understand
yaml, and to make the examples stand out from the comments.
 2019-02-19 13:54:29 +00:00
Erik Johnston
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation 
Batch cache invalidation over replication
 2019-02-19 13:14:30 +00:00
Erik Johnston
62175a20e5 Docs 2019-02-19 11:38:40 +00:00
Erik Johnston
1bb35e3a83 Use itertools 2019-02-19 11:34:40 +00:00
Erik Johnston
bc8fa1509d Documentation 2019-02-19 11:24:59 +00:00
Brendan Abolivier
a288bdf0b1
Merge pull request #4652 from matrix-org/babolivier/acme-delegated 
Support .well-known delegation when issuing certificates through ACME
 2019-02-19 11:15:38 +00:00
Brendan Abolivier
5a707a2f9a Improve config documentation 2019-02-19 10:59:26 +00:00
Brendan Abolivier
a8626901cd Fetch ACME domain into an instance member 2019-02-19 10:54:33 +00:00
Erik Johnston
32590b7139
Merge pull request #4670 from matrix-org/erikj/register_login_split 
Split /login into client_reader
 2019-02-19 10:46:22 +00:00
Richard van der Hoff
7c70b8f8a6
Try and make TLS federation client code faster (#4674) 
* fix to use makeContext so that we don't need to rebuild the certificateoptions each time
 2019-02-19 10:19:16 +00:00
Andrew Morgan
968a30a75c
Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade 
Transfer bans on room upgrade
 2019-02-19 10:16:42 +00:00
Amber Brown
561eebe170 fix to use makeContext so that we don't need to rebuild the certificateoptions each time 2019-02-19 16:18:05 +11:00
Andrew Morgan
34ac75ce2c lint 2019-02-18 18:23:37 +00:00
Erik Johnston
a9b5ea6fc1 Batch cache invalidation over replication 
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.

Hopefully this will reduce load on the master process by substantially
reducing traffic.
 2019-02-18 17:53:31 +00:00
Andrew Morgan
f8b9ca53ce Move member event processing and changelog fix 2019-02-18 17:28:52 +00:00
Erik Johnston
d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry 
Add basic optional sentry.io integration
 2019-02-18 17:22:45 +00:00
Erik Johnston
4cc4400b4d Split /login into client_reader 2019-02-18 17:19:01 +00:00
Erik Johnston
fc2c245a1f
Merge pull request #4666 from matrix-org/erikj/register_login_split 
Split out registration to worker
 2019-02-18 17:18:06 +00:00
Erik Johnston
459d3d5046
Merge pull request #4668 from matrix-org/erikj/catch_exceptions 
Correctly handle HttpResponseException when handling device updates
 2019-02-18 16:55:43 +00:00
Erik Johnston
d328a93b51 Fixup error handling and message 2019-02-18 16:53:56 +00:00
Erik Johnston
af691e415c Move register_device into handler 2019-02-18 16:49:38 +00:00
Erik Johnston
028267acd2
Merge pull request #4669 from matrix-org/erikj/log_exception 
Cleanup top level request exception logging
 2019-02-18 16:41:41 +00:00
Erik Johnston
d08bac4136
Merge pull request #4651 from matrix-org/matthew/well-known-cors 
set CORS on .well-known URI to unbreak modular
 2019-02-18 16:38:11 +00:00
Erik Johnston
c30f73c86a
Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers 
Fix kicking guest users in worker mode
 2019-02-18 15:55:48 +00:00
Erik Johnston
092b541401 Fixup 2019-02-18 15:52:26 +00:00
Brendan Abolivier
45bb55c6de Use a configuration parameter to give the domain to generate a certificate for 2019-02-18 15:46:23 +00:00
Erik Johnston
8b9ae6d3a6 Update docs 2019-02-18 15:26:13 +00:00
Erik Johnston
94960cef03 pep8 2019-02-18 15:24:13 +00:00
Erik Johnston
fe725f7e45 Cleanup top level request exception logging 
Firstly, we always logged that the request was being handled via
`JsonResource._async_render`, so we change that to use the servlet name
we add to the request.

Secondly, we pass the exception information to the logger rather than
formatting it manually. This makes it consistent with other exception
logging, allwoing logging hooks and formatters to access the exception
information.
 2019-02-18 15:11:04 +00:00
Matthew Hodgson
04dad5ac16 fix missig import 2019-02-18 15:59:23 +01:00
Erik Johnston
e07cc31cb8 Correctly handle HttpResponseException 2019-02-18 14:55:09 +00:00
Brendan Abolivier
68a53f825f Merge branch 'develop' into babolivier/acme-delegated 2019-02-18 14:52:23 +00:00
Erik Johnston
32e54b472a Fix kicking guest users in worker mode 
When guest_access changes from allowed to forbidden all local guest
users should be kicked from the room. This did not happen when
revocation was received from federation on a worker.

Presumably broken in #4141
 2019-02-18 14:16:07 +00:00
Andrew Morgan
915421065b Membership events are done later 2019-02-18 14:02:09 +00:00
Erik Johnston
d1b060b492
Merge pull request #4643 from matrix-org/erikj/catch_exceptions 
Correctly handle RequestSendFailed exceptions
 2019-02-18 13:53:24 +00:00
Andrew Morgan
9caab0c364 Transfer bans on room upgrade 2019-02-18 13:52:51 +00:00
Erik Johnston
dc5efc92a8 Fixup 2019-02-18 13:52:49 +00:00
Erik Johnston
eb2b8523ae Split out registration to worker 
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.

Note: due to the in-memory session map all registration requests must be
handled by the same worker.
 2019-02-18 12:12:57 +00:00
Andrew Morgan
6d02a13d81
Typo in info log 
Co-Authored-By: babolivier <contact@brendanabolivier.com>
 2019-02-18 11:36:34 +00:00
Juuso "Linda" Lapinlampi
68d2869c8d config: Remove a repeated word from a logger warning 
The warning for missing macaroon_secret_key was "missing missing".
 2019-02-15 22:24:53 -07:00
Erik Johnston
bd4505f765
Merge pull request #4647 from matrix-org/erikj/add_room_publishing_rules 
Add configurable room list publishing rules
 2019-02-15 22:11:01 +00:00
Brendan Abolivier
f86b695cbd
Various cosmetics to make TravisCI happy 2019-02-15 12:29:34 +00:00
Brendan Abolivier
af8a2f679b
Remove unused import 2019-02-15 12:27:43 +00:00
Brendan Abolivier
1895d14e12
Support .well-known delegation when issuing certificates through ACME 2019-02-15 12:05:08 +00:00
Erik Johnston
b99c532c1c Move defaults up into code 2019-02-15 10:53:39 +00:00
Erik Johnston
02c729d6b0 Hoist up checks to reduce overall work 2019-02-15 10:20:02 +00:00
Erik Johnston
02c46acc6a Fixup comments 2019-02-15 10:17:13 +00:00
Matthew Hodgson
bfcefbb230 tabs 2019-02-14 19:53:12 +00:00
Matthew Hodgson
6f47bc3fb2 set CORS on .well-known URI to unbreak modular 
otherwise a riot/web running on foo.riot.im can't query
the .well-known on foo.modular.im...
 2019-02-14 19:51:41 +00:00
Erik Johnston
8e32f26cb8 Clarify comments 2019-02-14 18:21:24 +00:00
Erik Johnston
cb12a37708 Clarify and fix behaviour when there are multiple aliases 2019-02-14 18:16:32 +00:00
Erik Johnston
f61b2068e6 Only fetch aliases when publishing rooms 2019-02-14 18:08:09 +00:00
Erik Johnston
f666fe36d7 Fixup comments 2019-02-14 18:07:24 +00:00
Richard van der Hoff
bf4fd14806 Merge branch 'master' into develop 2019-02-14 17:34:09 +00:00
Richard van der Hoff
f595d6ac57 0.99.1.1 2019-02-14 17:20:02 +00:00
Richard van der Hoff
f311018823
Fix errors in acme provisioning (#4648) 
* Better logging for errors on startup

* Fix "TypeError: '>' not supported" when starting without an existing
  certificate
* Fix a bug where an existing certificate would be reprovisoned every day
 2019-02-14 17:10:36 +00:00
Erik Johnston
eaf4d11af9 Add configurable room list publishing rules 
This allows specifying who and what is allowed to be published onto the
public room list
 2019-02-14 16:02:23 +00:00
Richard van der Hoff
b02465b9db Merge branch 'master' into develop 2019-02-14 14:42:03 +00:00
Richard van der Hoff
06cd757ae7 0.99.1 2019-02-14 14:24:24 +00:00
Erik Johnston
7fc1196a36 Correctly handle RequestSendFailed exceptions 
This mainly reduces the number of exceptions we log.
 2019-02-14 14:01:04 +00:00
Erik Johnston
6cb415b63f Fixup comments and add warning 2019-02-13 16:15:11 +00:00
Richard van der Hoff
c6e75c9f2d
Merge pull request #4450 from 14mRh4X0r/fix-dependency-message 
Fix error message for optional dependencies
 2019-02-13 16:12:49 +00:00
Richard van der Hoff
3bc238629e 0.99.1rc2 2019-02-13 14:46:18 +00:00
Richard van der Hoff
c1dfd6a18a Merge remote-tracking branch 'origin/release-v0.99.1' into develop 2019-02-13 14:27:45 +00:00
Erik Johnston
309f3bb322
Update synapse/app/_base.py 
Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>
 2019-02-13 13:24:27 +00:00
Amber Brown
bb4fd8f927
Run black on user directory code (#4635) 2019-02-13 23:05:32 +11:00
Richard van der Hoff
767686af48 Use listen_tcp for the replication listener 
Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with
what we do elsewhere.
 2019-02-13 11:59:04 +00:00
Richard van der Hoff
2a5a15aff8 Improve logging around listening services 
I wanted to bring listen_tcp into line with listen_ssl in terms of returning a
list of ports, and wanted to check that was a safe thing to do - hence the
logging in `refresh_certificate`.

Also, pull the 'Synapse now listening' message up to homeserver.py, because it
was being duplicated everywhere else.
 2019-02-13 11:58:54 +00:00
Richard van der Hoff
e3a0300431 Special-case the default bind_addresses for metrics listener 
turns out it doesn't really support ipv6, so let's hack around that by only
listening on ipv4 by default.
 2019-02-13 11:48:56 +00:00
Erik Johnston
93f7d2df3e Comments 2019-02-12 16:03:40 +00:00
Erik Johnston
6a8f902edb Raise an appropriate error message if sentry_sdk missing 2019-02-12 16:01:41 +00:00
Erik Johnston
ef2228c890 Basic sentry integration 2019-02-12 13:55:58 +00:00
Erik Johnston
d2fa7b7e99 Update changelog and version 2019-02-12 13:22:25 +00:00
Erik Johnston
ba3f27b69a
Merge pull request #4608 from matrix-org/anoa/acls_room_upgrade 
Transfer Server ACLs on room upgrade
 2019-02-12 13:20:06 +00:00
Erik Johnston
cf82338930
Merge pull request #4627 from matrix-org/erikj/user_ips_analyze 
Analyze user_ips before running deduplication
 2019-02-12 13:05:09 +00:00
Erik Johnston
3df8fcca25
Merge pull request #4626 from matrix-org/erikj/fixup_user_ips_dedupe 
Reduce user_ips bloat during dedupe background update
 2019-02-12 13:02:58 +00:00
Erik Johnston
495ea92350 Fix pep8 2019-02-12 12:40:42 +00:00
Erik Johnston
483ba85c7a Analyze user_ips before running deduplication 
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
 2019-02-12 11:55:27 +00:00
Erik Johnston
362d80b770 Reduce user_ips bloat during dedupe background update 
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.

This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.
 2019-02-12 11:39:34 +00:00
Erik Johnston
3c03c37883
Merge pull request #4625 from matrix-org/rav/fix_generate_config_warnings 
fix self-signed cert notice from generate-config
 2019-02-12 11:24:45 +00:00
Richard van der Hoff
a4ce91396b
Disable TLS by default (#4614) 2019-02-12 10:52:08 +00:00
Richard van der Hoff
32b781bfe2
Fix error when loading cert if tls is disabled (#4618) 
If TLS is disabled, it should not be an error if no cert is given.

Fixes #4554.
 2019-02-12 10:51:31 +00:00
Richard van der Hoff
dfc846a316 fix self-signed cert notice from generate-config 
fixes #4620
 2019-02-12 10:37:59 +00:00
Richard van der Hoff
0ca2908653 fix tests 2019-02-11 22:01:27 +00:00
Richard van der Hoff
4fddf8fc77 Infer no_tls from presence of TLS listeners 
Rather than have to specify `no_tls` explicitly, infer whether we need to load
the TLS keys etc from whether we have any TLS-enabled listeners.
 2019-02-11 21:39:14 +00:00
Richard van der Hoff
15272f837c Merge branch 'rav/no_create_server_contexts_if_no_tls' into rav/tls_cert/work 2019-02-11 21:34:19 +00:00
Richard van der Hoff
9645728619 Don't create server contexts when TLS is disabled 
we aren't going to use them anyway.
 2019-02-11 21:32:01 +00:00
Richard van der Hoff
be794c7cf7 Merge branch 'rav/tls_config_logging_fixes' into rav/tls_cert/work 2019-02-11 21:16:00 +00:00
Richard van der Hoff
2129dd1a02 Fail cleanly if listener config lacks a 'port' 
... otherwise we would fail with a mysterious KeyError or something later.
 2019-02-11 21:15:01 +00:00
Richard van der Hoff
086f6f27d4 Logging improvements around TLS certs 
Log which file we're reading keys and certs from, and refactor the code a bit
in preparation for other work
 2019-02-11 21:02:06 +00:00
Richard van der Hoff
5d27730a73
Move ClientTLSOptionsFactory init out of refresh_certificates (#4611) 
It's nothing to do with refreshing the certificates. No idea why it was here.
 2019-02-11 18:03:30 +00:00
Erik Johnston
719e073f00
Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating 
add updating of backup versions
 2019-02-11 13:45:49 +00:00
Richard van der Hoff
24b7f3916d
Clean up default listener configuration (#4586) 
Rearrange the comments to try to clarify them, and expand on what some of it
means.

Use a sensible default 'bind_addresses' setting.

For the insecure port, only bind to localhost, and enable x_forwarded, since
apparently it's for use behind a load-balancer.
 2019-02-11 12:50:30 +00:00
Andrew Morgan
a126f86eec Transfer Server ACLs on room upgrade 2019-02-11 11:30:37 +00:00
Amber Brown
6e2a5aa050 ACME Reprovisioning (#4522) 2019-02-11 10:36:26 +00:00
Amber Brown
4ffd10f46d Be tolerant of blank TLS fingerprints config (#4589) 2019-02-11 10:04:27 +00:00
Erik Johnston
b201149c7e
Merge pull request #4420 from matrix-org/jaywink/openid-listener 
New listener resource for the federation API "openid/userinfo" endpoint
 2019-02-11 09:44:00 +00:00
Valentin Anger
2dc2b6e9f1 Allow "unavailable" presence status for /sync (#4592) 
* Allow "unavailable" presence status for /sync

Closes #3772, closes #3779

Signed-off-by: Valentin Anger <valentin.an.1999@gmail.com>

* Add changelog for PR 4592
 2019-02-08 21:09:56 +00:00
Richard van der Hoff
56710c7df5
Fix 'no unique or exclusion constraint' error (#4591) 
Add more tables to the list of tables which need a background update to
complete before we can upsert into them, which fixes a race against the
background updates.
 2019-02-08 18:30:46 +00:00
Amber Brown
9cd33d2f4b
Deduplicate some code in synapse.app (#4567) 2019-02-08 17:25:57 +00:00
Hubert Chathi
afae8442b5 make sure version is in body and wrap in linearizer queue 
also add tests
 2019-02-08 01:32:45 -05:00
Hubert Chathi
d9e424bf64 re-try to make isort happy 2019-02-06 22:18:41 -05:00
Hubert Chathi
9ff620a518 fix import to make isort happy 2019-02-06 21:32:52 -05:00
Hubert Chathi
8248637173 add new endpoint to update backup versions 2019-02-06 17:57:10 -05:00
Hubert Chathi
664c81e8b7 return proper error codes for some 404s 2019-02-06 17:47:22 -05:00
Richard van der Hoff
2475434080 Merge branch 'master' into develop 2019-02-05 18:44:49 +00:00
Richard van der Hoff
3bd9daf4b8 v0.99.0 2019-02-05 18:33:02 +00:00
Andrew Morgan
627ecd358e Filter user directory state query to a subset of state events (#4462) 
* Filter user directory state query to a subset of state events

* Add changelog
 2019-02-05 12:16:28 +00:00
Richard van der Hoff
bf1e4d96ad
Fix default ACME config for py2 (#4564) 
Fixes #4559
 2019-02-05 11:37:33 +00:00
Richard van der Hoff
d7e27a1f08
fix typo in config comments (#4557) 2019-02-05 11:32:45 +00:00
Richard van der Hoff
30fd2f89db 0.99.0rc4 2019-02-01 15:52:28 +00:00
Richard van der Hoff
ef43a03fc5
Merge pull request #4546 from matrix-org/rav/silence_critical_error_from_federation 
Fix noisy "twisted.internet.task.TaskStopped" errors in logs
 2019-02-01 14:37:22 +00:00
Richard van der Hoff
f8db967d5a Merge remote-tracking branch 'origin/release-v0.99.0' into develop 2019-02-01 13:20:15 +00:00
Richard van der Hoff
fa794980ec
Merge pull request #4544 from matrix-org/rav/skip_invalid_well_known 
Treat an invalid .well-known the same as an absent one
 2019-02-01 13:18:36 +00:00
Richard van der Hoff
e9779a6f8f Fix b'ab' noise in logs 2019-02-01 12:34:31 +00:00
Richard van der Hoff
9763a73af0 Merge branch 'release-v0.99.0' into develop 2019-02-01 12:30:22 +00:00
Richard van der Hoff
f0ba34f581 Fix noisy "twisted.internet.task.TaskStopped" errors in logs 
Fixes #4003
 2019-02-01 12:22:57 +00:00
Richard van der Hoff
8a21b03fba Treat an invalid .well-known the same as an absent one 
... basically, carry on and fall back to SRV etc.
 2019-02-01 11:37:31 +00:00
Richard van der Hoff
3c8a41140e Cache failures to parse .well-known 
Also add a Measure block around the .well-known fetch
 2019-02-01 00:37:52 +00:00
Richard van der Hoff
24d59c7568 better logging for federation connections 2019-01-31 23:18:20 +00:00
Richard van der Hoff
d428b46346 Update federation routing logic to check .well-known before SRV 2019-01-31 23:14:18 +00:00
Andrew Morgan
c45fd0dda0
Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable 
Copy over non-federatable trait on room upgrade
 2019-01-31 20:28:37 +00:00
Richard van der Hoff
625385d684 Merge branch 'release-v0.99.0' into develop 2019-01-31 18:43:20 +00:00
Richard van der Hoff
85129d7068 v0.99.0rc3 2019-01-31 18:35:38 +00:00
Andrew Morgan
d239f67c25 Raise an exception instead of returning None 2019-01-31 18:34:15 +00:00
Richard van der Hoff
07dfe148de
Add some debug for membership syncing issues (#4538) 
I can't figure out what's going on with #4422 and #4436; perhaps this will help.
 2019-01-31 18:30:40 +00:00
Andrew Morgan
3ed3cb4339 New function for getting room's create event 2019-01-31 18:21:39 +00:00
Andrew Morgan
bbb97a35fd Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable 2019-01-31 18:11:00 +00:00
Richard van der Hoff
e707e7b38d
Fix infinite loop when an event is redacted in a v3 room (#4535) 2019-01-31 15:34:17 +00:00
Andrew Morgan
563f6a832b Reject large transactions on federation (#4513) 
* Reject large transactions on federation

* Add changelog

* lint

* Simplify large transaction handling
 2019-01-31 11:44:04 +00:00
Andrew Morgan
fb50934b8f lint 2019-01-31 11:34:45 +00:00
Richard van der Hoff
6fba9fd20c Merge remote-tracking branch 'origin/release-v0.99.0' into develop 2019-01-30 17:02:32 +00:00
Andrew Morgan
d621c5562e Copy over non-federatable trait on room upgrade 2019-01-30 16:33:51 +00:00
Richard van der Hoff
b8b898666e v0.99.0rc2 2019-01-30 16:31:07 +00:00
Matthew Hodgson
ad7ac8853c by default include m.room.encryption on invites (#3902) 
* by default include m.room.encryption on invites

* fix constant

* changelog
 2019-01-30 16:26:13 +00:00
Richard van der Hoff
c74b96755c Merge remote-tracking branch 'origin/develop' into release-v0.99.0 2019-01-30 16:23:28 +00:00
Neil Johnson
81b7e7eed3
Update constants.py 
remove trailing ,
 2019-01-30 16:11:36 +00:00
Erik Johnston
ed8c5e4cda Fix remote invite rejections not comming down sync 
This was broken in PR #4405, commit 886e5ac, where we changed remote
rejections to be outliers.

The fix is to explicitly add the leave event in when we know its an out
of band invite. We can't always add the event as if the server is/was in
the room there might be more events to send down the sync than just the
leave.
 2019-01-30 15:46:27 +00:00
Richard van der Hoff
a5d0c771a3 0.99.0rc1 2019-01-30 15:11:18 +00:00
Erik Johnston
6587b0b89b
Merge pull request #4472 from matrix-org/neilj/room_capabilities 
Server capabilities support
 2019-01-30 14:26:56 +00:00
Erik Johnston
a4f52a33fe Fix replication for room v3 (#4523) 
* Fix replication for room v3

We were not correctly quoting the path fragments over http replication,
which meant that it exploded when the event IDs had a slash in them

* Newsfile
 2019-01-30 14:19:52 +00:00
Richard van der Hoff
7615a8ced1 ACME config cleanups (#4525) 
* Handle listening for ACME requests on IPv6 addresses

the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses
without extra quoting. Building a string which you are about to parse again
seems like a weird choice. Let's just use listenTCP, which is consistent with
what we do elsewhere.

* Clean up the default ACME config

make it look a bit more consistent with everything else, and tweak the defaults
to listen on port 80.

* newsfile
 2019-01-30 14:17:55 +00:00
Erik Johnston
e87d7a4b0f Raise ConfigError instead 2019-01-30 12:48:09 +00:00
Erik Johnston
270f212a2a _listener_http should return a list 2019-01-30 12:14:50 +00:00
Richard van der Hoff
a79034aedf
Merge pull request #4521 from matrix-org/rav/fed_routing/cleanups 
Tiny .well-known fixes
 2019-01-30 11:47:24 +00:00
Richard van der Hoff
c7b24ac3d0
Follow redirects on .well-known (#4520) 2019-01-30 11:43:33 +00:00
Amber Brown
f6813919e8
SIGHUP for TLS cert reloading (#4495) 2019-01-30 11:00:02 +00:00
Richard van der Hoff
09a1a6b55e fix exception text 2019-01-30 10:58:52 +00:00
Richard van der Hoff
928c50b59a Also jitter the invalid cache period 2019-01-30 10:58:52 +00:00
Neil Johnson
b37e8c9572 Merge branch 'neilj/room_capabilities' of github.com:matrix-org/synapse into neilj/room_capabilities 2019-01-30 10:56:47 +00:00
Neil Johnson
f834d98402 isort 2019-01-30 10:55:42 +00:00
Richard van der Hoff
bc5f6e1797
Add a caching layer to .well-known responses (#4516) 2019-01-30 10:55:25 +00:00
Amber Brown
3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Neil Johnson
ee4df7fd7a
Merge branch 'develop' into neilj/room_capabilities 2019-01-30 10:28:08 +00:00
Neil Johnson
c5a0f82cca define room dispositions for use in exposing room capabilities 2019-01-30 10:24:24 +00:00
Neil Johnson
9c850d9d5e formatting and use constants where available 2019-01-30 10:23:26 +00:00
Neil Johnson
2f46804055 Populate default room version from Constants 2019-01-30 09:39:10 +00:00
Neil Johnson
c7837dce24 reflect that rooms v3 is a stable room version 2019-01-30 09:33:30 +00:00
Richard van der Hoff
457fbfaf22
Merge pull request #4486 from xperimental/workaround-4216 
Implement workaround for login error.
 2019-01-30 07:06:11 +00:00
Robert Jacob
2a7f0b8953 Implement workaround for login error. 
Signed-off-by: Robert Jacob <xperimental@solidproject.de>
 2019-01-30 01:06:39 +01:00
Erik Johnston
0b24d58e05 No vdh tests! 2019-01-29 23:11:48 +00:00
Erik Johnston
a1b0e1879b Enable room version v3 2019-01-29 23:09:10 +00:00
Erik Johnston
67b82f1336 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah 2019-01-29 22:58:38 +00:00
Erik Johnston
afeea319df Fixup comment 2019-01-29 22:55:29 +00:00
Erik Johnston
7740eddd04
Merge pull request #4514 from matrix-org/erikj/remove_event_id 
Remove usages of event ID's domain
 2019-01-29 22:54:25 +00:00
Erik Johnston
ff2f65d737 Update comment 2019-01-29 22:35:36 +00:00
Erik Johnston
655ce037fd check event format version not room version 2019-01-29 22:33:43 +00:00
Erik Johnston
f46a818ce5 kill vdh test some more 2019-01-29 22:02:58 +00:00
Erik Johnston
a696c48133 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah 2019-01-29 22:00:33 +00:00
Erik Johnston
c21b7cbc09
Update synapse/storage/events_worker.py 2019-01-29 21:53:48 +00:00
Erik Johnston
4db252c073 Check redaction state when event is pulled out of the database 2019-01-29 21:48:36 +00:00
Erik Johnston
6d23ec2111 Fix typo 2019-01-29 21:45:53 +00:00
Erik Johnston
b5d510ad64 Remove unused arg 2019-01-29 21:45:28 +00:00
Erik Johnston
6f9cdc2d47
Merge pull request #4483 from matrix-org/erikj/event_v2 
Implement event format V2
 2019-01-29 21:40:00 +00:00
Erik Johnston
47e2dd1994 Drop vdh support 2019-01-29 21:24:34 +00:00
Erik Johnston
38590a4870 Add docstring 2019-01-29 21:22:47 +00:00
Richard van der Hoff
82165eeb05
Update synapse/storage/events_worker.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-01-29 21:14:39 +00:00
Andrew Morgan
03b086647f
Merge pull request #4512 from matrix-org/anoa/consent_dir 
Check consent dir path on startup
 2019-01-29 20:08:18 +00:00
Erik Johnston
84af577356 Implement event format v2 2019-01-29 18:06:11 +00:00
Neil Johnson
e4bef9d470 rework format of change password capability 2019-01-29 18:04:56 +00:00
Erik Johnston
610f0830b0 Don't assert an event must have an event ID 2019-01-29 18:02:26 +00:00
Erik Johnston
840068bd78 Only check event ID domain for signatures for V1 events 
In future version events won't have an event ID, so we won't be able to
do this check.
 2019-01-29 18:02:02 +00:00
Erik Johnston
8e3d34e3c5 Use event origin for filtering incoming events 
We only process events sent to us from a server if the event ID matches
the server, to help guard against federation storms. We replace this
with a check against the event origin.
 2019-01-29 16:57:00 +00:00
Erik Johnston
55d9024835 Use snder and not event ID domain to check if ours 
The transaction queue only sends out events that we generate. This was
done by checking domain of event ID, but that can no longer be used.
Instead, we may as well use the sender field.
 2019-01-29 16:54:23 +00:00
Richard van der Hoff
cc2d650ef7
Relax requirement for a content-type on .well-known (#4511) 2019-01-29 16:49:17 +00:00
Erik Johnston
b1fffca345 Remove event ID usage when checking if new room 
The event ID is changing, so we can no longer get the domain from it. On
the other hand, the check is unnecessary.
 2019-01-29 16:15:02 +00:00
Erik Johnston
770b823445 Only check event IDs domain signed event for V1 and V2 
Since newer versions of events don't have the same format for event ID.
 2019-01-29 16:15:00 +00:00
Travis Ralston
d02c5ccb11
Merge pull request #4498 from matrix-org/travis/fix-docs-public_baseurl 
Don't recommend :8448 to people on public_baseurl
 2019-01-29 09:06:16 -07:00
Neil Johnson
f03b3a7a3a support change_password in capabilities end-point 2019-01-29 15:58:37 +00:00
Andrew Morgan
e65a17b26f Check consent dir path on startup 2019-01-29 15:30:33 +00:00
Amber Brown
6bd4374636
Do not generate self-signed TLS certificates by default. (#4509) 2019-01-29 14:09:10 +00:00
Erik Johnston
b8d75ef53e
Merge pull request #4481 from matrix-org/erikj/event_builder 
Refactor event building into EventBuilder
 2019-01-29 14:07:23 +00:00
Richard van der Hoff
99e36d5e24
Implement MSC1708 (.well-known lookups for server routing) (#4489) 2019-01-29 13:53:02 +00:00
Erik Johnston
b82a76c384 Finish comment... 2019-01-29 13:50:59 +00:00
Erik Johnston
2562319821
Merge pull request #4510 from matrix-org/erikj/fixup_compute_event_signature 
Fixup calls to `comput_event_signature`
 2019-01-29 13:35:19 +00:00
Erik Johnston
5891a6edc8 Correctly set context.app_service 2019-01-29 12:09:10 +00:00
Erik Johnston
fb99dae9c8 Don't set event_id twice 2019-01-29 12:08:23 +00:00
Erik Johnston
64c1bd1d21 Remove dead function 2019-01-29 12:06:28 +00:00
Erik Johnston
7709d2bd16 Implement rechecking of redactions 2019-01-29 11:56:20 +00:00
Erik Johnston
7a3ec5b022 Add RoomVersions.V3 constant, without enabling it 
We add the constant, but don't add it to the known room versions. This
lets us start adding V3 logic, but the servers will never join or create
V3 rooms
 2019-01-29 11:55:33 +00:00
Erik Johnston
6598992b01 Fixup calls to comput_event_signature 
We currently pass FrozenEvent instead of `dict` to
`compute_event_signature`, which works by accident due to `dict(event)`
producing the correct result.

This fixes PR #4493 commit 855a151
 2019-01-29 11:41:58 +00:00
Neil Johnson
a124025dab enforce auth for capabilities endpoint 2019-01-29 11:37:56 +00:00
Erik Johnston
ff37acb8ce
Merge pull request #4496 from matrix-org/erikj/invite_fallback 
Implement fallback for V2 invite API
 2019-01-29 11:28:23 +00:00
Erik Johnston
5180f12bae Replace usage of builder.user_id with builder.sender 
`.user_id` is proxed to `.sender` in FrozenEvent, so this has no
functional change
 2019-01-29 11:18:38 +00:00
Erik Johnston
be47cfa9c9 Refactor event building into EventBuilder 
This is so that everything is done in one place, making it easier to
change the event format based on room version
 2019-01-29 11:13:00 +00:00
Erik Johnston
554ca58ea1 Make add_hashes_and_signatures operate on dicts 2019-01-29 11:12:38 +00:00
Amber Brown
f815bd7feb
Make linearizer more quiet (#4507) 2019-01-29 11:05:31 +00:00
Erik Johnston
073f6c2e5e
Merge pull request #4494 from matrix-org/erikj/fixup_event_validator 
Split up event validation between event and builder
 2019-01-29 10:55:07 +00:00
Erik Johnston
40638ae7f5 Remove duplicate checks 2019-01-29 10:37:40 +00:00
Erik Johnston
9fa3c6ffa3 Fix up error messages 2019-01-29 10:36:46 +00:00
Erik Johnston
28efc80723 Fold validate into validate_new 2019-01-29 10:34:49 +00:00
Erik Johnston
b6b73a0bcf Fix receiving events from federation via a worker 
This bug was introduced in PR #4470, commit 678a92cb56
 2019-01-29 10:30:26 +00:00
Neil Johnson
327b992e17 register capabilities servlet 2019-01-29 10:28:35 +00:00
Amber Brown
94fb63e44f
Fix typo in upserts code (#4505) 
* fix obvious problem :|

* changelog
 2019-01-29 10:04:23 +00:00
Erik Johnston
17709f8f9c
Merge pull request #4493 from matrix-org/erikj/refactor_event_signing 
Refactor event signing to work on dicts
 2019-01-29 09:52:54 +00:00
Erik Johnston
f1a04462eb
Merge pull request #4482 from matrix-org/erikj/event_auth_room_version 
Pass through room version to event auth
 2019-01-28 20:09:38 +00:00
Travis Ralston
6901ac7e9d
Don't recommend :8448 to people on public_baseurl 2019-01-28 12:15:22 -07:00
Erik Johnston
d414f30019 Implement fallback for V2 invite API 
If the room version is either 1 or 2 then a server should retry failed
`/v2/invite` requests with the v1 API
 2019-01-28 17:33:25 +00:00
Erik Johnston
d758d5310e Correctly use default room version if none is set 2019-01-28 17:26:39 +00:00
Amber Brown
5d976c0c7c
Fix worker TLS (#4492) 
* load cert

* changelog

* fix
 2019-01-28 17:18:33 +00:00
Erik Johnston
b872c7b1b4 Split up event validation between event and builder 
The validator was being run on the EventBuilder objects, and so the
validator only checked a subset of fields. With the upcoming
EventBuilder refactor even fewer fields will be there to validate.

To get around this we split the validation into those that can be run
against an EventBuilder and those run against a fully fledged event.
 2019-01-28 17:00:14 +00:00
Erik Johnston
855a151015 Refactor event signing to work on dicts 
This is in preparation for making EventBuilder format agnostic, which
means event signing should be done against the event dict rather than
the EventBuilder object.
 2019-01-28 16:42:10 +00:00
Amber Brown
7072fe3084
Fix UPSERTs on SQLite 3.24+ (#4477) 2019-01-28 15:43:32 +00:00
Andrew Morgan
f0e96ab66a Change return syntax in doc string 2019-01-28 14:09:45 +00:00
Andrew Morgan
4026d555fa Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade 2019-01-28 14:08:24 +00:00
Andrew Morgan
1ce463963d Reuse predecessor method 2019-01-28 14:08:18 +00:00
Richard van der Hoff
3bd0f1a4a3 docstrings for _RoutingResult 2019-01-28 12:43:09 +00:00
Richard van der Hoff
0fd5b3b53e Handle IP literals explicitly 
We don't want to be doing .well-known lookups on these guys.
 2019-01-28 10:34:30 +00:00
Richard van der Hoff
51958df766 MatrixFederationAgent: factor out routing logic 
This is going to get too big and unmanageable.
 2019-01-28 10:34:30 +00:00
Richard van der Hoff
d840019192
Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487) 
Turns out that the library does a better job of parsing URIs than our
reinvented wheel. Who knew.

There are two things going on here. The first is that, unlike
parse_server_name, URI.fromBytes will strip off square brackets from IPv6
literals, which means that it is valid input to ClientTLSOptionsFactory and
HostnameEndpoint.

The second is that we stay in `bytes` throughout (except for the argument to
ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up
with idna-encoded values being held in `unicode` variables. TBH it probably
would have been ok but it made the tests fragile.
 2019-01-28 09:56:59 +00:00
Erik Johnston
ae2a957dba Pass through room version to event auth 2019-01-25 18:31:41 +00:00
Neil Johnson
893107be78 backout v3 2019-01-25 17:27:36 +00:00
Erik Johnston
b6dce9b9fd
Merge pull request #4470 from matrix-org/erikj/require_format_version 
Require event format version to parse or create events
 2019-01-25 15:59:36 +00:00
Erik Johnston
431e485914 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert 2019-01-25 14:11:17 +00:00
Richard van der Hoff
4a3f138832
Fix quoting for allowed_local_3pids example config (#4476) 
If you use double-quotes here, you have to escape your backslashes. It's much
easier with single-quotes.

(Note that the existing double-backslashes are already interpreted by python's
""" parsing.)
 2019-01-25 13:57:52 +00:00
Richard van der Hoff
8520bc3109
Fix Host header sent by MatrixFederationAgent (#4468) 
Move the Host header logic down here so that (a) it is used if we reuse the
agent elsewhere, and (b) we can mess about with it with .well-known.
 2019-01-25 12:38:16 +00:00
Andrew Morgan
9244a3089e Fixes 2019-01-25 11:48:38 +00:00
Andrew Morgan
da0d2219d2 Clean up direct_rooms access 2019-01-25 11:37:12 +00:00
Andrew Morgan
8265995498 Use python magic 2019-01-25 11:26:06 +00:00
Andrew Morgan
0b3fd1401f Don't require sqlite3 when using postgres (#4466) 2019-01-25 22:25:02 +11:00
Andrew Morgan
c4cdafa81f Destructure account data tuple before use 2019-01-25 11:24:28 +00:00
Andrew Morgan
516456b763 Remove unnecessary null check 2019-01-25 11:22:14 +00:00
Andrew Morgan
6f3fda79ce Move room_tag declaration to be closer to its use 2019-01-25 11:21:25 +00:00
Neil Johnson
95f871fc0d Support room version capabilities in CS API (MSC1804) 2019-01-25 11:16:29 +00:00
Andrew Morgan
821b65aeb5 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade 2019-01-25 11:09:53 +00:00
Andrew Morgan
0862d35b8e Move tag and direct state copying into separate function 2019-01-25 11:09:34 +00:00
Neil Johnson
53ef4da8c2 track unstable room v3 2019-01-25 11:04:11 +00:00
Andrew Morgan
b1b6dba2d2
Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms 
Ability to search entire room history after upgrading room
 2019-01-25 10:49:58 +00:00
Erik Johnston
1953067136 Disable native upserts for sqlite, as they don't work 2019-01-25 10:46:49 +00:00
Erik Johnston
678a92cb56 Replace missed usages of FrozenEvent 2019-01-25 10:32:30 +00:00
Erik Johnston
a50cf929c1 Require event format version to parse or create events 2019-01-25 10:32:19 +00:00
Erik Johnston
62514bb81b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/msc_1813 2019-01-25 10:07:08 +00:00
Erik Johnston
edc1e21dbe
Merge pull request #4448 from matrix-org/erikj/get_pdu_versions 
Add room_version param to get_pdu
 2019-01-24 18:47:15 +00:00
Erik Johnston
80bcca659e
Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites 
Store rejected remote invite events as outliers
 2019-01-24 18:33:27 +00:00
Erik Johnston
664b7a2920
Merge pull request #4437 from matrix-org/erikj/event_format_version_v2 
Add support for persisting event format versions
 2019-01-24 18:32:30 +00:00
Erik Johnston
d148c43050 Review comments 2019-01-24 18:31:23 +00:00
Erik Johnston
26f44164c8 Review comments 2019-01-24 18:28:00 +00:00
Erik Johnston
5ee1f997a8 Update make_membership_event docs 2019-01-24 18:08:08 +00:00
Erik Johnston
9139b87be4 Remove unecessary setting of outlier bit 2019-01-24 18:04:02 +00:00
Erik Johnston
b8082a5445 Use term 'out of band membership' instead 2019-01-24 17:33:19 +00:00
Andrew Morgan
a383289b0d Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms 2019-01-24 17:23:51 +00:00
Andrew Morgan
e1781b043b Remove redundant create event None check 2019-01-24 17:23:39 +00:00
Richard van der Hoff
03c85335d1
Apply suggestions from code review 
Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>
 2019-01-24 17:22:09 +00:00
Richard van der Hoff
8ea509a935
Update synapse/api/filtering.py 
Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>
 2019-01-24 17:21:35 +00:00
Andrew Morgan
50c396a7ee
Merge pull request #4461 from matrix-org/anoa/room_dir_quick_fix 
Add a 60s timeout to filtered room directory queries
 2019-01-24 15:24:35 +00:00
Andrew Morgan
075ff3ede9 Change default timeout value from 0 to None 2019-01-24 15:10:22 +00:00
Andrew Morgan
5b1dc94083 Use self.clock instead of datetime 2019-01-24 14:59:50 +00:00
Andrew Morgan
a2d85144e5 isort 2019-01-24 14:22:26 +00:00
Richard van der Hoff
4a6e863843
Merge pull request #4464 from matrix-org/rav/fix_srv_lookup 
MatrixFederationAgent: Look up the right SRV record
 2019-01-24 13:57:51 +00:00
Richard van der Hoff
f4697b5ec1
Fix UnboundLocalError in post_urlencoded_get_json (#4460) 
This could cause exceptions if the id server returned 4xx responses.
 2019-01-24 13:38:29 +00:00
Richard van der Hoff
afd69a0920 Look up the right SRV record 2019-01-24 13:31:43 +00:00
Neil Johnson
10b89d5c2e
Merge pull request #4435 from matrix-org/neilj/fix_threepid_auth_check 
Neilj/fix threepid auth check
 2019-01-24 13:02:50 +00:00
Andrew Morgan
5541645e80 lint 2019-01-24 12:45:32 +00:00
Andrew Morgan
068aa1d228 Time out filtered room dir queries after 60s 2019-01-24 12:44:27 +00:00
Neil Johnson
5c41b22359
Merge pull request #4458 from matrix-org/dbkr/public_baseurl_doc 
Clarify docs for public_baseurl
 2019-01-24 12:36:09 +00:00
Amber Brown
0e27501ee5
Fix UPSERT check (#4459) 2019-01-24 22:57:41 +11:00
David Baker
92d8a068ad Clarify docs for public_baseurl 
This is leading to problems with people upgrading to clients that
support MSC1730 because people have this misconfigured, so try
to make the docs completely unambiguous.
 2019-01-24 10:52:06 +00:00
Amber Brown
58f6c48183
Use native UPSERTs where possible (#4306) 2019-01-24 21:31:54 +11:00
Richard van der Hoff
97fd29c019
Don't send IP addresses as SNI (#4452) 
The problem here is that we have cut-and-pasted an impl from Twisted, and then
failed to maintain it. It was fixed in Twisted in
https://github.com/twisted/twisted/pull/1047/files; let's do the same here.
 2019-01-24 09:34:44 +00:00
Erik Johnston
7c288c2250 Clarify the invite flows 2019-01-23 20:07:47 +00:00
Erik Johnston
07f62da55a Remove unnecessary '_sign_event' 2019-01-23 20:04:57 +00:00
Erik Johnston
886e5acc76 Store rejected remote invite events as outliers 
Currently they're stored as non-outliers even though the server isn't in
the room, which can be problematic in places where the code assumes it
has the state for all non outlier events.

In particular, there is an edge case where persisting the leave event
triggers a state resolution, which requires looking up the room version
from state. Since the server doesn't have the state, this causes an
exception to be thrown.
 2019-01-23 20:04:57 +00:00
Willem Mulder
d528406cb8 Fix error message for optional dependencies 
Signed-off-by: Willem Mulder <willemmaster@hotmail.com>
 2019-01-23 18:44:57 +01:00
Erik Johnston
6a41d2a187 Add room_version param to get_pdu 
When we add new event format we'll need to know the event format or room
version when parsing events.
 2019-01-23 17:19:58 +00:00
Erik Johnston
67cd4dad81 Implement MSC 1813 - Add room version to make APIs 
We also implement `make_membership_event` converting the returned
room version to an event format version.
 2019-01-23 16:51:46 +00:00
Andrew Morgan
6b90ae6efc
Merge pull request #4445 from matrix-org/anoa/user_dir_develop_backport 
Add metric for user dir current event stream position
 2019-01-23 15:58:40 +00:00
Richard van der Hoff
a0ae475219
Merge pull request #4428 from matrix-org/rav/matrix_federation_agent 
Move SRV magic into an Agent-like thing
 2019-01-23 15:50:25 +00:00
Andrew Morgan
82a92ba535 Add metric for user dir current event stream position 2019-01-23 15:34:47 +00:00
Erik Johnston
be1065af59 isort 2019-01-23 11:48:16 +00:00
Erik Johnston
c5a296b10c Add support for persisting event format versions 
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.

We specify the format version rather than the room version as:

1. We don't necessarily know the room version, existing events may be
   either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
   events in the same room reported to be of different versions, which
   sounds annoying.
 2019-01-23 11:30:01 +00:00
Jason Robinson
6f680241bd Fix flake8 issues 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:53:48 +02:00
Erik Johnston
90743c9d89 Fixup removal of duplicate user_ips rows (#4432) 
* Remove unnecessary ORDER BY clause

* Add logging

* Newsfile
 2019-01-23 19:45:18 +11:00
Amber Brown
6129e52f43
Support ACME for certificate provisioning (#4384) 2019-01-23 19:39:06 +11:00
Jason Robinson
0516dc4d85 Remove openid resource from default config 
Instead document it commented out.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
d39b7b6d38 Document servlet_groups parameters 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
82e13662c0 Split federation OpenID userinfo endpoint out of the federation resource 
This allows the OpenID userinfo endpoint to be active even if the
federation resource is not active. The OpenID userinfo endpoint
is called by integration managers to verify user actions using the
client API OpenID access token. Without this verification, the
integration manager cannot know that the access token is valid.

The OpenID userinfo endpoint will be loaded in the case that either
"federation" or "openid" resource is defined. The new "openid"
resource is defaulted to active in default configuration.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
a17bac171f Make SynapseHomeServer _http_listener use self.get_reactor() 
For all the homeserver classes, only the FrontendProxyServer passes
its reactor when doing the http listen. Looking at previous PR's looks
like this was introduced to make it possible to write a test, otherwise
when you try to run a test with the test homeserver it tries to
do a real bind to a port. Passing the reactor that the homeserver
is instantiated with should probably be the right thing to do anyway?

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
4f8f41c824 Make FederationReaderServer _http_listen use self.get_reactor() 
For all the homeserver classes, only the FrontendProxyServer passes
its reactor when doing the http listen. Looking at previous PR's looks
like this was introduced to make it possible to write a test, otherwise
when you try to run a test with the test homeserver it tries to
do a real bind to a port. Passing the reactor that the homeserver
is instantiated with should probably be the right thing to do anyway?

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Jason Robinson
899e60be80 Add parameterized Python module to test dependencies 
Allows running parameterized tests. BSD license.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-01-23 10:32:41 +02:00
Richard van der Hoff
7021784d46 put resolve_service in an object 
this makes it easier to stub things out for tests.
 2019-01-22 20:35:12 +00:00
Richard van der Hoff
53a327b4d5 Require that service_name be a byte string 
it is only ever a bytes now, so let's enforce that.
 2019-01-22 20:35:12 +00:00
Richard van der Hoff
fe212bbe4a Kill off matrix_federation_endpoint 
this thing is now redundant.
 2019-01-22 20:34:35 +00:00
Richard van der Hoff
7871146667 Make MatrixFederationClient use MatrixFederationAgent 
... instead of the matrix_federation_endpoint
 2019-01-22 20:34:35 +00:00
Richard van der Hoff
44be7513bf MatrixFederationAgent 
Pull the magic that is currently in matrix_federation_endpoint and friends into
an agent-like thing
 2019-01-22 20:34:35 +00:00
Neil Johnson
c99c2d58d7 move guard out of is_threepid_reserved and into register.py 2019-01-22 17:47:00 +00:00
Neil Johnson
d619b113ed Fix None guard in config.server.is_threepid_reserved 2019-01-22 16:52:29 +00:00
Erik Johnston
12699a701f
Merge pull request #4434 from matrix-org/erikj/fix_user_ips_dedup 
Fix bug when removing duplicate rows from user_ips
 2019-01-22 16:51:57 +00:00
Erik Johnston
7f503f83b9 Refactor to rewrite the SQL instead 2019-01-22 16:31:05 +00:00
Neil Johnson
388c164aea
Merge pull request #4423 from matrix-org/neilj/disable_msisdn_on_registration 
Config option to disable requesting MSISDN on registration
 2019-01-22 16:23:08 +00:00
Erik Johnston
1c9704f8ab Don't shadow params 2019-01-22 16:20:33 +00:00
Andrew Morgan
766a172b99 lint 2019-01-22 13:51:40 +00:00
Erik Johnston
2557531f0f Fix bug when removing duplicate rows from user_ips 
This was caused by accidentally overwritting a `last_seen` variable
in a for loop, causing the wrong value to be written to the progress
table. The result of which was that we didn't scan sections of the table
when searching for duplicates, and so some duplicates did not get
deleted.
 2019-01-22 13:33:46 +00:00
Andrew Morgan
117bc94cd2 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade 2019-01-22 13:16:51 +00:00
Andrew Morgan
277e50462d Do not return in a deferred function 2019-01-22 12:40:26 +00:00
Andrew Morgan
c4875d8c76 Prevent duplicate room IDs in m.direct 2019-01-22 12:13:46 +00:00
Andrew Morgan
c433f61091 Ensure new filter is actually created 2019-01-22 12:06:36 +00:00
Andrew Morgan
c9bfb058d8 Fix a bug with single-room search searching all rooms 
* Create a new method for getting predecessor rooms
* Remove formatting change
 2019-01-22 12:00:41 +00:00
Andrew Morgan
8086a5c05e Fix comments 2019-01-22 11:16:23 +00:00
Richard van der Hoff
6bfa735a69
Make key fetches use regular federation client (#4426) 
All this magic is redundant.
 2019-01-22 11:04:20 +00:00
Andrew Morgan
48951f437f Join logic covers both room creator and arbitrary users 2019-01-22 11:00:04 +00:00
Andrew Morgan
8c85f0833d tags, m.direct copying over correctly 2019-01-22 11:00:04 +00:00
Andrew Morgan
25d64a846a Fix typos 2019-01-22 11:00:04 +00:00
Andrew Morgan
ea8903fcc9 Migrating dm and room tags work for migrator 2019-01-22 11:00:04 +00:00
Andrew Morgan
887ca93a1b Prevent crash on user who doesn't have any direct rooms 2019-01-22 11:00:04 +00:00
Andrew Morgan
4ff6d22245 Preserve DM status of a room on upgrade 
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
 2019-01-22 11:00:04 +00:00
Andrew Morgan
75942af1db Fix typo 2019-01-22 11:00:04 +00:00
Richard van der Hoff
33a55289cb
Refactor and bugfix for resove_service (#4427) 2019-01-22 10:59:27 +00:00
Amber Brown
23b0813599
Require ECDH key exchange & remove dh_params (#4429) 
* remove dh_params and set better cipher string
 2019-01-22 21:58:50 +11:00
Neil Johnson
1b53cc3cb4 fix line length 2019-01-21 15:17:20 +00:00
Neil Johnson
5349262302 Config option to disable requesting MSISDN on registration 2019-01-21 14:59:37 +00:00
Erik Johnston
35e1d67b4e Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_v2_invite_server 2019-01-21 14:04:19 +00:00
Erik Johnston
5f54765587
Merge pull request #4390 from matrix-org/erikj/versioned_fed_apis 
Add groundwork for new versions of federation APIs
 2019-01-21 11:44:05 +00:00
Andrew Morgan
702c4b750c Migrate encryption state on room upgrade (#4411) 
* Migrate encryption state on room upgrade

Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>

* Add changelog file
 2019-01-21 20:42:58 +11:00
Erik Johnston
25dd56ace3 Fix race when persisting create event (#4404) 
* Fix race when persisting create event

When persisting a chunk of DAG it is sometimes requried to do a state
resolution, which requires knowledge of the room version. If this
happens while we're persisting the create event then we need to use that
event rather than attempting to look it up in the database.
 2019-01-18 23:17:04 +11:00
Richard van der Hoff
de6888e7ce Remove redundant WrappedConnection (#4409) 
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
 2019-01-18 23:07:38 +11:00
Andrew Morgan
df3a661e4a Search for messages across predecessor rooms 
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
 2019-01-18 11:19:20 +00:00
Richard van der Hoff
676cf2ee26
Fix incorrect logcontexts after a Deferred was cancelled (#4407) 2019-01-17 14:00:23 +00:00
Richard van der Hoff
9feb5d0b71
sign_request -> build_auth_headers (#4408) 
Just got very confused about the fact that the headers are only an output, not
an input.
 2019-01-17 12:40:09 +00:00
Richard van der Hoff
3982a6ee07
Changing macaroon_secret_key no longer logs you out (#4387) 2019-01-16 23:14:41 +00:00
Richard van der Hoff
05e1296649
don't store more remote device lists if they have more than 1K devices (#4397) 2019-01-16 23:14:11 +00:00
Andrej Shadura
64cf6788d9
Depend on pymacaroons >= 0.13.0 instead on pymacaroons-pynacl 
Since 0.13.0, pymacaroons works correctly with pynacl, so there
isn’t any more reason to depend on an outdated pynacl fork.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
 2019-01-16 10:54:41 +01:00
Andrej Shadura
fab948120f
Use msgpack instead of msgpack-python 
The package msgpack-python has been deprecated.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
 2019-01-16 10:53:49 +01:00
Neil Johnson
9ec56d6935 ALL_USER_TYPES should be a tuple 2019-01-15 14:38:15 +00:00
Erik Johnston
4a4d2e17bc Add /v2/invite federation API 2019-01-15 13:22:44 +00:00
Erik Johnston
bb63e7ca4f Add groundwork for new versions of federation APIs 2019-01-15 11:14:34 +00:00
Amber Brown
a35c66a00b
Remove duplicates in the user_ips table and add an index (#4370) 2019-01-12 06:21:50 +11:00
Amber Brown
522dada206 Merge remote-tracking branch 'origin/master' into develop 2019-01-11 02:22:48 +11:00
Amber Brown
6dc06c3775 version 2019-01-11 01:56:37 +11:00
Richard van der Hoff
e0910d0145 Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 
Fixes #4371
 2019-01-10 14:12:50 +00:00
Richard van der Hoff
ba41aeed6a Revert "Fix macaroon_secret_key fallback logic" 
This is already fixed in 0.34.1, by 59f93bb

This reverts commit efc522c55e.
 2019-01-10 14:09:26 +00:00
Richard van der Hoff
aa70d24125 Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34 2019-01-10 12:58:33 +00:00
Richard van der Hoff
efc522c55e Fix macaroon_secret_key fallback logic 2019-01-10 12:57:27 +00:00
Richard van der Hoff
566947ff34 Skip macaroon check for access tokens in the db 2019-01-10 12:57:21 +00:00
Richard van der Hoff
353f2407b7 Fix fallback to signing key for macaroon-secret-key 2019-01-10 12:42:56 +00:00
Richard van der Hoff
d566e6b17a Merge branch 'master' into develop 2019-01-09 15:00:46 +00:00
Richard van der Hoff
998f5225c1 0.34.1 2019-01-09 14:53:54 +00:00
Erik Johnston
55c3e853c5
Merge pull request #4368 from matrix-org/erikj/better_errors 
Fixup docstrings for matrixfederationclient
 2019-01-09 11:39:28 +00:00
Erik Johnston
484867d35d Merge branch 'develop' into release-v0.34.1 2019-01-09 11:38:13 +00:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Erik Johnston
34ea14139d Fixup docstrings for matrixfederationclient 2019-01-09 09:25:59 +00:00
Erik Johnston
d422570e9b Use RequestSendFailed when fail to parse content type headers 2019-01-08 14:22:18 +00:00
Erik Johnston
1371d5b798 Don't log stack traces for HTTP error responses 2019-01-08 12:28:30 +00:00
Richard van der Hoff
ba85786d71 0.34.1rc1 2019-01-08 11:15:34 +00:00
Erik Johnston
b970cb0e96 Refactor request sending to have better excpetions (#4358) 
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
 2019-01-08 11:04:28 +00:00
Amber Brown
1dcb086f33
Fix synapse.config.__main__ on python 3 (#4356) 2019-01-08 10:03:09 +11:00
Travis Ralston
bc1fa8cd01 Add GET account data routes (#4303) 
As per https://github.com/matrix-org/matrix-doc/issues/1339
 2019-01-07 10:27:54 +00:00
Richard van der Hoff
b7c0218812
Check jinja version for consent resource (#4327) 
* Raise a ConfigError if an invalid resource is specified

* Require Jinja 2.9 for the consent resource

* changelog
 2019-01-07 10:14:31 +00:00
Michael Telatynski
a27e501b09 fix the check for whether is_url to match all the other ones in codebase (#3405) 
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
 2019-01-06 21:43:25 +00:00
Matthew Hodgson
27128145e6 fix NPE in /messages by checking if all events were filtered out (#4330) 2019-01-02 23:37:39 +00:00
Neil Johnson
84b6fae1f5 Ensure synchrotrons can access is_support_user in the storage layer 2019-01-02 10:19:59 +00:00
Amber Brown
d7843f47b6
Remove v1 only REST APIs now we don't ship matrix console (#4334) 2018-12-29 23:12:30 +11:00
Will Hunt
0708f437cc Log roomid along with Unknown room (#4297) 2018-12-24 10:49:42 +01:00
Richard van der Hoff
cd018e3543
Merge pull request #4307 from matrix-org/erikj/v2_rooms 
Add v2 room version
 2018-12-24 10:48:59 +01:00
Richard van der Hoff
f85676cc93
Return well_known in /login response (#4319) 
... as per MSC1730.
 2018-12-24 10:44:33 +01:00
Richard van der Hoff
9c2af7b2c5 Add a script to generate a clean config file (#4315) 2018-12-22 02:04:57 +11:00
Richard van der Hoff
f3561f8d86 Fix indentation in default config (#4313) 
These settings are not supposed to be under 'listeners'.
 2018-12-22 02:01:43 +11:00
Amber Brown
ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Amber Brown
c26f49a664
Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298) 2018-12-22 01:37:26 +11:00
Richard van der Hoff
0b26feb422 Merge branch 'master' into develop 2018-12-20 16:48:47 +00:00
Richard van der Hoff
6dd7271243 Prepare 0.34 release 2018-12-20 12:07:15 +00:00
Richard van der Hoff
d3c9c562c6 Debian packaging via dh_virtualenv 2018-12-20 11:35:23 +00:00
Richard van der Hoff
1a6d5bfa08 Debian packaging via dh_virtualenv (#4285) 2018-12-20 11:33:29 +00:00
David Baker
1c0051114a Add 'sandbox' to CSP for media repo (#4284) 
* Add 'sandbox' to the CSP for media repo

* Changelog
 2018-12-20 11:09:18 +00:00
Nathan Pennie
81b513416e
Fixed line length 2018-12-19 14:12:33 -05:00
Nathan Pennie
668e6625b0
Fix the variable names used for account_data 2018-12-19 06:53:02 -05:00
Erik Johnston
df89f8afb8 Add v2 room version 2018-12-18 18:10:37 +00:00
Neil Johnson
7e22cd90f5
ensure can report mau stats when hs.config.mau_stats_only is set (#4305) 
* ensure can report mau stats when hs.config.mau_stats_only is set
 2018-12-18 14:36:11 +00:00
Neil Johnson
d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Nad Chishtie
128294d3e5 Improved welcome page (#4294) 
* Improved static/index.html look, feel and content.

* Added accessibility tags to in-line SVG.

* Grammar.
 2018-12-14 05:13:56 +11:00
Amber Brown
c7144b105c Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:13:33 +11:00
Amber Brown
e93a0ebf50
Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:10:31 +11:00
Nad Chishtie
cef8ae272a Improved welcome page (#4294) 
* Improved static/index.html look, feel and content.

* Added accessibility tags to in-line SVG.

* Grammar.
 2018-12-14 02:42:51 +11:00
Richard van der Hoff
f208f608cb Merge branch 'release-v0.34.0' into develop 2018-12-11 15:43:20 +00:00
Richard van der Hoff
eb835bf65b 0.34.0rc2 2018-12-11 15:35:47 +00:00
Richard van der Hoff
188945713e
Merge pull request #4290 from matrix-org/rav/remove_webclient 
Stop installing Matrix Console by default
 2018-12-11 16:24:15 +01:00
Richard van der Hoff
95c1f6500b
Merge pull request #4289 from matrix-org/rav/welcome_page 
Add a welcome page to the static resources
 2018-12-11 16:24:06 +01:00
Richard van der Hoff
f537432ef9 Add a welcome page to the static resources 
This is largely a precursor for the removal of the bundled webclient. The idea
is to present a page at / which reassures people that something is working, and
to give them some links for next steps.

The welcome page lives at `/_matrix/static/`, so is enabled alongside the other
`static` resources (which, in practice, means the client API is enabled). We'll
redirect to it from `/` if we have nothing better to display there.

It would be nice to have a way to disable it (in the same way that you might
disable the nginx welcome page), but I can't really think of a good way to do
that without a load of ickiness.

It's based on the work done by @krombel for #2601.
 2018-12-11 13:26:22 +00:00
Richard van der Hoff
df96177ca7 Stop installing Matrix Console by default 
This is based on the work done by @krombel in #2601.
 2018-12-11 13:20:33 +00:00
David Baker
89ac2a5bdb Add 'sandbox' to CSP for media repo (#4284) 
* Add 'sandbox' to the CSP for media repo

* Changelog
 2018-12-11 04:05:02 +11:00
Will Hunt
fee831c040 Move imports to one line 2018-12-10 13:52:33 +00:00
Will Hunt
466c1f3e01
Use send_cors 2018-12-10 13:11:37 +00:00
Will Hunt
91206e09f2 changelog & isort 2018-12-09 17:39:44 +00:00
Will Hunt
dbf736ba66
Make /config more CORS-y 2018-12-09 13:27:22 +00:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Richard van der Hoff
c7401a697f
Implement SAML2 authentication (#4267) 
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.

What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
 2018-12-07 13:11:11 +01:00
Richard van der Hoff
c588b9b9e4
Factor SSO success handling out of CAS login (#4264) 
This is mostly factoring out the post-CAS-login code to somewhere we can reuse
it for other SSO flows, but it also fixes the userid mapping while we're at it.
 2018-12-07 13:10:07 +01:00
rkfg
ae19a7db8c Prevent crash on pagination. 2018-12-06 13:32:05 +03:00
Richard van der Hoff
b0c24a66ec Rip out half-implemented m.login.saml2 support (#4265) 
* Rip out half-implemented m.login.saml2 support

This was implemented in an odd way that left most of the work to the client, in
a way that I really didn't understand. It's going to be a pain to maintain, so
let's start by ripping it out.

* drop undocumented dependency on dateutil

It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 19:44:38 +11:00
Richard van der Hoff
9a3e24a13d drop undocumented dependency on dateutil (#4266) 
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 04:52:42 +11:00
Richard van der Hoff
e8d98466b0
Implement .well-known handling (#4262) 
Sometimes it's useful for synapse to generate its own .well-known file.
 2018-12-05 14:38:58 +01:00
Richard van der Hoff
e55983defe Prepare 0.34.0rc1 2018-12-04 13:52:16 +00:00
Travis Ralston
1737753a62 Add an option to enable recording IPs for appservice users (#3831) 2018-12-04 12:44:41 +01:00
Amber Brown
fd96dd75a3 Fix non-ASCII pushrules (#4248) 2018-12-04 12:44:02 +01:00
Travis Ralston
158ffb92f1 Add an option to disable search for homeservers which may not be interested in it (#4230) 
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
 2018-12-04 12:01:02 +01:00
Ben Parsons
b5ac0ffa0a add more detail to logging regarding "More than one row matched" error (#4234) 2018-12-04 11:57:39 +01:00
Richard van der Hoff
ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english (#4253) 
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
 2018-12-04 11:55:52 +01:00
Richard van der Hoff
f144c0a210
Merge pull request #4244 from aaronraimist/drop-sent-txt 
Drop sent_transactions
 2018-12-04 11:41:28 +01:00
Richard van der Hoff
a077e710a3
Merge pull request #4250 from matrix-org/hawkowl/pusher-remove-py3 
Fix removing pushers on python 3
 2018-12-04 11:22:46 +01:00
Richard van der Hoff
52e87fbfbe
Run the AS senders as background processes (#4189) 
This should fix some "Starting db connection from sentinel context" warnings,
and will mean we get metrics for these processes.
 2018-12-04 10:53:49 +01:00
Amber Brown
d3c61ef906 fix type error 2018-12-03 22:27:41 +11:00
Richard van der Hoff
c03324294d Workaround for non-ascii event ids (#4241) 
It turns out that we accept events with non-ascii IDs, which would later cause
an explosion during state res.

Fixes #4226
 2018-12-03 21:47:48 +11:00
Aaron Raimist
704c5298f0
Drop sent_transactions 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2018-12-01 23:07:35 -06:00
Neil Johnson
7039ece8fb Neilj/fix autojoin (#4223) 
* Fix auto join failures for servers that require user consent

* Fix auto join failures for servers that require user consent
 2018-11-28 22:24:57 +11:00
Neil Johnson
f9b136a886 Neilj/fix mau initial reserved users (#4211) 
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail

* towncrier

* white space
 2018-11-28 20:33:41 +11:00
Richard van der Hoff
944d524f18 Support m.login.sso (#4220) 
* Clean up the CSS for the fallback login form

I was finding this hard to work with, so simplify a bunch of things. Each
flow is now a form inside a div of class login_flow.

The login_flow class now has a fixed width, as that looks much better than each
flow having a differnt width.

* Support m.login.sso

MSC1721 renames m.login.cas to m.login.sso. This implements the change
(retaining support for m.login.cas for older clients).

* changelog
 2018-11-27 18:51:52 +11:00
Richard van der Hoff
6c18cc4b50 Ignore __pycache__ directories in schema delta dir 
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
 2018-11-20 22:52:34 +00:00
Neil Johnson
78ba0e7ab8 Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207) 2018-11-20 12:29:25 +01:00
Richard van der Hoff
416c671474
Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes 
Fix some logcontext leaks
 2018-11-20 12:19:19 +01:00
Amber Brown
31425d82a3 Merge remote-tracking branch 'origin/master' into develop 2018-11-19 12:55:25 -06:00
Amber Brown
d102e19e47 version 2018-11-19 12:42:49 -06:00
Amber Brown
80cac86b2c
Fix fallback auth on Python 3 (#4197) 2018-11-19 12:27:33 -06:00
Richard van der Hoff
f5faf6bc14 Fix logcontext leak in EmailPusher 2018-11-19 17:07:01 +00:00
Travis Ralston
ab4526a153 Remove duplicate slashes in generated consent URLs 2018-11-15 20:41:53 -07:00
Amber Brown
8b1affe7d5
Fix Content-Disposition in media repository (#4176) 2018-11-15 15:55:58 -06:00
Travis Ralston
835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Amber Brown
df758e155d
Use <meta> tags to discover the per-page encoding of html previews (#4183) 2018-11-15 11:05:08 -06:00
Neil Johnson
bf648c37e7 release 0.33.9rc1 2018-11-14 11:45:52 +00:00
Richard van der Hoff
c1efcd7c6a Add a test for the public T&Cs form 2018-11-14 10:46:27 +00:00
Richard van der Hoff
83a5f459aa Fix an internal server error when viewing the public privacy policy 2018-11-14 10:21:07 +00:00
David Baker
0869566ad3
Merge pull request #4113 from matrix-org/dbkr/e2e_backup_versions_are_numbers 
Make e2e backup versions numeric in the DB
 2018-11-14 07:55:48 +00:00
David Baker
bca3b91c2d Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers 2018-11-09 18:35:02 +00:00
Erik Johnston
db5a1c059a
Merge pull request #4166 from matrix-org/erikj/drop_unknown_events 
Drop incoming events from federation for unknown rooms
 2018-11-09 17:59:34 +00:00
Erik Johnston
dc59ad5334 Remove hack to support rejoining rooms 2018-11-09 14:58:09 +00:00
David Baker
d44dea0223 pep8 2018-11-09 14:38:31 +00:00
David Baker
4f93abd62d add docs 2018-11-09 13:25:38 +00:00
Erik Johnston
30dd27afff Simplify to always drop events if server isn't in the room 2018-11-09 11:36:45 +00:00
Richard van der Hoff
3cecf5340d
Update synapse/federation/federation_server.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2018-11-09 11:28:25 +00:00
Richard van der Hoff
9bce065a53
Update synapse/federation/federation_server.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2018-11-09 11:28:22 +00:00
David Baker
d3fa6194f7 Remove unnecessary str() 2018-11-09 11:11:31 +00:00
Erik Johnston
7b22421a7b
Merge pull request #4164 from matrix-org/erikj/fix_device_comparison 
Fix noop checks when updating device keys
 2018-11-08 14:37:20 +00:00
Erik Johnston
5ebed18692 Lets convert bytes to unicode instead 2018-11-08 12:33:13 +00:00
Erik Johnston
b1a22b24ab Fix noop checks when updating device keys 
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.

The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
 2018-11-08 12:18:38 +00:00
Erik Johnston
9417986f77 Drop PDUs of unknown rooms 
When we receive events over federation we will need to know the room
version to be able to correctly handle them, e.g. once we start changing
event formats. Currently, we attempt to handle events in unknown rooms.
 2018-11-08 12:11:20 +00:00
Richard van der Hoff
0a1fc52971 fix parse_string docstring 2018-11-08 11:12:29 +00:00
hera
2b075fb03a Fix encoding error for consent form on python3 
The form was rendering this as "b'01234....'".

-- richvdh
 2018-11-08 11:05:39 +00:00
Amber Brown
b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00
Travis Ralston
0f5e51f726 Add config variables for enabling terms auth and the policy name (#4142) 
So people can still collect consent the old way if they want to.
 2018-11-06 10:32:34 +00:00
Hubert Chathi
f1087106cf
handle empty backups according to latest spec proposal (#4123) 
fixes #4056
 2018-11-05 17:59:29 -05:00
Amber Brown
efdcbbe46b
Tests for user consent resource (#4140) 2018-11-06 05:53:44 +11:00
Erik Johnston
bc80b3f454 Add helpers for getting prev and auth events (#4139) 
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
 2018-11-06 00:35:15 +11:00
Amber Brown
0467384d2f
Set the encoding to UTF8 in the default logconfig (#4138) 2018-11-03 02:28:07 +11:00
Erik Johnston
90d713b8c6
Merge pull request #4137 from matrix-org/erikj/clean_up_events 
Clean up event accesses and tests
 2018-11-02 14:12:49 +00:00
Erik Johnston
b86d05a279 Clean up event accesses and tests 
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
 2018-11-02 13:44:14 +00:00
Amber Brown
cb7a6b2379
Fix typing being reset causing infinite syncs (#4127) 2018-11-03 00:19:23 +11:00
Richard van der Hoff
efb9343c8c
Merge pull request #4132 from matrix-org/rav/fix_device_list_locking 
Fix locked upsert on device_lists_remote_cache
 2018-11-02 10:50:53 +00:00
Richard van der Hoff
00f12e00f8
Merge pull request #4133 from matrix-org/travis/fix-terms-auth 
Fix logic error that prevented guests from seeing the privacy policy
 2018-11-02 10:50:43 +00:00
Erik Johnston
b199534518
Merge pull request #4135 from matrix-org/erikj/fix_state_res_none 
Fix None exception in state res v2
 2018-11-02 10:45:57 +00:00
Richard van der Hoff
350f654e7b Add unique indexes to a couple of tables 
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
 2018-11-02 10:36:13 +00:00
Richard van der Hoff
50e328d1e7 Remove redundant database locks for device list updates 
We can rely on the application-level per-user linearizer.
 2018-11-02 10:36:13 +00:00
Erik Johnston
54aec35867 Fix None exception in state res v2 2018-11-02 10:29:19 +00:00
Travis Ralston
642505abc3 Fix logic error that prevented guests from seeing the privacy policy 2018-11-01 16:48:32 -06:00
Travis Ralston
c68aab1536
Merge pull request #4004 from matrix-org/travis/login-terms 
Add m.login.terms to the registration flow
 2018-11-01 11:03:38 -06:00
Erik Johnston
1b21e771d0
Merge pull request #4128 from matrix-org/erikj/state_res_v2_version 
Add STATE_V2_TEST room version
 2018-11-01 13:17:57 +00:00
Erik Johnston
b3dd6fa981 Add STATE_V2_TEST room version 2018-11-01 11:43:46 +00:00
Amber Brown
073d400b84 Merge branch 'master' into develop 2018-11-01 21:32:12 +11:00
Amber Brown
aa98e38896 version 2018-11-01 21:28:35 +11:00
Travis Ralston
a8d41c6aff Include a version query string arg for the consent route 2018-10-31 13:19:28 -06:00
Travis Ralston
d1e7b9c44c Merge branch 'develop' into travis/login-terms 2018-10-31 13:15:14 -06:00
Richard van der Hoff
1729ba1650
Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades 
Attempt to move room aliases on room upgrades
 2018-10-31 17:52:18 +00:00
Richard van der Hoff
0f8591a5a8 Avoid else clause on exception for clarity 2018-10-31 15:43:57 +00:00
Richard van der Hoff
94c7fadc98 Attempt to move room aliases on room upgrades 2018-10-31 15:43:57 +00:00
Richard van der Hoff
9b827c40ca
Log some bits about event creation (#4121) 
I found these helpful in debugging my room upgrade tests.
 2018-10-31 15:42:23 +00:00
Amber Brown
916efc8249
Remove fetching keys via the deprecated v1 kex method (#4120) 2018-10-31 23:14:39 +11:00
Amber Brown
f79f454485
Remove deprecated v1 key exchange endpoint (#4119) 2018-10-31 22:29:02 +11:00
Amber Brown
67c1924899 version bump 2018-10-31 06:27:05 +11:00
Amber Brown
086e1a8f3e Fix search 500ing (#4122) 2018-10-31 06:26:46 +11:00
Amber Brown
3bade14ec0
Fix search 500ing (#4122) 2018-10-31 04:33:41 +11:00
Amber Brown
2e223a8c22
Remove the unused /pull federation API (#4118) 2018-10-31 04:24:59 +11:00
Erik Johnston
0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups 
Delete unreferenced state groups during purge
 2018-10-30 16:58:22 +00:00
Amber Brown
0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
David Baker
e0934acdbb Cast to int here too 2018-10-30 11:12:23 +00:00
David Baker
12941f5f8b Cast bacjup version to int when querying 2018-10-30 11:01:07 +00:00
David Baker
2f0f911c52 Convert version back to a string 2018-10-30 10:35:18 +00:00
David Baker
64fa557f80 Try & make it work on postgres 2018-10-30 09:51:04 +00:00
David Baker
563f9b61b1 Make e2e backup versions numeric in the DB 
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
 2018-10-29 21:01:22 +00:00
Erik Johnston
88e5ffe6fe Deduplicate device updates sent over replication 
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
 2018-10-29 17:34:34 +00:00
Erik Johnston
a163b748a5 Don't truncate command name in metrics 2018-10-29 17:34:21 +00:00
Erik Johnston
ad88460e0d Move _find_unreferenced_groups 2018-10-29 14:24:19 +00:00
Erik Johnston
664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston
f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston
b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Amber Brown
4cd1c9f2ff
Delete the disused & unspecced identicon functionality (#4106) 2018-10-29 23:57:24 +11:00
Richard van der Hoff
7fbfea062e
Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar 
Remember to copy the avatar on room upgrades
 2018-10-29 12:49:21 +00:00
Richard van der Hoff
56ca578f77
Merge pull request #4099 from matrix-org/rav/upgrade_odd_pls 
Better handling of odd PLs during room upgrades
 2018-10-29 12:48:51 +00:00
Richard van der Hoff
bf33eed609
Merge pull request #4091 from matrix-org/rav/room_version_upgrades 
Room version upgrade support
 2018-10-29 12:47:20 +00:00
Amber Brown
c4b3698a80
Make the replication logger quieter (#4108) 2018-10-29 22:59:44 +11:00
Amber Brown
2b791865c4 version bump 2018-10-29 21:52:52 +11:00
Richard van der Hoff
db24d7f15e Better handling of odd PLs during room upgrades 
Fixes handling of rooms where we have permission to send the tombstone, but not
other state. We need to (a) fail more gracefully when we can't send the PLs in
the old room, and (b) not set the PLs in the new room until we are done with
the other stuff.
 2018-10-27 00:54:26 +01:00
Richard van der Hoff
5caf79b312 Remember to copy the avatar on room upgrades 2018-10-26 23:56:40 +01:00
Richard van der Hoff
54bbe71867 optimise state copying 2018-10-26 22:51:34 +01:00
Richard van der Hoff
193cadc988 Address review comments 
Improve comments, get old room state from the context we already have
 2018-10-26 17:10:30 +01:00
Erik Johnston
03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Amber Brown
77d70a7646
Port register_new_matrix_user to Python 3 and add tests (#4085) 2018-10-26 22:05:22 +11:00
Richard van der Hoff
e6babc27d5 restrict PLs in old room 2018-10-25 19:18:25 +01:00
Richard van der Hoff
3a263bf3ae copy state 2018-10-25 19:18:25 +01:00
Richard van der Hoff
1b9f253e20 preserve PLs 2018-10-25 19:10:24 +01:00
Richard van der Hoff
4cda300058 preserve room visibility 2018-10-25 19:10:24 +01:00
Richard van der Hoff
0f7d1c9906 Basic initial support for room upgrades 
Currently just creates a new, empty, room, and sends a tombstone in the old
room.
 2018-10-25 19:10:24 +01:00
Richard van der Hoff
e1948175ee Allow power_level_content_override=None for _send_events_for_new_room 2018-10-25 19:10:24 +01:00
Richard van der Hoff
7f7b2cd3de Make room_member_handler a member of RoomCreationHandler 
... to save passing it into `_send_events_for_new_room`
 2018-10-25 19:10:18 +01:00
Richard van der Hoff
871c4abfec Factor _generate_room_id out of create_room 
we're going to need this for room upgrades.
 2018-10-25 18:23:09 +01:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Erik Johnston
c85e063302
Merge pull request #4051 from matrix-org/erikj/alias_disallow_list 
Add config option to control alias creation
 2018-10-25 17:04:59 +01:00
Neil Johnson
95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init 
fix race condiftion in calling initialise_reserved_users
 2018-10-25 16:33:40 +01:00
Neil Johnson
fcbd488e9a add new line 2018-10-25 16:13:43 +01:00
Erik Johnston
b94a43d5b5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list 2018-10-25 15:25:31 +01:00
Erik Johnston
e5481b22aa Use allow/deny 2018-10-25 15:25:21 +01:00
Neil Johnson
f8fe98812b improve comments 2018-10-25 14:58:59 +01:00
Neil Johnson
f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Travis Ralston
54def42c19 Merge branch 'develop' into travis/login-terms 2018-10-24 13:22:59 -06:00
Neil Johnson
ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson
663d9db8e7 commit transaction before closing 2018-10-24 17:17:30 +01:00
Neil Johnson
07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Neil Johnson
9ec2186586 isort 2018-10-24 16:09:21 +01:00
Neil Johnson
9532caf6ef remove trailing whiter space 2018-10-24 16:08:25 +01:00
Richard van der Hoff
94a49e0636
fix tuple 
Co-Authored-By: neilisfragile <neil@matrix.org>
 2018-10-24 14:39:23 +01:00
Richard van der Hoff
3ad359e5be Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation 2018-10-24 11:23:49 +01:00
Richard van der Hoff
7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8 
Fix a number of flake8 errors
 2018-10-24 11:23:35 +01:00
Erik Johnston
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase 
Add v2 state resolution algorithm
 2018-10-24 11:12:12 +01:00
Richard van der Hoff
7e07d25ed6 Allow backslashes in event field filters 
Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which
meant that single backslashes were not allowed in event field filters.

The intention here is to allow single-backslashes, but disallow
double-backslashes.
 2018-10-24 11:11:24 +01:00
Richard van der Hoff
ef771cc4c2 Fix a number of flake8 errors 
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
 2018-10-24 10:39:03 +01:00
Erik Johnston
47a9ba435d Use match rather than search 2018-10-24 09:54:57 +01:00
Richard van der Hoff
e0b9d5f0af
Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts 
Clean up the way logcontexts and threads work in the pushers
 2018-10-24 09:53:57 +01:00
Erik Johnston
dacbeb2e03 Comment 2018-10-24 09:47:49 +01:00
Erik Johnston
810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Erik Johnston
cb23aa4c42
Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation 
Clean up room alias creation
 2018-10-24 09:40:27 +01:00
Richard van der Hoff
c573794b22 Fix start_pushers vs _start_pushers confusion 2018-10-24 09:24:55 +01:00
Richard van der Hoff
e564306e31 sanity-check the is_processing flag 
... and rename it, for even more sanity
 2018-10-24 09:23:33 +01:00
Neil Johnson
a67d8ace9b remove errant exception and style 2018-10-23 17:44:39 +01:00
Travis Ralston
43c3f0b02f
Merge pull request #3969 from turt2live/travis/fix-federated-group-requests 
Handle HttpResponseException more safely for federated groups
 2018-10-23 10:41:04 -06:00
Travis Ralston
3e704822be
Comments help 2018-10-23 10:25:31 -06:00
Neil Johnson
329d18b39c remove white space 2018-10-23 15:27:20 +01:00
Neil Johnson
6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Richard van der Hoff
b3f6dddad2
Give some more things logcontexts (#4077) 2018-10-23 14:29:17 +01:00
Richard van der Hoff
5c445114d3
Correctly account for cpu usage by background threads (#4074) 
Wrap calls to deferToThread() in a thing which uses a child logcontext to
attribute CPU usage to the right request.

While we're in the area, remove the logcontext_tracer stuff, which is never
used, and afaik doesn't work.

Fixes #4064
 2018-10-23 13:12:32 +01:00
Richard van der Hoff
026cd91ac8 Run PusherPool.start as a background process 
We don't do anything with the result, so this is needed to give this code a
logcontext.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
f749607c91 Make on_started synchronous too 
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
e7a16c6210 Remove redundant run_as_background_process() from pusherpool 
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
c7273c11bc Give pushers their own background logcontext 
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
5110f4e425 move get_all_pushers call down 
simplifies the interface to _start_pushers
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
04277d0ed8 Factor PusherPool._start_pusher out of _start_pushers 
... and use it from start_pusher_by_id. This mostly simplifies
start_pusher_by_id.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
3e8b02c939 Rename _refresh_pusher 
This is public (or at least, called from outside the class), so ought to have a
better name.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
7aea00069c
Merge pull request #4076 from matrix-org/rav/fix_init_logcontexts 
Run MAU queries as background processes
 2018-10-22 14:46:59 +01:00
Richard van der Hoff
911db96658
Merge pull request #4073 from matrix-org/rav/require_psutil 
Make psutil an explicit dependency
 2018-10-22 12:33:21 +01:00
Richard van der Hoff
a6f421e812 Run MAU queries as background processes 
Fixes #3820
 2018-10-20 02:14:35 +01:00
Amber Brown
e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Richard van der Hoff
e5b52d0f94 Make psutil an explicit dependency 
As of #4027, we require psutil to be installed, so it should be in our
dependency list. We can also remove some of the conditional import code
introduced by #992.

Fixes #4062.
 2018-10-19 21:51:15 +01:00
Richard van der Hoff
f62c597d14
Merge pull request #4069 from matrix-org/rav/fix_email_templates_4065 
Calculate absolute path for email templates
 2018-10-19 16:33:54 +01:00
Erik Johnston
67f7b9cb50 pep8 2018-10-19 16:06:59 +01:00
Erik Johnston
056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00
Erik Johnston
47a9da28ca Batch process handling state groups 2018-10-19 15:48:15 +01:00
Richard van der Hoff
cc325c7069 Calculate absolute path for email templates 2018-10-19 14:01:59 +01:00
Amber Brown
e404ba9aac
Fix manhole on py3 (pt 2) (#4067) 2018-10-19 22:26:00 +11:00
Amber Brown
b69216f768
Make the metrics less racy (#4061) 2018-10-19 21:45:45 +11:00
Erik Johnston
9fafdfa97d Anchor returned regex to start and end of string 2018-10-19 10:22:45 +01:00
Erik Johnston
084046456e Add config option to control alias creation 2018-10-19 10:22:45 +01:00
Erik Johnston
74e7617083 Clean up room alias creation 2018-10-19 10:11:56 +01:00
Amber Brown
a36b0ec195 make a bytestring 2018-10-19 09:24:00 +11:00
Travis Ralston
49a044aa5f Merge branch 'develop' into travis/login-terms 2018-10-18 09:57:58 -06:00
Richard van der Hoff
c632bc8654 Merge branch 'master' into develop 2018-10-18 15:07:03 +01:00
Richard van der Hoff
926da4dda8 0.33.7 2018-10-18 14:57:32 +01:00
Richard van der Hoff
c7d0f34a3c v0.33.7rc2 2018-10-17 17:40:19 +01:00
Richard van der Hoff
0fd2321629 Fix incorrect truncation in get_missing_events 
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
 2018-10-17 17:35:26 +01:00
Richard van der Hoff
c8f2c19991 Put the warning blob at the top of the file 2018-10-17 16:56:22 +01:00
Richard van der Hoff
1519572961 Ship the email templates as package_data 
move the example email templates into the synapse package so that they can be
used as package data, which should mean that all of the packaging mechanisms
(pip, docker, debian, arch, etc) should now come with the example templates.

In order to grandfather in people who relied on the templates being in the old
place, check for that situation and fall back to using the defaults if the
templates directory does not exist.
 2018-10-17 16:46:02 +01:00
Erik Johnston
f6a0a02a62 Fix bug where we raised StopIteration in a generator 
This made python 3.7 unhappy
 2018-10-17 16:10:52 +01:00
Will Hunt
d6a7797dd1 Fix roomlist since tokens on Python 3 (#4046) 
Thanks @Half-Shot !!!
 2018-10-17 23:04:55 +11:00
Neil Johnson
c6584f4b5f clean up config error logic and imports 2018-10-17 11:36:41 +01:00
Richard van der Hoff
fc0f13dd03 Fix incorrect truncation in get_missing_events 
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
 2018-10-16 21:10:04 +01:00
Erik Johnston
4a28d3d36f Update event_auth table for rejected events 2018-10-16 16:28:42 +01:00
Erik Johnston
15133477ee Fix up use of resolve_events_with_factory 2018-10-16 16:28:42 +01:00
Erik Johnston
6bd856caa2 User event.sender rather than alias event.user_id 2018-10-16 16:16:13 +01:00
Erik Johnston
e238013c44 Add v2 state res algorithm. 
We hook this up to the vdh test room version.
 2018-10-16 16:16:13 +01:00
Richard van der Hoff
b8a5b0097c
Various cleanups in the federation client code (#4031) 
- Improve logging: log things in the right order, include destination and txids
  in all log lines, don't log successful responses twice

- Fix the docstring on TransportLayerClient.send_transaction

- Don't use treq.request, which is overcomplicated for our purposes: just use a
  twisted.web.client.Agent.

- simplify the logic for setting up the bodyProducer

- fix bytes/str confusions
 2018-10-16 10:44:49 +01:00
Travis Ralston
a8ed93a4b5 pep8 2018-10-15 16:10:29 -06:00
Travis Ralston
442734ff9e Ensure the terms params are actually provided 2018-10-15 14:56:13 -06:00
Travis Ralston
762a0982aa Python is hard 2018-10-15 14:46:09 -06:00
Travis Ralston
f293d124b6 Merge branch 'develop' into travis/login-terms 2018-10-15 14:44:32 -06:00
Amber Brown
f726f2dc6c version bump 2018-10-15 22:21:45 +11:00
David Baker
03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups 
E2E backups
 2018-10-15 10:19:43 +01:00
Ivan Shapovalov
06bc8d2fe5 synapse/app: frontend_proxy.py: actually make workers work on py3 2018-10-14 20:08:39 +03:00
Ivan Shapovalov
fb216a22db synapse/visibility.py: fix SyntaxError on py3.7 2018-10-14 20:08:17 +03:00
Neil Johnson
1ccafb0c5e no need to join room if creator 2018-10-13 21:14:21 +01:00
Travis Ralston
dd99db846d Update login terms structure for the proposed language support 2018-10-12 18:03:27 -06:00
Travis Ralston
5119818e9d Rely on the lack of ?u to represent public access 
also general cleanup
 2018-10-12 18:03:17 -06:00
Travis Ralston
7ede650956 Merge branch 'develop' into travis/login-terms 2018-10-12 16:24:07 -06:00
Travis Ralston
164f8e4843 isort 2018-10-12 15:11:59 -06:00
Travis Ralston
7bb651de6a More sane handling of group errors and pep8 2018-10-12 14:53:30 -06:00
Travis Ralston
e3586f7c06 Merge branch 'develop' into travis/fix-federated-group-requests 2018-10-12 14:49:58 -06:00
Neil Johnson
a2bfb778c8 improve auto room join logic, comments and tests 2018-10-12 18:17:36 +01:00
David Baker
a45f2c3a00 missed one 2018-10-12 14:33:55 +01:00
Amber Brown
381d2cfdf0
Make workers work on Py3 (#4027) 2018-10-13 00:14:08 +11:00
Erik Johnston
67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
David Baker
8c0ff0287a Linting soothes the savage PEP8 monster 2018-10-12 13:47:43 +01:00
David Baker
306361b31b Misc PR feedback bits 2018-10-12 11:48:56 +01:00
David Baker
bddfad253a Don't mangle exceptions 2018-10-12 11:48:02 +01:00
David Baker
86ef9760a7 Split /room_keys/version into 2 servlets 2018-10-12 11:35:08 +01:00
David Baker
83e72bb2f0 PR feedback pt. 1 2018-10-12 11:26:18 +01:00
Richard van der Hoff
8ddd0f273c Comments on get_all_new_events_stream 
just some docstrings to clarify the behaviour here
 2018-10-12 09:55:41 +01:00
Erik Johnston
7e561b5c1a Add description to counter metric 2018-10-10 11:41:15 +01:00
Erik Johnston
3cbe8331e6 Track number of non-empty sync responses instead 2018-10-10 11:23:17 +01:00
Erik Johnston
395276b405 Append _total to metric and fix up spelling 2018-10-10 09:24:39 +01:00
David Baker
b8d9e108be Fix mergefail 2018-10-09 18:04:21 +01:00
Erik Johnston
bdc27d6716 Add metric to count lazy member sync requests 2018-10-09 14:17:52 +01:00
David Baker
d3464ce708 isort 2018-10-09 10:33:59 +01:00
Erik Johnston
9eb1a79100
Merge pull request #4008 from matrix-org/erikj/log_looping_exceptions 
Log looping call exceptions
 2018-10-09 10:12:10 +01:00
David Baker
dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker
f4a4dbcad1 Apparently this blank line is Very Important 2018-10-09 09:47:04 +01:00
David Baker
0c905ee015 be python3 compatible 2018-10-09 09:39:13 +01:00
Erik Johnston
6982320572 Remove unnecessary extra function call layer 2018-10-08 14:06:19 +01:00