Commit Graph

10717 Commits

Author SHA1 Message Date
Erik Johnston
898dde981b Add received_ts column to redactions.
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
Erik Johnston
a27fb7d5ca Don't repeatedly attempt to censor events we don't have.
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.

When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
Erik Johnston
1d349fb159 Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop 2019-10-01 10:17:24 +01:00
Erik Johnston
9267741a5f Fix devices_last_seen background update.
Fixes #6134.
2019-09-30 11:58:36 +01:00
Richard van der Hoff
16cb9a71b8
Drop unused tables (#6115)
These tables are unused since #5893 (as amended by #6047), so we can now drop
them.

Fixes #6048.
2019-09-30 09:38:41 +01:00
Andrew Morgan
f3451118a6
Edit SimpleHttpClient to reference that header keys can be passed as str or bytes (#6077) 2019-09-27 17:59:18 +01:00
Brendan Abolivier
3e42d47a5a
Incorporate review 2019-09-27 16:15:01 +01:00
Brendan Abolivier
af92110c46
Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:12:15 +01:00
Brendan Abolivier
bbe2a0f339
Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:36 +01:00
Brendan Abolivier
873fe7883c
Lint 2019-09-27 15:21:03 +01:00
Brendan Abolivier
72a2708ac6
Fixup and add some logging 2019-09-27 15:13:39 +01:00
Brendan Abolivier
5257a2fb1c
Reject pending invites on deactivation 2019-09-27 14:49:53 +01:00
Andrew Morgan
8c27bc8b60
Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978)
Just to have all the methods that make calls to identity services in one place.
2019-09-27 10:36:20 +01:00
Erik Johnston
67ee18daea
Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span
Add some helpful opentracing tags and remove get_user_by_req span
2019-09-27 10:02:57 +01:00
Erik Johnston
3423633d50 Fix 'redaction_retention_period' sampel config to match guidelines 2019-09-26 16:43:52 +01:00
Richard van der Hoff
54569c787b
Kill off half-implemented password-reset via sms (#6101)
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.

See also matrix-org/matrix-doc#2303.
2019-09-26 15:38:25 +01:00
Neil Johnson
8b8f8c7b3c Explicitly log when a homeserver does not have a trusted key server configured (#6090) 2019-09-26 12:57:01 +01:00
Richard van der Hoff
2927c6bc4c bump version 2019-09-26 12:29:59 +01:00
Neil Johnson
034db2ba21 Fix dummy event insertion consent bug (#6053)
Fixes #5905
2019-09-26 11:47:53 +01:00
Richard van der Hoff
a96318127d Update comments and docstring 2019-09-25 18:17:39 +01:00
Erik Johnston
4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston
9614d3c9d1
Merge pull request #6089 from matrix-org/erikj/cleanup_user_ips
Move last seen info into devices table
2019-09-25 17:42:39 +01:00
Neil Johnson
a4f3ca48b5 Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. (#5884) 2019-09-25 17:27:35 +01:00
Erik Johnston
39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Erik Johnston
d2bd0bc6b1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips 2019-09-25 17:16:28 +01:00
Erik Johnston
50572db837
Use if is not None
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
Neil Johnson
77dc7093a7 Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. (#6104)
Fixes #6103
2019-09-25 12:29:35 +01:00
Richard van der Hoff
990928abde
Stop advertising unsupported flows for registration (#6107)
If email or msisdn verification aren't supported, let's stop advertising them
for registration.

Fixes #6100.
2019-09-25 12:10:26 +01:00
Erik Johnston
dc2c97e1a3 isort 2019-09-25 11:59:05 +01:00
Erik Johnston
dc01cad690 Add device and appservice tags 2019-09-25 11:59:00 +01:00
Erik Johnston
5d99713854 Add tags for event_id and txn_id in event sending
This will make it easier to search for sending event requests.
2019-09-25 11:39:15 +01:00
Richard van der Hoff
2cd98812ba
Refactor the user-interactive auth handling (#6105)
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.

This is also preparation for some more intelligent advertising of flows, as per #6100
2019-09-25 11:33:03 +01:00
Erik Johnston
fde4ce2213 Don't create new span for get_user_by_req
We don't actually care about what happens in `get_user_by_req` and
having it as a separate span means that the entity tag isn't added to
the servlet spans, making it harder to search.
2019-09-25 11:32:41 +01:00
Richard van der Hoff
8004d6ca2f
Refactor code for calculating registration flows (#6106)
because, frankly, it looked like it was written by an axe-murderer.

This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
2019-09-25 11:32:05 +01:00
Brendan Abolivier
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
Richard van der Hoff
9b7c4f4078 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 17:03:50 +01:00
Richard van der Hoff
566ac40939
remove unused parameter to get_user_id_by_threepid (#6099)
Added in #5377, apparently in error
2019-09-24 17:01:09 +01:00
Erik Johnston
242017db8b Prune rows in user_ips older than configured period
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
Erik Johnston
2135c198d1 Add has_completed_background_update
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Erik Johnston
367158a609 Add wrap_as_background_process decorator.
This does the same thing as `run_as_background_process` but means we
don't need to create superfluous functions.
2019-09-24 15:53:17 +01:00
Richard van der Hoff
bb82be9851
Merge pull request #6069 from matrix-org/rav/fix_attribute_mapping
Fix a bug with saml attribute maps.
2019-09-24 15:07:26 +01:00
Brendan Abolivier
12fe2a29bc
Incorporate review 2019-09-24 14:43:38 +01:00
J. Ryan Stinnett
40fb00f5b7
Add sid to next_link for email validation (#6097) 2019-09-24 14:39:50 +01:00
Brendan Abolivier
f02f14e09a
Fix logging 2019-09-24 14:39:07 +01:00
Richard van der Hoff
ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Richard van der Hoff
a25b66d3f9 docstrings and comments 2019-09-24 11:15:08 +01:00
Andrew Morgan
50776261e1 Add submit_url response parameter to msisdn /requestToken (#6079)
Second part of solving #6076
Fixes #6076

We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
2019-09-23 21:21:03 +01:00
Andrew Morgan
2b071a2ff1 Add an unstable feature flag for separate add/bind 3pid APIs (#6044)
Add a m.separate_add_and_bind flag set to True. See MSC2290's Backward Compatibility section for details.
2019-09-23 20:46:34 +01:00
Richard van der Hoff
78e8ec368e
Merge pull request #6064 from matrix-org/rav/saml_config_cleanup
Make the sample saml config closer to our standards
2019-09-23 20:36:51 +01:00
Andrew Morgan
e08ea43463 Use the federation blacklist for requests to untrusted Identity Servers (#6000)
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.

Fixes #5935
2019-09-23 20:23:20 +01:00
Andrew Morgan
1ea3ed7620 Add m.id_access_token to /versions unstable_features (MSC2264) (#5974)
Adds a flag to /versions' unstable_features section indicating that this Synapse understands what an id_access_token is, as per MSC2264.

Fixes #5927
2019-09-23 20:19:25 +01:00
Andrew Morgan
b38aa82b83 Add m.require_identity_server to /versions unstable_flags (#5972)
As MSC2263 states, m.require_identity_server must be set to false when it does not require an identity server to be provided by the client for the purposes of email registration or password reset.

Adds an m.require_identity_server flag to /versionss unstable_flags section. This will advertise that Synapse no longer needs id_server as a parameter.
2019-09-23 18:52:43 +01:00
Andrew Morgan
2c99c63453 Add POST submit_token endpoint for MSISDN (#6078)
First part of solving #6076
2019-09-23 17:49:00 +01:00
Brendan Abolivier
323d685bf7
Typo 2019-09-23 17:23:49 +01:00
Brendan Abolivier
2858d10671
Fix the return value in the users_set_deactivated_flag background job 2019-09-23 17:22:01 +01:00
Erik Johnston
51d28272e2 Query devices table for last seen info.
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.

The storage functions now no longer return the access_token, since it
was unused.
2019-09-23 16:59:45 +01:00
Erik Johnston
ed80231ade Add BG update to populate devices last seen info 2019-09-23 16:59:45 +01:00
Erik Johnston
2ade05dca3 Add last seen info to devices table.
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
Andrew Morgan
30af161af2 Implement MSC2290 (#6043)
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.

This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints.

Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
Andrew Morgan
1b519e0272 Disable /register/available if registration is disabled (#6082)
Fixes #6066

This register endpoint should be disabled if registration is disabled, otherwise we're giving anyone the ability to check if a username exists on a server when we don't need to be.

Error code is 403 (Forbidden) as that's the same returned by /register when registration is disabled.
2019-09-23 15:38:38 +01:00
Andrew Morgan
1c9feadf4b Generalize email sending logging (#6075)
In ancient times Synapse would only send emails when it was notifying a user about a message they received...

Now it can do all sorts of neat things!

Change the logging so it's not just about notifications.
2019-09-23 14:38:19 +01:00
Andrew Morgan
885a4726b7 Return timeout error to user for identity server calls (#6073) 2019-09-23 14:37:23 +01:00
Richard van der Hoff
0660f8e367
Merge pull request #6072 from matrix-org/rav/fix_retry_reset
Fix exception when resetting retry timings
2019-09-23 14:31:47 +01:00
Matthew Hodgson
7ef319aefe fix broken copyrights 2019-09-23 12:28:01 +01:00
Andrew Morgan
df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Andrew Morgan
7763dd3e95
Remove trailing slash ability from password reset's submit_token endpoint (#6074)
Remove trailing slash ability from the password reset submit_token endpoint. Since we provide the link in an email, and have never sent it with a trailing slash, there's no point for us to accept them on the endpoint.
2019-09-20 14:58:37 +01:00
Richard van der Hoff
9d94313209 Fix exception when resetting retry timings
Fixes:
> TypeError: set_destination_retry_timings() missing 1 required positional
argument: 'retry_interval'

Introduced in #6016.
2019-09-20 12:09:39 +01:00
Richard van der Hoff
366dc7d2d4 Merge commit '33757bad1' into rav/saml_mapping_work 2019-09-20 11:21:40 +01:00
Richard van der Hoff
33757bad19 More better logging 2019-09-20 11:20:02 +01:00
Andrew Morgan
aeb40f355c
Ensure email validation link parameters are URL-encoded (#6063)
The validation links sent via email had their query parameters inserted without any URL-encoding. Surprisingly this didn't seem to cause any issues, but if a user were to put a `/` in their client_secret it could lead to problems.
2019-09-20 10:46:59 +01:00
Andrew Morgan
3ac614eb6c
Drop support for bind param on POST /account/3pid (MSC2290) (#6067)
As per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/files#diff-05cde9463e9209b701312b3baf2fb2ebR151), we're dropping the bind parameter from `/account/3pid`. This endpoint can now only be used for adding threepid's to the user's account on the homeserver.
2019-09-20 10:46:34 +01:00
J. Ryan Stinnett
36015d68ef Use unstable prefix for 3PID unbind API (#6062) 2019-09-19 22:28:29 +01:00
Richard van der Hoff
284e1cb027 Merge branch 'develop' into rav/fix_attribute_mapping 2019-09-19 20:32:25 +01:00
Richard van der Hoff
b74606ea22 Fix a bug with saml attribute maps.
Fixes a bug where the default attribute maps were prioritised over
user-specified ones, resulting in incorrect mappings.

The problem is that if you call SPConfig.load() multiple times, it adds new
attribute mappers to a list. So by calling it with the default config first,
and then the user-specified config, we would always get the default mappers
before the user-specified mappers.

To solve this, let's merge the config dicts first, and then pass them to
SPConfig.
2019-09-19 20:32:14 +01:00
Richard van der Hoff
b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff
7423fade92 better logging 2019-09-19 17:16:50 +01:00
Richard van der Hoff
bcd9132869
Undo the deletion of some tables (#6047)
This is a partial revert of #5893. The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.

So let's leave the tables in place for now, and remember to drop them in a
subsequent release.

(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
2019-09-19 15:06:27 +01:00
Richard van der Hoff
b789c7eb03 Merge branch 'develop' into rav/saml_config_cleanup 2019-09-19 15:05:31 +01:00
Andrew Morgan
62e3ff92fd
Remove POST method from password reset submit_token endpoint (#6056)
Removes the POST method from `/password_reset/<medium>/submit_token/` as it's only used by phone number verification which Synapse does not support yet.
2019-09-19 10:53:14 +01:00
Jorik Schellekens
38fd1f8e3f Fix typo in account_threepid_delegates config (#6028) 2019-09-18 22:30:44 +01:00
J. Ryan Stinnett
a86a290850 Fix logcontext spam on non-Linux platforms (#6059)
This checks whether the current platform supports thread resource usage tracking
before logging a warning to avoid log spam.

Fixes https://github.com/matrix-org/synapse/issues/6055
2019-09-18 21:55:37 +01:00
Andrew Morgan
6670bd4072
v2 3PID Invites (part of MSC2140) (#5979)
3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is.

These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens).

Requires: ~~#5976~~
2019-09-17 18:05:13 +01:00
dstipp
379d2a8c39 (#5849) Convert rst to markdown (#6040)
Converting some of the rst documentation to markdown.  Attempted to
preserve whitespace and line breaks to minimize cosmetic change.
2019-09-17 12:55:29 +01:00
Erik Johnston
70c52821ce Fix race condition in room stats. (#6029)
Broke in #5971

Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)
2019-09-17 12:41:23 +01:00
Richard van der Hoff
1e19ce00bf
Add 'failure_ts' column to 'destinations' table (#6016)
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
Amber Brown
850dcfd2d3
Fix well-known lookups with the federation certificate whitelist (#5997) 2019-09-14 04:58:38 +10:00
Richard van der Hoff
a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Richard van der Hoff
785cbd3999 Make the sample saml config closer to our standards
It' still not great, thanks to the nested dictionaries, but it's better.
2019-09-13 12:07:03 +01:00
Amber Brown
b617864cd9
Fix for structured logging tests stomping on logs (#6023) 2019-09-13 02:29:55 +10:00
Richard van der Hoff
3d882a7ba5
Remove the cap on federation retry interval. (#6026)
Essentially the intention here is to end up blacklisting servers which never
respond to federation requests.

Fixes https://github.com/matrix-org/synapse/issues/5113.
2019-09-12 13:00:13 +01:00
Richard van der Hoff
0388beafe4
Fix bug in calculating the federation retry backoff period (#6025)
This was intended to introduce an element of jitter; instead it gave you a
30/60 chance of resetting to zero.
2019-09-12 12:59:43 +01:00
David Baker
642fad8bd4 Fix SSO fallback login
Well, it worked, but forgot to remove the thing saying login was
unavailable.
2019-09-12 11:42:47 +01:00
Sorunome
dd2e5b0038 add report_stats_endpoint config option (#6012)
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
2019-09-12 11:24:57 +01:00
Jorik Schellekens
a8251da10f
Blow up config if opentracing is missing (#5985)
* Blow up config if opentracing is missing
2019-09-12 10:57:37 +01:00
Jason Robinson
6d847d8ce6 Ensure support users can be registered even if MAU limit is reached
This allows support users to be created even on MAU limits via
the admin API. Support users are excluded from MAU after creation,
so it makes sense to exclude them in creation - except if the
whole host is in disabled state.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-11 20:48:31 +03:00
Andrew Morgan
9fc71dc5ee
Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)
This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted.

Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140).

Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
2019-09-11 16:02:42 +01:00
Erik Johnston
cbcbfe64a2
Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction
Allow use of different ratelimits for admin redactions.
2019-09-11 15:39:38 +01:00
Richard van der Hoff
7902bf1e1d
Clean up some code in the retry logic (#6017)
* remove some unused code
* make things which were constants into constants for efficiency and clarity
2019-09-11 15:14:56 +01:00
Andrew Morgan
9c555f37e3
Add note about extra arg to send_membership_event, remove arg in remote_reject_invite (#6009)
Some small fixes to `room_member.py` found while doing other PRs.

1. Add requester to the base `_remote_reject_invite` method.
2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.
2019-09-11 14:23:24 +01:00
Jorik Schellekens
6604b64fae
Check dependencies on setup in the nicer way. (#5989) 2019-09-11 14:00:37 +01:00