Hubert Chathi
691dd67fcd
Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version
...
make version optional in body of e2e backup version update
2019-10-11 10:11:59 -04:00
Erik Johnston
6f5c6c8f60
Merge pull request #6195 from matrix-org/erikj/opentracing_preview_url
...
Trace non-JSON APIs, /media, /key etc
2019-10-11 13:23:52 +01:00
Valérian Rousset
be9b55e0d2
cas: support setting display name ( #6114 )
...
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Erik Johnston
dfbb62c28d
Port synaps/rest/client/users.py to async/await
2019-10-11 12:21:40 +01:00
Erik Johnston
f95325e22a
Port synaps/rest/client/server_notice_servlet.py to async/await
2019-10-11 12:21:40 +01:00
Erik Johnston
281f887090
Port synaps/rest/client/media.py to async/await
2019-10-11 12:21:40 +01:00
Erik Johnston
fca3a541e7
Port rest/admin/__init__.py to async/await
2019-10-11 12:17:52 +01:00
Erik Johnston
f3ceaf4323
Trace non-JSON APIs, /media, /key etc
2019-10-11 11:58:52 +01:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. ( #6168 )
2019-10-11 09:38:26 +01:00
Hubert Chathi
7a0dce9259
make sure we actually return something
2019-10-10 20:31:30 -04:00
Hubert Chathi
4908fb3b30
make storage layer in charge of interpreting the device key data
2019-10-10 19:15:30 -04:00
Erik Johnston
83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any
...
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
Erik Johnston
b54b1e759a
Fix SQLite take 2
2019-10-10 16:19:40 +01:00
Erik Johnston
afb6d9d53b
Fix SQLite
2019-10-10 15:55:41 +01:00
Erik Johnston
3bc687508f
Remove add_in_list_sql_clause
2019-10-10 15:35:46 +01:00
Erik Johnston
ca3e01e50d
Fix store_url_cache using bytes
2019-10-10 14:52:29 +01:00
Erik Johnston
9d06fb9cb1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any
2019-10-10 14:38:21 +01:00
Erik Johnston
8bc529c04d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 14:30:51 +01:00
Erik Johnston
933034e2fe
Merge pull request #6127 from matrix-org/erikj/patch_inner
...
Add more log context checks when patching inlineCallbacks
2019-10-10 13:47:50 +01:00
Erik Johnston
fe1c1e6c28
Fixup comments
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-10 13:17:19 +01:00
Erik Johnston
b161786c14
Replace IN usage with helper funcs
2019-10-10 13:15:49 +01:00
Erik Johnston
b4fbf71187
Add helper funcs to use postgres ANY
...
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
Erik Johnston
91f43dca39
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 13:10:57 +01:00
werner291
b5b03b7079
Add domain validation when creating room with list of invitees ( #6121 )
2019-10-10 13:05:48 +01:00
Michael Telatynski
9a84d74417
before fulfilling a group invite,check if user is already joined/invited ( #3436 )
...
Fixes vector-im/riot-web#5645
2019-10-10 13:03:44 +01:00
krombel
2efd050c9d
send 404 as http-status when filter-id is unknown to the server ( #2380 )
...
This fixed the weirdness of 400 vs 404 as http status code in the case
the filter id is not known by the server.
As e.g. matrix-js-sdk expects 404 to catch this situation this leads
to unwanted behaviour.
2019-10-10 12:59:55 +01:00
Erik Johnston
59e0ed8306
Fix py3.5
2019-10-10 12:47:07 +01:00
Erik Johnston
c349e3ebaf
Fix py3.5
2019-10-10 12:29:38 +01:00
Erik Johnston
f735aeec65
sort
2019-10-10 12:20:29 +01:00
Erik Johnston
941edad583
Appease mypy
2019-10-10 12:15:17 +01:00
Erik Johnston
791a8c559b
Add coments
2019-10-10 11:53:57 +01:00
Erik Johnston
9970f955ce
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-10 11:51:50 +01:00
Richard van der Hoff
a139420a3c
Fix races in room stats (and other) updates. ( #6187 )
...
Hopefully this will fix the occasional failures we were seeing in the room directory.
The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.
We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
Richard van der Hoff
562b4e51dd
Rewrite the user_filter migration again ( #6184 )
...
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
Erik Johnston
128d5948c4
Fix packaging
2019-10-10 11:16:26 +01:00
Erik Johnston
ec0596f2ab
Log correct context
2019-10-10 11:11:38 +01:00
Erik Johnston
3e4272961a
Test for sentinel commit
2019-10-10 10:58:32 +01:00
Erik Johnston
1d6dd1c294
Move patch_inline_callbacks into synapse/
2019-10-10 10:53:06 +01:00
Andrew Morgan
da815c1f69
Move tag/push rules room upgrade checking ealier ( #6155 )
...
It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers.
The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).
2019-10-10 10:06:45 +01:00
Amber Brown
f743108a94
Refactor HomeserverConfig so it can be typechecked ( #6137 )
2019-10-10 09:39:35 +01:00
Hubert Chathi
4535a07f4a
make version optional in body of e2e backup version update
...
to agree with latest version of the MSC
2019-10-09 17:54:03 -04:00
Erik Johnston
5c1f886c75
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-09 16:52:21 +01:00
Erik Johnston
7f18b3d526
Do the update as a background index
2019-10-09 16:03:24 +01:00
Erik Johnston
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
...
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
Brendan Abolivier
59d6290ed9
Merge pull request #6178 from matrix-org/babolivier/factor_out_bg_updates
...
Factor out backgroung updates
2019-10-09 12:29:01 +01:00
Erik Johnston
1d3858371e
Disable bytes usage with postgres
...
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
Erik Johnston
e7631d84e6
Fix existing hex encoded json values in DB
2019-10-08 16:18:43 +01:00
Erik Johnston
ced4784592
Fix inserting bytes as text
2019-10-08 16:18:43 +01:00
Brendan Abolivier
c69324ffb5
Fix RegistrationStore
2019-10-08 14:48:33 +01:00
Brendan Abolivier
b1c0a4ceb3
Cleanup client_ips
2019-10-08 14:38:14 +01:00
Brendan Abolivier
8f1b385acc
Don't end up with 4 classes in registration
2019-10-08 14:36:33 +01:00
Anshul Angaria
474abf1eb6
add M_TOO_LARGE error code for uploading a too large file ( #6151 )
...
Fixes #6109
2019-10-08 13:55:16 +01:00
Andrew Morgan
ea7d938bca
Remove unused public room list timeout param ( #6179 )
...
* Remove unused public room list timeout param
* Add changelog
2019-10-08 13:51:25 +01:00
Richard van der Hoff
b94a401852
Fix /federation/v1/state for recent room versions ( #6170 )
...
* Fix /federation/v1/state for recent room versions
Turns out this endpoint was completely broken for v3 rooms. Hopefully this
re-signing code is irrelevant nowadays anyway.
2019-10-08 09:35:37 +01:00
Brendan Abolivier
66ebea1723
Lint
2019-10-07 17:44:41 +01:00
Brendan Abolivier
0496eafbf4
Move roommember's bg updates to a dedicated store
2019-10-07 17:35:01 +01:00
Brendan Abolivier
e106a0e4db
Move user_directory's bg updates to a dedicated store
2019-10-07 17:34:45 +01:00
Brendan Abolivier
cfccd2d78a
Move state's bg updates to a dedicated store
2019-10-07 17:34:39 +01:00
Brendan Abolivier
841054ad96
Move search's bg updates to a dedicated store
2019-10-07 17:34:35 +01:00
Brendan Abolivier
81e6ffb536
Move registration's bg updates to a dedicated store
2019-10-07 17:34:29 +01:00
Brendan Abolivier
54f87e0734
Move media_repository's bg updates to a dedicated store
2019-10-07 17:34:26 +01:00
Brendan Abolivier
cef9f6753e
Move devices's bg updates to a dedicated store
2019-10-07 17:34:20 +01:00
Brendan Abolivier
2d3b4f42f0
Move deviceinbox's bg updates to a dedicated store
2019-10-07 17:34:16 +01:00
Brendan Abolivier
88957199e7
Move client_ips's bg updates to a dedicated store
2019-10-07 17:34:12 +01:00
Erik Johnston
2cb7466993
Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure
...
Don't regenerate numeric user ID if registration fails.
2019-10-07 16:56:23 +01:00
Michael Kaye
dc795ba709
Log responder we are using. ( #6139 )
...
This prevents us logging "Responding to media request with responder %s".
2019-10-07 15:41:25 +01:00
Richard van der Hoff
276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test ( #6167 )
2019-10-07 14:41:39 +01:00
Brendan Abolivier
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
...
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
Brendan Abolivier
c8e6c308c6
Fix unique_user_filter_index schema update
2019-10-07 13:15:35 +01:00
Erik Johnston
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships
...
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
Erik Johnston
5119a4cac7
Fix bug where we didn't pull out event ID
2019-10-07 12:21:17 +01:00
Brendan Abolivier
ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
...
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
Brendan Abolivier
21d51ab598
Typo
2019-10-04 11:21:24 +01:00
Brendan Abolivier
4676732ca0
Lint
2019-10-04 11:18:28 +01:00
Brendan Abolivier
81d51ce48b
Incorporate review
2019-10-04 11:16:19 +01:00
Alexander Maznev
13c4345c84
Update user_filters
table to have a unique index, and non-null columns ( #1172 )
2019-10-04 10:34:16 +01:00
Robert Swain
39b40d6d99
media/thumbnailer: Better quality for 1-bit / 8-bit color palette images ( #2142 )
...
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
Erik Johnston
6511071837
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships
2019-10-03 17:51:06 +01:00
Richard van der Hoff
66537e10ce
add some metrics on the federation sender ( #6160 )
2019-10-03 17:47:20 +01:00
Erik Johnston
ab8a64772b
Remove unused variable
2019-10-03 17:42:32 +01:00
Erik Johnston
693156aaf4
Don't regenerate numeric user ID if registration fails.
...
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
Erik Johnston
91f61fc6d7
Use the right error....
2019-10-03 17:28:31 +01:00
Erik Johnston
84691da6c3
pep8
2019-10-03 17:27:18 +01:00
Erik Johnston
a9610cdf02
Fixup names and comments
2019-10-03 17:26:56 +01:00
Erik Johnston
d89ebf7c25
cachedList descriptor doesn't like typing
2019-10-03 17:23:11 +01:00
Erik Johnston
c8145af8a9
Cache room membership lookups in _get_joined_users_from_context
2019-10-03 17:11:04 +01:00
Andrew Morgan
6018bec919
Merge branch 'master' into develop
2019-10-03 13:48:45 +01:00
Andrew Morgan
ecb69d824a
1.4.0
2019-10-03 13:22:44 +01:00
Andrew Morgan
0f46bf5737
Replace client_secret with <redacted> in server logs ( #6158 )
...
Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
2019-10-03 12:57:26 +01:00
Brendan Abolivier
6527fa18c1
Add test case
2019-10-03 11:24:36 +01:00
Erik Johnston
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
...
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
Erik Johnston
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
...
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
Erik Johnston
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
...
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
Erik Johnston
7a5f080f91
Fix appservice room list pagination
2019-10-02 15:47:22 +01:00
Erik Johnston
4c4f44930d
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 15:20:36 +01:00
Erik Johnston
03cf4385e0
Fix public room list pagination.
...
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
Erik Johnston
a5166e4d5f
Land improved room list based on room stats ( #6019 )
...
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Andrew Morgan
aec1377d0b
1.4.0rc2
2019-10-02 13:55:00 +01:00
Amber Brown
864f144543
Fix up some typechecking ( #6150 )
...
* type checking fixes
* changelog
2019-10-02 05:29:01 -07:00
Brendan Abolivier
972c9f65d7
Lint
2019-10-02 12:17:46 +01:00
Andrew Morgan
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade ( #6144 )
...
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.
For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.
Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
Erik Johnston
d69fd53f74
Bound find_next_generated_user_id DB query.
...
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
Brendan Abolivier
5705ecaec6
Don't 500 code when trying to exchange a revoked 3PID invite
...
While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state.
When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events.
When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it).
This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
2019-10-02 11:19:43 +01:00
Erik Johnston
ecd254bc49
Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop
2019-10-02 11:08:07 +01:00
Erik Johnston
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
...
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
Erik Johnston
35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions
...
Fix fetching censored redactions from DB
2019-10-02 10:57:30 +01:00
Erik Johnston
f44f1d2e83
Fix errors storing large retry intervals.
...
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.
To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
Erik Johnston
ce7a3e7e27
Fix fetching censored redactions from DB
...
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
Erik Johnston
5e8387af9e
Use received_ts
to find uncensored redacted events
...
Joining against `events` and ordering by `stream_ordering` is
inefficient as it forced scanning the entirety of the redactions table.
This isn't the case if we use `redactions.received_ts` column as we can
then use an index.
2019-10-01 13:43:48 +01:00
Erik Johnston
898dde981b
Add received_ts column to redactions.
...
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
Erik Johnston
a27fb7d5ca
Don't repeatedly attempt to censor events we don't have.
...
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.
When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
Erik Johnston
1d349fb159
Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop
2019-10-01 10:17:24 +01:00
Erik Johnston
9267741a5f
Fix devices_last_seen
background update.
...
Fixes #6134 .
2019-09-30 11:58:36 +01:00
Richard van der Hoff
16cb9a71b8
Drop unused tables ( #6115 )
...
These tables are unused since #5893 (as amended by #6047 ), so we can now drop
them.
Fixes #6048 .
2019-09-30 09:38:41 +01:00
Andrew Morgan
f3451118a6
Edit SimpleHttpClient to reference that header keys can be passed as str or bytes ( #6077 )
2019-09-27 17:59:18 +01:00
Brendan Abolivier
3e42d47a5a
Incorporate review
2019-09-27 16:15:01 +01:00
Brendan Abolivier
af92110c46
Update synapse/handlers/deactivate_account.py
...
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:12:15 +01:00
Brendan Abolivier
bbe2a0f339
Update synapse/handlers/deactivate_account.py
...
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:36 +01:00
Brendan Abolivier
873fe7883c
Lint
2019-09-27 15:21:03 +01:00
Brendan Abolivier
72a2708ac6
Fixup and add some logging
2019-09-27 15:13:39 +01:00
Erik Johnston
132279a46f
Patch inlinecallbacks for log contexts
2019-09-27 15:11:14 +01:00
Brendan Abolivier
5257a2fb1c
Reject pending invites on deactivation
2019-09-27 14:49:53 +01:00
Andrew Morgan
8c27bc8b60
Move lookup-related functions from RoomMemberHandler to IdentityHandler ( #5978 )
...
Just to have all the methods that make calls to identity services in one place.
2019-09-27 10:36:20 +01:00
Erik Johnston
67ee18daea
Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span
...
Add some helpful opentracing tags and remove get_user_by_req span
2019-09-27 10:02:57 +01:00
Erik Johnston
3423633d50
Fix 'redaction_retention_period' sampel config to match guidelines
2019-09-26 16:43:52 +01:00
Richard van der Hoff
54569c787b
Kill off half-implemented password-reset via sms ( #6101 )
...
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.
See also matrix-org/matrix-doc#2303 .
2019-09-26 15:38:25 +01:00
Neil Johnson
8b8f8c7b3c
Explicitly log when a homeserver does not have a trusted key server configured ( #6090 )
2019-09-26 12:57:01 +01:00
Richard van der Hoff
2927c6bc4c
bump version
2019-09-26 12:29:59 +01:00
Neil Johnson
034db2ba21
Fix dummy event insertion consent bug ( #6053 )
...
Fixes #5905
2019-09-26 11:47:53 +01:00
Richard van der Hoff
a96318127d
Update comments and docstring
2019-09-25 18:17:39 +01:00
Erik Johnston
4fb3c129aa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2
2019-09-25 17:53:13 +01:00
Erik Johnston
9614d3c9d1
Merge pull request #6089 from matrix-org/erikj/cleanup_user_ips
...
Move last seen info into devices table
2019-09-25 17:42:39 +01:00
Neil Johnson
a4f3ca48b5
Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ( #5884 )
2019-09-25 17:27:35 +01:00
Erik Johnston
39b50ad42a
Review comments
2019-09-25 17:22:33 +01:00
Erik Johnston
d2bd0bc6b1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips
2019-09-25 17:16:28 +01:00
Erik Johnston
50572db837
Use if is not None
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
Neil Johnson
77dc7093a7
Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. ( #6104 )
...
Fixes #6103
2019-09-25 12:29:35 +01:00
Richard van der Hoff
990928abde
Stop advertising unsupported flows for registration ( #6107 )
...
If email or msisdn verification aren't supported, let's stop advertising them
for registration.
Fixes #6100 .
2019-09-25 12:10:26 +01:00
Erik Johnston
dc2c97e1a3
isort
2019-09-25 11:59:05 +01:00
Erik Johnston
dc01cad690
Add device and appservice tags
2019-09-25 11:59:00 +01:00
Erik Johnston
5d99713854
Add tags for event_id and txn_id in event sending
...
This will make it easier to search for sending event requests.
2019-09-25 11:39:15 +01:00
Richard van der Hoff
2cd98812ba
Refactor the user-interactive auth handling ( #6105 )
...
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.
This is also preparation for some more intelligent advertising of flows, as per #6100
2019-09-25 11:33:03 +01:00
Erik Johnston
fde4ce2213
Don't create new span for get_user_by_req
...
We don't actually care about what happens in `get_user_by_req` and
having it as a separate span means that the entity tag isn't added to
the servlet spans, making it harder to search.
2019-09-25 11:32:41 +01:00
Richard van der Hoff
8004d6ca2f
Refactor code for calculating registration flows ( #6106 )
...
because, frankly, it looked like it was written by an axe-murderer.
This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
2019-09-25 11:32:05 +01:00
Hubert Chathi
c3635c9459
make isort happy
2019-09-24 16:21:03 -04:00
Hubert Chathi
f4b6d43ec3
add some comments
2019-09-24 16:19:54 -04:00
Hubert Chathi
39864f45ec
drop some logger lines to debug
2019-09-24 15:26:45 -04:00
Hubert Chathi
26113fb7de
make changes based on PR feedback
2019-09-24 14:12:20 -04:00
Brendan Abolivier
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
...
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
Richard van der Hoff
9b7c4f4078
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 17:03:50 +01:00
Richard van der Hoff
566ac40939
remove unused parameter to get_user_id_by_threepid ( #6099 )
...
Added in #5377 , apparently in error
2019-09-24 17:01:09 +01:00
Erik Johnston
242017db8b
Prune rows in user_ips older than configured period
...
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
Erik Johnston
2135c198d1
Add has_completed_background_update
...
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Erik Johnston
367158a609
Add wrap_as_background_process decorator.
...
This does the same thing as `run_as_background_process` but means we
don't need to create superfluous functions.
2019-09-24 15:53:17 +01:00
Richard van der Hoff
bb82be9851
Merge pull request #6069 from matrix-org/rav/fix_attribute_mapping
...
Fix a bug with saml attribute maps.
2019-09-24 15:07:26 +01:00
Brendan Abolivier
12fe2a29bc
Incorporate review
2019-09-24 14:43:38 +01:00
J. Ryan Stinnett
40fb00f5b7
Add sid to next_link for email validation ( #6097 )
2019-09-24 14:39:50 +01:00
Brendan Abolivier
f02f14e09a
Fix logging
2019-09-24 14:39:07 +01:00
Richard van der Hoff
ed8b92f0d2
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 12:57:32 +01:00
Richard van der Hoff
a25b66d3f9
docstrings and comments
2019-09-24 11:15:08 +01:00
Andrew Morgan
50776261e1
Add submit_url response parameter to msisdn /requestToken ( #6079 )
...
Second part of solving #6076
Fixes #6076
We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
2019-09-23 21:21:03 +01:00
Andrew Morgan
2b071a2ff1
Add an unstable feature flag for separate add/bind 3pid APIs ( #6044 )
...
Add a m.separate_add_and_bind flag set to True. See MSC2290's Backward Compatibility section for details.
2019-09-23 20:46:34 +01:00
Richard van der Hoff
78e8ec368e
Merge pull request #6064 from matrix-org/rav/saml_config_cleanup
...
Make the sample saml config closer to our standards
2019-09-23 20:36:51 +01:00
Andrew Morgan
e08ea43463
Use the federation blacklist for requests to untrusted Identity Servers ( #6000 )
...
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.
Fixes #5935
2019-09-23 20:23:20 +01:00
Andrew Morgan
1ea3ed7620
Add m.id_access_token to /versions unstable_features (MSC2264) ( #5974 )
...
Adds a flag to /versions' unstable_features section indicating that this Synapse understands what an id_access_token is, as per MSC2264.
Fixes #5927
2019-09-23 20:19:25 +01:00
Andrew Morgan
b38aa82b83
Add m.require_identity_server to /versions unstable_flags ( #5972 )
...
As MSC2263 states, m.require_identity_server must be set to false when it does not require an identity server to be provided by the client for the purposes of email registration or password reset.
Adds an m.require_identity_server flag to /versionss unstable_flags section. This will advertise that Synapse no longer needs id_server as a parameter.
2019-09-23 18:52:43 +01:00
Andrew Morgan
2c99c63453
Add POST submit_token endpoint for MSISDN ( #6078 )
...
First part of solving #6076
2019-09-23 17:49:00 +01:00
Brendan Abolivier
323d685bf7
Typo
2019-09-23 17:23:49 +01:00
Brendan Abolivier
2858d10671
Fix the return value in the users_set_deactivated_flag background job
2019-09-23 17:22:01 +01:00
Erik Johnston
51d28272e2
Query devices table for last seen info.
...
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.
The storage functions now no longer return the access_token, since it
was unused.
2019-09-23 16:59:45 +01:00
Erik Johnston
ed80231ade
Add BG update to populate devices last seen info
2019-09-23 16:59:45 +01:00
Erik Johnston
2ade05dca3
Add last seen info to devices table.
...
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
Andrew Morgan
30af161af2
Implement MSC2290 ( #6043 )
...
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.
This PR also removes the blacklist on some 3PID tests which occurs in #6042 , as the corresponding Sytest PR changes them to use the new endpoints.
Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
Andrew Morgan
1b519e0272
Disable /register/available if registration is disabled ( #6082 )
...
Fixes #6066
This register endpoint should be disabled if registration is disabled, otherwise we're giving anyone the ability to check if a username exists on a server when we don't need to be.
Error code is 403 (Forbidden) as that's the same returned by /register when registration is disabled.
2019-09-23 15:38:38 +01:00
Andrew Morgan
1c9feadf4b
Generalize email sending logging ( #6075 )
...
In ancient times Synapse would only send emails when it was notifying a user about a message they received...
Now it can do all sorts of neat things!
Change the logging so it's not just about notifications.
2019-09-23 14:38:19 +01:00
Andrew Morgan
885a4726b7
Return timeout error to user for identity server calls ( #6073 )
2019-09-23 14:37:23 +01:00
Richard van der Hoff
0660f8e367
Merge pull request #6072 from matrix-org/rav/fix_retry_reset
...
Fix exception when resetting retry timings
2019-09-23 14:31:47 +01:00
Matthew Hodgson
7ef319aefe
fix broken copyrights
2019-09-23 12:28:01 +01:00
Andrew Morgan
df3401a71d
Allow HS to send emails when adding an email to the HS ( #6042 )
2019-09-20 15:21:30 +01:00
Andrew Morgan
7763dd3e95
Remove trailing slash ability from password reset's submit_token endpoint ( #6074 )
...
Remove trailing slash ability from the password reset submit_token endpoint. Since we provide the link in an email, and have never sent it with a trailing slash, there's no point for us to accept them on the endpoint.
2019-09-20 14:58:37 +01:00
Richard van der Hoff
9d94313209
Fix exception when resetting retry timings
...
Fixes:
> TypeError: set_destination_retry_timings() missing 1 required positional
argument: 'retry_interval'
Introduced in #6016 .
2019-09-20 12:09:39 +01:00
Richard van der Hoff
366dc7d2d4
Merge commit '33757bad1' into rav/saml_mapping_work
2019-09-20 11:21:40 +01:00
Richard van der Hoff
33757bad19
More better logging
2019-09-20 11:20:02 +01:00
Andrew Morgan
aeb40f355c
Ensure email validation link parameters are URL-encoded ( #6063 )
...
The validation links sent via email had their query parameters inserted without any URL-encoding. Surprisingly this didn't seem to cause any issues, but if a user were to put a `/` in their client_secret it could lead to problems.
2019-09-20 10:46:59 +01:00
Andrew Morgan
3ac614eb6c
Drop support for bind param on POST /account/3pid (MSC2290) ( #6067 )
...
As per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/files#diff-05cde9463e9209b701312b3baf2fb2ebR151 ), we're dropping the bind parameter from `/account/3pid`. This endpoint can now only be used for adding threepid's to the user's account on the homeserver.
2019-09-20 10:46:34 +01:00
J. Ryan Stinnett
36015d68ef
Use unstable prefix for 3PID unbind API ( #6062 )
2019-09-19 22:28:29 +01:00
Richard van der Hoff
284e1cb027
Merge branch 'develop' into rav/fix_attribute_mapping
2019-09-19 20:32:25 +01:00
Richard van der Hoff
b74606ea22
Fix a bug with saml attribute maps.
...
Fixes a bug where the default attribute maps were prioritised over
user-specified ones, resulting in incorrect mappings.
The problem is that if you call SPConfig.load() multiple times, it adds new
attribute mappers to a list. So by calling it with the default config first,
and then the user-specified config, we would always get the default mappers
before the user-specified mappers.
To solve this, let's merge the config dicts first, and then pass them to
SPConfig.
2019-09-19 20:32:14 +01:00
Richard van der Hoff
b65327ff66
Merge branch 'develop' into rav/saml_mapping_work
2019-09-19 18:13:31 +01:00
Richard van der Hoff
7423fade92
better logging
2019-09-19 17:16:50 +01:00
Richard van der Hoff
bcd9132869
Undo the deletion of some tables ( #6047 )
...
This is a partial revert of #5893 . The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.
So let's leave the tables in place for now, and remember to drop them in a
subsequent release.
(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
2019-09-19 15:06:27 +01:00
Richard van der Hoff
b789c7eb03
Merge branch 'develop' into rav/saml_config_cleanup
2019-09-19 15:05:31 +01:00
Andrew Morgan
62e3ff92fd
Remove POST method from password reset submit_token endpoint ( #6056 )
...
Removes the POST method from `/password_reset/<medium>/submit_token/` as it's only used by phone number verification which Synapse does not support yet.
2019-09-19 10:53:14 +01:00
Jorik Schellekens
38fd1f8e3f
Fix typo in account_threepid_delegates config ( #6028 )
2019-09-18 22:30:44 +01:00
J. Ryan Stinnett
a86a290850
Fix logcontext spam on non-Linux platforms ( #6059 )
...
This checks whether the current platform supports thread resource usage tracking
before logging a warning to avoid log spam.
Fixes https://github.com/matrix-org/synapse/issues/6055
2019-09-18 21:55:37 +01:00
Andrew Morgan
6670bd4072
v2 3PID Invites (part of MSC2140) ( #5979 )
...
3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is.
These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens).
Requires: ~~#5976~~
2019-09-17 18:05:13 +01:00
dstipp
379d2a8c39
( #5849 ) Convert rst to markdown ( #6040 )
...
Converting some of the rst documentation to markdown. Attempted to
preserve whitespace and line breaks to minimize cosmetic change.
2019-09-17 12:55:29 +01:00
Erik Johnston
70c52821ce
Fix race condition in room stats. ( #6029 )
...
Broke in #5971
Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)
2019-09-17 12:41:23 +01:00
Richard van der Hoff
1e19ce00bf
Add 'failure_ts' column to 'destinations' table ( #6016 )
...
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
Amber Brown
850dcfd2d3
Fix well-known lookups with the federation certificate whitelist ( #5997 )
2019-09-14 04:58:38 +10:00
Richard van der Hoff
a8ac40445c
Record mappings from saml users in an external table
...
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Richard van der Hoff
785cbd3999
Make the sample saml config closer to our standards
...
It' still not great, thanks to the nested dictionaries, but it's better.
2019-09-13 12:07:03 +01:00
Amber Brown
b617864cd9
Fix for structured logging tests stomping on logs ( #6023 )
2019-09-13 02:29:55 +10:00
Richard van der Hoff
3d882a7ba5
Remove the cap on federation retry interval. ( #6026 )
...
Essentially the intention here is to end up blacklisting servers which never
respond to federation requests.
Fixes https://github.com/matrix-org/synapse/issues/5113 .
2019-09-12 13:00:13 +01:00
Richard van der Hoff
0388beafe4
Fix bug in calculating the federation retry backoff period ( #6025 )
...
This was intended to introduce an element of jitter; instead it gave you a
30/60 chance of resetting to zero.
2019-09-12 12:59:43 +01:00
David Baker
642fad8bd4
Fix SSO fallback login
...
Well, it worked, but forgot to remove the thing saying login was
unavailable.
2019-09-12 11:42:47 +01:00
Sorunome
dd2e5b0038
add report_stats_endpoint config option ( #6012 )
...
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
2019-09-12 11:24:57 +01:00
Jorik Schellekens
a8251da10f
Blow up config if opentracing is missing ( #5985 )
...
* Blow up config if opentracing is missing
2019-09-12 10:57:37 +01:00
Jason Robinson
6d847d8ce6
Ensure support users can be registered even if MAU limit is reached
...
This allows support users to be created even on MAU limits via
the admin API. Support users are excluded from MAU after creation,
so it makes sense to exclude them in creation - except if the
whole host is in disabled state.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-11 20:48:31 +03:00
Andrew Morgan
9fc71dc5ee
Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) ( #5976 )
...
This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted.
Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134 ) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ).
Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
2019-09-11 16:02:42 +01:00
Erik Johnston
cbcbfe64a2
Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction
...
Allow use of different ratelimits for admin redactions.
2019-09-11 15:39:38 +01:00
Richard van der Hoff
7902bf1e1d
Clean up some code in the retry logic ( #6017 )
...
* remove some unused code
* make things which were constants into constants for efficiency and clarity
2019-09-11 15:14:56 +01:00
Andrew Morgan
9c555f37e3
Add note about extra arg to send_membership_event, remove arg in remote_reject_invite ( #6009 )
...
Some small fixes to `room_member.py` found while doing other PRs.
1. Add requester to the base `_remote_reject_invite` method.
2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.
2019-09-11 14:23:24 +01:00
Jorik Schellekens
6604b64fae
Check dependencies on setup in the nicer way. ( #5989 )
2019-09-11 14:00:37 +01:00
Erik Johnston
57dd41a45b
Fix comments
...
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-11 13:54:50 +01:00
Andrew Morgan
3505ffcda7
Fix existing v2 identity server calls (MSC2140) ( #6013 )
...
Two things I missed while implementing [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R80 ).
1. Access tokens should be provided to the identity server as `access_token`, not `id_access_token`, even though the homeserver may accept the tokens as `id_access_token`.
2. Access tokens must be sent to the identity server in a query parameter, the JSON body is not allowed.
We now send the access token as part of an `Authorization: ...` header, which fixes both things.
The breaking code was added in https://github.com/matrix-org/synapse/pull/5892
Sytest PR: https://github.com/matrix-org/sytest/pull/697
2019-09-11 11:59:45 +01:00
Erik Johnston
c64c3bb4c5
Fix how we check for self redaction
2019-09-11 11:18:04 +01:00
Erik Johnston
54ce81c86d
Allow use of different ratelimits for admin redactions.
...
This is useful to allow room admins to quickly deal with a large number
of abusive messages.
2019-09-11 10:46:38 +01:00
Andrew Morgan
cd17a2085e
Remove origin parameter from add_display_name_to_third_party_invite and add params to docstring ( #6010 )
...
Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument.
Since it's not used anywhere in the handler function though, we should remove it from the function arguments.
2019-09-11 10:37:17 +01:00
Erik Johnston
5e9b05d7da
Merge pull request #6011 from matrix-org/anoa/fix_3pid_validation
...
Use account_threepid_delegate for 3pid validation
2019-09-10 18:15:07 +01:00
Andrew Morgan
b5833a2abf
Add changelog
2019-09-10 17:56:10 +01:00
Andrew Morgan
60d3c57bd0
Use account_threepid_delegate for 3pid validation
2019-09-10 17:56:10 +01:00
Jason Robinson
63f9317b8e
Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users
...
Only count real users when checking for auto-creation of auto-join room
2019-09-09 17:37:52 +03:00
Erik Johnston
470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions
...
Censor redactions in DB after a month
2019-09-09 15:29:39 +01:00
Amber Brown
aeb9b2179e
Add a build info metric to Prometheus ( #6005 )
2019-09-10 00:14:58 +10:00
Jason Robinson
aaed6b39e1
Fix code style, again
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 17:10:02 +03:00
Erik Johnston
580f3df9b2
Fix comments
2019-09-09 15:08:24 +01:00
Jason Robinson
e89fea4f04
Simplify count_real_users SQL to only count user_type is null rows
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 16:43:32 +03:00
Jason Robinson
8c03cd0e5f
Simplify is_real_user_txn check to trust user_type is null if real user
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 16:40:40 +03:00
Erik Johnston
8b9ade8c78
Default to censoring redactions after seven days
2019-09-09 13:55:28 +01:00
Erik Johnston
e7184a4370
Use better names in SQL
2019-09-09 13:33:38 +01:00
Erik Johnston
916c697228
Fixup comment
2019-09-09 13:31:00 +01:00
Erik Johnston
fffe17b77d
Don't start looping call unless enabled
2019-09-09 13:24:24 +01:00
Erik Johnston
80e14a8546
Handle setting retention period to 0
2019-09-09 13:23:41 +01:00
Jason Robinson
62fac9d969
Auto-fix a few code style issues
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:59:35 +03:00
Jason Robinson
be618e0551
Only count real users when checking for auto-creation of auto-join room
...
Previously if the first registered user was a "support" or "bot" user,
when the first real user registers, the auto-join rooms were not
created.
Fix to exclude non-real (ie users with a special user type) users
when counting how many users there are to determine whether we should
auto-create a room.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-09-09 14:48:08 +03:00
Erik Johnston
05bae6b4fc
Add opentracing span for HTTP push
2019-09-09 10:24:14 +01:00
Hubert Chathi
d3f2fbcfe5
add function docs
2019-09-07 14:13:18 -04:00
Hubert Chathi
8e86f5b65c
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-09-07 13:20:34 -04:00
Hubert Chathi
0d61d1d735
Merge branch 'develop' into cross-signing_sig_upload
2019-09-07 13:14:45 -04:00
Hubert Chathi
415d0a00e0
run black
2019-09-06 16:46:45 -04:00
Hubert Chathi
561cbba057
split out signature processing into separate functions
2019-09-06 16:44:24 -04:00
Amber Brown
55d5b3af88
Servers-known-about statistic ( #5981 )
2019-09-07 01:45:51 +10:00
Andrew Morgan
78801e7f9e
Ensure a sid parameter is passed to bind_threepid ( #5995 )
...
`sid` is required to be part of `three_pid_creds`. We were 500'ing if it wasn't provided instead of returning `M_MISSING_PARAM`.
2019-09-06 15:36:50 +01:00
Erik Johnston
a2a695b7ec
Merge pull request #5998 from matrix-org/erikj/fixup_federate_flag
...
Correctly handle non-bool m.federate flag
2019-09-06 15:32:43 +01:00
Erik Johnston
142c9325c2
Correctly handle non-bool m.federate flag
2019-09-06 14:21:06 +01:00
Erik Johnston
30b67e0f63
Merge pull request #5993 from matrix-org/anoa/worker_store_reg
...
Move get_threepid_validation_session and delete_threepid_session into RegistrationWorkerStore
2019-09-06 14:10:02 +01:00
Andrew Morgan
cf5a420c8a
Apply suggestions from code review
...
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-09-06 13:34:42 +01:00
Andrew Morgan
ca74b140f2
Fix destructuring assumption bug
2019-09-06 13:25:06 +01:00
Andrew Morgan
6ddda8152e
Move delete_threepid_session into RegistrationWorkerStore
2019-09-06 13:23:10 +01:00
Andrew Morgan
e059c5e648
Move get_threepid_validation_session into RegistrationWorkerStore
2019-09-06 13:10:11 +01:00
Erik Johnston
146af7b47f
Merge pull request #5991 from matrix-org/erikj/fix_tracing_funcs
...
Don't assume there is a 'self' arg in @trace decorator
2019-09-06 11:42:45 +01:00
Andrew Morgan
0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation ( #5987 )
...
This is a combination of a few different PRs, finally all being merged into `develop`:
* #5875
* #5876
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)
) in #5969 . What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835
* #5969
* #5940
Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.
UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Erik Johnston
d8517da85b
Don't assume there is a 'self' arg in @trace decorator
2019-09-06 10:07:12 +01:00
Hubert Chathi
369462da74
avoid modifying input parameter
2019-09-05 17:03:31 -04:00
Jorik Schellekens
f7c873a643
Trace how long it takes for the send trasaction to complete, including retrys ( #5986 )
2019-09-05 17:44:55 +01:00
Jorik Schellekens
bc604e7f94
Gracefully handle log context slips and missing opentracing import errors. ( #5988 )
2019-09-05 17:33:29 +01:00
Erik Johnston
591d82f06b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions
2019-09-05 17:27:46 +01:00
Erik Johnston
3ff0422d2d
Make redaction retention period configurable
2019-09-05 17:16:45 +01:00
Erik Johnston
1a6ae33309
Merge pull request #5984 from matrix-org/joriks/opentracing_link_send_to_edu_contexts
...
Link the send loop with the edus contexts
2019-09-05 15:22:24 +01:00
Jorik Schellekens
ef20aa52eb
use access methods (duh..)
...
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-09-05 15:07:17 +01:00
Jorik Schellekens
5ade977d08
Opentracing context cannot be none
2019-09-05 15:06:13 +01:00
Jorik Schellekens
909827b422
Add opentracing to all client servlets ( #5983 )
2019-09-05 14:46:04 +01:00
Jorik Schellekens
1d65292e94
Link the send loop with the edus contexts
...
The contexts were being filtered too early so the send loop wasn't
being linked to them unless the destination
was whitelisted.
2019-09-05 14:42:37 +01:00
Andrew Morgan
a0d294c306
Switch to using v2 Identity Service APIs other than lookup (MSC 2140) ( #5892 )
2019-09-05 14:31:22 +01:00
Jorik Schellekens
b9cfd3c375
Fix opentracing contexts missing from outbound replication requests ( #5982 )
2019-09-05 14:22:15 +01:00
Andrew Morgan
90d17a3d28
Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) ( #5980 )
...
Implements `POST /_matrix/client/r0/account/3pid/unbind` from [MSC2140](https://github.com/matrix-org/matrix-doc/blob/dbkr/tos_2/proposals/2140-terms-of-service-2.md#post-_matrixclientr0account3pidunbind ).
2019-09-05 14:00:30 +01:00
Hubert Chathi
c8dc740a94
update with newer coding style
2019-09-04 22:30:45 -04:00
Hubert Chathi
7d6c70fc7a
make black happy
2019-09-04 22:04:12 -04:00
Hubert Chathi
ac4746ac4b
allow uploading signatures of master key signed by devices
2019-09-04 22:03:41 -04:00
Hubert Chathi
4bb4544784
implement device signature uploading/fetching
2019-09-04 20:02:56 -04:00
Hubert Chathi
a22d58c96c
add user signature stream change cache to slaved device store
2019-09-04 19:32:35 -04:00
Hubert Chathi
faf72a4c40
Merge branch 'develop' into cross-signing_keys
2019-09-04 19:12:29 -04:00
Andrew Morgan
b736c6cd3a
Remove bind_email and bind_msisdn ( #5964 )
...
Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107 ).
2019-09-04 18:24:23 +01:00
Andrew Morgan
b09d443632
Cleanup event auth type initialisation ( #5975 )
...
Very small code cleanup.
2019-09-04 16:16:56 +01:00
Erik Johnston
6e834e94fc
Fix and refactor room and user stats ( #5971 )
...
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
Andrew Morgan
ea128a3e8e
code cleanups
2019-09-03 21:05:06 +01:00
Travis Ralston
2f416fc997
Ensure the list media admin API is always available ( #5966 )
...
* Ensure the list media admin API is always available
This API is required for some external media repo implementations to operate (mostly for doing quarantine operations on a room).
* changelog
2019-09-03 13:35:20 -06:00
Andrew Morgan
6b6086b8bf
Fix docstring
2019-09-03 20:00:09 +01:00
Andrew Morgan
a98b8583c6
Remove unnecessary variable declaration
2019-09-03 19:58:51 +01:00
Travis Ralston
0eac7077c9
Ensure an auth instance is available to ListMediaInRoom ( #5967 )
...
* Ensure an auth instance is available to ListMediaInRoom
Fixes https://github.com/matrix-org/synapse/issues/5737
* Changelog
2019-09-03 09:01:30 -06:00
Matthew Hodgson
8401bcd206
fix typo
2019-09-03 12:44:14 +01:00
Andrew Morgan
2a44782666
Remove double return statements ( #5962 )
...
Remove all the "double return" statements which were a result of us removing all the instances of
```
defer.returnValue(...)
return
```
statements when we switched to python3 fully.
2019-09-03 11:42:45 +01:00
Jorik Schellekens
a90d16dabc
Opentrace device lists ( #5853 )
...
Trace device list changes.
2019-09-03 10:21:30 +01:00
Andrew Morgan
36f34e6f3d
Remove unused methods from c/s api v1 in register.py ( #5963 )
...
These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
2019-09-02 18:29:21 +01:00
L0ric0
ce7803b8b0
fix thumbnail storage location ( #5915 )
...
* fix thumbnail storage location
Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
* Add changelog file.
Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
* Update Changelog
Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
2019-09-02 12:18:41 +01:00
Andrew Morgan
2a012e8a04
Revert "Add m.id_access_token flag ( #5930 )" ( #5945 )
...
This reverts commit 4765f0cfd9
.
2019-08-30 17:13:37 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements ( #5931 )
...
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Andrew Morgan
4765f0cfd9
Add m.id_access_token flag ( #5930 )
...
Adds a flag to `/versions`' `unstable_features` section indicating that this Synapse understands what an `id_access_token` is, as per https://github.com/matrix-org/synapse/issues/5927#issuecomment-523566043
Fixes #5927
2019-08-30 15:22:51 +01:00
Andrew Morgan
3057095a5d
Revert "Use the v2 lookup API for 3PID invites ( #5897 )" ( #5937 )
...
This reverts commit 71fc04069a
.
This broke 3PID invites as #5892 was required for it to work correctly.
2019-08-30 12:00:20 +01:00
Erik Johnston
a4bf72c30c
Censor redactions in DB after a month
2019-08-30 11:29:17 +01:00
Hubert Chathi
e3d3fbf63f
Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys
2019-08-28 17:36:46 -07:00
Hubert Chathi
72d296a7f3
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-08-28 17:33:44 -07:00
Hubert Chathi
96bda56370
black
2019-08-28 17:18:40 -07:00
Hubert Chathi
3b0b22cb05
use stream ID generator instead of timestamp
2019-08-28 17:17:21 -07:00
Jorik Schellekens
92c1550f4a
Add a link to python's logging config schema ( #5926 )
2019-08-28 19:08:32 +01:00
Will Hunt
c8fa620d7a
Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent
...
Exempt support users from consent
2019-08-28 16:31:40 +01:00
Jorik Schellekens
deca277d09
Let synctl use a config directory. ( #5904 )
...
* Let synctl use a config directory.
2019-08-28 15:55:58 +01:00
Andrew Morgan
71fc04069a
Use the v2 lookup API for 3PID invites ( #5897 )
...
Fixes https://github.com/matrix-org/synapse/issues/5861
Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134 ). Currently this is only used for 3PID invites.
Sytest PR: https://github.com/matrix-org/sytest/pull/679
2019-08-28 14:59:26 +02:00
Jorik Schellekens
6d97843793
Config templating ( #5900 )
...
Template config files
* Imagine a system composed entirely of x, y, z etc and the basic operations..
Wait George, why XOR? Why not just neq?
George: Eh, I didn't think of that..
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-08-28 13:12:22 +01:00
Amber Brown
7dc398586c
Implement a structured logging output system. ( #5680 )
2019-08-28 21:18:53 +10:00
Richard van der Hoff
49ef8ec399
Fix a cache-invalidation bug for worker-based deployments ( #5920 )
...
Some of the caches on worker processes were not being correctly invalidated
when a room's state was changed in a way that did not affect the membership
list of the room.
We need to make sure we send out cache invalidations even when no memberships
are changing.
2019-08-28 10:18:16 +01:00
reivilibre
a3f0635686
Merge pull request #5914 from matrix-org/rei/admin_getadmin
...
Add GET method to admin API /users/@user:dom/admin
2019-08-28 09:44:22 +01:00
reivilibre
7ccc251415
Merge pull request #5859 from matrix-org/rei/msc2197
...
MSC2197 Search Filters over Federation
2019-08-28 09:00:21 +01:00
Erik Johnston
dfd10f5133
Merge pull request #5864 from matrix-org/erikj/reliable_lookups
...
Refactor MatrixFederationAgent to retry SRV.
2019-08-27 16:54:06 +01:00
Erik Johnston
91caa5b430
Fix off by one error in SRV result shuffling
2019-08-27 13:56:42 +01:00
Olivier Wilkinson (reivilibre)
c88a119259
Add GET method to admin API /users/@user:dom/admin
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 13:12:27 +01:00
reivilibre
322ccac33f
Allow schema deltas to be engine-specific ( #5911 )
...
* Allow schema deltas to be engine-specific
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Code style (Black)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 11:53:21 +01:00
Richard van der Hoff
ccb15a5bbe
Merge pull request #5906 from matrix-org/neilj/increase_display_name_limit
...
Increase profile display name limit
2019-08-27 11:52:59 +01:00
Erik Johnston
f5b50d0871
Merge pull request #5895 from matrix-org/erikj/notary_key
...
Add config option to sign remote key query responses with a separate key.
2019-08-27 11:51:37 +01:00
Richard van der Hoff
7837a5f2ea
Merge pull request #5909 from aaronraimist/public_base_url
...
public_base_url is actually public_baseurl
2019-08-27 11:49:59 +01:00
reivilibre
1a7e6eb633
Add Admin API capability to set adminship of a user ( #5878 )
...
Admin API: Set adminship of a user
2019-08-27 10:14:00 +01:00
Olivier Wilkinson (reivilibre)
d1e0b91083
Code style (Black)
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:39:11 +01:00
Olivier Wilkinson (reivilibre)
aefa76f5cd
Allow schema deltas to be engine-specific
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 09:14:00 +01:00
Aaron Raimist
e8e3e033ee
public_base_url is actually public_baseurl
...
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-08-26 21:01:56 -05:00
Neil Johnson
27d3fc421a
Increase max display name limit
2019-08-24 22:33:43 +01:00
Erik Johnston
fbb758a7ce
Fixup comments
2019-08-23 15:37:20 +01:00
Erik Johnston
e70f0081da
Fix logcontexts
2019-08-23 15:37:20 +01:00
Erik Johnston
fe0ac98e66
Don't implicitly include server signing key
2019-08-23 15:36:28 +01:00
Erik Johnston
7af5a63063
Fixup review comments
2019-08-23 15:36:28 +01:00
Will Hunt
c998f25006
Apply suggestions from code review
...
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-08-23 10:28:54 +01:00
Half-Shot
9ba32f6573
Exempt bot users
2019-08-23 09:56:31 +01:00
Half-Shot
ffa5b757c7
Merge branch 'hs/bot-user-type' into hs/exempt-support-users-from-consent
2019-08-23 09:55:57 +01:00
Half-Shot
d9b8cf81be
Add bot type
2019-08-23 09:52:09 +01:00
Half-Shot
ae38e0569f
Ignore consent for support users
2019-08-23 09:15:10 +01:00
Half-Shot
886eceba3e
Return user_type in get_user_by_id
2019-08-23 09:14:52 +01:00
Jorik Schellekens
8767b63a82
Propagate opentracing contexts through EDUs ( #5852 )
...
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Richard van der Hoff
0b39fa53b6
Merge pull request #5877 from Awesome-Technologies/remove_shared_secret_registration
...
Remove shared secret registration
2019-08-22 18:12:25 +01:00
Jorik Schellekens
812ed6b0d5
Opentracing across workers ( #5771 )
...
Propagate opentracing contexts across workers
Also includes some Convenience modifications to opentracing for servlets, notably:
- Add boolean to skip the whitelisting check on inject
extract methods. - useful when injecting into carriers
locally. Otherwise we'd always have to include our
own servername and whitelist our servername
- start_active_span_from_request instead of header
- Add boolean to decide whether to extract context
from a request to a servlet
2019-08-22 18:08:07 +01:00
Manuel Stahl
0bab582fd6
Remove shared secret registration from client/r0/register endpoint
...
This type of registration was probably never used. It only includes the
user name in the HMAC but not the password.
Shared secret registration is still available via
client/r0/admin/register.
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2019-08-22 18:04:08 +02:00
Brendan Abolivier
dbd46decad
Revert "Do not send consent notices if "no-consent-required" is set"
...
This reverts commit 27a686e53b
.
2019-08-22 14:47:43 +01:00
Brendan Abolivier
1c5b8c6222
Revert "Add "require_consent" parameter for registration"
...
This reverts commit 3320aaab3a
.
2019-08-22 14:47:34 +01:00
Half-Shot
27a686e53b
Do not send consent notices if "no-consent-required" is set
2019-08-22 14:22:04 +01:00
Half-Shot
3320aaab3a
Add "require_consent" parameter for registration
2019-08-22 14:21:54 +01:00
Erik Johnston
1e4b4d85e7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reliable_lookups
2019-08-22 13:41:57 +01:00
Erik Johnston
1b09cf8658
Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_fail
...
Retry well known on fail
2019-08-22 13:17:05 +01:00
Jorik Schellekens
9a6f2be572
Opentrace e2e keys ( #5855 )
...
Add opentracing tags and logs for e2e keys
2019-08-22 11:28:12 +01:00
Richard van der Hoff
c9f11d09fc
Add missing index on users_in_public_rooms. ( #5894 )
2019-08-22 10:43:13 +01:00
Richard van der Hoff
119aa31b10
Servlet to purge old rooms ( #5845 )
2019-08-22 10:42:59 +01:00
Richard van der Hoff
ef1c524bb3
Improve error msg when key-fetch fails ( #5896 )
...
There's no point doing a raise_from here, because the exception is always
logged at warn with no stacktrace in the caller. Instead, let's try to give
better messages to reduce confusion.
In particular, this means that we won't log 'Failed to connect to remote
server' when we don't even attempt to connect to the remote server due to
blacklisting.
2019-08-22 10:42:06 +01:00
Hubert Chathi
814f253f1b
make isort happy
2019-08-21 13:22:15 -07:00
Hubert Chathi
7c3abc6572
apply PR review suggestions
2019-08-21 13:19:35 -07:00
Richard van der Hoff
4dab867288
Drop some unused tables. ( #5893 )
...
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
Erik Johnston
97cbc96093
Only sign when we respond to remote key requests
2019-08-21 11:21:58 +01:00
Erik Johnston
5906be8589
Add config option for keys to use to sign keys
...
This allows servers to separate keys that are used to sign remote keys
when acting as a notary server.
2019-08-21 10:44:58 +01:00
Richard van der Hoff
72bc285669
Refactor the Appservice scheduler code ( #5886 )
...
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).
Add some docstrings too.
2019-08-20 17:42:45 +01:00
Richard van der Hoff
baa3f4a80d
Avoid deep recursion in appservice recovery ( #5885 )
...
Hopefully, this will fix a stack overflow when recovering an appservice.
The recursion here leads to a huge chain of deferred callbacks, which then
overflows the stack when the chain completes. `inlineCallbacks` makes a better
job of this if we use iteration instead.
Clean up the code a bit too, while we're there.
2019-08-20 17:39:38 +01:00
Jorik Schellekens
c886f976e0
Opentracing doc update ( #5776 )
...
Update opentracing docs to use the unified 'trace' method
2019-08-20 13:56:03 +01:00
Erik Johnston
74f016d343
Remove now unused pick_server_from_list
2019-08-20 12:37:08 +01:00
Erik Johnston
1f9df1cc7b
Fixup _sort_server_list to be slightly more efficient
...
Also document that we are using the algorithm described in RFC2782 and
ensure we handle zero weight correctly.
2019-08-20 12:36:11 +01:00