Commit Graph

11801 Commits

Author SHA1 Message Date
Patrick Cloke
ac3264bf1e 1.13.0 2020-05-19 09:19:09 -04:00
Patrick Cloke
3c8a57f080 1.13.0rc3 2020-05-18 10:58:51 -04:00
Patrick Cloke
56db0b1365
Hash passwords earlier in the registration process (#7523) 2020-05-18 09:46:18 -04:00
Richard van der Hoff
623abbeb8d 1.13.0rc2 2020-05-14 11:01:24 +01:00
Richard van der Hoff
a0e063387d
Stop get_joined_users corruption from custom statuses (#7376)
Fix a bug where the `get_joined_users` cache could be corrupted by custom
status events (or other state events with a state_key matching the user ID).

The bug was introduced by #2229, but has largely gone unnoticed since then.

Fixes #7099, #7373.
2020-05-14 10:07:54 +01:00
Patrick Cloke
5d64fefd6c
Do not validate that the client dict is stable during UI Auth. (#7483)
This backs out some of the validation for the client dictionary and logs if
this changes during a user interactive authentication session instead.
2020-05-13 14:26:44 -04:00
Patrick Cloke
edd3b0747c
Fix new flake8 errors (#7489)
This is a cherry-pick of 1a1da60ad2 (#7470)
to the release-v1.13.0 branch.
2020-05-13 08:24:50 -04:00
Andrew Morgan
20ffaa7209 1.13.0rc1 2020-05-11 14:54:38 +01:00
Patrick Cloke
0ad6d28b0d
Rework UI Auth session validation for registration (#7455)
Be less strict about validation of UI authentication sessions during
registration to match client expecations.
2020-05-08 16:08:58 -04:00
Richard van der Hoff
aa5aa6f96a
Fix errors from malformed log line (#7454) 2020-05-07 19:51:38 +01:00
Richard van der Hoff
da9b2db3af
Drop support for redis.dbid (#7450)
Since we only use pubsub, the dbid is irrelevant.
2020-05-07 16:46:15 +01:00
Brendan Abolivier
d9b8d27494
Add a configuration setting for the dummy event threshold (#7422)
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
Patrick Cloke
d7c2df2fa3
Improve per-block CPU and DB usage metrics (#7426) 2020-05-06 16:43:39 -04:00
Andrew Morgan
4162c39dcf
Port group attestation renewal slow down from matrix-org-hotfixes (#7442) 2020-05-06 20:21:38 +01:00
Richard van der Hoff
e053c86a96
Make redis go faster with hiredis (#7439)
For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.
2020-05-06 17:36:46 +01:00
Andrew Morgan
aee9130a83
Stop Auth methods from polling the config on every req. (#7420) 2020-05-06 15:54:58 +01:00
Erik Johnston
b26f3e582c
Merge pull request #7423 from matrix-org/erikj/faster_device_lists_fetch
Speed up fetching device lists changes in sync.
2020-05-06 11:14:13 +01:00
Richard van der Hoff
a8c17da245 Merge branch 'release-v1.13.0' into rav/fix_dropped_messages 2020-05-05 23:01:12 +01:00
Richard van der Hoff
1242267316 Merge branch 'release-v1.13.0' into rav/fix_dropped_messages 2020-05-05 22:38:44 +01:00
Richard van der Hoff
7f7eedbebb Wait for a POSITION on the right connection before accepting RDATA
... otherwise we can believe we're up to date when we're not.
2020-05-05 22:38:16 +01:00
Brendan Abolivier
5b8023dc7f
Move logs about discarded RDATA to debug (#7421) 2020-05-05 21:07:33 +02:00
Richard van der Hoff
d78265af0c Wait to subscribe before sending REPLICATE 2020-05-05 19:31:37 +01:00
Richard van der Hoff
13dd458b8d Merge branch 'release-v1.13.0' into erikj/faster_device_lists_fetch 2020-05-05 18:14:00 +01:00
Erik Johnston
f9073893af Speed up fetching device lists changes in sync.
Currently we copy `users_who_share_room` needlessly about three times,
which is expensive when the set is large (which it can easily be).
2020-05-05 17:40:29 +01:00
Patrick Cloke
fe69fb6263
Add backwards compatibility codepath to LoggingContext. (#7408) 2020-05-05 09:21:34 -04:00
Erik Johnston
7941a70fa8
Fix bug in EventContext.deserialize. (#7393)
This caused `prev_state_ids` to be incorrect if the state event was not
replacing an existing state entry.
2020-05-05 14:17:27 +01:00
Erik Johnston
350421e058
Fix redis password support. (#7401)
We forgot to set the password on the subscriber connection, as well as
not calling super methods for overridden connectionMade/connectionLost
functions.
2020-05-04 14:04:09 +01:00
Patrick Cloke
032e5a2aca
Convert synapse.server_notices to async/await. (#7394) 2020-05-01 15:28:59 -04:00
Patrick Cloke
b0cbc57375
Convert the synapse.notifier module to async/await. (#7395) 2020-05-01 15:14:49 -04:00
Richard van der Hoff
97ef1471c6
Wait for current_state_events_membership before populate_stats_process_rooms (#7387)
populate_stats_process_rooms was added in #5971 / v1.4.0; current_state_events_membership was added in #5706 / v1.3.0.

Fixes #7380.
2020-05-01 18:05:58 +01:00
Erik Johnston
0e719f2398
Thread through instance name to replication client. (#7369)
For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
2020-05-01 17:19:56 +01:00
Erik Johnston
3085cde577
Use stream.current_token() and remove stream_positions() (#7172)
We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
2020-05-01 15:21:35 +01:00
Andrew Morgan
6b22921b19
async/await is_server_admin (#7363) 2020-05-01 15:15:36 +01:00
Andrew Morgan
2e8955f4a6
Further improvements to requesting the public rooms list on a homeserver which has it set to private (#7368) 2020-05-01 15:15:08 +01:00
Richard van der Hoff
b2dba06079
Workaround for assertion errors from db_query_to_update_function (#7378)
Hopefully this is no worse than what we have on master...
2020-05-01 09:25:16 +01:00
Patrick Cloke
627b0f5f27
Persist user interactive authentication sessions (#7302)
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
2020-04-30 13:47:49 -04:00
Andrew Morgan
9d8ecc9e6c
Apply federation check for /publicRooms with filter list (#7367) 2020-04-30 11:38:07 +01:00
Erik Johnston
37f6823f5b
Add instance name to RDATA/POSITION commands (#7364)
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
2020-04-29 16:23:08 +01:00
Erik Johnston
3eab76ad43
Don't relay REMOTE_SERVER_UP cmds to same conn. (#7352)
For direct TCP connections we need the master to relay REMOTE_SERVER_UP
commands to the other connections so that all instances get notified
about it. The old implementation just relayed to all connections,
assuming that sending back to the original sender of the command was
safe. This is not true for redis, where commands sent get echoed back to
the sender, which was causing master to effectively infinite loop
sending and then re-receiving REMOTE_SERVER_UP commands that it sent.

The fix is to ensure that we only relay to *other* connections and not
to the connection we received the notification from.

Fixes #7334.
2020-04-29 14:10:59 +01:00
Richard van der Hoff
c2e1a2110f
Fix limit logic for EventsStream (#7358)
* Factor out functions for injecting events into database

I want to add some more flexibility to the tools for injecting events into the
database, and I don't want to clutter up HomeserverTestCase with them, so let's
factor them out to a new file.

* Rework TestReplicationDataHandler

This wasn't very easy to work with: the mock wrapping was largely superfluous,
and it's useful to be able to inspect the received rows, and clear out the
received list.

* Fix AssertionErrors being thrown by EventsStream

Part of the problem was that there was an off-by-one error in the assertion,
but also the limit logic was too simple. Fix it all up and add some tests.
2020-04-29 12:30:36 +01:00
Andrew Morgan
eeef9633af
Fix fallback value for account_threepid_delegates.email (#7316) 2020-04-29 11:30:06 +01:00
Manuel Stahl
04dd7d182d
Return total number of users and profile attributes in admin users endpoint (#6881)
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-04-28 18:19:36 +01:00
Andrew Morgan
ce207aa0ff Fix typo 'datbases' in ConfigError 2020-04-28 16:53:10 +01:00
Erik Johnston
38919b521e
Run replication streamers on workers (#7146)
Currently we never write to streams from workers, but that will change soon
2020-04-28 13:34:12 +01:00
Richard van der Hoff
07337fe30b
Fix incorrect metrics reporting for renew_attestations (#7344)
We need to wait for the renewals to finish, so that the metrics are correctly
reported.
2020-04-27 22:20:10 +01:00
Brendan Abolivier
cc9eceb00d
Don't crash when one of the configuration files is empty (#7341)
If the admin adds a `.yaml` file that's either empty or doesn't parse into a dict to a config directory (e.g. `conf.d` for debs installs), stuff like https://github.com/matrix-org/synapse/issues/7322 would happen. This PR checks that the file is correctly parsed into a dict, or ignores it with a warning if it parses into any other type (including `None` for empty files).

Fixes https://github.com/matrix-org/synapse/issues/7322
2020-04-27 15:01:03 +02:00
Patrick Cloke
7bfe0902ce
Add documentation to the sample config about the templates for SSO. (#7343) 2020-04-24 15:03:49 -04:00
Patrick Cloke
33bceb7f70
Convert some of the federation handler methods to async/await. (#7338) 2020-04-24 14:36:38 -04:00
Richard van der Hoff
69a1ac00b2
Merge pull request #7337 from matrix-org/rav/fix_update_limit_assertion
Fix assertions being thrown by the EventsStream update function
2020-04-24 14:00:29 +01:00
Richard van der Hoff
ce428a1abe Fix EventsStream raising assertions when it falls behind
Figuring out how to correctly limit updates from this stream without dropping
entries is far more complicated than just counting the number of rows being
returned. We need to consider each query separately and, if any one query hits
the limit, truncate the results from the others.

I think this also fixes some potentially long-standing bugs where events or
state changes could get missed if we hit the limit on either query.
2020-04-24 13:59:21 +01:00