Commit Graph

927 Commits

Author SHA1 Message Date
Patrick Cloke
1619802228
Various clean-ups to the logging context code (#8935) 2020-12-14 14:19:47 -05:00
Patrick Cloke
344ab0b53a
Default to blacklisting reserved IP ranges and add a whitelist. (#8870)
This defaults `ip_range_blacklist` to reserved IP ranges and also adds an
`ip_range_whitelist` setting to override it.
2020-12-09 13:56:06 -05:00
Richard van der Hoff
ab7a24cc6b
Better formatting for config errors from modules (#8874)
The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError,
and it will be magically turned into a legible error message. There's a few components to it:

* Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it
   into an absolute path.
* Generally improving the way ConfigErrors get printed.
* Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
2020-12-08 14:04:35 +00:00
Richard van der Hoff
025fa06fc7
Clarify config template comments (#8891) 2020-12-08 14:03:08 +00:00
Patrick Cloke
96358cb424
Add authentication to replication endpoints. (#8853)
Authentication is done by checking a shared secret provided
in the Synapse configuration file.
2020-12-04 10:56:28 -05:00
Patrick Cloke
30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Andrew Morgan
17fa58bdd1
Add a config option to change whether unread push notification counts are per-message or per-room (#8820)
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).

However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
2020-11-30 18:43:54 +00:00
Andrew Morgan
2b110dda2a
Fix the formatting of push config section (#8818)
This PR updates the push config's formatting to better align with our [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).
2020-11-25 21:02:53 +00:00
Ben Banfield-Zanin
53a6f5ddf0
SAML: Allow specifying the IdP entityid to use. (#8630)
If the SAML metadata includes multiple IdPs it is necessary to
specify which IdP to redirect users to for authentication.
2020-11-19 09:57:13 -05:00
Marcus Schopen
d356588339
SAML: Document allowing a clock/time difference from IdP (#8731)
Updates the sample configuration with the pysaml2 configuration for
accepting clock skew/drift between the homeserver and IdP.
2020-11-18 07:36:28 -05:00
Adrian Wannenmacher
f1de4bb58b
Clarify the usecase for an msisdn delegate (#8734)
Signed-off-by: Adrian Wannenmacher <tfld@tfld.dev>
2020-11-14 23:09:36 +00:00
Marcus Schopen
68fc0dcb5a
SAML: add <mdui:UIInfo> element examples (#8718)
add some mdui:UIInfo element examples for saml2_config in homeserver.yaml
2020-11-13 12:07:50 +00:00
Patrick Cloke
8f1aefa694
Improve the sample config for SSO (OIDC, SAML, and CAS). (#8635) 2020-10-30 10:01:59 -04:00
Patrick Cloke
00b24aa545
Support generating structured logs in addition to standard logs. (#8607)
This modifies the configuration of structured logging to be usable from
the standard Python logging configuration.

This also separates the formatting of logs from the transport allowing
JSON logs to files or standard logs to sockets.
2020-10-29 07:27:37 -04:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Will Hunt
c276bd9969
Send some ephemeral events to appservices (#8437)
Optionally sends typing, presence, and read receipt information to appservices.
2020-10-15 12:33:28 -04:00
Mateusz Przybyłowicz
ca2db5dd0c
Increase default max_upload_size from 10M to 50M (#8502)
Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
2020-10-09 16:58:23 +01:00
Richard van der Hoff
785437dc0d
Update default room version to 6 (#8461)
Per https://github.com/matrix-org/matrix-doc/pull/2788
2020-10-05 21:40:51 +01:00
Patrick Cloke
62894673e6
Allow background tasks to be run on a separate worker. (#8369) 2020-10-02 08:23:15 -04:00
Richard van der Hoff
462e681c79 Synapse 1.21.0rc2 (2020-10-02)
==============================
 
 Features
 --------
 
 - Convert additional templates from inline HTML to Jinja2 templates. ([\#8444](https://github.com/matrix-org/synapse/issues/8444))
 
 Bugfixes
 --------
 
 - Fix a regression in v1.21.0rc1 which broke thumbnails of remote media. ([\#8438](https://github.com/matrix-org/synapse/issues/8438))
 - Do not expose the experimental `uk.half-shot.msc2778.login.application_service` flow in the login API, which caused a compatibility problem with Element iOS. ([\#8440](https://github.com/matrix-org/synapse/issues/8440))
 - Fix malformed log line in new federation "catch up" logic. ([\#8442](https://github.com/matrix-org/synapse/issues/8442))
 - Fix DB query on startup for negative streams which caused long start up times. Introduced in [\#8374](https://github.com/matrix-org/synapse/issues/8374). ([\#8447](https://github.com/matrix-org/synapse/issues/8447))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl93FccACgkQOSor00I9
 eP9/Egf7B4YOF6tniyAXxZvmvFOwV1WNw4sbFmF+czUKHBTAwS/Ij9MbutulD4OB
 +yqHAvu15qUCQR/G+KGjyHBDtESEUtn5SRy8znLYlR2n3qfEdEpd5y6LJSq4s7sr
 NjFVNVI1g5L8PmbvvWCINfpPm2JSm8zyOdyxy4KZifex1B+8YgPILeQOB59sWL/H
 1maFbHCgepqO3jotsA8PUXQZx5oScABmqYYe92b4sLna00uFBq2NWp0NA654dRqK
 VRFlGzId1fZNWTy1jzfOY2sJKpBCy4cMrtfGJ/eqMtryHqbnBFT6hgB8FyTNg0h0
 oew+BLV/mcJLcvB0ALRMFS7xZHdoxQ==
 =+3N3
 -----END PGP SIGNATURE-----

Merge tag 'v1.21.0rc2' into develop

Synapse 1.21.0rc2 (2020-10-02)
==============================

Features
--------

- Convert additional templates from inline HTML to Jinja2 templates. ([\#8444](https://github.com/matrix-org/synapse/issues/8444))

Bugfixes
--------

- Fix a regression in v1.21.0rc1 which broke thumbnails of remote media. ([\#8438](https://github.com/matrix-org/synapse/issues/8438))
- Do not expose the experimental `uk.half-shot.msc2778.login.application_service` flow in the login API, which caused a compatibility problem with Element iOS. ([\#8440](https://github.com/matrix-org/synapse/issues/8440))
- Fix malformed log line in new federation "catch up" logic. ([\#8442](https://github.com/matrix-org/synapse/issues/8442))
- Fix DB query on startup for negative streams which caused long start up times. Introduced in [\#8374](https://github.com/matrix-org/synapse/issues/8374). ([\#8447](https://github.com/matrix-org/synapse/issues/8447))
2020-10-02 12:59:17 +01:00
Patrick Cloke
34ff8da83b
Convert additional templates to Jinja (#8444)
This converts a few more of our inline HTML templates to Jinja. This is somewhat part of #7280 and should make it a bit easier to customize these in the future.
2020-10-02 11:15:53 +01:00
BBBSnowball
05ee048f2c
Add config option for always using "userinfo endpoint" for OIDC (#7658)
This allows for connecting to certain IdPs, e.g. GitLab.
2020-10-01 13:54:35 -04:00
Patrick Cloke
4ff0201e62
Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
Richard van der Hoff
c1ef579b63
Add prometheus metrics to track federation delays (#8430)
Add a pair of federation metrics to track the delays in sending PDUs to/from 
particular servers.
2020-10-01 11:09:12 +01:00
Patrick Cloke
8b40843392
Allow additional SSO properties to be passed to the client (#8413) 2020-09-30 13:02:43 -04:00
Aaron Raimist
8238b55e08
Update description of server_name config option (#8415) 2020-09-29 13:50:25 -04:00
Tdxdxoz
abd04b6af0
Allow existing users to login via OpenID Connect. (#8345)
Co-authored-by: Benjamin Koch <bbbsnowball@gmail.com>

This adds configuration flags that will match a user to pre-existing users
when logging in via OpenID Connect. This is useful when switching to
an existing SSO system.
2020-09-25 07:01:45 -04:00
Andrew Morgan
c77c4a2fcd Merge branch 'master' into develop 2020-09-24 17:00:33 +01:00
Andrew Morgan
3f4a2a7064
Hotfix: disable autoescape by default when rendering Jinja2 templates (#8394)
#8037 changed the default `autoescape` option when rendering Jinja2 templates from `False` to `True`. This caused some bugs, noticeably around redirect URLs being escaped in SAML2 auth confirmation templates, causing those URLs to break for users.

This change returns the previous behaviour as it stood. We may want to look at each template individually and see whether autoescaping is a good idea at some point, but for now lets just fix the breakage.
2020-09-24 16:24:08 +01:00
Patrick Cloke
8a4a4186de
Simplify super() calls to Python 3 syntax. (#8344)
This converts calls like super(Foo, self) -> super().

Generated with:

    sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
2020-09-18 09:56:44 -04:00
Jonathan de Jong
837293c314
Remove obsolete __future__ imports (#8337) 2020-09-17 08:37:01 -04:00
Patrick Cloke
6605470bfb
Improve SAML error messages (#8248) 2020-09-14 09:05:36 -04:00
Erik Johnston
04cc249b43
Add experimental support for sharding event persister. Again. (#8294)
This is *not* ready for production yet. Caveats:

1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-14 10:16:41 +01:00
Andrew Morgan
a3a90ee031
Show a confirmation page during user password reset (#8004)
This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset.

This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
2020-09-10 11:45:12 +01:00
Andrew Morgan
094896a69d
Add a config option for validating 'next_link' parameters against a domain whitelist (#8275)
This is a config option ported over from DINUM's Sydent: https://github.com/matrix-org/sydent/pull/285

They've switched to validating 3PIDs via Synapse rather than Sydent, and would like to retain this functionality.

This original purpose for this change is phishing prevention. This solution could also potentially be replaced by a similar one to https://github.com/matrix-org/synapse/pull/8004, but across all `*/submit_token` endpoint.

This option may still be useful to enterprise even with that safeguard in place though, if they want to be absolutely sure that their employees don't follow links to other domains.
2020-09-08 16:03:09 +01:00
Richard van der Hoff
77794ebc77
Fix stack overflow when logging system encounters an error (#8268) 2020-09-07 16:54:30 +01:00
Patrick Cloke
db7de4d182
Fix a regression from calling read_templates. (#8252)
Regressed in #8037.
2020-09-04 09:10:33 -04:00
Patrick Cloke
c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Brendan Abolivier
9f8abdcc38
Revert "Add experimental support for sharding event persister. (#8170)" (#8242)
* Revert "Add experimental support for sharding event persister. (#8170)"

This reverts commit 82c1ee1c22.

* Changelog
2020-09-04 10:19:42 +01:00
Erik Johnston
82c1ee1c22
Add experimental support for sharding event persister. (#8170)
This is *not* ready for production yet. Caveats:

1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-02 15:48:37 +01:00
Brendan Abolivier
420484a334
Allow capping a room's retention policy (#8104) 2020-08-24 18:21:04 +01:00
Andrew Morgan
8b6c176aee
Add resources.consent conditional dependency back (#8107)
Turns out that part of the codebase (synapse.config.server) checks for this key explicitly. Remove that check.
2020-08-18 10:59:54 +01:00
Andrew Morgan
e04e465b4d
Use the default templates when a custom template file cannot be found (#8037)
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
Erik Johnston
0304ad0c3d Move setting of Filter into code.
We do this to prevent foot guns. The default config uses a MemoryFilter,
but users are free to change to logging to files directly. If they do
then they have to ensure to set the `filters: [context]` on the right
handler, otherwise records get written with the wrong context.

Instead we move the logic to happen when we generate a record, which is
when we *log* rather than *handle*.

(It's possible to add filters to loggers in the config, however they
don't apply to descendant loggers and so they have to be manually set on
*every* logger used in the code base)
2020-08-11 18:10:46 +01:00
Erik Johnston
db131b6b22
Change the default log config to reduce disk I/O and storage (#8040)
* Change default log config to buffer by default.

This batches up writes to the filesystem, which is more efficient for
disk I/O. This means that it can take some time for logs to get written
to disk. Note that ERROR logs (and above) immediately flush the buffer.

This only effects new installs, as we only write the log config if
started with `--generate-config` (in the same way we do for generating
signing keys).

* Default to keeping last 4 days of logs.

This hopefully reduces the amount of logs kept for new servers. Keeping
the last 1GB of logs is likely overkill for new servers, but equally may
not be enough for busy ones.

Instead, we keep the last four days worth of logs, enough so that admins
can investigate any problems that happened over e.g. a long weekend.
2020-08-11 18:09:46 +01:00
Richard van der Hoff
0cb169900e
Implement login blocking based on SAML attributes (#8052)
Hopefully this mostly speaks for itself. I also did a bit of cleaning up of the
error handling.

Fixes #8047
2020-08-11 16:08:10 +01:00
Brendan Abolivier
5c43c43240
Typo 2020-08-10 11:23:24 +01:00
Brendan Abolivier
1a3aabcf3f
Lint 2020-08-10 11:13:21 +01:00
Brendan Abolivier
cee6c6012e
why mypy why 2020-08-10 11:10:34 +01:00
Brendan Abolivier
367e9e6e9e
Lint 2020-08-06 17:57:58 +01:00