Commit Graph

1189 Commits

Author SHA1 Message Date
Brendan Abolivier
9f8abdcc38
Revert "Add experimental support for sharding event persister. (#8170)" (#8242)
* Revert "Add experimental support for sharding event persister. (#8170)"

This reverts commit 82c1ee1c22.

* Changelog
2020-09-04 10:19:42 +01:00
Erik Johnston
82c1ee1c22
Add experimental support for sharding event persister. (#8170)
This is *not* ready for production yet. Caveats:

1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-02 15:48:37 +01:00
Brendan Abolivier
420484a334
Allow capping a room's retention policy (#8104) 2020-08-24 18:21:04 +01:00
Andrew Morgan
8b6c176aee
Add resources.consent conditional dependency back (#8107)
Turns out that part of the codebase (synapse.config.server) checks for this key explicitly. Remove that check.
2020-08-18 10:59:54 +01:00
Andrew Morgan
e04e465b4d
Use the default templates when a custom template file cannot be found (#8037)
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
Erik Johnston
0304ad0c3d Move setting of Filter into code.
We do this to prevent foot guns. The default config uses a MemoryFilter,
but users are free to change to logging to files directly. If they do
then they have to ensure to set the `filters: [context]` on the right
handler, otherwise records get written with the wrong context.

Instead we move the logic to happen when we generate a record, which is
when we *log* rather than *handle*.

(It's possible to add filters to loggers in the config, however they
don't apply to descendant loggers and so they have to be manually set on
*every* logger used in the code base)
2020-08-11 18:10:46 +01:00
Erik Johnston
db131b6b22
Change the default log config to reduce disk I/O and storage (#8040)
* Change default log config to buffer by default.

This batches up writes to the filesystem, which is more efficient for
disk I/O. This means that it can take some time for logs to get written
to disk. Note that ERROR logs (and above) immediately flush the buffer.

This only effects new installs, as we only write the log config if
started with `--generate-config` (in the same way we do for generating
signing keys).

* Default to keeping last 4 days of logs.

This hopefully reduces the amount of logs kept for new servers. Keeping
the last 1GB of logs is likely overkill for new servers, but equally may
not be enough for busy ones.

Instead, we keep the last four days worth of logs, enough so that admins
can investigate any problems that happened over e.g. a long weekend.
2020-08-11 18:09:46 +01:00
Richard van der Hoff
0cb169900e
Implement login blocking based on SAML attributes (#8052)
Hopefully this mostly speaks for itself. I also did a bit of cleaning up of the
error handling.

Fixes #8047
2020-08-11 16:08:10 +01:00
Brendan Abolivier
5c43c43240
Typo 2020-08-10 11:23:24 +01:00
Brendan Abolivier
1a3aabcf3f
Lint 2020-08-10 11:13:21 +01:00
Brendan Abolivier
cee6c6012e
why mypy why 2020-08-10 11:10:34 +01:00
Brendan Abolivier
367e9e6e9e
Lint 2020-08-06 17:57:58 +01:00
Brendan Abolivier
bf33d5c457
Incorporate review 2020-08-06 17:52:34 +01:00
Brendan Abolivier
118a9eafb3 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules 2020-08-06 10:52:50 +01:00
Brendan Abolivier
dd11f575a2
Incorporate review 2020-08-06 10:52:26 +01:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Brendan Abolivier
1678057b56
Back out the database hack and replace it with a temporary config setting 2020-08-03 11:22:22 +01:00
Erik Johnston
faba873d4b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/add_rate_limiting_to_joins 2020-07-31 15:07:01 +01:00
Erik Johnston
18de00adb4 Add ratelimiting on joins 2020-07-31 15:06:56 +01:00
Olivier Wilkinson (reivilibre)
3aa36b782c Merge branch 'master' into develop 2020-07-30 15:18:36 +01:00
Erik Johnston
2c1b9d6763
Update worker docs with recent enhancements (#7969) 2020-07-29 23:22:13 +01:00
Aaron Raimist
2184f61fae
Various improvements to the docs (#7899) 2020-07-29 10:35:44 -04:00
lugino-emeritus
3857de2194
Option to allow server admins to join complex rooms (#7902)
Fixes #7901.

Signed-off-by: Niklas Tittjung <nik_t.01@web.de>
2020-07-28 13:41:44 +01:00
Adrian
64d2280299
Fix a typo in the sample config. (#7890) 2020-07-20 13:42:52 -04:00
Andrew Morgan
5ecf98f59e
Change sample config's postgres user to synapse_user (#7889)
The [postgres setup docs](https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md#set-up-database) recommend setting up your database with user `synapse_user`.

However, uncommenting the postgres defaults in the sample config leave you with user `synapse`.

This PR switches the sample config to recommend `synapse_user`. Took a me a second to figure this out, so assume this will beneficial to others.
2020-07-20 18:29:25 +01:00
Patrick Cloke
852930add7
Add a default limit (of 100) to get/sync operations. (#7858) 2020-07-17 07:59:23 -04:00
Erik Johnston
f2e38ca867
Allow moving typing off master (#7869) 2020-07-16 15:12:54 +01:00
Erik Johnston
649a7ead5c
Add ability to run multiple pusher instances (#7855)
This reuses the same scheme as federation sender sharding
2020-07-16 14:06:28 +01:00
Brendan Abolivier
85223106f3
Allow email subjects to be customised through Synapse's configuration (#7846) 2020-07-14 19:10:42 +01:00
Patrick Cloke
77d2c05410
Add the option to validate the iss and aud claims for JWT logins. (#7827) 2020-07-14 07:16:43 -04:00
Brendan Abolivier
504c8f3483
Fix handling of "off" in encryption_enabled_by_default_for_room_type (#7822)
Fixes https://github.com/matrix-org/synapse/issues/7821, introduced in https://github.com/matrix-org/synapse/pull/7639

Turns out PyYAML translates `off` into a `False` boolean if it's
unquoted (see https://stackoverflow.com/questions/36463531/pyyaml-automatically-converting-certain-keys-to-boolean-values),
which seems to be a liberal interpretation of this bit of the YAML spec: https://yaml.org/spec/1.1/current.html#id864510

An alternative fix would be to implement the solution mentioned in the
SO post linked above, but I'm aware it might break existing setups
(which might use these values in the configuration file) so it's
probably better just to add an extra check for this one. We should be
aware that this is a thing for the next times we do that though.

I didn't find any other occurrence of this bug elsewhere in the
codebase.
2020-07-13 17:14:42 +01:00
Erik Johnston
f299441cc6
Add ability to shard the federation sender (#7798) 2020-07-10 18:26:36 +01:00
Patrick Cloke
38e1fac886
Fix some spelling mistakes / typos. (#7811) 2020-07-09 09:52:58 -04:00
Patrick Cloke
2a266f4511
Add documentation for JWT login type and improve sample config. (#7776) 2020-07-06 08:31:51 -04:00
Will Hunt
62b1ce8539
isort 5 compatibility (#7786)
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
2020-07-05 16:32:02 +01:00
Patrick Cloke
71cccf1593
Additional configuration options for auto-join rooms (#7763) 2020-06-30 15:41:36 -04:00
Erik Johnston
b44bdd7f7b
Support running multiple media repos. (#7706)
This requires a new config option to specify which media repo should be
responsible for running background jobs to e.g. clear out expired URL
preview caches.
2020-06-17 14:13:30 +01:00
Richard van der Hoff
e452973fd2
fix broken link in sample config (#7712) 2020-06-16 19:50:16 +01:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Richard van der Hoff
03619324fc
Create a ListenerConfig object (#7681)
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Patrick Cloke
b9df7f70bb
Increase the default SAML session expirary time to 15 minutes. (#7664) 2020-06-11 07:55:45 -04:00
wondratsch
c746889bb0
fix typo in sample_config.yaml (#7652)
Just a simple typo fix.

Signed-off-by: wondratsch 28294257+wondratsch@users.noreply.github.com
2020-06-11 11:51:10 +01:00
Richard van der Hoff
0df618f813
Take out a lock before modifying _CACHES (#7663)
This should fix #7610.
2020-06-10 18:27:49 +01:00
Andrew Morgan
fcd6961441
Add option to enable encryption by default for new rooms (#7639)
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on #7637
2020-06-10 17:44:34 +01:00
Travis Ralston
09099313e6
Add an option to disable autojoin for guest accounts (#6637)
Fixes https://github.com/matrix-org/synapse/issues/3177
2020-06-05 18:18:15 +01:00
WGH
e55ee7c32f
Add support for webp thumbnailing (#7586)
Closes #4382

Signed-off-by: Maxim Plotnikov <wgh@torlan.ru>
2020-06-05 11:54:27 +01:00
Andrew Morgan
f4e6495b5d
Performance improvements and refactor of Ratelimiter (#7595)
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:

* Rather undocumented, and
* causing a *lot* of config checks

This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. 

Best to be reviewed commit-by-commit.
2020-06-05 10:47:20 +01:00
Richard van der Hoff
11de843626
Cleanups to the OpenID Connect integration (#7628)
docs, default configs, comments. Nothing very significant.
2020-06-03 21:13:17 +01:00
Richard van der Hoff
1bbc9e2df6
Clean up exception handling in SAML2ResponseResource (#7614)
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string

* Clean up exception handling in SAML2ResponseResource

  * use the existing code in `return_html_error` instead of re-implementing it
    (giving it a jinja2 template rather than inventing a new form of template)

  * do the exception-catching in the REST layer rather than in the handler
    layer, to make sure we catch all exceptions.
2020-06-03 10:41:12 +01:00
Jason Robinson
4be968d05d
Fix sample config docs error (#7581)
'client_auth_method' commented out value was erronously 'client_auth_basic',
when code and docstring says it should be 'client_secret_basic'.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2020-05-27 13:52:18 +01:00
Erik Johnston
d7d8a2e7ee Fix up comments 2020-05-27 13:34:46 +01:00
Erik Johnston
4ba55559ac
Fix specifying cache factors via env vars with * in name. (#7580)
This mostly applise to `*stateGroupCache*` and co.

Broke in #6391.
2020-05-27 13:17:01 +01:00
Erik Johnston
e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Richard van der Hoff
66a564c859
Fix some DETECTED VIOLATIONS in the config file (#7550)
consistency ftw
2020-05-22 10:11:50 +01:00
Erik Johnston
1de36407d1
Add instance_map config and route replication calls (#7495) 2020-05-14 14:00:58 +01:00
Paul Tötterman
46cb2550bb
Fix copypasted comment (#7477)
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2020-05-13 16:55:43 +01:00
Erik Johnston
1a1da60ad2
Fix new flake8 errors (#7470) 2020-05-12 11:20:48 +01:00
Amber Brown
7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Andrew Morgan
5cf758cdd6 Merge branch 'release-v1.13.0' into develop
* release-v1.13.0:
  Don't UPGRADE database rows
  RST indenting
  Put rollback instructions in upgrade notes
  Fix changelog typo
  Oh yeah, RST
  Absolute URL it is then
  Fix upgrade notes link
  Provide summary of upgrade issues in changelog. Fix )
  Move next version notes from changelog to upgrade notes
  Changelog fixes
  1.13.0rc1
  Documentation on setting up redis (#7446)
  Rework UI Auth session validation for registration (#7455)
  Fix errors from malformed log line (#7454)
  Drop support for redis.dbid (#7450)
2020-05-11 16:46:33 +01:00
Andrew Morgan
67feea8044
Extend spam checker to allow for multiple modules (#7435) 2020-05-08 19:25:48 +01:00
Quentin Gliech
616af44137
Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Richard van der Hoff
da9b2db3af
Drop support for redis.dbid (#7450)
Since we only use pubsub, the dbid is irrelevant.
2020-05-07 16:46:15 +01:00
Brendan Abolivier
d9b8d27494
Add a configuration setting for the dummy event threshold (#7422)
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
Andrew Morgan
eeef9633af
Fix fallback value for account_threepid_delegates.email (#7316) 2020-04-29 11:30:06 +01:00
Andrew Morgan
ce207aa0ff Fix typo 'datbases' in ConfigError 2020-04-28 16:53:10 +01:00
Brendan Abolivier
cc9eceb00d
Don't crash when one of the configuration files is empty (#7341)
If the admin adds a `.yaml` file that's either empty or doesn't parse into a dict to a config directory (e.g. `conf.d` for debs installs), stuff like https://github.com/matrix-org/synapse/issues/7322 would happen. This PR checks that the file is correctly parsed into a dict, or ignores it with a warning if it parses into any other type (including `None` for empty files).

Fixes https://github.com/matrix-org/synapse/issues/7322
2020-04-27 15:01:03 +02:00
Patrick Cloke
7bfe0902ce
Add documentation to the sample config about the templates for SSO. (#7343) 2020-04-24 15:03:49 -04:00
Brendan Abolivier
2e3b9a0fcb
Revert "Revert "Merge pull request #7315 from matrix-org/babolivier/request_token""
This reverts commit 1adf6a5587.
2020-04-23 11:23:53 +02:00
Erik Johnston
51f7eaf908
Add ability to run replication protocol over redis. (#7040)
This is configured via the `redis` config options.
2020-04-22 13:07:41 +01:00
Lars Franke
13917232d5
Fix indention in generated config file (#7300)
Also adjust sample_config.yaml

Signed-off-by: Lars Franke <frcl@mailbox.org>
2020-04-20 16:51:27 +01:00
Patrick Cloke
054c231e58
Use a template for the SSO success page to allow for customization. (#7279) 2020-04-17 13:34:55 -04:00
Tristan Lins
c07fca9e2f
Clarify the comments for media_storage_providers options (#7272) 2020-04-17 07:09:33 -04:00
Andrew Morgan
a48138784e
Allow specifying the value of Accept-Language header for URL previews (#7265) 2020-04-15 13:35:29 +01:00
Patrick Cloke
b85d7652ff
Do not allow a deactivated user to login via SSO. (#7240) 2020-04-09 13:28:13 -04:00
Richard van der Hoff
5a709630bf
Fix --help commandline argument (#7249)
I don't really remember why this was so complicated; I think it dates
back to the time when we had to instantiate the Config classes before
we could call `add_arguments` - ie before #5597. In any case, I don't
think there's a good reason for it any more, and the impact of it
being complicated is that `--help` doesn't work correctly.
2020-04-09 12:44:37 +01:00
Andrew Morgan
29b7e22b93
Add documentation to password_providers config option (#7238) 2020-04-08 00:46:50 +01:00
Martin Milata
b0db928c63
Extend web_client_location to handle absolute URLs (#7006)
Log warning when filesystem path is used.

Signed-off-by: Martin Milata <martin@martinmilata.cz>
2020-04-03 11:57:34 -04:00
Andrew Morgan
d9f29f8dae
Fix a small typo in the metrics_flags config option. (#7171) 2020-03-30 17:38:21 +01:00
Richard van der Hoff
b7da598a61 Always whitelist the login fallback for SSO (#7153)
That fallback sets the redirect URL to itself (so it can process the login
token then return gracefully to the client). This would make it pointless to
ask the user for confirmation, since the URL the confirmation page would be
showing wouldn't be the client's.
2020-03-27 20:24:52 +00:00
Dirk Klimpel
8327eb9280
Add options to prevent users from changing their profile. (#7096) 2020-03-27 19:15:23 +00:00
Nektarios Katakis
825fb5d0a5
Don't default to an invalid sqlite config if no database configuration is provided (#6573) 2020-03-26 17:13:14 +00:00
Dirk Klimpel
e8e2ddb60a
Allow server admins to define and enforce a password policy (MSC2000). (#7118) 2020-03-26 16:51:13 +00:00
Aaron Raimist
6ca5e56fd1
Remove unused captcha_bypass_secret option (#7137)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-03-25 17:49:34 +00:00
Richard van der Hoff
c165c1233b
Improve database configuration docs (#6988)
Attempts to clarify the sample config for databases, and add some stuff about
tcp keepalives to `postgres.md`.
2020-03-20 15:24:22 +00:00
Richard van der Hoff
6a35046363 Revert "Add options to disable setting profile info for prevent changes. (#7053)"
This reverts commit 54dd28621b, reversing
changes made to 6640460d05.
2020-03-17 11:25:01 +00:00
Brendan Abolivier
0de9f9486a
Lint 2020-03-11 20:39:18 +00:00
Brendan Abolivier
f9e98176bf
Put the file in the templates directory 2020-03-11 20:31:42 +00:00
Brendan Abolivier
900bca9707
Update wording and config 2020-03-11 19:40:30 +00:00
Brendan Abolivier
b8cfe79ffc
Move the default SAML2 error HTML to a dedicated file
Also add some JS to it to process any error we might have in the URI
(see #6893).
2020-03-11 19:33:16 +00:00
Brendan Abolivier
54dd28621b
Add options to disable setting profile info for prevent changes. (#7053) 2020-03-10 22:23:01 +00:00
Dirk Klimpel
42ac4ca477
Update synapse/config/registration.py
Co-Authored-By: Brendan Abolivier <github@brendanabolivier.com>
2020-03-10 21:26:55 +01:00
Brendan Abolivier
8f826f98ac
Rephrase default message 2020-03-10 17:22:45 +00:00
Brendan Abolivier
5ec2077bf9
Lint 2020-03-10 14:04:20 +00:00
Brendan Abolivier
6b0efe73e2
SAML2: render a comprehensible error page if something goes wrong
If an error happened while processing a SAML AuthN response, or a client
ends up doing a `GET` request to `/authn_response`, then render a
customisable error page rather than a confusing error.
2020-03-10 13:59:22 +00:00
dklimpel
885134529f updates after review 2020-03-09 22:09:29 +01:00
dklimpel
99bbe177b6 add disable_3pid_changes 2020-03-08 21:58:12 +01:00
dklimpel
20545a2199 lint2 2020-03-08 15:28:00 +01:00
dklimpel
fb078f921b changelog 2020-03-08 15:19:07 +01:00
dklimpel
1f5f3ae8b1 Add options to disable setting profile info for prevent changes. 2020-03-08 14:49:33 +01:00
Brendan Abolivier
43f874055d
Merge branch 'master' into develop 2020-03-03 15:20:49 +00:00
Brendan Abolivier
65c73cdfec Factor out complete_sso_login and expose it to the Module API 2020-03-03 10:54:44 +00:00
Richard van der Hoff
b68041df3d Add a whitelist for the SSO confirmation step. 2020-03-02 17:05:09 +00:00
Brendan Abolivier
b2bd54a2e3 Add a confirmation step to the SSO login flow 2020-03-02 16:36:32 +00:00
Richard van der Hoff
4c2ed3f20e
Fix minor issues with email config (#6962)
* Give `notif_template_html`, `notif_template_text` default values (fixes #6960)
 * Don't complain if `smtp_host` and `smtp_port` are unset, since they have sensible defaults (fixes #6961)
 * Set the example for `enable_notifs` to `True`, for consistency and because it's more useful
 * Raise errors as ConfigError rather than RuntimeError for nicer formatting
2020-02-24 15:18:38 +00:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Brendan Abolivier
d484126bf7
Merge pull request #6907 from matrix-org/babolivier/acme-config
Add mention and warning about ACME v1 deprecation to the TLS config
2020-02-18 16:11:31 +00:00
Richard van der Hoff
97a42bbc3a
Add a warning about indentation to generated config (#6920)
Fixes #6916.
2020-02-14 16:22:30 +00:00
Brendan Abolivier
36af094017
Linters are hard but in they end they just want what's best for us 2020-02-13 17:03:41 +00:00
Brendan Abolivier
ef9c275d96
Add a separator for the config warning 2020-02-13 15:44:14 +00:00
Brendan Abolivier
5820ed905f
Add mention and warning about ACME v1 deprecation to the Synapse config 2020-02-13 14:20:08 +00:00
timfi
bce557175b
Allow empty federation_certificate_verification_whitelist (#6849) 2020-02-06 14:45:01 +00:00
Andrew Morgan
198d52da3a Fix empty account_validity config block 2020-01-20 14:05:29 +00:00
Brendan Abolivier
a17f64361c
Add more logging around message retention policies support (#6717)
So we can debug issues like #6683 more easily
2020-01-17 20:51:44 +00:00
Richard van der Hoff
2b6a77fcde
Delegate remote_user_id mapping to the saml mapping provider (#6723)
Turns out that figuring out a remote user id for the SAML user isn't quite as obvious as it seems. Factor it out to the SamlMappingProvider so that it's easy to control.
2020-01-17 10:32:47 +00:00
Richard van der Hoff
5ce0b17e38
Clarify the account_validity and email sections of the sample configuration. (#6685)
Generally try to make this more comprehensible, and make it match the
conventions.

I've removed the documentation for all the settings which allow you to change
the names of the template files, because I can't really see why they are
useful.
2020-01-17 10:04:15 +00:00
Brendan Abolivier
2b6b7f482a
Merge pull request #6621 from matrix-org/babolivier/purge_job_config_typo
Fix a typo in the purge jobs configuration example
2020-01-07 16:17:40 +01:00
Brendan Abolivier
391fb47791
Reword 2020-01-07 14:54:32 +00:00
Brendan Abolivier
3a86477162
Change the example from 5min to 12h
Have a purge job running every 5min is probably not something we want to advise admins to do as a sort-of default.
2020-01-07 14:53:07 +00:00
Erik Johnston
9f6c1befbb
Add experimental 'databases' config (#6580) 2020-01-06 14:44:01 +00:00
Richard van der Hoff
08815566bc
Automate generation of the sample and debian log configs (#6627) 2020-01-03 17:14:00 +00:00
Richard van der Hoff
e484101306
Raise an error if someone tries to use the log_file config option (#6626)
This has caused some confusion for people who didn't notice it going away.
2020-01-03 17:11:29 +00:00
Richard van der Hoff
98247c4a0e
Remove unused, undocumented "content repo" resource (#6628)
This looks like it got half-killed back in #888.

Fixes #6567.
2020-01-03 17:10:52 +00:00
Brendan Abolivier
4efe1d4d3f
Fix a typo in the purge jobs configuration example 2020-01-03 12:57:24 +01:00
Erik Johnston
75d8f26ac8
Split state groups into a separate data store (#6296) 2019-12-20 10:48:24 +00:00
Richard van der Hoff
b95b762560
Add an export_signing_key script (#6546)
I want to do some key rotation, and it is silly that we don't have a way to do
this.
2019-12-19 11:11:14 +00:00
Erik Johnston
2284eb3a53
Add database config class (#6513)
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Will Hunt
bfb95654c9 Add option to allow profile queries without sharing a room (#6523) 2019-12-16 16:11:55 +00:00
Erik Johnston
495005360c Bump version of mypy 2019-12-12 15:21:12 +00:00
Andrew Morgan
4947de5a14
Allow SAML username provider plugins (#6411) 2019-12-10 17:30:16 +00:00
Neil Johnson
cb0aeb147e
privacy by default for room dir (#6355)
Ensure that the the default settings for the room directory are that the it is hidden from public view by default.
2019-12-04 09:46:16 +00:00
Brendan Abolivier
54dd5dc12b
Add ephemeral messages support (MSC2228) (#6409)
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:

* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
2019-12-03 19:19:45 +00:00
Richard van der Hoff
c48ea98007
Clarifications for the email configuration settings. (#6423)
Cf #6422
2019-11-28 09:29:18 +00:00
Brendan Abolivier
d31f69afa0
Merge pull request #6358 from matrix-org/babolivier/message_retention
Implement message retention policies (MSC1763)
2019-11-27 15:04:38 +00:00
Andrew Morgan
ce578031f4
Remove assertion and provide a clear warning on startup for missing public_baseurl (#6379) 2019-11-26 18:42:27 +00:00
Brendan Abolivier
9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Brendan Abolivier
97b863fe32
Lint again 2019-11-19 13:33:58 +00:00
Brendan Abolivier
bf9a11c54d
Lint again 2019-11-19 13:30:04 +00:00
Brendan Abolivier
7c24d0f443
Lint 2019-11-19 13:22:37 +00:00
Andrew Morgan
473acedcdd Merge branch 'develop' of github.com:matrix-org/synapse into anoa/homeserver_copy
* 'develop' of github.com:matrix-org/synapse:
  Blacklist PurgeRoomTestCase (#6361)
  Set room version default to 5
2019-11-14 10:26:27 +00:00
Brendan Abolivier
a42567e4a8
Merge pull request #6220 from matrix-org/neilj/set_room_version_default_to_5
Set room version default to 5
2019-11-14 10:21:00 +00:00
Andrew Morgan
73d091be48 A couple more instances 2019-11-12 13:12:25 +00:00
Andrew Morgan
bc29a19731 Replace instance variations of homeserver with correct case/spacing 2019-11-12 13:08:12 +00:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Andrew Morgan
fe1f2b4520
Remove last usages of deprecated logging.warn method (#6314) 2019-11-01 12:03:44 +00:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Andrew Morgan
46c12918ad
Fix typo in domain name in account_threepid_delegates config option (#6273) 2019-10-30 11:07:42 +00:00
Neil Johnson
2794b79052 Option to suppress resource exceeded alerting (#6173)
The expected use case is to suppress MAU limiting on small instances
2019-10-24 11:48:46 +01:00
Andrew Morgan
409c62b27b
Add config linting script that checks for bool casing (#6203)
Add a linting script that enforces all boolean values in the default config be lowercase.

This has annoyed me for a while so I decided to fix it.
2019-10-23 13:22:54 +01:00
Neil Johnson
82c8799ec7 Set room version default to 5 2019-10-19 09:06:15 +01:00
Richard van der Hoff
560c122267
Fix logging config for the docker image (#6197)
Turns out that loggers that are instantiated before the config is loaded get
turned off.

Also bring the logging config that is generated by --generate-config into line.

Fixes #6194.
2019-10-18 13:34:33 +02:00
Valérian Rousset
be9b55e0d2 cas: support setting display name (#6114)
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Amber Brown
f743108a94
Refactor HomeserverConfig so it can be typechecked (#6137) 2019-10-10 09:39:35 +01:00
Amber Brown
864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Erik Johnston
3423633d50 Fix 'redaction_retention_period' sampel config to match guidelines 2019-09-26 16:43:52 +01:00
Neil Johnson
8b8f8c7b3c Explicitly log when a homeserver does not have a trusted key server configured (#6090) 2019-09-26 12:57:01 +01:00
Erik Johnston
4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Neil Johnson
a4f3ca48b5 Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. (#5884) 2019-09-25 17:27:35 +01:00
Erik Johnston
39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Richard van der Hoff
9b7c4f4078 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 17:03:50 +01:00
Erik Johnston
242017db8b Prune rows in user_ips older than configured period
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
Richard van der Hoff
bb82be9851
Merge pull request #6069 from matrix-org/rav/fix_attribute_mapping
Fix a bug with saml attribute maps.
2019-09-24 15:07:26 +01:00
Richard van der Hoff
ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Richard van der Hoff
a25b66d3f9 docstrings and comments 2019-09-24 11:15:08 +01:00
Andrew Morgan
50776261e1 Add submit_url response parameter to msisdn /requestToken (#6079)
Second part of solving #6076
Fixes #6076

We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
2019-09-23 21:21:03 +01:00
Richard van der Hoff
78e8ec368e
Merge pull request #6064 from matrix-org/rav/saml_config_cleanup
Make the sample saml config closer to our standards
2019-09-23 20:36:51 +01:00
Andrew Morgan
e08ea43463 Use the federation blacklist for requests to untrusted Identity Servers (#6000)
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.

Fixes #5935
2019-09-23 20:23:20 +01:00
Matthew Hodgson
7ef319aefe fix broken copyrights 2019-09-23 12:28:01 +01:00
Andrew Morgan
df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Richard van der Hoff
284e1cb027 Merge branch 'develop' into rav/fix_attribute_mapping 2019-09-19 20:32:25 +01:00
Richard van der Hoff
b74606ea22 Fix a bug with saml attribute maps.
Fixes a bug where the default attribute maps were prioritised over
user-specified ones, resulting in incorrect mappings.

The problem is that if you call SPConfig.load() multiple times, it adds new
attribute mappers to a list. So by calling it with the default config first,
and then the user-specified config, we would always get the default mappers
before the user-specified mappers.

To solve this, let's merge the config dicts first, and then pass them to
SPConfig.
2019-09-19 20:32:14 +01:00
Richard van der Hoff
b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff
b789c7eb03 Merge branch 'develop' into rav/saml_config_cleanup 2019-09-19 15:05:31 +01:00
Jorik Schellekens
38fd1f8e3f Fix typo in account_threepid_delegates config (#6028) 2019-09-18 22:30:44 +01:00
dstipp
379d2a8c39 (#5849) Convert rst to markdown (#6040)
Converting some of the rst documentation to markdown.  Attempted to
preserve whitespace and line breaks to minimize cosmetic change.
2019-09-17 12:55:29 +01:00
Amber Brown
850dcfd2d3
Fix well-known lookups with the federation certificate whitelist (#5997) 2019-09-14 04:58:38 +10:00
Richard van der Hoff
a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Richard van der Hoff
785cbd3999 Make the sample saml config closer to our standards
It' still not great, thanks to the nested dictionaries, but it's better.
2019-09-13 12:07:03 +01:00
Amber Brown
b617864cd9
Fix for structured logging tests stomping on logs (#6023) 2019-09-13 02:29:55 +10:00
Sorunome
dd2e5b0038 add report_stats_endpoint config option (#6012)
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
2019-09-12 11:24:57 +01:00
Jorik Schellekens
a8251da10f
Blow up config if opentracing is missing (#5985)
* Blow up config if opentracing is missing
2019-09-12 10:57:37 +01:00
Erik Johnston
cbcbfe64a2
Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction
Allow use of different ratelimits for admin redactions.
2019-09-11 15:39:38 +01:00
Jorik Schellekens
6604b64fae
Check dependencies on setup in the nicer way. (#5989) 2019-09-11 14:00:37 +01:00
Erik Johnston
57dd41a45b
Fix comments
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-11 13:54:50 +01:00
Erik Johnston
54ce81c86d Allow use of different ratelimits for admin redactions.
This is useful to allow room admins to quickly deal with a large number
of abusive messages.
2019-09-11 10:46:38 +01:00
Erik Johnston
470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions
Censor redactions in DB after a month
2019-09-09 15:29:39 +01:00
Erik Johnston
8b9ade8c78 Default to censoring redactions after seven days 2019-09-09 13:55:28 +01:00
Erik Johnston
80e14a8546 Handle setting retention period to 0 2019-09-09 13:23:41 +01:00
Amber Brown
55d5b3af88
Servers-known-about statistic (#5981) 2019-09-07 01:45:51 +10:00
Andrew Morgan
0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Erik Johnston
591d82f06b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions 2019-09-05 17:27:46 +01:00
Erik Johnston
3ff0422d2d Make redaction retention period configurable 2019-09-05 17:16:45 +01:00
Erik Johnston
6e834e94fc
Fix and refactor room and user stats (#5971)
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
Jorik Schellekens
92c1550f4a Add a link to python's logging config schema (#5926) 2019-08-28 19:08:32 +01:00
Jorik Schellekens
deca277d09
Let synctl use a config directory. (#5904)
* Let synctl use a config directory.
2019-08-28 15:55:58 +01:00
Jorik Schellekens
6d97843793
Config templating (#5900)
Template config files

* Imagine a system composed entirely of x, y, z etc and the basic operations..

Wait George, why XOR? Why not just neq?

George: Eh, I didn't think of that..

Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-08-28 13:12:22 +01:00
Amber Brown
7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Erik Johnston
f5b50d0871
Merge pull request #5895 from matrix-org/erikj/notary_key
Add config option to sign remote key query responses with a separate key.
2019-08-27 11:51:37 +01:00
Aaron Raimist
e8e3e033ee
public_base_url is actually public_baseurl
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-08-26 21:01:56 -05:00
Erik Johnston
fe0ac98e66 Don't implicitly include server signing key 2019-08-23 15:36:28 +01:00
Erik Johnston
5906be8589 Add config option for keys to use to sign keys
This allows servers to separate keys that are used to sign remote keys
when acting as a notary server.
2019-08-21 10:44:58 +01:00
Andrew Morgan
b13cac896d
Fix up password reset template config names (#5863)
Fixes #5833

The emailconfig code was attempting to pull incorrect config file names. This corrects that, while also marking a difference between a config file variable that's a filepath versus a str containing HTML.
2019-08-15 16:27:11 +01:00
Amber H. Brown
18bdac8ee4 fix config being a dict, actually 2019-08-14 02:06:42 +10:00
Amber Brown
0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Brendan Abolivier
f25f638c35
Lint 2019-08-01 12:19:08 +02:00
Brendan Abolivier
a4a9ded4d0
Allow defining HTML templates to serve the user on account renewal 2019-08-01 11:59:27 +02:00
Richard van der Hoff
8c97f6414c
Remove non-functional 'expire_access_token' setting (#5782)
The `expire_access_token` didn't do what it sounded like it should do. What it
actually did was make Synapse enforce the 'time' caveat on macaroons used as
access tokens, but since our access token macaroons never contained such a
caveat, it was always a no-op.

(The code to add 'time' caveats was removed back in v0.18.5, in #1656)
2019-07-30 08:25:02 +01:00
Amber Brown
865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Jorik Schellekens
3641784e8c
Make Jaeger fully configurable (#5694)
* Allow Jaeger to be configured

* Update sample config
2019-07-23 15:46:04 +01:00
Amber Brown
0d0f6d12bc Fix logging in workers (#5729)
This also adds a worker blacklist.
2019-07-22 16:05:00 +01:00
Jorik Schellekens
0fd171770a Merge branch 'release-v1.2.0' into develop 2019-07-22 11:18:50 +01:00
Jorik Schellekens
826e6ec3bd
Opentracing Documentation (#5703)
* Opentracing survival guide

* Update decorator names in doc

* Doc cleanup

These are all alterations as a result of comments in #5703, it
includes mostly typos and clarifications. The most interesting
changes are:

- Split developer and user docs into two sections
- Add a high level description of OpenTracing

* newsfile

* Move contributer specific info to docstring.

* Sample config.

* Trailing whitespace.

* Update 5703.misc

* Apply suggestions from code review

Mostly just rewording parts of the docs for clarity.

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-07-22 11:15:21 +01:00
Amber Brown
6a85cb5ef7
Remove non-dedicated logging options and command line arguments (#5678) 2019-07-19 01:40:08 +10:00
Neil Johnson
cfc00068bd enable aggregations support by default 2019-07-18 15:56:59 +01:00
Richard van der Hoff
82345bc09a
Clean up opentracing configuration options (#5712)
Clean up config settings and dead code.

This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular:
 * There should be a blank line after `## Section ##' headings
 * There should be a blank line between each config setting
 * There should be a `#`-only line between a comment and the setting it describes
 * We don't really do the `#  #` style commenting-out of whole sections if we can help it
 * rename `tracer_enabled` to `enabled`

While we're here, do more config parsing upfront, which makes it easier to use
later on.

Also removes redundant code from LogContextScopeManager.

Also changes the changelog fragment to a `feature` - it's exciting!
2019-07-18 15:06:54 +01:00
Erik Johnston
f44354e17f Clean up arg name and remove lying comment 2019-07-16 11:39:40 +01:00
Erik Johnston
03cc8c4b5d Fix invoking add_argument from homeserver.py 2019-07-15 14:25:05 +01:00
Erik Johnston
e8c53b07f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_api_cmd 2019-07-15 14:13:22 +01:00
Erik Johnston
fdefb9e29a Move creation of ArgumentParser to caller 2019-07-15 14:09:35 +01:00
Erik Johnston
37b524f971 Fix up comments 2019-07-15 14:09:35 +01:00
Erik Johnston
823e13ddf4 Change add_arguments to be a static method 2019-07-15 14:09:33 +01:00
Richard van der Hoff
5f158ec039
Implement access token expiry (#5660)
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Jorik Schellekens
38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Richard van der Hoff
80cc82a445
Remove support for invite_3pid_guest. (#5625)
This has never been documented, and I'm not sure it's ever been used outside
sytest.

It's quite a lot of poorly-maintained code, so I'd like to get rid of it.

For now I haven't removed the database table; I suggest we leave that for a
future clearout.
2019-07-05 16:47:58 +01:00
Richard van der Hoff
9481707a52
Fixes to the federation rate limiter (#5621)
- Put the default window_size back to 1000ms (broken by #5181)
- Make the `rc_federation` config actually do something
- fix an off-by-one error in the 'concurrent' limit
- Avoid creating an unused `_PerHostRatelimiter` object for every single
  incoming request
2019-07-05 11:10:19 +01:00
Andrew Morgan
0e5434264f
Make errors about email password resets much clearer (#5616)
The runtime errors that dealt with local email password resets talked about config options that users may not even have in their config file yet (if upgrading). Instead, the cryptic errors are now replaced with hopefully much more helpful ones.
2019-07-05 10:44:12 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Erik Johnston
9f3c0a8556 Add basic admin cmd app 2019-07-02 17:12:48 +01:00
Richard van der Hoff
b4fd86a9b4 Merge branch 'develop' into rav/saml2_client 2019-07-01 14:21:03 +01:00
Amber Brown
be3b901ccd
Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550) 2019-06-28 18:19:09 +10:00
Daniel Hoffend
9646a593ac Added possibilty to disable local password authentication (#5092)
Signed-off-by: Daniel Hoffend <dh@dotlan.net>
2019-06-27 18:37:29 +01:00
Andrew Morgan
c548dbc4b1
Make it clearer that the template dir is relative to synapse's root dir (#5543)
Helps address #5444
2019-06-27 18:20:17 +01:00
Richard van der Hoff
28db0ae537 cleanups 2019-06-27 00:37:41 +01:00
Richard van der Hoff
36f4953dec Add support for tracking SAML2 sessions.
This allows us to correctly handle `allow_unsolicited: False`.
2019-06-26 23:50:55 +01:00
Richard van der Hoff
a4daa899ec Merge branch 'develop' into rav/saml2_client 2019-06-26 22:34:41 +01:00
Richard van der Hoff
4ac7ef4b67
Merge pull request #5524 from matrix-org/rav/new_cmdline_options
Add --data-dir and --open-private-ports options.
2019-06-24 17:25:57 +01:00
Richard van der Hoff
af8a962905
Merge pull request #5523 from matrix-org/rav/arg_defaults
Stop conflating generated config and default config
2019-06-24 17:24:35 +01:00
Brendan Abolivier
deb4fe6ef3
Merge pull request #5534 from matrix-org/babolivier/federation-publicrooms
Split public rooms directory auth config in two
2019-06-24 16:08:02 +01:00
Brendan Abolivier
bfe84e051e Split public rooms directory auth config in two 2019-06-24 15:42:31 +01:00
Richard van der Hoff
3f8a252dd8 Add "--open-private-ports" cmdline option
This is helpful when generating a config file for running synapse under docker.
2019-06-24 14:15:34 +01:00
Richard van der Hoff
6a92b06cbb Add --data-directory commandline argument
We don't necessarily want to put the data in the cwd.
2019-06-24 14:15:34 +01:00
Richard van der Hoff
16b52642e2 Don't load the generated config as the default.
It's too confusing.
2019-06-24 14:14:52 +01:00
Richard van der Hoff
7c2f8881a9 Ensure that all config options have sensible defaults
This will enable us to skip the unintuitive behaviour where the generated
config and default config are the same thing.
2019-06-24 14:14:52 +01:00
Richard van der Hoff
367a8263b3 Remove unused Config.config_dir_path attribute
This is no longer used and only serves to confuse.
2019-06-24 13:51:22 +01:00
Richard van der Hoff
edea4bb5be Allow configuration of the path used for ACME account keys.
Because sticking it in the same place as the config isn't necessarily the right
thing to do.
2019-06-24 13:51:22 +01:00
Richard van der Hoff
c3c6b00d95
Pass config_dir_path and data_dir_path into Config.read_config. (#5522)
* Pull config_dir_path and data_dir_path calculation out of read_config_files

* Pass config_dir_path and data_dir_path into read_config
2019-06-24 11:34:45 +01:00
Richard van der Hoff
6cda36777b Drop support for cpu_affinity (#5525)
This has no useful purpose on python3, and is generally a source of confusion.
2019-06-22 11:01:55 +10:00
Richard van der Hoff
e1a795758c
Improve help and cmdline option names for --generate-config options (#5512)
* group the arguments together into a group
* add new names "--generate-missing-config" and "--config-directory" for
  existing cmdline options "--generate-keys" and "--keys-dir", which better
  reflect their purposes.
2019-06-21 18:50:43 +01:00
Richard van der Hoff
03cea2b0fe
Refactor Config parser and add some comments. (#5511)
Add some comments, and simplify `read_config_files`.
2019-06-21 17:43:38 +01:00
Richard van der Hoff
37933a3bf8
Improve logging when generating config files (#5510)
Make it a bit clearer what's going on.
2019-06-21 17:14:56 +01:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Brendan Abolivier
f12e1f029c
Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules
Allow server admins to define implementations of extra rules for allowing or denying incoming events
2019-06-14 19:37:59 +01:00
Brendan Abolivier
f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Andrew Morgan
2ddc13577c
Don't warn user about password reset disabling through config code (#5387)
Moves the warning about password resets being disabled to the point where a user actually tries to reset their password. Is this an appropriate place for it to happen?

Also removed the disabling of msisdn password resets when you don't have an email config, as that just doesn't make sense.

Also change the error a user receives upon disabled passwords to specify that only email-based password reset is disabled.
2019-06-11 00:25:07 +01:00
Richard van der Hoff
426049247b Code cleanups and simplifications.
Also: share the saml client between redirect and response handlers.
2019-06-11 00:03:57 +01:00
Richard van der Hoff
69a43d9974 Merge remote-tracking branch 'origin/develop' into rav/saml2_client 2019-06-10 20:28:08 +01:00
Richard van der Hoff
88d7182ada
Improve startup checks for insecure notary configs (#5392)
It's not really a problem to trust notary responses signed by the old key so
long as we are also doing TLS validation.

This commit adds a check to the config parsing code at startup to check that
we do not have the insecure matrix.org key without tls validation, and refuses
to start without it.

This allows us to remove the rather alarming-looking warning which happens at
runtime.
2019-06-10 10:33:00 +01:00
Neil Johnson
a11865016e
Set default room version to v4. (#5379)
Set default room version to v4.
2019-06-06 20:13:47 +01:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Richard van der Hoff
9fbb20a531
Stop hardcoding trust of old matrix.org key (#5374)
There are a few changes going on here:

* We make checking the signature on a key server response optional: if no
  verify_keys are specified, we trust to TLS to validate the connection.

* We change the default config so that it does not require responses to be
  signed by the old key.

* We replace the old 'perspectives' config with 'trusted_key_servers', which
  is also formatted slightly differently.

* We emit a warning to the logs every time we trust a key server response
  signed by the old key.
2019-06-06 17:33:11 +01:00
Richard van der Hoff
7603a706eb Merge branch 'rav/fix_custom_ca' into rav/enable_tls_verification 2019-06-05 16:32:35 +01:00
Richard van der Hoff
f8a45302c9 Fix federation_custom_ca_list configuration option.
Previously, setting this option would cause an exception at startup.
2019-06-05 16:19:07 +01:00
Richard van der Hoff
e2dfb922e1 Validate federation server TLS certificates by default. 2019-06-05 14:17:50 +01:00
Neil Johnson
26713515de
Neilj/mau tracking config explainer (#5284)
Improve documentation of monthly active user blocking and mau_trial_days
2019-06-05 13:16:23 +01:00
Brendan Abolivier
f6dd12d1e2
Merge pull request #5341 from matrix-org/babolivier/email_config
Make account validity renewal emails work when email notifs are disabled
2019-06-04 14:49:06 +01:00
Brendan Abolivier
2f62e1f6ff
Only parse from email if provided 2019-06-04 14:24:36 +01:00
Brendan Abolivier
1cc5fc1f6c
Lint 2019-06-04 13:51:23 +01:00
Brendan Abolivier
ac3cc32367
Make account validity renewal emails work when email notifs are disabled 2019-06-04 13:47:44 +01:00
Alexander Trost
dc3e586938
SAML2 Improvements and redirect stuff
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2019-06-02 18:14:40 +02:00
Erik Johnston
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
Brendan Abolivier
4d794dae21 Move delta from +10% to -10% 2019-05-31 11:09:39 +01:00
Brendan Abolivier
0c2362861e Gah python 2019-05-31 09:56:52 +01:00
Brendan Abolivier
847b9dcd1c Make max_delta equal to period * 10% 2019-05-31 09:54:46 +01:00
Aaron Raimist
9b6f72663e
Fix docs on resetting the user directory (#5036)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-05-28 20:54:01 -05:00
Brendan Abolivier
52839886d6
Allow configuring a range for the account validity startup job
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
2019-05-28 16:52:45 +01:00
Richard van der Hoff
dba9152d15
Add missing blank line in config (#5249) 2019-05-24 14:12:38 +01:00
Andrew Morgan
6368150a74
Add config option for setting homeserver's default room version (#5223)
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
2019-05-23 15:00:20 +01:00
Amber Brown
4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Brendan Abolivier
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
Brendan Abolivier
384122efa8
Doc 2019-05-21 14:39:36 +01:00
Erik Johnston
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
Brendan Abolivier
cd32375846 Add option to disable per-room profiles 2019-05-16 14:34:28 +01:00
Erik Johnston
a0603523d2 Add aggregations API 2019-05-16 09:37:20 +01:00
Amber Brown
f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Andrew Morgan
5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
Matthew Hodgson
c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083)
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
2019-05-08 18:26:56 +01:00
Richard van der Hoff
836d3adcce Merge branch 'master' into develop 2019-05-03 19:25:01 +01:00
Richard van der Hoff
1565ebec2c more config comment updates 2019-05-03 15:50:59 +01:00
Richard van der Hoff
1a7104fde3 Blacklist 0.0.0.0 and :: by default for URL previews 2019-05-03 15:35:49 +01:00
Brendan Abolivier
c193b39134
Merge pull request #5124 from matrix-org/babolivier/aliases
Add some limitations to alias creation
2019-05-02 11:22:40 +01:00
Brendan Abolivier
84196cb231 Add some limitations to alias creation 2019-05-02 11:05:11 +01:00
Richard van der Hoff
8e9ca83537 Move admin API to a new prefix 2019-05-01 15:44:30 +01:00
Andrew Morgan
6824ddd93d Config option for verifying federation certificates (MSC 1711) (#4967) 2019-04-25 14:22:49 +01:00
Brendan Abolivier
f8826d31cd Don't crash on lack of expiry templates 2019-04-18 14:50:05 +01:00
Brendan Abolivier
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier
20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Andrew Morgan
caa76e6021
Remove periods from copyright headers (#5046) 2019-04-11 17:08:13 +01:00
Brendan Abolivier
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
Add time-based account expiration
2019-04-09 17:02:41 +01:00
Brendan Abolivier
747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Neil Johnson
b25e387c0d
add context to phonehome stats (#5020)
add context to phonehome stats
2019-04-08 15:47:39 +01:00
Brendan Abolivier
8e85493b0c
Add config option to block users from looking up 3PIDs (#5010) 2019-04-04 17:25:47 +01:00
Richard van der Hoff
8530090b16
Add config.signing_key_path. (#4974)
As requested by @andrewshadura
2019-04-02 16:59:27 +01:00
Erik Johnston
3677548a82 Use yaml safe_load 2019-03-22 10:20:17 +00:00
Erik Johnston
09f991a63d
Merge pull request #4896 from matrix-org/erikj/disable_room_directory
Add option to disable search room lists
2019-03-21 10:16:54 +00:00
Erik Johnston
263f2c9ce1
Merge pull request #4895 from matrix-org/erikj/disable_user_search
Add option to disable searching in the user dir
2019-03-20 16:47:15 +00:00
Richard van der Hoff
cdb8036161
Add a config option for torture-testing worker replication. (#4902)
Setting this to 50 or so makes a bunch of sytests fail in worker mode.
2019-03-20 16:04:35 +00:00
Richard van der Hoff
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890)
Rate-limit outgoing read-receipts as per #4730.
2019-03-20 16:02:25 +00:00
Erik Johnston
cd8c5b91ad Fix up sample config 2019-03-20 14:35:41 +00:00
Richard van der Hoff
ab20f85c59
Update synapse/config/user_directory.py
Co-Authored-By: erikjohnston <erikj@jki.re>
2019-03-20 14:33:11 +00:00
Erik Johnston
926f29ea6d Fix up config comments 2019-03-20 14:24:53 +00:00
Erik Johnston
213c98c00a Add option to disable search room lists
This disables both local and remote room list searching.
2019-03-19 17:10:52 +00:00
Erik Johnston
320667a479 Add option to disable searching in the user dir
We still populate it, as it can still be accessed via the admin API.
2019-03-19 16:40:19 +00:00
Richard van der Hoff
13bc1e0746 Use a regular HomeServerConfig object for unit tests
Rather than using a Mock for the homeserver config, use a genuine
HomeServerConfig object. This makes for a more realistic test, and means that
we don't have to keep remembering to add things to the mock config every time
we add a new config setting.
2019-03-19 11:44:43 +00:00
Richard van der Hoff
fd463b4f5d
Comment out most options in the generated config. (#4863)
Make it so that most options in the config are optional, and commented out in
the generated config.

The reasons this is a good thing are as follows:

* If we decide that we should change the default for an option, we can do so,
  and only those admins that have deliberately chosen to override that option
  will be stuck on the old setting.

* It moves us towards a point where we can get rid of the super-surprising
  feature of synapse where the default settings for the config come from the
  generated yaml.

* It makes setting up a test config for unit testing an order of magnitude
  easier (see forthcoming PR).

* It makes the generated config more consistent, and hopefully easier for users
  to understand.
2019-03-19 10:06:40 +00:00
Brendan Abolivier
651ad8bc96 Add ratelimiting on failed login attempts (#4865) 2019-03-18 12:57:20 +00:00
Brendan Abolivier
899e523d6d
Add ratelimiting on login (#4821)
Add two ratelimiters on login (per-IP address and per-userID).
2019-03-15 17:46:16 +00:00
Erik Johnston
9ad448c1e5 Correctly handle all command line options 2019-03-14 13:32:14 +00:00
Erik Johnston
72bfaf746d Allow passing --daemonize to workers 2019-03-13 17:33:54 +00:00
Andrew Morgan
7998ca3a66
Document using a certificate with a full chain (#4849) 2019-03-13 15:26:29 +00:00
Aaron Raimist
8ea1b41a0e Clarify what registration_shared_secret allows for (#2885) (#4844)
* Clarify what registration_shared_secret allows for (#2885)

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-03-11 18:21:52 +00:00
Brendan Abolivier
067ce795c0 Move settings from registration to ratelimiting in config file 2019-03-05 18:03:14 +00:00
Brendan Abolivier
a4c3a361b7
Add rate-limiting on registration (#4735)
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
2019-03-05 14:25:33 +00:00
Richard van der Hoff
8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Richard van der Hoff
641c409e4e Fix ACME config for python 2. (#4717)
Fixes #4675.
2019-02-25 11:16:33 -08:00
Matthew Hodgson
70ea2f4e1d switch from google.com to recaptcha.net for reCAPTCHA (#4731)
* add trivial clarification about jemalloc

* switch from google.com to recaptcha.net

because https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally
2019-02-25 11:15:36 -08:00
Erik Johnston
82fca11fc1
Merge pull request #4694 from matrix-org/erikj/fix_sentry_config_format
Fixup generated metrics config
2019-02-20 14:13:38 +00:00
Erik Johnston
3d672fec51 Fixup generated metrics config 2019-02-20 13:39:37 +00:00
Richard van der Hoff
5f9bdf90fe Attempt to make default config more consistent
The general idea here is that config examples should just have a hash and no
extraneous whitespace, both to make it easier for people who don't understand
yaml, and to make the examples stand out from the comments.
2019-02-19 13:54:29 +00:00
Brendan Abolivier
a288bdf0b1
Merge pull request #4652 from matrix-org/babolivier/acme-delegated
Support .well-known delegation when issuing certificates through ACME
2019-02-19 11:15:38 +00:00
Brendan Abolivier
5a707a2f9a Improve config documentation 2019-02-19 10:59:26 +00:00
Erik Johnston
d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry
Add basic optional sentry.io integration
2019-02-18 17:22:45 +00:00
Erik Johnston
d328a93b51 Fixup error handling and message 2019-02-18 16:53:56 +00:00
Brendan Abolivier
45bb55c6de Use a configuration parameter to give the domain to generate a certificate for 2019-02-18 15:46:23 +00:00
Erik Johnston
dc5efc92a8 Fixup 2019-02-18 13:52:49 +00:00
Juuso "Linda" Lapinlampi
68d2869c8d config: Remove a repeated word from a logger warning
The warning for missing macaroon_secret_key was "missing missing".
2019-02-15 22:24:53 -07:00
Erik Johnston
bd4505f765
Merge pull request #4647 from matrix-org/erikj/add_room_publishing_rules
Add configurable room list publishing rules
2019-02-15 22:11:01 +00:00
Erik Johnston
b99c532c1c Move defaults up into code 2019-02-15 10:53:39 +00:00
Erik Johnston
02c729d6b0 Hoist up checks to reduce overall work 2019-02-15 10:20:02 +00:00
Erik Johnston
02c46acc6a Fixup comments 2019-02-15 10:17:13 +00:00
Erik Johnston
8e32f26cb8 Clarify comments 2019-02-14 18:21:24 +00:00
Erik Johnston
cb12a37708 Clarify and fix behaviour when there are multiple aliases 2019-02-14 18:16:32 +00:00
Erik Johnston
f666fe36d7 Fixup comments 2019-02-14 18:07:24 +00:00
Richard van der Hoff
f311018823
Fix errors in acme provisioning (#4648)
* Better logging for errors on startup

* Fix "TypeError: '>' not supported" when starting without an existing
  certificate
* Fix a bug where an existing certificate would be reprovisoned every day
2019-02-14 17:10:36 +00:00
Erik Johnston
eaf4d11af9 Add configurable room list publishing rules
This allows specifying who and what is allowed to be published onto the
public room list
2019-02-14 16:02:23 +00:00
Erik Johnston
6cb415b63f Fixup comments and add warning 2019-02-13 16:15:11 +00:00
Richard van der Hoff
e3a0300431 Special-case the default bind_addresses for metrics listener
turns out it doesn't really support ipv6, so let's hack around that by only
listening on ipv4 by default.
2019-02-13 11:48:56 +00:00
Erik Johnston
6a8f902edb Raise an appropriate error message if sentry_sdk missing 2019-02-12 16:01:41 +00:00
Erik Johnston
ef2228c890 Basic sentry integration 2019-02-12 13:55:58 +00:00
Erik Johnston
3c03c37883
Merge pull request #4625 from matrix-org/rav/fix_generate_config_warnings
fix self-signed cert notice from generate-config
2019-02-12 11:24:45 +00:00