Commit Graph

1244 Commits

Author SHA1 Message Date
David Robertson
313581e4e9
Use importlib.metadata to read requirements (#12088)
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`

I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.

I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.

I hope you agree that this is clearer.

* Test cases
2022-03-01 17:44:41 +00:00
reivilibre
2cc5ea933d
Add support for MSC3202: sending one-time key counts and fallback key usage states to Application Services. (#11617)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-02-24 17:55:45 +00:00
Brendan Abolivier
250104d357
Implement account status endpoints (MSC3720) (#12001)
See matrix-org/matrix-doc#3720

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
Patrick Cloke
7bcc28f82f
Use room version 9 as the default room version (per MSC3589). (#12058) 2022-02-22 14:09:40 +00:00
Erik Johnston
3f4d25a48b
Remove unstable MSC3283 flags (#12018)
Fixes #11962
2022-02-17 17:22:55 +00:00
Richard van der Hoff
da0e9f8efd
Faster joins: parse msc3706 fields in send_join response (#12011)
Part of my work on #11249: add code to handle the new fields added in MSC3706.
2022-02-17 16:11:59 +00:00
Patrick Cloke
e69f8f0a8e
Remove support for the legacy structured logging configuration. (#12008) 2022-02-17 13:32:18 +00:00
H. Shay
9c4563c5cd remove empty file, reword/rename newsfragment, and add line to upgrade doc 2022-02-14 10:21:00 -08:00
David Robertson
4ae956c8bb
Use version string helper from matrix-common (#11979)
* Require latest matrix-common
* Use the common function
2022-02-14 13:12:22 +00:00
Richard van der Hoff
63c46349c4
Implement MSC3706: partial state in /send_join response (#11967)
* Make `get_auth_chain_ids` return a Set

It has a set internally, and a set is often useful where it gets used, so let's
avoid converting to an intermediate list.

* Minor refactors in `on_send_join_request`

A little bit of non-functional groundwork

* Implement MSC3706: partial state in /send_join response
2022-02-12 10:44:16 +00:00
Shay
b2b971f28a
Enable cache time-based expiry by default (#11849) 2022-02-11 11:05:20 -08:00
Patrick Cloke
8c94b3abe9
Experimental support to include bundled aggregations in search results (MSC3666) (#11837) 2022-02-08 09:21:20 -05:00
Patrick Cloke
119edf51eb
Remove support for the webclient listener. (#11895)
Also remove support for non-HTTP(S) web_client_location.
2022-02-03 18:36:49 +00:00
Brendan Abolivier
d80d39b035
Add a ratelimiter for 3pid invite (#11892) 2022-02-03 14:28:15 +01:00
Patrick Cloke
acda9f07c8
Revert experimental push rules from #7997. (#11884)
Manually reverts the merge from cdbb8e6d6e.
2022-02-02 09:49:31 -05:00
Andrew Morgan
64ec45fc1b
Send to-device messages to application services (#11215)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
Patrick Cloke
02755c3188
Remove the obsolete MSC1849 configuration flag. (#11843)
MSC1849 was replaced by MSC2675, which was merged.
The configuration flag, which defaulted to true, is no
longer useful.
2022-01-31 10:13:32 -05:00
Brendan Abolivier
bf60da1a60
Configurable limits on avatars (#11846)
Only allow files which file size and content types match configured
limits to be set as avatar.

Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
2022-01-28 15:41:33 +01:00
Dirk Klimpel
fd65139714
Fix some indentation inconsistencies in the sample config (modules) (#11838) 2022-01-27 10:06:29 +00:00
Brendan Abolivier
95b3f952fa
Add a config flag to inhibit M_USER_IN_USE during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
Patrick Cloke
91221b6961
Add deprecation warnings for webclient listener and non-HTTP(S) web_client_location. (#11774)
This changes the behaviour of the root endpoint to redirect
directly to the configuration of `web_client_location` if it is
given an HTTP(S) URL.
2022-01-20 14:21:06 +00:00
Richard van der Hoff
86615aa965
Fix up docs for track_puppeted_user_ips (again) (#11757)
Fixes #11741
2022-01-17 14:55:30 +00:00
Richard van der Hoff
b0352f9c08
Update documentation for configuring facebook login (#11755)
... and a minor thinko fix in the sample config.
2022-01-17 12:35:00 +00:00
Jason Robinson
904bb04409
Fix sample_config.yaml in regards track_puppeted_user_ips (#11749)
* Fix sample_config.yaml in regards track_puppeted_user_ips

Closes #11741

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2022-01-14 14:11:55 +00:00
qwertyforce
b9632046fb
update room spec url in config files (#11739)
* change spec url in config files

* Create 11739.txt

* .txt -> .doc
2022-01-13 17:09:15 +00:00
Patrick Cloke
10a88ba91c
Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
Jason Robinson
2560b1b6b2
Allow tracking puppeted users for MAU (#11561)
Currently when puppeting another user, the user doing the puppeting is
tracked for client IPs and MAU (if configured).

When tracking MAU is important, it becomes necessary to be possible to
also track the client IPs and MAU of puppeted users. As an example a
client that manages user creation and creation of tokens via the Synapse
admin API, passing those tokens for the client to use.

This PR adds optional configuration to enable tracking of puppeted users
into monthly active users. The default behaviour stays the same.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2022-01-12 16:09:36 +00:00
Philipp Matthias Schäfer
eedb4527f1
Fix link from generated configuration file to documentation (#11678)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: reivilibre <oliverw@matrix.org>
2022-01-05 12:16:52 +00:00
reivilibre
84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common #11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
Patrick Cloke
8422a7f7f6
Include the topic event in the prejoin state, per MSC3173. (#11666)
Invites and knocks will now include the topic in the stripped state
send to clients before joining the room.
2022-01-04 11:08:08 -05:00
Patrick Cloke
cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
Eric Eastwood
43f5cc7adc
Add MSC2716 and MSC3030 to /versions -> unstable_features (#11582)
As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
2021-12-16 11:25:37 -06:00
reivilibre
17886d2603
Add experimental support for MSC3202: allowing application services to masquerade as specific devices. (#11538) 2021-12-15 10:40:52 +00:00
Patrick Cloke
2519beaad2
Add missing type hints to synapse.appservice (#11360) 2021-12-14 17:02:46 +00:00
Patrick Cloke
0cc3bf97b4
Additional type hints for the config module, part 2. (#11480) 2021-12-09 16:15:46 +00:00
Sean Quah
088d748f2c
Revert "Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) (#11527)
This reverts commit a77c369897.
2021-12-07 13:51:11 +00:00
reivilibre
2f053f3f82
Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435) 2021-12-06 19:11:43 +00:00
Sean Quah
a77c369897
Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) 2021-12-06 11:36:08 +00:00
reivilibre
637df95de6
Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445) 2021-12-03 16:42:44 +00:00
Eric Eastwood
a6f1a3abec
Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)
MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030

Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about.
```
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Federation API endpoint:
```
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-12-02 01:02:20 -06:00
Patrick Cloke
f44d729d4c
Additional type hints for config module. (#11465)
This adds some misc. type hints to helper methods used
in the `synapse.config` module.
2021-12-01 07:28:23 -05:00
reivilibre
a82b90ab32
Add type annotations to some of the configuration surrounding refresh tokens. (#11428) 2021-11-29 13:34:14 +00:00
reivilibre
1d8b80b334
Support expiry of refresh tokens and expiry of the overall session when refresh tokens are in use. (#11425) 2021-11-26 14:27:14 +00:00
reivilibre
f25c75d376
Rename unstable access_token_lifetime configuration option to refreshable_access_token_lifetime to make it clear it only concerns refreshable access tokens. (#11388) 2021-11-23 17:01:34 +00:00
Patrick Cloke
55669bd3de
Add missing type hints to config base classes (#11377) 2021-11-23 15:21:19 +00:00
Kostas
1035663833
Add config for customizing the claim used for JWT logins. (#11361)
Allows specifying a different claim (from the default "sub") to use
when calculating the localpart of the Matrix ID used during the
JWT login.
2021-11-22 13:01:03 -05:00
Dirk Klimpel
ea20937084
Add an admin API to run background jobs. (#11352)
Instead of having admins poke into the database directly.

Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
Patrick Cloke
7ae559944a
Fix checking whether a room can be published on creation. (#11392)
If `room_list_publication_rules` was configured with a rule with a
non-wildcard alias and a room was created with an alias then an
internal server error would have been thrown.

This fixes the error and properly applies the publication rules
during room creation.
2021-11-19 15:19:32 +00:00
Patrick Cloke
4d6d38ac2f
Remove dead code from acme support. (#11393) 2021-11-19 07:07:22 -05:00
Shay
92b75388f5
Remove legacy code related to deprecated trust_identity_server_for_password_resets config flag (#11333)
* remove code legacy code related to deprecated config flag "trust_identity_server_for_password_resets" from synapse/config/emailconfig.py

* remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/config/registration.py

* remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/handlers/identity.py

* add tests to ensure config error is thrown and synapse refuses to start when depreciated config flag is found

* add changelog

* slightly change behavior to only check for deprecated flag if set to 'true'

* Update changelog.d/11333.misc

Co-authored-by: reivilibre <oliverw@matrix.org>

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-18 10:56:32 -08:00
Nicolai Søborg
e2dabec996
Docs: Quote wildcard federation_certificate_verification_whitelist (#11381)
Otherwise I get this beautiful stacktrace:

```
python3 -m synapse.app.homeserver --config-path /etc/matrix/homeserver.yaml
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/synapse/synapse/app/homeserver.py", line 455, in <module>
    main()
  File "/root/synapse/synapse/app/homeserver.py", line 445, in main
    hs = setup(sys.argv[1:])
  File "/root/synapse/synapse/app/homeserver.py", line 345, in setup
    config = HomeServerConfig.load_or_generate_config(
  File "/root/synapse/synapse/config/_base.py", line 671, in load_or_generate_config
    config_dict = read_config_files(config_files)
  File "/root/synapse/synapse/config/_base.py", line 717, in read_config_files
    yaml_config = yaml.safe_load(file_stream)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 379, in parse_block_sequence_first_entry
    return self.parse_block_sequence_entry()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/parser.py", line 384, in parse_block_sequence_entry
    if not self.check_token(BlockEntryToken, BlockEndToken):
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 227, in fetch_more_tokens
    return self.fetch_alias()
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 610, in fetch_alias
    self.tokens.append(self.scan_anchor(AliasToken))
  File "/root/synapse/env/lib/python3.8/site-packages/yaml/scanner.py", line 922, in scan_anchor
    raise ScannerError("while scanning an %s" % name, start_mark,
yaml.scanner.ScannerError: while scanning an alias
  in "/etc/matrix/homeserver.yaml", line 614, column 5
expected alphabetic or numeric character, but found '.'
  in "/etc/matrix/homeserver.yaml", line 614, column 6
```

Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
2021-11-18 12:24:40 +00:00
Richard van der Hoff
86a497efaa
Default value for public_baseurl (#11210)
We might as well use a default value for `public_baseurl` based on
`server_name` - in many cases, it will be correct.
2021-11-08 14:13:10 +00:00
Nick Barrett
af54167516
Enable passing typing stream writers as a list. (#11237)
This makes the typing stream writer config match the other stream writers
that only currently support a single worker.
2021-11-03 14:25:47 +00:00
Richard van der Hoff
71f9966f27
Support for serving server well-known files (#11211)
Fixes https://github.com/matrix-org/synapse/issues/8308
2021-11-01 15:10:16 +00:00
Sean Quah
2b82ec425f
Add type hints for most HomeServer parameters (#11095) 2021-10-22 18:15:41 +01:00
Jason Robinson
b9ce53e878
Fix synapse.config module "read" command (#11145)
`synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse.

 The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example.

 Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-22 12:00:52 +02:00
Patrick Cloke
ba00e20234
Add a thread relation type per MSC3440. (#11088)
Adds experimental support for MSC3440's `io.element.thread` relation
type (and the aggregation for it).
2021-10-21 14:39:16 -04:00
Patrick Cloke
5573133348
Move experimental & retention config out of the server module. (#11070) 2021-10-15 14:30:48 +00:00
Azrenbeth
cdd308845b
Port the Password Auth Providers module interface to the new generic interface (#10548)
Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-13 11:21:52 +00:00
David Robertson
797ee7812d
Relax ignore-missing-imports for modules that have stubs now and update mypy (#11006)
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
Patrick Cloke
f4b1a9a527
Require direct references to configuration variables. (#10985)
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
2021-10-06 10:47:41 -04:00
Patrick Cloke
94b620a5ed
Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
Patrick Cloke
eb2c7e51c4
Clean-up type hints in server config (#10915)
By using attrs instead of dicts to store configuration.

Also updates some of the attrs classes to use proper type
hints and auto_attribs.
2021-09-28 09:24:40 -04:00
Sean Quah
6c83c27107
Fix race conditions when creating media store and config directories (#10913) 2021-09-27 11:29:23 +01:00
Patrick Cloke
bb7fdd821b
Use direct references for configuration variables (part 5). (#10897) 2021-09-24 07:25:21 -04:00
Patrick Cloke
47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Patrick Cloke
e584534403
Use direct references for some configuration variables (part 3) (#10885)
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.

It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
Andrew Morgan
aa2c027792
Remove unnecessary parentheses around tuples returned from methods (#10889) 2021-09-23 11:59:07 +01:00
David Robertson
60453315bd
Always add local users to the user directory (#10796)
It's a simplification, but one that'll help make the user directory logic easier
to follow with the other changes upcoming. It's not strictly required for those
changes, but this will help simplify the resulting logic that listens for
`m.room.member` events and generally make the logic easier to follow.

This means the config option `search_all_users` ends up controlling the
search query only, and not the data we store. The cost of doing so is an
extra row in the `user_directory` and `user_directory_search` tables for
each local user which

- belongs to no public rooms
- belongs to no private rooms of size ≥ 2

I think the cost of this will be marginal (since they'll already have entries
 in `users` and `profiles` anyway).

As a small upside, a homeserver whose directory was built with this
change can toggle `search_all_users` without having to rebuild their
directory.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-21 12:02:34 +00:00
Patrick Cloke
b3590614da
Require type hints in the handlers module. (#10831)
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.

This also removes the unused construct_auth_difference method
from the FederationHandler.
2021-09-20 08:56:23 -04:00
Patrick Cloke
01c88a09cd
Use direct references for some configuration variables (#10798)
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
BramvdnHeuvel
9f111075e8
Fix copy-paste error in the password section of the sample-config. (#10804) 2021-09-13 08:58:34 -04:00
reivilibre
524b8ead77
Add types to synapse.util. (#10601) 2021-09-10 17:03:18 +01:00
Sean
273b6861f2
Remove unstable MSC2858 API, including experimental.msc2858_enabled config option (#10693)
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-09 17:59:59 +01:00
Patrick Cloke
580a15e039
Request JSON for oEmbed requests (and ignore XML only providers). (#10759)
This adds the format to the request arguments / URL to
ensure that JSON data is returned (which is all that
Synapse supports).

This also adds additional error checking / filtering to the
configuration file to ignore XML-only providers.
2021-09-08 07:17:52 -04:00
Brendan Abolivier
287108fb2e Merge tag 'v1.42.0rc2' into develop
Synapse 1.42.0rc2 (2021-09-06)
==============================

This version of Synapse removes deprecated room-management admin APIs, removes out-of-date
email pushers, and improves error handling for fallback templates for user-interactive
authentication. For more information on these points, server administrators are
encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).

Features
--------

- Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747))

Internal Changes
----------------

- Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))
2021-09-06 16:29:59 +01:00
Azrenbeth
6e895366ea
Add config option to use non-default manhole password and keys (#10643) 2021-09-06 16:08:03 +01:00
Brendan Abolivier
8c9e723fe0
Add a warning when using deprecated template_dir settings (#10768)
The deprecation itself happened in #10596 which shipped with Synapse v1.41.0. However, it doesn't seem fair to suddenly drop support for these settings in ~4-6w without being more vocal about said deprecation.
2021-09-06 16:23:50 +02:00
Patrick Cloke
e2481dbe93
Allow configuration of the oEmbed URLs. (#10714)
This adds configuration options (under an `oembed` section) to
configure which URLs are matched to use oEmbed for URL
previews.
2021-08-31 18:37:07 -04:00
Erik Johnston
cd22fb568a Merge branch 'master' into develop 2021-08-24 17:13:29 +01:00
Callum Brown
947dbbdfd1
Implement MSC3231: Token authenticated registration (#10142)
Signed-off-by: Callum Brown <callum@calcuode.com>

This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
2021-08-21 22:14:43 +01:00
Patrick Cloke
5cda75fede
Set room version 8 as preferred for restricted rooms. (#10571) 2021-08-20 07:17:50 -04:00
Dirk Klimpel
b5fef6054a
Support MSC3283: Expose enable_set_displayname in capabilities (#10452) 2021-08-19 09:40:40 +00:00
Erik Johnston
78a70a2e0b Merge branch 'release-v1.41' into develop 2021-08-18 17:02:47 +01:00
Azrenbeth
1a9f531c79
Port the PresenceRouter module interface to the new generic interface (#10524)
Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
2021-08-17 13:22:45 +00:00
Brendan Abolivier
5639759980
Centralise the custom template directory (#10596)
Several configuration sections are using separate settings for custom template directories, which can be confusing. This PR adds a new top-level configuration for a custom template directory which is then used for every module. The only exception is the consent templates, since the consent template directory require a specific hierarchy, so it's probably better that it stays separate from everything else.
2021-08-17 14:45:24 +02:00
reivilibre
5f7b1e1f27
Make PeriodicallyFlushingMemoryHandler the default logging handler. (#10518) 2021-08-17 13:13:11 +01:00
Brendan Abolivier
ae2714c1f3
Allow using several custom template directories (#10587)
Allow using several directories in read_templates.
2021-08-17 10:23:14 +00:00
Michael Telatynski
0ace38b7b3
Experimental support for MSC3266 Room Summary API. (#10394) 2021-08-16 14:49:12 +00:00
Richard van der Hoff
74d7336686
Add a setting to disable TLS for sending email (#10546)
This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
2021-08-06 10:13:34 +00:00
Kento Okamoto
72935b7c50
Add warnings to ip_range_blacklist usage with proxies (#10129)
Per issue #9812 using `url_preview_ip_range_blacklist` with a proxy via `HTTPS_PROXY` or `HTTP_PROXY` environment variables has some inconsistent bahavior than mentioned. This PR changes the following:

- Changes the Sample Config file to include a note mentioning that `url_preview_ip_range_blacklist` and `ip_range_blacklist` is ignored when using a proxy
- Changes some logic in synapse/config/repository.py to send a warning when both `*ip_range_blacklist` configs and a proxy environment variable are set and but no longer throws an error.

Signed-off-by: Kento Okamoto <kentokamoto@protonmail.com>
2021-08-03 18:13:34 +00:00
Richard van der Hoff
4b10880da3
Make sync response cache time configurable. (#10513) 2021-08-03 14:45:04 +01:00
reivilibre
f4ac934afe
Revert use of PeriodicallyFlushingMemoryHandler by default (#10515) 2021-08-03 11:30:39 +01:00
Toni Spets
ba5287f5e8
Allow setting transaction limit for db connections (#10440)
Setting the value will help PostgreSQL free up memory by recycling
the connections in the connection pool.

Signed-off-by: Toni Spets <toni.spets@iki.fi>
2021-08-02 13:24:43 +00:00
Šimon Brandner
c3b037795a
Support for MSC2285 (hidden read receipts) (#10413)
Implementation of matrix-org/matrix-doc#2285
2021-07-28 10:05:11 +02:00
reivilibre
e16eab29d6
Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-27 14:32:05 +01:00
Patrick Cloke
d15e72e511
Update the notification email subject when invited to a space. (#10426) 2021-07-21 17:29:54 +00:00
Michael Telatynski
69226c1ab4
MSC3244 room capabilities implementation (#10283) 2021-07-20 12:59:23 +01:00
Brendan Abolivier
a743bf4694
Port the ThirdPartyEventRules module interface to the new generic interface (#10386)
Port the third-party event rules interface to the generic module interface introduced in v1.37.0
2021-07-20 12:39:46 +02:00
Jonathan de Jong
95e47b2e78
[pyupgrade] synapse/ (#10348)
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of #9744
2021-07-19 15:28:05 +01:00
Brendan Abolivier
36dc15412d
Add a module type for account validity (#9884)
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.

To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16 18:11:53 +02:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) (#10380) 2021-07-15 11:02:43 +01:00
Cristina
f6767abc05
Remove functionality associated with unused historical stats tables (#9721)
Fixes #9602
2021-07-08 16:57:13 +01:00
Dirk Klimpel
56fd5fa8e1
Update links to documentation in sample config (#10287)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-07 12:35:45 +01:00
Erik Johnston
7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Quentin Gliech
bd4919fb72
MSC2918 Refresh tokens implementation (#9450)
This implements refresh tokens, as defined by MSC2918

This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235

The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-06-24 14:33:20 +01:00
Brendan Abolivier
34db6bb9f5
Warn users trying to use the deprecated spam checker interface (#10210)
So admins aren't surprised if things break when we remove this code in a couple of months.
2021-06-22 12:24:10 +02:00
Eric Eastwood
96f6293de5
Add endpoints for backfilling history (MSC2716) (#9247)
Work on https://github.com/matrix-org/matrix-doc/pull/2716
2021-06-22 10:02:53 +01:00
jkanefendt
756fd513df
Implement config option sso.update_profile_information (#10108)
Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname.

Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>
2021-06-21 23:48:57 +01:00
Erik Johnston
a5cd05beee
Fix performance of responding to user key requests over federation (#10221)
We were repeatedly looking up a config option in a loop (using the
unclassed config style), which is expensive enough that it can cause
large CPU usage.
2021-06-21 14:38:59 +01:00
Richard van der Hoff
107c06081f
Ensure that errors during startup are written to the logs and the console. (#10191)
* Defer stdio redirection until we are about to start the reactor

* Catch and handle exceptions during startup
2021-06-21 11:41:25 +01:00
Brendan Abolivier
1b3e398bea
Standardise the module interface (#10062)
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
2021-06-18 12:15:52 +01:00
Brendan Abolivier
08c8469322
Remove support for ACME v1 (#10194)
Fixes #9778

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
Patrick Cloke
76f9c701c3
Always require users to re-authenticate for dangerous operations. (#10184)
Dangerous actions means deactivating an account, modifying an account
password, or adding a 3PID.

Other actions (deleting devices, uploading keys) can re-use the same UI
auth session if ui_auth.session_timeout is configured.
2021-06-16 11:07:28 -04:00
Patrick Cloke
9e5ab6dd58
Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167)
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
2021-06-15 07:45:14 -04:00
Michael Kutzner
aac2c49b9b
Fix 'ip_range_whitelist' not working for federation servers (#10115)
Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers.

Signed-off-by: Michael Kutzner 1mikure@gmail.com
2021-06-15 08:53:55 +01:00
Aaron Raimist
e6245e6d48
Mention that you need to configure max upload size in reverse proxy as well (#10122)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-06-10 11:40:24 +01:00
Sorunome
d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Richard van der Hoff
fe5dad46b0
Remove redundant code to reload tls cert (#10054)
we don't need to reload the tls cert if we don't have any tls listeners.

Follow-up to #9280.
2021-05-27 10:34:24 +01:00
Patrick Cloke
f42e4c4eb9
Remove the experimental spaces enabled flag. (#10063)
In lieu of just always enabling the unstable spaces endpoint and
unstable room version.
2021-05-26 14:35:16 -04:00
Patrick Cloke
7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Jerin J Titus
057ce7b754
Remove tls_fingerprints option (#9280)
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
2021-05-24 17:43:30 +01:00
Patrick Cloke
316f89e87f
Enable experimental spaces by default. (#10011)
The previous spaces_enabled flag now defaults to true and
is exposed in the sample config.
2021-05-24 08:57:14 -04:00
Dan Callahan
ebdef256b3
Remove superfluous call to bool() (#9986)
Our strtobool already returns a bool, so no need to re-cast here

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:57 +01:00
Richard van der Hoff
c14f99be46
Support enabling opentracing by user (#9978)
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
2021-05-14 10:51:08 +01:00
Patrick Cloke
28c6841102
Send the m.room.create stripped event with invites (support MSC1772). (#9966)
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
2021-05-11 10:58:58 -04:00
Brendan Abolivier
652a6b094d Merge branch 'master' into develop 2021-05-11 14:15:30 +01:00
Aaron Raimist
dc6366a9bd
Add config option to hide device names over federation (#9945)
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-11 14:03:23 +01:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Erik Johnston
ef889c98a6
Optionally track memory usage of each LruCache (#9881)
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
2021-05-05 16:54:36 +01:00
Erik Johnston
1fb9a2d0bf
Limit how often GC happens by time. (#9902)
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.

This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.

Adds a `gc_min_seconds_between` config option to override the defaults.

Fixes #9890.
2021-05-05 16:53:45 +01:00
Andrew Morgan
b85821aca2
Add port parameter to the sample config for psycopg2 args (#9911)
Adds the `port` option with the default value to the sample config file.
2021-05-04 13:28:59 +01:00
Richard van der Hoff
3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Andrew Morgan
695b73c861
Allow OIDC cookies to work on non-root public baseurls (#9726)
Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574.

As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
2021-04-23 18:22:47 +01:00
Erik Johnston
9d25a0ae65
Split presence out of master (#9820) 2021-04-23 12:21:55 +01:00
Richard van der Hoff
294c675033
Remove synapse.types.Collection (#9856)
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Patrick Cloke
683d6f75af
Rename handler and config modules which end in handler/config. (#9816) 2021-04-20 14:55:20 -04:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
rkfg
c9a2b5d402
More robust handling of the Content-Type header for thumbnail generation (#9788)
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
2021-04-14 16:30:59 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Dan Callahan
1d5f0e3529
Bump black configuration to target py36 (#9781)
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Andrew Morgan
04819239ba
Add a Synapse Module for configuring presence update routing (#9491)
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
Patrick Cloke
35c5ef2d24
Add an experimental room version to support restricted join rules. (#9717)
Per MSC3083.
2021-03-31 16:39:08 -04:00
Denis Kasak
5ff8eb97c6
Make sample config allowed_local_3pids regex stricter. (#9719)
The regex should be terminated so that subdomain matches of another
domain are not accepted. Just ensuring that someone doesn't shoot
themselves in the foot by copying our example.

Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-03-31 12:27:20 +00:00
Richard van der Hoff
4dabcf026e
Include m.room.create in invite_room_state for Spaces (#9710) 2021-03-30 14:03:17 +01:00
Richard van der Hoff
f02663c4dd
Replace room_invite_state_types with room_prejoin_state (#9700)
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
2021-03-30 12:12:44 +01:00
Richard van der Hoff
4bbd535450
Update the OIDC sample config (#9695)
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.

I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
2021-03-29 15:40:11 +01:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy (#9692)
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Erik Johnston
e550ab17ad
Increase default join burst ratelimiting (#9674)
It's legitimate behaviour to try and join a bunch of rooms at once.
2021-03-23 14:52:20 +00:00