Standardise the module interface (#10062)

This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
This commit is contained in:
Brendan Abolivier 2021-06-18 13:15:52 +02:00 committed by GitHub
parent 91fa9cca99
commit 1b3e398bea
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
23 changed files with 768 additions and 187 deletions

View File

@ -85,6 +85,23 @@ for example:
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
Upgrading to v1.37.0
====================
Deprecation of the current spam checker interface
-------------------------------------------------
The current spam checker interface is deprecated in favour of a new generic modules system.
Authors of spam checker modules can refer to `this documentation <https://matrix-org.github.io/synapse/develop/modules.html#porting-an-existing-module-that-uses-the-old-interface>`_
to update their modules. Synapse administrators can refer to `this documentation <https://matrix-org.github.io/synapse/develop/modules.html#using-modules>`_
to update their configuration once the modules they are using have been updated.
We plan to remove support for the current spam checker interface in August 2021.
More module interfaces will be ported over to this new generic system in future versions
of Synapse.
Upgrading to v1.34.0 Upgrading to v1.34.0
==================== ====================

View File

@ -0,0 +1 @@
Standardised the module interface.

View File

@ -0,0 +1 @@
The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/master/UPGRADE.rst#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system.

View File

@ -35,7 +35,7 @@
- [URL Previews](url_previews.md) - [URL Previews](url_previews.md)
- [User Directory](user_directory.md) - [User Directory](user_directory.md)
- [Message Retention Policies](message_retention_policies.md) - [Message Retention Policies](message_retention_policies.md)
- [Pluggable Modules]() - [Pluggable Modules](modules.md)
- [Third Party Rules]() - [Third Party Rules]()
- [Spam Checker](spam_checker.md) - [Spam Checker](spam_checker.md)
- [Presence Router](presence_router_module.md) - [Presence Router](presence_router_module.md)

258
docs/modules.md Normal file
View File

@ -0,0 +1,258 @@
# Modules
Synapse supports extending its functionality by configuring external modules.
## Using modules
To use a module on Synapse, add it to the `modules` section of the configuration file:
```yaml
modules:
- module: my_super_module.MySuperClass
config:
do_thing: true
- module: my_other_super_module.SomeClass
config: {}
```
Each module is defined by a path to a Python class as well as a configuration. This
information for a given module should be available in the module's own documentation.
**Note**: When using third-party modules, you effectively allow someone else to run
custom code on your Synapse homeserver. Server admins are encouraged to verify the
provenance of the modules they use on their homeserver and make sure the modules aren't
running malicious code on their instance.
Also note that we are currently in the process of migrating module interfaces to this
system. While some interfaces might be compatible with it, others still require
configuring modules in another part of Synapse's configuration file. Currently, only the
spam checker interface is compatible with this new system.
## Writing a module
A module is a Python class that uses Synapse's module API to interact with the
homeserver. It can register callbacks that Synapse will call on specific operations, as
well as web resources to attach to Synapse's web server.
When instantiated, a module is given its parsed configuration as well as an instance of
the `synapse.module_api.ModuleApi` class. The configuration is a dictionary, and is
either the output of the module's `parse_config` static method (see below), or the
configuration associated with the module in Synapse's configuration file.
See the documentation for the `ModuleApi` class
[here](https://github.com/matrix-org/synapse/blob/master/synapse/module_api/__init__.py).
### Handling the module's configuration
A module can implement the following static method:
```python
@staticmethod
def parse_config(config: dict) -> dict
```
This method is given a dictionary resulting from parsing the YAML configuration for the
module. It may modify it (for example by parsing durations expressed as strings (e.g.
"5d") into milliseconds, etc.), and return the modified dictionary. It may also verify
that the configuration is correct, and raise an instance of
`synapse.module_api.errors.ConfigError` if not.
### Registering a web resource
Modules can register web resources onto Synapse's web server using the following module
API method:
```python
def ModuleApi.register_web_resource(path: str, resource: IResource)
```
The path is the full absolute path to register the resource at. For example, if you
register a resource for the path `/_synapse/client/my_super_module/say_hello`, Synapse
will serve it at `http(s)://[HS_URL]/_synapse/client/my_super_module/say_hello`. Note
that Synapse does not allow registering resources for several sub-paths in the `/_matrix`
namespace (such as anything under `/_matrix/client` for example). It is strongly
recommended that modules register their web resources under the `/_synapse/client`
namespace.
The provided resource is a Python class that implements Twisted's [IResource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.IResource.html)
interface (such as [Resource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resource.html)).
Only one resource can be registered for a given path. If several modules attempt to
register a resource for the same path, the module that appears first in Synapse's
configuration file takes priority.
Modules **must** register their web resources in their `__init__` method.
### Registering a callback
Modules can use Synapse's module API to register callbacks. Callbacks are functions that
Synapse will call when performing specific actions. Callbacks must be asynchronous, and
are split in categories. A single module may implement callbacks from multiple categories,
and is under no obligation to implement all callbacks from the categories it registers
callbacks for.
#### Spam checker callbacks
To register one of the callbacks described in this section, a module needs to use the
module API's `register_spam_checker_callbacks` method. The callback functions are passed
to `register_spam_checker_callbacks` as keyword arguments, with the callback name as the
argument name and the function as its value. This is demonstrated in the example below.
The available spam checker callbacks are:
```python
def check_event_for_spam(event: "synapse.events.EventBase") -> Union[bool, str]
```
Called when receiving an event from a client or via federation. The module can return
either a `bool` to indicate whether the event must be rejected because of spam, or a `str`
to indicate the event must be rejected because of spam and to give a rejection reason to
forward to clients.
```python
def user_may_invite(inviter: str, invitee: str, room_id: str) -> bool
```
Called when processing an invitation. The module must return a `bool` indicating whether
the inviter can invite the invitee to the given room. Both inviter and invitee are
represented by their Matrix user ID (i.e. `@alice:example.com`).
```python
def user_may_create_room(user: str) -> bool
```
Called when processing a room creation request. The module must return a `bool` indicating
whether the given user (represented by their Matrix user ID) is allowed to create a room.
```python
def user_may_create_room_alias(user: str, room_alias: "synapse.types.RoomAlias") -> bool
```
Called when trying to associate an alias with an existing room. The module must return a
`bool` indicating whether the given user (represented by their Matrix user ID) is allowed
to set the given alias.
```python
def user_may_publish_room(user: str, room_id: str) -> bool
```
Called when trying to publish a room to the homeserver's public rooms directory. The
module must return a `bool` indicating whether the given user (represented by their
Matrix user ID) is allowed to publish the given room.
```python
def check_username_for_spam(user_profile: Dict[str, str]) -> bool
```
Called when computing search results in the user directory. The module must return a
`bool` indicating whether the given user profile can appear in search results. The profile
is represented as a dictionary with the following keys:
* `user_id`: The Matrix ID for this user.
* `display_name`: The user's display name.
* `avatar_url`: The `mxc://` URL to the user's avatar.
The module is given a copy of the original dictionary, so modifying it from within the
module cannot modify a user's profile when included in user directory search results.
```python
def check_registration_for_spam(
email_threepid: Optional[dict],
username: Optional[str],
request_info: Collection[Tuple[str, str]],
auth_provider_id: Optional[str] = None,
) -> "synapse.spam_checker_api.RegistrationBehaviour"
```
Called when registering a new user. The module must return a `RegistrationBehaviour`
indicating whether the registration can go through or must be denied, or whether the user
may be allowed to register but will be shadow banned.
The arguments passed to this callback are:
* `email_threepid`: The email address used for registering, if any.
* `username`: The username the user would like to register. Can be `None`, meaning that
Synapse will generate one later.
* `request_info`: A collection of tuples, which first item is a user agent, and which
second item is an IP address. These user agents and IP addresses are the ones that were
used during the registration process.
* `auth_provider_id`: The identifier of the SSO authentication provider, if any.
```python
def check_media_file_for_spam(
file_wrapper: "synapse.rest.media.v1.media_storage.ReadableFileWrapper",
file_info: "synapse.rest.media.v1._base.FileInfo"
) -> bool
```
Called when storing a local or remote file. The module must return a boolean indicating
whether the given file can be stored in the homeserver's media store.
### Porting an existing module that uses the old interface
In order to port a module that uses Synapse's old module interface, its author needs to:
* ensure the module's callbacks are all asynchronous.
* register their callbacks using one or more of the `register_[...]_callbacks` methods
from the `ModuleApi` class in the module's `__init__` method (see [this section](#registering-a-web-resource)
for more info).
Additionally, if the module is packaged with an additional web resource, the module
should register this resource in its `__init__` method using the `register_web_resource`
method from the `ModuleApi` class (see [this section](#registering-a-web-resource) for
more info).
The module's author should also update any example in the module's configuration to only
use the new `modules` section in Synapse's configuration file (see [this section](#using-modules)
for more info).
### Example
The example below is a module that implements the spam checker callback
`user_may_create_room` to deny room creation to user `@evilguy:example.com`, and registers
a web resource to the path `/_synapse/client/demo/hello` that returns a JSON object.
```python
import json
from twisted.web.resource import Resource
from twisted.web.server import Request
from synapse.module_api import ModuleApi
class DemoResource(Resource):
def __init__(self, config):
super(DemoResource, self).__init__()
self.config = config
def render_GET(self, request: Request):
name = request.args.get(b"name")[0]
request.setHeader(b"Content-Type", b"application/json")
return json.dumps({"hello": name})
class DemoModule:
def __init__(self, config: dict, api: ModuleApi):
self.config = config
self.api = api
self.api.register_web_resource(
path="/_synapse/client/demo/hello",
resource=DemoResource(self.config),
)
self.api.register_spam_checker_callbacks(
user_may_create_room=self.user_may_create_room,
)
@staticmethod
def parse_config(config):
return config
async def user_may_create_room(self, user: str) -> bool:
if user == "@evilguy:example.com":
return False
return True
```

View File

@ -31,6 +31,22 @@
# #
# [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html # [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
## Modules ##
# Server admins can expand Synapse's functionality with external modules.
#
# See https://matrix-org.github.io/synapse/develop/modules.html for more
# documentation on how to configure or create custom modules for Synapse.
#
modules:
# - module: my_super_module.MySuperClass
# config:
# do_thing: true
# - module: my_other_super_module.SomeClass
# config: {}
## Server ## ## Server ##
# The public-facing domain of the server # The public-facing domain of the server
@ -2491,19 +2507,6 @@ push:
#group_unread_count_by_room: false #group_unread_count_by_room: false
# Spam checkers are third-party modules that can block specific actions
# of local users, such as creating rooms and registering undesirable
# usernames, as well as remote users by redacting incoming events.
#
spam_checker:
#- module: "my_custom_project.SuperSpamChecker"
# config:
# example_option: 'things'
#- module: "some_other_project.BadEventStopper"
# config:
# example_stop_events_from: ['@bad:example.com']
## Rooms ## ## Rooms ##
# Controls whether locally-created rooms should be end-to-end encrypted by # Controls whether locally-created rooms should be end-to-end encrypted by

View File

@ -1,3 +1,7 @@
**Note: this page of the Synapse documentation is now deprecated. For up to date
documentation on setting up or writing a spam checker module, please see
[this page](https://matrix-org.github.io/synapse/develop/modules.html).**
# Handling spam in Synapse # Handling spam in Synapse
Synapse has support to customize spam checking behavior. It can plug into a Synapse has support to customize spam checking behavior. It can plug into a

View File

@ -35,6 +35,7 @@ from synapse.app import check_bind_error
from synapse.app.phone_stats_home import start_phone_stats_home from synapse.app.phone_stats_home import start_phone_stats_home
from synapse.config.homeserver import HomeServerConfig from synapse.config.homeserver import HomeServerConfig
from synapse.crypto import context_factory from synapse.crypto import context_factory
from synapse.events.spamcheck import load_legacy_spam_checkers
from synapse.logging.context import PreserveLoggingContext from synapse.logging.context import PreserveLoggingContext
from synapse.metrics.background_process_metrics import wrap_as_background_process from synapse.metrics.background_process_metrics import wrap_as_background_process
from synapse.metrics.jemalloc import setup_jemalloc_stats from synapse.metrics.jemalloc import setup_jemalloc_stats
@ -330,6 +331,14 @@ async def start(hs: "synapse.server.HomeServer"):
# Start the tracer # Start the tracer
synapse.logging.opentracing.init_tracer(hs) # type: ignore[attr-defined] # noqa synapse.logging.opentracing.init_tracer(hs) # type: ignore[attr-defined] # noqa
# Instantiate the modules so they can register their web resources to the module API
# before we start the listeners.
module_api = hs.get_module_api()
for module, config in hs.config.modules.loaded_modules:
module(config=config, api=module_api)
load_legacy_spam_checkers(hs)
# It is now safe to start your Synapse. # It is now safe to start your Synapse.
hs.start_listening() hs.start_listening()
hs.get_datastore().db_pool.start_profiling() hs.get_datastore().db_pool.start_profiling()

View File

@ -354,6 +354,10 @@ class GenericWorkerServer(HomeServer):
if name == "replication": if name == "replication":
resources[REPLICATION_PREFIX] = ReplicationRestResource(self) resources[REPLICATION_PREFIX] = ReplicationRestResource(self)
# Attach additional resources registered by modules.
resources.update(self._module_web_resources)
self._module_web_resources_consumed = True
root_resource = create_resource_tree(resources, OptionsResource()) root_resource = create_resource_tree(resources, OptionsResource())
_base.listen_tcp( _base.listen_tcp(

View File

@ -124,6 +124,10 @@ class SynapseHomeServer(HomeServer):
) )
resources[path] = resource resources[path] = resource
# Attach additional resources registered by modules.
resources.update(self._module_web_resources)
self._module_web_resources_consumed = True
# try to find something useful to redirect '/' to # try to find something useful to redirect '/' to
if WEB_CLIENT_PREFIX in resources: if WEB_CLIENT_PREFIX in resources:
root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX) root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)

View File

@ -16,6 +16,7 @@ from synapse.config import (
key, key,
logger, logger,
metrics, metrics,
modules,
oidc, oidc,
password_auth_providers, password_auth_providers,
push, push,
@ -85,6 +86,7 @@ class RootConfig:
thirdpartyrules: third_party_event_rules.ThirdPartyRulesConfig thirdpartyrules: third_party_event_rules.ThirdPartyRulesConfig
tracer: tracer.TracerConfig tracer: tracer.TracerConfig
redis: redis.RedisConfig redis: redis.RedisConfig
modules: modules.ModulesConfig
config_classes: List = ... config_classes: List = ...
def __init__(self) -> None: ... def __init__(self) -> None: ...

View File

@ -1,5 +1,4 @@
# Copyright 2014-2016 OpenMarket Ltd # Copyright 2021 The Matrix.org Foundation C.I.C.
# Copyright 2018 New Vector Ltd
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -30,6 +29,7 @@ from .jwt import JWTConfig
from .key import KeyConfig from .key import KeyConfig
from .logger import LoggingConfig from .logger import LoggingConfig
from .metrics import MetricsConfig from .metrics import MetricsConfig
from .modules import ModulesConfig
from .oidc import OIDCConfig from .oidc import OIDCConfig
from .password_auth_providers import PasswordAuthProviderConfig from .password_auth_providers import PasswordAuthProviderConfig
from .push import PushConfig from .push import PushConfig
@ -56,6 +56,7 @@ from .workers import WorkerConfig
class HomeServerConfig(RootConfig): class HomeServerConfig(RootConfig):
config_classes = [ config_classes = [
ModulesConfig,
ServerConfig, ServerConfig,
TlsConfig, TlsConfig,
FederationConfig, FederationConfig,

49
synapse/config/modules.py Normal file
View File

@ -0,0 +1,49 @@
# Copyright 2021 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from typing import Any, Dict, List, Tuple
from synapse.config._base import Config, ConfigError
from synapse.util.module_loader import load_module
class ModulesConfig(Config):
section = "modules"
def read_config(self, config: dict, **kwargs):
self.loaded_modules: List[Tuple[Any, Dict]] = []
configured_modules = config.get("modules") or []
for i, module in enumerate(configured_modules):
config_path = ("modules", "<item %i>" % i)
if not isinstance(module, dict):
raise ConfigError("expected a mapping", config_path)
self.loaded_modules.append(load_module(module, config_path))
def generate_config_section(self, **kwargs):
return """
## Modules ##
# Server admins can expand Synapse's functionality with external modules.
#
# See https://matrix-org.github.io/synapse/develop/modules.html for more
# documentation on how to configure or create custom modules for Synapse.
#
modules:
# - module: my_super_module.MySuperClass
# config:
# do_thing: true
# - module: my_other_super_module.SomeClass
# config: {}
"""

View File

@ -42,18 +42,3 @@ class SpamCheckerConfig(Config):
self.spam_checkers.append(load_module(spam_checker, config_path)) self.spam_checkers.append(load_module(spam_checker, config_path))
else: else:
raise ConfigError("spam_checker syntax is incorrect") raise ConfigError("spam_checker syntax is incorrect")
def generate_config_section(self, **kwargs):
return """\
# Spam checkers are third-party modules that can block specific actions
# of local users, such as creating rooms and registering undesirable
# usernames, as well as remote users by redacting incoming events.
#
spam_checker:
#- module: "my_custom_project.SuperSpamChecker"
# config:
# example_option: 'things'
#- module: "some_other_project.BadEventStopper"
# config:
# example_stop_events_from: ['@bad:example.com']
"""

View File

@ -15,7 +15,18 @@
import inspect import inspect
import logging import logging
from typing import TYPE_CHECKING, Any, Collection, Dict, List, Optional, Tuple, Union from typing import (
TYPE_CHECKING,
Any,
Awaitable,
Callable,
Collection,
Dict,
List,
Optional,
Tuple,
Union,
)
from synapse.rest.media.v1._base import FileInfo from synapse.rest.media.v1._base import FileInfo
from synapse.rest.media.v1.media_storage import ReadableFileWrapper from synapse.rest.media.v1.media_storage import ReadableFileWrapper
@ -29,20 +40,186 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
CHECK_EVENT_FOR_SPAM_CALLBACK = Callable[
["synapse.events.EventBase"],
Awaitable[Union[bool, str]],
]
USER_MAY_INVITE_CALLBACK = Callable[[str, str, str], Awaitable[bool]]
USER_MAY_CREATE_ROOM_CALLBACK = Callable[[str], Awaitable[bool]]
USER_MAY_CREATE_ROOM_ALIAS_CALLBACK = Callable[[str, RoomAlias], Awaitable[bool]]
USER_MAY_PUBLISH_ROOM_CALLBACK = Callable[[str, str], Awaitable[bool]]
CHECK_USERNAME_FOR_SPAM_CALLBACK = Callable[[Dict[str, str]], Awaitable[bool]]
LEGACY_CHECK_REGISTRATION_FOR_SPAM_CALLBACK = Callable[
[
Optional[dict],
Optional[str],
Collection[Tuple[str, str]],
],
Awaitable[RegistrationBehaviour],
]
CHECK_REGISTRATION_FOR_SPAM_CALLBACK = Callable[
[
Optional[dict],
Optional[str],
Collection[Tuple[str, str]],
Optional[str],
],
Awaitable[RegistrationBehaviour],
]
CHECK_MEDIA_FILE_FOR_SPAM_CALLBACK = Callable[
[ReadableFileWrapper, FileInfo],
Awaitable[bool],
]
def load_legacy_spam_checkers(hs: "synapse.server.HomeServer"):
"""Wrapper that loads spam checkers configured using the old configuration, and
registers the spam checker hooks they implement.
"""
spam_checkers = [] # type: List[Any]
api = hs.get_module_api()
for module, config in hs.config.spam_checkers:
# Older spam checkers don't accept the `api` argument, so we
# try and detect support.
spam_args = inspect.getfullargspec(module)
if "api" in spam_args.args:
spam_checkers.append(module(config=config, api=api))
else:
spam_checkers.append(module(config=config))
# The known spam checker hooks. If a spam checker module implements a method
# which name appears in this set, we'll want to register it.
spam_checker_methods = {
"check_event_for_spam",
"user_may_invite",
"user_may_create_room",
"user_may_create_room_alias",
"user_may_publish_room",
"check_username_for_spam",
"check_registration_for_spam",
"check_media_file_for_spam",
}
for spam_checker in spam_checkers:
# Methods on legacy spam checkers might not be async, so we wrap them around a
# wrapper that will call maybe_awaitable on the result.
def async_wrapper(f: Optional[Callable]) -> Optional[Callable[..., Awaitable]]:
# f might be None if the callback isn't implemented by the module. In this
# case we don't want to register a callback at all so we return None.
if f is None:
return None
if f.__name__ == "check_registration_for_spam":
checker_args = inspect.signature(f)
if len(checker_args.parameters) == 3:
# Backwards compatibility; some modules might implement a hook that
# doesn't expect a 4th argument. In this case, wrap it in a function
# that gives it only 3 arguments and drops the auth_provider_id on
# the floor.
def wrapper(
email_threepid: Optional[dict],
username: Optional[str],
request_info: Collection[Tuple[str, str]],
auth_provider_id: Optional[str],
) -> Union[Awaitable[RegistrationBehaviour], RegistrationBehaviour]:
# We've already made sure f is not None above, but mypy doesn't
# do well across function boundaries so we need to tell it f is
# definitely not None.
assert f is not None
return f(
email_threepid,
username,
request_info,
)
f = wrapper
elif len(checker_args.parameters) != 4:
raise RuntimeError(
"Bad signature for callback check_registration_for_spam",
)
def run(*args, **kwargs):
# We've already made sure f is not None above, but mypy doesn't do well
# across function boundaries so we need to tell it f is definitely not
# None.
assert f is not None
return maybe_awaitable(f(*args, **kwargs))
return run
# Register the hooks through the module API.
hooks = {
hook: async_wrapper(getattr(spam_checker, hook, None))
for hook in spam_checker_methods
}
api.register_spam_checker_callbacks(**hooks)
class SpamChecker: class SpamChecker:
def __init__(self, hs: "synapse.server.HomeServer"): def __init__(self):
self.spam_checkers = [] # type: List[Any] self._check_event_for_spam_callbacks: List[CHECK_EVENT_FOR_SPAM_CALLBACK] = []
api = hs.get_module_api() self._user_may_invite_callbacks: List[USER_MAY_INVITE_CALLBACK] = []
self._user_may_create_room_callbacks: List[USER_MAY_CREATE_ROOM_CALLBACK] = []
self._user_may_create_room_alias_callbacks: List[
USER_MAY_CREATE_ROOM_ALIAS_CALLBACK
] = []
self._user_may_publish_room_callbacks: List[USER_MAY_PUBLISH_ROOM_CALLBACK] = []
self._check_username_for_spam_callbacks: List[
CHECK_USERNAME_FOR_SPAM_CALLBACK
] = []
self._check_registration_for_spam_callbacks: List[
CHECK_REGISTRATION_FOR_SPAM_CALLBACK
] = []
self._check_media_file_for_spam_callbacks: List[
CHECK_MEDIA_FILE_FOR_SPAM_CALLBACK
] = []
for module, config in hs.config.spam_checkers: def register_callbacks(
# Older spam checkers don't accept the `api` argument, so we self,
# try and detect support. check_event_for_spam: Optional[CHECK_EVENT_FOR_SPAM_CALLBACK] = None,
spam_args = inspect.getfullargspec(module) user_may_invite: Optional[USER_MAY_INVITE_CALLBACK] = None,
if "api" in spam_args.args: user_may_create_room: Optional[USER_MAY_CREATE_ROOM_CALLBACK] = None,
self.spam_checkers.append(module(config=config, api=api)) user_may_create_room_alias: Optional[
else: USER_MAY_CREATE_ROOM_ALIAS_CALLBACK
self.spam_checkers.append(module(config=config)) ] = None,
user_may_publish_room: Optional[USER_MAY_PUBLISH_ROOM_CALLBACK] = None,
check_username_for_spam: Optional[CHECK_USERNAME_FOR_SPAM_CALLBACK] = None,
check_registration_for_spam: Optional[
CHECK_REGISTRATION_FOR_SPAM_CALLBACK
] = None,
check_media_file_for_spam: Optional[CHECK_MEDIA_FILE_FOR_SPAM_CALLBACK] = None,
):
"""Register callbacks from module for each hook."""
if check_event_for_spam is not None:
self._check_event_for_spam_callbacks.append(check_event_for_spam)
if user_may_invite is not None:
self._user_may_invite_callbacks.append(user_may_invite)
if user_may_create_room is not None:
self._user_may_create_room_callbacks.append(user_may_create_room)
if user_may_create_room_alias is not None:
self._user_may_create_room_alias_callbacks.append(
user_may_create_room_alias,
)
if user_may_publish_room is not None:
self._user_may_publish_room_callbacks.append(user_may_publish_room)
if check_username_for_spam is not None:
self._check_username_for_spam_callbacks.append(check_username_for_spam)
if check_registration_for_spam is not None:
self._check_registration_for_spam_callbacks.append(
check_registration_for_spam,
)
if check_media_file_for_spam is not None:
self._check_media_file_for_spam_callbacks.append(check_media_file_for_spam)
async def check_event_for_spam( async def check_event_for_spam(
self, event: "synapse.events.EventBase" self, event: "synapse.events.EventBase"
@ -60,9 +237,10 @@ class SpamChecker:
True or a string if the event is spammy. If a string is returned it True or a string if the event is spammy. If a string is returned it
will be used as the error message returned to the user. will be used as the error message returned to the user.
""" """
for spam_checker in self.spam_checkers: for callback in self._check_event_for_spam_callbacks:
if await maybe_awaitable(spam_checker.check_event_for_spam(event)): res = await callback(event) # type: Union[bool, str]
return True if res:
return res
return False return False
@ -81,15 +259,8 @@ class SpamChecker:
Returns: Returns:
True if the user may send an invite, otherwise False True if the user may send an invite, otherwise False
""" """
for spam_checker in self.spam_checkers: for callback in self._user_may_invite_callbacks:
if ( if await callback(inviter_userid, invitee_userid, room_id) is False:
await maybe_awaitable(
spam_checker.user_may_invite(
inviter_userid, invitee_userid, room_id
)
)
is False
):
return False return False
return True return True
@ -105,11 +276,8 @@ class SpamChecker:
Returns: Returns:
True if the user may create a room, otherwise False True if the user may create a room, otherwise False
""" """
for spam_checker in self.spam_checkers: for callback in self._user_may_create_room_callbacks:
if ( if await callback(userid) is False:
await maybe_awaitable(spam_checker.user_may_create_room(userid))
is False
):
return False return False
return True return True
@ -128,13 +296,8 @@ class SpamChecker:
Returns: Returns:
True if the user may create a room alias, otherwise False True if the user may create a room alias, otherwise False
""" """
for spam_checker in self.spam_checkers: for callback in self._user_may_create_room_alias_callbacks:
if ( if await callback(userid, room_alias) is False:
await maybe_awaitable(
spam_checker.user_may_create_room_alias(userid, room_alias)
)
is False
):
return False return False
return True return True
@ -151,13 +314,8 @@ class SpamChecker:
Returns: Returns:
True if the user may publish the room, otherwise False True if the user may publish the room, otherwise False
""" """
for spam_checker in self.spam_checkers: for callback in self._user_may_publish_room_callbacks:
if ( if await callback(userid, room_id) is False:
await maybe_awaitable(
spam_checker.user_may_publish_room(userid, room_id)
)
is False
):
return False return False
return True return True
@ -177,15 +335,11 @@ class SpamChecker:
Returns: Returns:
True if the user is spammy. True if the user is spammy.
""" """
for spam_checker in self.spam_checkers: for callback in self._check_username_for_spam_callbacks:
# For backwards compatibility, only run if the method exists on the # Make a copy of the user profile object to ensure the spam checker cannot
# spam checker # modify it.
checker = getattr(spam_checker, "check_username_for_spam", None) if await callback(user_profile.copy()):
if checker: return True
# Make a copy of the user profile object to ensure the spam checker
# cannot modify it.
if await maybe_awaitable(checker(user_profile.copy())):
return True
return False return False
@ -211,33 +365,13 @@ class SpamChecker:
Enum for how the request should be handled Enum for how the request should be handled
""" """
for spam_checker in self.spam_checkers: for callback in self._check_registration_for_spam_callbacks:
# For backwards compatibility, only run if the method exists on the behaviour = await (
# spam checker callback(email_threepid, username, request_info, auth_provider_id)
checker = getattr(spam_checker, "check_registration_for_spam", None) )
if checker: assert isinstance(behaviour, RegistrationBehaviour)
# Provide auth_provider_id if the function supports it if behaviour != RegistrationBehaviour.ALLOW:
checker_args = inspect.signature(checker) return behaviour
if len(checker_args.parameters) == 4:
d = checker(
email_threepid,
username,
request_info,
auth_provider_id,
)
elif len(checker_args.parameters) == 3:
d = checker(email_threepid, username, request_info)
else:
logger.error(
"Invalid signature for %s.check_registration_for_spam. Denying registration",
spam_checker.__module__,
)
return RegistrationBehaviour.DENY
behaviour = await maybe_awaitable(d)
assert isinstance(behaviour, RegistrationBehaviour)
if behaviour != RegistrationBehaviour.ALLOW:
return behaviour
return RegistrationBehaviour.ALLOW return RegistrationBehaviour.ALLOW
@ -275,13 +409,9 @@ class SpamChecker:
allowed. allowed.
""" """
for spam_checker in self.spam_checkers: for callback in self._check_media_file_for_spam_callbacks:
# For backwards compatibility, only run if the method exists on the spam = await callback(file_wrapper, file_info)
# spam checker if spam:
checker = getattr(spam_checker, "check_media_file_for_spam", None) return True
if checker:
spam = await maybe_awaitable(checker(file_wrapper, file_info))
if spam:
return True
return False return False

View File

@ -195,7 +195,7 @@ class RegistrationHandler(BaseHandler):
bind_emails: list of emails to bind to this account. bind_emails: list of emails to bind to this account.
by_admin: True if this registration is being made via the by_admin: True if this registration is being made via the
admin api, otherwise False. admin api, otherwise False.
user_agent_ips: Tuples of IP addresses and user-agents used user_agent_ips: Tuples of user-agents and IP addresses used
during the registration process. during the registration process.
auth_provider_id: The SSO IdP the user used, if any. auth_provider_id: The SSO IdP the user used, if any.
Returns: Returns:

View File

@ -16,6 +16,7 @@ import logging
from typing import TYPE_CHECKING, Any, Generator, Iterable, List, Optional, Tuple from typing import TYPE_CHECKING, Any, Generator, Iterable, List, Optional, Tuple
from twisted.internet import defer from twisted.internet import defer
from twisted.web.resource import IResource
from synapse.events import EventBase from synapse.events import EventBase
from synapse.http.client import SimpleHttpClient from synapse.http.client import SimpleHttpClient
@ -42,7 +43,7 @@ class ModuleApi:
can register new users etc if necessary. can register new users etc if necessary.
""" """
def __init__(self, hs, auth_handler): def __init__(self, hs: "HomeServer", auth_handler):
self._hs = hs self._hs = hs
self._store = hs.get_datastore() self._store = hs.get_datastore()
@ -56,6 +57,33 @@ class ModuleApi:
self._http_client = hs.get_simple_http_client() # type: SimpleHttpClient self._http_client = hs.get_simple_http_client() # type: SimpleHttpClient
self._public_room_list_manager = PublicRoomListManager(hs) self._public_room_list_manager = PublicRoomListManager(hs)
self._spam_checker = hs.get_spam_checker()
#################################################################################
# The following methods should only be called during the module's initialisation.
@property
def register_spam_checker_callbacks(self):
"""Registers callbacks for spam checking capabilities."""
return self._spam_checker.register_callbacks
def register_web_resource(self, path: str, resource: IResource):
"""Registers a web resource to be served at the given path.
This function should be called during initialisation of the module.
If multiple modules register a resource for the same path, the module that
appears the highest in the configuration file takes priority.
Args:
path: The path to register the resource for.
resource: The resource to attach to this path.
"""
self._hs.register_module_web_resource(path, resource)
#########################################################################
# The following methods can be called by the module at any point in time.
@property @property
def http_client(self): def http_client(self):
"""Allows making outbound HTTP requests to remote resources. """Allows making outbound HTTP requests to remote resources.

View File

@ -15,3 +15,4 @@
"""Exception types which are exposed as part of the stable module API""" """Exception types which are exposed as part of the stable module API"""
from synapse.api.errors import RedirectException, SynapseError # noqa: F401 from synapse.api.errors import RedirectException, SynapseError # noqa: F401
from synapse.config._base import ConfigError # noqa: F401

View File

@ -1,6 +1,4 @@
# Copyright 2014-2016 OpenMarket Ltd # Copyright 2021 The Matrix.org Foundation C.I.C.
# Copyright 2017-2018 New Vector Ltd
# Copyright 2019 The Matrix.org Foundation C.I.C.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -39,6 +37,7 @@ import twisted.internet.tcp
from twisted.internet import defer from twisted.internet import defer
from twisted.mail.smtp import sendmail from twisted.mail.smtp import sendmail
from twisted.web.iweb import IPolicyForHTTPS from twisted.web.iweb import IPolicyForHTTPS
from twisted.web.resource import IResource
from synapse.api.auth import Auth from synapse.api.auth import Auth
from synapse.api.filtering import Filtering from synapse.api.filtering import Filtering
@ -258,6 +257,38 @@ class HomeServer(metaclass=abc.ABCMeta):
self.datastores = None # type: Optional[Databases] self.datastores = None # type: Optional[Databases]
self._module_web_resources: Dict[str, IResource] = {}
self._module_web_resources_consumed = False
def register_module_web_resource(self, path: str, resource: IResource):
"""Allows a module to register a web resource to be served at the given path.
If multiple modules register a resource for the same path, the module that
appears the highest in the configuration file takes priority.
Args:
path: The path to register the resource for.
resource: The resource to attach to this path.
Raises:
SynapseError(500): A module tried to register a web resource after the HTTP
listeners have been started.
"""
if self._module_web_resources_consumed:
raise RuntimeError(
"Tried to register a web resource from a module after startup",
)
# Don't register a resource that's already been registered.
if path not in self._module_web_resources.keys():
self._module_web_resources[path] = resource
else:
logger.warning(
"Module tried to register a web resource for path %s but another module"
" has already registered a resource for this path.",
path,
)
def get_instance_id(self) -> str: def get_instance_id(self) -> str:
"""A unique ID for this synapse process instance. """A unique ID for this synapse process instance.
@ -646,7 +677,7 @@ class HomeServer(metaclass=abc.ABCMeta):
@cache_in_self @cache_in_self
def get_spam_checker(self) -> SpamChecker: def get_spam_checker(self) -> SpamChecker:
return SpamChecker(self) return SpamChecker()
@cache_in_self @cache_in_self
def get_third_party_event_rules(self) -> ThirdPartyEventRules: def get_third_party_event_rules(self) -> ThirdPartyEventRules:

View File

@ -51,21 +51,26 @@ def load_module(provider: dict, config_path: Iterable[str]) -> Tuple[Type, Any]:
# Load the module config. If None, pass an empty dictionary instead # Load the module config. If None, pass an empty dictionary instead
module_config = provider.get("config") or {} module_config = provider.get("config") or {}
try: if hasattr(provider_class, "parse_config"):
provider_config = provider_class.parse_config(module_config) try:
except jsonschema.ValidationError as e: provider_config = provider_class.parse_config(module_config)
raise json_error_to_config_error(e, itertools.chain(config_path, ("config",))) except jsonschema.ValidationError as e:
except ConfigError as e: raise json_error_to_config_error(
raise _wrap_config_error( e, itertools.chain(config_path, ("config",))
"Failed to parse config for module %r" % (modulename,), )
prefix=itertools.chain(config_path, ("config",)), except ConfigError as e:
e=e, raise _wrap_config_error(
) "Failed to parse config for module %r" % (modulename,),
except Exception as e: prefix=itertools.chain(config_path, ("config",)),
raise ConfigError( e=e,
"Failed to parse config for module %r" % (modulename,), )
path=itertools.chain(config_path, ("config",)), except Exception as e:
) from e raise ConfigError(
"Failed to parse config for module %r" % (modulename,),
path=itertools.chain(config_path, ("config",)),
) from e
else:
provider_config = module_config
return provider_class, provider_config return provider_class, provider_config

View File

@ -27,6 +27,58 @@ from tests.utils import mock_getRawHeaders
from .. import unittest from .. import unittest
class TestSpamChecker:
def __init__(self, config, api):
api.register_spam_checker_callbacks(
check_registration_for_spam=self.check_registration_for_spam,
)
@staticmethod
def parse_config(config):
return config
async def check_registration_for_spam(
self,
email_threepid,
username,
request_info,
auth_provider_id,
):
pass
class DenyAll(TestSpamChecker):
async def check_registration_for_spam(
self,
email_threepid,
username,
request_info,
auth_provider_id,
):
return RegistrationBehaviour.DENY
class BanAll(TestSpamChecker):
async def check_registration_for_spam(
self,
email_threepid,
username,
request_info,
auth_provider_id,
):
return RegistrationBehaviour.SHADOW_BAN
class BanBadIdPUser(TestSpamChecker):
async def check_registration_for_spam(
self, email_threepid, username, request_info, auth_provider_id=None
):
# Reject any user coming from CAS and whose username contains profanity
if auth_provider_id == "cas" and "flimflob" in username:
return RegistrationBehaviour.DENY
return RegistrationBehaviour.ALLOW
class RegistrationTestCase(unittest.HomeserverTestCase): class RegistrationTestCase(unittest.HomeserverTestCase):
"""Tests the RegistrationHandler.""" """Tests the RegistrationHandler."""
@ -42,6 +94,11 @@ class RegistrationTestCase(unittest.HomeserverTestCase):
hs_config["limit_usage_by_mau"] = True hs_config["limit_usage_by_mau"] = True
hs = self.setup_test_homeserver(config=hs_config) hs = self.setup_test_homeserver(config=hs_config)
module_api = hs.get_module_api()
for module, config in hs.config.modules.loaded_modules:
module(config=config, api=module_api)
return hs return hs
def prepare(self, reactor, clock, hs): def prepare(self, reactor, clock, hs):
@ -465,34 +522,30 @@ class RegistrationTestCase(unittest.HomeserverTestCase):
self.handler.register_user(localpart=invalid_user_id), SynapseError self.handler.register_user(localpart=invalid_user_id), SynapseError
) )
@override_config(
{
"modules": [
{
"module": TestSpamChecker.__module__ + ".DenyAll",
}
]
}
)
def test_spam_checker_deny(self): def test_spam_checker_deny(self):
"""A spam checker can deny registration, which results in an error.""" """A spam checker can deny registration, which results in an error."""
class DenyAll:
def check_registration_for_spam(
self, email_threepid, username, request_info
):
return RegistrationBehaviour.DENY
# Configure a spam checker that denies all users.
spam_checker = self.hs.get_spam_checker()
spam_checker.spam_checkers = [DenyAll()]
self.get_failure(self.handler.register_user(localpart="user"), SynapseError) self.get_failure(self.handler.register_user(localpart="user"), SynapseError)
@override_config(
{
"modules": [
{
"module": TestSpamChecker.__module__ + ".BanAll",
}
]
}
)
def test_spam_checker_shadow_ban(self): def test_spam_checker_shadow_ban(self):
"""A spam checker can choose to shadow-ban a user, which allows registration to succeed.""" """A spam checker can choose to shadow-ban a user, which allows registration to succeed."""
class BanAll:
def check_registration_for_spam(
self, email_threepid, username, request_info
):
return RegistrationBehaviour.SHADOW_BAN
# Configure a spam checker that denies all users.
spam_checker = self.hs.get_spam_checker()
spam_checker.spam_checkers = [BanAll()]
user_id = self.get_success(self.handler.register_user(localpart="user")) user_id = self.get_success(self.handler.register_user(localpart="user"))
# Get an access token. # Get an access token.
@ -512,22 +565,17 @@ class RegistrationTestCase(unittest.HomeserverTestCase):
self.assertTrue(requester.shadow_banned) self.assertTrue(requester.shadow_banned)
@override_config(
{
"modules": [
{
"module": TestSpamChecker.__module__ + ".BanBadIdPUser",
}
]
}
)
def test_spam_checker_receives_sso_type(self): def test_spam_checker_receives_sso_type(self):
"""Test rejecting registration based on SSO type""" """Test rejecting registration based on SSO type"""
class BanBadIdPUser:
def check_registration_for_spam(
self, email_threepid, username, request_info, auth_provider_id=None
):
# Reject any user coming from CAS and whose username contains profanity
if auth_provider_id == "cas" and "flimflob" in username:
return RegistrationBehaviour.DENY
return RegistrationBehaviour.ALLOW
# Configure a spam checker that denies a certain user on a specific IdP
spam_checker = self.hs.get_spam_checker()
spam_checker.spam_checkers = [BanBadIdPUser()]
f = self.get_failure( f = self.get_failure(
self.handler.register_user(localpart="bobflimflob", auth_provider_id="cas"), self.handler.register_user(localpart="bobflimflob", auth_provider_id="cas"),
SynapseError, SynapseError,

View File

@ -312,15 +312,13 @@ class UserDirectoryTestCase(unittest.HomeserverTestCase):
s = self.get_success(self.handler.search_users(u1, "user2", 10)) s = self.get_success(self.handler.search_users(u1, "user2", 10))
self.assertEqual(len(s["results"]), 1) self.assertEqual(len(s["results"]), 1)
async def allow_all(user_profile):
# Allow all users.
return False
# Configure a spam checker that does not filter any users. # Configure a spam checker that does not filter any users.
spam_checker = self.hs.get_spam_checker() spam_checker = self.hs.get_spam_checker()
spam_checker._check_username_for_spam_callbacks = [allow_all]
class AllowAll:
async def check_username_for_spam(self, user_profile):
# Allow all users.
return False
spam_checker.spam_checkers = [AllowAll()]
# The results do not change: # The results do not change:
# We get one search result when searching for user2 by user1. # We get one search result when searching for user2 by user1.
@ -328,12 +326,11 @@ class UserDirectoryTestCase(unittest.HomeserverTestCase):
self.assertEqual(len(s["results"]), 1) self.assertEqual(len(s["results"]), 1)
# Configure a spam checker that filters all users. # Configure a spam checker that filters all users.
class BlockAll: async def block_all(user_profile):
async def check_username_for_spam(self, user_profile): # All users are spammy.
# All users are spammy. return True
return True
spam_checker.spam_checkers = [BlockAll()] spam_checker._check_username_for_spam_callbacks = [block_all]
# User1 now gets no search results for any of the other users. # User1 now gets no search results for any of the other users.
s = self.get_success(self.handler.search_users(u1, "user2", 10)) s = self.get_success(self.handler.search_users(u1, "user2", 10))

View File

@ -27,6 +27,7 @@ from PIL import Image as Image
from twisted.internet import defer from twisted.internet import defer
from twisted.internet.defer import Deferred from twisted.internet.defer import Deferred
from synapse.events.spamcheck import load_legacy_spam_checkers
from synapse.logging.context import make_deferred_yieldable from synapse.logging.context import make_deferred_yieldable
from synapse.rest import admin from synapse.rest import admin
from synapse.rest.client.v1 import login from synapse.rest.client.v1 import login
@ -535,6 +536,8 @@ class SpamCheckerTestCase(unittest.HomeserverTestCase):
self.download_resource = self.media_repo.children[b"download"] self.download_resource = self.media_repo.children[b"download"]
self.upload_resource = self.media_repo.children[b"upload"] self.upload_resource = self.media_repo.children[b"upload"]
load_legacy_spam_checkers(hs)
def default_config(self): def default_config(self):
config = default_config("test") config = default_config("test")