Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-10-01 08:25:44 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'develop' into invite_power_level

Browse Source
This commit is contained in:
Paul "LeoNerd" Evans 2015-04-27 17:09:25 +01:00
commit 38432d8c25
17 changed files with 367 additions and 74 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
README.rst
View File

@ -86,7 +86,7 @@ Homeserver Installation
=======================
System requirements:
- POSIX-compliant system (tested on Linux & OSX)
- POSIX-compliant system (tested on Linux & OS X)
- Python 2.7
Synapse is written in python but some of the libraries is uses are written in
@ -128,6 +128,15 @@ To set up your homeserver, run (in your virtualenv, as before)::
Substituting your host and domain name as appropriate.
This will generate you a config file that you can then customise, but it will
also generate a set of keys for you. These keys will allow your Home Server to
identify itself to other Home Servers, so don't lose or delete them. It would be
wise to back them up somewhere safe. If, for whatever reason, you do need to
change your Home Server's keys, you may find that other Home Servers have the
old key cached. If you update the signing key, you should change the name of the
key in the <server name>.signing.key file (the second word, which by default is
, 'auto') to something different.
By default, registration of new users is disabled. You can either enable
registration in the config by specifying ``enable_registration: true``
(it is then recommended to also set up CAPTCHA), or
@ -367,10 +376,6 @@ SRV record, as that is the name other machines will expect it to have::
You may additionally want to pass one or more "-v" options, in order to
increase the verbosity of logging output; at least for initial testing.
For the initial alpha release, the homeserver is not speaking TLS for
either client-server or server-server traffic for ease of debugging. We have
also not spent any time yet getting the homeserver to run behind loadbalancers.
Running a Demo Federation of Homeservers
----------------------------------------
@ -433,7 +438,7 @@ track 3PID logins and publish end-user public keys.
It's currently early days for identity servers as Matrix is not yet using 3PIDs
as the primary means of identity and E2E encryption is not complete. As such,
we are running a single identity server (http://matrix.org:8090) at the current
we are running a single identity server (https://matrix.org) at the current
time.

34
UPGRADE.rst
View File

@ -1,3 +1,37 @@
Upgrading to v0.x.x
===================
Application services have had a breaking API change in this version.
They can no longer register themselves with a home server using the AS HTTP API. This
decision was made because a compromised application service with free reign to register
any regex in effect grants full read/write access to the home server if a regex of ``.*``
is used. An attack where a compromised AS re-registers itself with ``.*`` was deemed too
big of a security risk to ignore, and so the ability to register with the HS remotely has
been removed.
It has been replaced by specifying a list of application service registrations in
``homeserver.yaml``::
app_service_config_files: ["registration-01.yaml", "registration-02.yaml"]
Where ``registration-01.yaml`` looks like::
url: <String> # e.g. "https://my.application.service.com"
as_token: <String>
hs_token: <String>
sender_localpart: <String> # This is a new field which denotes the user_id localpart when using the AS token
namespaces:
users:
- exclusive: <Boolean>
regex: <String> # e.g. "@prefix_.*"
aliases:
- exclusive: <Boolean>
regex: <String>
rooms:
- exclusive: <Boolean>
regex: <String>
Upgrading to v0.8.0
===================

50
docs/metrics-howto.rst Normal file
View File

@ -0,0 +1,50 @@
How to monitor Synapse metrics using Prometheus
===============================================
1: Install prometheus:
Follow instructions at http://prometheus.io/docs/introduction/install/
2: Enable synapse metrics:
Simply setting a (local) port number will enable it. Pick a port.
prometheus itself defaults to 9090, so starting just above that for
locally monitored services seems reasonable. E.g. 9092:
Add to homeserver.yaml
metrics_port: 9092
Restart synapse
3: Check out synapse-prometheus-config
https://github.com/matrix-org/synapse-prometheus-config
4: Add ``synapse.html`` and ``synapse.rules``
The ``.html`` file needs to appear in prometheus's ``consoles`` directory,
and the ``.rules`` file needs to be invoked somewhere in the main config
file. A symlink to each from the git checkout into the prometheus directory
might be easiest to ensure ``git pull`` keeps it updated.
5: Add a prometheus target for synapse
This is easiest if prometheus runs on the same machine as synapse, as it can
then just use localhost::
global: {
rule_file: "synapse.rules"
}
job: {
name: "synapse"
target_group: {
target: "http://localhost:9092/"
}
}
6: Start prometheus::
./prometheus -config.file=prometheus.conf
7: Wait a few seconds for it to start and perform the first scrape,
then visit the console:
http://server-where-prometheus-runs:9090/consoles/synapse.html

48
synapse/api/auth.py
View File

@ -183,17 +183,7 @@ class Auth(object):
else:
join_rule = JoinRules.INVITE
user_level = self._get_power_level_from_event_state(
event,
event.user_id,
auth_events,
)
# TODO(paul): There's an awful lot of int()-casting in this code;
# surely we should be squashing strings to integers at a higher
# level, maybe when we insert?
if user_level is not None:
user_level = int(user_level)
user_level = self._get_user_power_level(event.user_id, auth_events)
# FIXME (erikj): What should we do here as the default?
ban_level = self._get_named_level(auth_events, "ban", 50)
@ -281,23 +271,26 @@ class Auth(object):
key = (EventTypes.PowerLevels, "", )
return auth_events.get(key)
def _get_power_level_from_event_state(self, event, user_id, auth_events):
def _get_user_power_level(self, user_id, auth_events):
power_level_event = self._get_power_level_event(auth_events)
level = None
if power_level_event:
level = power_level_event.content.get("users", {}).get(user_id)
if not level:
level = power_level_event.content.get("users_default", 0)
if level is None:
return 0
else:
return int(level)
else:
key = (EventTypes.Create, "", )
create_event = auth_events.get(key)
if (create_event is not None and
create_event.content["creator"] == user_id):
return 100
return level
else:
return 0
def _get_named_level(self, auth_events, name, default):
power_level_event = self._get_power_level_event(auth_events)
@ -504,16 +497,7 @@ class Auth(object):
else:
send_level = 0
user_level = self._get_power_level_from_event_state(
event,
event.user_id,
auth_events,
)
if user_level:
user_level = int(user_level)
else:
user_level = 0
user_level = self._get_user_power_level(event.user_id, auth_events)
if user_level < send_level:
raise AuthError(
@ -545,11 +529,7 @@ class Auth(object):
return True
def _check_redaction(self, event, auth_events):
user_level = self._get_power_level_from_event_state(
event,
event.user_id,
auth_events,
)
user_level = self._get_user_power_level(event.user_id, auth_events)
redact_level = self._get_named_level(auth_events, "redact", 50)
@ -579,11 +559,7 @@ class Auth(object):
if not current_state:
return
user_level = self._get_power_level_from_event_state(
event,
event.user_id,
auth_events,
)
user_level = self._get_user_power_level(event.user_id, auth_events)
# Check other levels:
levels_to_check = [

4
synapse/api/errors.py
View File

@ -35,8 +35,8 @@ class Codes(object):
LIMIT_EXCEEDED = "M_LIMIT_EXCEEDED"
CAPTCHA_NEEDED = "M_CAPTCHA_NEEDED"
CAPTCHA_INVALID = "M_CAPTCHA_INVALID"
MISSING_PARAM = "M_MISSING_PARAM",
TOO_LARGE = "M_TOO_LARGE",
MISSING_PARAM = "M_MISSING_PARAM"
TOO_LARGE = "M_TOO_LARGE"
EXCLUSIVE = "M_EXCLUSIVE"

1
synapse/config/registration.py
View File

@ -53,6 +53,7 @@ class RegistrationConfig(Config):
@classmethod
def generate_config(cls, args, config_dir_path):
super(RegistrationConfig, cls).generate_config(args, config_dir_path)
if args.enable_registration is None:
args.enable_registration = False

20
synapse/crypto/keyring.py
View File

@ -24,6 +24,8 @@ from synapse.api.errors import SynapseError, Codes
from synapse.util.retryutils import get_retry_limiter
from synapse.util.async import create_observer
from OpenSSL import crypto
import logging
@ -38,6 +40,8 @@ class Keyring(object):
self.clock = hs.get_clock()
self.hs = hs
self.key_downloads = {}
@defer.inlineCallbacks
def verify_json_for_server(self, server_name, json_object):
logger.debug("Verifying for %s", server_name)
@ -97,6 +101,22 @@ class Keyring(object):
defer.returnValue(cached[0])
return
download = self.key_downloads.get(server_name)
if download is None:
download = self._get_server_verify_key_impl(server_name, key_ids)
self.key_downloads[server_name] = download
@download.addBoth
def callback(ret):
del self.key_downloads[server_name]
return ret
r = yield create_observer(download)
defer.returnValue(r)
@defer.inlineCallbacks
def _get_server_verify_key_impl(self, server_name, key_ids):
# Try to fetch the key from the remote server.
limiter = yield get_retry_limiter(

53
synapse/handlers/presence.py
View File

@ -36,6 +36,9 @@ metrics = synapse.metrics.get_metrics_for(__name__)
# Don't bother bumping "last active" time if it differs by less than 60 seconds
LAST_ACTIVE_GRANULARITY = 60*1000
# Keep no more than this number of offline serial revisions
MAX_OFFLINE_SERIALS = 1000
# TODO(paul): Maybe there's one of these I can steal from somewhere
def partition(l, func):
@ -135,6 +138,9 @@ class PresenceHandler(BaseHandler):
self._remote_sendmap = {}
# map remote users to sets of local users who're interested in them
self._remote_recvmap = {}
# list of (serial, set of(userids)) tuples, ordered by serial, latest
# first
self._remote_offline_serials = []
# map any user to a UserPresenceCache
self._user_cachemap = {}
@ -714,8 +720,24 @@ class PresenceHandler(BaseHandler):
statuscache=statuscache,
)
user_id = user.to_string()
if state["presence"] == PresenceState.OFFLINE:
self._remote_offline_serials.insert(
0,
(self._user_cachemap_latest_serial, set([user_id]))
)
while len(self._remote_offline_serials) > MAX_OFFLINE_SERIALS:
self._remote_offline_serials.pop() # remove the oldest
del self._user_cachemap[user]
else:
# Remove the user from remote_offline_serials now that they're
# no longer offline
for idx, elem in enumerate(self._remote_offline_serials):
(_, user_ids) = elem
user_ids.discard(user_id)
if not user_ids:
self._remote_offline_serials.pop(idx)
for poll in content.get("poll", []):
user = UserID.from_string(poll)
@ -836,6 +858,8 @@ class PresenceEventSource(object):
presence = self.hs.get_handlers().presence_handler
cachemap = presence._user_cachemap
clock = self.clock
latest_serial = None
updates = []
# TODO(paul): use a DeferredList ? How to limit concurrency.
@ -845,18 +869,31 @@ class PresenceEventSource(object):
if cached.serial <= from_key:
continue
if (yield self.is_visible(observer_user, observed_user)):
updates.append((observed_user, cached))
if not (yield self.is_visible(observer_user, observed_user)):
continue
if latest_serial is None or cached.serial > latest_serial:
latest_serial = cached.serial
updates.append(cached.make_event(user=observed_user, clock=clock))
# TODO(paul): limit
for serial, user_ids in presence._remote_offline_serials:
if serial < from_key:
break
for u in user_ids:
updates.append({
"type": "m.presence",
"content": {"user_id": u, "presence": PresenceState.OFFLINE},
})
# TODO(paul): For the v2 API we want to tell the client their from_key
# is too old if we fell off the end of the _remote_offline_serials
# list, and get them to invalidate+resync. In v1 we have no such
# concept so this is a best-effort result.
if updates:
clock = self.clock
latest_serial = max([x[1].serial for x in updates])
data = [x[1].make_event(user=x[0], clock=clock) for x in updates]
defer.returnValue((data, latest_serial))
defer.returnValue((updates, latest_serial))
else:
defer.returnValue(([], presence._user_cachemap_latest_serial))

56
synapse/push/baserules.py
View File

@ -1,3 +1,17 @@
# Copyright 2015 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from synapse.push.rulekinds import PRIORITY_CLASS_MAP, PRIORITY_CLASS_INVERSE_MAP
@ -112,7 +126,25 @@ def make_base_prepend_override_rules():
def make_base_append_override_rules():
return [
{
'rule_id': 'global/override/.m.rule.call',
'rule_id': 'global/override/.m.rule.suppress_notices',
'conditions': [
{
'kind': 'event_match',
'key': 'content.msgtype',
'pattern': 'm.notice',
}
],
'actions': [
'dont_notify',
]
}
]
def make_base_append_underride_rules(user):
return [
{
'rule_id': 'global/underride/.m.rule.call',
'conditions': [
{
'kind': 'event_match',
@ -131,19 +163,6 @@ def make_base_append_override_rules():
}
]
},
{
'rule_id': 'global/override/.m.rule.suppress_notices',
'conditions': [
{
'kind': 'event_match',
'key': 'content.msgtype',
'pattern': 'm.notice',
}
],
'actions': [
'dont_notify',
]
},
{
'rule_id': 'global/override/.m.rule.contains_display_name',
'conditions': [
@ -162,7 +181,7 @@ def make_base_append_override_rules():
]
},
{
'rule_id': 'global/override/.m.rule.room_one_to_one',
'rule_id': 'global/underride/.m.rule.room_one_to_one',
'conditions': [
{
'kind': 'room_member_count',
@ -179,12 +198,7 @@ def make_base_append_override_rules():
'value': False
}
]
}
]
def make_base_append_underride_rules(user):
return [
},
{
'rule_id': 'global/underride/.m.rule.invite_for_me',
'conditions': [

14
synapse/push/rulekinds.py
View File

@ -1,3 +1,17 @@
# Copyright 2015 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
PRIORITY_CLASS_MAP = {
'underride': 1,
'sender': 2,

14
synapse/python_dependencies.py
View File

@ -1,3 +1,17 @@
# Copyright 2015 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
from distutils.version import LooseVersion

4
synapse/rest/media/v1/base_resource.py
View File

@ -25,6 +25,8 @@ from twisted.internet import defer
from twisted.web.resource import Resource
from twisted.protocols.basic import FileSender
from synapse.util.async import create_observer
import os
import logging
@ -87,7 +89,7 @@ class BaseMediaResource(Resource):
def callback(media_info):
del self.downloads[key]
return media_info
return download
return create_observer(download)
@defer.inlineCallbacks
def _get_remote_media_impl(self, server_name, media_id):

14
synapse/rest/media/v1/identicon_resource.py
View File

@ -1,3 +1,17 @@
# Copyright 2015 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from pydenticon import Generator
from twisted.web.resource import Resource

14
synapse/storage/schema/delta/14/upgrade_appservice_db.py
View File

@ -1,3 +1,17 @@
# Copyright 2015 OpenMarket Ltd
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import logging

14
synapse/storage/schema/delta/14/v14.sql
View File

@ -1,3 +1,17 @@
/* Copyright 2015 OpenMarket Ltd
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
CREATE TABLE IF NOT EXISTS push_rules_enable (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_name TEXT NOT NULL,

19
synapse/util/async.py
View File

@ -32,3 +32,22 @@ def run_on_reactor():
iteration of the main loop
"""
return sleep(0)
def create_observer(deferred):
"""Creates a deferred that observes the result or failure of the given
deferred *without* affecting the given deferred.
"""
d = defer.Deferred()
def callback(r):
d.callback(r)
return r
def errback(f):
d.errback(f)
return f
deferred.addCallbacks(callback, errback)
return d

65
tests/handlers/test_presence.py
View File

@ -878,6 +878,71 @@ class PresencePushTestCase(MockedDatastorePresenceTestCase):
state
)
@defer.inlineCallbacks
def test_recv_remote_offline(self):
""" Various tests relating to SYN-261 """
potato_set = self.handler._remote_recvmap.setdefault(self.u_potato,
set())
potato_set.add(self.u_apple)
self.room_members = [self.u_banana, self.u_potato]
self.assertEquals(self.event_source.get_current_key(), 0)
yield self.mock_federation_resource.trigger("PUT",
"/_matrix/federation/v1/send/1000000/",
_make_edu_json("elsewhere", "m.presence",
content={
"push": [
{"user_id": "@potato:remote",
"presence": "offline"},
],
}
)
)
self.assertEquals(self.event_source.get_current_key(), 1)
(events, _) = yield self.event_source.get_new_events_for_user(
self.u_apple, 0, None
)
self.assertEquals(events,
[
{"type": "m.presence",
"content": {
"user_id": "@potato:remote",
"presence": OFFLINE,
}}
]
)
yield self.mock_federation_resource.trigger("PUT",
"/_matrix/federation/v1/send/1000001/",
_make_edu_json("elsewhere", "m.presence",
content={
"push": [
{"user_id": "@potato:remote",
"presence": "online"},
],
}
)
)
self.assertEquals(self.event_source.get_current_key(), 2)
(events, _) = yield self.event_source.get_new_events_for_user(
self.u_apple, 0, None
)
self.assertEquals(events,
[
{"type": "m.presence",
"content": {
"user_id": "@potato:remote",
"presence": ONLINE,
}}
]
)
@defer.inlineCallbacks
def test_join_room_local(self):
self.room_members = [self.u_apple, self.u_banana]