Merge remote-tracking branch 'origin/develop' into room-initial-sync

This commit is contained in:
Paul "LeoNerd" Evans 2014-11-18 15:48:30 +00:00
commit e9f587ecba
8 changed files with 209 additions and 167 deletions

View File

@ -4,9 +4,9 @@ Introduction
Matrix is an ambitious new ecosystem for open federated Instant Messaging and Matrix is an ambitious new ecosystem for open federated Instant Messaging and
VoIP. The basics you need to know to get up and running are: VoIP. The basics you need to know to get up and running are:
- Chatrooms are distributed and do not exist on any single server. Rooms - Everything in Matrix happens in a room. Rooms are distributed and do not
can be found using aliases like ``#matrix:matrix.org`` or exist on any single server. Rooms can be located using convenience aliases
``#test:localhost:8008`` or they can be ephemeral. like ``#matrix:matrix.org`` or ``#test:localhost:8008``.
- Matrix user IDs look like ``@matthew:matrix.org`` (although in the future - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
you will normally refer to yourself and others using a 3PID: email you will normally refer to yourself and others using a 3PID: email
@ -17,56 +17,12 @@ The overall architecture is::
client <----> homeserver <=====================> homeserver <----> client client <----> homeserver <=====================> homeserver <----> client
https://somewhere.org/_matrix https://elsewhere.net/_matrix https://somewhere.org/_matrix https://elsewhere.net/_matrix
WARNING ``#matrix:matrix.org`` is the official support room for Matrix, and can be
======= accessed by the web client at http://matrix.org/alpha or via an IRC bridge at
irc://irc.freenode.net/matrix.
**Synapse is currently in a state of rapid development, and not all features
are yet functional. Critically, some security features are still in
development, which means Synapse can *not* be considered secure or reliable at
this point.** For instance:
- **SSL Certificates used by server-server federation are not yet validated.**
- **Room permissions are not yet enforced on traffic received via federation.**
- **Homeservers do not yet cryptographically sign their events to avoid
tampering**
- Default configuration provides open signup to the service from the internet
Despite this, we believe Synapse is more than useful as a way for experimenting
and exploring Synapse, and the missing features will land shortly. **Until
then, please do *NOT* use Synapse for any remotely important or secure
communication.**
Quick Start
===========
System requirements:
- POSIX-compliant system (tested on Linux & OSX)
- Python 2.7
To get up and running:
- To simply play with an **existing** homeserver you can
just go straight to http://matrix.org/alpha.
- To run your own **private** homeserver on localhost:8008, generate a basic
config file: ``./synctl start`` will give you instructions on how to do this.
For this purpose, you can use 'localhost' or your hostname as a server name.
Once you've done so, running ``./synctl start`` again will start your private
home server. You will find a webclient running at http://localhost:8008.
Please use a recent Chrome or Firefox for now (or Safari if you don't need
VoIP support).
- To run a **public** homeserver and let it exchange messages with other
homeservers and participate in the global Matrix federation, you must expose
port 8448 to the internet and edit homeserver.yaml to specify server_name
(the public DNS entry for this server) and then run ``synctl start``. If you
changed the server_name, you may need to move the old database
(homeserver.db) out of the way first. Then come join ``#matrix:matrix.org``
and say hi! :)
For more detailed setup instructions, please see further down this document.
Synapse is currently in rapid development, but as of version 0.5 we believe it
is sufficiently stable to be run as an internet-facing service for real usage!
About Matrix About Matrix
============ ============
@ -76,10 +32,10 @@ which handle:
- Creating and managing fully distributed chat rooms with no - Creating and managing fully distributed chat rooms with no
single points of control or failure single points of control or failure
- Eventually-consistent cryptographically secure[1] synchronisation of room - Eventually-consistent cryptographically secure synchronisation of room
state across a global open network of federated servers and services state across a global open network of federated servers and services
- Sending and receiving extensible messages in a room with (optional) - Sending and receiving extensible messages in a room with (optional)
end-to-end encryption[2] end-to-end encryption[1]
- Inviting, joining, leaving, kicking, banning room members - Inviting, joining, leaving, kicking, banning room members
- Managing user accounts (registration, login, logout) - Managing user accounts (registration, login, logout)
- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
@ -111,25 +67,31 @@ Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
web client demo implemented in AngularJS) and cmdclient (a basic Python web client demo implemented in AngularJS) and cmdclient (a basic Python
command line utility which lets you easily see what the JSON APIs are up to). command line utility which lets you easily see what the JSON APIs are up to).
We'd like to invite you to take a look at the Matrix spec, try to run a Meanwhile, iOS and Android SDKs and clients are currently in development and available from:
homeserver, and join the existing Matrix chatrooms already out there,
experiment with the APIs and the demo clients, and let us know your thoughts at
https://github.com/matrix-org/synapse/issues or at matrix@matrix.org.
Thanks for trying Matrix! * https://github.com/matrix-org/matrix-ios-sdk
* https://github.com/matrix-org/matrix-android-sdk
[1] Cryptographic signing of messages isn't turned on yet We'd like to invite you to join #matrix:matrix.org (via http://matrix.org/alpha), run a homeserver, take a look at the Matrix spec at
http://matrix.org/docs/spec, experiment with the APIs and the demo
clients, and report any bugs via http://matrix.org/jira.
[2] End-to-end encryption is currently in development Thanks for using Matrix!
[1] End-to-end encryption is currently in development
Homeserver Installation Homeserver Installation
======================= =======================
System requirements:
- POSIX-compliant system (tested on Linux & OSX)
- Python 2.7
Synapse is written in python but some of the libraries is uses are written in Synapse is written in python but some of the libraries is uses are written in
C. So before we can install synapse itself we need a working C compiler and the C. So before we can install synapse itself we need a working C compiler and the
header files for python C extensions. header files for python C extensions.
Installing prerequisites on Ubuntu:: Installing prerequisites on Ubuntu or Debian::
$ sudo apt-get install build-essential python2.7-dev libffi-dev \ $ sudo apt-get install build-essential python2.7-dev libffi-dev \
python-pip python-setuptools python-pip python-setuptools
@ -137,8 +99,62 @@ Installing prerequisites on Ubuntu::
Installing prerequisites on Mac OS X:: Installing prerequisites on Mac OS X::
$ xcode-select --install $ xcode-select --install
To install the synapse homeserver run::
Synapse uses NaCl (http://nacl.cr.yp.to/) for encryption and digital signatures. $ pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
This installs synapse, along with the libraries it uses, into
``$HOME/.local/lib/`` on Linux or ``$HOME/Library/Python/2.7/lib/`` on OSX.
Troubleshooting Installation
----------------------------
Synapse requires pip 1.7 or later, so if your OS provides too old a version and
you get errors about ``error: no such option: --process-dependency-links`` you
may need to manually upgrade it::
$ sudo pip install --upgrade pip
If pip crashes mid-installation for reason (e.g. lost terminal), pip may
refuse to run until you remove the temporary installation directory it
created. To reset the installation::
$ rm -rf /tmp/pip_install_matrix
pip seems to leak *lots* of memory during installation. For instance, a Linux
host with 512MB of RAM may run out of memory whilst installing Twisted. If this
happens, you will have to individually install the dependencies which are
failing, e.g.:
$ pip install --user twisted
Running Your Homeserver
=======================
To actually run your new homeserver, pick a working directory for Synapse to run
(e.g. ``~/.synapse``), and::
$ mkdir ~/.synapse
$ cd ~/.synapse
$ # on Linux
$ ~/.local/bin/synctl start
$ # on OSX
$ ~/Library/Python/2.7/bin/synctl start
Troubleshooting Running
-----------------------
If ``synctl`` fails with ``pkg_resources.DistributionNotFound`` errors you may
need a newer version of setuptools than that provided by your OS.
$ sudo pip install setuptools --upgrade
If synapse fails with ``missing "sodium.h"`` crypto errors, you may need
to manually upgrade PyNaCL, as synapse uses NaCl (http://nacl.cr.yp.to/) for
encryption and digital signatures.
Unfortunately PyNACL currently has a few issues Unfortunately PyNACL currently has a few issues
(https://github.com/pyca/pynacl/issues/53) and (https://github.com/pyca/pynacl/issues/53) and
(https://github.com/pyca/pynacl/issues/79) that mean it may not install (https://github.com/pyca/pynacl/issues/79) that mean it may not install
@ -151,21 +167,6 @@ fix try re-installing from PyPI or directly from
$ # Install from github $ # Install from github
$ pip install --user https://github.com/pyca/pynacl/tarball/master $ pip install --user https://github.com/pyca/pynacl/tarball/master
On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'``
you will need to ``export CFLAGS=-Qunused-arguments``.
To install the synapse homeserver run::
$ pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
This installs synapse, along with the libraries it uses, into
``$HOME/.local/lib/``.
To actually run your new homeserver, pick a working directory for Synapse to run (e.g. ``~/.synapse``), and::
$ mkdir ~/.synapse
$ cd ~/.synapse
$ synctl start
Homeserver Development Homeserver Development
====================== ======================
@ -200,8 +201,8 @@ This should end with a 'PASSED' result::
Upgrading an existing homeserver Upgrading an existing homeserver
================================ ================================
Before upgrading an existing homeserver to a new version, please refer to IMPORTANT: Before upgrading an existing homeserver to a new version, please
UPGRADE.rst for any additional instructions. refer to UPGRADE.rst for any additional instructions.
Setting up Federation Setting up Federation
@ -230,10 +231,7 @@ For the first form, simply pass the required hostname (of the machine) as the
--generate-config --generate-config
$ python -m synapse.app.homeserver --config-path homeserver.config $ python -m synapse.app.homeserver --config-path homeserver.config
Alternatively, you can run synapse via synctl - running ``synctl start`` to Alternatively, you can run ``synctl start`` to guide you through the process.
generate a homeserver.yaml config file, where you can then edit server-name to
specify machine.my.domain.name, and then set the actual server running again
with synctl start.
For the second form, first create your SRV record and publish it in DNS. This For the second form, first create your SRV record and publish it in DNS. This
needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@ -270,6 +268,8 @@ private federation (``localhost:8080``, ``localhost:8081`` and
http://localhost:8080. Simply run:: http://localhost:8080. Simply run::
$ demo/start.sh $ demo/start.sh
This is mainly useful just for development purposes.
Running The Demo Web Client Running The Demo Web Client
=========================== ===========================
@ -328,13 +328,14 @@ time.
Where's the spec?! Where's the spec?!
================== ==================
For now, please go spelunking in the ``docs/`` directory to find out. The source of the matrix spec lives at https://github.com/matrix-org/matrix-doc.
A recent HTML snapshot of this lives at http://matrix.org/docs/spec
Building Internal API Documentation Building Internal API Documentation
=================================== ===================================
Before building internal API documentation install spinx and Before building internal API documentation install sphinx and
sphinxcontrib-napoleon:: sphinxcontrib-napoleon::
$ pip install sphinx $ pip install sphinx

View File

@ -21,7 +21,7 @@ from synapse.api.constants import Membership, JoinRules
from synapse.api.errors import AuthError, StoreError, Codes, SynapseError from synapse.api.errors import AuthError, StoreError, Codes, SynapseError
from synapse.api.events.room import ( from synapse.api.events.room import (
RoomMemberEvent, RoomPowerLevelsEvent, RoomRedactionEvent, RoomMemberEvent, RoomPowerLevelsEvent, RoomRedactionEvent,
RoomJoinRulesEvent, RoomCreateEvent, RoomJoinRulesEvent, RoomCreateEvent, RoomAliasesEvent,
) )
from synapse.util.logutils import log_function from synapse.util.logutils import log_function
from syutil.base64util import encode_base64 from syutil.base64util import encode_base64
@ -63,6 +63,10 @@ class Auth(object):
# FIXME # FIXME
return True return True
# FIXME: Temp hack
if event.type == RoomAliasesEvent.TYPE:
return True
if event.type == RoomMemberEvent.TYPE: if event.type == RoomMemberEvent.TYPE:
allowed = self.is_membership_change_allowed(event) allowed = self.is_membership_change_allowed(event)
if allowed: if allowed:
@ -144,6 +148,17 @@ class Auth(object):
@log_function @log_function
def is_membership_change_allowed(self, event): def is_membership_change_allowed(self, event):
membership = event.content["membership"]
# Check if this is the room creator joining:
if len(event.prev_events) == 1 and Membership.JOIN == membership:
# Get room creation event:
key = (RoomCreateEvent.TYPE, "", )
create = event.old_state_events.get(key)
if event.prev_events[0][0] == create.event_id:
if create.content["creator"] == event.state_key:
return True
target_user_id = event.state_key target_user_id = event.state_key
# get info about the caller # get info about the caller
@ -159,8 +174,6 @@ class Auth(object):
target_in_room = target and target.membership == Membership.JOIN target_in_room = target and target.membership == Membership.JOIN
membership = event.content["membership"]
key = (RoomJoinRulesEvent.TYPE, "", ) key = (RoomJoinRulesEvent.TYPE, "", )
join_rule_event = event.old_state_events.get(key) join_rule_event = event.old_state_events.get(key)
if join_rule_event: if join_rule_event:
@ -255,6 +268,12 @@ class Auth(object):
level = power_level_event.content.get("users", {}).get(user_id) level = power_level_event.content.get("users", {}).get(user_id)
if not level: if not level:
level = power_level_event.content.get("users_default", 0) level = power_level_event.content.get("users_default", 0)
else:
key = (RoomCreateEvent.TYPE, "", )
create_event = event.old_state_events.get(key)
if (create_event is not None and
create_event.content["creator"] == user_id):
return 100
return level return level

View File

@ -184,15 +184,7 @@ class SynapseHomeServer(HomeServer):
logger.info("Synapse now listening on port %d", unsecure_port) logger.info("Synapse now listening on port %d", unsecure_port)
def setup(): def setup(config, run_http=True):
config = HomeServerConfig.load_config(
"Synapse Homeserver",
sys.argv[1:],
generate_section="Homeserver"
)
config.setup_logging()
logger.info("Server hostname: %s", config.server_name) logger.info("Server hostname: %s", config.server_name)
if re.search(":[0-9]+$", config.server_name): if re.search(":[0-9]+$", config.server_name):
@ -212,12 +204,13 @@ def setup():
content_addr=config.content_addr, content_addr=config.content_addr,
) )
hs.register_servlets() if run_http:
hs.register_servlets()
hs.create_resource_tree( hs.create_resource_tree(
web_client=config.webclient, web_client=config.webclient,
redirect_root_to_web_client=True, redirect_root_to_web_client=True,
) )
db_name = hs.get_db_name() db_name = hs.get_db_name()
@ -237,11 +230,18 @@ def setup():
f.namespace['hs'] = hs f.namespace['hs'] = hs
reactor.listenTCP(config.manhole, f, interface='127.0.0.1') reactor.listenTCP(config.manhole, f, interface='127.0.0.1')
bind_port = config.bind_port if run_http:
if config.no_tls: bind_port = config.bind_port
bind_port = None if config.no_tls:
hs.start_listening(bind_port, config.unsecure_port) bind_port = None
hs.start_listening(bind_port, config.unsecure_port)
hs.config = config
return hs
def run(config):
if config.daemonize: if config.daemonize:
print config.pid_file print config.pid_file
daemon = Daemonize( daemon = Daemonize(
@ -257,13 +257,26 @@ def setup():
else: else:
reactor.run() reactor.run()
def run():
with LoggingContext("run"):
reactor.run()
def main(): def main(args, run_http=True):
with LoggingContext("main"): with LoggingContext("main"):
setup() config = HomeServerConfig.load_config(
"Synapse Homeserver",
args,
generate_section="Homeserver"
)
config.setup_logging()
hs = setup(config, run_http=run_http)
def r():
run(config)
hs.run = r
return hs
if __name__ == '__main__': if __name__ == '__main__':
main() hs = main(sys.argv[1:])
hs.run()

View File

@ -56,8 +56,8 @@ def main():
elif action == "stop": elif action == "stop":
stop() stop()
elif action == "restart": elif action == "restart":
start()
stop() stop()
start()
else: else:
sys.stderr.write("Usage: %s [start|stop|restart]\n" % (sys.argv[0],)) sys.stderr.write("Usage: %s [start|stop|restart]\n" % (sys.argv[0],))
sys.exit(1) sys.exit(1)

View File

@ -21,8 +21,6 @@ from synapse.api.errors import SynapseError
from synapse.api.events.room import RoomAliasesEvent from synapse.api.events.room import RoomAliasesEvent
import logging import logging
import sqlite3
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -56,17 +54,11 @@ class DirectoryHandler(BaseHandler):
if not servers: if not servers:
raise SynapseError(400, "Failed to get server list") raise SynapseError(400, "Failed to get server list")
try: yield self.store.create_room_alias_association(
yield self.store.create_room_alias_association( room_alias,
room_alias, room_id,
room_id, servers
servers )
)
except sqlite3.IntegrityError:
defer.returnValue("Already exists")
# TODO: Send the room event.
yield self._update_room_alias_events(user_id, room_id)
@defer.inlineCallbacks @defer.inlineCallbacks
def delete_association(self, user_id, room_alias): def delete_association(self, user_id, room_alias):
@ -136,7 +128,7 @@ class DirectoryHandler(BaseHandler):
}) })
@defer.inlineCallbacks @defer.inlineCallbacks
def _update_room_alias_events(self, user_id, room_id): def send_room_alias_update_event(self, user_id, room_id):
aliases = yield self.store.get_aliases_for_room(room_id) aliases = yield self.store.get_aliases_for_room(room_id)
event = self.event_factory.create_event( event = self.event_factory.create_event(

View File

@ -106,20 +106,37 @@ class RoomCreationHandler(BaseHandler):
if not room_id: if not room_id:
raise StoreError(500, "Couldn't generate a room ID.") raise StoreError(500, "Couldn't generate a room ID.")
if room_alias:
directory_handler = self.hs.get_handlers().directory_handler
yield directory_handler.create_association(
user_id=user_id,
room_id=room_id,
room_alias=room_alias,
servers=[self.hs.hostname],
)
user = self.hs.parse_userid(user_id) user = self.hs.parse_userid(user_id)
creation_events = self._create_events_for_new_room( creation_events = self._create_events_for_new_room(
user, room_id, is_public=is_public user, room_id, is_public=is_public
) )
room_member_handler = self.hs.get_handlers().room_member_handler
@defer.inlineCallbacks @defer.inlineCallbacks
def handle_event(event): def handle_event(event):
snapshot = yield self.store.snapshot_room(event) snapshot = yield self.store.snapshot_room(event)
logger.debug("Event: %s", event) logger.debug("Event: %s", event)
yield self._on_new_room_event( if event.type == RoomMemberEvent.TYPE:
event, snapshot, extra_users=[user], suppress_auth=True yield room_member_handler.change_membership(
) event,
do_auth=True
)
else:
yield self._on_new_room_event(
event, snapshot, extra_users=[user], suppress_auth=True
)
for event in creation_events: for event in creation_events:
yield handle_event(event) yield handle_event(event)
@ -146,16 +163,6 @@ class RoomCreationHandler(BaseHandler):
yield handle_event(topic_event) yield handle_event(topic_event)
content = {"membership": Membership.JOIN}
join_event = self.event_factory.create_event(
etype=RoomMemberEvent.TYPE,
state_key=user_id,
room_id=room_id,
user_id=user_id,
membership=Membership.JOIN,
content=content
)
content = {"membership": Membership.INVITE} content = {"membership": Membership.INVITE}
for invitee in invite_list: for invitee in invite_list:
invite_event = self.event_factory.create_event( invite_event = self.event_factory.create_event(
@ -165,35 +172,22 @@ class RoomCreationHandler(BaseHandler):
user_id=user_id, user_id=user_id,
content=content content=content
) )
yield handle_event(invite_event)
yield self.hs.get_handlers().room_member_handler.change_membership(
invite_event,
do_auth=False
)
yield self.hs.get_handlers().room_member_handler.change_membership(
join_event,
do_auth=False
)
result = {"room_id": room_id} result = {"room_id": room_id}
if room_alias: if room_alias:
result["room_alias"] = room_alias.to_string() result["room_alias"] = room_alias.to_string()
directory_handler = self.hs.get_handlers().directory_handler directory_handler.send_room_alias_update_event(user_id, room_id)
yield directory_handler.create_association(
user_id=user_id,
room_id=room_id,
room_alias=room_alias,
servers=[self.hs.hostname],
)
defer.returnValue(result) defer.returnValue(result)
def _create_events_for_new_room(self, creator, room_id, is_public=False): def _create_events_for_new_room(self, creator, room_id, is_public=False):
creator_id = creator.to_string()
event_keys = { event_keys = {
"room_id": room_id, "room_id": room_id,
"user_id": creator.to_string(), "user_id": creator_id,
} }
def create(etype, **content): def create(etype, **content):
@ -208,6 +202,16 @@ class RoomCreationHandler(BaseHandler):
creator=creator.to_string(), creator=creator.to_string(),
) )
join_event = self.event_factory.create_event(
etype=RoomMemberEvent.TYPE,
state_key=creator_id,
content={
"membership": Membership.JOIN,
},
**event_keys
)
power_levels_event = self.event_factory.create_event( power_levels_event = self.event_factory.create_event(
etype=RoomPowerLevelsEvent.TYPE, etype=RoomPowerLevelsEvent.TYPE,
content={ content={
@ -236,6 +240,7 @@ class RoomCreationHandler(BaseHandler):
return [ return [
creation_event, creation_event,
join_event,
power_levels_event, power_levels_event,
join_rules_event, join_rules_event,
] ]

View File

@ -70,9 +70,11 @@ class ClientDirectoryServer(RestServlet):
dir_handler = self.handlers.directory_handler dir_handler = self.handlers.directory_handler
try: try:
user_id = user.to_string()
yield dir_handler.create_association( yield dir_handler.create_association(
user.to_string(), room_alias, room_id, servers user_id, room_alias, room_id, servers
) )
yield dir_handler.send_room_alias_update_event(user_id, room_id)
except SynapseError as e: except SynapseError as e:
raise e raise e
except: except:

View File

@ -14,10 +14,15 @@
# limitations under the License. # limitations under the License.
from ._base import SQLBaseStore from ._base import SQLBaseStore
from synapse.api.errors import SynapseError
from twisted.internet import defer from twisted.internet import defer
from collections import namedtuple from collections import namedtuple
import sqlite3
RoomAliasMapping = namedtuple( RoomAliasMapping = namedtuple(
"RoomAliasMapping", "RoomAliasMapping",
@ -75,13 +80,18 @@ class DirectoryStore(SQLBaseStore):
Returns: Returns:
Deferred Deferred
""" """
yield self._simple_insert( try:
"room_aliases", yield self._simple_insert(
{ "room_aliases",
"room_alias": room_alias.to_string(), {
"room_id": room_id, "room_alias": room_alias.to_string(),
}, "room_id": room_id,
) },
)
except sqlite3.IntegrityError:
raise SynapseError(
409, "Room alias %s already exists" % room_alias.to_string()
)
for server in servers: for server in servers:
# TODO(erikj): Fix this to bulk insert # TODO(erikj): Fix this to bulk insert