Merge branch develop into server2server_signing

Conflicts:
	synapse/app/homeserver.py

docs/client-server/howto.rst

@@ -31,7 +31,7 @@ Registration
 The aim of registration is to get a user ID and access token which you will need
 when accessing other APIs::
 
-    curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/register"
+    curl -XPOST -d '{"user":"example", "password":"wordpass", "type":"m.login.password"}' "http://localhost:8008/_matrix/client/api/v1/register"
 
     {
         "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", 
@@ -39,14 +39,15 @@ when accessing other APIs::
         "user_id": "@example:localhost"
     }
 
-NB: If a ``user_id`` is not specified, one will be randomly generated for you. 
+NB: If a ``user`` is not specified, one will be randomly generated for you. 
 If you do not specify a ``password``, you will be unable to login to the account
 if you forget the ``access_token``.
 
 Implementation note: The matrix specification does not enforce how users 
 register with a server. It just specifies the URL path and absolute minimum 
 keys. The reference home server uses a username/password to authenticate user,
-but other home servers may use different methods.
+but other home servers may use different methods. This is why you need to
+specify the ``type`` of method.
 
 Login
 -----

docs/client-server/model/presence.rst

@@ -1,103 +1,3 @@
-========
-Presence
-========
-
-A description of presence information and visibility between users.
-
-Overview
-========
-
-Each user has the concept of Presence information. This encodes a sense of the
-"availability" of that user, suitable for display on other user's clients.
-
-
-Presence Information
-====================
-
-The basic piece of presence information is an enumeration of a small set of
-state; such as "free to chat", "online", "busy", or "offline". The default state
-unless the user changes it is "online". Lower states suggest some amount of
-decreased availability from normal, which might have some client-side effect
-like muting notification sounds and suggests to other users not to bother them
-unless it is urgent. Equally, the "free to chat" state exists to let the user
-announce their general willingness to receive messages moreso than default.
-
-Home servers should also allow a user to set their state as "hidden" - a state
-which behaves as offline, but allows the user to see the client state anyway and
-generally interact with client features such as reading message history or
-accessing contacts in the address book.
-
-This basic state field applies to the user as a whole, regardless of how many
-client devices they have connected. The home server should synchronise this
-status choice among multiple devices to ensure the user gets a consistent
-experience.
-
-Idle Time
----------
-
-As well as the basic state field, the presence information can also show a sense
-of an "idle timer". This should be maintained individually by the user's
-clients, and the homeserver can take the highest reported time as that to
-report. Likely this should be presented in fairly coarse granularity; possibly
-being limited to letting the home server automatically switch from a "free to
-chat" or "online" mode into "idle".
-
-When a user is offline, the Home Server can still report when the user was last
-seen online, again perhaps in a somewhat coarse manner.
-
-Device Type
------------
-
-Client devices that may limit the user experience somewhat (such as "mobile"
-devices with limited ability to type on a real keyboard or read large amounts of
-text) should report this to the home server, as this is also useful information
-to report as "presence" if the user cannot be expected to provide a good typed
-response to messages.
-
-
-Presence List
-=============
-
-Each user's home server stores a "presence list" for that user. This stores a
-list of other user IDs the user has chosen to add to it (remembering any ACL
-Pointer if appropriate).
-
-To be added to a contact list, the user being added must grant permission. Once
-granted, both user's HS(es) store this information, as it allows the user who
-has added the contact some more abilities; see below. Since such subscriptions
-are likely to be bidirectional, HSes may wish to automatically accept requests
-when a reverse subscription already exists.
-
-As a convenience, presence lists should support the ability to collect users
-into groups, which could allow things like inviting the entire group to a new
-("ad-hoc") chat room, or easy interaction with the profile information ACL
-implementation of the HS.
-
-
-Presence and Permissions
-========================
-
-For a viewing user to be allowed to see the presence information of a target
-user, either
-
- * The target user has allowed the viewing user to add them to their presence
-   list, or
-
- * The two users share at least one room in common
-
-In the latter case, this allows for clients to display some minimal sense of
-presence information in a user list for a room.
-
-Home servers can also use the user's choice of presence state as a signal for
-how to handle new private one-to-one chat message requests. For example, it
-might decide:
-
-  "free to chat": accept anything
-  "online": accept from anyone in my addres book list
-  "busy": accept from anyone in this "important people" group in my address
-    book list
-
-
 API Efficiency
 ==============
 

docs/client-server/swagger_matrix/api-docs-directory

@@ -48,6 +48,22 @@
               "paramType": "body"
             }
           ]
+        },
+        {
+          "method": "DELETE",
+          "summary": "Removes a mapping of room alias to room ID.",
+          "notes": "Only privileged users can perform this action.",
+          "type": "void",
+          "nickname": "remove_room_alias",
+          "parameters": [
+            {
+              "name": "roomAlias",
+              "description": "The room alias to remove.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
         }
       ]
     }

docs/definitions.rst

@@ -0,0 +1,53 @@
+Definitions
+===========
+
+# *Event* -- A JSON object that represents a piece of information to be
+distributed to the the room. The object includes a payload and metadata,
+including a `type` used to indicate what the payload is for and how to process
+them. It also includes one or more references to previous events.
+
+# *Event graph* -- Events and their references to previous events form a
+directed acyclic graph. All events must be a descendant of the first event in a
+room, except for a few special circumstances.
+
+# *State event* -- A state event is an event that has a non-null string valued
+`state_key` field. It may also include a `prev_state` key referencing exactly
+one state event with the same type and state key, in the same event graph.
+
+# *State tree* -- A state tree is a tree formed by a collection of state events
+that have the same type and state key (all in the same event graph.
+
+# *State resolution algorithm* -- An algorithm that takes a state tree as input
+and selects a single leaf node.
+
+# *Current state event* -- The leaf node of a given state tree that has been
+selected by the state resolution algorithm.
+
+# *Room state* / *state dictionary* / *current state* -- A mapping of the pair
+(event type, state key) to the current state event for that pair.
+
+# *Room* -- An event graph and its associated state dictionary. An event is in
+the room if it is part of the event graph.
+
+# *Topological ordering* -- The partial ordering that can be extracted from the
+event graph due to it being a DAG.
+
+(The state definitions are purposely slightly ill-defined, since if we allow
+deleting events we might end up with multiple state trees for a given event
+type and state key pair.)
+
+Federation specific
+-------------------
+# *(Persistent data unit) PDU* -- An encoding of an event for distribution of
+the server to server protocol.
+
+# *(Ephemeral data unit) EDU* -- A piece of information that is sent between
+servers and doesn't encode an event.
+
+Client specific
+---------------
+# *Child events* -- Events that reference a single event in the same room
+independently of the event graph.
+
+# *Collapsed events* -- Events that have all child events that reference it
+included in the JSON object.

docs/specification-NOTHAVE.rst

@@ -0,0 +1,30 @@
+Matrix Specification NOTHAVEs
+=============================
+
+This document contains sections of the main specification that have been
+temporarily removed, because they specify intentions or aspirations that have
+in no way yet been implemented. Rather than outright-deleting them, they have
+been moved here so as to stand as an initial version for such time as they
+become extant.
+
+
+Presence
+========
+
+Idle Time
+---------
+As well as the basic ``presence`` field, the presence information can also show
+a sense of an "idle timer". This should be maintained individually by the
+user's clients, and the home server can take the highest reported time as that
+to report. When a user is offline, the home server can still report when the
+user was last seen online.
+
+Device Type
+-----------
+
+Client devices that may limit the user experience somewhat (such as "mobile"
+devices with limited ability to type on a real keyboard or read large amounts of
+text) should report this to the home server, as this is also useful information
+to report as "presence" if the user cannot be expected to provide a good typed
+response to messages.
+

docs/state_resolution.rst

@@ -0,0 +1,51 @@
+State Resolution
+================
+This section describes why we need state resolution and how it works.
+
+
+Motivation
+-----------
+We want to be able to associate some shared state with rooms, e.g. a room name
+or members list. This is done by having a current state dictionary that maps
+from the pair event type and state key to an event.
+
+However, since the servers involved in the room are distributed we need to be
+able to handle the case when two (or more) servers try and update the state at
+the same time. This is done via the state resolution algorithm.
+
+
+State Tree
+------------
+State events contain a reference to the state it is trying to replace. These
+relations form a tree where the current state is one of the leaf nodes.
+
+Note that state events are events, and so are part of the PDU graph. Thus we
+can be sure that (modulo the internet being particularly broken) we will see
+all state events eventually.
+
+
+Algorithm requirements
+----------------------
+We want the algorithm to have the following properties:
+- Since we aren't guaranteed what order we receive state events in, except that
+  we see parents before children, the state resolution algorithm must not depend
+  on the order and must always come to the same result. 
+- If we receive a state event whose parent is the current state, then the
+  algorithm will select it.
+- The algorithm does not depend on internal state, ensuring all servers should
+  come to the same decision.
+
+These three properties mean it is enough to keep track of the current state and
+compare it with any new proposed state, rather than having to keep track of all
+the leafs of the tree and recomputing across the entire state tree.
+
+
+Current Implementation
+----------------------
+The current implementation works as follows: Upon receipt of a newly proposed
+state change we first find the common ancestor. Then we take the maximum
+across each branch of the users' power levels, if one is higher then it is
+selected as the current state. Otherwise, we check if one chain is longer than
+the other, if so we choose that one. If that also fails, then we concatenate
+all the pdu ids and take a SHA1 hash and compare them to select a common
+ancestor.