From 392dc8af59e333ad90bd598a6ab355a5e6bd14bf Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Tue, 30 Sep 2014 18:11:24 +0100 Subject: [PATCH 01/48] Annotate all the 'TODO' marks as relating to either the specification itself or the documentation thereof --- docs/specification.rst | 104 +++++++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 50 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 07c57f9dd..6dcdea306 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -118,7 +118,7 @@ the account and looks like:: The ``localpart`` of a user ID may be a user name, or an opaque ID identifying this user. They are case-insensitive. -.. TODO +.. TODO-spec - Need to specify precise grammar for Matrix IDs A "Home Server" is a server which provides C-S APIs and has the ability to @@ -366,7 +366,7 @@ events which are visible to the client will appear in the event stream. When the request returns, an ``end`` token is included in the response. This token can be used in the next request to continue where the client left off. -.. TODO +.. TODO-spec How do we filter the event stream? Do we ever return multiple events in a single request? Don't we get lots of request setup RTT latency if we only do one event per request? Do we ever support streaming @@ -704,7 +704,7 @@ Rooms Creation -------- .. TODO kegan - - TODO: Key for invite these users? + - TODO-spec: Key for invite these users? To create a room, a client has to use the |createRoom|_ API. There are various options which can be set when creating a room: @@ -799,7 +799,7 @@ Modifying aliases .. NOTE:: This section is a work in progress. -.. TODO kegan +.. TODO-doc kegan - path to edit aliases - PUT /directory/room/ { room_id : foo } - GET /directory/room/ { room_id : foo, servers: [a.com, b.com] } @@ -811,11 +811,11 @@ Permissions .. NOTE:: This section is a work in progress. -.. TODO kegan - - TODO: What is a power level? How do they work? Defaults / required levels for X. How do they change +.. TODO-doc kegan + - What is a power level? How do they work? Defaults / required levels for X. How do they change as people join and leave rooms? What do you do if you get a clash? Examples. - - TODO: List all actions which use power levels (sending msgs, inviting users, banning people, etc...) - - TODO: Room config - what is the event and what are the keys/values and explanations for them. + - List all actions which use power levels (sending msgs, inviting users, banning people, etc...) + - Room config - what is the event and what are the keys/values and explanations for them. Link through to respective sections where necessary. How does this tie in with permissions, e.g. give example of creating a read-only room. @@ -845,7 +845,7 @@ defined in the following state events: Joining rooms ------------- -.. TODO kegan +.. TODO-doc kegan - TODO: What does the home server have to do to join a user to a room? Users need to join a room in order to send and receive events in that room. A @@ -883,7 +883,7 @@ received an invite. Inviting users -------------- -.. TODO kegan +.. TODO-doc kegan - Can invite users to a room if the room config key TODO is set to TODO. Must have required power level. - Outline invite join dance. What is it? Why is it required? How does it work? - What does the home server have to do? @@ -921,7 +921,7 @@ See the `Room events`_ section for more information on ``m.room.member``. Leaving rooms ------------- -.. TODO kegan +.. TODO-spec kegan - TODO: Grace period before deletion? - TODO: Under what conditions should a room NOT be purged? @@ -1076,7 +1076,7 @@ presence events will also be returned. There are two APIs provided: - |/rooms//initialSync|_ : A sync scoped to a single room. Presents room and event information for this room only. -.. TODO kegan +.. TODO-doc kegan - TODO: JSON response format for both types - TODO: when would you use global? when would you use scoped? @@ -1098,7 +1098,7 @@ There are several APIs provided to ``GET`` events for a room: Response format: ``[ { state event }, { state event }, ... ]`` Example: - TODO + TODO-doc |/rooms//members|_ @@ -1107,7 +1107,7 @@ There are several APIs provided to ``GET`` events for a room: Response format: ``{ "start": "", "end": "", "chunk": [ { m.room.member event }, ... ] }`` Example: - TODO + TODO-doc |/rooms//messages|_ Description: @@ -1117,16 +1117,16 @@ There are several APIs provided to ``GET`` events for a room: Response format: ``{ "start": "", "end": "" }`` Example: - TODO + TODO-doc |/rooms//initialSync|_ Description: Get all relevant events for a room. This includes state events, paginated non-state events and presence events. Response format: - `` { TODO } `` + `` { TODO-doc } `` Example: - TODO + TODO-doc Redactions ---------- @@ -1153,7 +1153,7 @@ Room Events .. NOTE:: This section is a work in progress. -.. TODO dave? +.. TODO-doc dave? - voip events? This specification outlines several standard event types, all of which are @@ -1232,7 +1232,7 @@ prefixed with ``m.`` Example: ``{ "join_rule": "public" }`` Description: - TODO : Use docs/models/rooms.rst + TODO-doc : Use docs/models/rooms.rst ``m.room.power_levels`` Summary: @@ -1480,8 +1480,9 @@ the following: - ``offline`` : The user is not connected to an event stream. - ``free_for_chat`` : The user is generally willing to receive messages moreso than default. - - ``hidden`` : TODO. Behaves as offline, but allows the user to see the - client state anyway and generally interact with client features. + - ``hidden`` : Behaves as offline, but allows the user to see the client + state anyway and generally interact with client features. (Not yet + implemented in synapse). This basic ``presence`` field applies to the user as a whole, regardless of how many client devices they have connected. The home server should synchronise @@ -1509,7 +1510,7 @@ Transmission .. NOTE:: This section is a work in progress. -.. TODO: +.. TODO-doc: - Transmitted as an EDU. - Presence lists determine who to send to. @@ -1534,17 +1535,23 @@ user, either: In the latter case, this allows for clients to display some minimal sense of presence information in a user list for a room. + Typing notifications ==================== .. NOTE:: This section is a work in progress. -.. TODO Leo +.. TODO-doc Leo - what is the event type. Are they bundled with other event types? If so, which. - what are the valid keys / values. What do they represent. Any gotchas? - Timeouts. How do they work, who sets them and how do they expire. Does one have priority over another? Give examples. +.. TODO-spec Leo + - actually define the client-server API; the only thing that currently + exists is entirely server-server + + Voice over IP ============= Matrix can also be used to set up VoIP calls. This is part of the core @@ -1681,12 +1688,13 @@ a call and the other party had accepted. Thusly, any media stream that had been setup for use on a call should be transferred and used for the call that replaces it. + Profiles ======== .. NOTE:: This section is a work in progress. -.. TODO +.. TODO-doc - Metadata extensibility - Changing profile info generates m.presence events ("presencelike") - keys on m.presence are optional, except presence which is required @@ -1712,9 +1720,10 @@ Identity .. NOTE:: This section is a work in progress. -.. TODO Dave +.. TODO-doc Dave - 3PIDs and identity server, functions + Federation ========== @@ -1884,10 +1893,9 @@ All PDUs have: The maximum depth of the previous PDUs plus one. -.. TODO paul - [[TODO(paul): Update this structure so that 'pdu_id' is a two-element - [origin,ref] pair like the prev_pdus are]] - +.. TODO-spec paul + - Update this structure so that 'pdu_id' is a two-element [origin,ref] pair + like the prev_pdus are For state updates: @@ -1967,18 +1975,10 @@ keys exist to support this: {..., "is_state":true, - "state_key":TODO - "power_level":TODO - "prev_state_id":TODO - "prev_state_origin":TODO} - -.. TODO paul - [[TODO(paul): At this point we should probably have a long description of how - State management works, with descriptions of clobbering rules, power levels, etc - etc... But some of that detail is rather up-in-the-air, on the whiteboard, and - so on. This part needs refining. And writing in its own document as the details - relate to the server/system as a whole, not specifically to server-server - federation.]] + "state_key":TODO-doc + "power_level":TODO-doc + "prev_state_id":TODO-doc + "prev_state_origin":TODO-doc} EDUs, by comparison to PDUs, do not have an ID, a context, or a list of "previous" IDs. The only mandatory fields for these are the type, origin and @@ -2005,7 +2005,7 @@ For active pushing of messages representing live activity "as it happens":: PUT .../send/:transaction_id/ Body: JSON encoding of a single Transaction - Response: TODO + Response: TODO-doc The transaction_id path argument will override any ID given in the JSON body. The destination name will be set to that of the receiving server itself. Each @@ -2068,7 +2068,7 @@ Backfilling .. NOTE:: This section is a work in progress. -.. TODO +.. TODO-doc - What it is, when is it used, how is it done SRV Records @@ -2076,9 +2076,10 @@ SRV Records .. NOTE:: This section is a work in progress. -.. TODO +.. TODO-doc - Why it is needed + Security ======== @@ -2119,7 +2120,7 @@ victim would then include in their view of the chatroom history. Other servers in the chatroom would reject the invalid messages and potentially reject the victims messages as well since they depended on the invalid messages. -.. TODO +.. TODO-spec Track trustworthiness of HS or users based on if they try to pretend they haven't seen recent events, and fake a splitbrain... --M @@ -2227,7 +2228,7 @@ standard error response of the form:: The ``retry_after_ms`` key SHOULD be included to tell the client how long they have to wait in milliseconds before they can try again. -.. TODO +.. TODO-spec - Surely we should recommend an algorithm for the rate limiting, rather than letting every homeserver come up with their own idea, causing totally unpredictable performance over federated rooms? @@ -2236,30 +2237,33 @@ have to wait in milliseconds before they can try again. - Lawful intercept + Key Escrow TODO Mark + Policy Servers ============== .. NOTE:: This section is a work in progress. -.. TODO +.. TODO-spec We should mention them in the Architecture section at least... - + + Content repository ================== .. NOTE:: This section is a work in progress. -.. TODO +.. TODO-spec - path to upload - format for thumbnail paths, mention what it is protecting against. - content size limit and associated M_ERROR. + Address book repository ======================= .. NOTE:: This section is a work in progress. -.. TODO +.. TODO-spec - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET. - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of contacts), etc. From c8d67beb9cf32f273fd5a78e26636aa4feedde25 Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Wed, 1 Oct 2014 15:51:49 +0100 Subject: [PATCH 02/48] remove "red", "blue" and "green" server_name mappings --- synapse/http/client.py | 7 ------- 1 file changed, 7 deletions(-) diff --git a/synapse/http/client.py b/synapse/http/client.py index eb11bfd4d..822afeec1 100644 --- a/synapse/http/client.py +++ b/synapse/http/client.py @@ -35,13 +35,6 @@ import urllib logger = logging.getLogger(__name__) -# FIXME: SURELY these should be killed?! -_destination_mappings = { - "red": "localhost:8080", - "blue": "localhost:8081", - "green": "localhost:8082", -} - class HttpClient(object): """ Interface for talking json over http From 166bec0c088de407cace0ec9c5293f0a65e74824 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Wed, 1 Oct 2014 17:32:49 +0100 Subject: [PATCH 03/48] Nuke the entire 'Typing Notifications' spec section given as they don't exist yet in the implementation --- docs/specification.rst | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 6dcdea306..f35ddec88 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1536,22 +1536,6 @@ In the latter case, this allows for clients to display some minimal sense of presence information in a user list for a room. -Typing notifications -==================== -.. NOTE:: - This section is a work in progress. - -.. TODO-doc Leo - - what is the event type. Are they bundled with other event types? If so, which. - - what are the valid keys / values. What do they represent. Any gotchas? - - Timeouts. How do they work, who sets them and how do they expire. Does one - have priority over another? Give examples. - -.. TODO-spec Leo - - actually define the client-server API; the only thing that currently - exists is entirely server-server - - Voice over IP ============= Matrix can also be used to set up VoIP calls. This is part of the core From a6d3be4dbf4504a1ec07ffa9c10febbcdc63590d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 1 Oct 2014 17:55:31 +0100 Subject: [PATCH 04/48] s/m.room.redacted/m.room.redaction/ --- docs/specification.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specification.rst b/docs/specification.rst index f35ddec88..6fe72da22 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1143,7 +1143,7 @@ is the event that caused it to be redacted, which may include a reason. Redacting an event cannot be undone, allowing server owners to delete the offending content from the databases. -Currently, only room admins can redact events by sending a ``m.room.redacted`` +Currently, only room admins can redact events by sending a ``m.room.redaction`` event, but server admins also need to be able to redact events by a similar mechanism. From 5813e81dc65764cbb8862e49f619d6ff71e8e004 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Wed, 1 Oct 2014 17:59:55 +0100 Subject: [PATCH 05/48] Move documented but-unimplemented 'presence idle times' into a new document to contain such features --- docs/specification-NOTHAVE.rst | 20 ++++++++++++++++++++ docs/specification.rst | 8 -------- 2 files changed, 20 insertions(+), 8 deletions(-) create mode 100644 docs/specification-NOTHAVE.rst diff --git a/docs/specification-NOTHAVE.rst b/docs/specification-NOTHAVE.rst new file mode 100644 index 000000000..6ed8298cd --- /dev/null +++ b/docs/specification-NOTHAVE.rst @@ -0,0 +1,20 @@ +Matrix Specification NOTHAVEs +============================= + +This document contains sections of the main specification that have been +temporarily removed, because they specify intentions or aspirations that have +in no way yet been implemented. Rather than outright-deleting them, they have +been moved here so as to stand as an initial version for such time as they +become extant. + + +Presence +======== + +Idle Time +--------- +As well as the basic ``presence`` field, the presence information can also show +a sense of an "idle timer". This should be maintained individually by the +user's clients, and the home server can take the highest reported time as that +to report. When a user is offline, the home server can still report when the +user was last seen online. diff --git a/docs/specification.rst b/docs/specification.rst index 6fe72da22..84801b7d6 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1497,14 +1497,6 @@ in the other direction will not). This timestamp is presented via a key called ``last_active_ago``, which gives the relative number of miliseconds since the message is generated/emitted, that the user was last seen active. -Idle Time ---------- -As well as the basic ``presence`` field, the presence information can also show -a sense of an "idle timer". This should be maintained individually by the -user's clients, and the home server can take the highest reported time as that -to report. When a user is offline, the home server can still report when the -user was last seen online. - Transmission ------------ .. NOTE:: From a940a87ddc5bca1eb497abfc91eb835a61328d5f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 1 Oct 2014 18:16:47 +0100 Subject: [PATCH 06/48] SPEC-25: Add details on how to prune redacted events. SPEC-25 #comment I've added the details of what the server should do on receipt of a redaction event. In reality it can do whatever it wants, and its probably a reasonable implementation to flag it up to a server admin for verification before actually redacting an event. --- docs/specification.rst | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/specification.rst b/docs/specification.rst index 84801b7d6..22c55ad86 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1147,6 +1147,36 @@ Currently, only room admins can redact events by sending a ``m.room.redaction`` event, but server admins also need to be able to redact events by a similar mechanism. +Upon receipt of a redaction event, the server should strip off any keys not in +the following list: + + - ``event_id`` + - ``type`` + - ``room_id`` + - ``user_id`` + - ``state_key`` + - ``prev_state`` + - ``content`` + +The content object should also be stripped of all keys, unless it is one of +one of the following event types: + + - ``m.room.member`` allows key ``membership`` + - ``m.room.create`` allows key ``creator`` + - ``m.room.join_rules`` allows key ``join_rule`` + - ``m.room.power_levels`` allows keys that are user ids or ``default`` + - ``m.room.add_state_level`` allows key ``level`` + - ``m.room.send_event_level`` allows key ``level`` + - ``m.room.ops_levels`` allows keys ``kick_level``, ``ban_level`` + and ``redact_level`` + - ``m.room.aliases`` allows key ``aliases`` + +The redaction event should be added under the key ``redacted_because``. + + +When a client receives a redaction event it should change the redacted event +in the same way a server does. + Room Events =========== From ee447abcad251cd499a9752d3dbdda004c940298 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Wed, 1 Oct 2014 18:34:00 +0100 Subject: [PATCH 07/48] Continue moving content out of docs/model/presence into the main spec; delete model docs that are duplicated --- docs/client-server/model/presence.rst | 100 -------------------------- docs/specification-NOTHAVE.rst | 10 +++ docs/specification.rst | 14 ++++ 3 files changed, 24 insertions(+), 100 deletions(-) diff --git a/docs/client-server/model/presence.rst b/docs/client-server/model/presence.rst index 7e5450536..811bac3fa 100644 --- a/docs/client-server/model/presence.rst +++ b/docs/client-server/model/presence.rst @@ -1,103 +1,3 @@ -======== -Presence -======== - -A description of presence information and visibility between users. - -Overview -======== - -Each user has the concept of Presence information. This encodes a sense of the -"availability" of that user, suitable for display on other user's clients. - - -Presence Information -==================== - -The basic piece of presence information is an enumeration of a small set of -state; such as "free to chat", "online", "busy", or "offline". The default state -unless the user changes it is "online". Lower states suggest some amount of -decreased availability from normal, which might have some client-side effect -like muting notification sounds and suggests to other users not to bother them -unless it is urgent. Equally, the "free to chat" state exists to let the user -announce their general willingness to receive messages moreso than default. - -Home servers should also allow a user to set their state as "hidden" - a state -which behaves as offline, but allows the user to see the client state anyway and -generally interact with client features such as reading message history or -accessing contacts in the address book. - -This basic state field applies to the user as a whole, regardless of how many -client devices they have connected. The home server should synchronise this -status choice among multiple devices to ensure the user gets a consistent -experience. - -Idle Time ---------- - -As well as the basic state field, the presence information can also show a sense -of an "idle timer". This should be maintained individually by the user's -clients, and the homeserver can take the highest reported time as that to -report. Likely this should be presented in fairly coarse granularity; possibly -being limited to letting the home server automatically switch from a "free to -chat" or "online" mode into "idle". - -When a user is offline, the Home Server can still report when the user was last -seen online, again perhaps in a somewhat coarse manner. - -Device Type ------------ - -Client devices that may limit the user experience somewhat (such as "mobile" -devices with limited ability to type on a real keyboard or read large amounts of -text) should report this to the home server, as this is also useful information -to report as "presence" if the user cannot be expected to provide a good typed -response to messages. - - -Presence List -============= - -Each user's home server stores a "presence list" for that user. This stores a -list of other user IDs the user has chosen to add to it (remembering any ACL -Pointer if appropriate). - -To be added to a contact list, the user being added must grant permission. Once -granted, both user's HS(es) store this information, as it allows the user who -has added the contact some more abilities; see below. Since such subscriptions -are likely to be bidirectional, HSes may wish to automatically accept requests -when a reverse subscription already exists. - -As a convenience, presence lists should support the ability to collect users -into groups, which could allow things like inviting the entire group to a new -("ad-hoc") chat room, or easy interaction with the profile information ACL -implementation of the HS. - - -Presence and Permissions -======================== - -For a viewing user to be allowed to see the presence information of a target -user, either - - * The target user has allowed the viewing user to add them to their presence - list, or - - * The two users share at least one room in common - -In the latter case, this allows for clients to display some minimal sense of -presence information in a user list for a room. - -Home servers can also use the user's choice of presence state as a signal for -how to handle new private one-to-one chat message requests. For example, it -might decide: - - "free to chat": accept anything - "online": accept from anyone in my addres book list - "busy": accept from anyone in this "important people" group in my address - book list - - API Efficiency ============== diff --git a/docs/specification-NOTHAVE.rst b/docs/specification-NOTHAVE.rst index 6ed8298cd..369594f6a 100644 --- a/docs/specification-NOTHAVE.rst +++ b/docs/specification-NOTHAVE.rst @@ -18,3 +18,13 @@ a sense of an "idle timer". This should be maintained individually by the user's clients, and the home server can take the highest reported time as that to report. When a user is offline, the home server can still report when the user was last seen online. + +Device Type +----------- + +Client devices that may limit the user experience somewhat (such as "mobile" +devices with limited ability to type on a real keyboard or read large amounts of +text) should report this to the home server, as this is also useful information +to report as "presence" if the user cannot be expected to provide a good typed +response to messages. + diff --git a/docs/specification.rst b/docs/specification.rst index 22c55ad86..ada40bdbe 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1527,6 +1527,15 @@ in the other direction will not). This timestamp is presented via a key called ``last_active_ago``, which gives the relative number of miliseconds since the message is generated/emitted, that the user was last seen active. +Home servers can also use the user's choice of presence state as a signal for +how to handle new private one-to-one chat message requests. For example, it +might decide: + + - ``free_for_chat`` : accept anything + - ``online`` : accept from anyone in my addres book list + - ``busy`` : accept from anyone in this "important people" group in my + address book list + Transmission ------------ .. NOTE:: @@ -1545,6 +1554,11 @@ granted, both user's HS(es) store this information. Since such subscriptions are likely to be bidirectional, HSes may wish to automatically accept requests when a reverse subscription already exists. +As a convenience, presence lists should support the ability to collect users +into groups, which could allow things like inviting the entire group to a new +("ad-hoc") chat room, or easy interaction with the profile information ACL +implementation of the HS. + Presence and Permissions ------------------------ For a viewing user to be allowed to see the presence information of a target From c5757a0266a26f8448e42bf504c38b379c5a37d6 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Wed, 1 Oct 2014 19:35:13 +0100 Subject: [PATCH 08/48] Define the client and server APIs for Presence --- docs/specification.rst | 111 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 102 insertions(+), 9 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index ada40bdbe..e270d8a6f 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1536,15 +1536,6 @@ might decide: - ``busy`` : accept from anyone in this "important people" group in my address book list -Transmission ------------- -.. NOTE:: - This section is a work in progress. - -.. TODO-doc: - - Transmitted as an EDU. - - Presence lists determine who to send to. - Presence List ------------- Each user's home server stores a "presence list" for that user. This stores a @@ -1571,6 +1562,108 @@ user, either: In the latter case, this allows for clients to display some minimal sense of presence information in a user list for a room. +Client API +---------- +The client API for presence is on the following set of REST calls. + +Fetching basic status:: + + GET $PREFIX/presence/:user_id/status + + Returned content: JSON object containing the following keys: + presence: "offline"|"unavailable"|"online"|"free_for_chat" + status_msg: (optional) string of freeform text + last_active_ago: miliseconds since the last activity by the user + +Setting basic status:: + + PUT $PREFIX/presence/:user_id/status + + Content: JSON object containing the following keys: + presence and status_msg: as above + +When setting the status, the activity time is updated to reflect that activity; +the client does not need to specify the ``last_active_ago`` field. + +Fetching the presence list:: + + GET $PREFIX/presence/list + + Returned content: JSON array containing objects; each object containing the + following keys: + user_id: observed user ID + presence: "offline"|"unavailable"|"online"|"free_for_chat" + status_msg: (optional) string of freeform text + last_active_ago: miliseconds since the last activity by the user + +Maintaining the presence list:: + + POST $PREFIX/presence/list + + Content: JSON object containing either or both of the following keys: + invite: JSON array of strings giving user IDs to send invites to + drop: JSON array of strings giving user IDs to remove from the list + +.. TODO-spec + - Define how users receive presence invites, and how they accept/decline them + +Server API +---------- +The server API for presence is based entirely on exchange of the following +EDUs. There are no PDUs or Federation Queries involved. + +Performing a presence update and poll subscription request:: + + EDU type: m.presence + + Content keys: + push: (optional): list of push operations. + Each should be an object with the following keys: + user_id: string containing a User ID + presence: "offline"|"unavailable"|"online"|"free_for_chat" + status_msg: (optional) string of freeform text + last_active_ago: miliseconds since the last activity by the user + + poll: (optional): list of strings giving User IDs + + unpoll: (optional): list of strings giving User IDs + +The presence of this combined message is two-fold: it informs the recipient +server of the current status of one or more users on the sending server (by the +``push`` key), and it maintains the list of users on the recipient server that +the sending server is interested in receiving updates for, by adding (by the +``poll`` key) or removing them (by the ``unpoll`` key). The ``poll`` and +``unpoll`` lists apply *changes* to the implied list of users; any existing IDs +that the server sent as ``poll`` operations in a previous message are not +removed until explicitly requested by a later ``unpoll``. + +On receipt of a message containing a non-empty ``poll`` list, the receiving +server should immediately send the sending server a presence update EDU of its +own, containing in a ``push`` list the current state of every user that was in +the orginal EDU's ``poll`` list. + +Sending a presence invite:: + + EDU type: m.presence_invite + + Content keys: + observed_user: string giving the User ID of the user whose presence is + requested (i.e. the recipient of the invite) + observer_user: string giving the User ID of the user who is requesting to + observe the presence (i.e. the sender of the invite) + +Accepting a presence invite:: + + EDU type: m.presence_accept + + Content keys - as for m.presence_invite + +Rejecting a presence invite:: + + EDU type: m.presence_deny + + Content keys - as for m.presence_invite + Voice over IP ============= From bf8b9b90cde80f4e60433378c552176f3a1d0619 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Wed, 1 Oct 2014 19:37:18 +0100 Subject: [PATCH 09/48] Added a TODO-doc marker about the presence timing system --- docs/specification.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/specification.rst b/docs/specification.rst index e270d8a6f..0a421e9e3 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1664,6 +1664,12 @@ Rejecting a presence invite:: Content keys - as for m.presence_invite +.. TODO-doc + - Explain the timing-based roundtrip reduction mechanism for presence + messages + - Explain the zero-byte presence inference logic + See also: docs/client-server/model/presence + Voice over IP ============= From a0b1b34c71e46304f7024f0570db92f4577303b5 Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 2 Oct 2014 09:55:26 +0100 Subject: [PATCH 10/48] Make instructions synctl gives for generateing a config file actuall generate a config file. Also, make synctil run synapse correctly by invoking a module such that the path is correct to pull in other bits from the working directory rather than requiring them to be on the PYTHONPATH (which would lead to people being very confused when they edit source in the working directory and their changes do not take effect). --- synctl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/synctl b/synctl index 0f83e9cb1..7523fd3db 100755 --- a/synctl +++ b/synctl @@ -1,6 +1,6 @@ #!/bin/bash -SYNAPSE="synapse/app/homeserver.py" +SYNAPSE="python -m synapse.app.homeserver" CONFIGFILE="homeserver.yaml" PIDFILE="homeserver.pid" @@ -14,7 +14,7 @@ case "$1" in start) if [ ! -f "$CONFIGFILE" ]; then echo "No config file found" - echo "To generate a config file, run 'python --generate-config'" + echo "To generate a config file, run '$SYNAPSE -c $CONFIGFILE --generate-config'" exit 1 fi From d1adb19b8af839afcb4fa4d302abb7036a9acd0a Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 2 Oct 2014 10:38:11 +0100 Subject: [PATCH 11/48] Re-apply a0b1b34c71e46304f7024f0570db92f4577303b5 to master (fixing synctl) --- synctl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/synctl b/synctl index 0f83e9cb1..7523fd3db 100755 --- a/synctl +++ b/synctl @@ -1,6 +1,6 @@ #!/bin/bash -SYNAPSE="synapse/app/homeserver.py" +SYNAPSE="python -m synapse.app.homeserver" CONFIGFILE="homeserver.yaml" PIDFILE="homeserver.pid" @@ -14,7 +14,7 @@ case "$1" in start) if [ ! -f "$CONFIGFILE" ]; then echo "No config file found" - echo "To generate a config file, run 'python --generate-config'" + echo "To generate a config file, run '$SYNAPSE -c $CONFIGFILE --generate-config'" exit 1 fi From 7a322b63264acbef7e60b511ad8d39ae4718386b Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 2 Oct 2014 10:43:22 +0100 Subject: [PATCH 12/48] Update README setup instructions to be correct. Make synapse spit out explanatory note when generating config to tell people to look at it and customise it. --- README.rst | 12 +++++++----- synapse/config/_base.py | 1 + 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/README.rst b/README.rst index 6f7940e74..1530e5caa 100644 --- a/README.rst +++ b/README.rst @@ -46,11 +46,13 @@ To get up and running: - To simply play with an **existing** homeserver you can just go straight to http://matrix.org/alpha. - - To run your own **private** homeserver on localhost:8008, install synapse with - ``python setup.py develop --user`` and then run ``./synctl start`` twice (once to - generate a config; once to actually run) - you will find a webclient running at - http://localhost:8008. Please use a recent Chrome, Safari or Firefox for now... - + - To run your own **private** homeserver on localhost:8008, generate a basic + config file: ``./synctl start`` will give you instructions on how to do this. + Once you've done so, running ``./synctl start`` again will start your private + home sserver. You will find a webclient running at http://localhost:8008. + Please use a recent Chrome or Firefox for now (or Safari if you don't need + VoIP support). + - To run a **public** homeserver and let it exchange messages with other homeservers and participate in the global Matrix federation, you must expose port 8448 to the internet and edit homeserver.yaml to specify server_name (the public DNS entry for diff --git a/synapse/config/_base.py b/synapse/config/_base.py index 35bcece2c..809f9c922 100644 --- a/synapse/config/_base.py +++ b/synapse/config/_base.py @@ -123,6 +123,7 @@ class Config(object): # style mode markers into the file, to hint to people that # this is a YAML file. yaml.dump(config, config_file, default_flow_style=False) + print "A config file has been generated in %s (your server name is '%s'). Please review this file and customise it to your needs." % (config_args.config_path, config['server_name']) sys.exit(0) return cls(args) From 45f7677bdcffdc1e0b64d05f1b7c23dab0d1342c Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Thu, 2 Oct 2014 11:00:21 +0100 Subject: [PATCH 13/48] Trivial formatting fixes for README. --- README.rst | 166 +++++++++++++++++++++++++++-------------------------- 1 file changed, 85 insertions(+), 81 deletions(-) diff --git a/README.rst b/README.rst index 1530e5caa..0459d5463 100644 --- a/README.rst +++ b/README.rst @@ -4,13 +4,13 @@ Introduction Matrix is an ambitious new ecosystem for open federated Instant Messaging and VoIP. The basics you need to know to get up and running are: - - Chatrooms are distributed and do not exist on any single server. Rooms - can be found using aliases like ``#matrix:matrix.org`` or - ``#test:localhost:8008`` or they can be ephemeral. - - - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future - you will normally refer to yourself and others using a 3PID: email - address, phone number, etc rather than manipulating Matrix user IDs) +- Chatrooms are distributed and do not exist on any single server. Rooms + can be found using aliases like ``#matrix:matrix.org`` or + ``#test:localhost:8008`` or they can be ephemeral. + +- Matrix user IDs look like ``@matthew:matrix.org`` (although in the future + you will normally refer to yourself and others using a 3PID: email + address, phone number, etc rather than manipulating Matrix user IDs) The overall architecture is:: @@ -20,70 +20,74 @@ The overall architecture is:: WARNING ======= -**Synapse is currently in a state of rapid development, and not all features are yet functional. -Critically, some security features are still in development, which means Synapse can *not* -be considered secure or reliable at this point.** For instance: +**Synapse is currently in a state of rapid development, and not all features +are yet functional. Critically, some security features are still in +development, which means Synapse can *not* be considered secure or reliable at +this point.** For instance: - **SSL Certificates used by server-server federation are not yet validated.** - **Room permissions are not yet enforced on traffic received via federation.** -- **Homeservers do not yet cryptographically sign their events to avoid tampering** +- **Homeservers do not yet cryptographically sign their events to avoid + tampering** - Default configuration provides open signup to the service from the internet -Despite this, we believe Synapse is more than useful as a way for experimenting and -exploring Synapse, and the missing features will land shortly. **Until then, please do *NOT* -use Synapse for any remotely important or secure communication.** +Despite this, we believe Synapse is more than useful as a way for experimenting +and exploring Synapse, and the missing features will land shortly. **Until +then, please do *NOT* use Synapse for any remotely important or secure +communication.** Quick Start =========== System requirements: - - POSIX-compliant system (tested on Linux & OSX) - - Python 2.7 +- POSIX-compliant system (tested on Linux & OSX) +- Python 2.7 To get up and running: - - - To simply play with an **existing** homeserver you can - just go straight to http://matrix.org/alpha. - - - To run your own **private** homeserver on localhost:8008, generate a basic - config file: ``./synctl start`` will give you instructions on how to do this. - Once you've done so, running ``./synctl start`` again will start your private - home sserver. You will find a webclient running at http://localhost:8008. - Please use a recent Chrome or Firefox for now (or Safari if you don't need - VoIP support). - - To run a **public** homeserver and let it exchange messages with other homeservers - and participate in the global Matrix federation, you must expose port 8448 to the - internet and edit homeserver.yaml to specify server_name (the public DNS entry for - this server) and then run ``synctl start``. If you changed the server_name, you may - need to move the old database (homeserver.db) out of the way first. Then come join - ``#matrix:matrix.org`` and say hi! :) +- To simply play with an **existing** homeserver you can + just go straight to http://matrix.org/alpha. + +- To run your own **private** homeserver on localhost:8008, generate a basic + config file: ``./synctl start`` will give you instructions on how to do this. + Once you've done so, running ``./synctl start`` again will start your private + home sserver. You will find a webclient running at http://localhost:8008. + Please use a recent Chrome or Firefox for now (or Safari if you don't need + VoIP support). + +- To run a **public** homeserver and let it exchange messages with other + homeservers and participate in the global Matrix federation, you must expose + port 8448 to the internet and edit homeserver.yaml to specify server_name + (the public DNS entry for this server) and then run ``synctl start``. If you + changed the server_name, you may need to move the old database + (homeserver.db) out of the way first. Then come join ``#matrix:matrix.org`` + and say hi! :) For more detailed setup instructions, please see further down this document. - + About Matrix ============ Matrix specifies a set of pragmatic RESTful HTTP JSON APIs as an open standard, which handle: - - Creating and managing fully distributed chat rooms with no - single points of control or failure - - Eventually-consistent cryptographically secure[1] synchronisation of room - state across a global open network of federated servers and services - - Sending and receiving extensible messages in a room with (optional) - end-to-end encryption[2] - - Inviting, joining, leaving, kicking, banning room members - - Managing user accounts (registration, login, logout) - - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, - Facebook accounts to authenticate, identify and discover users on Matrix. - - Placing 1:1 VoIP and Video calls +- Creating and managing fully distributed chat rooms with no + single points of control or failure +- Eventually-consistent cryptographically secure[1] synchronisation of room + state across a global open network of federated servers and services +- Sending and receiving extensible messages in a room with (optional) + end-to-end encryption[2] +- Inviting, joining, leaving, kicking, banning room members +- Managing user accounts (registration, login, logout) +- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, + Facebook accounts to authenticate, identify and discover users on Matrix. +- Placing 1:1 VoIP and Video calls These APIs are intended to be implemented on a wide range of servers, services -and clients, letting developers build messaging and VoIP functionality on top of -the entirely open Matrix ecosystem rather than using closed or proprietary +and clients, letting developers build messaging and VoIP functionality on top +of the entirely open Matrix ecosystem rather than using closed or proprietary solutions. The hope is for Matrix to act as the building blocks for a new generation of fully open and interoperable messaging and VoIP apps for the internet. @@ -98,17 +102,17 @@ In Matrix, every user runs one or more Matrix clients, which connect through to a Matrix homeserver which stores all their personal chat history and user account information - much as a mail client connects through to an IMAP/SMTP server. Just like email, you can either run your own Matrix homeserver and -control and own your own communications and history or use one hosted by someone -else (e.g. matrix.org) - there is no single point of control or mandatory -service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc. +control and own your own communications and history or use one hosted by +someone else (e.g. matrix.org) - there is no single point of control or +mandatory service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc. Synapse ships with two basic demo Matrix clients: webclient (a basic group chat web client demo implemented in AngularJS) and cmdclient (a basic Python command line utility which lets you easily see what the JSON APIs are up to). We'd like to invite you to take a look at the Matrix spec, try to run a -homeserver, and join the existing Matrix chatrooms already out there, experiment -with the APIs and the demo clients, and let us know your thoughts at +homeserver, and join the existing Matrix chatrooms already out there, +experiment with the APIs and the demo clients, and let us know your thoughts at https://github.com/matrix-org/synapse/issues or at matrix@matrix.org. Thanks for trying Matrix! @@ -121,14 +125,14 @@ Thanks for trying Matrix! Homeserver Installation ======================= -First, the dependencies need to be installed. Start by installing +First, the dependencies need to be installed. Start by installing 'python2.7-dev' and the various tools of the compiler toolchain. - Installing prerequisites on Ubuntu:: +Installing prerequisites on Ubuntu:: $ sudo apt-get install build-essential python2.7-dev libffi-dev - Installing prerequisites on Mac OS X:: +Installing prerequisites on Mac OS X:: $ xcode-select --install @@ -136,24 +140,24 @@ The homeserver has a number of external dependencies, that are easiest to install by making setup.py do so, in --user mode:: $ python setup.py develop --user - + You'll need a version of setuptools new enough to know about git, so you -may need to also run: +may need to also run:: $ sudo apt-get install python-pip $ sudo pip install --upgrade setuptools - + If you don't have access to github, then you may need to install ``syutil`` -manually by checking it out and running ``python setup.py develop --user`` on it -too. - +manually by checking it out and running ``python setup.py develop --user`` on +it too. + If you get errors about ``sodium.h`` being missing, you may also need to manually install a newer PyNaCl via pip as setuptools installs an old one. Or you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and -installing it. Installing PyNaCl using pip may also work (remember to remove any -other versions installed by setuputils in, for example, ~/.local/lib). +installing it. Installing PyNaCl using pip may also work (remember to remove +any other versions installed by setuputils in, for example, ~/.local/lib). -On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` +On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` you will need to ``export CFLAGS=-Qunused-arguments``. This will run a process of downloading and installing into your @@ -177,7 +181,7 @@ Upgrading an existing homeserver Before upgrading an existing homeserver to a new version, please refer to UPGRADE.rst for any additional instructions. - + Setting up Federation ===================== @@ -187,14 +191,14 @@ be publicly visible on the internet, and they will need to know its host name. You have two choices here, which will influence the form of your Matrix user IDs: - 1) Use the machine's own hostname as available on public DNS in the form of its - A or AAAA records. This is easier to set up initially, perhaps for testing, - but lacks the flexibility of SRV. +1) Use the machine's own hostname as available on public DNS in the form of + its A or AAAA records. This is easier to set up initially, perhaps for + testing, but lacks the flexibility of SRV. - 2) Set up a SRV record for your domain name. This requires you create a SRV - record in DNS, but gives the flexibility to run the server on your own - choice of TCP port, on a machine that might not be the same name as the - domain name. +2) Set up a SRV record for your domain name. This requires you create a SRV + record in DNS, but gives the flexibility to run the server on your own + choice of TCP port, on a machine that might not be the same name as the + domain name. For the first form, simply pass the required hostname (of the machine) as the --host parameter:: @@ -204,10 +208,10 @@ For the first form, simply pass the required hostname (of the machine) as the --config-path homeserver.config \ --generate-config $ python synapse/app/homeserver.py --config-path homeserver.config - -Alternatively, you can run synapse via synctl - running ``synctl start`` to -generate a homeserver.yaml config file, where you can then edit server-name to -specify machine.my.domain.name, and then set the actual server running again + +Alternatively, you can run synapse via synctl - running ``synctl start`` to +generate a homeserver.yaml config file, where you can then edit server-name to +specify machine.my.domain.name, and then set the actual server running again with synctl start. For the second form, first create your SRV record and publish it in DNS. This @@ -269,8 +273,8 @@ account. Your name will take the form of:: Specify your desired localpart in the topmost box of the "Register for an account" form, and click the "Register" button. Hostnames can contain ports if -required due to lack of SRV records (e.g. @matthew:localhost:8080 on an internal -synapse sandbox running on localhost) +required due to lack of SRV records (e.g. @matthew:localhost:8080 on an +internal synapse sandbox running on localhost) Logging In To An Existing Account @@ -285,9 +289,9 @@ Identity Servers The job of authenticating 3PIDs and tracking which 3PIDs are associated with a given Matrix user is very security-sensitive, as there is obvious risk of spam -if it is too easy to sign up for Matrix accounts or harvest 3PID data. Meanwhile -the job of publishing the end-to-end encryption public keys for Matrix users is -also very security-sensitive for similar reasons. +if it is too easy to sign up for Matrix accounts or harvest 3PID data. +Meanwhile the job of publishing the end-to-end encryption public keys for +Matrix users is also very security-sensitive for similar reasons. Therefore the role of managing trusted identity in the Matrix ecosystem is farmed out to a cluster of known trusted ecosystem partners, who run 'Matrix @@ -296,7 +300,7 @@ track 3PID logins and publish end-user public keys. It's currently early days for identity servers as Matrix is not yet using 3PIDs as the primary means of identity and E2E encryption is not complete. As such, -we are running a single identity server (http://matrix.org:8090) at the current +we are running a single identity server (http://matrix.org:8090) at the current time. From 4f11518934c3e032a763f115a73261414d67f87b Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Thu, 2 Oct 2014 13:57:48 +0100 Subject: [PATCH 14/48] Split PlainHttpClient into separate clients for talking to Identity servers and talking to Capatcha servers --- synapse/app/homeserver.py | 4 +- synapse/handlers/directory.py | 4 +- synapse/handlers/login.py | 6 +- synapse/handlers/register.py | 11 +- synapse/http/client.py | 296 ++++++++++++++++--------------- tests/handlers/test_directory.py | 4 +- 6 files changed, 166 insertions(+), 159 deletions(-) diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py index 2f1b95490..61d574a00 100755 --- a/synapse/app/homeserver.py +++ b/synapse/app/homeserver.py @@ -25,7 +25,7 @@ from twisted.web.static import File from twisted.web.server import Site from synapse.http.server import JsonResource, RootRedirect from synapse.http.content_repository import ContentRepoResource -from synapse.http.client import TwistedHttpClient +from synapse.http.client import MatrixHttpClient from synapse.api.urls import ( CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX ) @@ -47,7 +47,7 @@ logger = logging.getLogger(__name__) class SynapseHomeServer(HomeServer): def build_http_client(self): - return TwistedHttpClient(self) + return MatrixHttpClient(self) def build_resource_for_client(self): return JsonResource() diff --git a/synapse/handlers/directory.py b/synapse/handlers/directory.py index 84c3a1d56..cec7737e0 100644 --- a/synapse/handlers/directory.py +++ b/synapse/handlers/directory.py @@ -18,7 +18,7 @@ from twisted.internet import defer from ._base import BaseHandler from synapse.api.errors import SynapseError -from synapse.http.client import HttpClient +from synapse.http.client import MatrixHttpClient from synapse.api.events.room import RoomAliasesEvent import logging @@ -98,7 +98,7 @@ class DirectoryHandler(BaseHandler): query_type="directory", args={ "room_alias": room_alias.to_string(), - HttpClient.RETRY_DNS_LOOKUP_FAILURES: False + MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES: False } ) diff --git a/synapse/handlers/login.py b/synapse/handlers/login.py index 80ffdd272..3f152e18f 100644 --- a/synapse/handlers/login.py +++ b/synapse/handlers/login.py @@ -17,7 +17,7 @@ from twisted.internet import defer from ._base import BaseHandler from synapse.api.errors import LoginError, Codes -from synapse.http.client import PlainHttpClient +from synapse.http.client import IdentityServerHttpClient from synapse.util.emailutils import EmailException import synapse.util.emailutils as emailutils @@ -97,10 +97,10 @@ class LoginHandler(BaseHandler): @defer.inlineCallbacks def _query_email(self, email): - httpCli = PlainHttpClient(self.hs) + httpCli = IdentityServerHttpClient(self.hs) data = yield httpCli.get_json( 'matrix.org:8090', # TODO FIXME This should be configurable. "/_matrix/identity/api/v1/lookup?medium=email&address=" + "%s" % urllib.quote(email) ) - defer.returnValue(data) \ No newline at end of file + defer.returnValue(data) diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py index a019d770d..266495056 100644 --- a/synapse/handlers/register.py +++ b/synapse/handlers/register.py @@ -22,7 +22,8 @@ from synapse.api.errors import ( ) from ._base import BaseHandler import synapse.util.stringutils as stringutils -from synapse.http.client import PlainHttpClient +from synapse.http.client import IdentityServerHttpClient +from synapse.http.client import CaptchaServerHttpClient import base64 import bcrypt @@ -154,7 +155,9 @@ class RegistrationHandler(BaseHandler): @defer.inlineCallbacks def _threepid_from_creds(self, creds): - httpCli = PlainHttpClient(self.hs) + # TODO: get this from the homeserver rather than creating a new one for + # each request + httpCli = IdentityServerHttpClient(self.hs) # XXX: make this configurable! trustedIdServers = ['matrix.org:8090'] if not creds['idServer'] in trustedIdServers: @@ -203,7 +206,9 @@ class RegistrationHandler(BaseHandler): @defer.inlineCallbacks def _submit_captcha(self, ip_addr, private_key, challenge, response): - client = PlainHttpClient(self.hs) + # TODO: get this from the homeserver rather than creating a new one for + # each request + client = CaptchaServerHttpClient(self.hs) data = yield client.post_urlencoded_get_raw( "www.google.com:80", "/recaptcha/api/verify", diff --git a/synapse/http/client.py b/synapse/http/client.py index 822afeec1..e02cce564 100644 --- a/synapse/http/client.py +++ b/synapse/http/client.py @@ -36,49 +36,6 @@ import urllib logger = logging.getLogger(__name__) -class HttpClient(object): - """ Interface for talking json over http - """ - RETRY_DNS_LOOKUP_FAILURES = "__retry_dns" - - def put_json(self, destination, path, data): - """ Sends the specifed json data using PUT - - Args: - destination (str): The remote server to send the HTTP request - to. - path (str): The HTTP path. - data (dict): A dict containing the data that will be used as - the request body. This will be encoded as JSON. - - Returns: - Deferred: Succeeds when we get a 2xx HTTP response. The result - will be the decoded JSON body. On a 4xx or 5xx error response a - CodeMessageException is raised. - """ - pass - - def get_json(self, destination, path, args=None): - """ Get's some json from the given host homeserver and path - - Args: - destination (str): The remote server to send the HTTP request - to. - path (str): The HTTP path. - args (dict): A dictionary used to create query strings, defaults to - None. - **Note**: The value of each key is assumed to be an iterable - and *not* a string. - - Returns: - Deferred: Succeeds when we get *any* HTTP response. - - The result of the deferred is a tuple of `(code, response)`, - where `response` is a dict representing the decoded JSON body. - """ - pass - - class MatrixHttpAgent(_AgentBase): def __init__(self, reactor, pool=None): @@ -102,113 +59,14 @@ class MatrixHttpAgent(_AgentBase): parsed_URI.originForm) -class TwistedHttpClient(HttpClient): - """ Wrapper around the twisted HTTP client api. - - Attributes: - agent (twisted.web.client.Agent): The twisted Agent used to send the - requests. +class BaseHttpClient(object): + """Base class for HTTP clients using twisted. """ def __init__(self, hs): self.agent = MatrixHttpAgent(reactor) self.hs = hs - @defer.inlineCallbacks - def put_json(self, destination, path, data, on_send_callback=None): - if destination in _destination_mappings: - destination = _destination_mappings[destination] - - response = yield self._create_request( - destination.encode("ascii"), - "PUT", - path.encode("ascii"), - producer=_JsonProducer(data), - headers_dict={"Content-Type": ["application/json"]}, - on_send_callback=on_send_callback, - ) - - logger.debug("Getting resp body") - body = yield readBody(response) - logger.debug("Got resp body") - - defer.returnValue((response.code, body)) - - @defer.inlineCallbacks - def get_json(self, destination, path, args={}): - if destination in _destination_mappings: - destination = _destination_mappings[destination] - - logger.debug("get_json args: %s", args) - - retry_on_dns_fail = True - if HttpClient.RETRY_DNS_LOOKUP_FAILURES in args: - # FIXME: This isn't ideal, but the interface exposed in get_json - # isn't comprehensive enough to give caller's any control over - # their connection mechanics. - retry_on_dns_fail = args.pop(HttpClient.RETRY_DNS_LOOKUP_FAILURES) - - query_bytes = urllib.urlencode(args, True) - logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail) - - response = yield self._create_request( - destination.encode("ascii"), - "GET", - path.encode("ascii"), - query_bytes=query_bytes, - retry_on_dns_fail=retry_on_dns_fail - ) - - body = yield readBody(response) - - defer.returnValue(json.loads(body)) - - @defer.inlineCallbacks - def post_urlencoded_get_json(self, destination, path, args={}): - if destination in _destination_mappings: - destination = _destination_mappings[destination] - - logger.debug("post_urlencoded_get_json args: %s", args) - query_bytes = urllib.urlencode(args, True) - - response = yield self._create_request( - destination.encode("ascii"), - "POST", - path.encode("ascii"), - producer=FileBodyProducer(StringIO(urllib.urlencode(args))), - headers_dict={"Content-Type": ["application/x-www-form-urlencoded"]} - ) - - body = yield readBody(response) - - defer.returnValue(json.loads(body)) - - # XXX FIXME : I'm so sorry. - @defer.inlineCallbacks - def post_urlencoded_get_raw(self, destination, path, accept_partial=False, args={}): - if destination in _destination_mappings: - destination = _destination_mappings[destination] - - query_bytes = urllib.urlencode(args, True) - - response = yield self._create_request( - destination.encode("ascii"), - "POST", - path.encode("ascii"), - producer=FileBodyProducer(StringIO(urllib.urlencode(args))), - headers_dict={"Content-Type": ["application/x-www-form-urlencoded"]} - ) - - try: - body = yield readBody(response) - defer.returnValue(body) - except PartialDownloadError as e: - if accept_partial: - defer.returnValue(e.response) - else: - raise e - - @defer.inlineCallbacks def _create_request(self, destination, method, path_bytes, param_bytes=b"", query_bytes=b"", producer=None, headers_dict={}, @@ -232,7 +90,6 @@ class TwistedHttpClient(HttpClient): retries_left = 5 - # TODO: setup and pass in an ssl_context to enable TLS endpoint = self._getEndpoint(reactor, destination); while True: @@ -283,6 +140,92 @@ class TwistedHttpClient(HttpClient): defer.returnValue(response) + +class MatrixHttpClient(BaseHttpClient): + """ Wrapper around the twisted HTTP client api. Implements + + Attributes: + agent (twisted.web.client.Agent): The twisted Agent used to send the + requests. + """ + + RETRY_DNS_LOOKUP_FAILURES = "__retry_dns" + + @defer.inlineCallbacks + def put_json(self, destination, path, data, on_send_callback=None): + """ Sends the specifed json data using PUT + + Args: + destination (str): The remote server to send the HTTP request + to. + path (str): The HTTP path. + data (dict): A dict containing the data that will be used as + the request body. This will be encoded as JSON. + + Returns: + Deferred: Succeeds when we get a 2xx HTTP response. The result + will be the decoded JSON body. On a 4xx or 5xx error response a + CodeMessageException is raised. + """ + response = yield self._create_request( + destination.encode("ascii"), + "PUT", + path.encode("ascii"), + producer=_JsonProducer(data), + headers_dict={"Content-Type": ["application/json"]}, + on_send_callback=on_send_callback, + ) + + logger.debug("Getting resp body") + body = yield readBody(response) + logger.debug("Got resp body") + + defer.returnValue((response.code, body)) + + @defer.inlineCallbacks + def get_json(self, destination, path, args={}): + """ Get's some json from the given host homeserver and path + + Args: + destination (str): The remote server to send the HTTP request + to. + path (str): The HTTP path. + args (dict): A dictionary used to create query strings, defaults to + None. + **Note**: The value of each key is assumed to be an iterable + and *not* a string. + + Returns: + Deferred: Succeeds when we get *any* HTTP response. + + The result of the deferred is a tuple of `(code, response)`, + where `response` is a dict representing the decoded JSON body. + """ + logger.debug("get_json args: %s", args) + + retry_on_dns_fail = True + if HttpClient.RETRY_DNS_LOOKUP_FAILURES in args: + # FIXME: This isn't ideal, but the interface exposed in get_json + # isn't comprehensive enough to give caller's any control over + # their connection mechanics. + retry_on_dns_fail = args.pop(HttpClient.RETRY_DNS_LOOKUP_FAILURES) + + query_bytes = urllib.urlencode(args, True) + logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail) + + response = yield self._create_request( + destination.encode("ascii"), + "GET", + path.encode("ascii"), + query_bytes=query_bytes, + retry_on_dns_fail=retry_on_dns_fail + ) + + body = yield readBody(response) + + defer.returnValue(json.loads(body)) + + def _getEndpoint(self, reactor, destination): return matrix_endpoint( reactor, destination, timeout=10, @@ -290,10 +233,69 @@ class TwistedHttpClient(HttpClient): ) -class PlainHttpClient(TwistedHttpClient): +class IdentityServerHttpClient(BaseHttpClient): + """Separate HTTP client for talking to the Identity servers since they + don't use SRV records and talk x-www-form-urlencoded rather than JSON. + """ + def _getEndpoint(self, reactor, destination): + #TODO: This should be talking TLS + return matrix_endpoint(reactor, destination, timeout=10) + + @defer.inlineCallbacks + def post_urlencoded_get_json(self, destination, path, args={}): + if destination in _destination_mappings: + destination = _destination_mappings[destination] + + logger.debug("post_urlencoded_get_json args: %s", args) + query_bytes = urllib.urlencode(args, True) + + response = yield self._create_request( + destination.encode("ascii"), + "POST", + path.encode("ascii"), + producer=FileBodyProducer(StringIO(urllib.urlencode(args))), + headers_dict={ + "Content-Type": ["application/x-www-form-urlencoded"] + } + ) + + body = yield readBody(response) + + defer.returnValue(json.loads(body)) + + +class CaptchaServerHttpClient(MatrixHttpClient): + """Separate HTTP client for talking to google's captcha servers""" + def _getEndpoint(self, reactor, destination): return matrix_endpoint(reactor, destination, timeout=10) - + + @defer.inlineCallbacks + def post_urlencoded_get_raw(self, destination, path, accept_partial=False, + args={}): + if destination in _destination_mappings: + destination = _destination_mappings[destination] + + query_bytes = urllib.urlencode(args, True) + + response = yield self._create_request( + destination.encode("ascii"), + "POST", + path.encode("ascii"), + producer=FileBodyProducer(StringIO(urllib.urlencode(args))), + headers_dict={ + "Content-Type": ["application/x-www-form-urlencoded"] + } + ) + + try: + body = yield readBody(response) + defer.returnValue(body) + except PartialDownloadError as e: + if accept_partial: + defer.returnValue(e.response) + else: + raise e def _print_ex(e): if hasattr(e, "reasons") and e.reasons: diff --git a/tests/handlers/test_directory.py b/tests/handlers/test_directory.py index dd5d85dde..0c31502dd 100644 --- a/tests/handlers/test_directory.py +++ b/tests/handlers/test_directory.py @@ -20,7 +20,7 @@ from twisted.internet import defer from mock import Mock from synapse.server import HomeServer -from synapse.http.client import HttpClient +from synapse.http.client import MatrixHttpClient from synapse.handlers.directory import DirectoryHandler from synapse.storage.directory import RoomAliasMapping @@ -95,7 +95,7 @@ class DirectoryTestCase(unittest.TestCase): query_type="directory", args={ "room_alias": "#another:remote", - HttpClient.RETRY_DNS_LOOKUP_FAILURES: False + MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES: False } ) From d694619a953adf6254e3960d2a4ec973d31dfcae Mon Sep 17 00:00:00 2001 From: David Baker Date: Thu, 2 Oct 2014 14:09:27 +0100 Subject: [PATCH 15/48] Fix ncorrect ports in documentation and add notes on how generate-config also generates certs bound to whatever hostname you give with --generate-config. SYN-87 #resolved --- README.rst | 5 +++-- synapse/config/_base.py | 3 ++- synctl | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/README.rst b/README.rst index 0459d5463..f40492b8a 100644 --- a/README.rst +++ b/README.rst @@ -51,6 +51,7 @@ To get up and running: - To run your own **private** homeserver on localhost:8008, generate a basic config file: ``./synctl start`` will give you instructions on how to do this. + For this purpose, you can use 'localhost' or your hostname as a server name. Once you've done so, running ``./synctl start`` again will start your private home sserver. You will find a webclient running at http://localhost:8008. Please use a recent Chrome or Firefox for now (or Safari if you don't need @@ -253,7 +254,7 @@ http://localhost:8080. Simply run:: Running The Demo Web Client =========================== -The homeserver runs a web client by default at http://localhost:8080. +The homeserver runs a web client by default at https://localhost:8448/. If this is the first time you have used the client from that browser (it uses HTML5 local storage to remember its config), you will need to log in to your @@ -273,7 +274,7 @@ account. Your name will take the form of:: Specify your desired localpart in the topmost box of the "Register for an account" form, and click the "Register" button. Hostnames can contain ports if -required due to lack of SRV records (e.g. @matthew:localhost:8080 on an +required due to lack of SRV records (e.g. @matthew:localhost:8448 on an internal synapse sandbox running on localhost) diff --git a/synapse/config/_base.py b/synapse/config/_base.py index 809f9c922..b3aeff327 100644 --- a/synapse/config/_base.py +++ b/synapse/config/_base.py @@ -123,7 +123,8 @@ class Config(object): # style mode markers into the file, to hint to people that # this is a YAML file. yaml.dump(config, config_file, default_flow_style=False) - print "A config file has been generated in %s (your server name is '%s'). Please review this file and customise it to your needs." % (config_args.config_path, config['server_name']) + print "A config file has been generated in %s for server name '%s') with corresponding SSL keys and self-signed certificates. Please review this file and customise it to your needs." % (config_args.config_path, config['server_name']) + print "If this server name is incorrect, you will need to regenerate the SSL certificates" sys.exit(0) return cls(args) diff --git a/synctl b/synctl index 7523fd3db..c227a9e1e 100755 --- a/synctl +++ b/synctl @@ -14,7 +14,7 @@ case "$1" in start) if [ ! -f "$CONFIGFILE" ]; then echo "No config file found" - echo "To generate a config file, run '$SYNAPSE -c $CONFIGFILE --generate-config'" + echo "To generate a config file, run '$SYNAPSE -c $CONFIGFILE --generate-config --server-name='" exit 1 fi From b2d41b1cd94d6278f5d3e090d45762fff4219d2f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:25:47 +0100 Subject: [PATCH 16/48] All room state is currently shared. --- docs/specification.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 0a421e9e3..f45c75fca 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -167,7 +167,7 @@ The following diagram shows an ``m.room.message`` event being sent in the room | matrix.org |<-------Federation------->| domain.com | +------------------+ +------------------+ | ................................. | - |______| Partially Shared State |_______| + |______| Shared State |_______| | Room ID: !qporfwt:matrix.org | | Servers: matrix.org, domain.com | | Members: | @@ -177,11 +177,10 @@ The following diagram shows an ``m.room.message`` event being sent in the room Federation maintains shared state between multiple home servers, such that when an event is sent to a room, the home server knows where to forward the event on -to, and how to process the event. Home servers do not need to have completely -shared state in order to participate in a room. State is scoped to a single -room, and federation ensures that all home servers have the information they -need, even if that means the home server has to request more information from -another home server before processing the event. +to, and how to process the event. State is scoped to a single room, and +federation ensures that all home servers have the information they need, even +if that means the home server has to request more information from another home +server before processing the event. Room Aliases ------------ From 574377636ee4eafba50580fc4d7a1d0793774332 Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Thu, 2 Oct 2014 14:09:15 +0100 Subject: [PATCH 17/48] Add a keyword argument to get_json to avoid retrying on DNS failures. Rather than passing MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES as a fake query string parameter --- synapse/federation/replication.py | 7 +++++-- synapse/federation/transport.py | 5 +++-- synapse/handlers/directory.py | 5 ++--- synapse/http/client.py | 9 +-------- tests/federation/test_federation.py | 5 +++-- tests/handlers/test_directory.py | 5 ++--- 6 files changed, 16 insertions(+), 20 deletions(-) diff --git a/synapse/federation/replication.py b/synapse/federation/replication.py index 96b82f00c..5f96f7999 100644 --- a/synapse/federation/replication.py +++ b/synapse/federation/replication.py @@ -159,7 +159,8 @@ class ReplicationLayer(object): return defer.succeed(None) @log_function - def make_query(self, destination, query_type, args): + def make_query(self, destination, query_type, args, + retry_on_dns_fail=True): """Sends a federation Query to a remote homeserver of the given type and arguments. @@ -174,7 +175,9 @@ class ReplicationLayer(object): a Deferred which will eventually yield a JSON object from the response """ - return self.transport_layer.make_query(destination, query_type, args) + return self.transport_layer.make_query( + destination, query_type, args, retry_on_dns_fail=retry_on_dns_fail + ) @defer.inlineCallbacks @log_function diff --git a/synapse/federation/transport.py b/synapse/federation/transport.py index afc777ec9..93296af20 100644 --- a/synapse/federation/transport.py +++ b/synapse/federation/transport.py @@ -193,13 +193,14 @@ class TransportLayer(object): @defer.inlineCallbacks @log_function - def make_query(self, destination, query_type, args): + def make_query(self, destination, query_type, args, retry_on_dns_fail): path = PREFIX + "/query/%s" % query_type response = yield self.client.get_json( destination=destination, path=path, - args=args + args=args, + retry_on_dns_fail=retry_on_dns_fail, ) defer.returnValue(response) diff --git a/synapse/handlers/directory.py b/synapse/handlers/directory.py index cec7737e0..a56830d52 100644 --- a/synapse/handlers/directory.py +++ b/synapse/handlers/directory.py @@ -18,7 +18,6 @@ from twisted.internet import defer from ._base import BaseHandler from synapse.api.errors import SynapseError -from synapse.http.client import MatrixHttpClient from synapse.api.events.room import RoomAliasesEvent import logging @@ -98,8 +97,8 @@ class DirectoryHandler(BaseHandler): query_type="directory", args={ "room_alias": room_alias.to_string(), - MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES: False - } + }, + retry_on_dns_fail=False, ) if result and "room_id" in result and "servers" in result: diff --git a/synapse/http/client.py b/synapse/http/client.py index e02cce564..57b49355f 100644 --- a/synapse/http/client.py +++ b/synapse/http/client.py @@ -183,7 +183,7 @@ class MatrixHttpClient(BaseHttpClient): defer.returnValue((response.code, body)) @defer.inlineCallbacks - def get_json(self, destination, path, args={}): + def get_json(self, destination, path, args={}, retry_on_dns_fail=True): """ Get's some json from the given host homeserver and path Args: @@ -203,13 +203,6 @@ class MatrixHttpClient(BaseHttpClient): """ logger.debug("get_json args: %s", args) - retry_on_dns_fail = True - if HttpClient.RETRY_DNS_LOOKUP_FAILURES in args: - # FIXME: This isn't ideal, but the interface exposed in get_json - # isn't comprehensive enough to give caller's any control over - # their connection mechanics. - retry_on_dns_fail = args.pop(HttpClient.RETRY_DNS_LOOKUP_FAILURES) - query_bytes = urllib.urlencode(args, True) logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail) diff --git a/tests/federation/test_federation.py b/tests/federation/test_federation.py index bb17e9aaf..d95b9013a 100644 --- a/tests/federation/test_federation.py +++ b/tests/federation/test_federation.py @@ -253,7 +253,7 @@ class FederationTestCase(unittest.TestCase): response = yield self.federation.make_query( destination="remote", query_type="a-question", - args={"one": "1", "two": "2"} + args={"one": "1", "two": "2"}, ) self.assertEquals({"your": "response"}, response) @@ -261,7 +261,8 @@ class FederationTestCase(unittest.TestCase): self.mock_http_client.get_json.assert_called_with( destination="remote", path="/_matrix/federation/v1/query/a-question", - args={"one": "1", "two": "2"} + args={"one": "1", "two": "2"}, + retry_on_dns_fail=True, ) @defer.inlineCallbacks diff --git a/tests/handlers/test_directory.py b/tests/handlers/test_directory.py index 0c31502dd..e10a49a8a 100644 --- a/tests/handlers/test_directory.py +++ b/tests/handlers/test_directory.py @@ -20,7 +20,6 @@ from twisted.internet import defer from mock import Mock from synapse.server import HomeServer -from synapse.http.client import MatrixHttpClient from synapse.handlers.directory import DirectoryHandler from synapse.storage.directory import RoomAliasMapping @@ -95,8 +94,8 @@ class DirectoryTestCase(unittest.TestCase): query_type="directory", args={ "room_alias": "#another:remote", - MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES: False - } + }, + retry_on_dns_fail=False, ) @defer.inlineCallbacks From ff553cc9dd4fe7c9203a7c4c414e5fa27348180a Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:26:58 +0100 Subject: [PATCH 18/48] Alias lookups return a server list. --- docs/specification.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/specification.rst b/docs/specification.rst index f45c75fca..e0609f707 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -190,7 +190,7 @@ Each room can also have multiple "Room Aliases", which looks like:: #room_alias:domain .. TODO - - Need to specify precise grammar for Room IDs + - Need to specify precise grammar for Room Aliases A room alias "points" to a room ID and is the human-readable label by which rooms are publicised and discovered. The room ID the alias is pointing to can @@ -200,6 +200,9 @@ over time to point to a different room ID. For this reason, Clients SHOULD resolve the room alias to a room ID once and then use that ID on subsequent requests. +When resolving a room alias the server will also respond with a list of servers +that are in the room that can be used to join via. + :: GET From 6860a18c12f9bedf59c70efca7ba2cda94f7b3dc Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:27:35 +0100 Subject: [PATCH 19/48] Be less alarmist about not using an ID server. --- docs/specification.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index e0609f707..26a9c17d9 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -241,8 +241,8 @@ authentication of the 3PID. Identity servers are also used to preserve the mapping indefinitely, by replicating the mappings across multiple ISes. Usage of an IS is not required in order for a client application to be part of -the Matrix ecosystem. However, by not using an IS, discovery of users is -greatly impacted. +the Matrix ecosystem. However, without one clients will not be able to look up +user IDs using 3PIDs. API Standards ------------- From cf3188352b898c91c9c3e0ef65c9fb120b00cc13 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:28:22 +0100 Subject: [PATCH 20/48] Fix default value and key names. --- docs/specification.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 26a9c17d9..e8790c2ad 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -721,7 +721,7 @@ options which can be set when creating a room: Description: A ``public`` visibility indicates that the room will be shown in the public room list. A ``private`` visibility will hide the room from the public room - list. Rooms default to ``public`` visibility if this key is not included. + list. Rooms default to ``private`` visibility if this key is not included. ``room_alias_name`` Type: @@ -2038,7 +2038,7 @@ For state updates: Description: The asserted power level of the user performing the update. -``min_update`` +``required_power_level`` Type: Integer Description: @@ -2056,7 +2056,7 @@ For state updates: Description: The PDU id of the update this replaces. -``user`` +``user_id`` Type: String Description: From 918e71adb7c645ee6edf3e796204f916cee26a42 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:31:21 +0100 Subject: [PATCH 21/48] Don't use spaces in example room alias --- docs/specification.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specification.rst b/docs/specification.rst index e8790c2ad..b5bfce59e 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -774,7 +774,7 @@ Example:: { "visibility": "public", - "room_alias_name": "the pub", + "room_alias_name": "thepub", "name": "The Grand Duke Pub", "topic": "All about happy hour" } From f368ad946ea987abfe5d7fc0d9f10971f8fecaef Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:33:26 +0100 Subject: [PATCH 22/48] m.room.ops_levels includes redact_level --- docs/specification.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index b5bfce59e..732b50c69 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -792,7 +792,7 @@ includes: - ``m.room.send_event_level`` : The power level required in order to send a message in this room. - ``m.room.ops_level`` : The power level required in order to kick or ban a - user from the room. + user from the room or redact an event in the room. See `Room Events`_ for more information on these events. @@ -1316,7 +1316,7 @@ prefixed with ``m.`` Type: State event JSON format: - ``{ "ban_level": , "kick_level": }`` + ``{ "ban_level": , "kick_level": , "redact_level": }`` Example: ``{ "ban_level": 5, "kick_level": 5 }`` Description: From 1561ef56ed6b514aca6c21f2151af5ba892a69d2 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:34:13 +0100 Subject: [PATCH 23/48] Remove note about assymetry of having left a room. Currently, if you leave a room you still appear in the members list. This is basically a bug with the current implementation/spec, rather than something that should happen. --- docs/specification.rst | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 732b50c69..081d949b4 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -947,11 +947,7 @@ directly by sending the following request to See the `Room events`_ section for more information on ``m.room.member``. Once a user has left a room, that room will no longer appear on the -|initialSync|_ API. Be aware that leaving a room is not equivalent to have -never been in that room. A user who has previously left a room still maintains -some residual state in that room. Their membership state will be marked as -``leave``. This contrasts with a user who has *never been invited or joined to -that room* who will not have any membership state for that room. +|initialSync|_ API. If all members in a room leave, that room becomes eligible for deletion. From b9cdc443d7acdd1f8095878e3c24b859f82ea31e Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Thu, 2 Oct 2014 14:37:30 +0100 Subject: [PATCH 24/48] Fix pyflakes errors --- synapse/handlers/register.py | 2 +- synapse/http/client.py | 10 ++-------- 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py index 266495056..df562aa76 100644 --- a/synapse/handlers/register.py +++ b/synapse/handlers/register.py @@ -176,7 +176,7 @@ class RegistrationHandler(BaseHandler): @defer.inlineCallbacks def _bind_threepid(self, creds, mxid): - httpCli = PlainHttpClient(self.hs) + httpCli = IdentityServerHttpClient(self.hs) data = yield httpCli.post_urlencoded_get_json( creds['idServer'], "/_matrix/identity/api/v1/3pid/bind", diff --git a/synapse/http/client.py b/synapse/http/client.py index 57b49355f..5c2fbd1f8 100644 --- a/synapse/http/client.py +++ b/synapse/http/client.py @@ -236,9 +236,6 @@ class IdentityServerHttpClient(BaseHttpClient): @defer.inlineCallbacks def post_urlencoded_get_json(self, destination, path, args={}): - if destination in _destination_mappings: - destination = _destination_mappings[destination] - logger.debug("post_urlencoded_get_json args: %s", args) query_bytes = urllib.urlencode(args, True) @@ -246,7 +243,7 @@ class IdentityServerHttpClient(BaseHttpClient): destination.encode("ascii"), "POST", path.encode("ascii"), - producer=FileBodyProducer(StringIO(urllib.urlencode(args))), + producer=FileBodyProducer(StringIO(query_bytes)), headers_dict={ "Content-Type": ["application/x-www-form-urlencoded"] } @@ -266,16 +263,13 @@ class CaptchaServerHttpClient(MatrixHttpClient): @defer.inlineCallbacks def post_urlencoded_get_raw(self, destination, path, accept_partial=False, args={}): - if destination in _destination_mappings: - destination = _destination_mappings[destination] - query_bytes = urllib.urlencode(args, True) response = yield self._create_request( destination.encode("ascii"), "POST", path.encode("ascii"), - producer=FileBodyProducer(StringIO(urllib.urlencode(args))), + producer=FileBodyProducer(StringIO(query_bytes)), headers_dict={ "Content-Type": ["application/x-www-form-urlencoded"] } From 82e278029c92c2460976142bb2aadb3b9dcdfa93 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:38:22 +0100 Subject: [PATCH 25/48] Remove incorrect reasons for empty PDU lists. --- docs/specification.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 081d949b4..dcb99d379 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1949,9 +1949,7 @@ is another list containing the EDUs. This key may be entirely absent if there are no EDUs to transfer. (* Normally the PDU list will be non-empty, but the server should cope with -receiving an "empty" transaction, as this is useful for informing peers of -other transaction IDs they should be aware of. This effectively acts as a push -mechanism to encourage peers to continue to replicate content.) +receiving an "empty" transaction.) PDUs and EDUs ------------- From 036333412d498b574075735ac5a49ce17036b11d Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Thu, 2 Oct 2014 14:38:53 +0100 Subject: [PATCH 26/48] Add todo notes --- docs/specification.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/specification.rst b/docs/specification.rst index dcb99d379..891470d5f 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -849,6 +849,7 @@ Joining rooms ------------- .. TODO-doc kegan - TODO: What does the home server have to do to join a user to a room? + See SPEC-30. Users need to join a room in order to send and receive events in that room. A user can join a room by making a request to |/join/|_ with:: @@ -1389,6 +1390,10 @@ prefixed with ``m.`` m.room.message msgtypes ----------------------- + +.. TODO-spec + How a client should handle unknown message types. + Each ``m.room.message`` MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as outlined below: From de38f54f22cf5d3fecbcf227fcd43744345c8817 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Thu, 2 Oct 2014 17:18:32 +0100 Subject: [PATCH 27/48] Document the Profile system --- docs/specification.rst | 109 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 101 insertions(+), 8 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index 0a421e9e3..f169cf02c 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1813,14 +1813,8 @@ Profiles .. NOTE:: This section is a work in progress. -.. TODO-doc +.. TODO-spec - Metadata extensibility - - Changing profile info generates m.presence events ("presencelike") - - keys on m.presence are optional, except presence which is required - - m.room.member is populated with the current displayname at that point in time. - - That is added by the HS, not you. - - Display name changes also generates m.room.member with displayname key f.e. room - the user is in. Internally within Matrix users are referred to by their user ID, which is typically a compact unique identifier. Profiles grant users the ability to see @@ -1831,7 +1825,106 @@ age or location. A Profile consists of a display name, an avatar picture, and a set of other metadata fields that the user may wish to publish (email address, phone numbers, website URLs, etc...). This specification puts no requirements on the -display name other than it being a valid unicode string. +display name other than it being a valid unicode string. Avatar images are not +stored directly; instead the home server stores an ``http``-scheme URL where +clients may fetch it from. + +Client API +---------- +The client API for profile management consists of the following REST calls. + +Fetching a user account displayname:: + + GET $PREFIX/profile/:user_id/displayname + + Returned content: JSON object containing the following keys: + displayname: string of freeform text + +This call may be used to fetch the user's own displayname or to query the name +of other users; either locally or on remote systems hosted on other home +servers. + +Setting a new displayname:: + + PUT $PREFIX/profile/:user_id/displayname + + Content: JSON object containing the following keys: + displayname: string of freeform text + +Fetching a user account avatar URL:: + + GET $PREFIX/profile/:user_id/avatar_url + + Returned content: JSON object containing the following keys: + avatar_url: string containing an http-scheme URL + +As with displayname, this call may be used to fetch either the user's own, or +other users' avatar URL. + +Setting a new avatar URL:: + + PUT $PREFIX/profile/:user_id/avatar_url + + Content: JSON object containing the following keys: + avatar_url: string containing an http-scheme URL + +Fetching combined account profile information:: + + GET $PREFIX/profile/:user_id + + Returned content: JSON object containing the following keys: + displayname: string of freeform text + avatar_url: string containing an http-scheme URL + +At the current time, this API simply returns the displayname and avatar URL +information, though it is intended to return more fields about the user's +profile once they are defined. Client implementations should take care not to +expect that these are the only two keys returned as future versions of this +specification may yield more keys here. + +Server API +---------- +The server API for profiles is based entirely on the following Federation +Queries. There are no additional EDU or PDU types involved, other than the +implicit ``m.presence`` and ``m.room.member`` events (see section below). + +Querying profile information:: + + Query type: profile + + Arguments: + user_id: the ID of the user whose profile to return + field: (optional) string giving a field name + + Returns: JSON object containing the following keys: + displayname: string of freeform text + avatar_url: string containing an http-scheme URL + +If the query contains the optional ``field`` key, it should give the name of a +result field. If such is present, then the result should contain only a field +of that name, with no others present. If not, the result should contain as much +of the user's profile as the home server has available and can make public. + +Events on Change of Profile Information +--------------------------------------- +Because the profile displayname and avatar information are likely to be used in +many places of a client's display, changes to these fields cause an automatic +propagation event to occur, informing likely-interested parties of the new +values. This change is conveyed using two separate mechanisms: + + - a ``m.room.member`` event is sent to every room the user is a member of, + to update the ``displayname`` and ``avatar_url``. + - a presence status update is sent, again containing the new values of the + ``displayname`` and ``avatar_url`` keys, in addition to the required + ``presence`` key containing the current presence state of the user. + +Both of these should be done automatically by the home server when a user +successfully changes their displayname or avatar URL fields. + +Additionally, when home servers emit room membership events for their own +users, they should include the displayname and avatar URL fields in these +events so that clients already have these details to hand, and do not have to +perform extra roundtrips to query it. Identity From bc1d685a8c32f4b4f0f94e58fed94ed3a8a8beb8 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Thu, 2 Oct 2014 18:00:31 +0100 Subject: [PATCH 28/48] Remove TODO note about VoIP events as they now have their own entire section --- docs/specification.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index d07a667a5..b6cb07058 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1182,9 +1182,6 @@ Room Events .. NOTE:: This section is a work in progress. -.. TODO-doc dave? - - voip events? - This specification outlines several standard event types, all of which are prefixed with ``m.`` From 1aa5cc917874c150e3e4af2483fe86213a2a067f Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Thu, 2 Oct 2014 18:11:04 +0100 Subject: [PATCH 29/48] Federation protocol URLs should have an H2 heading, not H1 --- docs/specification.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specification.rst b/docs/specification.rst index b6cb07058..dfaf46017 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -2203,7 +2203,7 @@ destination home server names, and the actual nested content. Protocol URLs -============= +------------- .. WARNING:: This section may be misleading or inaccurate. From 7e1437c6b17c5139303f330dc92f55847b6f3807 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 3 Oct 2014 10:34:29 +0100 Subject: [PATCH 30/48] Add more information to TODOs. Explain m.room.join_rules. --- docs/specification.rst | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index dfaf46017..a58ec6645 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -847,9 +847,8 @@ defined in the following state events: Joining rooms ------------- -.. TODO-doc kegan - - TODO: What does the home server have to do to join a user to a room? - See SPEC-30. +.. TODO-doc What does the home server have to do to join a user to a room? + - See SPEC-30. Users need to join a room in order to send and receive events in that room. A user can join a room by making a request to |/join/|_ with:: @@ -886,20 +885,21 @@ received an invite. Inviting users -------------- -.. TODO-doc kegan - - Can invite users to a room if the room config key TODO is set to TODO. Must have required power level. +.. TODO-doc Invite-join dance - Outline invite join dance. What is it? Why is it required? How does it work? - What does the home server have to do? - - TODO: In what circumstances will direct member editing NOT be equivalent to ``/invite``? The purpose of inviting users to a room is to notify them that the room exists so they can choose to become a member of that room. Some rooms require that all users who join a room are previously invited to it (an "invite-only" room). Whether a given room is an "invite-only" room is determined by the room config -key ``TODO``. It can have one of the following values: +key ``m.room.join_rules``. It can have one of the following values: - - TODO Room config invite only value explanation - - TODO Room config free-to-join value explanation +``public`` + This room is free for anyone to join without an invite. + +``invite`` + This room can only be joined if you were invited. Only users who have a membership state of ``join`` in a room can invite new users to said room. The person being invited must not be in the ``join`` state @@ -924,9 +924,14 @@ See the `Room events`_ section for more information on ``m.room.member``. Leaving rooms ------------- -.. TODO-spec kegan - - TODO: Grace period before deletion? - - TODO: Under what conditions should a room NOT be purged? +.. TODO-spec - HS deleting rooms they are no longer a part of. Not implemented. + - This is actually Very Tricky. If all clients a HS is serving leave a room, + the HS will no longer get any new events for that room, because the servers + who get the events are determined on the *membership list*. There should + probably be a way for a HS to lurk on a room even if there are 0 of their + members in the room. + - Grace period before deletion? + - Under what conditions should a room NOT be purged? A user can leave a room to stop receiving events for that room. A user must @@ -1078,6 +1083,10 @@ presence events will also be returned. There are two APIs provided: .. TODO-doc kegan - TODO: JSON response format for both types - TODO: when would you use global? when would you use scoped? + - Room-scoped initial sync is Very Tricky because typically people would + want to sync the room then listen for any new content from that point + onwards. The event stream cannot do this for a single room currently. + Not sure if room-scoped initial sync should be included at this time. Getting events for a room ------------------------- From ba11afafb908fa712aa347e956fec2af9737ad2c Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 3 Oct 2014 14:39:58 +0100 Subject: [PATCH 31/48] Flesh out room alias section. --- docs/specification.rst | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index a58ec6645..e6b85dba3 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -705,9 +705,6 @@ Rooms Creation -------- -.. TODO kegan - - TODO-spec: Key for invite these users? - To create a room, a client has to use the |createRoom|_ API. There are various options which can be set when creating a room: @@ -801,12 +798,35 @@ Modifying aliases .. NOTE:: This section is a work in progress. -.. TODO-doc kegan - - path to edit aliases - - PUT /directory/room/ { room_id : foo } - - GET /directory/room/ { room_id : foo, servers: [a.com, b.com] } - - format when retrieving list of aliases. NOT complete list. - - format for adding/removing aliases. +Room aliases can be created by sending a ``PUT /directory/room/``:: + + { + "room_id": + } + +They can be deleted by sending a ``DELETE /directory/room/`` with +no content. Only some privileged users may be able to delete room aliases, e.g. +server admins, the creator of the room alias, etc. This specification does not +outline the privilege level required for deleting room aliases. + +Rooms store a *partial* list of room aliases via the ``m.room.aliases`` state +event. This alias list is partial because it cannot guarantee that the alias +list is in any way accurate or up-to-date, as room aliases can point to +different room IDs over time. Crucially, the aliases in this event are +**purely informational** and SHOULD NOT be treated as accurate. They SHOULD +be checked before they are used or shared with another user. If a room +appears to have a room alias of ``#alias:example.com``, this SHOULD be checked +to make sure that the room's ID matches the ``room_id`` returned from the +request. + +Room aliases can be checked in the same way they are resolved; by sending a +``GET /directory/room/``:: + + { + "room_id": , + "servers": [ , , ] + } + Permissions ----------- From ca0e8dedfba6dd643e9f996baca052b7a0aac401 Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Fri, 3 Oct 2014 14:45:42 +0100 Subject: [PATCH 32/48] Clarify how m.room.alias event works --- docs/specification.rst | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index a58ec6645..d8e754701 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1338,10 +1338,32 @@ prefixed with ``m.`` Example: ``{ "aliases": ["#foo:example.com"] }`` Description: - A server `may` inform the room that it has added or removed an alias for - the room. This is purely for informational purposes and may become stale. - Clients `should` check that the room alias is still valid before using it. - The ``state_key`` of the event is the homeserver which owns the room alias. + This event is sent by a homeserver directly to inform of changes to the + list of aliases it knows about for that room. As a special-case, the + ``state_key`` of the event is the homeserver which owns the room alias. + For example, an event might look like:: + + { + "type": "m.room.aliases", + "event_id": "012345678ab", + "room_id": "!xAbCdEfG:example.com", + "state_key": "example.com", + "content": { + "aliases": ["#foo:example.com"] + } + } + + The event contains the full list of aliases now stored by the home server + that emitted it; additions or deletions are not explicitly mentioned as + being such. The entire set of known aliases for the room is then the union + of the individual lists declared by all such keys, one from each home + server holding at least one alias. + + Clients `should` check the validity of any room alias given in this list + before presenting it to the user as trusted fact. The lists given by this + event should be considered simply as advice on which aliases might exist, + for which the client can perform the lookup to confirm whether it receives + the correct room ID. ``m.room.message`` Summary: From 02a44664b90a81f4a7d880fdeef1262cbabe4168 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 3 Oct 2014 17:38:30 +0100 Subject: [PATCH 33/48] More spec work. --- docs/specification.rst | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index f2e973de3..eff9dcd03 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -216,10 +216,6 @@ that are in the room that can be used to join via. | #golf >> !wfeiofh:sport.com | | #bike >> !4rguxf:matrix.org | |________________________________| - -.. TODO kegan - - show the actual API rather than pseudo-API? - Identity -------- @@ -793,8 +789,8 @@ includes: See `Room Events`_ for more information on these events. -Modifying aliases ------------------ +Room aliases +------------ .. NOTE:: This section is a work in progress. @@ -834,9 +830,6 @@ Permissions This section is a work in progress. .. TODO-doc kegan - - What is a power level? How do they work? Defaults / required levels for X. How do they change - as people join and leave rooms? What do you do if you get a clash? Examples. - - List all actions which use power levels (sending msgs, inviting users, banning people, etc...) - Room config - what is the event and what are the keys/values and explanations for them. Link through to respective sections where necessary. How does this tie in with permissions, e.g. give example of creating a read-only room. @@ -847,12 +840,15 @@ action in a room a user must have a suitable power level. Power levels for users are defined in ``m.room.power_levels``, where both a default and specific users' power levels can be set. By default all users have a power level of 0, other than the room creator whose power level defaults to -100. Power levels for users are tracked per-room even if the user is not -present in the room. +100. Users can grant other users increased power levels up to their own power +level. For example, user A with a power level of 50 could increase the power +level of user B to a maximum of level 50. Power levels for users are tracked +per-room even if the user is not present in the room. State events may contain a ``required_power_level`` key, which indicates the minimum power a user must have before they can update that state key. The only -exception to this is when a user leaves a room. +exception to this is when a user leaves a room, which revokes the user's right +to update state events in that room. To perform certain actions there are additional power level requirements defined in the following state events: @@ -861,8 +857,9 @@ defined in the following state events: events. Defaults to 50. - ``m.room.add_state_level`` defines the minimum level for adding new state, rather than updating existing state. Defaults to 50. -- ``m.room.ops_level`` defines the minimum levels to ban and kick other users. - This defaults to a kick and ban levels of 50 each. +- ``m.room.ops_level`` defines the minimum ``ban_level`` and ``kick_level`` to + ban and kick other users respectively. This defaults to a kick and ban levels + of 50 each. Joining rooms From 78a3f43d9d6be03c16ea478671e45de7d3a7dcea Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 6 Oct 2014 09:23:19 +0100 Subject: [PATCH 34/48] swagger: Added DELETE method for directory server. --- .../swagger_matrix/api-docs-directory | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/client-server/swagger_matrix/api-docs-directory b/docs/client-server/swagger_matrix/api-docs-directory index ce12be8c9..5dda58065 100644 --- a/docs/client-server/swagger_matrix/api-docs-directory +++ b/docs/client-server/swagger_matrix/api-docs-directory @@ -48,6 +48,22 @@ "paramType": "body" } ] + }, + { + "method": "DELETE", + "summary": "Removes a mapping of room alias to room ID.", + "notes": "Only privileged users can perform this action.", + "type": "void", + "nickname": "remove_room_alias", + "parameters": [ + { + "name": "roomAlias", + "description": "The room alias to remove.", + "required": true, + "type": "string", + "paramType": "path" + } + ] } ] } From 51276c60bf113400f38299fc813511642f60d510 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 6 Oct 2014 10:32:04 +0100 Subject: [PATCH 35/48] Add information about the initialSync API. Outline and describe the keys from the initial sync API. Hide room-scoped initial sync API for now as it is not implemented and needs more thought before it can be specced. --- docs/specification.rst | 54 +++++++++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 8 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index eff9dcd03..edd8f5dbf 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1089,21 +1089,59 @@ When a client logs in, they may have a list of rooms which they have already joined. These rooms may also have a list of events associated with them. The purpose of 'syncing' is to present the current room and event information in a convenient, compact manner. The events returned are not limited to room events; -presence events will also be returned. There are two APIs provided: +presence events will also be returned. A single syncing API is provided: - |initialSync|_ : A global sync which will present room and event information for all rooms the user has joined. +.. TODO-spec room-scoped initial sync - |/rooms//initialSync|_ : A sync scoped to a single room. Presents room and event information for this room only. + - Room-scoped initial sync is Very Tricky because typically people would + want to sync the room then listen for any new content from that point + onwards. The event stream cannot do this for a single room currently. + As a result, commenting room-scoped initial sync at this time. -.. TODO-doc kegan - - TODO: JSON response format for both types - - TODO: when would you use global? when would you use scoped? - - Room-scoped initial sync is Very Tricky because typically people would - want to sync the room then listen for any new content from that point - onwards. The event stream cannot do this for a single room currently. - Not sure if room-scoped initial sync should be included at this time. +The |initialSync|_ API contains the following keys: + +``presence`` + Description: + Contains a list of presence information for users the client is interested + in. + Format: + A JSON array of ``m.presence`` events. + +``end`` + Description: + Contains an event stream token which can be used with the `Event Stream`_. + Format: + A string containing the event stream token. + +``rooms`` + Description: + Contains a list of room information for all rooms the client has joined, + and limited room information on rooms the client has been invited to. + Format: + A JSON array containing Room Information JSON objects. + +Room Information: + Description: + Contains all state events for the room, along with a limited amount of + the most recent non-state events, configured via the ``limit`` query + parameter. Also contains additional keys with room metadata, such as the + ``room_id`` and the client's ``membership`` to the room. + Format: + A JSON object with the following keys: + ``room_id`` + A string containing the ID of the room being described. + ``membership`` + A string representing the client's membership status in this room. + ``messages`` + An event stream JSON object containing a ``chunk`` of recent non-state + events, along with an ``end`` token. *NB: The name of this key will be + changed in a later version.* + ``state`` + A JSON array containing all the current state events for this room. Getting events for a room ------------------------- From 94982392bef6654199160732944257c4117c7a40 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 6 Oct 2014 12:41:48 +0100 Subject: [PATCH 36/48] Clarify room permission / power level information. --- docs/specification.rst | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index edd8f5dbf..eea892c90 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -829,21 +829,24 @@ Permissions .. NOTE:: This section is a work in progress. -.. TODO-doc kegan - - Room config - what is the event and what are the keys/values and explanations for them. - Link through to respective sections where necessary. How does this tie in with permissions, e.g. - give example of creating a read-only room. - Permissions for rooms are done via the concept of power levels - to do any -action in a room a user must have a suitable power level. +action in a room a user must have a suitable power level. Power levels are +stored as state events in a given room. Power levels for users are defined in ``m.room.power_levels``, where both a -default and specific users' power levels can be set. By default all users have -a power level of 0, other than the room creator whose power level defaults to -100. Users can grant other users increased power levels up to their own power -level. For example, user A with a power level of 50 could increase the power -level of user B to a maximum of level 50. Power levels for users are tracked -per-room even if the user is not present in the room. +default and specific users' power levels can be set:: + + { + "": , + "": , + "default": 0 + } + +By default all users have a power level of 0, other than the room creator whose +power level defaults to 100. Users can grant other users increased power levels +up to their own power level. For example, user A with a power level of 50 could +increase the power level of user B to a maximum of level 50. Power levels for +users are tracked per-room even if the user is not present in the room. State events may contain a ``required_power_level`` key, which indicates the minimum power a user must have before they can update that state key. The only @@ -853,10 +856,10 @@ to update state events in that room. To perform certain actions there are additional power level requirements defined in the following state events: -- ``m.room.send_event_level`` defines the minimum level for sending non-state - events. Defaults to 50. -- ``m.room.add_state_level`` defines the minimum level for adding new state, - rather than updating existing state. Defaults to 50. +- ``m.room.send_event_level`` defines the minimum ``level`` for sending + non-state events. Defaults to 50. +- ``m.room.add_state_level`` defines the minimum ``level`` for adding new + state, rather than updating existing state. Defaults to 50. - ``m.room.ops_level`` defines the minimum ``ban_level`` and ``kick_level`` to ban and kick other users respectively. This defaults to a kick and ban levels of 50 each. From aaf1d499bfb648b8fa24eb9de0ca9772b80823a3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 6 Oct 2014 13:18:52 +0100 Subject: [PATCH 37/48] Add more section headings. --- docs/specification.rst | 42 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-) diff --git a/docs/specification.rst b/docs/specification.rst index eea892c90..de3d04f24 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -2378,6 +2378,15 @@ SRV Records .. TODO-doc - Why it is needed +State Conflict Resolution +------------------------- +.. NOTE:: + This section is a work in progress. + +.. TODO-doc + - How do conflicts arise (diagrams?) + - How are they resolved (incl tie breaks) + - How does this work with deleting current state Security ======== @@ -2385,6 +2394,29 @@ Security .. NOTE:: This section is a work in progress. +Server-Server Authentication +---------------------------- + +.. TODO-doc + - Why is this needed. + - High level overview of process. + - Transaction/PDU signing + - How does this work with redactions? (eg hashing required keys only) + +End-to-End Encryption +--------------------- + +.. TODO-doc + - Why is this needed. + - Overview of process + - Implementation + +Lawful Interception +------------------- + +Key Escrow Servers +~~~~~~~~~~~~~~~~~~ + Threat Model ------------ @@ -2531,10 +2563,6 @@ have to wait in milliseconds before they can try again. - Surely we should recommend an algorithm for the rate limiting, rather than letting every homeserver come up with their own idea, causing totally unpredictable performance over federated rooms? - - crypto (s-s auth) - - E2E - - Lawful intercept + Key Escrow - TODO Mark Policy Servers @@ -2543,7 +2571,11 @@ Policy Servers This section is a work in progress. .. TODO-spec - We should mention them in the Architecture section at least... + We should mention them in the Architecture section at least: how they fit + into the picture. + +Enforcing policies +------------------ Content repository From 3ef2c946d58034173f4fb54b4708ac108f1cf3f7 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 6 Oct 2014 14:52:46 +0100 Subject: [PATCH 38/48] Update JSFiddles/how-to to support the new registration format. --- docs/client-server/howto.rst | 2 +- jsfiddles/example_app/demo.js | 2 +- jsfiddles/register_login/demo.js | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/client-server/howto.rst b/docs/client-server/howto.rst index ec941eddd..9913e8db1 100644 --- a/docs/client-server/howto.rst +++ b/docs/client-server/howto.rst @@ -31,7 +31,7 @@ Registration The aim of registration is to get a user ID and access token which you will need when accessing other APIs:: - curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/register" + curl -XPOST -d '{"user":"example", "password":"wordpass", "type":"m.login.password"}' "http://localhost:8008/_matrix/client/api/v1/register" { "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", diff --git a/jsfiddles/example_app/demo.js b/jsfiddles/example_app/demo.js index ad79fcca2..13c9c2b33 100644 --- a/jsfiddles/example_app/demo.js +++ b/jsfiddles/example_app/demo.js @@ -110,7 +110,7 @@ $('.register').live('click', function() { url: "http://localhost:8008/_matrix/client/api/v1/register", type: "POST", contentType: "application/json; charset=utf-8", - data: JSON.stringify({ user_id: user, password: password }), + data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), dataType: "json", success: function(data) { onLoggedIn(data); diff --git a/jsfiddles/register_login/demo.js b/jsfiddles/register_login/demo.js index fffa9e055..2e6957b63 100644 --- a/jsfiddles/register_login/demo.js +++ b/jsfiddles/register_login/demo.js @@ -14,7 +14,7 @@ $('.register').live('click', function() { url: "http://localhost:8008/_matrix/client/api/v1/register", type: "POST", contentType: "application/json; charset=utf-8", - data: JSON.stringify({ user_id: user, password: password }), + data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), dataType: "json", success: function(data) { showLoggedIn(data); From c72074b48e1714c2c41f71b0f81474ecafa29144 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 6 Oct 2014 14:57:26 +0100 Subject: [PATCH 39/48] Clarify how-to some more. --- docs/client-server/howto.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/client-server/howto.rst b/docs/client-server/howto.rst index 9913e8db1..37346224a 100644 --- a/docs/client-server/howto.rst +++ b/docs/client-server/howto.rst @@ -39,14 +39,15 @@ when accessing other APIs:: "user_id": "@example:localhost" } -NB: If a ``user_id`` is not specified, one will be randomly generated for you. +NB: If a ``user`` is not specified, one will be randomly generated for you. If you do not specify a ``password``, you will be unable to login to the account if you forget the ``access_token``. Implementation note: The matrix specification does not enforce how users register with a server. It just specifies the URL path and absolute minimum keys. The reference home server uses a username/password to authenticate user, -but other home servers may use different methods. +but other home servers may use different methods. This is why you need to +specify the ``type`` of method. Login ----- From 2fc00508fb23800a3d4494680947a7f402910dd1 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Mon, 6 Oct 2014 17:34:44 +0100 Subject: [PATCH 40/48] Add quick and dirty doc about state resolution --- docs/state_resolution.rst | 51 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/state_resolution.rst diff --git a/docs/state_resolution.rst b/docs/state_resolution.rst new file mode 100644 index 000000000..fec290dd7 --- /dev/null +++ b/docs/state_resolution.rst @@ -0,0 +1,51 @@ +State Resolution +================ +This section describes why we need state resolution and how it works. + + +Motivation +----------- +We want to be able to associate some shared state with rooms, e.g. a room name +or members list. This is done by having a current state dictionary that maps +from the pair event type and state key to an event. + +However, since the servers involved in the room are distributed we need to be +able to handle the case when two (or more) servers try and update the state at +the same time. This is done via the state resolution algorithm. + + +State Tree +------------ +State events contain a reference to the state it is trying to replace. These +relations form a tree where the current state is one of the leaf nodes. + +Note that state events are events, and so are part of the PDU graph. Thus we +can be sure that (modulo the internet being particularly broken) we will see +all state events eventually. + + +Algorithm requirements +---------------------- +We want the algorithm to have the following properties: +- Since we aren't guaranteed what order we receive state events in, except that + we see parents before children, the state resolution algorithm must not depend + on the order and must always come to the same result. +- If we receive a state event whose parent is the current state, then the + algorithm will select it. +- The algorithm does not depend on internal state, ensuring all servers should + come to the same decision. + +These three properties mean it is enough to keep track of the current state and +compare it with any new proposed state, rather than having to keep track of all +the leafs of the tree and recomputing across the entire state tree. + + +Current Implementation +---------------------- +The current implementation works as follows: Upon receipt of a newly proposed +state change we first find the common ancestor. Then we take the maximum +across each branch of the users' power levels, if one is higher then it is +selected as the current state. Otherwise, we check if one chain is longer than +the other, if so we choose that one. If that also fails, then we concatenate +all the pdu ids and take a SHA1 hash and compare them to select a common +ancestor. From 9ac53ef8cfe595b5606b97c52fb4aaf6fb35e9df Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 7 Oct 2014 11:38:02 +0100 Subject: [PATCH 41/48] SPEC-3: First hack at defining some of the various event related concepts --- docs/definitions.rst | 53 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 docs/definitions.rst diff --git a/docs/definitions.rst b/docs/definitions.rst new file mode 100644 index 000000000..b0f95ae9d --- /dev/null +++ b/docs/definitions.rst @@ -0,0 +1,53 @@ +Definitions +=========== + +# *Event* -- A JSON object that represents a piece of information to be +distributed to the the room. The object includes a payload and metadata, +including a `type` used to indicate what the payload is for and how to process +them. It also includes one or more references to previous events. + +# *Event graph* -- Events and their references to previous events form a +directed acyclic graph. All events must be a descendant of the first event in a +room, except for a few special circumstances. + +# *State event* -- A state event is an event that has a non-null string valued +`state_key` field. It may also include a `prev_state` key referencing exactly +one state event with the same type and state key, in the same event graph. + +# *State tree* -- A state tree is a tree formed by a collection of state events +that have the same type and state key (all in the same event graph. + +# *State resolution algorithm* -- An algorithm that takes a state tree as input +and selects a single leaf node. + +# *Current state event* -- The leaf node of a given state tree that has been +selected by the state resolution algorithm. + +# *Room state* / *state dictionary* / *current state* -- A mapping of the pair +(event type, state key) to the current state event for that pair. + +# *Room* -- An event graph and its associated state dictionary. An event is in +the room if it is part of the event graph. + +# *Topological ordering* -- The partial ordering that can be extracted from the +event graph due to it being a DAG. + +(The state definitions are purposely slightly ill-defined, since if we allow +deleting events we might end up with multiple state trees for a given event +type and state key pair.) + +Federation specific +------------------- +# *(Persistent data unit) PDU* -- An encoding of an event for distribution of +the server to server protocol. + +# *(Ephemeral data unit) EDU* -- A piece of information that is sent between +servers and doesn't encode an event. + +Client specific +--------------- +# *Child events* -- Events that reference a single event in the same room +independently of the event graph. + +# *Collapsed events* -- Events that have all child events that reference it +included in the JSON object. From 917af4705bcf425f04a5f17f61edfda00651017b Mon Sep 17 00:00:00 2001 From: "Paul \"LeoNerd\" Evans" Date: Tue, 7 Oct 2014 16:22:57 +0100 Subject: [PATCH 42/48] Clarify that room alias domain names will be server-scoped; nonlocal edits are unliekly to work but nonlocal lookups will --- docs/specification.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/specification.rst b/docs/specification.rst index de3d04f24..84722aa28 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -805,6 +805,11 @@ no content. Only some privileged users may be able to delete room aliases, e.g. server admins, the creator of the room alias, etc. This specification does not outline the privilege level required for deleting room aliases. +As room aliases are scoped to a particular home server domain name, it is +likely that a home server will reject attempts to maintain aliases on other +domain names. This specification does not provide a way for home servers to +send update requests to other servers. + Rooms store a *partial* list of room aliases via the ``m.room.aliases`` state event. This alias list is partial because it cannot guarantee that the alias list is in any way accurate or up-to-date, as room aliases can point to @@ -823,6 +828,9 @@ Room aliases can be checked in the same way they are resolved; by sending a "servers": [ , , ] } +Home servers can respond to resolve requests for aliases on other domains than +their own by using the federation API to ask other domain name home servers. + Permissions ----------- From 6045bd89fb50c1b45ffb2a373a8ec5f5b93dd236 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 8 Oct 2014 15:16:03 +0100 Subject: [PATCH 43/48] Break unit test. --- tests/test_state.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_state.py b/tests/test_state.py index b1624f0b2..c000eedc0 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -71,7 +71,7 @@ class StateTestCase(unittest.TestCase): is_new = yield self.state.handle_new_state(new_pdu) - self.assertTrue(is_new) + self.assertFalse(is_new) self.persistence.get_unresolved_state_tree.assert_called_once_with( new_pdu From 72aef114ab1201f5a5cd734220c9ec738c4e2910 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 8 Oct 2014 15:18:19 +0100 Subject: [PATCH 44/48] Fix unit test. --- tests/test_state.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_state.py b/tests/test_state.py index c000eedc0..b1624f0b2 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -71,7 +71,7 @@ class StateTestCase(unittest.TestCase): is_new = yield self.state.handle_new_state(new_pdu) - self.assertFalse(is_new) + self.assertTrue(is_new) self.persistence.get_unresolved_state_tree.assert_called_once_with( new_pdu From 83c53113af50b8c838e5a6795f049f63178641c9 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 9 Oct 2014 15:51:05 +0100 Subject: [PATCH 45/48] Break a test. --- tests/test_state.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_state.py b/tests/test_state.py index b1624f0b2..c000eedc0 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -71,7 +71,7 @@ class StateTestCase(unittest.TestCase): is_new = yield self.state.handle_new_state(new_pdu) - self.assertTrue(is_new) + self.assertFalse(is_new) self.persistence.get_unresolved_state_tree.assert_called_once_with( new_pdu From 3db09c4d1576c96c41c825f04aab6deeaadb18de Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 9 Oct 2014 15:53:40 +0100 Subject: [PATCH 46/48] Still broken. --- tests/test_state.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_state.py b/tests/test_state.py index c000eedc0..5714d85b3 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -71,7 +71,8 @@ class StateTestCase(unittest.TestCase): is_new = yield self.state.handle_new_state(new_pdu) - self.assertFalse(is_new) + self.assertTrue(is_new) + self.assertTrue(False) self.persistence.get_unresolved_state_tree.assert_called_once_with( new_pdu From 868eb478d8b0a5691c540a74a06bafe8864abf9a Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 9 Oct 2014 15:55:07 +0100 Subject: [PATCH 47/48] Fixed test. --- tests/test_state.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_state.py b/tests/test_state.py index 5714d85b3..b1624f0b2 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -72,7 +72,6 @@ class StateTestCase(unittest.TestCase): is_new = yield self.state.handle_new_state(new_pdu) self.assertTrue(is_new) - self.assertTrue(False) self.persistence.get_unresolved_state_tree.assert_called_once_with( new_pdu From 66df7f1aaf73a4f5774aee705de4dac2c8d8ad5e Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 12 Oct 2014 00:00:37 +0100 Subject: [PATCH 48/48] remove wishlist in favour of jira --- WISHLIST.rst | 9 --------- 1 file changed, 9 deletions(-) delete mode 100644 WISHLIST.rst diff --git a/WISHLIST.rst b/WISHLIST.rst deleted file mode 100644 index a0713f196..000000000 --- a/WISHLIST.rst +++ /dev/null @@ -1,9 +0,0 @@ -Broad-sweeping stuff which would be nice to have -================================================ - - - Additional SQL backends beyond sqlite - - homeserver implementation in go - - homeserver implementation in node.js - - client SDKs - - libpurple library - - irssi plugin?