Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
7,479 Commits 5 Branches 723 Tags 124 MiB
aa7b890cfe
Commit Graph

5258 Commits

Author SHA1 Message Date
Erik Johnston
b260f92936 Ignore AlreadyCalled errors on timer cancel 2016-07-31 16:00:12 +01:00
David Baker
271d3e7865 Fix adding emails on registration 
Synapse was not adding email addresses to accounts registered with an email address, due to too many different variables called 'result'. Rename both of them. Also remove the defer.returnValue() with no params because that's not a thing.
 2016-07-29 15:25:24 +01:00
Paul Evans
18b7eb830b Merge pull request #958 from matrix-org/paul/SYN-738 
Forbid non-ASes from registering users whose names begin with '_'
 2016-07-29 14:10:45 +01:00
Erik Johnston
2a9ce8c422 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:33:46 +01:00
Erik Johnston
cbea0c7044 Merge pull request #964 from matrix-org/erikj/fed_join_fix 
Handle the case of missing auth events when joining a room
 2016-07-29 11:33:16 +01:00
Erik Johnston
5aa024e501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:24:56 +01:00
Erik Johnston
c51a52f300 Mention that func will fetch auth events 2016-07-29 11:17:04 +01:00
Erik Johnston
3d13c3a295 Update docstring 2016-07-29 10:45:05 +01:00
Mark Haines
8dad08a950 Fix SQL to supply arguments in the same order 2016-07-29 09:57:13 +01:00
Mark Haines
0a7d3cd00f Create separate methods for getting messages to push 
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
 2016-07-28 20:24:24 +01:00
Erik Johnston
ec8b217722 Add destination retry to slave store 2016-07-28 17:35:53 +01:00
Erik Johnston
76b89d0edb Add slace storage functions for public room list 2016-07-28 17:03:40 +01:00
Kegan Dougal
370135ad0b Comment get_unread_push_actions_for_user_in_range function 2016-07-28 16:47:37 +01:00
Erik Johnston
0fcbca531f Add get_auth_chain to slave store 2016-07-28 16:36:28 +01:00
Erik Johnston
1e2740caab Handle the case of missing auth events when joining a room 2016-07-28 16:08:33 +01:00
Erik Johnston
6ede23ff1b Add more key storage funcs into slave store 2016-07-28 15:41:26 +01:00
Erik Johnston
5fb41a955c Merge branch 'release-v0.17.0' of github.com:matrix-org/synapse into develop 2016-07-28 14:56:32 +01:00
Mark Haines
4329f20e3f Merge pull request #962 from matrix-org/markjh/retry 
Fix retry utils to check if the exception is a subclass of CME
 2016-07-28 11:12:19 +01:00
Erik Johnston
a285194021 Merge branch 'erikj/key_client_fix' of github.com:matrix-org/synapse into release-v0.17.0 2016-07-28 10:47:06 +01:00
Mark Haines
bf81e38d36 Fix retry utils to check if the exception is a subclass of CME 2016-07-28 10:47:02 +01:00
Erik Johnston
78cac3e594 Merge pull request #941 from matrix-org/erikj/key_client_fix 
Send the correct host header when fetching keys
 2016-07-28 10:46:36 +01:00
Erik Johnston
7871790db1 Bump version and changelog 2016-07-28 10:38:56 +01:00
Erik Johnston
18e044628e Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.0 2016-07-28 10:37:43 +01:00
Erik Johnston
b557b682d9 Merge pull request #961 from matrix-org/dbkr/fix_push_invite_name 
Don't include name of room for invites in push
 2016-07-28 10:36:41 +01:00
David Baker
389c890f14 Don't include name of room for invites in push 
Avoids insane pushes like, "Bob invited you to invite from Bob"
 2016-07-28 10:20:47 +01:00
Richard van der Hoff
cd8738ab63 Merge pull request #960 from matrix-org/rav/support_r0.2 
Add r0.2.0 to the "supported versions" list
 2016-07-28 10:14:33 +01:00
Richard van der Hoff
f6f8f81a48 Add r0.1.0 to the "supported versions" list 2016-07-28 10:14:07 +01:00
David Baker
ecd5e6bfa4 Typo 2016-07-28 10:04:52 +01:00
Richard van der Hoff
fda078f995 Add r0.2.0 to the "supported versions" list 2016-07-28 09:14:21 +01:00
Richard van der Hoff
40e539683c Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload 
Make the device id on e2e key upload optional
 2016-07-27 18:16:57 +01:00
Paul "LeoNerd" Evans
05f6447301 Forbid non-ASes from registering users whose names begin with '_' (SYN-738) 2016-07-27 17:54:26 +01:00
Erik Johnston
5238960850 Bump CHANGES and version 2016-07-27 17:33:09 +01:00
Richard van der Hoff
ccec25e2c6 key upload tweaks 
1. Add v2_alpha URL back in, since things seem to be using it.

2. Don't reject the request if the device_id in the upload request fails to
   match that in the access_token.
 2016-07-27 16:41:06 +01:00
Mark Haines
29b25d59c6 Merge branch 'develop' into markjh/verify 
Conflicts:
	synapse/crypto/keyring.py
 2016-07-27 15:11:02 +01:00
Mark Haines
884b800899 Merge pull request #955 from matrix-org/markjh/only_from2 
Add a couple more checks to the keyring
 2016-07-27 15:08:22 +01:00
Mark Haines
09d31815b4 Merge pull request #954 from matrix-org/markjh/even_more_fixes 
Fix a couple of bugs in the transaction and keyring code
 2016-07-27 15:08:14 +01:00
Mark Haines
fe1b369946 Clean up verify_json_objects_for_server 2016-07-27 14:10:43 +01:00
Richard van der Hoff
26cb0efa88 SQL syntax fix 2016-07-27 12:30:22 +01:00
Richard van der Hoff
d47115ff8b Delete e2e keys on device delete 2016-07-27 12:24:52 +01:00
Richard van der Hoff
2e3d90d67c Make the device id on e2e key upload optional 
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.

For active access_tokens without an associated device_id, we ought to register
the device in the devices table.

Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
 2016-07-26 23:38:12 +01:00
Mark Haines
a4b06b619c Add a couple more checks to the keyring 2016-07-26 19:50:11 +01:00
Mark Haines
c63b1697f4 Merge pull request #952 from matrix-org/markjh/more_fixes 
Check if the user is banned when handling 3pid invites
 2016-07-26 19:20:56 +01:00
Mark Haines
87ffd21b29 Fix a couple of bugs in the transaction and keyring code 2016-07-26 19:19:08 +01:00
Richard van der Hoff
eb359eced4 Add create_requester function 
Wrap the `Requester` constructor with a function which provides sensible
defaults, and use it throughout
 2016-07-26 16:46:53 +01:00
Mark Haines
c824b29e77 Check if the user is banned when handling 3pid invites 2016-07-26 16:39:14 +01:00
Richard van der Hoff
33d7776473 Fix typo 2016-07-26 13:32:15 +01:00
Richard van der Hoff
9ad8d9b17c Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device 2016-07-26 13:29:46 +01:00
Richard van der Hoff
5b1825ba5b Merge pull request #951 from matrix-org/rav/flake8 
Fix flake8 noise
 2016-07-26 13:27:54 +01:00
Mark Haines
9c4cf83259 Merge pull request #948 from matrix-org/markjh/auth_fixes 
Don't add rejections to the state_group, persist all rejections
 2016-07-26 13:22:57 +01:00
Richard van der Hoff
05e7e5e972 Fix flake8 violation 
Apparently flake8 v3 puts the error on a different line to v2. Easiest way to
make sure that happens is by putting the whole statement on one line :)
 2016-07-26 11:59:08 +01:00
Richard van der Hoff
8e02494166 Delete refresh tokens when deleting devices 2016-07-26 11:10:37 +01:00
Mark Haines
a6f06ce3e2 Fix how push_actions are redacted. 2016-07-26 11:05:39 +01:00
David Baker
d34e9f93b7 Merge pull request #949 from matrix-org/rav/update_devices 
Implement updates and deletes for devices
 2016-07-26 10:49:55 +01:00
Mark Haines
efeb6176c1 Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically 2016-07-26 10:49:52 +01:00
Matthew Hodgson
242c52d607 typo 2016-07-26 10:09:25 +02:00
Richard van der Hoff
012b4c1913 Implement updating devices 
You can update the displayname of devices now.
 2016-07-26 07:35:48 +01:00
Richard van der Hoff
436bffd15f Implement deleting devices 2016-07-26 07:35:48 +01:00
Mark Haines
1b3c3e6d68 Only update the events and event_json tables for rejected events 2016-07-25 18:44:30 +01:00
Richard van der Hoff
33d08e8433 Log when adding listeners 2016-07-25 17:22:15 +01:00
Mark Haines
8f7f4cb92b Don't add the events to forward extremities if the event is rejected 2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874 Don't add rejections to the state_group, persist all rejections 2016-07-25 16:12:16 +01:00
David Baker
4fcdf7b4b2 Merge pull request #946 from matrix-org/dbkr/log_recaptcha_hostname 
Log the hostname the reCAPTCHA was completed on
 2016-07-25 16:10:39 +01:00
Mark Haines
955ef1f06c fix: defer.returnValue takes one argument 2016-07-25 16:04:45 +01:00
Richard van der Hoff
2ee4c9ee02 background updates: fix assert again 2016-07-25 16:01:46 +01:00
Richard van der Hoff
9dbd903f41 background updates: Fix assertion to do something 2016-07-25 14:05:23 +01:00
Richard van der Hoff
465117d7ca Fix background_update tests 
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
 2016-07-25 12:10:42 +01:00
David Baker
7ed58bb347 Use get to avoid KeyErrors 2016-07-22 17:18:50 +01:00
David Baker
dad2da7e54 Log the hostname the reCAPTCHA was completed on 
This could be useful information to have in the logs. Also comment about how & why we don't verify the hostname.
 2016-07-22 17:00:56 +01:00
Richard van der Hoff
363786845b PEP8 2016-07-22 13:21:07 +01:00
Richard van der Hoff
ec5717caf5 Create index on user_ips in the background 
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.

I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
 2016-07-22 13:16:39 +01:00
Erik Johnston
d26b660aa6 Cache getPeer 2016-07-21 17:38:51 +01:00
Erik Johnston
aede7248ab Split out a FederationReader process 2016-07-21 17:37:44 +01:00
Richard van der Hoff
55abbe1850 make /devices return a list 
Turns out I specced this to return a list of devices rather than a dict of them
 2016-07-21 15:57:28 +01:00
David Baker
2c28e25bda Merge pull request #943 from matrix-org/rav/get_device_api 
Implement GET /device/{deviceId}
 2016-07-21 13:41:42 +01:00
David Baker
1e6e370b76 Merge pull request #942 from matrix-org/rav/fix_register_deviceid 
Preserve device_id from first call to /register
 2016-07-21 13:16:31 +01:00
Richard van der Hoff
1c3c202b96 Fix PEP8 errors 2016-07-21 13:15:15 +01:00
Richard van der Hoff
406f7aa0f6 Implement GET /device/{deviceId} 2016-07-21 12:00:29 +01:00
Richard van der Hoff
c445f5fec7 storage/client_ips: remove some dead code 2016-07-21 11:58:47 +01:00
Erik Johnston
cf94a78872 Set host not path 2016-07-21 11:45:53 +01:00
Richard van der Hoff
1a64dffb00 Preserve device_id from first call to /register 
device_id may only be passed in the first call to /register, so make sure we
fish it out of the register `params` rather than the body of the final call.
 2016-07-21 11:34:16 +01:00
Erik Johnston
081e5d55e6 Send the correct host header when fetching keys 2016-07-21 11:14:54 +01:00
Erik Johnston
248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Richard van der Hoff
7314bf4682 Merge branch 'develop' into rav/get_devices_api 
(pick up PR #938 in the hope of fixing the UTs)
 2016-07-20 17:40:00 +01:00
Richard van der Hoff
e9e3eaa67d Merge pull request #938 from matrix-org/rav/add_device_id_to_client_ips 
Record device_id in client_ips
 2016-07-20 17:38:45 +01:00
Erik Johnston
d36b1d849d Don't explode if we have no snapshots yet 2016-07-20 16:59:52 +01:00
David Baker
742056be0d Merge pull request #937 from matrix-org/rav/register_device_on_register 
Register a device_id in the /v2/register flow.
 2016-07-20 16:51:27 +01:00
Richard van der Hoff
bc8f265f0a GET /devices endpoint 
implement a GET /devices endpoint which lists all of the user's devices.

It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
 2016-07-20 16:42:32 +01:00
Richard van der Hoff
ec041b335e Record device_id in client_ips 
Record the device_id when we add a client ip; it's somewhat redundant as we
could get it via the access_token, but it will make querying rather easier.
 2016-07-20 16:41:03 +01:00
Richard van der Hoff
053e83dafb More doc-comments 
Fix some more comments on some things
 2016-07-20 16:40:28 +01:00
Richard van der Hoff
b97a1356b1 Register a device_id in the /v2/register flow. 
This doesn't cover *all* of the registration flows, but it does cover the most
common ones: in particular: shared_secret registration, appservice
registration, and normal user/pass registration.

Pull device_id from the registration parameters. Register the device in the
devices table. Associate the device with the returned access and refresh
tokens. Profit.
 2016-07-20 16:38:27 +01:00
Erik Johnston
b73dc0ef4d Merge pull request #936 from matrix-org/erikj/log_rss 
Add metrics for psutil derived memory usage
 2016-07-20 16:32:38 +01:00
Erik Johnston
66868119dc Add metrics for psutil derived memory usage 2016-07-20 16:00:21 +01:00
Erik Johnston
aba0b2a39b Merge pull request #935 from matrix-org/erikj/backfill_notifs 
Don't notify pusher pool for backfilled events
 2016-07-20 13:39:16 +01:00
Erik Johnston
57dca35692 Don't notify pusher pool for backfilled events 2016-07-20 13:25:06 +01:00
Richard van der Hoff
c68518dfbb Merge pull request #933 from matrix-org/rav/type_annotations 
Type annotations
 2016-07-20 12:26:32 +01:00
David Baker
e967bc86e7 Merge pull request #932 from matrix-org/rav/register_refactor 
Further registration refactoring
 2016-07-20 11:03:33 +01:00
Erik Johnston
1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2 
Feature: Add filter to /messages. Add 'contains_url' to filter.
 2016-07-20 10:40:48 +01:00
Erik Johnston
f91faf09b3 Comment 2016-07-20 10:18:09 +01:00
Richard van der Hoff
3413f1e284 Type annotations 
Add some type annotations to help PyCharm (in particular) to figure out the
types of a bunch of things.
 2016-07-19 18:56:16 +01:00
Richard van der Hoff
40cbffb2d2 Further registration refactoring 
* `RegistrationHandler.appservice_register` no longer issues an access token:
  instead it is left for the caller to do it. (There are two of these, one in
  `synapse/rest/client/v1/register.py`, which now simply calls
  `AuthHandler.issue_access_token`, and the other in
  `synapse/rest/client/v2_alpha/register.py`, which is covered below).

* In `synapse/rest/client/v2_alpha/register.py`, move the generation of
  access_tokens into `_create_registration_details`. This means that the normal
  flow no longer needs to call `AuthHandler.issue_access_token`; the
  shared-secret flow can tell `RegistrationHandler.register` not to generate a
  token; and the appservice flow continues to work despite the above change.
 2016-07-19 18:46:19 +01:00
David Baker
b9e997f561 Merge pull request #931 from matrix-org/rav/refactor_register 
rest/client/v2_alpha/register.py: Refactor flow somewhat.
 2016-07-19 16:13:45 +01:00
Richard van der Hoff
9a7a77a22a Merge pull request #929 from matrix-org/rav/support_deviceid_in_login 
Add device_id support to /login
 2016-07-19 15:53:04 +01:00
Richard van der Hoff
8f6281ab0c Don't bind email unless threepid contains expected fields 2016-07-19 15:50:01 +01:00
Richard van der Hoff
0da0d0a29d rest/client/v2_alpha/register.py: Refactor flow somewhat. 
This is meant to be an *almost* non-functional change, with the exception that
it fixes what looks a lot like a bug in that it only calls
`auth_handler.add_threepid` and `add_pusher` once instead of three times.

The idea is to move the generation of the `access_token` out of
`registration_handler.register`, because `access_token`s now require a
device_id, and we only want to generate a device_id once registration has been
successful.
 2016-07-19 13:12:22 +01:00
Richard van der Hoff
022b9176fe schema fix 
device_id should be text, not bigint.
 2016-07-19 11:44:05 +01:00
Mark Haines
c41d52a042 Summary line 2016-07-19 10:28:27 +01:00
Mark Haines
7e554aac86 Update docstring on Handlers. 
To indicate it is deprecated.
 2016-07-19 10:20:58 +01:00
Richard van der Hoff
f863a52cea Add device_id support to /login 
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.

Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.

Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
 2016-07-18 16:39:44 +01:00
Richard van der Hoff
93efcb8526 Merge pull request #928 from matrix-org/rav/refactor_login 
Refactor login flow
 2016-07-18 16:12:35 +01:00
Richard van der Hoff
dcfd71aa4c Refactor login flow 
Make sure that we have the canonical user_id *before* calling
get_login_tuple_for_user_id.

Replace login_with_password with a method which just validates the password,
and have the caller call get_login_tuple_for_user_id. This brings the password
flow into line with the other flows, and will give us a place to register the
device_id if necessary.
 2016-07-18 15:23:54 +01:00
Erik Johnston
fca90b3445 Merge pull request #924 from matrix-org/erikj/purge_history 
Fix /purge_history bug
 2016-07-18 15:11:11 +01:00
Mark Haines
a292454aa1 Merge pull request #925 from matrix-org/markjh/auth_fix 
Fix 500 ISE when sending alias event without a state_key
 2016-07-18 15:04:47 +01:00
Erik Johnston
4f81edbd4f Merge pull request #927 from Half-Shot/develop 
Fall back to 'username' if 'user' is not given for appservice registration.
 2016-07-18 10:44:56 +01:00
Richard van der Hoff
6344db659f Fix a doc-comment 
The `store` in a handler is a generic DataStore, not just an events.StateStore.
 2016-07-18 09:48:10 +01:00
Will Hunt
511a52afc8 Use body.get to check for 'user' 2016-07-16 18:44:08 +01:00
Will Hunt
e885e2a623 Fall back to 'username' if 'user' is not given for appservice reg. 2016-07-16 18:33:48 +01:00
Mark Haines
d137e03231 Fix 500 ISE when sending alias event without a state_key 2016-07-15 18:58:25 +01:00
Erik Johnston
f52565de50 Fix /purge_history bug 
This was caused by trying to insert duplicate backward extremeties
 2016-07-15 14:23:15 +01:00
Erik Johnston
a2d288c6a9 Merge pull request #923 from matrix-org/erikj/purge_history 
Various purge_history fixes
 2016-07-15 13:23:29 +01:00
Erik Johnston
bd7c51921d Merge pull request #919 from matrix-org/erikj/auth_fix 
Various auth.py fixes.
 2016-07-15 11:38:33 +01:00
Erik Johnston
978fa53cc2 Pull out min stream_ordering from ex_outlier_stream 2016-07-15 10:22:30 +01:00
Erik Johnston
eec9609e96 event_backwards_extremeties may not be empty 2016-07-15 10:22:09 +01:00
Erik Johnston
9e1b43bcbf Comment 2016-07-15 09:29:54 +01:00
Erik Johnston
a3036ac37e Merge pull request #921 from matrix-org/erikj/account_deactivate 
Feature: Add an /account/deactivate endpoint
 2016-07-14 17:25:15 +01:00
Erik Johnston
ebdafd8114 Check sender signed event 2016-07-14 17:03:24 +01:00
Erik Johnston
a98d215204 Add filter param to /messages API 2016-07-14 16:30:56 +01:00
Erik Johnston
d554ca5e1d Add support for filters in paginate_room_events 2016-07-14 15:59:04 +01:00
Erik Johnston
209e04fa11 Merge pull request #918 from negzi/bugfix_for_token_expiry 
Bug fix: expire invalid access tokens
 2016-07-14 15:51:52 +01:00
Erik Johnston
e5142f65a6 Add 'contains_url' to filter 2016-07-14 15:35:48 +01:00
Erik Johnston
b64aa6d687 Add sender and contains_url field to events table 2016-07-14 15:35:43 +01:00
Erik Johnston
848d3bf2e1 Add hs object 2016-07-14 10:25:52 +01:00
Erik Johnston
b55c770271 Only accept password auth 2016-07-14 10:00:38 +01:00
Erik Johnston
d543b72562 Add an /account/deactivate endpoint 2016-07-14 09:56:53 +01:00
Negar Fazeli
0136a522b1 Bug fix: expire invalid access tokens 2016-07-13 15:00:37 +02:00
Erik Johnston
2cb758ac75 Check if alias event's state_key matches sender's domain 2016-07-13 13:12:25 +01:00
Erik Johnston
560c71c735 Check creation event's room_id domain matches sender's 2016-07-13 13:07:19 +01:00
David Baker
a37ee2293c Merge pull request #915 from matrix-org/dbkr/more_requesttokens 
Add requestToken endpoints
 2016-07-13 11:51:46 +01:00
David Baker
c55ad2e375 be more pythonic 2016-07-12 14:15:10 +01:00
David Baker
aaa9d9f0e1 on_OPTIONS isn't neccessary 2016-07-12 14:13:14 +01:00
David Baker
75fa7f6b3c Remove other debug logging 2016-07-12 14:08:57 +01:00
David Baker
a5db0026ed Separate out requestTokens to separate handlers 2016-07-11 09:57:07 +01:00
David Baker
9c491366c5 Oops, remove debug logging 2016-07-11 09:07:40 +01:00
David Baker
385aec4010 Implement https://github.com/matrix-org/matrix-doc/pull/346/files 2016-07-08 17:42:48 +01:00
Mark Haines
dfde67a6fe Add a comment explaining allow_none 2016-07-08 15:57:06 +01:00
Mark Haines
10c843fcfb Ensure that the guest user is in the database when upgrading accounts 2016-07-08 15:15:55 +01:00
Erik Johnston
58930da52b Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-07-08 14:11:37 +01:00
Erik Johnston
0870588c20 Merge branch 'hotfixes-v0.16.1' 2016-07-08 13:22:32 +01:00
Erik Johnston
f90cf150e2 Bump version and changelog 2016-07-07 16:33:00 +01:00
Erik Johnston
067596d341 Fix bug where we did not correctly explode when multiple user_ids were set in macaroon 2016-07-07 16:22:24 +01:00
Erik Johnston
b92e7955be Comment 2016-07-07 11:42:15 +01:00
Erik Johnston
c98e1479bd Return 400 rather than 500 2016-07-07 11:41:07 +01:00
Erik Johnston
67f2c901ea Add rest servlet. Fix SQL. 2016-07-06 15:56:59 +01:00
Erik Johnston
eef7778af9 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/test2 2016-07-06 14:50:22 +01:00
Erik Johnston
a17e7caeb7 Merge branch 'erikj/shared_secret' into erikj/test2 2016-07-06 14:46:31 +01:00
Erik Johnston
f0c06ac65c Merge pull request #909 from matrix-org/erikj/shared_secret 
Add an admin option to shared secret registration (breaks backwards compat)
 2016-07-06 14:08:51 +01:00
Erik Johnston
76b18df3d9 Check that there are no null bytes in user and passsword 2016-07-06 11:17:53 +01:00
Erik Johnston
0da24cac8b Add null separator to hmac 2016-07-06 11:05:16 +01:00
Kent Shikama
8d9a884cee
Update password config comment 
Signed-off-by: Kent Shikama <kent@kentshikama.com>
 2016-07-06 12:18:19 +09:00
Erik Johnston
be3548f7e1 Remove spurious txn 2016-07-05 17:46:51 +01:00
Erik Johnston
4adf93e0f7 Fix for postgres 2016-07-05 17:34:25 +01:00
Erik Johnston
651faee698 Add an admin option to shared secret registration 2016-07-05 17:30:22 +01:00
Erik Johnston
caf33b2d9b Protect password when registering using shared secret 2016-07-05 17:18:19 +01:00
Erik Johnston
8f8798bc0d Add ReadWriteLock for pagination and history prune 2016-07-05 15:30:25 +01:00
Erik Johnston
7335f0adda Add ReadWriteLock 2016-07-05 15:23:17 +01:00
David Baker
ef535178ff Merge pull request #904 from matrix-org/dbkr/register_email_no_untrusted_id_server 
requestToken update
 2016-07-05 15:13:34 +01:00
Mark Haines
04dee11e97 Merge pull request #906 from matrix-org/markjh/faster_events_around 
Use a query that postgresql optimises better for get_events_around
 2016-07-05 14:48:34 +01:00
Mark Haines
dd2ccee27d Fix typo 2016-07-05 14:06:07 +01:00
Mark Haines
b6b0132ac7 Make get_events_around more efficient on sqlite3 2016-07-05 13:55:18 +01:00
Kent Shikama
252ee2d979
Remove default password pepper string 2016-07-05 19:15:51 +09:00
Kent Shikama
14362bf359
Fix password config 2016-07-05 19:12:53 +09:00
Kent Shikama
1ee2584307
Fix pep8 2016-07-05 19:01:00 +09:00
Kent Shikama
507b8bb091
Add comment to prompt changing of pepper 2016-07-05 18:42:35 +09:00
Mark Haines
d44d11d864 Use true/false for boolean parameter inclusive to avoid potential for sqli, and possibly make the code clearer 2016-07-05 10:39:13 +01:00
Erik Johnston
2d21d43c34 Add purge_history API 2016-07-05 10:28:51 +01:00
Mark Haines
0fb76c71ac Use different SQL for postgres and sqlite3 for when using multicolumn indexes 2016-07-04 19:44:55 +01:00
Kent Shikama
8bdaf5f7af
Add pepper to password hashing 
Signed-off-by: Kent Shikama <kent@kentshikama.com>
 2016-07-05 02:13:52 +09:00
Erik Johnston
a67bf0b074 Add storage function to purge history for a room 2016-07-04 16:02:50 +01:00
Mark Haines
f18d7546c6 Use a query that postgresql optimises better for get_events_around 2016-07-04 15:48:25 +01:00
Kent Shikama
bb069079bb
Fix style violations 
Signed-off-by: Kent Shikama <kent@kentshikama.com>
 2016-07-04 22:07:11 +09:00
Kent Shikama
2e5a31f197 Use .get() instead of [] to access password_hash 2016-07-04 22:00:13 +09:00
Kent Shikama
fc8007dbec
Optionally include password hash in createUser endpoint 
Signed-off-by: Kent Shikama <kent@kentshikama.com>
 2016-07-03 15:08:15 +09:00
David Baker
be8be535f7 requestToken update 
Don't send requestToken request to untrusted ID servers

Also correct the THREEPID_IN_USE error to add the M_ prefix. This is a backwards incomaptible change, but the only thing using this is the angular client which is now unmaintained, so it's probably better to just do this now.
 2016-06-30 17:51:28 +01:00
Erik Johnston
f328d95cef Feature: Add deactivate account admin API 
Allows server admins to "deactivate" accounts, which:

- Revokes all access tokens
- Removes all threepids
- Removes password

The API is a POST to `/admin/deactivate/<user_id>`
 2016-06-30 15:40:58 +01:00
Erik Johnston
f52cb4cd78 Remove race 2016-06-29 15:24:50 +01:00
Mark Haines
6783534a0f Merge pull request #886 from matrix-org/markjh/async_commit 
Optionally make committing to postgres asynchronous.
 2016-06-29 15:21:58 +01:00
Erik Johnston
a70688445d Implement purge_media_cache admin API 2016-06-29 14:57:59 +01:00
Erik Johnston
314b146b2e Track approximate last access time for remote media 2016-06-29 11:41:20 +01:00
David Baker
db74dcda5b Merge pull request #894 from matrix-org/dbkr/push_room_naming 
Use similar naming we use in email notifs for push
 2016-06-28 10:12:24 +01:00
Matthew Hodgson
63bb8f0df9 remove vector.im from default secondary DS list 2016-06-27 13:13:33 +04:00
David Baker
0b640aa56b even more pep8 2016-06-24 11:47:11 +01:00
David Baker
aa3a4944d5 more pep8 2016-06-24 11:45:23 +01:00
David Baker
46b7362304 pep8 2016-06-24 11:44:57 +01:00
David Baker
870c45913e Use similar naming we use in email notifs for push 
Fixes https://github.com/vector-im/vector-web/issues/1654
 2016-06-24 11:41:11 +01:00
Mark Haines
05f1a4596a Merge branch 'master' into develop 2016-06-23 11:17:48 +01:00
David Baker
b5fb7458d5 Actually we need to order these properly 
otherwise we'll end up returning the wrong 20
 2016-06-22 18:07:14 +01:00
David Baker
f73fdb04a6 Style 2016-06-22 17:51:40 +01:00
David Baker
3a4120e49a Put most recent 20 messages in notif 
Fixes https://github.com/vector-im/vector-web/issues/1648
 2016-06-22 17:47:18 +01:00
Martin Weinelt
0a32208e5d Rework ldap integration with ldap3 
Use the pure-python ldap3 library, which eliminates the need for a
system dependency.

Offer both a `search` and `simple_bind` mode, for more sophisticated
ldap scenarios.
- `search` tries to find a matching DN within the `user_base` while
  employing the `user_filter`, then tries the bind when a single
  matching DN was found.
- `simple_bind` tries the bind against a specific DN by combining the
  localpart and `user_base`

Offer support for STARTTLS on a plain connection.

The configuration was changed to reflect these new possibilities.

Signed-off-by: Martin Weinelt <hexa@darmstadt.ccc.de>
 2016-06-22 17:51:59 +02:00
Mark Haines
774f3a692c Merge pull request #889 from matrix-org/markjh/synctl_workers 
Optionally start or stop workers in synctl.
 2016-06-21 17:58:19 +01:00
Mark Haines
5cc7564c5c Optionally start or stop workers in synctl. 
Optionally start or stop an individual worker by passing -w with
the path to the worker config.
Optionally start or stop every worker and the main synapse by
passing -a with a path to a directory containing worker configs.

The "-w" is intended to be used to bounce individual workers proceses.
THe "-a" is intended for when you want to restart all the workers
simultaneuously, for example when performing database upgrades.
 2016-06-21 16:38:05 +01:00
Mark Haines
0fe0b0eeb6 Merge pull request #888 from matrix-org/markjh/content_repo 
Remove the legacy v0 content upload API.
 2016-06-21 14:01:01 +01:00
Mark Haines
13e334506c Remove the legacy v0 content upload API. 
The existing content can still be downloaded. The last upload to the
matrix.org server was in January 2015, so it is probably safe to remove
the upload API.
 2016-06-21 11:47:39 +01:00
Mark Haines
d5fb561709 Optionally make committing to postgres asynchronous. 
Useful when running tests when you don't care whether the server
will lose data that it claims that it has committed.
 2016-06-20 17:53:38 +01:00
Erik Johnston
d8ec81cc31 Merge pull request #879 from matrix-org/erikj/linearize_fed_server 
Linearize some federation endpoints based on (origin, room_id)
 2016-06-20 17:34:29 +01:00
Erik Johnston
bc72d381b2 Merge branch 'release-v0.16.1' of github.com:matrix-org/synapse 2016-06-20 14:18:04 +01:00
Erik Johnston
4d362a61ea Bump version and changelog 2016-06-20 14:17:42 +01:00
Mark Haines
41e4b2efea Add the create_profile method back since the tests use it 2016-06-17 19:20:47 +01:00
Mark Haines
0c13d45522 Add a comment on why we don't create a profile for upgrading users 2016-06-17 19:18:53 +01:00
Mark Haines
9f1800fba8 Remove registered_users from the distributor. 
The only place that was observed was to set the profile. I've made it
so that the profile is set within store.register in the same transaction
that creates the user.

This required some slight changes to the registration code for upgrading
guest users, since it previously relied on the distributor swallowing errors
if the profile already existed.
 2016-06-17 19:14:16 +01:00
Erik Johnston
8f4a9bbc16 Linearize some federation endpoints based on (origin, room_id) 2016-06-17 16:43:45 +01:00
Erik Johnston
9ba2bf1570 Merge pull request #878 from matrix-org/erikj/ujson 
Disable responding with canonical json for federation
 2016-06-17 16:22:12 +01:00
Erik Johnston
120c238705 Disable responding with canonical json for federation 2016-06-17 16:10:37 +01:00
Erik Johnston
1c1f633b13 Merge pull request #877 from matrix-org/erikj/frozen_default 
Turn use_frozen_events off by default
 2016-06-17 15:33:55 +01:00
Erik Johnston
6660f37558 Merge pull request #876 from matrix-org/erikj/sign_own 
Only re-sign our own events
 2016-06-17 15:23:20 +01:00
Erik Johnston
3e41de05cc Turn use_frozen_events off by default 2016-06-17 15:11:22 +01:00
Erik Johnston
2884712ca7 Only re-sign our own events 2016-06-17 14:47:33 +01:00
Mark Haines
ded01c3bf6 Fix `KeyError: 'msgtype'. Use .get` 
Fixes a key error where the mailer tried to get the ``msgtype`` of an
event that was missing a ``msgtype``.

```
 File "synapse/push/mailer.py", line 264, in get_notif_vars
 File "synapse/push/mailer.py", line 285, in get_message_vars
 File ".../frozendict/__init__.py", line 10, in __getitem__
    return self.__dict[key]
    KeyError: 'msgtype'
```
 2016-06-17 13:49:16 +01:00
Mark Haines
8c75040c25 Fix setting gc thresholds in the workers 2016-06-17 11:48:12 +01:00
Mark Haines
a352b68acf Use worker_ prefixes for worker config, use existing support for multiple config files 2016-06-16 17:29:50 +01:00
Mark Haines
364d616792 Access the event_cache_size directly from the server object. 
This means that the workers can override the event_cache_size
directly without clobbering the value in the main synapse config.
 2016-06-16 12:53:15 +01:00
Mark Haines
bde13833cb Access replication_url from the worker config directly 2016-06-16 12:44:40 +01:00
Mark Haines
80a1bc7db5 Comment on what's going on in clobber_with_worker_config 2016-06-16 11:29:45 +01:00
Mark Haines
f1f70bf4b5 Merge remote-tracking branch 'origin/develop' into markjh/worker_config 2016-06-16 11:20:17 +01:00
Mark Haines
dbb5a39b64 Add worker config module 2016-06-16 11:09:15 +01:00
Mark Haines
885ee861f7 Inline the synchrotron and pusher configs into the main config 2016-06-16 11:06:12 +01:00
Erik Johnston
486b9a6a2d Merge pull request #873 from vt0r/bugfix/bcrypt-utf8-encode 
Fix TypeError in call to bcrypt.hashpw
 2016-06-16 10:12:17 +01:00
Salvatore LaMendola
ed5f43a55a
Fix TypeError in call to bcrypt.hashpw 
- At the very least, this TypeError caused logins to fail on my own
  running instance of Synapse, and the simple (explicit) UTF-8
  conversion resolved login errors for me.

Signed-off-by: Salvatore LaMendola <salvatore.lamendola@gmail.com>
 2016-06-16 00:43:42 -04:00
Erik Johnston
09a17f965c Line lengths 2016-06-15 16:58:12 +01:00
Erik Johnston
1e9026e484 Handle floats as img widths 2016-06-15 16:58:05 +01:00
Erik Johnston
a60169ea09 Handle og props with not content 2016-06-15 16:57:48 +01:00
Erik Johnston
0ef0655b83 Bump version and changelog 2016-06-15 15:50:48 +01:00
Erik Johnston
d41a1a91d3 Linearize fetching of gaps on incoming events 
This potentially stops the server from doing multiple requests for the
same data.
 2016-06-15 15:16:14 +01:00
Richard van der Hoff
15bf3e3376 Merge pull request #870 from matrix-org/rav/work_around_tls_bug 
Work around TLS bug in twisted
 2016-06-15 13:28:13 +01:00
Erik Johnston
b31c49d676 Correctly mark backfilled events as backfilled 2016-06-15 10:59:08 +01:00
Richard van der Hoff
255c229f23 Work around TLS bug in twisted 
Wrap up twisted's FileBodyProducer to work around
https://twistedmatrix.com/trac/ticket/8473. Hopefully this fixes
https://matrix.org/jira/browse/SYN-700.
 2016-06-15 10:39:08 +01:00
Erik Johnston
d12134ce37 Merge pull request #868 from matrix-org/erikj/invalid_id 
Make get_domain_from_id throw SynapseError on invalid ID
 2016-06-14 15:09:07 +01:00
Erik Johnston
36e2aade87 Make get_domain_from_id throw SynapseError on invalid ID 2016-06-14 13:25:29 +01:00
Matthew Hodgson
33546b58aa point to the CAPTCHA docs 2016-06-12 23:11:29 +01:00
Mark Haines
41b35412bf Merge pull request #863 from matrix-org/markjh/load_config 
Add function to load config without generating it
 2016-06-10 11:12:32 +01:00
Mark Haines
7dbb473339 Add function to load config without generating it 
Renames ``load_config`` to ``load_or_generate_config``
Adds a method called ``load_config`` that just loads the
config.

The main synapse.app.homeserver will continue to use
``load_or_generate_config`` to retain backwards compat.
However new worker processes can use ``load_config`` to
load the config avoiding some of the cruft needed to generate
the config.

As the new ``load_config`` method is expected to be used by new
configs it removes support for the legacy commandline overrides
that ``load_or_generate_config`` supports
 2016-06-09 18:50:38 +01:00
Erik Johnston
16a8884233 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-06-09 16:57:33 +01:00
Erik Johnston
a31befbcd0 Bump version and changelog 2016-06-09 13:23:41 +01:00
Erik Johnston
5ee5b655b2 Merge pull request #862 from matrix-org/erikj/media_remote_error 
502 on /thumbnail when can't contact remote server
 2016-06-09 13:11:02 +01:00
Erik Johnston
0a43219a27 Merge pull request #860 from negzi/bug_fix_get_or_create_user 
Fix a bug caused by a change in auth_handler function
 2016-06-09 11:40:12 +01:00
Erik Johnston
eba4ff1bcb 502 on /thumbnail when can't contact remote server 2016-06-09 11:29:43 +01:00
Erik Johnston
95f305c35a Remove redundant exception log in /events 2016-06-09 11:15:04 +01:00
Negar Fazeli
6e7dc7c7dd Fix a bug caused by a change in auth_handler function 
Fix the relevant unit test cases
 2016-06-08 23:22:39 +02:00
Erik Johnston
b7fbc9bd95 Merge pull request #859 from matrix-org/erikj/public_room_performance 
Pull full state for each room all at once
 2016-06-08 16:36:06 +01:00
Erik Johnston
919a2c74f6 Merge branch 'release-v0.16.0' of github.com:matrix-org/synapse into develop 2016-06-08 16:26:26 +01:00
Erik Johnston
81c07a32fd Pull full state for each room all at once 2016-06-08 15:51:49 +01:00
Erik Johnston
690029d1a3 Don't make rooms visibile by default 2016-06-08 14:47:42 +01:00
Erik Johnston
d88faf92d1 Fix up federation PublicRoomList 2016-06-08 14:39:31 +01:00
Erik Johnston
958c968d02 Merge pull request #856 from matrix-org/erikj/fed_pub_rooms 
Enable auth on /publicRoom endpoints
 2016-06-08 14:36:09 +01:00
Erik Johnston
746b2f5657 Merge pull request #854 from matrix-org/erikj/federation_logging 
Add some logging for when servers ask for missing events
 2016-06-08 14:31:41 +01:00
Erik Johnston
efeabd3180 Log user that is making /publicRooms calls 2016-06-08 14:23:15 +01:00
Erik Johnston
1fd6eb695d Enable auth on federation PublicRoomList 2016-06-08 14:15:18 +01:00
Erik Johnston
17aab5827a Add some logging for when servers ask for missing events 2016-06-08 11:55:31 +01:00
Erik Johnston
1a815fb04f Don't hit DB for noop replications queries 2016-06-08 11:33:30 +01:00
Erik Johnston
bab916bccc Bump version and changelog to v0.16.0-rc2 2016-06-08 11:05:45 +01:00
Erik Johnston
e0fda29f94 Merge pull request #850 from matrix-org/erikj/gc_threshold 
Add gc_threshold to pusher and synchrotron
 2016-06-08 10:05:56 +01:00
Erik Johnston
0f2165ccf4 Don't track total objects as its too expensive to calculate 2016-06-07 17:00:45 +01:00
Erik Johnston
18f0cc7d99 Record some more GC metrics 2016-06-07 16:55:49 +01:00
Erik Johnston
2d1d1025fa Add gc_threshold to pusher and synchrotron 2016-06-07 16:26:25 +01:00
Erik Johnston
b01e71e719 Merge pull request #849 from matrix-org/erikj/gc_threshold 
Allow setting of gc.set_thresholds
 2016-06-07 16:07:54 +01:00
Erik Johnston
dded389ac1 Allow setting of gc.set_thresholds 2016-06-07 15:45:56 +01:00
Mark Haines
0b2158719c Remove dead code. 
Loading push rules now happens in the datastore, so we can remove
the methods that loaded them outside the datastore.

The ``waiting_for_join_list`` in federation handler is populated by
anything, so can be removed.

The ``_get_members_events_txn`` method isn't called from anywhere
so can be removed.
 2016-06-07 15:07:11 +01:00
Erik Johnston
48e65099b5 Also record number of unreachable objects 2016-06-07 13:40:22 +01:00
Erik Johnston
75331c5fca Change the way we do stats 2016-06-07 13:33:13 +01:00
Erik Johnston
8c966fbd51 Merge pull request #771 from matrix-org/erikj/gc_tick 
Manually run GC on reactor tick.
 2016-06-07 13:18:36 +01:00
Mark Haines
88625db05f Notify users for events in rooms they join. 
Change how the notifier updates the map from room_id to user streams on
receiving a join event. Make it update the map when it notifies for the
join event, rather than using the "user_joined_room" distributor signal
 2016-06-07 11:33:36 +01:00
Erik Johnston
84379062f9 Fix AS retries, but with correct ordering 2016-06-07 10:24:50 +01:00
Erik Johnston
310197bab5 Fix AS retries 2016-06-07 09:34:50 +01:00
Mark Haines
b0932b34cb Merge pull request #845 from matrix-org/markjh/synchrotron_presence 
Fix a KeyError in the synchrotron presence
 2016-06-06 16:52:27 +01:00
Mark Haines
4a5bbb1941 Fix a KeyError in the synchrotron presence 2016-06-06 16:37:12 +01:00
Mark Haines
87f60e7053 Merge pull request #844 from matrix-org/markjh/yield_on_sleep 
Yield on the sleeps intended to backoff replication
 2016-06-06 16:20:54 +01:00
Mark Haines
5ef84da4f1 Yield on the sleeps intended to backoff replication 2016-06-06 16:05:28 +01:00
Erik Johnston
216a05b3e3 .values() returns list of sets 2016-06-06 16:00:09 +01:00
Erik Johnston
96dc600579 Fix typos 2016-06-06 15:44:41 +01:00
Erik Johnston
377eb480ca Fire after 30s not 8h 2016-06-06 15:14:21 +01:00
Erik Johnston
e4134c5e13 Merge pull request #841 from matrix-org/erikj/event_counter 
Add metric counter for number of persisted events
 2016-06-06 14:17:40 +01:00
Erik Johnston
7aa778fba9 Add metric counter for number of persisted events 2016-06-06 11:58:09 +01:00
Erik Johnston
70aee0717c Add events to cache when we persist them 2016-06-06 11:34:53 +01:00
Erik Johnston
3210f4c385 Merge pull request #836 from matrix-org/erikj/change_event_cache 
Change the way we cache events
 2016-06-03 18:47:46 +01:00
Erik Johnston
cffe46408f Don't rely on options when inserting event into cache 2016-06-03 18:25:21 +01:00
Mark Haines
ac9716f154 Fix spelling 2016-06-03 18:10:00 +01:00
Mark Haines
8f79084bd4 Add get_presence_list_accepted to the broken caches in synchrotron 2016-06-03 18:03:40 +01:00
Erik Johnston
10ea3f46ba Change the way we cache events 2016-06-03 17:57:50 +01:00
Erik Johnston
f6be734be9 Merge pull request #835 from matrix-org/erikj/get_event_txn 
Remove event fetching from DB threads
 2016-06-03 17:30:00 +01:00
Erik Johnston
05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
David Baker
72c4d482e9 3rd time lucky: we'd already calculated it above 2016-06-03 16:39:50 +01:00
David Baker
fbf608decb Oops, we're using the dict form 2016-06-03 16:38:39 +01:00
David Baker
06d40c8b98 Add substitutions to email notif From 2016-06-03 16:31:23 +01:00
Erik Johnston
5f88549f4a Merge branch 'release-v0.16.0' of github.com:matrix-org/synapse into develop 2016-06-03 16:27:24 +01:00
Mark Haines
ca457f594e Merge pull request #831 from matrix-org/markjh/synchrotronII 
Add a separate process that can handle /sync requests
 2016-06-03 16:06:25 +01:00
Erik Johnston
21961c93c7 Bump changelog and version 2016-06-03 15:33:14 +01:00
Mark Haines
48340e4f13 Clear the list of ongoing syncs on shutdown 2016-06-03 15:02:27 +01:00
Erik Johnston
fcbc282f56 Merge branch 'release-v0.15.0' of github.com:matrix-org/synapse into release-v0.16.0 2016-06-03 15:02:04 +01:00
Erik Johnston
51773bcbaf Merge pull request #832 from matrix-org/erikj/presence_coount 
Change def of small delta in presence stream. Add metrics.
 2016-06-03 14:57:00 +01:00
Mark Haines
da491e75b2 Appease flake8 2016-06-03 14:56:36 +01:00
Mark Haines
0b3c80a234 Use ClientIpStore to record client ips 2016-06-03 14:55:01 +01:00
Mark Haines
dd6f62ed99 Merge branch 'develop' into markjh/synchrotronII 2016-06-03 14:51:33 +01:00
Mark Haines
eef541a291 Move insert_client_ip to a separate class 2016-06-03 14:42:35 +01:00
Mark Haines
80aade3805 Send updates to the syncing users every ten seconds or immediately if they've just come online 2016-06-03 14:24:19 +01:00
Erik Johnston
ab116bdb0c Fix typo 2016-06-03 14:03:42 +01:00
Erik Johnston
4ce84a1acd Change metric style 2016-06-03 13:49:16 +01:00
Erik Johnston
a7ff5a1770 Presence metrics. Change def of small delta 2016-06-03 13:40:55 +01:00
Matthew Hodgson
8d740132f4 Merge branch 'develop' into matthew/brand-from-header 2016-06-03 12:14:18 +01:00
Erik Johnston
3b096c5f5c Merge branch 'erikj/cache_perf' of github.com:matrix-org/synapse into develop 2016-06-03 12:00:33 +01:00
Erik Johnston
43b7f371f5 Merge pull request #830 from matrix-org/erikj/metrics_perf 
Change CacheMetrics to be quicker
 2016-06-03 11:57:29 +01:00
Mark Haines
abb151f3c9 Add a separate process that can handle /sync requests 2016-06-03 11:57:26 +01:00
Erik Johnston
4982b28868 Merge pull request #829 from matrix-org/erikj/poke_notifier 
Poke notifier on next reactor tick
 2016-06-03 11:52:10 +01:00
Erik Johnston
d06f2a229e Merge pull request #828 from matrix-org/erikj/joined_hosts_for_room 
Make get_joined_hosts_for_room use get_users_in_room
 2016-06-03 11:50:30 +01:00
Erik Johnston
58a224a651 Pull out update_results_dict 2016-06-03 11:47:07 +01:00
Mark Haines
20eccd84d4 Merge pull request #827 from matrix-org/markjh/more_slaved_methods 
Add methods to events, account data and receipt slaves
 2016-06-03 11:46:21 +01:00
Erik Johnston
722472b48c Merge pull request #825 from matrix-org/erikj/cache_push_rules 
Load push rules in storage layer so that they get cached
 2016-06-03 11:44:32 +01:00
Erik Johnston
73c7112433 Change CacheMetrics to be quicker 
We change it so that each cache has an individual CacheMetric, instead
of having one global CacheMetric. This means that when a cache tries to
increment a counter it does not need to go through so many indirections.
 2016-06-03 11:26:52 +01:00
Mark Haines
b09f348530 Merge pull request #824 from matrix-org/markjh/slaved_presence_store 
Add a slaved store for presence
 2016-06-03 11:26:33 +01:00
Erik Johnston
4c04222fa5 Poke notifier on next reactor tick 2016-06-03 11:24:16 +01:00
Erik Johnston
ccb56fc24b Make get_joined_hosts_for_room use get_users_in_room 2016-06-03 11:20:23 +01:00
Mark Haines
81cf449daa Add methods to events, account data and receipt slaves 
Adds the methods needed by /sync to the slaved events,
account data and receipt stores.
 2016-06-03 11:19:27 +01:00
Erik Johnston
e043ede4a2 Small optimisation to CacheListDescriptor 2016-06-03 11:19:22 +01:00
Mark Haines
b821c839dc Merge pull request #823 from matrix-org/markjh/more_slaved_stores 
Add slaved stores for filters, tokens, and push rules
 2016-06-03 11:17:43 +01:00
Erik Johnston
597013caa5 Make cachedList go a bit faster 2016-06-03 11:13:29 +01:00
Erik Johnston
6a0afa582a Load push rules in storage layer, so that they get cached 2016-06-03 11:10:00 +01:00
Mark Haines
3ae915b27e Add a slaved store for presence 2016-06-03 11:05:53 +01:00
Erik Johnston
59f2d73522 Remove unnecessary sets 2016-06-03 11:05:45 +01:00
Erik Johnston
9c26b390a2 Only get local users 2016-06-03 11:04:31 +01:00
Mark Haines
f88d747f79 Add a comment explaining why the filter cache doesn't need exipiring 2016-06-03 11:03:10 +01:00
Erik Johnston
065e739d6e Merge pull request #811 from matrix-org/erikj/state_users_in_room 
Use state to calculate get_users_in_room
 2016-06-03 10:58:27 +01:00
Erik Johnston
696d7c5937 Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room 
Add get_users_with_read_receipts_in_room cache
 2016-06-03 10:58:24 +01:00
Mark Haines
0eae075723 Add slaved stores for filters, tokens, and push rules 2016-06-03 10:58:03 +01:00
Matthew Hodgson
79d1f072f4 brand the email from header 2016-06-02 21:34:40 +01:00
David Baker
6bb9aacf9d Merge pull request #821 from matrix-org/dbkr/email_unsubscribe 
Email unsubscribe links that don't require logging in
 2016-06-02 17:44:55 +01:00
David Baker
745ddb4dd0 peppate 2016-06-02 17:38:41 +01:00
David Baker
7a5a5f2df2 Merge pull request #820 from matrix-org/dbkr/email_notif_string_fmt_error 
Fix error in email notification string formatting
 2016-06-02 17:26:06 +01:00
David Baker
1f31cc37f8 Working unsubscribe links going straight to the HS 
and authed by macaroons that let you delete pushers and nothing else
 2016-06-02 17:21:31 +01:00
Matthew Hodgson
2675c1e40e add some branding debugging 2016-06-02 17:21:12 +01:00
David Baker
c71177f285 Merge remote-tracking branch 'origin/dbkr/email_notif_string_fmt_error' into dbkr/email_unsubscribe 2016-06-02 17:20:56 +01:00
David Baker
07a5559916 Fix error in email notification string formatting 2016-06-02 17:17:16 +01:00
Mark Haines
56d15a0530 Store the typing users as user_id strings. (#819) 
Rather than storing them as UserID objects.
 2016-06-02 16:28:54 +01:00
David Baker
812b5de0fe Merge remote-tracking branch 'origin/develop' into dbkr/email_unsubscribe 2016-06-02 15:33:28 +01:00
Mark Haines
80f34d7b57 Fix setting the _clock in SQLBaseStore 2016-06-02 15:23:56 +01:00
Mark Haines
661a540dd1 Deduplicate presence entries in sync (#818) 2016-06-02 15:20:28 +01:00
Mark Haines
70599ce925 Allow external processes to mark a user as syncing. (#812) 
* Add infrastructure to the presence handler to track sync requests in external processes

* Expire stale entries for dead external processes

* Add an http endpoint for making users as syncing

Add some docstrings and comments.

* Fixes
 2016-06-02 15:20:15 +01:00
David Baker
fb2193cc63 Merge pull request #817 from matrix-org/dbkr/split_out_auth_handler 
Split out the auth handler
 2016-06-02 14:31:35 +01:00
Erik Johnston
356f13c069 Disable INCLUDE_ALL_UNREAD_NOTIFS 2016-06-02 14:07:38 +01:00
Erik Johnston
02ac463dbf Merge pull request #800 from matrix-org/erikj/sync_refactor 
Refactor SyncHandler
 2016-06-02 14:02:13 +01:00
Matthew Hodgson
c5af1b6b00 Merge pull request #814 from matrix-org/matthew/3pid_invite_auth 
special case m.room.third_party_invite event auth to match invites,
 2016-06-02 13:47:40 +01:00
David Baker
3a3fb2f6f9 Merge branch 'dbkr/split_out_auth_handler' into dbkr/email_unsubscribe 2016-06-02 13:35:25 +01:00
David Baker
4a10510cd5 Split out the auth handler 2016-06-02 13:31:45 +01:00
Matthew Hodgson
f84b89f0c6 if an email pusher specifies a brand param, use it 2016-06-02 13:29:48 +01:00
David Baker
a15ad60849 Email unsubscribing that may in theory, work 
Were it not for that fact that you can't use the base handler in the pusher because it pulls in the world. Comitting while I fix that on a different branch.
 2016-06-02 11:44:15 +01:00
David Baker
e793866398 Use user_id in email greeting if display name is null 2016-06-02 09:41:13 +01:00
Matthew Hodgson
aaa70e26a2 special case m.room.third_party_invite event auth to match invites, otherwise they get out of sync and you get https://github.com/vector-im/vector-web/issues/1208 2016-06-01 22:13:47 +01:00
Erik Johnston
a04a2d043c Merge pull request #807 from matrix-org/erikj/push_rules_cache 
Ensure we always return boolean in push rules
 2016-06-01 18:07:48 +01:00
Erik Johnston
0f06b496d1 Merge pull request #806 from matrix-org/erikj/hash_cache 
Cache get_event_reference_hashes
 2016-06-01 18:07:42 +01:00
David Baker
e0deeff23e Fix room list spidering 2016-06-01 17:58:58 +01:00
David Baker
991af8b0d6 WIP on unsubscribing email notifs without logging in 2016-06-01 17:40:52 +01:00
David Baker
00c487a8db Merge pull request #808 from matrix-org/dbkr/room_list_spider 
Add secondary_directory_servers option to fetch room list from other servers
 2016-06-01 15:32:52 +01:00
Erik Johnston
c8285564a3 Use state to calculate get_users_in_room 2016-06-01 15:25:25 +01:00
David Baker
d60eed0710 Limit number of notifications in an email notification 2016-06-01 11:45:43 +01:00
Erik Johnston
43db0d9f6a Add get_users_with_read_receipts_in_room cache 2016-06-01 10:54:32 +01:00
David Baker
8e539f13c0 Merge remote-tracking branch 'origin/develop' into dbkr/room_list_spider 2016-06-01 09:54:36 +01:00
David Baker
6ecb2ca4ec pep8 2016-06-01 09:48:55 +01:00
David Baker
2a449fec4d Add cache to remote room lists 
Poll for updates from remote servers, waiting for the poll if there's no cache entry.
 2016-05-31 18:27:23 +01:00
David Baker
6ca4d3ae9a Add vector.im to default secondary_directory_servers and add comment explaining it's not a permanent solution 2016-05-31 17:24:50 +01:00
Erik Johnston
dea9f20f8c Force boolean 2016-05-31 17:24:30 +01:00
David Baker
963e3ed282 Apparently I am not permitted to have two blank lines here 2016-05-31 17:22:53 +01:00
David Baker
d240796ded Basic, un-cached support for secondary_directory_servers 2016-05-31 17:20:07 +01:00
Mark Haines
c8c5bf950a Fix synapse/storage/schema/delta/30/as_users.py 2016-05-31 17:10:40 +01:00
Erik Johnston
c9ca285d33 Merge pull request #805 from matrix-org/erikj/push_rules_cache 
Fix GET /push_rules
 2016-05-31 16:42:21 +01:00
Erik Johnston
1d4ee854e2 Fix typo 2016-05-31 15:45:53 +01:00
Erik Johnston
cca0093fa9 Change fix 2016-05-31 15:44:08 +01:00
Erik Johnston
4efa389299 Fix GET /push_rules 2016-05-31 15:37:53 +01:00
Erik Johnston
aefd2d1cbc Cache get_event_reference_hashes 2016-05-31 15:32:32 +01:00
Erik Johnston
10de8c2631 Merge pull request #804 from matrix-org/erikj/push_rules_cache 
Add caches to bulk_get_push_rules*
 2016-05-31 15:04:40 +01:00
Mark Haines
c626fc576a Move the AS handler out of the Handlers object. 
Access it directly from the homeserver itself. It already wasn't
inheriting from BaseHandler storing it on the Handlers object was
already somewhat dubious.
 2016-05-31 13:53:48 +01:00
Erik Johnston
e5b0bbcd33 Add caches to bulk_get_push_rules* 2016-05-31 13:46:58 +01:00
David Baker
70ecb415f5 Fix c+p fail 2016-05-31 12:00:54 +01:00
David Baker
e1625d62a8 Add federation room list servlet 2016-05-31 11:55:57 +01:00
David Baker
887c6e6f05 Split out the room list handler 
So I can use it from federation bits without pulling in all the handlers.
 2016-05-31 11:05:16 +01:00
Erik Johnston
85b992f621 Fix to allow start with postgres 2016-05-27 10:44:44 +01:00
Erik Johnston
cc84f7cb8e Send down correct error response if user not found 2016-05-27 10:35:15 +01:00
David Baker
209ba4d024 Merge pull request #795 from matrix-org/dbkr/delete_push_actions_after_a_month 
Only delete push actions after 30 days
 2016-05-24 16:22:13 +01:00
David Baker
b007ee4606 Check for presence of 'avatar_url' key 2016-05-24 15:12:05 +01:00
David Baker
cbf8d146ac Merge pull request #799 from matrix-org/matthew/quieter-email-notifs 
Tune email notifs to make them quieter:
 2016-05-24 14:30:51 +01:00
Erik Johnston
faad233ea6 Change short circuit path 2016-05-24 14:27:19 +01:00
Erik Johnston
6900303997 Don't send down all ephemeral events 2016-05-24 11:44:55 +01:00
David Baker
37b7e84620 Include the ts the notif was received at 2016-05-24 11:33:32 +01:00
Erik Johnston
1c5ed2a19b Only work out newly_joined_users for incremental sync 2016-05-24 11:21:34 +01:00
Erik Johnston
b08ad0389e Only include non-offline presence in initial sync 2016-05-24 11:15:05 +01:00
Erik Johnston
be2c677386 Spell builder correctly 2016-05-24 10:53:03 +01:00
Erik Johnston
79bea8ab9a Inline function. Make load_filtered_recents private 2016-05-24 10:22:24 +01:00
Erik Johnston
84f94e4cbb Add comments 2016-05-24 10:14:53 +01:00
Erik Johnston
137e6a4557 Shuffle things room 2016-05-24 09:50:55 +01:00
Matthew Hodgson
680f1d9387 catch thinko in presentable names 2016-05-23 22:55:11 +01:00
Matthew Hodgson
cb8a321bdd fix NPE in room ordering 2016-05-23 22:54:56 +01:00
Matthew Hodgson
88ea5ab2c3 consistency is the better part of valour 2016-05-23 19:33:45 +01:00
Matthew Hodgson
989bdc9e56 Tune email notifs to make them quieter: 
* After initial 10 minute window, only alert every 24h for room notifs
 * Reset room state after 6h of idleness
 * Synchronise throttles for messages sent in the same notif, so the 24 hourly notifs 'line up'
 * Fix the email subjects to say what triggered the notification
 * Order the rooms in reverse activity order in the email, so the 'reason' room should always come first
 2016-05-23 19:24:11 +01:00
Negi Fazeli
6fe04ffef2 Fix set profile error with Requester. 
Replace flush_user with delete access token due to function removal
Add a new test case for if the user is already registered
 2016-05-23 19:50:28 +02:00
David Baker
b791a530da Actually make the 'read' flag correct 2016-05-23 18:48:02 +01:00
David Baker
a24bc5b2dc Add GET /notifications API 2016-05-23 18:33:51 +01:00
Erik Johnston
c0c79ef444 Add back concurrently_execute 2016-05-23 18:21:27 +01:00
Erik Johnston
b5605dfecc Refactor SyncHandler 2016-05-23 18:08:18 +01:00
David Baker
31b5395ab6 Remove debug logging 2016-05-23 16:32:01 +01:00
David Baker
c2da3406fc Oops, missing comma 2016-05-20 18:03:31 +01:00
David Baker
ccffb0965d Remove stale line 2016-05-20 17:59:10 +01:00
David Baker
18d68bfee4 Handle empty events table 2016-05-20 17:58:09 +01:00
David Baker
d4503e25ed Make deleting push actions more efficient 
There's no index on received_ts, so manually binary search using the stream_ordering index, and only update it once an hour.
 2016-05-20 17:56:10 +01:00
David Baker
149fa411e2 Only delete push actions after 30 days 2016-05-20 15:25:12 +01:00
Kegan Dougal
332d7e9b97 Allow clients to specify a server_name to avoid 'No known servers' 
Multiple server_names are supported via ?server_name=foo&server_name=bar
 2016-05-19 13:50:52 +01:00
Matthew Hodgson
6fb51eaf7b Merge pull request #793 from matrix-org/matthew/one-push-badge-per-convo 
increment badge count per missed convo, not per msg
 2016-05-18 13:56:38 +01:00
Matthew Hodgson
e837df6adb increment badge count per missed convo, not per msg 2016-05-18 11:53:25 +01:00
Erik Johnston
42368ea8db Add desc to get_presence_for_users 2016-05-18 11:38:10 +01:00
Mark Haines
0cb441fedd Move typing handler out of the Handlers object 2016-05-17 15:58:46 +01:00
Mark Haines
6a30a0bfd3 Move the functions for parsing app service config 2016-05-17 11:28:58 +01:00
Mark Haines
523d5bcd0b Merge remote-tracking branch 'origin/develop' into markjh/liberate_sync_handler 2016-05-17 10:43:58 +01:00
Matthew Hodgson
43e1e0489c Merge pull request #786 from matrix-org/matthew/email_notifs_tuning 
tune email notifs, fix CSS a bit, and add debugging details
 2016-05-17 10:43:24 +01:00
Mark Haines
526bf8126f Remove unused get_joined_rooms_for_user 2016-05-17 10:20:51 +01:00
Mark Haines
425e6b4983 Merge branch 'develop' into markjh/member_cleanup 2016-05-17 10:13:16 +01:00
Mark Haines
b153f5b150 Merge pull request #787 from matrix-org/markjh/liberate_presence_handler 
Move the presence handler out of the Handlers object
 2016-05-17 10:09:43 +01:00
Mark Haines
3fd8a07fca Merge pull request #788 from matrix-org/markjh/domian 
Spell "domain" correctly
 2016-05-17 10:09:34 +01:00
Mark Haines
f68eea808a Move SyncHandler out of the Handlers object 2016-05-16 20:19:26 +01:00
Mark Haines
53e171f345 Merge branch 'markjh/liberate_presence_handler' into markjh/liberate_sync_handler 2016-05-16 20:08:32 +01:00
Mark Haines
80cb9becd8 Remove get_joined_rooms_for_user from RoomMemberHandler 2016-05-16 20:06:55 +01:00
Mark Haines
816df9f267 get_room_members is unused now 2016-05-16 19:51:43 +01:00
Mark Haines
821306120a Replaces calls to fetch_room_distributions_into with get_joined_hosts_for_room 2016-05-16 19:48:07 +01:00
Mark Haines
1a3a2002ff Spell "domain" correctly 
s/domian/domain/g
 2016-05-16 19:17:23 +01:00
Mark Haines
e168abbcff Don't inherit PresenceHandler from BaseHandler, remove references to self.hs from presence handler 2016-05-16 19:08:40 +01:00
Matthew Hodgson
cbd2adc95e tune email notifs, fix CSS a bit, and add debugging details 2016-05-16 18:58:38 +01:00
Mark Haines
3b86ecfa79 Move the presence handler out of the Handlers object 2016-05-16 18:56:37 +01:00
David Baker
647781ca56 Fix emailpusher import 
Try importing at the root level rather than conditionally importing, as per comment
 2016-05-16 18:41:32 +01:00
Erik Johnston
c39f305067 os.environ requires a string 2016-05-16 17:21:30 +01:00
Erik Johnston
678c8a7f1e Merge pull request #785 from matrix-org/erikj/cache_factor_synctly 
Make synctl read a cache factor from config file
 2016-05-16 17:07:35 +01:00
Erik Johnston
c5c5a7403b Make synctl read a cache factor from config file 2016-05-16 17:01:57 +01:00
Mark Haines
eb79110beb Clean up the blacklist/whitelist handling. 
Always set the config key with an empty list, even if a list isn't specified.
This means that the codepaths are the same for both the empty list and
for a missing key. Since the behaviour is the same for both cases this
makes the code somewhat easier to reason about.
 2016-05-16 13:03:59 +01:00
Mark Haines
dd95eb4cb5 Merge branch 'develop' into matthew/preview_url_ip_whitelist 2016-05-16 12:59:41 +01:00
Erik Johnston
60d53f9e95 Count number of GC collects 2016-05-16 09:34:42 +01:00
Matthew Hodgson
782471b7e1 fix matrix.to URLs 2016-05-13 17:50:16 +01:00
Mark Haines
0466454b00 Assert that stream replicated stream positions are ints 2016-05-13 17:33:44 +01:00
Mark Haines
b3f29dc1e5 Manually expire broken caches like the who_forgot_in_room 2016-05-13 17:16:27 +01:00
Mark Haines
f03ddc98ec Use the SlavedAccountDataStore 2016-05-13 17:01:28 +01:00
Mark Haines
1f71f386f6 Merge branch 'develop' into dbkr/email_notifs_on_pusher 2016-05-13 16:59:56 +01:00
Mark Haines
206eb9fd94 Shift some of the state_group methods into the SlavedEventStore 2016-05-13 16:58:14 +01:00
Erik Johnston
7d6e89ed22 Add a comment 2016-05-13 16:31:08 +01:00
Mark Haines
21018c2c13 Merge pull request #783 from matrix-org/markjh/slave_account_data 
Add a slaved datastore for account data
 2016-05-13 15:56:04 +01:00
Mark Haines
b7381d5338 Allow receipts for events we haven't seen in the db 2016-05-13 15:46:41 +01:00
Mark Haines
3abab26458 Add a slaved datastore for account data 2016-05-13 15:34:06 +01:00
Erik Johnston
99b5a2e560 Merge pull request #741 from negzi/create_user_with_expiry 
Create user with expiry
 2016-05-13 14:46:53 +01:00
Erik Johnston
ba5c616ff4 Merge pull request #778 from matrix-org/erikj/add_pusher 
Fixup add_pusher
 2016-05-13 14:43:23 +01:00