Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-10-01 08:25:44 -04:00
Code Issues Packages Projects Releases Wiki Activity
8,412 Commits 4 Branches 609 Tags 60 MiB
4bd597d9fc
Commit Graph

5982 Commits

Author SHA1 Message Date
Erik Johnston
0163466d72 Ignore AS users when fetching push rules 
By ignoring AS users early on when fetching push rules for a room we can
avoid needlessly hitting the DB and filling up the caches.
 2016-11-23 11:01:01 +00:00
Erik Johnston
4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston
54fed21c04 Fix tests and flake8 2016-11-22 18:18:31 +00:00
Erik Johnston
90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00
Kegan Dougal
83bcdcee61 Return early on /sync code paths if a '*' filter is used 
This is currently very conservative in that it only does this if there is no
`since` token. This limits the risk to clients likely to be doing one-off
syncs (like bridges), but does mean that normal human clients won't benefit
from the time savings here. If the savings are large enough, I would consider
generalising this to just check the filter.
 2016-11-22 16:38:35 +00:00
Kegsay
d4a459f7cb Merge pull request #1638 from matrix-org/kegan/sync-event-fields 
Implement "event_fields" in filters
 2016-11-22 14:02:38 +00:00
Kegan Dougal
c3d963ac24 Review comments 2016-11-22 13:42:11 +00:00
Kegan Dougal
6d4e6d4cba Also check for dict since sometimes they aren't frozen 2016-11-22 10:39:41 +00:00
Erik Johnston
baf9e74a73 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-11-22 10:31:48 +00:00
Kegan Dougal
cea4e4e7b2 Glue only_event_fields into the sync rest servlet 2016-11-22 10:14:05 +00:00
Kegan Dougal
0a8b0eeca1 More tests 2016-11-22 09:59:27 +00:00
Erik Johnston
51e89709aa Comments 2016-11-21 17:59:39 +00:00
Kegan Dougal
70a2157b64 Start adding some tests 2016-11-21 17:52:45 +00:00
Kegan Dougal
f97511a1f3 Move event_fields filtering to serialize_event 
Also make it an inclusive not exclusive filter, as the spec demands.
 2016-11-21 17:42:16 +00:00
Erik Johnston
88d85ebae1 Add some metrics 2016-11-21 17:36:05 +00:00
Erik Johnston
50934ce460 Comments 2016-11-21 16:55:23 +00:00
Kegan Dougal
e90fcd9edd Add filter_event_fields and filter_field to FilterCollection 2016-11-21 15:18:18 +00:00
Erik Johnston
9687e039e7 Remove explicit calls to send_pdu 2016-11-21 14:48:51 +00:00
Kegan Dougal
a2a6c1c22f Fail with a coherent error message if /sync?filter= is invalid 2016-11-21 13:15:25 +00:00
Erik Johnston
524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Mark Haines
a289150943 Fix flake8 2016-11-18 17:15:02 +00:00
David Baker
544722bad2 Work around client replacing reg params 
Works around https://github.com/vector-im/vector-android/issues/715
and equivalent for iOS
 2016-11-18 17:07:35 +00:00
Erik Johnston
f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston
1587b5a033 Add initial cut of federation send queue 2016-11-16 14:47:52 +00:00
Erik Johnston
59ef517e6b Use new federation_sender DI 2016-11-16 14:47:52 +00:00
Erik Johnston
847d5db1d1 Add transaction queue and transport layer to DI 2016-11-16 14:47:52 +00:00
Erik Johnston
daec6fc355 Move logic into transaction_queue 2016-11-16 14:47:52 +00:00
Erik Johnston
0e830d3770 Rename transaction queue functions to send_* 2016-11-16 14:47:52 +00:00
Erik Johnston
dc6cede78e Merge pull request #1628 from matrix-org/erikj/ldap_split_out 
Use external ldap auth pacakge
 2016-11-15 16:53:34 +00:00
Erik Johnston
c7546b3cdb Merge pull request #1617 from matrix-org/erikj/intern_state_dict 
Correctly intern keys in state cache
 2016-11-15 16:45:55 +00:00
Erik Johnston
d56c39cf24 Use external ldap auth pacakge 2016-11-15 13:03:19 +00:00
Erik Johnston
f9d156d270 New Flake8 fixes 2016-11-15 11:22:29 +00:00
Erik Johnston
9d58ccc547 Bump changelog and version 2016-11-14 15:05:04 +00:00
Kegsay
9355a5c42b Merge pull request #1624 from matrix-org/kegan/idempotent-requests 
Store Promise<Response> instead of Response for HTTP API transactions
 2016-11-14 12:45:30 +00:00
Kegan Dougal
3991b4cbdb Clean transactions based on time. Add HttpTransactionCache tests. 2016-11-14 11:19:24 +00:00
Kegan Dougal
af4a1bac50 Move .observe() up to the cache to make things neater 2016-11-14 09:52:41 +00:00
Erik Johnston
0964005d84 Merge pull request #1625 from DanielDent/patch-1 
Add support for durations in minutes
 2016-11-12 11:20:46 +00:00
Daniel Dent
1c93cd9f9f Add support for durations in minutes 2016-11-12 00:10:23 -08:00
Kegan Dougal
8ecaff51a1 Review comments 2016-11-11 17:47:03 +00:00
Kegan Dougal
f6c48802f5 More flake8 2016-11-11 15:08:24 +00:00
Kegan Dougal
a88bc67f88 Flake8 and fix whoopsie 2016-11-11 15:02:29 +00:00
Kegan Dougal
42c43cfafd Use ObservableDeferreds instead of Deferreds as they behave as intended 2016-11-11 14:54:10 +00:00
Kegan Dougal
c7daf3136c Use observable deferreds because they are sane 2016-11-11 14:13:32 +00:00
Erik Johnston
64038b806c Comments 2016-11-11 10:42:08 +00:00
Erik Johnston
2bd4513a4d Limit the number of events that can be created on a given room concurretnly 2016-11-10 16:44:35 +00:00
Erik Johnston
d073cb7ead Add Limiter: limit concurrent access to resource 2016-11-10 16:29:51 +00:00
Kegan Dougal
8a8ad46f48 Flake8 2016-11-10 15:22:11 +00:00
Kegan Dougal
2771447c29 Store Promise<Response> instead of Response for HTTP API transactions 
This fixes a race whereby:
 - User hits an endpoint.
 - No cached transaction so executes main code.
 - User hits same endpoint.
 - No cache transaction so executes main code.
 - Main code finishes executing and caches response and returns.
 - Main code finishes executing and caches response and returns.

 This race is common in the wild when Synapse is struggling under load.
 This commit fixes the race by:
  - User hits an endpoint.
  - Caches the promise to execute the main code and executes main code.
  - User hits same endpoint.
  - Yields on the same promise as the first request.
  - Main code finishes executing and returns, unblocking both requests.
 2016-11-10 14:49:26 +00:00
Erik Johnston
ac507e7ab8 Don't assume providers raise ConfigError's 2016-11-08 17:23:28 +00:00
Erik Johnston
e6651e8046 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-11-08 14:43:49 +00:00
Erik Johnston
3c09818d91 Bump version and changelog 2016-11-08 14:39:55 +00:00
Erik Johnston
27d3f2e7ab Explicitly set authentication mode in ldap3 
This only makes a difference for versions of ldap3 before 1.0, but a)
its best to be explicit and b) there are distributions that package
ancient versions for ldap3 (e.g. debian).
 2016-11-08 14:35:25 +00:00
Erik Johnston
17e0a58020 Merge pull request #1615 from matrix-org/erikj/limit_prev_events 
Limit the number of prev_events of new events
 2016-11-08 12:06:15 +00:00
Erik Johnston
587d8ac60f Correctly intern keys in state cache 2016-11-08 11:53:25 +00:00
Erik Johnston
a4632783fb Sample correctly 2016-11-08 11:20:26 +00:00
Erik Johnston
24772ba56e Respect use_frozen_dicts option in workers 2016-11-08 11:07:18 +00:00
Erik Johnston
eeda4e618c Limit the number of prev_events of new events 2016-11-08 11:02:29 +00:00
Erik Johnston
d24197bead Merge pull request #1198 from euank/more-ip-blacklist 
default config: blacklist more internal ips
 2016-11-07 09:41:34 +00:00
Euan Kemp
c6bbad109b default config: blacklist more internal ips 2016-11-06 17:02:25 -08:00
Erik Johnston
63772443e6 Comment 2016-11-04 10:53:42 +00:00
Erik Johnston
a3f6576084 Remove unused but buggy function 2016-11-04 10:48:20 +00:00
Paul "LeoNerd" Evans
2938a00825 Rename the python-specific metrics now the docs claim that we have done 2016-11-03 17:03:52 +00:00
Paul "LeoNerd" Evans
5219f7e060 Since we don't export per-filetype fd counts any more, delete all the code related to that too 2016-11-03 16:41:32 +00:00
Paul "LeoNerd" Evans
93ebeb2aa8 Remove now-unused 'resource' import 2016-11-03 16:37:09 +00:00
Paul "LeoNerd" Evans
c1b077cd19 Now we have new-style metrics don't bother exporting legacy-named process ones 2016-11-03 16:34:16 +00:00
Erik Johnston
64c6566980 Remove spurious comment 2016-11-03 15:04:32 +00:00
Erik Johnston
8fd4d9129f Replace postgres GIN with GIST 
This is because GIN can be slow to write too, especially when the table
gets large.
 2016-11-03 15:00:03 +00:00
David Baker
9084720993 Don't error on non-ascii passwords 2016-11-03 10:42:14 +00:00
Mark Haines
b1c27975d0 Set CORs headers on responses from the media repo 2016-11-02 11:29:25 +00:00
Erik Johnston
2746e805fe Merge pull request #1188 from matrix-org/erikj/sent_transactions 
Remove sent_transactions table.
 2016-11-01 14:52:55 +00:00
Erik Johnston
3c91c5b216 Bump version and changelog 2016-11-01 13:06:36 +00:00
Erik Johnston
760469c812 Continue to clean up received_transactions 2016-11-01 11:42:08 +00:00
Erik Johnston
f09d2b692f Removed unused stuff 2016-10-31 17:10:56 +00:00
Erik Johnston
4c3eb14d68 Increase batching of sent transaction inserts 
This should further reduce the number of individual inserts,
transactions and updates that are required for keeping sent_transactions
up to date.
 2016-10-31 16:07:45 +00:00
Mark Haines
82e71a259c Bump changelog and version 2016-10-28 11:16:05 +01:00
Mark Haines
490b97d3e7 Merge branch 'develop' into release-v0.18.2 2016-10-28 11:12:31 +01:00
Paul "LeoNerd" Evans
1cc22da600 Set up the process collector during metrics __init__; that way all split-process workers have it 2016-10-27 18:09:34 +01:00
Paul "LeoNerd" Evans
aac13b1f9a Pass the Metrics group into the process collector instead of having it find its own one; this avoids it needing to import from synapse.metrics 2016-10-27 18:08:15 +01:00
Paul "LeoNerd" Evans
ccc1a3d54d Allow creation of a 'subspace' within a Metrics object, returning another one 2016-10-27 18:07:34 +01:00
Erik Johnston
665e53524e Bump changelog and version 2016-10-27 14:52:47 +01:00
Erik Johnston
a9111786f9 Use most recently added binding, not most recently seen user. 2016-10-27 14:32:45 +01:00
Erik Johnston
1fc1bc2a51 Fix user_threepids schema delta 
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.

This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
 2016-10-27 14:14:44 +01:00
Erik Johnston
ab731d8f8e Bump changelog and version 2016-10-27 11:07:02 +01:00
Mark Haines
177f104432 Merge pull request #1098 from matrix-org/markjh/bearer_token 
Allow clients to supply access_tokens as headers
 2016-10-25 17:33:15 +01:00
Erik Johnston
22fbf86e4f Merge branch 'release-v0.18.2' of github.com:matrix-org/synapse into develop 2016-10-25 15:09:46 +01:00
Erik Johnston
855645c719 Bump version and changelog 2016-10-25 11:16:16 +01:00
Erik Johnston
2ef617bc06 Fix infinite typing bug 
There's a bug somewhere that causes typing notifications to not be timed
out properly. By adding a paranoia timer and using correct inequalities
notifications should stop being stuck, even if it the root cause hasn't
been fixed.
 2016-10-24 15:51:22 +01:00
Erik Johnston
b6800a8ecd Actually use the new function 2016-10-24 13:39:49 +01:00
Erik Johnston
d04e2ff3a4 Fix incredubly slow back pagination query 
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
 2016-10-24 13:35:51 +01:00
Paul Evans
a842fed418 Merge pull request #1177 from matrix-org/paul/standard-metric-names 
Standardise prometheus metrics
 2016-10-21 13:06:19 +01:00
Luke Barnard
e01a1bc92d Merge pull request #1175 from matrix-org/luke/feature-configurable-as-rate-limiting 
Allow Configurable Rate Limiting Per AS
 2016-10-20 16:21:10 +01:00
Luke Barnard
6fdd31915b Style 2016-10-20 13:53:15 +01:00
Luke Barnard
07caa749bf Closing brace on following line 2016-10-20 12:07:16 +01:00
Luke Barnard
f09db236b1 as_user->app_service, less redundant comments, better positioned comments 2016-10-20 12:04:54 +01:00
Luke Barnard
8bfd01f619 flake8 2016-10-20 11:52:46 +01:00
Luke Barnard
1b17d1a106 Use real AS object by passing it through the requester 
This means synapse does not have to check if the AS is interested, but instead it effectively re-uses what it already knew about the requesting user
 2016-10-20 11:43:05 +01:00
Paul "LeoNerd" Evans
b01aaadd48 Split callback metric lambda functions down onto their own lines to keep line lengths under 90 2016-10-19 18:26:13 +01:00
Paul "LeoNerd" Evans
1071c7d963 Adjust code for <100 char line limit 2016-10-19 18:23:25 +01:00
Paul "LeoNerd" Evans
6453d03edd Cut the raw /proc/self/stat line up into named fields at collection time 2016-10-19 18:21:40 +01:00
Paul "LeoNerd" Evans
3ae48a1f99 Move the process metrics collector code into its own file 2016-10-19 18:10:24 +01:00
Paul "LeoNerd" Evans
4cedd53224 A slightly neater way to manage metric collector functions 2016-10-19 17:54:09 +01:00
Paul "LeoNerd" Evans
5663137e03 appease pep8 2016-10-19 16:09:42 +01:00
Paul "LeoNerd" Evans
b202531be6 Also guard /proc/self/fds-related code with a suitable psuedoconstant 2016-10-19 15:37:41 +01:00
Paul "LeoNerd" Evans
1b179455fc Guard registration of process-wide metrics by existence of the requisite /proc entries 2016-10-19 15:34:38 +01:00
Paul "LeoNerd" Evans
981f852d54 Add standard process_start_time_seconds metric 2016-10-19 15:05:22 +01:00
Paul "LeoNerd" Evans
def63649df Add standard process_max_fds metric 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans
06f1ad1625 Add standard process_open_fds metric 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans
95fc70216d Add standard process_*_memory_bytes metrics 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans
9b0316c75a Use /proc/self/stat to generate the new process_cpu_*_seconds_total metrics 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans
03c2720940 Export CPU usage metrics also under prometheus-standard metric name 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans
b21b9dbc37 Callback metric values might not just be integers - allow floats 2016-10-19 15:05:15 +01:00
Erik Johnston
78c083f159 Merge pull request #1164 from pik/error-codes 
Clarify Error codes for GET /filter/
 2016-10-19 14:26:17 +01:00
Erik Johnston
3aa8925091 Merge pull request #1176 from matrix-org/erikj/eager_ratelimit_check 
Check whether to ratelimit sooner to avoid work
 2016-10-19 14:25:52 +01:00
Erik Johnston
f2f74ffce6 Comment 2016-10-19 14:21:28 +01:00
David Baker
0108ed8ae6 Latest delta is now 37 2016-10-19 11:40:35 +01:00
David Baker
a7f48320b1 Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive 2016-10-19 11:28:56 +01:00
David Baker
df2a616c7b Convert emails to lowercase when storing 
And db migration sql to convert existing addresses.
 2016-10-19 11:13:55 +01:00
Erik Johnston
550308c7a1 Check whether to ratelimit sooner to avoid work 2016-10-19 10:45:24 +01:00
pik
e8b1d2a452 Refactor test_filter to use real DataStore 
* add tests for filter api errors
 2016-10-18 12:17:38 -05:00
Luke Barnard
5b54d51d1e Allow Configurable Rate Limiting Per AS 
This adds a flag loaded from the registration file of an AS that will determine whether or not its users are rate limited (by ratelimit in _base.py). Needed for IRC bridge reasons - see https://github.com/matrix-org/matrix-appservice-irc/issues/240.
 2016-10-18 17:04:09 +01:00
Erik Johnston
8ca05b5755 Fix push notifications for a single unread message 2016-10-18 10:57:33 +01:00
Erik Johnston
f0ca088280 Reduce redundant database work in email pusher 
Update the last stream ordering if the
`get_unread_push_actions_for_user_in_range_for_email` returns no new
push actions. This reduces the range that it needs to check next
iteration.
 2016-10-18 10:52:47 +01:00
Erik Johnston
b95dbdcba4 Bump version and changelog 2016-10-17 15:59:12 +01:00
Erik Johnston
6942d68247 Bump schema version 2016-10-17 11:17:45 +01:00
Erik Johnston
b59994b454 Remove TODO 2016-10-17 11:17:02 +01:00
Erik Johnston
816988baaa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth 2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7 Drop some unused indices 2016-10-17 11:08:19 +01:00
Erik Johnston
a68ade6ed3 Merge pull request #1162 from larroy/master 
Use sys.executable instead of hardcoded python. fixes #1161
 2016-10-14 21:42:55 +01:00
David Baker
29c5922021 Revert part of 6207399 
older sqlite doesn't support indexes on expressions, lets just
store things lowercase in the db
 2016-10-14 16:20:24 +01:00
Alexander Maznev
d9350b0db8 Error codes for filters 
* add tests

Signed-off-by: Alexander Maznev <alexander.maznev@gmail.com>
 2016-10-14 10:18:28 -05:00
David Baker
bcb1245a2d Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive 2016-10-14 15:10:38 +01:00
David Baker
62073992c5 Make password reset email field case insensitive 2016-10-14 13:56:53 +01:00
Erik Johnston
6f7540ada4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_email_notifs 2016-10-14 10:22:43 +01:00
Erik Johnston
1d107d8484 Fix email push notifs being dropped 
A lot of email push notifications were failing to be sent due to an
exception being thrown along one of the (many) paths. This was due to a
change where we moved from pulling out the full state for each room, but
rather pulled out the event ids for the state and separately loaded the
full events when needed.
 2016-10-13 13:40:38 +01:00
Richard van der Hoff
f7aed3d7a2 Merge pull request #1168 from matrix-org/rav/ui_auth_on_device_delete 
User-interactive auth on delete device
 2016-10-13 09:38:41 +01:00
Richard van der Hoff
9009143fb9 Handle delete device requests with no body 
We should probably return a 401 rather than a 400 for existing clients that
don't know they have to do the UIA dance to delete a device.
 2016-10-12 18:47:28 +01:00
Richard van der Hoff
fbd3866bc6 User-interactive auth on delete device 2016-10-12 16:16:31 +01:00
Mark Haines
9e18e0b1cb Merge pull request #1167 from matrix-org/markjh/fingerprints 
Add config option for adding additional TLS fingerprints
 2016-10-12 15:27:44 +01:00
Mark Haines
c61ddeedac Explain how long the servers can cache the TLS fingerprints for 2016-10-12 14:48:24 +01:00
Mark Haines
0af6213019 Improve comment formatting 2016-10-12 14:45:13 +01:00
Erik Johnston
35e2cc8b52 Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth 
Implement pluggable password auth
 2016-10-12 11:41:20 +01:00
Mark Haines
6e9f3ab415 Add config option for adding additional TLS fingerprints 2016-10-11 19:14:46 +01:00
Erik Johnston
3061dac53e Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop 2016-10-11 14:08:29 +01:00
Erik Johnston
668f91d707 Fix check of wrong variable 2016-10-11 13:57:22 +01:00
Richard van der Hoff
0061e8744f Merge pull request #1166 from matrix-org/rav/grandfather_broken_riot_signup 
Work around email-spamming Riot bug
 2016-10-11 11:58:58 +01:00
Richard van der Hoff
fa74fcf512 Work around email-spamming Riot bug 
5d9546f9 introduced a change to synapse behaviour, in that failures in the
interactive-auth process would return the flows and params data as well as an
error code (as specced in https://github.com/matrix-org/matrix-doc/pull/397).

That change exposed a bug in Riot which would make it request a new validation
token (and send a new email) each time it got a 401 with a `flows` parameter
(see https://github.com/vector-im/vector-web/issues/2447 and the fix at
https://github.com/matrix-org/matrix-react-sdk/pull/510).

To preserve compatibility with broken versions of Riot, grandfather in the old
behaviour for the email validation stage.
 2016-10-11 11:34:40 +01:00
Erik Johnston
a2f2516199 Merge pull request #1157 from Rugvip/nolimit 
Remove rate limiting from app service senders and fix get_or_create_user requester
 2016-10-11 11:20:54 +01:00
Erik Johnston
a940618c94 Merge pull request #1150 from Rugvip/state_key 
api/auth: fix for not being allowed to set your own state_key
 2016-10-11 11:19:55 +01:00
Pedro Larroy
c57f871184 Use sys.executable instead of hardcoded python. fixes #1161 2016-10-08 23:55:20 +02:00
Richard van der Hoff
8681aff4f1 Merge pull request #1160 from matrix-org/rav/401_on_password_fail 
Interactive Auth: Return 401 from for incorrect password
 2016-10-07 10:57:43 +01:00
Richard van der Hoff
5d9546f9f4 Interactive Auth: Return 401 from for incorrect password 
This requires a bit of fettling, because I want to return a helpful error
message too but we don't want to distinguish between unknown user and invalid
password. To avoid hardcoding the error message into 15 places in the code,
I've had to refactor a few methods to return None instead of throwing.

Fixes https://matrix.org/jira/browse/SYN-744
 2016-10-07 00:00:00 +01:00
Patrik Oldsberg
7b5546d077 rest/client/v1/register: use the correct requester in createUser 
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
 2016-10-06 22:12:32 +02:00
Richard van der Hoff
5d34e32d42 Merge pull request #1159 from matrix-org/rav/uia_fallback_postmessage 
window.postmessage for Interactive Auth fallback
 2016-10-06 19:56:43 +01:00
Richard van der Hoff
f382117852 window.postmessage for Interactive Auth fallback 
If you're a webapp running the fallback in an iframe, you can't set set a
window.onAuthDone function. Let's post a message back to window.opener instead.
 2016-10-06 18:16:59 +01:00
Patrik Oldsberg
3de7c8a4d0 handlers/profile: added admin override for set_displayname and set_avatar_url 
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
 2016-10-06 15:24:59 +02:00
Patrik Oldsberg
2ff2d36b80 handers: do not ratelimit app service senders 
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
 2016-10-06 15:24:59 +02:00
Patrik Oldsberg
9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Erik Johnston
4285be791d Bump changelog and version 2016-10-05 14:40:38 +01:00
Erik Johnston
850b103b36 Implement pluggable password auth 
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
 2016-10-03 10:36:40 +01:00
Patrik Oldsberg
24a70e19c7 api/auth: fix for not being allowed to set your own state_key 
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
 2016-09-30 13:08:25 +02:00
Erik Johnston
04aa2f2863 Bump version and changelog 2016-09-30 10:34:57 +01:00
Martin Weinelt
3027ea22b0 Restructure ldap authentication 
- properly parse return values of ldap bind() calls
- externalize authentication methods
- change control flow to be more error-resilient
- unbind ldap connections in many places
- improve log messages and loglevels
 2016-09-29 15:30:08 +01:00
Erik Johnston
5875a65253 Merge pull request #1145 from matrix-org/erikj/fix_reindex 
Fix background reindex of origin_server_ts
 2016-09-29 13:53:48 +01:00
Erik Johnston
9040c9ffa1 Fix background reindex of origin_server_ts 
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
 2016-09-27 11:23:49 +01:00
Erik Johnston
4974147aa3 Remove duplication 2016-09-27 09:27:54 +01:00
Erik Johnston
13122e5e24 Remove unused variable 2016-09-27 09:21:51 +01:00
Erik Johnston
cf3e1cc200 Fix perf of fetching state in SQLite 2016-09-26 17:16:24 +01:00
Erik Johnston
a38d46249e Merge pull request #1140 from matrix-org/erikj/typing_fed_timeout 
Time out typing over federation
 2016-09-26 11:24:14 +01:00
Erik Johnston
748d8fdc7b Reduce DB hits for replication 
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.

This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
 2016-09-23 16:49:21 +01:00
Erik Johnston
655891d179 Move FEDERATION_PING_INTERVAL timer. Update log line 2016-09-23 15:43:34 +01:00
Erik Johnston
4225a97f4e Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-09-23 15:36:59 +01:00
Erik Johnston
22578545a0 Time out typing over federation 2016-09-23 14:00:52 +01:00
Erik Johnston
667fcd54e8 Merge pull request #1136 from matrix-org/erikj/fix_signed_3pid 
Allow invites via 3pid to bypass sender sig check
 2016-09-22 13:41:49 +01:00
Erik Johnston
f96020550f Update comments 2016-09-22 12:54:22 +01:00
Erik Johnston
81964aeb90 Merge pull request #1132 from matrix-org/erikj/initial_sync_split 
Support /initialSync in synchrotron worker
 2016-09-22 12:45:02 +01:00
Erik Johnston
2e9ee30969 Add comments 2016-09-22 11:59:46 +01:00
Erik Johnston
a61e4522b5 Shuffle things around to make unit tests work 2016-09-22 11:08:12 +01:00
Erik Johnston
1168cbd54d Allow invites via 3pid to bypass sender sig check 
When a server sends a third party invite another server may be the one
that the inviting user registers with. In this case it is that remote
server that will issue an actual invitation, and wants to do it "in the
name of" the original invitee. However, the new proper invite will not
be signed by the original server, and thus other servers would reject
the invite if it was seen as coming from the original user.

To fix this, a special case has been added to the auth rules whereby
another server can send an invite "in the name of" another server's
user, so long as that user had previously issued a third party invite
that is now being accepted.
 2016-09-22 10:56:53 +01:00
Erik Johnston
bbc0d9617f Merge pull request #1134 from matrix-org/erikj/fix_stream_public_deletion 
Fix _delete_old_forward_extrem_cache query
 2016-09-21 17:04:04 +01:00
Erik Johnston
8009d84364 Match against event_id, rather than room_id 2016-09-21 16:46:59 +01:00
Erik Johnston
dc692556d6 Remove spurious AS clause 2016-09-21 16:28:47 +01:00
Erik Johnston
dc78db8c56 Update correct table 2016-09-21 15:52:44 +01:00
Erik Johnston
4f78108d8c Readd entries to public_room_list_stream that were deleted 2016-09-21 15:24:22 +01:00
Erik Johnston
0b78d8adf2 Fix _delete_old_forward_extrem_cache query 2016-09-21 15:20:56 +01:00
Erik Johnston
90c070c850 Add total_room_count_estimate to /publicRooms 2016-09-21 13:30:05 +01:00
Erik Johnston
87528f0756 Support /initialSync in synchrotron worker 2016-09-21 11:46:28 +01:00
Erik Johnston
2b8ff4659f Bump version and changelog 2016-09-19 17:16:56 +01:00
Erik Johnston
ddfcdd4778 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.18.0 2016-09-19 17:15:24 +01:00
Erik Johnston
6f0c5e5d9b Merge pull request #1131 from matrix-org/matthew/e2e-notifs 
Notify on e2e events
 2016-09-19 10:50:26 +01:00
Erik Johnston
49cf205dc7 _id field must uniquely identify different conditions 2016-09-19 10:34:01 +01:00
Erik Johnston
39af634dd2 Merge pull request #1130 from matrix-org/erikj/fix_pubroom_pag 
Handle fact that _generate_room_entry may not return a room entry
 2016-09-19 10:13:59 +01:00
Matthew Hodgson
3f6ec271ba proposal for notifying on e2e events 2016-09-17 22:05:06 +01:00
Erik Johnston
4d49e0bdfd PEP8 2016-09-17 18:09:22 +01:00
Erik Johnston
81570abfb2 Handle fact that _generate_room_entry may not return a room entry 2016-09-17 18:01:54 +01:00
Erik Johnston
ddc89df89d Enable guest access to POST /publicRooms 2016-09-17 15:55:24 +01:00
Erik Johnston
eb24aecf8c Merge pull request #1129 from matrix-org/erikj/fix_pubroom_pag 
Fix and clean up publicRooms pagination
 2016-09-17 15:30:34 +01:00
Erik Johnston
e1ba98d724 Merge pull request #1127 from matrix-org/dbkr/publicroom_search_case_insensitive 
Make public room search case insensitive
 2016-09-17 15:01:17 +01:00
Erik Johnston
a298331de4 Spelling 2016-09-17 14:59:40 +01:00
Erik Johnston
71edaae981 Fix and clean up publicRooms pagination 2016-09-17 14:46:19 +01:00
Matthew Hodgson
883df2e983 fix logger for client_reader worker 2016-09-17 14:12:04 +01:00
David Baker
5336acd46f Make public room search case insensitive 2016-09-16 19:02:42 +01:00
Erik Johnston
19fe990476 Update changelog and bump version 2016-09-16 17:30:59 +01:00
Erik Johnston
995f2f032f Fix public room pagination for client_reader app 2016-09-16 14:48:21 +01:00
Erik Johnston
a68807d426 Comment 2016-09-16 11:36:20 +01:00
Erik Johnston
2e67cabd7f Make POST /publicRooms require auth 2016-09-16 11:32:51 +01:00
Erik Johnston
b7b62bf9ea Comment 2016-09-16 11:00:29 +01:00
Erik Johnston
d84319ae10 Add remote reoom cache 2016-09-16 10:31:59 +01:00
Erik Johnston
23b6701a28 Support filtering remote room lists 2016-09-16 10:24:15 +01:00
Erik Johnston
e58a9d781c Filter remote rooms lists locally 2016-09-16 10:19:32 +01:00
Erik Johnston
74d4cdee25 Don't cache searched in /publicRooms 2016-09-16 09:05:11 +01:00
Erik Johnston
418bcd4309 Add new storage function to slave store 2016-09-16 08:37:39 +01:00
Erik Johnston
098db4aa52 Add very basic filter API to /publicRooms 2016-09-15 17:50:16 +01:00
Erik Johnston
c33b25fd8d Change the way we calculate new_limit in /publicRooms and add POST API 2016-09-15 17:35:20 +01:00
Erik Johnston
de4f798f01 Handling expiring stream extrems correctly. 2016-09-15 17:34:59 +01:00
Erik Johnston
ea6dc356b0 Merge pull request #1125 from matrix-org/erikj/public_room_cache 
Change get_pos_of_last_change to return upper bound
 2016-09-15 15:48:53 +01:00
Erik Johnston
955f34d23e Change get_pos_of_last_change to return upper bound 2016-09-15 15:12:07 +01:00
Erik Johnston
241d7d2d62 Merge pull request #1124 from matrix-org/erikj/enable_state_caching_workers 
Enable state caches on workers
 2016-09-15 15:01:47 +01:00
Erik Johnston
4be85281f9 Enable state caches on workers 2016-09-15 14:31:22 +01:00
Erik Johnston
cb3edec6af Use stream_change cache to make get_forward_extremeties_for_room cache more effective 2016-09-15 14:28:13 +01:00
Erik Johnston
55e6fc917c Add cache to get_forward_extremeties_for_room 2016-09-15 14:04:28 +01:00
Erik Johnston
68c1ed4d1a Remove default public rooms limit 2016-09-15 13:56:20 +01:00
Erik Johnston
b82fa849c8 Merge pull request #1120 from matrix-org/erikj/push_invite_cache 
Ensure we don't mutate the cache of push rules
 2016-09-15 13:27:18 +01:00
Erik Johnston
e457034e99 Merge pull request #1121 from matrix-org/erikj/public_room_paginate 
Add pagination support to publicRooms
 2016-09-15 13:27:09 +01:00
Erik Johnston
1d98cf26be By default limit /publicRooms to 100 entries 2016-09-15 13:18:35 +01:00
Erik Johnston
211786ecd6 Stream public room changes down replication 2016-09-15 11:47:23 +01:00
Erik Johnston
4fb65a1091 Base public room list off of public_rooms stream 2016-09-15 11:27:04 +01:00
Erik Johnston
5810cffd33 Pass since/from parameters over federation 2016-09-15 10:36:19 +01:00
Erik Johnston
f3eead0660 Allow paginating both forwards and backwards 2016-09-15 10:15:37 +01:00
Erik Johnston
4131381123 Remove support for aggregate room lists 2016-09-15 09:28:15 +01:00
Erik Johnston
6a5ded5988 Ensure we don't mutate the cache of push rules 2016-09-15 09:16:13 +01:00
Erik Johnston
4f181f361d Accept optional token to public room list 2016-09-15 09:08:57 +01:00
Erik Johnston
c566f0ee17 Calculate the public room list from a stream_ordering 2016-09-14 17:42:47 +01:00
Erik Johnston
772c6067a3 Refactor public rooms to not pull out the full state for each room 2016-09-14 17:29:25 +01:00
Erik Johnston
baffe96d95 Add a room visibility stream 2016-09-14 17:29:19 +01:00
Erik Johnston
264a48aedf Merge pull request #1117 from matrix-org/erikj/fix_state 
Ensure we don't mutate state cache entries
 2016-09-14 16:50:37 +01:00
Erik Johnston
ed992ae6ba Add a DB index to figure out past state at a stream ordering in a room 2016-09-14 16:20:27 +01:00
Erik Johnston
e0b6db29ed Split out public room list into a worker process 2016-09-14 14:42:51 +01:00
Erik Johnston
a70a43bc51 Move RoomListHandler into a separate file 2016-09-14 14:07:37 +01:00
Erik Johnston
f2b2cd8eb4 Amalgamate two identical consecutive if statements 2016-09-14 11:16:22 +01:00
Erik Johnston
00f51493f5 Fix reindex 2016-09-14 10:18:30 +01:00
Erik Johnston
d5ae1f1291 Ensure we don't mutate state cache entries 2016-09-14 10:03:48 +01:00
Matthew Hodgson
1b01488d27 Merge pull request #1111 from matrix-org/matthew/device-ids 
make device IDs more useful for human disambiguation
 2016-09-14 01:18:02 +01:00
Paul Evans
0f73f0e70e Merge pull request #1116 from matrix-org/paul/tiny-fixes 
Fix typo "persiting"
 2016-09-13 13:27:42 +01:00
Paul "LeoNerd" Evans
ca35e54d6b Fix typo "persiting" 2016-09-13 13:26:33 +01:00
Mark Haines
ad816b0add Limit how often we ask for keys from dead servers 2016-09-13 11:53:50 +01:00
Erik Johnston
43253c10b8 Remove redundant event_auth index 2016-09-13 11:47:48 +01:00
Mark Haines
18ab019a4a Move the E2E key handling into the e2e handler 2016-09-13 11:35:35 +01:00
Mark Haines
76b09c29b0 Merge pull request #1110 from matrix-org/markjh/e2e_timeout 
Add a timeout parameter for end2end key queries.
 2016-09-13 10:50:45 +01:00
Erik Johnston
ba6bc2faa0 Merge pull request #1109 from matrix-org/erikj/partial_indices 
Add WHERE clause support to index creation
 2016-09-13 09:06:16 +01:00
Matthew Hodgson
edbcb4152b make device IDs more useful for human disambiguation 2016-09-13 00:02:39 +01:00
Mark Haines
949c2c5435 Add a timeout parameter for end2end key queries. 
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.

Fetch keys from different servers in parallel rather than in series.

Set the default timeout to 10s.
 2016-09-12 18:17:09 +01:00
Erik Johnston
b17af156c7 Remove where clause 2016-09-12 17:05:54 +01:00
Erik Johnston
1c9da43a95 Merge pull request #1108 from matrix-org/erikj/create_dm 
Add is_direct param to /createRoom
 2016-09-12 16:57:16 +01:00
Erik Johnston
0b32bb20bb Index contains_url for file search queries 2016-09-12 16:57:05 +01:00
Erik Johnston
c94de0ab60 Add WHERE clause support to index creation 2016-09-12 16:55:01 +01:00
Erik Johnston
502c901e11 Merge pull request #1107 from matrix-org/erikj/backfill_none 
Fix backfill when cannot find an event.
 2016-09-12 16:48:01 +01:00
Erik Johnston
48a5a7552d Add is_direct param to /createRoom 2016-09-12 16:34:20 +01:00
Erik Johnston
706b5d76ed Fix backfill when cannot find an event. 
`get_pdu` can succeed but return None.
 2016-09-12 14:59:51 +01:00
Erik Johnston
7c679b1118 Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent 
Create new index concurrently
 2016-09-12 14:38:40 +01:00
Erik Johnston
d080b3425c Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast 
Optimise /notifications query
 2016-09-12 14:34:12 +01:00
Erik Johnston
03a98aff3c Create new index concurrently 2016-09-12 14:27:01 +01:00
Erik Johnston
fa20c9ce94 Change the index to be stream_ordering, highlight 2016-09-12 14:04:08 +01:00
Erik Johnston
5ef5435529 Remove unused import 2016-09-12 13:32:58 +01:00
Mark Haines
aa7b890cfe Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync 
Fix direct to device messages recieved over federation to notify sync
 2016-09-12 13:25:23 +01:00
Erik Johnston
7cd6edb947 Use register_background_index_update 2016-09-12 12:54:48 +01:00
Erik Johnston
0294c14ec4 Add back in query change 2016-09-12 12:43:56 +01:00
Erik Johnston
7fe42cf949 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast 2016-09-12 12:37:09 +01:00
Erik Johnston
15ca0c6a4d Make reindex happen in bg 2016-09-12 12:36:36 +01:00
Mark Haines
a232e06100 Fix direct to device messages recieved over federation to notify sync 2016-09-12 12:30:46 +01:00
Mark Haines
4a32d25d4c Merge branch 'develop' into markjh/bearer_token 2016-09-12 11:14:56 +01:00
Mark Haines
31f85f9db9 Add comments to existing schema deltas that used "CREATE INDEX" directly 2016-09-12 11:00:26 +01:00
Mark Haines
ec609f8094 Fix unit tests 2016-09-12 10:46:02 +01:00
Erik Johnston
54417999b6 Revert "Add index to event_push_actions" 2016-09-12 10:39:55 +01:00
Erik Johnston
45dc260060 Merge pull request #1101 from matrix-org/erikj/state_types_idx 
Change state fetch query for postgres to be faster
 2016-09-12 10:20:38 +01:00
Erik Johnston
d1c217c823 Merge pull request #1097 from matrix-org/erikj/replication_typing_rest 
Correctly handle typing stream id resetting
 2016-09-12 10:10:15 +01:00
Erik Johnston
897d57bc58 Change state fetch query for postgres to be faster 
It turns out that postgres doesn't like doing a list of OR's and is
about 1000x slower, so we just issue a query for each specific type
seperately.
 2016-09-12 10:05:07 +01:00
Erik Johnston
555460ae1b Merge pull request #1095 from matrix-org/erikj/batch_edus 
Clobber EDUs in send queue
 2016-09-12 08:04:15 +01:00
Richard van der Hoff
4162f820ff Merge pull request #1100 from VShell/fix-cas 
Conform better to the CAS protocol specification
 2016-09-09 21:57:42 +01:00
Shell Turner
29205e9596 Conform better to the CAS protocol specification 
Redirect to CAS's /login endpoint properly, and
don't require an <attributes> element.

Signed-off-by: Shell Turner <cam.turn@gmail.com>
 2016-09-09 21:20:14 +01:00
David Baker
b91e2833b3 Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast 2016-09-09 19:11:34 +01:00
David Baker
f2acc3dcf9 Add index to event_push_actions 
and remove room_id caluse so it uses it

Mostly from @negativemjark
 2016-09-09 18:54:54 +01:00
Mark Haines
3ddec016ff Merge branch 'develop' into markjh/bearer_token 2016-09-09 18:51:22 +01:00
Mark Haines
8e01263587 Allow clients to supply access_tokens as headers 
Clients can continue to supply access tokens as query parameters
or can supply the token as a header:

   Authorization: Bearer <access_token_goes_here>

This matches the ouath2 format of
https://tools.ietf.org/html/rfc6750#section-2.1
 2016-09-09 18:17:42 +01:00
Erik Johnston
3265def8c7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/batch_edus 2016-09-09 18:06:01 +01:00
Erik Johnston
af4701b311 Fix incorrect attribute name 2016-09-09 17:36:56 +01:00
Erik Johnston
44330a21e9 Comment 2016-09-09 17:22:07 +01:00
Erik Johnston
464ffd1b5e Comment 2016-09-09 17:17:23 +01:00
Erik Johnston
327425764e Add edu.type as part of key. Remove debug logging 2016-09-09 17:13:30 +01:00
Mark Haines
dbff7e9436 Merge pull request #1096 from matrix-org/markjh/get_access_token 
Add helper function for getting access_tokens from requests
 2016-09-09 17:09:27 +01:00
Erik Johnston
a4339de9de Correctly handle typing stream id resetting 2016-09-09 16:44:26 +01:00
Mark Haines
8aee5aa068 Add helper function for getting access_tokens from requests 
Rather than reimplementing the token parsing in the various places.
This will make it easier to change the token parsing to allow access_tokens
in HTTP headers.
 2016-09-09 16:33:15 +01:00
Erik Johnston
52b2318777 Clobber EDUs in send queue 2016-09-09 15:59:08 +01:00
Paul Evans
56f38d1776 Merge pull request #1091 from matrix-org/paul/third-party-lookup 
Improvements to 3PE lookup API
 2016-09-09 15:43:11 +01:00
Paul Evans
8cb252d00c Merge pull request #1094 from matrix-org/paul/get-state-whole-event 
Allow clients to ask for the whole of a single state event
 2016-09-09 15:26:17 +01:00
Paul "LeoNerd" Evans
776594f99d Log if rejecting 3PE query metadata result due to type check 2016-09-09 15:11:31 +01:00
Paul "LeoNerd" Evans
ed44c475d8 Reject malformed 3PE query metadata results earlier in AS API handling code 2016-09-09 15:07:04 +01:00
Erik Johnston
ab80d5e0a9 Drop replication log levels 2016-09-09 14:56:50 +01:00
Paul "LeoNerd" Evans
f25d74f69c Minor fixes from PR comments 2016-09-09 14:54:16 +01:00
Paul "LeoNerd" Evans
d271383e63 Filter returned events for client-facing format 2016-09-09 14:40:15 +01:00
Paul "LeoNerd" Evans
0fc0a3bdff Allow clients to specify the format a room state event is returned in 2016-09-09 14:34:29 +01:00
Erik Johnston
6c4d582144 Deduplicate presence in _update_states 2016-09-09 14:28:22 +01:00
Erik Johnston
685da5a3b0 Merge pull request #1092 from matrix-org/erikj/transaction_queue_check 
Check if destination is ready for retry earlier
 2016-09-09 13:49:56 +01:00
Erik Johnston
a6c6750166 Check if destination is ready for retry earlier 2016-09-09 13:46:05 +01:00
Paul "LeoNerd" Evans
bdbcfc2a80 appease pep8 2016-09-09 13:31:39 +01:00
Paul "LeoNerd" Evans
6eb0c8a2e4 Python isn't JavaScript; have to quote dict keys 2016-09-09 13:31:17 +01:00
Mark Haines
6b54fa81de Merge pull request #1089 from matrix-org/markjh/direct_to_device_stream 
Track the max device stream_id in a separate table,
 2016-09-09 13:31:07 +01:00
Paul "LeoNerd" Evans
25eb769b26 Efficiency fix for lookups of a single protocol 2016-09-09 13:25:02 +01:00
Paul "LeoNerd" Evans
3328428d05 Allow lookup of a single 3PE protocol query metadata 2016-09-09 13:19:04 +01:00
Erik Johnston
4598682b43 Fix tightloop on sending transaction 2016-09-09 13:12:53 +01:00
Paul "LeoNerd" Evans
033d43e419 Don't corrupt shared cache on subsequent protocol requests 2016-09-09 13:10:36 +01:00
Mark Haines
647c724573 Use the previous MAX value if any to set the stream_id 2016-09-09 11:52:44 +01:00
Erik Johnston
a15ba15e64 Merge pull request #1088 from matrix-org/erikj/transaction_queue_check 
Correctly guard against multiple concurrent transactions
 2016-09-09 11:49:06 +01:00
Mark Haines
6a6cbfcf1e Track the max_stream_device_id in a separate table, since we delete from the inbox table 2016-09-09 11:48:23 +01:00
Erik Johnston
d2688d7f03 Correctly guard against multiple concurrent transactions 2016-09-09 11:44:36 +01:00
Mark Haines
303b6f29f0 Merge pull request #1087 from matrix-org/markjh/reapply_delta 
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
 2016-09-09 11:24:27 +01:00
Erik Johnston
1fe7ca1362 Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop 2016-09-09 11:15:40 +01:00
Erik Johnston
66efcbbff1 Bump changelog and version 2016-09-09 11:07:39 +01:00
Mark Haines
0877157353 Just move the schema and add some DROPs 2016-09-09 11:04:47 +01:00
Erik Johnston
2ffec928e2 Reduce batch size to be under SQL limit 2016-09-09 11:03:31 +01:00
Erik Johnston
b390756150 Update last_device_stream_id_by_dest if there is nothing to send 2016-09-09 11:00:15 +01:00
Matthew Hodgson
b8f84f99ff Merge pull request #1081 from matrix-org/dbkr/notifications_only_highlight 
Implement `only=highlight` on `/notifications`
 2016-09-09 00:09:51 +01:00
Mark Haines
43b77c5d97 Only catch databas errors 2016-09-08 17:44:21 +01:00
Paul "LeoNerd" Evans
2f267ee160 Collect up all the "instances" lists of individual AS protocol results into one combined answer to the client 2016-09-08 17:43:53 +01:00
Mark Haines
7d5b142547 Add a stub run_upgrade 2016-09-08 17:39:11 +01:00
David Baker
c3276aef25 Merge pull request #1080 from matrix-org/dbkr/fix_notifications_api_with_from 
Fix /notifications API when used with `from` param
 2016-09-08 17:35:35 +01:00
Mark Haines
fa722a699c Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop 2016-09-08 17:35:16 +01:00
Erik Johnston
5c688739d6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reindex_state_groups 2016-09-08 16:52:09 +01:00
Erik Johnston
ebb46497ba Add delta file 2016-09-08 16:38:54 +01:00
Mark Haines
91ec972277 Merge pull request #1084 from matrix-org/markjh/direct_to_device_wildcard 
Support wildcard device_ids for direct to device messages
 2016-09-08 16:28:08 +01:00
Erik Johnston
5beda10bbd Reindex state_groups_state after pruning 2016-09-08 16:18:01 +01:00
Erik Johnston
257025ac89 Merge pull request #1082 from matrix-org/erikj/remote_public_rooms 
Add server param to /publicRooms
 2016-09-08 16:04:22 +01:00
Erik Johnston
3f9889bfd6 Use parse_string 2016-09-08 15:51:10 +01:00
Erik Johnston
caa22334b3 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-09-08 15:33:00 +01:00
Erik Johnston
b152ee71fe Bump version and changelog 2016-09-08 15:23:44 +01:00
Mark Haines
a1c8f268e5 Support wildcard device_ids for direct to device messages 2016-09-08 15:13:05 +01:00
Erik Johnston
8b93af662d Check the user_id for presence/typing matches origin 2016-09-08 15:07:38 +01:00
Mark Haines
fa9d36e050 Merge branch 'develop' into markjh/direct_to_device_federation 2016-09-08 13:43:43 +01:00
David Baker
4ef222ab61 Implement only=highlight on /notifications 2016-09-08 13:43:35 +01:00
Erik Johnston
61cd9af09b Log delta files we're applying 2016-09-08 13:40:46 +01:00
Erik Johnston
791658b576 Add server param to /publicRooms 2016-09-08 11:53:05 +01:00
David Baker
c5b49eb7ca Fix /notifications API when used with from param 2016-09-08 09:40:10 +01:00
Erik Johnston
b568ca309c Temporarily disable sequential scans for state fetching 2016-09-08 09:38:54 +01:00
Mark Haines
3c320c006c Merge pull request #1077 from matrix-org/markjh/device_logging 
Log the types and values when failing to store devices
 2016-09-07 18:24:24 +01:00
Mark Haines
85b51fdd6b Log the types and values when failing to store devices 2016-09-07 17:19:18 +01:00
Mark Haines
43954d000e Add a new method to enqueue the device messages rather than sending a dummy EDU 2016-09-07 16:10:51 +01:00
Mark Haines
2a0159b8ae Fix the stream change cache to work over replication 2016-09-07 15:58:00 +01:00
Mark Haines
cb98ac261b Move the check for federated device_messages. 
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
 2016-09-07 15:39:13 +01:00
Mark Haines
31a07d2335 Add stream change caches for device messages 2016-09-07 15:27:07 +01:00
Erik Johnston
513188aa56 Comment 2016-09-07 14:53:23 +01:00
Erik Johnston
fadb01551a Add appopriate framing clause 2016-09-07 14:39:01 +01:00
Erik Johnston
d25c20ccbe Use windowing function to make use of index 2016-09-07 14:22:22 +01:00
Mark Haines
7d893beebe Comment the add_messages storage functions 2016-09-07 12:03:37 +01:00
Erik Johnston
94a83b534f Merge pull request #1065 from matrix-org/erikj/state_storage 
Move to storing state_groups_state as deltas
 2016-09-07 09:39:58 +01:00
Mark Haines
d4a35ada28 Send device messages over federation 2016-09-06 18:16:20 +01:00
Mark Haines
e020834e4f Add storage methods for federated device messages 2016-09-06 15:12:13 +01:00
Mark Haines
2ad72da931 Add tables for federated device messages 
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
 2016-09-06 15:10:40 +01:00
Erik Johnston
3c4208a057 Record counts of state changes 2016-09-06 11:31:01 +01:00
Mark Haines
f4164edb70 Move _add_messages_to_device_inbox_txn into a separate method 2016-09-06 11:26:37 +01:00
Erik Johnston
2eed4d7af4 Merge pull request #1072 from matrix-org/erikj/presence_fiddle 
Fiddle should_notify to better report stats
 2016-09-06 10:57:48 +01:00
Erik Johnston
438ef47637 Short circuit if presence is the same 2016-09-06 10:28:35 +01:00
Erik Johnston
74a3b4a650 Fiddle should_notify to better report stats 2016-09-06 10:23:38 +01:00
Erik Johnston
d51b8a1674 Add quotes and be explicity about script-src 2016-09-05 17:35:01 +01:00
Erik Johnston
662b031a30 Allow PDF to be rendered from media repo 2016-09-05 17:25:26 +01:00
Erik Johnston
0595413c0f Scale the batch size so that we're not bitten by the minimum 2016-09-05 15:49:57 +01:00
Erik Johnston
a7032abb2e Correctly handle reindexing state groups that already have an edge 2016-09-05 15:07:23 +01:00
Erik Johnston
9e6d88f4e2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-05 15:01:33 +01:00
Erik Johnston
8c93e0bae7 Merge pull request #1070 from matrix-org/erikj/presence_stats 
Record why we have chosen to notify
 2016-09-05 15:01:30 +01:00
Erik Johnston
70332a12dd Take value in a better way 2016-09-05 14:57:14 +01:00
Erik Johnston
373654c635 Comment about sqlite and WITH RECURSIVE 2016-09-05 14:50:36 +01:00
Erik Johnston
485d999c8a Correctly delete old state groups in purge history API 2016-09-05 14:49:08 +01:00
Erik Johnston
69054e3d4c Record why we have chosen to notify 2016-09-05 14:12:11 +01:00
Erik Johnston
69a2d4e38c Use get_joined_users_from_context instead of manually looking up hosts 2016-09-05 13:44:40 +01:00
Erik Johnston
f12993ec16 Bump changelog and version 2016-09-05 13:01:14 +01:00
Erik Johnston
bb36b93f71 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-05 11:53:11 +01:00
Erik Johnston
d87b87adf7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/bulk_push 2016-09-05 11:52:49 +01:00
Erik Johnston
caed150363 Remove unused imports 2016-09-05 10:52:01 +01:00
Erik Johnston
80a6a445fa Only fetch local pushers 2016-09-05 10:43:32 +01:00
Erik Johnston
628e65721b Add comments 2016-09-05 10:41:27 +01:00
Mark Haines
274c2f50a5 Merge pull request #1067 from matrix-org/markjh/idempotent 
Fix membership changes to be idempotent
 2016-09-05 10:21:25 +01:00
Erik Johnston
a99e933550 Add upgrade script that will slowly prune state_groups_state entries 2016-09-05 10:05:36 +01:00
Erik Johnston
3847fa38c4 Make bulk_get_push_rules_for_room use get_joined_users_from_context cache 2016-09-05 10:02:38 +01:00
Mark Haines
f2690c6423 Fix membership changes to be idempotent 2016-09-02 19:23:22 +01:00
Mark Haines
81b94c5750 Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound 
Only return new device messages in /sync
 2016-09-02 16:18:34 +01:00
Mark Haines
65fa37ac5e Only return new device messages in /sync 2016-09-02 15:50:37 +01:00
Erik Johnston
3baf641a48 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-02 14:54:07 +01:00
Erik Johnston
c0238ecbed Explicitly specify state_key for history_visibility fetching 2016-09-02 14:53:46 +01:00
Erik Johnston
273b6bcf22 Merge pull request #1064 from matrix-org/erikj/on_receive_check 
Only check if host is in room if we have state and auth_chain
 2016-09-02 14:43:35 +01:00
Erik Johnston
f7f1027d3d Comment on when auth chain and state are None 2016-09-02 14:42:38 +01:00
Erik Johnston
34e5e17f91 Comment 2016-09-02 14:26:07 +01:00
Erik Johnston
b96c6c3185 Docstrings 2016-09-02 14:19:22 +01:00
Erik Johnston
1ffe9578d1 Merge pull request #1063 from matrix-org/erikj/pull_out_ids_only 
Only pull out IDs from DB for /state_ids/ request
 2016-09-02 14:16:28 +01:00
Erik Johnston
cce957e254 Bump max_entries on get_destination_retry_timings 2016-09-02 14:08:33 +01:00
Erik Johnston
bd9b8d87ae Only check if host is in room if we have state and auth_chain 2016-09-02 13:40:28 +01:00
Erik Johnston
657847e4c6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-02 11:04:48 +01:00
Mark Haines
965168a842 Merge branch 'develop' into markjh/direct_to_device_synchrotron 2016-09-02 10:59:24 +01:00
Erik Johnston
2854ee2a52 Only pull out IDs from DB for /state_ids/ request 2016-09-02 10:53:36 +01:00
Erik Johnston
598317927c Limit the length of state chains 2016-09-02 10:41:38 +01:00
Mark Haines
7ed5acacf4 Fix up the calls to the notifier for device messages 2016-09-01 18:08:40 +01:00
Erik Johnston
051a9ea921 Linearize state resolution to help caches 2016-09-01 14:55:03 +01:00
Erik Johnston
265d847ffd Fix typo in log line 2016-09-01 14:50:06 +01:00
Erik Johnston
f4778d4cd9 Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop 2016-09-01 14:40:36 +01:00
Erik Johnston
9e25443db8 Move to storing state_groups_state as deltas 2016-09-01 14:31:26 +01:00
Erik Johnston
44982606ee Merge pull request #1060 from matrix-org/erikj/state_ids 
Assign state groups in state handler.
 2016-09-01 14:20:42 +01:00
Erik Johnston
516a272aca Ensure we only return a validated pdu in get_pdu 2016-09-01 10:55:02 +01:00
Erik Johnston
0cfd6c3161 Use state_groups table to test existence 2016-08-31 16:25:57 +01:00
Erik Johnston
5405351b14 Lower get_linearized_receipts_for_room cache size 2016-08-31 16:19:44 +01:00
Erik Johnston
ed7a703d4c Handle the fact that workers can't generate state groups 2016-08-31 15:53:19 +01:00
Erik Johnston
1671913287 Merge pull request #1059 from matrix-org/erikj/sent_transaction_delete 
Clean up old sent transactions
 2016-08-31 15:07:33 +01:00
Erik Johnston
f51888530d Always specify state_group so that its in the cache 2016-08-31 14:58:11 +01:00
Erik Johnston
826ca61745 Add storage function to SlaveStore 2016-08-31 14:45:04 +01:00
Erik Johnston
fbd2615de4 Merge pull request #1057 from matrix-org/erikj/fix_email_name 
Fix email notifs by adding missing param
 2016-08-31 14:33:45 +01:00
Erik Johnston
c10cb581c6 Correctly handle the difference between prev and current state 2016-08-31 14:26:22 +01:00
Erik Johnston
ef0cc648cf Clean up old sent transactions 2016-08-31 11:12:02 +01:00
Mark Haines
761f9fccff Merge pull request #1058 from matrix-org/markjh/direct_to_device_synchrotron 
Add a replication stream for direct to device messages
 2016-08-31 11:08:18 +01:00
Mark Haines
a662252758 Return the current stream position from add_messages_to_device_inbox 2016-08-31 10:42:52 +01:00
Mark Haines
1aa3e1d287 Add a replication stream for direct to device messages 2016-08-31 10:38:58 +01:00
Erik Johnston
1bb8ec296d Generate state group ids in state layer 2016-08-31 10:09:46 +01:00
Erik Johnston
d80f64d370 Fix email notifs by adding missing param 2016-08-30 21:46:39 +01:00
Kegan Dougal
c882783535 flake8 2016-08-30 17:20:31 +01:00
Kegan Dougal
572acde483 Use None instead of the empty string 
Change how we validate the 'url' field as a result.
 2016-08-30 17:16:00 +01:00
Erik Johnston
5dc2a702cf Make _state_groups_id_gen a normal IdGenerator 2016-08-30 16:55:11 +01:00
Erik Johnston
3e784eff74 Remove state replication stream 2016-08-30 16:51:36 +01:00
Kegan Dougal
16b652f0a3 Flake8 2016-08-30 16:30:12 +01:00
Kegan Dougal
e82247f990 Allow application services to have an optional 'url' 
If 'url' is not specified, they will not be pushed for events or queries. This
is useful for bots who simply wish to reserve large chunks of user/alias
namespace, and don't care about being pushed for events.
 2016-08-30 16:21:16 +01:00
Erik Johnston
c7f665d700 Merge pull request #1055 from matrix-org/erikj/occaisonally_persist 
Occaisonally persist unpersisted presence updates
 2016-08-30 15:59:00 +01:00
Erik Johnston
097330bae8 Check correct variable 2016-08-30 15:50:20 +01:00
Erik Johnston
21b977ccfe Occaisonally persist unpersisted presence updates 2016-08-30 15:39:50 +01:00
Erik Johnston
bc1a8b1f7a Don't notify for online -> online transitions. 
Specifically, if currently_active remains true then we should not notify
if only the last active time changes.
 2016-08-30 15:05:32 +01:00
Erik Johnston
67f0c990f8 Merge pull request #1051 from matrix-org/erikj/fix_push_names 
Fix push room names for rooms with only an alias
 2016-08-30 11:50:34 +01:00
Erik Johnston
c8cd87b21b Comment about message deletion 2016-08-30 11:23:26 +01:00
Erik Johnston
55e17d3697 Fix push room names for rooms with only an alias 2016-08-30 11:19:59 +01:00
Erik Johnston
1ee6285905 Fix check 2016-08-30 11:17:46 +01:00
Erik Johnston
68e1a872fd Noop get_new_messages_for_device if token hasn't changed 2016-08-30 10:58:46 +01:00
Erik Johnston
55fc17cf4b Merge pull request #1049 from matrix-org/erikj/presence_users_in_room 
Use state handler instead of get_users_in_room/get_joined_hosts
 2016-08-30 10:50:37 +01:00
Erik Johnston
41788bba50 Add to slave store 2016-08-30 09:55:17 +01:00
Erik Johnston
873f870e5a Add new direct message storage functions to slave store 2016-08-30 09:40:32 +01:00
Erik Johnston
bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Mark Haines
4bbef62124 Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device 2016-08-26 14:35:31 +01:00
Erik Johnston
a234e895cf Fix room name in email notifs 2016-08-26 14:10:21 +01:00
Erik Johnston
4daa397a00 Add is_host_joined to slave storage 2016-08-26 13:10:56 +01:00
Erik Johnston
c7cd35d682 Typo 2016-08-26 11:23:58 +01:00
Erik Johnston
54cc69154e Make None optional 2016-08-26 11:20:59 +01:00
Erik Johnston
11faa4296d Measure _filter_events_for_server 2016-08-26 11:15:40 +01:00
Erik Johnston
f6338d6a3e Don't pull out full state for _filter_events_for_server 2016-08-26 11:13:16 +01:00
Erik Johnston
1ccdc1e93a Cache check_host_in_room 2016-08-26 10:59:40 +01:00
Erik Johnston
25414b44a2 Add measure on check_host_in_room 2016-08-26 10:47:00 +01:00
Erik Johnston
50943ab942 Add new state storage funcs to replication 2016-08-26 09:57:32 +01:00
Erik Johnston
30961182f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids 2016-08-26 09:48:13 +01:00
Erik Johnston
c1a133a6b6 Merge pull request #1043 from matrix-org/erikj/backfill_fix 
Fix None check in backfill
 2016-08-26 09:07:05 +01:00
Erik Johnston
778fa85f47 Make sync not pull out full state 2016-08-25 18:59:44 +01:00
Erik Johnston
7356d52e73 Fix up push to use get_current_state_ids 2016-08-25 18:35:49 +01:00
Paul "LeoNerd" Evans
9459137f1e Just sprintf the 'kind' argument into uri directly 2016-08-25 18:35:38 +01:00
Paul "LeoNerd" Evans
1294d4a329 Move ThirdPartyEntityKind into api.constants so the expectation becomes that the value is significant 2016-08-25 18:34:47 +01:00
Mark Haines
ab34fdecb7 Merge branch 'develop' into markjh/direct_to_device 2016-08-25 18:34:46 +01:00
Mark Haines
b162cb2e41 Add some TODOs 2016-08-25 18:18:53 +01:00
Erik Johnston
0e1900d819 Pull out full state less 2016-08-25 18:15:51 +01:00
Mark Haines
641efb6a39 Fix the deduplication of incoming direct-to-device messages 2016-08-25 18:14:02 +01:00
Paul "LeoNerd" Evans
e7af8be5ae Root the 3PE lookup API within /_matrix/app/unstable instead of at toplevel 2016-08-25 18:06:29 +01:00
Paul "LeoNerd" Evans
142983b4ea APP_SERVICE_PREFIX is never used; don't bother 2016-08-25 18:06:05 +01:00
Erik Johnston
721414d98a Add desc 2016-08-25 17:49:05 +01:00
Mark Haines
e993925279 Add store-and-forward direct-to-device messaging 2016-08-25 17:35:37 +01:00
Erik Johnston
a3dc1e9cbe Replace context.current_state with context.current_state_ids 2016-08-25 17:32:22 +01:00
Paul Evans
d9dcb2ba3a Merge pull request #1041 from matrix-org/paul/third-party-lookup 
Extend 3PE lookup APIs for metadata query
 2016-08-25 17:06:53 +01:00
Paul "LeoNerd" Evans
adf53f04ce appease pep8 2016-08-25 16:00:31 +01:00
Paul "LeoNerd" Evans
c435bfee9c Don't need toplevel cache on 3PE lookup metadata any more 2016-08-25 15:57:07 +01:00
Paul "LeoNerd" Evans
db7283cc6b Implement a ResponseCache around 3PE lookup metadata lookups 2016-08-25 15:56:27 +01:00
Paul "LeoNerd" Evans
d0b8d49f71 Kill PROTOCOL_META since I'm not using it any more 2016-08-25 15:45:28 +01:00
Paul "LeoNerd" Evans
5474824975 Actually query over AS API for 3PE lookup metadata 2016-08-25 15:29:36 +01:00
Erik Johnston
17f4f14df7 Pull out event ids rather than full events for state 2016-08-25 13:42:44 +01:00
Erik Johnston
cd5b264b03 Fix None check in backfill 2016-08-25 10:39:19 +01:00
Erik Johnston
60a015550a Bump changelog and version 2016-08-24 14:31:13 +01:00
Erik Johnston
90d5983d7a Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.1 2016-08-24 14:28:57 +01:00
Erik Johnston
c20cb5160d Remove tracer 2016-08-24 13:22:47 +01:00
Erik Johnston
fda97dd58a Merge branch 'develop' of github.com:matrix-org/synapse into erikj/preserve_log_contexts 2016-08-24 13:22:02 +01:00
Paul "LeoNerd" Evans
8e1ed09dff Move static knowledge of protocol metadata into AS handler; cache the result 2016-08-24 13:01:53 +01:00
Paul "LeoNerd" Evans
965f33c901 Declare 'gitter' known protocol, with user lookup 2016-08-24 12:34:03 +01:00
Paul "LeoNerd" Evans
9899824b85 Initial hack at the 3PN protocols metadata lookup API 2016-08-24 12:33:01 +01:00
Erik Johnston
9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Paul "LeoNerd" Evans
63c19e1df9 Move 3PU/3PL lookup APIs into /thirdparty containing entity 2016-08-24 11:55:57 +01:00
Erik Johnston
86bcf4d6a7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/pagination 2016-08-24 10:38:21 +01:00
Erik Johnston
ba07d4a70e Add None checks to backfill 2016-08-24 10:31:05 +01:00
Kegan Dougal
4b31426a02 Pass through user-supplied content in /join/$room_id 
It was always intended to allow custom keys on the join event, but this has
at some point been lost. Restore it.

If the user specifies keys like "avatar_url" then they will be clobbered.
 2016-08-23 16:32:04 +01:00
Erik Johnston
122c7a43c9 Merge pull request #1038 from matrix-org/erikj/receved_txn_purge 
Delete old received_transactions rows
 2016-08-23 11:02:40 +01:00
Erik Johnston
14047126d8 Bump changelog and version 2016-08-22 18:26:07 +01:00
Erik Johnston
d143f211c8 Merge pull request #1028 from matrix-org/dbkr/notifications_api 
Add the Notifications API
 2016-08-22 18:23:24 +01:00
Erik Johnston
aad8a1a825 Delete old received_transactions 2016-08-22 16:29:46 +01:00
Mark Haines
689f4cb914 Update comment 2016-08-22 16:17:31 +01:00
Mark Haines
c8f9b45bc2 Add usage stats to prometheus monitoring 2016-08-22 15:34:38 +01:00
Erik Johnston
e65bc7d315 Merge pull request #1031 from matrix-org/erikj/measure_notifier 
Add more Measure blocks
 2016-08-22 12:13:07 +01:00
Erik Johnston
33f3624ff7 Add exception logging. Fix typo 2016-08-22 10:49:31 +01:00
Erik Johnston
8c52160b07 Allow request handlers to override metric name 2016-08-22 10:44:45 +01:00
Erik Johnston
a093fab253 Use top level measure 2016-08-22 10:18:12 +01:00
Matthew Hodgson
6e80c03d45 Merge branch 'develop' into dbkr/notifications_api 2016-08-20 00:16:18 +01:00
Erik Johnston
27e0178da9 Add a top level measure 2016-08-19 18:49:37 +01:00
Erik Johnston
6d1a94d218 Remove redundant measure 2016-08-19 18:40:31 +01:00
Erik Johnston
8731197e54 Only abort Measure on Exceptions 2016-08-19 18:23:45 +01:00
Erik Johnston
afbf6b33fc defer.returnValue must not be called within Measure 2016-08-19 18:23:44 +01:00
Erik Johnston
37adde32dc Move defer.returnValue out of Measure 2016-08-19 18:23:44 +01:00