Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
7,937 Commits 4 Branches 615 Tags 63 MiB
c95e9fff99
Commit Graph

5603 Commits

Author SHA1 Message Date
Erik Johnston
ebb46497ba Add delta file 2016-09-08 16:38:54 +01:00
Mark Haines
91ec972277 Merge pull request #1084 from matrix-org/markjh/direct_to_device_wildcard 
Support wildcard device_ids for direct to device messages
 2016-09-08 16:28:08 +01:00
Erik Johnston
5beda10bbd Reindex state_groups_state after pruning 2016-09-08 16:18:01 +01:00
Erik Johnston
257025ac89 Merge pull request #1082 from matrix-org/erikj/remote_public_rooms 
Add server param to /publicRooms
 2016-09-08 16:04:22 +01:00
Erik Johnston
3f9889bfd6 Use parse_string 2016-09-08 15:51:10 +01:00
Erik Johnston
caa22334b3 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-09-08 15:33:00 +01:00
Erik Johnston
b152ee71fe Bump version and changelog 2016-09-08 15:23:44 +01:00
Mark Haines
a1c8f268e5 Support wildcard device_ids for direct to device messages 2016-09-08 15:13:05 +01:00
Erik Johnston
8b93af662d Check the user_id for presence/typing matches origin 2016-09-08 15:07:38 +01:00
Mark Haines
fa9d36e050 Merge branch 'develop' into markjh/direct_to_device_federation 2016-09-08 13:43:43 +01:00
David Baker
4ef222ab61 Implement only=highlight on /notifications 2016-09-08 13:43:35 +01:00
Erik Johnston
61cd9af09b Log delta files we're applying 2016-09-08 13:40:46 +01:00
Erik Johnston
791658b576 Add server param to /publicRooms 2016-09-08 11:53:05 +01:00
David Baker
c5b49eb7ca Fix /notifications API when used with from param 2016-09-08 09:40:10 +01:00
Erik Johnston
b568ca309c Temporarily disable sequential scans for state fetching 2016-09-08 09:38:54 +01:00
Mark Haines
3c320c006c Merge pull request #1077 from matrix-org/markjh/device_logging 
Log the types and values when failing to store devices
 2016-09-07 18:24:24 +01:00
Mark Haines
85b51fdd6b Log the types and values when failing to store devices 2016-09-07 17:19:18 +01:00
Mark Haines
43954d000e Add a new method to enqueue the device messages rather than sending a dummy EDU 2016-09-07 16:10:51 +01:00
Mark Haines
2a0159b8ae Fix the stream change cache to work over replication 2016-09-07 15:58:00 +01:00
Mark Haines
cb98ac261b Move the check for federated device_messages. 
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
 2016-09-07 15:39:13 +01:00
Mark Haines
31a07d2335 Add stream change caches for device messages 2016-09-07 15:27:07 +01:00
Erik Johnston
513188aa56 Comment 2016-09-07 14:53:23 +01:00
Erik Johnston
fadb01551a Add appopriate framing clause 2016-09-07 14:39:01 +01:00
Erik Johnston
d25c20ccbe Use windowing function to make use of index 2016-09-07 14:22:22 +01:00
Mark Haines
7d893beebe Comment the add_messages storage functions 2016-09-07 12:03:37 +01:00
Erik Johnston
94a83b534f Merge pull request #1065 from matrix-org/erikj/state_storage 
Move to storing state_groups_state as deltas
 2016-09-07 09:39:58 +01:00
Mark Haines
d4a35ada28 Send device messages over federation 2016-09-06 18:16:20 +01:00
Mark Haines
e020834e4f Add storage methods for federated device messages 2016-09-06 15:12:13 +01:00
Mark Haines
2ad72da931 Add tables for federated device messages 
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
 2016-09-06 15:10:40 +01:00
Erik Johnston
3c4208a057 Record counts of state changes 2016-09-06 11:31:01 +01:00
Mark Haines
f4164edb70 Move _add_messages_to_device_inbox_txn into a separate method 2016-09-06 11:26:37 +01:00
Erik Johnston
2eed4d7af4 Merge pull request #1072 from matrix-org/erikj/presence_fiddle 
Fiddle should_notify to better report stats
 2016-09-06 10:57:48 +01:00
Erik Johnston
438ef47637 Short circuit if presence is the same 2016-09-06 10:28:35 +01:00
Erik Johnston
74a3b4a650 Fiddle should_notify to better report stats 2016-09-06 10:23:38 +01:00
Erik Johnston
d51b8a1674 Add quotes and be explicity about script-src 2016-09-05 17:35:01 +01:00
Erik Johnston
662b031a30 Allow PDF to be rendered from media repo 2016-09-05 17:25:26 +01:00
Erik Johnston
0595413c0f Scale the batch size so that we're not bitten by the minimum 2016-09-05 15:49:57 +01:00
Erik Johnston
a7032abb2e Correctly handle reindexing state groups that already have an edge 2016-09-05 15:07:23 +01:00
Erik Johnston
9e6d88f4e2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-05 15:01:33 +01:00
Erik Johnston
8c93e0bae7 Merge pull request #1070 from matrix-org/erikj/presence_stats 
Record why we have chosen to notify
 2016-09-05 15:01:30 +01:00
Erik Johnston
70332a12dd Take value in a better way 2016-09-05 14:57:14 +01:00
Erik Johnston
373654c635 Comment about sqlite and WITH RECURSIVE 2016-09-05 14:50:36 +01:00
Erik Johnston
485d999c8a Correctly delete old state groups in purge history API 2016-09-05 14:49:08 +01:00
Erik Johnston
69054e3d4c Record why we have chosen to notify 2016-09-05 14:12:11 +01:00
Erik Johnston
69a2d4e38c Use get_joined_users_from_context instead of manually looking up hosts 2016-09-05 13:44:40 +01:00
Erik Johnston
f12993ec16 Bump changelog and version 2016-09-05 13:01:14 +01:00
Erik Johnston
bb36b93f71 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-05 11:53:11 +01:00
Erik Johnston
d87b87adf7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/bulk_push 2016-09-05 11:52:49 +01:00
Erik Johnston
caed150363 Remove unused imports 2016-09-05 10:52:01 +01:00
Erik Johnston
80a6a445fa Only fetch local pushers 2016-09-05 10:43:32 +01:00
Erik Johnston
628e65721b Add comments 2016-09-05 10:41:27 +01:00
Mark Haines
274c2f50a5 Merge pull request #1067 from matrix-org/markjh/idempotent 
Fix membership changes to be idempotent
 2016-09-05 10:21:25 +01:00
Erik Johnston
a99e933550 Add upgrade script that will slowly prune state_groups_state entries 2016-09-05 10:05:36 +01:00
Erik Johnston
3847fa38c4 Make bulk_get_push_rules_for_room use get_joined_users_from_context cache 2016-09-05 10:02:38 +01:00
Mark Haines
f2690c6423 Fix membership changes to be idempotent 2016-09-02 19:23:22 +01:00
Mark Haines
81b94c5750 Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound 
Only return new device messages in /sync
 2016-09-02 16:18:34 +01:00
Mark Haines
65fa37ac5e Only return new device messages in /sync 2016-09-02 15:50:37 +01:00
Erik Johnston
3baf641a48 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-02 14:54:07 +01:00
Erik Johnston
c0238ecbed Explicitly specify state_key for history_visibility fetching 2016-09-02 14:53:46 +01:00
Erik Johnston
273b6bcf22 Merge pull request #1064 from matrix-org/erikj/on_receive_check 
Only check if host is in room if we have state and auth_chain
 2016-09-02 14:43:35 +01:00
Erik Johnston
f7f1027d3d Comment on when auth chain and state are None 2016-09-02 14:42:38 +01:00
Erik Johnston
34e5e17f91 Comment 2016-09-02 14:26:07 +01:00
Erik Johnston
b96c6c3185 Docstrings 2016-09-02 14:19:22 +01:00
Erik Johnston
1ffe9578d1 Merge pull request #1063 from matrix-org/erikj/pull_out_ids_only 
Only pull out IDs from DB for /state_ids/ request
 2016-09-02 14:16:28 +01:00
Erik Johnston
cce957e254 Bump max_entries on get_destination_retry_timings 2016-09-02 14:08:33 +01:00
Erik Johnston
bd9b8d87ae Only check if host is in room if we have state and auth_chain 2016-09-02 13:40:28 +01:00
Erik Johnston
657847e4c6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-02 11:04:48 +01:00
Mark Haines
965168a842 Merge branch 'develop' into markjh/direct_to_device_synchrotron 2016-09-02 10:59:24 +01:00
Erik Johnston
2854ee2a52 Only pull out IDs from DB for /state_ids/ request 2016-09-02 10:53:36 +01:00
Erik Johnston
598317927c Limit the length of state chains 2016-09-02 10:41:38 +01:00
Mark Haines
7ed5acacf4 Fix up the calls to the notifier for device messages 2016-09-01 18:08:40 +01:00
Erik Johnston
051a9ea921 Linearize state resolution to help caches 2016-09-01 14:55:03 +01:00
Erik Johnston
265d847ffd Fix typo in log line 2016-09-01 14:50:06 +01:00
Erik Johnston
f4778d4cd9 Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop 2016-09-01 14:40:36 +01:00
Erik Johnston
9e25443db8 Move to storing state_groups_state as deltas 2016-09-01 14:31:26 +01:00
Erik Johnston
44982606ee Merge pull request #1060 from matrix-org/erikj/state_ids 
Assign state groups in state handler.
 2016-09-01 14:20:42 +01:00
Erik Johnston
516a272aca Ensure we only return a validated pdu in get_pdu 2016-09-01 10:55:02 +01:00
Erik Johnston
0cfd6c3161 Use state_groups table to test existence 2016-08-31 16:25:57 +01:00
Erik Johnston
5405351b14 Lower get_linearized_receipts_for_room cache size 2016-08-31 16:19:44 +01:00
Erik Johnston
ed7a703d4c Handle the fact that workers can't generate state groups 2016-08-31 15:53:19 +01:00
Erik Johnston
1671913287 Merge pull request #1059 from matrix-org/erikj/sent_transaction_delete 
Clean up old sent transactions
 2016-08-31 15:07:33 +01:00
Erik Johnston
f51888530d Always specify state_group so that its in the cache 2016-08-31 14:58:11 +01:00
Erik Johnston
826ca61745 Add storage function to SlaveStore 2016-08-31 14:45:04 +01:00
Erik Johnston
fbd2615de4 Merge pull request #1057 from matrix-org/erikj/fix_email_name 
Fix email notifs by adding missing param
 2016-08-31 14:33:45 +01:00
Erik Johnston
c10cb581c6 Correctly handle the difference between prev and current state 2016-08-31 14:26:22 +01:00
Erik Johnston
ef0cc648cf Clean up old sent transactions 2016-08-31 11:12:02 +01:00
Mark Haines
761f9fccff Merge pull request #1058 from matrix-org/markjh/direct_to_device_synchrotron 
Add a replication stream for direct to device messages
 2016-08-31 11:08:18 +01:00
Mark Haines
a662252758 Return the current stream position from add_messages_to_device_inbox 2016-08-31 10:42:52 +01:00
Mark Haines
1aa3e1d287 Add a replication stream for direct to device messages 2016-08-31 10:38:58 +01:00
Erik Johnston
1bb8ec296d Generate state group ids in state layer 2016-08-31 10:09:46 +01:00
Erik Johnston
d80f64d370 Fix email notifs by adding missing param 2016-08-30 21:46:39 +01:00
Kegan Dougal
c882783535 flake8 2016-08-30 17:20:31 +01:00
Kegan Dougal
572acde483 Use None instead of the empty string 
Change how we validate the 'url' field as a result.
 2016-08-30 17:16:00 +01:00
Erik Johnston
5dc2a702cf Make _state_groups_id_gen a normal IdGenerator 2016-08-30 16:55:11 +01:00
Erik Johnston
3e784eff74 Remove state replication stream 2016-08-30 16:51:36 +01:00
Kegan Dougal
16b652f0a3 Flake8 2016-08-30 16:30:12 +01:00
Kegan Dougal
e82247f990 Allow application services to have an optional 'url' 
If 'url' is not specified, they will not be pushed for events or queries. This
is useful for bots who simply wish to reserve large chunks of user/alias
namespace, and don't care about being pushed for events.
 2016-08-30 16:21:16 +01:00
Erik Johnston
c7f665d700 Merge pull request #1055 from matrix-org/erikj/occaisonally_persist 
Occaisonally persist unpersisted presence updates
 2016-08-30 15:59:00 +01:00
Erik Johnston
097330bae8 Check correct variable 2016-08-30 15:50:20 +01:00
Erik Johnston
21b977ccfe Occaisonally persist unpersisted presence updates 2016-08-30 15:39:50 +01:00
Erik Johnston
bc1a8b1f7a Don't notify for online -> online transitions. 
Specifically, if currently_active remains true then we should not notify
if only the last active time changes.
 2016-08-30 15:05:32 +01:00
Erik Johnston
67f0c990f8 Merge pull request #1051 from matrix-org/erikj/fix_push_names 
Fix push room names for rooms with only an alias
 2016-08-30 11:50:34 +01:00
Erik Johnston
c8cd87b21b Comment about message deletion 2016-08-30 11:23:26 +01:00
Erik Johnston
55e17d3697 Fix push room names for rooms with only an alias 2016-08-30 11:19:59 +01:00
Erik Johnston
1ee6285905 Fix check 2016-08-30 11:17:46 +01:00
Erik Johnston
68e1a872fd Noop get_new_messages_for_device if token hasn't changed 2016-08-30 10:58:46 +01:00
Erik Johnston
55fc17cf4b Merge pull request #1049 from matrix-org/erikj/presence_users_in_room 
Use state handler instead of get_users_in_room/get_joined_hosts
 2016-08-30 10:50:37 +01:00
Erik Johnston
41788bba50 Add to slave store 2016-08-30 09:55:17 +01:00
Erik Johnston
873f870e5a Add new direct message storage functions to slave store 2016-08-30 09:40:32 +01:00
Erik Johnston
bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Mark Haines
4bbef62124 Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device 2016-08-26 14:35:31 +01:00
Erik Johnston
a234e895cf Fix room name in email notifs 2016-08-26 14:10:21 +01:00
Erik Johnston
4daa397a00 Add is_host_joined to slave storage 2016-08-26 13:10:56 +01:00
Erik Johnston
c7cd35d682 Typo 2016-08-26 11:23:58 +01:00
Erik Johnston
54cc69154e Make None optional 2016-08-26 11:20:59 +01:00
Erik Johnston
11faa4296d Measure _filter_events_for_server 2016-08-26 11:15:40 +01:00
Erik Johnston
f6338d6a3e Don't pull out full state for _filter_events_for_server 2016-08-26 11:13:16 +01:00
Erik Johnston
1ccdc1e93a Cache check_host_in_room 2016-08-26 10:59:40 +01:00
Erik Johnston
25414b44a2 Add measure on check_host_in_room 2016-08-26 10:47:00 +01:00
Erik Johnston
50943ab942 Add new state storage funcs to replication 2016-08-26 09:57:32 +01:00
Erik Johnston
30961182f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids 2016-08-26 09:48:13 +01:00
Erik Johnston
c1a133a6b6 Merge pull request #1043 from matrix-org/erikj/backfill_fix 
Fix None check in backfill
 2016-08-26 09:07:05 +01:00
Erik Johnston
778fa85f47 Make sync not pull out full state 2016-08-25 18:59:44 +01:00
Erik Johnston
7356d52e73 Fix up push to use get_current_state_ids 2016-08-25 18:35:49 +01:00
Paul "LeoNerd" Evans
9459137f1e Just sprintf the 'kind' argument into uri directly 2016-08-25 18:35:38 +01:00
Paul "LeoNerd" Evans
1294d4a329 Move ThirdPartyEntityKind into api.constants so the expectation becomes that the value is significant 2016-08-25 18:34:47 +01:00
Mark Haines
ab34fdecb7 Merge branch 'develop' into markjh/direct_to_device 2016-08-25 18:34:46 +01:00
Mark Haines
b162cb2e41 Add some TODOs 2016-08-25 18:18:53 +01:00
Erik Johnston
0e1900d819 Pull out full state less 2016-08-25 18:15:51 +01:00
Mark Haines
641efb6a39 Fix the deduplication of incoming direct-to-device messages 2016-08-25 18:14:02 +01:00
Paul "LeoNerd" Evans
e7af8be5ae Root the 3PE lookup API within /_matrix/app/unstable instead of at toplevel 2016-08-25 18:06:29 +01:00
Paul "LeoNerd" Evans
142983b4ea APP_SERVICE_PREFIX is never used; don't bother 2016-08-25 18:06:05 +01:00
Erik Johnston
721414d98a Add desc 2016-08-25 17:49:05 +01:00
Mark Haines
e993925279 Add store-and-forward direct-to-device messaging 2016-08-25 17:35:37 +01:00
Erik Johnston
a3dc1e9cbe Replace context.current_state with context.current_state_ids 2016-08-25 17:32:22 +01:00
Paul Evans
d9dcb2ba3a Merge pull request #1041 from matrix-org/paul/third-party-lookup 
Extend 3PE lookup APIs for metadata query
 2016-08-25 17:06:53 +01:00
Paul "LeoNerd" Evans
adf53f04ce appease pep8 2016-08-25 16:00:31 +01:00
Paul "LeoNerd" Evans
c435bfee9c Don't need toplevel cache on 3PE lookup metadata any more 2016-08-25 15:57:07 +01:00
Paul "LeoNerd" Evans
db7283cc6b Implement a ResponseCache around 3PE lookup metadata lookups 2016-08-25 15:56:27 +01:00
Paul "LeoNerd" Evans
d0b8d49f71 Kill PROTOCOL_META since I'm not using it any more 2016-08-25 15:45:28 +01:00
Paul "LeoNerd" Evans
5474824975 Actually query over AS API for 3PE lookup metadata 2016-08-25 15:29:36 +01:00
Erik Johnston
17f4f14df7 Pull out event ids rather than full events for state 2016-08-25 13:42:44 +01:00
Erik Johnston
cd5b264b03 Fix None check in backfill 2016-08-25 10:39:19 +01:00
Erik Johnston
60a015550a Bump changelog and version 2016-08-24 14:31:13 +01:00
Erik Johnston
90d5983d7a Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.1 2016-08-24 14:28:57 +01:00
Erik Johnston
c20cb5160d Remove tracer 2016-08-24 13:22:47 +01:00
Erik Johnston
fda97dd58a Merge branch 'develop' of github.com:matrix-org/synapse into erikj/preserve_log_contexts 2016-08-24 13:22:02 +01:00
Paul "LeoNerd" Evans
8e1ed09dff Move static knowledge of protocol metadata into AS handler; cache the result 2016-08-24 13:01:53 +01:00
Paul "LeoNerd" Evans
965f33c901 Declare 'gitter' known protocol, with user lookup 2016-08-24 12:34:03 +01:00
Paul "LeoNerd" Evans
9899824b85 Initial hack at the 3PN protocols metadata lookup API 2016-08-24 12:33:01 +01:00
Erik Johnston
9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Paul "LeoNerd" Evans
63c19e1df9 Move 3PU/3PL lookup APIs into /thirdparty containing entity 2016-08-24 11:55:57 +01:00
Erik Johnston
86bcf4d6a7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/pagination 2016-08-24 10:38:21 +01:00
Erik Johnston
ba07d4a70e Add None checks to backfill 2016-08-24 10:31:05 +01:00
Kegan Dougal
4b31426a02 Pass through user-supplied content in /join/$room_id 
It was always intended to allow custom keys on the join event, but this has
at some point been lost. Restore it.

If the user specifies keys like "avatar_url" then they will be clobbered.
 2016-08-23 16:32:04 +01:00
Erik Johnston
122c7a43c9 Merge pull request #1038 from matrix-org/erikj/receved_txn_purge 
Delete old received_transactions rows
 2016-08-23 11:02:40 +01:00
Erik Johnston
14047126d8 Bump changelog and version 2016-08-22 18:26:07 +01:00
Erik Johnston
d143f211c8 Merge pull request #1028 from matrix-org/dbkr/notifications_api 
Add the Notifications API
 2016-08-22 18:23:24 +01:00
Erik Johnston
aad8a1a825 Delete old received_transactions 2016-08-22 16:29:46 +01:00
Mark Haines
689f4cb914 Update comment 2016-08-22 16:17:31 +01:00
Mark Haines
c8f9b45bc2 Add usage stats to prometheus monitoring 2016-08-22 15:34:38 +01:00
Erik Johnston
e65bc7d315 Merge pull request #1031 from matrix-org/erikj/measure_notifier 
Add more Measure blocks
 2016-08-22 12:13:07 +01:00
Erik Johnston
33f3624ff7 Add exception logging. Fix typo 2016-08-22 10:49:31 +01:00
Erik Johnston
8c52160b07 Allow request handlers to override metric name 2016-08-22 10:44:45 +01:00
Erik Johnston
a093fab253 Use top level measure 2016-08-22 10:18:12 +01:00
Matthew Hodgson
6e80c03d45 Merge branch 'develop' into dbkr/notifications_api 2016-08-20 00:16:18 +01:00
Erik Johnston
27e0178da9 Add a top level measure 2016-08-19 18:49:37 +01:00
Erik Johnston
6d1a94d218 Remove redundant measure 2016-08-19 18:40:31 +01:00
Erik Johnston
8731197e54 Only abort Measure on Exceptions 2016-08-19 18:23:45 +01:00
Erik Johnston
afbf6b33fc defer.returnValue must not be called within Measure 2016-08-19 18:23:44 +01:00
Erik Johnston
37adde32dc Move defer.returnValue out of Measure 2016-08-19 18:23:44 +01:00
Erik Johnston
04fc8bbcb0 Update keyring Measure 2016-08-19 18:23:44 +01:00
Erik Johnston
39b900b316 Measure http.server render 2016-08-19 18:23:44 +01:00
Erik Johnston
47dd8f02a1 Measure _get_event_from_row 2016-08-19 18:23:44 +01:00
Erik Johnston
2426c2f21a Measure keyrings 2016-08-19 18:23:44 +01:00
Erik Johnston
39242090e3 Add measure blocks to notifier 2016-08-19 18:23:44 +01:00
Erik Johnston
e6784daf07 Merge pull request #1030 from matrix-org/erikj/cache_contexts 
Add concept of cache contexts
 2016-08-19 16:29:58 +01:00
Erik Johnston
45fd2c8942 Ensure invalidation list does not grow unboundedly 2016-08-19 16:09:16 +01:00
Erik Johnston
c0d7d9d642 Rename to on_invalidate 2016-08-19 15:13:58 +01:00
Erik Johnston
dc76a3e909 Make cache_context an explicit option 2016-08-19 15:02:38 +01:00
Erik Johnston
f164fd9220 Move _bulk_get_push_rules_for_room to storage layer 2016-08-19 14:29:20 +01:00
Erik Johnston
ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston
4161ff2fc4 Add concept of cache contexts 2016-08-19 14:17:07 +01:00
Erik Johnston
b770435389 Make get_new_events_for_appservice use indices 2016-08-19 10:28:42 +01:00
Paul Evans
5674ea3e6c Merge pull request #1026 from matrix-org/paul/thirdpartylookup 
3rd party entity lookup
 2016-08-18 20:52:50 +01:00
David Baker
1e4217c90c Explicit join 2016-08-18 17:53:44 +01:00
David Baker
0acdd0f1ea Use tuple comparison 
Hopefully easier to read
 2016-08-18 17:51:08 +01:00
Paul "LeoNerd" Evans
65201631a4 Move validation logic for AS 3PE query response into ApplicationServiceApi class, to keep the handler logic neater 2016-08-18 17:33:56 +01:00
Paul "LeoNerd" Evans
697872cf08 More warnings about invalid results from AS 3PE query 2016-08-18 17:24:39 +01:00
Paul "LeoNerd" Evans
b515f844ee Avoid so much copypasta between 3PU and 3PL query by unifying around a ThirdPartyEntityKind enumeration 2016-08-18 17:19:55 +01:00
David Baker
602c84cd9c Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api 2016-08-18 17:15:26 +01:00
Paul "LeoNerd" Evans
2a91799fcc Minor syntax neatenings 2016-08-18 16:58:25 +01:00
Paul "LeoNerd" Evans
fcf1dec809 Appease pep8 2016-08-18 16:26:19 +01:00
Paul "LeoNerd" Evans
105ff162d4 Authenticate 3PE lookup requests 2016-08-18 16:19:23 +01:00
Paul "LeoNerd" Evans
06964c4a0a Copypasta the 3PU support code to also do 3PL 2016-08-18 16:09:50 +01:00
Paul "LeoNerd" Evans
f3afd6ef1a Remove TODO note about request fields being strings - they're always strings 2016-08-18 15:53:01 +01:00
Erik Johnston
bcbd74dc5b Remove log lines 2016-08-18 15:52:10 +01:00
Paul "LeoNerd" Evans
d7b42afc74 Log a warning if an AS yields an invalid 3PU lookup result 2016-08-18 15:49:55 +01:00
Paul "LeoNerd" Evans
80f4740c8f Scattergather the call out to ASes; validate received results 2016-08-18 15:40:41 +01:00
Erik Johnston
07229bbdae Add appservice worker 2016-08-18 14:59:55 +01:00
Paul "LeoNerd" Evans
434bbf2cb5 Filter 3PU lookups by only ASes that declare knowledge of that protocol 2016-08-18 14:56:02 +01:00
Paul "LeoNerd" Evans
d5bf7a4a99 Merge remote-tracking branch 'origin/develop' into paul/thirdpartylookup 2016-08-18 14:21:01 +01:00
Paul "LeoNerd" Evans
718ffcf8bb Since empty lookups now return 200/empty list not 404, we can safely log failures as exceptions 2016-08-18 14:18:37 +01:00
Paul "LeoNerd" Evans
3856582741 Ensure that 3PU lookup request fields actually get passed in 2016-08-18 14:06:02 +01:00
Paul "LeoNerd" Evans
f0c73a1e7a Extend individual list results into the main return list, don't append 2016-08-18 13:53:54 +01:00
Paul "LeoNerd" Evans
b3511adb92 Don't catch the return-value-as-exception that @defer.inlineCallbacks will use 2016-08-18 13:45:18 +01:00
Erik Johnston
9da84a9a1e Make AppserviceHandler stream events from database 
This is for two reasons:

1. Suppresses duplicates correctly, as the notifier doesn't do any
   duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
   stream.
 2016-08-18 11:54:41 +01:00
Mark Haines
403ecd8a2c Missed a s/federation reader/media repository/ in a log message 2016-08-18 10:26:15 +01:00
Mark Haines
47fbff7a05 Merge pull request #1024 from matrix-org/markjh/media_repository 
Add a media repository worker
 2016-08-18 10:12:17 +01:00
Mark Haines
396624864a Add a media repository worker 2016-08-18 09:38:42 +01:00
Erik Johnston
e73dcb66da Merge pull request #1022 from matrix-org/erikj/as_notify_perf 
Make notify_interested_services faster
 2016-08-18 09:35:19 +01:00
Paul "LeoNerd" Evans
3ec10dffd6 Actually make 3PU lookup calls out to ASes 2016-08-18 00:39:09 +01:00
Erik Johnston
732cf72b86 Fix push_display_name_rename schema update 2016-08-17 18:12:21 +01:00
Erik Johnston
abcb9aee5b Make push Measure finer grained 2016-08-17 18:00:18 +01:00
Erik Johnston
2a0d8f8219 Merge pull request #1021 from matrix-org/erikj/mediasecurity_policy 
Set `Content-Security-Policy` on media repo
 2016-08-17 17:22:43 +01:00
Erik Johnston
320dfe523c Make notify_interested_services faster 2016-08-17 17:20:50 +01:00
Erik Johnston
0af9e1a637 Set Content-Security-Policy on media repo 
This is to inform browsers that they should sandbox the returned
media. This is particularly cruical for javascript/HTML files.
 2016-08-17 16:27:39 +01:00
Paul "LeoNerd" Evans
fa87c981e1 Thread 3PU lookup through as far as the AS API object; which currently noöps it 2016-08-17 16:17:28 +01:00
David Baker
0d7cef0943 Merge pull request #1011 from matrix-org/dbkr/contains_display_name_override 
Move display name rule
 2016-08-17 16:15:45 +01:00
Erik Johnston
f90b3d83a3 Add None check to _iterate_over_text 2016-08-17 15:17:17 +01:00
Matrix
f743471380 Change name of metric 2016-08-17 15:05:50 +01:00
Erik Johnston
b9e888858c Move Measure block inside loop 2016-08-17 14:52:26 +01:00
Erik Johnston
973d67a033 Merge pull request #1019 from matrix-org/erikj/appservice_clean 
Clean up _ServiceQueuer
 2016-08-17 14:37:21 +01:00
Paul "LeoNerd" Evans
e3e3fbc23a Initial empty implementation that just registers an API endpoint handler 2016-08-17 12:46:49 +01:00
Erik Johnston
7321f45457 Clean up _ServiceQueuer 2016-08-17 12:03:04 +01:00
Erik Johnston
b9abf3e4e3 Remove dead appservice code 2016-08-17 11:48:23 +01:00
Erik Johnston
62c5245c87 Measure notify_interested_services 2016-08-17 11:12:29 +01:00
Erik Johnston
949629291c Do it in storage function 2016-08-16 17:05:34 +01:00
David Baker
ad42322257 Add migration script 
To port existing rule actions & enable entries to the new name
 2016-08-16 16:56:30 +01:00
David Baker
0bba2799b6 Merge remote-tracking branch 'origin/develop' into dbkr/contains_display_name_override 2016-08-16 16:46:37 +01:00
Erik Johnston
64a2acb161 Don't update caches replication stream if tokens haven't advanced 2016-08-16 16:44:19 +01:00
David Baker
1594eba29e s/underride/override/ in the rule_id too 2016-08-16 16:44:07 +01:00
Erik Johnston
109a560905 Flake8 2016-08-16 14:57:21 +01:00
Erik Johnston
48b5829aea Fix up preview URL API. Add tests. 
This includes:

- Splitting out methods of a class into stand alone functions, to make
  them easier to test.
- Adding unit tests to split out functions, testing HTML -> preview.
- Handle the fact that elements in lxml may have tail text.
 2016-08-16 14:53:24 +01:00
Erik Johnston
7c6f4f9427 Merge pull request #1012 from matrix-org/erikj/limit_backfill_uri 
Limit number of extremeties in backfill request
 2016-08-16 12:55:42 +01:00
Erik Johnston
25c2332071 Merge pull request #1010 from matrix-org/erikj/refactor_deletions 
Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.
 2016-08-16 11:37:53 +01:00
Erik Johnston
2ee1bd124c Limit number of extremeties in backfill request 
This works around a bug where if we make a backfill request with too
many extremeties it causes the request URI to be too long.
 2016-08-16 11:34:36 +01:00
Erik Johnston
a2427981b7 Use cached get_user_by_access_token in slaves 2016-08-16 11:24:32 +01:00
Erik Johnston
6cbd1b495e Merge branch 'fix_integrity_retry' of https://github.com/Ralith/synapse into Ralith-fix_integrity_retry 2016-08-16 10:50:24 +01:00
David Baker
1c7c317df1 Move display name rule 
As per https://github.com/matrix-org/matrix-doc/pull/373 and comment
 2016-08-15 18:34:53 +01:00
Erik Johnston
dc3a00f24f Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves. 2016-08-15 17:04:39 +01:00
Erik Johnston
75299af4fc Merge pull request #1009 from matrix-org/erikj/event_split 
Split out /events to synchrotron
 2016-08-15 15:39:05 +01:00
Erik Johnston
89e786bd85 Doc get_next() context manager usage 2016-08-15 13:45:26 +01:00
Erik Johnston
d9664344ec Rename table. Add docs. 2016-08-15 11:45:57 +01:00
Erik Johnston
784a2d4f2c Remove broken cache stuff 2016-08-15 11:25:48 +01:00
Erik Johnston
0be963472b Use cached version of get_aliases_for_room 2016-08-15 11:24:12 +01:00
Erik Johnston
64e7e11853 Implement cache replication stream 2016-08-15 11:16:45 +01:00
Erik Johnston
4d70d1f80e Add some invalidations to a cache_stream 2016-08-15 11:15:17 +01:00
Erik Johnston
99bbd90b0d Always run txn.after_callbacks 2016-08-15 09:45:44 +01:00
Benjamin Saunders
8a57cc3123 Add missing database corruption recovery case 
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
 2016-08-14 11:50:22 -07:00
Daniel Ehlers
dfaf0fee31
Log the value which is observed in the first place. 
The name 'result' is of bool type and has no len property,
resulting in a TypeError. Futhermore in the flow control
conn.response is observed and hence should be reported.

Signed-off-by: Daniel Ehlers <sargon@toppoint.de>
 2016-08-14 16:49:05 +02:00
Daniel Ehlers
e380538b59
Fix AttributeError when bind_dn is not defined. 
In case one does not define bind_dn in ldap configuration, filter
attribute is not declared. Since auth code only uses ldap_filter attribute
when according LDAP mode is selected, it is safe to only declare the
attribute in that case.

Signed-off-by: Daniel Ehlers <sargon@toppoint.de>
 2016-08-14 16:48:33 +02:00
Erik Johnston
4e1cebd56f Make synchrotron accept /events 2016-08-12 15:31:44 +01:00
Erik Johnston
866a5320de Dont invoke get_handlers fromClientV1RestServlet 
hs.get_handlers() can not be invoked from split out processes. Moving
the invocations down a level means that we can slowly split out
individual servlets.
 2016-08-12 10:03:19 +01:00
Erik Johnston
448ac6cf0d Only process one local membership event per room at a time 2016-08-12 09:32:19 +01:00
Erik Johnston
832799dbff Merge pull request #997 from Half-Shot/develop 
Don't change status_msg on /sync
 2016-08-11 14:10:55 +01:00
David Baker
b4ecf0b886 Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api 2016-08-11 14:09:13 +01:00
Will Hunt
5b5148b7ec Synced up synchrotron set_state with PresenceHandler set_state 2016-08-11 11:48:30 +01:00
Erik Johnston
1d40373c9d Include prev_content in redacted state events 2016-08-11 10:24:41 +01:00
Erik Johnston
c315922b5f PEP8 2016-08-10 16:34:10 +01:00
Erik Johnston
ca8abfbf30 Clean up TransactionQueue 2016-08-10 16:24:16 +01:00
Erik Johnston
5aeadb7414 Merge pull request #999 from matrix-org/erikj/measure_more 
Measure federation send transaction resources
 2016-08-10 14:16:14 +01:00
Erik Johnston
487bc49bf8 Don't stop on 4xx series errors 2016-08-10 13:39:12 +01:00
Erik Johnston
739ea29d1e Also check if server is in the room 2016-08-10 13:32:23 +01:00
Erik Johnston
ea8c4094db Also pull out rejected events 2016-08-10 13:26:13 +01:00
Erik Johnston
7f41bcbeec Correctly auth /event/ requests 2016-08-10 13:22:20 +01:00
Erik Johnston
11fdfaf03b Only resign our own events 2016-08-10 13:16:58 +01:00
Will Hunt
2510db3e76 Don't change status_msg on /sync 2016-08-10 12:59:59 +01:00
Erik Johnston
f91df1f761 Store if we fail to fetch an event from a destination 2016-08-10 11:31:46 +01:00
Erik Johnston
3bc9629be5 Measure federation send transaction resources 2016-08-10 10:56:38 +01:00
Erik Johnston
d45489474d Merge pull request #996 from matrix-org/erikj/tls_error 
Don't print stack traces when failing to get remote keys
 2016-08-10 10:56:08 +01:00
Erik Johnston
fa1ce4d8ad Don't print stack traces when failing to get remote keys 2016-08-10 10:44:37 +01:00
Richard van der Hoff
79ebfbe7c6 /login: Respond with a 403 when we get an invalid m.login.token 2016-08-09 16:29:28 +01:00
David Baker
cd41c6ece2 Merge pull request #995 from matrix-org/rav/clean_up_cas_login 
Clean up CAS login code
 2016-08-09 10:21:56 +01:00
David Baker
27771b2495 Merge pull request #994 from matrix-org/rav/fix_cas_login 
Fix CAS login
 2016-08-08 17:47:45 +01:00
Richard van der Hoff
d3250499c1 Merge pull request #993 from matrix-org/rav/fix_token_login 
Fix token login
 2016-08-08 17:43:02 +01:00
Richard van der Hoff
65666fedd5 Clean up CAS login code 
Remove some apparently unused code.

Clean up parse_cas_response, mostly to catch the exception if the CAS response
isn't valid XML.
 2016-08-08 17:17:25 +01:00
Richard van der Hoff
0682ca04b3 Fix CAS login 
Attempting to log in with CAS was giving a 500 error.
 2016-08-08 17:01:30 +01:00
Richard van der Hoff
6fe6a6f029 Fix login with m.login.token 
login with token (as used by CAS auth) was broken by 067596d, such that it
always returned a 401.
 2016-08-08 16:40:39 +01:00
Erik Johnston
27fe3e2d4f Bump changelog and version 2016-08-08 11:31:12 +01:00
Erik Johnston
3410142741 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.0 2016-08-08 11:24:53 +01:00
Erik Johnston
7c1a92274c Make psutil optional 2016-08-08 11:12:21 +01:00
Erik Johnston
f5deaff424 Merge pull request #991 from matrix-org/erikj/retry_make 
Retry joining via other servers if first one failed. Fix some other bugs.
 2016-08-05 18:21:27 +01:00
Erik Johnston
5f360182c6 Fix a couple of python bugs 2016-08-05 18:08:32 +01:00
Erik Johnston
46453bfc2f Retry joining via other servers if first one failed 2016-08-05 18:02:03 +01:00
Erik Johnston
6bf6bc1d1d Merge pull request #990 from matrix-org/erikj/fed_vers 
Add federation /version API
 2016-08-05 16:59:16 +01:00
Erik Johnston
24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston
597c79be10 Change the way we specify if we require auth or not 2016-08-05 16:17:04 +01:00
Erik Johnston
e6021c370e Merge pull request #989 from matrix-org/erikj/raise_404 
Raise 404 when couldn't find event
 2016-08-05 15:54:16 +01:00
Erik Johnston
a8b946decb Raise 404 when couldn't find event 2016-08-05 15:31:02 +01:00
Erik Johnston
5bcccfde6c Don't include html comments in description 2016-08-05 14:45:11 +01:00
Erik Johnston
4d87d3659a Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.0 2016-08-05 13:19:22 +01:00
Erik Johnston
93acf49e9b Fix backfill auth events 2016-08-05 12:59:04 +01:00
Erik Johnston
b2c290a6e5 Bump version and changelog 2016-08-05 11:16:54 +01:00
Erik Johnston
499dc1b349 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.0 2016-08-05 11:15:48 +01:00
Erik Johnston
2d4de61fb7 Fix typo 2016-08-05 10:48:56 +01:00
Erik Johnston
fccadb7719 Check if we already have the events returned by /state/ 2016-08-05 10:43:47 +01:00
Erik Johnston
f0fa66f495 Delete more tables 2016-08-05 10:40:08 +01:00
Erik Johnston
1515d1b581 Fallback to /state/ on both 400 and 404 2016-08-05 10:24:23 +01:00
Benjamin Saunders
a2b7102eea Tweak integrity error recovery to work as intended 2016-08-04 20:38:08 -07:00
Erik Johnston
a5d7968b3e Merge pull request #973 from matrix-org/erikj/xpath_fix 
Change the way we summarize URLs
 2016-08-04 16:37:25 +01:00
Erik Johnston
b5525c76d1 Typo 2016-08-04 16:10:08 +01:00
Erik Johnston
e97648c4e2 Test summarization 2016-08-04 16:09:09 +01:00
Erik Johnston
835ceeee76 Merge pull request #983 from matrix-org/erikj/retry_on_integrity_error 
Retry event persistence on IntegrityError
 2016-08-04 15:40:03 +01:00
Erik Johnston
b3682df2ca Merge branch 'develop' of github.com:matrix-org/synapse into erikj/xpath_fix 2016-08-04 15:29:45 +01:00
Erik Johnston
8ad8490cff Fix typo 2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88 Retry event persistence on IntegrityError 
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.

This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
 2016-08-04 15:02:15 +01:00
Erik Johnston
257c41cc2e Fix typos. 2016-08-04 14:05:45 +01:00
Erik Johnston
b4e2290d89 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api 2016-08-04 14:04:35 +01:00
Erik Johnston
e3ee63578f Tidy up get_events 2016-08-04 14:01:18 +01:00
Richard van der Hoff
f131cd9e53 keys/query: Omit device displayname if null 
... which makes it more consistent with user displaynames.
 2016-08-04 10:59:51 +01:00
Erik Johnston
edb33eb163 Rename fields to _ids 2016-08-03 17:19:15 +01:00
Erik Johnston
bcc9cda8ca Fix copy + paste fails 2016-08-03 17:17:26 +01:00
Richard van der Hoff
98385888b8 PEP8 2016-08-03 15:42:08 +01:00
Richard van der Hoff
68264d7404 Include device name in /keys/query response 
Add an 'unsigned' section which includes the device display name.
 2016-08-03 15:42:08 +01:00
Richard van der Hoff
91fa69e029 keys/query: return all users which were asked for 
In the situation where all of a user's devices get deleted, we want to
indicate this to a client, so we want to return an empty dictionary, rather
than nothing at all.
 2016-08-03 15:41:44 +01:00
Erik Johnston
4c56bedee3 Actually call get_room_state 2016-08-03 15:04:29 +01:00
Erik Johnston
520ee9bd2c Fix syntax error 2016-08-03 15:03:15 +01:00
Erik Johnston
a60a2eaa02 Comment 2016-08-03 14:52:43 +01:00
Erik Johnston
e3a720217a Add /state_ids federation API 
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
 2016-08-03 14:47:37 +01:00
Richard van der Hoff
530bc862dc Merge branch 'rav/null_default_device_displayname' into develop 2016-08-03 14:30:32 +01:00
Richard van der Hoff
a6f5cc65d9 PEP8 2016-08-03 14:30:06 +01:00
Richard van der Hoff
e555bc6551 Merge branch 'rav/refactor_device_query' into develop 2016-08-03 14:26:01 +01:00
Richard van der Hoff
a843868fe9 E2eKeysHandler: minor tweaks 
PR feedback
 2016-08-03 14:24:33 +01:00
Erik Johnston
80ad710217 Remove other bit of deduplication 2016-08-03 13:25:59 +01:00
Richard van der Hoff
4fec5e57be Default device_display_name to null 
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
 2016-08-03 11:53:00 +01:00
Erik Johnston
a8a32d2714 Ensure we only persist an event once at a time 2016-08-03 11:23:39 +01:00
Mark Haines
921f17f938 Merge branch 'develop' into rav/refactor_device_query 2016-08-03 11:12:47 +01:00
Erik Johnston
58c9653c6b Don't infer paragrahs from newlines 2016-08-02 18:50:24 +01:00
Erik Johnston
6b58ade2f0 Comment on why we clone 2016-08-02 18:41:22 +01:00
Erik Johnston
9e66c58ceb Spelling. 2016-08-02 18:37:31 +01:00
Erik Johnston
f83f5fbce8 Make it actually compile 2016-08-02 18:32:42 +01:00
Erik Johnston
aecaec3e10 Change the way we summarize URLs 
Using XPath is slow on some machines (for unknown reasons), so use a
different approach to get a list of text nodes.

Try to generate a summary that respect paragraph and then word
boundaries, adding ellipses when appropriate.
 2016-08-02 18:25:53 +01:00
Richard van der Hoff
1efee2f52b E2E keys: Make federation query share code with client query 
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
 2016-08-02 18:12:00 +01:00
Erik Johnston
49e047c55e Bump version and changelog 2016-08-02 17:10:26 +01:00
Erik Johnston
59a2c6d60e Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.0 2016-08-02 17:07:54 +01:00
Erik Johnston
c9154b970c Don't double wrap 200 2016-08-02 16:45:53 +01:00
Erik Johnston
b3d5c4ad9d Fix response cache 2016-08-02 16:42:21 +01:00
Erik Johnston
8f650bd338 Bump changeog and version 2016-08-02 15:43:52 +01:00
Erik Johnston
7b0f6293f2 Merge pull request #940 from matrix-org/erikj/fed_state_cache 
Cache federation state responses
 2016-08-02 15:21:37 +01:00
Erik Johnston
342e072024 Merge pull request #967 from matrix-org/erikj/fed_reader 
Split out the federation reading portions into a separate.
 2016-08-02 13:59:47 +01:00
Mark Haines
54de6a812a Merge branch 'develop' into dbkr/fix_add_email_on_register 2016-08-02 10:18:15 +01:00
Erik Johnston
733bf44290 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-08-02 09:56:31 +01:00
Richard van der Hoff
986615b0b2 Move e2e query logic into a handler 2016-08-01 18:02:07 +01:00
Erik Johnston
b260f92936 Ignore AlreadyCalled errors on timer cancel 2016-07-31 16:00:12 +01:00
David Baker
271d3e7865 Fix adding emails on registration 
Synapse was not adding email addresses to accounts registered with an email address, due to too many different variables called 'result'. Rename both of them. Also remove the defer.returnValue() with no params because that's not a thing.
 2016-07-29 15:25:24 +01:00
Paul Evans
18b7eb830b Merge pull request #958 from matrix-org/paul/SYN-738 
Forbid non-ASes from registering users whose names begin with '_'
 2016-07-29 14:10:45 +01:00
Erik Johnston
2a9ce8c422 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:33:46 +01:00
Erik Johnston
cbea0c7044 Merge pull request #964 from matrix-org/erikj/fed_join_fix 
Handle the case of missing auth events when joining a room
 2016-07-29 11:33:16 +01:00
Erik Johnston
5aa024e501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:24:56 +01:00
Erik Johnston
c51a52f300 Mention that func will fetch auth events 2016-07-29 11:17:04 +01:00
Erik Johnston
3d13c3a295 Update docstring 2016-07-29 10:45:05 +01:00
Mark Haines
8dad08a950 Fix SQL to supply arguments in the same order 2016-07-29 09:57:13 +01:00
Mark Haines
0a7d3cd00f Create separate methods for getting messages to push 
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
 2016-07-28 20:24:24 +01:00
Erik Johnston
ec8b217722 Add destination retry to slave store 2016-07-28 17:35:53 +01:00
Erik Johnston
76b89d0edb Add slace storage functions for public room list 2016-07-28 17:03:40 +01:00
Kegan Dougal
370135ad0b Comment get_unread_push_actions_for_user_in_range function 2016-07-28 16:47:37 +01:00
Erik Johnston
0fcbca531f Add get_auth_chain to slave store 2016-07-28 16:36:28 +01:00
Erik Johnston
1e2740caab Handle the case of missing auth events when joining a room 2016-07-28 16:08:33 +01:00
Erik Johnston
6ede23ff1b Add more key storage funcs into slave store 2016-07-28 15:41:26 +01:00
Erik Johnston
5fb41a955c Merge branch 'release-v0.17.0' of github.com:matrix-org/synapse into develop 2016-07-28 14:56:32 +01:00
Mark Haines
4329f20e3f Merge pull request #962 from matrix-org/markjh/retry 
Fix retry utils to check if the exception is a subclass of CME
 2016-07-28 11:12:19 +01:00
Erik Johnston
a285194021 Merge branch 'erikj/key_client_fix' of github.com:matrix-org/synapse into release-v0.17.0 2016-07-28 10:47:06 +01:00
Mark Haines
bf81e38d36 Fix retry utils to check if the exception is a subclass of CME 2016-07-28 10:47:02 +01:00
Erik Johnston
78cac3e594 Merge pull request #941 from matrix-org/erikj/key_client_fix 
Send the correct host header when fetching keys
 2016-07-28 10:46:36 +01:00
Erik Johnston
7871790db1 Bump version and changelog 2016-07-28 10:38:56 +01:00
Erik Johnston
18e044628e Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.17.0 2016-07-28 10:37:43 +01:00
Erik Johnston
b557b682d9 Merge pull request #961 from matrix-org/dbkr/fix_push_invite_name 
Don't include name of room for invites in push
 2016-07-28 10:36:41 +01:00
David Baker
389c890f14 Don't include name of room for invites in push 
Avoids insane pushes like, "Bob invited you to invite from Bob"
 2016-07-28 10:20:47 +01:00
Richard van der Hoff
cd8738ab63 Merge pull request #960 from matrix-org/rav/support_r0.2 
Add r0.2.0 to the "supported versions" list
 2016-07-28 10:14:33 +01:00
Richard van der Hoff
f6f8f81a48 Add r0.1.0 to the "supported versions" list 2016-07-28 10:14:07 +01:00
David Baker
ecd5e6bfa4 Typo 2016-07-28 10:04:52 +01:00
Richard van der Hoff
fda078f995 Add r0.2.0 to the "supported versions" list 2016-07-28 09:14:21 +01:00
Richard van der Hoff
40e539683c Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload 
Make the device id on e2e key upload optional
 2016-07-27 18:16:57 +01:00
Paul "LeoNerd" Evans
05f6447301 Forbid non-ASes from registering users whose names begin with '_' (SYN-738) 2016-07-27 17:54:26 +01:00
Erik Johnston
5238960850 Bump CHANGES and version 2016-07-27 17:33:09 +01:00
Richard van der Hoff
ccec25e2c6 key upload tweaks 
1. Add v2_alpha URL back in, since things seem to be using it.

2. Don't reject the request if the device_id in the upload request fails to
   match that in the access_token.
 2016-07-27 16:41:06 +01:00
Mark Haines
29b25d59c6 Merge branch 'develop' into markjh/verify 
Conflicts:
	synapse/crypto/keyring.py
 2016-07-27 15:11:02 +01:00
Mark Haines
884b800899 Merge pull request #955 from matrix-org/markjh/only_from2 
Add a couple more checks to the keyring
 2016-07-27 15:08:22 +01:00
Mark Haines
09d31815b4 Merge pull request #954 from matrix-org/markjh/even_more_fixes 
Fix a couple of bugs in the transaction and keyring code
 2016-07-27 15:08:14 +01:00
Mark Haines
fe1b369946 Clean up verify_json_objects_for_server 2016-07-27 14:10:43 +01:00
Richard van der Hoff
26cb0efa88 SQL syntax fix 2016-07-27 12:30:22 +01:00
Richard van der Hoff
d47115ff8b Delete e2e keys on device delete 2016-07-27 12:24:52 +01:00
Richard van der Hoff
2e3d90d67c Make the device id on e2e key upload optional 
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.

For active access_tokens without an associated device_id, we ought to register
the device in the devices table.

Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
 2016-07-26 23:38:12 +01:00
Mark Haines
a4b06b619c Add a couple more checks to the keyring 2016-07-26 19:50:11 +01:00
Mark Haines
c63b1697f4 Merge pull request #952 from matrix-org/markjh/more_fixes 
Check if the user is banned when handling 3pid invites
 2016-07-26 19:20:56 +01:00
Mark Haines
87ffd21b29 Fix a couple of bugs in the transaction and keyring code 2016-07-26 19:19:08 +01:00
Richard van der Hoff
eb359eced4 Add create_requester function 
Wrap the `Requester` constructor with a function which provides sensible
defaults, and use it throughout
 2016-07-26 16:46:53 +01:00
Mark Haines
c824b29e77 Check if the user is banned when handling 3pid invites 2016-07-26 16:39:14 +01:00
Richard van der Hoff
33d7776473 Fix typo 2016-07-26 13:32:15 +01:00
Richard van der Hoff
9ad8d9b17c Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device 2016-07-26 13:29:46 +01:00
Richard van der Hoff
5b1825ba5b Merge pull request #951 from matrix-org/rav/flake8 
Fix flake8 noise
 2016-07-26 13:27:54 +01:00
Mark Haines
9c4cf83259 Merge pull request #948 from matrix-org/markjh/auth_fixes 
Don't add rejections to the state_group, persist all rejections
 2016-07-26 13:22:57 +01:00
Richard van der Hoff
05e7e5e972 Fix flake8 violation 
Apparently flake8 v3 puts the error on a different line to v2. Easiest way to
make sure that happens is by putting the whole statement on one line :)
 2016-07-26 11:59:08 +01:00
Richard van der Hoff
8e02494166 Delete refresh tokens when deleting devices 2016-07-26 11:10:37 +01:00
Mark Haines
a6f06ce3e2 Fix how push_actions are redacted. 2016-07-26 11:05:39 +01:00
David Baker
d34e9f93b7 Merge pull request #949 from matrix-org/rav/update_devices 
Implement updates and deletes for devices
 2016-07-26 10:49:55 +01:00
Mark Haines
efeb6176c1 Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically 2016-07-26 10:49:52 +01:00
Matthew Hodgson
242c52d607 typo 2016-07-26 10:09:25 +02:00
Richard van der Hoff
012b4c1913 Implement updating devices 
You can update the displayname of devices now.
 2016-07-26 07:35:48 +01:00
Richard van der Hoff
436bffd15f Implement deleting devices 2016-07-26 07:35:48 +01:00
Mark Haines
1b3c3e6d68 Only update the events and event_json tables for rejected events 2016-07-25 18:44:30 +01:00
Richard van der Hoff
33d08e8433 Log when adding listeners 2016-07-25 17:22:15 +01:00
Mark Haines
8f7f4cb92b Don't add the events to forward extremities if the event is rejected 2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874 Don't add rejections to the state_group, persist all rejections 2016-07-25 16:12:16 +01:00
David Baker
4fcdf7b4b2 Merge pull request #946 from matrix-org/dbkr/log_recaptcha_hostname 
Log the hostname the reCAPTCHA was completed on
 2016-07-25 16:10:39 +01:00
Mark Haines
955ef1f06c fix: defer.returnValue takes one argument 2016-07-25 16:04:45 +01:00
Richard van der Hoff
2ee4c9ee02 background updates: fix assert again 2016-07-25 16:01:46 +01:00
Richard van der Hoff
9dbd903f41 background updates: Fix assertion to do something 2016-07-25 14:05:23 +01:00
Richard van der Hoff
465117d7ca Fix background_update tests 
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
 2016-07-25 12:10:42 +01:00
David Baker
7ed58bb347 Use get to avoid KeyErrors 2016-07-22 17:18:50 +01:00
David Baker
dad2da7e54 Log the hostname the reCAPTCHA was completed on 
This could be useful information to have in the logs. Also comment about how & why we don't verify the hostname.
 2016-07-22 17:00:56 +01:00
Richard van der Hoff
363786845b PEP8 2016-07-22 13:21:07 +01:00
Richard van der Hoff
ec5717caf5 Create index on user_ips in the background 
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.

I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
 2016-07-22 13:16:39 +01:00
Erik Johnston
d26b660aa6 Cache getPeer 2016-07-21 17:38:51 +01:00
Erik Johnston
aede7248ab Split out a FederationReader process 2016-07-21 17:37:44 +01:00
Richard van der Hoff
55abbe1850 make /devices return a list 
Turns out I specced this to return a list of devices rather than a dict of them
 2016-07-21 15:57:28 +01:00
David Baker
2c28e25bda Merge pull request #943 from matrix-org/rav/get_device_api 
Implement GET /device/{deviceId}
 2016-07-21 13:41:42 +01:00
David Baker
1e6e370b76 Merge pull request #942 from matrix-org/rav/fix_register_deviceid 
Preserve device_id from first call to /register
 2016-07-21 13:16:31 +01:00
Richard van der Hoff
1c3c202b96 Fix PEP8 errors 2016-07-21 13:15:15 +01:00
Richard van der Hoff
406f7aa0f6 Implement GET /device/{deviceId} 2016-07-21 12:00:29 +01:00
Richard van der Hoff
c445f5fec7 storage/client_ips: remove some dead code 2016-07-21 11:58:47 +01:00
Erik Johnston
cf94a78872 Set host not path 2016-07-21 11:45:53 +01:00
Richard van der Hoff
1a64dffb00 Preserve device_id from first call to /register 
device_id may only be passed in the first call to /register, so make sure we
fish it out of the register `params` rather than the body of the final call.
 2016-07-21 11:34:16 +01:00
Erik Johnston
081e5d55e6 Send the correct host header when fetching keys 2016-07-21 11:14:54 +01:00
Erik Johnston
248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Richard van der Hoff
7314bf4682 Merge branch 'develop' into rav/get_devices_api 
(pick up PR #938 in the hope of fixing the UTs)
 2016-07-20 17:40:00 +01:00
Richard van der Hoff
e9e3eaa67d Merge pull request #938 from matrix-org/rav/add_device_id_to_client_ips 
Record device_id in client_ips
 2016-07-20 17:38:45 +01:00
Erik Johnston
d36b1d849d Don't explode if we have no snapshots yet 2016-07-20 16:59:52 +01:00
David Baker
742056be0d Merge pull request #937 from matrix-org/rav/register_device_on_register 
Register a device_id in the /v2/register flow.
 2016-07-20 16:51:27 +01:00
Richard van der Hoff
bc8f265f0a GET /devices endpoint 
implement a GET /devices endpoint which lists all of the user's devices.

It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
 2016-07-20 16:42:32 +01:00
Richard van der Hoff
ec041b335e Record device_id in client_ips 
Record the device_id when we add a client ip; it's somewhat redundant as we
could get it via the access_token, but it will make querying rather easier.
 2016-07-20 16:41:03 +01:00
Richard van der Hoff
053e83dafb More doc-comments 
Fix some more comments on some things
 2016-07-20 16:40:28 +01:00
Richard van der Hoff
b97a1356b1 Register a device_id in the /v2/register flow. 
This doesn't cover *all* of the registration flows, but it does cover the most
common ones: in particular: shared_secret registration, appservice
registration, and normal user/pass registration.

Pull device_id from the registration parameters. Register the device in the
devices table. Associate the device with the returned access and refresh
tokens. Profit.
 2016-07-20 16:38:27 +01:00
Erik Johnston
b73dc0ef4d Merge pull request #936 from matrix-org/erikj/log_rss 
Add metrics for psutil derived memory usage
 2016-07-20 16:32:38 +01:00
Erik Johnston
66868119dc Add metrics for psutil derived memory usage 2016-07-20 16:00:21 +01:00
Erik Johnston
aba0b2a39b Merge pull request #935 from matrix-org/erikj/backfill_notifs 
Don't notify pusher pool for backfilled events
 2016-07-20 13:39:16 +01:00
Erik Johnston
57dca35692 Don't notify pusher pool for backfilled events 2016-07-20 13:25:06 +01:00
Richard van der Hoff
c68518dfbb Merge pull request #933 from matrix-org/rav/type_annotations 
Type annotations
 2016-07-20 12:26:32 +01:00
David Baker
e967bc86e7 Merge pull request #932 from matrix-org/rav/register_refactor 
Further registration refactoring
 2016-07-20 11:03:33 +01:00
Erik Johnston
1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2 
Feature: Add filter to /messages. Add 'contains_url' to filter.
 2016-07-20 10:40:48 +01:00
Erik Johnston
f91faf09b3 Comment 2016-07-20 10:18:09 +01:00
Richard van der Hoff
3413f1e284 Type annotations 
Add some type annotations to help PyCharm (in particular) to figure out the
types of a bunch of things.
 2016-07-19 18:56:16 +01:00
Richard van der Hoff
40cbffb2d2 Further registration refactoring 
* `RegistrationHandler.appservice_register` no longer issues an access token:
  instead it is left for the caller to do it. (There are two of these, one in
  `synapse/rest/client/v1/register.py`, which now simply calls
  `AuthHandler.issue_access_token`, and the other in
  `synapse/rest/client/v2_alpha/register.py`, which is covered below).

* In `synapse/rest/client/v2_alpha/register.py`, move the generation of
  access_tokens into `_create_registration_details`. This means that the normal
  flow no longer needs to call `AuthHandler.issue_access_token`; the
  shared-secret flow can tell `RegistrationHandler.register` not to generate a
  token; and the appservice flow continues to work despite the above change.
 2016-07-19 18:46:19 +01:00
David Baker
b9e997f561 Merge pull request #931 from matrix-org/rav/refactor_register 
rest/client/v2_alpha/register.py: Refactor flow somewhat.
 2016-07-19 16:13:45 +01:00
Richard van der Hoff
9a7a77a22a Merge pull request #929 from matrix-org/rav/support_deviceid_in_login 
Add device_id support to /login
 2016-07-19 15:53:04 +01:00
Richard van der Hoff
8f6281ab0c Don't bind email unless threepid contains expected fields 2016-07-19 15:50:01 +01:00
Richard van der Hoff
0da0d0a29d rest/client/v2_alpha/register.py: Refactor flow somewhat. 
This is meant to be an *almost* non-functional change, with the exception that
it fixes what looks a lot like a bug in that it only calls
`auth_handler.add_threepid` and `add_pusher` once instead of three times.

The idea is to move the generation of the `access_token` out of
`registration_handler.register`, because `access_token`s now require a
device_id, and we only want to generate a device_id once registration has been
successful.
 2016-07-19 13:12:22 +01:00
Richard van der Hoff
022b9176fe schema fix 
device_id should be text, not bigint.
 2016-07-19 11:44:05 +01:00
Mark Haines
c41d52a042 Summary line 2016-07-19 10:28:27 +01:00