Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-22 04:24:57 -05:00
Code Issues Packages Projects Releases Wiki Activity
12,607 Commits 4 Branches 609 Tags 60 MiB
cb7a6b2379
Commit Graph

2532 Commits

Author SHA1 Message Date
Erik Johnston
360ba89c50 Don't fail requests to unbind 3pids for non supporting ID servers 
Older identity servers may not support the unbind 3pid request, so we
shouldn't fail the requests if we received one of 400/404/501. The
request still fails if we receive e.g. 500 responses, allowing clients
to retry requests on transient identity server errors that otherwise do
support the API.

Fixes #3661
 2018-08-08 12:06:18 +01:00
Richard van der Hoff
53bca4690b more metrics for the federation and appservice senders 2018-08-07 19:09:48 +01:00
Richard van der Hoff
3523f5432a Don't expose default_room_version as config opt 2018-08-07 12:51:57 +01:00
Erik Johnston
495cb100d1 Allow profile changes to happen on workers 2018-08-07 10:50:26 +01:00
Erik Johnston
a3f5bf79a0 Add EDU/query handling over replication 2018-08-06 15:23:31 +01:00
Erik Johnston
e26dbd82ef Add replication APIs for persisting federation events 2018-08-06 15:02:28 +01:00
Richard van der Hoff
f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Neil Johnson
e40a510fbf py3 fix 2018-08-03 23:19:13 +01:00
Neil Johnson
886be75ad1 bug fixes 2018-08-03 22:29:03 +01:00
Will Hunt
16d9701892 Return M_NOT_FOUND when a profile could not be found. (#3596) 2018-08-03 19:08:05 +01:00
Richard van der Hoff
0ca459ea33 Basic support for room versioning 
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
 2018-08-03 16:08:32 +01:00
Neil Johnson
897c51d274 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-03 13:40:47 +01:00
Erik Johnston
cb298ff623 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet 2018-08-03 09:25:15 +01:00
Richard van der Hoff
1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling 
Clean up handling of errors from outbound requests
 2018-08-02 17:38:24 +01:00
Neil Johnson
74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Richard van der Hoff
704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff
0bf5ec0db7 Check room visibility for /event/ requests 
Make sure that the user has permission to view the requeseted event for
/event/{eventId} and /room/{roomId}/event/{eventId} requests.

Also check that the event is in the given room for
/room/{roomId}/event/{eventId}, for sanity.
 2018-08-02 15:03:27 +01:00
Richard van der Hoff
14fa9d4d92 Avoid extra db lookups 
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
 2018-08-02 13:55:51 +01:00
Richard van der Hoff
0a65450d04 Validation for events/rooms in fed requests 
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
 2018-08-02 13:48:40 +01:00
Erik Johnston
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store 
Split out DB writes in federation handler
 2018-08-02 10:41:42 +01:00
Neil Johnson
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits 
Initial impl of capping MAU
 2018-08-01 15:58:45 +00:00
Richard van der Hoff
01e93f48ed Kill off MatrixCodeMessageException 
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).

Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
 2018-08-01 16:02:46 +01:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson
303f1c851f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits 2018-08-01 13:42:50 +01:00
Erik Johnston
a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Erik Johnston
4b256b9271 _persist_auth_tree no longer returns anything 2018-08-01 13:39:07 +01:00
Neil Johnson
0aba3d361a count_monthly_users() async 2018-08-01 11:47:58 +01:00
Neil Johnson
7931393495 make count_monthly_users async synapse/handlers/auth.py 2018-08-01 10:21:56 +01:00
Richard van der Hoff
bdbdceeafa rename replication_layer to federation_client 
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
 2018-07-31 15:44:05 +01:00
Erik Johnston
443da003bc Use new helper base class for membership requests 2018-07-31 14:32:23 +01:00
Erik Johnston
729b672823 Use new helper base class for ReplicationSendEventRestServlet 2018-07-31 14:32:23 +01:00
Neil Johnson
df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Neil Johnson
251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Erik Johnston
143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Matthew Hodgson
e9b2d047f6
make /context lazyload & filter aware (#3567) 
make /context lazyload & filter aware.
 2018-07-27 15:12:50 +01:00
Richard van der Hoff
b0b5566f36
Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid 
if inviter_display_name == ""||None then default to inviter MXID
 2018-07-27 10:08:39 +01:00
Matthew Hodgson
a75231b507
Deduplicate redundant lazy-loaded members (#3331) 
* attempt at deduplicating lazy-loaded members

as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again.  For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up).  For now we hardcode X to 30.

* add include_redundant_members filter option & make it work

* remove stale todo

* add tests for _get_some_state_from_cache

* incorporate review
 2018-07-26 22:51:30 +01:00
Richard van der Hoff
03751a6420 Fix some looping_call calls which were broken in #3604 
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
 2018-07-26 11:48:08 +01:00
Matthew Hodgson
1bcd0490c2
Merge pull request #2970 from matrix-org/matthew/filter_members 
Implement the lazy_load_members room state filter parameter
 2018-07-26 00:03:01 +01:00
Travis Ralston
d8e65ed7e1
Fix a minor documentation typo in on_make_leave 2018-07-25 15:44:41 -06:00
Matthew Hodgson
2565804030 Merge branch 'develop' into matthew/filter_members 2018-07-25 17:27:49 +01:00
Erik Johnston
78a691d005 Split out DB writes in federation handler 
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
 2018-07-25 16:22:56 +01:00
Richard van der Hoff
55acd6856c Fix updating of cached remote profiles 
_update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when
it was called, would just return a generator object, without actually running
any of the method body.
 2018-07-25 10:34:48 +01:00
Michael Telatynski
87951d3891
Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid 2018-07-24 17:17:46 +01:00
Erik Johnston
8b8c4f34a3 Replace usage of get_current_toke with StreamToken.START 
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
 2018-07-24 16:49:17 +01:00
Richard van der Hoff
a321f78991
Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups 
Fixes and optimisations for resolve_state_groups
 2018-07-24 14:11:45 +01:00
Matthew Hodgson
1a01a5b964 clarify comment on p_ids 2018-07-24 14:03:15 +01:00
Matthew Hodgson
eb1d911ab7 rather than adding ll_ids, remove them from p_ids 2018-07-24 13:40:49 +01:00
Matthew Hodgson
d19fba3655 Merge branch 'develop' into matthew/filter_members 2018-07-24 12:39:54 +01:00
Matthew Hodgson
cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Richard van der Hoff
a678145010
Merge branch 'develop' into rav/logcontext_fixes 2018-07-24 10:43:30 +01:00
Erik Johnston
536bc63a4e
Merge branch 'develop' into erikj/client_apis_move 2018-07-24 09:57:05 +01:00
Richard van der Hoff
cf2d15c6a9 another couple of logcontext leaks 2018-07-24 00:57:48 +01:00
Richard van der Hoff
8dff6e0322 Logcontext fixes 
Fix some random logcontext leaks.
 2018-07-24 00:37:17 +01:00
Richard van der Hoff
c1f80effbe Handle delta_ids being None in _update_context_for_auth_events 
it's easier to create the new state group as a delta from the existing one.

(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364)
 2018-07-23 22:06:50 +01:00
Matthew Hodgson
adfe29ec0b Merge branch 'develop' into matthew/filter_members 2018-07-23 19:21:37 +01:00
Matthew Hodgson
254fb430d1 incorporate review 2018-07-23 19:21:20 +01:00
Erik Johnston
0faa3223cd Fix missing attributes on workers. 
This was missed during the transition from attribute to getter for
getting state from context.
 2018-07-23 16:28:00 +01:00
Erik Johnston
0b0b24cb82 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move 2018-07-23 13:21:15 +01:00
Erik Johnston
027bc01a1b Add support for updating state 2018-07-23 13:17:25 +01:00
Erik Johnston
e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Amber Brown
3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Erik Johnston
5c88bb722f Move PaginationHandler to its own file 2018-07-20 15:32:23 +01:00
Erik Johnston
0ecf68aedc Move check_in_room_or_world_readable to Auth 2018-07-20 15:30:59 +01:00
Richard van der Hoff
7c712f95bb Combine Limiter and Linearizer 
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
 2018-07-20 13:11:43 +01:00
Richard van der Hoff
8462c26485 Improvements to the Limiter 
* give them names, to improve logging
* use a deque rather than a list for efficiency
 2018-07-20 12:50:27 +01:00
Matthew Hodgson
bcaec2915a incorporate review 2018-07-19 19:03:50 +01:00
Matthew Hodgson
924eb34d94 add a filtered_types param to limit filtering to specific types 2018-07-19 18:32:02 +01:00
Matthew Hodgson
be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Erik Johnston
bacdf0cbf9 Move RoomContextHandler out of Handlers 
This is in preparation for moving GET /context/ to a worker
 2018-07-18 15:33:03 +01:00
Erik Johnston
8cb8df55e9 Split MessageHandler into read only and writers 
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
 2018-07-18 15:33:03 +01:00
Richard van der Hoff
dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks 
Remove event re-signing hacks
 2018-07-18 12:46:27 +01:00
Richard van der Hoff
9c04b4abf9
Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server 
Refactor and optimze filter_events_for_server
 2018-07-17 14:01:39 +01:00
Richard van der Hoff
ea69d35651 Move filter_events_for_server out of FederationHandler 
for easier unit testing.
 2018-07-16 13:06:24 +01:00
Matthew Hodgson
ba22b6a456 typo 2018-07-13 12:03:39 +01:00
Matthew Hodgson
ea752bdd99 s/becuase/because/g 2018-07-10 17:58:18 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
a4ab491371
Merge branch 'develop' into rav/drop_re_signing_hacks 2018-07-04 07:13:38 +01:00
Erik Johnston
e3b4043800
Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking 
Check the state of prev_events a bit more thoroughly when coming over federation
 2018-06-29 13:55:02 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Amber Brown
99800de63d try and clean up 2018-06-27 11:40:27 +01:00
Amber Brown
f03a5d1a17 pep8 2018-06-27 11:38:14 +01:00
Amber Brown
a7ecf34b70 cleanups 2018-06-27 11:36:03 +01:00
Amber Brown
77078d6c8e handle federation not telling us about prev_events 2018-06-27 11:27:32 +01:00
Matthew Hodgson
c7f6b420ae
Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api 
add GDPR erase param to deactivate API
 2018-06-26 10:43:14 +01:00
Matthew Hodgson
1e788db430 add GDPR erase param to deactivate API 2018-06-26 10:26:54 +01:00
Erik Johnston
d72fb9a448
Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts 
allow non-consented users to still part rooms (to let us autopart them)
 2018-06-25 20:14:18 +01:00
Erik Johnston
df48f7ef37 Actually fix it 2018-06-25 20:03:41 +01:00
Erik Johnston
a0e8a53c6d Comment 2018-06-25 19:57:38 +01:00
Erik Johnston
7bdc5c8fa3 Fix bug with assuming wrong type 2018-06-25 19:56:02 +01:00
Erik Johnston
ea7a9c0483 Add fast path to _filter_events_for_server 
Most rooms have a trivial history visibility like "shared" or
"world_readable", especially large rooms, so lets not bother getting the
full membership of those rooms in that case.
 2018-06-25 19:49:13 +01:00
Matthew Hodgson
0269367f18 allow non-consented users to still part rooms (to let us autopart them) 2018-06-25 17:56:10 +01:00
Matthew Hodgson
784189b1f4 typos 2018-06-25 17:37:16 +01:00
Matthew Hodgson
6eb861b67f typo 2018-06-25 17:37:16 +01:00
Erik Johnston
33fdcfa957
Merge pull request #3441 from matrix-org/erikj/redo_erasure 
Fix user erasure and re-enable
 2018-06-25 14:37:01 +01:00
Amber Brown
07cad26d65
Remove all global reactor imports & pass it around explicitly (#3424) 2018-06-25 14:08:28 +01:00
Erik Johnston
244484bf3c Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" 
This reverts commit 1d009013b3.
 2018-06-25 13:42:55 +01:00
Richard van der Hoff
1d009013b3 Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" 
This reverts commit ce0d911156, reversing
changes made to b4a5d767a9.
 2018-06-22 16:35:10 +01:00
Erik Johnston
28ddc6cfbe Also log number of events for serach context 2018-06-22 15:42:11 +01:00
Erik Johnston
4b4cec3989 Add some logging to search queries 2018-06-22 15:42:11 +01:00
Erik Johnston
ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility 
Support hiding events from deleted users
 2018-06-22 15:06:44 +01:00
Erik Johnston
b4a5d767a9
Merge pull request #3428 from matrix-org/erikj/persisted_pdu 
Simplify get_persisted_pdu
 2018-06-22 14:47:43 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
Richard van der Hoff
02bfc581f8
Merge pull request #3399 from costacruise/master 
Add error code to room creation error
 2018-06-20 17:26:25 +01:00
Michael Wagner
19cd3120ec Add error code to room creation error 
This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom
 2018-06-14 14:08:40 +02:00
Amber Brown
a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Michael Telatynski
94700e55fa
if inviter_display_name == ""||None then default to inviter MXID 
to prevent email invite from "None"
 2018-06-13 10:31:01 +01:00
Richard van der Hoff
b6faef2ad7 Filter out erased messages 
Redact any messges sent by erased users.
 2018-06-12 09:53:18 +01:00
Richard van der Hoff
f1023ebf4b mark accounts as erased when requested 2018-06-12 09:53:18 +01:00
Richard van der Hoff
9fc5b74b24 simplify get_persisted_pdu 
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
 2018-06-12 09:51:31 +01:00
David Baker
187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind 
Remove email addresses / phone numbers from ID servers when they're removed from synapse
 2018-06-11 16:02:00 +01:00
Matthew Hodgson
c96d882a02 Merge branch 'develop' into matthew/filter_members 2018-06-10 12:26:14 +03:00
Richard van der Hoff
c6b1441c52 Fix event filtering in get_missing_events handler 2018-06-08 14:15:31 +01:00
Richard van der Hoff
0834b49c6a Fix event filtering in get_missing_events handler 2018-06-08 11:34:46 +01:00
Richard van der Hoff
8503dd0047 Remove event re-signing hacks 
These "temporary fixes" have been here three and a half years, and I can't find
any events in the matrix.org database where the calculated signature differs
from what's in the db. It's time for them to go away.
 2018-06-07 16:08:29 +01:00
David Baker
d62162bbec doc fixes 2018-06-05 18:09:13 +01:00
David Baker
6a29e815fc Fix comment 2018-06-04 12:01:23 +01:00
David Baker
e44150a6de Missing yield 2018-06-04 12:01:13 +01:00
David Baker
f731e42baf docstring 2018-06-04 12:00:51 +01:00
Matthew Hodgson
28f09fcdd5 Merge branch 'develop' into matthew/filter_members 2018-06-04 00:09:17 +03:00
Matthew Hodgson
5f6122fe10 more comments 2018-06-04 00:08:52 +03:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Richard van der Hoff
34b85df7f5 Update some comments and docstrings in SyncHandler 2018-05-29 22:31:18 +01:00
Richard van der Hoff
4a9cbdbc15 Exempt AS-registered users from doing gdpr 2018-05-29 19:54:32 +01:00
Matthew Hodgson
a6c8f7c875 add pydoc 2018-05-29 01:09:55 +01:00
Matthew Hodgson
7a6df013cc merge develop 2018-05-29 00:25:22 +01:00
Amber Brown
754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Richard van der Hoff
41921ac01b
Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room 
Let users leave the server notice room after joining
 2018-05-25 11:15:45 +01:00
Richard van der Hoff
757ed27258 Let users leave the server notice room after joining 
They still can't reject invites, but we let them leave it.
 2018-05-25 11:07:21 +01:00
Adrian Tschira
17a70cf6e9 Misc. py3 fixes 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-24 20:20:33 +02:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4 
replace some iteritems with six
 2018-05-24 13:04:07 -05:00
David Baker
77a23e2e05 Merge remote-tracking branch 'origin/develop' into dbkr/unbind 2018-05-24 16:20:53 +01:00
David Baker
7a1af504d7 Remove users from user directory on deactivate 2018-05-24 15:59:58 +01:00
Erik Johnston
f72d5a44d5
Merge pull request #3261 from matrix-org/erikj/pagination_fixes 
Fix federation backfill bugs
 2018-05-24 14:52:03 +01:00
Erik Johnston
68399fc4de
Merge pull request #3267 from matrix-org/erikj/iter_filter 
Use iter* methods for _filter_events_for_server
 2018-05-24 14:44:57 +01:00
David Baker
9700d15611 pep8 2018-05-24 11:23:15 +01:00
David Baker
a21a41bad7 comment 2018-05-24 11:19:59 +01:00
David Baker
b3bff53178 Unbind 3pids when they're deleted too 2018-05-24 11:08:05 +01:00
Richard van der Hoff
a0b3946fe2
Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri 2018-05-23 16:06:03 +01:00
Richard van der Hoff
3ff6f50eac Use the localpart in the consent uri 
... because it's shorter.
 2018-05-23 15:28:23 +01:00
Richard van der Hoff
82191b08f6 Support for putting %(consent_uri)s in messages 
Make it possible to put the URI in the error message and the server notice that
get sent by the server
 2018-05-23 15:24:31 +01:00
David Baker
2c7866d664 Hit the 3pid unbind endpoint on deactivation 2018-05-23 14:38:56 +01:00
Erik Johnston
5aaa3189d5 s/values/itervalues/ 2018-05-23 10:13:05 +01:00
Erik Johnston
0a4bca4134 Use iter* methods for _filter_events_for_server 2018-05-23 10:04:23 +01:00
Amber Brown
53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Erik Johnston
e85b5a0ff7 Use iter* methods 2018-05-22 19:02:48 +01:00
Erik Johnston
586b66b197 Fix that states is a dict of dicts 2018-05-22 19:02:36 +01:00
Richard van der Hoff
08bfc48abf custom error code for not leaving server notices room 2018-05-22 17:27:27 +01:00
David Baker
0a078026ea comment typo 2018-05-22 17:14:06 +01:00
Amber Brown
8f5a688d42 cleanups, self-registration 2018-05-22 10:56:03 -05:00
Amber Brown
a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Erik Johnston
cb2a2ad791 get_domains_from_state returns list of tuples 2018-05-22 16:23:39 +01:00
Richard van der Hoff
a5e2941aad Reject attempts to send event before privacy consent is given 
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
 2018-05-22 12:00:47 +01:00
Richard van der Hoff
8810685df9 Stub out ServerNoticesSender on the workers 
... and have the sync endpoints call it directly rather than obsure indirection
via PresenceHandler
 2018-05-22 11:54:51 +01:00
Richard van der Hoff
9ea219c514 Send users a server notice about consent 
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
 2018-05-22 11:54:51 +01:00
Amber Brown
df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Adrian Tschira
933bf2dd35 replace some iteritems with six 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-19 17:59:26 +02:00
Richard van der Hoff
6d9dc67139
Merge pull request #3232 from matrix-org/rav/server_notices_room 
Infrastructure for a server notices room
 2018-05-18 11:28:04 +01:00
Richard van der Hoff
67af392712
Merge pull request #3233 from matrix-org/rav/remove_dead_code 
Remove unused `update_external_syncs`
 2018-05-18 11:22:43 +01:00
Richard van der Hoff
26305788fe Make sure we reject attempts to invite the notices user 2018-05-18 11:18:39 +01:00
Erik Johnston
fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token 
Make purge_history operate on tokens
 2018-05-18 10:35:23 +01:00
Richard van der Hoff
88d3405332 fix missing yield for server_notices_room 2018-05-17 18:33:45 +01:00
Richard van der Hoff
d43d480d86 Remove unused update_external_syncs 
This method isn't used anywhere. Burninate it.
 2018-05-17 18:22:19 +01:00
Richard van der Hoff
fed62e21ad Infrastructure for a server notices room 
Server Notices use a special room which the user can't dismiss. They are
created on demand when some other bit of the code calls send_notice.

(This doesn't actually do much yet becuse we don't call send_notice anywhere)
 2018-05-17 17:58:25 +01:00
Richard van der Hoff
c46367d0d7 Move RoomCreationHandler out of synapse.handlers.Handlers 
Handlers is deprecated nowadays, so let's move this out before I add a new
dependency on it.

Also fix the docstrings on create_room.
 2018-05-17 09:08:42 +01:00
Erik Johnston
5f27ed75ad Make purge_history operate on tokens 
As we're soon going to change how topological_ordering works
 2018-05-15 16:23:50 +01:00
David Baker
8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate 
Part user from rooms on account deactivate
 2018-05-14 11:31:48 +01:00
David Baker
6b49628e3b Catch failure to part user from room 2018-05-10 12:23:53 +01:00
David Baker
217bc53c98 Many docstrings 2018-05-10 12:20:40 +01:00
David Baker
4d298506dd Oops, don't call function passed to run_in_background 2018-05-10 11:57:13 +01:00
Erik Johnston
bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync 
Refactor sync APIs to reuse pagination API
 2018-05-09 16:16:56 +01:00
Erik Johnston
d26bec8a43 Add comment to sync as to why code path is split 2018-05-09 15:56:07 +01:00
Erik Johnston
fcf55f2255 Fix returned token is no longer a tuple 2018-05-09 15:43:00 +01:00
David Baker
294e9a0c9b Prefix internal functions 2018-05-09 15:10:37 +01:00
David Baker
52281e4c54 Indent fail 2018-05-09 15:06:16 +01:00
David Baker
7e8726b8fb Part deactivated users in the background 
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
 2018-05-09 14:54:28 +01:00
Erik Johnston
e2accd7f1d Refactor sync APIs to reuse pagination API 
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
 2018-05-09 13:43:39 +01:00
Erik Johnston
c4af4c24ca Refactor get_recent_events_for_room return type 
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
 2018-05-09 11:55:34 +01:00
David Baker
bf98fa0864 Part user from rooms on account deactivate 
This implements this very crudely: this probably isn't viable
because parting a user from all their rooms could take a long time,
and if the HS gets restarted in that time the process will be
aborted.
 2018-05-08 15:58:35 +01:00
Richard van der Hoff
ca7211104e Merge branch 'release-v0.28.1' into develop 2018-05-01 18:16:57 +01:00
Richard van der Hoff
33f469ba19 Apply some limits to depth to counter abuse 
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth

As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
 2018-05-01 17:54:19 +01:00
Richard van der Hoff
63ae5cbf34
Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn 
Remove redundant call to preserve_fn
 2018-04-30 10:23:59 +01:00
Richard van der Hoff
bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib 
move httplib import to six
 2018-04-30 01:26:42 +01:00
Richard van der Hoff
683149c1f9
Merge pull request #3151 from NotAFile/py3-xrange-1 
Move more xrange to six
 2018-04-30 01:20:06 +01:00
Richard van der Hoff
3b0e431c82
Merge pull request #3150 from NotAFile/py3-listcomp-yield 
Don't yield in list comprehensions
 2018-04-30 01:11:41 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1 2018-04-30 01:02:25 +01:00
Richard van der Hoff
af3cc50511 Remove redundant call to preserve_fn 
submit_event_for_as doesn't return a deferred anyway, so this is pointless.
 2018-04-30 00:48:36 +01:00
Adrian Tschira
d82b6ea9e6 Move more xrange to six 
plus a bonus next()

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-28 13:57:00 +02:00
Adrian Tschira
94f4d7f49e move httplib import to six 2018-04-28 13:43:34 +02:00
Adrian Tschira
cdb4647a80 Don't yield in list comprehensions 
I've tried to grep for more of this with no success.

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-28 13:36:30 +02:00
Richard van der Hoff
453adf00b6 pep8; remove spurious import 2018-04-27 14:32:08 +01:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn 
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
 2018-04-27 12:55:51 +01:00
Richard van der Hoff
41d4b07a53
Merge pull request #3142 from matrix-org/rav/reraise 
reraise exceptions more carefully
 2018-04-27 12:16:19 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions 
Improve exception handling for background processes
 2018-04-27 11:47:49 +01:00
Richard van der Hoff
6493b22b42 reraise exceptions more carefully 
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
 2018-04-27 11:40:06 +01:00
Richard van der Hoff
605defb9e4 Add missing consumeErrors 
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
 2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17 Improve exception handling for background processes 
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
 2018-04-27 11:07:40 +01:00
Richard van der Hoff
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events 
Reject events which have lots of prev_events
 2018-04-23 17:51:38 +01:00
Richard van der Hoff
9b9c38373c Remove spurious param 2018-04-23 12:00:06 +01:00
Richard van der Hoff
8dc4a6144b
Merge pull request #3107 from NotAFile/py3-bool-nonzero 
add __bool__ alias to __nonzero__ methods
 2018-04-20 15:43:39 +01:00
Richard van der Hoff
3de7d9fe99 accept stupid events over backfill 2018-04-20 11:41:03 +01:00
Richard van der Hoff
11a67b7c9d
Merge pull request #3093 from matrix-org/rav/response_cache_wrap 
Refactor ResponseCache usage
 2018-04-20 11:31:17 +01:00
Richard van der Hoff
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events 
Refactor store.have_events
 2018-04-20 10:26:12 +01:00
Richard van der Hoff
b1dfbc3c40 Refactor store.have_events 
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
 2018-04-20 10:25:56 +01:00
Richard van der Hoff
1f4b498b73 Add some comments 2018-04-18 00:15:36 +01:00
Richard van der Hoff
e585228860 Check events on backfill too 2018-04-18 00:06:42 +01:00
Richard van der Hoff
9b7794262f Reject events which have too many auth_events or prev_events 
... this should protect us from being dossed by people making silly events
(deliberately or otherwise)
 2018-04-18 00:06:42 +01:00
Richard van der Hoff
639480e14a Avoid creating events with huge numbers of prev_events 
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
 2018-04-16 18:41:37 +01:00
Adrian Tschira
f63ff73c7f add __bool__ alias to __nonzero__ methods 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-15 20:40:47 +02:00
Richard van der Hoff
b78395b7fe Refactor ResponseCache usage 
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.

This will be largely non-functional, but does include the following functional
changes:

* federation_server.on_context_state_request: drops use of _server_linearizer
  which looked redundant and could cause incorrect cache misses by yielding
  between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
  on production.
 2018-04-12 13:02:15 +01:00
Richard van der Hoff
d5c74b9f6c
Merge pull request #3092 from matrix-org/rav/response_cache_metrics 
Add metrics for ResponseCache
 2018-04-12 12:59:36 +01:00
Erik Johnston
19ceb4851f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag 2018-04-12 11:36:07 +01:00
Erik Johnston
f67e906e18 Set all metrics at the same time 2018-04-12 11:18:19 +01:00
Erik Johnston
971059a733
Merge pull request #3088 from matrix-org/erikj/as_parallel 
Send events to ASes concurrently
 2018-04-12 10:42:36 +01:00
Erik Johnston
4dae4a97ed Track last processed event received_ts 2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5 Track where event stream processing have gotten up to 2018-04-11 12:13:40 +01:00
Erik Johnston
121591568b Send events to ASes concurrently 2018-04-11 09:56:00 +01:00
Richard van der Hoff
b3384232a0 Add metrics for ResponseCache 2018-04-10 23:14:47 +01:00
Erik Johnston
eaa2ebf20b
Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends 
Limit concurrent event sends for a room
 2018-04-10 16:43:58 +01:00
Erik Johnston
f8e8ec013b Note why we're limiting concurrent event sends 2018-04-10 14:00:46 +01:00
Richard van der Hoff
a3599dda97
Merge pull request #2996 from krombel/allow_auto_join_rooms 
move handling of auto_join_rooms to RegisterHandler
 2018-04-10 01:11:00 +01:00
Richard van der Hoff
d2c6f4d626
Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin 
Return a 404 rather than a 500 on rejoining empty rooms
 2018-04-09 17:32:36 +01:00
Richard van der Hoff
f3ef60662f Return a 404 rather than a 500 on rejoining empty rooms 
Filter ourselves out of the server list before checking for an empty remote
host list, to fix 500 error

Fixes #2141
 2018-04-09 12:56:22 +01:00
Erik Johnston
e5082494eb Limit concurrent event sends for a room 2018-04-09 12:07:39 +01:00
David Baker
b370fe61c0 Implement group join API 2018-04-06 15:43:27 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable 
Add joinability for groups
 2018-04-05 15:57:49 +01:00
Richard van der Hoff
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style 
Replace old style error catching with 'as' keyword
 2018-04-04 10:45:22 +01:00
Luke Barnard
eb8d8d6f57 Use join_policy API instead of joinable 
The API is now under
 /groups/$group_id/setting/m.join_policy

and expects a JSON blob of the shape

```json
{
  "m.join_policy": {
    "type": "invite"
  }
}
```

where "invite" could alternatively be "open".
 2018-04-03 16:16:40 +01:00
Richard van der Hoff
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson 
Use static JSONEncoders
 2018-04-03 15:18:32 +01:00
Richard van der Hoff
05630758f2 Use static JSONEncoders 
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
 2018-03-29 23:13:33 +01:00
Richard van der Hoff
fcfe7f6ad3 Use simplejson throughout 
Let's use simplejson rather than json, for consistency.
 2018-03-29 22:45:52 +01:00
Richard van der Hoff
9a0db062af
Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors 
Fix error when claiming e2e keys from offline servers
 2018-03-28 14:50:38 +01:00
David Baker
79452edeee Add joinability for groups 
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
 2018-03-28 14:03:37 +01:00
Krombel
6152e253d8 Merge branch 'develop' of into allow_auto_join_rooms 2018-03-28 14:45:28 +02:00
Richard van der Hoff
a134c572a6 Stringify exceptions for keys/{query,claim} 
Make sure we stringify any exceptions we return from keys/query and keys/claim,
to avoid a 'not JSON serializable' error later

Fixes #3010
 2018-03-27 17:15:06 +01:00
Richard van der Hoff
c2a5cf2fe3 factor out exception handling for keys/claim and keys/query 
this stuff is badly c&p'ed
 2018-03-27 17:11:23 +01:00
Erik Johnston
3e0c0660b3 Also do check inside linearizer 2018-03-27 13:01:34 +01:00
Erik Johnston
44cd6e1358 PEP8 2018-03-26 12:06:48 +01:00
Erik Johnston
a052aa42e7 Linearize calls to _generate_user_id 2018-03-26 12:02:20 +01:00
Erik Johnston
fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Matthew Hodgson
b2f2282947 make lazy_load_members configurable in filters 2018-03-19 01:15:13 +00:00
Matthew Hodgson
478af0f720 reshuffle todo & comments 2018-03-19 01:00:12 +00:00
Matthew Hodgson
366f730bf6 only get member state IDs for incremental syncs if we're filtering 2018-03-18 21:40:43 +00:00
Matthew Hodgson
fc5397fdf5 remove debug 2018-03-16 01:44:55 +00:00
Matthew Hodgson
4f0493c850 fix tsm search again 2018-03-16 01:43:37 +00:00
Matthew Hodgson
f7dcc404f2 add state_ids for timeline entries 2018-03-16 01:37:53 +00:00
Matthew Hodgson
5b3b3aada8 simplify timeline_start_members 2018-03-16 01:17:34 +00:00
Matthew Hodgson
3bc5bd2d22 make incr syncs work 2018-03-16 00:52:04 +00:00
Erik Johnston
926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00