Richard van der Hoff
33f469ba19
Apply some limits to depth to counter abuse
...
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth
As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
2018-05-01 17:54:19 +01:00
Richard van der Hoff
63ae5cbf34
Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn
...
Remove redundant call to preserve_fn
2018-04-30 10:23:59 +01:00
Richard van der Hoff
bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib
...
move httplib import to six
2018-04-30 01:26:42 +01:00
Richard van der Hoff
683149c1f9
Merge pull request #3151 from NotAFile/py3-xrange-1
...
Move more xrange to six
2018-04-30 01:20:06 +01:00
Richard van der Hoff
3b0e431c82
Merge pull request #3150 from NotAFile/py3-listcomp-yield
...
Don't yield in list comprehensions
2018-04-30 01:11:41 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
Richard van der Hoff
af3cc50511
Remove redundant call to preserve_fn
...
submit_event_for_as doesn't return a deferred anyway, so this is pointless.
2018-04-30 00:48:36 +01:00
Adrian Tschira
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Adrian Tschira
94f4d7f49e
move httplib import to six
2018-04-28 13:43:34 +02:00
Adrian Tschira
cdb4647a80
Don't yield in list comprehensions
...
I've tried to grep for more of this with no success.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:36:30 +02:00
Richard van der Hoff
453adf00b6
pep8; remove spurious import
2018-04-27 14:32:08 +01:00
Richard van der Hoff
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff
41d4b07a53
Merge pull request #3142 from matrix-org/rav/reraise
...
reraise exceptions more carefully
2018-04-27 12:16:19 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
Richard van der Hoff
6493b22b42
reraise exceptions more carefully
...
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
2018-04-27 11:40:06 +01:00
Richard van der Hoff
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Richard van der Hoff
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events
...
Reject events which have lots of prev_events
2018-04-23 17:51:38 +01:00
Richard van der Hoff
9b9c38373c
Remove spurious param
2018-04-23 12:00:06 +01:00
Richard van der Hoff
8dc4a6144b
Merge pull request #3107 from NotAFile/py3-bool-nonzero
...
add __bool__ alias to __nonzero__ methods
2018-04-20 15:43:39 +01:00
Richard van der Hoff
3de7d9fe99
accept stupid events over backfill
2018-04-20 11:41:03 +01:00
Richard van der Hoff
11a67b7c9d
Merge pull request #3093 from matrix-org/rav/response_cache_wrap
...
Refactor ResponseCache usage
2018-04-20 11:31:17 +01:00
Richard van der Hoff
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
Richard van der Hoff
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
Richard van der Hoff
1f4b498b73
Add some comments
2018-04-18 00:15:36 +01:00
Richard van der Hoff
e585228860
Check events on backfill too
2018-04-18 00:06:42 +01:00
Richard van der Hoff
9b7794262f
Reject events which have too many auth_events or prev_events
...
... this should protect us from being dossed by people making silly events
(deliberately or otherwise)
2018-04-18 00:06:42 +01:00
Richard van der Hoff
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira
f63ff73c7f
add __bool__ alias to __nonzero__ methods
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:40:47 +02:00
Richard van der Hoff
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
Richard van der Hoff
d5c74b9f6c
Merge pull request #3092 from matrix-org/rav/response_cache_metrics
...
Add metrics for ResponseCache
2018-04-12 12:59:36 +01:00
Erik Johnston
19ceb4851f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag
2018-04-12 11:36:07 +01:00
Erik Johnston
f67e906e18
Set all metrics at the same time
2018-04-12 11:18:19 +01:00
Erik Johnston
971059a733
Merge pull request #3088 from matrix-org/erikj/as_parallel
...
Send events to ASes concurrently
2018-04-12 10:42:36 +01:00
Erik Johnston
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
Erik Johnston
121591568b
Send events to ASes concurrently
2018-04-11 09:56:00 +01:00
Richard van der Hoff
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
Erik Johnston
eaa2ebf20b
Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends
...
Limit concurrent event sends for a room
2018-04-10 16:43:58 +01:00
Erik Johnston
f8e8ec013b
Note why we're limiting concurrent event sends
2018-04-10 14:00:46 +01:00
Richard van der Hoff
a3599dda97
Merge pull request #2996 from krombel/allow_auto_join_rooms
...
move handling of auto_join_rooms to RegisterHandler
2018-04-10 01:11:00 +01:00
Richard van der Hoff
d2c6f4d626
Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin
...
Return a 404 rather than a 500 on rejoining empty rooms
2018-04-09 17:32:36 +01:00
Richard van der Hoff
f3ef60662f
Return a 404 rather than a 500 on rejoining empty rooms
...
Filter ourselves out of the server list before checking for an empty remote
host list, to fix 500 error
Fixes #2141
2018-04-09 12:56:22 +01:00
Erik Johnston
e5082494eb
Limit concurrent event sends for a room
2018-04-09 12:07:39 +01:00
David Baker
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
Richard van der Hoff
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style
...
Replace old style error catching with 'as' keyword
2018-04-04 10:45:22 +01:00
Luke Barnard
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
Richard van der Hoff
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
Richard van der Hoff
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
Richard van der Hoff
9a0db062af
Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors
...
Fix error when claiming e2e keys from offline servers
2018-03-28 14:50:38 +01:00
David Baker
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
Krombel
6152e253d8
Merge branch 'develop' of into allow_auto_join_rooms
2018-03-28 14:45:28 +02:00
Richard van der Hoff
a134c572a6
Stringify exceptions for keys/{query,claim}
...
Make sure we stringify any exceptions we return from keys/query and keys/claim,
to avoid a 'not JSON serializable' error later
Fixes #3010
2018-03-27 17:15:06 +01:00
Richard van der Hoff
c2a5cf2fe3
factor out exception handling for keys/claim and keys/query
...
this stuff is badly c&p'ed
2018-03-27 17:11:23 +01:00
Erik Johnston
3e0c0660b3
Also do check inside linearizer
2018-03-27 13:01:34 +01:00
Erik Johnston
44cd6e1358
PEP8
2018-03-26 12:06:48 +01:00
Erik Johnston
a052aa42e7
Linearize calls to _generate_user_id
2018-03-26 12:02:20 +01:00
Erik Johnston
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
Erik Johnston
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
NotAFile
2cc9f76bc3
replace old style error catching with 'as' keyword
...
This is both easier to read and compatible with python3 (not that that
matters)
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-15 16:11:17 +01:00
Krombel
91ea0202e6
move handling of auto_join_rooms to RegisterHandler
...
Currently the handling of auto_join_rooms only works when a user
registers itself via public register api. Registrations via
registration_shared_secret and ModuleApi do not work
This auto_joins the users in the registration handler which enables
the auto join feature for all 3 registration paths.
This is related to issue #2725
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-14 16:45:37 +01:00
Erik Johnston
57db62e554
Merge pull request #2992 from matrix-org/erikj/implement_member_workre
...
Implement RoomMemberWorkerHandler
2018-03-14 14:29:33 +00:00
Erik Johnston
62ad701326
s/join/joined/ in notify_user_membership_change
2018-03-14 14:17:43 +00:00
Erik Johnston
3f0f06cb31
Split RoomMemberWorkerHandler to separate file
2018-03-14 11:41:45 +00:00
Erik Johnston
3e839e0548
Merge pull request #2989 from matrix-org/erikj/profile_cache_master
...
Only update remote profile cache on master
2018-03-14 09:42:27 +00:00
Erik Johnston
b27320b550
Implement RoomMemberWorkerHandler
2018-03-13 18:26:00 +00:00
Erik Johnston
350331d466
_remote_join and co take a requester
2018-03-13 17:50:39 +00:00
Erik Johnston
1a69c6d590
Merge pull request #2987 from matrix-org/erikj/split_room_member_handler
...
Split RoomMemberHandler into base and master class
2018-03-13 17:40:00 +00:00
Erik Johnston
df8ff682a7
Only update remote profile cache on master
2018-03-13 17:38:21 +00:00
Erik Johnston
6dbebef141
Add missing param to docstrings
2018-03-13 17:15:32 +00:00
Erik Johnston
16adb11cc0
Correct import order
2018-03-13 16:57:07 +00:00
Erik Johnston
82f16faa78
Move user_*_room distributor stuff to master class
...
I added yields when calling user_left_room, but they shouldn't matter on
the master process as they always return None anyway.
2018-03-13 16:38:15 +00:00
Erik Johnston
b78717b87b
Split RoomMemberHandler into base and master class
...
The intention here is to split the class into the bits that can be done
on workers and the bits that have to be done on the master.
In future there will also be a class that can be run on the worker,
which will delegate work to the master when necessary.
2018-03-13 16:37:41 +00:00
Erik Johnston
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer
...
Remove ReplicationLayer and user Client/Server directly
2018-03-13 15:45:08 +00:00
Erik Johnston
5d8476d8ff
Merge pull request #2981 from matrix-org/erikj/factor_remote_leave
...
Factor out _remote_reject_invite in RoomMember
2018-03-13 15:44:56 +00:00
Erik Johnston
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers
...
Don't build handlers on workers unnecessarily
2018-03-13 13:46:38 +00:00
Erik Johnston
cb9f8e527c
s/replication_client/federation_client/
2018-03-13 13:26:52 +00:00
Erik Johnston
bf8e97bd3c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave
2018-03-13 13:17:08 +00:00
Erik Johnston
ea3442c15c
Add docstring
2018-03-13 13:16:21 +00:00
Erik Johnston
16469a4f15
Merge pull request #2980 from matrix-org/erikj/rm_priv
...
Make RoomMemberHandler functions private that can be
2018-03-13 13:11:11 +00:00
Erik Johnston
c82111a55f
Merge pull request #2982 from matrix-org/erikj/fix_extra_users
...
extra_users is actually a list of UserIDs
2018-03-13 13:11:04 +00:00
Erik Johnston
da87791975
Merge pull request #2983 from matrix-org/erikj/rename_register_3pid
...
Refactor get_or_register_3pid_guest
2018-03-13 13:10:52 +00:00
Erik Johnston
8b3573a8b2
Refactor get_or_register_3pid_guest
2018-03-13 12:08:58 +00:00
Erik Johnston
0f942f68c1
Factor out _remote_reject_invite in RoomMember
2018-03-13 11:22:45 +00:00
Erik Johnston
d0fcc48f9d
extra_users is actually a list of UserIDs
2018-03-13 11:20:06 +00:00
Erik Johnston
31becf4ac3
Make functions private that can be
2018-03-13 11:15:16 +00:00
Erik Johnston
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
Erik Johnston
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
Richard van der Hoff
e48c7aac4d
Add transactional API to history purge
...
Make the purge request return quickly, and allow scripts to poll for updates.
2018-03-12 16:22:55 +00:00
Richard van der Hoff
1708412f56
Return an error when doing two purges on a room
...
Queuing up purges doesn't sound like a good thing.
2018-03-12 16:22:54 +00:00
Erik Johnston
a56d54dcb7
Fix up log message
2018-03-07 11:55:31 +00:00
Erik Johnston
02a1296ad6
Fix typo
2018-03-07 11:55:31 +00:00
Erik Johnston
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
Richard van der Hoff
8ffaacbee3
Merge pull request #2949 from krombel/use_bcrypt_checkpw
...
use bcrypt.checkpw
2018-03-06 11:56:06 +00:00
Richard van der Hoff
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
Krombel
ed9b5eced4
use bcrypt.checkpw
...
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago)
This makes use of that with enhancements which might get introduced
by that
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-05 18:02:59 +01:00
Erik Johnston
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00