Erik Johnston
1aeb5e28a9
Merge pull request #3193 from matrix-org/erikj/pagination_refactor
...
Refactor /context to reuse pagination storage functions
2018-05-09 10:22:38 +01:00
Erik Johnston
23ec51c94c
Fix up comments and make function private
2018-05-09 09:55:19 +01:00
Erik Johnston
696f532453
Reuse existing pagination code for context API
2018-05-08 16:20:19 +01:00
Erik Johnston
3e6d306e94
Parse tokens before calling DB function
2018-05-08 16:18:58 +01:00
Erik Johnston
274b8c6025
Only fetch required fields from database
2018-05-08 16:15:25 +01:00
Erik Johnston
06c0d0ed08
Split paginate_room_events storage function
2018-05-08 16:14:26 +01:00
David Baker
bf98fa0864
Part user from rooms on account deactivate
...
This implements this very crudely: this probably isn't viable
because parting a user from all their rooms could take a long time,
and if the HS gets restarted in that time the process will be
aborted.
2018-05-08 15:58:35 +01:00
Richard van der Hoff
678e649b78
Merge pull request #3190 from mujx/notif-token-fix
...
notifications: Convert next_token to string according to the spec
2018-05-08 13:54:47 +01:00
Konstantinos Sideris
88868b2839
notifications: Convert next_token to string according to the spec
...
Currently the parameter is serialized as an integer.
Signed-off-by: Konstantinos Sideris <sideris.konstantin@gmail.com>
2018-05-05 12:55:02 +03:00
Erik Johnston
6d8ec3462d
Note that label values can be anything
2018-05-03 16:25:05 +01:00
Erik Johnston
95b6912045
Fix metrics that have integer value labels
2018-05-03 15:51:04 +01:00
Richard van der Hoff
093d8c415a
Merge remote-tracking branch 'origin/develop' into rav/warn_on_logcontext_fail
2018-05-03 14:59:29 +01:00
Richard van der Hoff
0ba609dc6f
Merge pull request #3183 from matrix-org/rav/moar_logcontext_leaks
...
Fix logcontext leaks in rate limiter
2018-05-03 14:58:13 +01:00
Richard van der Hoff
2117f84323
Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdown
...
Fix 'Unhandled Error' logs with Twisted 18.4
2018-05-03 12:40:11 +01:00
Richard van der Hoff
a7fe62f0cb
Fix logcontext leaks in rate limiter
2018-05-03 12:31:59 +01:00
Will Hunt
2e7a94c36b
Don't abortConnection() if the transport connection has already closed.
2018-05-03 12:31:47 +01:00
Richard van der Hoff
a2aaa9cb3c
Merge pull request #3178 from matrix-org/rav/fix_request_timeouts
...
fix http request timeout code
2018-05-03 11:33:26 +01:00
Erik Johnston
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
...
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
Erik Johnston
31c7c29d43
Fix up grammar
2018-05-03 10:38:58 +01:00
Richard van der Hoff
902673e356
Merge pull request #3161 from NotAFile/remove-v1auth
...
Make Client-Server API return 403 for invalid token
2018-05-03 10:10:57 +01:00
Erik Johnston
53a5fdf312
Merge pull request #3175 from matrix-org/erikj/escape_metric_values
...
Escape label values in prometheus metrics
2018-05-03 10:01:04 +01:00
Richard van der Hoff
1dfd650348
add missing param to cancelled_to_request_timed_out_error
...
This gets two arguments, not one.
2018-05-02 22:42:36 +01:00
Erik Johnston
a41117c63b
Make _escape_character take MatchObject
2018-05-02 17:27:27 +01:00
Erik Johnston
32015e1109
Escape label values in prometheus metrics
2018-05-02 16:52:42 +01:00
Richard van der Hoff
3a42aed9a1
Merge pull request #3170 from matrix-org/rav/more_logcontext_leaks
...
Fix a class of logcontext leaks
2018-05-02 16:45:51 +01:00
Richard van der Hoff
415c6b672e
Merge branch 'develop' into rav/more_logcontext_leaks
2018-05-02 16:16:01 +01:00
Richard van der Hoff
be31adb036
Fix logcontext leak in media repo
...
Make FileResponder.write_to_consumer uphold the logcontext contract
2018-05-02 16:14:50 +01:00
Richard van der Hoff
f22e7cda2c
Fix a class of logcontext leaks
...
So, it turns out that if you have a first `Deferred` `D1`, you can add a
callback which returns another `Deferred` `D2`, and `D2` must then complete
before any further callbacks on `D1` will execute (and later callbacks on `D1`
get the *result* of `D2` rather than `D2` itself).
So, `D1` might have `called=True` (as in, it has started running its
callbacks), but any new callbacks added to `D1` won't get run until `D2`
completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield`
will 'block'.
In conclusion: some of our assumptions in `logcontext` were invalid. We need to
make sure that we don't optimise out the logcontext juggling when this
situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
2018-05-02 11:58:00 +01:00
Richard van der Hoff
a8d8bf92e0
Merge pull request #3168 from matrix-org/rav/fix_logformatter
...
Fix incorrect reference to StringIO
2018-05-02 10:03:36 +01:00
Richard van der Hoff
e482f8cd85
Fix incorrect reference to StringIO
...
This was introduced in 4f2f5171
2018-05-02 09:12:26 +01:00
Neil Johnson
2414178ed6
Merge branch 'master' into develop
2018-05-01 18:53:56 +01:00
Neil Johnson
8570bb84cc
Update __init__.py
...
bump version
2018-05-01 18:22:53 +01:00
Richard van der Hoff
ca7211104e
Merge branch 'release-v0.28.1' into develop
2018-05-01 18:16:57 +01:00
Richard van der Hoff
d5eee5d601
Merge commit '33f469b' into release-v0.28.1
2018-05-01 18:14:18 +01:00
Richard van der Hoff
d858f3bd4e
Miscellaneous fixes to python_dependencies
...
* add some doc about wtf this thing does
* pin Twisted to < 18.4
* add explicit dep on six (fixes #3089 )
2018-05-01 18:13:54 +01:00
Richard van der Hoff
33f469ba19
Apply some limits to depth to counter abuse
...
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth
As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
2018-05-01 17:54:19 +01:00
Neil Johnson
dd1a832419
remove user agent from data model, will just join on user_ips
2018-05-01 12:13:49 +01:00
Neil Johnson
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
Neil Johnson
5917562b60
10 mins seems more reasonable that every minute
2018-05-01 12:12:22 +01:00
Adrian Tschira
6495dbb326
Burminate v1auth
...
This closes #2602
v1auth was created to account for the differences in status code between
the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid
tokens). However since those protocols were merged, this makes the r0
version/endpoint internally inconsistent, and violates the
specification for the r0 endpoint.
This might break clients that rely on this inconsistency with the
specification. This is said to affect the legacy angular reference
client. However, I feel that restoring parity with the spec is more
important. Either way, it is critical to inform developers about this
change, in case they rely on the illegal behaviour.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-30 22:20:43 +02:00
Will Hunt
2ad3fc36e6
Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve ( #3157 )
...
fixes #3135
Signed-off-by: Will Hunt will@half-shot.uk
2018-04-30 16:21:11 +01:00
Krombel
576b71dd3d
add guard for None on purge_history api
2018-04-30 14:29:48 +02:00
Matthew Hodgson
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Richard van der Hoff
63ae5cbf34
Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn
...
Remove redundant call to preserve_fn
2018-04-30 10:23:59 +01:00
Richard van der Hoff
fdb6849b81
Merge pull request #3144 from matrix-org/rav/run_in_background_exception_handling
...
Trap exceptions thrown within run_in_background
2018-04-30 10:23:02 +01:00
Richard van der Hoff
01e8a52825
Merge pull request #3102 from NotAFile/py3-attributeerror
...
Make event properties raise AttributeError instead
2018-04-30 09:22:09 +01:00
Adrian Tschira
0c9db26260
add comment explaining attributeerror
2018-04-30 09:49:10 +02:00
Richard van der Hoff
950a32eb47
Merge pull request #3152 from NotAFile/py3-local-imports
...
make imports local
2018-04-30 01:28:13 +01:00
Richard van der Hoff
bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib
...
move httplib import to six
2018-04-30 01:26:42 +01:00
Richard van der Hoff
683149c1f9
Merge pull request #3151 from NotAFile/py3-xrange-1
...
Move more xrange to six
2018-04-30 01:20:06 +01:00
Richard van der Hoff
3b0e431c82
Merge pull request #3150 from NotAFile/py3-listcomp-yield
...
Don't yield in list comprehensions
2018-04-30 01:11:41 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
Richard van der Hoff
2fd96727b1
Merge pull request #3085 from NotAFile/py3-config-text-mode
...
Open config file in non-bytes mode
2018-04-30 01:00:23 +01:00
Richard van der Hoff
b8ee12b978
Merge pull request #3084 from NotAFile/py3-certs-byte-mode
...
Open certificate files as bytes
2018-04-30 01:00:05 +01:00
Richard van der Hoff
049b0b5af2
Merge pull request #3154 from NotAFile/py3-stringio
...
Replace stringIO imports with six
2018-04-30 00:59:04 +01:00
Richard van der Hoff
af3cc50511
Remove redundant call to preserve_fn
...
submit_event_for_as doesn't return a deferred anyway, so this is pointless.
2018-04-30 00:48:36 +01:00
Richard van der Hoff
dbf6f28d64
Merge pull request #3155 from NotAFile/py3-bytes-1
...
more bytes strings
2018-04-30 00:38:21 +01:00
Richard van der Hoff
aab2e4da60
Merge pull request #3140 from matrix-org/rav/use_run_in_background
...
Use run_in_background in preference to preserve_fn
2018-04-30 00:34:28 +01:00
Richard van der Hoff
1315d374cc
Merge pull request #3156 from NotAFile/py3-hmac-bytes
...
Construct HMAC as bytes on py3
2018-04-30 00:33:20 +01:00
Richard van der Hoff
9e2601f830
Merge pull request #3108 from NotAFile/py3-six-urlparse
...
Use six.moves.urlparse
2018-04-30 00:33:05 +01:00
Adrian Tschira
122593265b
Construct HMAC as bytes on py3
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:19:41 +02:00
Adrian Tschira
e9143b6593
more bytes strings
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:13:57 +02:00
Matthew Hodgson
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
Matthew Hodgson
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
Matthew Hodgson
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
Adrian Tschira
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Adrian Tschira
4f2f5171b7
replace stringIO imports
2018-04-28 13:46:23 +02:00
Adrian Tschira
94f4d7f49e
move httplib import to six
2018-04-28 13:43:34 +02:00
Adrian Tschira
57b58e2174
make imports local
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:41:41 +02:00
Adrian Tschira
cdb4647a80
Don't yield in list comprehensions
...
I've tried to grep for more of this with no success.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:36:30 +02:00
Adrian Tschira
a376d8f761
open log_config in text mode too
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:34:13 +02:00
Richard van der Hoff
453adf00b6
pep8; remove spurious import
2018-04-27 14:32:08 +01:00
Richard van der Hoff
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
Richard van der Hoff
6146332387
Merge remote-tracking branch 'origin/develop' into rav/deferred_timeout
2018-04-27 14:18:00 +01:00
Richard van der Hoff
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff
3d1ae61399
Merge branch 'develop' into rav/deferred_timeout
2018-04-27 12:54:43 +01:00
Richard van der Hoff
9d2c1b8429
Backport deferred.addTimeout
...
Twisted 16.0 doesn't have addTimeout, so let's backport it.
2018-04-27 12:52:30 +01:00
Richard van der Hoff
13843f771e
Trap exceptions thrown within run_in_background
...
Turn any exceptions that get thrown synchronously within run_in_background into
Failures instead.
2018-04-27 12:17:13 +01:00
Richard van der Hoff
41d4b07a53
Merge pull request #3142 from matrix-org/rav/reraise
...
reraise exceptions more carefully
2018-04-27 12:16:19 +01:00
Richard van der Hoff
268e40341b
Merge pull request #3136 from matrix-org/rav/fix_dependencies
...
Miscellaneous fixes to python_dependencies
2018-04-27 11:48:28 +01:00
Richard van der Hoff
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
Richard van der Hoff
53494c34df
Merge pull request #3139 from matrix-org/rav/consume_errors
...
Add missing consumeErrors to improve exception handling
2018-04-27 11:47:21 +01:00
Richard van der Hoff
6493b22b42
reraise exceptions more carefully
...
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
2018-04-27 11:40:06 +01:00
Erik Johnston
6e10eed28e
Refactor event storage to not require state
...
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.
We also add a check to see if the state groups of the old extremities
are the same as the new ones.
2018-04-27 11:38:02 +01:00
Richard van der Hoff
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
Richard van der Hoff
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Neil Johnson
28dd536e80
update changelog and bump version to 0.28.0
2018-04-26 15:51:39 +01:00
Richard van der Hoff
d78ada3166
Miscellaneous fixes to python_dependencies
...
* add some doc about wtf this thing does
* pin Twisted to < 18.4
* add explicit dep on six (fixes #3089 )
2018-04-26 13:11:03 +01:00
Erik Johnston
7ec8e798b4
Fix media admin APIs
2018-04-26 11:31:22 +01:00
Neil Johnson
fb6015d0a6
pep8
2018-04-25 17:56:11 +01:00
Neil Johnson
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
Erik Johnston
22881b3d69
Also fix reindexing of search
2018-04-25 15:32:04 +01:00
Erik Johnston
ba3166743c
Fix quarantine media admin API
2018-04-25 15:11:18 +01:00
Matthew Hodgson
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
Neil Johnson
2c3e995f38
Bump version and update changelog
2018-04-24 15:33:22 +01:00
Richard van der Hoff
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events
...
Reject events which have lots of prev_events
2018-04-23 17:51:38 +01:00
Richard van der Hoff
9b9c38373c
Remove spurious param
2018-04-23 12:00:06 +01:00
Richard van der Hoff
1ea904b9f0
Use deferred.addTimeout instead of time_bound_deferred
...
This doesn't feel like a wheel we need to reinvent.
2018-04-23 00:53:18 +01:00
Richard van der Hoff
dc875d2712
Merge pull request #3106 from NotAFile/py3-six-itervalues-1
...
Use six.itervalues in some places
2018-04-20 15:43:52 +01:00
Richard van der Hoff
8dc4a6144b
Merge pull request #3107 from NotAFile/py3-bool-nonzero
...
add __bool__ alias to __nonzero__ methods
2018-04-20 15:43:39 +01:00
Richard van der Hoff
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
...
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
Richard van der Hoff
692a3cc806
Merge pull request #3103 from NotAFile/py3-baseexcepton-message
...
Use str(e) instead of e.message
2018-04-20 15:34:49 +01:00
Richard van der Hoff
3de7d9fe99
accept stupid events over backfill
2018-04-20 11:41:03 +01:00
Richard van der Hoff
11a67b7c9d
Merge pull request #3093 from matrix-org/rav/response_cache_wrap
...
Refactor ResponseCache usage
2018-04-20 11:31:17 +01:00
Richard van der Hoff
0c280d4d99
Reinstate linearizer for federation_server.on_context_state_request
2018-04-20 11:10:04 +01:00
Richard van der Hoff
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
Richard van der Hoff
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
Richard van der Hoff
1f4b498b73
Add some comments
2018-04-18 00:15:36 +01:00
Richard van der Hoff
e585228860
Check events on backfill too
2018-04-18 00:06:42 +01:00
Richard van der Hoff
9b7794262f
Reject events which have too many auth_events or prev_events
...
... this should protect us from being dossed by people making silly events
(deliberately or otherwise)
2018-04-18 00:06:42 +01:00
Richard van der Hoff
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira
878995e660
Replace Queue with six.moves.queue
...
and a six.range change which I missed the last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
Adrian Tschira
2a3c33ff03
Use six.moves.urlparse
...
The imports were shuffled around a bunch in py3
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 21:22:43 +02:00
Adrian Tschira
f63ff73c7f
add __bool__ alias to __nonzero__ methods
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:40:47 +02:00
Adrian Tschira
36c59ce669
Use six.itervalues in some places
...
There's more where that came from
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
Adrian Tschira
1515560f5c
Use str(e) instead of e.message
...
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:32:42 +02:00
Adrian Tschira
bfc2ade9b3
Make event properties raise AttributeError instead
...
They raised KeyError before. I'm changing this because the code uses
hasattr() to check for the presence of a key. This worked accidentally
before, because hasattr() silences all exceptions in python 2. However,
in python3, this isn't the case anymore.
I had a look around to see if anything depended on this raising a
KeyError and I couldn't find anything. Of course, I could have simply
missed it.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:16:59 +02:00
Neil Johnson
154b44c249
Merge branch 'master' of https://github.com/matrix-org/synapse into develop
2018-04-13 17:07:54 +01:00
Matthew Hodgson
78a9698650
fix federation_domain_whitelist
...
we were checking the wrong server_name on inbound requests
2018-04-13 15:47:43 +01:00
Matthew Hodgson
25b0ba30b1
revert last to PR properly
2018-04-13 15:46:37 +01:00
Matthew Hodgson
f8d46cad3c
correctly auth inbound federation_domain_whitelist reqs
2018-04-13 15:41:52 +01:00
Neil Johnson
d4b2e05852
Merge branch 'master' of https://github.com/matrix-org/synapse into develop
2018-04-13 12:16:27 +01:00
Neil Johnson
51d628d28d
Bump version and Change log
2018-04-13 12:08:19 +01:00
Richard van der Hoff
d3347ad485
Revert "Use sortedcontainers instead of blist"
...
This reverts commit 9fbe70a7dc
.
It turns out that sortedcontainers.SortedDict is not an exact match for
blist.sorteddict; in particular, `popitem()` removes things from the opposite
end of the dict.
This is trivial to fix, but I want to add some unit tests, and potentially some
more thought about it, before we do so.
2018-04-13 11:16:43 +01:00
Richard van der Hoff
fac3f9e678
Bump canonicaljson to 1.1.3
...
1.1.2 was a bit broken too :/
2018-04-13 10:21:38 +01:00
Richard van der Hoff
60f6014bb7
ResponseCache: fix handling of completed results
...
Turns out that ObservableDeferred.observe doesn't return a deferred if the
result is already completed. Fix handling and improve documentation.
2018-04-13 07:32:29 +01:00
Richard van der Hoff
119596ab8f
Update canonicaljson dependency
...
1.1.0 and 1.1.1 were broken, so we're updating this to help people make sure
they don't end up on a broken version.
Also, 1.1.0 is speedier...
2018-04-12 17:31:44 +01:00
Richard van der Hoff
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
Richard van der Hoff
d5c74b9f6c
Merge pull request #3092 from matrix-org/rav/response_cache_metrics
...
Add metrics for ResponseCache
2018-04-12 12:59:36 +01:00
Erik Johnston
415aeefd89
Format docstring
2018-04-12 12:07:09 +01:00
Erik Johnston
19ceb4851f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag
2018-04-12 11:36:07 +01:00
Richard van der Hoff
261124396e
Merge pull request #3059 from matrix-org/rav/doc_response_cache
...
Document the behaviour of ResponseCache
2018-04-12 11:22:30 +01:00
Erik Johnston
23a7f9d7f4
Doc we raise on unknown event
2018-04-12 11:20:51 +01:00
Erik Johnston
d7bf3a68f0
s/list/tuple
2018-04-12 11:19:04 +01:00
Erik Johnston
f67e906e18
Set all metrics at the same time
2018-04-12 11:18:19 +01:00
Erik Johnston
971059a733
Merge pull request #3088 from matrix-org/erikj/as_parallel
...
Send events to ASes concurrently
2018-04-12 10:42:36 +01:00
Erik Johnston
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
Erik Johnston
ab825aa328
Add GaugeMetric
2018-04-11 12:13:40 +01:00
Richard van der Hoff
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
Neil Johnson
781cd8c54f
bump version/changelog
2018-04-11 10:54:43 +01:00
Neil Johnson
9ef0b179e0
Merge commit '11d2609da70af797405241cdf7d9df19db5628f2' of https://github.com/matrix-org/synapse into release-v0.27.0
2018-04-11 10:51:59 +01:00
Erik Johnston
121591568b
Send events to ASes concurrently
2018-04-11 09:56:00 +01:00
Richard van der Hoff
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
Neil Johnson
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
Erik Johnston
eaa2ebf20b
Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends
...
Limit concurrent event sends for a room
2018-04-10 16:43:58 +01:00
Erik Johnston
9daf82278f
Merge pull request #3078 from matrix-org/erikj/federation_sender
...
Send federation events concurrently
2018-04-10 16:43:48 +01:00
Adrian Tschira
a3f9ddbede
Open certificate files as bytes
...
That's what pyOpenSSL expects on python3
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-10 17:36:29 +02:00
Adrian Tschira
7f8eebc8ee
Open config file in non-bytes mode
...
Nothing written into it is encoded, so it makes little sense, but it
does break in python3 the way it was before.
The variable names were adjusted to be less misleading.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-10 17:32:40 +02:00
Neil Johnson
dd723267b2
Merge branch 'release-v0.27.0' of https://github.com/matrix-org/synapse into develop
2018-04-10 15:03:29 +01:00
Erik Johnston
a060dfa132
Use run_in_background instead
2018-04-10 14:25:11 +01:00
Erik Johnston
f8e8ec013b
Note why we're limiting concurrent event sends
2018-04-10 14:00:46 +01:00
Erik Johnston
1246d23710
Preserve log contexts correctly
2018-04-10 12:04:32 +01:00
Erik Johnston
d49cbf712f
Log event ID on exception
2018-04-10 12:03:41 +01:00
Erik Johnston
ce72d590ed
Merge pull request #3082 from matrix-org/erikj/urlencode_paths
...
URL quote path segments over federation
2018-04-10 11:31:16 +01:00
Erik Johnston
11d2609da7
Ensure slashes are escaped
2018-04-10 11:24:40 +01:00
Erik Johnston
dab87b84a3
URL quote path segments over federation
2018-04-10 11:16:08 +01:00
Vincent Breitmoser
6d7f0f8dd3
Don't disable GC when running on PyPy
...
PyPy's incminimark GC can't be triggered manually. From what I observed
there are no obvious issues with just letting it run normally. And
unlike CPython, it actually returns unused RAM to the system.
Signed-off-by: Vincent Breitmoser <look@my.amazin.horse>
2018-04-10 11:35:34 +02:00
Vincent Breitmoser
f4284d943a
In DomainSpecificString, override __repr__ in addition to __str__
...
For some reason, string interpolation on a DomainSpecificString object
like "%r" % (domainSpecificStringObj) fails under PyPy, because the
default __repr__ implementation wants to iterate over the object. I'm
not sure why that happens, but overriding __repr__ instead of __str__
fixes this problem, and is arguably the more appropriate thing to do
anyways.
2018-04-10 11:35:29 +02:00
Richard van der Hoff
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
Vincent Breitmoser
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
Vincent Breitmoser
9fbe70a7dc
Use sortedcontainers instead of blist
...
This commit drop-in replaces blist with SortedContainers. They are
written in pure python so work with pypy, but perform as good as
native implementations, at least in a couple benchmarks:
http://www.grantjenks.com/docs/sortedcontainers/performance.html
2018-04-10 11:29:51 +02:00
Richard van der Hoff
a3599dda97
Merge pull request #2996 from krombel/allow_auto_join_rooms
...
move handling of auto_join_rooms to RegisterHandler
2018-04-10 01:11:00 +01:00
Richard van der Hoff
87478c5a60
Merge pull request #3061 from NotAFile/add-some-byte-strings
...
Add b prefixes to some strings that are bytes in py3
2018-04-09 23:54:05 +01:00
Richard van der Hoff
c508b2f2f0
Merge pull request #3073 from NotAFile/use-six-reraise
...
Replace old-style raise with six.reraise
2018-04-09 23:53:40 +01:00
Richard van der Hoff
37354b55c9
Merge pull request #2938 from dklug/develop
...
Return 401 for invalid access_token on logout
2018-04-09 23:52:56 +01:00
Richard van der Hoff
0e9aa1d091
Merge pull request #3074 from NotAFile/fix-py3-prints
...
use python3-compatible prints
2018-04-09 23:44:41 +01:00
Richard van der Hoff
8eaa141d8f
Merge pull request #3075 from NotAFile/six-type-checks
...
Replace some type checks with six type checks
2018-04-09 23:40:44 +01:00
Richard van der Hoff
664adb4236
Merge pull request #3016 from silkeh/improve-service-lookups
...
Improve handling of SRV records for federation connections
2018-04-09 23:40:06 +01:00
Neil Johnson
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
Neil Johnson
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
Neil Johnson
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
Neil Johnson
64bc2162ef
Fix psycopg2 interpolation
2018-04-09 17:50:36 +01:00
Richard van der Hoff
d2c6f4d626
Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin
...
Return a 404 rather than a 500 on rejoining empty rooms
2018-04-09 17:32:36 +01:00
Neil Johnson
5232d3bfb1
version bump v0.27.3-rc2
2018-04-09 17:25:57 +01:00
Neil Johnson
5e785d4d5b
Merge branch 'develop' of https://github.com/matrix-org/synapse into release-v0.27.0
2018-04-09 17:21:34 +01:00
Erik Johnston
6e025a97b4
Handle all events in a room correctly
2018-04-09 16:02:48 +01:00
Neil Johnson
d9ae2bc826
bump version to release candidate
2018-04-09 16:00:31 +01:00
Neil Johnson
072fb59446
bump version
2018-04-09 13:49:25 +01:00
Richard van der Hoff
13decdbf96
Revert "Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics"
...
We aren't ready to release this yet, so I'm reverting it for now.
This reverts commit d1679a4ed7
, reversing
changes made to e089100c62
.
2018-04-09 12:59:12 +01:00
Richard van der Hoff
f3ef60662f
Return a 404 rather than a 500 on rejoining empty rooms
...
Filter ourselves out of the server list before checking for an empty remote
host list, to fix 500 error
Fixes #2141
2018-04-09 12:56:22 +01:00
Erik Johnston
e5082494eb
Limit concurrent event sends for a room
2018-04-09 12:07:39 +01:00
Erik Johnston
56b0589865
Use create_and_send_nonmember_event everywhere
2018-04-09 12:04:18 +01:00
Erik Johnston
11974f3787
Send federation events concurrently
2018-04-09 11:47:10 +01:00
Erik Johnston
145d14656b
Handle exceptions in get_hosts_for_room when sending events over federation
2018-04-09 11:47:01 +01:00
Adrian Tschira
e54c202b81
Replace some type checks with six type checks
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-07 01:02:32 +02:00
Adrian Tschira
b0500d3774
use python3-compatible prints
2018-04-06 23:35:27 +02:00
Adrian Tschira
4f40d058cc
Replace old-style raise with six.reraise
...
The old style raise is invalid syntax in python3. As noted in the docs,
this adds one more frame in the traceback, but I think this is
acceptable:
<ipython-input-7-bcc5cba3de3f> in <module>()
16 except:
17 pass
---> 18 six.reraise(*x)
/usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
691 if value.__traceback__ is not tb:
692 raise value.with_traceback(tb)
--> 693 raise value
694 finally:
695 value = None
<ipython-input-7-bcc5cba3de3f> in <module>()
9
10 try:
---> 11 x()
12 except:
13 x = sys.exc_info()
Also note that this uses six, which is not formally a dependency yet,
but is included indirectly since most packages depend on it.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-06 23:06:24 +02:00
Luke Barnard
135fc5b9cd
Merge pull request #3046 from matrix-org/dbkr/join_group
...
Implement group join API
2018-04-06 16:24:32 +01:00
Luke Barnard
020a501354
de-lint, quote consistency
2018-04-06 16:02:06 +01:00
Luke Barnard
db2fd801f7
Explicitly grab individual columns from group object
2018-04-06 15:57:25 +01:00
Luke Barnard
7945435587
When exposing group state, return is_openly_joinable
...
as opposed to join_policy, which is really only pertinent to the
synapse implementation of the group server.
By doing this we keep the group server concept extensible by
allowing arbitrarily complex rules for deciding whether a group
is openly joinable.
2018-04-06 15:43:27 +01:00
Luke Barnard
6bd1b7053e
By default, join policy is "invite"
2018-04-06 15:43:27 +01:00
Luke Barnard
b4478e586f
add_user -> _add_user
2018-04-06 15:43:27 +01:00
Luke Barnard
112c2253e2
pep8
2018-04-06 15:43:27 +01:00
Luke Barnard
6850f8aea3
Get group_info from existing call to check_group_is_ours
2018-04-06 15:43:27 +01:00
Luke Barnard
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
Luke Barnard
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
Luke Barnard
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
Luke Barnard
f8d1917fce
Fix federation client set_group_joinable
typo
2018-04-06 15:43:27 +01:00
Luke Barnard
6eb3aa94b6
Factor out add_user from accept_invite and join_group
2018-04-06 15:43:27 +01:00
David Baker
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
David Baker
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
Erik Johnston
7b824f1475
Add response size metrics
2018-04-06 13:20:11 +01:00
Krombel
1d71f484d4
use PUT instead of POST for federating groups/m.join_policy
2018-04-06 12:54:09 +02:00
Richard van der Hoff
15e8ed874f
more verbosity in synctl
2018-04-06 09:28:36 +01:00
Richard van der Hoff
c7f0969731
Merge pull request #2986 from jplatte/join_reponse_room_id
...
Add room_id to the response of `rooms/{roomId}/join`
2018-04-05 17:29:06 +01:00
Richard van der Hoff
3449da3bc7
Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation
...
Improve database cache performance
2018-04-05 17:21:44 +01:00
Richard van der Hoff
d1679a4ed7
Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics
...
Remove redundant metrics which were deprecated in 0.27.0.
2018-04-05 17:21:18 +01:00
Richard van der Hoff
01afc563c3
Fix overzealous cache invalidation
...
Fixes an issue where a cache invalidation would invalidate *all* pending
entries, rather than just the entry that we intended to invalidate.
2018-04-05 16:24:04 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
Richard van der Hoff
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
Luke Barnard
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
Luke Barnard
104c0bc1d5
Use "/settings/" (plural)
2018-04-05 14:07:16 +01:00
Luke Barnard
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
Luke Barnard
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
Neil Johnson
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
Richard van der Hoff
518f6de088
Remove redundant metrics which were deprecated in 0.27.0.
2018-04-04 19:46:28 +01:00
Jan Christian Grünhage
e4570c53dd
phone home cache size configurations
2018-04-04 16:46:58 +01:00
Adrian Tschira
6168351877
Add b prefixes to some strings that are bytes in py3
...
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
Richard van der Hoff
f92963f5db
Revert "improve mxid check performance"
2018-04-04 12:08:29 +01:00
Silke
72251d1b97
Remove address resolution of hosts in SRV records
...
Signed-off-by: Silke Hofstra <silke@slxh.eu>
2018-04-04 12:26:50 +02:00
Richard van der Hoff
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style
...
Replace old style error catching with 'as' keyword
2018-04-04 10:45:22 +01:00
Richard van der Hoff
a89f9f830c
Merge pull request #3044 from matrix-org/michaelk/performance_stats
...
Add basic performance statistics to phone home
2018-04-04 10:37:25 +01:00
Richard van der Hoff
39ce38b024
Merge pull request #3053 from NotAFile/speedup-mxid-check
...
improve mxid check performance
2018-04-04 10:19:52 +01:00
Richard van der Hoff
a9a74101a4
Document the behaviour of ResponseCache
...
it looks like everything that uses ResponseCache expects to have to
`make_deferred_yieldable` its results. It's debatable whether that is the best
approach, but let's document it for now to avoid further confusion.
2018-04-04 09:06:22 +01:00
Luke Barnard
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
Richard van der Hoff
3ee4ad09eb
Fix json encoding bug in replication
...
json encoders have an encode method, not a dumps method.
2018-04-03 15:09:48 +01:00
Adrian Tschira
11597ddea5
improve mxid check performance ~4x
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-31 02:00:22 +02:00
Richard van der Hoff
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
Richard van der Hoff
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
Richard van der Hoff
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
Neil Johnson
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
Neil Johnson
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
Erik Johnston
88cc9cc69e
Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation
...
Measure time it takes to calculate state group ID
2018-03-28 17:40:14 +01:00
Neil Johnson
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
David Baker
c5de6987c2
This should probably be a PUT
2018-03-28 16:44:11 +01:00
Neil Johnson
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
David Baker
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
Richard van der Hoff
9a0db062af
Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors
...
Fix error when claiming e2e keys from offline servers
2018-03-28 14:50:38 +01:00
David Baker
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
Neil Johnson
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
Neil Johnson
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
David Baker
32260baa41
pep8
2018-03-28 14:29:42 +01:00
Michael Kaye
33f6195d9a
Handle review comments
2018-03-28 14:25:25 +01:00
David Baker
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
David Baker
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
David Baker
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00