Neil Johnson
6023cdd227
remove errant print
2018-08-01 10:27:17 +01:00
Neil Johnson
7931393495
make count_monthly_users async synapse/handlers/auth.py
2018-08-01 10:21:56 +01:00
Neil Johnson
c507fa15ce
only need to loop if mau limiting is enabled
2018-08-01 10:20:42 +01:00
Travis Ralston
37be52ac34
limt -> limit
2018-07-31 16:29:09 -06:00
Serban Constantin
70af98e361
return NotFoundError if room not found
...
Per the Client-Server API[0] we should return
`M_NOT_FOUND` if the room isn't found instead
of generic SynapseError.
This ensures that /directory/list API returns
404 for room not found instead of 400.
[0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid
Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2018-07-31 21:47:23 +03:00
Travis Ralston
5e2ee64660
Merge pull request #3628 from turt2live/travis/goodby-pdu-failures
...
Remove pdu_failures from transactions
2018-07-31 12:13:09 -06:00
Neil Johnson
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
Richard van der Hoff
bdbdceeafa
rename replication_layer to federation_client
...
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
2018-07-31 15:44:05 +01:00
Erik Johnston
443da003bc
Use new helper base class for membership requests
2018-07-31 14:32:23 +01:00
Erik Johnston
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
Erik Johnston
d81602b75a
Add helper base class for generating new replication endpoints
...
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
Richard van der Hoff
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
...
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
Neil Johnson
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
Richard van der Hoff
0bc9b9e397
reinstate explicit include of EventsWorkerStore
2018-07-31 13:11:04 +01:00
Travis Ralston
e908b86832
Remove pdu_failures from transactions
...
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
Krombel
254e8267e2
Only import secrets when available
...
secrets got introduced in python 3.6 so this class is not available
in 3.5 and before.
This now checks for the current running version and only tries using
secrets if the version is 3.6 or above
Signed-Off-By: Matthias Kesler <krombel@krombel.de>
2018-07-30 23:59:02 +02:00
Neil Johnson
21276ff846
remove errant logging
2018-07-30 22:42:12 +01:00
Neil Johnson
fef7e58ac6
actually close conn
2018-07-30 22:29:44 +01:00
Neil Johnson
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
Neil Johnson
251e6c1210
limit register and sign in on number of monthly users
2018-07-30 15:55:57 +01:00
Erik Johnston
143f1a2532
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store
2018-07-30 09:56:18 +01:00
Jeroen
2903e65aff
fix isort
2018-07-29 19:47:08 +02:00
Richard van der Hoff
a8cbce0ced
fix invalidation
2018-07-27 16:17:17 +01:00
Matthew Hodgson
e9b2d047f6
make /context lazyload & filter aware ( #3567 )
...
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
Richard van der Hoff
f102c05856
Rewrite cache list decorator
...
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
2018-07-27 13:47:04 +01:00
Jeroen
8e3f75b39a
fix accidental removal of hs
2018-07-27 12:17:31 +02:00
Richard van der Hoff
b0b5566f36
Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid
...
if inviter_display_name == ""||None then default to inviter MXID
2018-07-27 10:08:39 +01:00
Richard van der Hoff
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests
2018-07-27 09:17:11 +01:00
Richard van der Hoff
d42455cdc9
Merge pull request #3616 from matrix-org/travis/event_id_send_leave
...
Update the send_leave path to be an event_id
2018-07-26 23:25:17 +01:00
Richard van der Hoff
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
...
Stop populating events.content
2018-07-26 22:54:08 +01:00
Matthew Hodgson
a75231b507
Deduplicate redundant lazy-loaded members ( #3331 )
...
* attempt at deduplicating lazy-loaded members
as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again. For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up). For now we hardcode X to 30.
* add include_redundant_members filter option & make it work
* remove stale todo
* add tests for _get_some_state_from_cache
* incorporate review
2018-07-26 22:51:30 +01:00
Richard van der Hoff
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
...
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
Travis Ralston
7d32f0d745
Update the send_leave path to be an event_id
...
It's still not used, however the parameter is an event ID not a transaction ID.
2018-07-26 14:41:59 -06:00
Richard van der Hoff
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
...
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
Richard van der Hoff
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
Richard van der Hoff
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff
bd4b25f4d0
Remove some redundant joins on event_edges.room_id
...
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.
event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
Richard van der Hoff
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
Richard van der Hoff
21e878ebb6
Make EventStore inherit from EventFederationStore
...
(since it uses methods therein)
Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
Richard van der Hoff
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Matthew Hodgson
1bcd0490c2
Merge pull request #2970 from matrix-org/matthew/filter_members
...
Implement the lazy_load_members room state filter parameter
2018-07-26 00:03:01 +01:00
Matthew Hodgson
bc7944e6d2
switch missing_types to be a bool
2018-07-25 23:36:31 +01:00
Travis Ralston
d8e65ed7e1
Fix a minor documentation typo in on_make_leave
2018-07-25 15:44:41 -06:00
Matthew Hodgson
2565804030
Merge branch 'develop' into matthew/filter_members
2018-07-25 17:27:49 +01:00
Matthew Hodgson
7d9fb88617
incorporate more review.
2018-07-25 16:33:50 +01:00
Erik Johnston
78a691d005
Split out DB writes in federation handler
...
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
Erik Johnston
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
...
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
Richard van der Hoff
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
...
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
Richard van der Hoff
32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
...
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
Richard van der Hoff
4081bd1560
Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache
...
Fix updating of cached remote profiles
2018-07-25 11:54:46 +01:00
Erik Johnston
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
Richard van der Hoff
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
Richard van der Hoff
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
Richard van der Hoff
55acd6856c
Fix updating of cached remote profiles
...
_update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when
it was called, would just return a generator object, without actually running
any of the method body.
2018-07-25 10:34:48 +01:00
Richard van der Hoff
f59be4eb0e
Fix unit tests
...
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
Erik Johnston
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
Richard van der Hoff
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Erik Johnston
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
Matthew Hodgson
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
Michael Telatynski
87951d3891
Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid
2018-07-24 17:17:46 +01:00
Erik Johnston
8b8c4f34a3
Replace usage of get_current_toke with StreamToken.START
...
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
2018-07-24 16:49:17 +01:00
Erik Johnston
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
Erik Johnston
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
Erik Johnston
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
Erik Johnston
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
Erik Johnston
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
Erik Johnston
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
Erik Johnston
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Richard van der Hoff
81946db9cf
Merge pull request #3587 from matrix-org/rav/better_exception_logging
...
Improve logging for exceptions when handling PDUs
2018-07-24 14:12:12 +01:00
Richard van der Hoff
a321f78991
Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups
...
Fixes and optimisations for resolve_state_groups
2018-07-24 14:11:45 +01:00
Richard van der Hoff
93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
...
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
Matthew Hodgson
454f59b7ad
Merge branch 'develop' into matthew/filter_members
2018-07-24 14:03:37 +01:00
Matthew Hodgson
1a01a5b964
clarify comment on p_ids
2018-07-24 14:03:15 +01:00
Erik Johnston
223341205e
Don't require to_delete to have event_ids
2018-07-24 14:02:40 +01:00
Matthew Hodgson
e22700c3dd
consider non-filter_type types as wildcards, thus missing from the state-group-cache
2018-07-24 13:59:07 +01:00
Matthew Hodgson
eb1d911ab7
rather than adding ll_ids, remove them from p_ids
2018-07-24 13:40:49 +01:00
Matthew Hodgson
d19fba3655
Merge branch 'develop' into matthew/filter_members
2018-07-24 12:39:54 +01:00
Matthew Hodgson
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
Richard van der Hoff
30bfed5aa5
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-24 11:46:09 +01:00
Erik Johnston
97acd385a3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta
2018-07-24 11:32:13 +01:00
Erik Johnston
0fa73e4a63
Remove unnecessary if
2018-07-24 11:19:23 +01:00
Erik Johnston
ff5426f6b8
Speed up _calculate_state_delta
2018-07-24 10:55:11 +01:00
Richard van der Hoff
a678145010
Merge branch 'develop' into rav/logcontext_fixes
2018-07-24 10:43:30 +01:00
Erik Johnston
d436ad332c
Merge pull request #3555 from matrix-org/erikj/client_apis_move
...
Make client_reader support some more read only APIs
2018-07-24 10:42:28 +01:00
Erik Johnston
536bc63a4e
Merge branch 'develop' into erikj/client_apis_move
2018-07-24 09:57:05 +01:00
Richard van der Hoff
cf2d15c6a9
another couple of logcontext leaks
2018-07-24 00:57:48 +01:00
Richard van der Hoff
8dff6e0322
Logcontext fixes
...
Fix some random logcontext leaks.
2018-07-24 00:37:17 +01:00
Richard van der Hoff
69fb5dbdab
fix idiocy
2018-07-24 00:04:44 +01:00
Richard van der Hoff
1938cffaea
Add some measure blocks to persist_events
...
... to help us figure out where 40% of CPU is going
2018-07-23 23:48:19 +01:00
Matthew Hodgson
efcdacad7d
handle case where types is [] on postgres correctly
2018-07-23 22:41:05 +01:00
Richard van der Hoff
ce0c18dec5
Improve logging for exceptions handling PDUs
...
when we get an exception handling a federation PDU, log the whole stacktrace.
2018-07-23 22:13:19 +01:00
Richard van der Hoff
c1f80effbe
Handle delta_ids being None in _update_context_for_auth_events
...
it's easier to create the new state group as a delta from the existing one.
(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364 )
2018-07-23 22:06:50 +01:00
Matthew Hodgson
adfe29ec0b
Merge branch 'develop' into matthew/filter_members
2018-07-23 19:21:37 +01:00
Matthew Hodgson
254fb430d1
incorporate review
2018-07-23 19:21:20 +01:00
Richard van der Hoff
5c705f70c9
Fixes and optimisations for resolve_state_groups
...
First of all, fix the logic which looks for identical input state groups so
that we actually use them. This turned out to be most easily done by factoring
the relevant code out to a separate function so that we could do an early
return.
Secondly, avoid building the whole `conflicted_state` dict (which was only ever
used as a boolean flag).
Thirdly, replace the construction of the `state` dict (which mapped from keys
to events that set them), with an optimistic construction of the resolution
result assuming there will be no conflicts. This should be no slower than
building the old `state` dict, and:
- in the conflicted case, we'll short-cut it, saving part of the work
- in the unconflicted case, it saves rebuilding the resolution from the
`state` dict.
Finally, do a couple of s/values/itervalues/.
2018-07-23 19:10:11 +01:00
Erik Johnston
8b9f164fff
Comments
2018-07-23 17:43:01 +01:00
Erik Johnston
50c60e5fad
Only get cached state from context in persist_event
...
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
2018-07-23 17:21:40 +01:00
Richard van der Hoff
4f5cc8e4e7
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-23 17:15:12 +01:00
Richard van der Hoff
dae6dc1e77
Remove redundant checks on room forgottenness
...
Fixes #3550
2018-07-23 17:13:34 +01:00
Erik Johnston
0faa3223cd
Fix missing attributes on workers.
...
This was missed during the transition from attribute to getter for
getting state from context.
2018-07-23 16:28:00 +01:00
Erik Johnston
999bcf9d01
Fix EventContext when using workers
...
We were:
1. Not correctly setting all attributes
2. Using defer.inlineCallbacks in a non-generator
2018-07-23 15:24:21 +01:00
Erik Johnston
4797ed000e
Update docstrings to make sense
2018-07-23 15:05:56 +01:00
Erik Johnston
0b0b24cb82
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move
2018-07-23 13:21:15 +01:00
Erik Johnston
027bc01a1b
Add support for updating state
2018-07-23 13:17:25 +01:00
Erik Johnston
e42510ba63
Use new getters
2018-07-23 13:17:22 +01:00
Erik Johnston
440b8845b5
Make EventContext lazy load state
2018-07-23 12:56:56 +01:00
Erik Johnston
842cdece42
pep8
2018-07-23 12:31:35 +01:00
Erik Johnston
acbfdc3442
Refcator EventContext to accept state during init
2018-07-23 12:20:55 +01:00
Matthew Hodgson
9b34f3ea3a
Merge branch 'develop' into matthew/sync_deleted_devices
2018-07-23 10:03:28 +01:00
Matthew Hodgson
c1bf2b587e
add trailing comma
2018-07-23 09:56:23 +01:00
Amber Brown
3132b89f12
Make the rest of the .iterwhatever go away ( #3562 )
2018-07-21 15:47:18 +10:00
Erik Johnston
5c88bb722f
Move PaginationHandler to its own file
2018-07-20 15:32:23 +01:00
Erik Johnston
0ecf68aedc
Move check_in_room_or_world_readable to Auth
2018-07-20 15:30:59 +01:00
Richard van der Hoff
3d6df84658
Test and fix support for cancellation in Linearizer
2018-07-20 13:59:55 +01:00
Richard van der Hoff
4f67623674
Merge pull request #3571 from matrix-org/rav/limiter_fixes
...
A set of improvements to the Limiter
2018-07-20 13:53:03 +01:00
Amber Brown
e1a237eaab
Admin API for creating new users ( #3415 )
2018-07-20 22:41:13 +10:00
Richard van der Hoff
7c712f95bb
Combine Limiter and Linearizer
...
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
2018-07-20 13:11:43 +01:00
Richard van der Hoff
8462c26485
Improvements to the Limiter
...
* give them names, to improve logging
* use a deque rather than a list for efficiency
2018-07-20 12:50:27 +01:00
Richard van der Hoff
d7275eecf3
Add a sleep to the Limiter to fix stack overflows.
...
Fixes #3570
2018-07-20 12:37:12 +01:00
Matthew Hodgson
2f558300cc
fix thinkos; unbreak tests
2018-07-19 19:22:27 +01:00
Matthew Hodgson
bcaec2915a
incorporate review
2018-07-19 19:03:50 +01:00
Matthew Hodgson
924eb34d94
add a filtered_types param to limit filtering to specific types
2018-07-19 18:32:02 +01:00
Richard van der Hoff
5f3658baf5
Merge pull request #3377 from Valodim/note-affinity
...
document that the affinity package is required for the cpu_affinity setting
2018-07-19 14:35:06 +01:00
Amber Brown
9a8acdc720
Merge branch 'master' into develop
2018-07-19 21:16:44 +10:00
Amber Brown
38f53399a2
0.33 final
2018-07-19 21:11:40 +10:00
Amber Brown
95ccb6e2ec
Don't spew errors because we can't save metrics ( #3563 )
2018-07-19 20:58:18 +10:00
Matthew Hodgson
be3adfc331
merge develop pydoc for _get_state_for_groups
2018-07-19 11:26:04 +01:00
Matthew Hodgson
9e40834f74
yes, we do need to invalidate the device_id_exists_cache when deleting a remote device
2018-07-19 11:15:10 +01:00
Richard van der Hoff
f1a15ea206
revert 00bc979
...
... we've fixed the things that caused the warnings, so we should reinstate the
warning.
2018-07-19 11:14:20 +01:00
Richard van der Hoff
1ffb7bec20
Merge remote-tracking branch 'origin/release-v0.33.0' into develop
2018-07-19 11:12:33 +01:00
Richard van der Hoff
c754e006f4
Merge pull request #3556 from matrix-org/rav/background_processes
...
Run things as background processes
2018-07-19 11:04:18 +01:00
Amber Brown
a97c845271
Move v1-only APIs into their own module & isolate deprecated ones ( #3460 )
2018-07-19 20:03:33 +10:00
Matthew Hodgson
c0685f67c0
spell out that include_deleted_devices requires include_all_devices
2018-07-19 10:59:02 +01:00
Richard van der Hoff
00bc979137
Disable logcontext warning
...
Temporary workaround to #3518 while we release 0.33.0.
2018-07-19 10:51:15 +01:00
Erik Johnston
6f62a6ef21
Merge pull request #3554 from matrix-org/erikj/response_metrics_code
...
Add response code to response timer metrics
2018-07-19 10:25:18 +01:00
Richard van der Hoff
8c69b735e3
Make Distributor run its processes as a background process
...
This is more involved than it might otherwise be, because the current
implementation just drops its logcontexts and runs everything in the sentinel
context.
It turns out that we aren't actually using a bunch of the functionality here
(notably suppress_failures and the fact that Distributor.fire returns a
deferred), so the easiest way to fix this is actually by simplifying a bunch of
code.
2018-07-18 20:55:05 +01:00
Richard van der Hoff
667fba68f3
Run things as background processes
...
This fixes #3518 , and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
Erik Johnston
9b596177ae
Add some room read only APIs to client_reader
2018-07-18 15:33:03 +01:00
Erik Johnston
bacdf0cbf9
Move RoomContextHandler out of Handlers
...
This is in preparation for moving GET /context/ to a worker
2018-07-18 15:33:03 +01:00
Erik Johnston
8cb8df55e9
Split MessageHandler into read only and writers
...
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
2018-07-18 15:33:03 +01:00
Richard van der Hoff
65d6a0e477
Merge pull request #3553 from matrix-org/rav/background_process_tracking
...
Resource tracking for background processes
2018-07-18 14:27:50 +01:00
Erik Johnston
5bd0a47fcd
pep8
2018-07-18 14:19:00 +01:00
Erik Johnston
e45a46b6e4
Add response code to response timer metrics
2018-07-18 13:59:36 +01:00
Richard van der Hoff
dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks
...
Remove event re-signing hacks
2018-07-18 12:46:27 +01:00
David Baker
c91a44572e
Merge pull request #3514 from matrix-org/dbkr/turn_dont_add_defaults
...
Comment dummy TURN parameters in default config
2018-07-18 11:56:18 +01:00
Richard van der Hoff
6e3fc657b4
Resource tracking for background processes
...
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.
This will help address #3518 , but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.
We *could* do this with Measure blocks, but:
- I think having them pulled out as a completely separate metric class will
make it easier to distinguish top-level processes from those which are
nested.
- I want to be able to report on in-flight background processes, and I don't
think we want to do this for *all* Measure blocks.
2018-07-18 10:50:33 +01:00
Amber Brown
5f3d02f6eb
bump to 0.33.0rc1
2018-07-18 12:52:56 +10:00
Richard van der Hoff
79eb339c66
add a comment
2018-07-17 14:53:34 +01:00
Richard van der Hoff
d897be6a98
Fix visibility of events from erased users over federation
2018-07-17 14:02:07 +01:00
Richard van der Hoff
9c04b4abf9
Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server
...
Refactor and optimze filter_events_for_server
2018-07-17 14:01:39 +01:00
Richard van der Hoff
94440ae994
fix imports
2018-07-17 11:51:26 +01:00
Amber Brown
bc006b3c9d
Refactor REST API tests to use explicit reactors ( #3351 )
2018-07-17 20:43:18 +10:00
Richard van der Hoff
2172a3d8cb
add a comment
2018-07-17 11:13:57 +01:00
Erik Johnston
b2aa05a8d6
Use efficient .intersection
2018-07-17 11:07:04 +01:00
Erik Johnston
547b1355d3
Fix perf regression in PR #3530
...
The get_entities_changed function was changed to return all changed
entities since the given stream position, rather than only those changed
from a given list of entities. This resulted in the function incorrectly
returning large numbers of entities that, for example, caused large
increases in database usage.
2018-07-17 10:27:51 +01:00
Amber Brown
3fe0938b76
Merge pull request #3530 from matrix-org/erikj/stream_cache
...
Don't return unknown entities in get_entities_changed
2018-07-17 13:44:46 +10:00
Richard van der Hoff
09e29fb58b
Attempt to make _filter_events_for_server more efficient
2018-07-16 14:06:09 +01:00
Krombel
78a9ddcf9a
rerun isort with latest version
2018-07-16 14:23:25 +02:00
Richard van der Hoff
ea69d35651
Move filter_events_for_server out of FederationHandler
...
for easier unit testing.
2018-07-16 13:06:24 +01:00
Krombel
4a27000548
check isort by travis
2018-07-16 13:57:33 +02:00
Jeroen
505530f36a
Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
...
# Conflicts:
# synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
Amber Brown
8a4f05fefb
Fix develop because I broke it :( ( #3535 )
2018-07-14 09:51:00 +10:00
Amber Brown
8532953c04
Merge pull request #3534 from krombel/use_parse_and_asserts_from_servlet
...
Use parse and asserts from http.servlet
2018-07-14 09:09:19 +10:00
Amber Brown
a2374b2c7f
fix sytests
2018-07-14 07:52:58 +10:00
Amber Brown
33b60c01b5
Make auth & transactions more testable ( #3499 )
2018-07-14 07:34:49 +10:00
Krombel
516f960ad8
add changelog
2018-07-13 22:19:19 +02:00
Krombel
3366b9c534
rename assert_params_in_request to assert_params_in_dict
...
the method "assert_params_in_request" does handle dicts and not
requests. A request body has to be parsed to json before this method
can be used
2018-07-13 21:53:01 +02:00
Krombel
32fd6910d0
Use parse_{int,str} and assert from http.servlet
...
parse_integer and parse_string can take a request and raise errors
in case we have wrong or missing params.
This PR tries to use them more to deduplicate some code and make it
better readable
2018-07-13 21:40:14 +02:00
Richard van der Hoff
2aba1f549c
Merge pull request #3533 from matrix-org/rav/fix_federation_ratelimite_queue
...
Make FederationRateLimiter queue requests properly
2018-07-13 16:59:18 +01:00
Richard van der Hoff
33b40d0a25
Make FederationRateLimiter queue requests properly
...
popitem removes the *most recent* item by default [1]. We want the oldest.
Fixes #3524
[1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem
2018-07-13 16:19:40 +01:00
Erik Johnston
77b692e65d
Don't return unknown entities in get_entities_changed
...
The stream cache keeps track of all entities that have changed since
a particular stream position, so get_entities_changed does not need to
return unknown entites when given a larger stream position.
This makes it consistent with the behaviour of has_entity_changed.
2018-07-13 15:26:10 +01:00
Matthew Hodgson
ba22b6a456
typo
2018-07-13 12:03:39 +01:00
Richard van der Hoff
6dff49b8a9
Merge pull request #3521 from matrix-org/rav/optimise_stream_change_cache
...
Reduce set building in get_entities_changed
2018-07-12 12:08:49 +01:00
Matthew Hodgson
12ec58301f
shift to using an explicit deleted flag on m.device_list_update EDUs
...
and generally make it work.
2018-07-12 11:39:43 +01:00
Richard van der Hoff
fa5c2bc082
Reduce set building in get_entities_changed
...
This line shows up as about 5% of cpu time on a synchrotron:
not_known_entities = set(entities) - set(self._entity_to_key)
Presumably the problem here is that _entity_to_key can be largeish, and
building a set for its keys every time this function is called is slow.
Here we rewrite the logic to avoid building so many sets.
2018-07-12 11:37:44 +01:00
Richard van der Hoff
482d17b58b
Merge branch 'develop' into rav/enforce_report_api
2018-07-12 09:56:28 +01:00
Erik Johnston
0456e05977
Merge pull request #3505 from matrix-org/erikj/receipts_cahce
...
Use stream cache in get_linearized_receipts_for_room
2018-07-12 09:46:29 +01:00
Erik Johnston
aff1dfdf3d
Update return value docstring
2018-07-12 09:45:37 +01:00
Matthew Hodgson
5797f5542b
WIP to announce deleted devices over federation
...
Previously we queued up the poke correctly when the device was deleted,
but then the actual EDU wouldn't get sent, as the device was no longer known.
Instead, we now send EDUs for deleted devices too if there's a poke for them.
2018-07-12 01:32:39 +01:00
David Baker
36f4fd3e1e
Comment dummy TURN parameters in default config
...
This default config is parsed and used a base before the actual
config is overlaid, so with these values not commented out, the
code to detect when no turn params were set and refuse to generate
credentials was never firing because the dummy default was always set.
2018-07-11 15:49:29 +01:00
Erik Johnston
6ccefef07a
Use 'is not None' and add comments
2018-07-10 18:12:39 +01:00
Matthew Hodgson
ea752bdd99
s/becuase/because/g
2018-07-10 17:58:18 +01:00
Erik Johnston
05f5dabc10
Use stream cache in get_linearized_receipts_for_room
...
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
Richard van der Hoff
c3c29aa196
Attempt to include db threads in cpu usage stats ( #3496 )
...
Let's try to include time spent in the DB threads in the per-request/block cpu
usage metrics.
2018-07-10 16:12:36 +01:00
Richard van der Hoff
55370331da
Refactor logcontext resource usage tracking ( #3501 )
...
Factor out the resource usage tracking out to a separate object, which can be
passed around and copied independently of the logcontext itself.
2018-07-10 13:56:07 +01:00
Matthew Hodgson
16b10666e7
another typo
2018-07-10 12:28:42 +01:00
Matthew Hodgson
4ea391a6ae
typo (i think)
2018-07-10 12:08:09 +01:00
Richard van der Hoff
e31e5dee38
Add CPU metrics for _fetch_event_list
...
add a Measure block on _fetch_event_list, in the hope that we can better
measure CPU usage here.
2018-07-09 18:15:54 +01:00
Jeroen
b5e157d895
Merge branch 'develop' into send_sni_for_federation_requests
...
# Conflicts:
# synapse/http/endpoint.py
2018-07-09 08:51:11 +02:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Amber Brown
3060bcc8e9
version
2018-07-07 10:48:06 +10:00
Amber Brown
e845fd41c2
Correct attrs package name in requirements ( #3492 )
2018-07-07 10:46:59 +10:00
Richard van der Hoff
1cfc2c4790
Prepare 0.32.1 release
2018-07-06 16:50:52 +01:00
Richard van der Hoff
1464a0578a
Add explicit dependency on netaddr
...
the dependencies file, causing failures on upgrade (and presumably for new
installs).
2018-07-06 16:27:17 +01:00
Neil Johnson
277c561766
0.32.0 version bump, update changelog
2018-07-06 15:07:29 +01:00
Amber Brown
d196fe42a9
bump version to 0.32.0rc1
2018-07-05 20:22:35 +10:00
Richard van der Hoff
3cf3e08a97
Implementation of server_acls
...
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw .
2018-07-04 19:06:20 +01:00
Richard van der Hoff
546bc9e28b
More server_name validation
...
We need to do a bit more validation when we get a server name, but don't want
to be re-doing it all over the shop, so factor out a separate
parse_and_validate_server_name, and do the extra validation.
Also, use it to verify the server name in the config file.
2018-07-04 18:59:51 +01:00
Erik Johnston
13f7adf84b
Merge pull request #3473 from matrix-org/erikj/thread_cache
...
Invalidate cache on correct thread
2018-07-04 10:11:38 +01:00
Erik Johnston
40252d13d1
Merge pull request #3474 from matrix-org/erikj/py3_auth
...
Fix up auth check
2018-07-04 09:41:33 +01:00
Richard van der Hoff
a4ab491371
Merge branch 'develop' into rav/drop_re_signing_hacks
2018-07-04 07:13:38 +01:00
Richard van der Hoff
508196e08a
Reject invalid server names ( #3480 )
...
Make sure that server_names used in auth headers are sane, and reject them with
a sensible error code, before they disappear off into the depths of the system.
2018-07-03 14:36:14 +01:00
Erik Johnston
2c33b55738
Avoid relying on int vs None comparison
...
Python 3 doesn't support comparing None to ints
2018-07-02 11:40:32 +01:00
Erik Johnston
cbf82dddf1
Ensure that we define sender_domain
2018-07-02 11:37:57 +01:00
Erik Johnston
3905c693c5
Invalidate cache on correct thread
2018-07-02 11:36:44 +01:00
Matthew Hodgson
fc4f8f33be
replace invalid utf8 with \ufffd
2018-07-02 11:33:02 +01:00
Matthew Hodgson
1c867f5391
a fix which doesn't NPE everywhere
2018-07-01 11:56:33 +01:00
Matthew Hodgson
f131bf8d3e
don't mix unicode strings with utf8-in-byte-strings
...
otherwise we explode with:
```
Traceback (most recent call last):
File /usr/lib/python2.7/logging/handlers.py, line 78, in emit
logging.FileHandler.emit(self, record)
File /usr/lib/python2.7/logging/__init__.py, line 950, in emit
StreamHandler.emit(self, record)
File /usr/lib/python2.7/logging/__init__.py, line 887, in emit
self.handleError(record)
File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError
None, sys.stderr)
File /usr/lib/python2.7/traceback.py, line 124, in print_exception
_print(file, 'Traceback (most recent call last):')
File /usr/lib/python2.7/traceback.py, line 13, in _print
file.write(str+terminator)
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write
self.log.emit(self.level, format=u{log_io}, log_io=line)
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit
self.observer(event)
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__
errorLogger = self._errorLoggerForObserver(brokenObserver)
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver
if obs is not observer
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__
self.log = Logger(observer=self)
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__
namespace = self._namespaceFromCallingContext()
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext
return currentframe(2).f_globals[__name__]
File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe
for x in range(n + 1):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Logged from file site.py, line 129
File /usr/lib/python2.7/logging/__init__.py, line 859, in emit
msg = self.format(record)
File /usr/lib/python2.7/logging/__init__.py, line 732, in format
return fmt.format(record)
File /usr/lib/python2.7/logging/__init__.py, line 471, in format
record.message = record.getMessage()
File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage
msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128)
Logged from file site.py, line 129
```
...where the logger apparently recurses whilst trying to log the error, hitting the
maximum recursion depth and killing everything badly.
2018-07-01 05:08:58 +01:00
Erik Johnston
e3b4043800
Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking
...
Check the state of prev_events a bit more thoroughly when coming over federation
2018-06-29 13:55:02 +01:00
Matthew Hodgson
e72234f6bd
fix tests
2018-06-28 20:56:07 +01:00
Matthew Hodgson
f4f1cda928
add ip_range_whitelist parameter to limit where ASes can connect from
2018-06-28 20:32:00 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
Amber Brown
72d2143ea8
Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" ( #3454 )
2018-06-28 11:04:18 +01:00
Amber Brown
99800de63d
try and clean up
2018-06-27 11:40:27 +01:00
Amber Brown
f03a5d1a17
pep8
2018-06-27 11:38:14 +01:00
Amber Brown
a7ecf34b70
cleanups
2018-06-27 11:36:03 +01:00
Amber Brown
77078d6c8e
handle federation not telling us about prev_events
2018-06-27 11:27:32 +01:00
Jeroen
95341a8f6f
take idna implementation from twisted
2018-06-26 21:15:14 +02:00
Jeroen
b7f34ee348
allow self-signed certificates
2018-06-26 20:41:05 +02:00
Matthew Hodgson
8057489b26
Revert "Try to not use as much CPU in the StreamChangeCache"
2018-06-26 18:09:01 +01:00
Matthew Hodgson
d91efb06cf
Merge pull request #3451 from matrix-org/hawkowl/sorteddict-api
...
Try to not use as much CPU in the cache
2018-06-26 17:49:55 +01:00
Amber Brown
1202508067
fixes
2018-06-26 17:29:01 +01:00
Amber Brown
bd3d329c88
fixes
2018-06-26 17:28:12 +01:00
Amber Brown
abfe4b2957
try and make loading items from the cache faster
2018-06-26 17:25:34 +01:00
David Baker
028490afd4
Fix error on deleting users pending deactivation
...
Use simple_delete instead of simple_delete_one as commented
2018-06-26 10:52:52 +01:00
Matthew Hodgson
c7f6b420ae
Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api
...
add GDPR erase param to deactivate API
2018-06-26 10:43:14 +01:00
Matthew Hodgson
9570aa82eb
update doc for deactivate API
2018-06-26 10:42:50 +01:00
Matthew Hodgson
1e788db430
add GDPR erase param to deactivate API
2018-06-26 10:26:54 +01:00
Amber Brown
1d62c4a127
Merge pull request #3438 from turt2live/travis/dont-print-access-tokens-in-logs
...
Stop including access tokens in warnings in the log
2018-06-26 09:55:55 +01:00
Erik Johnston
d72fb9a448
Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts
...
allow non-consented users to still part rooms (to let us autopart them)
2018-06-25 20:14:18 +01:00
Erik Johnston
df48f7ef37
Actually fix it
2018-06-25 20:03:41 +01:00
Erik Johnston
a0e8a53c6d
Comment
2018-06-25 19:57:38 +01:00
Erik Johnston
7bdc5c8fa3
Fix bug with assuming wrong type
2018-06-25 19:56:02 +01:00
Erik Johnston
ea7a9c0483
Add fast path to _filter_events_for_server
...
Most rooms have a trivial history visibility like "shared" or
"world_readable", especially large rooms, so lets not bother getting the
full membership of those rooms in that case.
2018-06-25 19:49:13 +01:00
Matthew Hodgson
0269367f18
allow non-consented users to still part rooms (to let us autopart them)
2018-06-25 17:56:10 +01:00
Matthew Hodgson
784189b1f4
typos
2018-06-25 17:37:16 +01:00
Matthew Hodgson
6eb861b67f
typo
2018-06-25 17:37:16 +01:00
Erik Johnston
947fea67cb
Need to pass reactor to endpoint fac
2018-06-25 15:22:57 +01:00
Erik Johnston
33fdcfa957
Merge pull request #3441 from matrix-org/erikj/redo_erasure
...
Fix user erasure and re-enable
2018-06-25 14:37:01 +01:00
Erik Johnston
eb50c44eaf
Add UserErasureWorkerStore to workers
2018-06-25 14:22:24 +01:00
Amber Brown
07cad26d65
Remove all global reactor imports & pass it around explicitly ( #3424 )
2018-06-25 14:08:28 +01:00
Erik Johnston
244484bf3c
Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
...
This reverts commit 1d009013b3
.
2018-06-25 13:42:55 +01:00
Jeroen
07b4f88de9
formatting changes for pep8
2018-06-25 12:31:16 +02:00
Jeroen
3d605853c8
send SNI for federation requests
2018-06-24 22:38:43 +02:00
Travis Ralston
ec1e799e17
Don't print invalid access tokens in the logs
...
Tokens shouldn't be appearing the logs, valid or invalid.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-24 12:17:01 -06:00
Richard van der Hoff
1d009013b3
Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
...
This reverts commit ce0d911156
, reversing
changes made to b4a5d767a9
.
2018-06-22 16:35:10 +01:00
Richard van der Hoff
516f884176
Merge pull request #3435 from matrix-org/rav/fix_event_push_actions_tablescan
...
Fix event_push_actions tablescan when reinserting events
2018-06-22 15:57:22 +01:00
Mark Haines
9850f66abe
Deleting from event_push_actions needs to use an index
2018-06-22 15:54:48 +01:00
Erik Johnston
28ddc6cfbe
Also log number of events for serach context
2018-06-22 15:42:11 +01:00
Erik Johnston
4b4cec3989
Add some logging to search queries
2018-06-22 15:42:11 +01:00
Richard van der Hoff
200e11c5bf
Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable
...
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:18:51 +01:00
Erik Johnston
f8272813a9
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:12:26 +01:00
Richard van der Hoff
1d7ad11747
Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation
...
Make push actions rotation configurable
2018-06-22 15:07:31 +01:00
Erik Johnston
ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility
...
Support hiding events from deleted users
2018-06-22 15:06:44 +01:00
Erik Johnston
b4a5d767a9
Merge pull request #3428 from matrix-org/erikj/persisted_pdu
...
Simplify get_persisted_pdu
2018-06-22 14:47:43 +01:00
Erik Johnston
f79abda87f
Merge pull request #3427 from matrix-org/erikj/remove_filters
...
remove dead filter_events_for_clients
2018-06-22 14:45:42 +01:00
Erik Johnston
75dc3ddeab
Make push actions rotation configurable
2018-06-22 14:44:37 +01:00
Richard van der Hoff
43e02c409d
Disable partial state group caching for wildcard lookups
...
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
2018-06-22 11:52:07 +01:00
Richard van der Hoff
240f192523
Merge pull request #3382 from matrix-org/rav/optimise_state_groups
...
Optimise state_group_cache update
2018-06-22 11:20:20 +01:00
Richard van der Hoff
70e6501913
Merge pull request #3419 from matrix-org/rav/events_per_request
...
Log number of events fetched from DB
2018-06-22 11:17:56 +01:00
Richard van der Hoff
0495fe0035
Indirect evt_count updates via method call
...
so that we can stub it for the sentinel and not have a billion failing UTs
2018-06-22 10:42:28 +01:00
Amber Brown
9a685d60ae
Merge pull request #3418 from matrix-org/rav/fix_metric_desc
...
Fix description of "python_gc_time" metric
2018-06-22 09:43:26 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
Richard van der Hoff
cbbfaa4be8
Fix description of "python_gc_time" metric
2018-06-21 10:02:42 +01:00
Amber Brown
c2eff937ac
Populate synapse_federation_client_sent_pdu_destinations:count again ( #3386 )
2018-06-21 09:39:58 +01:00
Amber Brown
99b77aa829
Fix tcp protocol metrics naming ( #3410 )
2018-06-21 09:39:27 +01:00
Richard van der Hoff
b088aafcae
Log number of events fetched from DB
...
When we finish processing a request, log the number of events we fetched from
the database to handle it.
[I'm trying to figure out which requests are responsible for large amounts of
event cache churn. It may turn out to be more helpful to add counts to the
prometheus per-request/block metrics, but that is an extension to this code
anyway.]
2018-06-21 06:15:03 +01:00
Richard van der Hoff
aff3d76920
Merge pull request #3416 from matrix-org/rav/restart_indicator
...
Write a clear restart indicator in logs
2018-06-20 18:14:08 +01:00
Richard van der Hoff
02bfc581f8
Merge pull request #3399 from costacruise/master
...
Add error code to room creation error
2018-06-20 17:26:25 +01:00
Richard van der Hoff
245d53d32a
Write a clear restart indicator in logs
...
I'm fed up with never being able to find the point a server restarted in the
logs.
2018-06-20 15:33:14 +01:00
Amber Brown
f6c4d74f96
Fix inflight requests metric (incorrect name & traceback) ( #3413 )
2018-06-20 11:18:57 +01:00
Matthew Hodgson
ccfdaf68be
spell gauge correctly
2018-06-16 07:10:34 +01:00
Richard van der Hoff
9a793f861c
Merge branch 'master' into develop
2018-06-14 16:36:01 +01:00
Richard van der Hoff
53969e1960
SECURITY UPDATE: Prevent unauthorised users from setting state events in a room
...
when there is no `m.room.power_levels` event in force in the room. (PR #3397 )
Discussion around the Matrix Spec change proposal for this change can be
followed at https://github.com/matrix-org/matrix-doc/issues/1304 .
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJbIop9AAoJEIofk9V1tejV9lsIAJVH0l5dXROmy1KH/zt16AUA
CXa6Vv4Vyo6hKad/fZ81OZVRr5ChK/TvbIJVn/SA/muCfdoIFdxhT8eo/pXzO2UW
zReuLsDhAg+gSvpNus37oWj2FVsAE1HYDZ60lfaapAdZnkFit68d5DQZjO6nZHHA
YUXcU3GUwj0ZYuUzFzYKMLu6uNNasNkN8h6SS2lF7Bm4JaKDW+mFMfCyJwdIVSEh
BGhHoVpXdxFysD9s6Mwxqrz3KKg1Jtp7idDkk0x2S2Eh+gxyiDQQokv0oQ3+0+HG
sgy5Iz2t2CkpS02/j+LOvAZljTmnD0bXu3srGR+25StsoDFP038Am3bfQwtD190=
=9jsT
-----END PGP SIGNATURE-----
Merge tag 'v0.31.2'
SECURITY UPDATE: Prevent unauthorised users from setting state events in a room
when there is no `m.room.power_levels` event in force in the room. (PR #3397 )
Discussion around the Matrix Spec change proposal for this change can be
followed at https://github.com/matrix-org/matrix-doc/issues/1304 .
2018-06-14 16:35:33 +01:00
Richard van der Hoff
7e1c616452
v0.31.2
2018-06-14 16:24:32 +01:00
Richard van der Hoff
61ab08a197
Merge pull request #3397 from matrix-org/rav/adjust_auth_rules
...
Adjust event auth rules when there is no PL event
2018-06-14 16:09:13 +01:00
Michael Wagner
19cd3120ec
Add error code to room creation error
...
This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom
2018-06-14 14:08:40 +02:00
Richard van der Hoff
5c9afd6f80
Make default state_default 50
...
Make it so that, before there is a power-levels event in the room, you need a
power level of at least 50 to send state.
Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192
2018-06-14 12:38:09 +01:00
Richard van der Hoff
52423607bd
Clarify interface for event_auth
...
stop pretending that it returns a boolean, which just almost gave me a heart
attack.
2018-06-14 12:26:17 +01:00
Amber Brown
f116f32ace
add a last seen metric ( #3396 )
2018-06-14 20:26:59 +10:00
Richard van der Hoff
557b686eac
Refactor get_send_level to take a power_levels event
...
it makes it easier for me to reason about
2018-06-14 11:26:27 +01:00
Amber Brown
a61738b316
Remove run_on_reactor ( #3395 )
2018-06-14 18:27:37 +10:00
Amber Brown
0fde1896cd
Merge pull request #3389 from turt2live/travis/name_metrics
...
Use the correct flag (enable_metrics) when warning about an incorrect metrics setup
2018-06-13 23:50:10 +10:00
Michael Telatynski
94700e55fa
if inviter_display_name == ""||None then default to inviter MXID
...
to prevent email invite from "None"
2018-06-13 10:31:01 +01:00
Travis Ralston
45768d1640
Use the RegistryProxy for appservices too
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:55:48 -06:00
Travis Ralston
12285a1a76
The flag is named enable_metrics, not collect_metrics
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:51:31 -06:00
Richard van der Hoff
b6faef2ad7
Filter out erased messages
...
Redact any messges sent by erased users.
2018-06-12 09:53:18 +01:00
Richard van der Hoff
f1023ebf4b
mark accounts as erased when requested
2018-06-12 09:53:18 +01:00
Richard van der Hoff
3ff8a619f5
UserErasureStore
...
to store which users have been erased
2018-06-12 09:52:22 +01:00
Richard van der Hoff
9fc5b74b24
simplify get_persisted_pdu
...
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
2018-06-12 09:51:31 +01:00
Richard van der Hoff
bd348f0af6
remove dead filter_events_for_clients
...
This is only used by filter_events_for_client, so we can simplify the whole
thing by just doing one user at a time, and removing a dead storage function to
boot.
2018-06-12 09:51:31 +01:00
Richard van der Hoff
eb32b2ca20
Optimise state_group_cache update
...
(1) matrix-org-hotfixes has removed the intern calls; let's do the same here.
(2) remove redundant iteritems() so we can used an optimised db update.
2018-06-11 22:56:11 +01:00
David Baker
187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind
...
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
Matthew Hodgson
d6cc369205
fix idiotic typo in state res
2018-06-11 14:43:55 +01:00
Matthew Hodgson
c96d882a02
Merge branch 'develop' into matthew/filter_members
2018-06-10 12:26:14 +03:00
Vincent Breitmoser
b800834351
add note that the affinity package is required for the cpu_affinity setting
2018-06-09 22:50:29 +02:00
Neil Johnson
ed5a0780a4
Merge branch 'master' into develop
2018-06-08 15:47:11 +01:00
Neil Johnson
1032393dfb
Changes in synapse v0.31.1 (2018-06-08)
...
=======================================
v0.31.1 fixes a security bug in the ``get_missing_events`` federation API
where event visibility rules were not applied correctly.
We are not aware of it being actively exploited but please upgrade asap.
Bug Fixes:
* Fix event filtering in get_missing_events handler (PR #3371 )
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsalP8ACgkQCTxDAAxP
S/RADw/+NeDu0LjVpS5Uc4ElHgRBuFSm6l2i4z8rZBBlKSYnuq0Em4WMvLloi/JF
iAvTOYE7OjmF+gNvmsdH1N7hc1lKdQ2gAlpvaQR/5Qz9NtOVmM3WPZxS7n5jZHvD
hVSxeO9+GQOwK7rJorqrrsnWHQt0OkLHV6WThFdrgZb1JjWCUDTvw+Hei2uMX2aq
y2mkMG4TLStHwMvL2qw0h+hFtXywXI796qJR73ZxbEn24YD+kOXeEVkIFi2LT0Pj
cgkg7WWT32JD43/ioumLupZuhCmpRyxn4fi5gIKpXe5kiLsxOdApzNQSwmoJ+WA8
7zlrWY+0QDN4pbA5ESLitWSWAT50Ul//uM4nwmM4xEBPHdljXvKyHPsaSCeKLvT9
RT8pc41TQAqSshlXF8zgIAtStnF3oGel3EBBl1mmM9Un1ULnBFwWDZhlIm+ZZGhJ
MWoAWNG7j8AQuy0BTUAUr76x7t+/cdSqDuyVl1GO1tbDh0DUWoHZGXCUKrAXnn2T
SbiFigwOLvEADbvkW7L9Je9CVOi2V5Pg/32X9O8YMEiSz+j5PQEiGefyVh/I/QvV
Ha/atRpZF2OZ+XUOO5DLZMP/XCXpVgvHuskzfU6LvvVQCXgExuJhRD1PrRGeBaWr
zjJW+rmY+VeredqX7QkTB3XOGLMLGJfx2FeSMb+j0w3a7/iFj+Q=
=J80S
-----END PGP SIGNATURE-----
Merge tag 'v0.31.1'
Changes in synapse v0.31.1 (2018-06-08)
=======================================
v0.31.1 fixes a security bug in the ``get_missing_events`` federation API
where event visibility rules were not applied correctly.
We are not aware of it being actively exploited but please upgrade asap.
Bug Fixes:
* Fix event filtering in get_missing_events handler (PR #3371 )
2018-06-08 15:46:18 +01:00
Neil Johnson
0eb4722932
changelog a bump version
2018-06-08 15:21:46 +01:00
Richard van der Hoff
c6b1441c52
Fix event filtering in get_missing_events handler
2018-06-08 14:15:31 +01:00
David Baker
0e505b1913
Merge pull request #3372 from matrix-org/rav/better_verification_logging
...
Try to log more helpful info when a sig verification fails
2018-06-08 13:32:02 +01:00
Richard van der Hoff
e82db24a0e
Try to log more helpful info when a sig verification fails
...
Firstly, don't swallow the reason for the failure
Secondly, don't assume all exceptions are verification failures
Thirdly, log a bit of info about the key being used if debug is enabled
2018-06-08 12:13:08 +01:00
Richard van der Hoff
0834b49c6a
Fix event filtering in get_missing_events handler
2018-06-08 11:34:46 +01:00
Richard van der Hoff
8503dd0047
Remove event re-signing hacks
...
These "temporary fixes" have been here three and a half years, and I can't find
any events in the matrix.org database where the calculated signature differs
from what's in the db. It's time for them to go away.
2018-06-07 16:08:29 +01:00
Will Hunt
13d211edc1
Merge pull request #3344 from Half-Shot/hs/as-metrics
...
Add metrics to track appservice transactions
2018-06-07 11:37:12 +01:00
Richard van der Hoff
1152f495a0
Merge pull request #3363 from matrix-org/rav/fix_purge
...
Fix event-purge-by-ts admin API
2018-06-07 11:34:46 +01:00
Richard van der Hoff
e2acf536d4
Merge pull request #3355 from matrix-org/rav/fix_federation_backfill
...
Fix federation backfill from sqlite servers
2018-06-07 10:18:01 +01:00
Amber Brown
0160f6601f
Merge pull request #3356 from matrix-org/rav/add_missing_attr_dep
...
Make sure that attr is installed
2018-06-07 16:33:30 +10:00
Richard van der Hoff
f4caf3f83d
fix log
2018-06-07 00:26:38 +01:00
Richard van der Hoff
0546715c18
Fix event-purge-by-ts admin API
...
This got completely broken in 0.30.
Fixes #3300 .
2018-06-07 00:15:49 +01:00
Richard van der Hoff
57e3f923d2
Add missing dependency on attr
...
We've rcently added a dep on `attr`. I don't know why the CI didn't pick this
up, but we should make it explicit anyway.
2018-06-06 17:12:41 +01:00
Richard van der Hoff
d3a8c9c55e
Fix sql error in _get_state_groups_from_groups
...
If this was called with a `(type, None)` entry in types (which is supposed to
return all state of type `type`), it would explode with a sql error.
2018-06-06 14:19:01 +01:00
Neil Johnson
fef6c2cdcc
Merge branch 'master' into develop
2018-06-06 12:28:15 +01:00
Neil Johnson
752b7b32ed
Changes in synapse v0.31.0 (2018-06-06)
...
======================================
Most notable change from v0.30.0 is to switch to python prometheus library to improve system
stats reporting. WARNING this changes a number of prometheus metrics in a
backwards-incompatible manner. For more details, see
`docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_.
Bug Fixes:
* Fix metric documentation tables (PR #3341 )
* Fix LaterGuage error handling (694968f
)
* Fix replication metrics (b7e7fd2
)
Changes in synapse v0.31.0-rc1 (2018-06-04)
==========================================
Features:
* Switch to the Python Prometheus library (PR #3256 , #3274 )
* Let users leave the server notice room after joining (PR #3287 )
Changes:
* daily user type phone home stats (PR #3264 )
* Use iter* methods for _filter_events_for_server (PR #3267 )
* Docs on consent bits (PR #3268 )
* Remove users from user directory on deactivate (PR #3277 )
* Avoid sending consent notice to guest users (PR #3288 )
* disable CPUMetrics if no /proc/self/stat (PR #3299 )
* Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312 ) Thanks to @thegcat!
* Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307 )
* Add private IPv6 addresses to example config for url preview blacklist (PR #3317 ) Thanks to @thegcat!
* Reduce stuck read-receipts: ignore depth when updating (PR #3318 )
* Put python's logs into Trial when running unit tests (PR #3319 )
Changes, python 3 migration:
* Replace some more comparisons with six (PR #3243 ) Thanks to @NotAFile!
* replace some iteritems with six (PR #3244 ) Thanks to @NotAFile!
* Add batch_iter to utils (PR #3245 ) Thanks to @NotAFile!
* use repr, not str (PR #3246 ) Thanks to @NotAFile!
* Misc Python3 fixes (PR #3247 ) Thanks to @NotAFile!
* Py3 storage/_base.py (PR #3278 ) Thanks to @NotAFile!
* more six iteritems (PR #3279 ) Thanks to @NotAFile!
* More Misc. py3 fixes (PR #3280 ) Thanks to @NotAFile!
* remaining isintance fixes (PR #3281 ) Thanks to @NotAFile!
* py3-ize state.py (PR #3283 ) Thanks to @NotAFile!
* extend tox testing for py3 to avoid regressions (PR #3302 ) Thanks to @krombel!
* use memoryview in py3 (PR #3303 ) Thanks to @NotAFile!
Bugs:
* Fix federation backfill bugs (PR #3261 )
* federation: fix LaterGauge usage (PR #3328 ) Thanks to @intelfx!
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsXxJAACgkQCTxDAAxP
S/SJTg//Wtr+Qop9LJh2/leAYXpyqW6P7Ftak0w3aJ3KL3+tYg32yYNoRADCqbp3
LkrHu8MwbZagHjRUyEWNfDk4jbfq5fwh0JVGmYuUKhG9aF0HYyytKkbW79YzuhdQ
dfHj9x0xSBOUvgt/husloZSDy0VHC6uyQSAFgFDyHS2y7RPAiGstqLGByv0ciZOk
pO7TdjkUQcx4Ps7Wgip31NuHy3GY2int6f540pUXoZHLXs7RkfqS2cpF9Z/sTXJ4
xDLiY7uYNsTcCblwqaiijY5c90xwRB2vLs5CJdKFgyB6PNgg/2wHJqP/WHHEj+F8
BoSm3Ts7NXQf23pP9CXICe7vXX3J+ruOnC7FOSRobr6KGjn6DUrIZxo1ZepTwpp9
DIq+1eOFKKjwLQM3Jdi8WBCP63LhYXrTZxreke3jpwdcD7oIO9v6/e9J1gU5xHWa
Izg+YnWn1JLfq/X8T7YTZddUXGGPkH5i6LZKKkyY8u7LkJ4WR7syuAceUzkOOIAq
UWO0uEV7IiLnZzZGTtNIlEqtuklmVQTm6bvAgTPabai2JQyngFpH5M/5mPpVSiLV
QRLwaM56c+5GGZJWef8vxdGeYn+8rFI/UUniJ7358kLJF2IHsxlZu8J0ZZO2HWI2
ze5Kz0AWRzXLhWzq62Qb2dsiGySrZ7hng1tDxLak2IiusY+9SjM=
=Mz9U
-----END PGP SIGNATURE-----
Merge tag 'v0.31.0'
Changes in synapse v0.31.0 (2018-06-06)
======================================
Most notable change from v0.30.0 is to switch to python prometheus library to improve system
stats reporting. WARNING this changes a number of prometheus metrics in a
backwards-incompatible manner. For more details, see
`docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_.
Bug Fixes:
* Fix metric documentation tables (PR #3341 )
* Fix LaterGuage error handling (694968f
)
* Fix replication metrics (b7e7fd2
)
Changes in synapse v0.31.0-rc1 (2018-06-04)
==========================================
Features:
* Switch to the Python Prometheus library (PR #3256 , #3274 )
* Let users leave the server notice room after joining (PR #3287 )
Changes:
* daily user type phone home stats (PR #3264 )
* Use iter* methods for _filter_events_for_server (PR #3267 )
* Docs on consent bits (PR #3268 )
* Remove users from user directory on deactivate (PR #3277 )
* Avoid sending consent notice to guest users (PR #3288 )
* disable CPUMetrics if no /proc/self/stat (PR #3299 )
* Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312 ) Thanks to @thegcat!
* Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307 )
* Add private IPv6 addresses to example config for url preview blacklist (PR #3317 ) Thanks to @thegcat!
* Reduce stuck read-receipts: ignore depth when updating (PR #3318 )
* Put python's logs into Trial when running unit tests (PR #3319 )
Changes, python 3 migration:
* Replace some more comparisons with six (PR #3243 ) Thanks to @NotAFile!
* replace some iteritems with six (PR #3244 ) Thanks to @NotAFile!
* Add batch_iter to utils (PR #3245 ) Thanks to @NotAFile!
* use repr, not str (PR #3246 ) Thanks to @NotAFile!
* Misc Python3 fixes (PR #3247 ) Thanks to @NotAFile!
* Py3 storage/_base.py (PR #3278 ) Thanks to @NotAFile!
* more six iteritems (PR #3279 ) Thanks to @NotAFile!
* More Misc. py3 fixes (PR #3280 ) Thanks to @NotAFile!
* remaining isintance fixes (PR #3281 ) Thanks to @NotAFile!
* py3-ize state.py (PR #3283 ) Thanks to @NotAFile!
* extend tox testing for py3 to avoid regressions (PR #3302 ) Thanks to @krombel!
* use memoryview in py3 (PR #3303 ) Thanks to @NotAFile!
Bugs:
* Fix federation backfill bugs (PR #3261 )
* federation: fix LaterGauge usage (PR #3328 ) Thanks to @intelfx!
2018-06-06 12:27:33 +01:00
Neil Johnson
61134debdc
bump version and changelog
2018-06-06 11:26:21 +01:00
Richard van der Hoff
ad459a106c
Merge pull request #3349 from t3chguy/redact_as_request_token
...
Redact AS tokens in log (fixes to #3327 )
2018-06-06 10:58:07 +01:00
Michael Telatynski
592c162516
also redact __str__ of ApplicationService used for logging
2018-06-06 10:35:29 +01:00
Michael Telatynski
330432031b
redact_uri in two missed log paths
2018-06-06 10:25:48 +01:00
David Baker
bf54c1cf6c
pep8
2018-06-06 10:15:33 +01:00
David Baker
3e4bc4488c
More doc fixes
2018-06-06 09:44:10 +01:00
Richard van der Hoff
592ee217a3
Merge commit 'b7e7fd2' into release-v0.31.0
2018-06-06 07:02:02 +01:00
Ivan Shapovalov
c88d50aa8f
federation/send_queue.py: fix usage of sortedcontainers.SortedDict
2018-06-06 02:45:18 +03:00
Richard van der Hoff
e316407b5d
Merge pull request #3327 from t3chguy/redact_as_request_token
...
Strip `access_token` from outgoing requests
2018-06-05 19:08:46 +01:00
Michael Telatynski
e6cbf47773
factor out uri redaction into a method on http
2018-06-05 18:31:40 +01:00
David Baker
607bd27c83
fix pep8
2018-06-05 18:10:35 +01:00
David Baker
d62162bbec
doc fixes
2018-06-05 18:09:13 +01:00
Richard van der Hoff
522bd3c8a3
Merge remote-tracking branch 'origin/master' into develop
2018-06-05 17:42:49 +01:00
Will Hunt
d6e3c2c79b
Let's try labels instead of label, that might work
2018-06-05 17:30:45 +01:00
Amber Brown
f7869f8f8b
Port to sortedcontainers (with tests!) ( #3332 )
2018-06-06 00:13:57 +10:00
Will Hunt
604cff1a06
Add metrics to track appservice transactions
2018-06-05 13:21:30 +01:00
Richard van der Hoff
f29b41fde9
Merge pull request #3324 from matrix-org/rav/remove_dead_method
...
Remove was_forgotten_at
2018-06-04 18:18:08 +01:00
Richard van der Hoff
28b0490dfd
Merge pull request #3334 from matrix-org/rav/cache_factor_override
...
Cache factor override system for specific caches
2018-06-04 17:19:33 +01:00
Richard van der Hoff
b7e7fd2d0e
Fix replication metrics
...
fix bug introduced in #3256
2018-06-04 16:23:05 +01:00
Neil Johnson
244ab974e7
bump version and changelog
2018-06-04 16:09:58 +01:00
Richard van der Hoff
694968fa81
Hopefully, fix LaterGuage error handling
2018-06-04 15:59:14 +01:00
Erik Johnston
042eedfa2b
Add hacky cache factor override system
2018-06-04 15:39:28 +01:00
David Baker
c5930d513a
Docstring
2018-06-04 12:05:58 +01:00
David Baker
6a29e815fc
Fix comment
2018-06-04 12:01:23 +01:00
David Baker
e44150a6de
Missing yield
2018-06-04 12:01:13 +01:00
David Baker
f731e42baf
docstring
2018-06-04 12:00:51 +01:00
Matthew Hodgson
28f09fcdd5
Merge branch 'develop' into matthew/filter_members
2018-06-04 00:09:17 +03:00
Matthew Hodgson
5f6122fe10
more comments
2018-06-04 00:08:52 +03:00
Ivan Shapovalov
7d9d75e4e8
federation/send_queue.py: fix usage of LaterGauge
...
Fixes a startup crash due to commit df9f72d9e5
"replacing portions".
2018-06-03 14:16:17 +03:00
Michael Telatynski
09503126df
Strip access_token
from outgoing requests using existing regex
2018-06-02 23:25:13 +01:00
Richard van der Hoff
c1f4118bb6
Remove was_forgotten_at
...
This is unused. IT MUST DIE!!!1
̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢
̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *
2018-06-01 18:21:49 +01:00
Richard van der Hoff
a9e97dcd65
Merge pull request #3317 from thegcat/feature/3312-add_ipv6_to_blacklist_example_config
...
Add private IPv6 addresses to example config for url preview blacklist
2018-06-01 14:45:14 +01:00
Neil Johnson
71477f3317
Merge pull request #3264 from matrix-org/neil/sign-up-stats
...
daily user type phone home stats
2018-06-01 13:42:01 +00:00
Richard van der Hoff
9f797a24a4
Handle RRs which arrive before their events
2018-06-01 14:01:43 +01:00
Richard van der Hoff
857e6fd8b6
Ignore depth when updating read-receipts
...
Order read receipts by stream ordering instead of depth
2018-06-01 12:18:11 +01:00
Felix Schäfer
4ef76f3ac4
Add private IPv6 addresses to preview blacklist #3312
...
The added addresses are expected to be local or loopback addresses and
shouldn't be spidered for previews.
Signed-off-by: Felix Schäfer <felix@thegcat.net>
2018-06-01 12:18:35 +02:00
Neil Johnson
4986b084f8
remove unnecessary INSERT
2018-06-01 10:50:40 +01:00
Richard van der Hoff
7e15410f02
Enforce the specified API for report_event
...
as per
https://matrix.org/docs/spec/client_server/unstable.html#post-matrix-client-r0-rooms-roomid-report-eventid
2018-05-31 18:17:11 +01:00
Amber Brown
febe0ec8fd
Run Prometheus on a different port, optionally. ( #3274 )
2018-05-31 19:04:50 +10:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy ( #3307 )
2018-05-31 19:03:47 +10:00
Richard van der Hoff
219c2a322b
remove trailing whitespace
2018-05-30 19:42:19 +01:00
Richard van der Hoff
2e4be8bfd9
fix english and wrap comment
2018-05-30 19:24:12 +01:00
Amber Brown
872cf43516
Merge pull request #3303 from NotAFile/py3-memoryview
...
use memoryview in py3
2018-05-30 12:51:42 +10:00
Amber Brown
debff7ae09
Merge pull request #3281 from NotAFile/py3-six-isinstance
...
remaining isintance fixes
2018-05-30 12:44:46 +10:00
Richard van der Hoff
34b85df7f5
Update some comments and docstrings in SyncHandler
2018-05-29 22:31:18 +01:00
Richard van der Hoff
4a9cbdbc15
Exempt AS-registered users from doing gdpr
2018-05-29 19:54:32 +01:00
Neil Johnson
ab0ef31dc7
create users index on creation_ts
2018-05-29 17:51:08 +01:00
Neil Johnson
558f3d376a
create index in background
2018-05-29 17:47:55 +01:00
Neil Johnson
c379acd4fd
bump version
2018-05-29 17:47:28 +01:00
Adrian Tschira
4b9d0cde97
add remaining memoryview changes
2018-05-29 17:42:43 +02:00
Adrian Tschira
7873cde526
pep8
2018-05-29 17:35:55 +02:00
Adrian Tschira
1afafb3497
use memoryview in py3
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-29 17:14:34 +02:00
Matthew Hodgson
9bbb9f5556
add lazy_load_members to the filter json schema
2018-05-29 04:26:10 +01:00
Matthew Hodgson
8df7bad839
pep8
2018-05-29 02:49:36 +01:00
Matthew Hodgson
ff1bc0a279
pep8
2018-05-29 02:32:15 +01:00
Matthew Hodgson
0a240ad36e
disable CPUMetrics if no /proc/self/stat
...
fixes build on macOS again
2018-05-29 02:23:30 +01:00
Matthew Hodgson
b69ff33d9e
disable CPUMetrics if no /proc/self/stat
...
fixes build on macOS again
2018-05-29 02:22:27 +01:00
Matthew Hodgson
a6c8f7c875
add pydoc
2018-05-29 01:09:55 +01:00
Matthew Hodgson
7a6df013cc
merge develop
2018-05-29 00:25:22 +01:00
Amber Brown
57ad76fa4a
fix up tests
2018-05-28 19:51:53 +10:00
Amber Brown
3ef5cd74a6
update to more consistently use seconds in any metrics or logging
2018-05-28 19:39:27 +10:00
Amber Brown
5c40ce3777
invalid syntax :(
2018-05-28 19:16:09 +10:00
Amber Brown
357c74a50f
add comment about why unreg
2018-05-28 19:14:41 +10:00
Amber Brown
a2eb5db4a0
update metrics to be in seconds
2018-05-28 19:10:27 +10:00
Amber Brown
754826a830
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-28 18:57:23 +10:00
Ruben Barkow
08ea5fe635
add link to thorough instruction how to configure consent
2018-05-25 23:19:55 +02:00
Richard van der Hoff
66bdae986f
Fix default for send_server_notice_to_guests
...
bool("False") == True...
2018-05-25 11:42:05 +01:00
Richard van der Hoff
ba1b163590
Avoid sending consent notice to guest users
...
we think it makes sense not to send the notices to guest users.
2018-05-25 11:36:43 +01:00
Richard van der Hoff
41921ac01b
Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room
...
Let users leave the server notice room after joining
2018-05-25 11:15:45 +01:00
Richard van der Hoff
757ed27258
Let users leave the server notice room after joining
...
They still can't reject invites, but we let them leave it.
2018-05-25 11:07:21 +01:00
Adrian Tschira
4ee4450d66
fix recursion error
2018-05-24 21:44:10 +02:00
Amber Brown
9c36c150e7
Merge pull request #3283 from NotAFile/py3-state
...
py3-ize state.py
2018-05-24 14:23:33 -05:00
Amber Brown
cc1349c06a
Merge pull request #3279 from NotAFile/py3-more-iteritems
...
more six iteritems
2018-05-24 14:23:13 -05:00
Adrian Tschira
0e61705661
py3-ize state.py
2018-05-24 20:59:00 +02:00
Adrian Tschira
dd068ca979
remaining isintance fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:55:08 +02:00
Adrian Tschira
17a70cf6e9
Misc. py3 fixes
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
Adrian Tschira
6c16a4ec1b
more iteritems
2018-05-24 20:19:06 +02:00
Amber Brown
7ea07c7305
Merge pull request #3278 from NotAFile/py3-storage-base
...
Py3 storage/_base.py
2018-05-24 13:08:09 -05:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
...
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown
c4fb15a06c
Merge pull request #3246 from NotAFile/py3-repr-string
...
use repr, not str
2018-05-24 13:00:20 -05:00
Amber Brown
36501068d8
Merge pull request #3247 from NotAFile/py3-misc
...
Misc Python3 fixes
2018-05-24 12:58:37 -05:00
Amber Brown
2aff6eab6d
Merge pull request #3245 from NotAFile/batch-iter
...
Add batch_iter to utils
2018-05-24 12:54:12 -05:00
Adrian Tschira
095292304f
Py3 storage/_base.py
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 18:24:12 +02:00
David Baker
77a23e2e05
Merge remote-tracking branch 'origin/develop' into dbkr/unbind
2018-05-24 16:20:53 +01:00
David Baker
ecc4b88bd1
Merge pull request #3277 from matrix-org/dbkr/remove_from_user_dir
...
Remove users from user directory on deactivate
2018-05-24 16:12:12 +01:00
Erik Johnston
46345187cc
Merge pull request #3243 from NotAFile/py3-six-3
...
Replace some more comparisons with six
2018-05-24 16:08:57 +01:00
Neil Johnson
037c6db85d
Merge branch 'master' into develop
2018-05-24 16:03:44 +01:00
David Baker
7a1af504d7
Remove users from user directory on deactivate
2018-05-24 15:59:58 +01:00
Erik Johnston
f72d5a44d5
Merge pull request #3261 from matrix-org/erikj/pagination_fixes
...
Fix federation backfill bugs
2018-05-24 14:52:03 +01:00
Erik Johnston
68399fc4de
Merge pull request #3267 from matrix-org/erikj/iter_filter
...
Use iter* methods for _filter_events_for_server
2018-05-24 14:44:57 +01:00
Neil Johnson
91d95a1d8e
bump version
2018-05-24 14:05:07 +01:00
David Baker
9700d15611
pep8
2018-05-24 11:23:15 +01:00
David Baker
a21a41bad7
comment
2018-05-24 11:19:59 +01:00
David Baker
b3bff53178
Unbind 3pids when they're deleted too
2018-05-24 11:08:05 +01:00
Richard van der Hoff
6abcb5d22d
Merge pull request #3273 from matrix-org/rav/server_notices_avatar_url
...
Allow overriding the server_notices user's avatar
2018-05-24 10:31:43 +01:00
Amber Brown
389dac2c15
pepeightttt
2018-05-23 13:08:59 -05:00
Amber Brown
472a5ec4e2
add back CPU metrics
2018-05-23 13:03:56 -05:00
Amber Brown
e987079037
fixes
2018-05-23 13:03:51 -05:00
Richard van der Hoff
9bf4b2bda3
Allow overriding the server_notices user's avatar
...
probably should have done this in the first place, like @turt2live suggested.
2018-05-23 17:43:30 +01:00
Richard van der Hoff
a0b3946fe2
Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri
2018-05-23 16:06:03 +01:00
Richard van der Hoff
2f7008d4eb
Merge pull request #3271 from matrix-org/rav/consent_uri_in_messages
...
Support for putting %(consent_uri)s in messages
2018-05-23 16:04:30 +01:00
Richard van der Hoff
dda40fb55d
fix typo
2018-05-23 15:30:26 +01:00
Richard van der Hoff
3ff6f50eac
Use the localpart in the consent uri
...
... because it's shorter.
2018-05-23 15:28:23 +01:00
Richard van der Hoff
82191b08f6
Support for putting %(consent_uri)s in messages
...
Make it possible to put the URI in the error message and the server notice that
get sent by the server
2018-05-23 15:24:31 +01:00
David Baker
2c7866d664
Hit the 3pid unbind endpoint on deactivation
2018-05-23 14:38:56 +01:00
Richard van der Hoff
321f02d263
Block attempts to send server notices to remote users
2018-05-23 14:30:47 +01:00
Neil Johnson
3601a240aa
bump version and changelog
2018-05-23 12:03:23 +01:00
Erik Johnston
5aaa3189d5
s/values/itervalues/
2018-05-23 10:13:05 +01:00
Erik Johnston
0a4bca4134
Use iter* methods for _filter_events_for_server
2018-05-23 10:04:23 +01:00
Amber Brown
b6063631c3
more cleanup
2018-05-22 17:36:20 -05:00
Amber Brown
53cc2cde1f
cleanup
2018-05-22 17:32:57 -05:00
Amber Brown
071206304d
cleanup pep8 errors
2018-05-22 16:54:22 -05:00
Amber Brown
85ba83eb51
fixes
2018-05-22 16:28:23 -05:00
Amber Brown
228f1f584e
fix the test failures
2018-05-22 15:02:38 -05:00
Erik Johnston
e85b5a0ff7
Use iter* methods
2018-05-22 19:02:48 +01:00
Erik Johnston
586b66b197
Fix that states is a dict of dicts
2018-05-22 19:02:36 +01:00
Erik Johnston
a17e901f4d
Remove unused string formatting param
2018-05-22 18:24:32 +01:00
Erik Johnston
5494c1d71e
Don't support limitless pagination
...
The pagination storage function supported not specifiying a limit on the
number of events returned. This was triggered when using the search or
context API with a limit of zero, which the storage function took to
mean not being limited.
2018-05-22 18:15:21 +01:00
Neil Johnson
d8cb7225d2
daily user type phone home stats
2018-05-22 18:09:09 +01:00
hera
ad2823ee27
fix synchrotron
2018-05-22 17:47:42 +01:00
Richard van der Hoff
08bfc48abf
custom error code for not leaving server notices room
2018-05-22 17:27:27 +01:00
David Baker
0a078026ea
comment typo
2018-05-22 17:14:06 +01:00
Amber Brown
8f5a688d42
cleanups, self-registration
2018-05-22 10:56:03 -05:00
Amber Brown
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
Erik Johnston
cb2a2ad791
get_domains_from_state returns list of tuples
2018-05-22 16:23:39 +01:00
Richard van der Hoff
08a14b32ae
Merge pull request #3262 from matrix-org/rav/has_already_consented
...
Add a 'has_consented' template var to consent forms
2018-05-22 15:35:10 +01:00
Richard van der Hoff
7b36d06a69
Add a 'has_consented' template var to consent forms
...
fixes #3260
2018-05-22 14:58:34 +01:00
Richard van der Hoff
669400e22f
Enable auto-escaping for the consent templates
...
... to reduce the risk of somebody introducing an html injection attack...
2018-05-22 14:58:34 +01:00
Richard van der Hoff
b5b2d5d64b
Fix dependency on jinja2
...
Delay the import of ConsentResource, so that we can get away without jinja2 if
people don't have the consent resource enabled.
Fixes #3259
2018-05-22 14:03:45 +01:00
Richard van der Hoff
a5e2941aad
Reject attempts to send event before privacy consent is given
...
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252 ) if consent is not yet
given.
2018-05-22 12:00:47 +01:00
Richard van der Hoff
8aeb529262
Merge pull request #3236 from matrix-org/rav/consent_notice
...
Send users a server notice about consent
2018-05-22 11:56:09 +01:00
Richard van der Hoff
8810685df9
Stub out ServerNoticesSender on the workers
...
... and have the sync endpoints call it directly rather than obsure indirection
via PresenceHandler
2018-05-22 11:54:51 +01:00
Richard van der Hoff
d5dca9a04f
Move consent config parsing into ConsentConfig
...
turns out we need to reuse this, so it's better in the config class.
2018-05-22 11:54:51 +01:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Richard van der Hoff
d14d7b8fdc
Rename 'version' param on user consent config
...
we're going to use it for the version we require too.
2018-05-22 11:54:51 +01:00
Erik Johnston
7948ecf234
Comment
2018-05-22 11:39:43 +01:00
Erik Johnston
020377a550
Fix logcontext resource usage tracking
2018-05-22 11:16:07 +01:00
Erik Johnston
13a8dfba0d
Merge pull request #3252 from matrix-org/erikj/in_flight_requests
...
Add in flight request metrics
2018-05-22 09:54:30 +01:00
Erik Johnston
c435b0b441
Don't store context
2018-05-22 09:34:26 +01:00
Erik Johnston
fb2806b186
Move in_flight_requests_count to be a callback metric
2018-05-22 09:31:53 +01:00
Amber Brown
fcc525b0b7
rest of the changes
2018-05-21 19:48:57 -05:00
Amber Brown
df9f72d9e5
replacing portions
2018-05-21 19:47:37 -05:00
Amber Brown
c60e0d5e02
don't need the resource portion
2018-05-21 17:03:20 -05:00
Amber Brown
02c1d29133
look at the Prometheus metrics instead
2018-05-21 17:02:20 -05:00
Amber Brown
f258deffcb
remove old metrics libs
2018-05-21 17:01:15 -05:00
Richard van der Hoff
413482f578
Merge pull request #3255 from matrix-org/rav/fix_transactions
...
Stop the transaction cache caching failures
2018-05-21 17:58:08 +01:00
Richard van der Hoff
6e1cb54a05
Fix logcontext leak in HttpTransactionCache
...
ONE DAY I WILL PURGE THE WORLD OF THIS EVIL
2018-05-21 16:58:20 +01:00
Richard van der Hoff
6d6e7288fe
Stop the transaction cache caching failures
...
The transaction cache has some code which tries to stop it caching failures,
but if the callback function failed straight away, then things would happen
backwards and we'd end up with the failure stuck in the cache.
2018-05-21 16:49:59 +01:00
Erik Johnston
dfa70adc33
Add in flight request metrics
...
This tracks CPU and DB usage while requests are in flight, rather than
when we write the response.
2018-05-21 16:23:06 +01:00
Adrian Tschira
933bf2dd35
replace some iteritems with six
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira
d9fe2b2d9d
Replace some more comparisons with six
...
plus a bonus b"" string I missed last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Adrian Tschira
45b55e23d3
Add batch_iter to utils
...
There's a frequent idiom I noticed where an iterable is split up into a
number of chunks/batches. Unfortunately that method does not work with
iterators like dict.keys() in python3. This implementation works with
iterators.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:48:30 +02:00
Adrian Tschira
dcc235b47d
use stand-in value if maxint is not available
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:44 +02:00
Adrian Tschira
73cbdef5f7
fix py3 intern and remove unnecessary py3 encode
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:31 +02:00
Adrian Tschira
aafb0f6b0d
py3-ize url preview
2018-05-19 17:35:20 +02:00
Adrian Tschira
b932b4ea25
use repr, not str
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:28:42 +02:00
Neil Johnson
644aac5f73
Tighter filtering for user_daily_visits
2018-05-18 17:10:35 +01:00
Neil Johnson
ef466b3a13
fix psql compatability bug
2018-05-18 15:51:21 +01:00
Neil Johnson
2725223f08
Merge branch 'master' into develop
2018-05-18 14:07:50 +01:00
Richard van der Hoff
6d9dc67139
Merge pull request #3232 from matrix-org/rav/server_notices_room
...
Infrastructure for a server notices room
2018-05-18 11:28:04 +01:00
Richard van der Hoff
ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred
...
Fix error in handling receipts
2018-05-18 11:23:11 +01:00
Richard van der Hoff
67af392712
Merge pull request #3233 from matrix-org/rav/remove_dead_code
...
Remove unused `update_external_syncs`
2018-05-18 11:22:43 +01:00
Richard van der Hoff
011e1f4010
Better docstrings
2018-05-18 11:22:12 +01:00
Richard van der Hoff
26305788fe
Make sure we reject attempts to invite the notices user
2018-05-18 11:18:39 +01:00
Richard van der Hoff
d10707c810
Replace inline docstrings with "Attributes" in class docstring
2018-05-18 11:00:55 +01:00
Erik Johnston
fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token
...
Make purge_history operate on tokens
2018-05-18 10:35:23 +01:00
Richard van der Hoff
8b1c856d81
Fix error in handling receipts
...
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):
> TypeError: argument of type 'ObservableDeferred' is not iterable
fixes #3234
2018-05-18 09:15:35 +01:00
Neil Johnson
6958459b50
bump version, change log
2018-05-17 21:35:07 +01:00
Richard van der Hoff
88d3405332
fix missing yield for server_notices_room
2018-05-17 18:33:45 +01:00
Richard van der Hoff
d43d480d86
Remove unused update_external_syncs
...
This method isn't used anywhere. Burninate it.
2018-05-17 18:22:19 +01:00
Richard van der Hoff
fed62e21ad
Infrastructure for a server notices room
...
Server Notices use a special room which the user can't dismiss. They are
created on demand when some other bit of the code calls send_notice.
(This doesn't actually do much yet becuse we don't call send_notice anywhere)
2018-05-17 17:58:25 +01:00
Richard van der Hoff
f8a1e76d64
Merge pull request #3225 from matrix-org/rav/move_creation_handler
...
Move RoomCreationHandler out of synapse.handlers.Handlers
2018-05-17 17:56:42 +01:00
Erik Johnston
f7906203f6
Merge pull request #3212 from matrix-org/erikj/epa_stream
...
Use stream rather depth ordering for push actions
2018-05-17 12:01:21 +01:00
Richard van der Hoff
c46367d0d7
Move RoomCreationHandler out of synapse.handlers.Handlers
...
Handlers is deprecated nowadays, so let's move this out before I add a new
dependency on it.
Also fix the docstrings on create_room.
2018-05-17 09:08:42 +01:00
Neil Johnson
85b8acdeb4
Merge branch 'master' into develop
2018-05-16 16:48:10 +01:00
Neil Johnson
3c099219e0
bump version and changelog for 0.29.0
2018-05-16 15:44:59 +01:00
Erik Johnston
680530cc7f
Clarify comment
2018-05-16 11:47:29 +01:00
Erik Johnston
43e6e82c4d
Comments
2018-05-16 11:13:31 +01:00
Neil Johnson
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
Erik Johnston
c945af8799
Move and rename variable
2018-05-16 10:52:06 +01:00
Neil Johnson
be11a02c4f
remove empty line
2018-05-16 10:45:40 +01:00
Neil Johnson
a2204cc9cc
remove unused method recurring_user_daily_visit_stats
2018-05-16 09:47:20 +01:00
Neil Johnson
31c2502ca8
style and further contraining query
2018-05-16 09:46:43 +01:00
Richard van der Hoff
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Neil Johnson
c92a8aa578
pep8
2018-05-15 17:31:11 +01:00
Neil Johnson
05ac15ae82
Limit query load of generate_user_daily_visits
...
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
Erik Johnston
5f27ed75ad
Make purge_history operate on tokens
...
As we're soon going to change how topological_ordering works
2018-05-15 16:23:50 +01:00
Erik Johnston
37dbee6490
Use events_to_purge table rather than token
2018-05-15 16:23:47 +01:00
Richard van der Hoff
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson
aea80a0118
v0.29.0-rc1: bump version and change log
2018-05-14 15:50:57 +01:00
Neil Johnson
f077e97914
instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day
2018-05-14 13:50:58 +01:00
David Baker
8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate
...
Part user from rooms on account deactivate
2018-05-14 11:31:48 +01:00
Neil Johnson
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
Richard van der Hoff
c25d7ba12e
Merge pull request #3208 from matrix-org/rav/more_refactor_request_handler
...
Set Server header in SynapseRequest
2018-05-11 16:07:47 +01:00
Erik Johnston
6406b70aeb
Use stream rather depth ordering for push actions
...
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Richard van der Hoff
23e2dfe940
Merge pull request #3209 from damir-manapov/master
...
transaction_id, destination defined twice
2018-05-11 00:35:13 +01:00
Damir Manapov
db18d854cd
transaction_id, destination twice
2018-05-10 22:13:31 +03:00
Richard van der Hoff
318711e139
Set Server header in SynapseRequest
...
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).
This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.
Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
2018-05-10 18:50:27 +01:00
David Baker
6b49628e3b
Catch failure to part user from room
2018-05-10 12:23:53 +01:00
David Baker
217bc53c98
Many docstrings
2018-05-10 12:20:40 +01:00
Richard van der Hoff
645cb4bf06
Remove redundant request_handler decorator
...
This is needless complexity; we might as well use the wrapper directly.
Also rename wrap_request_handler->wrap_json_request_handler.
2018-05-10 12:19:53 +01:00
Richard van der Hoff
09f570b935
Factor wrap_request_handler_with_logging out of wrap_request_handler
...
... so that it can be used on non-JSON endpoints
2018-05-10 12:19:52 +01:00
Richard van der Hoff
9589a1925e
Remove include_metrics param
...
The metrics are now available via the request, so this is redundant and can go
away at last.
2018-05-10 12:19:52 +01:00
Richard van der Hoff
49e5a613f1
Move outgoing_responses_counter handling to RequestMetrics
...
it's much neater there.
2018-05-10 12:19:52 +01:00
Richard van der Hoff
b8700dd7d0
Bump requests_counter in wrapped_request_handler
...
less magic
2018-05-10 12:19:52 +01:00
Richard van der Hoff
c6f730282c
Move RequestMetrics handling into SynapseRequest.processing()
...
It fits quite nicely here, and opens the path to getting rid of the
"include_metrics" mess.
2018-05-10 12:19:51 +01:00
Richard van der Hoff
09b29f9c4a
Make RequestMetrics take a raw time rather than a clock
...
... which is going to make it easier to move around.
2018-05-10 12:18:52 +01:00
David Baker
4d298506dd
Oops, don't call function passed to run_in_background
2018-05-10 11:57:13 +01:00
Richard van der Hoff
8460e48d06
Move request_id management into SynapseRequest
2018-05-10 11:48:17 +01:00
Richard van der Hoff
18e144fe08
Move RequestsMetrics to its own file
...
This is useful in its own right, because server.py is full of stuff; but more
importantly, I want to do some refactoring that will cause a circular reference
as it is.
2018-05-09 19:55:03 +01:00
Erik Johnston
bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync
...
Refactor sync APIs to reuse pagination API
2018-05-09 16:16:56 +01:00
Erik Johnston
5adb75bcba
Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination
...
Refactor get_recent_events_for_room return type
2018-05-09 16:07:14 +01:00
Erik Johnston
d26bec8a43
Add comment to sync as to why code path is split
2018-05-09 15:56:07 +01:00
Erik Johnston
fcf55f2255
Fix returned token is no longer a tuple
2018-05-09 15:43:00 +01:00
Erik Johnston
7ce98804ff
Fix up comment
2018-05-09 15:42:39 +01:00
Erik Johnston
cddf91c8b9
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change
2018-05-09 15:32:07 +01:00
Erik Johnston
9896dab8f6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination
2018-05-09 15:31:33 +01:00
Erik Johnston
1e5280b7d0
Merge pull request #3196 from matrix-org/erikj/pagination_return
...
Refactor pagination DB API to return concrete type
2018-05-09 15:27:17 +01:00
Erik Johnston
75552d2148
Update comments
2018-05-09 15:15:38 +01:00
David Baker
294e9a0c9b
Prefix internal functions
2018-05-09 15:10:37 +01:00
David Baker
46df23f581
Add the schema file
2018-05-09 15:07:54 +01:00
David Baker
52281e4c54
Indent fail
2018-05-09 15:06:16 +01:00
David Baker
7e8726b8fb
Part deactivated users in the background
...
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
2018-05-09 14:54:28 +01:00
Erik Johnston
c0e08dc45b
Remove unused code path from member change DB func
...
The function is never called without a from_key, so we can remove all
the handling for that scenario.
2018-05-09 14:31:32 +01:00
Erik Johnston
0461ef01b7
Merge pull request #3195 from matrix-org/erikj/pagination_refactor
...
Refactor recent events func to use pagination func
2018-05-09 14:12:24 +01:00
Erik Johnston
e2accd7f1d
Refactor sync APIs to reuse pagination API
...
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
2018-05-09 13:43:39 +01:00
Erik Johnston
e5ab9cd24b
Don't unnecessarily require token to be stream token
...
This allows calling the `get_recent_event_ids_for_room` function in more
situations.
2018-05-09 11:58:35 +01:00
Erik Johnston
c4af4c24ca
Refactor get_recent_events_for_room return type
...
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
2018-05-09 11:55:34 +01:00
Erik Johnston
05e0a2462c
Refactor pagination DB API to return concrete type
...
This makes it easier to document what is being returned by the storage
functions and what some functions expect as arguments.
2018-05-09 11:34:24 +01:00
Erik Johnston
7dd13415db
Remove unused from_token param
2018-05-09 10:58:16 +01:00
Erik Johnston
27cf170558
Refactor recent events func to use pagination func
...
This also removes a cache that is unlikely to ever get hit.
2018-05-09 10:55:55 +01:00
Erik Johnston
1aeb5e28a9
Merge pull request #3193 from matrix-org/erikj/pagination_refactor
...
Refactor /context to reuse pagination storage functions
2018-05-09 10:22:38 +01:00
Erik Johnston
23ec51c94c
Fix up comments and make function private
2018-05-09 09:55:19 +01:00
Erik Johnston
696f532453
Reuse existing pagination code for context API
2018-05-08 16:20:19 +01:00
Erik Johnston
3e6d306e94
Parse tokens before calling DB function
2018-05-08 16:18:58 +01:00
Erik Johnston
274b8c6025
Only fetch required fields from database
2018-05-08 16:15:25 +01:00
Erik Johnston
06c0d0ed08
Split paginate_room_events storage function
2018-05-08 16:14:26 +01:00
David Baker
bf98fa0864
Part user from rooms on account deactivate
...
This implements this very crudely: this probably isn't viable
because parting a user from all their rooms could take a long time,
and if the HS gets restarted in that time the process will be
aborted.
2018-05-08 15:58:35 +01:00
Richard van der Hoff
678e649b78
Merge pull request #3190 from mujx/notif-token-fix
...
notifications: Convert next_token to string according to the spec
2018-05-08 13:54:47 +01:00
Konstantinos Sideris
88868b2839
notifications: Convert next_token to string according to the spec
...
Currently the parameter is serialized as an integer.
Signed-off-by: Konstantinos Sideris <sideris.konstantin@gmail.com>
2018-05-05 12:55:02 +03:00
Erik Johnston
6d8ec3462d
Note that label values can be anything
2018-05-03 16:25:05 +01:00
Erik Johnston
95b6912045
Fix metrics that have integer value labels
2018-05-03 15:51:04 +01:00
Richard van der Hoff
093d8c415a
Merge remote-tracking branch 'origin/develop' into rav/warn_on_logcontext_fail
2018-05-03 14:59:29 +01:00
Richard van der Hoff
0ba609dc6f
Merge pull request #3183 from matrix-org/rav/moar_logcontext_leaks
...
Fix logcontext leaks in rate limiter
2018-05-03 14:58:13 +01:00
Richard van der Hoff
2117f84323
Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdown
...
Fix 'Unhandled Error' logs with Twisted 18.4
2018-05-03 12:40:11 +01:00
Richard van der Hoff
a7fe62f0cb
Fix logcontext leaks in rate limiter
2018-05-03 12:31:59 +01:00
Will Hunt
2e7a94c36b
Don't abortConnection() if the transport connection has already closed.
2018-05-03 12:31:47 +01:00
Richard van der Hoff
a2aaa9cb3c
Merge pull request #3178 from matrix-org/rav/fix_request_timeouts
...
fix http request timeout code
2018-05-03 11:33:26 +01:00
Erik Johnston
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
...
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
Erik Johnston
31c7c29d43
Fix up grammar
2018-05-03 10:38:58 +01:00
Richard van der Hoff
902673e356
Merge pull request #3161 from NotAFile/remove-v1auth
...
Make Client-Server API return 403 for invalid token
2018-05-03 10:10:57 +01:00
Erik Johnston
53a5fdf312
Merge pull request #3175 from matrix-org/erikj/escape_metric_values
...
Escape label values in prometheus metrics
2018-05-03 10:01:04 +01:00
Richard van der Hoff
1dfd650348
add missing param to cancelled_to_request_timed_out_error
...
This gets two arguments, not one.
2018-05-02 22:42:36 +01:00
Erik Johnston
a41117c63b
Make _escape_character take MatchObject
2018-05-02 17:27:27 +01:00
Erik Johnston
32015e1109
Escape label values in prometheus metrics
2018-05-02 16:52:42 +01:00
Richard van der Hoff
3a42aed9a1
Merge pull request #3170 from matrix-org/rav/more_logcontext_leaks
...
Fix a class of logcontext leaks
2018-05-02 16:45:51 +01:00
Richard van der Hoff
415c6b672e
Merge branch 'develop' into rav/more_logcontext_leaks
2018-05-02 16:16:01 +01:00
Richard van der Hoff
be31adb036
Fix logcontext leak in media repo
...
Make FileResponder.write_to_consumer uphold the logcontext contract
2018-05-02 16:14:50 +01:00
Richard van der Hoff
f22e7cda2c
Fix a class of logcontext leaks
...
So, it turns out that if you have a first `Deferred` `D1`, you can add a
callback which returns another `Deferred` `D2`, and `D2` must then complete
before any further callbacks on `D1` will execute (and later callbacks on `D1`
get the *result* of `D2` rather than `D2` itself).
So, `D1` might have `called=True` (as in, it has started running its
callbacks), but any new callbacks added to `D1` won't get run until `D2`
completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield`
will 'block'.
In conclusion: some of our assumptions in `logcontext` were invalid. We need to
make sure that we don't optimise out the logcontext juggling when this
situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
2018-05-02 11:58:00 +01:00
Richard van der Hoff
a8d8bf92e0
Merge pull request #3168 from matrix-org/rav/fix_logformatter
...
Fix incorrect reference to StringIO
2018-05-02 10:03:36 +01:00
Richard van der Hoff
e482f8cd85
Fix incorrect reference to StringIO
...
This was introduced in 4f2f5171
2018-05-02 09:12:26 +01:00
Neil Johnson
2414178ed6
Merge branch 'master' into develop
2018-05-01 18:53:56 +01:00
Neil Johnson
8570bb84cc
Update __init__.py
...
bump version
2018-05-01 18:22:53 +01:00
Richard van der Hoff
ca7211104e
Merge branch 'release-v0.28.1' into develop
2018-05-01 18:16:57 +01:00
Richard van der Hoff
d5eee5d601
Merge commit '33f469b' into release-v0.28.1
2018-05-01 18:14:18 +01:00
Richard van der Hoff
d858f3bd4e
Miscellaneous fixes to python_dependencies
...
* add some doc about wtf this thing does
* pin Twisted to < 18.4
* add explicit dep on six (fixes #3089 )
2018-05-01 18:13:54 +01:00
Richard van der Hoff
33f469ba19
Apply some limits to depth to counter abuse
...
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth
As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
2018-05-01 17:54:19 +01:00
Neil Johnson
dd1a832419
remove user agent from data model, will just join on user_ips
2018-05-01 12:13:49 +01:00
Neil Johnson
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
Neil Johnson
5917562b60
10 mins seems more reasonable that every minute
2018-05-01 12:12:22 +01:00
Adrian Tschira
6495dbb326
Burminate v1auth
...
This closes #2602
v1auth was created to account for the differences in status code between
the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid
tokens). However since those protocols were merged, this makes the r0
version/endpoint internally inconsistent, and violates the
specification for the r0 endpoint.
This might break clients that rely on this inconsistency with the
specification. This is said to affect the legacy angular reference
client. However, I feel that restoring parity with the spec is more
important. Either way, it is critical to inform developers about this
change, in case they rely on the illegal behaviour.
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-30 22:20:43 +02:00
Will Hunt
2ad3fc36e6
Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve ( #3157 )
...
fixes #3135
Signed-off-by: Will Hunt will@half-shot.uk
2018-04-30 16:21:11 +01:00
Krombel
576b71dd3d
add guard for None on purge_history api
2018-04-30 14:29:48 +02:00
Matthew Hodgson
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Richard van der Hoff
63ae5cbf34
Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn
...
Remove redundant call to preserve_fn
2018-04-30 10:23:59 +01:00
Richard van der Hoff
fdb6849b81
Merge pull request #3144 from matrix-org/rav/run_in_background_exception_handling
...
Trap exceptions thrown within run_in_background
2018-04-30 10:23:02 +01:00
Richard van der Hoff
01e8a52825
Merge pull request #3102 from NotAFile/py3-attributeerror
...
Make event properties raise AttributeError instead
2018-04-30 09:22:09 +01:00
Adrian Tschira
0c9db26260
add comment explaining attributeerror
2018-04-30 09:49:10 +02:00
Richard van der Hoff
950a32eb47
Merge pull request #3152 from NotAFile/py3-local-imports
...
make imports local
2018-04-30 01:28:13 +01:00
Richard van der Hoff
bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib
...
move httplib import to six
2018-04-30 01:26:42 +01:00
Richard van der Hoff
683149c1f9
Merge pull request #3151 from NotAFile/py3-xrange-1
...
Move more xrange to six
2018-04-30 01:20:06 +01:00
Richard van der Hoff
3b0e431c82
Merge pull request #3150 from NotAFile/py3-listcomp-yield
...
Don't yield in list comprehensions
2018-04-30 01:11:41 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
Richard van der Hoff
2fd96727b1
Merge pull request #3085 from NotAFile/py3-config-text-mode
...
Open config file in non-bytes mode
2018-04-30 01:00:23 +01:00
Richard van der Hoff
b8ee12b978
Merge pull request #3084 from NotAFile/py3-certs-byte-mode
...
Open certificate files as bytes
2018-04-30 01:00:05 +01:00
Richard van der Hoff
049b0b5af2
Merge pull request #3154 from NotAFile/py3-stringio
...
Replace stringIO imports with six
2018-04-30 00:59:04 +01:00