Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
15,003 Commits 4 Branches 615 Tags 63 MiB
470dc621ae
Commit Graph

10553 Commits

Author SHA1 Message Date
Richard van der Hoff
baa3f4a80d
Avoid deep recursion in appservice recovery (#5885) 
Hopefully, this will fix a stack overflow when recovering an appservice.

The recursion here leads to a huge chain of deferred callbacks, which then
overflows the stack when the chain completes. `inlineCallbacks` makes a better
job of this if we use iteration instead.

Clean up the code a bit too, while we're there.
 2019-08-20 17:39:38 +01:00
Jorik Schellekens
c886f976e0
Opentracing doc update (#5776) 
Update opentracing docs to use the unified 'trace' method
 2019-08-20 13:56:03 +01:00
Erik Johnston
74f016d343 Remove now unused pick_server_from_list 2019-08-20 12:37:08 +01:00
Erik Johnston
1f9df1cc7b Fixup _sort_server_list to be slightly more efficient 
Also document that we are using the algorithm described in RFC2782 and
ensure we handle zero weight correctly.
 2019-08-20 12:36:11 +01:00
Richard van der Hoff
5019945828 Refactor the Appservice scheduler code 
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).

Add some docstrings too.
 2019-08-20 11:50:23 +01:00
Erik Johnston
1dec31560e Change jitter to be a factor rather than absolute value 2019-08-20 11:46:00 +01:00
Olivier Wilkinson (reivilibre)
bb29bc2937 Use MSC2197 on stable prefix as it has almost finished FCP 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-20 08:49:31 +01:00
Erik Johnston
d514dac0b2
Merge pull request #5860 from matrix-org/erikj/update_5704_comments 
Remove logging for #5407 and update comments
 2019-08-19 10:20:59 +01:00
Brendan Abolivier
bdd201ea7f Merge branch 'master' into develop 2019-08-17 10:50:42 +01:00
Richard van der Hoff
74fb729213 1.3.1 2019-08-17 09:16:17 +01:00
Richard van der Hoff
412c6e21a8
Drop dependency on sdnotify (#5871) 
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
 2019-08-17 09:09:52 +01:00
Hubert Chathi
8a5f6ed130
Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index 
add the version field to the index for e2e_room_keys
 2019-08-16 17:45:50 -07:00
Chris Moos
6d86df73f1 Fix issue with Synapse not starting up. Fixes #5866. 
Signed-off-by: Chris Moos <chris@chrismoos.com>
 2019-08-16 22:16:13 +01:00
Jorik Schellekens
87fa26006b
Opentracing misc (#5856) 
Add authenticated_entity and servlet_names tags.

Functionally:
- Add a tag for authenticated_entity
- Add a tag for servlet_names

Stylistically:
Moved to importing methods directly from opentracing.
 2019-08-16 16:13:25 +01:00
Erik Johnston
861d663c15 Fixup changelog and remove debug logging 2019-08-16 13:15:26 +01:00
Andrew Morgan
b13cac896d
Fix up password reset template config names (#5863) 
Fixes #5833

The emailconfig code was attempting to pull incorrect config file names. This corrects that, while also marking a difference between a config file variable that's a filepath versus a str containing HTML.
 2019-08-15 16:27:11 +01:00
Erik Johnston
f299c5414c Refactor MatrixFederationAgent to retry SRV. 
This refactors MatrixFederationAgent to move the SRV lookup into the
endpoint code, this has two benefits:
	1. Its easier to retry different host/ports in the same way as
	   HostnameEndpoint.
	2. We avoid SRV lookups if we have a free connection in the pool
 2019-08-15 15:43:22 +01:00
Brendan Abolivier
ce5f1cb98c Merge branch 'master' into develop 2019-08-15 12:38:21 +01:00
Brendan Abolivier
fb5acd7039 1.3.0 2019-08-15 12:05:24 +01:00
Erik Johnston
748aa38378 Remove logging for #5407 and update comments 2019-08-15 12:02:18 +01:00
Olivier Wilkinson (reivilibre)
2253b083d9 Add support for inbound MSC2197 requests on unstable Federation API 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-15 11:06:21 +01:00
Olivier Wilkinson (reivilibre)
6fadb560fc Support MSC2197 outbound with unstable prefix 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-15 10:59:37 +01:00
Michael Telatynski
baee288fb4 Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633) 2019-08-15 09:45:57 +01:00
Erik Johnston
e6e136decc Retry well known on fail. 
If we have recently seen a valid well-known for a domain we want to
retry on (non-final) errors a few times, to handle temporary blips in
networking/etc.
 2019-08-15 09:28:58 +01:00
Hubert Chathi
c058aeb88d update set_e2e_room_key to agree with fixed index 2019-08-14 18:02:58 -07:00
Hubert Chathi
b7f7cc7ace add the version field to the index for e2e_room_keys 2019-08-14 17:14:40 -07:00
Olivier Wilkinson (reivilibre)
1b63ccd848 Wrap get_local_public_room_list call in maybeDeferred because it 
is cached and so does not always return a `Deferred`.
`await` does not silently pass-through non-Deferreds like `yield` used to.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-14 14:53:49 +01:00
Erik Johnston
09f6152a11
Merge pull request #5844 from matrix-org/erikj/retry_well_known_lookup 
Retry well-known lookup before expiry.
 2019-08-14 09:53:33 +01:00
Brendan Abolivier
f70d0a1dd9 1.3.0rc1 2019-08-13 18:20:09 +01:00
Amber H. Brown
18bdac8ee4 fix config being a dict, actually 2019-08-14 02:06:42 +10:00
Erik Johnston
17e1e80726 Retry well-known lookup before expiry. 
This gives a bit of a grace period where we can attempt to refetch a
remote `well-known`, while still using the cached result if that fails.

Hopefully this will make the well-known resolution a bit more torelant
of failures, rather than it immediately treating failures as "no result"
and caching that for an hour.
 2019-08-13 16:20:38 +01:00
Erik Johnston
af187805b3
Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop 
Handle pusher being deleted during processing.
 2019-08-13 14:08:29 +01:00
Erik Johnston
96bdd661b8
Remove redundant return 2019-08-13 12:50:36 +01:00
Amber Brown
0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Erik Johnston
e9906b0772
Merge pull request #5836 from matrix-org/erikj/lower_bound_ttl_well_known 
Add a lower bound to well-known TTL.
 2019-08-13 12:41:16 +01:00
Erik Johnston
fb3469f53a
Clarify docstring 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-08-13 10:17:23 +01:00
Erik Johnston
f218705d2a Make default well known cache global again. 2019-08-13 10:06:51 +01:00
Erik Johnston
2546f32b90
Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses 
Don't unnecessarily block notifying of new events.
 2019-08-13 09:36:25 +01:00
Richard van der Hoff
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions 
Fix handling of redactions of redactions
 2019-08-12 12:25:19 +01:00
Erik Johnston
107ad133fc Move well known lookup into a separate clas 2019-08-07 15:36:38 +01:00
Erik Johnston
af9f1c0764 Add a lower bound for TTL on well known results. 
It costs both us and the remote server for us to fetch the well known
for every single request we send, so we add a minimum cache period. This
is set to 5m so that we still honour the basic premise of "refetch
frequently".
 2019-08-06 17:01:23 +01:00
Erik Johnston
d1b5b055be
Merge pull request #5825 from matrix-org/erikj/fix_empty_limited_sync 
Handle TimelineBatch being limited and empty.
 2019-08-06 15:39:44 +01:00
Andrew Morgan
edeae53221
Return 404 instead of 403 when retrieving an event without perms (#5798) 
Part of fixing matrix-org/sytest#652

Sytest PR: matrix-org/sytest#667
 2019-08-06 13:33:55 +01:00
Erik Johnston
bf4db42920 Don't unnecessarily block notifying of new events. 
When persisting events we calculate new stream orderings up front.
Before we notify about an event all events with lower stream orderings
must have finished being persisted.

This PR moves the assignment of stream ordering till *after* calculated
the new current state and split the batch of events into separate chunks
for persistence. This means that if it takes a long time to calculate
new current state then it will not block events in other rooms being
notified about.

This should help reduce some global pauses in the events stream which
can last for tens of seconds (if not longer), caused by some
particularly expensive state resolutions.
 2019-08-06 13:32:02 +01:00
Erik Johnston
6881f21f3e Handle TimelineBatch being limited and empty. 
This hopefully addresses #5407 by gracefully handling an empty but
limited TimelineBatch. We also add some logging to figure out how this
is happening.
 2019-08-06 12:59:00 +01:00
Brendan Abolivier
8ed9e63432
Account validity: allow defining HTML templates to serve the us… (#5807) 
Account validity: allow defining HTML templates to serve the user on account renewal attempt
 2019-08-01 16:09:25 +02:00
Erik Johnston
d55bc4a8bf
Merge pull request #5810 from matrix-org/erikj/no_server_reachable 
Return 502 not 500 when failing to reach any remote server.
 2019-08-01 14:19:39 +01:00
Andrew Morgan
5d018d23f0
Have ClientReaderSlavedStore inherit RegistrationStore (#5806) 
Fixes #5803
 2019-08-01 13:54:56 +01:00
Erik Johnston
3c076c79c5
Merge pull request #5808 from matrix-org/erikj/parse_decode_error 
Handle incorrectly encoded query params correctly
 2019-08-01 13:48:10 +01:00
Erik Johnston
a8f40a8302 Return 502 not 500 when failing to reach any remote server. 2019-08-01 13:47:31 +01:00
Erik Johnston
55a0c98d16
Merge pull request #5805 from matrix-org/erikj/validate_state 
Validate well known state events are state events.
 2019-08-01 13:45:48 +01:00
Erik Johnston
0b36decfb6
Merge pull request #5801 from matrix-org/erikj/recursive_tombstone 
Don't allow clients to send tombstones that reference the same room
 2019-08-01 13:45:35 +01:00
Erik Johnston
d02e41dcb2 Handle pusher being deleted during processing. 
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
 2019-08-01 13:44:12 +01:00
Erik Johnston
d2e3d5b9db Handle incorrectly encoded query params correctly 2019-08-01 13:23:00 +01:00
Erik Johnston
76a58fdcce Fix spelling. 
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2019-08-01 13:17:55 +01:00
Erik Johnston
58af30a6c7
Merge pull request #5802 from matrix-org/erikj/deny_redacting_different_room 
Deny redaction of events in a different room.
 2019-08-01 13:14:46 +01:00
Erik Johnston
0f632f3a57
Merge pull request #5790 from matrix-org/erikj/groups_request_errors 
Handle RequestSendFailed exception correctly in more places.
 2019-08-01 13:14:08 +01:00
Brendan Abolivier
f25f638c35
Lint 2019-08-01 12:19:08 +02:00
Brendan Abolivier
a4a9ded4d0
Allow defining HTML templates to serve the user on account renewal 2019-08-01 11:59:27 +02:00
Erik Johnston
dc4d74e44a Validate well-known state events are state events. 
Lets disallow sending things like memberships, topics etc as non-state
events.
 2019-07-31 16:36:20 +01:00
Erik Johnston
2e697d3013 Explicitly check that tombstone is a state event before notifying. 2019-07-31 16:32:03 +01:00
Erik Johnston
cf89266b98 Deny redaction of events in a different room. 
We already correctly filter out such redactions, but we should also deny
them over the CS API.
 2019-07-31 16:12:27 +01:00
Erik Johnston
f31d4cb7a2 Don't allow clients to send tombstones that reference the same room 2019-07-31 15:52:27 +01:00
Andrew Morgan
72167fb394
Change user deactivated errcode to USER_DEACTIVATED and use it (#5686) 
This is intended as an amendment to #5674 as using M_UNKNOWN as the errcode makes it hard for clients to differentiate between an invalid password and a deactivated user (the problem we were trying to solve in the first place).

M_UNKNOWN was originally chosen as it was presumed than an MSC would have to be carried out to add a new code, but as Synapse often is the testing bed for new MSC implementations, it makes sense to try it out first in the wild and then add it into the spec if it is successful. Thus this PR return a new M_USER_DEACTIVATED code when a deactivated user attempts to login.
 2019-07-31 15:19:06 +01:00
Andrew Morgan
58a755cdc3 Remove duplicate return statement 2019-07-31 13:24:51 +01:00
Erik Johnston
a9bcae9f50 Share SSL options for well-known requests 2019-07-31 10:39:24 +01:00
Erik Johnston
123c04daa7 Don't recreate column 2019-07-30 16:42:48 +01:00
Erik Johnston
15056ca208 Fix current_state_events membership background update. 
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
 2019-07-30 14:51:41 +01:00
Erik Johnston
b4d5ff0af7 Don't log as exception when failing durig backfill 2019-07-30 13:19:22 +01:00
Erik Johnston
1ec7d656dd Unwrap error 2019-07-30 13:09:02 +01:00
Erik Johnston
458e51df7a Fix error handling when fetching remote device keys 2019-07-30 13:07:02 +01:00
Richard van der Hoff
8c97f6414c
Remove non-functional 'expire_access_token' setting (#5782) 
The `expire_access_token` didn't do what it sounded like it should do. What it
actually did was make Synapse enforce the 'time' caveat on macaroons used as
access tokens, but since our access token macaroons never contained such a
caveat, it was always a no-op.

(The code to add 'time' caveats was removed back in v0.18.5, in #1656)
 2019-07-30 08:25:02 +01:00
Richard van der Hoff
4e97eb89e5 Handle loops in redaction events 2019-07-30 00:00:34 +01:00
Richard van der Hoff
448bcfd0f9 recursively fetch redactions 2019-07-30 00:00:34 +01:00
Richard van der Hoff
e6a6c4fbab split _get_events_from_db out of _enqueue_events 2019-07-29 23:15:15 +01:00
Richard van der Hoff
c9964ba600 Return dicts from _fetch_event_list 2019-07-29 23:15:15 +01:00
Amber Brown
865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Erik Johnston
aecae8f397 Correctly handle errors doing requests to group servers 2019-07-29 17:21:57 +01:00
Erik Johnston
7c8c3b8437
Merge pull request #5774 from matrix-org/erikj/fix_rejected_membership 
Fix room summary when rejected events are in state
 2019-07-29 17:15:15 +01:00
Erik Johnston
3e013b7c8e
Merge pull request #5752 from matrix-org/erikj/forgotten_user 
Remove some more joins on room_memberships
 2019-07-29 17:15:01 +01:00
Erik Johnston
2a12d76646
Merge pull request #5770 from matrix-org/erikj/fix_current_state_event_sqlite 
Fix current_state bg update to work on old SQLite
 2019-07-29 17:09:01 +01:00
Amber Brown
97a8b4caf7
Move some timeout checking logs to DEBUG #5785 2019-07-30 02:02:18 +10:00
Erik Johnston
df3a5db629 Expand comment 2019-07-29 16:40:25 +01:00
Jorik Schellekens
85b0bd8fe0 Update the device list cache when keys/query is called (#5693) 2019-07-29 16:34:44 +01:00
Erik Johnston
105e7f6ed3 Remove lost comment 2019-07-29 16:09:48 +01:00
Erik Johnston
45df38e61b Fix current_state bg update to work on old SQLite 2019-07-29 13:04:10 +01:00
Brendan Abolivier
244953be3f
Add kwargs and doc 2019-07-29 10:03:14 +02:00
Brendan Abolivier
08352d44f8
Add ability to pass arguments to looping calls 2019-07-29 09:54:37 +02:00
Richard van der Hoff
d74595e2ca Merge branch 'master' into develop 2019-07-26 12:39:33 +01:00
Richard van der Hoff
1a93daf353
Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch 
Log when we receive a /make_* request from a different origin
 2019-07-26 12:38:37 +01:00
Richard van der Hoff
dde6ea7ff6 1.2.1 2019-07-26 11:33:16 +01:00
Erik Johnston
14c24c9037 Fix room summary when rejected events are in state 
Annoyingly, `current_state_events` table can include rejected events,
in which case the membership column will be null. To work around this
lets just always filter out null membership for now.
 2019-07-26 10:11:36 +01:00
Richard van der Hoff
a0ee2ec458 Merge branch 'erikj/log_leave_origin_mismatch' into release-v1.2.1 2019-07-26 10:09:36 +01:00
Richard van der Hoff
d1020653fc Log when we receive a /make_* request from a different origin 2019-07-26 10:08:22 +01:00
Erik Johnston
1f8bae7724 Log when we receive receipt from a different origin 2019-07-26 07:55:25 +01:00
Richard van der Hoff
0f2ecb961e Fix DoS when there is a cycle in redaction events 
Make sure that synapse doesn't explode when a redaction redacts itself, or
there is a larger cycle.
 2019-07-26 06:36:48 +00:00
Richard van der Hoff
26d742fed6
Merge pull request #5767 from matrix-org/rav/redactions/cross_room_id 
log when a redaction attempts to redact an event in a different room
 2019-07-25 18:49:56 +01:00
Richard van der Hoff
b1605cdd23 log when a redaction attempts to redact an event in a different room 2019-07-25 18:26:20 +01:00
Richard van der Hoff
618bd1ee76
Fix some error cases in the caching layer. (#5749) 
There was some inconsistent behaviour in the caching layer around how
exceptions were handled - particularly synchronously-thrown ones.

This seems to be most easily handled by pushing the creation of
ObservableDeferreds down from CacheDescriptor to the Cache.
 2019-07-25 15:59:45 +01:00
Andrew Morgan
f16aa3a44b Merge branch 'master' into develop 2019-07-25 15:19:22 +01:00
Andrew Morgan
c0a1301ccd 1.2.0 2019-07-25 14:10:32 +01:00
Andrew Morgan
baf081cd3b Bugfixes 
--------
 
 - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl04Ur0THGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9F4oD/0TY6S/SEd2uAmzor64ojmbX5BOwPzf
 j/wzUTrfvuf40EvkNPDpnejNZSvy/ysbaGQaQusv0SQKlV3xrvdn4RuMvnOWVWck
 kBsO+lvzOaUTR0KHDxN4y9F5eI2NdPbub4847PPVzyqSIHAd+kolxXS8kSBBhwpL
 yfaICWV/AOy5L7xN+JZ9IQpnegVAvUj5DmgXzDHd6VdeiHDVJuARaBgrR5uCkwVS
 ZoLRqZ95XV/qiguMAUvPOwyEqht2mwO64989MswP16YYm8oMkB5QA6I5nYnACsTP
 qk9YcN/oNvEfQXUhttku6MxK1/4yUMPUhEoDBDH7ebc0440QDtWN+IHTdA6oPVZB
 IuStL9YGY16m7Ltx37ZUA4URfNMiSeLHo3zKc/mCAcwxN4HyOjJewtxbG5zKQAOZ
 SMs8UcDwGR4zL1hnt8ZDNYtWwfzJBQIdGjoHvjXJEY7/1csTv2lmAwewFTXiqSAr
 30GW5ews94kotqBK53zZT6V0F5gHNqgGHniOz1ZpqLLxYLqO3LSAGe97CrqlWUdX
 GkhA9tZyweknociD9fyyBmKdcFJ4mL4a+oGI5CMnSMph8UvCY8Y5XMb1T+iYEABI
 tA9G3mBvgkLPj+5V+8QggNkBafSigW2Q4FX7enGsDmiiskZOtfeKrAcVkapD4ooi
 3I7IW5aetZr2IQ==
 =+JBn
 -----END PGP SIGNATURE-----

Merge tag 'v1.2.0rc2' into develop

Bugfixes
--------

- Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
 2019-07-24 13:47:51 +01:00
Andrew Morgan
2d573e2e2b 1.2.0rc2 2019-07-24 13:43:40 +01:00
Erik Johnston
2276936bac
Merge pull request #5743 from matrix-org/erikj/log_origin_receipts_mismatch 
Log when we receive receipt from a different origin
 2019-07-24 13:27:57 +01:00
Richard van der Hoff
f30a71a67b
Stop trying to fetch events with event_id=None. (#5753) 
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.

I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
 2019-07-24 13:16:18 +01:00
Jorik Schellekens
cf2972c818
Fix servlet metric names (#5734) 
* Fix servlet metric names

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Remove redundant check

* Cover all return paths
 2019-07-24 13:07:35 +01:00
Erik Johnston
0c4a99607e Remove join when calculating room summaries. 2019-07-24 11:49:15 +01:00
Erik Johnston
62921fb53e Remove join on room_memberships when fetching rooms for user. 2019-07-24 11:45:58 +01:00
Erik Johnston
32768e96d4 Add function to get all forgotten rooms for user 
This will allow us to efficiently filter out rooms that have been
forgotten in other queries without having to join against the
`room_memberships` table.
 2019-07-24 11:44:23 +01:00
Richard van der Hoff
418635e68a
Add a prometheus metric for active cache lookups. (#5750) 
* Add a prometheus metric for active cache lookups.

* changelog
 2019-07-24 11:33:13 +01:00
Jorik Schellekens
3641784e8c
Make Jaeger fully configurable (#5694) 
* Allow Jaeger to be configured

* Update sample config
 2019-07-23 15:46:04 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Jorik Schellekens
18a466b84e
Opentracing Utils (#5722) 
* Add decerators for tracing functions

* Use the new clean contexts

* Context and edu utils

* Move opentracing setters

* Move whitelisting

* Sectioning comments

* Better args wrapper

* Docstrings

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Remove unused methods.

* Don't use global

* One tracing decorator to rule them all.
 2019-07-23 13:31:16 +01:00
Erik Johnston
3db1377b26 Log when we receive receipt from a different origin 2019-07-23 13:31:03 +01:00
Erik Johnston
841b12867e
Merge pull request #5732 from matrix-org/erikj/sdnotify 
Add process hooks to tell systemd our state.
 2019-07-23 13:06:53 +01:00
Erik Johnston
b2a629ef49 Speed up current state background update. 
Turns out that storing huge JSON arrays in the progress JSON isn't
something that postgres particularly likes.
 2019-07-23 10:01:30 +01:00
Amber Brown
0d0f6d12bc Fix logging in workers (#5729) 
This also adds a worker blacklist.
 2019-07-22 16:05:00 +01:00
Erik Johnston
80cfad233e Call startup commands as system triggers. 
This helps ensures that we only consider ourselves "up" once all the
startup functions have completed.
 2019-07-22 15:22:14 +01:00
Erik Johnston
720d30469f
Merge pull request #5730 from matrix-org/erikj/cache_versions 
Cache get_version_string.
 2019-07-22 14:52:52 +01:00
Erik Johnston
c560b791e1 Add process hooks to tell systemd our state. 
Fixes #5676.
 2019-07-22 14:52:18 +01:00
Jason Robinson
8e513e7afc
Merge pull request #5731 from matrix-org/jaywink/admin-user-list-user-type 
Add `user_type` to returned fields in admin API user list endpoints
 2019-07-22 16:28:51 +03:00
Richard van der Hoff
0cb72812f9
Fix stack overflow in Keyring (#5724) 
* Refactor Keyring._start_key_lookups

There's an awful lot of deferreds and dictionaries flying around here. The
whole thing can be made much simpler and achieve the same effect.

* Add a delay to key lookup lock release to fix stack overflow

A tactical call_later here should fix #5723

* changelog
 2019-07-22 13:51:22 +01:00
Andrew Morgan
f477ce4b1a v1.2.0rc1 
Features
 --------
 
 - Add support for opentracing. ([\#5544](https://github.com/matrix-org/synapse/issues/5544), [\#5712](https://github.com/matrix-org/synapse/issues/5712))
 - Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](https://github.com/matrix-org/synapse/issues/5589))
 - Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](https://github.com/matrix-org/synapse/issues/5597))
 - Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](https://github.com/matrix-org/synapse/issues/5613))
 - Add default push rule to ignore reactions. ([\#5623](https://github.com/matrix-org/synapse/issues/5623))
 - Include the original event when asking for its relations. ([\#5626](https://github.com/matrix-org/synapse/issues/5626))
 - Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](https://github.com/matrix-org/synapse/issues/5660))
 - Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](https://github.com/matrix-org/synapse/issues/5674))
 - Enable aggregations support by default ([\#5714](https://github.com/matrix-org/synapse/issues/5714))
 
 Bugfixes
 --------
 
 - Fix 'utime went backwards' errors on daemonization. ([\#5609](https://github.com/matrix-org/synapse/issues/5609))
 - Various minor fixes to the federation request rate limiter. ([\#5621](https://github.com/matrix-org/synapse/issues/5621))
 - Forbid viewing relations on an event once it has been redacted. ([\#5629](https://github.com/matrix-org/synapse/issues/5629))
 - Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](https://github.com/matrix-org/synapse/issues/5638))
 - Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](https://github.com/matrix-org/synapse/issues/5644))
 - Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](https://github.com/matrix-org/synapse/issues/5654))
 - Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](https://github.com/matrix-org/synapse/issues/5658))
 - Fix some problems with authenticating redactions in recent room versions. ([\#5699](https://github.com/matrix-org/synapse/issues/5699), [\#5700](https://github.com/matrix-org/synapse/issues/5700), [\#5707](https://github.com/matrix-org/synapse/issues/5707))
 - Ignore redactions of m.room.create events. ([\#5701](https://github.com/matrix-org/synapse/issues/5701))
 
 Updates to the Docker image
 ---------------------------
 
 - Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](https://github.com/matrix-org/synapse/issues/5619))
 - Add missing space in default logging file format generated by the Docker image. ([\#5620](https://github.com/matrix-org/synapse/issues/5620))
 
 Improved Documentation
 ----------------------
 
 - Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](https://github.com/matrix-org/synapse/issues/5397))
 - --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](https://github.com/matrix-org/synapse/issues/5651))
 - Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](https://github.com/matrix-org/synapse/issues/5661))
 - Minor tweaks to postgres documentation. ([\#5675](https://github.com/matrix-org/synapse/issues/5675))
 
 Deprecations and Removals
 -------------------------
 
 - Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](https://github.com/matrix-org/synapse/issues/5625))
 
 Internal Changes
 ----------------
 
 - Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](https://github.com/matrix-org/synapse/issues/5606), [\#5617](https://github.com/matrix-org/synapse/issues/5617))
 - Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](https://github.com/matrix-org/synapse/issues/5611))
 - Make runtime errors surrounding password reset emails much clearer. ([\#5616](https://github.com/matrix-org/synapse/issues/5616))
 - Remove dead code for persiting outgoing federation transactions. ([\#5622](https://github.com/matrix-org/synapse/issues/5622))
 - Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](https://github.com/matrix-org/synapse/issues/5627))
 - Move RegistrationHandler.get_or_create_user to test code. ([\#5628](https://github.com/matrix-org/synapse/issues/5628))
 - Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](https://github.com/matrix-org/synapse/issues/5630))
 - Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](https://github.com/matrix-org/synapse/issues/5636))
 - Unblacklist some user_directory sytests. ([\#5637](https://github.com/matrix-org/synapse/issues/5637))
 - Factor out some redundant code in the login implementation. ([\#5639](https://github.com/matrix-org/synapse/issues/5639))
 - Update ModuleApi to avoid register(generate_token=True). ([\#5640](https://github.com/matrix-org/synapse/issues/5640))
 - Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](https://github.com/matrix-org/synapse/issues/5641))
 - Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](https://github.com/matrix-org/synapse/issues/5642))
 - Improve logging for auto-join when a new user is created. ([\#5643](https://github.com/matrix-org/synapse/issues/5643))
 - Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](https://github.com/matrix-org/synapse/issues/5645))
 - Fix a small typo in a code comment. ([\#5655](https://github.com/matrix-org/synapse/issues/5655))
 - Clean up exception handling around client access tokens. ([\#5656](https://github.com/matrix-org/synapse/issues/5656))
 - Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](https://github.com/matrix-org/synapse/issues/5657))
 - Inline issue_access_token. ([\#5659](https://github.com/matrix-org/synapse/issues/5659))
 - Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](https://github.com/matrix-org/synapse/issues/5664))
 - Add a `docker` type to the towncrier configuration. ([\#5673](https://github.com/matrix-org/synapse/issues/5673))
 - Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](https://github.com/matrix-org/synapse/issues/5689))
 - Documentation for opentracing. ([\#5703](https://github.com/matrix-org/synapse/issues/5703))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl01rA4THGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9GYqD/sHgmTXL33/TL6ytdWfcrngo3+zyqz3
 eMTvLggwFojTEzE2pO9R3UsaEZI+gMZEKfm+EEU4dXEnXxRxppMniHX3mMball7Q
 Xz7TgmediE6nW+m8FDbAuHsMYa7GY3p+u5ZyleO78zIjIu8YahcpKwX40JCJNCE1
 Vzz1T24i+lIsGB+3ahzabxkG1QHMppfClEFEpC8jYb+DL8BQzdMl0HAvdw4tAbDW
 ONPAdGkjtPUv6TcfBDXoMXVdoAA4CFyDuCWR0O57I3Re5b7ARy6Dx0vcQbMCTEgQ
 GnT1/QWtrpkX04OezMre2FAI+VQpQu/Hlhrl3+h8uQT1jBqYpWfR9jy/Y/iJWH0a
 Dam0XzvZrMM1wrzd6exLMWS76W8EWKT9IV7xR9kf4HwnOxGrmusshzkCEUAxwxJX
 jmFfTUw2f0yKt6lfNWaa67wP3suOlb++g33becFMKEM2DIb3iFxdk8U9p50406fL
 asCfMsNWNKrcIkAnsgvPXfZMhUX0bgHBtkiZxTZ8/cJK01N+9pVvlb6ZZ7j30aIH
 r4I6MAfaA57QOIla9Ml3LrT49dRskJpbReIqPiPxfH47W7hZhaCt0mF95SXzogF0
 cP6t36ths/ywJ85EQUYY+9VgYK26jObm2xFTN2rUnZ1hnCEhKY4/Lwz0jPRQ8au3
 2XHFZfqeAVpvvw==
 =ZrnB
 -----END PGP SIGNATURE-----

Merge tag 'v1.2.0rc1' into develop

v1.2.0rc1

Features
--------

- Add support for opentracing. ([\#5544](https://github.com/matrix-org/synapse/issues/5544), [\#5712](https://github.com/matrix-org/synapse/issues/5712))
- Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](https://github.com/matrix-org/synapse/issues/5589))
- Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](https://github.com/matrix-org/synapse/issues/5597))
- Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](https://github.com/matrix-org/synapse/issues/5613))
- Add default push rule to ignore reactions. ([\#5623](https://github.com/matrix-org/synapse/issues/5623))
- Include the original event when asking for its relations. ([\#5626](https://github.com/matrix-org/synapse/issues/5626))
- Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](https://github.com/matrix-org/synapse/issues/5660))
- Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](https://github.com/matrix-org/synapse/issues/5674))
- Enable aggregations support by default ([\#5714](https://github.com/matrix-org/synapse/issues/5714))

Bugfixes
--------

- Fix 'utime went backwards' errors on daemonization. ([\#5609](https://github.com/matrix-org/synapse/issues/5609))
- Various minor fixes to the federation request rate limiter. ([\#5621](https://github.com/matrix-org/synapse/issues/5621))
- Forbid viewing relations on an event once it has been redacted. ([\#5629](https://github.com/matrix-org/synapse/issues/5629))
- Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](https://github.com/matrix-org/synapse/issues/5638))
- Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](https://github.com/matrix-org/synapse/issues/5644))
- Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](https://github.com/matrix-org/synapse/issues/5654))
- Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](https://github.com/matrix-org/synapse/issues/5658))
- Fix some problems with authenticating redactions in recent room versions. ([\#5699](https://github.com/matrix-org/synapse/issues/5699), [\#5700](https://github.com/matrix-org/synapse/issues/5700), [\#5707](https://github.com/matrix-org/synapse/issues/5707))
- Ignore redactions of m.room.create events. ([\#5701](https://github.com/matrix-org/synapse/issues/5701))

Updates to the Docker image
---------------------------

- Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](https://github.com/matrix-org/synapse/issues/5619))
- Add missing space in default logging file format generated by the Docker image. ([\#5620](https://github.com/matrix-org/synapse/issues/5620))

Improved Documentation
----------------------

- Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](https://github.com/matrix-org/synapse/issues/5397))
- --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](https://github.com/matrix-org/synapse/issues/5651))
- Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](https://github.com/matrix-org/synapse/issues/5661))
- Minor tweaks to postgres documentation. ([\#5675](https://github.com/matrix-org/synapse/issues/5675))

Deprecations and Removals
-------------------------

- Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](https://github.com/matrix-org/synapse/issues/5625))

Internal Changes
----------------

- Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](https://github.com/matrix-org/synapse/issues/5606), [\#5617](https://github.com/matrix-org/synapse/issues/5617))
- Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](https://github.com/matrix-org/synapse/issues/5611))
- Make runtime errors surrounding password reset emails much clearer. ([\#5616](https://github.com/matrix-org/synapse/issues/5616))
- Remove dead code for persiting outgoing federation transactions. ([\#5622](https://github.com/matrix-org/synapse/issues/5622))
- Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](https://github.com/matrix-org/synapse/issues/5627))
- Move RegistrationHandler.get_or_create_user to test code. ([\#5628](https://github.com/matrix-org/synapse/issues/5628))
- Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](https://github.com/matrix-org/synapse/issues/5630))
- Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](https://github.com/matrix-org/synapse/issues/5636))
- Unblacklist some user_directory sytests. ([\#5637](https://github.com/matrix-org/synapse/issues/5637))
- Factor out some redundant code in the login implementation. ([\#5639](https://github.com/matrix-org/synapse/issues/5639))
- Update ModuleApi to avoid register(generate_token=True). ([\#5640](https://github.com/matrix-org/synapse/issues/5640))
- Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](https://github.com/matrix-org/synapse/issues/5641))
- Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](https://github.com/matrix-org/synapse/issues/5642))
- Improve logging for auto-join when a new user is created. ([\#5643](https://github.com/matrix-org/synapse/issues/5643))
- Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](https://github.com/matrix-org/synapse/issues/5645))
- Fix a small typo in a code comment. ([\#5655](https://github.com/matrix-org/synapse/issues/5655))
- Clean up exception handling around client access tokens. ([\#5656](https://github.com/matrix-org/synapse/issues/5656))
- Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](https://github.com/matrix-org/synapse/issues/5657))
- Inline issue_access_token. ([\#5659](https://github.com/matrix-org/synapse/issues/5659))
- Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](https://github.com/matrix-org/synapse/issues/5664))
- Add a `docker` type to the towncrier configuration. ([\#5673](https://github.com/matrix-org/synapse/issues/5673))
- Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](https://github.com/matrix-org/synapse/issues/5689))
- Documentation for opentracing. ([\#5703](https://github.com/matrix-org/synapse/issues/5703))
 2019-07-22 13:49:16 +01:00
Jason Robinson
66f5ff72fd Add user_type to returned fields in admin API user list endpoints 
Mostly user type will be empty (normal user) but there is also the
"support" user type.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-07-22 15:29:18 +03:00
Erik Johnston
5ea773c505 Cache get_version_string. 
The version of a module isn't going to change over the lifetime of the
process (assuming no funky hot reloading is going on, which it isn't),
so let's just cache the result to avoid spawning lots of git
subprocesses.

Fixes #5672.
 2019-07-22 13:15:08 +01:00
Andrew Morgan
54437c48ca 1.2.0rc1 2019-07-22 12:59:04 +01:00
Jorik Schellekens
0fd171770a Merge branch 'release-v1.2.0' into develop 2019-07-22 11:18:50 +01:00
Jorik Schellekens
826e6ec3bd
Opentracing Documentation (#5703) 
* Opentracing survival guide

* Update decorator names in doc

* Doc cleanup

These are all alterations as a result of comments in #5703, it
includes mostly typos and clarifications. The most interesting
changes are:

- Split developer and user docs into two sections
- Add a high level description of OpenTracing

* newsfile

* Move contributer specific info to docstring.

* Sample config.

* Trailing whitespace.

* Update 5703.misc

* Apply suggestions from code review

Mostly just rewording parts of the docs for clarity.

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-07-22 11:15:21 +01:00
Richard van der Hoff
dcca56baba Add a delay to key lookup lock release to fix stack overflow 
A tactical call_later here should fix #5723
 2019-07-19 17:57:00 +01:00
Richard van der Hoff
c7095be913 Refactor Keyring._start_key_lookups 
There's an awful lot of deferreds and dictionaries flying around here. The
whole thing can be made much simpler and achieve the same effect.
 2019-07-19 17:49:19 +01:00
Erik Johnston
7704873cb8
Merge pull request #5720 from matrix-org/erikj/transactions_upsert 
Use upsert when updating destination retry interval
 2019-07-19 16:51:16 +01:00
Erik Johnston
d7bd9651bc
Merge pull request #5713 from matrix-org/erikj/use_cache_for_filtered_state 
Delegate to cached version when using get_filtered_current_state_ids
 2019-07-19 16:30:49 +01:00
Erik Johnston
5c07c97c09
Merge pull request #5706 from matrix-org/erikj/add_memberships_to_current_state 
Add membership column to current_state_events table
 2019-07-19 16:30:33 +01:00
Jorik Schellekens
7b8bc61834
Don't accept opentracing data from clients. (#5715) 
* Don't accept opentracing data from clients.

* newsfile
 2019-07-19 16:29:57 +01:00
Erik Johnston
2410335507 Use upsert when updating destination retry interval 2019-07-19 13:40:24 +01:00
Erik Johnston
bd2e1a2aa8 LoggingTransaction accepts None for callback lists. 
Its a bit disingenuousto give LoggingTransaction lists to append
callbacks to if we're not going to run the callbacks.
 2019-07-19 13:36:04 +01:00
Erik Johnston
ebc5ed1296 Update comment for new column 2019-07-19 13:29:02 +01:00
Neil Johnson
5c05ae7ba0 Add 'rel' attribute to default welcome page. (#5695) 
add rel attribute as a precaution against reverse tabnabbing in future
 2019-07-19 12:03:36 +01:00
Amber Brown
6a85cb5ef7
Remove non-dedicated logging options and command line arguments (#5678) 2019-07-19 01:40:08 +10:00
Neil Johnson
cfc00068bd enable aggregations support by default 2019-07-18 15:56:59 +01:00
Erik Johnston
10523241d8 Delegate to cached version when using get_filtered_current_state_ids 
In the case where it gets called with `StateFilter.all()`
 2019-07-18 15:17:39 +01:00
Richard van der Hoff
82345bc09a
Clean up opentracing configuration options (#5712) 
Clean up config settings and dead code.

This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular:
 * There should be a blank line after `## Section ##' headings
 * There should be a blank line between each config setting
 * There should be a `#`-only line between a comment and the setting it describes
 * We don't really do the `#  #` style commenting-out of whole sections if we can help it
 * rename `tracer_enabled` to `enabled`

While we're here, do more config parsing upfront, which makes it easier to use
later on.

Also removes redundant code from LogContextScopeManager.

Also changes the changelog fragment to a `feature` - it's exciting!
 2019-07-18 15:06:54 +01:00
Amber Brown
7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Andrew Morgan
b2a382efdb
Remove the ability to query relations when the original event was redacted. (#5629) 
Fixes #5594

Forbid viewing relations on an event once it has been redacted.
 2019-07-18 14:41:42 +01:00
Erik Johnston
8e1ada9e6f Use the current_state_events.membership column 2019-07-18 14:16:01 +01:00
Erik Johnston
059d8c1a4e Track if current_state_events.membership is up to date 2019-07-18 14:16:01 +01:00
Erik Johnston
c618a5d348 Add background update for current_state_events.membership column 2019-07-18 14:16:01 +01:00
Erik Johnston
6de09e07a6 Add membership column to current_state_events table. 
It turns out that doing a join is surprisingly expensive for the DB to
do when room_membership table is larger than the disk cache.
 2019-07-18 14:15:57 +01:00
Richard van der Hoff
fa8271c5ac
Convert synapse.federation.transport.server to async (#5689) 
* Convert BaseFederationServlet._wrap to async

Empirically, this fixes some lost stacktraces. It should be safe because the
wrapped function is called from JsonResource._async_render, which is already
async.

* Convert the rest of synapse.federation.transport.server to async

We may as well do the whole file while we're here.

* changelog

* flake8
 2019-07-18 11:46:47 +01:00
Richard van der Hoff
9c70a02a9c
Ignore redactions of m.room.create events (#5701) 2019-07-17 19:08:02 +01:00