Commit Graph

707 Commits

Author SHA1 Message Date
Erik Johnston
2ace775d88 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_database_class 2019-12-06 11:33:34 +00:00
Erik Johnston
1a0997bbd5 Port rest/v1 to async/await 2019-12-05 15:57:28 +00:00
Erik Johnston
4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston
756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston
ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Erik Johnston
c2f525a525 Don't call SQLBaseStore methods from outside stores 2019-12-04 16:23:43 +00:00
Andrew Morgan
9fb350af65 Merge branch 'master' into develop 2019-11-26 14:15:30 +00:00
Erik Johnston
65d54c5e8c Fix phone home stats (#6418)
Fix phone home stats
2019-11-26 13:28:40 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Erik Johnston
052513958d Fix phone home stats 2019-11-05 17:44:09 +00:00
Amber Brown
4e1c7b79fa
Remove the psutil dependency (#6318)
* remove psutil and replace with resource
2019-11-05 05:05:48 +11:00
Richard van der Hoff
1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston
c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Amber Brown
864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Sorunome
dd2e5b0038 add report_stats_endpoint config option (#6012)
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
2019-09-12 11:24:57 +01:00
Andrew Morgan
0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Amber Brown
7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Richard van der Hoff
412c6e21a8
Drop dependency on sdnotify (#5871)
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
2019-08-17 09:09:52 +01:00
Chris Moos
6d86df73f1 Fix issue with Synapse not starting up. Fixes #5866.
Signed-off-by: Chris Moos <chris@chrismoos.com>
2019-08-16 22:16:13 +01:00
Amber Brown
0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Erik Johnston
80cfad233e Call startup commands as system triggers.
This helps ensures that we only consider ourselves "up" once all the
startup functions have completed.
2019-07-22 15:22:14 +01:00
Erik Johnston
c560b791e1 Add process hooks to tell systemd our state.
Fixes #5676.
2019-07-22 14:52:18 +01:00
Amber Brown
7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Erik Johnston
5ed7853bb0 Remove pointless description 2019-07-16 11:45:57 +01:00
Erik Johnston
d0d479c1af
Fix typo in synapse/app/admin_cmd.py
Co-Authored-By: Aaron Raimist <aaron@raim.ist>
2019-07-16 09:52:56 +01:00
Erik Johnston
eca4f5ac73 s/exfiltrate_user_data/export_user_data/ 2019-07-15 14:17:28 +01:00
Erik Johnston
1b2067f53d Add FileExfiltrationWriter 2019-07-15 14:15:22 +01:00
Erik Johnston
e8c53b07f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_api_cmd 2019-07-15 14:13:22 +01:00
Erik Johnston
c8f35d8d38 Use set_defaults(func=) style 2019-07-15 14:09:35 +01:00
Erik Johnston
fdefb9e29a Move creation of ArgumentParser to caller 2019-07-15 14:09:35 +01:00
Erik Johnston
37b524f971 Fix up comments 2019-07-15 14:09:35 +01:00
Jorik Schellekens
38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Richard van der Hoff
cb8d568cf9 Fix 'utime went backwards' errors on daemonization. (#5609)
* Fix 'utime went backwards' errors on daemonization.

Fixes #5608

* remove spurious debug
2019-07-03 22:40:45 +10:00
Erik Johnston
9f3c0a8556 Add basic admin cmd app 2019-07-02 17:12:48 +01:00
Richard van der Hoff
6cda36777b Drop support for cpu_affinity (#5525)
This has no useful purpose on python3, and is generally a source of confusion.
2019-06-22 11:01:55 +10:00
Erik Johnston
7eadb74056 Fix /messages on worker when no token supplied 2019-06-21 14:15:59 +01:00
Erik Johnston
f3ab533374 Support pagination API in client_reader worker 2019-06-21 10:43:12 +01:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
7e68691ce9 Merge branch 'master' of github.com:matrix-org/synapse into develop 2019-06-11 17:25:16 +01:00
Erik Johnston
10383e6e6f Change password reset links to /_matrix. 2019-06-11 11:34:33 +01:00
Neil Johnson
94dac0f3e5
add monthly active users to phonehome stats (#5252)
* add monthly active users to phonehome stats
2019-06-10 23:33:59 +01:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Amber Brown
2889b05554
Unify v1 and v2 REST client APIs (#5226) 2019-06-03 21:28:59 +10:00
Richard van der Hoff
f76d407ef3
Fix dropped logcontexts during high outbound traffic. (#5277)
Fixes #5271.
2019-05-29 09:17:33 +01:00
Neil Johnson
66b75e2d81
Neilj/ensure get profileinfo available in client reader slaved store (#5213)
* expose SlavedProfileStore to ClientReaderSlavedStore
2019-05-22 13:55:32 +01:00
Richard van der Hoff
24b93b9c76 Revert "expose SlavedProfileStore to ClientReaderSlavedStore (#5200)"
This reverts commit ce5bcefc60.

This caused:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/client_reader.py", line 32, in <module>
    from synapse.replication.slave.storage import SlavedProfileStore
ImportError: cannot import name 'SlavedProfileStore' from 'synapse.replication.slave.storage' (/home/synapse/src/synapse/replication/slave/storage/__init__.py)
error starting synapse.app.client_reader('/home/synapse/config/workers/client_reader.yaml') (exit code: 1); see above for logs
```
2019-05-20 16:21:34 +01:00
Neil Johnson
ce5bcefc60
expose SlavedProfileStore to ClientReaderSlavedStore (#5200)
* expose SlavedProfileStore to ClientReaderSlavedStore
2019-05-17 13:27:19 +01:00
Erik Johnston
c8c069db92
Merge pull request #5037 from matrix-org/erikj/limit_inflight_dns
Limit in flight DNS requests
2019-05-08 17:11:03 +01:00
Richard van der Hoff
40e576e29c Move admin api impl to its own package
It doesn't really belong under rest/client/v1 any more.
2019-05-01 15:44:30 +01:00
Richard van der Hoff
8e9ca83537 Move admin API to a new prefix 2019-05-01 15:44:30 +01:00
Erik Johnston
38642614cf VersionRestServlet doesn't take a param 2019-04-15 19:39:47 +01:00
Erik Johnston
6e27a8620f
Merge pull request #5063 from matrix-org/erikj/move_endpoints
Move some rest endpoints to client reader
2019-04-15 18:55:01 +01:00
Erik Johnston
d5adf297e6 Move some rest endpoints to client reader 2019-04-15 17:21:03 +01:00
Erik Johnston
a0fc256d65 Limit in flight DNS requests
This is to work around a bug in twisted where a large number of
concurrent DNS requests cause it to tight loop forever.

c.f. https://twistedmatrix.com/trac/ticket/9620#ticket
2019-04-09 17:23:42 +01:00
Neil Johnson
b25e387c0d
add context to phonehome stats (#5020)
add context to phonehome stats
2019-04-08 15:47:39 +01:00
Richard van der Hoff
4b91c313a9 Combine the CurrentStateDeltaStream into the EventStream 2019-03-27 22:07:05 +00:00
Richard van der Hoff
1f6d6f918a Make EventStream rows have a type
... as a precursor to combining it with the CurrentStateDelta stream.
2019-03-27 22:07:05 +00:00
Richard van der Hoff
a5798de067 Move replication.tcp.streams into a package 2019-03-27 21:13:14 +00:00
Richard van der Hoff
2dee441bdb
Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker
Move client receipt processing to federation sender worker.
2019-03-15 12:30:30 +00:00
Erik Johnston
b0fa3f6ff3
Merge pull request #4853 from matrix-org/erikj/worker_docker_ci
Allow passing --daemonize to workers
2019-03-15 10:35:38 +00:00
Erik Johnston
9ad448c1e5 Correctly handle all command line options 2019-03-14 13:32:14 +00:00
Richard van der Hoff
fdcad8eabd Move client receipt processing to federation sender worker.
This is mostly a prerequisite for #4730, but also fits with the general theme
of "move everything off the master that we possibly can".
2019-03-13 17:21:19 +00:00
Amber Brown
8da22e2b53 master startup 2019-03-11 21:13:35 +11:00
Erik Johnston
bfa7d46a10 Allow /keys/{changes,query} API to run on worker 2019-03-04 18:30:01 +00:00
Erik Johnston
76550c58d2
Merge pull request #4759 from matrix-org/erikj/3pid_client_reader
Move /account/3pid to client_reader
2019-02-27 16:11:21 +00:00
Erik Johnston
54f9ce11a7 Move /account/3pid to client_reader 2019-02-27 14:26:08 +00:00
Erik Johnston
4cff9376f7 Move server key queries to federation reader 2019-02-27 13:43:53 +00:00
Richard van der Hoff
f191be822b
Add database version to phonehome stats. (#4753) 2019-02-27 10:21:49 +00:00
Erik Johnston
47a7e3928d Correctly proxy exception in frontend_proxy worker 2019-02-23 15:17:57 +00:00
Erik Johnston
32590b7139
Merge pull request #4670 from matrix-org/erikj/register_login_split
Split /login into client_reader
2019-02-19 10:46:22 +00:00
Erik Johnston
d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry
Add basic optional sentry.io integration
2019-02-18 17:22:45 +00:00
Erik Johnston
4cc4400b4d Split /login into client_reader 2019-02-18 17:19:01 +00:00
Erik Johnston
fc2c245a1f
Merge pull request #4666 from matrix-org/erikj/register_login_split
Split out registration to worker
2019-02-18 17:18:06 +00:00
Erik Johnston
32e54b472a Fix kicking guest users in worker mode
When guest_access changes from allowed to forbidden all local guest
users should be kicked from the room. This did not happen when
revocation was received from federation on a worker.

Presumably broken in #4141
2019-02-18 14:16:07 +00:00
Erik Johnston
eb2b8523ae Split out registration to worker
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.

Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
Richard van der Hoff
f311018823
Fix errors in acme provisioning (#4648)
* Better logging for errors on startup

* Fix "TypeError: '>' not supported" when starting without an existing
  certificate
* Fix a bug where an existing certificate would be reprovisoned every day
2019-02-14 17:10:36 +00:00
Erik Johnston
6cb415b63f Fixup comments and add warning 2019-02-13 16:15:11 +00:00
Erik Johnston
309f3bb322
Update synapse/app/_base.py
Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>
2019-02-13 13:24:27 +00:00
Richard van der Hoff
767686af48 Use listen_tcp for the replication listener
Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with
what we do elsewhere.
2019-02-13 11:59:04 +00:00
Richard van der Hoff
2a5a15aff8 Improve logging around listening services
I wanted to bring listen_tcp into line with listen_ssl in terms of returning a
list of ports, and wanted to check that was a safe thing to do - hence the
logging in `refresh_certificate`.

Also, pull the 'Synapse now listening' message up to homeserver.py, because it
was being duplicated everywhere else.
2019-02-13 11:58:54 +00:00
Richard van der Hoff
e3a0300431 Special-case the default bind_addresses for metrics listener
turns out it doesn't really support ipv6, so let's hack around that by only
listening on ipv4 by default.
2019-02-13 11:48:56 +00:00
Erik Johnston
93f7d2df3e Comments 2019-02-12 16:03:40 +00:00
Erik Johnston
ef2228c890 Basic sentry integration 2019-02-12 13:55:58 +00:00
Richard van der Hoff
32b781bfe2
Fix error when loading cert if tls is disabled (#4618)
If TLS is disabled, it should not be an error if no cert is given.

Fixes #4554.
2019-02-12 10:51:31 +00:00
Richard van der Hoff
4fddf8fc77 Infer no_tls from presence of TLS listeners
Rather than have to specify `no_tls` explicitly, infer whether we need to load
the TLS keys etc from whether we have any TLS-enabled listeners.
2019-02-11 21:39:14 +00:00
Richard van der Hoff
9645728619 Don't create server contexts when TLS is disabled
we aren't going to use them anyway.
2019-02-11 21:32:01 +00:00
Richard van der Hoff
086f6f27d4 Logging improvements around TLS certs
Log which file we're reading keys and certs from, and refactor the code a bit
in preparation for other work
2019-02-11 21:02:06 +00:00
Richard van der Hoff
5d27730a73
Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
It's nothing to do with refreshing the certificates. No idea why it was here.
2019-02-11 18:03:30 +00:00
Amber Brown
6e2a5aa050 ACME Reprovisioning (#4522) 2019-02-11 10:36:26 +00:00
Erik Johnston
b201149c7e
Merge pull request #4420 from matrix-org/jaywink/openid-listener
New listener resource for the federation API "openid/userinfo" endpoint
2019-02-11 09:44:00 +00:00
Amber Brown
9cd33d2f4b
Deduplicate some code in synapse.app (#4567) 2019-02-08 17:25:57 +00:00
Richard van der Hoff
7615a8ced1 ACME config cleanups (#4525)
* Handle listening for ACME requests on IPv6 addresses

the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses
without extra quoting. Building a string which you are about to parse again
seems like a weird choice. Let's just use listenTCP, which is consistent with
what we do elsewhere.

* Clean up the default ACME config

make it look a bit more consistent with everything else, and tweak the defaults
to listen on port 80.

* newsfile
2019-01-30 14:17:55 +00:00
Erik Johnston
e87d7a4b0f Raise ConfigError instead 2019-01-30 12:48:09 +00:00
Erik Johnston
270f212a2a _listener_http should return a list 2019-01-30 12:14:50 +00:00
Amber Brown
f6813919e8
SIGHUP for TLS cert reloading (#4495) 2019-01-30 11:00:02 +00:00