Commit Graph

666 Commits

Author SHA1 Message Date
Richard van der Hoff
0b794cbd7b
Fix sdnotify with acme enabled (#6571)
If acme was enabled, the sdnotify startup hook would never be run because we
would try to add it to a hook which had already fired.

There's no need to delay it: we can sdnotify as soon as we've started the
listeners.
2019-12-19 14:52:52 +00:00
Erik Johnston
b8e4b39b69
Merge pull request #6511 from matrix-org/erikj/remove_db_config_from_apps
Move database config from apps into HomeServer object
2019-12-12 10:37:56 +00:00
Erik Johnston
adb3a873fd Synapse 1.7.0rc2 (2019-12-11)
=============================
 
 Bugfixes
 --------
 
 - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
 - Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
 - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl3w+RwQHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHijfD/43EQ58jqKvD+qKZwVFOE6JJ3SCS7UJbi3f
 zq9KWDuCB6EAFjAgmJikbBBqHPO5qq2WtNXaCpexx44s8Mk8SZKHg56dP6Fk651C
 assAb/Nsh6CAlPUcRkx8I0L/kYXMPDyATlLVBHVOi3pFDJ093mdOQ4q8yP9iUTM+
 OPsbT8k/pMhrhCH951bGmB6/SEcju+ubObW+bRFe8o3v1KE9jVYQjUGMhuoXp3pM
 z/OB8idZcqOvCc6HMo83tg9FuI613Jy80PMIc1ofyJgvnu+aDBepWuldvFEMnmSR
 D862jMor7+WdnDOTeWZrC+DXjl0qCP8F6ahs5rEllRglt/Ep2wEDPA/8YrRoEI3V
 RWe9W7XDdFCXdzlvXheOfETqTu9kdsurTwBEeJrWQ0vOLY86hxt9KKKcHVhy5eKq
 kNfRtvSVLmRhIssp7hVBcywRwnaxN7R2OoRq/TWnTZz+xEOPzYFU6r0l9kk5dbg6
 fqYYxIXbgZlhjihLWNIMNwwZH9ll/eoPiinkRTZNz40THP/VDTR9DM4tQ6jrhrr6
 sP0qM7LljvrdiimXtV00tUDyUspNgJl6xDJyGDHWM9uoCB7uorEpMQZSzlZhZe8s
 6q+fQPHlfW4JYOejfihPkjrV1ViawvEucqWPaIsD+v26C4RP7qCTtYj3NPiA65of
 zhOjomWWcg==
 =ABoZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.7.0rc2' into develop

Synapse 1.7.0rc2 (2019-12-11)
=============================

Bugfixes
--------

- Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
- Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
- Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
2019-12-11 14:14:30 +00:00
Erik Johnston
bc5cb8bfe8 Remove database config parsing from apps. 2019-12-10 14:34:17 +00:00
Erik Johnston
663238aeb4 Phone home stats DB reporting should not assume a single DB. 2019-12-10 13:21:04 +00:00
Brendan Abolivier
2ac78438d8
Make the PusherSlaveStore inherit from the slave RoomStore
So that it has access to the get_retention_policy_for_room function which is required by filter_events_for_client.
2019-12-10 12:31:03 +00:00
Erik Johnston
75f87450d8 Move start up DB checks to main data store. 2019-12-06 16:02:21 +00:00
Erik Johnston
d64bb32a73 Move are_all_users_on_domain checks to main data store. 2019-12-06 13:43:40 +00:00
Erik Johnston
9a4fb457cf Change DataStores to accept 'database' param. 2019-12-06 13:30:06 +00:00
Erik Johnston
2ace775d88 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_database_class 2019-12-06 11:33:34 +00:00
Erik Johnston
1a0997bbd5 Port rest/v1 to async/await 2019-12-05 15:57:28 +00:00
Erik Johnston
4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston
756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston
ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Erik Johnston
c2f525a525 Don't call SQLBaseStore methods from outside stores 2019-12-04 16:23:43 +00:00
Andrew Morgan
9fb350af65 Merge branch 'master' into develop 2019-11-26 14:15:30 +00:00
Erik Johnston
65d54c5e8c Fix phone home stats (#6418)
Fix phone home stats
2019-11-26 13:28:40 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Erik Johnston
052513958d Fix phone home stats 2019-11-05 17:44:09 +00:00
Amber Brown
4e1c7b79fa
Remove the psutil dependency (#6318)
* remove psutil and replace with resource
2019-11-05 05:05:48 +11:00
Richard van der Hoff
1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston
c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Amber Brown
864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Sorunome
dd2e5b0038 add report_stats_endpoint config option (#6012)
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
2019-09-12 11:24:57 +01:00
Andrew Morgan
0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Amber Brown
7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Richard van der Hoff
412c6e21a8
Drop dependency on sdnotify (#5871)
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
2019-08-17 09:09:52 +01:00
Chris Moos
6d86df73f1 Fix issue with Synapse not starting up. Fixes #5866.
Signed-off-by: Chris Moos <chris@chrismoos.com>
2019-08-16 22:16:13 +01:00
Amber Brown
0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Erik Johnston
80cfad233e Call startup commands as system triggers.
This helps ensures that we only consider ourselves "up" once all the
startup functions have completed.
2019-07-22 15:22:14 +01:00
Erik Johnston
c560b791e1 Add process hooks to tell systemd our state.
Fixes #5676.
2019-07-22 14:52:18 +01:00
Amber Brown
7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Erik Johnston
5ed7853bb0 Remove pointless description 2019-07-16 11:45:57 +01:00
Erik Johnston
d0d479c1af
Fix typo in synapse/app/admin_cmd.py
Co-Authored-By: Aaron Raimist <aaron@raim.ist>
2019-07-16 09:52:56 +01:00
Erik Johnston
eca4f5ac73 s/exfiltrate_user_data/export_user_data/ 2019-07-15 14:17:28 +01:00
Erik Johnston
1b2067f53d Add FileExfiltrationWriter 2019-07-15 14:15:22 +01:00
Erik Johnston
e8c53b07f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_api_cmd 2019-07-15 14:13:22 +01:00
Erik Johnston
c8f35d8d38 Use set_defaults(func=) style 2019-07-15 14:09:35 +01:00
Erik Johnston
fdefb9e29a Move creation of ArgumentParser to caller 2019-07-15 14:09:35 +01:00
Erik Johnston
37b524f971 Fix up comments 2019-07-15 14:09:35 +01:00
Jorik Schellekens
38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Richard van der Hoff
cb8d568cf9 Fix 'utime went backwards' errors on daemonization. (#5609)
* Fix 'utime went backwards' errors on daemonization.

Fixes #5608

* remove spurious debug
2019-07-03 22:40:45 +10:00
Erik Johnston
9f3c0a8556 Add basic admin cmd app 2019-07-02 17:12:48 +01:00
Richard van der Hoff
6cda36777b Drop support for cpu_affinity (#5525)
This has no useful purpose on python3, and is generally a source of confusion.
2019-06-22 11:01:55 +10:00
Erik Johnston
7eadb74056 Fix /messages on worker when no token supplied 2019-06-21 14:15:59 +01:00