Commit Graph

9615 Commits

Author SHA1 Message Date
Erik Johnston
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation
Batch cache invalidation over replication
2019-02-19 13:14:30 +00:00
Erik Johnston
62175a20e5 Docs 2019-02-19 11:38:40 +00:00
Erik Johnston
1bb35e3a83 Use itertools 2019-02-19 11:34:40 +00:00
Erik Johnston
bc8fa1509d Documentation 2019-02-19 11:24:59 +00:00
Brendan Abolivier
a288bdf0b1
Merge pull request #4652 from matrix-org/babolivier/acme-delegated
Support .well-known delegation when issuing certificates through ACME
2019-02-19 11:15:38 +00:00
Brendan Abolivier
5a707a2f9a Improve config documentation 2019-02-19 10:59:26 +00:00
Brendan Abolivier
a8626901cd Fetch ACME domain into an instance member 2019-02-19 10:54:33 +00:00
Erik Johnston
32590b7139
Merge pull request #4670 from matrix-org/erikj/register_login_split
Split /login into client_reader
2019-02-19 10:46:22 +00:00
Richard van der Hoff
7c70b8f8a6
Try and make TLS federation client code faster (#4674)
* fix to use makeContext so that we don't need to rebuild the certificateoptions each time
2019-02-19 10:19:16 +00:00
Andrew Morgan
968a30a75c
Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade
Transfer bans on room upgrade
2019-02-19 10:16:42 +00:00
Amber Brown
561eebe170 fix to use makeContext so that we don't need to rebuild the certificateoptions each time 2019-02-19 16:18:05 +11:00
Andrew Morgan
34ac75ce2c lint 2019-02-18 18:23:37 +00:00
Erik Johnston
a9b5ea6fc1 Batch cache invalidation over replication
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.

Hopefully this will reduce load on the master process by substantially
reducing traffic.
2019-02-18 17:53:31 +00:00
Andrew Morgan
f8b9ca53ce Move member event processing and changelog fix 2019-02-18 17:28:52 +00:00
Erik Johnston
d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry
Add basic optional sentry.io integration
2019-02-18 17:22:45 +00:00
Erik Johnston
4cc4400b4d Split /login into client_reader 2019-02-18 17:19:01 +00:00
Erik Johnston
fc2c245a1f
Merge pull request #4666 from matrix-org/erikj/register_login_split
Split out registration to worker
2019-02-18 17:18:06 +00:00
Erik Johnston
459d3d5046
Merge pull request #4668 from matrix-org/erikj/catch_exceptions
Correctly handle HttpResponseException when handling device updates
2019-02-18 16:55:43 +00:00
Erik Johnston
d328a93b51 Fixup error handling and message 2019-02-18 16:53:56 +00:00
Erik Johnston
af691e415c Move register_device into handler 2019-02-18 16:49:38 +00:00
Erik Johnston
028267acd2
Merge pull request #4669 from matrix-org/erikj/log_exception
Cleanup top level request exception logging
2019-02-18 16:41:41 +00:00
Erik Johnston
d08bac4136
Merge pull request #4651 from matrix-org/matthew/well-known-cors
set CORS on .well-known URI to unbreak modular
2019-02-18 16:38:11 +00:00
Erik Johnston
c30f73c86a
Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers
Fix kicking guest users in worker mode
2019-02-18 15:55:48 +00:00
Erik Johnston
092b541401 Fixup 2019-02-18 15:52:26 +00:00
Brendan Abolivier
45bb55c6de Use a configuration parameter to give the domain to generate a certificate for 2019-02-18 15:46:23 +00:00
Erik Johnston
8b9ae6d3a6 Update docs 2019-02-18 15:26:13 +00:00
Erik Johnston
94960cef03 pep8 2019-02-18 15:24:13 +00:00
Erik Johnston
fe725f7e45 Cleanup top level request exception logging
Firstly, we always logged that the request was being handled via
`JsonResource._async_render`, so we change that to use the servlet name
we add to the request.

Secondly, we pass the exception information to the logger rather than
formatting it manually. This makes it consistent with other exception
logging, allwoing logging hooks and formatters to access the exception
information.
2019-02-18 15:11:04 +00:00
Matthew Hodgson
04dad5ac16 fix missig import 2019-02-18 15:59:23 +01:00
Erik Johnston
e07cc31cb8 Correctly handle HttpResponseException 2019-02-18 14:55:09 +00:00
Brendan Abolivier
68a53f825f Merge branch 'develop' into babolivier/acme-delegated 2019-02-18 14:52:23 +00:00
Erik Johnston
32e54b472a Fix kicking guest users in worker mode
When guest_access changes from allowed to forbidden all local guest
users should be kicked from the room. This did not happen when
revocation was received from federation on a worker.

Presumably broken in #4141
2019-02-18 14:16:07 +00:00
Andrew Morgan
915421065b Membership events are done later 2019-02-18 14:02:09 +00:00
Erik Johnston
d1b060b492
Merge pull request #4643 from matrix-org/erikj/catch_exceptions
Correctly handle RequestSendFailed exceptions
2019-02-18 13:53:24 +00:00
Andrew Morgan
9caab0c364 Transfer bans on room upgrade 2019-02-18 13:52:51 +00:00
Erik Johnston
dc5efc92a8 Fixup 2019-02-18 13:52:49 +00:00
Erik Johnston
eb2b8523ae Split out registration to worker
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.

Note: due to the in-memory session map all registration requests must be
handled by the same worker.
2019-02-18 12:12:57 +00:00
Andrew Morgan
6d02a13d81
Typo in info log
Co-Authored-By: babolivier <contact@brendanabolivier.com>
2019-02-18 11:36:34 +00:00
Juuso "Linda" Lapinlampi
68d2869c8d config: Remove a repeated word from a logger warning
The warning for missing macaroon_secret_key was "missing missing".
2019-02-15 22:24:53 -07:00
Erik Johnston
bd4505f765
Merge pull request #4647 from matrix-org/erikj/add_room_publishing_rules
Add configurable room list publishing rules
2019-02-15 22:11:01 +00:00
Brendan Abolivier
f86b695cbd
Various cosmetics to make TravisCI happy 2019-02-15 12:29:34 +00:00
Brendan Abolivier
af8a2f679b
Remove unused import 2019-02-15 12:27:43 +00:00
Brendan Abolivier
1895d14e12
Support .well-known delegation when issuing certificates through ACME 2019-02-15 12:05:08 +00:00
Erik Johnston
b99c532c1c Move defaults up into code 2019-02-15 10:53:39 +00:00
Erik Johnston
02c729d6b0 Hoist up checks to reduce overall work 2019-02-15 10:20:02 +00:00
Erik Johnston
02c46acc6a Fixup comments 2019-02-15 10:17:13 +00:00
Matthew Hodgson
bfcefbb230 tabs 2019-02-14 19:53:12 +00:00
Matthew Hodgson
6f47bc3fb2 set CORS on .well-known URI to unbreak modular
otherwise a riot/web running on foo.riot.im can't query
the .well-known on foo.modular.im...
2019-02-14 19:51:41 +00:00
Erik Johnston
8e32f26cb8 Clarify comments 2019-02-14 18:21:24 +00:00
Erik Johnston
cb12a37708 Clarify and fix behaviour when there are multiple aliases 2019-02-14 18:16:32 +00:00
Erik Johnston
f61b2068e6 Only fetch aliases when publishing rooms 2019-02-14 18:08:09 +00:00
Erik Johnston
f666fe36d7 Fixup comments 2019-02-14 18:07:24 +00:00
Richard van der Hoff
bf4fd14806 Merge branch 'master' into develop 2019-02-14 17:34:09 +00:00
Richard van der Hoff
f595d6ac57 0.99.1.1 2019-02-14 17:20:02 +00:00
Richard van der Hoff
f311018823
Fix errors in acme provisioning (#4648)
* Better logging for errors on startup

* Fix "TypeError: '>' not supported" when starting without an existing
  certificate
* Fix a bug where an existing certificate would be reprovisoned every day
2019-02-14 17:10:36 +00:00
Erik Johnston
eaf4d11af9 Add configurable room list publishing rules
This allows specifying who and what is allowed to be published onto the
public room list
2019-02-14 16:02:23 +00:00
Richard van der Hoff
b02465b9db Merge branch 'master' into develop 2019-02-14 14:42:03 +00:00
Richard van der Hoff
06cd757ae7 0.99.1 2019-02-14 14:24:24 +00:00
Erik Johnston
7fc1196a36 Correctly handle RequestSendFailed exceptions
This mainly reduces the number of exceptions we log.
2019-02-14 14:01:04 +00:00
Erik Johnston
6cb415b63f Fixup comments and add warning 2019-02-13 16:15:11 +00:00
Richard van der Hoff
c6e75c9f2d
Merge pull request #4450 from 14mRh4X0r/fix-dependency-message
Fix error message for optional dependencies
2019-02-13 16:12:49 +00:00
Richard van der Hoff
3bc238629e 0.99.1rc2 2019-02-13 14:46:18 +00:00
Richard van der Hoff
c1dfd6a18a Merge remote-tracking branch 'origin/release-v0.99.1' into develop 2019-02-13 14:27:45 +00:00
Erik Johnston
309f3bb322
Update synapse/app/_base.py
Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>
2019-02-13 13:24:27 +00:00
Amber Brown
bb4fd8f927
Run black on user directory code (#4635) 2019-02-13 23:05:32 +11:00
Richard van der Hoff
767686af48 Use listen_tcp for the replication listener
Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with
what we do elsewhere.
2019-02-13 11:59:04 +00:00
Richard van der Hoff
2a5a15aff8 Improve logging around listening services
I wanted to bring listen_tcp into line with listen_ssl in terms of returning a
list of ports, and wanted to check that was a safe thing to do - hence the
logging in `refresh_certificate`.

Also, pull the 'Synapse now listening' message up to homeserver.py, because it
was being duplicated everywhere else.
2019-02-13 11:58:54 +00:00
Richard van der Hoff
e3a0300431 Special-case the default bind_addresses for metrics listener
turns out it doesn't really support ipv6, so let's hack around that by only
listening on ipv4 by default.
2019-02-13 11:48:56 +00:00
Erik Johnston
93f7d2df3e Comments 2019-02-12 16:03:40 +00:00
Erik Johnston
6a8f902edb Raise an appropriate error message if sentry_sdk missing 2019-02-12 16:01:41 +00:00
Erik Johnston
ef2228c890 Basic sentry integration 2019-02-12 13:55:58 +00:00
Erik Johnston
d2fa7b7e99 Update changelog and version 2019-02-12 13:22:25 +00:00
Erik Johnston
ba3f27b69a
Merge pull request #4608 from matrix-org/anoa/acls_room_upgrade
Transfer Server ACLs on room upgrade
2019-02-12 13:20:06 +00:00
Erik Johnston
cf82338930
Merge pull request #4627 from matrix-org/erikj/user_ips_analyze
Analyze user_ips before running deduplication
2019-02-12 13:05:09 +00:00
Erik Johnston
3df8fcca25
Merge pull request #4626 from matrix-org/erikj/fixup_user_ips_dedupe
Reduce user_ips bloat during dedupe background update
2019-02-12 13:02:58 +00:00
Erik Johnston
495ea92350 Fix pep8 2019-02-12 12:40:42 +00:00
Erik Johnston
483ba85c7a Analyze user_ips before running deduplication
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
2019-02-12 11:55:27 +00:00
Erik Johnston
362d80b770 Reduce user_ips bloat during dedupe background update
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.

This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.
2019-02-12 11:39:34 +00:00
Erik Johnston
3c03c37883
Merge pull request #4625 from matrix-org/rav/fix_generate_config_warnings
fix self-signed cert notice from generate-config
2019-02-12 11:24:45 +00:00
Richard van der Hoff
a4ce91396b
Disable TLS by default (#4614) 2019-02-12 10:52:08 +00:00
Richard van der Hoff
32b781bfe2
Fix error when loading cert if tls is disabled (#4618)
If TLS is disabled, it should not be an error if no cert is given.

Fixes #4554.
2019-02-12 10:51:31 +00:00
Richard van der Hoff
dfc846a316 fix self-signed cert notice from generate-config
fixes #4620
2019-02-12 10:37:59 +00:00
Richard van der Hoff
0ca2908653 fix tests 2019-02-11 22:01:27 +00:00
Richard van der Hoff
4fddf8fc77 Infer no_tls from presence of TLS listeners
Rather than have to specify `no_tls` explicitly, infer whether we need to load
the TLS keys etc from whether we have any TLS-enabled listeners.
2019-02-11 21:39:14 +00:00
Richard van der Hoff
15272f837c Merge branch 'rav/no_create_server_contexts_if_no_tls' into rav/tls_cert/work 2019-02-11 21:34:19 +00:00
Richard van der Hoff
9645728619 Don't create server contexts when TLS is disabled
we aren't going to use them anyway.
2019-02-11 21:32:01 +00:00
Richard van der Hoff
be794c7cf7 Merge branch 'rav/tls_config_logging_fixes' into rav/tls_cert/work 2019-02-11 21:16:00 +00:00
Richard van der Hoff
2129dd1a02 Fail cleanly if listener config lacks a 'port'
... otherwise we would fail with a mysterious KeyError or something later.
2019-02-11 21:15:01 +00:00
Richard van der Hoff
086f6f27d4 Logging improvements around TLS certs
Log which file we're reading keys and certs from, and refactor the code a bit
in preparation for other work
2019-02-11 21:02:06 +00:00
Richard van der Hoff
5d27730a73
Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
It's nothing to do with refreshing the certificates. No idea why it was here.
2019-02-11 18:03:30 +00:00
Erik Johnston
719e073f00
Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating
add updating of backup versions
2019-02-11 13:45:49 +00:00
Richard van der Hoff
24b7f3916d
Clean up default listener configuration (#4586)
Rearrange the comments to try to clarify them, and expand on what some of it
means.

Use a sensible default 'bind_addresses' setting.

For the insecure port, only bind to localhost, and enable x_forwarded, since
apparently it's for use behind a load-balancer.
2019-02-11 12:50:30 +00:00
Andrew Morgan
a126f86eec Transfer Server ACLs on room upgrade 2019-02-11 11:30:37 +00:00
Amber Brown
6e2a5aa050 ACME Reprovisioning (#4522) 2019-02-11 10:36:26 +00:00
Amber Brown
4ffd10f46d Be tolerant of blank TLS fingerprints config (#4589) 2019-02-11 10:04:27 +00:00
Erik Johnston
b201149c7e
Merge pull request #4420 from matrix-org/jaywink/openid-listener
New listener resource for the federation API "openid/userinfo" endpoint
2019-02-11 09:44:00 +00:00
Valentin Anger
2dc2b6e9f1 Allow "unavailable" presence status for /sync (#4592)
* Allow "unavailable" presence status for /sync

Closes #3772, closes #3779

Signed-off-by: Valentin Anger <valentin.an.1999@gmail.com>

* Add changelog for PR 4592
2019-02-08 21:09:56 +00:00
Richard van der Hoff
56710c7df5
Fix 'no unique or exclusion constraint' error (#4591)
Add more tables to the list of tables which need a background update to
complete before we can upsert into them, which fixes a race against the
background updates.
2019-02-08 18:30:46 +00:00
Amber Brown
9cd33d2f4b
Deduplicate some code in synapse.app (#4567) 2019-02-08 17:25:57 +00:00
Hubert Chathi
afae8442b5 make sure version is in body and wrap in linearizer queue
also add tests
2019-02-08 01:32:45 -05:00
Hubert Chathi
d9e424bf64 re-try to make isort happy 2019-02-06 22:18:41 -05:00
Hubert Chathi
9ff620a518 fix import to make isort happy 2019-02-06 21:32:52 -05:00
Hubert Chathi
8248637173 add new endpoint to update backup versions 2019-02-06 17:57:10 -05:00
Hubert Chathi
664c81e8b7 return proper error codes for some 404s 2019-02-06 17:47:22 -05:00
Richard van der Hoff
2475434080 Merge branch 'master' into develop 2019-02-05 18:44:49 +00:00
Richard van der Hoff
3bd9daf4b8 v0.99.0 2019-02-05 18:33:02 +00:00
Andrew Morgan
627ecd358e Filter user directory state query to a subset of state events (#4462)
* Filter user directory state query to a subset of state events

* Add changelog
2019-02-05 12:16:28 +00:00
Richard van der Hoff
bf1e4d96ad
Fix default ACME config for py2 (#4564)
Fixes #4559
2019-02-05 11:37:33 +00:00
Richard van der Hoff
d7e27a1f08
fix typo in config comments (#4557) 2019-02-05 11:32:45 +00:00
Richard van der Hoff
30fd2f89db 0.99.0rc4 2019-02-01 15:52:28 +00:00
Richard van der Hoff
ef43a03fc5
Merge pull request #4546 from matrix-org/rav/silence_critical_error_from_federation
Fix noisy "twisted.internet.task.TaskStopped" errors in logs
2019-02-01 14:37:22 +00:00
Richard van der Hoff
f8db967d5a Merge remote-tracking branch 'origin/release-v0.99.0' into develop 2019-02-01 13:20:15 +00:00
Richard van der Hoff
fa794980ec
Merge pull request #4544 from matrix-org/rav/skip_invalid_well_known
Treat an invalid .well-known the same as an absent one
2019-02-01 13:18:36 +00:00
Richard van der Hoff
e9779a6f8f Fix b'ab' noise in logs 2019-02-01 12:34:31 +00:00
Richard van der Hoff
9763a73af0 Merge branch 'release-v0.99.0' into develop 2019-02-01 12:30:22 +00:00
Richard van der Hoff
f0ba34f581 Fix noisy "twisted.internet.task.TaskStopped" errors in logs
Fixes #4003
2019-02-01 12:22:57 +00:00
Richard van der Hoff
8a21b03fba Treat an invalid .well-known the same as an absent one
... basically, carry on and fall back to SRV etc.
2019-02-01 11:37:31 +00:00
Richard van der Hoff
3c8a41140e Cache failures to parse .well-known
Also add a Measure block around the .well-known fetch
2019-02-01 00:37:52 +00:00
Richard van der Hoff
24d59c7568 better logging for federation connections 2019-01-31 23:18:20 +00:00
Richard van der Hoff
d428b46346 Update federation routing logic to check .well-known before SRV 2019-01-31 23:14:18 +00:00
Andrew Morgan
c45fd0dda0
Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable
Copy over non-federatable trait on room upgrade
2019-01-31 20:28:37 +00:00
Richard van der Hoff
625385d684 Merge branch 'release-v0.99.0' into develop 2019-01-31 18:43:20 +00:00
Richard van der Hoff
85129d7068 v0.99.0rc3 2019-01-31 18:35:38 +00:00
Andrew Morgan
d239f67c25 Raise an exception instead of returning None 2019-01-31 18:34:15 +00:00
Richard van der Hoff
07dfe148de
Add some debug for membership syncing issues (#4538)
I can't figure out what's going on with #4422 and #4436; perhaps this will help.
2019-01-31 18:30:40 +00:00
Andrew Morgan
3ed3cb4339 New function for getting room's create event 2019-01-31 18:21:39 +00:00
Andrew Morgan
bbb97a35fd Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable 2019-01-31 18:11:00 +00:00
Richard van der Hoff
e707e7b38d
Fix infinite loop when an event is redacted in a v3 room (#4535) 2019-01-31 15:34:17 +00:00
Andrew Morgan
563f6a832b Reject large transactions on federation (#4513)
* Reject large transactions on federation

* Add changelog

* lint

* Simplify large transaction handling
2019-01-31 11:44:04 +00:00
Andrew Morgan
fb50934b8f lint 2019-01-31 11:34:45 +00:00
Richard van der Hoff
6fba9fd20c Merge remote-tracking branch 'origin/release-v0.99.0' into develop 2019-01-30 17:02:32 +00:00
Andrew Morgan
d621c5562e Copy over non-federatable trait on room upgrade 2019-01-30 16:33:51 +00:00
Richard van der Hoff
b8b898666e v0.99.0rc2 2019-01-30 16:31:07 +00:00
Matthew Hodgson
ad7ac8853c by default include m.room.encryption on invites (#3902)
* by default include m.room.encryption on invites

* fix constant

* changelog
2019-01-30 16:26:13 +00:00
Richard van der Hoff
c74b96755c Merge remote-tracking branch 'origin/develop' into release-v0.99.0 2019-01-30 16:23:28 +00:00
Neil Johnson
81b7e7eed3
Update constants.py
remove trailing ,
2019-01-30 16:11:36 +00:00
Erik Johnston
ed8c5e4cda Fix remote invite rejections not comming down sync
This was broken in PR #4405, commit 886e5ac, where we changed remote
rejections to be outliers.

The fix is to explicitly add the leave event in when we know its an out
of band invite. We can't always add the event as if the server is/was in
the room there might be more events to send down the sync than just the
leave.
2019-01-30 15:46:27 +00:00
Richard van der Hoff
a5d0c771a3 0.99.0rc1 2019-01-30 15:11:18 +00:00
Erik Johnston
6587b0b89b
Merge pull request #4472 from matrix-org/neilj/room_capabilities
Server capabilities support
2019-01-30 14:26:56 +00:00
Erik Johnston
a4f52a33fe Fix replication for room v3 (#4523)
* Fix replication for room v3

We were not correctly quoting the path fragments over http replication,
which meant that it exploded when the event IDs had a slash in them

* Newsfile
2019-01-30 14:19:52 +00:00
Richard van der Hoff
7615a8ced1 ACME config cleanups (#4525)
* Handle listening for ACME requests on IPv6 addresses

the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses
without extra quoting. Building a string which you are about to parse again
seems like a weird choice. Let's just use listenTCP, which is consistent with
what we do elsewhere.

* Clean up the default ACME config

make it look a bit more consistent with everything else, and tweak the defaults
to listen on port 80.

* newsfile
2019-01-30 14:17:55 +00:00
Erik Johnston
e87d7a4b0f Raise ConfigError instead 2019-01-30 12:48:09 +00:00
Erik Johnston
270f212a2a _listener_http should return a list 2019-01-30 12:14:50 +00:00
Richard van der Hoff
a79034aedf
Merge pull request #4521 from matrix-org/rav/fed_routing/cleanups
Tiny .well-known fixes
2019-01-30 11:47:24 +00:00
Richard van der Hoff
c7b24ac3d0
Follow redirects on .well-known (#4520) 2019-01-30 11:43:33 +00:00
Amber Brown
f6813919e8
SIGHUP for TLS cert reloading (#4495) 2019-01-30 11:00:02 +00:00
Richard van der Hoff
09a1a6b55e fix exception text 2019-01-30 10:58:52 +00:00
Richard van der Hoff
928c50b59a Also jitter the invalid cache period 2019-01-30 10:58:52 +00:00
Neil Johnson
b37e8c9572 Merge branch 'neilj/room_capabilities' of github.com:matrix-org/synapse into neilj/room_capabilities 2019-01-30 10:56:47 +00:00
Neil Johnson
f834d98402 isort 2019-01-30 10:55:42 +00:00