Erik Johnston
3baf6e1667
Fix ExpiringCache.__len__ to be accurate
...
It used to try and produce an estimate, which was sometimes negative.
This caused metrics to be sad, so lets always just calculate it from
scratch.
(This appears to have been a longstanding bug, but one which has been made more
of a problem by #3932 and #3933 ).
(This was originally done by Erik as part of #3933 . I'm cherry-picking it
because really it's a fix in its own right)
2018-09-26 12:32:29 +01:00
Richard van der Hoff
f65163627f
Merge pull request #3911 from matrix-org/jcgruenhage/docker-support-python3
...
make python 3 work in the docker container
2018-09-25 15:18:09 +01:00
Jan Christian Grünhage
df55a943ca
Update Dockerfile
2018-09-25 14:33:38 +02:00
Jan Christian Grünhage
e7fa032126
Update .dockerignore
2018-09-25 14:33:19 +02:00
Richard van der Hoff
0649306fde
Update grafana dashboard
2018-09-25 13:29:28 +01:00
Richard van der Hoff
4c3e7eeec5
Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches
...
Fix some instances of ExpiringCache not expiring cache items
2018-09-25 12:02:57 +01:00
Jérémy Farnaud
6cf261930a
added "media-src: 'self'" to CSP for resources ( #3578 )
...
Synapse doesn’t allow for media resources to be played directly from
Chrome. It is a problem for users on other networks (e.g. IRC)
communicating with Matrix users through a gateway. The gateway sends
them the raw URL for the resource when a Matrix user uploads a video
and the video cannot be played directly in Chrome using that URL.
Chrome argues it is not authorized to play the video because of the
Content Security Policy. Chrome checks for the "media-src" policy which
is missing, and defauts to the "default-src" policy which is "none".
As Synapse already sends "object-src: 'self'" I thought it wouldn’t be
a problem to add "media-src: 'self'" to the CSP to fix this problem.
2018-09-25 11:55:02 +01:00
Richard van der Hoff
94f7befc31
Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered
...
Fix spurious exceptions when client closes conncetion
2018-09-25 11:52:06 +01:00
Richard van der Hoff
e4e96486a9
Merge pull request #3947 from matrix-org/rav/attr_version
...
We require attrs 16.0.0
2018-09-25 11:16:42 +01:00
Richard van der Hoff
45a1053d44
changelog
2018-09-25 10:45:34 +01:00
Richard van der Hoff
a9d84f4e44
We require attrs 16.0.0
...
Ref: https://github.com/matrix-org/synapse/issues/3945
2018-09-25 10:43:39 +01:00
Matthew Hodgson
787d22ed6c
Only lazy load self-members on initial sync
...
Given we have disabled lazy loading for incr syncs in #3840 , we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
2018-09-25 00:49:26 +01:00
Amber Brown
fbe5ba25f6
Merge branch 'master' into develop
2018-09-25 03:10:01 +10:00
Amber Brown
5121ae97f5
Internal Changes
...
----------------
- Fix incompatibility with older Twisted version in tests. Thanks
@OlegGirko!
([\#3940](https://github.com/matrix-org/synapse/issues/3940 ))
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlupGfUACgkQIwi0edOS
ShH/rgf5AdLpXHVmWN1YiKuwwgiK9i8/41pqcj5ZiVn+UQjzUIBNc0ERNf2gfoRU
iwECt3rDMatB2Xxd4VFNoTn6wFtuqNV4ke7jeghMU2KxAzmwX8F6b7qJWz9sywyO
7cOtByqVAKN2v8qLDI+wcGuBkxkFHMF8iKBQApkDn1gqvvqHs/yk9FprhLJOYvaz
iOvrgm4T3XLiMOqDmVH1byemynLQCagol/8rn63NnnDVaoHd9Cgar0qRMta2MYBL
xXjOJ2HS35Y7ZuIPhRAmOVhs6DXmjBnEgfxeDV+GAF1Co8qEH5Jbfq/vu/2gpKKf
TsA5HG3mDGBit67X9evU9SN5WFlcQw==
=CZ+X
-----END PGP SIGNATURE-----
Merge tag 'v0.33.5.1'
Internal Changes
----------------
- Fix incompatibility with older Twisted version in tests. Thanks
@OlegGirko!
([\#3940](https://github.com/matrix-org/synapse/issues/3940 ))
2018-09-25 03:09:30 +10:00
Amber Brown
fc691ca97c
changelog
2018-09-25 02:54:59 +10:00
Amber Brown
6b6cb32297
bump version
2018-09-25 02:54:34 +10:00
Amber Brown
e37c221b97
changelog for 3940
2018-09-25 02:51:55 +10:00
Oleg Girko
7d3f639844
Fix compatibility issue with older Twisted in tests.
...
Older Twisted (18.4.0) returns TimeoutError instead of
ConnectingCancelledError when connection times out.
This change allows tests to be compatible with this behaviour.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
2018-09-25 02:51:18 +10:00
Amber Brown
04eed80a73
Merge branch 'master' into develop
2018-09-24 23:42:25 +10:00
Amber Brown
829213523e
Features
...
--------
- Python 3.5 and 3.6 support is now in beta.
([\#3576](https://github.com/matrix-org/synapse/issues/3576 ))
- Implement `event_format` filter param in `/sync`
([\#3790](https://github.com/matrix-org/synapse/issues/3790 ))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](https://github.com/matrix-org/synapse/issues/3846 ))
Bugfixes
--------
- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](https://github.com/matrix-org/synapse/issues/3788 ))
- guest users should not be part of mau total
([\#3800](https://github.com/matrix-org/synapse/issues/3800 ))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](https://github.com/matrix-org/synapse/issues/3804 ))
- Fix existing room tags not coming down sync when joining a room
([\#3810](https://github.com/matrix-org/synapse/issues/3810 ))
- Fix jwt import check
([\#3824](https://github.com/matrix-org/synapse/issues/3824 ))
- fix VOIP crashes under Python 3 (#3821 )
([\#3835](https://github.com/matrix-org/synapse/issues/3835 ))
- Fix manhole so that it works with latest openssh clients
([\#3841](https://github.com/matrix-org/synapse/issues/3841 ))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](https://github.com/matrix-org/synapse/issues/3845 ))
- Show heroes if room name/canonical alias has been deleted
([\#3851](https://github.com/matrix-org/synapse/issues/3851 ))
- Fix handling of redacted events from federation
([\#3859](https://github.com/matrix-org/synapse/issues/3859 ))
- ([\#3874](https://github.com/matrix-org/synapse/issues/3874 ))
- Mitigate outbound federation randomly becoming wedged
([\#3875](https://github.com/matrix-org/synapse/issues/3875 ))
Internal Changes
----------------
- CircleCI tests now run on the potential merge of a PR.
([\#3704](https://github.com/matrix-org/synapse/issues/3704 ))
- http/ is now ported to Python 3.
([\#3771](https://github.com/matrix-org/synapse/issues/3771 ))
- Improve human readable error messages for threepid
registration/account update
([\#3789](https://github.com/matrix-org/synapse/issues/3789 ))
- Make /sync slightly faster by avoiding needless copies
([\#3795](https://github.com/matrix-org/synapse/issues/3795 ))
- handlers/ is now ported to Python 3.
([\#3803](https://github.com/matrix-org/synapse/issues/3803 ))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](https://github.com/matrix-org/synapse/issues/3805 ))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](https://github.com/matrix-org/synapse/issues/3806 ))
- tests/ is now ported to Python 3.
([\#3808](https://github.com/matrix-org/synapse/issues/3808 ))
- crypto/ is now ported to Python 3.
([\#3822](https://github.com/matrix-org/synapse/issues/3822 ))
- rest/ is now ported to Python 3.
([\#3823](https://github.com/matrix-org/synapse/issues/3823 ))
- add some logging for the keyring queue
([\#3826](https://github.com/matrix-org/synapse/issues/3826 ))
- speed up lazy loading by 2-3x
([\#3827](https://github.com/matrix-org/synapse/issues/3827 ))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](https://github.com/matrix-org/synapse/issues/3834 ))
- Disable lazy loading for incremental syncs for now
([\#3840](https://github.com/matrix-org/synapse/issues/3840 ))
- federation/ is now ported to Python 3.
([\#3847](https://github.com/matrix-org/synapse/issues/3847 ))
- Log when we retry outbound requests
([\#3853](https://github.com/matrix-org/synapse/issues/3853 ))
- Removed some excess logging messages.
([\#3855](https://github.com/matrix-org/synapse/issues/3855 ))
- Speed up purge history for rooms that have been previously purged
([\#3856](https://github.com/matrix-org/synapse/issues/3856 ))
- Refactor some HTTP timeout code.
([\#3857](https://github.com/matrix-org/synapse/issues/3857 ))
- Fix running merged builds on CircleCI
([\#3858](https://github.com/matrix-org/synapse/issues/3858 ))
- Fix typo in replication stream exception.
([\#3860](https://github.com/matrix-org/synapse/issues/3860 ))
- Add in flight real time metrics for Measure blocks
([\#3871](https://github.com/matrix-org/synapse/issues/3871 ))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872 ))
- mention jemalloc in the README
([\#3877](https://github.com/matrix-org/synapse/issues/3877 ))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](https://github.com/matrix-org/synapse/issues/3888 ))
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAluo6WIACgkQIwi0edOS
ShHFIwf/ZURGbih+6q4jy8voc4wUbgtuNGabmFLQeXGhIXIOYltvKOrB8wcCEi/4
jODztPqewikZoI7NFA4M44GToQKacuIEOYK/M513l3t/mEpBQYDdvLGDccM+6XGc
RcrWqQEfJYkiNyWNmLx5TJ4bYaNelrpfUyNbbONph0wLO7o6f7ANoGyqqVKm/7pD
ufUwwBLqzd2U2vuvNCbQLSW0LkognP91vYytHMN5NTkoAR4BSYfDxQCShUSoNSlJ
098CEYjXZaDcY9ypGA6wlflbNfl3NC2KcTajLOCHdKe2U+FbEMc/9Ou690GbDRJ9
6L+Bonfwx0tsF04LHsteTdyTIOVp+Q==
=Dv5r
-----END PGP SIGNATURE-----
Merge tag 'v0.33.5'
Features
--------
- Python 3.5 and 3.6 support is now in beta.
([\#3576](https://github.com/matrix-org/synapse/issues/3576 ))
- Implement `event_format` filter param in `/sync`
([\#3790](https://github.com/matrix-org/synapse/issues/3790 ))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](https://github.com/matrix-org/synapse/issues/3846 ))
Bugfixes
--------
- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](https://github.com/matrix-org/synapse/issues/3788 ))
- guest users should not be part of mau total
([\#3800](https://github.com/matrix-org/synapse/issues/3800 ))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](https://github.com/matrix-org/synapse/issues/3804 ))
- Fix existing room tags not coming down sync when joining a room
([\#3810](https://github.com/matrix-org/synapse/issues/3810 ))
- Fix jwt import check
([\#3824](https://github.com/matrix-org/synapse/issues/3824 ))
- fix VOIP crashes under Python 3 (#3821 )
([\#3835](https://github.com/matrix-org/synapse/issues/3835 ))
- Fix manhole so that it works with latest openssh clients
([\#3841](https://github.com/matrix-org/synapse/issues/3841 ))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](https://github.com/matrix-org/synapse/issues/3845 ))
- Show heroes if room name/canonical alias has been deleted
([\#3851](https://github.com/matrix-org/synapse/issues/3851 ))
- Fix handling of redacted events from federation
([\#3859](https://github.com/matrix-org/synapse/issues/3859 ))
- ([\#3874](https://github.com/matrix-org/synapse/issues/3874 ))
- Mitigate outbound federation randomly becoming wedged
([\#3875](https://github.com/matrix-org/synapse/issues/3875 ))
Internal Changes
----------------
- CircleCI tests now run on the potential merge of a PR.
([\#3704](https://github.com/matrix-org/synapse/issues/3704 ))
- http/ is now ported to Python 3.
([\#3771](https://github.com/matrix-org/synapse/issues/3771 ))
- Improve human readable error messages for threepid
registration/account update
([\#3789](https://github.com/matrix-org/synapse/issues/3789 ))
- Make /sync slightly faster by avoiding needless copies
([\#3795](https://github.com/matrix-org/synapse/issues/3795 ))
- handlers/ is now ported to Python 3.
([\#3803](https://github.com/matrix-org/synapse/issues/3803 ))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](https://github.com/matrix-org/synapse/issues/3805 ))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](https://github.com/matrix-org/synapse/issues/3806 ))
- tests/ is now ported to Python 3.
([\#3808](https://github.com/matrix-org/synapse/issues/3808 ))
- crypto/ is now ported to Python 3.
([\#3822](https://github.com/matrix-org/synapse/issues/3822 ))
- rest/ is now ported to Python 3.
([\#3823](https://github.com/matrix-org/synapse/issues/3823 ))
- add some logging for the keyring queue
([\#3826](https://github.com/matrix-org/synapse/issues/3826 ))
- speed up lazy loading by 2-3x
([\#3827](https://github.com/matrix-org/synapse/issues/3827 ))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](https://github.com/matrix-org/synapse/issues/3834 ))
- Disable lazy loading for incremental syncs for now
([\#3840](https://github.com/matrix-org/synapse/issues/3840 ))
- federation/ is now ported to Python 3.
([\#3847](https://github.com/matrix-org/synapse/issues/3847 ))
- Log when we retry outbound requests
([\#3853](https://github.com/matrix-org/synapse/issues/3853 ))
- Removed some excess logging messages.
([\#3855](https://github.com/matrix-org/synapse/issues/3855 ))
- Speed up purge history for rooms that have been previously purged
([\#3856](https://github.com/matrix-org/synapse/issues/3856 ))
- Refactor some HTTP timeout code.
([\#3857](https://github.com/matrix-org/synapse/issues/3857 ))
- Fix running merged builds on CircleCI
([\#3858](https://github.com/matrix-org/synapse/issues/3858 ))
- Fix typo in replication stream exception.
([\#3860](https://github.com/matrix-org/synapse/issues/3860 ))
- Add in flight real time metrics for Measure blocks
([\#3871](https://github.com/matrix-org/synapse/issues/3871 ))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872 ))
- mention jemalloc in the README
([\#3877](https://github.com/matrix-org/synapse/issues/3877 ))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](https://github.com/matrix-org/synapse/issues/3888 ))
2018-09-24 23:41:35 +10:00
Amber Brown
e3aa2c0b75
towncrier
2018-09-24 23:40:27 +10:00
Amber Brown
e302f40e20
update version
2018-09-24 23:40:05 +10:00
Erik Johnston
d3f80cbc9c
Newsfile
2018-09-21 14:24:39 +01:00
Erik Johnston
8601c24287
Fix some instances of ExpiringCache not expiring cache items
...
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.
This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
2018-09-21 14:19:46 +01:00
Erik Johnston
ad53a5497d
Merge pull request #3927 from matrix-org/erikj/handle_background_errors
...
Handle exceptions thrown by background tasks
2018-09-21 09:26:30 +01:00
Matthew Hodgson
a2ddaa90f2
Always LL ourselves if we're in a room to simplify clients ( #3916 )
...
Should fix https://github.com/vector-im/riot-web/issues/7209
2018-09-20 21:21:54 +01:00
Erik Johnston
94ae1dea3c
Add missing logger
2018-09-20 17:05:34 +01:00
Erik Johnston
168491c412
Newsfile
2018-09-20 16:16:52 +01:00
Erik Johnston
9ea408441f
Handle exceptions thrown by background tasks
...
Fixes #3921
2018-09-20 16:15:21 +01:00
Jan Christian Grünhage
3af7a95a9d
add changelog
2018-09-20 14:55:11 +02:00
Jan Christian Grünhage
8dfb33d325
make python 3 work in the docker container
2018-09-20 14:55:11 +02:00
Erik Johnston
13f6f1624b
Newsfile
2018-09-20 13:52:09 +01:00
Erik Johnston
b28a7ed503
Fix spurious exceptions when client closes conncetion
...
If a HTTP handler throws an exception while processing a request we
automatically write a JSON error response. If the handler had already
started writing a response twisted throws an exception.
We should check for this case and simple abort the connection if there
was an error after the response had started being written.
2018-09-20 13:44:20 +01:00
Neil Johnson
23b53b4ef8
Merge pull request #3868 from matrix-org/neilj/fix_room_invite_mail_links
...
Neilj/fix room invite mail links
2018-09-20 13:32:38 +01:00
Amber Brown
1f3f5fcf52
Fix client IPs being broken on Python 3 ( #3908 )
2018-09-20 20:14:34 +10:00
Erik Johnston
3fd68d533b
Merge pull request #3914 from matrix-org/erikj/remove_retry_cache
...
Remove get_destination_retry_timings cache
2018-09-20 10:54:49 +01:00
Amber Brown
741571cf22
Add a way to run tests in PostgreSQL in Docker ( #3699 )
2018-09-20 18:12:45 +10:00
Amber Brown
aeca5a5ed5
Add a regression test for logging on failed connections ( #3912 )
2018-09-20 16:28:18 +10:00
Richard van der Hoff
642199570c
Improve the logging when handling a federation transaction ( #3904 )
...
Let's try to rationalise the logging that happens when we are processing an
incoming transaction, to make it easier to figure out what is going wrong when
they take ages. In particular:
- make everything start with a [room_id event_id] prefix
- make sure we log a warning when catching exceptions rather than just turning
them into other, more cryptic, exceptions.
2018-09-19 17:28:18 +01:00
Erik Johnston
bbab6ebfd9
Fix up changelog and remove spurious comment
2018-09-19 14:45:14 +01:00
Erik Johnston
392a54128c
pep8
2018-09-19 14:37:49 +01:00
Erik Johnston
83ee5592a5
Newsfile
2018-09-19 14:22:59 +01:00
Erik Johnston
b9158ac2bf
Remove get_destination_retry_timings cache
...
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.
Fixes #3798
2018-09-19 14:22:57 +01:00
Erik Johnston
cb016baa37
Merge pull request #3910 from matrix-org/erikj/update_timeout
...
Update to use new timeout function everywhere.
2018-09-19 11:45:11 +01:00
Erik Johnston
80d2d50f47
Fixup
2018-09-19 11:19:47 +01:00
Erik Johnston
9407bcf37a
Replace custom DeferredTimeoutError with defer.TimeoutError
2018-09-19 11:07:29 +01:00
Erik Johnston
6c48aa0256
Run canceller first to allow it to generate correct error
2018-09-19 11:07:27 +01:00
Erik Johnston
6bbe3d5732
Newsfile
2018-09-19 10:43:13 +01:00
Erik Johnston
a334e1cace
Update to use new timeout function everywhere.
...
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842 .
The new helper function doesn't suffer from this problem.
2018-09-19 10:39:40 +01:00
Richard van der Hoff
05b9937cd7
update changelog for #3909
2018-09-19 09:17:54 +01:00