Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-10-01 08:25:44 -04:00
Code Issues Packages Projects Releases Wiki Activity
12,199 Commits 4 Branches 609 Tags 60 MiB
948a6d8776
Commit Graph

12199 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Richard van der Hoff
948a6d8776 changelog 2018-09-27 11:37:39 +01:00
Richard van der Hoff
333bee27f5 Include event when resolving state for missing prevs 
If we have a forward extremity for a room as `E`, and you receive `A`, `B`,
s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need
to do state res between `X` and `E`.

When that happens, we need to make sure we include `X` in the state that goes
into the state res alg.

Fixes #3934.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff
bd61c82bdf Include state from remote servers in pdu handling 
If we've fetched state events from remote servers in order to resolve the state
for a new event, we need to actually pass those events into
resolve_events_with_factory (so that it can do the state res) and then persist
the ones we need - otherwise other bits of the codebase get confused about why
we have state groups pointing to non-existent events.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff
a215b698c4 Fix "unhashable type: 'list'" exception in federation handling 
get_state_groups returns a map from state_group_id to a list of FrozenEvents,
so was very much the wrong thing to be putting as one of the entries in the
list passed to resolve_events_with_factory (which expects maps from
(event_type, state_key) to event id).

We actually want get_state_groups_ids().values() rather than
get_state_groups().

This fixes the main problem in #3923, but there are other problems with this
bit of code which get discovered once you do so.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff
28223841e0 more comments 2018-09-27 11:31:51 +01:00
Richard van der Hoff
ad8e137062 changelog 2018-09-27 11:31:51 +01:00
Richard van der Hoff
e3c159863d Clarifications in FederationHandler 
* add some comments on things that look a bit bogus
* rename this `state` variable to avoid confusion with the `state` used
  elsewhere in this function. (There was no actual conflict, but it was
  a confusing bit of spaghetti.)
 2018-09-27 11:31:51 +01:00
Richard van der Hoff
92abd3d6d6
Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 
Logging improvements
 2018-09-27 11:27:46 +01:00
Richard van der Hoff
4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959) 
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
 2018-09-27 11:25:34 +01:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state (#3958) 
I spent ages trying to figure out how I was going mad...
 2018-09-27 11:22:25 +01:00
Amber Brown
2c695fd1aa
Merge pull request #3963 from matrix-org/rav/get_state_for_room_docstring 
fix docstring for FederationClient.get_state_for_room
 2018-09-27 18:06:38 +10:00
Richard van der Hoff
e70b4ce069 Logging improvements 
Some logging tweaks to help with debugging incoming federation transactions
 2018-09-26 17:36:14 +01:00
Richard van der Hoff
ab59f3d8da changelog 2018-09-26 16:52:24 +01:00
Richard van der Hoff
607eec0456 fix docstring for FederationClient.get_state_for_room 
trivial fixes for docstring
 2018-09-26 16:52:24 +01:00
Amber Brown
d4e0861ff9
Reduce the load on our CI (#3957) 
* changelog

* reduce circleci config

* plus a handy script

* fix regex
 2018-09-27 00:23:21 +10:00
Richard van der Hoff
bf01efb864 Merge branch 'develop' into rav/hacky_cache_factor_fix 2018-09-26 13:24:07 +01:00
Erik Johnston
8834396b8a Actuall set cache factors in workers 2018-09-26 13:23:02 +01:00
Richard van der Hoff
4e8276a34a
Merge pull request #3956 from matrix-org/rav/fix_expiring_cache_len 
Fix ExpiringCache.__len__ to be accurate
 2018-09-26 13:10:13 +01:00
Richard van der Hoff
7ee94fc1ba Log which cache is throwing exceptions 2018-09-26 12:43:08 +01:00
Amber Brown
66a1d57adb
Merge pull request #3948 from matrix-org/rav/no_symlink_synctl 
Move synctl into top dir to avoid a symlink
 2018-09-26 21:41:58 +10:00
Amber Brown
c2185f14d7
Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu 
Comments and interface cleanup for on_receive_pdu
 2018-09-26 21:41:26 +10:00
Richard van der Hoff
8ac9fa7375 changelog 2018-09-26 12:35:10 +01:00
Erik Johnston
3baf6e1667 Fix ExpiringCache.__len__ to be accurate 
It used to try and produce an estimate, which was sometimes negative.
This caused metrics to be sad, so lets always just calculate it from
scratch.

(This appears to have been a longstanding bug, but one which has been made more
of a problem by #3932 and #3933).

(This was originally done by Erik as part of #3933. I'm cherry-picking it
because really it's a fix in its own right)
 2018-09-26 12:32:29 +01:00
Richard van der Hoff
f65163627f
Merge pull request #3911 from matrix-org/jcgruenhage/docker-support-python3 
make python 3 work in the docker container
 2018-09-25 15:18:09 +01:00
Jan Christian Grünhage
df55a943ca
Update Dockerfile 2018-09-25 14:33:38 +02:00
Jan Christian Grünhage
e7fa032126
Update .dockerignore 2018-09-25 14:33:19 +02:00
Richard van der Hoff
0649306fde Update grafana dashboard 2018-09-25 13:29:28 +01:00
Richard van der Hoff
4c3e7eeec5
Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches 
Fix some instances of ExpiringCache not expiring cache items
 2018-09-25 12:02:57 +01:00
Jérémy Farnaud
6cf261930a added "media-src: 'self'" to CSP for resources (#3578) 
Synapse doesn’t allow for media resources to be played directly from
Chrome. It is a problem for users on other networks (e.g. IRC)
communicating with Matrix users through a gateway. The gateway sends
them the raw URL for the resource when a Matrix user uploads a video
and the video cannot be played directly in Chrome using that URL.

Chrome argues it is not authorized to play the video because of the
Content Security Policy. Chrome checks for the "media-src" policy which
is missing, and defauts to the "default-src" policy which is "none".

As Synapse already sends "object-src: 'self'" I thought it wouldn’t be
a problem to add "media-src: 'self'" to the CSP to fix this problem.
 2018-09-25 11:55:02 +01:00
Richard van der Hoff
94f7befc31
Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered 
Fix spurious exceptions when client closes conncetion
 2018-09-25 11:52:06 +01:00
Richard van der Hoff
bd469adaa9 changelog 2018-09-25 11:22:17 +01:00
Richard van der Hoff
c53336986d Move synctl into top dir to avoid a symlink 
symlinks apparently break setuptools on python3 and alpine
(https://bugs.python.org/issue31940), so let's stop using a symlink and just
use the file directly.
 2018-09-25 11:19:27 +01:00
Richard van der Hoff
e4e96486a9
Merge pull request #3947 from matrix-org/rav/attr_version 
We require attrs 16.0.0
 2018-09-25 11:16:42 +01:00
Richard van der Hoff
45a1053d44 changelog 2018-09-25 10:45:34 +01:00
Richard van der Hoff
a9d84f4e44 We require attrs 16.0.0 
Ref: https://github.com/matrix-org/synapse/issues/3945
 2018-09-25 10:43:39 +01:00
Matthew Hodgson
787d22ed6c
Only lazy load self-members on initial sync 
Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync.  Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
 2018-09-25 00:49:26 +01:00
Amber Brown
fbe5ba25f6 Merge branch 'master' into develop 2018-09-25 03:10:01 +10:00
Amber Brown
5121ae97f5 Internal Changes 
----------------
 
 - Fix incompatibility with older Twisted version in tests. Thanks
   @OlegGirko!
 ([\#3940](https://github.com/matrix-org/synapse/issues/3940))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlupGfUACgkQIwi0edOS
 ShH/rgf5AdLpXHVmWN1YiKuwwgiK9i8/41pqcj5ZiVn+UQjzUIBNc0ERNf2gfoRU
 iwECt3rDMatB2Xxd4VFNoTn6wFtuqNV4ke7jeghMU2KxAzmwX8F6b7qJWz9sywyO
 7cOtByqVAKN2v8qLDI+wcGuBkxkFHMF8iKBQApkDn1gqvvqHs/yk9FprhLJOYvaz
 iOvrgm4T3XLiMOqDmVH1byemynLQCagol/8rn63NnnDVaoHd9Cgar0qRMta2MYBL
 xXjOJ2HS35Y7ZuIPhRAmOVhs6DXmjBnEgfxeDV+GAF1Co8qEH5Jbfq/vu/2gpKKf
 TsA5HG3mDGBit67X9evU9SN5WFlcQw==
 =CZ+X
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.5.1'

Internal Changes
----------------

- Fix incompatibility with older Twisted version in tests. Thanks
  @OlegGirko!
([\#3940](https://github.com/matrix-org/synapse/issues/3940))
 2018-09-25 03:09:30 +10:00
Amber Brown
fc691ca97c changelog 2018-09-25 02:54:59 +10:00
Amber Brown
6b6cb32297 bump version 2018-09-25 02:54:34 +10:00
Amber Brown
e37c221b97 changelog for 3940 2018-09-25 02:51:55 +10:00
Oleg Girko
7d3f639844 Fix compatibility issue with older Twisted in tests. 
Older Twisted (18.4.0) returns TimeoutError instead of
ConnectingCancelledError when connection times out.
This change allows tests to be compatible with this behaviour.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
 2018-09-25 02:51:18 +10:00
Amber Brown
04eed80a73 Merge branch 'master' into develop 2018-09-24 23:42:25 +10:00
Amber Brown
829213523e Features 
--------
 
 - Python 3.5 and 3.6 support is now in beta.
 ([\#3576](https://github.com/matrix-org/synapse/issues/3576))
 - Implement `event_format` filter param in `/sync`
 ([\#3790](https://github.com/matrix-org/synapse/issues/3790))
 - Add synapse_admin_mau:registered_reserved_users metric to expose
 number of real reaserved users
 ([\#3846](https://github.com/matrix-org/synapse/issues/3846))
 
 Bugfixes
 --------
 
 - Remove connection ID for replication prometheus metrics, as it creates
 a large number of new series.
 ([\#3788](https://github.com/matrix-org/synapse/issues/3788))
 - guest users should not be part of mau total
 ([\#3800](https://github.com/matrix-org/synapse/issues/3800))
 - Bump dependency on pyopenssl 16.x, to avoid incompatibility with
 recent Twisted.
 ([\#3804](https://github.com/matrix-org/synapse/issues/3804))
 - Fix existing room tags not coming down sync when joining a room
 ([\#3810](https://github.com/matrix-org/synapse/issues/3810))
 - Fix jwt import check
 ([\#3824](https://github.com/matrix-org/synapse/issues/3824))
 - fix VOIP crashes under Python 3 (#3821)
 ([\#3835](https://github.com/matrix-org/synapse/issues/3835))
 - Fix manhole so that it works with latest openssh clients
 ([\#3841](https://github.com/matrix-org/synapse/issues/3841))
 - Fix outbound requests occasionally wedging, which can result in
 federation breaking between servers.
 ([\#3845](https://github.com/matrix-org/synapse/issues/3845))
 - Show heroes if room name/canonical alias has been deleted
 ([\#3851](https://github.com/matrix-org/synapse/issues/3851))
 - Fix handling of redacted events from federation
 ([\#3859](https://github.com/matrix-org/synapse/issues/3859))
 -  ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
 - Mitigate outbound federation randomly becoming wedged
 ([\#3875](https://github.com/matrix-org/synapse/issues/3875))
 
 Internal Changes
 ----------------
 
 - CircleCI tests now run on the potential merge of a PR.
 ([\#3704](https://github.com/matrix-org/synapse/issues/3704))
 - http/ is now ported to Python 3.
 ([\#3771](https://github.com/matrix-org/synapse/issues/3771))
 - Improve human readable error messages for threepid
 registration/account update
 ([\#3789](https://github.com/matrix-org/synapse/issues/3789))
 - Make /sync slightly faster by avoiding needless copies
 ([\#3795](https://github.com/matrix-org/synapse/issues/3795))
 - handlers/ is now ported to Python 3.
 ([\#3803](https://github.com/matrix-org/synapse/issues/3803))
 - Limit the number of PDUs/EDUs per federation transaction
 ([\#3805](https://github.com/matrix-org/synapse/issues/3805))
 - Only start postgres instance for postgres tests on Travis CI
 ([\#3806](https://github.com/matrix-org/synapse/issues/3806))
 - tests/ is now ported to Python 3.
 ([\#3808](https://github.com/matrix-org/synapse/issues/3808))
 - crypto/ is now ported to Python 3.
 ([\#3822](https://github.com/matrix-org/synapse/issues/3822))
 - rest/ is now ported to Python 3.
 ([\#3823](https://github.com/matrix-org/synapse/issues/3823))
 - add some logging for the keyring queue
 ([\#3826](https://github.com/matrix-org/synapse/issues/3826))
 - speed up lazy loading by 2-3x
 ([\#3827](https://github.com/matrix-org/synapse/issues/3827))
 - Improved Dockerfile to remove build requirements after building
 reducing the image size.
 ([\#3834](https://github.com/matrix-org/synapse/issues/3834))
 - Disable lazy loading for incremental syncs for now
 ([\#3840](https://github.com/matrix-org/synapse/issues/3840))
 - federation/ is now ported to Python 3.
 ([\#3847](https://github.com/matrix-org/synapse/issues/3847))
 - Log when we retry outbound requests
 ([\#3853](https://github.com/matrix-org/synapse/issues/3853))
 - Removed some excess logging messages.
 ([\#3855](https://github.com/matrix-org/synapse/issues/3855))
 - Speed up purge history for rooms that have been previously purged
 ([\#3856](https://github.com/matrix-org/synapse/issues/3856))
 - Refactor some HTTP timeout code.
 ([\#3857](https://github.com/matrix-org/synapse/issues/3857))
 - Fix running merged builds on CircleCI
 ([\#3858](https://github.com/matrix-org/synapse/issues/3858))
 - Fix typo in replication stream exception.
 ([\#3860](https://github.com/matrix-org/synapse/issues/3860))
 - Add in flight real time metrics for Measure blocks
 ([\#3871](https://github.com/matrix-org/synapse/issues/3871))
 - Disable buffering and automatic retrying in treq requests to prevent
 timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
 - mention jemalloc in the README
 ([\#3877](https://github.com/matrix-org/synapse/issues/3877))
 - Remove unmaintained "nuke-room-from-db.sh" script
 ([\#3888](https://github.com/matrix-org/synapse/issues/3888))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAluo6WIACgkQIwi0edOS
 ShHFIwf/ZURGbih+6q4jy8voc4wUbgtuNGabmFLQeXGhIXIOYltvKOrB8wcCEi/4
 jODztPqewikZoI7NFA4M44GToQKacuIEOYK/M513l3t/mEpBQYDdvLGDccM+6XGc
 RcrWqQEfJYkiNyWNmLx5TJ4bYaNelrpfUyNbbONph0wLO7o6f7ANoGyqqVKm/7pD
 ufUwwBLqzd2U2vuvNCbQLSW0LkognP91vYytHMN5NTkoAR4BSYfDxQCShUSoNSlJ
 098CEYjXZaDcY9ypGA6wlflbNfl3NC2KcTajLOCHdKe2U+FbEMc/9Ou690GbDRJ9
 6L+Bonfwx0tsF04LHsteTdyTIOVp+Q==
 =Dv5r
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.5'

Features
--------

- Python 3.5 and 3.6 support is now in beta.
([\#3576](https://github.com/matrix-org/synapse/issues/3576))
- Implement `event_format` filter param in `/sync`
([\#3790](https://github.com/matrix-org/synapse/issues/3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](https://github.com/matrix-org/synapse/issues/3846))

Bugfixes
--------

- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](https://github.com/matrix-org/synapse/issues/3788))
- guest users should not be part of mau total
([\#3800](https://github.com/matrix-org/synapse/issues/3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](https://github.com/matrix-org/synapse/issues/3804))
- Fix existing room tags not coming down sync when joining a room
([\#3810](https://github.com/matrix-org/synapse/issues/3810))
- Fix jwt import check
([\#3824](https://github.com/matrix-org/synapse/issues/3824))
- fix VOIP crashes under Python 3 (#3821)
([\#3835](https://github.com/matrix-org/synapse/issues/3835))
- Fix manhole so that it works with latest openssh clients
([\#3841](https://github.com/matrix-org/synapse/issues/3841))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](https://github.com/matrix-org/synapse/issues/3845))
- Show heroes if room name/canonical alias has been deleted
([\#3851](https://github.com/matrix-org/synapse/issues/3851))
- Fix handling of redacted events from federation
([\#3859](https://github.com/matrix-org/synapse/issues/3859))
-  ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
- Mitigate outbound federation randomly becoming wedged
([\#3875](https://github.com/matrix-org/synapse/issues/3875))

Internal Changes
----------------

- CircleCI tests now run on the potential merge of a PR.
([\#3704](https://github.com/matrix-org/synapse/issues/3704))
- http/ is now ported to Python 3.
([\#3771](https://github.com/matrix-org/synapse/issues/3771))
- Improve human readable error messages for threepid
registration/account update
([\#3789](https://github.com/matrix-org/synapse/issues/3789))
- Make /sync slightly faster by avoiding needless copies
([\#3795](https://github.com/matrix-org/synapse/issues/3795))
- handlers/ is now ported to Python 3.
([\#3803](https://github.com/matrix-org/synapse/issues/3803))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](https://github.com/matrix-org/synapse/issues/3805))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](https://github.com/matrix-org/synapse/issues/3806))
- tests/ is now ported to Python 3.
([\#3808](https://github.com/matrix-org/synapse/issues/3808))
- crypto/ is now ported to Python 3.
([\#3822](https://github.com/matrix-org/synapse/issues/3822))
- rest/ is now ported to Python 3.
([\#3823](https://github.com/matrix-org/synapse/issues/3823))
- add some logging for the keyring queue
([\#3826](https://github.com/matrix-org/synapse/issues/3826))
- speed up lazy loading by 2-3x
([\#3827](https://github.com/matrix-org/synapse/issues/3827))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](https://github.com/matrix-org/synapse/issues/3834))
- Disable lazy loading for incremental syncs for now
([\#3840](https://github.com/matrix-org/synapse/issues/3840))
- federation/ is now ported to Python 3.
([\#3847](https://github.com/matrix-org/synapse/issues/3847))
- Log when we retry outbound requests
([\#3853](https://github.com/matrix-org/synapse/issues/3853))
- Removed some excess logging messages.
([\#3855](https://github.com/matrix-org/synapse/issues/3855))
- Speed up purge history for rooms that have been previously purged
([\#3856](https://github.com/matrix-org/synapse/issues/3856))
- Refactor some HTTP timeout code.
([\#3857](https://github.com/matrix-org/synapse/issues/3857))
- Fix running merged builds on CircleCI
([\#3858](https://github.com/matrix-org/synapse/issues/3858))
- Fix typo in replication stream exception.
([\#3860](https://github.com/matrix-org/synapse/issues/3860))
- Add in flight real time metrics for Measure blocks
([\#3871](https://github.com/matrix-org/synapse/issues/3871))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
- mention jemalloc in the README
([\#3877](https://github.com/matrix-org/synapse/issues/3877))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](https://github.com/matrix-org/synapse/issues/3888))
 2018-09-24 23:41:35 +10:00
Amber Brown
e3aa2c0b75 towncrier 2018-09-24 23:40:27 +10:00
Amber Brown
e302f40e20 update version 2018-09-24 23:40:05 +10:00
Erik Johnston
d3f80cbc9c Newsfile 2018-09-21 14:24:39 +01:00
Erik Johnston
8601c24287 Fix some instances of ExpiringCache not expiring cache items 
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.

This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
 2018-09-21 14:19:46 +01:00
Erik Johnston
ad53a5497d
Merge pull request #3927 from matrix-org/erikj/handle_background_errors 
Handle exceptions thrown by background tasks
 2018-09-21 09:26:30 +01:00
Matthew Hodgson
a2ddaa90f2
Always LL ourselves if we're in a room to simplify clients (#3916) 
Should fix https://github.com/vector-im/riot-web/issues/7209
 2018-09-20 21:21:54 +01:00