Richard van der Hoff
74854a9719
Use recaptcha_ajax.js directly from Google
...
This was originally done in commit c75b71a397
,
but got reverted on this branch due to the PR (#3677 ) being based on the wrong
branch.
We're ready to merge this to master now, so let's make it match
release-v0.33.3.
2018-08-22 14:30:49 +01:00
Richard van der Hoff
48fec67536
Features
...
--------
- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439 ))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184 ))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568 ))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574 ))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589 ))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633 ))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653 ))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654 ))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655 ))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662 ))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664 ))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670 ))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687 ))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694 ))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708 ))
Bugfixes
--------
- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658 ))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661 ))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676 ))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677 ))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681 ))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689 ))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692 ))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700 ))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701 ))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705 ), [\#3713](https://github.com/matrix-org/synapse/issues/3713 ))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710 ))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719 ))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723 ))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732 ))
Deprecations and Removals
-------------------------
- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst ). ([\#3703](https://github.com/matrix-org/synapse/issues/3703 ))
Internal Changes
----------------
- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423 ))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632 ))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647 ))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660 ))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668 ))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669 ))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678 ))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679 ))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684 ))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690 ))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707 ))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709 ))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712 ))
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJbfWKDAAoJEIofk9V1tejVHOMH/RmP+axF2K+Socnnb5Fa6RTo
pDJXVwsvNQxcwzZIJXh2TRilVr88jx7A1PHUde0TmkHRmis/bDzZxa59QRA4rFEy
fLYMx9Tyt0RzAJrEd0p7jbIkJxxPm5PbMWfCgVfzg44q0kepRW6ZVl1+kFHDrsCo
ocE03WdLe2sRHSudaVtBLOVMA1lDE22v40/7xiu9IbUOnF+WZ3cjXYX+SNr5eb6g
7YGbXOk4/E8XoUnpzH5oEnNtiDK5piRrr7o1282kecY20ZMNzKgseBHU6nYMv5ZO
TUpCPwbukmzeGHITupY1RkzpmBbEIJTCnmLP8fiUw+RWznYyFC/aRFMiWTzle3A=
=ngmn
-----END PGP SIGNATURE-----
Merge tag 'v0.33.3'
Features
--------
- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439 ))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184 ))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568 ))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574 ))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589 ))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633 ))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653 ))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654 ))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655 ))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662 ))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664 ))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670 ))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687 ))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694 ))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708 ))
Bugfixes
--------
- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658 ))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661 ))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676 ))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677 ))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681 ))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689 ))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692 ))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700 ))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701 ))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705 ), [\#3713](https://github.com/matrix-org/synapse/issues/3713 ))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710 ))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719 ))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723 ))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732 ))
Deprecations and Removals
-------------------------
- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst ). ([\#3703](https://github.com/matrix-org/synapse/issues/3703 ))
Internal Changes
----------------
- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423 ))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632 ))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647 ))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660 ))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668 ))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669 ))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678 ))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679 ))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684 ))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690 ))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707 ))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709 ))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712 ))
2018-08-22 14:28:55 +01:00
Richard van der Hoff
4e5a4549b6
bump version to 0.33.3
2018-08-22 14:07:10 +01:00
Erik Johnston
db10f553ba
Merge pull request #3724 from Half-Shot/hs/guest-fetch-event
...
Allow guests to use /rooms/:roomId/event/:eventId
2018-08-22 13:41:08 +01:00
Erik Johnston
764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles
...
Allow profile updates to happen on workers
2018-08-22 11:35:55 +01:00
Erik Johnston
8432e2ebd7
Rename WorkerProfileHandler to BaseProfileHandler
2018-08-22 10:13:40 +01:00
Erik Johnston
a81f140880
Add assert to ensure handler is only run on master
2018-08-22 10:11:21 +01:00
Erik Johnston
47b25ba5f3
Remove redundant vars
2018-08-22 10:09:05 +01:00
Erik Johnston
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
...
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Richard van der Hoff
a4cf660a32
Merge pull request #3735 from matrix-org/travis/federation-spelling
...
limt -> limit
2018-08-22 09:34:21 +01:00
Richard van der Hoff
0d568ff403
Merge remote-tracking branch 'origin/release-v0.33.3' into develop
2018-08-22 09:15:44 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two ( #3726 )
...
Splits the state_group_cache in two.
One half contains normal state events; the other contains member events.
The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.
Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720 .
2018-08-22 00:56:37 +02:00
Richard van der Hoff
f7bf181a90
fix another consent encoding fail
2018-08-21 23:14:25 +01:00
Richard van der Hoff
f7baff6f7b
Fix 500 error from /consent form
...
Fixes #3731
2018-08-21 22:47:07 +01:00
Richard van der Hoff
a52f276990
Bugfixes
...
--------
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
([\#3723](https://github.com/matrix-org/synapse/issues/3723 ))
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS
ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6
Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9
698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ
YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV
ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU
Z2lL1C5ggBCkD43g3orDvehbNWCL5g==
=0vEe
-----END PGP SIGNATURE-----
Merge tag 'v0.33.3rc2' into develop
Bugfixes
--------
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
([\#3723](https://github.com/matrix-org/synapse/issues/3723 ))
2018-08-21 20:30:09 +01:00
Hubert Chathi
42a394caa2
allow session_data to be any JSON instead of just a string
2018-08-21 14:51:34 -04:00
Erik Johnston
46c832eaac
Merge pull request #3727 from matrix-org/erikj/dont_error_on_missing_keys
...
Don't log exceptions when failing to fetch server keys
2018-08-21 17:07:20 +01:00
Erik Johnston
cd6937fb26
Fix typo
2018-08-21 16:28:10 +01:00
Hubert Chathi
8550a7e9c2
allow auth_data to be any JSON instead of a string
2018-08-21 10:38:00 -04:00
Erik Johnston
c2c153dd3b
Log more detail when we fail to authenticate request
2018-08-21 11:42:49 +01:00
Erik Johnston
808d8e06aa
Don't log exceptions when failing to fetch server keys
...
Not being able to resolve or connect to remote servers is an expected
error, so we shouldn't log at ERROR with stacktraces.
2018-08-21 11:19:26 +01:00
Erik Johnston
3f6762f0bb
isort
2018-08-21 09:38:38 +01:00
Amber Brown
3b5b64ac99
changelog
2018-08-21 03:48:55 +10:00
Will Hunt
f89f6b7c09
Allow guests to access /rooms/:roomId/event/:eventId
2018-08-20 18:25:54 +01:00
Richard van der Hoff
be6527325a
Fix exceptions when a connection is closed before we read the headers
...
This fixes bugs introduced in #3700 , by making sure that we behave sanely
when an incoming connection is closed before the headers are read.
2018-08-20 18:21:10 +01:00
Richard van der Hoff
55e6bdf287
Robustness fix for logcontext filter
...
Make the logcontext filter not explode if it somehow ends up with a logcontext
of None, since that infinite-loops the whole logging system.
2018-08-20 18:20:07 +01:00
Erik Johnston
b01a755498
Make the in flight requests metrics thread safe
2018-08-20 17:27:52 +01:00
Erik Johnston
1058d14127
Make the in flight background process metrics thread safe
2018-08-20 17:27:24 +01:00
Amber Brown
9a2f960736
version
2018-08-21 00:00:19 +10:00
Amber Brown
324525f40c
Port over enough to get some sytests running on Python 3 ( #3668 )
2018-08-20 23:54:49 +10:00
Erik Johnston
4d664278af
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler
2018-08-20 14:49:43 +01:00
Erik Johnston
8dee601054
Remove redundant room_version checks
2018-08-20 14:48:53 +01:00
Erik Johnston
e21c368b8b
Revert spurious change
2018-08-20 13:54:51 +01:00
Erik Johnston
f2a48d87df
Use get_cache_factor_for function for state_cache
...
This allows the cache factor for `state_cache` to be individually
specified in the enviroment
2018-08-20 13:01:46 +01:00
Erik Johnston
2aa7cc6a46
Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging
...
Fix logging bug in EDU handling over replication
2018-08-20 10:51:45 +01:00
Neil Johnson
e07970165f
rename error code
2018-08-18 14:39:45 +01:00
Neil Johnson
c5171bf171
special case server_notices_mxid
2018-08-18 12:33:07 +01:00
Neil Johnson
ba1fbf7d5b
special case server_notices_mxid
2018-08-18 12:31:08 +01:00
Richard van der Hoff
3cef867cc1
Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commands
...
Logcontexts for replication command handlers
2018-08-17 16:22:07 +01:00
Richard van der Hoff
c144252a8c
Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates
...
Fix logcontexts for running pushers
2018-08-17 16:21:49 +01:00
Amber Brown
c334ca67bb
Integrate presence from hotfixes ( #3694 )
2018-08-18 01:08:45 +10:00
Amber Brown
04f5d2db62
Remove v1/register's broken shared secret functionality ( #3703 )
2018-08-18 00:55:01 +10:00
Erik Johnston
ab822a2d1f
Add some fixmes
2018-08-17 15:31:50 +01:00
Erik Johnston
91cdb6de08
Call UserDirectoryHandler methods directly
...
Turns out that the user directory handling is fairly racey as a bunch
of stuff assumes that the processing happens on master, which it doesn't
when there is a synapse.app.user_dir worker. So lets just call the
function directly until we actually get round to fixing it, since it
doesn't make the situation any worse.
2018-08-17 15:26:13 +01:00
Neil Johnson
d49b77404b
clean up, no functional changes
2018-08-17 15:21:34 +01:00
Richard van der Hoff
63260397c6
Merge pull request #3701 from matrix-org/rav/use_producer_for_responses
...
Use a producer to stream back responses
2018-08-17 14:58:45 +01:00
Richard van der Hoff
3f8709ffe4
Merge pull request #3700 from matrix-org/rav/wait_for_producers
...
Refactor request logging code
2018-08-17 14:57:45 +01:00
Neil Johnson
3ee57bdcbb
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-17 14:34:10 +01:00
Erik Johnston
782689bd40
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-17 14:15:48 +01:00
Erik Johnston
ca87ad1def
Split ProfileHandler into master and worker
2018-08-17 14:15:14 +01:00
Neil Johnson
b5f638f1f4
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-17 14:04:15 +01:00
Neil Johnson
9fd161c6fb
Merge branch 'neilj/limit_exceeded_error' of github.com:matrix-org/synapse into neilj/limit_exceeded_error
2018-08-17 13:58:40 +01:00
Neil Johnson
0195dfbf52
server limits config docs
2018-08-17 13:58:25 +01:00
Neil Johnson
69c49d3fa3
Merge branch 'develop' into neilj/limit_exceeded_error
2018-08-17 12:44:26 +00:00
Erik Johnston
38f708a2bb
Remote profile cache should remain in master worker
2018-08-17 11:37:42 +01:00
Erik Johnston
3b2dcfff78
Fix logging bug in EDU handling over replication
2018-08-17 11:11:06 +01:00
Neil Johnson
521d369e7a
remove errant yield
2018-08-17 10:12:11 +01:00
Richard van der Hoff
0e8d78f6aa
Logcontexts for replication command handlers
...
Run the handlers for replication commands as background processes. This should
improve the visibility in our metrics, and reduce the number of "running db
transaction from sentinel context" warnings.
Ideally it means converting the things that fire off deferreds into the night
into things that actually return a Deferred when they are done. I've made a bit
of a stab at this, but it will probably be leaky.
2018-08-17 00:43:43 +01:00
Richard van der Hoff
66f7dc8c87
Fix logcontexts for running pushers
...
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.
Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
Neil Johnson
bcfeb44afe
call reap on start up and fix under reaping bug
2018-08-16 22:55:32 +01:00
Neil Johnson
372bf073c1
block event creation and room creation on hitting resource limits
2018-08-16 21:25:16 +01:00
Neil Johnson
13ad9930c8
add new error type ResourceLimit
2018-08-16 18:02:02 +01:00
Neil Johnson
51b17ec566
flake8
2018-08-16 17:32:22 +01:00
Neil Johnson
3c1080b6e4
refactor for readability, and reuse caching for setting tags
2018-08-16 17:02:04 +01:00
Neil Johnson
eff3ae3b9a
add room tagging
2018-08-16 15:48:34 +01:00
Erik Johnston
7d5b1a60a3
Fix inbound federation on reader worker
...
Inbound federation requires calculating push, which in turn relies on
having access to account data.
2018-08-16 15:32:20 +01:00
Neil Johnson
bfb6c58624
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-16 15:10:16 +01:00
Neil Johnson
a675f9c556
check for room state before deciding on action
2018-08-16 14:53:35 +01:00
Will Hunt
c151b32b1d
Add GET media/v1/config ( #3184 )
2018-08-16 14:23:38 +01:00
Matthew Hodgson
762a758fea
lazyload aware /messages ( #3589 )
2018-08-16 14:22:47 +01:00
Neil Johnson
25d2b5d55f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-16 11:37:17 +01:00
Neil Johnson
df1e4f259f
WIP impl commiting to get feedback
2018-08-16 11:10:53 +01:00
Neil Johnson
c055c91655
fix case where empty string state check is evaulated as False
2018-08-16 11:10:19 +01:00
Matthew Hodgson
3f543dc021
initial cut at a room summary API ( #3574 )
2018-08-16 09:46:50 +01:00
Neil Johnson
8cfad2e686
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-15 17:20:38 +01:00
Neil Johnson
81d727efa9
Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus
...
Fix Mau off by one errors
2018-08-15 16:19:41 +00:00
Matthew Hodgson
2f78f432c4
speed up /members and add at= and membership params ( #3568 )
2018-08-15 16:35:22 +01:00
Neil Johnson
fc5d937550
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-15 16:31:40 +01:00
Neil Johnson
86a00e05e1
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-15 16:27:08 +01:00
Neil Johnson
eabc5f8271
wip cut at sending resource server notices
2018-08-15 15:04:52 +01:00
Neil Johnson
c24fc9797b
add new event types
2018-08-15 15:04:30 +01:00
Richard van der Hoff
afcd655ab6
Use a producer to stream back responses
...
The problem with dumping all of the json response into the Request object at
once is that doing so starts the timeout for the next request to be received:
so if it takes longer than 60s to stream back the response to the client, the
client never gets it.
The correct solution is to use a Producer; then the timeout is only started
once all of the content is sent over the TCP connection.
2018-08-15 15:04:16 +01:00
Erik Johnston
dc56c47dc0
Merge pull request #3653 from matrix-org/erikj/split_federation
...
Move more federation APIs to workers
2018-08-15 14:59:02 +01:00
Neil Johnson
4601129c44
Merge pull request #3687 from matrix-org/neilj/admin_email
...
support admin_email config and pass through into blocking errors,
2018-08-15 13:52:25 +00:00
Erik Johnston
ef184caf30
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-15 14:25:46 +01:00
Erik Johnston
488ffe6fdb
Use federation handler function rather than duplicate
...
This involves renaming _persist_events to be a public function.
2018-08-15 14:17:18 +01:00
Erik Johnston
773db62a22
Rename slave TransactionStore to SlaveTransactionStore
2018-08-15 14:17:06 +01:00
Richard van der Hoff
eaaa2248ff
Refactor request logging code
...
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.
Advantages of this are:
* we get logs for *all* requests (including OPTIONS and HEADs), rather than
just those that end up hitting handlers we've remembered to decorate
correctly.
* when a request handler wires up a Producer (as the media stuff does
currently, and as other things will do soon), we log at the point that all
of the traffic has been sent to the client.
2018-08-15 13:47:52 +01:00
Neil Johnson
87a824bad1
clean up AuthError
2018-08-15 11:58:03 +01:00
Neil Johnson
c4eb97518f
Merge branch 'neilj/update_limits_error_codes' of github.com:matrix-org/synapse into neilj/admin_email
2018-08-15 11:53:01 +01:00
Neil Johnson
55afba0fc5
update admin email to uri
2018-08-15 11:41:18 +01:00
Neil Johnson
1c5e690a6b
Merge pull request #3690 from matrix-org/neilj/change_prometheus_mau_metric_name
...
combine mau metrics into one group
2018-08-15 09:53:59 +00:00
Erik Johnston
fef2e65d12
Merge pull request #3667 from matrix-org/erikj/fixup_unbind
...
Don't fail requests to unbind 3pids for non supporting ID servers
2018-08-15 10:32:12 +01:00
Neil Johnson
b8429c7c81
update error codes for resource limiting
2018-08-15 10:19:48 +01:00
Neil Johnson
ab035bdeac
replace admin_email with admin_uri for greater flexibility
2018-08-15 10:16:41 +01:00
Neil Johnson
19b433e3f4
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email
2018-08-14 17:44:46 +01:00
Neil Johnson
b586b8b986
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 17:43:22 +01:00
Neil Johnson
70e48cbbb1
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/change_prometheus_mau_metric_name
2018-08-14 17:40:59 +01:00
Neil Johnson
9ecbaf8ba8
adding missing yield
2018-08-14 16:55:28 +01:00
Neil Johnson
e5962f845c
pep8
2018-08-14 16:36:14 +01:00
Neil Johnson
e7d091fb86
combine mau metrics into one group
2018-08-14 16:26:55 +01:00
Neil Johnson
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
...
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
Neil Johnson
2545993ce4
make comments clearer
2018-08-14 15:48:12 +01:00
Neil Johnson
8f9a7eb58d
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-14 15:11:54 +01:00
Neil Johnson
c74c71128d
remove blank line
2018-08-14 15:06:24 +01:00
Neil Johnson
ed4bc3d2fc
fix off by 1s on mau
2018-08-14 15:04:48 +01:00
Neil Johnson
bd92c8eaa7
Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:56:45 +01:00
Neil Johnson
99ebaed8e6
Update register.py
...
remove comments
2018-08-14 14:55:55 +01:00
Neil Johnson
9b5bf3d858
Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:51:38 +01:00
Neil Johnson
e25d87d97b
Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:32:18 +01:00
Neil Johnson
e2c9fe0a6a
backout ability to pass in event type to server notices
2018-08-14 13:32:56 +01:00
Amber Brown
591bf87c6a
Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres
2018-08-14 20:56:23 +10:00
Neil Johnson
9b75c78b4d
support server notice state events for resource limits
2018-08-14 11:20:41 +01:00
Neil Johnson
63417c31e9
fix typo
2018-08-13 22:36:52 +01:00
Neil Johnson
f4b49152e2
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-13 21:09:47 +01:00
Richard van der Hoff
50bcaf1a27
Merge pull request #3677 from andrewshadura/master
...
Use recaptcha_ajax.js directly from Google
2018-08-13 18:52:56 +01:00
Richard van der Hoff
34f51babc5
Revert "Use recaptcha_ajax.js directly from Google"
2018-08-13 18:48:03 +01:00
Neil Johnson
ce7de9ae6b
Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases"
...
This reverts commit 0d43f991a1
.
2018-08-13 18:06:18 +01:00
Neil Johnson
0d43f991a1
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-13 18:00:23 +01:00
Amber Brown
99dd975dae
Run tests under PostgreSQL ( #3423 )
2018-08-13 16:47:46 +10:00
Matthew Hodgson
4f7064f6b5
missing import
2018-08-12 19:14:31 -04:00
Matthew Hodgson
f0cede5556
missing import
2018-08-12 19:14:31 -04:00
Matthew Hodgson
54ac18e832
use parse_string
2018-08-12 19:14:31 -04:00
Matthew Hodgson
66a4ca1d28
404 nicely if you try to interact with a missing current version
2018-08-12 19:14:31 -04:00
Matthew Hodgson
72788cf9c1
support DELETE /version with no args
2018-08-12 19:14:31 -04:00
Matthew Hodgson
edc427a351
flake8
2018-08-12 19:14:31 -04:00
Matthew Hodgson
fe87890b18
implement remaining tests and make them work
2018-08-12 19:14:31 -04:00
Matthew Hodgson
15d513f16f
fix idiocies and so make tests pass
2018-08-12 19:14:31 -04:00
Matthew Hodgson
b5eee511c7
don't needlessly return user_id
2018-08-12 19:14:31 -04:00
Matthew Hodgson
93d174bcc4
improve docstring
2018-08-12 19:14:31 -04:00
Matthew Hodgson
5e42c45c96
switch get_current_version_info back to being get_version_info
2018-08-12 19:14:31 -04:00
Matthew Hodgson
982edca380
fix flakes
2018-08-12 19:14:31 -04:00
Matthew Hodgson
234611f347
fix typos
2018-08-12 19:14:31 -04:00
Matthew Hodgson
14b3da63a3
add a tonne of docstring; make upload_room_keys properly assert version
2018-08-12 19:14:31 -04:00
Matthew Hodgson
9f0791b7bd
add a tonne of docstring; make upload_room_keys properly assert version
2018-08-12 19:14:31 -04:00
Matthew Hodgson
9f500cb39e
more docstring for the e2e_room_keys rest
2018-08-12 19:14:31 -04:00
Matthew Hodgson
8d14598e90
add storage docstring; remove unused set_e2e_room_keys
2018-08-12 19:14:31 -04:00
Matthew Hodgson
ca0b052307
fix factoring out of _should_replace_room_key
2018-08-12 19:14:31 -04:00
Matthew Hodgson
cac0253799
rename room_key_version table correctly, and fix opt args
2018-08-12 19:14:31 -04:00
Matthew Hodgson
0abb205b47
blindly incorporate PR review - needs testing & fixing
2018-08-12 19:14:31 -04:00
Matthew Hodgson
69e51c7ba4
make /room_keys/version work
2018-08-12 19:14:31 -04:00
Matthew Hodgson
8ae64b270f
implement /room_keys/version too (untested)
2018-08-12 19:14:31 -04:00
Matthew Hodgson
cf1e2000f6
document the API
2018-08-12 19:13:09 -04:00
Matthew Hodgson
6b8c07abc2
make it work and fix pep8
2018-08-12 19:13:09 -04:00
Matthew Hodgson
0bc4627a73
interim WIP checkin; doesn't build yet
2018-08-12 18:23:10 -04:00
Matthew Hodgson
53ace904b2
total WIP skeleton for /room_keys API
2018-08-12 18:23:10 -04:00
Neil Johnson
31fa743567
fix sqlite/postgres incompatibility in reap_monthly_active_users
2018-08-11 22:38:34 +01:00
Neil Johnson
6c6aba76e1
implementation of server notices to alert on hitting resource limits
2018-08-10 15:12:59 +01:00
Richard van der Hoff
c31793a784
Merge branch 'rav/fix_linearizer_cancellation' into develop
2018-08-10 14:57:27 +01:00
Richard van der Hoff
c08f9d95b2
log *after* reloading log config
...
... because logging *before* reloading means the log message gets lost in the old MemoryLogger
2018-08-10 14:56:48 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async
is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
Andrej Shadura
c75b71a397
Use recaptcha_ajax.js directly from Google
...
The script recaptcha_ajax.js contains minified non-free code which
we probably cannot redistribute. Since it talks to Google servers
anyway, it is better to just download it from Google directly instead
of shipping it.
This fixes #1932 .
2018-08-10 13:36:14 +02:00
Richard van der Hoff
3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
...
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Richard van der Hoff
638d35ef08
Fix linearizer cancellation on twisted < 18.7
...
Turns out that cancellation of inlineDeferreds didn't really work properly
until Twisted 18.7. This commit refactors Linearizer.queue to avoid
inlineCallbacks.
2018-08-10 10:59:09 +01:00
Neil Johnson
01021c812f
wip at implementing MSC 7075
2018-08-09 22:16:00 +01:00
Jeroen
2e9c73e8ca
more generic conversion of str/bytes to unicode
2018-08-09 21:31:26 +02:00
Jeroen
64899341dc
include private functions from twisted
2018-08-09 21:04:22 +02:00
Neil Johnson
885ea9c602
rename _user_last_seen_monthly_active
2018-08-09 18:02:12 +01:00
Neil Johnson
c1f9dec92a
fix errant parenthesis
2018-08-09 17:43:26 +01:00
Neil Johnson
09cf130898
only block on sync where user is not part of the mau cohort
2018-08-09 17:39:12 +01:00
Erik Johnston
bb99b1f550
Add fast path in state res for zero prev events
2018-08-09 14:58:49 +01:00
Erik Johnston
ce6db0e547
Choose state algorithm based on room version
2018-08-09 14:58:47 +01:00
Erik Johnston
152c0aa58e
Add constants for room versions
2018-08-09 14:55:47 +01:00
Erik Johnston
119451dcd1
Refactor state module
...
We split out the actual state resolution algorithm to prepare for having
multiple versions.
2018-08-09 14:55:47 +01:00
Neil Johnson
69ce057ea6
block sync if auth checks fail
2018-08-09 12:26:27 +01:00
Neil Johnson
3dce9050cf
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block
2018-08-09 11:42:02 +01:00
Neil Johnson
0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs
...
Flag to disable HS without disabling federation
2018-08-09 10:41:43 +00:00
Neil Johnson
a5ef110749
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block
2018-08-09 11:40:37 +01:00
Erik Johnston
a6c813761a
Docstrings
2018-08-09 10:41:08 +01:00
Erik Johnston
b179537f2a
Move clean_room_for_join to master
2018-08-09 10:37:38 +01:00
Erik Johnston
72d1902bbe
Fixup doc comments
2018-08-09 10:23:49 +01:00
Amber Brown
984376745b
Merge branch 'master' into develop
2018-08-09 19:23:47 +10:00
Amber Brown
dafe90a6be
0.33.2
2018-08-09 19:20:41 +10:00
Erik Johnston
484a0ebdfc
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-09 10:16:29 +01:00
Erik Johnston
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
Erik Johnston
bf7598f582
Log when we 3pid/unbind request fails
2018-08-09 10:09:56 +01:00
Erik Johnston
2bdafaf3c1
Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet
...
Add helper base class for generating new replication endpoints
2018-08-09 10:06:23 +01:00
Erik Johnston
62564797f5
Fixup wording and remove dead code
2018-08-09 09:56:10 +01:00
Richard van der Hoff
bb89c84614
Merge pull request #3664 from matrix-org/rav/federation_metrics
...
more metrics for the federation and appservice senders
2018-08-08 23:16:58 +01:00
Jeroen
d5c0ce4cad
updated docstring for ServerContextFactory
2018-08-08 19:25:01 +02:00
Neil Johnson
e92fb00f32
sync auth blocking
2018-08-08 17:54:49 +01:00
Neil Johnson
839a317c96
fix pep8 too many lines
2018-08-08 17:39:04 +01:00
Neil Johnson
5298d79fb5
Merge branch 'develop' into neilj/disable_hs
2018-08-08 16:13:03 +00:00
Richard van der Hoff
8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
...
Support for room versioning
2018-08-08 17:10:53 +01:00
Neil Johnson
d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs
2018-08-08 15:55:47 +00:00
Neil Johnson
2b5baebeba
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-08 13:47:07 +01:00
Neil Johnson
be59910b93
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users
2018-08-08 13:45:21 +01:00
Neil Johnson
990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
...
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
Neil Johnson
312ae74746
typos
2018-08-08 13:33:16 +01:00
Erik Johnston
360ba89c50
Don't fail requests to unbind 3pids for non supporting ID servers
...
Older identity servers may not support the unbind 3pid request, so we
shouldn't fail the requests if we received one of 400/404/501. The
request still fails if we receive e.g. 500 responses, allowing clients
to retry requests on transient identity server errors that otherwise do
support the API.
Fixes #3661
2018-08-08 12:06:18 +01:00
Erik Johnston
bebe325e6c
Rename POST param to METHOD
2018-08-08 10:36:18 +01:00
Erik Johnston
5011417632
Fixup logging and docstrings
2018-08-08 10:29:58 +01:00
Richard van der Hoff
bab94da79c
fix metric name
2018-08-07 22:11:45 +01:00
Richard van der Hoff
53bca4690b
more metrics for the federation and appservice senders
2018-08-07 19:09:48 +01:00
Neil Johnson
ef3589063a
prevent total number of reserved users being too large
2018-08-07 17:59:27 +01:00
Neil Johnson
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
Richard van der Hoff
3523f5432a
Don't expose default_room_version as config opt
2018-08-07 12:51:57 +01:00
Amber Brown
848431be1d
fix for rc1
2018-08-07 21:13:22 +10:00
Amber Brown
3b9662339b
version
2018-08-07 21:10:52 +10:00
Erik Johnston
cd9765805e
Allow ratelimiting on workers
2018-08-07 10:50:28 +01:00
Erik Johnston
495cb100d1
Allow profile changes to happen on workers
2018-08-07 10:50:26 +01:00
Richard van der Hoff
ca9bc1f4fe
Fix occasional glitches in the synapse_event_persisted_position metric
...
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
Neil Johnson
a74b25faaa
WIP building out mau reserved users
2018-08-06 23:25:25 +01:00
Neil Johnson
7daa8a78c5
load mau limit threepids
2018-08-06 22:55:05 +01:00
Neil Johnson
89834d9a29
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:56:56 +01:00
Neil Johnson
e54794f5b6
Fix postgres compatibility bug
2018-08-06 21:55:54 +01:00
Neil Johnson
7bcf126b18
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:39:44 +01:00
Neil Johnson
1911c037cb
update comments to reflect new sig
2018-08-06 18:01:46 +01:00
Neil Johnson
33bd07d062
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 17:52:28 +01:00
Neil Johnson
16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
Richard van der Hoff
19a17068f1
Check m.room.create for sane room_versions
2018-08-06 16:11:24 +01:00
Erik Johnston
96a9a29645
Pull in necessary stores in federation_reader
2018-08-06 15:23:57 +01:00
Erik Johnston
62ace05c45
Move necessary storage functions to worker classes
2018-08-06 15:23:38 +01:00
Erik Johnston
1e2bed9656
Import all functions from TransactionStore
2018-08-06 15:23:38 +01:00
Erik Johnston
a3f5bf79a0
Add EDU/query handling over replication
2018-08-06 15:23:31 +01:00
Erik Johnston
e26dbd82ef
Add replication APIs for persisting federation events
2018-08-06 15:02:28 +01:00
Erik Johnston
051a99c400
Fix isort
2018-08-06 14:29:31 +01:00
Richard van der Hoff
f900d50824
include known room versions in outgoing make_joins
2018-08-06 13:45:37 +01:00
Neil Johnson
42c6823827
disable HS from config
2018-08-04 22:07:04 +01:00
Neil Johnson
e40a510fbf
py3 fix
2018-08-03 23:19:13 +01:00
Neil Johnson
886be75ad1
bug fixes
2018-08-03 22:29:03 +01:00
Will Hunt
16d9701892
Return M_NOT_FOUND when a profile could not be found. ( #3596 )
2018-08-03 19:08:05 +01:00
Neil Johnson
e10830e976
wip commit - tests failing
2018-08-03 17:55:50 +01:00
Richard van der Hoff
3777fa26aa
sanity check response from make_join
2018-08-03 16:08:32 +01:00
Richard van der Hoff
0d63d93ca8
Enforce compatibility when processing make_join requests
...
Reject make_join requests from servers which do not support the room version.
Also include the room version in the response.
2018-08-03 16:08:32 +01:00
Richard van der Hoff
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Richard van der Hoff
15c1ae45e5
Docstrings for BaseFederationServlet
...
... to save me reverse-engineering this stuff again.
2018-08-03 16:08:32 +01:00
Neil Johnson
5593ff6773
fix (lots of) py3 test failures
2018-08-03 14:59:17 +01:00
Neil Johnson
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
Neil Johnson
897c51d274
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-03 13:40:47 +01:00
Erik Johnston
cb298ff623
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet
2018-08-03 09:25:15 +01:00
Neil Johnson
c0affa7b4f
update generate_monthly_active_users, and reap_monthly_active_users
2018-08-02 23:03:01 +01:00
Neil Johnson
5e2f7b8084
typo
2018-08-02 22:47:48 +01:00
Neil Johnson
4ecb4bdac9
wip attempt at caching
2018-08-02 22:41:05 +01:00
Richard van der Hoff
1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling
...
Clean up handling of errors from outbound requests
2018-08-02 17:38:24 +01:00
Richard van der Hoff
bdae8f2e68
Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling
...
Factor out exception handling in federation_client
2018-08-02 17:37:46 +01:00
Neil Johnson
74b1d46ad9
do mau checks based on monthly_active_users table
2018-08-02 16:57:35 +01:00
Neil Johnson
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
Richard van der Hoff
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
Richard van der Hoff
43ecfe0b10
Synapse 0.33.1 (2018-08-02)
...
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx
Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7
SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g
UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd
BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP
RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc=
=U/ir
-----END PGP SIGNATURE-----
Merge tag 'v0.33.1'
Synapse 0.33.1 (2018-08-02)
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
2018-08-02 15:40:44 +01:00
Richard van der Hoff
14a4e7d5a4
Prepare 0.33.1
2018-08-02 15:31:04 +01:00
Richard van der Hoff
0bf5ec0db7
Check room visibility for /event/ requests
...
Make sure that the user has permission to view the requeseted event for
/event/{eventId} and /room/{roomId}/event/{eventId} requests.
Also check that the event is in the given room for
/room/{roomId}/event/{eventId}, for sanity.
2018-08-02 15:03:27 +01:00
Richard van der Hoff
14fa9d4d92
Avoid extra db lookups
...
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
Richard van der Hoff
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
Neil Johnson
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
Neil Johnson
4a6725d9d1
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-02 11:04:18 +01:00
Neil Johnson
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
.
2018-08-02 10:59:58 +01:00
Erik Johnston
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
...
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
Neil Johnson
08281fe6b7
self.db_conn unused
2018-08-01 23:26:24 +01:00
Neil Johnson
c21d82bab3
normalise reaping query
2018-08-01 23:24:38 +01:00
Neil Johnson
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
Neil Johnson
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
Neil Johnson
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
...
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
Richard van der Hoff
b8d7d3996b
Merge pull request #3620 from fuzzmz/return-404-room-not-found
...
return 404 if room not found
2018-08-01 16:34:32 +01:00
Neil Johnson
b7f203a566
count_monthly_users is now async
2018-08-01 16:17:42 +01:00
Neil Johnson
7ff44d9215
improve clarity
2018-08-01 16:17:00 +01:00
Richard van der Hoff
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
Richard van der Hoff
018d75a148
Refactor code for turning HttpResponseException into SynapseError
...
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.
The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
2018-08-01 16:02:46 +01:00
Richard van der Hoff
fa7dc889f1
Be more careful which errors we send back over the C-S API
...
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.
That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.
The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
Richard van der Hoff
c82ccd3027
Factor out exception handling in federation_client
...
Factor out the error handling from make_membership_event, send_join, and
send_leave, so that it can be shared.
2018-08-01 16:01:04 +01:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses ( #3569 )
2018-08-02 00:54:06 +10:00
Neil Johnson
303f1c851f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits
2018-08-01 13:42:50 +01:00
Erik Johnston
a6d7b74915
update docs
2018-08-01 13:39:14 +01:00
Erik Johnston
4b256b9271
_persist_auth_tree no longer returns anything
2018-08-01 13:39:07 +01:00
Neil Johnson
4e5ac901dd
clean up
2018-08-01 12:03:57 +01:00
Neil Johnson
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
Neil Johnson
0aba3d361a
count_monthly_users() async
2018-08-01 11:47:58 +01:00
Neil Johnson
2c54f1c225
remove need to plot limit_usage_by_mau
2018-08-01 11:46:59 +01:00
Richard van der Hoff
f49147d14f
Merge pull request #3634 from matrix-org/rav/wtf_is_a_replication_layer
...
rename replication_layer to federation_client
2018-08-01 10:29:29 +01:00
Richard van der Hoff
cab782c17e
Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator
...
Rewrite cache list decorator
2018-08-01 10:28:56 +01:00
Neil Johnson
6023cdd227
remove errant print
2018-08-01 10:27:17 +01:00
Neil Johnson
7931393495
make count_monthly_users async synapse/handlers/auth.py
2018-08-01 10:21:56 +01:00
Neil Johnson
c507fa15ce
only need to loop if mau limiting is enabled
2018-08-01 10:20:42 +01:00
Travis Ralston
37be52ac34
limt -> limit
2018-07-31 16:29:09 -06:00
Serban Constantin
70af98e361
return NotFoundError if room not found
...
Per the Client-Server API[0] we should return
`M_NOT_FOUND` if the room isn't found instead
of generic SynapseError.
This ensures that /directory/list API returns
404 for room not found instead of 400.
[0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid
Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2018-07-31 21:47:23 +03:00
Travis Ralston
5e2ee64660
Merge pull request #3628 from turt2live/travis/goodby-pdu-failures
...
Remove pdu_failures from transactions
2018-07-31 12:13:09 -06:00
Neil Johnson
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
Richard van der Hoff
bdbdceeafa
rename replication_layer to federation_client
...
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
2018-07-31 15:44:05 +01:00
Erik Johnston
443da003bc
Use new helper base class for membership requests
2018-07-31 14:32:23 +01:00
Erik Johnston
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
Erik Johnston
d81602b75a
Add helper base class for generating new replication endpoints
...
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
Richard van der Hoff
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
...
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
Neil Johnson
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
Richard van der Hoff
0bc9b9e397
reinstate explicit include of EventsWorkerStore
2018-07-31 13:11:04 +01:00
Travis Ralston
e908b86832
Remove pdu_failures from transactions
...
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
Krombel
254e8267e2
Only import secrets when available
...
secrets got introduced in python 3.6 so this class is not available
in 3.5 and before.
This now checks for the current running version and only tries using
secrets if the version is 3.6 or above
Signed-Off-By: Matthias Kesler <krombel@krombel.de>
2018-07-30 23:59:02 +02:00
Neil Johnson
21276ff846
remove errant logging
2018-07-30 22:42:12 +01:00
Neil Johnson
fef7e58ac6
actually close conn
2018-07-30 22:29:44 +01:00
Neil Johnson
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
Neil Johnson
251e6c1210
limit register and sign in on number of monthly users
2018-07-30 15:55:57 +01:00
Erik Johnston
143f1a2532
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store
2018-07-30 09:56:18 +01:00
Jeroen
2903e65aff
fix isort
2018-07-29 19:47:08 +02:00
Richard van der Hoff
a8cbce0ced
fix invalidation
2018-07-27 16:17:17 +01:00
Matthew Hodgson
e9b2d047f6
make /context lazyload & filter aware ( #3567 )
...
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
Richard van der Hoff
f102c05856
Rewrite cache list decorator
...
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
2018-07-27 13:47:04 +01:00
Jeroen
8e3f75b39a
fix accidental removal of hs
2018-07-27 12:17:31 +02:00
Richard van der Hoff
b0b5566f36
Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid
...
if inviter_display_name == ""||None then default to inviter MXID
2018-07-27 10:08:39 +01:00
Richard van der Hoff
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests
2018-07-27 09:17:11 +01:00
Richard van der Hoff
d42455cdc9
Merge pull request #3616 from matrix-org/travis/event_id_send_leave
...
Update the send_leave path to be an event_id
2018-07-26 23:25:17 +01:00
Richard van der Hoff
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
...
Stop populating events.content
2018-07-26 22:54:08 +01:00
Matthew Hodgson
a75231b507
Deduplicate redundant lazy-loaded members ( #3331 )
...
* attempt at deduplicating lazy-loaded members
as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again. For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up). For now we hardcode X to 30.
* add include_redundant_members filter option & make it work
* remove stale todo
* add tests for _get_some_state_from_cache
* incorporate review
2018-07-26 22:51:30 +01:00
Richard van der Hoff
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
...
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
Travis Ralston
7d32f0d745
Update the send_leave path to be an event_id
...
It's still not used, however the parameter is an event ID not a transaction ID.
2018-07-26 14:41:59 -06:00
Richard van der Hoff
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
...
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
Richard van der Hoff
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
Richard van der Hoff
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff
bd4b25f4d0
Remove some redundant joins on event_edges.room_id
...
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.
event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
Richard van der Hoff
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
Richard van der Hoff
21e878ebb6
Make EventStore inherit from EventFederationStore
...
(since it uses methods therein)
Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
Richard van der Hoff
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Matthew Hodgson
1bcd0490c2
Merge pull request #2970 from matrix-org/matthew/filter_members
...
Implement the lazy_load_members room state filter parameter
2018-07-26 00:03:01 +01:00
Matthew Hodgson
bc7944e6d2
switch missing_types to be a bool
2018-07-25 23:36:31 +01:00
Travis Ralston
d8e65ed7e1
Fix a minor documentation typo in on_make_leave
2018-07-25 15:44:41 -06:00
Matthew Hodgson
2565804030
Merge branch 'develop' into matthew/filter_members
2018-07-25 17:27:49 +01:00
Matthew Hodgson
7d9fb88617
incorporate more review.
2018-07-25 16:33:50 +01:00
Erik Johnston
78a691d005
Split out DB writes in federation handler
...
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
Erik Johnston
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
...
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
Richard van der Hoff
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
...
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
Richard van der Hoff
32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
...
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
Richard van der Hoff
4081bd1560
Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache
...
Fix updating of cached remote profiles
2018-07-25 11:54:46 +01:00
Erik Johnston
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
Richard van der Hoff
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
Richard van der Hoff
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
Richard van der Hoff
55acd6856c
Fix updating of cached remote profiles
...
_update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when
it was called, would just return a generator object, without actually running
any of the method body.
2018-07-25 10:34:48 +01:00
Richard van der Hoff
f59be4eb0e
Fix unit tests
...
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
Erik Johnston
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
Richard van der Hoff
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Erik Johnston
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
Matthew Hodgson
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
Michael Telatynski
87951d3891
Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid
2018-07-24 17:17:46 +01:00
Erik Johnston
8b8c4f34a3
Replace usage of get_current_toke with StreamToken.START
...
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
2018-07-24 16:49:17 +01:00
Erik Johnston
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
Erik Johnston
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
Erik Johnston
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
Erik Johnston
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
Erik Johnston
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
Erik Johnston
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
Erik Johnston
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Richard van der Hoff
81946db9cf
Merge pull request #3587 from matrix-org/rav/better_exception_logging
...
Improve logging for exceptions when handling PDUs
2018-07-24 14:12:12 +01:00
Richard van der Hoff
a321f78991
Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups
...
Fixes and optimisations for resolve_state_groups
2018-07-24 14:11:45 +01:00
Richard van der Hoff
93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
...
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
Matthew Hodgson
454f59b7ad
Merge branch 'develop' into matthew/filter_members
2018-07-24 14:03:37 +01:00
Matthew Hodgson
1a01a5b964
clarify comment on p_ids
2018-07-24 14:03:15 +01:00
Erik Johnston
223341205e
Don't require to_delete to have event_ids
2018-07-24 14:02:40 +01:00
Matthew Hodgson
e22700c3dd
consider non-filter_type types as wildcards, thus missing from the state-group-cache
2018-07-24 13:59:07 +01:00
Matthew Hodgson
eb1d911ab7
rather than adding ll_ids, remove them from p_ids
2018-07-24 13:40:49 +01:00
Matthew Hodgson
d19fba3655
Merge branch 'develop' into matthew/filter_members
2018-07-24 12:39:54 +01:00
Matthew Hodgson
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
Richard van der Hoff
30bfed5aa5
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-24 11:46:09 +01:00
Erik Johnston
97acd385a3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta
2018-07-24 11:32:13 +01:00