Commit Graph

127 Commits

Author SHA1 Message Date
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. (#9640)
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.

Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
2021-03-18 15:52:26 +00:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily (#9561)
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.

Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
2021-03-15 14:42:40 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Andrew Morgan
168ba00d01
Fix RoomDirectoryFederationTests and make them actually run (#8998)
The `RoomDirectoryFederationTests` tests were not being run unless explicitly called as an `__init__.py` file was not present in `tests/federation/transport/`. Thus the folder was not a python module, and `trial` did not look inside for any test cases to run. This was found while working on #6739.

This PR adds a `__init__.py` and also fixes the test in a couple ways:

- Switch to subclassing `unittest.FederatingHomeserverTestCase` instead, which sets up federation endpoints for us.
- Supply a `federation_auth_origin` to `make_request` in order to more act like the request is coming from another server, instead of just an unauthenicated client requesting a federation endpoint.

I found that the second point makes no difference to the test passing, but felt like the right thing to do if we're testing over federation.
2020-12-30 19:27:32 +00:00
Richard van der Hoff
394516ad1b Remove spurious "SynapseRequest" result from `make_request"
This was never used, so let's get rid of it.
2020-12-15 22:35:40 +00:00
Richard van der Hoff
acfe3b3065 Remove redundant HomeserverTestCase.render 2020-11-16 18:24:08 +00:00
reivilibre
36efbcaf51
Catch-up after Federation Outage (bonus): Catch-up on Synapse Startup (#8322)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Fix _set_destination_retry_timings

This came about because the code assumed that retry_interval
could not be NULL — which has been challenged by catch-up.
2020-09-18 14:59:13 +01:00
reivilibre
576bc37d31
Catch-up after Federation Outage (split, 4): catch-up loop (#8272) 2020-09-15 09:07:19 +01:00
Patrick Cloke
a9dbe98ef9 Synapse 1.20.0rc3 (2020-09-11)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl9bbA8ACgkQM/xY9qcR
 MEjjJg//ZGcvPLr8y3F5JXptcjWA3AFja3DgztBA3uNFoDrwVhR9m3+F4rL/Fbat
 gnaccWsKIXOjZw5WOThQEUIfYMKmHEsaDl7xtlUPC4ylmSV+31ypu1KiRSxND2kT
 TmOupvqw+4/E4RajpYX6WT3e2oQUpIcBAKsUyZL3ZXECR/yNYrA6w2w7i4wrv3cU
 QDzBFrCcr5aJql7VUc88BlaZUG6xom2/kkPtjmO6imPnPGBHLN22uPU7zQ4n7Vgi
 Nrg45v5WHVCx57WoXqAEap1zdKgoRCna1x0NkqYh0OAXq1l6aVwlf/Pdynt91f2x
 yyaMYjMskjyatHlzONMV4kz0w03dUrGJXiAx2ldEDd32SKBUxLJ/CYtNEdoXZ4Mm
 o0OJfDbaSJwlqK7FhdZJE3kCSkUvlmVasDErXfQRDVHy5sx9Ufr4P0kkaZyXDys1
 UAWFcCJwy9dgFR2679PoXk1s/gHf10wuk6FOs8ESbcJJOvCljxWrOIqKNzSUL3HQ
 Hj5V1zHYXB6HtgiI7tbKxSn/d2uajHI9b8LVOkyV9RKsb1DGGRgNQQ5Kz4Zud7dC
 vuQ2jVw7JIo41W3QIXvcL1KdQeje2nwYwuwNREZaEDZJsGLjxaJ6dNyBVgkmYYFP
 gxhplSOSsFjQRGEsBrc2utWxDnLVGPHEAt7iHVSqjwVUyARvxjo=
 =e2T1
 -----END PGP SIGNATURE-----

Merge tag 'v1.20.0rc3' into develop

Synapse 1.20.0rc3 (2020-09-11)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
2020-09-11 08:30:36 -04:00
Patrick Cloke
b86764662b
Fix the exception that is raised when invalid JSON is encountered. (#8291) 2020-09-10 14:55:25 -04:00
Patrick Cloke
cef00211c8
Allow for make_awaitable's return value to be re-used. (#8261) 2020-09-08 07:26:55 -04:00
reivilibre
765437df54
Add tests for last_successful_stream_ordering (#8258) 2020-09-07 10:11:38 +01:00
reivilibre
58f61f10f7
Catch-up after Federation Outage (split, 1) (#8230)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2020-09-04 12:22:23 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Patrick Cloke
e00816ad98
Do not yield on awaitables in tests. (#8193) 2020-08-27 17:24:46 -04:00
Patrick Cloke
dd8f28bd3f
Fix unawaited coroutine error in tests. (#8072) 2020-08-13 07:11:39 -04:00
Patrick Cloke
c978f6c451
Convert federation client to async/await. (#7975) 2020-07-30 08:01:33 -04:00
lugino-emeritus
3857de2194
Option to allow server admins to join complex rooms (#7902)
Fixes #7901.

Signed-off-by: Niklas Tittjung <nik_t.01@web.de>
2020-07-28 13:41:44 +01:00
Patrick Cloke
b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Andrew Morgan
fcd6961441
Add option to enable encryption by default for new rooms (#7639)
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on #7637
2020-06-10 17:44:34 +01:00
Andrew Morgan
c389bfb6ea
Fix encryption algorithm typos in tests/comments (#7637)
@uhoreg has confirmed these were both typos. They are only in comments and tests though, rather than anything critical.

Introduced in:

* https://github.com/matrix-org/synapse/pull/7157
* https://github.com/matrix-org/synapse/pull/5726
2020-06-04 20:03:40 +01:00
Erik Johnston
1531b214fc
Add ability to wait for replication streams (#7542)
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
2020-05-22 14:21:54 +01:00
Richard van der Hoff
daa1ac89a0
Fix device list update stream ids going backward (#7158)
Occasionally we could get a federation device list update transaction which
looked like:

```
[
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D2', 'prev_id': [], 'stream_id': 12, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D1', 'prev_id': [12], 'stream_id': 11, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D3', 'prev_id': [11], 'stream_id': 13, 'deleted': True}}
]
```

Having `stream_ids` which are lower than `prev_ids` looks odd. It might work
(I'm not actually sure), but in any case it doesn't seem like a reasonable
thing to expect other implementations to support.
2020-04-03 10:40:22 +01:00
Richard van der Hoff
7966a1cde9
Rewrite prune_old_outbound_device_pokes for efficiency (#7159)
make sure we clear out all but one update for the user
2020-03-30 19:06:52 +01:00
Richard van der Hoff
665630fcaa Add tests for outbound device pokes 2020-03-27 12:01:37 +00:00
Richard van der Hoff
28d9d6e8a9 Remove spurious "name" parameter to default_config
this is never set to anything other than "test", and is a source of unnecessary
boilerplate.
2020-03-24 18:33:49 +00:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Richard van der Hoff
799001f2c0
Add a make_event_from_dict method (#6858)
... and use it in places where it's trivial to do so.

This will make it easier to pass room versions into the FrozenEvent
constructors.
2020-02-07 15:30:04 +00:00
Neil Johnson
cb0aeb147e
privacy by default for room dir (#6355)
Ensure that the the default settings for the room directory are that the it is hidden from public view by default.
2019-12-04 09:46:16 +00:00
Amber Brown
0f87b912ab
Implementation of MSC2314 (#6176) 2019-11-28 08:54:07 +11:00
Erik Johnston
7777d353bf Remove test debugs 2019-08-20 11:46:59 +01:00
Amber Brown
865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Amber Brown
b36c82576e
Run Black on the tests again (#5170) 2019-05-10 00:12:11 -05:00
Richard van der Hoff
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890)
Rate-limit outgoing read-receipts as per #4730.
2019-03-20 16:02:25 +00:00
black
8b3d9b6b19 Run black. 2018-08-10 23:54:09 +10:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
3cf3e08a97 Implementation of server_acls
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
2018-07-04 19:06:20 +01:00
Erik Johnston
9959d9ece8 Remove redundated BaseHomeServer 2016-01-26 13:52:29 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Erik Johnston
bd3de8f39a Update tests 2015-11-17 18:38:48 +00:00
Erik Johnston
9236136f3a Make work in both Maria and SQLite. Fix tests 2015-04-01 14:12:33 +01:00
Mark Haines
896253e085 Factor out some of the common homeserver setup code into a
setup_test_homeserver function in utils.
2015-02-11 11:37:30 +00:00
Mark Haines
041ac476a5 Supply auth_chain along with current state in '/state/', fetch auth events from a remote server if we are missing some of them 2014-12-18 18:47:13 +00:00
Erik Johnston
3fecacd86b Fix replication tests 2014-12-11 17:11:06 +00:00
Matthew Hodgson
8ada2d2018 fix UTs by telling all the mock stores about the new methods for tracking retries 2014-12-09 23:53:07 +00:00
Mark Haines
c5eabe3143 replace user_id with sender 2014-11-19 16:38:40 +00:00
Mark Haines
cb4b6c844a Merge PDUs and Events into one object 2014-11-14 21:25:02 +00:00
Mark Haines
8d8a133c89 SYN-103: Remove "origin" and "destination" keys from edus 2014-11-13 15:49:03 +00:00
Erik Johnston
b15e8d5bbc event <-> pdu mappings are now trivial and will soon be scrapped 2014-11-04 16:20:02 +00:00
Erik Johnston
24305ba5bf Fix up federation tests 2014-11-04 16:15:30 +00:00
Mark Haines
dc3c2823ac Merge branch 'develop' into event_signing
Conflicts:
	synapse/federation/replication.py
2014-10-17 17:33:58 +01:00
Mark Haines
82c5820767 keep 'origin_server_ts' as 'ts' in the database to avoid needlessly updating schema 2014-10-17 17:31:48 +01:00
Mark Haines
f5cf7ac25b SPEC-7: Rename 'ts' to 'origin_server_ts' 2014-10-17 17:12:25 +01:00
Mark Haines
bb04447c44 Include hashes of previous pdus when referencing them 2014-10-16 23:25:12 +01:00
Mark Haines
66104da10c Sign outgoing PDUs. 2014-10-16 00:09:48 +01:00
Mark Haines
1c445f88f6 persist hashes and origin signatures for PDUs 2014-10-15 17:09:04 +01:00
Mark Haines
9aed791fc3 SYN-103: Ignore the 'origin' key in received EDUs. Instead take the origin from the transaction itself 2014-10-14 16:44:27 +01:00
Mark Haines
6684855767 Verify signatures for server2server requests 2014-10-13 14:37:46 +01:00
Mark Haines
10ef8e6e4b SYN-75 sign at the request level rather than the transaction level 2014-10-13 11:49:55 +01:00
Mark Haines
984e207b59 Merge branch develop into server2server_signing
Conflicts:
	synapse/app/homeserver.py
2014-10-13 10:58:50 +01:00
Mark Haines
574377636e Add a keyword argument to get_json to avoid retrying on DNS failures. Rather than passing MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES as a fake query string parameter 2014-10-02 14:26:13 +01:00
Mark Haines
b95a178584 SYN-75 Verify signatures on server to server transactions 2014-09-30 15:15:10 +01:00
Mark Haines
52ca867670 Sign federation transactions 2014-09-24 17:25:41 +01:00
Erik Johnston
0897a09f49 Fix unit tests after adding extra argument on put_json 2014-09-15 16:24:03 +01:00
Paul "LeoNerd" Evans
cd62ee3f29 Have all unit tests import from our own subclass of trial's unittest TestCase; set up logging in ONE PLACE ONLY 2014-09-12 18:46:13 +01:00
Matthew Hodgson
8a7c1d6a00 fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org hasn't been incorporated in time for launch. 2014-09-03 17:31:57 +01:00
Matthew Hodgson
8fe912d95c change the world: make the default matrix API URL prefix /_matrix rather than /matrix to make it easier for existing websites to mount a HS in their namespace without collisions.
perl -pi -e 's#/matrix#/_matrix#g' ./cmdclient/console.py ./docs/client-server/howto.rst ./docs/client-server/specification.rst ./docs/client-server/swagger_matrix/directory ./docs/client-server/swagger_matrix/events ./docs/client-server/swagger_matrix/login ./docs/client-server/swagger_matrix/presence ./docs/client-server/swagger_matrix/profile ./docs/client-server/swagger_matrix/registration ./docs/client-server/swagger_matrix/rooms ./docs/server-server/specification.rst ./graph/graph.py ./jsfiddles/create_room_send_msg/demo.js ./jsfiddles/event_stream/demo.js ./jsfiddles/example_app/demo.js ./jsfiddles/register_login/demo.js ./jsfiddles/room_memberships/demo.js ./synapse/api/urls.py ./tests/federation/test_federation.py ./tests/handlers/test_presence.py ./tests/handlers/test_typing.py ./tests/rest/test_events.py ./tests/rest/test_presence.py ./tests/rest/test_profile.py ./tests/rest/test_rooms.py ./webclient/components/fileUpload/file-upload-service.js ./webclient/components/matrix/matrix-service.js
2014-08-31 14:51:37 +01:00
Mark Haines
64e2a5d58e Move pdu and event persistence into a single persist_event function 2014-08-26 18:01:36 +01:00
Paul "LeoNerd" Evans
bb793019a5 Rename MockHttpServer to MockHttpResource as it stands for one server resource rather than an entire server 2014-08-19 11:16:23 +01:00
Kegan Dougal
9a1638ed21 Removed http_server from HomeServer. Updated unit tests to use either resource_for_federation or resource_for_client depending on what is being tested. 2014-08-14 10:18:54 +01:00
Paul "LeoNerd" Evans
fc778e2bce Move MockClock into tests.utils so we can reüse it 2014-08-13 19:19:15 +01:00
Paul "LeoNerd" Evans
827de7cee9 Define the concept of a 'federation Query'; creating API for making and handling Queries on the Federation's increasingly-inaccurately-named ReplicationLayer 2014-08-13 17:23:49 +01:00
Paul "LeoNerd" Evans
820ed34abe Namespace all the Federation HTTP URLs to /matrix/federation/v1/... 2014-08-13 15:07:55 +01:00
Matthew Hodgson
3e35a9f96e add in copyrights to everything, not just the synapse subdir, and add a copyrighter.pl whilst we're at it 2014-08-13 03:32:18 +01:00
matrix.org
4f475c7697 Reference Matrix Home Server 2014-08-12 15:10:52 +01:00