Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
14,374 Commits 5 Branches 723 Tags 124 MiB
f682af052d
Commit Graph

497 Commits

Author SHA1 Message Date
Richard van der Hoff
a215b698c4 Fix "unhashable type: 'list'" exception in federation handling 
get_state_groups returns a map from state_group_id to a list of FrozenEvents,
so was very much the wrong thing to be putting as one of the entries in the
list passed to resolve_events_with_factory (which expects maps from
(event_type, state_key) to event id).

We actually want get_state_groups_ids().values() rather than
get_state_groups().

This fixes the main problem in #3923, but there are other problems with this
bit of code which get discovered once you do so.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff
28223841e0 more comments 2018-09-27 11:31:51 +01:00
Richard van der Hoff
e3c159863d Clarifications in FederationHandler 
* add some comments on things that look a bit bogus
* rename this `state` variable to avoid confusion with the `state` used
  elsewhere in this function. (There was no actual conflict, but it was
  a confusing bit of spaghetti.)
 2018-09-27 11:31:51 +01:00
Richard van der Hoff
92abd3d6d6
Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 
Logging improvements
 2018-09-27 11:27:46 +01:00
Richard van der Hoff
4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959) 
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
 2018-09-27 11:25:34 +01:00
Richard van der Hoff
e70b4ce069 Logging improvements 
Some logging tweaks to help with debugging incoming federation transactions
 2018-09-26 17:36:14 +01:00
Richard van der Hoff
703de4ec13 Comments and interface cleanup for on_receive_pdu 
Add some informative comments about what's going on here.

Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart
from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of
`get_missing` and use `sent_to_us_directly` consistently.
 2018-09-20 13:06:55 +01:00
Richard van der Hoff
642199570c
Improve the logging when handling a federation transaction (#3904) 
Let's try to rationalise the logging that happens when we are processing an
incoming transaction, to make it easier to figure out what is going wrong when
they take ages. In particular:

- make everything start with a [room_id event_id] prefix
- make sure we log a warning when catching exceptions rather than just turning
  them into other, more cryptic, exceptions.
 2018-09-19 17:28:18 +01:00
Richard van der Hoff
550007cb0e Bump timeout on get_missing_events request 2018-09-18 15:02:51 +01:00
Amber Brown
2608ebc04c
Port handlers/ to Python 3 (#3803) 2018-09-07 00:22:23 +10:00
Erik Johnston
5c261107c9 Remove unnecessary resolve_events_with_state_map 
We only ever used the synchronous resolve_events_with_state_map in one
place, which is trivial to replace with the async version.
 2018-08-22 15:41:15 +01:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston
8dee601054 Remove redundant room_version checks 2018-08-20 14:48:53 +01:00
Richard van der Hoff
66f7dc8c87 Fix logcontexts for running pushers 
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.

Instead, give them their own "background process" logcontexts.
 2018-08-17 00:32:39 +01:00
Erik Johnston
ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Erik Johnston
488ffe6fdb Use federation handler function rather than duplicate 
This involves renaming _persist_events to be a public function.
 2018-08-15 14:17:18 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Erik Johnston
ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston
b179537f2a Move clean_room_for_join to master 2018-08-09 10:37:38 +01:00
Erik Johnston
5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00
Erik Johnston
a3f5bf79a0 Add EDU/query handling over replication 2018-08-06 15:23:31 +01:00
Erik Johnston
e26dbd82ef Add replication APIs for persisting federation events 2018-08-06 15:02:28 +01:00
Richard van der Hoff
f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Richard van der Hoff
704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff
14fa9d4d92 Avoid extra db lookups 
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
 2018-08-02 13:55:51 +01:00
Richard van der Hoff
0a65450d04 Validation for events/rooms in fed requests 
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
 2018-08-02 13:48:40 +01:00
Erik Johnston
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store 
Split out DB writes in federation handler
 2018-08-02 10:41:42 +01:00
Erik Johnston
a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Erik Johnston
4b256b9271 _persist_auth_tree no longer returns anything 2018-08-01 13:39:07 +01:00
Richard van der Hoff
bdbdceeafa rename replication_layer to federation_client 
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
 2018-07-31 15:44:05 +01:00
Erik Johnston
143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Travis Ralston
d8e65ed7e1
Fix a minor documentation typo in on_make_leave 2018-07-25 15:44:41 -06:00
Erik Johnston
78a691d005 Split out DB writes in federation handler 
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
 2018-07-25 16:22:56 +01:00
Richard van der Hoff
c1f80effbe Handle delta_ids being None in _update_context_for_auth_events 
it's easier to create the new state group as a delta from the existing one.

(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364)
 2018-07-23 22:06:50 +01:00
Erik Johnston
027bc01a1b Add support for updating state 2018-07-23 13:17:25 +01:00
Erik Johnston
e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Amber Brown
3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Richard van der Hoff
dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks 
Remove event re-signing hacks
 2018-07-18 12:46:27 +01:00
Richard van der Hoff
ea69d35651 Move filter_events_for_server out of FederationHandler 
for easier unit testing.
 2018-07-16 13:06:24 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
a4ab491371
Merge branch 'develop' into rav/drop_re_signing_hacks 2018-07-04 07:13:38 +01:00
Amber Brown
99800de63d try and clean up 2018-06-27 11:40:27 +01:00
Amber Brown
f03a5d1a17 pep8 2018-06-27 11:38:14 +01:00
Amber Brown
a7ecf34b70 cleanups 2018-06-27 11:36:03 +01:00
Amber Brown
77078d6c8e handle federation not telling us about prev_events 2018-06-27 11:27:32 +01:00
Erik Johnston
df48f7ef37 Actually fix it 2018-06-25 20:03:41 +01:00
Erik Johnston
a0e8a53c6d Comment 2018-06-25 19:57:38 +01:00
Erik Johnston
7bdc5c8fa3 Fix bug with assuming wrong type 2018-06-25 19:56:02 +01:00
Erik Johnston
ea7a9c0483 Add fast path to _filter_events_for_server 
Most rooms have a trivial history visibility like "shared" or
"world_readable", especially large rooms, so lets not bother getting the
full membership of those rooms in that case.
 2018-06-25 19:49:13 +01:00
Erik Johnston
244484bf3c Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" 
This reverts commit 1d009013b3.
 2018-06-25 13:42:55 +01:00
Richard van der Hoff
1d009013b3 Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" 
This reverts commit ce0d911156, reversing
changes made to b4a5d767a9.
 2018-06-22 16:35:10 +01:00
Erik Johnston
ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility 
Support hiding events from deleted users
 2018-06-22 15:06:44 +01:00
Erik Johnston
b4a5d767a9
Merge pull request #3428 from matrix-org/erikj/persisted_pdu 
Simplify get_persisted_pdu
 2018-06-22 14:47:43 +01:00
Amber Brown
a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Richard van der Hoff
b6faef2ad7 Filter out erased messages 
Redact any messges sent by erased users.
 2018-06-12 09:53:18 +01:00
Richard van der Hoff
9fc5b74b24 simplify get_persisted_pdu 
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
 2018-06-12 09:51:31 +01:00
Richard van der Hoff
0834b49c6a Fix event filtering in get_missing_events handler 2018-06-08 11:34:46 +01:00
Richard van der Hoff
8503dd0047 Remove event re-signing hacks 
These "temporary fixes" have been here three and a half years, and I can't find
any events in the matrix.org database where the calculated signature differs
from what's in the db. It's time for them to go away.
 2018-06-07 16:08:29 +01:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Adrian Tschira
17a70cf6e9 Misc. py3 fixes 
Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-05-24 20:20:33 +02:00
Erik Johnston
f72d5a44d5
Merge pull request #3261 from matrix-org/erikj/pagination_fixes 
Fix federation backfill bugs
 2018-05-24 14:52:03 +01:00
Erik Johnston
5aaa3189d5 s/values/itervalues/ 2018-05-23 10:13:05 +01:00
Erik Johnston
0a4bca4134 Use iter* methods for _filter_events_for_server 2018-05-23 10:04:23 +01:00
Erik Johnston
e85b5a0ff7 Use iter* methods 2018-05-22 19:02:48 +01:00
Erik Johnston
586b66b197 Fix that states is a dict of dicts 2018-05-22 19:02:36 +01:00
Erik Johnston
cb2a2ad791 get_domains_from_state returns list of tuples 2018-05-22 16:23:39 +01:00
Richard van der Hoff
26305788fe Make sure we reject attempts to invite the notices user 2018-05-18 11:18:39 +01:00
Richard van der Hoff
bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib 
move httplib import to six
 2018-04-30 01:26:42 +01:00
Adrian Tschira
94f4d7f49e move httplib import to six 2018-04-28 13:43:34 +02:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn 
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
 2018-04-27 12:55:51 +01:00
Richard van der Hoff
6493b22b42 reraise exceptions more carefully 
We need to be careful (under python 2, at least) that when we reraise an
exception after doing some error handling, we actually reraise the original
exception rather than anything that might have been raised (and handled) during
the error handling.
 2018-04-27 11:40:06 +01:00
Richard van der Hoff
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events 
Reject events which have lots of prev_events
 2018-04-23 17:51:38 +01:00
Richard van der Hoff
9b9c38373c Remove spurious param 2018-04-23 12:00:06 +01:00
Richard van der Hoff
3de7d9fe99 accept stupid events over backfill 2018-04-20 11:41:03 +01:00
Richard van der Hoff
b1dfbc3c40 Refactor store.have_events 
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
 2018-04-20 10:25:56 +01:00
Richard van der Hoff
1f4b498b73 Add some comments 2018-04-18 00:15:36 +01:00
Richard van der Hoff
e585228860 Check events on backfill too 2018-04-18 00:06:42 +01:00
Richard van der Hoff
9b7794262f Reject events which have too many auth_events or prev_events 
... this should protect us from being dossed by people making silly events
(deliberately or otherwise)
 2018-04-18 00:06:42 +01:00
Erik Johnston
cb9f8e527c s/replication_client/federation_client/ 2018-03-13 13:26:52 +00:00
Erik Johnston
265b993b8a Split replication layer into two 2018-03-13 10:55:47 +00:00
Erik Johnston
784f036306 Move RoomMemberHandler out of Handlers 2018-03-01 14:36:50 +00:00
Erik Johnston
d874d4f2d7 Delete from push_actions_staging in federation too 2018-02-20 11:37:52 +00:00
Erik Johnston
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation 
Split event creation into a separate handler
 2018-02-06 17:01:17 +00:00
Erik Johnston
3e1e69ccaf Update copyright 2018-02-06 16:40:38 +00:00
Erik Johnston
770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784) 
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
 2018-02-06 14:31:24 +00:00
Erik Johnston
3fa362502c Update places where we create events 2018-02-05 16:01:48 +00:00
Richard van der Hoff
0cbda53819 Rename resolve_state_groups -> resolve_state_groups_for_events 
(to make way for a method that actually just does the state group resolution)
 2018-02-01 12:40:00 +00:00
Matthew Hodgson
ab9f844aaf
Add federation_domain_whitelist option (#2820) 
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
 2018-01-22 19:11:18 +01:00
Richard van der Hoff
e148438e97 s/items/iteritems/ 2017-11-08 09:21:41 +00:00
Richard van der Hoff
780dbb378f Update deltas when doing auth resolution 
Fixes a bug where the persisted state groups were different to those actually
being used after auth resolution.
 2017-11-07 16:43:00 +00:00
Richard van der Hoff
1ca4288135 factor out _update_context_for_auth_events 
This is duplicated, so let's factor it out before fixing it
 2017-11-07 16:43:00 +00:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:' 
what could possibly go wrong
 2017-10-23 15:52:32 +01:00
Erik Johnston
84e27a592d Merge pull request #2490 from matrix-org/erikj/drop_left_room_events 
Ignore incoming events for rooms that we have left
 2017-10-10 11:58:32 +01:00
Richard van der Hoff
c8f568ddf9 Fix up deferred handling in federation.py 
* Avoid preserve_context_over_deferred, which is broken

* set consumeErrors=True on defer.gatherResults, to avoid spurious "unhandled
  failure" erros
 2017-10-06 22:14:24 +01:00
David Baker
f878e6f8af Spam checking: add the invitee to user_may_invite 2017-10-05 14:02:28 +01:00
David Baker
1e375468de pass room id too 2017-10-03 17:13:14 +01:00
Erik Johnston
e4ab96021e Update comments 2017-10-03 14:10:41 +01:00
David Baker
2a7ed700d5 Fix param name & lint 2017-10-03 14:04:10 +01:00