Commit Graph

2551 Commits

Author SHA1 Message Date
Erik Johnston
810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Erik Johnston
cb23aa4c42
Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation
Clean up room alias creation
2018-10-24 09:40:27 +01:00
Neil Johnson
a67d8ace9b remove errant exception and style 2018-10-23 17:44:39 +01:00
Travis Ralston
43c3f0b02f
Merge pull request #3969 from turt2live/travis/fix-federated-group-requests
Handle HttpResponseException more safely for federated groups
2018-10-23 10:41:04 -06:00
Travis Ralston
3e704822be
Comments help 2018-10-23 10:25:31 -06:00
Richard van der Hoff
b3f6dddad2
Give some more things logcontexts (#4077) 2018-10-23 14:29:17 +01:00
Richard van der Hoff
5c445114d3
Correctly account for cpu usage by background threads (#4074)
Wrap calls to deferToThread() in a thing which uses a child logcontext to
attribute CPU usage to the right request.

While we're in the area, remove the logcontext_tracer stuff, which is never
used, and afaik doesn't work.

Fixes #4064
2018-10-23 13:12:32 +01:00
Richard van der Hoff
e7a16c6210 Remove redundant run_as_background_process() from pusherpool
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
2018-10-22 16:12:11 +01:00
Erik Johnston
084046456e Add config option to control alias creation 2018-10-19 10:22:45 +01:00
Erik Johnston
74e7617083 Clean up room alias creation 2018-10-19 10:11:56 +01:00
Travis Ralston
49a044aa5f Merge branch 'develop' into travis/login-terms 2018-10-18 09:57:58 -06:00
Will Hunt
d6a7797dd1 Fix roomlist since tokens on Python 3 (#4046)
Thanks @Half-Shot !!!
2018-10-17 23:04:55 +11:00
Neil Johnson
c6584f4b5f clean up config error logic and imports 2018-10-17 11:36:41 +01:00
Richard van der Hoff
fc0f13dd03 Fix incorrect truncation in get_missing_events
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
2018-10-16 21:10:04 +01:00
Erik Johnston
15133477ee Fix up use of resolve_events_with_factory 2018-10-16 16:28:42 +01:00
Travis Ralston
a8ed93a4b5 pep8 2018-10-15 16:10:29 -06:00
Travis Ralston
442734ff9e Ensure the terms params are actually provided 2018-10-15 14:56:13 -06:00
Travis Ralston
762a0982aa Python is hard 2018-10-15 14:46:09 -06:00
Travis Ralston
f293d124b6 Merge branch 'develop' into travis/login-terms 2018-10-15 14:44:32 -06:00
David Baker
03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups
E2E backups
2018-10-15 10:19:43 +01:00
Neil Johnson
1ccafb0c5e no need to join room if creator 2018-10-13 21:14:21 +01:00
Travis Ralston
dd99db846d Update login terms structure for the proposed language support 2018-10-12 18:03:27 -06:00
Travis Ralston
7ede650956 Merge branch 'develop' into travis/login-terms 2018-10-12 16:24:07 -06:00
Travis Ralston
164f8e4843 isort 2018-10-12 15:11:59 -06:00
Travis Ralston
7bb651de6a More sane handling of group errors and pep8 2018-10-12 14:53:30 -06:00
Travis Ralston
e3586f7c06 Merge branch 'develop' into travis/fix-federated-group-requests 2018-10-12 14:49:58 -06:00
Neil Johnson
a2bfb778c8 improve auto room join logic, comments and tests 2018-10-12 18:17:36 +01:00
David Baker
8c0ff0287a Linting soothes the savage PEP8 monster 2018-10-12 13:47:43 +01:00
David Baker
83e72bb2f0 PR feedback pt. 1 2018-10-12 11:26:18 +01:00
Erik Johnston
7e561b5c1a Add description to counter metric 2018-10-10 11:41:15 +01:00
Erik Johnston
3cbe8331e6 Track number of non-empty sync responses instead 2018-10-10 11:23:17 +01:00
Erik Johnston
395276b405 Append _total to metric and fix up spelling 2018-10-10 09:24:39 +01:00
Erik Johnston
bdc27d6716 Add metric to count lazy member sync requests 2018-10-09 14:17:52 +01:00
David Baker
dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker
f4a4dbcad1 Apparently this blank line is Very Important 2018-10-09 09:47:04 +01:00
David Baker
0c905ee015 be python3 compatible 2018-10-09 09:39:13 +01:00
Erik Johnston
6982320572 Remove unnecessary extra function call layer 2018-10-08 14:06:19 +01:00
Erik Johnston
8a1817f0d2 Use errback pattern and catch async failures 2018-10-08 13:29:47 +01:00
Neil Johnson
ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Neil Johnson
2dadc092b8 move logic into register, fix room alias localpart bug, tests 2018-10-04 17:00:27 +01:00
Travis Ralston
537d0b7b36 Use a flag rather than a new route for the public policy
This also means that the template now has optional parameters, which will need to be documented somehow.
2018-10-03 17:50:11 -06:00
Travis Ralston
149c4f1765 Supply params for terms auth stage
As per https://github.com/matrix-org/matrix-doc/pull/1692
2018-10-03 15:57:42 -06:00
Travis Ralston
fd99787162 Incorporate Dave's work for GDPR login flows
As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117
2018-10-03 15:57:42 -06:00
Erik Johnston
495a9d06bb Fix exception handling in fetching remote profiles 2018-10-03 11:34:30 +01:00
Richard van der Hoff
8c41b0ca66
Merge pull request #3989 from matrix-org/rav/better_stacktraces
Avoid reraise, to improve stacktraces
2018-10-02 10:47:38 +01:00
Erik Johnston
bc29946809
Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state
Fix lazy loaded sync with rejected state events
2018-10-02 11:38:35 +02:00
Richard van der Hoff
8174c6725b Avoid reraise, to improve stacktraces 2018-10-01 18:50:34 +01:00
Richard van der Hoff
b5b93f45d5
Merge pull request #3968 from matrix-org/rav/fix_federation_errors
Fix exceptions when handling incoming transactions
2018-10-01 15:54:24 +01:00
Erik Johnston
82f922b4af Fix lazy loaded sync with rejected state events
In particular, we assume that the name and canonical alias events in
the state have not been rejected. In practice this may not be the case
(though we should probably think about fixing that) so lets ensure that
we gracefully handle that case, rather than 404'ing the sync request
like we do now.
2018-10-01 14:19:40 +01:00
Matthew Hodgson
23b6a0537f emit room aliases event 2018-09-29 02:19:37 +01:00
Matthew Hodgson
5b68f29f48 fix thinkos 2018-09-29 02:14:40 +01:00
Matthew Hodgson
8f646f2d04 fix UTs 2018-09-28 15:37:28 +01:00
Richard van der Hoff
3deaad2fb4
Merge pull request #3964 from matrix-org/rav/remove_localhost_checks
remove spurious federation checks on localhost
2018-09-28 13:35:47 +01:00
Richard van der Hoff
965154d60a Fix complete fail to do the right thing 2018-09-28 12:45:54 +01:00
Matthew Hodgson
07340cdaca untested stab at autocreating autojoin rooms 2018-09-28 01:42:53 +01:00
Richard van der Hoff
f094f715cf Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors 2018-09-27 15:18:21 +01:00
Richard van der Hoff
36c62a67c4
Merge pull request #3794 from matrix-org/erikj/faster_typing
Improve performance of getting typing updates for replication
2018-09-27 15:14:26 +01:00
Richard van der Hoff
e1e3e77bfd
Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups
Clarifications in FederationHandler
2018-09-27 15:06:59 +01:00
Amber Brown
a512e637ac
Merge pull request #3970 from schnuffle/develop-py3
Replaced all occurences of e.message with str(e)
2018-09-27 23:39:45 +10:00
Schnuffle
dc5db01ff2 Replaced all occurences of e.message with str(e)
Signed-off-by: Schnuffle  <schnuffle@github.com>
2018-09-27 13:38:50 +02:00
Richard van der Hoff
333bee27f5 Include event when resolving state for missing prevs
If we have a forward extremity for a room as `E`, and you receive `A`, `B`,
s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need
to do state res between `X` and `E`.

When that happens, we need to make sure we include `X` in the state that goes
into the state res alg.

Fixes #3934.
2018-09-27 11:37:39 +01:00
Richard van der Hoff
bd61c82bdf Include state from remote servers in pdu handling
If we've fetched state events from remote servers in order to resolve the state
for a new event, we need to actually pass those events into
resolve_events_with_factory (so that it can do the state res) and then persist
the ones we need - otherwise other bits of the codebase get confused about why
we have state groups pointing to non-existent events.
2018-09-27 11:37:39 +01:00
Richard van der Hoff
a215b698c4 Fix "unhashable type: 'list'" exception in federation handling
get_state_groups returns a map from state_group_id to a list of FrozenEvents,
so was very much the wrong thing to be putting as one of the entries in the
list passed to resolve_events_with_factory (which expects maps from
(event_type, state_key) to event id).

We actually want get_state_groups_ids().values() rather than
get_state_groups().

This fixes the main problem in #3923, but there are other problems with this
bit of code which get discovered once you do so.
2018-09-27 11:37:39 +01:00
Richard van der Hoff
28223841e0 more comments 2018-09-27 11:31:51 +01:00
Richard van der Hoff
e3c159863d Clarifications in FederationHandler
* add some comments on things that look a bit bogus
* rename this `state` variable to avoid confusion with the `state` used
  elsewhere in this function. (There was no actual conflict, but it was
  a confusing bit of spaghetti.)
2018-09-27 11:31:51 +01:00
Richard van der Hoff
92abd3d6d6
Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2
Logging improvements
2018-09-27 11:27:46 +01:00
Richard van der Hoff
4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959)
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
2018-09-27 11:25:34 +01:00
Travis Ralston
82fa31799c Remove debugging statement 2018-09-26 14:01:02 -06:00
Travis Ralston
b4c3bc1734 Handle HttpResponseException more safely for federated groups 2018-09-26 13:48:04 -06:00
Richard van der Hoff
e70b4ce069 Logging improvements
Some logging tweaks to help with debugging incoming federation transactions
2018-09-26 17:36:14 +01:00
Amber Brown
c2185f14d7
Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu
Comments and interface cleanup for on_receive_pdu
2018-09-26 21:41:26 +10:00
Matthew Hodgson
787d22ed6c
Only lazy load self-members on initial sync
Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync.  Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
2018-09-25 00:49:26 +01:00
Matthew Hodgson
a2ddaa90f2
Always LL ourselves if we're in a room to simplify clients (#3916)
Should fix https://github.com/vector-im/riot-web/issues/7209
2018-09-20 21:21:54 +01:00
Richard van der Hoff
703de4ec13 Comments and interface cleanup for on_receive_pdu
Add some informative comments about what's going on here.

Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart
from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of
`get_missing` and use `sent_to_us_directly` consistently.
2018-09-20 13:06:55 +01:00
Richard van der Hoff
642199570c
Improve the logging when handling a federation transaction (#3904)
Let's try to rationalise the logging that happens when we are processing an
incoming transaction, to make it easier to figure out what is going wrong when
they take ages. In particular:

- make everything start with a [room_id event_id] prefix
- make sure we log a warning when catching exceptions rather than just turning
  them into other, more cryptic, exceptions.
2018-09-19 17:28:18 +01:00
Erik Johnston
ce846bb620 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing 2018-09-19 15:08:36 +01:00
Amber Brown
f773ecbd61
Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout
Bump timeout on get_missing_events request
2018-09-19 17:57:48 +10:00
Richard van der Hoff
a219ce8726
Use directory server for room joins (#3899)
When we do a join, always try the server we used for the alias lookup first.

Fixes #2418
2018-09-18 18:27:37 +01:00
Richard van der Hoff
550007cb0e Bump timeout on get_missing_events request 2018-09-18 15:02:51 +01:00
Richard van der Hoff
286d6930b7
Merge pull request #3879 from matrix-org/matthew/fix-autojoin
don't ratelimit autojoins
2018-09-18 13:07:01 +01:00
Richard van der Hoff
f75b9961c6 Reinstate missing null check 2018-09-17 16:52:02 +01:00
Richard van der Hoff
85a43f4167 Return a 404 when deleting unknown room alias
As per https://github.com/matrix-org/matrix-doc/issues/1675

Fixes https://github.com/matrix-org/synapse/issues/2782
2018-09-17 13:19:00 +01:00
Matthew Hodgson
d42d79e3c3 don't ratelimit autojoins 2018-09-15 22:27:41 +01:00
Matthew Hodgson
024be6cf18
don't filter membership events based on history visibility (#3874)
don't filter membership events based on history visibility
as we will already have filtered the messages in the timeline, and state events
are always visible.

and because @erikjohnston said so.
2018-09-14 18:12:52 +01:00
Matthew Hodgson
2ac1abbc7e
show heroes if a room has a 'deleted' name/canonical_alias (#3851) 2018-09-12 17:11:05 +01:00
Matthew Hodgson
0403cf0783 argh pep8 2018-09-12 16:54:28 +01:00
Matthew Hodgson
0e200e366d correctly log gappy sync metrics 2018-09-12 16:47:20 +01:00
Matthew Hodgson
11bfc2af1c fix logline 2018-09-12 16:45:42 +01:00
Matthew Hodgson
b041115415
Speed up lazy loading (#3827)
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
2018-09-12 00:50:39 +01:00
Erik Johnston
f60c9e2a01 Don't send empty tags list down sync 2018-09-06 17:01:41 +01:00
Erik Johnston
7baf66ef5d Send existing room tags down sync on join
When a user joined a room any existing tags were not sent down the sync
stream. Ordinarily this isn't a problem because the user needs to be in
the room to have set tags in it, however synapse will sometimes add tags
for a user to a room, e.g. for server notices, which need to come down
sync.
2018-09-06 16:46:51 +01:00
Hubert Chathi
3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Amber Brown
ee7c8bd2b5
Merge pull request #3795 from matrix-org/erikj/faster_sync_state
User iter* during sync state calculations
2018-09-07 00:24:43 +10:00
Amber Brown
2608ebc04c
Port handlers/ to Python 3 (#3803) 2018-09-07 00:22:23 +10:00
Erik Johnston
7419764351 User iter* during sync state calculations 2018-09-05 16:19:50 +01:00
Erik Johnston
5f02017aea Improve performance of getting typing updates for replication
Fetching the list of all new typing notifications involved iterating
over all rooms and comparing their serial. Lets move to using a stream
change cache, like we do for other streams.
2018-09-05 10:20:40 +01:00
Erik Johnston
567363e497
Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func
Remove unnecessary resolve_events_with_state_map
2018-09-03 16:19:41 +01:00
Neil Johnson
ea068d6f3c fix bug where preserved threepid user comes to sign up and server is mau blocked 2018-08-31 10:49:14 +01:00
Matthew Hodgson
b1580f50fe
don't return non-LL-member state in incremental sync state blocks (#3760)
don't return non-LL-member state in incremental sync state blocks
2018-08-28 23:25:58 +01:00
Hubert Chathi
16a31c6fce update to newer Synapse APIs 2018-08-24 22:51:25 -04:00