Commit Graph

39 Commits

Author SHA1 Message Date
Richard van der Hoff
72bc285669
Refactor the Appservice scheduler code ()
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).

Add some docstrings too.
2019-08-20 17:42:45 +01:00
Richard van der Hoff
baa3f4a80d
Avoid deep recursion in appservice recovery ()
Hopefully, this will fix a stack overflow when recovering an appservice.

The recursion here leads to a huge chain of deferred callbacks, which then
overflows the stack when the chain completes. `inlineCallbacks` makes a better
job of this if we use iteration instead.

Clean up the code a bit too, while we're there.
2019-08-20 17:39:38 +01:00
Richard van der Hoff
5019945828 Refactor the Appservice scheduler code
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).

Add some docstrings too.
2019-08-20 11:50:23 +01:00
Amber Brown
4806651744
Replace returnValue with return () 2019-07-23 23:00:55 +10:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ () 2019-07-04 00:07:04 +10:00
Amber Brown
32e7c9e7f2
Run Black. () 2019-06-20 19:32:02 +10:00
Richard van der Hoff
52e87fbfbe
Run the AS senders as background processes ()
This should fix some "Starting db connection from sentinel context" warnings,
and will mean we get metrics for these processes.
2018-12-04 10:53:49 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff
9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston
9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Matrix
f743471380 Change name of metric 2016-08-17 15:05:50 +01:00
Erik Johnston
b9e888858c Move Measure block inside loop 2016-08-17 14:52:26 +01:00
Erik Johnston
7321f45457 Clean up _ServiceQueuer 2016-08-17 12:03:04 +01:00
Mark Haines
c626fc576a Move the AS handler out of the Handlers object.
Access it directly from the homeserver itself. It already wasn't
inheriting from BaseHandler storing it on the Handlers object was
already somewhat dubious.
2016-05-31 13:53:48 +01:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Kegan Dougal
4dec901c76 Cap the time to retry txns to appservices to 8.5 minutes
There's been numerous issues with people playing around with their
application service and then not receiving events from their HS for
ages due to backoff timers reaching crazy heights (albeit capped at
< 1 day).

Reduce the max time between pokes to be 8.5 minutes (2^9 secs) which
is quick enough for people to wait it out (avg wait time being 4.25 min)
but long enough to actually give the AS breathing room if it needs it.
2015-10-21 10:10:55 +01:00
Kegan Dougal
32206dde3f Fixes from PR comments 2015-03-26 10:11:52 +00:00
Kegan Dougal
d04fa1f712 Implement ServiceQueuer with tests. 2015-03-16 14:03:16 +00:00
Kegan Dougal
6279285b2a Replace EventGrouper for ServiceQueuer to move to push-based txns. Fix tests and add stub tests for ServiceQueuer. 2015-03-16 13:15:40 +00:00
Kegan Dougal
c9c444f562 Wrap polling/retry blocks in try/excepts to avoid sending to other ASes breaking permanently should an error occur. 2015-03-16 10:38:02 +00:00
Kegan Dougal
835e01fc70 Minor PR comment tweaks. 2015-03-16 10:16:59 +00:00
Kegan Dougal
db1fbc6c6f Fix remaining scheduler bugs. Add more informative logging. 2015-03-10 10:04:20 +00:00
Kegan Dougal
04c9751f24 Bug fixes whilst putting it all together 2015-03-09 17:45:41 +00:00
Kegan Dougal
21fd84dcb8 Use seconds; start gluing in the AS scheduler into the AS handler. 2015-03-09 17:01:19 +00:00
Kegan Dougal
64345b7559 Upper bound the backoff. 2015-03-06 16:41:19 +00:00
Kegan Dougal
2602ddc379 Apply clarity and docstrings 2015-03-06 16:16:14 +00:00
Kegan Dougal
0354659f9d Finish synapse.appservice.scheduler implementation.
With tests to assert behaviour. Not hooked up yet. Stub datastore methods
not implemented yet.
2015-03-06 16:09:05 +00:00
Kegan Dougal
7d3491c741 Add some loggers 2015-03-06 15:17:50 +00:00
Kegan Dougal
f260cb72cd Flesh out more stub functions. 2015-03-06 15:12:24 +00:00
Kegan Dougal
141ec04d19 Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions. 2015-03-06 14:53:35 +00:00
Kegan Dougal
0fbfe1b08a Add more tests; fix bugs. 2015-03-06 14:36:52 +00:00
Kegan Dougal
192e228a98 Start adding some tests 2015-03-06 11:50:27 +00:00
Kegan Dougal
d516d68b29 Rejig structure given the appservice_handler already filters the correct ASes to use. 2015-03-06 10:25:50 +00:00
Kegan Dougal
0c838f9f5e Minor tweaks 2015-03-05 17:45:52 +00:00
Kegan Dougal
773cb3b688 Add stub architecture for txn reliability. 2015-03-05 17:35:07 +00:00
Kegan Dougal
e319071191 Add stub scheduler module for txn reliability 2015-03-05 16:30:33 +00:00