Commit Graph

35 Commits

Author SHA1 Message Date
David Robertson
797ee7812d
Relax ignore-missing-imports for modules that have stubs now and update mypy ()
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
Jonathan de Jong
95e47b2e78
[pyupgrade] synapse/ ()
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of 
2021-07-19 15:28:05 +01:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) () 2021-07-15 11:02:43 +01:00
Richard van der Hoff
b2557cbf42
opentracing: use a consistent name for background processes ()
... otherwise we tend to get a namespace clash between the bg process and the
functions that it calls.
2021-06-07 17:57:49 +01:00
Richard van der Hoff
ed53bf314f
Set opentracing priority before setting other tags ()
... because tags on spans which aren't being sampled get thrown away.
2021-05-28 16:14:08 +01:00
Andrew Morgan
4b2217ace2 Merge branch 'master' into develop 2021-04-21 14:55:06 +01:00
Richard van der Hoff
5d281c10dd
Stop BackgroundProcessLoggingContext making new prometheus timeseries ()
This undoes part of b076bc276e.
2021-04-21 10:03:31 +01:00
Andrew Morgan
6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Patrick Cloke
b076bc276e
Always use the name as the log ID. ()
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke
48d44ab142
Record more information into structured logs. ()
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ()
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
1619802228
Various clean-ups to the logging context code () 2020-12-14 14:19:47 -05:00
David Teller
f14428b25c
Allow spam-checker modules to be provide async methods. ()
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
2020-12-11 14:05:15 -05:00
Erik Johnston
2b7c180879
Start fewer opentracing spans ()
 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.

This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.

I don't really know how much this will help.
2020-10-26 09:30:19 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. () 2020-10-23 12:38:40 -04:00
Erik Johnston
1fcdbeb3ab
Start an opentracing span for background processes. ()
This should reduce the number of `There was no active span` errors we
see.

Fixes .
2020-10-19 12:26:26 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object () 2020-09-04 06:54:56 -04:00
Patrick Cloke
d89692ea84
Convert runWithConnection to async. () 2020-08-19 07:09:24 -04:00
Patrick Cloke
c36228c403
Convert run_as_background_process inner function to async. () 2020-08-06 08:20:42 -04:00
Richard van der Hoff
8ca39bd2c3
Improve stacktraces from exceptions in background processes ()
use `Failure()` to fish out the real exception.
2020-07-09 13:01:33 +01:00
Erik Johnston
f5353eff21
Make inflight background metrics more efficient. ()
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-05-29 13:25:32 +01:00
Richard van der Hoff
8c75667ad7
Add prometheus metrics for the number of active pushers () 2020-03-19 10:00:24 +00:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 ()
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Richard van der Hoff
a96318127d Update comments and docstring 2019-09-25 18:17:39 +01:00
Erik Johnston
367158a609 Add wrap_as_background_process decorator.
This does the same thing as `run_as_background_process` but means we
don't need to create superfluous functions.
2019-09-24 15:53:17 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ () 2019-07-04 00:07:04 +10:00
Amber Brown
32e7c9e7f2
Run Black. () 2019-06-20 19:32:02 +10:00
Erik Johnston
7c570bff74 Fix exception in background metrics collection
We attempted to iterate through a list on a separate thread without
doing the necessary copying.
2018-10-03 11:28:01 +01:00
Erik Johnston
94ae1dea3c Add missing logger 2018-09-20 17:05:34 +01:00
Erik Johnston
9ea408441f Handle exceptions thrown by background tasks
Fixes 
2018-09-20 16:15:21 +01:00
Erik Johnston
3f6762f0bb isort 2018-08-21 09:38:38 +01:00
Erik Johnston
1058d14127 Make the in flight background process metrics thread safe 2018-08-20 17:27:24 +01:00
Richard van der Hoff
03751a6420 Fix some looping_call calls which were broken in
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in .

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Richard van der Hoff
6e3fc657b4 Resource tracking for background processes
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.

This will help address , but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.

We *could* do this with Measure blocks, but:
 - I think having them pulled out as a completely separate metric class will
   make it easier to distinguish top-level processes from those which are
   nested.

 - I want to be able to report on in-flight background processes, and I don't
   think we want to do this for *all* Measure blocks.
2018-07-18 10:50:33 +01:00