Commit Graph

52 Commits

Author SHA1 Message Date
David Robertson
6ba732fefe
Type tests.utils ()
* Cast to postgres types when handling postgres db

* Remove unused method

* Easy annotations

* Annotate create_room

* Use `ParamSpec` to annotate looping_call

* Annotate `default_config`

* Track `now` as a float

`time_ms` returns an int like the proper Synapse `Clock`

* Introduce a `Timer` dataclass

* Introduce a Looper type

* Suppress checking of a mock

* tests.utils is typed

* Changelog

* Whoops, import ParamSpec from typing_extensions

* ditch the psycopg2 casts
2022-07-05 15:13:47 +01:00
Shay
cde8af9a49
Add config flags to allow for cache auto-tuning () 2022-05-13 12:32:39 -07:00
Erik Johnston
0dbbe33a65
Track cache invalidations ()
Currently we only track evictions due to size or time constraints.
2022-02-15 14:31:04 +00:00
David Robertson
f510fba4ba
Describe prune_unread_entries in docstrings ()
Should have been caught in .
2022-02-02 15:11:23 +00:00
Shay
fcfe67578f
Update to the current version of Black and run it on Synapse codebase ()
* update black version

* run updated version of black on code

* newsfragment

* enumerate python versions
2021-12-23 20:22:15 -08:00
Sean Quah
7ff22d6da4
Fix LruCache corruption bug with a size_callback that can return 0 ()
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.

Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
2021-11-30 16:28:02 +00:00
Sean Quah
5a0b652d36
Eliminate a few Anys in LruCache type hints () 2021-11-30 15:39:07 +00:00
David Robertson
f8d0f72b27
More types for synapse.util, part 1 ()
The following modules now pass `disallow_untyped_defs`:

* synapse.util.caches.cached_call 
* synapse.util.caches.lrucache
* synapse.util.caches.response_cache 
* synapse.util.caches.stream_change_cache
* synapse.util.caches.ttlcache pass
* synapse.util.daemonize
* synapse.util.patch_inline_callbacks pass `no-untyped-defs`
* synapse.util.versionstring

Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch 
2021-10-06 11:20:49 +01:00
David Robertson
724aef9a87
Opt out of cache expiry for get_users_who_share_room_with_user ()
* Allow LruCaches to opt out of time-based expiry
* Don't expire `get_users_who_share_room` & friends
2021-09-22 14:21:58 +01:00
David Robertson
a2d7195e01
Track why we're evicting from caches ()
So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
2021-09-22 10:59:52 +01:00
reivilibre
524b8ead77
Add types to synapse.util. () 2021-09-10 17:03:18 +01:00
Jonathan de Jong
95e47b2e78
[pyupgrade] synapse/ ()
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of 
2021-07-19 15:28:05 +01:00
Jonathan de Jong
bdfde6dca1
Use inline type hints in http/federation/, storage/ and util/ () 2021-07-15 12:46:54 -04:00
Erik Johnston
7a5873277e
Add support for evicting cache entries based on last access time. () 2021-07-05 16:32:12 +01:00
Richard van der Hoff
224f2f949b
Combine LruCache.invalidate and invalidate_many ()
* Make `invalidate` and `invalidate_many` do the same thing

... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.

* Kill off `invalidate_many`

* changelog
2021-05-27 10:33:56 +01:00
Richard van der Hoff
c0df6bae06
Remove keylen from LruCache. ()
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
2021-05-24 14:02:01 +01:00
Erik Johnston
ef889c98a6
Optionally track memory usage of each LruCache ()
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
2021-05-05 16:54:36 +01:00
Erik Johnston
391bfe9a7b
Reduce memory footprint of caches () 2021-04-28 11:59:28 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes ()
Part of 

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Richard van der Hoff
96e7d3c4a0
Fix 'LruCache' object has no attribute '_on_resize' ()
We need to make sure we are readu for the `set_cache_factor` callback.
2020-10-19 21:13:50 +01:00
Richard van der Hoff
97647b33c2
Replace DeferredCache with LruCache where possible ()
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
2020-10-19 12:20:29 +01:00
Richard van der Hoff
6d7b22041d review comments 2020-10-16 16:25:15 +01:00
Richard van der Hoff
995cc615a0
Apply suggestions from code review
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-16 16:14:42 +01:00
Richard van der Hoff
0ec0bc3886 type annotations for LruCache 2020-10-16 15:56:39 +01:00
Richard van der Hoff
3ee17585cd
Make LruCache register its own metrics ()
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
2020-10-16 15:51:57 +01:00
Richard van der Hoff
7eff59ec91 Add some more type annotations to Cache 2020-10-14 23:38:14 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object () 2020-09-04 06:54:56 -04:00
Erik Johnston
eefc6b3a0d
Don't apply cache factor to event cache. ()
This is already correctly done when we instansiate the cache, but wasn't
when it got reloaded (which always happens at least once on startup).
2020-05-27 12:04:37 +01:00
Amber Brown
7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables () 2020-05-11 18:45:23 +01:00
Amber Brown
32e7c9e7f2
Run Black. () 2019-06-20 19:32:02 +10:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Erik Johnston
9a0d783c11 Add comments 2018-03-19 11:35:53 +00:00
Erik Johnston
7c7706f42b Fix bug where state cache used lots of memory
The state cache bases its size on the sum of the size of entries. The
size of the entry is calculated once on insertion, so it is important
that the size of entries does not change.

The DictionaryCache modified the entries size, which caused the state
cache to incorrectly think it was smaller than it actually was.
2018-03-15 15:46:54 +00:00
Richard van der Hoff
bc496df192 report metrics on number of cache evictions 2018-02-05 15:34:01 +00:00
Erik Johnston
c430111d0e Update LruCache size estimate on clear 2017-01-18 14:55:23 +00:00
Erik Johnston
1ccd5676e3 Remove needless call to evict() 2017-01-17 11:42:26 +00:00
Erik Johnston
f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Erik Johnston
2fae34bd2c Optionally measure size of cache by sum of length of values 2017-01-13 17:46:17 +00:00
Erik Johnston
45fd2c8942 Ensure invalidation list does not grow unboundedly 2016-08-19 16:09:16 +01:00
Erik Johnston
4161ff2fc4 Add concept of cache contexts 2016-08-19 14:17:07 +01:00
Erik Johnston
c4a8cbd15a Make LruCache use a dedicated _Node class 2016-03-22 16:06:21 +00:00
Erik Johnston
c046630c33 Remove spurious self.size 2016-01-29 11:17:54 +00:00
Erik Johnston
766526e114 Make TreeCache keep track of its own size. 2016-01-29 10:11:21 +00:00
Erik Johnston
50e18938a9 Reset size on clear 2016-01-29 10:00:45 +00:00
David Baker
7cd418d38e Don't add the member functiopn if we're not using treecache 2016-01-22 13:40:37 +00:00
David Baker
cd80019eec docs 2016-01-22 12:21:13 +00:00
David Baker
d552861346 Revert all the bits changing keys of eeverything that used LRUCaches to tuples 2016-01-22 12:18:14 +00:00
David Baker
10f76dc5da Make LRU cache not default to treecache & add options to use it 2016-01-22 12:10:33 +00:00
David Baker
f1f8122120 Change LRUCache to be tree-based so we can delete subtrees. 2016-01-21 19:16:25 +00:00