Commit Graph

311 Commits

Author SHA1 Message Date
Matthew Hodgson
3241c7aac3 untested WIP but might actually work 2017-11-29 18:27:05 +00:00
Richard van der Hoff
77a1227870 Fix broken ref to IntegrityError 2017-11-16 16:03:38 +00:00
Richard van der Hoff
10aaa1bc15 _simple_upsert: retry on IntegrityError
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
2017-11-16 15:30:15 +00:00
Richard van der Hoff
cdc9e50a5d Cleanup in _simple_upsert_txn
Bail out early to reduce indentation
2017-11-16 15:29:10 +00:00
Richard van der Hoff
4dd1bfa8c1 Revert "Revert "move _state_group_cache to statestore""
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.

This reverts commit ab335edb02.
2017-11-14 11:43:58 +00:00
Richard van der Hoff
6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston
ab335edb02 Revert "move _state_group_cache to statestore"
This reverts commit f5cf3638e9.
2017-11-13 10:05:33 +00:00
Richard van der Hoff
f5cf3638e9 move _state_group_cache to statestore
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston
97c544f91f Add _simple_update 2017-08-25 11:11:37 +01:00
Erik Johnston
b5e8d529e6 Define CACHE_SIZE_FACTOR once 2017-07-04 09:56:44 +01:00
Erik Johnston
c62c480dc6 Merge pull request #2259 from matrix-org/erikj/fix_state_woes
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0 Fix bug where state_group tables got corrupted
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.

We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston
21e255a8f1 Split the table in two 2017-06-01 14:50:46 +01:00
Erik Johnston
bfbc907cec Prefill state caches 2017-05-15 15:11:13 +01:00
Erik Johnston
587f07543f Revert "Prefill state caches" 2017-05-04 15:07:27 +01:00
Erik Johnston
a2c89a225c Prefill state caches 2017-05-02 10:40:31 +01:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Luke Barnard
bbeeb97f75 Implement _simple_delete_many_txn, use it to delete devices
(But this doesn't implement the same for deleting access tokens or e2e keys.

Also respond to code review.
2017-03-13 17:53:23 +00:00
Richard van der Hoff
6ad71cc29d Remove spurious SQL logging (#1972)
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
2017-03-08 18:00:44 +00:00
Erik Johnston
b84907bdbb Intern table column names once 2017-02-28 14:38:16 +00:00
Erik Johnston
1a4f8022e6 Strip newlines from SQL queries 2017-02-23 11:15:31 +00:00
Morteza Araby
2849d3f29d admin,storage: added more administrator functionalities
administrators can now:
 - Set displayname of users
 - Update user avatars
 - Search for users by user_id
 - Browse all users in a paginated API
 - Reset user passwords
 - Deactivate users

Helpers for doing paginated queries has also been added to storage

Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
Erik Johnston
2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Erik Johnston
d906206049 Increase state_group_cache_size 2017-01-17 11:31:08 +00:00
Erik Johnston
dd52d4de4c Limit number of entries to prefill from cache
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.

In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
Erik Johnston
524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
pik
e8b1d2a452 Refactor test_filter to use real DataStore
* add tests for filter api errors
2016-10-18 12:17:38 -05:00
Erik Johnston
ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston
949629291c Do it in storage function 2016-08-16 17:05:34 +01:00
Erik Johnston
a2427981b7 Use cached get_user_by_access_token in slaves 2016-08-16 11:24:32 +01:00
Erik Johnston
89e786bd85 Doc get_next() context manager usage 2016-08-15 13:45:26 +01:00
Erik Johnston
d9664344ec Rename table. Add docs. 2016-08-15 11:45:57 +01:00
Erik Johnston
64e7e11853 Implement cache replication stream 2016-08-15 11:16:45 +01:00
Erik Johnston
4d70d1f80e Add some invalidations to a cache_stream 2016-08-15 11:15:17 +01:00
Erik Johnston
99bbd90b0d Always run txn.after_callbacks 2016-08-15 09:45:44 +01:00
Richard van der Hoff
053e83dafb More doc-comments
Fix some more comments on some things
2016-07-20 16:40:28 +01:00
Erik Johnston
f328d95cef Feature: Add deactivate account admin API
Allows server admins to "deactivate" accounts, which:

- Revokes all access tokens
- Removes all threepids
- Removes password

The API is a POST to `/admin/deactivate/<user_id>`
2016-06-30 15:40:58 +01:00
Mark Haines
80f34d7b57 Fix setting the _clock in SQLBaseStore 2016-06-02 15:23:56 +01:00
David Baker
d4503e25ed Make deleting push actions more efficient
There's no index on received_ts, so manually binary search using the stream_ordering index, and only update it once an hour.
2016-05-20 17:56:10 +01:00
Erik Johnston
0c11c1be88 Spelling 2016-05-13 14:42:25 +01:00
Erik Johnston
13d37c3c56 Fixup add_pusher 2016-05-13 11:25:02 +01:00
Mark Haines
7e2f971c08 Remove some unused functions (#711)
* Remove some unused functions

* get_room_events_stream is only used in tests

* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
Mark Haines
a1e0d316ea Move _get_cache_dict into the SQLBaseStore 2016-04-06 13:05:19 +01:00
Erik Johnston
acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Erik Johnston
b591277620 Make stateGroupCache honour CACHE_SIZE_FACTOR 2016-03-22 10:32:50 +00:00
Mark Haines
a1cf9e3bf3 Add a stream for push rule updates 2016-03-01 18:16:37 +00:00
Erik Johnston
2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00