Erik Johnston
99c7fbfef7
Fix to work with SQLite
2015-10-14 09:52:40 +01:00
Erik Johnston
858634e1d0
Remove unused room_id arg
2015-10-14 09:31:20 +01:00
Erik Johnston
7ecd11accb
Add paranoia limit
2015-10-13 15:50:56 +01:00
Erik Johnston
3e2a1297b5
Remove constraints in preperation of using filters
2015-10-13 15:22:14 +01:00
Mark Haines
cacf0688c6
Add a get_invites_for_user method to the storage to find out the rooms a user is invited to
2015-10-13 14:08:38 +01:00
Erik Johnston
88971fd034
Merge branch 'erikj/store_engine' into erikj/search
2015-10-13 14:03:30 +01:00
Erik Johnston
17c80c8a3d
rename schema_prepare to prepare_database
2015-10-13 13:56:22 +01:00
Erik Johnston
cfd39d6b55
Add SQLite support
2015-10-13 13:47:50 +01:00
Erik Johnston
f9340ea0d5
Merge branch 'erikj/store_engine' into erikj/search
2015-10-13 13:29:02 +01:00
Erik Johnston
ec398af41c
Expose error more nicely
2015-10-13 11:43:43 +01:00
Erik Johnston
40b6a5aad1
Split out the schema preparation and update logic into its own module
2015-10-13 11:38:48 +01:00
Erik Johnston
1a40afa756
Add sqlite schema
2015-10-13 10:36:25 +01:00
Erik Johnston
ca53ad7425
Filter events to only thsoe that the user is allowed to see
2015-10-12 15:52:55 +01:00
Erik Johnston
927004e349
Remove unused room_id parameter
2015-10-12 15:06:14 +01:00
Erik Johnston
61561b9df7
Keep FTS indexes up to date. Only search through rooms currently joined
2015-10-12 10:49:53 +01:00
Erik Johnston
c85c912562
Add basic full text search impl.
2015-10-09 15:48:31 +01:00
Mark Haines
301141515a
Merge pull request #288 from matrix-org/markjh/unused_definitions
...
Remove some of the unused definitions from synapse
2015-09-28 14:22:44 +01:00
Mark Haines
cf1100887b
Fix order of ON constraints in _get_rooms_for_user_where_membership_is_txn
2015-09-24 17:35:10 +01:00
Mark Haines
1cd65a8d1e
synapse/storage/state.py: _make_group_id was unused
2015-09-23 10:37:58 +01:00
Mark Haines
973ebb66ba
Remove unused functions from synapse/storage/signatures.py
2015-09-23 10:36:33 +01:00
Mark Haines
e51aa4be96
synapse/storage/roommember.py:_get_members_query was unused
2015-09-23 10:35:10 +01:00
Mark Haines
92d8d724c5
Remove unused functions from synapse/storage/events.py
2015-09-23 10:33:06 +01:00
Mark Haines
c292dba70c
Remove unused functions from synapse/storage/event_federation.py
2015-09-23 10:31:25 +01:00
Mark Haines
396834f1c0
synapse/storage/_base.py:_simple_max_id was unused
2015-09-23 10:30:38 +01:00
Mark Haines
1d9036aff2
synapse/storage/_base.py:_simple_delete was unused
2015-09-23 10:30:25 +01:00
Mark Haines
1ee3d26432
synapse/storage/_base.py:_simple_selectupdate_one was unused
2015-09-23 10:30:03 +01:00
Mark Haines
3559a835a2
synapse/storage/event_federation.py:_get_auth_events is unused
2015-09-22 18:39:46 +01:00
Mark Haines
7dd4f79c49
synapse/storage/_base.py:_execute_and_decode was unused
2015-09-22 18:37:07 +01:00
Mark Haines
527d95dea0
synapse/storage/_base.py:Table was unused
2015-09-22 18:14:15 +01:00
Daniel Wagner-Hall
6d59ffe1ce
Add some docstrings
2015-09-22 13:47:40 +01:00
Daniel Wagner-Hall
eb011cd99b
Add docstring
2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Mark Haines
ee2d722f0f
Merge pull request #276 from matrix-org/markjh/history_for_rooms_that_have_been_left
...
SPEC-216: Allow users to view the history of rooms that they have left.
2015-09-21 14:38:13 +01:00
Erik Johnston
2048388cfd
Merge pull request #281 from matrix-org/erikj/update_extremeties
...
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:57:25 +01:00
Erik Johnston
8148c48f11
"Comments"
2015-09-15 16:54:48 +01:00
Erik Johnston
3b05b67c89
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:34:42 +01:00
Erik Johnston
dffc9c4ae0
Drop unused index
2015-09-14 14:41:37 +01:00
Mark Haines
09cb5c7d33
Allow users that have left a room to get the messages that happend in the room before they left
2015-09-09 17:31:09 +01:00
Mark Haines
3c166a24c5
Remove undocumented and unimplemented 'feedback' parameter from the Client-Server API
2015-09-09 16:05:09 +01:00
Mark Haines
89ae0166de
Allow room initialSync for users that have left the room, returning a snapshot of how the room was when they left it
2015-09-09 13:25:22 +01:00
Mark Haines
417485eefa
Include the event_id and stream_ordering of membership events when looking up which rooms a user is in
2015-09-08 18:14:54 +01:00
Erik Johnston
b62c1395d6
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-09-01 13:11:55 +01:00
Erik Johnston
cd800ad99a
Lower size of 'stateGroupCache' now that we have data from matrix.org to support doing so
2015-09-01 10:09:03 +01:00
Mark Haines
a7122692d9
Merge branch 'release-v0.10.0' into develop
...
Conflicts:
synapse/handlers/auth.py
synapse/python_dependencies.py
synapse/rest/client/v1/login.py
2015-08-28 11:15:27 +01:00
Daniel Wagner-Hall
d3c0e48859
Merge erikj/user_dedup to develop
2015-08-26 13:42:45 +01:00
Daniel Wagner-Hall
a2355fae7e
Merge pull request #251 from matrix-org/removeadmin
...
Stop looking up "admin", which we never read
2015-08-25 17:23:05 +01:00
Daniel Wagner-Hall
ee3fa1a99c
Merge pull request #248 from matrix-org/deviceid
...
Remove completely unused concepts from codebase
2015-08-25 17:19:06 +01:00
Daniel Wagner-Hall
a9d8bd95e7
Stop looking up "admin", which we never read
2015-08-25 16:29:39 +01:00
Daniel Wagner-Hall
a0b181bd17
Remove completely unused concepts from codebase
...
Removes device_id and ClientInfo
device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.
This doesn't remove the columns from the database, because that's
fiddly.
2015-08-25 16:23:06 +01:00
Mark Haines
1c847af28a
Merge pull request #243 from matrix-org/markjh/remove_syutil
...
Replace syutil dependency with smaller, single-purpose libraries
2015-08-25 10:52:16 +01:00
Mark Haines
037481a033
Remove autoincrement since we incrementing the ID in the storage layer
2015-08-24 17:48:57 +01:00
Mark Haines
01fc3943f1
Fix indent
2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
Erik Johnston
457970c724
Don't insert events into 'event_*_extremeties' tables if they're outliers
2015-08-23 13:44:23 +01:00
Erik Johnston
1bd1a43073
Actually check if event_id isn't returned by _get_state_groups
2015-08-21 14:30:34 +01:00
Erik Johnston
745b72660a
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-08-21 11:39:38 +01:00
Erik Johnston
42f12ad92f
When logging in fetch user by user_id case insensitively, *unless* there are multiple case insensitive matches, in which case require the exact user_id
2015-08-21 11:38:44 +01:00
Erik Johnston
aa3c9c7bd0
Don't allow people to register user ids which only differ by case to an existing one
2015-08-21 10:57:47 +01:00
Erik Johnston
1f7642efa9
Fix bug where we didn't correctly serialize the redacted_because key over federation
2015-08-21 09:36:07 +01:00
Daniel Wagner-Hall
e8cf77fa49
Merge branch 'develop' into refresh
...
Conflicts:
synapse/rest/client/v1/login.py
2015-08-20 16:25:40 +01:00
Daniel Wagner-Hall
cecbd636e9
/tokenrefresh POST endpoint
...
This allows refresh tokens to be exchanged for (access_token,
refresh_token).
It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
Daniel Wagner-Hall
13a6517d89
s/by_token/by_access_token/g
...
We're about to have two kinds of token, access and refresh
2015-08-20 16:01:29 +01:00
Mark Haines
a0b8e5f2fe
Merge pull request #211 from matrix-org/email_in_use
...
Changes for unique emails
2015-08-20 10:04:04 +01:00
Erik Johnston
e624cdec64
Merge pull request #228 from matrix-org/erikj/_get_state_for_groups
...
Ensure we never return a None event from _get_state_for_groups
2015-08-18 16:30:17 +01:00
Erik Johnston
38a965b816
Merge pull request #227 from matrix-org/erikj/receipts_take2
...
Re-enable receipts API.
2015-08-18 16:30:04 +01:00
Erik Johnston
a82938416d
Remove newline because vertical whitespace makes mjark sad
2015-08-18 16:28:13 +01:00
Erik Johnston
0bfdaf1f4f
Rejig the code to make it nicer
2015-08-18 16:26:07 +01:00
Erik Johnston
128ed32e6b
Bump size of get_presence_state cache
2015-08-18 15:51:23 +01:00
Erik Johnston
f704c10f29
Rename unhelpful variable name
2015-08-18 11:54:03 +01:00
Erik Johnston
8199475ce0
Ensure we never return a None event from _get_state_for_groups
2015-08-18 11:44:10 +01:00
Erik Johnston
85d0bc3bdc
Reduce cache size from obscenely large to quite large
2015-08-18 11:00:38 +01:00
Erik Johnston
cfc503681f
Comments
2015-08-18 10:49:23 +01:00
Erik Johnston
8f4165628b
Add index receipts_linearized_room_stream
2015-08-17 14:43:54 +01:00
Erik Johnston
d3d582bc1c
Remove unused import
2015-08-17 13:38:09 +01:00
Erik Johnston
afef6f5d16
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2
2015-08-17 13:23:44 +01:00
Erik Johnston
2d97e65558
Remember to invalidate caches
2015-08-17 10:46:55 +01:00
Erik Johnston
1a9510bb84
Implement a batched presence_handler.get_state and use it
2015-08-17 10:40:23 +01:00
Erik Johnston
47abebfd6d
Add batched version of store.get_presence_state
2015-08-17 09:50:50 +01:00
Erik Johnston
f9d4da7f45
Fix bug where we were leaking None into state event lists
2015-08-17 09:39:45 +01:00
Daniel Wagner-Hall
30883d8409
Merge pull request #221 from matrix-org/auth
...
Simplify LoginHander and AuthHandler
2015-08-14 17:02:22 +01:00
Erik Johnston
68b255c5a1
Batch _get_linearized_receipts_for_rooms
2015-08-14 15:06:22 +01:00
Erik Johnston
129ee4e149
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2
2015-08-13 17:28:43 +01:00
Erik Johnston
2bb2c02571
Remove some vertical space
2015-08-13 17:11:30 +01:00
Erik Johnston
57877b01d7
Replace list comprehension
2015-08-13 17:00:17 +01:00
Erik Johnston
5db5677969
Add metrics to the receipts cache
2015-08-13 16:58:23 +01:00
Erik Johnston
0fbed2a8fa
Comment
2015-08-12 17:22:54 +01:00
Erik Johnston
c10ac7806e
Explain why we're prefilling dict with Nones
2015-08-12 17:16:30 +01:00
Erik Johnston
101ee3fd00
Better variable name
2015-08-12 17:08:05 +01:00
Erik Johnston
df361d08f7
Split _get_state_for_group_from_cache into two
2015-08-12 17:06:21 +01:00
Daniel Wagner-Hall
415c2f0549
Simplify LoginHander and AuthHandler
...
* Merge LoginHandler -> AuthHandler
* Add a bunch of documentation
* Improve some naming
* Remove unused branches
I will start merging the actual logic of the two handlers shortly
2015-08-12 15:49:37 +01:00
Erik Johnston
2df8dd9b37
Move all the caches into their own package, synapse.util.caches
2015-08-11 18:00:59 +01:00
Erik Johnston
53a817518b
Comments
2015-08-11 11:40:40 +01:00
Erik Johnston
6eaa116867
Comment
2015-08-11 11:35:24 +01:00
Erik Johnston
4762c276cb
Docs
2015-08-11 11:33:41 +01:00
Erik Johnston
1b994a97dd
Fix application of ACLs
2015-08-11 10:41:40 +01:00
Erik Johnston
10b874067b
Fix state cache
2015-08-11 09:12:41 +01:00
Erik Johnston
017b798e4f
Clean up StateStore
2015-08-10 15:01:06 +01:00
Erik Johnston
2c019eea11
Remove unused function
2015-08-10 14:44:41 +01:00
Erik Johnston
bb0a475c30
Comments
2015-08-10 14:27:38 +01:00
Mark Haines
559c51debc
Use TypeError instead of ValueError and give a nicer error mesasge
...
when someone calls Cache.invalidate with the wrong type.
2015-08-10 14:07:17 +01:00
Erik Johnston
7ce71f2ffc
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 13:47:51 +01:00
Erik Johnston
8c3a62b5c7
Merge pull request #215 from matrix-org/erikj/cache_varargs_interface
...
Change Cache to not use *args in its interface
2015-08-10 13:47:45 +01:00
Erik Johnston
86eaaa885b
Rename keyargs to args in CacheDescriptor
2015-08-10 13:44:44 +01:00
Erik Johnston
e0b6e49466
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 10:39:22 +01:00
Erik Johnston
2cd6cb9f65
Rename keyargs to args in Cache
2015-08-10 10:38:47 +01:00
Erik Johnston
aa88582e00
Do bounds check
2015-08-10 10:08:15 +01:00
Erik Johnston
5119e416e8
Line length
2015-08-10 10:05:30 +01:00
Erik Johnston
8f04b6fa7a
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:30:25 +01:00
Erik Johnston
7dec0b2bee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:28:39 +01:00
Erik Johnston
06218ab125
Merge pull request #212 from matrix-org/erikj/cache_deferreds
...
Make CacheDescriptor cache deferreds rather than the deferreds' values
2015-08-07 19:28:05 +01:00
Erik Johnston
2352974aab
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:26:54 +01:00
Erik Johnston
9c5385b53a
s/observed/observer/
2015-08-07 19:26:38 +01:00
Erik Johnston
ffab798a38
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:18:47 +01:00
Erik Johnston
62126c996c
Propogate stale cache errors to calling functions
2015-08-07 19:17:58 +01:00
Erik Johnston
3213ff630c
Remove unnecessary cache
2015-08-07 19:14:05 +01:00
Erik Johnston
20addfa358
Change Cache to not use *args in its interface
2015-08-07 18:32:47 +01:00
Erik Johnston
9eb5b23d3a
Batch up various DB requests for event -> state
2015-08-07 18:16:02 +01:00
Erik Johnston
0211890134
Implement a CacheListDescriptor
2015-08-07 18:14:49 +01:00
Paul "LeoNerd" Evans
0db40d3e93
Don't complain about extra .pyc files we find while hunting for database schemas
2015-08-07 17:22:11 +01:00
Erik Johnston
b2c7bd4b09
Cache get_recent_events_for_room
2015-08-07 14:42:34 +01:00
Erik Johnston
b3768ec10a
Remove unncessary cache
2015-08-07 13:41:05 +01:00
Erik Johnston
b8e386db59
Change Cache to not use *args in its interface
2015-08-07 11:52:21 +01:00
Erik Johnston
fe994e728f
Store absence of state in cache
2015-08-07 10:17:38 +01:00
Erik Johnston
1d08bf7c17
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 14:03:15 +01:00
Erik Johnston
63b1eaf32c
Docs
2015-08-06 14:02:50 +01:00
Erik Johnston
b811c98574
Remove failed deferreds from cache
2015-08-06 14:01:27 +01:00
Erik Johnston
433314cc34
Re-implement DEBUG_CACHES flag
2015-08-06 14:01:05 +01:00
Erik Johnston
f596ff402e
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 13:37:56 +01:00
Erik Johnston
2efb93af52
Merge branch 'erikj/cached_keyword_args' into erikj/cache_deferreds
2015-08-06 13:35:28 +01:00
Erik Johnston
953dbd28a7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cached_keyword_args
2015-08-06 13:35:03 +01:00
Erik Johnston
7eea3e356f
Make @cached cache deferreds rather than the deferreds' values
2015-08-06 13:33:34 +01:00
Erik Johnston
3e1b77efc2
Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 16:45:56 +01:00
Erik Johnston
b52b4a84ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 15:41:20 +01:00
Erik Johnston
1e62a3d3a9
Up the cache size for 'get_joined_hosts_for_room' and 'get_users_in_room'
2015-08-05 15:40:40 +01:00
Erik Johnston
a89559d797
Use LRU cache by default
2015-08-05 15:39:47 +01:00
Erik Johnston
07507643cb
Use dictionary cache to do group -> state fetching
2015-08-05 15:11:42 +01:00
Erik Johnston
c67ba143fa
Move DictionaryCache
2015-08-04 15:58:28 +01:00
Erik Johnston
e7768e77f5
Add basic dictionary cache
2015-08-04 15:56:56 +01:00
David Baker
e124128542
Bump schema version
2015-08-04 14:50:31 +01:00
David Baker
c77048e12f
Add endpoint that proxies ID server request token and errors if the given email is in use on this Home Server.
2015-08-04 14:37:09 +01:00
Erik Johnston
413a4c289b
Add comment
2015-08-04 11:08:07 +01:00
Erik Johnston
4d6cb8814e
Speed up event filtering (for ACL) logic
2015-08-04 09:32:23 +01:00
Erik Johnston
39e21ea51c
Add support for using keyword arguments with cached functions
2015-07-27 13:57:29 +01:00
David Baker
20c0324e9c
Dodesn't seem to make any difference: guess it does work with the object reference
2015-07-21 16:21:37 -07:00
David Baker
cf7a40b08a
I think this was what was intended...
2015-07-21 16:08:00 -07:00
Mark Haines
3b5823c74d
s/take/claim/ for end to end key APIs
2015-07-20 18:23:54 +01:00
Matrix
4f973eb657
Up default cache size for _RoomStreamChangeCache
2015-07-18 19:07:33 +01:00
Erik Johnston
b6d4a4c6d8
Merge pull request #199 from matrix-org/erikj/receipts
...
Implement read receipts.
2015-07-16 18:18:36 +01:00
Erik Johnston
baa55fb69e
Merge pull request #193 from matrix-org/erikj/bulk_persist_event
...
Add bulk insert events API
2015-07-14 10:49:24 +01:00
Erik Johnston
4624d6035e
Docs
2015-07-14 10:19:07 +01:00
Erik Johnston
5989637f37
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-13 13:50:57 +01:00
Erik Johnston
e5991af629
Comments
2015-07-13 13:30:43 +01:00
Erik Johnston
17bb9a7eb9
Remove commented out code
2015-07-10 14:07:57 +01:00
Erik Johnston
532fcc997a
Merge pull request #196 from matrix-org/erikj/room_history
...
Add ability to restrict room history.
2015-07-10 13:47:04 +01:00
Erik Johnston
0b3389bcd2
Merge pull request #194 from matrix-org/erikj/bulk_verify_sigs
...
Implement bulk verify_signed_json API
2015-07-10 13:46:53 +01:00
Erik Johnston
0d7f0febf4
Uniquely name unique constraint
2015-07-10 13:43:03 +01:00
Erik Johnston
ed88720952
Handle error slightly better
2015-07-09 16:14:46 +01:00
Mark Haines
bf0d59ed30
Don't bother with a timeout for one time keys on the server.
2015-07-09 14:04:03 +01:00
Erik Johnston
1af188209a
Change format of receipts to allow inclusion of data
2015-07-09 11:39:30 +01:00
Mark Haines
8fb79eeea4
Only remove one time keys when new one time keys are added
2015-07-08 17:04:29 +01:00
Erik Johnston
304343f4d7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-08 15:37:33 +01:00
Erik Johnston
af812b68dd
Add a cache to fetching of receipt streams
2015-07-08 15:35:00 +01:00
Erik Johnston
87311d1b8c
Hook up receipts to v1 initialSync
2015-07-08 11:02:04 +01:00
Erik Johnston
ca041d5526
Wire together receipts and the notifer/federation
2015-07-07 15:25:30 +01:00
Erik Johnston
716e426933
Fix various typos
2015-07-07 10:55:31 +01:00
Mark Haines
b5770f8947
Add store for client end to end keys
2015-07-06 18:46:47 +01:00
Erik Johnston
65e69dec8b
Don't explode if we don't recognize one of the event_ids in the backfill request
2015-07-06 09:33:03 +01:00
Erik Johnston
1a60545626
Add basic impl for room history ACL on GET /messages client API
2015-07-02 16:20:10 +01:00
Erik Johnston
ac78e60de6
Add stream_id index
2015-07-02 13:18:41 +01:00
Erik Johnston
80a61330ee
Add basic storage functions for handling of receipts
2015-07-01 17:19:12 +01:00
Erik Johnston
901f56fa63
Add tables for receipts
2015-06-30 15:29:47 +01:00
Erik Johnston
b5f55a1d85
Implement bulk verify_signed_json API
2015-06-26 10:39:34 +01:00
Erik Johnston
5130d80d79
Add bulk insert events API
2015-06-25 17:29:34 +01:00
Erik Johnston
6924852592
Batch SELECTs in _get_auth_chain_ids_txn
2015-06-23 11:01:04 +01:00
Erik Johnston
2eda996a63
Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py
2015-06-13 12:21:58 +01:00
Erik Johnston
1e365e88bd
Bump schema version
2015-06-11 15:50:39 +01:00
David Baker
da84946de4
pep8
2015-06-04 16:43:45 +01:00
David Baker
63a7b3ad1e
Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them.
2015-06-04 16:16:01 +01:00
Erik Johnston
0d7d9c37b6
Add cache to get_state_groups
2015-06-03 14:45:55 +01:00
Erik Johnston
d8866d7277
Caches should be bound to instances.
...
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
2015-06-03 14:45:17 +01:00
Erik Johnston
6e17463228
Don't explode if we don't have the event
2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf
Add config option to turn off freezing events. Use new encode_json api and ujson.loads
2015-05-29 12:17:33 +01:00
Mark Haines
d9f60e8dc8
Merge pull request #163 from matrix-org/markjh/presence_list_cache
...
Add a cache for the presence list
2015-05-22 17:02:23 +01:00
Mark Haines
0e42dfbe22
Merge pull request #164 from matrix-org/markjh/pusher_performance_2
...
Add a cache for get_push rules for user, fix cache invalidation
2015-05-22 17:01:56 +01:00
Mark Haines
1a9a9abcc7
Add a cache for getting the presence list for a user
2015-05-22 16:11:17 +01:00
Erik Johnston
74b7de83ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-22 16:10:42 +01:00
Mark Haines
36317f3dad
Merge pull request #156 from matrix-org/erikj/join_perf
...
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
Mark Haines
b6adfc59f5
Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities
2015-05-22 13:01:03 +01:00
Mark Haines
a04cde613e
Add a cache for get_push rules for user, fix cache invalidation
2015-05-22 10:39:45 +01:00
Erik Johnston
4429e720ae
Merge branch 'master' of github.com:matrix-org/synapse into develop
2015-05-22 10:33:00 +01:00
Mark Haines
2043527b9b
Don't try to use a txn when not in one, remove spurious debug logging
2015-05-21 16:53:03 +01:00
Mark Haines
53447e9cd3
Add caches for things requested by the pushers
2015-05-21 16:41:39 +01:00
Mark Haines
d61ce3f670
Add a cache for get_current_state with state_key
2015-05-21 16:41:39 +01:00
Erik Johnston
a910984b58
Actually return something from lambda
2015-05-21 15:58:41 +01:00
Erik Johnston
e309b1045d
Sort backfill events
2015-05-21 15:57:35 +01:00
Erik Johnston
0180bfe4aa
Remove dead code
2015-05-21 15:53:41 +01:00
Erik Johnston
1f3d1d85a9
Only get non-state
2015-05-21 15:52:29 +01:00
Erik Johnston
39a3340f73
Skip events we've already seen
2015-05-21 15:48:56 +01:00
Erik Johnston
ae3bff3491
Correctly prepopulate queue
2015-05-21 15:46:07 +01:00
Erik Johnston
dc085ddf8c
Don't prepopulate event_results
2015-05-21 15:44:05 +01:00
Erik Johnston
73d23c6ae8
Don't readd things that are already in event_results
2015-05-21 15:40:22 +01:00
Erik Johnston
6189d8e54d
PriorityQueue gives lowest first
2015-05-21 15:38:08 +01:00
Erik Johnston
115ef3ddac
Correctly capture Queue.Empty exception
2015-05-21 15:37:43 +01:00
Erik Johnston
c2633907c5
Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-21 14:58:47 +01:00
Erik Johnston
ebfdd2eb5b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf
2015-05-21 14:54:52 +01:00
Erik Johnston
a551c5dad7
Merge pull request #155 from matrix-org/erikj/perf
...
Bulk and batch retrieval of events.
2015-05-21 14:54:40 +01:00
Erik Johnston
27e4b45c06
s/for events/for requests for events/
2015-05-21 14:52:23 +01:00
Erik Johnston
ac5f2bf9db
s/for events/for requests for events/
2015-05-21 14:50:57 +01:00
Erik Johnston
80a167b1f0
Add comments
2015-05-21 11:19:04 +01:00
Erik Johnston
9118a92862
Split up _get_events into defer and txn versions
2015-05-20 13:27:16 +01:00
Erik Johnston
ab45e12d31
Make not return a deferred _get_event_from_row_txn
2015-05-20 13:07:19 +01:00
Erik Johnston
f407cbd2f1
PEP8
2015-05-20 13:02:01 +01:00
Erik Johnston
227f8ef031
Split out _get_event_from_row back into defer and _txn version
2015-05-20 13:00:57 +01:00
Erik Johnston
2bc60c55af
Fix _get_backfill_events to return events in the correct order
2015-05-20 12:57:00 +01:00
Erik Johnston
9084cdd70f
Ensure event_results is a set
2015-05-19 16:34:31 +01:00
Erik Johnston
aa729349dd
Fix event_backwards_extrem insertion to ignore outliers
2015-05-19 15:27:00 +01:00
Mark Haines
99914ec9f8
Merge pull request #152 from matrix-org/notifier_performance
...
Notifier performance
2015-05-18 17:49:59 +01:00
Erik Johnston
165eb2dbe6
Comments and shuffle of functions
2015-05-18 15:18:41 +01:00
Erik Johnston
65878a2319
Remove unused metric
2015-05-18 14:06:30 +01:00
Erik Johnston
4d1b6f4ad1
Remove rejected events if we don't want rejected events
2015-05-18 14:03:46 +01:00
Mark Haines
0b0033c40b
Merge branch 'develop' into notifier_performance
2015-05-18 13:50:01 +01:00
Erik Johnston
10f1bdb9a2
Move get_events functions to storage.events
2015-05-18 10:21:40 +01:00
Erik Johnston
c71176858b
Newline, remove debug logging
2015-05-18 10:11:14 +01:00
Erik Johnston
c3b37abdfd
PEP8
2015-05-15 16:59:58 +01:00
Erik Johnston
6c74fd62a0
Revert limiting of fetching, it didn't help perf.
2015-05-15 16:45:35 +01:00
Erik Johnston
9ff7f66a2b
init j
2015-05-15 16:36:03 +01:00
Erik Johnston
70f272f71c
Don't completely drain the list
2015-05-15 16:34:17 +01:00
Erik Johnston
8763dd80ef
Don't fetch prev_content for current_state
2015-05-15 15:33:01 +01:00
Erik Johnston
acb12cc811
Make store.get_current_state fetch events asyncly
2015-05-15 15:20:05 +01:00
Erik Johnston
d62dee7eae
Remove more debug logging
2015-05-15 15:06:37 +01:00
Erik Johnston
0f29cfabc3
Remove debug logging
2015-05-15 14:06:42 +01:00
Erik Johnston
e275a9c0d9
preserve log context
2015-05-15 11:54:51 +01:00
Erik Johnston
aa32bd38e4
Add a wait
2015-05-15 11:35:04 +01:00
Erik Johnston
372d4c6d7b
Srsly. Don't use closures. Baaaaaad
2015-05-15 11:26:00 +01:00
Erik Johnston
575ec91d82
Correctly pass through params
2015-05-15 11:15:10 +01:00
Erik Johnston
de01438a57
Sort out error handling
2015-05-15 11:00:50 +01:00
Erik Johnston
a2c4f3f150
Fix daedlock
2015-05-15 10:54:04 +01:00
Erik Johnston
1d566edb81
Remove race condition
2015-05-14 16:54:35 +01:00
Erik Johnston
ef3d8754f5
Call from right thread
2015-05-14 15:41:55 +01:00
Erik Johnston
142934084a
Count and loop
2015-05-14 15:40:21 +01:00
Erik Johnston
96c5b9f87c
Don't start up more fetch_events
2015-05-14 15:36:04 +01:00
Erik Johnston
7cd6a6f6cf
Awful idea for speeding up fetching of events
2015-05-14 15:34:02 +01:00
Erik Johnston
7f4105a5c9
Turn off preemptive transactions
2015-05-14 14:51:06 +01:00
Erik Johnston
f4d58deba1
PEP8
2015-05-14 14:45:42 +01:00
Erik Johnston
386b7330d2
Move from _base to events
2015-05-14 14:45:22 +01:00
Erik Johnston
7d6a1dae31
Jump out early
2015-05-14 14:27:58 +01:00
Erik Johnston
656223fbd3
Actually, we probably want to run this in a transaction
2015-05-14 14:26:35 +01:00
Erik Johnston
2f7f8e1c2b
Preemptively jump into a transaction if we ask for get_prev_content
2015-05-14 14:17:36 +01:00
Erik Johnston
e1e9f0c5b2
loop -> gatherResults
2015-05-14 13:58:49 +01:00
Erik Johnston
ab78a8926e
Err, we probably want a bigger limit
2015-05-14 13:47:16 +01:00
Erik Johnston
f6f902d459
Move fetching of events into their own transactions
2015-05-14 13:45:48 +01:00
Erik Johnston
cdb3757942
Refactor _get_events
2015-05-14 13:31:55 +01:00
Erik Johnston
36ea26c5c0
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf
2015-05-14 12:01:38 +01:00
David Baker
7c549dd557
Add ID generator for push_rules_enable to #resolve SYN-378
2015-05-14 11:44:03 +01:00
Mark Haines
899d4675dd
Merge branch 'notifier_unify' into notifier_performance
2015-05-14 11:36:44 +01:00
Erik Johnston
4f1d984e56
Add index on events
2015-05-13 17:22:26 +01:00
Erik Johnston
968b01a91a
Actually use async method
2015-05-13 17:02:46 +01:00
Erik Johnston
4071f29653
Fetch events from events_id in their own transactions
2015-05-13 16:59:41 +01:00
Erik Johnston
a988361aea
Typo
2015-05-13 15:44:15 +01:00
Erik Johnston
8888982db3
Don't insert None
2015-05-13 15:43:32 +01:00
Erik Johnston
cf706cc6ef
Don't return None
2015-05-13 15:31:25 +01:00
Erik Johnston
5971d240d4
Limit batch size
2015-05-13 15:26:49 +01:00
Erik Johnston
ca4f458787
Fetch events in bulk
2015-05-13 15:13:42 +01:00
Erik Johnston
6edff11a88
Don't fetch redaction and rejection stuff for each event, so we can use index only scan
2015-05-13 14:39:05 +01:00
Mark Haines
63878c0379
Don't bother checking for updates if the stream token hasn't advanced for a user
2015-05-13 13:42:21 +01:00
Erik Johnston
02590c3e1d
Temp turn off checking for rejections and redactions
2015-05-13 11:31:28 +01:00
Erik Johnston
619a21812b
defer.gatherResults loop
2015-05-13 11:29:03 +01:00
Erik Johnston
fec4485e28
Batch fetching of events for state groups
2015-05-13 11:22:42 +01:00
Erik Johnston
409bcc76bd
Load events for state group seperately
2015-05-13 11:13:31 +01:00
Erik Johnston
e122685978
You need to call contextmanager
2015-05-12 16:12:37 +01:00
Mark Haines
a6fb2aa2a5
Merge pull request #144 from matrix-org/erikj/logging_context
...
Preserving logging contexts
2015-05-12 15:23:50 +01:00
Mark Haines
1fce36b111
Merge pull request #149 from matrix-org/erikj/backfill
...
Backfill support
2015-05-12 15:20:32 +01:00
Erik Johnston
8b28209c60
Err, delete the right stuff
2015-05-12 15:02:53 +01:00
Erik Johnston
30c72d377e
Newlines
2015-05-12 14:47:40 +01:00
Erik Johnston
e4eddf9b36
We do actually want to delete rows out of event_backward_extremities
2015-05-12 14:47:23 +01:00
Erik Johnston
c1779a79bc
Fix up _handle_prev_events to not try to insert duplicate rows
2015-05-12 14:41:50 +01:00
Erik Johnston
74850d7f75
Do state groups persistence /after/ checking if we have already persisted the event
2015-05-12 14:14:58 +01:00
Erik Johnston
da6a7bbdde
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context
2015-05-12 13:10:42 +01:00
Mark Haines
5002056b16
SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes.
2015-05-12 11:20:40 +01:00
Erik Johnston
4df11b5039
Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards'
2015-05-12 10:28:10 +01:00
Erik Johnston
84e6b4001f
Initial hack at wiring together pagination and backfill
2015-05-11 18:01:31 +01:00
Erik Johnston
17653a5dfe
Move storage.stream._StreamToken to types.RoomStreamToken
2015-05-11 18:01:01 +01:00
Erik Johnston
b036596b75
Prefer to use _simple_*.
2015-05-11 11:24:01 +01:00
Erik Johnston
cd525c0f5a
push_rules table expects an 'id' field
2015-05-11 11:24:01 +01:00
Erik Johnston
22d2f498fa
Fix push rule bug: can't insert bool into small int column
2015-05-10 10:50:51 +01:00
Erik Johnston
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
Mark Haines
1c7912751e
Drop the old table not the new table
2015-05-08 16:04:32 +01:00
Mark Haines
9d36eb4eab
Rename unique constraint
2015-05-08 16:01:55 +01:00
Mark Haines
b0f71db3ff
Remove unsigned
2015-05-08 15:59:51 +01:00
Mark Haines
84e1cacea4
Bump schema version
2015-05-08 15:58:14 +01:00
Mark Haines
6538d445e8
Make the timestamps in server_keys_json bigints
2015-05-08 15:55:17 +01:00
Erik Johnston
0c4ac271ca
Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371
2015-05-05 18:21:19 +01:00
Erik Johnston
0cf7e480b4
And use buffer(...) there as well
2015-05-05 18:20:01 +01:00
Erik Johnston
ed2584050f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany
2015-05-05 18:15:20 +01:00
Erik Johnston
977338a7af
Use buffer(...) when inserting into bytea column
2015-05-05 18:12:53 +01:00
Mark Haines
31049c4d72
Merge pull request #139 from matrix-org/bugs/SYN-369
...
Fix race with cache invalidation. SYN-369
2015-05-05 17:46:13 +01:00
Mark Haines
deb0237166
Add some doc-string
2015-05-05 17:45:11 +01:00
Mark Haines
d18f37e026
Collect the invalidate callbacks on the transaction object rather than passing around a separate list
2015-05-05 17:32:21 +01:00
Erik Johnston
9951542393
Add a comment about the zip(*[zip(sorted(...),...)])
2015-05-05 17:06:55 +01:00
Mark Haines
041b6cba61
SYN-369: Add comments to the sequence number logic in the cache
2015-05-05 16:32:44 +01:00
Mark Haines
63075118a5
Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access
2015-05-05 16:24:04 +01:00
Erik Johnston
531d7955fd
Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely
2015-05-05 16:12:28 +01:00
Mark Haines
bfa4a7f8b0
Invalidate the room_member cache if the current state events updates
2015-05-05 15:43:49 +01:00
Mark Haines
d0fece8d3c
Missing return for when the event was already persisted
2015-05-05 15:39:09 +01:00
Erik Johnston
bdcd7693c8
Fix indentation
2015-05-05 15:14:48 +01:00
Erik Johnston
43c2e8deae
Add support for using executemany
2015-05-05 15:13:25 +01:00
Erik Johnston
1692dc019d
Don't call 'encode_parameter' no-op
2015-05-05 15:00:30 +01:00
Mark Haines
a9aea68fd5
Invalidate the caches from the correct thread
2015-05-05 14:57:08 +01:00
Mark Haines
261d809a47
Sequence the modifications to the cache so that selects don't race with inserts
2015-05-05 14:13:50 +01:00
Erik Johnston
d9cc5de9e5
Correctly name transaction
2015-05-05 10:24:10 +01:00
Erik Johnston
b8940cd902
Remove some unused indexes
2015-05-01 16:14:25 +01:00
David Baker
eb9bd2d949
user_id now in user_threepids
2015-05-01 15:04:37 +01:00
David Baker
9eeb03c0dd
Don't use self.execute: it's designed for fetching stuff
2015-05-01 14:21:25 +01:00
David Baker
7693f24792
No id field on user
2015-05-01 13:55:42 +01:00
David Baker
92b20713d7
More missed get_user_by_id API changes
2015-05-01 13:45:54 +01:00
Erik Johnston
4011cf1c42
Cache latest_event_ids_in_room
2015-05-01 13:06:26 +01:00
Erik Johnston
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
Erik Johnston
fabb7acd45
Fix bug where we reconnected to the database on every query.
2015-05-01 10:24:24 +01:00
Erik Johnston
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
Erik Johnston
8be5284e91
Remove pointless join
2015-04-30 18:49:26 +01:00
Erik Johnston
503e4d3d52
Fix broken SQL
2015-04-30 18:44:47 +01:00
Erik Johnston
00718ae7a9
Need more yield
2015-04-30 18:43:39 +01:00
Erik Johnston
0465560c1a
Add missing param
2015-04-30 18:42:44 +01:00
Erik Johnston
61d05daab1
More join conditions
2015-04-30 18:41:05 +01:00
Erik Johnston
6ead27ddda
Add more conditions on JOINs to make postgres go a little faster.
2015-04-30 18:32:03 +01:00
Erik Johnston
f383d5a801
Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction
2015-04-30 17:12:52 +01:00
Erik Johnston
69d4063651
Add get_rooms_for_user cache
2015-04-30 16:47:51 +01:00
Erik Johnston
3c4c229788
Don't use sub queries, it makes postgres sad
2015-04-30 16:16:53 +01:00
Mark Haines
29400b45b9
SYN-367: Use upsert rather than insert_or_replace
2015-04-30 15:21:31 +01:00
Erik Johnston
2366d28780
Don't needlessly join on state_events
2015-04-30 14:02:06 +01:00
Erik Johnston
1d7702833d
Make simple query rather than long one and then throw away half the results
2015-04-30 10:16:12 +01:00
Erik Johnston
b1ca784aca
Correctly decode, for sqlite and postgres, rows from pushers table
2015-04-29 19:41:14 +01:00
Erik Johnston
4a9dc5b2f5
pushkey' are also bytes.
2015-04-29 19:27:02 +01:00
Erik Johnston
0ade2712d1
Typo
2015-04-29 19:17:25 +01:00
Erik Johnston
50f96f256f
Also remove yield from within lock in the other generator
2015-04-29 19:17:00 +01:00
Erik Johnston
d2d61a8288
Fix deadlock in id_generators. No idea why this was an actual deadlock.
2015-04-29 19:15:23 +01:00
Erik Johnston
3864b3a8e6
Actually return rows
2015-04-29 18:07:36 +01:00
Erik Johnston
0618978238
Typo, args wrong way round
2015-04-29 18:04:35 +01:00
Erik Johnston
09177f4f2e
Decode buffers in same thread
2015-04-29 18:03:42 +01:00
Erik Johnston
472be88674
We store pusher data as bytes
2015-04-29 17:43:46 +01:00
David Baker
12d381bd5d
Decode the data json in the storage layer (was moved but this part was missed)
2015-04-29 17:13:51 +01:00
David Baker
f8c30faf25
Oops, update the contraint too
2015-04-29 16:58:42 +01:00
David Baker
61cd5d9045
Be more postgressive
2015-04-29 16:57:14 +01:00
David Baker
fb95035a65
Be postgressive
2015-04-29 16:53:41 +01:00
David Baker
4669def000
Oops, forgot the schema delta file
2015-04-29 16:50:16 +01:00
Erik Johnston
0337eaf321
txn.execute doesn't return cursors
2015-04-29 16:43:39 +01:00
Erik Johnston
884fb88e28
txn.execute doesn't return cursors
2015-04-29 16:35:20 +01:00
Erik Johnston
d76c058eea
Fix invalid SQL to work in postgres land
2015-04-29 16:30:25 +01:00
Erik Johnston
c8ed9bd278
pushers table requires an access_token
2015-04-29 15:33:27 +01:00
Erik Johnston
970a9b9d2b
We can't use REPLACE when upgrading databases now we have postgres
2015-04-29 13:55:44 +01:00
