Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-12-28 05:19:22 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,901 Commits 4 Branches 615 Tags 63 MiB
2c86187a1b
Commit Graph

1308 Commits

Author SHA1 Message Date
Mark Haines
527d95dea0 synapse/storage/_base.py:Table was unused 2015-09-22 18:14:15 +01:00
Daniel Wagner-Hall
6d59ffe1ce Add some docstrings 2015-09-22 13:47:40 +01:00
Daniel Wagner-Hall
eb011cd99b Add docstring 2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083 Implement configurable stats reporting 
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
 2015-09-22 12:57:40 +01:00
Mark Haines
ee2d722f0f Merge pull request #276 from matrix-org/markjh/history_for_rooms_that_have_been_left 
SPEC-216: Allow users to view the history of rooms that they have left.
 2015-09-21 14:38:13 +01:00
Erik Johnston
2048388cfd Merge pull request #281 from matrix-org/erikj/update_extremeties 
When updating a stored event from outlier to non-outlier, remember to update the extremeties
 2015-09-15 16:57:25 +01:00
Erik Johnston
8148c48f11 "Comments" 2015-09-15 16:54:48 +01:00
Erik Johnston
3b05b67c89 When updating a stored event from outlier to non-outlier, remember to update the extremeties 2015-09-15 16:34:42 +01:00
Erik Johnston
dffc9c4ae0 Drop unused index 2015-09-14 14:41:37 +01:00
Mark Haines
09cb5c7d33 Allow users that have left a room to get the messages that happend in the room before they left 2015-09-09 17:31:09 +01:00
Mark Haines
3c166a24c5 Remove undocumented and unimplemented 'feedback' parameter from the Client-Server API 2015-09-09 16:05:09 +01:00
Mark Haines
89ae0166de Allow room initialSync for users that have left the room, returning a snapshot of how the room was when they left it 2015-09-09 13:25:22 +01:00
Mark Haines
417485eefa Include the event_id and stream_ordering of membership events when looking up which rooms a user is in 2015-09-08 18:14:54 +01:00
Erik Johnston
b62c1395d6 Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop 2015-09-01 13:11:55 +01:00
Erik Johnston
cd800ad99a Lower size of 'stateGroupCache' now that we have data from matrix.org to support doing so 2015-09-01 10:09:03 +01:00
Mark Haines
a7122692d9 Merge branch 'release-v0.10.0' into develop 
Conflicts:
	synapse/handlers/auth.py
	synapse/python_dependencies.py
	synapse/rest/client/v1/login.py
 2015-08-28 11:15:27 +01:00
Daniel Wagner-Hall
d3c0e48859 Merge erikj/user_dedup to develop 2015-08-26 13:42:45 +01:00
Daniel Wagner-Hall
a2355fae7e Merge pull request #251 from matrix-org/removeadmin 
Stop looking up "admin", which we never read
 2015-08-25 17:23:05 +01:00
Daniel Wagner-Hall
ee3fa1a99c Merge pull request #248 from matrix-org/deviceid 
Remove completely unused concepts from codebase
 2015-08-25 17:19:06 +01:00
Daniel Wagner-Hall
a9d8bd95e7 Stop looking up "admin", which we never read 2015-08-25 16:29:39 +01:00
Daniel Wagner-Hall
a0b181bd17 Remove completely unused concepts from codebase 
Removes device_id and ClientInfo

device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.

This doesn't remove the columns from the database, because that's
fiddly.
 2015-08-25 16:23:06 +01:00
Mark Haines
1c847af28a Merge pull request #243 from matrix-org/markjh/remove_syutil 
Replace syutil dependency with smaller, single-purpose libraries
 2015-08-25 10:52:16 +01:00
Mark Haines
037481a033 Remove autoincrement since we incrementing the ID in the storage layer 2015-08-24 17:48:57 +01:00
Mark Haines
01fc3943f1 Fix indent 2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3 Remove syutil dependency in favour of smaller single-purpose libraries 2015-08-24 16:17:38 +01:00
Erik Johnston
457970c724 Don't insert events into 'event_*_extremeties' tables if they're outliers 2015-08-23 13:44:23 +01:00
Erik Johnston
1bd1a43073 Actually check if event_id isn't returned by _get_state_groups 2015-08-21 14:30:34 +01:00
Erik Johnston
745b72660a Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop 2015-08-21 11:39:38 +01:00
Erik Johnston
42f12ad92f When logging in fetch user by user_id case insensitively, *unless* there are multiple case insensitive matches, in which case require the exact user_id 2015-08-21 11:38:44 +01:00
Erik Johnston
aa3c9c7bd0 Don't allow people to register user ids which only differ by case to an existing one 2015-08-21 10:57:47 +01:00
Erik Johnston
1f7642efa9 Fix bug where we didn't correctly serialize the redacted_because key over federation 2015-08-21 09:36:07 +01:00
Daniel Wagner-Hall
e8cf77fa49 Merge branch 'develop' into refresh 
Conflicts:
	synapse/rest/client/v1/login.py
 2015-08-20 16:25:40 +01:00
Daniel Wagner-Hall
cecbd636e9 /tokenrefresh POST endpoint 
This allows refresh tokens to be exchanged for (access_token,
refresh_token).

It also starts issuing them on login, though no clients currently
interpret them.
 2015-08-20 16:21:35 +01:00
Daniel Wagner-Hall
13a6517d89 s/by_token/by_access_token/g 
We're about to have two kinds of token, access and refresh
 2015-08-20 16:01:29 +01:00
Mark Haines
a0b8e5f2fe Merge pull request #211 from matrix-org/email_in_use 
Changes for unique emails
 2015-08-20 10:04:04 +01:00
Erik Johnston
e624cdec64 Merge pull request #228 from matrix-org/erikj/_get_state_for_groups 
Ensure we never return a None event from _get_state_for_groups
 2015-08-18 16:30:17 +01:00
Erik Johnston
38a965b816 Merge pull request #227 from matrix-org/erikj/receipts_take2 
Re-enable receipts API.
 2015-08-18 16:30:04 +01:00
Erik Johnston
a82938416d Remove newline because vertical whitespace makes mjark sad 2015-08-18 16:28:13 +01:00
Erik Johnston
0bfdaf1f4f Rejig the code to make it nicer 2015-08-18 16:26:07 +01:00
Erik Johnston
128ed32e6b Bump size of get_presence_state cache 2015-08-18 15:51:23 +01:00
Erik Johnston
f704c10f29 Rename unhelpful variable name 2015-08-18 11:54:03 +01:00
Erik Johnston
8199475ce0 Ensure we never return a None event from _get_state_for_groups 2015-08-18 11:44:10 +01:00
Erik Johnston
85d0bc3bdc Reduce cache size from obscenely large to quite large 2015-08-18 11:00:38 +01:00
Erik Johnston
cfc503681f Comments 2015-08-18 10:49:23 +01:00
Erik Johnston
8f4165628b Add index receipts_linearized_room_stream 2015-08-17 14:43:54 +01:00
Erik Johnston
d3d582bc1c Remove unused import 2015-08-17 13:38:09 +01:00
Erik Johnston
afef6f5d16 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2 2015-08-17 13:23:44 +01:00
Erik Johnston
2d97e65558 Remember to invalidate caches 2015-08-17 10:46:55 +01:00
Erik Johnston
1a9510bb84 Implement a batched presence_handler.get_state and use it 2015-08-17 10:40:23 +01:00
Erik Johnston
47abebfd6d Add batched version of store.get_presence_state 2015-08-17 09:50:50 +01:00
Erik Johnston
f9d4da7f45 Fix bug where we were leaking None into state event lists 2015-08-17 09:39:45 +01:00
Daniel Wagner-Hall
30883d8409 Merge pull request #221 from matrix-org/auth 
Simplify LoginHander and AuthHandler
 2015-08-14 17:02:22 +01:00
Erik Johnston
68b255c5a1 Batch _get_linearized_receipts_for_rooms 2015-08-14 15:06:22 +01:00
Erik Johnston
129ee4e149 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2 2015-08-13 17:28:43 +01:00
Erik Johnston
2bb2c02571 Remove some vertical space 2015-08-13 17:11:30 +01:00
Erik Johnston
57877b01d7 Replace list comprehension 2015-08-13 17:00:17 +01:00
Erik Johnston
5db5677969 Add metrics to the receipts cache 2015-08-13 16:58:23 +01:00
Erik Johnston
0fbed2a8fa Comment 2015-08-12 17:22:54 +01:00
Erik Johnston
c10ac7806e Explain why we're prefilling dict with Nones 2015-08-12 17:16:30 +01:00
Erik Johnston
101ee3fd00 Better variable name 2015-08-12 17:08:05 +01:00
Erik Johnston
df361d08f7 Split _get_state_for_group_from_cache into two 2015-08-12 17:06:21 +01:00
Daniel Wagner-Hall
415c2f0549 Simplify LoginHander and AuthHandler 
* Merge LoginHandler -> AuthHandler
 * Add a bunch of documentation
 * Improve some naming
 * Remove unused branches

I will start merging the actual logic of the two handlers shortly
 2015-08-12 15:49:37 +01:00
Erik Johnston
2df8dd9b37 Move all the caches into their own package, synapse.util.caches 2015-08-11 18:00:59 +01:00
Erik Johnston
53a817518b Comments 2015-08-11 11:40:40 +01:00
Erik Johnston
6eaa116867 Comment 2015-08-11 11:35:24 +01:00
Erik Johnston
4762c276cb Docs 2015-08-11 11:33:41 +01:00
Erik Johnston
1b994a97dd Fix application of ACLs 2015-08-11 10:41:40 +01:00
Erik Johnston
10b874067b Fix state cache 2015-08-11 09:12:41 +01:00
Erik Johnston
017b798e4f Clean up StateStore 2015-08-10 15:01:06 +01:00
Erik Johnston
2c019eea11 Remove unused function 2015-08-10 14:44:41 +01:00
Erik Johnston
bb0a475c30 Comments 2015-08-10 14:27:38 +01:00
Mark Haines
559c51debc Use TypeError instead of ValueError and give a nicer error mesasge 
when someone calls Cache.invalidate with the wrong type.
 2015-08-10 14:07:17 +01:00
Erik Johnston
7ce71f2ffc Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-10 13:47:51 +01:00
Erik Johnston
8c3a62b5c7 Merge pull request #215 from matrix-org/erikj/cache_varargs_interface 
Change Cache to not use *args in its interface
 2015-08-10 13:47:45 +01:00
Erik Johnston
86eaaa885b Rename keyargs to args in CacheDescriptor 2015-08-10 13:44:44 +01:00
Erik Johnston
e0b6e49466 Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-10 10:39:22 +01:00
Erik Johnston
2cd6cb9f65 Rename keyargs to args in Cache 2015-08-10 10:38:47 +01:00
Erik Johnston
aa88582e00 Do bounds check 2015-08-10 10:08:15 +01:00
Erik Johnston
5119e416e8 Line length 2015-08-10 10:05:30 +01:00
Erik Johnston
8f04b6fa7a Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-07 19:30:25 +01:00
Erik Johnston
7dec0b2bee Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-07 19:28:39 +01:00
Erik Johnston
06218ab125 Merge pull request #212 from matrix-org/erikj/cache_deferreds 
Make CacheDescriptor cache deferreds rather than the deferreds' values
 2015-08-07 19:28:05 +01:00
Erik Johnston
2352974aab Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface 2015-08-07 19:26:54 +01:00
Erik Johnston
9c5385b53a s/observed/observer/ 2015-08-07 19:26:38 +01:00
Erik Johnston
ffab798a38 Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface 2015-08-07 19:18:47 +01:00
Erik Johnston
62126c996c Propogate stale cache errors to calling functions 2015-08-07 19:17:58 +01:00
Erik Johnston
3213ff630c Remove unnecessary cache 2015-08-07 19:14:05 +01:00
Erik Johnston
20addfa358 Change Cache to not use *args in its interface 2015-08-07 18:32:47 +01:00
Erik Johnston
9eb5b23d3a Batch up various DB requests for event -> state 2015-08-07 18:16:02 +01:00
Erik Johnston
0211890134 Implement a CacheListDescriptor 2015-08-07 18:14:49 +01:00
Paul "LeoNerd" Evans
0db40d3e93 Don't complain about extra .pyc files we find while hunting for database schemas 2015-08-07 17:22:11 +01:00
Erik Johnston
b2c7bd4b09 Cache get_recent_events_for_room 2015-08-07 14:42:34 +01:00
Erik Johnston
b3768ec10a Remove unncessary cache 2015-08-07 13:41:05 +01:00
Erik Johnston
b8e386db59 Change Cache to not use *args in its interface 2015-08-07 11:52:21 +01:00
Erik Johnston
fe994e728f Store absence of state in cache 2015-08-07 10:17:38 +01:00
Erik Johnston
1d08bf7c17 Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache 2015-08-06 14:03:15 +01:00
Erik Johnston
63b1eaf32c Docs 2015-08-06 14:02:50 +01:00
Erik Johnston
b811c98574 Remove failed deferreds from cache 2015-08-06 14:01:27 +01:00
Erik Johnston
433314cc34 Re-implement DEBUG_CACHES flag 2015-08-06 14:01:05 +01:00
Erik Johnston
f596ff402e Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache 2015-08-06 13:37:56 +01:00
Erik Johnston
2efb93af52 Merge branch 'erikj/cached_keyword_args' into erikj/cache_deferreds 2015-08-06 13:35:28 +01:00
Erik Johnston
953dbd28a7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cached_keyword_args 2015-08-06 13:35:03 +01:00
Erik Johnston
7eea3e356f Make @cached cache deferreds rather than the deferreds' values 2015-08-06 13:33:34 +01:00
Erik Johnston
3e1b77efc2 Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-05 16:45:56 +01:00
Erik Johnston
b52b4a84ec Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-05 15:41:20 +01:00
Erik Johnston
1e62a3d3a9 Up the cache size for 'get_joined_hosts_for_room' and 'get_users_in_room' 2015-08-05 15:40:40 +01:00
Erik Johnston
a89559d797 Use LRU cache by default 2015-08-05 15:39:47 +01:00
Erik Johnston
07507643cb Use dictionary cache to do group -> state fetching 2015-08-05 15:11:42 +01:00
Erik Johnston
c67ba143fa Move DictionaryCache 2015-08-04 15:58:28 +01:00
Erik Johnston
e7768e77f5 Add basic dictionary cache 2015-08-04 15:56:56 +01:00
David Baker
e124128542 Bump schema version 2015-08-04 14:50:31 +01:00
David Baker
c77048e12f Add endpoint that proxies ID server request token and errors if the given email is in use on this Home Server. 2015-08-04 14:37:09 +01:00
Erik Johnston
413a4c289b Add comment 2015-08-04 11:08:07 +01:00
Erik Johnston
4d6cb8814e Speed up event filtering (for ACL) logic 2015-08-04 09:32:23 +01:00
Erik Johnston
39e21ea51c Add support for using keyword arguments with cached functions 2015-07-27 13:57:29 +01:00
David Baker
20c0324e9c Dodesn't seem to make any difference: guess it does work with the object reference 2015-07-21 16:21:37 -07:00
David Baker
cf7a40b08a I think this was what was intended... 2015-07-21 16:08:00 -07:00
Mark Haines
3b5823c74d s/take/claim/ for end to end key APIs 2015-07-20 18:23:54 +01:00
Matrix
4f973eb657 Up default cache size for _RoomStreamChangeCache 2015-07-18 19:07:33 +01:00
Erik Johnston
b6d4a4c6d8 Merge pull request #199 from matrix-org/erikj/receipts 
Implement read receipts.
 2015-07-16 18:18:36 +01:00
Erik Johnston
baa55fb69e Merge pull request #193 from matrix-org/erikj/bulk_persist_event 
Add bulk insert events API
 2015-07-14 10:49:24 +01:00
Erik Johnston
4624d6035e Docs 2015-07-14 10:19:07 +01:00
Erik Johnston
5989637f37 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts 2015-07-13 13:50:57 +01:00
Erik Johnston
e5991af629 Comments 2015-07-13 13:30:43 +01:00
Erik Johnston
17bb9a7eb9 Remove commented out code 2015-07-10 14:07:57 +01:00
Erik Johnston
532fcc997a Merge pull request #196 from matrix-org/erikj/room_history 
Add ability to restrict room history.
 2015-07-10 13:47:04 +01:00
Erik Johnston
0b3389bcd2 Merge pull request #194 from matrix-org/erikj/bulk_verify_sigs 
Implement bulk verify_signed_json API
 2015-07-10 13:46:53 +01:00
Erik Johnston
0d7f0febf4 Uniquely name unique constraint 2015-07-10 13:43:03 +01:00
Erik Johnston
ed88720952 Handle error slightly better 2015-07-09 16:14:46 +01:00
Mark Haines
bf0d59ed30 Don't bother with a timeout for one time keys on the server. 2015-07-09 14:04:03 +01:00
Erik Johnston
1af188209a Change format of receipts to allow inclusion of data 2015-07-09 11:39:30 +01:00
Mark Haines
8fb79eeea4 Only remove one time keys when new one time keys are added 2015-07-08 17:04:29 +01:00
Erik Johnston
304343f4d7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts 2015-07-08 15:37:33 +01:00
Erik Johnston
af812b68dd Add a cache to fetching of receipt streams 2015-07-08 15:35:00 +01:00
Erik Johnston
87311d1b8c Hook up receipts to v1 initialSync 2015-07-08 11:02:04 +01:00
Erik Johnston
ca041d5526 Wire together receipts and the notifer/federation 2015-07-07 15:25:30 +01:00
Erik Johnston
716e426933 Fix various typos 2015-07-07 10:55:31 +01:00
Mark Haines
b5770f8947 Add store for client end to end keys 2015-07-06 18:46:47 +01:00
Erik Johnston
65e69dec8b Don't explode if we don't recognize one of the event_ids in the backfill request 2015-07-06 09:33:03 +01:00
Erik Johnston
1a60545626 Add basic impl for room history ACL on GET /messages client API 2015-07-02 16:20:10 +01:00
Erik Johnston
ac78e60de6 Add stream_id index 2015-07-02 13:18:41 +01:00
Erik Johnston
80a61330ee Add basic storage functions for handling of receipts 2015-07-01 17:19:12 +01:00
Erik Johnston
901f56fa63 Add tables for receipts 2015-06-30 15:29:47 +01:00
Erik Johnston
b5f55a1d85 Implement bulk verify_signed_json API 2015-06-26 10:39:34 +01:00
Erik Johnston
5130d80d79 Add bulk insert events API 2015-06-25 17:29:34 +01:00
Erik Johnston
6924852592 Batch SELECTs in _get_auth_chain_ids_txn 2015-06-23 11:01:04 +01:00
Erik Johnston
2eda996a63 Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py 2015-06-13 12:21:58 +01:00
Erik Johnston
1e365e88bd Bump schema version 2015-06-11 15:50:39 +01:00
David Baker
da84946de4 pep8 2015-06-04 16:43:45 +01:00
David Baker
63a7b3ad1e Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them. 2015-06-04 16:16:01 +01:00
Erik Johnston
0d7d9c37b6 Add cache to get_state_groups 2015-06-03 14:45:55 +01:00
Erik Johnston
d8866d7277 Caches should be bound to instances. 
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
 2015-06-03 14:45:17 +01:00
Erik Johnston
6e17463228 Don't explode if we don't have the event 2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf Add config option to turn off freezing events. Use new encode_json api and ujson.loads 2015-05-29 12:17:33 +01:00
Mark Haines
d9f60e8dc8 Merge pull request #163 from matrix-org/markjh/presence_list_cache 
Add a cache for the presence list
 2015-05-22 17:02:23 +01:00
Mark Haines
0e42dfbe22 Merge pull request #164 from matrix-org/markjh/pusher_performance_2 
Add a cache for get_push rules for user, fix cache invalidation
 2015-05-22 17:01:56 +01:00
Mark Haines
1a9a9abcc7 Add a cache for getting the presence list for a user 2015-05-22 16:11:17 +01:00
Erik Johnston
74b7de83ec Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-22 16:10:42 +01:00
Mark Haines
36317f3dad Merge pull request #156 from matrix-org/erikj/join_perf 
Make joining #matrix:matrix.org over federation quicker
 2015-05-22 16:09:54 +01:00
Mark Haines
b6adfc59f5 Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities 2015-05-22 13:01:03 +01:00
Mark Haines
a04cde613e Add a cache for get_push rules for user, fix cache invalidation 2015-05-22 10:39:45 +01:00
Erik Johnston
4429e720ae Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-22 10:33:00 +01:00
Mark Haines
2043527b9b Don't try to use a txn when not in one, remove spurious debug logging 2015-05-21 16:53:03 +01:00
Mark Haines
53447e9cd3 Add caches for things requested by the pushers 2015-05-21 16:41:39 +01:00
Mark Haines
d61ce3f670 Add a cache for get_current_state with state_key 2015-05-21 16:41:39 +01:00
Erik Johnston
a910984b58 Actually return something from lambda 2015-05-21 15:58:41 +01:00
Erik Johnston
e309b1045d Sort backfill events 2015-05-21 15:57:35 +01:00
Erik Johnston
0180bfe4aa Remove dead code 2015-05-21 15:53:41 +01:00
Erik Johnston
1f3d1d85a9 Only get non-state 2015-05-21 15:52:29 +01:00
Erik Johnston
39a3340f73 Skip events we've already seen 2015-05-21 15:48:56 +01:00
Erik Johnston
ae3bff3491 Correctly prepopulate queue 2015-05-21 15:46:07 +01:00
Erik Johnston
dc085ddf8c Don't prepopulate event_results 2015-05-21 15:44:05 +01:00
Erik Johnston
73d23c6ae8 Don't readd things that are already in event_results 2015-05-21 15:40:22 +01:00
Erik Johnston
6189d8e54d PriorityQueue gives lowest first 2015-05-21 15:38:08 +01:00
Erik Johnston
115ef3ddac Correctly capture Queue.Empty exception 2015-05-21 15:37:43 +01:00
Erik Johnston
c2633907c5 Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-21 14:58:47 +01:00
Erik Johnston
ebfdd2eb5b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-21 14:54:52 +01:00
Erik Johnston
a551c5dad7 Merge pull request #155 from matrix-org/erikj/perf 
Bulk and batch retrieval of events.
 2015-05-21 14:54:40 +01:00
Erik Johnston
27e4b45c06 s/for events/for requests for events/ 2015-05-21 14:52:23 +01:00
Erik Johnston
ac5f2bf9db s/for events/for requests for events/ 2015-05-21 14:50:57 +01:00
Erik Johnston
80a167b1f0 Add comments 2015-05-21 11:19:04 +01:00
Erik Johnston
9118a92862 Split up _get_events into defer and txn versions 2015-05-20 13:27:16 +01:00
Erik Johnston
ab45e12d31 Make not return a deferred _get_event_from_row_txn 2015-05-20 13:07:19 +01:00
Erik Johnston
f407cbd2f1 PEP8 2015-05-20 13:02:01 +01:00
Erik Johnston
227f8ef031 Split out _get_event_from_row back into defer and _txn version 2015-05-20 13:00:57 +01:00
Erik Johnston
2bc60c55af Fix _get_backfill_events to return events in the correct order 2015-05-20 12:57:00 +01:00
Erik Johnston
9084cdd70f Ensure event_results is a set 2015-05-19 16:34:31 +01:00
Erik Johnston
aa729349dd Fix event_backwards_extrem insertion to ignore outliers 2015-05-19 15:27:00 +01:00
Mark Haines
99914ec9f8 Merge pull request #152 from matrix-org/notifier_performance 
Notifier performance
 2015-05-18 17:49:59 +01:00
Erik Johnston
165eb2dbe6 Comments and shuffle of functions 2015-05-18 15:18:41 +01:00
Erik Johnston
65878a2319 Remove unused metric 2015-05-18 14:06:30 +01:00
Erik Johnston
4d1b6f4ad1 Remove rejected events if we don't want rejected events 2015-05-18 14:03:46 +01:00
Mark Haines
0b0033c40b Merge branch 'develop' into notifier_performance 2015-05-18 13:50:01 +01:00
Erik Johnston
10f1bdb9a2 Move get_events functions to storage.events 2015-05-18 10:21:40 +01:00
Erik Johnston
c71176858b Newline, remove debug logging 2015-05-18 10:11:14 +01:00
Erik Johnston
c3b37abdfd PEP8 2015-05-15 16:59:58 +01:00
Erik Johnston
6c74fd62a0 Revert limiting of fetching, it didn't help perf. 2015-05-15 16:45:35 +01:00
Erik Johnston
9ff7f66a2b init j 2015-05-15 16:36:03 +01:00
Erik Johnston
70f272f71c Don't completely drain the list 2015-05-15 16:34:17 +01:00
Erik Johnston
8763dd80ef Don't fetch prev_content for current_state 2015-05-15 15:33:01 +01:00
Erik Johnston
acb12cc811 Make store.get_current_state fetch events asyncly 2015-05-15 15:20:05 +01:00
Erik Johnston
d62dee7eae Remove more debug logging 2015-05-15 15:06:37 +01:00
Erik Johnston
0f29cfabc3 Remove debug logging 2015-05-15 14:06:42 +01:00
Erik Johnston
e275a9c0d9 preserve log context 2015-05-15 11:54:51 +01:00
Erik Johnston
aa32bd38e4 Add a wait 2015-05-15 11:35:04 +01:00
Erik Johnston
372d4c6d7b Srsly. Don't use closures. Baaaaaad 2015-05-15 11:26:00 +01:00
Erik Johnston
575ec91d82 Correctly pass through params 2015-05-15 11:15:10 +01:00
Erik Johnston
de01438a57 Sort out error handling 2015-05-15 11:00:50 +01:00
Erik Johnston
a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston
1d566edb81 Remove race condition 2015-05-14 16:54:35 +01:00
Erik Johnston
ef3d8754f5 Call from right thread 2015-05-14 15:41:55 +01:00
Erik Johnston
142934084a Count and loop 2015-05-14 15:40:21 +01:00
Erik Johnston
96c5b9f87c Don't start up more fetch_events 2015-05-14 15:36:04 +01:00
Erik Johnston
7cd6a6f6cf Awful idea for speeding up fetching of events 2015-05-14 15:34:02 +01:00
Erik Johnston
7f4105a5c9 Turn off preemptive transactions 2015-05-14 14:51:06 +01:00
Erik Johnston
f4d58deba1 PEP8 2015-05-14 14:45:42 +01:00
Erik Johnston
386b7330d2 Move from _base to events 2015-05-14 14:45:22 +01:00
Erik Johnston
7d6a1dae31 Jump out early 2015-05-14 14:27:58 +01:00
Erik Johnston
656223fbd3 Actually, we probably want to run this in a transaction 2015-05-14 14:26:35 +01:00
Erik Johnston
2f7f8e1c2b Preemptively jump into a transaction if we ask for get_prev_content 2015-05-14 14:17:36 +01:00
Erik Johnston
e1e9f0c5b2 loop -> gatherResults 2015-05-14 13:58:49 +01:00
Erik Johnston
ab78a8926e Err, we probably want a bigger limit 2015-05-14 13:47:16 +01:00
Erik Johnston
f6f902d459 Move fetching of events into their own transactions 2015-05-14 13:45:48 +01:00
Erik Johnston
cdb3757942 Refactor _get_events 2015-05-14 13:31:55 +01:00
Erik Johnston
36ea26c5c0 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf 2015-05-14 12:01:38 +01:00
David Baker
7c549dd557 Add ID generator for push_rules_enable to #resolve SYN-378 2015-05-14 11:44:03 +01:00
Mark Haines
899d4675dd Merge branch 'notifier_unify' into notifier_performance 2015-05-14 11:36:44 +01:00
Erik Johnston
4f1d984e56 Add index on events 2015-05-13 17:22:26 +01:00
Erik Johnston
968b01a91a Actually use async method 2015-05-13 17:02:46 +01:00
Erik Johnston
4071f29653 Fetch events from events_id in their own transactions 2015-05-13 16:59:41 +01:00
Erik Johnston
a988361aea Typo 2015-05-13 15:44:15 +01:00
Erik Johnston
8888982db3 Don't insert None 2015-05-13 15:43:32 +01:00
Erik Johnston
cf706cc6ef Don't return None 2015-05-13 15:31:25 +01:00
Erik Johnston
5971d240d4 Limit batch size 2015-05-13 15:26:49 +01:00
Erik Johnston
ca4f458787 Fetch events in bulk 2015-05-13 15:13:42 +01:00
Erik Johnston
6edff11a88 Don't fetch redaction and rejection stuff for each event, so we can use index only scan 2015-05-13 14:39:05 +01:00
Mark Haines
63878c0379 Don't bother checking for updates if the stream token hasn't advanced for a user 2015-05-13 13:42:21 +01:00
Erik Johnston
02590c3e1d Temp turn off checking for rejections and redactions 2015-05-13 11:31:28 +01:00
Erik Johnston
619a21812b defer.gatherResults loop 2015-05-13 11:29:03 +01:00
Erik Johnston
fec4485e28 Batch fetching of events for state groups 2015-05-13 11:22:42 +01:00
Erik Johnston
409bcc76bd Load events for state group seperately 2015-05-13 11:13:31 +01:00
Erik Johnston
e122685978 You need to call contextmanager 2015-05-12 16:12:37 +01:00
Mark Haines
a6fb2aa2a5 Merge pull request #144 from matrix-org/erikj/logging_context 
Preserving logging contexts
 2015-05-12 15:23:50 +01:00
Mark Haines
1fce36b111 Merge pull request #149 from matrix-org/erikj/backfill 
Backfill support
 2015-05-12 15:20:32 +01:00
Erik Johnston
8b28209c60 Err, delete the right stuff 2015-05-12 15:02:53 +01:00
Erik Johnston
30c72d377e Newlines 2015-05-12 14:47:40 +01:00
Erik Johnston
e4eddf9b36 We do actually want to delete rows out of event_backward_extremities 2015-05-12 14:47:23 +01:00
Erik Johnston
c1779a79bc Fix up _handle_prev_events to not try to insert duplicate rows 2015-05-12 14:41:50 +01:00
Erik Johnston
74850d7f75 Do state groups persistence /after/ checking if we have already persisted the event 2015-05-12 14:14:58 +01:00
Erik Johnston
da6a7bbdde Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context 2015-05-12 13:10:42 +01:00
Mark Haines
5002056b16 SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes. 2015-05-12 11:20:40 +01:00
Erik Johnston
4df11b5039 Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards' 2015-05-12 10:28:10 +01:00
Erik Johnston
84e6b4001f Initial hack at wiring together pagination and backfill 2015-05-11 18:01:31 +01:00
Erik Johnston
17653a5dfe Move storage.stream._StreamToken to types.RoomStreamToken 2015-05-11 18:01:01 +01:00
Erik Johnston
b036596b75 Prefer to use _simple_*. 2015-05-11 11:24:01 +01:00
Erik Johnston
cd525c0f5a push_rules table expects an 'id' field 2015-05-11 11:24:01 +01:00
Erik Johnston
22d2f498fa Fix push rule bug: can't insert bool into small int column 2015-05-10 10:50:51 +01:00
Erik Johnston
476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
Mark Haines
1c7912751e Drop the old table not the new table 2015-05-08 16:04:32 +01:00
Mark Haines
9d36eb4eab Rename unique constraint 2015-05-08 16:01:55 +01:00
Mark Haines
b0f71db3ff Remove unsigned 2015-05-08 15:59:51 +01:00
Mark Haines
84e1cacea4 Bump schema version 2015-05-08 15:58:14 +01:00
Mark Haines
6538d445e8 Make the timestamps in server_keys_json bigints 2015-05-08 15:55:17 +01:00
Erik Johnston
0c4ac271ca Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371 2015-05-05 18:21:19 +01:00
Erik Johnston
0cf7e480b4 And use buffer(...) there as well 2015-05-05 18:20:01 +01:00
Erik Johnston
ed2584050f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany 2015-05-05 18:15:20 +01:00
Erik Johnston
977338a7af Use buffer(...) when inserting into bytea column 2015-05-05 18:12:53 +01:00
Mark Haines
31049c4d72 Merge pull request #139 from matrix-org/bugs/SYN-369 
Fix race with cache invalidation. SYN-369
 2015-05-05 17:46:13 +01:00
Mark Haines
deb0237166 Add some doc-string 2015-05-05 17:45:11 +01:00
Mark Haines
d18f37e026 Collect the invalidate callbacks on the transaction object rather than passing around a separate list 2015-05-05 17:32:21 +01:00
Erik Johnston
9951542393 Add a comment about the zip(*[zip(sorted(...),...)]) 2015-05-05 17:06:55 +01:00
Mark Haines
041b6cba61 SYN-369: Add comments to the sequence number logic in the cache 2015-05-05 16:32:44 +01:00
Mark Haines
63075118a5 Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access 2015-05-05 16:24:04 +01:00
Erik Johnston
531d7955fd Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely 2015-05-05 16:12:28 +01:00
Mark Haines
bfa4a7f8b0 Invalidate the room_member cache if the current state events updates 2015-05-05 15:43:49 +01:00
Mark Haines
d0fece8d3c Missing return for when the event was already persisted 2015-05-05 15:39:09 +01:00
Erik Johnston
bdcd7693c8 Fix indentation 2015-05-05 15:14:48 +01:00
Erik Johnston
43c2e8deae Add support for using executemany 2015-05-05 15:13:25 +01:00
Erik Johnston
1692dc019d Don't call 'encode_parameter' no-op 2015-05-05 15:00:30 +01:00
Mark Haines
a9aea68fd5 Invalidate the caches from the correct thread 2015-05-05 14:57:08 +01:00
Mark Haines
261d809a47 Sequence the modifications to the cache so that selects don't race with inserts 2015-05-05 14:13:50 +01:00
Erik Johnston
d9cc5de9e5 Correctly name transaction 2015-05-05 10:24:10 +01:00
Erik Johnston
b8940cd902 Remove some unused indexes 2015-05-01 16:14:25 +01:00
David Baker
eb9bd2d949 user_id now in user_threepids 2015-05-01 15:04:37 +01:00
David Baker
9eeb03c0dd Don't use self.execute: it's designed for fetching stuff 2015-05-01 14:21:25 +01:00
David Baker
7693f24792 No id field on user 2015-05-01 13:55:42 +01:00
David Baker
92b20713d7 More missed get_user_by_id API changes 2015-05-01 13:45:54 +01:00
Erik Johnston
4011cf1c42 Cache latest_event_ids_in_room 2015-05-01 13:06:26 +01:00
Erik Johnston
657298cebd Don't lock user_ips table for upsert. 2015-05-01 10:46:48 +01:00
Erik Johnston
fabb7acd45 Fix bug where we reconnected to the database on every query. 2015-05-01 10:24:24 +01:00
Erik Johnston
23c639ff32 Split a storage function in two so that we don't have to do extra work. 2015-05-01 10:17:19 +01:00
Erik Johnston
8be5284e91 Remove pointless join 2015-04-30 18:49:26 +01:00
Erik Johnston
503e4d3d52 Fix broken SQL 2015-04-30 18:44:47 +01:00
Erik Johnston
00718ae7a9 Need more yield 2015-04-30 18:43:39 +01:00
Erik Johnston
0465560c1a Add missing param 2015-04-30 18:42:44 +01:00
Erik Johnston
61d05daab1 More join conditions 2015-04-30 18:41:05 +01:00
Erik Johnston
6ead27ddda Add more conditions on JOINs to make postgres go a little faster. 2015-04-30 18:32:03 +01:00
Erik Johnston
f383d5a801 Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction 2015-04-30 17:12:52 +01:00
Erik Johnston
69d4063651 Add get_rooms_for_user cache 2015-04-30 16:47:51 +01:00
Erik Johnston
3c4c229788 Don't use sub queries, it makes postgres sad 2015-04-30 16:16:53 +01:00
Mark Haines
29400b45b9 SYN-367: Use upsert rather than insert_or_replace 2015-04-30 15:21:31 +01:00
Erik Johnston
2366d28780 Don't needlessly join on state_events 2015-04-30 14:02:06 +01:00
Erik Johnston
1d7702833d Make simple query rather than long one and then throw away half the results 2015-04-30 10:16:12 +01:00
Erik Johnston
b1ca784aca Correctly decode, for sqlite and postgres, rows from pushers table 2015-04-29 19:41:14 +01:00
Erik Johnston
4a9dc5b2f5 pushkey' are also bytes. 2015-04-29 19:27:02 +01:00
Erik Johnston
0ade2712d1 Typo 2015-04-29 19:17:25 +01:00
Erik Johnston
50f96f256f Also remove yield from within lock in the other generator 2015-04-29 19:17:00 +01:00
Erik Johnston
d2d61a8288 Fix deadlock in id_generators. No idea why this was an actual deadlock. 2015-04-29 19:15:23 +01:00
Erik Johnston
3864b3a8e6 Actually return rows 2015-04-29 18:07:36 +01:00
Erik Johnston
0618978238 Typo, args wrong way round 2015-04-29 18:04:35 +01:00
Erik Johnston
09177f4f2e Decode buffers in same thread 2015-04-29 18:03:42 +01:00
Erik Johnston
472be88674 We store pusher data as bytes 2015-04-29 17:43:46 +01:00
David Baker
12d381bd5d Decode the data json in the storage layer (was moved but this part was missed) 2015-04-29 17:13:51 +01:00
David Baker
f8c30faf25 Oops, update the contraint too 2015-04-29 16:58:42 +01:00
David Baker
61cd5d9045 Be more postgressive 2015-04-29 16:57:14 +01:00
David Baker
fb95035a65 Be postgressive 2015-04-29 16:53:41 +01:00
David Baker
4669def000 Oops, forgot the schema delta file 2015-04-29 16:50:16 +01:00
Erik Johnston
0337eaf321 txn.execute doesn't return cursors 2015-04-29 16:43:39 +01:00
Erik Johnston
884fb88e28 txn.execute doesn't return cursors 2015-04-29 16:35:20 +01:00
Erik Johnston
d76c058eea Fix invalid SQL to work in postgres land 2015-04-29 16:30:25 +01:00
Erik Johnston
c8ed9bd278 pushers table requires an access_token 2015-04-29 15:33:27 +01:00
Erik Johnston
970a9b9d2b We can't use REPLACE when upgrading databases now we have postgres 2015-04-29 13:55:44 +01:00
Erik Johnston
64991b0c8b Merge pull request #129 from matrix-org/key_distribution 
Key distribution v2
 2015-04-29 13:34:38 +01:00
Mark Haines
e26a3d8d9e bump database schema version 2015-04-29 13:32:32 +01:00
Mark Haines
a9549fdce3 Use bytea rather than BLOB 2015-04-29 13:16:09 +01:00
Mark Haines
4ad8b45155 Merge branch 'develop' into key_distribution 
Conflicts:
	synapse/config/homeserver.py
 2015-04-29 13:15:14 +01:00
Erik Johnston
cd0864121b Make postgres database error slightly more helpful 2015-04-29 12:12:25 +01:00
Erik Johnston
4932a7e2d9 Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__ 2015-04-29 12:12:25 +01:00
Erik Johnston
a08bf11138 Appease PEP8 2015-04-29 11:44:48 +01:00
Erik Johnston
204132a998 Check that postgres database has correct charset set 2015-04-29 11:42:28 +01:00
Erik Johnston
327ca883ec Merge branch 'develop' of github.com:matrix-org/synapse into postgres 2015-04-28 13:39:42 +01:00
Erik Johnston
18f8247701 Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit 2015-04-28 12:41:33 +01:00
Mark Haines
9182f87664 Merge pull request #126 from matrix-org/csauth 
Client / Server Auth Refactor
 2015-04-28 11:00:27 +01:00
Mark Haines
55e1bc8920 And don't bump the schema version unnecessarily 2015-04-28 10:54:15 +01:00
Mark Haines
b96c133034 Add server_keys.sql to the current delta rather than creating a new delta 2015-04-28 10:50:00 +01:00
Erik Johnston
8558e1ec73 Make get_max_token into inlineCallbacks so that the lock works. 2015-04-27 15:19:44 +01:00
Erik Johnston
6f8e2d517e Merge branch 'develop' of github.com:matrix-org/synapse into postgres 2015-04-27 14:41:40 +01:00
Erik Johnston
2732be83d9 Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible. 2015-04-27 13:22:30 +01:00
Erik Johnston
e4c4664d73 Handle the fact that postgres databases can be restarted from under us 2015-04-27 12:40:49 +01:00
David Baker
8a5be236e0 pep8 2015-04-27 11:49:18 +01:00
David Baker
b02e1006b9 Run database check before daemonizing, at the cost of database hygiene. 2015-04-27 11:46:00 +01:00
David Baker
f8152f2708 rename db method to be more informative 2015-04-27 10:16:26 +01:00
David Baker
2f475bd5d5 pep8 2015-04-24 18:15:07 +01:00
David Baker
a7b51f4539 Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266 2015-04-24 18:11:21 +01:00
Mark Haines
eede182df7 Merge branch 'develop' into key_distribution 2015-04-24 10:35:49 +01:00
Mark Haines
4e2f8b8722 Copyright notices 2015-04-24 10:35:29 +01:00
Mark Haines
f30d47c876 Implement remote key lookup api 2015-04-22 14:21:08 +01:00
David Baker
91c8f828e1 pep8 2015-04-17 19:56:04 +01:00
David Baker
8db6832db8 Password reset, finally. 2015-04-17 19:53:47 +01:00
David Baker
117f35ac4a Add endpoint to get threepids from server 2015-04-17 17:20:18 +01:00