Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
12,019 Commits 4 Branches 615 Tags 63 MiB
546aee7e52
Commit Graph

358 Commits

Author SHA1 Message Date
Erik Johnston
8ad8490cff Fix typo 2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88 Retry event persistence on IntegrityError 
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.

This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
 2016-08-04 15:02:15 +01:00
Erik Johnston
80ad710217 Remove other bit of deduplication 2016-08-03 13:25:59 +01:00
Erik Johnston
a8a32d2714 Ensure we only persist an event once at a time 2016-08-03 11:23:39 +01:00
Mark Haines
a6f06ce3e2 Fix how push_actions are redacted. 2016-07-26 11:05:39 +01:00
Mark Haines
efeb6176c1 Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically 2016-07-26 10:49:52 +01:00
Mark Haines
1b3c3e6d68 Only update the events and event_json tables for rejected events 2016-07-25 18:44:30 +01:00
Mark Haines
8f7f4cb92b Don't add the events to forward extremities if the event is rejected 2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874 Don't add rejections to the state_group, persist all rejections 2016-07-25 16:12:16 +01:00
Erik Johnston
1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2 
Feature: Add filter to /messages. Add 'contains_url' to filter.
 2016-07-20 10:40:48 +01:00
Erik Johnston
f52565de50 Fix /purge_history bug 
This was caused by trying to insert duplicate backward extremeties
 2016-07-15 14:23:15 +01:00
Erik Johnston
eec9609e96 event_backwards_extremeties may not be empty 2016-07-15 10:22:09 +01:00
Erik Johnston
b64aa6d687 Add sender and contains_url field to events table 2016-07-14 15:35:43 +01:00
Erik Johnston
b92e7955be Comment 2016-07-07 11:42:15 +01:00
Erik Johnston
c98e1479bd Return 400 rather than 500 2016-07-07 11:41:07 +01:00
Erik Johnston
67f2c901ea Add rest servlet. Fix SQL. 2016-07-06 15:56:59 +01:00
Erik Johnston
2d21d43c34 Add purge_history API 2016-07-05 10:28:51 +01:00
Erik Johnston
a67bf0b074 Add storage function to purge history for a room 2016-07-04 16:02:50 +01:00
David Baker
870c45913e Use similar naming we use in email notifs for push 
Fixes https://github.com/vector-im/vector-web/issues/1654
 2016-06-24 11:41:11 +01:00
Erik Johnston
e4134c5e13 Merge pull request #841 from matrix-org/erikj/event_counter 
Add metric counter for number of persisted events
 2016-06-06 14:17:40 +01:00
Erik Johnston
7aa778fba9 Add metric counter for number of persisted events 2016-06-06 11:58:09 +01:00
Erik Johnston
70aee0717c Add events to cache when we persist them 2016-06-06 11:34:53 +01:00
Erik Johnston
cffe46408f Don't rely on options when inserting event into cache 2016-06-03 18:25:21 +01:00
Erik Johnston
10ea3f46ba Change the way we cache events 2016-06-03 17:57:50 +01:00
Erik Johnston
05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Erik Johnston
c8285564a3 Use state to calculate get_users_in_room 2016-06-01 15:25:25 +01:00
David Baker
997db04648 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-05-10 14:40:19 +02:00
Erik Johnston
fcb2c3f0db Remove unused import 2016-05-06 15:47:40 +01:00
Erik Johnston
fd85b167ec Pull loop one level up 2016-05-06 15:38:42 +01:00
Erik Johnston
b6e0be701e Queue events for persistence 2016-05-06 14:31:44 +01:00
David Baker
acded821c4 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-04-29 10:05:20 +01:00
Mark Haines
871357d539 Check that somethign has happend before running the selects 2016-04-27 11:54:13 +01:00
David Baker
07d765209d First bits of emailpusher 
Mostly logic of when to send an email
 2016-04-19 14:24:36 +01:00
Erik Johnston
762ada1e07 Add back backfilled parameter that was removed 2016-04-12 11:58:04 +01:00
Erik Johnston
cccf86dd05 Check if we've already backfilled events 2016-04-12 11:19:32 +01:00
David Baker
9c99ab4572 Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions 2016-04-07 16:35:22 +01:00
David Baker
0fd1cd2400 pep8 2016-04-06 16:50:47 +01:00
David Baker
7e2c89a37f Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do: 
* Make badges work again
 * Remove old, unused code
 2016-04-06 15:42:15 +01:00
Mark Haines
75fb9ac1be Add a slaved events store class 
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
 2016-04-06 14:18:35 +01:00
Erik Johnston
3d76b7cb2b Store invites in a separate table. 2016-04-04 16:30:15 +01:00
Mark Haines
f2b916534b Merge pull request #684 from matrix-org/markjh/backfill_id_gen 
Use a stream id generator for backfilled ids
 2016-04-01 15:13:14 +01:00
Mark Haines
e36bfbab38 Use a stream id generator for backfilled ids 2016-04-01 13:29:05 +01:00
Mark Haines
7753fc6570 Fix the invalidation of the names and aliases cache 2016-04-01 10:34:51 +01:00
Mark Haines
5d06929169 Move the check for backfilled outside the for loop 2016-03-31 15:09:09 +01:00
Mark Haines
76503f95ed Remove the is_new_state argument to persist event. 
Move the checks for whether an event is new state inside persist
event itself.

This was harder than expected because there wasn't enough information
passed to persist event to correctly handle invites from remote servers
for new rooms.
 2016-03-31 15:00:42 +01:00
Mark Haines
62e395f0e3 Merge pull request #676 from matrix-org/markjh/replicate_stateIII 
Add replication streams for ex outliers and current state resets
 2016-03-31 11:20:57 +01:00
Mark Haines
2ec5426035 Use a namedtuple rather than tuple unpacking 2016-03-31 10:33:02 +01:00
Mark Haines
1fbb094c6f Add replication streams for ex outliers and current state resets 2016-03-30 17:19:56 +01:00
Mark Haines
8b8052909f return the state_group for backfill 2016-03-30 16:20:07 +01:00
Mark Haines
61407986b4 Add a entry to current_state_resets table when the current state is reset 2016-03-30 16:18:46 +01:00
Mark Haines
31a9eceda5 Add a replication stream for state groups 2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6 Use a stream id generator to assign state group ids 2016-03-30 12:55:02 +01:00
Erik Johnston
e14baa7a3b Merge pull request #665 from matrix-org/erikj/dont_cache_events 
Only cache events in the event cache
 2016-03-23 13:19:12 +00:00
Erik Johnston
8b0dfc9fc4 Don't cache events in get_current_state_for_key 2016-03-23 11:42:17 +00:00
Erik Johnston
99f929f36b Make StateHandler._state_cache only store event_ids. 2016-03-22 16:06:04 +00:00
Mark Haines
5244c0b48e Remove unused backfilled parameter from persist_event 2016-03-21 18:06:08 +00:00
Erik Johnston
67ed8065db Dedupe requested event list in _get_events 2016-03-18 14:31:31 +00:00
Mark Haines
7076082ae6 Fix relative imports so they work in both py3 and py27 2016-03-08 11:45:50 +00:00
Mark Haines
a612ce6659 Merge pull request #489 from matrix-org/markjh/replication 
Add a /replication API for extracting the updates that happened on synapse.
 2016-03-01 15:08:24 +00:00
Mark Haines
60a0f81c7a Add a /replication API for extracting the updates that happened on 
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
 2016-03-01 14:49:41 +00:00
Mark Haines
54172924c8 Load the current id in the IdGenerator constructor 
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.

It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
 2016-03-01 14:32:56 +00:00
Erik Johnston
42109a62a4 Remove unused param from get_max_token 2016-02-18 16:37:28 +00:00
Erik Johnston
0214745239 Rename functions 2016-02-10 11:09:56 +00:00
Erik Johnston
7b0d846407 Atomically persit push actions when we persist the event 2016-02-09 16:19:15 +00:00
Erik Johnston
2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston
00cb3eb24b Cache tags and account data 2016-01-28 16:37:41 +00:00
Erik Johnston
e1941442d4 Invalidate caches properly. Remove unused arg 2016-01-28 15:02:41 +00:00
Erik Johnston
e7febf4fbb PEP8 2016-01-27 17:33:27 +00:00
Erik Johnston
aca3193efb Use the same path for incremental with gap or without gap 2016-01-27 17:33:27 +00:00
Erik Johnston
b97f6626b6 Add cache to room stream 2016-01-27 17:33:26 +00:00
Erik Johnston
87f9477b10 Add a Homeserver.setup method. 
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
 2016-01-26 15:51:06 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
5880de186b Log when we skip daily messages 2016-01-06 13:52:16 +00:00
Mark Haines
6a5ff5f223 Track the time spent in the database per request. 
and track the number of transactions that request started.
 2015-12-07 17:56:11 +00:00
Erik Johnston
4dcaa42b6d Allow paginating search ordered by recents 2015-11-30 17:45:31 +00:00
Richard van der Hoff
e4d622aaaf Implementation of state rollback in /sync 
Implementation of SPEC-254: roll back the state dictionary to how it looked at
the start of the timeline.

Merged PR https://github.com/matrix-org/synapse/pull/373
 2015-11-13 10:58:56 +00:00
Daniel Wagner-Hall
cf437900e0 Return world_readable and guest_can_join in /publicRooms 2015-11-10 17:10:27 +00:00
Daniel Wagner-Hall
ca2f90742d Open up /events to anonymous users for room events only 
Squash-merge of PR #345 from daniel/anonymousevents
 2015-11-05 14:32:26 +00:00
Erik Johnston
61561b9df7 Keep FTS indexes up to date. Only search through rooms currently joined 2015-10-12 10:49:53 +01:00
Mark Haines
92d8d724c5 Remove unused functions from synapse/storage/events.py 2015-09-23 10:33:06 +01:00
Daniel Wagner-Hall
eb011cd99b Add docstring 2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083 Implement configurable stats reporting 
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
 2015-09-22 12:57:40 +01:00
Erik Johnston
2048388cfd Merge pull request #281 from matrix-org/erikj/update_extremeties 
When updating a stored event from outlier to non-outlier, remember to update the extremeties
 2015-09-15 16:57:25 +01:00
Erik Johnston
3b05b67c89 When updating a stored event from outlier to non-outlier, remember to update the extremeties 2015-09-15 16:34:42 +01:00
Mark Haines
01fc3943f1 Fix indent 2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3 Remove syutil dependency in favour of smaller single-purpose libraries 2015-08-24 16:17:38 +01:00
Erik Johnston
1f7642efa9 Fix bug where we didn't correctly serialize the redacted_because key over federation 2015-08-21 09:36:07 +01:00
Erik Johnston
20addfa358 Change Cache to not use *args in its interface 2015-08-07 18:32:47 +01:00
Erik Johnston
17bb9a7eb9 Remove commented out code 2015-07-10 14:07:57 +01:00
Erik Johnston
5130d80d79 Add bulk insert events API 2015-06-25 17:29:34 +01:00
Erik Johnston
6e17463228 Don't explode if we don't have the event 2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf Add config option to turn off freezing events. Use new encode_json api and ujson.loads 2015-05-29 12:17:33 +01:00
Erik Johnston
4429e720ae Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-22 10:33:00 +01:00
Mark Haines
53447e9cd3 Add caches for things requested by the pushers 2015-05-21 16:41:39 +01:00
Mark Haines
d61ce3f670 Add a cache for get_current_state with state_key 2015-05-21 16:41:39 +01:00
Erik Johnston
a551c5dad7 Merge pull request #155 from matrix-org/erikj/perf 
Bulk and batch retrieval of events.
 2015-05-21 14:54:40 +01:00
Erik Johnston
27e4b45c06 s/for events/for requests for events/ 2015-05-21 14:52:23 +01:00
Erik Johnston
ac5f2bf9db s/for events/for requests for events/ 2015-05-21 14:50:57 +01:00
Erik Johnston
80a167b1f0 Add comments 2015-05-21 11:19:04 +01:00
Erik Johnston
9118a92862 Split up _get_events into defer and txn versions 2015-05-20 13:27:16 +01:00
Erik Johnston
ab45e12d31 Make not return a deferred _get_event_from_row_txn 2015-05-20 13:07:19 +01:00
Erik Johnston
f407cbd2f1 PEP8 2015-05-20 13:02:01 +01:00
Erik Johnston
227f8ef031 Split out _get_event_from_row back into defer and _txn version 2015-05-20 13:00:57 +01:00
Mark Haines
99914ec9f8 Merge pull request #152 from matrix-org/notifier_performance 
Notifier performance
 2015-05-18 17:49:59 +01:00
Erik Johnston
165eb2dbe6 Comments and shuffle of functions 2015-05-18 15:18:41 +01:00
Erik Johnston
4d1b6f4ad1 Remove rejected events if we don't want rejected events 2015-05-18 14:03:46 +01:00
Erik Johnston
10f1bdb9a2 Move get_events functions to storage.events 2015-05-18 10:21:40 +01:00
Erik Johnston
c3b37abdfd PEP8 2015-05-15 16:59:58 +01:00
Erik Johnston
6c74fd62a0 Revert limiting of fetching, it didn't help perf. 2015-05-15 16:45:35 +01:00
Erik Johnston
9ff7f66a2b init j 2015-05-15 16:36:03 +01:00
Erik Johnston
70f272f71c Don't completely drain the list 2015-05-15 16:34:17 +01:00
Erik Johnston
acb12cc811 Make store.get_current_state fetch events asyncly 2015-05-15 15:20:05 +01:00
Erik Johnston
d62dee7eae Remove more debug logging 2015-05-15 15:06:37 +01:00
Erik Johnston
0f29cfabc3 Remove debug logging 2015-05-15 14:06:42 +01:00
Erik Johnston
e275a9c0d9 preserve log context 2015-05-15 11:54:51 +01:00
Erik Johnston
aa32bd38e4 Add a wait 2015-05-15 11:35:04 +01:00
Erik Johnston
372d4c6d7b Srsly. Don't use closures. Baaaaaad 2015-05-15 11:26:00 +01:00
Erik Johnston
575ec91d82 Correctly pass through params 2015-05-15 11:15:10 +01:00
Erik Johnston
de01438a57 Sort out error handling 2015-05-15 11:00:50 +01:00
Erik Johnston
a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston
1d566edb81 Remove race condition 2015-05-14 16:54:35 +01:00
Erik Johnston
ef3d8754f5 Call from right thread 2015-05-14 15:41:55 +01:00
Erik Johnston
142934084a Count and loop 2015-05-14 15:40:21 +01:00
Erik Johnston
96c5b9f87c Don't start up more fetch_events 2015-05-14 15:36:04 +01:00
Erik Johnston
7cd6a6f6cf Awful idea for speeding up fetching of events 2015-05-14 15:34:02 +01:00
Erik Johnston
7f4105a5c9 Turn off preemptive transactions 2015-05-14 14:51:06 +01:00
Erik Johnston
f4d58deba1 PEP8 2015-05-14 14:45:42 +01:00
Erik Johnston
386b7330d2 Move from _base to events 2015-05-14 14:45:22 +01:00
Mark Haines
899d4675dd Merge branch 'notifier_unify' into notifier_performance 2015-05-14 11:36:44 +01:00
Mark Haines
63878c0379 Don't bother checking for updates if the stream token hasn't advanced for a user 2015-05-13 13:42:21 +01:00
Erik Johnston
e122685978 You need to call contextmanager 2015-05-12 16:12:37 +01:00
Mark Haines
1fce36b111 Merge pull request #149 from matrix-org/erikj/backfill 
Backfill support
 2015-05-12 15:20:32 +01:00
Erik Johnston
74850d7f75 Do state groups persistence /after/ checking if we have already persisted the event 2015-05-12 14:14:58 +01:00
Mark Haines
5002056b16 SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes. 2015-05-12 11:20:40 +01:00
Erik Johnston
0c4ac271ca Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371 2015-05-05 18:21:19 +01:00
Erik Johnston
ed2584050f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany 2015-05-05 18:15:20 +01:00
Mark Haines
d18f37e026 Collect the invalidate callbacks on the transaction object rather than passing around a separate list 2015-05-05 17:32:21 +01:00
Erik Johnston
531d7955fd Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely 2015-05-05 16:12:28 +01:00
Mark Haines
bfa4a7f8b0 Invalidate the room_member cache if the current state events updates 2015-05-05 15:43:49 +01:00
Mark Haines
d0fece8d3c Missing return for when the event was already persisted 2015-05-05 15:39:09 +01:00
Erik Johnston
43c2e8deae Add support for using executemany 2015-05-05 15:13:25 +01:00
Mark Haines
a9aea68fd5 Invalidate the caches from the correct thread 2015-05-05 14:57:08 +01:00
Erik Johnston
2732be83d9 Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible. 2015-04-27 13:22:30 +01:00
Erik Johnston
b8092fbc82 Go back to storing JSON in TEXT 2015-04-16 11:17:52 +01:00
Erik Johnston
ded4128965 Use True for True rather than 1 2015-04-15 10:24:24 +01:00
Erik Johnston
58d8339966 Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS. 2015-04-14 13:53:20 +01:00
Erik Johnston
8bf285e082 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-09 13:13:26 +01:00
Erik Johnston
8ad0f4912e
Stream ordering and out of order insertions. 
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
 2015-04-09 11:41:36 +01:00
Erik Johnston
0af5f5efaf Don't use multiple UNIQUE constraints; it will cause deadlocks 2015-04-07 12:08:35 +01:00
Erik Johnston
304111afd0 Don't use AUTOINCREMENT, use an in memory version 2015-04-07 12:05:36 +01:00
Paul "LeoNerd" Evans
033a517feb Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file 2015-03-25 19:06:05 +00:00
Erik Johnston
0e8f5095c7 Fix unicode database support 2015-03-25 17:15:20 +00:00
Erik Johnston
f488293d96 Don't reinsert into event_edges 2015-03-24 16:20:26 +00:00
Erik Johnston
f6583796fe Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-03-20 16:31:48 +00:00
Erik Johnston
9517f4da4d Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement 2015-03-20 16:02:47 +00:00
Erik Johnston
87db64b839 Rearrange storage modules 2015-03-20 14:11:38 +00:00
Erik Johnston
cb8162d3d1 Rearrange storage modules 2015-03-20 13:52:56 +00:00