Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
10,936 Commits 4 Branches 615 Tags 63 MiB
284e36ad04
Commit Graph

308 Commits

Author SHA1 Message Date
Luke Barnard
73880268ef Refactor event ordering check to events store 2017-04-11 17:34:09 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall 
This prevents unnecessary construction of lists
 2017-03-23 17:53:49 +00:00
Erik Johnston
37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed 
Don't send the full event json over replication
 2017-03-21 13:11:15 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae Avoid resetting state on rejected events 
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.

This should fix https://github.com/matrix-org/synapse/issues/1935.
 2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5 Refactoring and cleanups 
A few non-functional changes:

* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
  readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
  mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
  mind-bending conditional.
 2017-03-17 15:06:07 +00:00
Erik Johnston
bb256ac96f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed 2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e Fix current_state_events table to not lie 
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
 2017-03-14 10:57:43 +00:00
Erik Johnston
8ffbe43ba1 Get current state by using current_state_events table 2017-03-10 17:39:35 +00:00
Erik Johnston
a41dce8f8a Remove needless check 2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776 It should be all 2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709 Don't fetch current state in common case 
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
 2017-02-27 18:33:41 +00:00
Erik Johnston
7723b4caa4 Ignore new rejected events when working out forward extremeties. 2017-02-08 14:48:06 +00:00
Erik Johnston
ebfaff84c9 Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events 
Add a small cache get_all_new_events
 2017-02-01 13:22:02 +00:00
Erik Johnston
368c88c487 Add a small cache get_all_new_events 2017-02-01 10:50:44 +00:00
Erik Johnston
458b6f4733 Only invalidate membership caches based on the cache stream 
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
 2017-01-31 16:09:03 +00:00
Erik Johnston
4b3403ca9b Stream cache invalidations for room membership storage functions 2017-01-30 17:28:22 +00:00
Erik Johnston
fdf2a31a51 Typo 2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092 Refactor to calculate state delta outside transaction 2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485 Comments 2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5 Spelling 2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67 Update all call sites after rename 2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9 Comments 2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e Calculate the forward extremeties once 2017-01-20 14:28:53 +00:00
Erik Johnston
09eb08f910 Derive current_state_events from state groups 2017-01-20 11:52:51 +00:00
Erik Johnston
4ec1cf49e2 Lower loading events log to DEBUG 2017-01-17 17:28:32 +00:00
Mark Haines
dd3df11c55 More logging for the linearizer and for get_events 2017-01-05 12:32:47 +00:00
Erik Johnston
f9d156d270 New Flake8 fixes 2016-11-15 11:22:29 +00:00
Erik Johnston
816988baaa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth 2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7 Drop some unused indices 2016-10-17 11:08:19 +01:00
Erik Johnston
9040c9ffa1 Fix background reindex of origin_server_ts 
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
 2016-09-27 11:23:49 +01:00
Erik Johnston
0b32bb20bb Index contains_url for file search queries 2016-09-12 16:57:05 +01:00
Erik Johnston
70332a12dd Take value in a better way 2016-09-05 14:57:14 +01:00
Erik Johnston
485d999c8a Correctly delete old state groups in purge history API 2016-09-05 14:49:08 +01:00
Erik Johnston
598317927c Limit the length of state chains 2016-09-02 10:41:38 +01:00
Erik Johnston
1bb8ec296d Generate state group ids in state layer 2016-08-31 10:09:46 +01:00
Erik Johnston
5dc2a702cf Make _state_groups_id_gen a normal IdGenerator 2016-08-30 16:55:11 +01:00
Erik Johnston
bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston
9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Erik Johnston
37adde32dc Move defer.returnValue out of Measure 2016-08-19 18:23:44 +01:00
Erik Johnston
47dd8f02a1 Measure _get_event_from_row 2016-08-19 18:23:44 +01:00
Benjamin Saunders
8a57cc3123 Add missing database corruption recovery case 
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
 2016-08-14 11:50:22 -07:00
Erik Johnston
a8b946decb Raise 404 when couldn't find event 2016-08-05 15:31:02 +01:00
Erik Johnston
f0fa66f495 Delete more tables 2016-08-05 10:40:08 +01:00
Benjamin Saunders
a2b7102eea Tweak integrity error recovery to work as intended 2016-08-04 20:38:08 -07:00
Erik Johnston
8ad8490cff Fix typo 2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88 Retry event persistence on IntegrityError 
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.

This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
 2016-08-04 15:02:15 +01:00
Erik Johnston
80ad710217 Remove other bit of deduplication 2016-08-03 13:25:59 +01:00
Erik Johnston
a8a32d2714 Ensure we only persist an event once at a time 2016-08-03 11:23:39 +01:00
Mark Haines
a6f06ce3e2 Fix how push_actions are redacted. 2016-07-26 11:05:39 +01:00
Mark Haines
efeb6176c1 Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically 2016-07-26 10:49:52 +01:00
Mark Haines
1b3c3e6d68 Only update the events and event_json tables for rejected events 2016-07-25 18:44:30 +01:00
Mark Haines
8f7f4cb92b Don't add the events to forward extremities if the event is rejected 2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874 Don't add rejections to the state_group, persist all rejections 2016-07-25 16:12:16 +01:00
Erik Johnston
1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2 
Feature: Add filter to /messages. Add 'contains_url' to filter.
 2016-07-20 10:40:48 +01:00
Erik Johnston
f52565de50 Fix /purge_history bug 
This was caused by trying to insert duplicate backward extremeties
 2016-07-15 14:23:15 +01:00
Erik Johnston
eec9609e96 event_backwards_extremeties may not be empty 2016-07-15 10:22:09 +01:00
Erik Johnston
b64aa6d687 Add sender and contains_url field to events table 2016-07-14 15:35:43 +01:00
Erik Johnston
b92e7955be Comment 2016-07-07 11:42:15 +01:00
Erik Johnston
c98e1479bd Return 400 rather than 500 2016-07-07 11:41:07 +01:00
Erik Johnston
67f2c901ea Add rest servlet. Fix SQL. 2016-07-06 15:56:59 +01:00
Erik Johnston
2d21d43c34 Add purge_history API 2016-07-05 10:28:51 +01:00
Erik Johnston
a67bf0b074 Add storage function to purge history for a room 2016-07-04 16:02:50 +01:00
David Baker
870c45913e Use similar naming we use in email notifs for push 
Fixes https://github.com/vector-im/vector-web/issues/1654
 2016-06-24 11:41:11 +01:00
Erik Johnston
e4134c5e13 Merge pull request #841 from matrix-org/erikj/event_counter 
Add metric counter for number of persisted events
 2016-06-06 14:17:40 +01:00
Erik Johnston
7aa778fba9 Add metric counter for number of persisted events 2016-06-06 11:58:09 +01:00
Erik Johnston
70aee0717c Add events to cache when we persist them 2016-06-06 11:34:53 +01:00
Erik Johnston
cffe46408f Don't rely on options when inserting event into cache 2016-06-03 18:25:21 +01:00
Erik Johnston
10ea3f46ba Change the way we cache events 2016-06-03 17:57:50 +01:00
Erik Johnston
05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Erik Johnston
c8285564a3 Use state to calculate get_users_in_room 2016-06-01 15:25:25 +01:00
David Baker
997db04648 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-05-10 14:40:19 +02:00
Erik Johnston
fcb2c3f0db Remove unused import 2016-05-06 15:47:40 +01:00
Erik Johnston
fd85b167ec Pull loop one level up 2016-05-06 15:38:42 +01:00
Erik Johnston
b6e0be701e Queue events for persistence 2016-05-06 14:31:44 +01:00
David Baker
acded821c4 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-04-29 10:05:20 +01:00
Mark Haines
871357d539 Check that somethign has happend before running the selects 2016-04-27 11:54:13 +01:00
David Baker
07d765209d First bits of emailpusher 
Mostly logic of when to send an email
 2016-04-19 14:24:36 +01:00
Erik Johnston
762ada1e07 Add back backfilled parameter that was removed 2016-04-12 11:58:04 +01:00
Erik Johnston
cccf86dd05 Check if we've already backfilled events 2016-04-12 11:19:32 +01:00
David Baker
9c99ab4572 Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions 2016-04-07 16:35:22 +01:00
David Baker
0fd1cd2400 pep8 2016-04-06 16:50:47 +01:00
David Baker
7e2c89a37f Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do: 
* Make badges work again
 * Remove old, unused code
 2016-04-06 15:42:15 +01:00
Mark Haines
75fb9ac1be Add a slaved events store class 
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
 2016-04-06 14:18:35 +01:00
Erik Johnston
3d76b7cb2b Store invites in a separate table. 2016-04-04 16:30:15 +01:00
Mark Haines
f2b916534b Merge pull request #684 from matrix-org/markjh/backfill_id_gen 
Use a stream id generator for backfilled ids
 2016-04-01 15:13:14 +01:00
Mark Haines
e36bfbab38 Use a stream id generator for backfilled ids 2016-04-01 13:29:05 +01:00
Mark Haines
7753fc6570 Fix the invalidation of the names and aliases cache 2016-04-01 10:34:51 +01:00
Mark Haines
5d06929169 Move the check for backfilled outside the for loop 2016-03-31 15:09:09 +01:00
Mark Haines
76503f95ed Remove the is_new_state argument to persist event. 
Move the checks for whether an event is new state inside persist
event itself.

This was harder than expected because there wasn't enough information
passed to persist event to correctly handle invites from remote servers
for new rooms.
 2016-03-31 15:00:42 +01:00
Mark Haines
62e395f0e3 Merge pull request #676 from matrix-org/markjh/replicate_stateIII 
Add replication streams for ex outliers and current state resets
 2016-03-31 11:20:57 +01:00
Mark Haines
2ec5426035 Use a namedtuple rather than tuple unpacking 2016-03-31 10:33:02 +01:00
Mark Haines
1fbb094c6f Add replication streams for ex outliers and current state resets 2016-03-30 17:19:56 +01:00
Mark Haines
8b8052909f return the state_group for backfill 2016-03-30 16:20:07 +01:00
Mark Haines
61407986b4 Add a entry to current_state_resets table when the current state is reset 2016-03-30 16:18:46 +01:00
Mark Haines
31a9eceda5 Add a replication stream for state groups 2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6 Use a stream id generator to assign state group ids 2016-03-30 12:55:02 +01:00
Erik Johnston
e14baa7a3b Merge pull request #665 from matrix-org/erikj/dont_cache_events 
Only cache events in the event cache
 2016-03-23 13:19:12 +00:00
Erik Johnston
8b0dfc9fc4 Don't cache events in get_current_state_for_key 2016-03-23 11:42:17 +00:00
Erik Johnston
99f929f36b Make StateHandler._state_cache only store event_ids. 2016-03-22 16:06:04 +00:00
Mark Haines
5244c0b48e Remove unused backfilled parameter from persist_event 2016-03-21 18:06:08 +00:00
Erik Johnston
67ed8065db Dedupe requested event list in _get_events 2016-03-18 14:31:31 +00:00
Mark Haines
7076082ae6 Fix relative imports so they work in both py3 and py27 2016-03-08 11:45:50 +00:00
Mark Haines
a612ce6659 Merge pull request #489 from matrix-org/markjh/replication 
Add a /replication API for extracting the updates that happened on synapse.
 2016-03-01 15:08:24 +00:00
Mark Haines
60a0f81c7a Add a /replication API for extracting the updates that happened on 
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
 2016-03-01 14:49:41 +00:00
Mark Haines
54172924c8 Load the current id in the IdGenerator constructor 
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.

It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
 2016-03-01 14:32:56 +00:00
Erik Johnston
42109a62a4 Remove unused param from get_max_token 2016-02-18 16:37:28 +00:00
Erik Johnston
0214745239 Rename functions 2016-02-10 11:09:56 +00:00
Erik Johnston
7b0d846407 Atomically persit push actions when we persist the event 2016-02-09 16:19:15 +00:00
Erik Johnston
2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston
00cb3eb24b Cache tags and account data 2016-01-28 16:37:41 +00:00
Erik Johnston
e1941442d4 Invalidate caches properly. Remove unused arg 2016-01-28 15:02:41 +00:00
Erik Johnston
e7febf4fbb PEP8 2016-01-27 17:33:27 +00:00
Erik Johnston
aca3193efb Use the same path for incremental with gap or without gap 2016-01-27 17:33:27 +00:00
Erik Johnston
b97f6626b6 Add cache to room stream 2016-01-27 17:33:26 +00:00
Erik Johnston
87f9477b10 Add a Homeserver.setup method. 
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
 2016-01-26 15:51:06 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
5880de186b Log when we skip daily messages 2016-01-06 13:52:16 +00:00
Mark Haines
6a5ff5f223 Track the time spent in the database per request. 
and track the number of transactions that request started.
 2015-12-07 17:56:11 +00:00
Erik Johnston
4dcaa42b6d Allow paginating search ordered by recents 2015-11-30 17:45:31 +00:00
Richard van der Hoff
e4d622aaaf Implementation of state rollback in /sync 
Implementation of SPEC-254: roll back the state dictionary to how it looked at
the start of the timeline.

Merged PR https://github.com/matrix-org/synapse/pull/373
 2015-11-13 10:58:56 +00:00
Daniel Wagner-Hall
cf437900e0 Return world_readable and guest_can_join in /publicRooms 2015-11-10 17:10:27 +00:00
Daniel Wagner-Hall
ca2f90742d Open up /events to anonymous users for room events only 
Squash-merge of PR #345 from daniel/anonymousevents
 2015-11-05 14:32:26 +00:00
Erik Johnston
61561b9df7 Keep FTS indexes up to date. Only search through rooms currently joined 2015-10-12 10:49:53 +01:00
Mark Haines
92d8d724c5 Remove unused functions from synapse/storage/events.py 2015-09-23 10:33:06 +01:00
Daniel Wagner-Hall
eb011cd99b Add docstring 2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083 Implement configurable stats reporting 
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
 2015-09-22 12:57:40 +01:00
Erik Johnston
2048388cfd Merge pull request #281 from matrix-org/erikj/update_extremeties 
When updating a stored event from outlier to non-outlier, remember to update the extremeties
 2015-09-15 16:57:25 +01:00
Erik Johnston
3b05b67c89 When updating a stored event from outlier to non-outlier, remember to update the extremeties 2015-09-15 16:34:42 +01:00
Mark Haines
01fc3943f1 Fix indent 2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3 Remove syutil dependency in favour of smaller single-purpose libraries 2015-08-24 16:17:38 +01:00
Erik Johnston
1f7642efa9 Fix bug where we didn't correctly serialize the redacted_because key over federation 2015-08-21 09:36:07 +01:00
Erik Johnston
20addfa358 Change Cache to not use *args in its interface 2015-08-07 18:32:47 +01:00
Erik Johnston
17bb9a7eb9 Remove commented out code 2015-07-10 14:07:57 +01:00
Erik Johnston
5130d80d79 Add bulk insert events API 2015-06-25 17:29:34 +01:00
Erik Johnston
6e17463228 Don't explode if we don't have the event 2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf Add config option to turn off freezing events. Use new encode_json api and ujson.loads 2015-05-29 12:17:33 +01:00
Erik Johnston
4429e720ae Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-22 10:33:00 +01:00
Mark Haines
53447e9cd3 Add caches for things requested by the pushers 2015-05-21 16:41:39 +01:00
Mark Haines
d61ce3f670 Add a cache for get_current_state with state_key 2015-05-21 16:41:39 +01:00
Erik Johnston
a551c5dad7 Merge pull request #155 from matrix-org/erikj/perf 
Bulk and batch retrieval of events.
 2015-05-21 14:54:40 +01:00
Erik Johnston
27e4b45c06 s/for events/for requests for events/ 2015-05-21 14:52:23 +01:00
Erik Johnston
ac5f2bf9db s/for events/for requests for events/ 2015-05-21 14:50:57 +01:00
Erik Johnston
80a167b1f0 Add comments 2015-05-21 11:19:04 +01:00