Erik Johnston
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
Erik Johnston
37adde32dc
Move defer.returnValue out of Measure
2016-08-19 18:23:44 +01:00
Erik Johnston
47dd8f02a1
Measure _get_event_from_row
2016-08-19 18:23:44 +01:00
Benjamin Saunders
8a57cc3123
Add missing database corruption recovery case
...
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
2016-08-14 11:50:22 -07:00
Erik Johnston
a8b946decb
Raise 404 when couldn't find event
2016-08-05 15:31:02 +01:00
Erik Johnston
f0fa66f495
Delete more tables
2016-08-05 10:40:08 +01:00
Benjamin Saunders
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
Erik Johnston
8ad8490cff
Fix typo
2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88
Retry event persistence on IntegrityError
...
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.
This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
2016-08-04 15:02:15 +01:00
Erik Johnston
80ad710217
Remove other bit of deduplication
2016-08-03 13:25:59 +01:00
Erik Johnston
a8a32d2714
Ensure we only persist an event once at a time
2016-08-03 11:23:39 +01:00
Mark Haines
a6f06ce3e2
Fix how push_actions are redacted.
2016-07-26 11:05:39 +01:00
Mark Haines
efeb6176c1
Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically
2016-07-26 10:49:52 +01:00
Mark Haines
1b3c3e6d68
Only update the events and event_json tables for rejected events
2016-07-25 18:44:30 +01:00
Mark Haines
8f7f4cb92b
Don't add the events to forward extremities if the event is rejected
2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874
Don't add rejections to the state_group, persist all rejections
2016-07-25 16:12:16 +01:00
Erik Johnston
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Erik Johnston
f52565de50
Fix /purge_history bug
...
This was caused by trying to insert duplicate backward extremeties
2016-07-15 14:23:15 +01:00
Erik Johnston
eec9609e96
event_backwards_extremeties may not be empty
2016-07-15 10:22:09 +01:00
Erik Johnston
b64aa6d687
Add sender and contains_url field to events table
2016-07-14 15:35:43 +01:00
Erik Johnston
b92e7955be
Comment
2016-07-07 11:42:15 +01:00
Erik Johnston
c98e1479bd
Return 400 rather than 500
2016-07-07 11:41:07 +01:00
Erik Johnston
67f2c901ea
Add rest servlet. Fix SQL.
2016-07-06 15:56:59 +01:00
Erik Johnston
2d21d43c34
Add purge_history API
2016-07-05 10:28:51 +01:00
Erik Johnston
a67bf0b074
Add storage function to purge history for a room
2016-07-04 16:02:50 +01:00
David Baker
870c45913e
Use similar naming we use in email notifs for push
...
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
Erik Johnston
e4134c5e13
Merge pull request #841 from matrix-org/erikj/event_counter
...
Add metric counter for number of persisted events
2016-06-06 14:17:40 +01:00
Erik Johnston
7aa778fba9
Add metric counter for number of persisted events
2016-06-06 11:58:09 +01:00
Erik Johnston
70aee0717c
Add events to cache when we persist them
2016-06-06 11:34:53 +01:00
Erik Johnston
cffe46408f
Don't rely on options when inserting event into cache
2016-06-03 18:25:21 +01:00
Erik Johnston
10ea3f46ba
Change the way we cache events
2016-06-03 17:57:50 +01:00
Erik Johnston
05e01f21d7
Remove event fetching from DB threads
2016-06-03 17:22:13 +01:00
Erik Johnston
c8285564a3
Use state to calculate get_users_in_room
2016-06-01 15:25:25 +01:00
David Baker
997db04648
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-05-10 14:40:19 +02:00
Erik Johnston
fcb2c3f0db
Remove unused import
2016-05-06 15:47:40 +01:00
Erik Johnston
fd85b167ec
Pull loop one level up
2016-05-06 15:38:42 +01:00
Erik Johnston
b6e0be701e
Queue events for persistence
2016-05-06 14:31:44 +01:00
David Baker
acded821c4
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-04-29 10:05:20 +01:00
Mark Haines
871357d539
Check that somethign has happend before running the selects
2016-04-27 11:54:13 +01:00
David Baker
07d765209d
First bits of emailpusher
...
Mostly logic of when to send an email
2016-04-19 14:24:36 +01:00
Erik Johnston
762ada1e07
Add back backfilled parameter that was removed
2016-04-12 11:58:04 +01:00
Erik Johnston
cccf86dd05
Check if we've already backfilled events
2016-04-12 11:19:32 +01:00
David Baker
9c99ab4572
Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions
2016-04-07 16:35:22 +01:00
David Baker
0fd1cd2400
pep8
2016-04-06 16:50:47 +01:00
David Baker
7e2c89a37f
Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do:
...
* Make badges work again
* Remove old, unused code
2016-04-06 15:42:15 +01:00
Mark Haines
75fb9ac1be
Add a slaved events store class
...
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
2016-04-06 14:18:35 +01:00
Erik Johnston
3d76b7cb2b
Store invites in a separate table.
2016-04-04 16:30:15 +01:00
Mark Haines
f2b916534b
Merge pull request #684 from matrix-org/markjh/backfill_id_gen
...
Use a stream id generator for backfilled ids
2016-04-01 15:13:14 +01:00
Mark Haines
e36bfbab38
Use a stream id generator for backfilled ids
2016-04-01 13:29:05 +01:00
Mark Haines
7753fc6570
Fix the invalidation of the names and aliases cache
2016-04-01 10:34:51 +01:00
Mark Haines
5d06929169
Move the check for backfilled outside the for loop
2016-03-31 15:09:09 +01:00
Mark Haines
76503f95ed
Remove the is_new_state argument to persist event.
...
Move the checks for whether an event is new state inside persist
event itself.
This was harder than expected because there wasn't enough information
passed to persist event to correctly handle invites from remote servers
for new rooms.
2016-03-31 15:00:42 +01:00
Mark Haines
62e395f0e3
Merge pull request #676 from matrix-org/markjh/replicate_stateIII
...
Add replication streams for ex outliers and current state resets
2016-03-31 11:20:57 +01:00
Mark Haines
2ec5426035
Use a namedtuple rather than tuple unpacking
2016-03-31 10:33:02 +01:00
Mark Haines
1fbb094c6f
Add replication streams for ex outliers and current state resets
2016-03-30 17:19:56 +01:00
Mark Haines
8b8052909f
return the state_group for backfill
2016-03-30 16:20:07 +01:00
Mark Haines
61407986b4
Add a entry to current_state_resets table when the current state is reset
2016-03-30 16:18:46 +01:00
Mark Haines
31a9eceda5
Add a replication stream for state groups
2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6
Use a stream id generator to assign state group ids
2016-03-30 12:55:02 +01:00
Erik Johnston
e14baa7a3b
Merge pull request #665 from matrix-org/erikj/dont_cache_events
...
Only cache events in the event cache
2016-03-23 13:19:12 +00:00
Erik Johnston
8b0dfc9fc4
Don't cache events in get_current_state_for_key
2016-03-23 11:42:17 +00:00
Erik Johnston
99f929f36b
Make StateHandler._state_cache only store event_ids.
2016-03-22 16:06:04 +00:00
Mark Haines
5244c0b48e
Remove unused backfilled parameter from persist_event
2016-03-21 18:06:08 +00:00
Erik Johnston
67ed8065db
Dedupe requested event list in _get_events
2016-03-18 14:31:31 +00:00
Mark Haines
7076082ae6
Fix relative imports so they work in both py3 and py27
2016-03-08 11:45:50 +00:00
Mark Haines
a612ce6659
Merge pull request #489 from matrix-org/markjh/replication
...
Add a /replication API for extracting the updates that happened on synapse.
2016-03-01 15:08:24 +00:00
Mark Haines
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Mark Haines
54172924c8
Load the current id in the IdGenerator constructor
...
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.
It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
2016-03-01 14:32:56 +00:00
Erik Johnston
42109a62a4
Remove unused param from get_max_token
2016-02-18 16:37:28 +00:00
Erik Johnston
0214745239
Rename functions
2016-02-10 11:09:56 +00:00
Erik Johnston
7b0d846407
Atomically persit push actions when we persist the event
2016-02-09 16:19:15 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Erik Johnston
00cb3eb24b
Cache tags and account data
2016-01-28 16:37:41 +00:00
Erik Johnston
e1941442d4
Invalidate caches properly. Remove unused arg
2016-01-28 15:02:41 +00:00
Erik Johnston
e7febf4fbb
PEP8
2016-01-27 17:33:27 +00:00
Erik Johnston
aca3193efb
Use the same path for incremental with gap or without gap
2016-01-27 17:33:27 +00:00
Erik Johnston
b97f6626b6
Add cache to room stream
2016-01-27 17:33:26 +00:00
Erik Johnston
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
5880de186b
Log when we skip daily messages
2016-01-06 13:52:16 +00:00
Mark Haines
6a5ff5f223
Track the time spent in the database per request.
...
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
Erik Johnston
4dcaa42b6d
Allow paginating search ordered by recents
2015-11-30 17:45:31 +00:00
Richard van der Hoff
e4d622aaaf
Implementation of state rollback in /sync
...
Implementation of SPEC-254: roll back the state dictionary to how it looked at
the start of the timeline.
Merged PR https://github.com/matrix-org/synapse/pull/373
2015-11-13 10:58:56 +00:00
Daniel Wagner-Hall
cf437900e0
Return world_readable and guest_can_join in /publicRooms
2015-11-10 17:10:27 +00:00
Daniel Wagner-Hall
ca2f90742d
Open up /events to anonymous users for room events only
...
Squash-merge of PR #345 from daniel/anonymousevents
2015-11-05 14:32:26 +00:00
Erik Johnston
61561b9df7
Keep FTS indexes up to date. Only search through rooms currently joined
2015-10-12 10:49:53 +01:00
Mark Haines
92d8d724c5
Remove unused functions from synapse/storage/events.py
2015-09-23 10:33:06 +01:00
Daniel Wagner-Hall
eb011cd99b
Add docstring
2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Erik Johnston
2048388cfd
Merge pull request #281 from matrix-org/erikj/update_extremeties
...
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:57:25 +01:00
Erik Johnston
3b05b67c89
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:34:42 +01:00
Mark Haines
01fc3943f1
Fix indent
2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
Erik Johnston
1f7642efa9
Fix bug where we didn't correctly serialize the redacted_because key over federation
2015-08-21 09:36:07 +01:00
Erik Johnston
20addfa358
Change Cache to not use *args in its interface
2015-08-07 18:32:47 +01:00
Erik Johnston
17bb9a7eb9
Remove commented out code
2015-07-10 14:07:57 +01:00
Erik Johnston
5130d80d79
Add bulk insert events API
2015-06-25 17:29:34 +01:00
Erik Johnston
6e17463228
Don't explode if we don't have the event
2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf
Add config option to turn off freezing events. Use new encode_json api and ujson.loads
2015-05-29 12:17:33 +01:00
