Mark Haines
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
Mark Haines
2a0159b8ae
Fix the stream change cache to work over replication
2016-09-07 15:58:00 +01:00
Mark Haines
31a07d2335
Add stream change caches for device messages
2016-09-07 15:27:07 +01:00
Erik Johnston
5dc2a702cf
Make _state_groups_id_gen a normal IdGenerator
2016-08-30 16:55:11 +01:00
Mark Haines
e993925279
Add store-and-forward direct-to-device messaging
2016-08-25 17:35:37 +01:00
Erik Johnston
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
Erik Johnston
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
Erik Johnston
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
Richard van der Hoff
f863a52cea
Add device_id support to /login
...
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.
Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.
Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
Erik Johnston
978fa53cc2
Pull out min stream_ordering from ex_outlier_stream
2016-07-15 10:22:30 +01:00
Mark Haines
eef541a291
Move insert_client_ip to a separate class
2016-06-03 14:42:35 +01:00
Mark Haines
3ae915b27e
Add a slaved store for presence
2016-06-03 11:05:53 +01:00
Erik Johnston
85b992f621
Fix to allow start with postgres
2016-05-27 10:44:44 +01:00
David Baker
d4503e25ed
Make deleting push actions more efficient
...
There's no index on received_ts, so manually binary search using the stream_ordering index, and only update it once an hour.
2016-05-20 17:56:10 +01:00
Mark Haines
9c272da05f
Add an openidish mechanism for proving to third parties that you own a given user_id
2016-05-05 13:42:44 +01:00
Erik Johnston
8e6a163f27
Add timestamp and auto incrementing ID
2016-05-04 15:19:12 +01:00
Mark Haines
a1e0d316ea
Move _get_cache_dict into the SQLBaseStore
2016-04-06 13:05:19 +01:00
Erik Johnston
df727f2126
Fix stuck invites
...
If rejecting a remote invite fails with an error response don't fail
the entire request; instead mark the invite as locally rejected.
This fixes the bug where users can get stuck invites which they can
neither accept nor reject.
2016-04-05 11:13:24 +01:00
Mark Haines
a2866e2e6a
Rename direction to step, apply checks consistently
2016-04-01 13:50:54 +01:00
Mark Haines
e36bfbab38
Use a stream id generator for backfilled ids
2016-04-01 13:29:05 +01:00
Mark Haines
1e25f62ee6
Use a stream id generator to assign state group ids
2016-03-30 12:55:02 +01:00
Mark Haines
b6e8420aee
Add replication stream for pushers
2016-03-15 17:33:10 +00:00
Mark Haines
7076082ae6
Fix relative imports so they work in both py3 and py27
2016-03-08 11:45:50 +00:00
Mark Haines
9848b54cac
Prefill from the correct stream
2016-03-04 16:20:22 +00:00
Mark Haines
deda48068c
prefill the push rules stream change cache
2016-03-04 16:19:42 +00:00
Mark Haines
1b4f4a936f
Hook up the push rules stream to account_data in /sync
2016-03-04 14:44:01 +00:00
Mark Haines
a1cf9e3bf3
Add a stream for push rule updates
2016-03-01 18:16:37 +00:00
Mark Haines
54172924c8
Load the current id in the IdGenerator constructor
...
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.
It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
2016-03-01 14:32:56 +00:00
Erik Johnston
e6c5e3f28a
Close cursor
2016-02-18 16:39:28 +00:00
Erik Johnston
112283e230
Prefix TS fields with _ts
2016-02-18 10:11:43 +00:00
Erik Johnston
e5999bfb1a
Initial cut
2016-02-17 15:40:50 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Erik Johnston
25c311eaf6
Cache get_room_changes_for_user
2016-01-29 16:52:48 +00:00
Erik Johnston
8da95b6f1b
Comment. Remove superfluous order by
2016-01-29 15:39:17 +00:00
Erik Johnston
b5dbced938
Don't prefill account data
2016-01-29 14:53:59 +00:00
Erik Johnston
3d60686c0c
Actually use cache
2016-01-29 14:49:11 +00:00
Erik Johnston
45488e0ffa
Max is not a function
2016-01-29 14:42:01 +00:00
Erik Johnston
f67d60496a
Convert param style
2016-01-29 14:41:16 +00:00
Erik Johnston
18579534ea
Prefill stream change caches
2016-01-29 14:37:59 +00:00
Erik Johnston
7ed2bbeb11
Clean up a bit. Add comment
2016-01-28 14:32:05 +00:00
Erik Johnston
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
David Baker
c232780081
Merge pull request #456 from matrix-org/store_event_actions
...
Send unread notification counts
2016-01-08 14:47:15 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
David Baker
c914d67cda
Rename event-actions to event_push_actions as per PR request
2016-01-04 14:05:37 +00:00
David Baker
aa667ee396
Save event actions to the db
2015-12-10 17:51:15 +00:00
Mark Haines
95f30ecd1f
Add API for setting account_data globaly or on a per room basis
2015-12-01 18:41:32 +00:00
Mark Haines
892e70ec84
Add APIs for adding and removing tags from rooms
2015-10-28 16:06:57 +00:00
Erik Johnston
f9340ea0d5
Merge branch 'erikj/store_engine' into erikj/search
2015-10-13 13:29:02 +01:00
Erik Johnston
ec398af41c
Expose error more nicely
2015-10-13 11:43:43 +01:00
Erik Johnston
40b6a5aad1
Split out the schema preparation and update logic into its own module
2015-10-13 11:38:48 +01:00
Erik Johnston
c85c912562
Add basic full text search impl.
2015-10-09 15:48:31 +01:00
Daniel Wagner-Hall
6d59ffe1ce
Add some docstrings
2015-09-22 13:47:40 +01:00
Daniel Wagner-Hall
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Daniel Wagner-Hall
a0b181bd17
Remove completely unused concepts from codebase
...
Removes device_id and ClientInfo
device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.
This doesn't remove the columns from the database, because that's
fiddly.
2015-08-25 16:23:06 +01:00
Daniel Wagner-Hall
cecbd636e9
/tokenrefresh POST endpoint
...
This allows refresh tokens to be exchanged for (access_token,
refresh_token).
It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
Erik Johnston
8f4165628b
Add index receipts_linearized_room_stream
2015-08-17 14:43:54 +01:00
Erik Johnston
7dec0b2bee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:28:39 +01:00
Paul "LeoNerd" Evans
0db40d3e93
Don't complain about extra .pyc files we find while hunting for database schemas
2015-08-07 17:22:11 +01:00
Erik Johnston
b8e386db59
Change Cache to not use *args in its interface
2015-08-07 11:52:21 +01:00
Erik Johnston
5989637f37
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-13 13:50:57 +01:00
Erik Johnston
716e426933
Fix various typos
2015-07-07 10:55:31 +01:00
Mark Haines
b5770f8947
Add store for client end to end keys
2015-07-06 18:46:47 +01:00
Erik Johnston
901f56fa63
Add tables for receipts
2015-06-30 15:29:47 +01:00
Erik Johnston
1e365e88bd
Bump schema version
2015-06-11 15:50:39 +01:00
David Baker
63a7b3ad1e
Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them.
2015-06-04 16:16:01 +01:00
Erik Johnston
4f1d984e56
Add index on events
2015-05-13 17:22:26 +01:00
Mark Haines
84e1cacea4
Bump schema version
2015-05-08 15:58:14 +01:00
Erik Johnston
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
Erik Johnston
970a9b9d2b
We can't use REPLACE when upgrading databases now we have postgres
2015-04-29 13:55:44 +01:00
Mark Haines
e26a3d8d9e
bump database schema version
2015-04-29 13:32:32 +01:00
Erik Johnston
327ca883ec
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-28 13:39:42 +01:00
David Baker
8a5be236e0
pep8
2015-04-27 11:49:18 +01:00
David Baker
b02e1006b9
Run database check before daemonizing, at the cost of database hygiene.
2015-04-27 11:46:00 +01:00
Erik Johnston
5a95cd4442
Rename user_ips.user -> user_id
2015-04-14 13:54:09 +01:00
Erik Johnston
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
Erik Johnston
ce797ad373
Bump schema version
2015-04-10 11:22:30 +01:00
Erik Johnston
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
Erik Johnston
0bfa78b39b
PEP8
2015-04-07 12:16:05 +01:00
Erik Johnston
6bc9edd8b2
Fix prepare_sqlite3_database's convert_param_style
2015-04-07 12:13:58 +01:00
Erik Johnston
05a35d62b6
Bump database version
2015-04-07 12:10:15 +01:00
Erik Johnston
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
Kegan Dougal
4edcbcee3b
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
Erik Johnston
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
Erik Johnston
9115421ace
Use _simple_upsert
2015-03-24 16:17:39 +00:00
Paul "LeoNerd" Evans
a63b4f7101
Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale
2015-03-20 18:25:49 +00:00
Erik Johnston
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
Erik Johnston
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
Erik Johnston
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
Erik Johnston
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
Erik Johnston
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
Kegan Dougal
835e01fc70
Minor PR comment tweaks.
2015-03-16 10:16:59 +00:00
Kegan Dougal
f9232c7917
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/appservice.py
2015-03-16 10:09:15 +00:00
Paul "LeoNerd" Evans
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
Paul "LeoNerd" Evans
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
David Baker
8916acbc13
These aren't defined for redacted events so don't crash
2015-03-10 11:21:37 +00:00
Kegan Dougal
21fd84dcb8
Use seconds; start gluing in the AS scheduler into the AS handler.
2015-03-09 17:01:19 +00:00
Kegan Dougal
01c099d9ef
Add appservice txns sql schema
2015-03-06 17:16:47 +00:00
Erik Johnston
d56c01fff4
Note that we don't specify execution order
2015-03-04 15:10:05 +00:00
Erik Johnston
17d319a20d
s/schema_deltas/applied_schema_deltas/
2015-03-04 15:06:22 +00:00
Erik Johnston
5681264faa
s/%r/%s/
2015-03-04 14:21:53 +00:00
Erik Johnston
f701197227
Add example directory structures in doc
2015-03-04 14:20:14 +00:00
Erik Johnston
16dd87d848
Don't assume db conn is a Context Manager.
...
Twisted adbapi wrapped connections aren't context managers.
2015-03-04 14:03:41 +00:00
Erik Johnston
5eefd1f618
Add unique constraint on schema_version.lock schema. Use conflict clause in sql.
2015-03-04 13:52:18 +00:00
Erik Johnston
640e53935d
Use context manager with db conn to correctly commit and rollback
2015-03-04 13:43:17 +00:00
Erik Johnston
c3530c3fb3
More docs. Rename 'schema/current' to 'schema/full_schemas'
2015-03-04 13:34:11 +00:00
Erik Johnston
811355ccd0
Add some docs and remove unused variables
2015-03-04 13:11:01 +00:00
Erik Johnston
82b34e813d
SYN-67: Finish up implementing new database schema management
2015-03-04 12:04:19 +00:00
Erik Johnston
8d33adfbbb
SYN-67: Begin changing the way we handle schema versioning
2015-03-02 18:23:55 +00:00
Erik Johnston
9f03553f48
Add missing comma
2015-03-02 16:38:40 +00:00
Erik Johnston
b41dc68773
We purposefully don't have a version 14 delta script.
2015-03-02 16:36:19 +00:00
Kegan Dougal
c3c01641d2
Run deltas and bump user_version in upgrade script
2015-03-02 13:38:57 +00:00
Erik Johnston
2c29ed3e84
Use absolute path when loading delta sql files
2015-02-17 17:22:24 +00:00
Erik Johnston
1a989c436c
Bump schema version
2015-02-17 15:45:55 +00:00
Mark Haines
0d872f5aa6
Merge pull request #50 from matrix-org/application-services
...
Application Services
2015-02-13 15:06:14 +00:00
Mark Haines
183b3d4e47
Prepare the database whenever a connection is opened from the db_pool so that in-memory databases will work
2015-02-13 14:38:24 +00:00
Erik Johnston
42bc56dad3
Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson
2015-02-11 17:01:38 +00:00
Erik Johnston
7eef84a95b
pyflakes
2015-02-11 16:52:22 +00:00
Kegan Dougal
f2fdcb7c4b
Merge branch 'develop' into application-services
2015-02-11 16:43:26 +00:00
Erik Johnston
ddb816cf60
Don't unfreeze when using FreezeEvent.get_dict, as we are using a JSONEncoder that understands FrozenDict
2015-02-11 15:44:28 +00:00
Mark Haines
ddc25cf4e2
Invalidate the cache for an event if it is redacted
2015-02-11 15:23:28 +00:00
Mark Haines
f5a70e0d2e
Add a cache for get_event
2015-02-11 15:01:15 +00:00
Erik Johnston
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
Kegan Dougal
951690e54d
Merge branch 'develop' into application-services
2015-02-05 14:28:03 +00:00
Erik Johnston
03d415a6a2
Brief comment on why we do some things on every call to persist_event and not others
2015-02-04 10:40:59 +00:00
Erik Johnston
c0462dbf15
Rearrange persist_event so that do all the queries that need to be done before returning early if we have already persisted that event.
2015-02-04 10:16:51 +00:00
Erik Johnston
02be8da5e1
Add doc to get_event
2015-02-03 17:34:07 +00:00
Erik Johnston
e7ca813dd4
Try to ensure we don't persist an event we have already persisted. In persist_event check if we already have the event, if so then update instead of replacing so that we don't cause a bump of the stream_ordering.
2015-02-03 10:39:41 +00:00
Kegan Dougal
1a2de0c5fe
Implement txns for AS (un)registration.
2015-02-02 17:39:41 +00:00
Erik Johnston
941f59101b
Don't fail an entire request if one of the returned events fails a signature check. If an event does fail a signature check, look in the local database and request it from the originator.
2015-02-02 16:56:01 +00:00
Kegan Dougal
a006d168c5
Actually merge into develop.
2015-02-02 16:05:34 +00:00
Kegan Dougal
c059c9fea5
Merge branch 'develop' into application-services
...
Conflicts:
synapse/handlers/__init__.py
synapse/storage/__init__.py
2015-02-02 15:57:59 +00:00
Mark Haines
9ec9d6f2cb
Merge pull request #42 from matrix-org/replication_split
...
Replication split
2015-01-30 15:14:10 +00:00
Erik Johnston
91015ad008
Remove merge conflict
2015-01-30 14:58:54 +00:00
Erik Johnston
4f7fe63b6d
Remember to add schema file to list
2015-01-30 14:57:53 +00:00
Erik Johnston
fdd2ac495a
Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage
...
Conflicts:
synapse/storage/__init__.py
2015-01-30 14:57:33 +00:00
Erik Johnston
471c47441d
Merge pull request #37 from matrix-org/client_v2_filter
...
Client v2 filter
2015-01-30 14:56:08 +00:00
David Baker
322a047502
Add room member count condition and default rule to make a noise on rooms of only 2 people.
2015-01-30 14:46:45 +00:00
Erik Johnston
84b78c3b5f
Merge branch 'rejections_storage' of github.com:matrix-org/synapse into replication_split
2015-01-30 14:17:47 +00:00
Erik Johnston
e0b7c521cb
Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage
...
Conflicts:
synapse/storage/__init__.py
synapse/storage/schema/delta/v12.sql
2015-01-30 14:08:28 +00:00
Mark Haines
9d8f798a3f
Merge changes from develop
2015-01-29 14:55:27 +00:00
David Baker
60b143a52e
Move pushers delta to v12 and bump schema version
2015-01-28 15:48:28 +00:00
Kegan Dougal
42876969b9
Add basic application_services SQL, and hook up parts of the appservice store to read from it.
2015-01-28 11:59:38 +00:00
Paul "LeoNerd" Evans
8398f19bce
Created schema delta
2015-01-27 19:00:09 +00:00
Paul "LeoNerd" Evans
06cc147012
Initial stab at real SQL storage implementation of user filter definitions
2015-01-27 18:46:03 +00:00
Paul "LeoNerd" Evans
54e513b4e6
Move storage of user filters into real datastore layer; now have to mock it out in the REST-level tests
2015-01-27 17:48:13 +00:00
Kegan Dougal
51449e0665
Add appservice handler and store. Glue together rest > handler > store.
2015-01-27 15:50:28 +00:00
David Baker
69a75b7ebe
Add brackets to make get room name / alias work
2015-01-26 16:52:47 +00:00
Erik Johnston
7b88619241
Split up replication_layer module into client, server and transaction queue
2015-01-26 10:45:24 +00:00
Erik Johnston
3b9cc882a5
Add storage method have_events
2015-01-23 15:42:52 +00:00
David Baker
673773b217
oops, this is not its own schema file
2015-01-22 18:27:07 +00:00
