Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-30 08:16:11 -05:00
Code Issues Packages Projects Releases Wiki Activity
21,440 Commits 4 Branches 609 Tags 60 MiB
29269d9d3f
Commit Graph

695 Commits

Author SHA1 Message Date
Erik Johnston
8abdd7b553 Fix up indices for users_who_share_rooms 2017-06-26 14:01:30 +01:00
Erik Johnston
7fe8ed1787 Store URL cache preview downloads seperately 
This makes it easier to clear old media out at a later date
 2017-06-23 11:14:11 +01:00
Erik Johnston
b8b936a6ea Add API to quarantine media 2017-06-19 17:39:21 +01:00
Erik Johnston
d7fe6b356c Add shutdown room API 2017-06-19 12:37:27 +01:00
Erik Johnston
ebcd55d641 Add DB schema for tracking users who share rooms 2017-06-15 09:45:48 +01:00
Erik Johnston
65f0513a33 Split up device_lists_outbound_pokes table for faster updates. 2017-06-07 11:02:38 +01:00
Erik Johnston
21e255a8f1 Split the table in two 2017-06-01 14:50:46 +01:00
Erik Johnston
5dd1b2c525 Use unique indices 2017-05-31 17:29:12 +01:00
Erik Johnston
5d79d728f5 Split out directory and search tables 2017-05-31 15:23:49 +01:00
Erik Johnston
350622a107 Handle the server leaving a public room 2017-05-31 15:11:36 +01:00
Erik Johnston
63fda37e20 Add comments 2017-05-31 15:00:29 +01:00
Erik Johnston
eeb2f9e546 Add user_directory to database 2017-05-31 11:51:01 +01:00
Erik Johnston
dd48f7204c Add comment 2017-05-30 15:01:22 +01:00
Erik Johnston
04095f7581 Add clobbered event_id 2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e Add current_state_delta_stream table 2017-05-30 14:44:09 +01:00
Erik Johnston
6e614e9e10 Add background task to clear out old event_auth 2017-05-24 15:23:34 +01:00
Richard van der Hoff
ff3d810ea8 Add a comment to old delta 2017-05-11 12:48:50 +01:00
Richard van der Hoff
baafb85ba4 Add an index to event_search 
- to make the purge API quicker
 2017-05-11 12:05:22 +01:00
Erik Johnston
b990b2fce5 Add per user ratelimiting overrides 2017-05-10 11:05:43 +01:00
Erik Johnston
d766343668 Add index to device_lists_stream 2017-03-01 15:56:30 +00:00
Erik Johnston
e4919b9329 Add stream_id index to device_lists_outbound_pokes 
As this is used for replication streaming
 2017-02-28 11:19:06 +00:00
David Baker
4aa29508af Use TEXT rather than VARCHAR 
While we're changing anyway
 2017-02-17 10:51:49 +00:00
David Baker
b4017539d4 Make the pushers lang field column longer 
To accommodate things like zh-Hans-CN

Fixes https://github.com/vector-im/riot-ios/issues/1031
 2017-02-17 10:42:57 +00:00
Erik Johnston
095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston
21b7375778 Add an index to make membership queries faster 2017-01-31 15:15:57 +00:00
Erik Johnston
d1e1fd6210 Add ts column to device_lists_outbound_pokes 2017-01-27 15:23:48 +00:00
Erik Johnston
fbfad76c03 Add comments 2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston
2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Erik Johnston
c050f493dd Add comment 2017-01-13 15:14:41 +00:00
Erik Johnston
caddadfc5a Change device_inbox stream index to include user 
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
 2017-01-10 15:04:57 +00:00
Erik Johnston
1574b839e0 Merge pull request #1676 from matrix-org/erikj/room_list 
Add new API appservice specific public room list
 2016-12-12 17:00:10 +00:00
Erik Johnston
c45d8e9ba2 Add profile data to the room_membership table for joins 2016-12-08 13:08:41 +00:00
Erik Johnston
2687af82d4 Comments 2016-12-07 09:58:33 +00:00
Erik Johnston
f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Erik Johnston
46ca345b06 Don't send old events as federation 2016-11-24 12:29:02 +00:00
Erik Johnston
2ac516850b More efficient notif count queries 2016-11-23 15:57:04 +00:00
Erik Johnston
7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston
8fd4d9129f Replace postgres GIN with GIST 
This is because GIN can be slow to write too, especially when the table
gets large.
 2016-11-03 15:00:03 +00:00
Erik Johnston
a9111786f9 Use most recently added binding, not most recently seen user. 2016-10-27 14:32:45 +01:00
Erik Johnston
1fc1bc2a51 Fix user_threepids schema delta 
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.

This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
 2016-10-27 14:14:44 +01:00
David Baker
0108ed8ae6 Latest delta is now 37 2016-10-19 11:40:35 +01:00
David Baker
a7f48320b1 Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive 2016-10-19 11:28:56 +01:00
David Baker
df2a616c7b Convert emails to lowercase when storing 
And db migration sql to convert existing addresses.
 2016-10-19 11:13:55 +01:00
Erik Johnston
6942d68247 Bump schema version 2016-10-17 11:17:45 +01:00
Erik Johnston
b59994b454 Remove TODO 2016-10-17 11:17:02 +01:00
Erik Johnston
816988baaa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth 2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7 Drop some unused indices 2016-10-17 11:08:19 +01:00
David Baker
62073992c5 Make password reset email field case insensitive 2016-10-14 13:56:53 +01:00
Erik Johnston
8009d84364 Match against event_id, rather than room_id 2016-09-21 16:46:59 +01:00
Erik Johnston
dc78db8c56 Update correct table 2016-09-21 15:52:44 +01:00
Erik Johnston
4f78108d8c Readd entries to public_room_list_stream that were deleted 2016-09-21 15:24:22 +01:00
Erik Johnston
baffe96d95 Add a room visibility stream 2016-09-14 17:29:19 +01:00
Erik Johnston
ed992ae6ba Add a DB index to figure out past state at a stream ordering in a room 2016-09-14 16:20:27 +01:00
Erik Johnston
43253c10b8 Remove redundant event_auth index 2016-09-13 11:47:48 +01:00
Erik Johnston
0b32bb20bb Index contains_url for file search queries 2016-09-12 16:57:05 +01:00
Erik Johnston
7fe42cf949 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast 2016-09-12 12:37:09 +01:00
Erik Johnston
15ca0c6a4d Make reindex happen in bg 2016-09-12 12:36:36 +01:00
Mark Haines
31f85f9db9 Add comments to existing schema deltas that used "CREATE INDEX" directly 2016-09-12 11:00:26 +01:00
Erik Johnston
54417999b6 Revert "Add index to event_push_actions" 2016-09-12 10:39:55 +01:00
David Baker
b91e2833b3 Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast 2016-09-09 19:11:34 +01:00
David Baker
f2acc3dcf9 Add index to event_push_actions 
and remove room_id caluse so it uses it

Mostly from @negativemjark
 2016-09-09 18:54:54 +01:00
Mark Haines
647c724573 Use the previous MAX value if any to set the stream_id 2016-09-09 11:52:44 +01:00
Mark Haines
6a6cbfcf1e Track the max_stream_device_id in a separate table, since we delete from the inbox table 2016-09-09 11:48:23 +01:00
Mark Haines
303b6f29f0 Merge pull request #1087 from matrix-org/markjh/reapply_delta 
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
 2016-09-09 11:24:27 +01:00
Mark Haines
0877157353 Just move the schema and add some DROPs 2016-09-09 11:04:47 +01:00
Mark Haines
43b77c5d97 Only catch databas errors 2016-09-08 17:44:21 +01:00
Mark Haines
7d5b142547 Add a stub run_upgrade 2016-09-08 17:39:11 +01:00
Mark Haines
fa722a699c Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop 2016-09-08 17:35:16 +01:00
Erik Johnston
ebb46497ba Add delta file 2016-09-08 16:38:54 +01:00
Mark Haines
fa9d36e050 Merge branch 'develop' into markjh/direct_to_device_federation 2016-09-08 13:43:43 +01:00
Mark Haines
d4a35ada28 Send device messages over federation 2016-09-06 18:16:20 +01:00
Mark Haines
2ad72da931 Add tables for federated device messages 
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
 2016-09-06 15:10:40 +01:00
Erik Johnston
485d999c8a Correctly delete old state groups in purge history API 2016-09-05 14:49:08 +01:00
Erik Johnston
a99e933550 Add upgrade script that will slowly prune state_groups_state entries 2016-09-05 10:05:36 +01:00
Erik Johnston
657847e4c6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage 2016-09-02 11:04:48 +01:00
Erik Johnston
9e25443db8 Move to storing state_groups_state as deltas 2016-09-01 14:31:26 +01:00
Erik Johnston
ef0cc648cf Clean up old sent transactions 2016-08-31 11:12:02 +01:00
Mark Haines
e993925279 Add store-and-forward direct-to-device messaging 2016-08-25 17:35:37 +01:00
Erik Johnston
aad8a1a825 Delete old received_transactions 2016-08-22 16:29:46 +01:00
Erik Johnston
9da84a9a1e Make AppserviceHandler stream events from database 
This is for two reasons:

1. Suppresses duplicates correctly, as the notifier doesn't do any
   duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
   stream.
 2016-08-18 11:54:41 +01:00
Erik Johnston
732cf72b86 Fix push_display_name_rename schema update 2016-08-17 18:12:21 +01:00
David Baker
ad42322257 Add migration script 
To port existing rule actions & enable entries to the new name
 2016-08-16 16:56:30 +01:00
Erik Johnston
d9664344ec Rename table. Add docs. 2016-08-15 11:45:57 +01:00
Erik Johnston
4d70d1f80e Add some invalidations to a cache_stream 2016-08-15 11:15:17 +01:00
Richard van der Hoff
4fec5e57be Default device_display_name to null 
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
 2016-08-03 11:53:00 +01:00
Richard van der Hoff
26cb0efa88 SQL syntax fix 2016-07-27 12:30:22 +01:00
Richard van der Hoff
2e3d90d67c Make the device id on e2e key upload optional 
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.

For active access_tokens without an associated device_id, we ought to register
the device in the devices table.

Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
 2016-07-26 23:38:12 +01:00
Richard van der Hoff
436bffd15f Implement deleting devices 2016-07-26 07:35:48 +01:00
Richard van der Hoff
ec5717caf5 Create index on user_ips in the background 
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.

I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
 2016-07-22 13:16:39 +01:00
Richard van der Hoff
bc8f265f0a GET /devices endpoint 
implement a GET /devices endpoint which lists all of the user's devices.

It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
 2016-07-20 16:42:32 +01:00
Erik Johnston
1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2 
Feature: Add filter to /messages. Add 'contains_url' to filter.
 2016-07-20 10:40:48 +01:00
Richard van der Hoff
022b9176fe schema fix 
device_id should be text, not bigint.
 2016-07-19 11:44:05 +01:00
Richard van der Hoff
f863a52cea Add device_id support to /login 
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.

Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.

Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
 2016-07-18 16:39:44 +01:00
Erik Johnston
b64aa6d687 Add sender and contains_url field to events table 2016-07-14 15:35:43 +01:00
Erik Johnston
314b146b2e Track approximate last access time for remote media 2016-06-29 11:41:20 +01:00
Mark Haines
c8c5bf950a Fix synapse/storage/schema/delta/30/as_users.py 2016-05-31 17:10:40 +01:00
Erik Johnston
c9aff0736c Remove topics table 2016-05-13 14:40:38 +01:00
Mark Haines
9295fa30a8 Annotate the removed indicies with why they were removed. 2016-05-13 14:16:57 +01:00
Erik Johnston
5e50058473 Remove unused indices 
This includes removing both unused indices and indices that are subsets
of other indices.
 2016-05-13 13:28:07 +01:00
Matrix
3b1930e8ec unbreak schema 2016-05-10 16:42:37 +01:00
David Baker
997db04648 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-05-10 14:40:19 +02:00
Mark Haines
9c272da05f Add an openidish mechanism for proving to third parties that you own a given user_id 2016-05-05 13:42:44 +01:00
Erik Johnston
fcd1eb642d Add primary key 2016-05-04 16:51:51 +01:00
Erik Johnston
8e6a163f27 Add timestamp and auto incrementing ID 2016-05-04 15:19:12 +01:00
Erik Johnston
984d4a2c0f Add /report endpoint 2016-05-04 11:28:10 +01:00
David Baker
acded821c4 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-04-29 10:05:20 +01:00
Erik Johnston
ae571810f2 Order NULLs first 2016-04-21 18:14:18 +01:00
Erik Johnston
8fae3d7b1e Use special UPDATE syntax 2016-04-21 18:01:49 +01:00
Erik Johnston
51bb339ab2 Create index concurrently 2016-04-21 17:16:11 +01:00
Erik Johnston
b743c1237e Add missing run_upgrade 2016-04-21 17:12:04 +01:00
Erik Johnston
c877f0f034 Optimise event_search in postgres 2016-04-21 16:56:14 +01:00
David Baker
07d765209d First bits of emailpusher 
Mostly logic of when to send an email
 2016-04-19 14:24:36 +01:00
Erik Johnston
96bcfb29c7 Add index 2016-04-14 11:26:33 +01:00
David Baker
2547dffccc Merge pull request #705 from matrix-org/dbkr/pushers_use_event_actions 
Change pushers to use the event_actions table
 2016-04-11 12:58:55 +01:00
Matthew Hodgson
6ff7a79308 move local_media_repository_url_cache.sql to schema v31 2016-04-08 19:09:02 +01:00
Matthew Hodgson
d6e7333ae4 Merge branch 'develop' into matthew/preview_urls 2016-04-07 17:26:44 +01:00
David Baker
05d044aac3 pep8 2016-04-07 16:45:38 +01:00
David Baker
2d5c693fd3 Fix port script for changes merged from develop 2016-04-07 16:43:54 +01:00
David Baker
9c99ab4572 Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions 2016-04-07 16:35:22 +01:00
David Baker
7e2c89a37f Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do: 
* Make badges work again
 * Remove old, unused code
 2016-04-06 15:42:15 +01:00
Erik Johnston
8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Erik Johnston
0c53d750e7 Docs and indents 2016-04-04 18:02:48 +01:00
Erik Johnston
92ab45a330 Add upgrade path, rename table 2016-04-04 17:07:43 +01:00
Erik Johnston
3d76b7cb2b Store invites in a separate table. 2016-04-04 16:30:15 +01:00
Matthew Hodgson
9f7dc2bef7 Merge branch 'develop' into matthew/preview_urls 2016-04-04 00:38:21 +01:00
Matthew Hodgson
7426c86eb8 add a persistent cache of URL lookups, and fix up the in-memory one to work 2016-04-03 00:31:57 +01:00
Mark Haines
1fbb094c6f Add replication streams for ex outliers and current state resets 2016-03-30 17:19:56 +01:00
Erik Johnston
b275765545 Comment about weird SQL 2016-03-23 14:15:32 +00:00
Erik Johnston
0c1a27b787 SQLite and postgres doesn't share a true literal 2016-03-23 14:10:49 +00:00
Erik Johnston
84afeb41f3 Ensure all old public rooms have aliases 2016-03-23 13:59:34 +00:00
Mark Haines
ba660ecde2 Add a comment to offer a hint to an explanation for why we have a unique constraint on (app_id, pushkey, user_id) 2016-03-16 10:35:00 +00:00
Mark Haines
b6e8420aee Add replication stream for pushers 2016-03-15 17:33:10 +00:00
Erik Johnston
9669a99d1a Update users table in a batched manner 2016-03-10 15:12:19 +00:00
Daniel Wagner-Hall
3b97797c8d Merge branch 'develop' into daniel/ick 2016-03-08 17:35:09 +00:00
Daniel Wagner-Hall
edca2d9891 Idempotent-ise schema update script 
If any ASes don't have an ID, the schema will fail, and then it will
error when trying to add the column again.
 2016-03-08 17:32:29 +00:00
Mark Haines
ebcbb23226 s/stream_ordering/event_stream_ordering/ in push 2016-03-04 16:15:23 +00:00
Mark Haines
ec7460b4f2 Merge branch 'develop' into markjh/pushrule_stream 2016-03-04 14:44:34 +00:00
Erik Johnston
dd463e246d Merge pull request #614 from matrix-org/erikj/alias_delete 
Allow alias creators to delete aliases
 2016-03-04 09:02:33 +00:00
Daniel Wagner-Hall
95481e7ba7 Merge pull request #571 from matrix-org/daniel/asids 
Mark AS users with their AS's ID
 2016-03-03 13:56:28 +00:00
Mark Haines
a1cf9e3bf3 Add a stream for push rule updates 2016-03-01 18:16:37 +00:00
Erik Johnston
f9af8962f8 Allow alias creators to delete aliases 2016-03-01 14:46:31 +00:00
Daniel Wagner-Hall
33300673b7 Generate guest access token on 3pid invites 
This means that following the same link across multiple sessions or
devices can re-use the same guest account.

Note that this is somewhat of an abuse vector; we can't throw up
captchas on this flow, so this is a way of registering ephemeral
accounts for spam, whose sign-up we don't rate limit.
 2016-02-24 14:41:25 +00:00
Erik Johnston
112283e230 Prefix TS fields with _ts 2016-02-18 10:11:43 +00:00
Erik Johnston
e5999bfb1a Initial cut 2016-02-17 15:40:50 +00:00
Daniel Wagner-Hall
763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Erik Johnston
8450114098 Merge pull request #554 from matrix-org/erikj/event_push 
Change event_push_actions_rm_tokens schema
 2016-02-03 15:02:47 +00:00
Erik Johnston
771528ab13 Change event_push_actions_rm_tokens schema 2016-02-03 10:50:49 +00:00
Erik Johnston
65e92eca49 Change the way we do public room list fetching 2016-02-02 15:21:10 +00:00
Erik Johnston
fdca8ec418 Add events index 2016-01-28 16:41:59 +00:00
Erik Johnston
86896408b0 Add index to event_push_actions 2016-01-25 15:30:32 +00:00
David Baker
d4315bbf6b Add index by user id on receipts_linearized 2016-01-20 15:33:27 +00:00
David Baker
fe56138142 Remove rogue 'admin' 2016-01-11 17:09:03 +00:00
David Baker
c232780081 Merge pull request #456 from matrix-org/store_event_actions 
Send unread notification counts
 2016-01-08 14:47:15 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
David Baker
ae1262a241 Add schema change file for is_guest flag 2016-01-06 11:58:20 +00:00
David Baker
c79f221192 Add is_guest flag to users db to track whether a user is a guest user or not. Use this so we can run _filter_events_for_client when calculating event_push_actions. 2016-01-06 11:38:09 +00:00
David Baker
eb03625626 Merge remote-tracking branch 'origin/develop' into store_event_actions 2016-01-05 18:39:50 +00:00
Daniel Wagner-Hall
cfd07aafff Allow guests to upgrade their accounts 2016-01-05 18:01:18 +00:00
David Baker
c914d67cda Rename event-actions to event_push_actions as per PR request 2016-01-04 14:05:37 +00:00
David Baker
42ad49f5b7 still very WIP, but now sends unread_notifications_count in the room object on sync (only actually corrrect in a full sync: hardcoded to 0 in incremental syncs). 2015-12-16 18:42:09 +00:00
David Baker
aa667ee396 Save event actions to the db 2015-12-10 17:51:15 +00:00
Mark Haines
d57c5cda71 Bump schema version. 
As we released version 26 in v0.11.1
 2015-12-04 15:28:39 +00:00
Erik Johnston
b8680b82c3 Merge pull request #414 from matrix-org/erikj/if_not_exists 
Older versions of SQLite don't like IF NOT EXISTS in virtual tables
 2015-12-03 13:21:09 +00:00
David Baker
e515b48929 Just replace the table definition with the one from full_schema 16 2015-12-02 17:23:52 +00:00
David Baker
8810eb8c39 Fix schema delta 15 on postgres in the very unlikley event that anyone upgrades to 15... 2015-12-02 17:19:11 +00:00
Erik Johnston
b2def42bfd Older versions of SQLite don't like IF NOT EXISTS in virtual tables 2015-12-02 13:29:14 +00:00
Mark Haines
95f30ecd1f Add API for setting account_data globaly or on a per room basis 2015-12-01 18:41:32 +00:00
Erik Johnston
4dcaa42b6d Allow paginating search ordered by recents 2015-11-30 17:45:31 +00:00
Daniel Wagner-Hall
7dfa455508 Remove size specifier for database column 
Postgres doesn't support them like this.

We don't have a bool type in common between postgres and sqlite.
 2015-11-23 18:35:25 +00:00
Paul "LeoNerd" Evans
1cfda3d2d8 Merge branch 'develop' into daniel/forgetrooms 2015-11-19 16:53:13 +00:00
Mark Haines
d153f482dd Rename the database table 2015-11-18 15:33:02 +00:00
Daniel Wagner-Hall
ba26eb3d5d Allow users to forget rooms 2015-11-17 17:17:30 -05:00
Erik Johnston
e21cef9bb5 Merge pull request #359 from matrix-org/markjh/incremental_indexing 
Incremental background updates for db indexes
 2015-11-11 17:19:51 +00:00
Mark Haines
e1627388d1 Fix param style to work on both sqlite and postgres 2015-11-11 17:14:56 +00:00
Mark Haines
940a161192 Fix the background update 2015-11-11 13:59:40 +00:00
Daniel Wagner-Hall
cf437900e0 Return world_readable and guest_can_join in /publicRooms 2015-11-10 17:10:27 +00:00
Mark Haines
90b503216c Use a background task to update databases to use the full text search 2015-11-10 16:20:13 +00:00
Mark Haines
c6a01f2ed0 Add storage module for tracking background updates. 
The progress for each background update is stored as a JSON blob in the
database. Each background update is broken up into separate batches.
The batch size is automatically tuned to try avoid blocking single
threaded databases for too long.
 2015-11-09 17:26:27 +00:00
Daniel Wagner-Hall
f2c4ee41b9 Remove accidentally added ID column 2015-11-06 14:27:49 +00:00
Daniel Wagner-Hall
ca2f90742d Open up /events to anonymous users for room events only 
Squash-merge of PR #345 from daniel/anonymousevents
 2015-11-05 14:32:26 +00:00
Mark Haines
6797fcd9ab Merge pull request #335 from matrix-org/markjh/room_tags 
Add APIs for adding and removing tags from rooms
 2015-11-03 16:45:53 +00:00
Mark Haines
ddd8566f41 Store room tag content and return the content in the m.tag event 2015-11-02 15:11:31 +00:00
Daniel Wagner-Hall
621e84d9a0 Add missing column 2015-10-30 16:25:53 +00:00
Mark Haines
892e70ec84 Add APIs for adding and removing tags from rooms 2015-10-28 16:06:57 +00:00
Erik Johnston
4cf633d5e9 Pull out sender when computing search results 2015-10-23 15:41:36 +01:00
Erik Johnston
0c36098c1f Implement rank function for SQLite FTS 2015-10-23 13:23:48 +01:00
Erik Johnston
f142898f52 PEP8 2015-10-22 11:18:01 +01:00
Erik Johnston
4d25bc6c92 Move FTS to delta 25 2015-10-22 11:12:28 +01:00
Erik Johnston
22a8c91448 Split up run_upgrade 2015-10-16 11:19:44 +01:00
Erik Johnston
88971fd034 Merge branch 'erikj/store_engine' into erikj/search 2015-10-13 14:03:30 +01:00
Erik Johnston
f9340ea0d5 Merge branch 'erikj/store_engine' into erikj/search 2015-10-13 13:29:02 +01:00
Erik Johnston
1a40afa756 Add sqlite schema 2015-10-13 10:36:25 +01:00
Erik Johnston
61561b9df7 Keep FTS indexes up to date. Only search through rooms currently joined 2015-10-12 10:49:53 +01:00
Erik Johnston
c85c912562 Add basic full text search impl. 2015-10-09 15:48:31 +01:00
Daniel Wagner-Hall
7213588083 Implement configurable stats reporting 
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
 2015-09-22 12:57:40 +01:00
Erik Johnston
dffc9c4ae0 Drop unused index 2015-09-14 14:41:37 +01:00
Mark Haines
037481a033 Remove autoincrement since we incrementing the ID in the storage layer 2015-08-24 17:48:57 +01:00
Daniel Wagner-Hall
e8cf77fa49 Merge branch 'develop' into refresh 
Conflicts:
	synapse/rest/client/v1/login.py
 2015-08-20 16:25:40 +01:00
Daniel Wagner-Hall
cecbd636e9 /tokenrefresh POST endpoint 
This allows refresh tokens to be exchanged for (access_token,
refresh_token).

It also starts issuing them on login, though no clients currently
interpret them.
 2015-08-20 16:21:35 +01:00
Mark Haines
a0b8e5f2fe Merge pull request #211 from matrix-org/email_in_use 
Changes for unique emails
 2015-08-20 10:04:04 +01:00
Erik Johnston
8f4165628b Add index receipts_linearized_room_stream 2015-08-17 14:43:54 +01:00
David Baker
c77048e12f Add endpoint that proxies ID server request token and errors if the given email is in use on this Home Server. 2015-08-04 14:37:09 +01:00
Erik Johnston
5989637f37 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts 2015-07-13 13:50:57 +01:00
Erik Johnston
0d7f0febf4 Uniquely name unique constraint 2015-07-10 13:43:03 +01:00
Mark Haines
bf0d59ed30 Don't bother with a timeout for one time keys on the server. 2015-07-09 14:04:03 +01:00
Erik Johnston
1af188209a Change format of receipts to allow inclusion of data 2015-07-09 11:39:30 +01:00
Mark Haines
b5770f8947 Add store for client end to end keys 2015-07-06 18:46:47 +01:00
Erik Johnston
ac78e60de6 Add stream_id index 2015-07-02 13:18:41 +01:00
Erik Johnston
80a61330ee Add basic storage functions for handling of receipts 2015-07-01 17:19:12 +01:00
Erik Johnston
901f56fa63 Add tables for receipts 2015-06-30 15:29:47 +01:00
Erik Johnston
2eda996a63 Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py 2015-06-13 12:21:58 +01:00
David Baker
da84946de4 pep8 2015-06-04 16:43:45 +01:00
David Baker
63a7b3ad1e Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them. 2015-06-04 16:16:01 +01:00
Erik Johnston
c71176858b Newline, remove debug logging 2015-05-18 10:11:14 +01:00
Erik Johnston
4f1d984e56 Add index on events 2015-05-13 17:22:26 +01:00
Mark Haines
1c7912751e Drop the old table not the new table 2015-05-08 16:04:32 +01:00
Mark Haines
9d36eb4eab Rename unique constraint 2015-05-08 16:01:55 +01:00
Mark Haines
b0f71db3ff Remove unsigned 2015-05-08 15:59:51 +01:00
Mark Haines
6538d445e8 Make the timestamps in server_keys_json bigints 2015-05-08 15:55:17 +01:00
Erik Johnston
b8940cd902 Remove some unused indexes 2015-05-01 16:14:25 +01:00
David Baker
f8c30faf25 Oops, update the contraint too 2015-04-29 16:58:42 +01:00
David Baker
61cd5d9045 Be more postgressive 2015-04-29 16:57:14 +01:00
David Baker
fb95035a65 Be postgressive 2015-04-29 16:53:41 +01:00
David Baker
4669def000 Oops, forgot the schema delta file 2015-04-29 16:50:16 +01:00
Erik Johnston
c8ed9bd278 pushers table requires an access_token 2015-04-29 15:33:27 +01:00
Mark Haines
e26a3d8d9e bump database schema version 2015-04-29 13:32:32 +01:00
Mark Haines
a9549fdce3 Use bytea rather than BLOB 2015-04-29 13:16:09 +01:00
Mark Haines
4ad8b45155 Merge branch 'develop' into key_distribution 
Conflicts:
	synapse/config/homeserver.py
 2015-04-29 13:15:14 +01:00
Erik Johnston
327ca883ec Merge branch 'develop' of github.com:matrix-org/synapse into postgres 2015-04-28 13:39:42 +01:00
Erik Johnston
18f8247701 Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit 2015-04-28 12:41:33 +01:00
Mark Haines
9182f87664 Merge pull request #126 from matrix-org/csauth 
Client / Server Auth Refactor
 2015-04-28 11:00:27 +01:00
Mark Haines
b96c133034 Add server_keys.sql to the current delta rather than creating a new delta 2015-04-28 10:50:00 +01:00
Erik Johnston
6f8e2d517e Merge branch 'develop' of github.com:matrix-org/synapse into postgres 2015-04-27 14:41:40 +01:00
Mark Haines
eede182df7 Merge branch 'develop' into key_distribution 2015-04-24 10:35:49 +01:00
Mark Haines
4e2f8b8722 Copyright notices 2015-04-24 10:35:29 +01:00
David Baker
cb03fafdf1 Merge branch 'develop' into csauth 2015-04-17 13:51:10 +01:00
Erik Johnston
0ec346d942 Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea 2015-04-17 11:10:20 +01:00
Erik Johnston
b8092fbc82 Go back to storing JSON in TEXT 2015-04-16 11:17:52 +01:00
Erik Johnston
ffad75bd62 Remove mysql/maria support 2015-04-15 17:00:50 +01:00
Mark Haines
a429515bdd Add methods for storing and retrieving the raw key json 2015-04-15 16:58:35 +01:00
Erik Johnston
cf04cedf21 Change full_schemas/11 to work with postgres 2015-04-15 16:53:47 +01:00
Erik Johnston
cce5d057d3 Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses 2015-04-15 11:43:25 +01:00
Mark Haines
32e14d8181 Return a sha256 fingerprint rather than the entire tls certificate 2015-04-14 19:10:09 +01:00
Erik Johnston
5a95cd4442 Rename user_ips.user -> user_id 2015-04-14 13:54:09 +01:00
Erik Johnston
58d8339966 Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS. 2015-04-14 13:53:20 +01:00
Erik Johnston
86fc9b617c For backwards compat, make state_groups.id have a type of int, not varchar 2015-04-13 17:03:49 +01:00
Erik Johnston
93937b2b31 Remove duplicate rows 2015-04-10 18:47:03 +01:00
Erik Johnston
4103b1c470 DROP indexes before recreating them 2015-04-10 18:46:09 +01:00
Erik Johnston
4d5b098626 Use LONGBLOB and TEXT for arbitary length rows 2015-04-10 18:45:09 +01:00
Erik Johnston
7e863c51e6 Use unsigned bigint 2015-04-10 11:22:04 +01:00
Erik Johnston
0f12772e32 SQLite wants INTEGER and not BIGINT for primary keys 2015-04-10 11:16:09 +01:00
Erik Johnston
d5d4281647 Update full_schemas/16 to match delta files. Add delta/16 scripts 2015-04-10 10:59:46 +01:00
Erik Johnston
cda4a6f93f Revert non-trivial changes to upgrade scripts 2015-04-10 10:19:50 +01:00
Erik Johnston
e2722f58ee Fix schema again 2015-04-10 10:16:29 +01:00
Erik Johnston
a1665c5094 Revert non-trivial schema changes and move them to a new schema version. 2015-04-10 10:05:44 +01:00
Erik Johnston
9707acfc40 Remove spurious spaces 2015-04-09 13:45:20 +01:00
Erik Johnston
4fe95094d1 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-07 18:05:39 +01:00
Erik Johnston
8574bf62dc Add index to presence table 2015-04-07 12:09:36 +01:00
Erik Johnston
0af5f5efaf Don't use multiple UNIQUE constraints; it will cause deadlocks 2015-04-07 12:08:35 +01:00
Erik Johnston
779f7b0f44 Fix unicode support 2015-04-02 10:06:22 +01:00
Erik Johnston
9236136f3a Make work in both Maria and SQLite. Fix tests 2015-04-01 14:12:33 +01:00
Kegan Dougal
c217504949 Edit SQL schema to use string IDs not ints. Use token as ID. Update tests. 2015-03-31 12:07:56 +01:00
David Baker
c1a256cc4c Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API. 2015-03-25 19:06:22 +00:00
Erik Johnston
0e8f5095c7 Fix unicode database support 2015-03-25 17:15:20 +00:00
David Baker
c7023f2155 1) Pushers are now associated with an access token 
2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
 2015-03-24 17:24:15 +00:00
Erik Johnston
0d0610870d Fix up schemas some more 2015-03-20 10:55:31 +00:00
Erik Johnston
58ed393235 Remove redundant key 2015-03-19 15:12:05 +00:00
Erik Johnston
fae059cc18 Fix up schemas to work with mariadb 2015-03-19 13:42:39 +00:00
Erik Johnston
89036579ed Update schema to work with mariadb 2015-03-18 11:18:49 +00:00
Kegan Dougal
b98cd03193 Use event IDs instead of dumping event content in the txns table. 2015-03-09 17:25:20 +00:00
Kegan Dougal
0a60bbf4fa Finish appservice txn storage impl and tests. 2015-03-09 15:53:03 +00:00
Kegan Dougal
34ce2ca62f Merge branch 'develop' into application-services-txn-reliability 2015-03-06 17:28:49 +00:00
Kegan Dougal
01c099d9ef Add appservice txns sql schema 2015-03-06 17:16:47 +00:00
David Baker
cf66ddc1b4 Schema change as delta in v14 2015-03-06 14:11:49 +00:00
David Baker
ed72fc3a50 Merge branch 'develop' into pushrules2 
Conflicts:
	synapse/storage/schema/pusher.sql
 2015-03-04 15:24:21 +00:00
Erik Johnston
17d319a20d s/schema_deltas/applied_schema_deltas/ 2015-03-04 15:06:22 +00:00
Erik Johnston
5eefd1f618 Add unique constraint on schema_version.lock schema. Use conflict clause in sql. 2015-03-04 13:52:18 +00:00
Erik Johnston
b4c38738f4 Change to use logger in db upgrade script 2015-03-04 13:43:35 +00:00
Erik Johnston
8c8354e85a Actually add full_schemas dir 2015-03-04 13:34:38 +00:00
Erik Johnston
c3530c3fb3 More docs. Rename 'schema/current' to 'schema/full_schemas' 2015-03-04 13:34:11 +00:00
Erik Johnston
82b34e813d SYN-67: Finish up implementing new database schema management 2015-03-04 12:04:19 +00:00
David Baker
94fa334b01 Add enable/disable overlay for push rules (REST API not yet hooked up) 2015-02-25 19:17:07 +00:00
Erik Johnston
1a989c436c Bump schema version 2015-02-17 15:45:55 +00:00
Mark Haines
0d872f5aa6 Merge pull request #50 from matrix-org/application-services 
Application Services
 2015-02-13 15:06:14 +00:00
Erik Johnston
83d41f25d8 Set database schema version in delta 2015-02-12 10:05:47 +00:00
Kegan Dougal
53557fc532 Merge branch 'develop' into application-services 2015-02-09 15:20:56 +00:00
Kegan Dougal
ac3183caaa Register a user account for the AS when the AS registers. Add 'sender' column to AS table. 2015-02-09 12:03:37 +00:00
Erik Johnston
0cd66885e3 Move delta/v13.sql to delta/v12.sql 2015-02-06 14:38:04 +00:00
Kegan Dougal
a3c6010718 Add delta sql file. 2015-02-05 16:48:57 +00:00
Kegan Dougal
951690e54d Merge branch 'develop' into application-services 2015-02-05 14:28:03 +00:00
Kegan Dougal
27091f146a Add hs_token column and generate a different token f.e application service. 2015-02-05 10:08:12 +00:00
David Baker
dc7bb70f22 s/instance_handle/profile_tag/ 2015-02-03 16:51:07 +00:00
Kegan Dougal
c059c9fea5 Merge branch 'develop' into application-services 
Conflicts:
	synapse/handlers/__init__.py
	synapse/storage/__init__.py
 2015-02-02 15:57:59 +00:00
Erik Johnston
fdd2ac495a Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage 
Conflicts:
	synapse/storage/__init__.py
 2015-01-30 14:57:33 +00:00
Erik Johnston
472cf532b7 Put CREATE rejections into seperate .sql 2015-01-30 14:48:03 +00:00
Erik Johnston
e0b7c521cb Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage 
Conflicts:
	synapse/storage/__init__.py
	synapse/storage/schema/delta/v12.sql
 2015-01-30 14:08:28 +00:00
Mark Haines
9d8f798a3f Merge changes from develop 2015-01-29 14:55:27 +00:00
Mark Haines
e4f50fa0aa Move bump schema delta 2015-01-29 14:53:18 +00:00
Kegan Dougal
3773759c0f Also edit the filter column on the delta SQL 2015-01-29 09:15:33 +00:00
Kegan Dougal
11634017f4 s/definition/filter_json/ since definition is now used to mean a component of the filter, rather than the complete json 2015-01-28 17:42:19 +00:00
David Baker
60b143a52e Move pushers delta to v12 and bump schema version 2015-01-28 15:48:28 +00:00
Kegan Dougal
42876969b9 Add basic application_services SQL, and hook up parts of the appservice store to read from it. 2015-01-28 11:59:38 +00:00
Paul "LeoNerd" Evans
8398f19bce Created schema delta 2015-01-27 19:00:09 +00:00
Paul "LeoNerd" Evans
06cc147012 Initial stab at real SQL storage implementation of user filter definitions 2015-01-27 18:46:03 +00:00
David Baker
ede491b4e0 Oops: second part of commit dc938606 2015-01-22 17:38:53 +00:00
Erik Johnston
b1b85753d7 Add support for storing rejected events in EventContext and data stores 2015-01-22 15:50:17 +00:00
David Baker
afb714f7be add instance_handles to pushers so we have a way to refer to them even if the push token changes. 2015-01-20 11:49:48 +00:00
David Baker
2d2953cf5f Require device language when adding a pusher. 
Because this seems like it might be useful to do sooner rather
than later.
 2015-01-16 11:24:10 +00:00
David Baker
2cb30767fa Honour the 'rejected' return from push gateways 
Add a timestamp to push tokens so we know the last time they we
got them from the device. Send it to the push gateways so it can
determine whether its failure is more recent than the token.
Stop and remove pushers that have been rejected.
 2015-01-13 19:48:37 +00:00
David Baker
c06a9063e1 Merge branch 'develop' into pushers 2015-01-13 13:15:51 +00:00
Erik Johnston
a01416cf21 Add delta and bump DB version 2015-01-06 15:42:18 +00:00
Erik Johnston
f6da237c35 Add index on transaction_id to sent_transcations 2015-01-06 15:40:38 +00:00
Mark Haines
adb04b1e57 Update copyright notices 2015-01-06 13:21:39 +00:00
David Baker
b56730bb6e Merge branch 'develop' into pushers 
Conflicts:
	synapse/api/errors.py
	synapse/server.py
	synapse/storage/__init__.py
 2014-12-18 15:15:22 +00:00
David Baker
afa953a293 schema version is now 10 2014-12-18 15:11:06 +00:00
David Baker
fc7c5e9cd7 Rename the pusher SQL delta to v9 which the next free one 2014-12-18 14:51:29 +00:00
David Baker
9728c305a3 after a few rethinks, a working implementation of pushers. 2014-12-18 14:49:22 +00:00
Erik Johnston
42b725ce52 Fix upgrade script to run all the missing deltas. 2014-12-16 15:13:34 +00:00
Erik Johnston
882dc8dcab Persist internal_metadata 2014-12-16 13:17:09 +00:00
Erik Johnston
cde840a82c Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 
Conflicts:
	setup.py
 2014-12-11 17:48:48 +00:00
Mark Haines
4f37c0ea9d Merge branch 'develop' into media_repository 2014-12-10 16:55:06 +00:00
Mark Haines
61fc37e467 Merge branch 'develop' into media_repository 2014-12-10 16:14:17 +00:00
Erik Johnston
1d2a0040cf Fix bug where we clobbered old state group values 2014-12-10 15:55:03 +00:00
Erik Johnston
02db7eb209 Fix bug when uploading state with empty state_key 2014-12-10 14:02:48 +00:00
Erik Johnston
aae8a37e63 Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 2014-12-10 13:18:40 +00:00
Matthew Hodgson
aed62a3583 track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover. 2014-12-07 02:26:07 +00:00
Mark Haines
a953be097f Add a method field to thumbnail storage 2014-12-05 16:31:56 +00:00
David Baker
88af58d41d Update to app_id / app_instance_id (partially) and mangle to be PEP8 compliant. 2014-12-03 13:37:02 +00:00
Mark Haines
279c48c8b4 Write the upload portion of version 1 of the media repository 2014-12-02 17:13:14 +00:00
David Baker
7642d95d5e Merge branch 'develop' into pushers 2014-12-02 13:50:05 +00:00
Erik Johnston
ec2b5d8c28 Store full JSON of events in db 2014-12-01 16:22:07 +00:00
Erik Johnston
cb76945688 Add update delta for schema change 2014-11-26 11:17:19 +00:00
Erik Johnston
87538711b6 Update schema to support multiple signatures 2014-11-26 11:14:30 +00:00
David Baker
eb6aedf92c More work on pushers. Attempt to do HTTP pokes. Not sure if the actual HTTP pokes work or not yet but the retry semantics are pretty good. 2014-11-21 12:21:00 +00:00
David Baker
74c3879760 Start creating a module to do generic notifications (just prints them to stdout currently!) 2014-11-19 18:20:59 +00:00
Erik Johnston
f04b3d5042 Store all signatures on events rather than just dropping them 2014-11-12 17:02:34 +00:00
Erik Johnston
58c0ef90c9 Add indices to state group tables 2014-11-12 14:33:48 +00:00
Erik Johnston
6cb6cb9e69 Tidy up some of the unused sql tables 2014-11-10 10:31:00 +00:00
Erik Johnston
bf6b72eb55 Start implementing auth chains 2014-11-07 10:53:38 +00:00
Erik Johnston
4317c8e583 Implement new replace_state and changed prev_state 
`prev_state` is now a list of previous state ids, similiar to
prev_events. `replace_state` now points to what we think was replaced.
 2014-11-06 15:10:55 +00:00
Erik Johnston
fb3a01fa3a Remove unused sql file. 2014-10-31 16:04:27 +00:00
Erik Johnston
2f39dc19a2 Remove more references to dead PDU tables 2014-10-31 14:27:14 +00:00
Erik Johnston
bfa36a72b9 Remove PDU tables. 2014-10-31 14:00:32 +00:00
Erik Johnston
e7858b6d7e Start filling out and using new events tables 2014-10-29 16:59:24 +00:00
Erik Johnston
2d1dfb3b34 Begin implementing all the PDU storage stuff in Events land 2014-10-28 16:42:35 +00:00
Erik Johnston
ad9226eeec Merge branch 'event_signing' of github.com:matrix-org/synapse into federation_authorization 
Conflicts:
	synapse/storage/__init__.py
 2014-10-27 11:58:32 +00:00
Mark Haines
acb2d171e8 Merge branch 'develop' into event_signing 2014-10-27 11:14:11 +00:00
Erik Johnston
bb4a20174c Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization 
Conflicts:
	synapse/federation/transport.py
	synapse/handlers/message.py
 2014-10-27 10:20:44 +00:00