Matthew Hodgson
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
David Baker
47ed4a4aa7
PR feedback
...
Unnecessary inlineCallbacks, missing yield, SQL comments & trailing
commas.
2018-04-25 13:58:37 +01:00
David Baker
3c446d0a81
Merge remote-tracking branch 'origin/dinsic' into dbkr/profile_replication
2018-04-16 18:35:25 +01:00
David Baker
e654230a51
Written but untested profile replication
2018-04-10 17:41:58 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
Luke Barnard
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
Luke Barnard
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
Luke Barnard
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
David Baker
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
David Baker
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
David Baker
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
David Baker
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
Neil Johnson
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
Erik Johnston
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
Erik Johnston
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
Erik Johnston
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
Erik Johnston
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
Richard van der Hoff
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Erik Johnston
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
Erik Johnston
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
Matthew Hodgson
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
Matthew Hodgson
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
Matthew Hodgson
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
Richard van der Hoff
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
Richard van der Hoff
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
Matthew Hodgson
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
Matthew Hodgson
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
Matthew Hodgson
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
Matthew Hodgson
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
Matthew Hodgson
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
Richard van der Hoff
356bcafc44
Remove the last vestiges of refresh_tokens
2017-10-31 20:35:58 +00:00
Luke Barnard
007cd48af6
Recreate groups table instead of adding column
...
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
Luke Barnard
713e60b9b6
Awful hack to get default true
2017-10-26 17:38:14 +01:00
Luke Barnard
9b2feef9eb
Add is_public to groups table to allow for private groups
...
Prevent group API access to non-members for private groups
Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston
c2c47550f9
Fix schema delta versions
2017-10-11 13:23:15 +01:00
David Baker
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
Erik Johnston
4dc07e93a8
Add old indices
2017-09-28 14:10:33 +01:00
Erik Johnston
ae79764fe5
Change expires column to expires_ts
2017-09-28 12:37:53 +01:00
Erik Johnston
77f1d24de3
More brackets
2017-09-28 12:23:15 +01:00
Erik Johnston
9ccb4226ba
Delete expired url cache data
2017-09-28 12:18:06 +01:00
Erik Johnston
a8e2a3df32
Add unique index to group_rooms table
2017-09-26 15:39:21 +01:00
Erik Johnston
27ebc5c8f2
Add remote profile cache
2017-08-25 11:25:47 +01:00
Erik Johnston
8b50fe5330
Use BOOLEAN rather than TEXT type
2017-08-21 16:37:29 +01:00
Erik Johnston
05e21285aa
Store whether the user wants to publicise their membership of a group
2017-08-08 13:01:46 +01:00
Erik Johnston
c544188ee3
Add groups to sync stream
2017-07-20 16:36:42 +01:00
Erik Johnston
508460f240
Remove sync stuff
2017-07-18 09:55:46 +01:00
Erik Johnston
4540730111
Remove unused tables
2017-07-18 09:38:15 +01:00
Erik Johnston
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
Erik Johnston
26451a09eb
Comments
2017-07-12 14:47:18 +01:00
Erik Johnston
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
Erik Johnston
6d586dc05c
Comment
2017-07-12 09:58:37 +01:00
Erik Johnston
410b4e14a1
Move comment
2017-07-11 15:44:18 +01:00
Erik Johnston
e52c391cd4
Rename column to attestation_json
2017-07-11 14:25:46 +01:00
Erik Johnston
0aac30d53b
Comments
2017-07-11 14:23:50 +01:00
Erik Johnston
83936293eb
Comments
2017-07-11 11:42:25 +01:00
Erik Johnston
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
Erik Johnston
8abdd7b553
Fix up indices for users_who_share_rooms
2017-06-26 14:01:30 +01:00
Erik Johnston
7fe8ed1787
Store URL cache preview downloads seperately
...
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
Erik Johnston
b8b936a6ea
Add API to quarantine media
2017-06-19 17:39:21 +01:00
Erik Johnston
d7fe6b356c
Add shutdown room API
2017-06-19 12:37:27 +01:00
Erik Johnston
ebcd55d641
Add DB schema for tracking users who share rooms
2017-06-15 09:45:48 +01:00
Erik Johnston
65f0513a33
Split up device_lists_outbound_pokes table for faster updates.
2017-06-07 11:02:38 +01:00
Erik Johnston
21e255a8f1
Split the table in two
2017-06-01 14:50:46 +01:00
Erik Johnston
5dd1b2c525
Use unique indices
2017-05-31 17:29:12 +01:00
Erik Johnston
5d79d728f5
Split out directory and search tables
2017-05-31 15:23:49 +01:00
Erik Johnston
350622a107
Handle the server leaving a public room
2017-05-31 15:11:36 +01:00
Erik Johnston
63fda37e20
Add comments
2017-05-31 15:00:29 +01:00
Erik Johnston
eeb2f9e546
Add user_directory to database
2017-05-31 11:51:01 +01:00
Erik Johnston
dd48f7204c
Add comment
2017-05-30 15:01:22 +01:00
Erik Johnston
04095f7581
Add clobbered event_id
2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e
Add current_state_delta_stream table
2017-05-30 14:44:09 +01:00
Erik Johnston
6e614e9e10
Add background task to clear out old event_auth
2017-05-24 15:23:34 +01:00
Richard van der Hoff
ff3d810ea8
Add a comment to old delta
2017-05-11 12:48:50 +01:00
Richard van der Hoff
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Erik Johnston
b990b2fce5
Add per user ratelimiting overrides
2017-05-10 11:05:43 +01:00
Erik Johnston
d766343668
Add index to device_lists_stream
2017-03-01 15:56:30 +00:00
Erik Johnston
e4919b9329
Add stream_id index to device_lists_outbound_pokes
...
As this is used for replication streaming
2017-02-28 11:19:06 +00:00
David Baker
4aa29508af
Use TEXT rather than VARCHAR
...
While we're changing anyway
2017-02-17 10:51:49 +00:00
David Baker
b4017539d4
Make the pushers lang field column longer
...
To accommodate things like zh-Hans-CN
Fixes https://github.com/vector-im/riot-ios/issues/1031
2017-02-17 10:42:57 +00:00
Erik Johnston
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
Erik Johnston
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
Erik Johnston
d1e1fd6210
Add ts column to device_lists_outbound_pokes
2017-01-27 15:23:48 +00:00
Erik Johnston
fbfad76c03
Add comments
2017-01-26 16:33:21 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
Erik Johnston
c050f493dd
Add comment
2017-01-13 15:14:41 +00:00
Erik Johnston
caddadfc5a
Change device_inbox stream index to include user
...
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
Erik Johnston
1574b839e0
Merge pull request #1676 from matrix-org/erikj/room_list
...
Add new API appservice specific public room list
2016-12-12 17:00:10 +00:00
Erik Johnston
c45d8e9ba2
Add profile data to the room_membership table for joins
2016-12-08 13:08:41 +00:00
Erik Johnston
2687af82d4
Comments
2016-12-07 09:58:33 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Erik Johnston
46ca345b06
Don't send old events as federation
2016-11-24 12:29:02 +00:00
Erik Johnston
2ac516850b
More efficient notif count queries
2016-11-23 15:57:04 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
8fd4d9129f
Replace postgres GIN with GIST
...
This is because GIN can be slow to write too, especially when the table
gets large.
2016-11-03 15:00:03 +00:00
Erik Johnston
a9111786f9
Use most recently added binding, not most recently seen user.
2016-10-27 14:32:45 +01:00
Erik Johnston
1fc1bc2a51
Fix user_threepids schema delta
...
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.
This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
2016-10-27 14:14:44 +01:00
David Baker
0108ed8ae6
Latest delta is now 37
2016-10-19 11:40:35 +01:00
David Baker
a7f48320b1
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-19 11:28:56 +01:00
David Baker
df2a616c7b
Convert emails to lowercase when storing
...
And db migration sql to convert existing addresses.
2016-10-19 11:13:55 +01:00
Erik Johnston
6942d68247
Bump schema version
2016-10-17 11:17:45 +01:00
Erik Johnston
b59994b454
Remove TODO
2016-10-17 11:17:02 +01:00
Erik Johnston
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
David Baker
62073992c5
Make password reset email field case insensitive
2016-10-14 13:56:53 +01:00
Erik Johnston
8009d84364
Match against event_id, rather than room_id
2016-09-21 16:46:59 +01:00
Erik Johnston
dc78db8c56
Update correct table
2016-09-21 15:52:44 +01:00
Erik Johnston
4f78108d8c
Readd entries to public_room_list_stream that were deleted
2016-09-21 15:24:22 +01:00
Erik Johnston
baffe96d95
Add a room visibility stream
2016-09-14 17:29:19 +01:00
Erik Johnston
ed992ae6ba
Add a DB index to figure out past state at a stream ordering in a room
2016-09-14 16:20:27 +01:00
Erik Johnston
43253c10b8
Remove redundant event_auth index
2016-09-13 11:47:48 +01:00
Erik Johnston
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
Erik Johnston
7fe42cf949
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast
2016-09-12 12:37:09 +01:00
Erik Johnston
15ca0c6a4d
Make reindex happen in bg
2016-09-12 12:36:36 +01:00
Mark Haines
31f85f9db9
Add comments to existing schema deltas that used "CREATE INDEX" directly
2016-09-12 11:00:26 +01:00
Erik Johnston
54417999b6
Revert "Add index to event_push_actions"
2016-09-12 10:39:55 +01:00
David Baker
b91e2833b3
Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast
2016-09-09 19:11:34 +01:00
David Baker
f2acc3dcf9
Add index to event_push_actions
...
and remove room_id caluse so it uses it
Mostly from @negativemjark
2016-09-09 18:54:54 +01:00
Mark Haines
647c724573
Use the previous MAX value if any to set the stream_id
2016-09-09 11:52:44 +01:00
Mark Haines
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
Mark Haines
303b6f29f0
Merge pull request #1087 from matrix-org/markjh/reapply_delta
...
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
2016-09-09 11:24:27 +01:00
Mark Haines
0877157353
Just move the schema and add some DROPs
2016-09-09 11:04:47 +01:00
Mark Haines
43b77c5d97
Only catch databas errors
2016-09-08 17:44:21 +01:00
Mark Haines
7d5b142547
Add a stub run_upgrade
2016-09-08 17:39:11 +01:00
Mark Haines
fa722a699c
Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop
2016-09-08 17:35:16 +01:00
Erik Johnston
ebb46497ba
Add delta file
2016-09-08 16:38:54 +01:00
Mark Haines
fa9d36e050
Merge branch 'develop' into markjh/direct_to_device_federation
2016-09-08 13:43:43 +01:00
Mark Haines
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
Mark Haines
2ad72da931
Add tables for federated device messages
...
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
2016-09-06 15:10:40 +01:00
Erik Johnston
485d999c8a
Correctly delete old state groups in purge history API
2016-09-05 14:49:08 +01:00
Erik Johnston
a99e933550
Add upgrade script that will slowly prune state_groups_state entries
2016-09-05 10:05:36 +01:00
Erik Johnston
657847e4c6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-02 11:04:48 +01:00
Erik Johnston
9e25443db8
Move to storing state_groups_state as deltas
2016-09-01 14:31:26 +01:00
Erik Johnston
ef0cc648cf
Clean up old sent transactions
2016-08-31 11:12:02 +01:00
Mark Haines
e993925279
Add store-and-forward direct-to-device messaging
2016-08-25 17:35:37 +01:00
Erik Johnston
aad8a1a825
Delete old received_transactions
2016-08-22 16:29:46 +01:00
Erik Johnston
9da84a9a1e
Make AppserviceHandler stream events from database
...
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
2016-08-18 11:54:41 +01:00
Erik Johnston
732cf72b86
Fix push_display_name_rename schema update
2016-08-17 18:12:21 +01:00
David Baker
ad42322257
Add migration script
...
To port existing rule actions & enable entries to the new name
2016-08-16 16:56:30 +01:00
Erik Johnston
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
Erik Johnston
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
Richard van der Hoff
4fec5e57be
Default device_display_name to null
...
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
2016-08-03 11:53:00 +01:00
Richard van der Hoff
26cb0efa88
SQL syntax fix
2016-07-27 12:30:22 +01:00
Richard van der Hoff
2e3d90d67c
Make the device id on e2e key upload optional
...
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.
For active access_tokens without an associated device_id, we ought to register
the device in the devices table.
Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
2016-07-26 23:38:12 +01:00
