Richard van der Hoff
c9f11d09fc
Add missing index on users_in_public_rooms. ( #5894 )
2019-08-22 10:43:13 +01:00
Richard van der Hoff
4dab867288
Drop some unused tables. ( #5893 )
...
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
Hubert Chathi
b7f7cc7ace
add the version field to the index for e2e_room_keys
2019-08-14 17:14:40 -07:00
Hubert Chathi
f63ba7a795
Cross-signing [1/4] -- hidden devices ( #5759 )
...
* allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
Hubert Chathi
fac1cdc562
make changes from PR review
2019-08-01 21:51:19 -04:00
Hubert Chathi
336c546d6a
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-08-01 16:31:40 -04:00
Hubert Chathi
c1f0a5636c
Merge branch 'develop' into cross-signing_hidden
2019-07-30 23:33:06 -04:00
Hubert Chathi
bc95890822
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-07-30 23:21:26 -04:00
Hubert Chathi
781ade836b
apply changes from PR review
2019-07-30 23:09:50 -04:00
Erik Johnston
123c04daa7
Don't recreate column
2019-07-30 16:42:48 +01:00
Erik Johnston
15056ca208
Fix current_state_events membership background update.
...
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
2019-07-30 14:51:41 +01:00
Erik Johnston
105e7f6ed3
Remove lost comment
2019-07-29 16:09:48 +01:00
Hubert Chathi
c659b9f94f
allow uploading keys for cross-signing
2019-07-25 11:08:24 -04:00
Hubert Chathi
d1c7c2a98a
allow devices to be marked as "hidden"
...
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-07-24 23:21:52 -04:00
Erik Johnston
32768e96d4
Add function to get all forgotten rooms for user
...
This will allow us to efficiently filter out rooms that have been
forgotten in other queries without having to join against the
`room_memberships` table.
2019-07-24 11:44:23 +01:00
Erik Johnston
ebc5ed1296
Update comment for new column
2019-07-19 13:29:02 +01:00
Erik Johnston
c618a5d348
Add background update for current_state_events.membership column
2019-07-18 14:16:01 +01:00
Erik Johnston
6de09e07a6
Add membership column to current_state_events table.
...
It turns out that doing a join is surprisingly expensive for the DB to
do when room_membership table is larger than the disk cache.
2019-07-18 14:15:57 +01:00
Richard van der Hoff
5f158ec039
Implement access token expiry ( #5660 )
...
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Amber Brown
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
Brendan Abolivier
d0530382ee
Track deactivated accounts in the database ( #5378 )
2019-06-14 13:18:24 +01:00
Erik Johnston
1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
...
Make a full SQL schema
2019-06-06 18:35:38 +01:00
Amber H. Brown
ed872db8df
fix maybe
2019-06-07 02:53:47 +10:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server ( #5377 )
...
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.
This PR is a culmination of 3 smaller PRs which have each been separately reviewed:
* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Amber H. Brown
837e32ef55
just user dir?
2019-06-07 01:49:25 +10:00
Amber H. Brown
3b6645d3bf
remove background updates that arent needed
2019-06-07 01:20:58 +10:00
Amber Brown
6362e3af14
add more comments
2019-06-06 04:20:35 +10:00
Amber Brown
2198b7ce2a
add stuff in bg updates
2019-06-04 01:06:00 +10:00
Amber Brown
4e75c5e02a
WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING
2019-06-03 22:42:12 +10:00
Amber Brown
be452fc9ac
more fix
2019-06-03 22:24:23 +10:00
Amber Brown
7f81b967ca
fix schemas
2019-06-03 22:23:40 +10:00
Amber Brown
dc72b90cd6
full schema
2019-06-03 22:03:28 +10:00
Erik Johnston
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
...
Speed up room stats background update
2019-05-31 10:48:51 +01:00
Erik Johnston
5037326d66
Add indices. Remove room_ids accidentally added
...
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.
We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
Erik Johnston
6cdfb0207e
Add index to temp table
2019-05-30 15:02:31 +01:00
Erik Johnston
640fcbb07f
Fixup comments and logging
2019-05-30 10:55:55 +01:00
Erik Johnston
d79c9994f4
Add DB bg update to cleanup extremities.
...
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
Brendan Abolivier
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
...
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
Brendan Abolivier
ba17de7fbc
Fix schema update for account validity
2019-05-28 10:11:38 +01:00
Richard van der Hoff
b75537beaf
Store key validity time in the storage layer
...
This is a first step to checking that the key is valid at the required moment.
The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Amber Brown
4a30e4acb4
Room Statistics ( #4338 )
2019-05-21 11:36:50 -05:00
Erik Johnston
efe3c7977a
Add simple send_relation API and track in DB
2019-05-15 13:36:51 +01:00
Brendan Abolivier
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
...
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier
20f0617e87
Send out emails with links to extend an account's validity period
2019-04-17 14:42:20 +01:00
Erik Johnston
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta
...
Fix schema upgrade when dropping tables
2019-04-10 10:22:35 +01:00
Brendan Abolivier
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
...
Add time-based account expiration
2019-04-09 17:02:41 +01:00
Brendan Abolivier
747aa9f8ca
Add account expiration feature
2019-04-09 16:46:04 +01:00
Erik Johnston
50d2a3059d
Fix schema upgrade when dropping tables
...
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.
Introduced in #4992 .
2019-04-09 14:39:18 +01:00
Richard van der Hoff
3352baac4b
Remove unused server_tls_certificates functions ( #5028 )
...
These have been unused since #4120 , and with the demise of perspectives, it is
unlikely that they will ever be used again.
2019-04-08 21:50:18 +01:00
Neil Johnson
2d951686a7
drop tables listed in #1830 ( #4992 )
...
Tables dropped:
* application_services,
* application_services_regex,
* transaction_id_to_pdu,
* stats_reporting
* current_state_resets
* event_content_hashes
* event_destinations
* event_edge_hashes
* event_signatures
* feedback
* room_hosts
* state_forward_extremities
2019-04-08 15:37:26 +01:00
Neil Johnson
e8419554ff
Remove presence lists ( #4989 )
...
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
Erik Johnston
c75e2017f1
Fixup docstrings
2019-04-02 11:20:06 +01:00
Erik Johnston
3715c124b3
Grandfather in existing user threepids
...
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
2019-04-01 15:25:19 +01:00
Erik Johnston
1666c0696a
Track IS used to bind 3PIDs
...
This will then be used to know which IS to default to when unbinding the
threepid.
2019-04-01 15:23:01 +01:00
Erik Johnston
e9eeca1314
Fix user directory background update ( #4887 )
2019-03-19 22:13:53 +11:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task ( #4864 )
2019-03-19 04:50:24 +11:00
Amber Brown
797b6a63fc
fixup
2019-03-13 01:17:51 +11:00
Amber Brown
10480c4348
fixup
2019-03-12 21:47:14 +11:00
Amber Brown
30a8deeb68
Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search
2019-03-11 21:12:30 +11:00
Amber Brown
4abc988c6a
initial
2019-03-11 21:11:36 +11:00
Erik Johnston
fe6c12e6cd
Add comment to schema
2019-03-08 16:38:23 +00:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster ( #4537 )
2019-03-07 01:22:53 -08:00
Erik Johnston
483ba85c7a
Analyze user_ips before running deduplication
...
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
2019-02-12 11:55:27 +00:00
Erik Johnston
c5a296b10c
Add support for persisting event format versions
...
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.
We specify the format version rather than the room version as:
1. We don't necessarily know the room version, existing events may be
either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
events in the same room reported to be of different versions, which
sounds annoying.
2019-01-23 11:30:01 +00:00
Amber Brown
a35c66a00b
Remove duplicates in the user_ips table and add an index ( #4370 )
2019-01-12 06:21:50 +11:00
Neil Johnson
d2f7c4e6b1
create support user ( #4141 )
...
Allow for the creation of a support user.
A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
Aaron Raimist
704c5298f0
Drop sent_transactions
...
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2018-12-01 23:07:35 -06:00
David Baker
bca3b91c2d
Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers
2018-11-09 18:35:02 +00:00
Richard van der Hoff
350f654e7b
Add unique indexes to a couple of tables
...
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
2018-11-02 10:36:13 +00:00
David Baker
64fa557f80
Try & make it work on postgres
2018-10-30 09:51:04 +00:00
David Baker
563f9b61b1
Make e2e backup versions numeric in the DB
...
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
2018-10-29 21:01:22 +00:00
Erik Johnston
056f099126
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups
2018-10-19 15:48:59 +01:00
David Baker
497444f1fd
Don't reuse backup versions
...
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.
Fixes https://github.com/vector-im/riot-web/issues/7448
2018-10-05 15:08:36 +01:00
Erik Johnston
4917ff5523
Add state_group index to event_to_state_groups
...
This is needed to efficiently check for unreferenced state groups during
purge.
2018-10-04 16:03:08 +01:00
David Baker
bc74925c5b
WIP e2e key backups
...
Continues from uhoreg's branch
This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
2018-09-13 17:02:59 +01:00
Hubert Chathi
83caead95a
Merge branch 'develop' into e2e_backups
2018-08-24 11:44:26 -04:00
Matthew Hodgson
15d513f16f
fix idiocies and so make tests pass
2018-08-12 19:14:31 -04:00
Matthew Hodgson
0abb205b47
blindly incorporate PR review - needs testing & fixing
2018-08-12 19:14:31 -04:00
Matthew Hodgson
69e51c7ba4
make /room_keys/version work
2018-08-12 19:14:31 -04:00
Matthew Hodgson
8ae64b270f
implement /room_keys/version too (untested)
2018-08-12 19:14:31 -04:00
Matthew Hodgson
6b8c07abc2
make it work and fix pep8
2018-08-12 19:13:09 -04:00
Matthew Hodgson
53ace904b2
total WIP skeleton for /room_keys API
2018-08-12 18:23:10 -04:00
Neil Johnson
312ae74746
typos
2018-08-08 13:33:16 +01:00
Neil Johnson
16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
Neil Johnson
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
.
2018-08-02 10:59:58 +01:00
Neil Johnson
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
Neil Johnson
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
Neil Johnson
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
Neil Johnson
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
Richard van der Hoff
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
Richard van der Hoff
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Erik Johnston
244484bf3c
Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
...
This reverts commit 1d009013b3
.
2018-06-25 13:42:55 +01:00
Richard van der Hoff
1d009013b3
Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
...
This reverts commit ce0d911156
, reversing
changes made to b4a5d767a9
.
2018-06-22 16:35:10 +01:00
Richard van der Hoff
3ff8a619f5
UserErasureStore
...
to store which users have been erased
2018-06-12 09:52:22 +01:00
Neil Johnson
4986b084f8
remove unnecessary INSERT
2018-06-01 10:50:40 +01:00
Neil Johnson
ab0ef31dc7
create users index on creation_ts
2018-05-29 17:51:08 +01:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Neil Johnson
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
Richard van der Hoff
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Richard van der Hoff
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
David Baker
46df23f581
Add the schema file
2018-05-09 15:07:54 +01:00
Neil Johnson
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
Matthew Hodgson
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Matthew Hodgson
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
Matthew Hodgson
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
Matthew Hodgson
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
Adrian Tschira
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Neil Johnson
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
Matthew Hodgson
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
Luke Barnard
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
Luke Barnard
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
Luke Barnard
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
Luke Barnard
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
David Baker
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
David Baker
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
David Baker
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
David Baker
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
Neil Johnson
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
Erik Johnston
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
Erik Johnston
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
Erik Johnston
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
Erik Johnston
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
Richard van der Hoff
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Erik Johnston
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
Erik Johnston
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
Matthew Hodgson
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
Matthew Hodgson
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
Matthew Hodgson
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
Richard van der Hoff
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
Richard van der Hoff
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
Matthew Hodgson
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
Matthew Hodgson
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
Matthew Hodgson
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
Matthew Hodgson
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
Matthew Hodgson
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
Richard van der Hoff
356bcafc44
Remove the last vestiges of refresh_tokens
2017-10-31 20:35:58 +00:00
Richard van der Hoff
1650eb5847
DB schema interface for password auth providers
...
Provide an interface by which password auth providers can register db schema
files to be run at startup
2017-10-31 14:01:53 +00:00
Luke Barnard
007cd48af6
Recreate groups table instead of adding column
...
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
Luke Barnard
713e60b9b6
Awful hack to get default true
2017-10-26 17:38:14 +01:00
Luke Barnard
9b2feef9eb
Add is_public to groups table to allow for private groups
...
Prevent group API access to non-members for private groups
Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston
c2c47550f9
Fix schema delta versions
2017-10-11 13:23:15 +01:00
David Baker
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00