Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-12 06:24:19 -05:00
Code Issues Packages Projects Releases Wiki Activity
15,208 Commits 4 Branches 609 Tags 60 MiB
c8e6c308c6
Commit Graph

3233 Commits

Author SHA1 Message Date
Erik Johnston
5037326d66 Add indices. Remove room_ids accidentally added 
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.

We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
 2019-05-31 10:26:56 +01:00
Brendan Abolivier
847b9dcd1c Make max_delta equal to period * 10% 2019-05-31 09:54:46 +01:00
Erik Johnston
5ac75fc9a2 Join against events to use its room_id index 2019-05-30 15:26:55 +01:00
Erik Johnston
e2c46ed851 Move deletion from table inside txn 2019-05-30 15:26:38 +01:00
Erik Johnston
04710cc2d7 Fetch membership counts all at once 2019-05-30 15:25:41 +01:00
Erik Johnston
54d50fbfdf Get events all at once 2019-05-30 15:15:13 +01:00
Erik Johnston
6cdfb0207e Add index to temp table 2019-05-30 15:02:31 +01:00
Erik Johnston
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities 
Add DB bg update to cleanup extremities.
 2019-05-30 14:28:26 +01:00
Erik Johnston
cb967e2346
Update synapse/storage/events_bg_updates.py 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-05-30 14:06:42 +01:00
Erik Johnston
468bd090ff Rename constant 2019-05-30 11:24:42 +01:00
Erik Johnston
5c1ece0ffc Move event background updates to a separate file 2019-05-30 11:22:59 +01:00
Erik Johnston
640fcbb07f Fixup comments and logging 2019-05-30 10:55:55 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Erik Johnston
7e8e683754 Log actual number of entries deleted 2019-05-29 15:11:28 +01:00
Erik Johnston
d79c9994f4 Add DB bg update to cleanup extremities. 
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
 2019-05-29 15:11:26 +01:00
Erik Johnston
58c8ed5b0d
Correctly filter out extremities with soft failed prevs (#5274) 
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
 2019-05-29 11:56:24 +01:00
Brendan Abolivier
52839886d6
Allow configuring a range for the account validity startup job 
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
 2019-05-28 16:52:45 +01:00
Brendan Abolivier
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema 
Fix schema update for account validity
 2019-05-28 10:30:07 +01:00
Brendan Abolivier
ba17de7fbc
Fix schema update for account validity 2019-05-28 10:11:38 +01:00
Travis Ralston
bc4b2ecf70 Fix logging for room stats background update 2019-05-25 12:02:48 -06:00
Richard van der Hoff
b75537beaf Store key validity time in the storage layer 
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
 2019-05-23 11:52:22 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234) 
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
 2019-05-23 11:45:39 +01:00
Richard van der Hoff
85d1e03b9d
Simplifications and comments in do_auth (#5227) 
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
 2019-05-23 11:17:42 +01:00
Amber Brown
4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Erik Johnston
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender 
Only count aggregations from distinct senders
 2019-05-21 17:10:48 +01:00
Erik Johnston
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions 
Block attempts to annotate the same event twice
 2019-05-21 17:08:14 +01:00
Richard van der Hoff
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146) 
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
 2019-05-21 16:10:54 +01:00
Erik Johnston
7b0e804a4a Fix get_max_topological_token to never return None 2019-05-21 15:21:43 +01:00
Brendan Abolivier
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date 
Add startup background job for account validity
 2019-05-21 14:55:15 +01:00
Brendan Abolivier
5ceee46c6b
Do the select and insert in a single transaction 2019-05-21 13:38:51 +01:00
Erik Johnston
c7ec06e8a6 Block attempts to annotate the same event twice 2019-05-20 17:39:05 +01:00
Erik Johnston
1dff859d6a Rename relation types to match MSC 2019-05-20 14:31:19 +01:00
Erik Johnston
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base 
Land basic reaction and edit support.
 2019-05-20 14:06:40 +01:00
Erik Johnston
935af0da38 Correctly update aggregation counts after redaction 2019-05-20 12:09:27 +01:00
Brendan Abolivier
ad5b4074e1
Add startup background job for account validity 
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
 2019-05-17 19:37:31 +01:00
Erik Johnston
b63cc325a9 Only count aggregations from distinct senders 
As a user isn't allowed to send a single emoji more than once.
 2019-05-17 18:03:10 +01:00
Erik Johnston
8dd9cca8ea Spelling and clarifications 2019-05-17 16:40:51 +01:00
Erik Johnston
5dbff34509 Fixup bsaed on review comments 2019-05-17 15:48:04 +01:00
Erik Johnston
d46aab3fa8 Add basic editing support 2019-05-16 16:54:45 +01:00
Erik Johnston
895179a4dc Update docstring 2019-05-16 16:41:05 +01:00
Erik Johnston
7a7eba8302 Move parsing of tokens out of storage layer 2019-05-16 14:26:23 +01:00
Erik Johnston
2c662ddde4 Indirect tuple conversion 2019-05-16 14:21:39 +01:00
Erik Johnston
b5c62c6b26 Fix relations in worker mode 2019-05-16 10:38:13 +01:00
Erik Johnston
33453419b0 Add cache to relations 2019-05-16 10:02:14 +01:00
Erik Johnston
a0603523d2 Add aggregations API 2019-05-16 09:37:20 +01:00
Erik Johnston
5be34fc3e3 Actually check for None rather falsey 2019-05-15 17:30:23 +01:00
Erik Johnston
b50641e357 Add simple pagination API 2019-05-15 13:36:51 +01:00
Erik Johnston
efe3c7977a Add simple send_relation API and track in DB 2019-05-15 13:36:51 +01:00
Erik Johnston
54d77107c1 Make generating SQL bounds for pagination generic 
This will allow us to reuse the same structure when we paginate e.g.
relations
 2019-05-15 11:30:05 +01:00
Erik Johnston
52ddc6c0ed
Update docstring with correct type 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-05-15 09:52:15 +01:00
Erik Johnston
4fb44fb5b9 Expose DataStore._get_events as get_events_as_list 
This is in preparation for reaction work which requires it.
 2019-05-14 13:37:44 +01:00
Quentin Dufour
11ea16777f Limit the number of EDUs in transactions to 100 as expected by receiver (#5138) 
Fixes #3951.
 2019-05-09 11:01:41 +01:00
Richard van der Hoff
bd0d45ca69 Fix infinite loop in presence handler 
Fixes #5102
 2019-04-26 11:14:49 +01:00
Erik Johnston
ca90336a69 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration 2019-04-17 19:44:40 +01:00
Brendan Abolivier
eaf41a943b Add management endpoints for account validity 2019-04-17 19:34:45 +01:00
Brendan Abolivier
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration 
Send out emails with links to extend an account's validity period
 2019-04-17 14:57:39 +01:00
Brendan Abolivier
20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Erik Johnston
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta 
Fix schema upgrade when dropping tables
 2019-04-10 10:22:35 +01:00
Brendan Abolivier
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration 
Add time-based account expiration
 2019-04-09 17:02:41 +01:00
Brendan Abolivier
747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Erik Johnston
50d2a3059d Fix schema upgrade when dropping tables 
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.

Introduced in #4992.
 2019-04-09 14:39:18 +01:00
Richard van der Hoff
18b69be00f Rewrite Datastore.get_server_verify_keys 
Rewrite this so that it doesn't hammer the database.
 2019-04-09 00:00:10 +01:00
Richard van der Hoff
3352baac4b
Remove unused server_tls_certificates functions (#5028) 
These have been unused since #4120, and with the demise of perspectives, it is
unlikely that they will ever be used again.
 2019-04-08 21:50:18 +01:00
Neil Johnson
2d951686a7
drop tables listed in #1830 (#4992) 
Tables dropped: 
 * application_services, 
 * application_services_regex, 
 * transaction_id_to_pdu, 
 * stats_reporting
 * current_state_resets
 * event_content_hashes
 * event_destinations
 * event_edge_hashes
 * event_signatures
 * feedback
 * room_hosts
 * state_forward_extremities
 2019-04-08 15:37:26 +01:00
Richard van der Hoff
7fc1e17f4c
Merge pull request #5001 from matrix-org/rav/keyring_cleanups 
Cleanups in the Keyring
 2019-04-08 12:47:09 +01:00
Richard van der Hoff
b43d9a920b Fix docstring on get_server_keys_json 2019-04-04 18:54:03 +01:00
Amber Brown
a33a5abc4c
Clean up the database pagination code (#5007) 
* rewrite & simplify

* changelog

* cleanup potential sql injection
 2019-04-05 00:21:16 +11:00
Erik Johnston
4a2e13631d Add functions to delete a group 2019-04-03 16:29:52 +01:00
Andrew Morgan
4a4d5c4fd6
Fix grammar and document get_current_users_in_room (#4998) 2019-04-03 14:32:20 +01:00
Neil Johnson
e8419554ff
Remove presence lists (#4989) 
Remove presence list support as per MSC 1819
 2019-04-03 11:11:15 +01:00
Erik Johnston
8f549c1177
Merge pull request #4982 from matrix-org/erikj/msc1915 
Implement MSC1915 - 3PID unbind APIs
 2019-04-03 11:07:09 +01:00
Amber Brown
7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Erik Johnston
5bec8d660d Make starting pushers faster during start up 
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.

Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
 2019-04-02 16:59:13 +01:00
Richard van der Hoff
297bf2547e
Fix sync bug when accepting invites (#4956) 
Hopefully this time we really will fix #4422.

We need to make sure that the cache on
`get_rooms_for_user_with_stream_ordering` is invalidated *before* the
SyncHandler is notified for the new events, and we can now do so reliably via
the `events` stream.
 2019-04-02 12:42:39 +01:00
Erik Johnston
c75e2017f1 Fixup docstrings 2019-04-02 11:20:06 +01:00
Erik Johnston
3715c124b3 Grandfather in existing user threepids 
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
 2019-04-01 15:25:19 +01:00
Erik Johnston
1666c0696a Track IS used to bind 3PIDs 
This will then be used to know which IS to default to when unbinding the
threepid.
 2019-04-01 15:23:01 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969) 
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
 2019-04-01 10:24:38 +01:00
Richard van der Hoff
902cdc63b6
Merge pull request #4955 from matrix-org/rav/merge_state_into_events 
Combine the CurrentStateDeltaStream into the EventStream
 2019-03-28 18:32:13 +00:00
Amber Brown
2e060774ad
Run black on some storage modules that the stats branch touches (#4959) 2019-03-29 00:37:16 +11:00
Richard van der Hoff
4b91c313a9 Combine the CurrentStateDeltaStream into the EventStream 2019-03-27 22:07:05 +00:00
Amber Brown
903f04c21f
Use the state event amount for userdir import batching, not room count (#4944) 2019-03-27 02:49:28 +11:00
Richard van der Hoff
9bde730ef8
Fix bug where read-receipts lost their timestamps (#4927) 
Make sure that they are sent correctly over the replication stream.

Fixes: #4898
 2019-03-25 16:38:05 +00:00
Amber Brown
ac396a0d32
Refactor out state delta handling into its own class (#4917) 2019-03-25 20:37:08 +11:00
Erik Johnston
3ecec5ede2 Fix upsert 2019-03-21 10:21:15 +00:00
Erik Johnston
8d8834d3e7 comment block_room 2019-03-20 17:49:56 +00:00
Erik Johnston
67d618e111 Allow blocking a room multiple times 2019-03-20 16:50:05 +00:00
Amber Brown
4d53017432
Batching in the user directory import (#4900) 2019-03-21 03:06:36 +11:00
Erik Johnston
e9eeca1314 Fix user directory background update (#4887) 2019-03-19 22:13:53 +11:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Erik Johnston
9073cfc8bd
Merge pull request #4846 from matrix-org/hawkowl/userdir-search 
Improve searching in the userdir
 2019-03-14 13:54:15 +00:00
Andrew Morgan
d42c81d724
Transfer local user's push rules on room upgrade (#4838) 
Transfer push rules (notifications) on room upgrade
 2019-03-12 14:42:53 +00:00
Amber Brown
c0332d095f fixup 2019-03-13 01:30:54 +11:00
Amber Brown
797b6a63fc fixup 2019-03-13 01:17:51 +11:00
Amber Brown
81d9d1bee6 fixup 2019-03-12 22:28:48 +11:00
Amber Brown
6f5890b2fa fixup 2019-03-12 22:27:56 +11:00
Amber Brown
10480c4348 fixup 2019-03-12 21:47:14 +11:00
Amber Brown
8b618041ef fixup 2019-03-12 18:06:28 +11:00
Amber Brown
c980c7e31f use the old method 2019-03-12 17:51:14 +11:00
Amber Brown
78a6b950b3 fix 2019-03-12 00:50:28 +11:00
Amber Brown
1b77bd69fb pep8 2019-03-12 00:39:12 +11:00
Amber Brown
5ba8ceab4c fixes 2019-03-12 00:35:31 +11:00
Amber Brown
30a8deeb68 Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search 2019-03-11 21:12:30 +11:00
Amber Brown
4abc988c6a
initial 2019-03-11 21:11:36 +11:00
Erik Johnston
fe6c12e6cd Add comment to schema 2019-03-08 16:38:23 +00:00
Erik Johnston
5536ddba75 Make prev_state field optional 
The `prev_state` field on events is not specced and so synapse shouldn't
explode if an event is missing the field.

Fixes #4787
 2019-03-08 15:05:32 +00:00
Erik Johnston
8c4896668f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl 2019-03-08 11:44:20 +00:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Erik Johnston
6d13bdec91 Add docstrings from matrix-org-hotfixes 2019-03-06 17:24:26 +00:00
Erik Johnston
a9de04be72 Implement soft fail 2019-03-06 16:22:16 +00:00
Erik Johnston
b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room 
Stop backpaginating when events not visible
 2019-03-05 09:32:33 +00:00
Erik Johnston
c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys 
Allow /keys/{changes,query} API to run on worker
 2019-03-05 09:06:25 +00:00
Richard van der Hoff
2db49ea476 Add some debug about processing read receipts. 
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
 2019-03-04 18:19:40 +00:00
Erik Johnston
a84b8d56c2 Fixup slave stores 2019-03-04 18:04:57 +00:00
Erik Johnston
8b63fe4c26 s/get_forward_events/get_successor_events/ 2019-03-04 11:56:03 +00:00
Erik Johnston
fbc047f2a5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room 2019-03-04 11:54:58 +00:00
Erik Johnston
54f9ce11a7 Move /account/3pid to client_reader 2019-02-27 14:26:08 +00:00
Erik Johnston
6bb1c028f1 Limit cache invalidation replication line length (#4748) 2019-02-27 10:28:37 +00:00
Richard van der Hoff
f191be822b
Add database version to phonehome stats. (#4753) 2019-02-27 10:21:49 +00:00
Erik Johnston
80467bbac3 Fix state cache invalidation on workers 2019-02-22 14:38:14 +00:00
Erik Johnston
71b625d808 Stop backpaginating when events not visible 2019-02-20 18:14:12 +00:00
Amber Brown
a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Erik Johnston
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation 
Batch cache invalidation over replication
 2019-02-19 13:14:30 +00:00
Erik Johnston
62175a20e5 Docs 2019-02-19 11:38:40 +00:00
Erik Johnston
1bb35e3a83 Use itertools 2019-02-19 11:34:40 +00:00
Erik Johnston
bc8fa1509d Documentation 2019-02-19 11:24:59 +00:00
Erik Johnston
a9b5ea6fc1 Batch cache invalidation over replication 
Currently whenever the current state changes in a room invalidate a lot
of caches, which cause *a lot* of traffic over replication. Instead,
lets batch up all those invalidations and send a single poke down
the replication streams.

Hopefully this will reduce load on the master process by substantially
reducing traffic.
 2019-02-18 17:53:31 +00:00
Erik Johnston
4cc4400b4d Split /login into client_reader 2019-02-18 17:19:01 +00:00
Erik Johnston
eb2b8523ae Split out registration to worker 
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.

Note: due to the in-memory session map all registration requests must be
handled by the same worker.
 2019-02-18 12:12:57 +00:00
Erik Johnston
eaf4d11af9 Add configurable room list publishing rules 
This allows specifying who and what is allowed to be published onto the
public room list
 2019-02-14 16:02:23 +00:00
Amber Brown
bb4fd8f927
Run black on user directory code (#4635) 2019-02-13 23:05:32 +11:00
Erik Johnston
cf82338930
Merge pull request #4627 from matrix-org/erikj/user_ips_analyze 
Analyze user_ips before running deduplication
 2019-02-12 13:05:09 +00:00
Erik Johnston
495ea92350 Fix pep8 2019-02-12 12:40:42 +00:00
Erik Johnston
483ba85c7a Analyze user_ips before running deduplication 
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
 2019-02-12 11:55:27 +00:00
Erik Johnston
362d80b770 Reduce user_ips bloat during dedupe background update 
The background update to remove duplicate rows naively deleted and
reinserted the duplicates. For large tables with a large number of
duplicates this causes a lot of bloat (with postgres), as the inserted
rows are appended to the table, since deleted rows will not be
overwritten until a VACUUM has happened.

This should hopefully also help ensure that the query in the last batch
uses the correct index, as inserting a large number of new rows without
analyzing will upset the query planner.
 2019-02-12 11:39:34 +00:00
Erik Johnston
719e073f00
Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating 
add updating of backup versions
 2019-02-11 13:45:49 +00:00
Richard van der Hoff
56710c7df5
Fix 'no unique or exclusion constraint' error (#4591) 
Add more tables to the list of tables which need a background update to
complete before we can upsert into them, which fixes a race against the
background updates.
 2019-02-08 18:30:46 +00:00
Hubert Chathi
8248637173 add new endpoint to update backup versions 2019-02-06 17:57:10 -05:00
Andrew Morgan
627ecd358e Filter user directory state query to a subset of state events (#4462) 
* Filter user directory state query to a subset of state events

* Add changelog
 2019-02-05 12:16:28 +00:00
Andrew Morgan
c45fd0dda0
Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable 
Copy over non-federatable trait on room upgrade
 2019-01-31 20:28:37 +00:00
Andrew Morgan
d239f67c25 Raise an exception instead of returning None 2019-01-31 18:34:15 +00:00
Andrew Morgan
3ed3cb4339 New function for getting room's create event 2019-01-31 18:21:39 +00:00
Richard van der Hoff
e707e7b38d
Fix infinite loop when an event is redacted in a v3 room (#4535) 2019-01-31 15:34:17 +00:00
Amber Brown
3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Erik Johnston
a696c48133 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah 2019-01-29 22:00:33 +00:00
Erik Johnston
c21b7cbc09
Update synapse/storage/events_worker.py 2019-01-29 21:53:48 +00:00
Erik Johnston
4db252c073 Check redaction state when event is pulled out of the database 2019-01-29 21:48:36 +00:00
Erik Johnston
6d23ec2111 Fix typo 2019-01-29 21:45:53 +00:00
Erik Johnston
b5d510ad64 Remove unused arg 2019-01-29 21:45:28 +00:00
Richard van der Hoff
82165eeb05
Update synapse/storage/events_worker.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-01-29 21:14:39 +00:00
Erik Johnston
7709d2bd16 Implement rechecking of redactions 2019-01-29 11:56:20 +00:00
Erik Johnston
be47cfa9c9 Refactor event building into EventBuilder 
This is so that everything is done in one place, making it easier to
change the event format based on room version
 2019-01-29 11:13:00 +00:00
Amber Brown
94fb63e44f
Fix typo in upserts code (#4505) 
* fix obvious problem :|

* changelog
 2019-01-29 10:04:23 +00:00
Amber Brown
7072fe3084
Fix UPSERTs on SQLite 3.24+ (#4477) 2019-01-28 15:43:32 +00:00
Andrew Morgan
4026d555fa Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade 2019-01-28 14:08:24 +00:00
Erik Johnston
b6dce9b9fd
Merge pull request #4470 from matrix-org/erikj/require_format_version 
Require event format version to parse or create events
 2019-01-25 15:59:36 +00:00
Erik Johnston
431e485914 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert 2019-01-25 14:11:17 +00:00
Andrew Morgan
0b3fd1401f Don't require sqlite3 when using postgres (#4466) 2019-01-25 22:25:02 +11:00
Andrew Morgan
821b65aeb5 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade 2019-01-25 11:09:53 +00:00
Andrew Morgan
b1b6dba2d2
Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms 
Ability to search entire room history after upgrading room
 2019-01-25 10:49:58 +00:00
Erik Johnston
1953067136 Disable native upserts for sqlite, as they don't work 2019-01-25 10:46:49 +00:00
Erik Johnston
678a92cb56 Replace missed usages of FrozenEvent 2019-01-25 10:32:30 +00:00
Erik Johnston
be6a7e47fa
Revert "Require event format version to parse or create events" 2019-01-25 10:23:51 +00:00
Erik Johnston
efb8ed1d45 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version 2019-01-24 18:52:34 +00:00
Erik Johnston
80bcca659e
Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites 
Store rejected remote invite events as outliers
 2019-01-24 18:33:27 +00:00
Erik Johnston
664b7a2920
Merge pull request #4437 from matrix-org/erikj/event_format_version_v2 
Add support for persisting event format versions
 2019-01-24 18:32:30 +00:00
Erik Johnston
b8082a5445 Use term 'out of band membership' instead 2019-01-24 17:33:19 +00:00
Andrew Morgan
a383289b0d Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms 2019-01-24 17:23:51 +00:00
Andrew Morgan
e1781b043b Remove redundant create event None check 2019-01-24 17:23:39 +00:00
Richard van der Hoff
03c85335d1
Apply suggestions from code review 
Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>
 2019-01-24 17:22:09 +00:00
Amber Brown
0e27501ee5
Fix UPSERT check (#4459) 2019-01-24 22:57:41 +11:00
Erik Johnston
e8c9f15397 Replace missed usages of FrozenEvent 2019-01-24 11:14:07 +00:00
Amber Brown
58f6c48183
Use native UPSERTs where possible (#4306) 2019-01-24 21:31:54 +11:00
Erik Johnston
17898a5ba6 Merge branch 'erikj/fixup_rejecting_invites' of github.com:matrix-org/synapse into erikj/require_format_version 2019-01-23 20:13:07 +00:00
Erik Johnston
7c288c2250 Clarify the invite flows 2019-01-23 20:07:47 +00:00
Erik Johnston
886e5acc76 Store rejected remote invite events as outliers 
Currently they're stored as non-outliers even though the server isn't in
the room, which can be problematic in places where the code assumes it
has the state for all non outlier events.

In particular, there is an edge case where persisting the leave event
triggers a state resolution, which requires looking up the room version
from state. Since the server doesn't have the state, this causes an
exception to be thrown.
 2019-01-23 20:04:57 +00:00
Erik Johnston
2a8edbaf74 Merge branch 'erikj/get_pdu_versions' into erikj/require_format_version 2019-01-23 17:27:49 +00:00
Erik Johnston
be1065af59 isort 2019-01-23 11:48:16 +00:00
Erik Johnston
c5a296b10c Add support for persisting event format versions 
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.

We specify the format version rather than the room version as:

1. We don't necessarily know the room version, existing events may be
   either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
   events in the same room reported to be of different versions, which
   sounds annoying.
 2019-01-23 11:30:01 +00:00
Erik Johnston
90743c9d89 Fixup removal of duplicate user_ips rows (#4432) 
* Remove unnecessary ORDER BY clause

* Add logging

* Newsfile
 2019-01-23 19:45:18 +11:00
Erik Johnston
7f503f83b9 Refactor to rewrite the SQL instead 2019-01-22 16:31:05 +00:00
Erik Johnston
1c9704f8ab Don't shadow params 2019-01-22 16:20:33 +00:00
Erik Johnston
2557531f0f Fix bug when removing duplicate rows from user_ips 
This was caused by accidentally overwritting a `last_seen` variable
in a for loop, causing the wrong value to be written to the progress
table. The result of which was that we didn't scan sections of the table
when searching for duplicates, and so some duplicates did not get
deleted.
 2019-01-22 13:33:46 +00:00
Andrew Morgan
277e50462d Do not return in a deferred function 2019-01-22 12:40:26 +00:00
Andrew Morgan
c433f61091 Ensure new filter is actually created 2019-01-22 12:06:36 +00:00
Andrew Morgan
c9bfb058d8 Fix a bug with single-room search searching all rooms 
* Create a new method for getting predecessor rooms
* Remove formatting change
 2019-01-22 12:00:41 +00:00
Andrew Morgan
25d64a846a Fix typos 2019-01-22 11:00:04 +00:00
Erik Johnston
25dd56ace3 Fix race when persisting create event (#4404) 
* Fix race when persisting create event

When persisting a chunk of DAG it is sometimes requried to do a state
resolution, which requires knowledge of the room version. If this
happens while we're persisting the create event then we need to use that
event rather than attempting to look it up in the database.
 2019-01-18 23:17:04 +11:00
Andrew Morgan
df3a661e4a Search for messages across predecessor rooms 
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
 2019-01-18 11:19:20 +00:00
Amber Brown
a35c66a00b
Remove duplicates in the user_ips table and add an index (#4370) 2019-01-12 06:21:50 +11:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Neil Johnson
84b6fae1f5 Ensure synchrotrons can access is_support_user in the storage layer 2019-01-02 10:19:59 +00:00
Will Hunt
0708f437cc Log roomid along with Unknown room (#4297) 2018-12-24 10:49:42 +01:00
Neil Johnson
d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Amber Brown
e93a0ebf50
Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:10:31 +11:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Richard van der Hoff
9a3e24a13d drop undocumented dependency on dateutil (#4266) 
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 04:52:42 +11:00
Travis Ralston
158ffb92f1 Add an option to disable search for homeservers which may not be interested in it (#4230) 
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
 2018-12-04 12:01:02 +01:00
Ben Parsons
b5ac0ffa0a add more detail to logging regarding "More than one row matched" error (#4234) 2018-12-04 11:57:39 +01:00
Richard van der Hoff
ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english (#4253) 
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
 2018-12-04 11:55:52 +01:00
Aaron Raimist
704c5298f0
Drop sent_transactions 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2018-12-01 23:07:35 -06:00
Neil Johnson
f9b136a886 Neilj/fix mau initial reserved users (#4211) 
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail

* towncrier

* white space
 2018-11-28 20:33:41 +11:00
Richard van der Hoff
6c18cc4b50 Ignore __pycache__ directories in schema delta dir 
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
 2018-11-20 22:52:34 +00:00
Travis Ralston
835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
David Baker
bca3b91c2d Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers 2018-11-09 18:35:02 +00:00
David Baker
d44dea0223 pep8 2018-11-09 14:38:31 +00:00
David Baker
4f93abd62d add docs 2018-11-09 13:25:38 +00:00
Erik Johnston
5ebed18692 Lets convert bytes to unicode instead 2018-11-08 12:33:13 +00:00
Erik Johnston
b1a22b24ab Fix noop checks when updating device keys 
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.

The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
 2018-11-08 12:18:38 +00:00
Erik Johnston
bc80b3f454 Add helpers for getting prev and auth events (#4139) 
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
 2018-11-06 00:35:15 +11:00
Richard van der Hoff
350f654e7b Add unique indexes to a couple of tables 
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
 2018-11-02 10:36:13 +00:00
Richard van der Hoff
50e328d1e7 Remove redundant database locks for device list updates 
We can rely on the application-level per-user linearizer.
 2018-11-02 10:36:13 +00:00
Erik Johnston
0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups 
Delete unreferenced state groups during purge
 2018-10-30 16:58:22 +00:00
David Baker
e0934acdbb Cast to int here too 2018-10-30 11:12:23 +00:00
David Baker
12941f5f8b Cast bacjup version to int when querying 2018-10-30 11:01:07 +00:00
David Baker
2f0f911c52 Convert version back to a string 2018-10-30 10:35:18 +00:00
David Baker
64fa557f80 Try & make it work on postgres 2018-10-30 09:51:04 +00:00
David Baker
563f9b61b1 Make e2e backup versions numeric in the DB 
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
 2018-10-29 21:01:22 +00:00
Erik Johnston
88e5ffe6fe Deduplicate device updates sent over replication 
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
 2018-10-29 17:34:34 +00:00
Erik Johnston
ad88460e0d Move _find_unreferenced_groups 2018-10-29 14:24:19 +00:00
Erik Johnston
664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston
f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston
b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Erik Johnston
03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Richard van der Hoff
4cda300058 preserve room visibility 2018-10-25 19:10:24 +01:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Neil Johnson
95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init 
fix race condiftion in calling initialise_reserved_users
 2018-10-25 16:33:40 +01:00
Neil Johnson
fcbd488e9a add new line 2018-10-25 16:13:43 +01:00
Neil Johnson
f8fe98812b improve comments 2018-10-25 14:58:59 +01:00
Neil Johnson
f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Neil Johnson
ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson
07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Richard van der Hoff
7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8 
Fix a number of flake8 errors
 2018-10-24 11:23:35 +01:00
Erik Johnston
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase 
Add v2 state resolution algorithm
 2018-10-24 11:12:12 +01:00
Richard van der Hoff
ef771cc4c2 Fix a number of flake8 errors 
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
 2018-10-24 10:39:03 +01:00
Neil Johnson
329d18b39c remove white space 2018-10-23 15:27:20 +01:00
Neil Johnson
6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Amber Brown
e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Erik Johnston
67f7b9cb50 pep8 2018-10-19 16:06:59 +01:00
Erik Johnston
056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00
Erik Johnston
47a9da28ca Batch process handling state groups 2018-10-19 15:48:15 +01:00
Richard van der Hoff
fc0f13dd03 Fix incorrect truncation in get_missing_events 
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
 2018-10-16 21:10:04 +01:00
Erik Johnston
4a28d3d36f Update event_auth table for rejected events 2018-10-16 16:28:42 +01:00
Erik Johnston
e238013c44 Add v2 state res algorithm. 
We hook this up to the vdh test room version.
 2018-10-16 16:16:13 +01:00
David Baker
03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups 
E2E backups
 2018-10-15 10:19:43 +01:00
Erik Johnston
67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
David Baker
306361b31b Misc PR feedback bits 2018-10-12 11:48:56 +01:00
Richard van der Hoff
8ddd0f273c Comments on get_all_new_events_stream 
just some docstrings to clarify the behaviour here
 2018-10-12 09:55:41 +01:00
David Baker
dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker
497444f1fd Don't reuse backup versions 
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.

Fixes https://github.com/vector-im/riot-web/issues/7448
 2018-10-05 15:08:36 +01:00