Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-20 12:34:20 -05:00
Code Issues Packages Projects Releases Wiki Activity
13,778 Commits 4 Branches 609 Tags 60 MiB
1666c0696a
Commit Graph

2956 Commits

Author SHA1 Message Date
Erik Johnston
be1065af59 isort 2019-01-23 11:48:16 +00:00
Erik Johnston
c5a296b10c Add support for persisting event format versions 
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.

We specify the format version rather than the room version as:

1. We don't necessarily know the room version, existing events may be
   either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
   events in the same room reported to be of different versions, which
   sounds annoying.
 2019-01-23 11:30:01 +00:00
Erik Johnston
90743c9d89 Fixup removal of duplicate user_ips rows (#4432) 
* Remove unnecessary ORDER BY clause

* Add logging

* Newsfile
 2019-01-23 19:45:18 +11:00
Erik Johnston
7f503f83b9 Refactor to rewrite the SQL instead 2019-01-22 16:31:05 +00:00
Erik Johnston
1c9704f8ab Don't shadow params 2019-01-22 16:20:33 +00:00
Erik Johnston
2557531f0f Fix bug when removing duplicate rows from user_ips 
This was caused by accidentally overwritting a `last_seen` variable
in a for loop, causing the wrong value to be written to the progress
table. The result of which was that we didn't scan sections of the table
when searching for duplicates, and so some duplicates did not get
deleted.
 2019-01-22 13:33:46 +00:00
Andrew Morgan
277e50462d Do not return in a deferred function 2019-01-22 12:40:26 +00:00
Andrew Morgan
c433f61091 Ensure new filter is actually created 2019-01-22 12:06:36 +00:00
Andrew Morgan
c9bfb058d8 Fix a bug with single-room search searching all rooms 
* Create a new method for getting predecessor rooms
* Remove formatting change
 2019-01-22 12:00:41 +00:00
Andrew Morgan
25d64a846a Fix typos 2019-01-22 11:00:04 +00:00
Erik Johnston
25dd56ace3 Fix race when persisting create event (#4404) 
* Fix race when persisting create event

When persisting a chunk of DAG it is sometimes requried to do a state
resolution, which requires knowledge of the room version. If this
happens while we're persisting the create event then we need to use that
event rather than attempting to look it up in the database.
 2019-01-18 23:17:04 +11:00
Andrew Morgan
df3a661e4a Search for messages across predecessor rooms 
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
 2019-01-18 11:19:20 +00:00
Amber Brown
a35c66a00b
Remove duplicates in the user_ips table and add an index (#4370) 2019-01-12 06:21:50 +11:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Neil Johnson
84b6fae1f5 Ensure synchrotrons can access is_support_user in the storage layer 2019-01-02 10:19:59 +00:00
Will Hunt
0708f437cc Log roomid along with Unknown room (#4297) 2018-12-24 10:49:42 +01:00
Neil Johnson
d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Amber Brown
e93a0ebf50
Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:10:31 +11:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Richard van der Hoff
9a3e24a13d drop undocumented dependency on dateutil (#4266) 
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 04:52:42 +11:00
Travis Ralston
158ffb92f1 Add an option to disable search for homeservers which may not be interested in it (#4230) 
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
 2018-12-04 12:01:02 +01:00
Ben Parsons
b5ac0ffa0a add more detail to logging regarding "More than one row matched" error (#4234) 2018-12-04 11:57:39 +01:00
Richard van der Hoff
ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english (#4253) 
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
 2018-12-04 11:55:52 +01:00
Aaron Raimist
704c5298f0
Drop sent_transactions 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2018-12-01 23:07:35 -06:00
Neil Johnson
f9b136a886 Neilj/fix mau initial reserved users (#4211) 
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail

* towncrier

* white space
 2018-11-28 20:33:41 +11:00
Richard van der Hoff
6c18cc4b50 Ignore __pycache__ directories in schema delta dir 
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
 2018-11-20 22:52:34 +00:00
Travis Ralston
835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
David Baker
bca3b91c2d Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers 2018-11-09 18:35:02 +00:00
David Baker
d44dea0223 pep8 2018-11-09 14:38:31 +00:00
David Baker
4f93abd62d add docs 2018-11-09 13:25:38 +00:00
Erik Johnston
5ebed18692 Lets convert bytes to unicode instead 2018-11-08 12:33:13 +00:00
Erik Johnston
b1a22b24ab Fix noop checks when updating device keys 
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.

The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
 2018-11-08 12:18:38 +00:00
Erik Johnston
bc80b3f454 Add helpers for getting prev and auth events (#4139) 
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
 2018-11-06 00:35:15 +11:00
Richard van der Hoff
350f654e7b Add unique indexes to a couple of tables 
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
 2018-11-02 10:36:13 +00:00
Richard van der Hoff
50e328d1e7 Remove redundant database locks for device list updates 
We can rely on the application-level per-user linearizer.
 2018-11-02 10:36:13 +00:00
Erik Johnston
0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups 
Delete unreferenced state groups during purge
 2018-10-30 16:58:22 +00:00
David Baker
e0934acdbb Cast to int here too 2018-10-30 11:12:23 +00:00
David Baker
12941f5f8b Cast bacjup version to int when querying 2018-10-30 11:01:07 +00:00
David Baker
2f0f911c52 Convert version back to a string 2018-10-30 10:35:18 +00:00
David Baker
64fa557f80 Try & make it work on postgres 2018-10-30 09:51:04 +00:00
David Baker
563f9b61b1 Make e2e backup versions numeric in the DB 
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
 2018-10-29 21:01:22 +00:00
Erik Johnston
88e5ffe6fe Deduplicate device updates sent over replication 
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
 2018-10-29 17:34:34 +00:00
Erik Johnston
ad88460e0d Move _find_unreferenced_groups 2018-10-29 14:24:19 +00:00
Erik Johnston
664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston
f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston
b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Erik Johnston
03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Richard van der Hoff
4cda300058 preserve room visibility 2018-10-25 19:10:24 +01:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Neil Johnson
95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init 
fix race condiftion in calling initialise_reserved_users
 2018-10-25 16:33:40 +01:00
Neil Johnson
fcbd488e9a add new line 2018-10-25 16:13:43 +01:00
Neil Johnson
f8fe98812b improve comments 2018-10-25 14:58:59 +01:00
Neil Johnson
f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Neil Johnson
ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson
07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Richard van der Hoff
7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8 
Fix a number of flake8 errors
 2018-10-24 11:23:35 +01:00
Erik Johnston
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase 
Add v2 state resolution algorithm
 2018-10-24 11:12:12 +01:00
Richard van der Hoff
ef771cc4c2 Fix a number of flake8 errors 
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
 2018-10-24 10:39:03 +01:00
Neil Johnson
329d18b39c remove white space 2018-10-23 15:27:20 +01:00
Neil Johnson
6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Amber Brown
e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Erik Johnston
67f7b9cb50 pep8 2018-10-19 16:06:59 +01:00
Erik Johnston
056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00
Erik Johnston
47a9da28ca Batch process handling state groups 2018-10-19 15:48:15 +01:00
Richard van der Hoff
fc0f13dd03 Fix incorrect truncation in get_missing_events 
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
 2018-10-16 21:10:04 +01:00
Erik Johnston
4a28d3d36f Update event_auth table for rejected events 2018-10-16 16:28:42 +01:00
Erik Johnston
e238013c44 Add v2 state res algorithm. 
We hook this up to the vdh test room version.
 2018-10-16 16:16:13 +01:00
David Baker
03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups 
E2E backups
 2018-10-15 10:19:43 +01:00
Erik Johnston
67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
David Baker
306361b31b Misc PR feedback bits 2018-10-12 11:48:56 +01:00
Richard van der Hoff
8ddd0f273c Comments on get_all_new_events_stream 
just some docstrings to clarify the behaviour here
 2018-10-12 09:55:41 +01:00
David Baker
dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker
497444f1fd Don't reuse backup versions 
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.

Fixes https://github.com/vector-im/riot-web/issues/7448
 2018-10-05 15:08:36 +01:00
Neil Johnson
ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Erik Johnston
4917ff5523 Add state_group index to event_to_state_groups 
This is needed to efficiently check for unreferenced state groups during
purge.
 2018-10-04 16:03:08 +01:00
Erik Johnston
17d585753f Delete unreferened state groups during purge 2018-10-04 16:03:06 +01:00
Richard van der Hoff
9693625e55 actually exclude outliers 2018-10-03 10:19:41 +01:00
Richard van der Hoff
3e39783d5d remove debugging 2018-10-02 23:44:14 +01:00
Richard van der Hoff
ae61ade891 Fix bug in forward_extremity update logic 
An event does not stop being a forward_extremity just because an outlier or
rejected event refers to it.
 2018-10-02 23:36:08 +01:00
Erik Johnston
7258d081a5 Fix bug when invalidating destination retry timings 2018-10-02 15:47:57 +01:00
Matthew Hodgson
5b68f29f48 fix thinkos 2018-09-29 02:14:40 +01:00
Richard van der Hoff
9c8cec5dab Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache 2018-09-28 10:51:09 +01:00
Richard van der Hoff
51d33d5178
Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts 
fix #3854 MAU transaction errors
 2018-09-27 12:31:37 +01:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state (#3958) 
I spent ages trying to figure out how I was going mad...
 2018-09-27 11:22:25 +01:00
Neil Johnson
28781b65e7 fix #3854 2018-09-26 16:16:41 +01:00
Erik Johnston
fdd1a62e8d Add a five minute cache to get_destination_retry_timings 
Hopefully helps with #3931
 2018-09-21 14:56:12 +01:00
Amber Brown
1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Erik Johnston
bbab6ebfd9 Fix up changelog and remove spurious comment 2018-09-19 14:45:14 +01:00
Erik Johnston
392a54128c pep8 2018-09-19 14:37:49 +01:00
Erik Johnston
b9158ac2bf Remove get_destination_retry_timings cache 
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.

Fixes #3798
 2018-09-19 14:22:57 +01:00
Richard van der Hoff
85a43f4167 Return a 404 when deleting unknown room alias 
As per https://github.com/matrix-org/matrix-doc/issues/1675

Fixes https://github.com/matrix-org/synapse/issues/2782
 2018-09-17 13:19:00 +01:00
David Baker
bc74925c5b WIP e2e key backups 
Continues from uhoreg's branch

This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
 2018-09-13 17:02:59 +01:00
Erik Johnston
ed5331a627 comment 2018-09-13 16:10:56 +01:00
Erik Johnston
9cbd0094f0 pep8 2018-09-13 15:15:35 +01:00
Erik Johnston
9dbe38ea7d Create indices after insertion 2018-09-13 15:05:52 +01:00
Erik Johnston
c857f5ef9b Make purge history slightly faster 
Don't pull out events that are outliers and won't be deleted, as nothing
should happen to them.
 2018-09-13 12:48:10 +01:00
Neil Johnson
f30a303590
Merge pull request #3846 from matrix-org/neilj/expose-registered-users 
expose number of real reserved users
 2018-09-12 17:14:04 +01:00
Neil Johnson
8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson
0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown
33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring 
add some logging for the keyring queue
 2018-09-12 20:43:47 +10:00
Matthew Hodgson
b041115415
Speed up lazy loading (#3827) 
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
 2018-09-12 00:50:39 +01:00
Richard van der Hoff
cd7ef43872 clearer logging when things fail, too 2018-09-06 23:56:47 +01:00
Neil Johnson
84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Hubert Chathi
3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Neil Johnson
61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Neil Johnson
0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson
09f3cf1a7e ensure post registration auth checks do not fail erroneously 2018-08-31 15:42:51 +01:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Hubert Chathi
83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston
cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston
7a0da69eee Add missing yield 2018-08-23 10:28:12 +01:00
Erik Johnston
764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles 
Allow profile updates to happen on workers
 2018-08-22 11:35:55 +01:00
Erik Johnston
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler 
Refactor state module to support multiple room versions
 2018-08-22 10:04:55 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two (#3726) 
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
 2018-08-22 00:56:37 +02:00
Hubert Chathi
42a394caa2 allow session_data to be any JSON instead of just a string 2018-08-21 14:51:34 -04:00
Hubert Chathi
8550a7e9c2 allow auth_data to be any JSON instead of a string 2018-08-21 10:38:00 -04:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston
782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston
38f708a2bb Remote profile cache should remain in master worker 2018-08-17 11:37:42 +01:00
Neil Johnson
bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Matthew Hodgson
3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Erik Johnston
ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Neil Johnson
9ecbaf8ba8 adding missing yield 2018-08-14 16:55:28 +01:00
Neil Johnson
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block 
Block ability to read via sync if mau limit exceeded
 2018-08-14 15:21:31 +00:00
Amber Brown
591bf87c6a Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres 2018-08-14 20:56:23 +10:00
Amber Brown
99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
Matthew Hodgson
f0cede5556 missing import 2018-08-12 19:14:31 -04:00
Matthew Hodgson
66a4ca1d28 404 nicely if you try to interact with a missing current version 2018-08-12 19:14:31 -04:00
Matthew Hodgson
edc427a351 flake8 2018-08-12 19:14:31 -04:00
Matthew Hodgson
fe87890b18 implement remaining tests and make them work 2018-08-12 19:14:31 -04:00
Matthew Hodgson
15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson
b5eee511c7 don't needlessly return user_id 2018-08-12 19:14:31 -04:00
Matthew Hodgson
982edca380 fix flakes 2018-08-12 19:14:31 -04:00
Matthew Hodgson
234611f347 fix typos 2018-08-12 19:14:31 -04:00
Matthew Hodgson
9f0791b7bd add a tonne of docstring; make upload_room_keys properly assert version 2018-08-12 19:14:31 -04:00
Matthew Hodgson
8d14598e90 add storage docstring; remove unused set_e2e_room_keys 2018-08-12 19:14:31 -04:00
Matthew Hodgson
cac0253799 rename room_key_version table correctly, and fix opt args 2018-08-12 19:14:31 -04:00
Matthew Hodgson
0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson
69e51c7ba4 make /room_keys/version work 2018-08-12 19:14:31 -04:00
Matthew Hodgson
8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Matthew Hodgson
6b8c07abc2 make it work and fix pep8 2018-08-12 19:13:09 -04:00
Matthew Hodgson
0bc4627a73 interim WIP checkin; doesn't build yet 2018-08-12 18:23:10 -04:00
Matthew Hodgson
53ace904b2 total WIP skeleton for /room_keys API 2018-08-12 18:23:10 -04:00
Neil Johnson
31fa743567 fix sqlite/postgres incompatibility in reap_monthly_active_users 2018-08-11 22:38:34 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Neil Johnson
885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Erik Johnston
ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston
484a0ebdfc Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-09 10:16:29 +01:00
Erik Johnston
5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00