Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-22 18:24:58 -05:00
Code Issues Packages Projects Releases Wiki Activity
13,161 Commits 4 Branches 609 Tags 60 MiB
bf1e4d96ad
Commit Graph

2891 Commits

Author SHA1 Message Date
Erik Johnston
4a28d3d36f Update event_auth table for rejected events 2018-10-16 16:28:42 +01:00
Erik Johnston
e238013c44 Add v2 state res algorithm. 
We hook this up to the vdh test room version.
 2018-10-16 16:16:13 +01:00
David Baker
03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups 
E2E backups
 2018-10-15 10:19:43 +01:00
Erik Johnston
67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
David Baker
306361b31b Misc PR feedback bits 2018-10-12 11:48:56 +01:00
Richard van der Hoff
8ddd0f273c Comments on get_all_new_events_stream 
just some docstrings to clarify the behaviour here
 2018-10-12 09:55:41 +01:00
David Baker
dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker
497444f1fd Don't reuse backup versions 
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.

Fixes https://github.com/vector-im/riot-web/issues/7448
 2018-10-05 15:08:36 +01:00
Neil Johnson
ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Erik Johnston
4917ff5523 Add state_group index to event_to_state_groups 
This is needed to efficiently check for unreferenced state groups during
purge.
 2018-10-04 16:03:08 +01:00
Erik Johnston
17d585753f Delete unreferened state groups during purge 2018-10-04 16:03:06 +01:00
Richard van der Hoff
9693625e55 actually exclude outliers 2018-10-03 10:19:41 +01:00
Richard van der Hoff
3e39783d5d remove debugging 2018-10-02 23:44:14 +01:00
Richard van der Hoff
ae61ade891 Fix bug in forward_extremity update logic 
An event does not stop being a forward_extremity just because an outlier or
rejected event refers to it.
 2018-10-02 23:36:08 +01:00
Erik Johnston
7258d081a5 Fix bug when invalidating destination retry timings 2018-10-02 15:47:57 +01:00
Matthew Hodgson
5b68f29f48 fix thinkos 2018-09-29 02:14:40 +01:00
Richard van der Hoff
9c8cec5dab Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache 2018-09-28 10:51:09 +01:00
Richard van der Hoff
51d33d5178
Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts 
fix #3854 MAU transaction errors
 2018-09-27 12:31:37 +01:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state (#3958) 
I spent ages trying to figure out how I was going mad...
 2018-09-27 11:22:25 +01:00
Neil Johnson
28781b65e7 fix #3854 2018-09-26 16:16:41 +01:00
Erik Johnston
fdd1a62e8d Add a five minute cache to get_destination_retry_timings 
Hopefully helps with #3931
 2018-09-21 14:56:12 +01:00
Amber Brown
1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Erik Johnston
bbab6ebfd9 Fix up changelog and remove spurious comment 2018-09-19 14:45:14 +01:00
Erik Johnston
392a54128c pep8 2018-09-19 14:37:49 +01:00
Erik Johnston
b9158ac2bf Remove get_destination_retry_timings cache 
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.

Fixes #3798
 2018-09-19 14:22:57 +01:00
Richard van der Hoff
85a43f4167 Return a 404 when deleting unknown room alias 
As per https://github.com/matrix-org/matrix-doc/issues/1675

Fixes https://github.com/matrix-org/synapse/issues/2782
 2018-09-17 13:19:00 +01:00
David Baker
bc74925c5b WIP e2e key backups 
Continues from uhoreg's branch

This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
 2018-09-13 17:02:59 +01:00
Erik Johnston
ed5331a627 comment 2018-09-13 16:10:56 +01:00
Erik Johnston
9cbd0094f0 pep8 2018-09-13 15:15:35 +01:00
Erik Johnston
9dbe38ea7d Create indices after insertion 2018-09-13 15:05:52 +01:00
Erik Johnston
c857f5ef9b Make purge history slightly faster 
Don't pull out events that are outliers and won't be deleted, as nothing
should happen to them.
 2018-09-13 12:48:10 +01:00
Neil Johnson
f30a303590
Merge pull request #3846 from matrix-org/neilj/expose-registered-users 
expose number of real reserved users
 2018-09-12 17:14:04 +01:00
Neil Johnson
8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson
0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown
33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring 
add some logging for the keyring queue
 2018-09-12 20:43:47 +10:00
Matthew Hodgson
b041115415
Speed up lazy loading (#3827) 
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
 2018-09-12 00:50:39 +01:00
Richard van der Hoff
cd7ef43872 clearer logging when things fail, too 2018-09-06 23:56:47 +01:00
Neil Johnson
84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Hubert Chathi
3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Neil Johnson
61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Neil Johnson
0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson
09f3cf1a7e ensure post registration auth checks do not fail erroneously 2018-08-31 15:42:51 +01:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Hubert Chathi
83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston
cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston
7a0da69eee Add missing yield 2018-08-23 10:28:12 +01:00
Erik Johnston
764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles 
Allow profile updates to happen on workers
 2018-08-22 11:35:55 +01:00
Erik Johnston
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler 
Refactor state module to support multiple room versions
 2018-08-22 10:04:55 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two (#3726) 
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
 2018-08-22 00:56:37 +02:00
Hubert Chathi
42a394caa2 allow session_data to be any JSON instead of just a string 2018-08-21 14:51:34 -04:00
Hubert Chathi
8550a7e9c2 allow auth_data to be any JSON instead of a string 2018-08-21 10:38:00 -04:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston
782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston
38f708a2bb Remote profile cache should remain in master worker 2018-08-17 11:37:42 +01:00
Neil Johnson
bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Matthew Hodgson
3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Erik Johnston
ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Neil Johnson
9ecbaf8ba8 adding missing yield 2018-08-14 16:55:28 +01:00
Neil Johnson
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block 
Block ability to read via sync if mau limit exceeded
 2018-08-14 15:21:31 +00:00
Amber Brown
591bf87c6a Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres 2018-08-14 20:56:23 +10:00
Amber Brown
99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
Matthew Hodgson
f0cede5556 missing import 2018-08-12 19:14:31 -04:00
Matthew Hodgson
66a4ca1d28 404 nicely if you try to interact with a missing current version 2018-08-12 19:14:31 -04:00
Matthew Hodgson
edc427a351 flake8 2018-08-12 19:14:31 -04:00
Matthew Hodgson
fe87890b18 implement remaining tests and make them work 2018-08-12 19:14:31 -04:00
Matthew Hodgson
15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson
b5eee511c7 don't needlessly return user_id 2018-08-12 19:14:31 -04:00
Matthew Hodgson
982edca380 fix flakes 2018-08-12 19:14:31 -04:00
Matthew Hodgson
234611f347 fix typos 2018-08-12 19:14:31 -04:00
Matthew Hodgson
9f0791b7bd add a tonne of docstring; make upload_room_keys properly assert version 2018-08-12 19:14:31 -04:00
Matthew Hodgson
8d14598e90 add storage docstring; remove unused set_e2e_room_keys 2018-08-12 19:14:31 -04:00
Matthew Hodgson
cac0253799 rename room_key_version table correctly, and fix opt args 2018-08-12 19:14:31 -04:00
Matthew Hodgson
0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson
69e51c7ba4 make /room_keys/version work 2018-08-12 19:14:31 -04:00
Matthew Hodgson
8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Matthew Hodgson
6b8c07abc2 make it work and fix pep8 2018-08-12 19:13:09 -04:00
Matthew Hodgson
0bc4627a73 interim WIP checkin; doesn't build yet 2018-08-12 18:23:10 -04:00
Matthew Hodgson
53ace904b2 total WIP skeleton for /room_keys API 2018-08-12 18:23:10 -04:00
Neil Johnson
31fa743567 fix sqlite/postgres incompatibility in reap_monthly_active_users 2018-08-11 22:38:34 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Neil Johnson
885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Erik Johnston
ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston
484a0ebdfc Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-09 10:16:29 +01:00
Erik Johnston
5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00
Richard van der Hoff
8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions 
Support for room versioning
 2018-08-08 17:10:53 +01:00
Neil Johnson
be59910b93 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users 2018-08-08 13:45:21 +01:00
Neil Johnson
990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker 
API for monthly_active_users table
 2018-08-08 12:44:15 +00:00
Neil Johnson
312ae74746 typos 2018-08-08 13:33:16 +01:00
Neil Johnson
ef3589063a prevent total number of reserved users being too large 2018-08-07 17:59:27 +01:00
Neil Johnson
e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Erik Johnston
cd9765805e Allow ratelimiting on workers 2018-08-07 10:50:28 +01:00
Erik Johnston
495cb100d1 Allow profile changes to happen on workers 2018-08-07 10:50:26 +01:00
Richard van der Hoff
ca9bc1f4fe Fix occasional glitches in the synapse_event_persisted_position metric 
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
 2018-08-07 10:00:03 +01:00
Neil Johnson
a74b25faaa WIP building out mau reserved users 2018-08-06 23:25:25 +01:00
Neil Johnson
e54794f5b6 Fix postgres compatibility bug 2018-08-06 21:55:54 +01:00
Neil Johnson
16d78be315 make use of _simple_select_one_onecol, improved comments 2018-08-06 17:51:15 +01:00
Erik Johnston
96a9a29645 Pull in necessary stores in federation_reader 2018-08-06 15:23:57 +01:00
Erik Johnston
62ace05c45 Move necessary storage functions to worker classes 2018-08-06 15:23:38 +01:00
Neil Johnson
886be75ad1 bug fixes 2018-08-03 22:29:03 +01:00
Neil Johnson
e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Richard van der Hoff
0ca459ea33 Basic support for room versioning 
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
 2018-08-03 16:08:32 +01:00
Neil Johnson
950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson
897c51d274 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-03 13:40:47 +01:00
Neil Johnson
5e2f7b8084 typo 2018-08-02 22:47:48 +01:00
Neil Johnson
4ecb4bdac9 wip attempt at caching 2018-08-02 22:41:05 +01:00
Neil Johnson
74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Neil Johnson
9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Richard van der Hoff
704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff
14fa9d4d92 Avoid extra db lookups 
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
 2018-08-02 13:55:51 +01:00
Richard van der Hoff
0a65450d04 Validation for events/rooms in fed requests 
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
 2018-08-02 13:48:40 +01:00
Neil Johnson
00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson
4a6725d9d1 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-02 11:04:18 +01:00
Neil Johnson
165e067033 Revert "change monthly_active_users table to be a single column" 
This reverts commit ec716a35b2.
 2018-08-02 10:59:58 +01:00
Erik Johnston
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store 
Split out DB writes in federation handler
 2018-08-02 10:41:42 +01:00
Neil Johnson
08281fe6b7 self.db_conn unused 2018-08-01 23:26:24 +01:00
Neil Johnson
c21d82bab3 normalise reaping query 2018-08-01 23:24:38 +01:00
Neil Johnson
ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson
d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits 
Initial impl of capping MAU
 2018-08-01 15:58:45 +00:00
Neil Johnson
7ff44d9215 improve clarity 2018-08-01 16:17:00 +01:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson
303f1c851f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits 2018-08-01 13:42:50 +01:00
Erik Johnston
a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Neil Johnson
4e5ac901dd clean up 2018-08-01 12:03:57 +01:00
Neil Johnson
f9f5559971 fix comment 2018-08-01 12:03:42 +01:00
Neil Johnson
6023cdd227 remove errant print 2018-08-01 10:27:17 +01:00
Neil Johnson
7931393495 make count_monthly_users async synapse/handlers/auth.py 2018-08-01 10:21:56 +01:00
Neil Johnson
6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Richard van der Hoff
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy 
Make EventStore inherit from EventFederationStore
 2018-07-31 13:44:04 +01:00
Neil Johnson
df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Richard van der Hoff
0bc9b9e397 reinstate explicit include of EventsWorkerStore 2018-07-31 13:11:04 +01:00
Neil Johnson
fef7e58ac6 actually close conn 2018-07-30 22:29:44 +01:00
Neil Johnson
9b13817e06 factor out metrics from __init__ to app/homeserver 2018-07-30 22:07:07 +01:00
Neil Johnson
251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Erik Johnston
143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Matthew Hodgson
e9b2d047f6
make /context lazyload & filter aware (#3567) 
make /context lazyload & filter aware.
 2018-07-27 15:12:50 +01:00
Richard van der Hoff
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content 
Stop populating events.content
 2018-07-26 22:54:08 +01:00
Richard van der Hoff
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id 
Remove some redundant joins on event_edges.room_id
 2018-07-26 22:31:01 +01:00
Richard van der Hoff
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls 
Fix some looping_call calls which were broken in #3604
 2018-07-26 14:55:57 +01:00
Richard van der Hoff
51d7df1915 Create the column nullable 
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
 2018-07-26 14:54:04 +01:00
Richard van der Hoff
5c1d301fd9 Stop populating events.content 
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
 2018-07-26 14:43:02 +01:00
Richard van der Hoff
bd4b25f4d0 Remove some redundant joins on event_edges.room_id 
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.

event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
 2018-07-26 13:19:08 +01:00
Richard van der Hoff
1b4d73fa52 comment on event_edges 2018-07-26 12:53:51 +01:00
Richard van der Hoff
21e878ebb6 Make EventStore inherit from EventFederationStore 
(since it uses methods therein)

Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
 2018-07-26 12:48:51 +01:00
Richard van der Hoff
03751a6420 Fix some looping_call calls which were broken in #3604 
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
 2018-07-26 11:48:08 +01:00
Matthew Hodgson
bc7944e6d2 switch missing_types to be a bool 2018-07-25 23:36:31 +01:00
Matthew Hodgson
2565804030 Merge branch 'develop' into matthew/filter_members 2018-07-25 17:27:49 +01:00
Matthew Hodgson
7d9fb88617 incorporate more review. 2018-07-25 16:33:50 +01:00
Erik Johnston
78a691d005 Split out DB writes in federation handler 
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
 2018-07-25 16:22:56 +01:00