Michael Kaye
39266a9c9f
Make user/room stats log line less verbose.
2019-10-24 17:55:53 +01:00
Erik Johnston
3aa2a90556
Move schema delta files to the correct data store.
...
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
Andrew Morgan
8831b04a53
1.5.0rc1
2019-10-24 12:47:58 +01:00
Neil Johnson
2794b79052
Option to suppress resource exceeded alerting ( #6173 )
...
The expected use case is to suppress MAU limiting on small instances
2019-10-24 11:48:46 +01:00
Andrew Morgan
92e88a71d3
Cleanup extra quotes from IDEs ( #6236 )
2019-10-23 16:49:05 +01:00
Erik Johnston
3ca4c7c516
Use new EventPersistenceStore
2019-10-23 16:15:03 +01:00
Erik Johnston
73cf63784b
Add DataStores and Storage classes.
2019-10-23 16:15:03 +01:00
Hubert Chathi
dc2cd6f79d
move get_e2e_cross_signing_key to EndToEndKeyWorkerStore so it works with workers
2019-10-23 09:13:47 -04:00
Erik Johnston
22a9847670
Move persist_events out from main data store.
...
This is in preparation for splitting out of state_groups_state from the
main store into it own one, as persisting events depends on calculating
state.
2019-10-23 13:29:44 +01:00
Andrew Morgan
409c62b27b
Add config linting script that checks for bool casing ( #6203 )
...
Add a linting script that enforces all boolean values in the default config be lowercase.
This has annoyed me for a while so I decided to fix it.
2019-10-23 13:22:54 +01:00
Hubert Chathi
480eac30eb
black
2019-10-22 22:37:16 -04:00
Hubert Chathi
404e8c8532
vendor-prefix the EDU name until MSC1756 is merged into the spec
2019-10-22 22:33:23 -04:00
Hubert Chathi
1fabf82d50
update to work with newer code, and fix formatting
2019-10-22 21:44:58 -04:00
Hubert Chathi
41ad35b523
add missing param
2019-10-22 19:06:29 -04:00
Hubert Chathi
cfdb84422d
make black happy
2019-10-22 19:06:06 -04:00
Hubert Chathi
a1aaf3eea6
don't crash if the user doesn't have cross-signing keys
2019-10-22 19:04:37 -04:00
Hubert Chathi
8d3542a64e
implement federation parts of cross-signing
2019-10-22 19:04:35 -04:00
Erik Johnston
c17efdc01c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores
2019-10-22 18:44:29 +01:00
Erik Johnston
23d62eded2
Clean up prepare_database.py a bit and add comments
2019-10-22 18:43:31 +01:00
Erik Johnston
6cc497f99b
Delete background_update table creation in main data_store
2019-10-22 18:02:50 +01:00
Erik Johnston
1bbc5444a8
Move README into synapse/storage/__init__.py
2019-10-22 17:59:31 +01:00
Hubert Chathi
2761731634
Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2
...
Cross-signing [3/4] -- uploading signatures edition
2019-10-22 17:05:06 +01:00
Erik Johnston
acf47c7698
Add a basic README to synapse.storage
2019-10-22 11:55:46 +01:00
Erik Johnston
bb6264be0b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores
2019-10-22 10:41:18 +01:00
Erik Johnston
ffd24545bb
Fix schema management to work with multiple data stores.
2019-10-21 16:08:40 +01:00
Erik Johnston
c66a06ac6b
Move storage classes into a main "data store".
...
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Richard van der Hoff
93eaeec75a
Remove Auth.check method ( #6217 )
...
This method was somewhat redundant, and confusing.
2019-10-18 19:43:36 +02:00
Hubert Chathi
36adfaedab
Merge branch 'develop' into cross-signing_sig_upload
2019-10-18 18:34:42 +01:00
Hubert Chathi
125eb45e19
fix doc strings
2019-10-18 16:56:16 +01:00
Hubert Chathi
0a9d22808c
Merge branch 'uhoreg/e2e_cross-signing_merged' into develop
2019-10-18 15:57:40 +01:00
Richard van der Hoff
2b22faded7
Merge remote-tracking branch 'origin/develop' into rav/event_auth/1
2019-10-18 12:35:13 +01:00
Richard van der Hoff
560c122267
Fix logging config for the docker image ( #6197 )
...
Turns out that loggers that are instantiated before the config is loaded get
turned off.
Also bring the logging config that is generated by --generate-config into line.
Fixes #6194 .
2019-10-18 13:34:33 +02:00
Hubert Chathi
cfc28325a6
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-10-18 11:39:40 +01:00
Brendan Abolivier
1ee97cbd01
Merge branch 'master' into develop
2019-10-18 11:00:43 +01:00
Hubert Chathi
f0f6a2b360
use the right function for when we're already in runInteraction
2019-10-18 10:56:54 +01:00
Erik Johnston
d98029ea89
Merge pull request #6196 from matrix-org/erikj/await
...
Move rest/admin to use async/await.
2019-10-18 11:53:02 +02:00
Brendan Abolivier
41b9faed16
1.4.1
2019-10-18 10:15:12 +01:00
Richard van der Hoff
80003dfcd5
Merge remote-tracking branch 'origin/develop' into rav/event_auth/1
2019-10-18 10:11:40 +01:00
Erik Johnston
5859a5c569
Fix presence timeouts when synchrotron restarts. ( #6212 )
...
* Fix presence timeouts when synchrotron restarts.
Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.
Fixes #3715 .
2019-10-18 06:42:26 +01:00
Richard van der Hoff
1ba359a11f
rip out some unreachable code
...
The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
2019-10-17 21:45:59 +01:00
Richard van der Hoff
70d9759cdf
Synapse 1.4.1rc1 (2019-10-17)
...
=============================
Bugfixes
--------
- Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185 ), [5b0e99485b0e9948eahttps://github.com/matrix-org/synapse/issues/6185 ), [5b0e99485b0e9948ea
2019-10-17 17:03:02 +01:00
Richard van der Hoff
71cd3fed66
1.4.1rc1
2019-10-17 16:40:56 +01:00
Erik Johnston
5b0e9948ea
Do the update as a background index
2019-10-17 16:31:42 +01:00
Erik Johnston
a2bb50c2eb
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
...
Fix inserting bytes as text in `censor_redactions`
2019-10-17 16:29:07 +01:00
Hubert Chathi
06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
...
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
Hubert Chathi
132b251e29
expand on comment
2019-10-11 14:24:52 -04:00
Erik Johnston
2e97a4c197
Port synapse/rest/client/_base.py to async/await
2019-10-11 15:29:26 +01:00
Erik Johnston
3c2d6c708c
Add maybe_awaitable and fix __init__ bugs
2019-10-11 15:26:09 +01:00
Hubert Chathi
691dd67fcd
Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version
...
make version optional in body of e2e backup version update
2019-10-11 10:11:59 -04:00
Erik Johnston
6f5c6c8f60
Merge pull request #6195 from matrix-org/erikj/opentracing_preview_url
...
Trace non-JSON APIs, /media, /key etc
2019-10-11 13:23:52 +01:00
Valérian Rousset
be9b55e0d2
cas: support setting display name ( #6114 )
...
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Erik Johnston
dfbb62c28d
Port synaps/rest/client/users.py to async/await
2019-10-11 12:21:40 +01:00
Erik Johnston
f95325e22a
Port synaps/rest/client/server_notice_servlet.py to async/await
2019-10-11 12:21:40 +01:00
Erik Johnston
281f887090
Port synaps/rest/client/media.py to async/await
2019-10-11 12:21:40 +01:00
Erik Johnston
fca3a541e7
Port rest/admin/__init__.py to async/await
2019-10-11 12:17:52 +01:00
Erik Johnston
f3ceaf4323
Trace non-JSON APIs, /media, /key etc
2019-10-11 11:58:52 +01:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. ( #6168 )
2019-10-11 09:38:26 +01:00
Hubert Chathi
7a0dce9259
make sure we actually return something
2019-10-10 20:31:30 -04:00
Hubert Chathi
4908fb3b30
make storage layer in charge of interpreting the device key data
2019-10-10 19:15:30 -04:00
Erik Johnston
83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any
...
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
Erik Johnston
b54b1e759a
Fix SQLite take 2
2019-10-10 16:19:40 +01:00
Erik Johnston
afb6d9d53b
Fix SQLite
2019-10-10 15:55:41 +01:00
Erik Johnston
3bc687508f
Remove add_in_list_sql_clause
2019-10-10 15:35:46 +01:00
Erik Johnston
ca3e01e50d
Fix store_url_cache using bytes
2019-10-10 14:52:29 +01:00
Erik Johnston
9d06fb9cb1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any
2019-10-10 14:38:21 +01:00
Erik Johnston
8bc529c04d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 14:30:51 +01:00
Erik Johnston
933034e2fe
Merge pull request #6127 from matrix-org/erikj/patch_inner
...
Add more log context checks when patching inlineCallbacks
2019-10-10 13:47:50 +01:00
Erik Johnston
fe1c1e6c28
Fixup comments
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-10 13:17:19 +01:00
Erik Johnston
b161786c14
Replace IN usage with helper funcs
2019-10-10 13:15:49 +01:00
Erik Johnston
b4fbf71187
Add helper funcs to use postgres ANY
...
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
Erik Johnston
91f43dca39
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 13:10:57 +01:00
werner291
b5b03b7079
Add domain validation when creating room with list of invitees ( #6121 )
2019-10-10 13:05:48 +01:00
Michael Telatynski
9a84d74417
before fulfilling a group invite,check if user is already joined/invited ( #3436 )
...
Fixes vector-im/riot-web#5645
2019-10-10 13:03:44 +01:00
krombel
2efd050c9d
send 404 as http-status when filter-id is unknown to the server ( #2380 )
...
This fixed the weirdness of 400 vs 404 as http status code in the case
the filter id is not known by the server.
As e.g. matrix-js-sdk expects 404 to catch this situation this leads
to unwanted behaviour.
2019-10-10 12:59:55 +01:00
Erik Johnston
59e0ed8306
Fix py3.5
2019-10-10 12:47:07 +01:00
Erik Johnston
c349e3ebaf
Fix py3.5
2019-10-10 12:29:38 +01:00
Erik Johnston
f735aeec65
sort
2019-10-10 12:20:29 +01:00
Erik Johnston
941edad583
Appease mypy
2019-10-10 12:15:17 +01:00
Erik Johnston
791a8c559b
Add coments
2019-10-10 11:53:57 +01:00
Erik Johnston
9970f955ce
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-10 11:51:50 +01:00
Richard van der Hoff
a139420a3c
Fix races in room stats (and other) updates. ( #6187 )
...
Hopefully this will fix the occasional failures we were seeing in the room directory.
The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.
We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
Richard van der Hoff
562b4e51dd
Rewrite the user_filter migration again ( #6184 )
...
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
Erik Johnston
128d5948c4
Fix packaging
2019-10-10 11:16:26 +01:00
Erik Johnston
ec0596f2ab
Log correct context
2019-10-10 11:11:38 +01:00
Erik Johnston
3e4272961a
Test for sentinel commit
2019-10-10 10:58:32 +01:00
Erik Johnston
1d6dd1c294
Move patch_inline_callbacks into synapse/
2019-10-10 10:53:06 +01:00
Andrew Morgan
da815c1f69
Move tag/push rules room upgrade checking ealier ( #6155 )
...
It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers.
The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).
2019-10-10 10:06:45 +01:00
Amber Brown
f743108a94
Refactor HomeserverConfig so it can be typechecked ( #6137 )
2019-10-10 09:39:35 +01:00
Hubert Chathi
4535a07f4a
make version optional in body of e2e backup version update
...
to agree with latest version of the MSC
2019-10-09 17:54:03 -04:00
Erik Johnston
5c1f886c75
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner
2019-10-09 16:52:21 +01:00
Erik Johnston
7f18b3d526
Do the update as a background index
2019-10-09 16:03:24 +01:00
Erik Johnston
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
...
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
Brendan Abolivier
59d6290ed9
Merge pull request #6178 from matrix-org/babolivier/factor_out_bg_updates
...
Factor out backgroung updates
2019-10-09 12:29:01 +01:00
Erik Johnston
1d3858371e
Disable bytes usage with postgres
...
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
Erik Johnston
e7631d84e6
Fix existing hex encoded json values in DB
2019-10-08 16:18:43 +01:00
Erik Johnston
ced4784592
Fix inserting bytes as text
2019-10-08 16:18:43 +01:00
Brendan Abolivier
c69324ffb5
Fix RegistrationStore
2019-10-08 14:48:33 +01:00
Brendan Abolivier
b1c0a4ceb3
Cleanup client_ips
2019-10-08 14:38:14 +01:00
Brendan Abolivier
8f1b385acc
Don't end up with 4 classes in registration
2019-10-08 14:36:33 +01:00
Anshul Angaria
474abf1eb6
add M_TOO_LARGE error code for uploading a too large file ( #6151 )
...
Fixes #6109
2019-10-08 13:55:16 +01:00
Andrew Morgan
ea7d938bca
Remove unused public room list timeout param ( #6179 )
...
* Remove unused public room list timeout param
* Add changelog
2019-10-08 13:51:25 +01:00
Richard van der Hoff
b94a401852
Fix /federation/v1/state for recent room versions ( #6170 )
...
* Fix /federation/v1/state for recent room versions
Turns out this endpoint was completely broken for v3 rooms. Hopefully this
re-signing code is irrelevant nowadays anyway.
2019-10-08 09:35:37 +01:00
Brendan Abolivier
66ebea1723
Lint
2019-10-07 17:44:41 +01:00
Brendan Abolivier
0496eafbf4
Move roommember's bg updates to a dedicated store
2019-10-07 17:35:01 +01:00
Brendan Abolivier
e106a0e4db
Move user_directory's bg updates to a dedicated store
2019-10-07 17:34:45 +01:00
Brendan Abolivier
cfccd2d78a
Move state's bg updates to a dedicated store
2019-10-07 17:34:39 +01:00
Brendan Abolivier
841054ad96
Move search's bg updates to a dedicated store
2019-10-07 17:34:35 +01:00
Brendan Abolivier
81e6ffb536
Move registration's bg updates to a dedicated store
2019-10-07 17:34:29 +01:00
Brendan Abolivier
54f87e0734
Move media_repository's bg updates to a dedicated store
2019-10-07 17:34:26 +01:00
Brendan Abolivier
cef9f6753e
Move devices's bg updates to a dedicated store
2019-10-07 17:34:20 +01:00
Brendan Abolivier
2d3b4f42f0
Move deviceinbox's bg updates to a dedicated store
2019-10-07 17:34:16 +01:00
Brendan Abolivier
88957199e7
Move client_ips's bg updates to a dedicated store
2019-10-07 17:34:12 +01:00
Erik Johnston
2cb7466993
Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure
...
Don't regenerate numeric user ID if registration fails.
2019-10-07 16:56:23 +01:00
Michael Kaye
dc795ba709
Log responder we are using. ( #6139 )
...
This prevents us logging "Responding to media request with responder %s".
2019-10-07 15:41:25 +01:00
Richard van der Hoff
276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test ( #6167 )
2019-10-07 14:41:39 +01:00
Brendan Abolivier
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
...
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
Brendan Abolivier
c8e6c308c6
Fix unique_user_filter_index schema update
2019-10-07 13:15:35 +01:00
Erik Johnston
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships
...
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
Erik Johnston
5119a4cac7
Fix bug where we didn't pull out event ID
2019-10-07 12:21:17 +01:00
Brendan Abolivier
ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
...
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
Brendan Abolivier
21d51ab598
Typo
2019-10-04 11:21:24 +01:00
Brendan Abolivier
4676732ca0
Lint
2019-10-04 11:18:28 +01:00
Brendan Abolivier
81d51ce48b
Incorporate review
2019-10-04 11:16:19 +01:00
Alexander Maznev
13c4345c84
Update user_filters table to have a unique index, and non-null columns ( #1172 )
2019-10-04 10:34:16 +01:00
Robert Swain
39b40d6d99
media/thumbnailer: Better quality for 1-bit / 8-bit color palette images ( #2142 )
...
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
Erik Johnston
6511071837
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships
2019-10-03 17:51:06 +01:00
Richard van der Hoff
66537e10ce
add some metrics on the federation sender ( #6160 )
2019-10-03 17:47:20 +01:00
Erik Johnston
ab8a64772b
Remove unused variable
2019-10-03 17:42:32 +01:00
Erik Johnston
693156aaf4
Don't regenerate numeric user ID if registration fails.
...
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
Erik Johnston
91f61fc6d7
Use the right error....
2019-10-03 17:28:31 +01:00
Erik Johnston
84691da6c3
pep8
2019-10-03 17:27:18 +01:00
Erik Johnston
a9610cdf02
Fixup names and comments
2019-10-03 17:26:56 +01:00
Erik Johnston
d89ebf7c25
cachedList descriptor doesn't like typing
2019-10-03 17:23:11 +01:00
Erik Johnston
c8145af8a9
Cache room membership lookups in _get_joined_users_from_context
2019-10-03 17:11:04 +01:00
Andrew Morgan
6018bec919
Merge branch 'master' into develop
2019-10-03 13:48:45 +01:00
Andrew Morgan
ecb69d824a
1.4.0
2019-10-03 13:22:44 +01:00
Andrew Morgan
0f46bf5737
Replace client_secret with <redacted> in server logs ( #6158 )
...
Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
2019-10-03 12:57:26 +01:00
Brendan Abolivier
6527fa18c1
Add test case
2019-10-03 11:24:36 +01:00
Erik Johnston
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
...
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
Erik Johnston
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
...
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
Erik Johnston
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
...
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
Erik Johnston
7a5f080f91
Fix appservice room list pagination
2019-10-02 15:47:22 +01:00
Erik Johnston
4c4f44930d
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 15:20:36 +01:00
Erik Johnston
03cf4385e0
Fix public room list pagination.
...
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
Erik Johnston
a5166e4d5f
Land improved room list based on room stats ( #6019 )
...
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Andrew Morgan
aec1377d0b
1.4.0rc2
2019-10-02 13:55:00 +01:00
Amber Brown
864f144543
Fix up some typechecking ( #6150 )
...
* type checking fixes
* changelog
2019-10-02 05:29:01 -07:00
Brendan Abolivier
972c9f65d7
Lint
2019-10-02 12:17:46 +01:00
Andrew Morgan
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade ( #6144 )
...
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.
For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.
Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
Erik Johnston
d69fd53f74
Bound find_next_generated_user_id DB query.
...
We can easily bound the set of user IDs we pull out of the DB, so lets
do that.
2019-10-02 11:45:31 +01:00
