Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-24 00:49:22 -05:00
Code Issues Packages Projects Releases Wiki Activity
13,160 Commits 4 Branches 609 Tags 60 MiB
ef43a03fc5
Commit Graph

9396 Commits

Author SHA1 Message Date
Andrew Morgan
48951f437f Join logic covers both room creator and arbitrary users 2019-01-22 11:00:04 +00:00
Andrew Morgan
8c85f0833d tags, m.direct copying over correctly 2019-01-22 11:00:04 +00:00
Andrew Morgan
25d64a846a Fix typos 2019-01-22 11:00:04 +00:00
Andrew Morgan
ea8903fcc9 Migrating dm and room tags work for migrator 2019-01-22 11:00:04 +00:00
Andrew Morgan
887ca93a1b Prevent crash on user who doesn't have any direct rooms 2019-01-22 11:00:04 +00:00
Andrew Morgan
4ff6d22245 Preserve DM status of a room on upgrade 
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
 2019-01-22 11:00:04 +00:00
Andrew Morgan
75942af1db Fix typo 2019-01-22 11:00:04 +00:00
Richard van der Hoff
33a55289cb
Refactor and bugfix for resove_service (#4427) 2019-01-22 10:59:27 +00:00
Amber Brown
23b0813599
Require ECDH key exchange & remove dh_params (#4429) 
* remove dh_params and set better cipher string
 2019-01-22 21:58:50 +11:00
Neil Johnson
1b53cc3cb4 fix line length 2019-01-21 15:17:20 +00:00
Neil Johnson
5349262302 Config option to disable requesting MSISDN on registration 2019-01-21 14:59:37 +00:00
Erik Johnston
35e1d67b4e Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_v2_invite_server 2019-01-21 14:04:19 +00:00
Erik Johnston
5f54765587
Merge pull request #4390 from matrix-org/erikj/versioned_fed_apis 
Add groundwork for new versions of federation APIs
 2019-01-21 11:44:05 +00:00
Andrew Morgan
702c4b750c Migrate encryption state on room upgrade (#4411) 
* Migrate encryption state on room upgrade

Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>

* Add changelog file
 2019-01-21 20:42:58 +11:00
Erik Johnston
25dd56ace3 Fix race when persisting create event (#4404) 
* Fix race when persisting create event

When persisting a chunk of DAG it is sometimes requried to do a state
resolution, which requires knowledge of the room version. If this
happens while we're persisting the create event then we need to use that
event rather than attempting to look it up in the database.
 2019-01-18 23:17:04 +11:00
Richard van der Hoff
de6888e7ce Remove redundant WrappedConnection (#4409) 
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
 2019-01-18 23:07:38 +11:00
Andrew Morgan
df3a661e4a Search for messages across predecessor rooms 
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
 2019-01-18 11:19:20 +00:00
Richard van der Hoff
676cf2ee26
Fix incorrect logcontexts after a Deferred was cancelled (#4407) 2019-01-17 14:00:23 +00:00
Richard van der Hoff
9feb5d0b71
sign_request -> build_auth_headers (#4408) 
Just got very confused about the fact that the headers are only an output, not
an input.
 2019-01-17 12:40:09 +00:00
Richard van der Hoff
3982a6ee07
Changing macaroon_secret_key no longer logs you out (#4387) 2019-01-16 23:14:41 +00:00
Richard van der Hoff
05e1296649
don't store more remote device lists if they have more than 1K devices (#4397) 2019-01-16 23:14:11 +00:00
Andrej Shadura
64cf6788d9
Depend on pymacaroons >= 0.13.0 instead on pymacaroons-pynacl 
Since 0.13.0, pymacaroons works correctly with pynacl, so there
isn’t any more reason to depend on an outdated pynacl fork.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
 2019-01-16 10:54:41 +01:00
Andrej Shadura
fab948120f
Use msgpack instead of msgpack-python 
The package msgpack-python has been deprecated.

Signed-off-by: Andrej Shadura <andrew.shadura@collabora.co.uk>
 2019-01-16 10:53:49 +01:00
Neil Johnson
9ec56d6935 ALL_USER_TYPES should be a tuple 2019-01-15 14:38:15 +00:00
Erik Johnston
4a4d2e17bc Add /v2/invite federation API 2019-01-15 13:22:44 +00:00
Erik Johnston
bb63e7ca4f Add groundwork for new versions of federation APIs 2019-01-15 11:14:34 +00:00
Amber Brown
a35c66a00b
Remove duplicates in the user_ips table and add an index (#4370) 2019-01-12 06:21:50 +11:00
Amber Brown
522dada206 Merge remote-tracking branch 'origin/master' into develop 2019-01-11 02:22:48 +11:00
Amber Brown
6dc06c3775 version 2019-01-11 01:56:37 +11:00
Richard van der Hoff
e0910d0145 Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1 
Fixes #4371
 2019-01-10 14:12:50 +00:00
Richard van der Hoff
ba41aeed6a Revert "Fix macaroon_secret_key fallback logic" 
This is already fixed in 0.34.1, by 59f93bb

This reverts commit efc522c55e.
 2019-01-10 14:09:26 +00:00
Richard van der Hoff
aa70d24125 Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34 2019-01-10 12:58:33 +00:00
Richard van der Hoff
efc522c55e Fix macaroon_secret_key fallback logic 2019-01-10 12:57:27 +00:00
Richard van der Hoff
566947ff34 Skip macaroon check for access tokens in the db 2019-01-10 12:57:21 +00:00
Richard van der Hoff
353f2407b7 Fix fallback to signing key for macaroon-secret-key 2019-01-10 12:42:56 +00:00
Richard van der Hoff
d566e6b17a Merge branch 'master' into develop 2019-01-09 15:00:46 +00:00
Richard van der Hoff
998f5225c1 0.34.1 2019-01-09 14:53:54 +00:00
Erik Johnston
55c3e853c5
Merge pull request #4368 from matrix-org/erikj/better_errors 
Fixup docstrings for matrixfederationclient
 2019-01-09 11:39:28 +00:00
Erik Johnston
484867d35d Merge branch 'develop' into release-v0.34.1 2019-01-09 11:38:13 +00:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Erik Johnston
34ea14139d Fixup docstrings for matrixfederationclient 2019-01-09 09:25:59 +00:00
Erik Johnston
d422570e9b Use RequestSendFailed when fail to parse content type headers 2019-01-08 14:22:18 +00:00
Erik Johnston
1371d5b798 Don't log stack traces for HTTP error responses 2019-01-08 12:28:30 +00:00
Richard van der Hoff
ba85786d71 0.34.1rc1 2019-01-08 11:15:34 +00:00
Erik Johnston
b970cb0e96 Refactor request sending to have better excpetions (#4358) 
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
 2019-01-08 11:04:28 +00:00
Amber Brown
1dcb086f33
Fix synapse.config.__main__ on python 3 (#4356) 2019-01-08 10:03:09 +11:00
Travis Ralston
bc1fa8cd01 Add GET account data routes (#4303) 
As per https://github.com/matrix-org/matrix-doc/issues/1339
 2019-01-07 10:27:54 +00:00
Richard van der Hoff
b7c0218812
Check jinja version for consent resource (#4327) 
* Raise a ConfigError if an invalid resource is specified

* Require Jinja 2.9 for the consent resource

* changelog
 2019-01-07 10:14:31 +00:00
Michael Telatynski
a27e501b09 fix the check for whether is_url to match all the other ones in codebase (#3405) 
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
 2019-01-06 21:43:25 +00:00
Matthew Hodgson
27128145e6 fix NPE in /messages by checking if all events were filtered out (#4330) 2019-01-02 23:37:39 +00:00
Neil Johnson
84b6fae1f5 Ensure synchrotrons can access is_support_user in the storage layer 2019-01-02 10:19:59 +00:00
Amber Brown
d7843f47b6
Remove v1 only REST APIs now we don't ship matrix console (#4334) 2018-12-29 23:12:30 +11:00
Will Hunt
0708f437cc Log roomid along with Unknown room (#4297) 2018-12-24 10:49:42 +01:00
Richard van der Hoff
cd018e3543
Merge pull request #4307 from matrix-org/erikj/v2_rooms 
Add v2 room version
 2018-12-24 10:48:59 +01:00
Richard van der Hoff
f85676cc93
Return well_known in /login response (#4319) 
... as per MSC1730.
 2018-12-24 10:44:33 +01:00
Richard van der Hoff
9c2af7b2c5 Add a script to generate a clean config file (#4315) 2018-12-22 02:04:57 +11:00
Richard van der Hoff
f3561f8d86 Fix indentation in default config (#4313) 
These settings are not supposed to be under 'listeners'.
 2018-12-22 02:01:43 +11:00
Amber Brown
ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Amber Brown
c26f49a664
Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298) 2018-12-22 01:37:26 +11:00
Richard van der Hoff
0b26feb422 Merge branch 'master' into develop 2018-12-20 16:48:47 +00:00
Richard van der Hoff
6dd7271243 Prepare 0.34 release 2018-12-20 12:07:15 +00:00
Richard van der Hoff
d3c9c562c6 Debian packaging via dh_virtualenv 2018-12-20 11:35:23 +00:00
Richard van der Hoff
1a6d5bfa08 Debian packaging via dh_virtualenv (#4285) 2018-12-20 11:33:29 +00:00
David Baker
1c0051114a Add 'sandbox' to CSP for media repo (#4284) 
* Add 'sandbox' to the CSP for media repo

* Changelog
 2018-12-20 11:09:18 +00:00
Nathan Pennie
81b513416e
Fixed line length 2018-12-19 14:12:33 -05:00
Nathan Pennie
668e6625b0
Fix the variable names used for account_data 2018-12-19 06:53:02 -05:00
Erik Johnston
df89f8afb8 Add v2 room version 2018-12-18 18:10:37 +00:00
Neil Johnson
7e22cd90f5
ensure can report mau stats when hs.config.mau_stats_only is set (#4305) 
* ensure can report mau stats when hs.config.mau_stats_only is set
 2018-12-18 14:36:11 +00:00
Neil Johnson
d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Nad Chishtie
128294d3e5 Improved welcome page (#4294) 
* Improved static/index.html look, feel and content.

* Added accessibility tags to in-line SVG.

* Grammar.
 2018-12-14 05:13:56 +11:00
Amber Brown
c7144b105c Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:13:33 +11:00
Amber Brown
e93a0ebf50
Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:10:31 +11:00
Nad Chishtie
cef8ae272a Improved welcome page (#4294) 
* Improved static/index.html look, feel and content.

* Added accessibility tags to in-line SVG.

* Grammar.
 2018-12-14 02:42:51 +11:00
Richard van der Hoff
f208f608cb Merge branch 'release-v0.34.0' into develop 2018-12-11 15:43:20 +00:00
Richard van der Hoff
eb835bf65b 0.34.0rc2 2018-12-11 15:35:47 +00:00
Richard van der Hoff
188945713e
Merge pull request #4290 from matrix-org/rav/remove_webclient 
Stop installing Matrix Console by default
 2018-12-11 16:24:15 +01:00
Richard van der Hoff
95c1f6500b
Merge pull request #4289 from matrix-org/rav/welcome_page 
Add a welcome page to the static resources
 2018-12-11 16:24:06 +01:00
Richard van der Hoff
f537432ef9 Add a welcome page to the static resources 
This is largely a precursor for the removal of the bundled webclient. The idea
is to present a page at / which reassures people that something is working, and
to give them some links for next steps.

The welcome page lives at `/_matrix/static/`, so is enabled alongside the other
`static` resources (which, in practice, means the client API is enabled). We'll
redirect to it from `/` if we have nothing better to display there.

It would be nice to have a way to disable it (in the same way that you might
disable the nginx welcome page), but I can't really think of a good way to do
that without a load of ickiness.

It's based on the work done by @krombel for #2601.
 2018-12-11 13:26:22 +00:00
Richard van der Hoff
df96177ca7 Stop installing Matrix Console by default 
This is based on the work done by @krombel in #2601.
 2018-12-11 13:20:33 +00:00
David Baker
89ac2a5bdb Add 'sandbox' to CSP for media repo (#4284) 
* Add 'sandbox' to the CSP for media repo

* Changelog
 2018-12-11 04:05:02 +11:00
Will Hunt
fee831c040 Move imports to one line 2018-12-10 13:52:33 +00:00
Will Hunt
466c1f3e01
Use send_cors 2018-12-10 13:11:37 +00:00
Will Hunt
91206e09f2 changelog & isort 2018-12-09 17:39:44 +00:00
Will Hunt
dbf736ba66
Make /config more CORS-y 2018-12-09 13:27:22 +00:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Richard van der Hoff
c7401a697f
Implement SAML2 authentication (#4267) 
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.

What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
 2018-12-07 13:11:11 +01:00
Richard van der Hoff
c588b9b9e4
Factor SSO success handling out of CAS login (#4264) 
This is mostly factoring out the post-CAS-login code to somewhere we can reuse
it for other SSO flows, but it also fixes the userid mapping while we're at it.
 2018-12-07 13:10:07 +01:00
Richard van der Hoff
b0c24a66ec Rip out half-implemented m.login.saml2 support (#4265) 
* Rip out half-implemented m.login.saml2 support

This was implemented in an odd way that left most of the work to the client, in
a way that I really didn't understand. It's going to be a pain to maintain, so
let's start by ripping it out.

* drop undocumented dependency on dateutil

It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 19:44:38 +11:00
Richard van der Hoff
9a3e24a13d drop undocumented dependency on dateutil (#4266) 
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 04:52:42 +11:00
Richard van der Hoff
e8d98466b0
Implement .well-known handling (#4262) 
Sometimes it's useful for synapse to generate its own .well-known file.
 2018-12-05 14:38:58 +01:00
Richard van der Hoff
e55983defe Prepare 0.34.0rc1 2018-12-04 13:52:16 +00:00
Travis Ralston
1737753a62 Add an option to enable recording IPs for appservice users (#3831) 2018-12-04 12:44:41 +01:00
Amber Brown
fd96dd75a3 Fix non-ASCII pushrules (#4248) 2018-12-04 12:44:02 +01:00
Travis Ralston
158ffb92f1 Add an option to disable search for homeservers which may not be interested in it (#4230) 
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
 2018-12-04 12:01:02 +01:00
Ben Parsons
b5ac0ffa0a add more detail to logging regarding "More than one row matched" error (#4234) 2018-12-04 11:57:39 +01:00
Richard van der Hoff
ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english (#4253) 
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
 2018-12-04 11:55:52 +01:00
Richard van der Hoff
f144c0a210
Merge pull request #4244 from aaronraimist/drop-sent-txt 
Drop sent_transactions
 2018-12-04 11:41:28 +01:00
Richard van der Hoff
a077e710a3
Merge pull request #4250 from matrix-org/hawkowl/pusher-remove-py3 
Fix removing pushers on python 3
 2018-12-04 11:22:46 +01:00
Richard van der Hoff
52e87fbfbe
Run the AS senders as background processes (#4189) 
This should fix some "Starting db connection from sentinel context" warnings,
and will mean we get metrics for these processes.
 2018-12-04 10:53:49 +01:00
Amber Brown
d3c61ef906 fix type error 2018-12-03 22:27:41 +11:00
Richard van der Hoff
c03324294d Workaround for non-ascii event ids (#4241) 
It turns out that we accept events with non-ascii IDs, which would later cause
an explosion during state res.

Fixes #4226
 2018-12-03 21:47:48 +11:00
Aaron Raimist
704c5298f0
Drop sent_transactions 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2018-12-01 23:07:35 -06:00
Neil Johnson
7039ece8fb Neilj/fix autojoin (#4223) 
* Fix auto join failures for servers that require user consent

* Fix auto join failures for servers that require user consent
 2018-11-28 22:24:57 +11:00
Neil Johnson
f9b136a886 Neilj/fix mau initial reserved users (#4211) 
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail

* towncrier

* white space
 2018-11-28 20:33:41 +11:00
Richard van der Hoff
944d524f18 Support m.login.sso (#4220) 
* Clean up the CSS for the fallback login form

I was finding this hard to work with, so simplify a bunch of things. Each
flow is now a form inside a div of class login_flow.

The login_flow class now has a fixed width, as that looks much better than each
flow having a differnt width.

* Support m.login.sso

MSC1721 renames m.login.cas to m.login.sso. This implements the change
(retaining support for m.login.cas for older clients).

* changelog
 2018-11-27 18:51:52 +11:00
Richard van der Hoff
6c18cc4b50 Ignore __pycache__ directories in schema delta dir 
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
 2018-11-20 22:52:34 +00:00
Neil Johnson
78ba0e7ab8 Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207) 2018-11-20 12:29:25 +01:00
Richard van der Hoff
416c671474
Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes 
Fix some logcontext leaks
 2018-11-20 12:19:19 +01:00
Amber Brown
31425d82a3 Merge remote-tracking branch 'origin/master' into develop 2018-11-19 12:55:25 -06:00
Amber Brown
d102e19e47 version 2018-11-19 12:42:49 -06:00
Amber Brown
80cac86b2c
Fix fallback auth on Python 3 (#4197) 2018-11-19 12:27:33 -06:00
Richard van der Hoff
f5faf6bc14 Fix logcontext leak in EmailPusher 2018-11-19 17:07:01 +00:00
Travis Ralston
ab4526a153 Remove duplicate slashes in generated consent URLs 2018-11-15 20:41:53 -07:00
Amber Brown
8b1affe7d5
Fix Content-Disposition in media repository (#4176) 2018-11-15 15:55:58 -06:00
Travis Ralston
835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Amber Brown
df758e155d
Use <meta> tags to discover the per-page encoding of html previews (#4183) 2018-11-15 11:05:08 -06:00
Neil Johnson
bf648c37e7 release 0.33.9rc1 2018-11-14 11:45:52 +00:00
Richard van der Hoff
c1efcd7c6a Add a test for the public T&Cs form 2018-11-14 10:46:27 +00:00
Richard van der Hoff
83a5f459aa Fix an internal server error when viewing the public privacy policy 2018-11-14 10:21:07 +00:00
David Baker
0869566ad3
Merge pull request #4113 from matrix-org/dbkr/e2e_backup_versions_are_numbers 
Make e2e backup versions numeric in the DB
 2018-11-14 07:55:48 +00:00
David Baker
bca3b91c2d Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers 2018-11-09 18:35:02 +00:00
Erik Johnston
db5a1c059a
Merge pull request #4166 from matrix-org/erikj/drop_unknown_events 
Drop incoming events from federation for unknown rooms
 2018-11-09 17:59:34 +00:00
Erik Johnston
dc59ad5334 Remove hack to support rejoining rooms 2018-11-09 14:58:09 +00:00
David Baker
d44dea0223 pep8 2018-11-09 14:38:31 +00:00
David Baker
4f93abd62d add docs 2018-11-09 13:25:38 +00:00
Erik Johnston
30dd27afff Simplify to always drop events if server isn't in the room 2018-11-09 11:36:45 +00:00
Richard van der Hoff
3cecf5340d
Update synapse/federation/federation_server.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2018-11-09 11:28:25 +00:00
Richard van der Hoff
9bce065a53
Update synapse/federation/federation_server.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2018-11-09 11:28:22 +00:00
David Baker
d3fa6194f7 Remove unnecessary str() 2018-11-09 11:11:31 +00:00
Erik Johnston
7b22421a7b
Merge pull request #4164 from matrix-org/erikj/fix_device_comparison 
Fix noop checks when updating device keys
 2018-11-08 14:37:20 +00:00
Erik Johnston
5ebed18692 Lets convert bytes to unicode instead 2018-11-08 12:33:13 +00:00
Erik Johnston
b1a22b24ab Fix noop checks when updating device keys 
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.

The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
 2018-11-08 12:18:38 +00:00
Erik Johnston
9417986f77 Drop PDUs of unknown rooms 
When we receive events over federation we will need to know the room
version to be able to correctly handle them, e.g. once we start changing
event formats. Currently, we attempt to handle events in unknown rooms.
 2018-11-08 12:11:20 +00:00
Richard van der Hoff
0a1fc52971 fix parse_string docstring 2018-11-08 11:12:29 +00:00
hera
2b075fb03a Fix encoding error for consent form on python3 
The form was rendering this as "b'01234....'".

-- richvdh
 2018-11-08 11:05:39 +00:00
Amber Brown
b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00
Travis Ralston
0f5e51f726 Add config variables for enabling terms auth and the policy name (#4142) 
So people can still collect consent the old way if they want to.
 2018-11-06 10:32:34 +00:00
Hubert Chathi
f1087106cf
handle empty backups according to latest spec proposal (#4123) 
fixes #4056
 2018-11-05 17:59:29 -05:00
Amber Brown
efdcbbe46b
Tests for user consent resource (#4140) 2018-11-06 05:53:44 +11:00
Erik Johnston
bc80b3f454 Add helpers for getting prev and auth events (#4139) 
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
 2018-11-06 00:35:15 +11:00
Amber Brown
0467384d2f
Set the encoding to UTF8 in the default logconfig (#4138) 2018-11-03 02:28:07 +11:00
Erik Johnston
90d713b8c6
Merge pull request #4137 from matrix-org/erikj/clean_up_events 
Clean up event accesses and tests
 2018-11-02 14:12:49 +00:00
Erik Johnston
b86d05a279 Clean up event accesses and tests 
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
 2018-11-02 13:44:14 +00:00
Amber Brown
cb7a6b2379
Fix typing being reset causing infinite syncs (#4127) 2018-11-03 00:19:23 +11:00
Richard van der Hoff
efb9343c8c
Merge pull request #4132 from matrix-org/rav/fix_device_list_locking 
Fix locked upsert on device_lists_remote_cache
 2018-11-02 10:50:53 +00:00
Richard van der Hoff
00f12e00f8
Merge pull request #4133 from matrix-org/travis/fix-terms-auth 
Fix logic error that prevented guests from seeing the privacy policy
 2018-11-02 10:50:43 +00:00
Erik Johnston
b199534518
Merge pull request #4135 from matrix-org/erikj/fix_state_res_none 
Fix None exception in state res v2
 2018-11-02 10:45:57 +00:00
Richard van der Hoff
350f654e7b Add unique indexes to a couple of tables 
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
 2018-11-02 10:36:13 +00:00
Richard van der Hoff
50e328d1e7 Remove redundant database locks for device list updates 
We can rely on the application-level per-user linearizer.
 2018-11-02 10:36:13 +00:00
Erik Johnston
54aec35867 Fix None exception in state res v2 2018-11-02 10:29:19 +00:00
Travis Ralston
642505abc3 Fix logic error that prevented guests from seeing the privacy policy 2018-11-01 16:48:32 -06:00
Travis Ralston
c68aab1536
Merge pull request #4004 from matrix-org/travis/login-terms 
Add m.login.terms to the registration flow
 2018-11-01 11:03:38 -06:00
Erik Johnston
1b21e771d0
Merge pull request #4128 from matrix-org/erikj/state_res_v2_version 
Add STATE_V2_TEST room version
 2018-11-01 13:17:57 +00:00
Erik Johnston
b3dd6fa981 Add STATE_V2_TEST room version 2018-11-01 11:43:46 +00:00
Amber Brown
073d400b84 Merge branch 'master' into develop 2018-11-01 21:32:12 +11:00
Amber Brown
aa98e38896 version 2018-11-01 21:28:35 +11:00
Travis Ralston
a8d41c6aff Include a version query string arg for the consent route 2018-10-31 13:19:28 -06:00
Travis Ralston
d1e7b9c44c Merge branch 'develop' into travis/login-terms 2018-10-31 13:15:14 -06:00
Richard van der Hoff
1729ba1650
Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades 
Attempt to move room aliases on room upgrades
 2018-10-31 17:52:18 +00:00
Richard van der Hoff
0f8591a5a8 Avoid else clause on exception for clarity 2018-10-31 15:43:57 +00:00
Richard van der Hoff
94c7fadc98 Attempt to move room aliases on room upgrades 2018-10-31 15:43:57 +00:00
Richard van der Hoff
9b827c40ca
Log some bits about event creation (#4121) 
I found these helpful in debugging my room upgrade tests.
 2018-10-31 15:42:23 +00:00
Amber Brown
916efc8249
Remove fetching keys via the deprecated v1 kex method (#4120) 2018-10-31 23:14:39 +11:00
Amber Brown
f79f454485
Remove deprecated v1 key exchange endpoint (#4119) 2018-10-31 22:29:02 +11:00
Amber Brown
67c1924899 version bump 2018-10-31 06:27:05 +11:00
Amber Brown
086e1a8f3e Fix search 500ing (#4122) 2018-10-31 06:26:46 +11:00
Amber Brown
3bade14ec0
Fix search 500ing (#4122) 2018-10-31 04:33:41 +11:00
Amber Brown
2e223a8c22
Remove the unused /pull federation API (#4118) 2018-10-31 04:24:59 +11:00
Erik Johnston
0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups 
Delete unreferenced state groups during purge
 2018-10-30 16:58:22 +00:00
Amber Brown
0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
David Baker
e0934acdbb Cast to int here too 2018-10-30 11:12:23 +00:00
David Baker
12941f5f8b Cast bacjup version to int when querying 2018-10-30 11:01:07 +00:00
David Baker
2f0f911c52 Convert version back to a string 2018-10-30 10:35:18 +00:00
David Baker
64fa557f80 Try & make it work on postgres 2018-10-30 09:51:04 +00:00
David Baker
563f9b61b1 Make e2e backup versions numeric in the DB 
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
 2018-10-29 21:01:22 +00:00
Erik Johnston
88e5ffe6fe Deduplicate device updates sent over replication 
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
 2018-10-29 17:34:34 +00:00
Erik Johnston
a163b748a5 Don't truncate command name in metrics 2018-10-29 17:34:21 +00:00
Erik Johnston
ad88460e0d Move _find_unreferenced_groups 2018-10-29 14:24:19 +00:00
Erik Johnston
664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston
f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston
b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Amber Brown
4cd1c9f2ff
Delete the disused & unspecced identicon functionality (#4106) 2018-10-29 23:57:24 +11:00
Richard van der Hoff
7fbfea062e
Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar 
Remember to copy the avatar on room upgrades
 2018-10-29 12:49:21 +00:00
Richard van der Hoff
56ca578f77
Merge pull request #4099 from matrix-org/rav/upgrade_odd_pls 
Better handling of odd PLs during room upgrades
 2018-10-29 12:48:51 +00:00
Richard van der Hoff
bf33eed609
Merge pull request #4091 from matrix-org/rav/room_version_upgrades 
Room version upgrade support
 2018-10-29 12:47:20 +00:00
Amber Brown
c4b3698a80
Make the replication logger quieter (#4108) 2018-10-29 22:59:44 +11:00
Amber Brown
2b791865c4 version bump 2018-10-29 21:52:52 +11:00
Richard van der Hoff
db24d7f15e Better handling of odd PLs during room upgrades 
Fixes handling of rooms where we have permission to send the tombstone, but not
other state. We need to (a) fail more gracefully when we can't send the PLs in
the old room, and (b) not set the PLs in the new room until we are done with
the other stuff.
 2018-10-27 00:54:26 +01:00
Richard van der Hoff
5caf79b312 Remember to copy the avatar on room upgrades 2018-10-26 23:56:40 +01:00
Richard van der Hoff
54bbe71867 optimise state copying 2018-10-26 22:51:34 +01:00
Richard van der Hoff
193cadc988 Address review comments 
Improve comments, get old room state from the context we already have
 2018-10-26 17:10:30 +01:00
Erik Johnston
03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Amber Brown
77d70a7646
Port register_new_matrix_user to Python 3 and add tests (#4085) 2018-10-26 22:05:22 +11:00
Richard van der Hoff
e6babc27d5 restrict PLs in old room 2018-10-25 19:18:25 +01:00
Richard van der Hoff
3a263bf3ae copy state 2018-10-25 19:18:25 +01:00
Richard van der Hoff
1b9f253e20 preserve PLs 2018-10-25 19:10:24 +01:00
Richard van der Hoff
4cda300058 preserve room visibility 2018-10-25 19:10:24 +01:00
Richard van der Hoff
0f7d1c9906 Basic initial support for room upgrades 
Currently just creates a new, empty, room, and sends a tombstone in the old
room.
 2018-10-25 19:10:24 +01:00
Richard van der Hoff
e1948175ee Allow power_level_content_override=None for _send_events_for_new_room 2018-10-25 19:10:24 +01:00
Richard van der Hoff
7f7b2cd3de Make room_member_handler a member of RoomCreationHandler 
... to save passing it into `_send_events_for_new_room`
 2018-10-25 19:10:18 +01:00
Richard van der Hoff
871c4abfec Factor _generate_room_id out of create_room 
we're going to need this for room upgrades.
 2018-10-25 18:23:09 +01:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Erik Johnston
c85e063302
Merge pull request #4051 from matrix-org/erikj/alias_disallow_list 
Add config option to control alias creation
 2018-10-25 17:04:59 +01:00
Neil Johnson
95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init 
fix race condiftion in calling initialise_reserved_users
 2018-10-25 16:33:40 +01:00
Neil Johnson
fcbd488e9a add new line 2018-10-25 16:13:43 +01:00
Erik Johnston
b94a43d5b5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list 2018-10-25 15:25:31 +01:00
Erik Johnston
e5481b22aa Use allow/deny 2018-10-25 15:25:21 +01:00
Neil Johnson
f8fe98812b improve comments 2018-10-25 14:58:59 +01:00
Neil Johnson
f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Travis Ralston
54def42c19 Merge branch 'develop' into travis/login-terms 2018-10-24 13:22:59 -06:00
Neil Johnson
ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson
663d9db8e7 commit transaction before closing 2018-10-24 17:17:30 +01:00
Neil Johnson
07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Neil Johnson
9ec2186586 isort 2018-10-24 16:09:21 +01:00
Neil Johnson
9532caf6ef remove trailing whiter space 2018-10-24 16:08:25 +01:00
Richard van der Hoff
94a49e0636
fix tuple 
Co-Authored-By: neilisfragile <neil@matrix.org>
 2018-10-24 14:39:23 +01:00
Richard van der Hoff
3ad359e5be Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation 2018-10-24 11:23:49 +01:00
Richard van der Hoff
7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8 
Fix a number of flake8 errors
 2018-10-24 11:23:35 +01:00
Erik Johnston
3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase 
Add v2 state resolution algorithm
 2018-10-24 11:12:12 +01:00
Richard van der Hoff
7e07d25ed6 Allow backslashes in event field filters 
Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which
meant that single backslashes were not allowed in event field filters.

The intention here is to allow single-backslashes, but disallow
double-backslashes.
 2018-10-24 11:11:24 +01:00
Richard van der Hoff
ef771cc4c2 Fix a number of flake8 errors 
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
 2018-10-24 10:39:03 +01:00
Erik Johnston
47a9ba435d Use match rather than search 2018-10-24 09:54:57 +01:00
Richard van der Hoff
e0b9d5f0af
Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts 
Clean up the way logcontexts and threads work in the pushers
 2018-10-24 09:53:57 +01:00
Erik Johnston
dacbeb2e03 Comment 2018-10-24 09:47:49 +01:00
Erik Johnston
810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Erik Johnston
cb23aa4c42
Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation 
Clean up room alias creation
 2018-10-24 09:40:27 +01:00
Richard van der Hoff
c573794b22 Fix start_pushers vs _start_pushers confusion 2018-10-24 09:24:55 +01:00
Richard van der Hoff
e564306e31 sanity-check the is_processing flag 
... and rename it, for even more sanity
 2018-10-24 09:23:33 +01:00
Neil Johnson
a67d8ace9b remove errant exception and style 2018-10-23 17:44:39 +01:00
Travis Ralston
43c3f0b02f
Merge pull request #3969 from turt2live/travis/fix-federated-group-requests 
Handle HttpResponseException more safely for federated groups
 2018-10-23 10:41:04 -06:00
Travis Ralston
3e704822be
Comments help 2018-10-23 10:25:31 -06:00
Neil Johnson
329d18b39c remove white space 2018-10-23 15:27:20 +01:00
Neil Johnson
6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Richard van der Hoff
b3f6dddad2
Give some more things logcontexts (#4077) 2018-10-23 14:29:17 +01:00
Richard van der Hoff
5c445114d3
Correctly account for cpu usage by background threads (#4074) 
Wrap calls to deferToThread() in a thing which uses a child logcontext to
attribute CPU usage to the right request.

While we're in the area, remove the logcontext_tracer stuff, which is never
used, and afaik doesn't work.

Fixes #4064
 2018-10-23 13:12:32 +01:00
Richard van der Hoff
026cd91ac8 Run PusherPool.start as a background process 
We don't do anything with the result, so this is needed to give this code a
logcontext.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
f749607c91 Make on_started synchronous too 
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
e7a16c6210 Remove redundant run_as_background_process() from pusherpool 
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
c7273c11bc Give pushers their own background logcontext 
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
5110f4e425 move get_all_pushers call down 
simplifies the interface to _start_pushers
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
04277d0ed8 Factor PusherPool._start_pusher out of _start_pushers 
... and use it from start_pusher_by_id. This mostly simplifies
start_pusher_by_id.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
3e8b02c939 Rename _refresh_pusher 
This is public (or at least, called from outside the class), so ought to have a
better name.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff
7aea00069c
Merge pull request #4076 from matrix-org/rav/fix_init_logcontexts 
Run MAU queries as background processes
 2018-10-22 14:46:59 +01:00
Richard van der Hoff
911db96658
Merge pull request #4073 from matrix-org/rav/require_psutil 
Make psutil an explicit dependency
 2018-10-22 12:33:21 +01:00
Richard van der Hoff
a6f421e812 Run MAU queries as background processes 
Fixes #3820
 2018-10-20 02:14:35 +01:00
Amber Brown
e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Richard van der Hoff
e5b52d0f94 Make psutil an explicit dependency 
As of #4027, we require psutil to be installed, so it should be in our
dependency list. We can also remove some of the conditional import code
introduced by #992.

Fixes #4062.
 2018-10-19 21:51:15 +01:00
Richard van der Hoff
f62c597d14
Merge pull request #4069 from matrix-org/rav/fix_email_templates_4065 
Calculate absolute path for email templates
 2018-10-19 16:33:54 +01:00
Erik Johnston
67f7b9cb50 pep8 2018-10-19 16:06:59 +01:00
Erik Johnston
056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00