Commit Graph

10865 Commits

Author SHA1 Message Date
Brendan Abolivier
14504ad573
Add CI for synapse_port_db (#6140)
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Richard van der Hoff
bcfc647e4d Synapse 1.5.0rc2 (2019-10-28)
=============================
 
 Bugfixes
 --------
 
 - Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
 - Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
 - Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))
 
 Internal Changes
 ----------------
 
 - Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
 - Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl225VkACgkQOSor00I9
 eP/7wwf6AkQE1dGYDPpmS29XYocfD3EbjT8FojaHTz1gbkIt5rEF88QHzwRo4P/x
 1S1kQZpEKxKrwmLMDmSbw9STK/76LKGk9k8VycVHfTdy8SICCiL50pVPpSTJ4nXV
 905E7nY0zau3wunWJD4gEjqll967W0AtNKwCHO0D6vNP4pcQNhTkSjhnmypXCFGo
 bvV9r0wq8mtq/FS1GX0Y2CEwbbV1MoPrvlXW37t/gBPk3P1QB/vPjakxO8LgeuZf
 JzAamP5jOYZJpPCpZxwDm6ZNEQ7T4jORdA623wp1gh1nTG88cObCLXJe2jCnpjnJ
 QVb1YlJiwcehXkclIoywgIzj13iIsw==
 =7ctT
 -----END PGP SIGNATURE-----

Merge tag 'v1.5.0rc2' into develop

Synapse 1.5.0rc2 (2019-10-28)
=============================

Bugfixes
--------

- Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
- Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
- Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))

Internal Changes
----------------

- Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
- Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
2019-10-28 12:59:13 +00:00
Richard van der Hoff
c482d45822 1.5.0rc2 2019-10-28 12:48:18 +00:00
Richard van der Hoff
172f264ed3
Improve signature checking on some federation APIs (#6262)
Make sure that we check that events sent over /send_join, /send_leave, and
/invite, are correctly signed and come from the expected servers.
2019-10-28 12:43:23 +00:00
Richard van der Hoff
9aee28927b Convert EventContext to attrs (#6218)
* make EventContext use an attr
2019-10-28 14:29:55 +02:00
Hubert Chathi
da78f61778
Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys
delete keys when deleting backup versions
2019-10-25 11:28:11 -04:00
Erik Johnston
64c2cfda8a Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:34:49 +01:00
Erik Johnston
87259b3a3a
Merge pull request #6255 from matrix-org/erikj/stats_config
Remove repeated calls to config.stats_enabled.
2019-10-25 11:34:38 +01:00
Erik Johnston
a71b8c87ec Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:32:24 +01:00
Erik Johnston
a411f2b177 Fix /keys/query API on workers.
The necessary getters were added only to the master store and not the
worker stores.
2019-10-25 11:08:03 +01:00
Erik Johnston
f666d6f5d7 Remove repeated calls to config.stats_enabled.
Turns out that fetching variables from the config object is expensive,
so doing it once at startup avoids unnecessary work.
2019-10-25 10:28:36 +01:00
Erik Johnston
44ab048cfe
Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_logging
Reduce debug logging overhead
2019-10-25 10:05:44 +01:00
Hubert Chathi
848cd388d9 delete keys when deleting backups 2019-10-24 21:21:51 -04:00
Michael Kaye
e4d98188da Address codestyle concerns 2019-10-24 18:43:13 +01:00
Michael Kaye
8f4a808d9d Delay printf until logging is required.
Using % will cause the string to be generated even if debugging
is off.
2019-10-24 18:31:53 +01:00
Michael Kaye
9eebc1e73b use %r to __repr__ objects
This avoids calculating __repr__ unless we are going to log.
2019-10-24 18:18:56 +01:00
Michael Kaye
f85b9842f0 Don't encode object as UTF-8 string if not needed.
I believe that string formatting ~10-15 sized events will
take a proportion of CPU time.
2019-10-24 18:08:45 +01:00
Michael Kaye
39266a9c9f Make user/room stats log line less verbose. 2019-10-24 17:55:53 +01:00
Erik Johnston
3aa2a90556 Move schema delta files to the correct data store.
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
Andrew Morgan
8831b04a53 1.5.0rc1 2019-10-24 12:47:58 +01:00
Neil Johnson
2794b79052 Option to suppress resource exceeded alerting (#6173)
The expected use case is to suppress MAU limiting on small instances
2019-10-24 11:48:46 +01:00
Andrew Morgan
92e88a71d3
Cleanup extra quotes from IDEs (#6236) 2019-10-23 16:49:05 +01:00
Andrew Morgan
409c62b27b
Add config linting script that checks for bool casing (#6203)
Add a linting script that enforces all boolean values in the default config be lowercase.

This has annoyed me for a while so I decided to fix it.
2019-10-23 13:22:54 +01:00
Erik Johnston
c17efdc01c Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 18:44:29 +01:00
Erik Johnston
23d62eded2 Clean up prepare_database.py a bit and add comments 2019-10-22 18:43:31 +01:00
Erik Johnston
6cc497f99b Delete background_update table creation in main data_store 2019-10-22 18:02:50 +01:00
Erik Johnston
1bbc5444a8 Move README into synapse/storage/__init__.py 2019-10-22 17:59:31 +01:00
Hubert Chathi
2761731634
Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2
Cross-signing [3/4] -- uploading signatures edition
2019-10-22 17:05:06 +01:00
Erik Johnston
acf47c7698 Add a basic README to synapse.storage 2019-10-22 11:55:46 +01:00
Erik Johnston
bb6264be0b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 10:41:18 +01:00
Erik Johnston
ffd24545bb Fix schema management to work with multiple data stores. 2019-10-21 16:08:40 +01:00
Erik Johnston
c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Richard van der Hoff
93eaeec75a
Remove Auth.check method (#6217)
This method was somewhat redundant, and confusing.
2019-10-18 19:43:36 +02:00
Hubert Chathi
36adfaedab Merge branch 'develop' into cross-signing_sig_upload 2019-10-18 18:34:42 +01:00
Hubert Chathi
125eb45e19 fix doc strings 2019-10-18 16:56:16 +01:00
Hubert Chathi
0a9d22808c Merge branch 'uhoreg/e2e_cross-signing_merged' into develop 2019-10-18 15:57:40 +01:00
Richard van der Hoff
2b22faded7 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 12:35:13 +01:00
Richard van der Hoff
560c122267
Fix logging config for the docker image (#6197)
Turns out that loggers that are instantiated before the config is loaded get
turned off.

Also bring the logging config that is generated by --generate-config into line.

Fixes #6194.
2019-10-18 13:34:33 +02:00
Hubert Chathi
cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Brendan Abolivier
1ee97cbd01
Merge branch 'master' into develop 2019-10-18 11:00:43 +01:00
Hubert Chathi
f0f6a2b360 use the right function for when we're already in runInteraction 2019-10-18 10:56:54 +01:00
Erik Johnston
d98029ea89
Merge pull request #6196 from matrix-org/erikj/await
Move rest/admin to use async/await.
2019-10-18 11:53:02 +02:00
Brendan Abolivier
41b9faed16 1.4.1 2019-10-18 10:15:12 +01:00
Richard van der Hoff
80003dfcd5 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 10:11:40 +01:00
Erik Johnston
5859a5c569 Fix presence timeouts when synchrotron restarts. (#6212)
* Fix presence timeouts when synchrotron restarts.

Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.

Fixes #3715.
2019-10-18 06:42:26 +01:00
Richard van der Hoff
1ba359a11f rip out some unreachable code
The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
2019-10-17 21:45:59 +01:00
Richard van der Hoff
70d9759cdf Synapse 1.4.1rc1 (2019-10-17)
=============================
 
 Bugfixes
 --------
 
 - Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](5b0e9948ea))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl2oj9sACgkQOSor00I9
 eP+ZQQf9GQbjQhq92G1Mt1dI3HxXQnIvelGWaLC9rB++eGcafBkP0EJgCbQZTKda
 WntghJib0oWVHVtPbwYKtwF3z365WbME3awO6RpRoMvrFL5PDqJ8Ma5LUgJ6/GuO
 yDBfABbZk7PVS5WibftXXqibK8MkSrnId57VN3tc/SUHj3DGe4vk+A7plwjNckBj
 2SLUA8DPw5I3Y4Z6peb6VweEA8M59+i3/0cZFAJ+4Ofy2Jad3z8kVXmJ1gTjBOdG
 1bjBfnS2SrZLZwD79zJzaUR0zxfIvj62mL/fjrtR66oRmKF2T+9UnM7+9gH9JHWZ
 ct3Mk4Nnr4VaBBWOWOcD2btvByckug==
 =9lJk
 -----END PGP SIGNATURE-----

Merge tag 'v1.4.1rc1' into develop

Synapse 1.4.1rc1 (2019-10-17)
=============================

Bugfixes
--------

- Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](5b0e9948ea))
2019-10-17 17:03:02 +01:00
Richard van der Hoff
71cd3fed66 1.4.1rc1 2019-10-17 16:40:56 +01:00
Erik Johnston
5b0e9948ea Do the update as a background index 2019-10-17 16:31:42 +01:00
Erik Johnston
a2bb50c2eb Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
Fix inserting bytes as text in `censor_redactions`
2019-10-17 16:29:07 +01:00