Commit Graph

73 Commits

Author SHA1 Message Date
David Robertson
586bfc6dc0
Use dummy fallback engines if imports fail () 2022-06-07 17:33:55 +01:00
David Robertson
1fe202a1a3
Tidy up and type-hint the database engine modules ()
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-05-17 00:34:38 +01:00
Shay
e78d4f61fc
Refuse to start if DB has an unsafe locale () 2022-03-23 10:23:05 -07:00
David Robertson
f4c5e5864c
Use psycopg2 type stubs () 2022-03-23 14:03:24 +00:00
Nick Barrett
b59d285f7c
Db txn set isolation level ()
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-25 15:14:46 +01:00
Shay
9006ee36d1
Drop support for and remove references to EOL Python 3.6 ()
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
Erik Johnston
329ef5c715
Fix the inbound PDU metric ()
This broke in 
2021-06-30 12:07:16 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Richard van der Hoff
3ada9b4264 Drop support for sqlite<3.22 as well 2021-04-08 16:42:32 +01:00
Richard van der Hoff
abade34633 Require py36 and Postgres 9.6 2021-04-08 16:42:32 +01:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ()
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Jonathan de Jong
2814028ce5
Add experimental support for PyPy. ()
* Adds proper dependencies.
* Minor fixes in database layer.
2021-02-04 08:29:47 -05:00
Erik Johnston
ae5b2a72c0
Reduce serialization errors in MultiWriterIdGen ()
We call `_update_stream_positions_table_txn` a lot, which is an UPSERT
that can conflict in `REPEATABLE READ` isolation level. Instead of doing
a transaction consisting of a single query we may as well run it outside
of a transaction.
2020-10-07 15:15:57 +01:00
Richard van der Hoff
3c36ae17a5 Use SequenceGenerator for state group ID allocation 2020-07-16 11:25:08 +01:00
Richard van der Hoff
244dbb04f7
Fix incorrect error message when database CTYPE was set incorrectly. () 2020-07-01 13:56:16 +01:00
Patrick Cloke
627b0f5f27
Persist user interactive authentication sessions ()
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
2020-04-30 13:47:49 -04:00
David Vo
fbf0782c63
Only import sqlite3 when type checking ()
Fixes: 
Signed-off-by: David Vo <david@vovo.id.au>
2020-03-27 13:20:00 +00:00
Richard van der Hoff
132b673dbe
Add some type annotations in synapse.storage ()
I cracked, and added some type definitions in synapse.storage.
2020-02-27 11:53:40 +00:00
Uday Bansal
7728d87fd7
Updated warning for incorrect database collation/ctype ()
Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
2020-02-26 15:17:03 +00:00
Erik Johnston
02b44db922
Warn if postgres database has non-C locale. ()
As using non-C locale can cause issues on upgrading OS.
2020-01-28 13:44:21 +00:00
Richard van der Hoff
bf46821180 Refuse to start if sqlite is older than 3.11.0 2020-01-09 18:11:04 +00:00
Richard van der Hoff
e48ba84e0b Check postgres version in check_database
this saves doing it on each connection, and will allow us to pass extra options
in.
2020-01-09 18:05:59 +00:00
Richard van der Hoff
e97d1cf001 Modify check_database to take a connection rather than a cursor
We might not need the cursor at all.
2020-01-09 18:05:50 +00:00
Erik Johnston
2284eb3a53
Add database config class ()
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Erik Johnston
83d86106a8
Merge pull request from matrix-org/erikj/postgres_any
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
Erik Johnston
afb6d9d53b Fix SQLite 2019-10-10 15:55:41 +01:00
Erik Johnston
3bc687508f Remove add_in_list_sql_clause 2019-10-10 15:35:46 +01:00
Erik Johnston
1d3858371e Disable bytes usage with postgres
More often than not passing bytes to `txn.execute` is a bug (where we
meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is
set to `ESCAPE`. However, this is a bit of a footgun so we want to
instead error when this happens, and force using `bytearray` if we
actually want to use bytes.
2019-10-08 16:28:57 +01:00
Erik Johnston
9267741a5f Fix devices_last_seen background update.
Fixes .
2019-09-30 11:58:36 +01:00
Amber Brown
32e7c9e7f2
Run Black. () 2019-06-20 19:32:02 +10:00
Amber Brown
eba7caf09f
Remove Postgres 9.4 support () 2019-06-18 00:59:00 +10:00
Amber Brown
7efd1d87c2 Run black on the rest of the storage module () 2019-04-03 10:07:29 +01:00
Richard van der Hoff
f191be822b
Add database version to phonehome stats. () 2019-02-27 10:21:49 +00:00
Amber Brown
7072fe3084
Fix UPSERTs on SQLite 3.24+ () 2019-01-28 15:43:32 +00:00
Erik Johnston
431e485914 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert 2019-01-25 14:11:17 +00:00
Andrew Morgan
0b3fd1401f Don't require sqlite3 when using postgres () 2019-01-25 22:25:02 +11:00
Erik Johnston
1953067136 Disable native upserts for sqlite, as they don't work 2019-01-25 10:46:49 +00:00
Amber Brown
58f6c48183
Use native UPSERTs where possible () 2019-01-24 21:31:54 +11:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 () 2018-08-31 00:19:58 +10:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
d1e56cfcd1 Fix pep8 error on psycopg2cffi hack 2018-04-10 11:35:29 +02:00
Vincent Breitmoser
89de934981 Use psycopg2cffi module instead of psycopg2 if running on pypy
The psycopg2 package isn't available for PyPy.  This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ()
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Mark Haines
d5fb561709 Optionally make committing to postgres asynchronous.
Useful when running tests when you don't care whether the server
will lose data that it claims that it has committed.
2016-06-20 17:53:38 +01:00
Erik Johnston
8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Daniel Wagner-Hall
763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Daniel Wagner-Hall
d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Erik Johnston
0c36098c1f Implement rank function for SQLite FTS 2015-10-23 13:23:48 +01:00
Erik Johnston
17c80c8a3d rename schema_prepare to prepare_database 2015-10-13 13:56:22 +01:00