Check that auto_vacuum is disabled when porting a SQLite database to Postgres, as VACUUMs must not be performed between runs of the script. (#13195)

This commit is contained in:
reivilibre 2022-07-07 11:08:04 +01:00 committed by GitHub
parent 57f6f59e3e
commit fb7d24ab6d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 43 additions and 0 deletions

1
changelog.d/13195.misc Normal file
View File

@ -0,0 +1 @@
Check that `auto_vacuum` is disabled when porting a SQLite database to Postgres, as `VACUUM`s must not be performed between runs of the script.

View File

@ -143,6 +143,14 @@ to do step 2.
It is safe to at any time kill the port script and restart it. It is safe to at any time kill the port script and restart it.
However, under no circumstances should the SQLite database be `VACUUM`ed between
multiple runs of the script. Doing so can lead to an inconsistent copy of your database
into Postgres.
To avoid accidental error, the script will check that SQLite's `auto_vacuum` mechanism
is disabled, but the script is not able to protect against a manual `VACUUM` operation
performed either by the administrator or by any automated task that the administrator
may have configured.
Note that the database may take up significantly more (25% - 100% more) Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres. space on disk after porting to Postgres.

View File

@ -621,6 +621,25 @@ class Porter:
self.postgres_store.db_pool.updates.has_completed_background_updates() self.postgres_store.db_pool.updates.has_completed_background_updates()
) )
@staticmethod
def _is_sqlite_autovacuum_enabled(txn: LoggingTransaction) -> bool:
"""
Returns true if auto_vacuum is enabled in SQLite.
https://www.sqlite.org/pragma.html#pragma_auto_vacuum
Vacuuming changes the rowids on rows in the database.
Auto-vacuuming is therefore dangerous when used in conjunction with this script.
Note that the auto_vacuum setting can't be changed without performing
a VACUUM after trying to change the pragma.
"""
txn.execute("PRAGMA auto_vacuum")
row = txn.fetchone()
assert row is not None, "`PRAGMA auto_vacuum` did not give a row."
(autovacuum_setting,) = row
# 0 means off. 1 means full. 2 means incremental.
return autovacuum_setting != 0
async def run(self) -> None: async def run(self) -> None:
"""Ports the SQLite database to a PostgreSQL database. """Ports the SQLite database to a PostgreSQL database.
@ -637,6 +656,21 @@ class Porter:
allow_outdated_version=True, allow_outdated_version=True,
) )
# For safety, ensure auto_vacuums are disabled.
if await self.sqlite_store.db_pool.runInteraction(
"is_sqlite_autovacuum_enabled", self._is_sqlite_autovacuum_enabled
):
end_error = (
"auto_vacuum is enabled in the SQLite database."
" (This is not the default configuration.)\n"
" This script relies on rowids being consistent and must not"
" be used if the database could be vacuumed between re-runs.\n"
" To disable auto_vacuum, you need to stop Synapse and run the following SQL:\n"
" PRAGMA auto_vacuum=off;\n"
" VACUUM;"
)
return
# Check if all background updates are done, abort if not. # Check if all background updates are done, abort if not.
updates_complete = ( updates_complete = (
await self.sqlite_store.db_pool.updates.has_completed_background_updates() await self.sqlite_store.db_pool.updates.has_completed_background_updates()