From 369b6d869a7b2d29979c8f72804f066e25353fef Mon Sep 17 00:00:00 2001 From: Fijxu Date: Sat, 18 Oct 2025 16:19:17 -0300 Subject: [PATCH 1/2] Add database backup guide --- docs/db-migration.md | 187 +++++++++++++++++++++++++++++++++++++++++++ docs/index.md | 1 + mkdocs.yml | 1 + 3 files changed, 189 insertions(+) create mode 100644 docs/db-migration.md diff --git a/docs/db-migration.md b/docs/db-migration.md new file mode 100644 index 0000000..ca55c04 --- /dev/null +++ b/docs/db-migration.md @@ -0,0 +1,187 @@ +# Database migration + +When Invidious gets a new feature, sometimes it comes with database changes that +need manual intervention. Invidious comes with a `--migrate` command that does +all the migrations for you, however, migrations can go wrong so it's recommended +to do a backup of your Invidious database before migrating your database. + +This guide is divided into 4 sections: + +- [Checking if you have free space to make a backup](#Checking-if-you-have-free-space-to-make-a-backup) +- [Doing a backup](#Doing-a-backup) +- [Migrating your database](#Migrating-your-database) +- [Recovering your database from a backup](#Recovering-your-database-from-a-backup): + This step is optional and you should read it if your database migration has + gone wrong. + +## Checking if you have free space to make a backup + +Doing a backup can take large portions of your storage if you have limited +resources (like running Invidious in a VPS), therefore you should check how many +free space you have in the server you are running Invidious on. + +Use one of the two methods listed bellow, this will depend of how you decided to +install Invidious from the [Installation instructions](./installation.md). + +### Docker + +If you are using docker, execute this command to check the size of your +database: + +```bash +$ docker system df -v | grep postgresdata +``` + +And you will get: + +``` +invidious_postgresdata 1 2.48GB +``` + +In this example, `2.48GB` is the size the database. + +If you system has less free space than the value that you got, you should free +some space before making a backup. + +### Manual Installation + +Assuming you already have PostresSQL installed from the +[Installation instructions](./installation.md), execute this command to check +the size of your database: + +```bash +$ sudo -i -u postgres +$ psql +# Write `SELECT pg_size_pretty(pg_database_size('invidious'));` into the psql interactive terminal +postgres=# SELECT pg_size_pretty(pg_database_size('invidious')); +``` + +And you will get: + +```postgres + pg_size_pretty +---------------- + 2 GB +``` + +In this example, `2 GB` is the size the database. + +If you system has less free space than the value that you got, you should free +some space before making a backup. + +## Doing a backup + +After you checked if you really have free space to make a backup, let's do a +backup! + +Use one of the two methods listed bellow, this will depend of how you decided to +install Invidious from the [Installation instructions](./installation.md). + +### Docker + +###### 1) Stop Invidious + +Go to the directory where the `docker-compose.yml` of Invidious is and use: + +```bash +$ docker compose down +``` + +If you are hosting a public instance, make sure to make your instance +inaccessible from the internet. + +###### 2) Clone the Docker Volume + +Since all the data of the database is inside a Docker Volume, you just need to +copy the volume into a new one using: + +```bash +$ sudo cp -ra /var/lib/docker/volumes/invidious_postgres /var/lib/docker/volumes/invidious_postgres_backup +``` + +###### 3) Restart Docker + +In order to make Docker able to detect the backup Docker Volume, you must +restart Docker using: + +```bash +$ sudo systemctl restart docker.service +``` + +###### 4) Check that Docker detects the volume + +Now check that Docker detects the backup volume using: + +``` +$ docker volume ls | grep invidious_postgresdata +``` + +If you get something like this: + +``` +local invidious_postgresdata +local invidious_postgresdata_backup # <- Your backup! +``` + +Then your backup volume is detected by Docker. + +###### 5) Checking your backup + +Now you should check if you backup really works before proceeding to the +migration. + +To do this, modify `docker-compose.yml` `volumes` declaration like this: + +``` +volumes: + postgresdata: + external: true + name: invidious_postgresdata_backup + companioncache: +``` + +And start Invidious using `docker compose up`, check some things like logging +into your account, your subscriptions and watch history, if everything loads +fine without any issues, your database is now successfully backed up and that +volume can be used to recover your database data if something goes wrong! + +### Manual Installation + +###### 1) Stop Invidious + +```bash +$ sudo systemctl stop invidious.service +``` + +If you are hosting a public instance, make sure to make your instance +inaccessible from the internet. + +###### 2) Dump the Invidious database + +```bash +$ sudo -i -u postgres +$ pg_dump invidious | gzip > invidious-db-backup.gz +``` + +And done, if your Invidious database is big, it may take a while to finish the +`pg_dump` process. + +## Migrating your Database + +### Docker + +###### 1) Start PostgreSQL + +Go to the directory where the `docker-compose.yml` of Invidious is and use: + +```bash +docker compose up -d invidious-db +``` + +###### 2) Migrate database + +```bash +docker compose run invidious sh -c "./invidious --migrate" +``` + +## Recovering your database from a backup diff --git a/docs/index.md b/docs/index.md index aef9be9..4343453 100644 --- a/docs/index.md +++ b/docs/index.md @@ -26,6 +26,7 @@ - [Caddy reverse proxy setup](./caddy.md) - [Apache2 reverse proxy setup](./apache2.md) - [Database maintenance](./db-maintenance.md) +- [Database migration](./db-migration.md) - [CAPTCHA bug on Debian and Ubuntu](./captcha-bug.md) - [Registering users manually](./register-user.md) - [Reset user password](./reset-password.md) diff --git a/mkdocs.yml b/mkdocs.yml index d75fd24..0c2322e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ nav: - 'caddy.md' - 'apache2.md' - 'db-maintenance.md' + - 'db-migration.md' - 'captcha-bug.md' - 'register-user.md' - 'reset-password.md' From e883c4206229781c58775e3a6141c1375aaaebe1 Mon Sep 17 00:00:00 2001 From: Fijxu Date: Sat, 13 Dec 2025 00:38:45 -0300 Subject: [PATCH 2/2] draft --- docs/db-migration-backup-docker.md | 134 ++++++++++++++++++++ docs/db-migration-backup-manual.md | 121 ++++++++++++++++++ docs/db-migration-info.md | 18 +++ docs/db-migration-restore-docker.md | 17 +++ docs/db-migration-restore-manual.md | 1 + docs/db-migration.md | 187 ---------------------------- docs/index.md | 2 +- mkdocs.yml | 7 +- 8 files changed, 298 insertions(+), 189 deletions(-) create mode 100644 docs/db-migration-backup-docker.md create mode 100644 docs/db-migration-backup-manual.md create mode 100644 docs/db-migration-info.md create mode 100644 docs/db-migration-restore-docker.md create mode 100644 docs/db-migration-restore-manual.md delete mode 100644 docs/db-migration.md diff --git a/docs/db-migration-backup-docker.md b/docs/db-migration-backup-docker.md new file mode 100644 index 0000000..94fc510 --- /dev/null +++ b/docs/db-migration-backup-docker.md @@ -0,0 +1,134 @@ +# Database backup and migration for Docker Compose installations + +This guide is divided into 4 sections: + +- [Checking if you have free space to make a backup](#checking-if-you-have-free-space-to-make-a-backup) +- [Deleting cached videos to reduce the size of the backup (recommended for public instances)](#deleting-cached-videos-to-reduce-the-size-of-the-backup-recommended-for-public-instances) +- [Doing a backup](#doing-a-backup) +- [Migrating your database](#migrating-your-database) + +## Checking if you have free space to make a backup + +Doing a backup can take large portions of your storage if you have limited +resources (like running Invidious in a VPS), therefore you should check how many +free space you have in the server you are running Invidious on. + +Follow this steps to check the size of your database: + +##### 1) Stop Invidious + +```bash +$ docker compose down +``` + +##### 2) Start the database + +```bash +$ docker compose up invidious-db -d +``` + +##### 3) Check the size of the database + +```bash +$ docker compose exec -i invidious-db psql -U kemal -d invidious +# Write `SELECT pg_size_pretty(pg_database_size('invidious'));` into the psql interactive terminal +postgres=# SELECT pg_size_pretty(pg_database_size('invidious')); +# Exit the psql interactive terminal with the Ctrl+D keybind. +``` + +And you will get: + +```postgres + pg_size_pretty +---------------- + 2 GB +``` + +In this example, `2 GB` is the size the database. + +If you system has less free space than the value that you got, you should free +some space before making a backup. + +## Deleting cached videos to reduce the size of the backup (recommended for public instances) + +**You can skip this step if you have enough free space**. + +Invidious caches video information so it can be later reused to display video +information, this is stored in the `videos` table and all the rows can be safely +deleted to free up some space when doing the backup. + +###### 1) Stop Invidious + +Go to the directory where the `docker-compose.yml` of Invidious is and use: + +```bash +$ docker compose down +``` + +###### 2) Start the database + +```bash +$ docker compose invidious-db -d +``` + +###### 3) Delete the video cache + +```bash +$ docker compose exec -i invidious-db psql -U kemal -d invidious +# Write `TRUNCATE TABLE videos;` into the psql interactive terminal +$ invidious=# TRUNCATE TABLE videos; +# Exit the psql interactive terminal with the Ctrl+D keybind. +``` + +## Doing a backup + +After you checked if you really have free space to make a backup, let's do a +backup! + +###### 1) Stop Invidious + +Go to the directory where the `docker-compose.yml` of Invidious is and use: + +```bash +$ docker compose down +``` + +##### 2) Start the database + +```bash +$ docker compose up invidious-db -d +``` + +###### 3) Backup the database + +```bash +$ docker compose exec -i invidious-db pg_dump -U kemal invidious > dump.sql +``` + +And done, if your Invidious database is big, it may take a while to finish the +`pg_dump` process. + +Your backup will be saved in the current directory where the command was +executed as `dump.sql` which you can later delete if everything went fine when +migrating your database. + +## Migrating your Database + +###### 1) Start the database + +```bash +docker compose up -d invidious-db +``` + +###### 2) Migrate database + +```bash +docker compose run invidious sh -c "./invidious --migrate" +``` + +--- + +And done, your Invidious instance should now be migrated to the latest database +version. If anything goes wrong, you can always restore your data reading the +[Database restore for Docker Compose installations](./db-migration-restore-docker.md) +guide. diff --git a/docs/db-migration-backup-manual.md b/docs/db-migration-backup-manual.md new file mode 100644 index 0000000..f5c7bc5 --- /dev/null +++ b/docs/db-migration-backup-manual.md @@ -0,0 +1,121 @@ +# Database backup and migration for manual installations + +This guide is divided into 4 sections: + +- [Checking if you have free space to make a backup](#checking-if-you-have-free-space-to-make-a-backup) +- [Deleting cached videos to reduce the size of the backup (recommended for public instances)](#deleting-cached-videos-to-reduce-the-size-of-the-backup-recommended-for-public-instances) +- [Doing a backup](#doing-a-backup) +- [Migrating your database](#migrating-your-database) + +## Checking if you have free space to make a backup + +Doing a backup can take large portions of your storage if you have limited +resources (like running Invidious in a VPS), therefore you should check how many +free space you have in the server you are running Invidious on. + +Follow this steps to check the size of your database: + +##### 1) Stop Invidious + +```bash +$ docker compose down +``` + +##### 2) Check the size of the database + +```bash +$ sudo -i -u postgres +$ psql +# Write `SELECT pg_size_pretty(pg_database_size('invidious'));` +# into the psql interactive terminal +postgres=# SELECT pg_size_pretty(pg_database_size('invidious')); +``` + +And you will get: + +```postgres + pg_size_pretty +---------------- + 2 GB +``` + +In this example, `2 GB` is the size the database. + +If you system has less free space than the value that you got, you should free +some space before making a backup. + +## Deleting cached videos to reduce the size of the backup (recommended for public instances) + +**You can skip this step if you have enough free space**. + +Invidious caches video information so it can be later reused to display video +information, this is stored in the `videos` table and all the rows can be safely +deleted to free up some space when doing the backup. + +###### 1) Stop invidious + +```bash +$ sudo systemctl stop invidious.service +``` + +###### 2) Delete the video cache + +```bash +$ sudo -i -u postgres +$ psql -d invidious +# Write `TRUNCATE TABLE videos;` +# into the psql interactive terminal +invidious=# TRUNCATE TABLE videos; +# Exit the psql interactive terminal with the Ctrl+D keybind. +``` + +## Doing a backup + +After you checked if you really have free space to make a backup, let's do a +backup! + +###### 1) Stop Invidious + +```bash +$ sudo systemctl stop invidious.service +``` + +###### 2) Dump the Invidious database + +```bash +$ sudo -i -u postgres +$ pg_dump invidious > dump.sql +``` + +And done, if your Invidious database is big, it may take a while to finish the +`pg_dump` process. + +Your backup will be saved in the current directory where the command was +executed as `dump.sql` which you can later delete if everything went fine when +migrating your database. + +## Migrating your Database + +### Manual installation + +Go to the directory where your Invidious installation is and follow this steps: + +###### 1) Go to the Invidious installation folder + +```bash +$ sudo su - invidious +$ cd invidious +``` + +###### 2) Migrate database + +```bash +$ ./invidious --migrate +``` + +--- + +And done, your Invidious instance should now be migrated to the latest database +version. If anything goes wrong, you can always restore your data reading the +[Database restore for Docker Compose installations](./db-migration-restore-docker.md) +guide. diff --git a/docs/db-migration-info.md b/docs/db-migration-info.md new file mode 100644 index 0000000..5499f45 --- /dev/null +++ b/docs/db-migration-info.md @@ -0,0 +1,18 @@ +# Database migration information + +When Invidious gets a new feature, sometimes it comes with database changes that +need manual intervention. Invidious comes with a `--migrate` command that does +all the migrations for you, however, migrations can go wrong. + +Because of this, we recommend you to do a backup of your Invidious database before running any migrations. + +We provide backup and migration instructions for: + +- [Docker Compose](./db-migration-backup-docker.md): For people that installed their Invidious instance with the [Installation instructions for Docker Compose (Production)](./installation.md#docker-compose-method-production) +- [Manual installation](./db-migration-backup-manual.md): For people that installed their Invidious instance with the [Installation instructions for a manual installation](./installation.md#manual-installation) + +And we also provide restore instructions for the same installations methods listed above: + +- [Docker Compose](./db-migration-restore-docker.md) +- [Manual installation](./db-migration-restore-manual.md) + diff --git a/docs/db-migration-restore-docker.md b/docs/db-migration-restore-docker.md new file mode 100644 index 0000000..2f6a913 --- /dev/null +++ b/docs/db-migration-restore-docker.md @@ -0,0 +1,17 @@ +# Database restore for Docker Compose installations + +In case a database migration went wrong, you can use a backup of your Invidious database to recover all the data or to simply revert it to a previous state. + +In order to recover your Invidious database from a backup follow the next steps: + +##### 1) Stop Invidious + +```bash +$ docker compose down +``` + +##### 2) Start the database + +```bash +$ docker compose up invidious-db -d +``` \ No newline at end of file diff --git a/docs/db-migration-restore-manual.md b/docs/db-migration-restore-manual.md new file mode 100644 index 0000000..0c321cc --- /dev/null +++ b/docs/db-migration-restore-manual.md @@ -0,0 +1 @@ +# Database restore for manual installations \ No newline at end of file diff --git a/docs/db-migration.md b/docs/db-migration.md deleted file mode 100644 index ca55c04..0000000 --- a/docs/db-migration.md +++ /dev/null @@ -1,187 +0,0 @@ -# Database migration - -When Invidious gets a new feature, sometimes it comes with database changes that -need manual intervention. Invidious comes with a `--migrate` command that does -all the migrations for you, however, migrations can go wrong so it's recommended -to do a backup of your Invidious database before migrating your database. - -This guide is divided into 4 sections: - -- [Checking if you have free space to make a backup](#Checking-if-you-have-free-space-to-make-a-backup) -- [Doing a backup](#Doing-a-backup) -- [Migrating your database](#Migrating-your-database) -- [Recovering your database from a backup](#Recovering-your-database-from-a-backup): - This step is optional and you should read it if your database migration has - gone wrong. - -## Checking if you have free space to make a backup - -Doing a backup can take large portions of your storage if you have limited -resources (like running Invidious in a VPS), therefore you should check how many -free space you have in the server you are running Invidious on. - -Use one of the two methods listed bellow, this will depend of how you decided to -install Invidious from the [Installation instructions](./installation.md). - -### Docker - -If you are using docker, execute this command to check the size of your -database: - -```bash -$ docker system df -v | grep postgresdata -``` - -And you will get: - -``` -invidious_postgresdata 1 2.48GB -``` - -In this example, `2.48GB` is the size the database. - -If you system has less free space than the value that you got, you should free -some space before making a backup. - -### Manual Installation - -Assuming you already have PostresSQL installed from the -[Installation instructions](./installation.md), execute this command to check -the size of your database: - -```bash -$ sudo -i -u postgres -$ psql -# Write `SELECT pg_size_pretty(pg_database_size('invidious'));` into the psql interactive terminal -postgres=# SELECT pg_size_pretty(pg_database_size('invidious')); -``` - -And you will get: - -```postgres - pg_size_pretty ----------------- - 2 GB -``` - -In this example, `2 GB` is the size the database. - -If you system has less free space than the value that you got, you should free -some space before making a backup. - -## Doing a backup - -After you checked if you really have free space to make a backup, let's do a -backup! - -Use one of the two methods listed bellow, this will depend of how you decided to -install Invidious from the [Installation instructions](./installation.md). - -### Docker - -###### 1) Stop Invidious - -Go to the directory where the `docker-compose.yml` of Invidious is and use: - -```bash -$ docker compose down -``` - -If you are hosting a public instance, make sure to make your instance -inaccessible from the internet. - -###### 2) Clone the Docker Volume - -Since all the data of the database is inside a Docker Volume, you just need to -copy the volume into a new one using: - -```bash -$ sudo cp -ra /var/lib/docker/volumes/invidious_postgres /var/lib/docker/volumes/invidious_postgres_backup -``` - -###### 3) Restart Docker - -In order to make Docker able to detect the backup Docker Volume, you must -restart Docker using: - -```bash -$ sudo systemctl restart docker.service -``` - -###### 4) Check that Docker detects the volume - -Now check that Docker detects the backup volume using: - -``` -$ docker volume ls | grep invidious_postgresdata -``` - -If you get something like this: - -``` -local invidious_postgresdata -local invidious_postgresdata_backup # <- Your backup! -``` - -Then your backup volume is detected by Docker. - -###### 5) Checking your backup - -Now you should check if you backup really works before proceeding to the -migration. - -To do this, modify `docker-compose.yml` `volumes` declaration like this: - -``` -volumes: - postgresdata: - external: true - name: invidious_postgresdata_backup - companioncache: -``` - -And start Invidious using `docker compose up`, check some things like logging -into your account, your subscriptions and watch history, if everything loads -fine without any issues, your database is now successfully backed up and that -volume can be used to recover your database data if something goes wrong! - -### Manual Installation - -###### 1) Stop Invidious - -```bash -$ sudo systemctl stop invidious.service -``` - -If you are hosting a public instance, make sure to make your instance -inaccessible from the internet. - -###### 2) Dump the Invidious database - -```bash -$ sudo -i -u postgres -$ pg_dump invidious | gzip > invidious-db-backup.gz -``` - -And done, if your Invidious database is big, it may take a while to finish the -`pg_dump` process. - -## Migrating your Database - -### Docker - -###### 1) Start PostgreSQL - -Go to the directory where the `docker-compose.yml` of Invidious is and use: - -```bash -docker compose up -d invidious-db -``` - -###### 2) Migrate database - -```bash -docker compose run invidious sh -c "./invidious --migrate" -``` - -## Recovering your database from a backup diff --git a/docs/index.md b/docs/index.md index 4343453..c4a8a69 100644 --- a/docs/index.md +++ b/docs/index.md @@ -26,7 +26,7 @@ - [Caddy reverse proxy setup](./caddy.md) - [Apache2 reverse proxy setup](./apache2.md) - [Database maintenance](./db-maintenance.md) -- [Database migration](./db-migration.md) +- [Database migration](./db-migration-info.md) - [CAPTCHA bug on Debian and Ubuntu](./captcha-bug.md) - [Registering users manually](./register-user.md) - [Reset user password](./reset-password.md) diff --git a/mkdocs.yml b/mkdocs.yml index 0c2322e..889a246 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,7 +30,12 @@ nav: - 'caddy.md' - 'apache2.md' - 'db-maintenance.md' - - 'db-migration.md' + - 'Database Migration': + - 'db-migration-info.md' + - 'db-migration-backup-docker.md' + - 'db-migration-backup-manual.md' + - 'db-migration-restore-docker.md' + - 'db-migration-restore-manual.md' - 'captcha-bug.md' - 'register-user.md' - 'reset-password.md'